mirror of
https://github.com/jmcorgan/fips.git
synced 2026-07-30 19:46:15 +00:00
Johnathan Corgan
2bc0345174
cost-avoidance and mixed-technology decide a parent by measured link cost (etx * (1 + srtt_ms/100)). Under the local CI's 4-way-parallel chaos the fast fiber link's measured srtt spikes from host scheduling and can exceed the Bluetooth link's, flipping the choice and reddening a run that proves nothing about the daemon. The old margin was ~0.4 cost units, which a ~24ms one-sided scheduling blip closes. Widen the Bluetooth delay to 150-250ms so the margin dwarfs any plausible one-sided spike. The link still establishes well within the handshake budget (a ~500ms RTT against a 30s stale-handshake window), so the losing candidate is still a real, established peer rather than an absent one. Add a netem mutation exclude_edges option and use it in mixed-technology to pin the two links n08 chooses between, so a random degradation can't flip the asserted comparison either. An unknown excluded edge is a hard error, since a silent no-op would reintroduce the flakiness it exists to remove. Validated by running both scenarios repeatedly under heavier-than-CI concurrent load: the parent choice holds every time.
187 lines
7.0 KiB
YAML
187 lines
7.0 KiB
YAML
# Mixed Technology: 10-node heterogeneous network
|
|
#
|
|
# Explicit topology with Bluetooth (L2CAP), WiFi, and fiber links.
|
|
# Tests that cost-based parent selection produces a tree favoring
|
|
# low-cost paths when multiple link technologies coexist.
|
|
#
|
|
# Topology:
|
|
#
|
|
# n01 (root)
|
|
# / | \
|
|
# f f f
|
|
# / | \
|
|
# n02 n03 n04
|
|
# | \ | \ | \
|
|
# f BT f f BT f
|
|
# | \ | | \ |
|
|
# n05 n06 n07 n08 n09
|
|
# \ /
|
|
# wifi---wifi
|
|
# n10
|
|
#
|
|
# Edges and link types:
|
|
# Fiber: n01-n02, n01-n03, n01-n04, n02-n05, n03-n07, n04-n09
|
|
# Bluetooth: n02-n06, n04-n08
|
|
# WiFi: n03-n06, n08-n10
|
|
# Fiber: n03-n08, n06-n10
|
|
#
|
|
# Test subjects:
|
|
# - n08 has fiber (n03) and Bluetooth (n04) parents — should pick n03.
|
|
# ASSERTED below. Holds in 4 of 4 runs under the pinned root.
|
|
# - n06 was documented as having "fiber (n03) and Bluetooth (n02) parents".
|
|
# CORRECTED 2026-07-23: that is wrong, and the netem block below is the
|
|
# evidence rather than the ASCII art above. `n03-n06` is a **WiFi** link
|
|
# (5-20ms, 1-3% loss) and `n02-n06` is Bluetooth (15-40ms, 2-8% loss), so
|
|
# n06 chooses between two impaired links, not between fiber and Bluetooth.
|
|
# n06's only fiber-grade link is `n06-n10`, and n10 sits at depth 3 so it
|
|
# is never a good parent. There is therefore no reason to expect n06 to
|
|
# prefer n03, and the mutation block (fraction 0.2, degraded 50-100ms)
|
|
# moves the two costs run to run: across four runs n06 took n03 twice and
|
|
# n02 twice. NOT asserted, and the observed split is the correct outcome
|
|
# rather than a defect — in the run checked, n03 measured a link cost of 3
|
|
# against n02's 1, so effective depth 1+3 beat by 1+1 and the daemon chose
|
|
# the cheaper path exactly as it should.
|
|
#
|
|
# Netem mutation shifts fiber-only links between normal and degraded.
|
|
#
|
|
# n08 IS now asserted (below); n06 is not, and both reasons trace to the root,
|
|
# which was found and fixed rather than papered over here.
|
|
#
|
|
# The diagram above puts n01 at the top, and both test subjects only make
|
|
# sense in a tree rooted there. The mesh does not root by topology: it roots
|
|
# at the numerically smallest NodeAddr (src/tree/state.rs:363-390), which is a
|
|
# hash of the node's key and bears no relation to the numbering. Until
|
|
# 2026-07-23 this scenario rooted at **n09** in every run, which put both
|
|
# criteria out of reach. Under that root n08's two candidates were not at
|
|
# equal depth, so its choice was settled by depth long before link technology
|
|
# could matter, and n08 taking n04 was CORRECT behaviour that these comments
|
|
# made look like a defect.
|
|
#
|
|
# What the archive shows, for anyone re-deriving this later: across the five
|
|
# runs under the old root, n06's parent is n10 in three and n03 in two, and
|
|
# n08's is n04 in four and n03 in one. Those numbers describe a tree rooted at
|
|
# n09 and must not be used to calibrate anything now.
|
|
#
|
|
# `topology.pin_root` (default true) now assigns identities in NodeAddr order,
|
|
# so n01 holds the smallest and the diagram describes the tree that forms.
|
|
# Under that root n08 takes fiber n03 as its subject assertion (below) now
|
|
# checks. n06 stays unasserted: its "fiber n03" was a WiFi link (see the
|
|
# corrected note above), so it was never a founded criterion. Tracked in
|
|
# ISSUE-2026-0085.
|
|
#
|
|
# The n08 assertion is made reliable, not merely reachable: the Bluetooth
|
|
# contrast is widened (netem below) and the two links n08 chooses between are
|
|
# excluded from the netem mutation, so neither host load nor a random
|
|
# degradation can flip the asserted comparison.
|
|
|
|
scenario:
|
|
name: "mixed-technology"
|
|
seed: 42
|
|
duration_secs: 90
|
|
|
|
topology:
|
|
algorithm: explicit
|
|
num_nodes: 10
|
|
params:
|
|
adjacency:
|
|
- [n01, n02]
|
|
- [n01, n03]
|
|
- [n01, n04]
|
|
- [n02, n05]
|
|
- [n02, n06]
|
|
- [n03, n06]
|
|
- [n03, n07]
|
|
- [n03, n08]
|
|
- [n04, n08]
|
|
- [n04, n09]
|
|
- [n06, n10]
|
|
- [n08, n10]
|
|
subnet: "172.20.0.0/24"
|
|
ip_start: 10
|
|
|
|
netem:
|
|
enabled: true
|
|
default_policy:
|
|
# Fiber-like defaults
|
|
delay_ms: [1, 5]
|
|
jitter_ms: [0, 1]
|
|
loss_pct: [0, 0.5]
|
|
link_policies:
|
|
# Bluetooth (L2CAP) links. Delay widened to 150-250ms so the
|
|
# fiber-vs-Bluetooth cost margin on n08's choice (fiber n03 vs Bluetooth
|
|
# n04) dwarfs any load-induced srtt spike on the fiber probe, matching
|
|
# cost-avoidance. Combined with excluding n03-n08/n04-n08 from the mutation
|
|
# (below), n08's asserted comparison is robust to both load and churn.
|
|
- edges: ["n02-n06", "n04-n08"]
|
|
policy:
|
|
delay_ms: [150, 250]
|
|
jitter_ms: [15, 30]
|
|
loss_pct: [2, 8]
|
|
# WiFi links (moderate latency, low loss)
|
|
- edges: ["n03-n06", "n08-n10"]
|
|
policy:
|
|
delay_ms: [5, 20]
|
|
jitter_ms: [2, 5]
|
|
loss_pct: [1, 3]
|
|
mutation:
|
|
interval_secs: {min: 30, max: 60}
|
|
fraction: 0.2
|
|
# Never mutate the two links n08's tree_parents assertion depends on: the
|
|
# degraded policy (50-100ms) could otherwise slow the fiber n03-n08 link
|
|
# enough to flip n08 to n04 independently of the widened Bluetooth contrast,
|
|
# making the assertion a coin toss. The mutation still churns the other ten
|
|
# links, so the dynamic behaviour it exists to exercise is unchanged.
|
|
exclude_edges: ["n03-n08", "n04-n08"]
|
|
policies:
|
|
normal:
|
|
delay_ms: [1, 10]
|
|
loss_pct: [0, 1]
|
|
degraded:
|
|
delay_ms: [50, 100]
|
|
jitter_ms: [10, 30]
|
|
loss_pct: [3, 8]
|
|
|
|
link_flaps:
|
|
enabled: false
|
|
|
|
traffic:
|
|
enabled: true
|
|
max_concurrent: 3
|
|
interval_secs: {min: 10, max: 30}
|
|
duration_secs: {min: 5, max: 15}
|
|
parallel_streams: 4
|
|
|
|
# Baseline: the mesh came up, agreed on a root, and took parents. This
|
|
# asserts nothing about parent choice, which this scenario cannot
|
|
# currently assert; it exists so that a run in which the mesh never formed
|
|
# cannot report success, which until now it could,
|
|
# because this scenario carried no assertions at all.
|
|
#
|
|
# Ten nodes, one root, nine parented and six to eight sessions across
|
|
# the five provably-completed archived runs. The session floor is set
|
|
# below the observed minimum deliberately: it is here to catch a mesh
|
|
# that carries no traffic at all, not to pin a throughput.
|
|
assertions:
|
|
baseline:
|
|
min_nodes_reporting: 10
|
|
max_roots: 1
|
|
min_nodes_parented: 9
|
|
min_sessions: 3
|
|
# The scenario's own subject, assertable for the first time now that the
|
|
# root is pinned to n01 and the topology above describes the tree that
|
|
# forms. n08's two candidates sit at equal depth, so the choice is decided
|
|
# by link cost alone: n03 over fiber-grade default netem against n04 over
|
|
# Bluetooth. That is the fiber-versus-Bluetooth preference this scenario
|
|
# exists to test.
|
|
#
|
|
# The criterion is encoded as written rather than calibrated from runs:
|
|
# "should pick n03" is the spec, and four of four runs agree. n06 is
|
|
# deliberately absent — see the corrected note in the header for why its
|
|
# documented criterion was never founded on this topology.
|
|
tree_parents:
|
|
n08: n03
|
|
|
|
logging:
|
|
rust_log: "info"
|
|
output_dir: "./sim-results"
|