mirror of
https://github.com/jmcorgan/fips.git
synced 2026-07-30 19:46:15 +00:00
Merge master: retire cost/congestion chaos scenarios
Bring the six retired cost-based parent-selection chaos scenarios up to next, resolving the ci-local.sh suite-list conflict (next carries the extra mixed-profile and admission-cap suites). next already has the equivalent sans-IO cost and transport-drop coverage, so src is unchanged; this takes only the scenario removals and the CI-list, README and comment updates.
This commit is contained in:
Johnathan Corgan
@@ -457,24 +457,6 @@ jobs:
|
||||
- suite: tcp-mesh
|
||||
type: chaos
|
||||
scenario: tcp-mesh
|
||||
- suite: bottleneck-parent
|
||||
type: chaos
|
||||
scenario: bottleneck-parent
|
||||
- suite: cost-avoidance
|
||||
type: chaos
|
||||
scenario: cost-avoidance
|
||||
- suite: cost-reeval
|
||||
type: chaos
|
||||
scenario: cost-reeval
|
||||
- suite: cost-stability
|
||||
type: chaos
|
||||
scenario: cost-stability
|
||||
- suite: depth-vs-cost
|
||||
type: chaos
|
||||
scenario: depth-vs-cost
|
||||
- suite: mixed-technology
|
||||
type: chaos
|
||||
scenario: mixed-technology
|
||||
- suite: congestion-stress
|
||||
type: chaos
|
||||
scenario: congestion-stress
|
||||
|
||||
+14
-33
@@ -44,41 +44,22 @@ Random topologies with increasing stressor intensity.
|
||||
simultaneously, bandwidth tiers (1/10/100/1000 Mbps), `protect_connectivity`
|
||||
disabled (partitions allowed).
|
||||
|
||||
### Cost-based parent selection
|
||||
### Cost-based parent selection — retired, now sans-IO unit tests
|
||||
|
||||
Explicit topologies with heterogeneous link types (fiber, Bluetooth, WiFi) to
|
||||
test that the spanning tree selects optimal parents based on link cost.
|
||||
The cost-selection scenarios (cost-avoidance, depth-vs-cost, bottleneck-parent,
|
||||
cost-reeval, cost-stability, mixed-technology) were retired on 2026-07-23.
|
||||
Their subject was the pure `TreeState::evaluate_parent` decision — which parent
|
||||
wins on `effective_depth = depth + link_cost`, when periodic re-evaluation
|
||||
switches, and when hysteresis suppresses a flap. A Docker mesh could not test
|
||||
that reliably: the root is whichever node holds the smallest `NodeAddr`, MMP
|
||||
costs take several measurement windows to settle, and hold-down plus hysteresis
|
||||
timing all confound the outcome (a deterministic `link_swap` attempt still
|
||||
produced zero periodic switches in a full run).
|
||||
|
||||
| Scenario | Nodes | Shape | Link types | Duration | What it tests |
|
||||
| ----------------- | ----- | --------------- | ------------------------ | -------- | ------------------------------------------------------------------- |
|
||||
| cost-avoidance | 4 | Diamond | Fiber + Bluetooth | 120s | n04 picks fiber parent (n03) over Bluetooth parent (n02) |
|
||||
| depth-vs-cost | 4 | Linear tree | Fiber + Bluetooth | 120s | Cost tradeoff: depth vs. Bluetooth link quality |
|
||||
| bottleneck-parent | 10 | Tree with BT | Fiber + Bluetooth | 120s | n06 avoids Bluetooth bottleneck via n02, picks fiber via n03 |
|
||||
| cost-mixed-7node | 7 | Multi-type tree | Fiber + Bluetooth + WiFi | 180s | n06 prefers fiber (n03) over WiFi (n04) |
|
||||
| cost-reeval | 4 | Diamond | Fiber (mutated) | 180s | Periodic re-evaluation triggers parent switch (reeval_interval=15s) |
|
||||
| cost-stability | 4 | Diamond | WiFi (all) | 180s | Hysteresis prevents flapping when costs vary within 20% band |
|
||||
|
||||
- **cost-avoidance**, **depth-vs-cost**: Minimal scenarios validating the core
|
||||
cost formula. Bluetooth (L2CAP) links use 15-40ms delay and 2-8% loss;
|
||||
fiber uses 1-5ms delay and 0-1% loss.
|
||||
- **bottleneck-parent**: Larger topology where some nodes have both fiber and
|
||||
Bluetooth paths to choose from, and one node (n09) is stuck with Bluetooth
|
||||
(no alternative).
|
||||
- **cost-mixed-7node**: Three link technologies in one mesh. Traffic enabled.
|
||||
- **cost-reeval**: Netem mutation (50% fraction, every 12-18s) degrades random
|
||||
links. FIPS override sets `reeval_interval_secs=15` so periodic re-evaluation
|
||||
catches cost asymmetry. Look for `trigger=periodic` in logs.
|
||||
- **cost-stability**: All links are WiFi. Mutation swings costs between
|
||||
`slightly_better` and `slightly_worse` — within the hysteresis band. Expect
|
||||
≤ 5 parent switches over 180s.
|
||||
|
||||
### Mixed-technology
|
||||
|
||||
Larger explicit topologies combining multiple link technologies.
|
||||
|
||||
| Scenario | Nodes | Link types | Duration | Netem mutation | What it tests |
|
||||
| ---------------- | ----- | ------------------------ | -------- | -------------- | ------------------------------------------------ |
|
||||
| mixed-technology | 10 | Fiber + Bluetooth + WiFi | 180s | 20%/30-60s | Tree convergence across heterogeneous link types |
|
||||
That logic is now covered by deterministic sans-IO unit tests in
|
||||
`src/tree/tests.rs` (`test_evaluate_parent_cost_*`, `..._hysteresis_*`,
|
||||
`..._effective_depth_*`), which run in the cargo quartet on every commit and
|
||||
can each be shown to fail by breaking the cost or hysteresis logic.
|
||||
|
||||
### Transport-specific
|
||||
|
||||
|
||||
@@ -1,107 +0,0 @@
|
||||
# Bottleneck Parent: 10-node focused Bluetooth bottleneck test
|
||||
#
|
||||
# Explicit topology with two Bluetooth (L2CAP) links that create
|
||||
# bottleneck parent candidates. Tests that nodes avoid choosing
|
||||
# Bluetooth parents when fiber alternatives exist at the same or
|
||||
# slightly greater depth.
|
||||
#
|
||||
# Topology:
|
||||
#
|
||||
# n01 (root)
|
||||
# / | \ \
|
||||
# f f f f
|
||||
# / | \ \
|
||||
# n02 n03 n04 n05
|
||||
# | / | \ |
|
||||
# BT f f BT
|
||||
# | / | \ |
|
||||
# n06 n07 n08 n09
|
||||
# |
|
||||
# f
|
||||
# |
|
||||
# n10
|
||||
#
|
||||
# Edges and link types:
|
||||
# Fiber: n01-n02, n01-n03, n01-n04, n01-n05,
|
||||
# n03-n06, n03-n07, n04-n08, n08-n10
|
||||
# Bluetooth: n02-n06, n05-n09
|
||||
#
|
||||
# Cross-links: n03-n08 (fiber) — gives n08 a fiber alternative to n04
|
||||
#
|
||||
# Test subjects:
|
||||
# - n06 has Bluetooth (n02) and fiber (n03) at depth 1 — should pick n03
|
||||
# - n09 has only Bluetooth (n05) — no alternative, stuck with BT parent
|
||||
|
||||
scenario:
|
||||
name: "bottleneck-parent"
|
||||
seed: 42
|
||||
duration_secs: 60
|
||||
|
||||
topology:
|
||||
algorithm: explicit
|
||||
num_nodes: 10
|
||||
params:
|
||||
adjacency:
|
||||
- [n01, n02]
|
||||
- [n01, n03]
|
||||
- [n01, n04]
|
||||
- [n01, n05]
|
||||
- [n02, n06]
|
||||
- [n03, n06]
|
||||
- [n03, n07]
|
||||
- [n03, n08]
|
||||
- [n04, n08]
|
||||
- [n05, n09]
|
||||
- [n08, n10]
|
||||
subnet: "172.20.0.0/24"
|
||||
ip_start: 10
|
||||
|
||||
netem:
|
||||
enabled: true
|
||||
default_policy:
|
||||
delay_ms: [1, 5]
|
||||
jitter_ms: [0, 1]
|
||||
loss_pct: [0, 0.5]
|
||||
link_policies:
|
||||
# Bluetooth (L2CAP) links
|
||||
- edges: ["n02-n06", "n05-n09"]
|
||||
policy:
|
||||
delay_ms: [15, 40]
|
||||
jitter_ms: [5, 15]
|
||||
loss_pct: [2, 8]
|
||||
mutation:
|
||||
interval_secs: {min: 30, max: 60}
|
||||
fraction: 0.2
|
||||
policies:
|
||||
normal:
|
||||
delay_ms: [1, 5]
|
||||
loss_pct: [0, 0.5]
|
||||
|
||||
link_flaps:
|
||||
enabled: false
|
||||
|
||||
traffic:
|
||||
enabled: true
|
||||
max_concurrent: 2
|
||||
interval_secs: {min: 15, max: 30}
|
||||
duration_secs: {min: 5, max: 10}
|
||||
parallel_streams: 2
|
||||
|
||||
# The two test subjects named at the top of this file, encoded. Both
|
||||
# existed only as comments and were checked by nothing.
|
||||
#
|
||||
# n06 has a Bluetooth parent (n02) and a fiber one (n03) at the same depth
|
||||
# and should take the fiber. n09 has only Bluetooth, so asserting it keeps
|
||||
# n05 is asserting that the absence of an alternative is handled without
|
||||
# thrashing, not that Bluetooth was preferred.
|
||||
#
|
||||
# From the archived corpus: across the six provably-completed runs, n06's
|
||||
# parent is n03 in all six and n09's is n05 in all six.
|
||||
assertions:
|
||||
tree_parents:
|
||||
n06: n03
|
||||
n09: n05
|
||||
|
||||
logging:
|
||||
rust_log: "info"
|
||||
output_dir: "./sim-results"
|
||||
@@ -7,17 +7,22 @@
|
||||
# Congestion detection signals exercised:
|
||||
# 1. MMP loss detection: netem loss exceeds the 5% loss_threshold,
|
||||
# triggering detect_congestion() via MMP metrics on transit nodes.
|
||||
# 2. Kernel socket drops: small recv_buf_size (8KB) combined with
|
||||
# traffic saturation causes SO_RXQ_OVFL on the UDP socket,
|
||||
# triggering the transport drop detection path.
|
||||
# 3. Ingress policing: tc policer on the receive side drops excess
|
||||
# 2. Ingress policing: tc policer on the receive side drops excess
|
||||
# inbound packets, creating bursty arrival patterns.
|
||||
# 3. ECN CE marking under the shaped bottleneck queue.
|
||||
#
|
||||
# The kernel socket-drop signal (SO_RXQ_OVFL) is NOT exercised here. This
|
||||
# scenario's 1 Mbps cap and ingress policer, which its ECN/MMP signals
|
||||
# require, make socket overflow impossible. It also cannot be provoked
|
||||
# deterministically anywhere in Docker — a fresh daemon reader keeps up
|
||||
# with container-speed traffic — so the FIPS drop-detection logic is
|
||||
# covered by transport-drop unit tests instead. See the note at the
|
||||
# assertions block below.
|
||||
#
|
||||
# ECN is explicitly enabled via fips_overrides.
|
||||
#
|
||||
# Success criteria (verified via post-run congestion snapshot):
|
||||
# - congestion_detected > 0 on at least one forwarding node
|
||||
# - kernel_drop_events > 0 on at least one node
|
||||
# - ce_forwarded > 0 on transit nodes
|
||||
# - ce_received > 0 on destination nodes
|
||||
#
|
||||
@@ -118,40 +123,28 @@ assertions:
|
||||
min_nodes_ce_forwarded: 1
|
||||
min_nodes_ce_received: 1
|
||||
|
||||
# The fourth criterion, "kernel_drop_events > 0 on at least one node", is
|
||||
# NOT asserted, because this scenario has never met it: across all 182
|
||||
# archived runs, including the six that meet the other three, no node has
|
||||
# reported a non-zero kernel_drop_events. Asserting it would red the scenario
|
||||
# permanently, and asserting a weakened version would assert nothing.
|
||||
# The kernel socket-drop criterion is NOT asserted here and no longer
|
||||
# belongs to this scenario. The FIPS drop-DETECTION logic is covered by
|
||||
# transport-drop unit tests (2026-07-23); the kernel dropping datagrams is a
|
||||
# kernel behaviour, not FIPS's to test, and could not be provoked in Docker.
|
||||
#
|
||||
# CORRECTED 2026-07-23. An earlier version of this note concluded that the
|
||||
# transport drop-detection path "is exercised by nothing". That is wrong, and
|
||||
# the error was one of scope: it looked only inside this scenario. Widening
|
||||
# the scan to every scenario's congestion snapshots -- 2317 files, 11778 node
|
||||
# records -- finds 51 records reporting kernel_drop_events > 0 across six
|
||||
# other scenarios (depth-vs-cost, bottleneck-parent, tcp-mesh, cost-avoidance,
|
||||
# maelstrom-sparse, mixed-technology), with raw drop counts up to 41165. The
|
||||
# path is live and plumbed through to the snapshot. What is dead is this
|
||||
# scenario's ability to provoke it.
|
||||
# Why it could never be met here: across all 182 archived runs of this
|
||||
# scenario, including the six that meet the three signals above, no node
|
||||
# ever reported a non-zero kernel_drop_events. SO_RXQ_OVFL counts datagrams
|
||||
# arriving at a FULL receive queue, and the 1 Mbps cap sets the arrival rate
|
||||
# to 125 kB/s per link. Linux doubles a requested SO_RCVBUF, so the queue is
|
||||
# 8192 bytes and filling it would take ~65 ms of reader stall (~22 ms even at
|
||||
# the busiest node's 3 Mbps aggregate). Traffic volume cannot cause the
|
||||
# overflow because the volume is capped below the rate the buffer drains, and
|
||||
# the ingress policer discards excess in tc before the socket ever sees it.
|
||||
# An unshaped attempt (congestion-drops, 2026-07-23) recorded zero raw drops
|
||||
# on every node even with a 4 KB buffer and heavy iperf: a fresh daemon
|
||||
# reader keeps up, so the overflow cannot be provoked deterministically. That
|
||||
# is why the detection edge is tested as a sans-IO unit test.
|
||||
#
|
||||
# The inversion is the lead: every scenario that DOES overflow runs with the
|
||||
# default 2 MB recv_buf_size, while this one, the only scenario that shrinks
|
||||
# it to 4 KB, never does. SO_RXQ_OVFL counts datagrams arriving at a full
|
||||
# queue, and the 1 Mbps cap below sets the arrival rate to 125 kB/s per link.
|
||||
# Linux doubles a requested SO_RCVBUF, so the queue is 8192 bytes and filling
|
||||
# it takes ~65 ms of reader stall -- ~22 ms even at the busiest node's 3 Mbps
|
||||
# aggregate. Traffic volume cannot cause the overflow because the volume is
|
||||
# capped below the rate the buffer drains. The ingress policer works against
|
||||
# the same signal, discarding excess in tc before the socket ever sees it.
|
||||
#
|
||||
# So signal 3 appears to suppress signal 2, and the two may not belong in one
|
||||
# scenario. Tracked with the experiment that would settle it (raise or drop
|
||||
# the bandwidth cap for one run and watch the counter) in ISSUE-2026-0084.
|
||||
#
|
||||
# Note also that the header above says "small recv_buf_size (8KB)" while
|
||||
# fips_overrides sets 4096. The 8 KB figure matches what Linux allocates after
|
||||
# doubling, so the comment may be describing the effective queue rather than
|
||||
# the setting; it reads as the setting and should say which it means.
|
||||
# The recv_buf_size: 4096 override below is kept because the three asserted
|
||||
# signals were validated with it in place; it is inert for socket overflow
|
||||
# under this scenario's cap.
|
||||
|
||||
logging:
|
||||
rust_log: "info"
|
||||
|
||||
@@ -1,96 +0,0 @@
|
||||
# Cost-Based Parent Selection: Bottleneck Avoidance Test
|
||||
#
|
||||
# Topology (explicit 4-node diamond):
|
||||
#
|
||||
# n01 (root — smallest addr)
|
||||
# / \
|
||||
# fiber fiber
|
||||
# / \
|
||||
# n02 n03
|
||||
# \ /
|
||||
# BT fiber
|
||||
# \ /
|
||||
# n04 (test subject)
|
||||
#
|
||||
# n04 has two candidate parents at depth 1: n02 (via Bluetooth L2CAP)
|
||||
# and n03 (via fiber). Cost-based selection should pick n03 because
|
||||
# link_cost = etx * (1 + srtt_ms/100), so the far slower Bluetooth link
|
||||
# costs far more:
|
||||
# effective_depth(n02) = 1 + ~4.5 (Bluetooth, ~150-250ms) = ~5.5
|
||||
# effective_depth(n03) = 1 + ~1.05 (fiber, ~1-5ms) = ~2.05
|
||||
#
|
||||
# The Bluetooth delay was widened deliberately, and the size is set by load,
|
||||
# not by realism. At the original 15-40ms the cost margin was ~0.4, and even
|
||||
# 80-120ms was not enough: under the CI's parallel-chaos contention the
|
||||
# *measured* srtt of the fiber link spikes (host scheduling, not netem) and
|
||||
# was observed to exceed a 120ms Bluetooth base, flipping n04 to n02. At
|
||||
# 150-250ms the gap dwarfs any plausible one-sided fiber spike, so the
|
||||
# assertion below is a reliable verdict rather than a coin toss under load.
|
||||
# n04 still establishes the n02 link (it was parented to it when this flaked),
|
||||
# so the choice remains real rather than vacuous.
|
||||
#
|
||||
# Validation: tree snapshot shows n04's parent is n03.
|
||||
|
||||
scenario:
|
||||
name: "cost-avoidance"
|
||||
seed: 42
|
||||
duration_secs: 45
|
||||
|
||||
topology:
|
||||
algorithm: explicit
|
||||
num_nodes: 4
|
||||
params:
|
||||
adjacency:
|
||||
- [n01, n02]
|
||||
- [n01, n03]
|
||||
- [n02, n04]
|
||||
- [n03, n04]
|
||||
subnet: "172.20.0.0/24"
|
||||
ip_start: 10
|
||||
|
||||
netem:
|
||||
enabled: true
|
||||
default_policy:
|
||||
# Fiber-like
|
||||
delay_ms: [1, 5]
|
||||
jitter_ms: [0, 1]
|
||||
loss_pct: [0, 0.5]
|
||||
link_policies:
|
||||
# Bluetooth (L2CAP) link from n02 to n04. Delay widened to 150-250ms so the
|
||||
# fiber-vs-Bluetooth cost margin survives CI load spikes on the fiber
|
||||
# probe's measured latency (see header). Well within handshake timeouts, so
|
||||
# the n02 link still establishes and the choice stays real.
|
||||
- edges: ["n02-n04"]
|
||||
policy:
|
||||
delay_ms: [150, 250]
|
||||
jitter_ms: [15, 30]
|
||||
loss_pct: [2, 8]
|
||||
|
||||
link_flaps:
|
||||
enabled: false
|
||||
|
||||
traffic:
|
||||
enabled: true
|
||||
max_concurrent: 2
|
||||
interval_secs: {min: 15, max: 30}
|
||||
duration_secs: {min: 5, max: 10}
|
||||
parallel_streams: 2
|
||||
|
||||
# The validation line at the top of this file, encoded. It existed only as
|
||||
# that comment and was checked by nothing, so the scenario could not fail
|
||||
# on the one thing it was built to test.
|
||||
#
|
||||
# From the archived corpus: of the six runs that carry a status.txt and are
|
||||
# therefore provably completed, n04's parent is n03 in all six. Among the
|
||||
# 176 older runs the picture is quite different -- n04 is absent from the
|
||||
# snapshot in 107 of them and is its own root in 49 -- but those are runs
|
||||
# whose tree had not converged when the snapshot was taken rather than runs
|
||||
# that chose the wrong parent, which is why the assertion fails an absent
|
||||
# or self-parented node explicitly instead of skipping it.
|
||||
assertions:
|
||||
tree_parents:
|
||||
n04: n03
|
||||
|
||||
logging:
|
||||
rust_log: "info"
|
||||
output_dir: "./sim-results"
|
||||
@@ -1,117 +0,0 @@
|
||||
# Periodic Cost Re-evaluation Test
|
||||
#
|
||||
# Topology (explicit 4-node diamond):
|
||||
#
|
||||
# n01 (root)
|
||||
# / \
|
||||
# fiber fiber
|
||||
# / \
|
||||
# n02 n03
|
||||
# \ /
|
||||
# fiber fiber
|
||||
# \ /
|
||||
# n04 (test subject)
|
||||
#
|
||||
# Initial state: All links are fiber. n04 has two candidate parents at
|
||||
# depth 1: n02 and n03. Both have identical costs (~1.01), so n04 picks
|
||||
# n02 (smaller NodeAddr, tiebreak rule).
|
||||
#
|
||||
# Mutation: Stochastic netem mutation with a single "degraded" policy
|
||||
# (Bluetooth-like: 15-40ms delay, 2-8% loss). With fraction=0.5, on
|
||||
# average 2 of 4 edges degrade each round. Over 12 mutation rounds
|
||||
# (180s / 15s interval), n02-n04 will be degraded in some rounds.
|
||||
#
|
||||
# Expected behavior: When n02-n04 is degraded and n03-n04 stays fiber
|
||||
# (or vice versa), periodic re-evaluation detects the cost asymmetry
|
||||
# and switches parents. Look for "trigger = periodic" in logs.
|
||||
#
|
||||
# FIPS overrides: reeval_interval_secs=15 (vs default 60) to increase
|
||||
# the chance of catching a cost asymmetry within the mutation window.
|
||||
#
|
||||
# Validation: grep n04 logs for "Parent switched via periodic cost
|
||||
# re-evaluation" (trigger=periodic). If mutation never creates enough
|
||||
# asymmetry in a particular seed, the test still validates that periodic
|
||||
# re-eval runs without interference.
|
||||
#
|
||||
# THIS SCENARIO COULD NOT EXERCISE ITS SUBJECT AT ALL until 2026-07-23, and
|
||||
# the reason is worth keeping because nothing about it was visible from here.
|
||||
# The mesh roots at the numerically smallest NodeAddr, a hash of the node's
|
||||
# key (src/tree/state.rs:363-390), and that turned out to be **n04** — this
|
||||
# scenario's own designated test subject. A root has no parent, and
|
||||
# evaluate_parent returns early for the smallest address, so the parent switch
|
||||
# described above could never happen. The tree was inverted: n04 at depth 0,
|
||||
# n01 a leaf at depth 2, in all five recent runs.
|
||||
#
|
||||
# The archived logs show exactly that shape. The live info! string above
|
||||
# appears in 14 of 105 historical runs, on n04 in 13 of them, back when keys
|
||||
# were regenerated per run and the root wandered. In the five runs before this
|
||||
# fix it appears only on **n01** — the node that actually had a parent choice.
|
||||
#
|
||||
# `topology.pin_root` (default true) now assigns identities in NodeAddr order
|
||||
# so n01 is root and n04 is the depth-1 subject this file describes.
|
||||
#
|
||||
# MEASURED after the fix, 2026-07-23: the periodic switch now happens, in 2 of
|
||||
# 3 runs on n04 (0, 1, 1 occurrences). Before the fix it was 0 in 5 of 5, and
|
||||
# structurally impossible rather than merely rare. So the subject is exercised
|
||||
# for the first time.
|
||||
#
|
||||
# STILL NOT ASSERTED, deliberately. A "at least one periodic switch" assertion
|
||||
# would red roughly one run in three on this evidence, and a single run reading
|
||||
# zero is an argument against asserting rather than for it. Two ways forward,
|
||||
# neither taken here: gather enough runs to know whether 2-in-3 is the real
|
||||
# rate, or reduce the variance -- the mutation is stochastic with fraction=0.5,
|
||||
# so a deterministic link-degradation schedule would make the asymmetry this
|
||||
# scenario needs occur every run and the outcome assertable.
|
||||
# Tracked in ISSUE-2026-0085.
|
||||
|
||||
scenario:
|
||||
name: "cost-reeval"
|
||||
seed: 42
|
||||
duration_secs: 180
|
||||
|
||||
topology:
|
||||
algorithm: explicit
|
||||
num_nodes: 4
|
||||
params:
|
||||
adjacency:
|
||||
- [n01, n02]
|
||||
- [n01, n03]
|
||||
- [n02, n04]
|
||||
- [n03, n04]
|
||||
subnet: "172.20.0.0/24"
|
||||
ip_start: 10
|
||||
|
||||
netem:
|
||||
enabled: true
|
||||
default_policy:
|
||||
# Fiber-like baseline
|
||||
delay_ms: [1, 5]
|
||||
jitter_ms: [0, 1]
|
||||
loss_pct: [0, 0.5]
|
||||
mutation:
|
||||
interval_secs: {min: 12, max: 18}
|
||||
fraction: 0.5
|
||||
policies:
|
||||
degraded:
|
||||
delay_ms: [15, 40]
|
||||
jitter_ms: [5, 15]
|
||||
loss_pct: [2, 8]
|
||||
|
||||
link_flaps:
|
||||
enabled: false
|
||||
|
||||
traffic:
|
||||
enabled: true
|
||||
max_concurrent: 1
|
||||
interval_secs: {min: 15, max: 30}
|
||||
duration_secs: {min: 5, max: 10}
|
||||
parallel_streams: 2
|
||||
|
||||
logging:
|
||||
rust_log: "info"
|
||||
output_dir: "./sim-results"
|
||||
|
||||
fips_overrides:
|
||||
node:
|
||||
tree:
|
||||
reeval_interval_secs: 15
|
||||
@@ -1,101 +0,0 @@
|
||||
# Cost-Based Parent Selection: Hysteresis Stability Test
|
||||
#
|
||||
# Topology (explicit 4-node diamond, symmetric):
|
||||
#
|
||||
# n01 (root)
|
||||
# / \
|
||||
# wifi wifi
|
||||
# / \
|
||||
# n02 n03
|
||||
# \ /
|
||||
# wifi wifi
|
||||
# \ /
|
||||
# n04 (test subject)
|
||||
#
|
||||
# All links are WiFi-like with similar characteristics. Aggressive
|
||||
# netem mutation shifts link qualities every 10-20s, but the changes
|
||||
# stay within the 20% hysteresis band. n04 should pick one parent
|
||||
# and mostly stick with it — flapping indicates insufficient hysteresis.
|
||||
#
|
||||
# Validation: n04 should switch parent at most 5 times. This is now the
|
||||
# max_parent_switches assertion below, scoped to n04. It previously
|
||||
# existed only as this comment and was checked by nothing, so the
|
||||
# scenario could not fail on it.
|
||||
#
|
||||
# The threshold of 5 is the one this comment always named, kept
|
||||
# deliberately rather than re-derived. What the number is worth, from 11
|
||||
# completed archived runs of this scenario: n04 was 1 or 2 every time and
|
||||
# never reached 3, while the mesh-wide total ranged 3 to 6. Two
|
||||
# consequences worth knowing before trusting a green result here.
|
||||
#
|
||||
# First, 5 is a long way above anything observed, so this catches only
|
||||
# near-pathological reparenting. The daemon's own 30s parent hold-down
|
||||
# caps non-mandatory switches at roughly 6 per 180s run, so the band
|
||||
# where this can fail is narrow. A hysteresis regression that added two
|
||||
# or three switches would pass.
|
||||
#
|
||||
# Second, the count spans the container's whole life, not just the
|
||||
# mutation phase: in the archived runs nearly every n04 switch lands in
|
||||
# the first 100ms during initial tree formation, before the first netem
|
||||
# mutation fires. So this number is dominated by convergence transients
|
||||
# rather than by the hysteresis behaviour the scenario is named for.
|
||||
#
|
||||
# Tightening to 3 would make it a real detector against the observed
|
||||
# distribution. That is a change to the stated criterion rather than a
|
||||
# fix, so it is left as a decision rather than taken here.
|
||||
|
||||
scenario:
|
||||
name: "cost-stability"
|
||||
seed: 42
|
||||
duration_secs: 180
|
||||
|
||||
topology:
|
||||
algorithm: explicit
|
||||
num_nodes: 4
|
||||
params:
|
||||
adjacency:
|
||||
- [n01, n02]
|
||||
- [n01, n03]
|
||||
- [n02, n04]
|
||||
- [n03, n04]
|
||||
subnet: "172.20.0.0/24"
|
||||
ip_start: 10
|
||||
|
||||
netem:
|
||||
enabled: true
|
||||
default_policy:
|
||||
# WiFi baseline
|
||||
delay_ms: [5, 20]
|
||||
jitter_ms: [2, 5]
|
||||
loss_pct: [1, 3]
|
||||
mutation:
|
||||
interval_secs: {min: 10, max: 20}
|
||||
fraction: 1.0
|
||||
policies:
|
||||
slightly_better:
|
||||
delay_ms: [3, 8]
|
||||
jitter_ms: [1, 3]
|
||||
loss_pct: [0, 1]
|
||||
slightly_worse:
|
||||
delay_ms: [15, 25]
|
||||
jitter_ms: [3, 8]
|
||||
loss_pct: [2, 4]
|
||||
|
||||
link_flaps:
|
||||
enabled: false
|
||||
|
||||
traffic:
|
||||
enabled: true
|
||||
max_concurrent: 2
|
||||
interval_secs: {min: 15, max: 30}
|
||||
duration_secs: {min: 5, max: 15}
|
||||
parallel_streams: 2
|
||||
|
||||
assertions:
|
||||
max_parent_switches:
|
||||
node: n04
|
||||
max_total: 5
|
||||
|
||||
logging:
|
||||
rust_log: "info"
|
||||
output_dir: "./sim-results"
|
||||
@@ -1,103 +0,0 @@
|
||||
# Cost-Based Parent Selection: Deeper Fiber Beats Shallow Bluetooth
|
||||
#
|
||||
# Topology (explicit 4-node):
|
||||
#
|
||||
# n01 (root)
|
||||
# / \
|
||||
# fiber BT
|
||||
# / \
|
||||
# n02 n04 (test subject)
|
||||
# | /
|
||||
# fiber fiber
|
||||
# | /
|
||||
# n03
|
||||
#
|
||||
# n04 has two candidate parents:
|
||||
# - n01 (root, depth 0) via Bluetooth L2CAP: effective_depth = 0 + ~1.4 = ~1.4
|
||||
# - n03 (depth 2) via fiber: effective_depth = 2 + ~1.01 = ~3.01
|
||||
#
|
||||
# Without cost-based selection, n04 would pick n01 (depth 0 < depth 2).
|
||||
# With cost-based selection and these Bluetooth impairments, n04 may
|
||||
# still pick n01 since the Bluetooth cost (~1.4) is modest. This tests
|
||||
# that the cost formula correctly weighs depth against link quality.
|
||||
#
|
||||
# Validation: tree snapshot shows n04's parent selection reflects the
|
||||
# actual cost tradeoff between depth and link quality.
|
||||
#
|
||||
# NO PARENT IS ASSERTED HERE, and unlike the other cost scenarios that is
|
||||
# not because the implementation disagrees with the comment. It is because
|
||||
# the validation line above does not name an outcome: "reflects the actual
|
||||
# cost tradeoff" is satisfied by either answer, so there is nothing to
|
||||
# encode. The comment above even says n04 "may still pick n01".
|
||||
#
|
||||
# The corpus agrees that both answers occur. Across the five provably
|
||||
# completed archived runs, n04's parent is n01 in three and n03 in two,
|
||||
# with the same seed each time, so the choice is not deterministic under
|
||||
# this configuration.
|
||||
#
|
||||
# Pinning whichever answer is more common would convert an unspecified
|
||||
# behaviour into a regression guard for something nobody decided, and
|
||||
# would go red roughly two runs in five besides. What this scenario needs
|
||||
# first is a decision about which parent is correct given the stated
|
||||
# effective depths, and that is a protocol question, not a harness one.
|
||||
|
||||
scenario:
|
||||
name: "depth-vs-cost"
|
||||
seed: 42
|
||||
duration_secs: 45
|
||||
|
||||
topology:
|
||||
algorithm: explicit
|
||||
num_nodes: 4
|
||||
params:
|
||||
adjacency:
|
||||
- [n01, n02]
|
||||
- [n02, n03]
|
||||
- [n03, n04]
|
||||
- [n01, n04]
|
||||
subnet: "172.20.0.0/24"
|
||||
ip_start: 10
|
||||
|
||||
netem:
|
||||
enabled: true
|
||||
default_policy:
|
||||
# Fiber-like
|
||||
delay_ms: [1, 5]
|
||||
jitter_ms: [0, 1]
|
||||
loss_pct: [0, 0.5]
|
||||
link_policies:
|
||||
# Bluetooth (L2CAP) link from n01 to n04
|
||||
- edges: ["n01-n04"]
|
||||
policy:
|
||||
delay_ms: [15, 40]
|
||||
jitter_ms: [5, 15]
|
||||
loss_pct: [2, 8]
|
||||
|
||||
link_flaps:
|
||||
enabled: false
|
||||
|
||||
traffic:
|
||||
enabled: true
|
||||
max_concurrent: 1
|
||||
interval_secs: {min: 15, max: 30}
|
||||
duration_secs: {min: 5, max: 10}
|
||||
parallel_streams: 2
|
||||
|
||||
# Baseline: the mesh came up, agreed on a root, and took parents. This
|
||||
# asserts nothing about which parent n04 chooses, which is unspecified;
|
||||
# it exists so that a run in which the mesh never formed cannot report
|
||||
# success, which until now it could,
|
||||
# because this scenario carried no assertions at all.
|
||||
#
|
||||
# Four nodes, one root, three parented and four sessions in all five
|
||||
# provably-completed archived runs.
|
||||
assertions:
|
||||
baseline:
|
||||
min_nodes_reporting: 4
|
||||
max_roots: 1
|
||||
min_nodes_parented: 3
|
||||
min_sessions: 2
|
||||
|
||||
logging:
|
||||
rust_log: "info"
|
||||
output_dir: "./sim-results"
|
||||
@@ -1,186 +0,0 @@
|
||||
# Mixed Technology: 10-node heterogeneous network
|
||||
#
|
||||
# Explicit topology with Bluetooth (L2CAP), WiFi, and fiber links.
|
||||
# Tests that cost-based parent selection produces a tree favoring
|
||||
# low-cost paths when multiple link technologies coexist.
|
||||
#
|
||||
# Topology:
|
||||
#
|
||||
# n01 (root)
|
||||
# / | \
|
||||
# f f f
|
||||
# / | \
|
||||
# n02 n03 n04
|
||||
# | \ | \ | \
|
||||
# f BT f f BT f
|
||||
# | \ | | \ |
|
||||
# n05 n06 n07 n08 n09
|
||||
# \ /
|
||||
# wifi---wifi
|
||||
# n10
|
||||
#
|
||||
# Edges and link types:
|
||||
# Fiber: n01-n02, n01-n03, n01-n04, n02-n05, n03-n07, n04-n09
|
||||
# Bluetooth: n02-n06, n04-n08
|
||||
# WiFi: n03-n06, n08-n10
|
||||
# Fiber: n03-n08, n06-n10
|
||||
#
|
||||
# Test subjects:
|
||||
# - n08 has fiber (n03) and Bluetooth (n04) parents — should pick n03.
|
||||
# ASSERTED below. Holds in 4 of 4 runs under the pinned root.
|
||||
# - n06 was documented as having "fiber (n03) and Bluetooth (n02) parents".
|
||||
# CORRECTED 2026-07-23: that is wrong, and the netem block below is the
|
||||
# evidence rather than the ASCII art above. `n03-n06` is a **WiFi** link
|
||||
# (5-20ms, 1-3% loss) and `n02-n06` is Bluetooth (15-40ms, 2-8% loss), so
|
||||
# n06 chooses between two impaired links, not between fiber and Bluetooth.
|
||||
# n06's only fiber-grade link is `n06-n10`, and n10 sits at depth 3 so it
|
||||
# is never a good parent. There is therefore no reason to expect n06 to
|
||||
# prefer n03, and the mutation block (fraction 0.2, degraded 50-100ms)
|
||||
# moves the two costs run to run: across four runs n06 took n03 twice and
|
||||
# n02 twice. NOT asserted, and the observed split is the correct outcome
|
||||
# rather than a defect — in the run checked, n03 measured a link cost of 3
|
||||
# against n02's 1, so effective depth 1+3 beat by 1+1 and the daemon chose
|
||||
# the cheaper path exactly as it should.
|
||||
#
|
||||
# Netem mutation shifts fiber-only links between normal and degraded.
|
||||
#
|
||||
# n08 IS now asserted (below); n06 is not, and both reasons trace to the root,
|
||||
# which was found and fixed rather than papered over here.
|
||||
#
|
||||
# The diagram above puts n01 at the top, and both test subjects only make
|
||||
# sense in a tree rooted there. The mesh does not root by topology: it roots
|
||||
# at the numerically smallest NodeAddr (src/tree/state.rs:363-390), which is a
|
||||
# hash of the node's key and bears no relation to the numbering. Until
|
||||
# 2026-07-23 this scenario rooted at **n09** in every run, which put both
|
||||
# criteria out of reach. Under that root n08's two candidates were not at
|
||||
# equal depth, so its choice was settled by depth long before link technology
|
||||
# could matter, and n08 taking n04 was CORRECT behaviour that these comments
|
||||
# made look like a defect.
|
||||
#
|
||||
# What the archive shows, for anyone re-deriving this later: across the five
|
||||
# runs under the old root, n06's parent is n10 in three and n03 in two, and
|
||||
# n08's is n04 in four and n03 in one. Those numbers describe a tree rooted at
|
||||
# n09 and must not be used to calibrate anything now.
|
||||
#
|
||||
# `topology.pin_root` (default true) now assigns identities in NodeAddr order,
|
||||
# so n01 holds the smallest and the diagram describes the tree that forms.
|
||||
# Under that root n08 takes fiber n03 as its subject assertion (below) now
|
||||
# checks. n06 stays unasserted: its "fiber n03" was a WiFi link (see the
|
||||
# corrected note above), so it was never a founded criterion. Tracked in
|
||||
# ISSUE-2026-0085.
|
||||
#
|
||||
# The n08 assertion is made reliable, not merely reachable: the Bluetooth
|
||||
# contrast is widened (netem below) and the two links n08 chooses between are
|
||||
# excluded from the netem mutation, so neither host load nor a random
|
||||
# degradation can flip the asserted comparison.
|
||||
|
||||
scenario:
|
||||
name: "mixed-technology"
|
||||
seed: 42
|
||||
duration_secs: 90
|
||||
|
||||
topology:
|
||||
algorithm: explicit
|
||||
num_nodes: 10
|
||||
params:
|
||||
adjacency:
|
||||
- [n01, n02]
|
||||
- [n01, n03]
|
||||
- [n01, n04]
|
||||
- [n02, n05]
|
||||
- [n02, n06]
|
||||
- [n03, n06]
|
||||
- [n03, n07]
|
||||
- [n03, n08]
|
||||
- [n04, n08]
|
||||
- [n04, n09]
|
||||
- [n06, n10]
|
||||
- [n08, n10]
|
||||
subnet: "172.20.0.0/24"
|
||||
ip_start: 10
|
||||
|
||||
netem:
|
||||
enabled: true
|
||||
default_policy:
|
||||
# Fiber-like defaults
|
||||
delay_ms: [1, 5]
|
||||
jitter_ms: [0, 1]
|
||||
loss_pct: [0, 0.5]
|
||||
link_policies:
|
||||
# Bluetooth (L2CAP) links. Delay widened to 150-250ms so the
|
||||
# fiber-vs-Bluetooth cost margin on n08's choice (fiber n03 vs Bluetooth
|
||||
# n04) dwarfs any load-induced srtt spike on the fiber probe, matching
|
||||
# cost-avoidance. Combined with excluding n03-n08/n04-n08 from the mutation
|
||||
# (below), n08's asserted comparison is robust to both load and churn.
|
||||
- edges: ["n02-n06", "n04-n08"]
|
||||
policy:
|
||||
delay_ms: [150, 250]
|
||||
jitter_ms: [15, 30]
|
||||
loss_pct: [2, 8]
|
||||
# WiFi links (moderate latency, low loss)
|
||||
- edges: ["n03-n06", "n08-n10"]
|
||||
policy:
|
||||
delay_ms: [5, 20]
|
||||
jitter_ms: [2, 5]
|
||||
loss_pct: [1, 3]
|
||||
mutation:
|
||||
interval_secs: {min: 30, max: 60}
|
||||
fraction: 0.2
|
||||
# Never mutate the two links n08's tree_parents assertion depends on: the
|
||||
# degraded policy (50-100ms) could otherwise slow the fiber n03-n08 link
|
||||
# enough to flip n08 to n04 independently of the widened Bluetooth contrast,
|
||||
# making the assertion a coin toss. The mutation still churns the other ten
|
||||
# links, so the dynamic behaviour it exists to exercise is unchanged.
|
||||
exclude_edges: ["n03-n08", "n04-n08"]
|
||||
policies:
|
||||
normal:
|
||||
delay_ms: [1, 10]
|
||||
loss_pct: [0, 1]
|
||||
degraded:
|
||||
delay_ms: [50, 100]
|
||||
jitter_ms: [10, 30]
|
||||
loss_pct: [3, 8]
|
||||
|
||||
link_flaps:
|
||||
enabled: false
|
||||
|
||||
traffic:
|
||||
enabled: true
|
||||
max_concurrent: 3
|
||||
interval_secs: {min: 10, max: 30}
|
||||
duration_secs: {min: 5, max: 15}
|
||||
parallel_streams: 4
|
||||
|
||||
# Baseline: the mesh came up, agreed on a root, and took parents. This
|
||||
# asserts nothing about parent choice, which this scenario cannot
|
||||
# currently assert; it exists so that a run in which the mesh never formed
|
||||
# cannot report success, which until now it could,
|
||||
# because this scenario carried no assertions at all.
|
||||
#
|
||||
# Ten nodes, one root, nine parented and six to eight sessions across
|
||||
# the five provably-completed archived runs. The session floor is set
|
||||
# below the observed minimum deliberately: it is here to catch a mesh
|
||||
# that carries no traffic at all, not to pin a throughput.
|
||||
assertions:
|
||||
baseline:
|
||||
min_nodes_reporting: 10
|
||||
max_roots: 1
|
||||
min_nodes_parented: 9
|
||||
min_sessions: 3
|
||||
# The scenario's own subject, assertable for the first time now that the
|
||||
# root is pinned to n01 and the topology above describes the tree that
|
||||
# forms. n08's two candidates sit at equal depth, so the choice is decided
|
||||
# by link cost alone: n03 over fiber-grade default netem against n04 over
|
||||
# Bluetooth. That is the fiber-versus-Bluetooth preference this scenario
|
||||
# exists to test.
|
||||
#
|
||||
# The criterion is encoded as written rather than calibrated from runs:
|
||||
# "should pick n03" is the spec, and four of four runs agree. n06 is
|
||||
# deliberately absent — see the corrected note in the header for why its
|
||||
# documented criterion was never founded on this topology.
|
||||
tree_parents:
|
||||
n08: n03
|
||||
|
||||
logging:
|
||||
rust_log: "info"
|
||||
output_dir: "./sim-results"
|
||||
@@ -46,11 +46,11 @@ class TopologyConfig:
|
||||
transport_mix: dict[str, float] | None = None
|
||||
# Assign derived identities in NodeAddr order so n01 holds the smallest and
|
||||
# is therefore the root every scenario diagram draws. Default on: without it
|
||||
# the root is effectively arbitrary, which left `cost-reeval` rooted at its
|
||||
# own designated test subject — so the parent switch it exists to observe
|
||||
# could not occur — and made `mixed-technology`'s parent criteria
|
||||
# unreachable. Set false where election from an arbitrary key distribution
|
||||
# is itself the subject.
|
||||
# the root is whichever node happens to hold the smallest key, which is
|
||||
# effectively arbitrary and leaves any scenario that reasons about a
|
||||
# specific root (e.g. bloom-storm's diamond, or a baseline max_roots check)
|
||||
# describing a tree that does not form. Set false where election from an
|
||||
# arbitrary key distribution is itself the subject.
|
||||
pin_root: bool = True
|
||||
# Set by --subnet to opt out of claiming a free range. Not a scenario-file
|
||||
# key: a scenario that hardcoded its range would reintroduce the collision
|
||||
|
||||
@@ -208,11 +208,10 @@ def generate_topology(
|
||||
# The mesh roots itself at the numerically smallest NodeAddr
|
||||
# (`src/tree/state.rs:363-390`), which is a hash of the node's public key
|
||||
# and so bears no relation to the node numbering. Every scenario diagram in
|
||||
# this tree draws n01 at the top, and before this ordering was applied that
|
||||
# held in only three of thirteen: `cost-reeval` rooted at n04 — its own
|
||||
# designated test subject, which therefore had no parent to switch and could
|
||||
# not exercise what the scenario exists to test — and `mixed-technology` at
|
||||
# n09, which made its two documented parent criteria unreachable.
|
||||
# this tree draws n01 at the top, and before this ordering was applied the
|
||||
# root landed on an arbitrary node in most scenarios — which is why the
|
||||
# cost-selection scenarios that reasoned about a specific root could never
|
||||
# be turned into reliable assertions and were moved to sans-IO unit tests.
|
||||
#
|
||||
# So derive the identities from the mesh name as before, then *assign* them
|
||||
# in NodeAddr order: n01 receives the smallest and is the root, n02 the next,
|
||||
|
||||
+12
-9
@@ -31,9 +31,7 @@
|
||||
# acl-allowlist, admission-cap, firewall, nat-cone, nat-symmetric,
|
||||
# nat-lan, nostr-publish-consume, stun-faults,
|
||||
# chaos-smoke-10, chaos-churn-mixed-10, chaos-ethernet-mesh,
|
||||
# chaos-ethernet-only, chaos-tcp-mesh, chaos-bottleneck-parent,
|
||||
# chaos-cost-avoidance, chaos-cost-reeval, chaos-cost-stability,
|
||||
# chaos-depth-vs-cost, chaos-mixed-technology, chaos-congestion-stress,
|
||||
# chaos-ethernet-only, chaos-tcp-mesh, chaos-congestion-stress,
|
||||
# chaos-bloom-storm,
|
||||
# sidecar, dns-resolver, deb-install
|
||||
#
|
||||
@@ -125,15 +123,20 @@ CHAOS_SUITES=(
|
||||
"ethernet-mesh ethernet-mesh"
|
||||
"ethernet-only ethernet-only"
|
||||
"tcp-mesh tcp-mesh"
|
||||
"bottleneck-parent bottleneck-parent"
|
||||
"cost-avoidance cost-avoidance"
|
||||
"cost-reeval cost-reeval"
|
||||
"cost-stability cost-stability"
|
||||
"depth-vs-cost depth-vs-cost"
|
||||
"mixed-technology mixed-technology"
|
||||
"congestion-stress congestion-stress"
|
||||
"bloom-storm bloom-storm"
|
||||
)
|
||||
# Scenarios retired 2026-07-23 because their subject was a pure decision the
|
||||
# Docker harness could not test reliably, now covered by sans-IO unit tests
|
||||
# of the parent-selection cost/depth/hysteresis decision and the transport
|
||||
# drop-detection edge:
|
||||
# - cost-reeval, cost-avoidance, cost-stability, depth-vs-cost,
|
||||
# mixed-technology, bottleneck-parent (root election, MMP measurement lag
|
||||
# and hold-down timing all confounded the Docker assertions).
|
||||
# - congestion-drops (never landed): a fresh daemon reader keeps up with
|
||||
# container-speed traffic, so SO_RXQ_OVFL could not be provoked.
|
||||
# congestion-stress stays: it exercises the ECN/MMP congestion signals,
|
||||
# which do need the real shaped bottleneck queue.
|
||||
SIDECAR_SUITES=(sidecar)
|
||||
GATEWAY_SUITES=(gateway)
|
||||
ACL_SUITES=(acl-allowlist)
|
||||
|
||||
Reference in New Issue
Block a user