Files
fips/testing/chaos/scenarios/congestion-stress.yaml
T
Johnathan Corgan 08a226fb63 Test cost-based parent selection and kernel-drop detection as unit tests
The cost-selection chaos scenarios (cost-reeval, cost-avoidance,
cost-stability, depth-vs-cost, mixed-technology, bottleneck-parent) tested
TreeState::evaluate_parent's decision logic through a Docker mesh that could
not exercise it reliably: the tree roots at whichever node holds the smallest
NodeAddr, MMP link costs take several measurement windows to settle, and the
parent hold-down plus hysteresis timing all confound the outcome. A
deterministic link-cost flap still produced zero periodic parent switches in a
full run.

Replace those six scenarios with deterministic unit tests in src/tree/tests.rs
that drive evaluate_parent directly: cheaper-link selection at equal depth,
switch-on-cost-change, hysteresis suppressing a marginal change while allowing
a significant one, and the depth-versus-cost effective-depth tradeoff. Each is
constructed so that breaking the cost or hysteresis logic makes it fail.

The congestion kernel-drop signal (SO_RXQ_OVFL) cannot be provoked
deterministically in Docker: a fresh daemon reader keeps up with
container-speed traffic, so the socket receive queue never overflows (an
unshaped run with a 4 KB buffer and heavy traffic recorded zero drops on every
node). Extract the drop-detection edge -- read the cumulative counter, fire an
event only on the transition into a new drop burst -- into
TransportDropState::observe_drops and unit-test it directly. congestion-stress
keeps its ECN and MMP congestion-signal assertions, which do need the real
shaped bottleneck queue.

Remove the retired scenarios from both CI runners and update the chaos README.
2026-07-23 23:33:01 +00:00

160 lines
5.4 KiB
YAML

# Congestion Stress: exercise both MMP-based and kernel-drop congestion detection
#
# Topology: 10-node tree with all links bandwidth-limited to 1 Mbps
# (egress HTB + ingress policing). Heavy iperf3 traffic (8 concurrent
# sessions, 8 parallel streams) combined with 5-10% netem loss.
#
# Congestion detection signals exercised:
# 1. MMP loss detection: netem loss exceeds the 5% loss_threshold,
# triggering detect_congestion() via MMP metrics on transit nodes.
# 2. Ingress policing: tc policer on the receive side drops excess
# inbound packets, creating bursty arrival patterns.
# 3. ECN CE marking under the shaped bottleneck queue.
#
# The kernel socket-drop signal (SO_RXQ_OVFL) is NOT exercised here. This
# scenario's 1 Mbps cap and ingress policer, which its ECN/MMP signals
# require, make socket overflow impossible. It also cannot be provoked
# deterministically anywhere in Docker — a fresh daemon reader keeps up
# with container-speed traffic — so the FIPS drop-detection logic is
# unit-tested directly in src/node/tests/unit.rs instead. See the note at
# the assertions block below.
#
# ECN is explicitly enabled via fips_overrides.
#
# Success criteria (verified via post-run congestion snapshot):
# - congestion_detected > 0 on at least one forwarding node
# - ce_forwarded > 0 on transit nodes
# - ce_received > 0 on destination nodes
#
# Topology:
#
# n01 (root)
# / \
# n02 n03
# / \ / \
# n04 n05 n06 n07
# | |
# n08 n09
# |
# n10
scenario:
name: "congestion-stress"
seed: 7
duration_secs: 120
topology:
algorithm: explicit
num_nodes: 10
params:
adjacency:
- [n01, n02]
- [n01, n03]
- [n02, n04]
- [n02, n05]
- [n03, n06]
- [n03, n07]
- [n04, n08]
- [n05, n09]
- [n08, n10]
subnet: "172.20.0.0/24"
ip_start: 10
bandwidth:
enabled: true
tiers_mbps: [1]
ingress:
enabled: true
tiers_kbps: [1000]
burst_bytes: 16000
netem:
enabled: true
default_policy:
delay_ms: [2, 5]
jitter_ms: [0, 1]
loss_pct: [5, 10]
mutation:
interval_secs: {min: 9999, max: 9999}
fraction: 0.0
policies:
baseline:
delay_ms: [2, 5]
loss_pct: [5, 10]
link_flaps:
enabled: false
traffic:
enabled: true
max_concurrent: 8
interval_secs: {min: 2, max: 5}
duration_secs: {min: 30, max: 60}
parallel_streams: 8
node_churn:
enabled: false
# Three of the four success criteria above, encoded. Until now all four
# existed only as that comment and were checked by nothing, so the scenario
# could not fail on any of them.
#
# The floors are 1 node each because that is what the criteria say ("on at
# least one forwarding node", "on transit nodes", "on destination nodes").
# They are deliberately not tightened to the observed counts: the criterion
# is the spec, and a floor invented from six runs would assert something
# nobody wrote down.
#
# What the floors are worth, from the archived corpus. The six runs that
# carry a status.txt -- the only ones provably completed, all between
# 2026-07-22 22:14 and 2026-07-23 02:02 -- meet all three with 5 to 9 nodes
# reporting each signal, so the margin over a floor of 1 is comfortable.
# The 176 older runs meet none of them. Those older runs did reach teardown
# (each wrote an analysis.txt), and ECN landed 2026-03-05, before all but
# one of them, so they are valid runs that observed no congestion rather
# than runs that died early. What changed on 2026-07-22 is not established:
# this file has not been touched since it was created, and neither have
# netem.py, traffic.py or control.py. Worth knowing before trusting a green
# result here, and worth its own investigation.
assertions:
congestion_signals:
min_nodes_detected: 1
min_nodes_ce_forwarded: 1
min_nodes_ce_received: 1
# The kernel socket-drop criterion is NOT asserted here and no longer
# belongs to this scenario. The FIPS drop-DETECTION logic is unit-tested in
# src/node/tests/unit.rs (2026-07-23); the kernel dropping datagrams is a
# kernel behaviour, not FIPS's to test, and could not be provoked in Docker.
#
# Why it could never be met here: across all 182 archived runs of this
# scenario, including the six that meet the three signals above, no node
# ever reported a non-zero kernel_drop_events. SO_RXQ_OVFL counts datagrams
# arriving at a FULL receive queue, and the 1 Mbps cap sets the arrival rate
# to 125 kB/s per link. Linux doubles a requested SO_RCVBUF, so the queue is
# 8192 bytes and filling it would take ~65 ms of reader stall (~22 ms even at
# the busiest node's 3 Mbps aggregate). Traffic volume cannot cause the
# overflow because the volume is capped below the rate the buffer drains, and
# the ingress policer discards excess in tc before the socket ever sees it.
# An unshaped attempt (congestion-drops, 2026-07-23) recorded zero raw drops
# on every node even with a 4 KB buffer and heavy iperf: a fresh daemon
# reader keeps up, so the overflow cannot be provoked deterministically. That
# is why the detection edge is tested as a sans-IO unit test.
#
# The recv_buf_size: 4096 override below is kept because the three asserted
# signals were validated with it in place; it is inert for socket overflow
# under this scenario's cap.
logging:
rust_log: "info"
output_dir: "./sim-results"
fips_overrides:
node:
ecn:
enabled: true
transports:
udp:
recv_buf_size: 4096