mirror of
https://github.com/jmcorgan/fips.git
synced 2026-07-30 19:46:15 +00:00
The cost-selection chaos scenarios (cost-reeval, cost-avoidance, cost-stability, depth-vs-cost, mixed-technology, bottleneck-parent) tested TreeState::evaluate_parent's decision logic through a Docker mesh that could not exercise it reliably: the tree roots at whichever node holds the smallest NodeAddr, MMP link costs take several measurement windows to settle, and the parent hold-down plus hysteresis timing all confound the outcome. A deterministic link-cost flap still produced zero periodic parent switches in a full run. Replace those six scenarios with deterministic unit tests in src/tree/tests.rs that drive evaluate_parent directly: cheaper-link selection at equal depth, switch-on-cost-change, hysteresis suppressing a marginal change while allowing a significant one, and the depth-versus-cost effective-depth tradeoff. Each is constructed so that breaking the cost or hysteresis logic makes it fail. The congestion kernel-drop signal (SO_RXQ_OVFL) cannot be provoked deterministically in Docker: a fresh daemon reader keeps up with container-speed traffic, so the socket receive queue never overflows (an unshaped run with a 4 KB buffer and heavy traffic recorded zero drops on every node). Extract the drop-detection edge -- read the cumulative counter, fire an event only on the transition into a new drop burst -- into TransportDropState::observe_drops and unit-test it directly. congestion-stress keeps its ECN and MMP congestion-signal assertions, which do need the real shaped bottleneck queue. Remove the retired scenarios from both CI runners and update the chaos README.
160 lines
5.4 KiB
YAML
160 lines
5.4 KiB
YAML
# Congestion Stress: exercise both MMP-based and kernel-drop congestion detection
|
|
#
|
|
# Topology: 10-node tree with all links bandwidth-limited to 1 Mbps
|
|
# (egress HTB + ingress policing). Heavy iperf3 traffic (8 concurrent
|
|
# sessions, 8 parallel streams) combined with 5-10% netem loss.
|
|
#
|
|
# Congestion detection signals exercised:
|
|
# 1. MMP loss detection: netem loss exceeds the 5% loss_threshold,
|
|
# triggering detect_congestion() via MMP metrics on transit nodes.
|
|
# 2. Ingress policing: tc policer on the receive side drops excess
|
|
# inbound packets, creating bursty arrival patterns.
|
|
# 3. ECN CE marking under the shaped bottleneck queue.
|
|
#
|
|
# The kernel socket-drop signal (SO_RXQ_OVFL) is NOT exercised here. This
|
|
# scenario's 1 Mbps cap and ingress policer, which its ECN/MMP signals
|
|
# require, make socket overflow impossible. It also cannot be provoked
|
|
# deterministically anywhere in Docker — a fresh daemon reader keeps up
|
|
# with container-speed traffic — so the FIPS drop-detection logic is
|
|
# unit-tested directly in src/node/tests/unit.rs instead. See the note at
|
|
# the assertions block below.
|
|
#
|
|
# ECN is explicitly enabled via fips_overrides.
|
|
#
|
|
# Success criteria (verified via post-run congestion snapshot):
|
|
# - congestion_detected > 0 on at least one forwarding node
|
|
# - ce_forwarded > 0 on transit nodes
|
|
# - ce_received > 0 on destination nodes
|
|
#
|
|
# Topology:
|
|
#
|
|
# n01 (root)
|
|
# / \
|
|
# n02 n03
|
|
# / \ / \
|
|
# n04 n05 n06 n07
|
|
# | |
|
|
# n08 n09
|
|
# |
|
|
# n10
|
|
|
|
scenario:
|
|
name: "congestion-stress"
|
|
seed: 7
|
|
duration_secs: 120
|
|
|
|
topology:
|
|
algorithm: explicit
|
|
num_nodes: 10
|
|
params:
|
|
adjacency:
|
|
- [n01, n02]
|
|
- [n01, n03]
|
|
- [n02, n04]
|
|
- [n02, n05]
|
|
- [n03, n06]
|
|
- [n03, n07]
|
|
- [n04, n08]
|
|
- [n05, n09]
|
|
- [n08, n10]
|
|
subnet: "172.20.0.0/24"
|
|
ip_start: 10
|
|
|
|
bandwidth:
|
|
enabled: true
|
|
tiers_mbps: [1]
|
|
|
|
ingress:
|
|
enabled: true
|
|
tiers_kbps: [1000]
|
|
burst_bytes: 16000
|
|
|
|
netem:
|
|
enabled: true
|
|
default_policy:
|
|
delay_ms: [2, 5]
|
|
jitter_ms: [0, 1]
|
|
loss_pct: [5, 10]
|
|
mutation:
|
|
interval_secs: {min: 9999, max: 9999}
|
|
fraction: 0.0
|
|
policies:
|
|
baseline:
|
|
delay_ms: [2, 5]
|
|
loss_pct: [5, 10]
|
|
|
|
link_flaps:
|
|
enabled: false
|
|
|
|
traffic:
|
|
enabled: true
|
|
max_concurrent: 8
|
|
interval_secs: {min: 2, max: 5}
|
|
duration_secs: {min: 30, max: 60}
|
|
parallel_streams: 8
|
|
|
|
node_churn:
|
|
enabled: false
|
|
|
|
# Three of the four success criteria above, encoded. Until now all four
|
|
# existed only as that comment and were checked by nothing, so the scenario
|
|
# could not fail on any of them.
|
|
#
|
|
# The floors are 1 node each because that is what the criteria say ("on at
|
|
# least one forwarding node", "on transit nodes", "on destination nodes").
|
|
# They are deliberately not tightened to the observed counts: the criterion
|
|
# is the spec, and a floor invented from six runs would assert something
|
|
# nobody wrote down.
|
|
#
|
|
# What the floors are worth, from the archived corpus. The six runs that
|
|
# carry a status.txt -- the only ones provably completed, all between
|
|
# 2026-07-22 22:14 and 2026-07-23 02:02 -- meet all three with 5 to 9 nodes
|
|
# reporting each signal, so the margin over a floor of 1 is comfortable.
|
|
# The 176 older runs meet none of them. Those older runs did reach teardown
|
|
# (each wrote an analysis.txt), and ECN landed 2026-03-05, before all but
|
|
# one of them, so they are valid runs that observed no congestion rather
|
|
# than runs that died early. What changed on 2026-07-22 is not established:
|
|
# this file has not been touched since it was created, and neither have
|
|
# netem.py, traffic.py or control.py. Worth knowing before trusting a green
|
|
# result here, and worth its own investigation.
|
|
assertions:
|
|
congestion_signals:
|
|
min_nodes_detected: 1
|
|
min_nodes_ce_forwarded: 1
|
|
min_nodes_ce_received: 1
|
|
|
|
# The kernel socket-drop criterion is NOT asserted here and no longer
|
|
# belongs to this scenario. The FIPS drop-DETECTION logic is unit-tested in
|
|
# src/node/tests/unit.rs (2026-07-23); the kernel dropping datagrams is a
|
|
# kernel behaviour, not FIPS's to test, and could not be provoked in Docker.
|
|
#
|
|
# Why it could never be met here: across all 182 archived runs of this
|
|
# scenario, including the six that meet the three signals above, no node
|
|
# ever reported a non-zero kernel_drop_events. SO_RXQ_OVFL counts datagrams
|
|
# arriving at a FULL receive queue, and the 1 Mbps cap sets the arrival rate
|
|
# to 125 kB/s per link. Linux doubles a requested SO_RCVBUF, so the queue is
|
|
# 8192 bytes and filling it would take ~65 ms of reader stall (~22 ms even at
|
|
# the busiest node's 3 Mbps aggregate). Traffic volume cannot cause the
|
|
# overflow because the volume is capped below the rate the buffer drains, and
|
|
# the ingress policer discards excess in tc before the socket ever sees it.
|
|
# An unshaped attempt (congestion-drops, 2026-07-23) recorded zero raw drops
|
|
# on every node even with a 4 KB buffer and heavy iperf: a fresh daemon
|
|
# reader keeps up, so the overflow cannot be provoked deterministically. That
|
|
# is why the detection edge is tested as a sans-IO unit test.
|
|
#
|
|
# The recv_buf_size: 4096 override below is kept because the three asserted
|
|
# signals were validated with it in place; it is inert for socket overflow
|
|
# under this scenario's cap.
|
|
|
|
logging:
|
|
rust_log: "info"
|
|
output_dir: "./sim-results"
|
|
|
|
fips_overrides:
|
|
node:
|
|
ecn:
|
|
enabled: true
|
|
transports:
|
|
udp:
|
|
recv_buf_size: 4096
|