fips/testing/chaos/README.md

Stochastic Network Simulation

Automated network testing for FIPS. Generates random or explicit topologies, spins up Docker containers, and applies configurable stressors (network impairment, link flaps, traffic generation, node churn) over a timed simulation run. Scenarios cover general stress testing, cost-based parent selection, mixed link technologies (fiber/Bluetooth/WiFi), and transport-specific validation (UDP, TCP, Ethernet). Logs are collected and analyzed automatically.

Prerequisites

• Docker with the compose plugin
• Rust toolchain (for building the FIPS binary)
• Python 3 with pyyaml and jinja2 packages

Quick Start

./testing/chaos/scripts/build.sh
./testing/chaos/scripts/chaos.sh churn-mixed

Available Scenarios

General stress tests

Random topologies with increasing stressor intensity.

Scenario	Nodes	Topology	Duration	Netem	Link Flaps	Traffic	Node Churn	Bandwidth
chaos-10	10	random_geometric	120s	yes	yes	yes	--	--
churn-10	10	random_geometric	600s	yes	yes	yes	yes	--
churn-20	20	erdos_renyi	600s	yes	yes	yes	yes	yes

• chaos-10: Network degradation (5-50ms delay, 0-2% loss), link flaps (max 2 down, 10-30s), and iperf traffic (max 3 concurrent). Netem mutates 30% of links every 15-30s between normal and degraded policies.
• churn-10: Extended run with node churn (1 node down at a time, 30-90s). Tests tree re-convergence after node departure/rejoin.
• churn-20: Aggressive scale test. Erdos-Renyi topology, up to 5 nodes down simultaneously, bandwidth tiers (1/10/100/1000 Mbps), protect_connectivity disabled (partitions allowed).

Cost-based parent selection — retired, now sans-IO unit tests

The cost-selection scenarios (cost-avoidance, depth-vs-cost, bottleneck-parent, cost-reeval, cost-stability, mixed-technology) were retired on 2026-07-23. Their subject was the pure TreeState::evaluate_parent decision — which parent wins on effective_depth = depth + link_cost, when periodic re-evaluation switches, and when hysteresis suppresses a flap. A Docker mesh could not test that reliably: the root is whichever node holds the smallest NodeAddr, MMP costs take several measurement windows to settle, and hold-down plus hysteresis timing all confound the outcome (a deterministic link_swap attempt still produced zero periodic switches in a full run).

That logic is now covered by deterministic sans-IO unit tests in src/tree/tests.rs (test_evaluate_parent_cost_*, ..._hysteresis_*, ..._effective_depth_*), which run in the cargo quartet on every commit and can each be shown to fail by breaking the cost or hysteresis logic.

Transport-specific

Explicit topologies exercising non-UDP transports.

Scenario	Nodes	Transport	Shape	Duration	Netem	Link Flaps	What it tests
ethernet-only	4	Ethernet	Ring	90s	yes	--	AF_PACKET transport with beacon discovery
ethernet-mesh	6	UDP + Ethernet	Mesh	120s	yes	yes	Mixed UDP/Ethernet, netem mutation + flaps
tcp-mesh	6	UDP + TCP	Mesh	120s	yes	yes	Mixed UDP/TCP, netem mutation + flaps

• ethernet-only: 4-node ring on raw Ethernet (AF_PACKET). Peers discovered via beacons, not static config. Minimal netem (1-5ms delay).
• ethernet-mesh: Mirrors tcp-mesh topology but with Ethernet instead of TCP. UDP edges use static config; Ethernet edges use beacon discovery.
• tcp-mesh: 6-node mesh with 4 UDP and 3 TCP edges. Both transports use static peer config. Netem mutation (30% fraction, every 20-40s) and link flaps (1 link max, 10-20s down).

Congestion and ECN

Scenarios testing ECN congestion signaling and transport-level congestion detection.

Scenario	Nodes	Topology	Duration	What it tests
congestion-stress	10	Tree	120s	CE marking under kernel drops and MMP loss detection
ecn-ab-on / ecn-ab-off	6	Tree	120s	A/B throughput comparison: ECN enabled vs disabled

• congestion-stress: 10-node tree with 1 Mbps egress bandwidth caps, 5-10% netem loss, and heavy iperf3 traffic. Ingress policing (1000 kbps) and small recv_buf_size (4 KB) trigger both MMP loss detection and SO_RXQ_OVFL kernel socket drops. Validates end-to-end CE propagation: transit nodes detect congestion, set CE flag, destinations receive CE-marked packets, ecn_ce_count reported in MMP.
• ecn-ab-on / ecn-ab-off: Paired scenarios with identical conditions (6-node tree, 10 Mbps egress, 1000 kbps ingress policing, 10ms link delay, 8 KB recv buffer) differing only in ecn.enabled. ecn-ab-compare.sh runs both and prints a side-by-side of throughput and congestion counters. It is a manual tool, not a test: it asserts nothing and no runner invokes it. The "+10.2% recv throughput with ECN enabled" figure once recorded here is not reproducible from anything on disk — the script read a fixed sim-results/ecn-ab-on/ path while the runner has written timestamped directories since 2026-03-20, and no ecn-ab result directory survives. The path bug is fixed; the figure is left out until a run produces one.

Ingress Traffic Control

Scenarios can include ingress configuration to simulate upstream bandwidth bottlenecks using tc ingress policing:

ingress:
  enabled: true
  tiers_kbps: [1000]         # per-peer rate limit in kbps
  burst_bytes: 10000         # policer burst allowance

Per-peer u32 filters on the ingress qdisc (parent ffff:) rate-limit inbound packets. Combined with small recv_buf_size, this reliably triggers SO_RXQ_OVFL kernel socket drops for congestion detection testing.

iperf3 JSON Capture

Traffic sessions capture iperf3 results using --json output. Results are collected per-session from containers and saved as iperf3-results.json in the scenario output directory, enabling automated throughput analysis across scenario runs.

CLI Options

Option	Description
-v, --verbose	Enable debug logging
--seed N	Override the scenario's random seed
--duration secs	Override the scenario's duration
--list	List available scenarios

The scenario argument accepts either a name (churn-10) or a file path (scenarios/churn-10.yaml).

Scenario YAML Format

Annotated example based on churn-10.yaml:

scenario:
  name: "churn-10"
  seed: 42                          # deterministic RNG seed
  duration_secs: 600                # total simulation time

topology:
  num_nodes: 10
  algorithm: random_geometric       # or erdos_renyi, chain
  params:
    radius: 0.5                     # algorithm-specific parameter
  ensure_connected: true            # retry until graph is connected
  subnet: "172.20.0.0/24"
  ip_start: 10                      # first node gets .10

netem:
  enabled: true
  default_policy:
    delay_ms: { min: 5, max: 50 }
    jitter_ms: { min: 1, max: 10 }
    loss_pct: { min: 0, max: 2 }
  mutation:
    interval_secs: { min: 20, max: 45 }  # re-roll interval
    fraction: 0.3                         # fraction of links mutated
    policies:                             # named policy profiles
      normal:
        delay_ms: [5, 20]
        loss_pct: [0, 1]
      degraded:
        delay_ms: [50, 100]
        jitter_ms: [10, 30]
        loss_pct: [3, 8]

link_flaps:
  enabled: true
  interval_secs: { min: 30, max: 60 }
  max_down_links: 2
  down_duration_secs: { min: 10, max: 30 }
  protect_connectivity: true        # never partition the graph

traffic:
  enabled: true
  max_concurrent: 3
  interval_secs: { min: 10, max: 30 }
  duration_secs: { min: 5, max: 15 }
  parallel_streams: 4

node_churn:
  enabled: true
  interval_secs: { min: 60, max: 180 }
  max_down_nodes: 1
  down_duration_secs: { min: 30, max: 90 }
  protect_connectivity: true        # never kill the last path

bandwidth:
  enabled: false                    # per-link HTB rate limiting
  tiers_mbps: [1, 10, 100, 1000]   # each link randomly assigned a tier

logging:
  rust_log: "debug"
  output_dir: "./sim-results"

Topology Algorithms

Algorithm	Parameters	Description
random_geometric	radius (default 0.5)	Place nodes in unit square, connect pairs within radius
erdos_renyi	p (default 0.3)	Include each edge independently with probability p
chain	--	Linear chain: n01--n02--...--nN
explicit	adjacency list	Hardcoded edges with optional per-edge transport type

When ensure_connected is true (default), the generator retries up to 50 times to produce a connected graph.

Directed Outbound Configs

The config generator assigns each static-config edge (UDP or TCP) to exactly one node for outbound connection using a BFS spanning tree rooted at the lowest node ID. Tree edges are assigned parent-to-child; non-tree edges are assigned from the lower node ID to the higher. This eliminates the dual-connect race condition where both sides initiate simultaneously, and creates a clear "owning side" for each link — relevant for auto-reconnect testing. Ethernet edges are excluded from static config since they use beacon discovery.

Output

Results written to sim-results/ (configurable via logging.output_dir):

• status.txt -- How the run ended, plus the scenario, the seed and the container names it used; one key=value per line
• analysis.txt -- Summary: panics, errors, sessions, metrics
• metadata.txt -- Seed, node count, edges, adjacency list
• runner.log -- Orchestration events (topology, netem, churn, traffic) with timestamps
• fips-node-nXX.log -- Per-node log output

The status field reads:

• completed -- ran for its configured duration
• interrupted -- a signal cut the run short, so the artifacts are real but describe less time than the scenario asked for
• aborted -- the run raised part way through; same caveat, and runner.log carries the traceback
• setup-failed -- the containers never started
• teardown-failed -- the mesh ran but its logs or analysis could not be produced

A setup-failed directory holds runner.log and status.txt and nothing else. Nothing is harvested, because container names are global to the host and reading them after a failed setup describes whichever run holds them now. So analysis.txt in a result directory is proof that this scenario's own mesh existed. A directory with no status.txt was written before this was the case and says nothing either way.

Exit codes:

• 0 -- Ran to completion, no panics, every assertion passed
• 1 -- The scenario file could not be loaded, or a second interrupt arrived while the first was being handled
• 2 -- Panics found in the collected node logs. Also what the argument parser exits with when it rejects the command line, before any run starts
• 3 -- A post-run assertion failed
• 4 -- Setup, warmup, the simulation loop or teardown raised, so the run did not complete; runner.log carries the traceback

Codes 2 and 3 describe what a mesh that ran did. Code 4 says there is nothing to describe, and takes precedence over both. Code 2 is dual-use: a run that never started cannot have panicked, so read it together with whether runner.log exists.

A run stopped by a signal exits on this same ladder rather than one of its own: what it collected before stopping is still worth reporting, and status.txt says it was cut short. chaos.sh reports 130 for a Ctrl-C of its own accord.

Creating Custom Scenarios

1. Copy an existing scenario from scenarios/.
2. Adjust topology size, algorithm, and stressor parameters.
3. Run with ./testing/chaos/scripts/chaos.sh path/to/custom.yaml.