fips/testing/chaos

Stochastic Network Simulation

Automated network testing for FIPS. Generates random or explicit topologies, spins up Docker containers, and applies configurable stressors (network impairment, link flaps, traffic generation, node churn) over a timed simulation run. Scenarios cover general stress testing, cost-based parent selection, mixed link technologies (fiber/Bluetooth/WiFi), and transport-specific validation (UDP, TCP, Ethernet). Logs are collected and analyzed automatically.

Prerequisites

• Docker with the compose plugin
• Rust toolchain (for building the FIPS binary)
• Python 3 with pyyaml and jinja2 packages

Quick Start

./testing/chaos/scripts/build.sh
./testing/chaos/scripts/chaos.sh smoke-10

Available Scenarios

General stress tests

Random topologies with increasing stressor intensity.

Scenario	Nodes	Topology	Duration	Netem	Link Flaps	Traffic	Node Churn	Bandwidth
smoke-10	10	random_geometric	60s	--	--	--	--	--
chaos-10	10	random_geometric	120s	yes	yes	yes	--	--
churn-10	10	random_geometric	600s	yes	yes	yes	yes	--
churn-20	20	erdos_renyi	600s	yes	yes	yes	yes	yes

• smoke-10: Baseline sanity check. No stressors, just verify tree convergence.
• chaos-10: Network degradation (5-50ms delay, 0-2% loss), link flaps (max 2 down, 10-30s), and iperf traffic (max 3 concurrent). Netem mutates 30% of links every 15-30s between normal and degraded policies.
• churn-10: Extended run with node churn (1 node down at a time, 30-90s). Tests tree re-convergence after node departure/rejoin.
• churn-20: Aggressive scale test. Erdos-Renyi topology, up to 5 nodes down simultaneously, bandwidth tiers (1/10/100/1000 Mbps), protect_connectivity disabled (partitions allowed).

Cost-based parent selection

Explicit topologies with heterogeneous link types (fiber, Bluetooth, WiFi) to test that the spanning tree selects optimal parents based on link cost.

Scenario	Nodes	Shape	Link types	Duration	What it tests
cost-avoidance	4	Diamond	Fiber + Bluetooth	120s	n04 picks fiber parent (n03) over Bluetooth parent (n02)
depth-vs-cost	4	Linear tree	Fiber + Bluetooth	120s	Cost tradeoff: depth vs. Bluetooth link quality
bottleneck-parent	10	Tree with BT	Fiber + Bluetooth	120s	n06 avoids Bluetooth bottleneck via n02, picks fiber via n03
cost-mixed-7node	7	Multi-type tree	Fiber + Bluetooth + WiFi	180s	n06 prefers fiber (n03) over WiFi (n04)
cost-reeval	4	Diamond	Fiber (mutated)	180s	Periodic re-evaluation triggers parent switch (reeval_interval=15s)
cost-stability	4	Diamond	WiFi (all)	180s	Hysteresis prevents flapping when costs vary within 20% band

• cost-avoidance, depth-vs-cost: Minimal scenarios validating the core cost formula. Bluetooth (L2CAP) links use 15-40ms delay and 2-8% loss; fiber uses 1-5ms delay and 0-1% loss.
• bottleneck-parent: Larger topology where some nodes have both fiber and Bluetooth paths to choose from, and one node (n09) is stuck with Bluetooth (no alternative).
• cost-mixed-7node: Three link technologies in one mesh. Traffic enabled.
• cost-reeval: Netem mutation (50% fraction, every 12-18s) degrades random links. FIPS override sets reeval_interval_secs=15 so periodic re-evaluation catches cost asymmetry. Look for trigger=periodic in logs.
• cost-stability: All links are WiFi. Mutation swings costs between slightly_better and slightly_worse — within the hysteresis band. Expect ≤ 5 parent switches over 180s.

Mixed-technology

Larger explicit topologies combining multiple link technologies.

Scenario	Nodes	Link types	Duration	Netem mutation	What it tests
mixed-technology	10	Fiber + Bluetooth + WiFi	180s	20%/30-60s	Tree convergence across heterogeneous link types

Transport-specific

Explicit topologies exercising non-UDP transports.

Scenario	Nodes	Transport	Shape	Duration	Netem	Link Flaps	What it tests
ethernet-only	4	Ethernet	Ring	90s	yes	--	AF_PACKET transport with beacon discovery
ethernet-mesh	6	UDP + Ethernet	Mesh	120s	yes	yes	Mixed UDP/Ethernet, netem mutation + flaps
tcp-only	4	TCP	Ring	90s	yes	--	TCP transport with static peer config
tcp-chain	4	TCP	Chain	90s	yes	--	TCP multi-hop routing through chain
tcp-mesh	6	UDP + TCP	Mesh	120s	yes	yes	Mixed UDP/TCP, netem mutation + flaps

• ethernet-only: 4-node ring on raw Ethernet (AF_PACKET). Peers discovered via beacons, not static config. Minimal netem (1-5ms delay).
• ethernet-mesh: Mirrors tcp-mesh topology but with Ethernet instead of TCP. UDP edges use static config; Ethernet edges use beacon discovery.
• tcp-only: 4-node ring using TCP on port 8443. Tests connect-on-send, FMP framing over TCP, and reconnection. Netem enabled (1-10ms delay, 0-1% loss).
• tcp-chain: 4-node linear chain, all TCP. Tests multi-hop routing over TCP-only mesh.
• tcp-mesh: 6-node mesh with 4 UDP and 3 TCP edges. Both transports use static peer config. Netem mutation (30% fraction, every 20-40s) and link flaps (1 link max, 10-20s down).

Congestion and ECN

Scenarios testing ECN congestion signaling and transport-level congestion detection.

Scenario	Nodes	Topology	Duration	What it tests
congestion-stress	10	Tree	120s	CE marking under kernel drops and MMP loss detection
ecn-ab-on / ecn-ab-off	6	Tree	120s	A/B throughput comparison: ECN enabled vs disabled

• congestion-stress: 10-node tree with 1 Mbps egress bandwidth caps, 5-10% netem loss, and heavy iperf3 traffic. Ingress policing (1000 kbps) and small recv_buf_size (4 KB) trigger both MMP loss detection and SO_RXQ_OVFL kernel socket drops. Validates end-to-end CE propagation: transit nodes detect congestion, set CE flag, destinations receive CE-marked packets, ecn_ce_count reported in MMP.
• ecn-ab-on / ecn-ab-off: Paired scenarios with identical conditions (6-node tree, 10 Mbps egress, 1000 kbps ingress policing, 10ms link delay, 8 KB recv buffer) differing only in ecn.enabled. ecn-ab-test.sh runs both and compares throughput and congestion counters. Initial results: +10.2% recv throughput with ECN enabled.

Ingress Traffic Control

Scenarios can include ingress configuration to simulate upstream bandwidth bottlenecks using tc ingress policing:

ingress:
  enabled: true
  tiers_kbps: [1000]         # per-peer rate limit in kbps
  burst_bytes: 10000         # policer burst allowance

Per-peer u32 filters on the ingress qdisc (parent ffff:) rate-limit inbound packets. Combined with small recv_buf_size, this reliably triggers SO_RXQ_OVFL kernel socket drops for congestion detection testing.

iperf3 JSON Capture

Traffic sessions capture iperf3 results using --json output. Results are collected per-session from containers and saved as iperf3-results.json in the scenario output directory, enabling automated throughput analysis across scenario runs.

CLI Options

Option	Description
-v, --verbose	Enable debug logging
--seed N	Override the scenario's random seed
--duration secs	Override the scenario's duration
--list	List available scenarios

The scenario argument accepts either a name (churn-10) or a file path (scenarios/churn-10.yaml).

Scenario YAML Format

Annotated example based on churn-10.yaml:

scenario:
  name: "churn-10"
  seed: 42                          # deterministic RNG seed
  duration_secs: 600                # total simulation time

topology:
  num_nodes: 10
  algorithm: random_geometric       # or erdos_renyi, chain
  params:
    radius: 0.5                     # algorithm-specific parameter
  ensure_connected: true            # retry until graph is connected
  subnet: "172.20.0.0/24"
  ip_start: 10                      # first node gets .10

netem:
  enabled: true
  default_policy:
    delay_ms: { min: 5, max: 50 }
    jitter_ms: { min: 1, max: 10 }
    loss_pct: { min: 0, max: 2 }
  mutation:
    interval_secs: { min: 20, max: 45 }  # re-roll interval
    fraction: 0.3                         # fraction of links mutated
    policies:                             # named policy profiles
      normal:
        delay_ms: [5, 20]
        loss_pct: [0, 1]
      degraded:
        delay_ms: [50, 100]
        jitter_ms: [10, 30]
        loss_pct: [3, 8]

link_flaps:
  enabled: true
  interval_secs: { min: 30, max: 60 }
  max_down_links: 2
  down_duration_secs: { min: 10, max: 30 }
  protect_connectivity: true        # never partition the graph

traffic:
  enabled: true
  max_concurrent: 3
  interval_secs: { min: 10, max: 30 }
  duration_secs: { min: 5, max: 15 }
  parallel_streams: 4

node_churn:
  enabled: true
  interval_secs: { min: 60, max: 180 }
  max_down_nodes: 1
  down_duration_secs: { min: 30, max: 90 }
  protect_connectivity: true        # never kill the last path

bandwidth:
  enabled: false                    # per-link HTB rate limiting
  tiers_mbps: [1, 10, 100, 1000]   # each link randomly assigned a tier

logging:
  rust_log: "debug"
  output_dir: "./sim-results"

Topology Algorithms

Algorithm	Parameters	Description
random_geometric	radius (default 0.5)	Place nodes in unit square, connect pairs within radius
erdos_renyi	p (default 0.3)	Include each edge independently with probability p
chain	--	Linear chain: n01--n02--...--nN
explicit	adjacency list	Hardcoded edges with optional per-edge transport type

When ensure_connected is true (default), the generator retries up to 50 times to produce a connected graph.

Directed Outbound Configs

The config generator assigns each static-config edge (UDP or TCP) to exactly one node for outbound connection using a BFS spanning tree rooted at the lowest node ID. Tree edges are assigned parent-to-child; non-tree edges are assigned from the lower node ID to the higher. This eliminates the dual-connect race condition where both sides initiate simultaneously, and creates a clear "owning side" for each link — relevant for auto-reconnect testing. Ethernet edges are excluded from static config since they use beacon discovery.

Output

Results written to sim-results/ (configurable via logging.output_dir):

• status.txt -- How the run ended, plus the scenario, the seed and the container names it used; one key=value per line
• analysis.txt -- Summary: panics, errors, sessions, metrics
• metadata.txt -- Seed, node count, edges, adjacency list
• runner.log -- Orchestration events (topology, netem, churn, traffic) with timestamps
• fips-node-nXX.log -- Per-node log output

The status field reads:

• completed -- ran for its configured duration
• interrupted -- a signal cut the run short, so the artifacts are real but describe less time than the scenario asked for
• aborted -- the run raised part way through; same caveat, and runner.log carries the traceback
• setup-failed -- the containers never started
• teardown-failed -- the mesh ran but its logs or analysis could not be produced

A setup-failed directory holds runner.log and status.txt and nothing else. Nothing is harvested, because container names are global to the host and reading them after a failed setup describes whichever run holds them now. So analysis.txt in a result directory is proof that this scenario's own mesh existed. A directory with no status.txt was written before this was the case and says nothing either way.

Exit codes:

• 0 -- Ran to completion, no panics, every assertion passed
• 1 -- The scenario file could not be loaded, or a second interrupt arrived while the first was being handled
• 2 -- Panics found in the collected node logs. Also what the argument parser exits with when it rejects the command line, before any run starts
• 3 -- A post-run assertion failed
• 4 -- Setup, warmup, the simulation loop or teardown raised, so the run did not complete; runner.log carries the traceback

Codes 2 and 3 describe what a mesh that ran did. Code 4 says there is nothing to describe, and takes precedence over both. Code 2 is dual-use: a run that never started cannot have panicked, so read it together with whether runner.log exists.

A run stopped by a signal exits on this same ladder rather than one of its own: what it collected before stopping is still worth reporting, and status.txt says it was cut short. chaos.sh reports 130 for a Ctrl-C of its own accord.

Creating Custom Scenarios

1. Copy an existing scenario from scenarios/.
2. Adjust topology size, algorithm, and stressor parameters.
3. Run with ./testing/chaos/scripts/chaos.sh path/to/custom.yaml.