Johnathan Corgan
ce0eb71722
Wires the bloom-storm chaos scenario into the mesh-lab harness as a first-class suite, with optional per-container CPU pinning to mimic GitHub Actions' 2-core ubuntu-latest budget. Dispatch path — three new run-loop.sh functions plus the dispatch_suite and dispatch_mechanism_match case-arm additions: - `run_bloom_storm` invokes `bash testing/chaos/scripts/chaos.sh bloom-storm` and captures stdout+stderr into the rep's test-output.log. Chaos uses its own python sim runner (`python3 -m sim`), not docker-compose, so this suite gets no per-container compose override, no separate `docker logs` capture, and no in-container netem injection — the chaos scenario yaml owns its own netem and link-swap config. - `parse_bloom_storm` extracts the bloom_send_rate result (pass/fail/unknown), ceiling, max-observed per-node delta, offenders list, full per-node delta distribution, the companion min_parent_switches result, and panic + error counts. Lands in the rep's signature.json. Two parser details: assertion greps are anchored on `^(PASS|FAIL)` so they only match the bare end-of-run summary line, not python-logger-prefixed lines that contain the same substring; and `grep -c` panic/error counts use `; true` + a defensive empty-string check instead of the common `|| echo 0` fallback (`grep -c` exits 1 on zero matches while also printing "0", so the fallback would corrupt the count to "0\\n0"). - `mechanism_match_bloom_storm` returns true when a rep both fails the bloom_send_rate assertion and the FAIL line carries a named offender (filtering the harness-side "failed to sample window endpoints" sub-failure out of the mechanism count). CPU-pinning sidecar — bloom-storm's chaos sim spawns containers directly via the docker SDK, so the mesh-lab compose-resource- limits override does not apply. A poll-and-pin loop around the chaos.sh invocation lists \`fips-*\` containers every 0.5 s and applies \`docker update --cpuset-cpus <set>\` to each. Pinning is idempotent (re-applying the same cpuset is a no-op). Default cpuset \`0,1\` mimics the GHA 2-core budget; override via \`FIPS_BLOOM_STORM_CPUSET=<set>\` (any comma-separated CPU list), or set to the empty string to disable. Only applies to the bloom-storm suite; other suites' dispatch paths are unchanged. README's "Suites supported" entry covers the assertion class, and the \`FIPS_BLOOM_STORM_CPUSET\` knob is documented alongside the other mesh-lab env-var knobs.
FIPS Testing
Integration and simulation test harnesses for FIPS, using Docker containers running the full protocol stack.
Test Harnesses
static/ -- Static Docker Network
Fixed topologies with manual scripts for building, config generation, connectivity tests (ping, iperf), and network impairment (netem). Useful for deterministic debugging and validating specific topology configurations.
| Topology | Nodes | Transport | Description |
|---|---|---|---|
| mesh | 5 | UDP | Sparse mesh, 6 links, multi-hop |
| chain | 5 | UDP | Linear chain, max 4-hop paths |
| mesh-public | 5+1 | UDP | Mesh with external public node |
| tcp-chain | 3 | TCP | Linear chain over TCP (port 8443) |
| rekey | 5 | UDP | Rekey integration test topology |
tor/ -- Tor Transport Integration
End-to-end Tor transport testing with Docker containers running real Tor daemons. Requires internet access for Tor bootstrapping.
| Scenario | Description |
|---|---|
| socks5-outbound | Outbound SOCKS5 connections through Tor to clearnet peer |
| directory-mode | Inbound via HiddenServiceDir onion service (co-located) |
nat/ -- NAT Traversal Lab
Real Docker NAT traversal tests for the Nostr/STUN bootstrap path,
using router containers with iptables-based NAT, a local Nostr relay,
and a local STUN responder.
| Scenario | Description |
|---|---|
| cone | Two NATed peers establish a UDP traversal path |
| symmetric | UDP traversal fails under symmetric NAT, TCP fallback wins |
| lan | Peers on the same LAN prefer local addresses over reflexive |
chaos/ -- Stochastic Simulation
Automated network testing with configurable node counts, topology algorithms (random geometric, Erdos-Renyi, chain, explicit), and fault injection (netem mutation, link flaps, traffic generation, node churn). 20 scenarios covering general stress testing, cost-based parent selection, mixed link technologies (fiber/Bluetooth/WiFi), transport-specific validation (UDP, TCP, Ethernet), and ECN/congestion testing. Scenarios are defined in YAML and executed via a Python harness that manages the full lifecycle: topology generation, Docker orchestration, fault scheduling, log collection, and analysis.
interop/ -- Mixed-Version Interop Harness
On-demand harness that runs an N-node full mesh from a node-spec where
each node can run a different build of the FIPS daemon, then attributes
every FMP/FSP/rekey/connectivity failure to a specific version pair
(same-version vs MIXED). Used to catch interop regressions between
builds, not as a per-commit CI gate; not part of ci-local.sh.
mesh-lab/ -- Mesh Reliability Lab
On-demand harness that runs a chosen integration suite N times under a
configurable host-pressure profile (idle / light / github-runner-
equivalent / heavy via stress-ng), per-container netem impairment,
and optional trace-level RUST_LOG, capturing per-rep diagnostics and a
mechanism-match summary across the run. Used for statistical reliability
characterization of known flake classes under calibrated stress, not as
a per-commit gate; not part of ci-local.sh.