fips/testing/mesh-lab

FIPS mesh-reliability lab

Local reproduction infrastructure for chronic CI integration-test flakiness. The goal: turn "happens occasionally on GitHub Actions" into "reproduces deterministically under controlled local pressure," then fix on bedrock instead of bumping timeouts.

Quick start

Prerequisites — same as testing/ci-local.sh:

• Docker daemon reachable.
• fips-test:latest and fips-test-app:latest Docker images built. The easiest way to (re)build them is to run testing/ci-local.sh --build-only once after a fresh checkout or after touching the daemon source — the lab itself does not rebuild between reps.
• Python 3 with pyyaml and jinja2 installed for the chaos suites (pip3 install --user pyyaml jinja2).
• stress-ng on the host for pressure profiles other than idle (sudo apt-get install stress-ng).

Simplest invocation — single rekey rep on the idle profile (no CPU pressure), output under a timestamped subdir of runs/:

bash testing/mesh-lab/run-loop.sh rekey

Twenty reps under the github-runner-equivalent pressure profile (the canonical Phase 1 acceptance-gate shape for the rekey Phase 5 flake class):

bash testing/mesh-lab/run-loop.sh rekey --reps 20 --profile github-runner-equivalent

The harness writes per-rep diagnostics to <runs-base>/runs/<timestamp>/rep-NN/ (raw logs, container state, exit codes) and a per-rep summary.json plus an aggregated <runs-base>/runs/<timestamp>/summary.json at the end. Where <runs-base> lands is controlled by the FIPS_MESH_LAB_RUNS_DIR environment variable (see below); by default it is the in-tree testing/mesh-lab/ directory. Raw artifacts are gitignored — they're big and per-developer. The summary.json shape is compact so triage doesn't require holding the raw log stream.

Suites supported

Initial target set:

• rekey, rekey-accept-off, rekey-outbound-only — rekey-suite Phase 5 post-second-rekey connectivity flake class.
• nat-lan — two-node NAT-traversal handshake-completion flake class.
• bloom-storm — chaos scenario; covers the bloom_send_rate per-node ceiling exceedance class (single node spiking above the ceiling while peers stay well under). Note that chaos uses its own python sim runner (not docker-compose), so the mesh-lab compose-resource-limits.yml and compose-trace.yml overrides do not apply to this suite; per-rep evidence comes from the captured test-output.log and the parsed signature.json (which extracts the bloom_send_rate and min_parent_switches assertion outcomes plus per-node delta distribution).

Adding more is straightforward — see the dispatch_suite function in run-loop.sh.

Pressure profiles

Defined in pressure-profiles.sh:

• idle — no pressure. Baseline; should produce zero failures on a healthy mesh.
• light — placeholder. Will be calibrated.
• github-runner-equivalent — placeholder. Will be calibrated to approximate the headroom an ubuntu-latest GitHub runner has while also juggling four parallel package-build workflows (estimate: 2-core / ~7 GiB total, so 1 stress-ng worker + memory ballast). The initial calibration target: this profile must reproduce the rekey Phase 5 flake class at ≥20% rate over 20 reps with mechanism-match.
• heavy — placeholder. Worst-case pressure for stall-finding work.

Environment-variable knobs

The harness reads three optional environment variables that shape what each rep does, set them in the invoking shell:

• FIPS_MESH_LAB_NETEM — netem argument string (e.g. "delay 10ms 5ms 25% loss 1%"). When set, the harness runs tc qdisc add dev eth0 root netem <args> inside each fips-node container after compose up. Bridge-level qdisc on the docker network does not shape inter-container traffic (Linux bridges forward port-to-port without packets traversing the bridge interface's egress qdisc), so per-container egress is the correct injection point.

• FIPS_MESH_LAB_TRACE — when set to any non-empty value, the harness layers a suite-specific trace-RUST_LOG compose override on top of the base stack. Module sets are per-suite:

  • rekey / rekey-accept-off / rekey-outbound-only — rekey, handshake, forwarding, session, encrypted, mmp (via compose-trace.yml).
  • nat-lan — discovery::nostr, transport::udp, node::lifecycle, handlers::handshake, dataplane::forwarding (via compose-trace-nat.yml, picked up by testing/nat/scripts/nat-test.sh through the FIPS_NAT_EXTRA_COMPOSE env-var hook).
  • bloom-storm — no compose override applies; chaos uses its own python sim runner.

  Use only when capturing primary failure-moment evidence for mechanism investigation — log volume increases substantially. Without this knob, daemon logs only capture state transitions and not per-datagram forwarding decisions, which makes evidence collection for routing-state stalls effectively impossible.

• FIPS_BLOOM_STORM_CPUSET — comma-separated CPU set for the bloom-storm dispatch's container-pinning sidecar (default 0,1). The sidecar polls for fips-* containers as the chaos sim spawns them and applies docker update --cpuset-cpus <set> to each, mimicking the 2-core constraint of a GHA ubuntu-latest runner. Set to a wider set (e.g. 0,1,2,3) to relax, or to the empty string to disable the sidecar entirely. Only applies to the bloom-storm suite; other suites ignore it.

• FIPS_NAT_LAN_CPUSET — comma-separated CPU set for the nat-lan dispatch's container-pinning sidecar (default 0,1). Same shape as FIPS_BLOOM_STORM_CPUSET, but the sidecar polls for fips-nat-lan-* containers and applies the cpuset as the compose-up creates them. The mesh-lab compose-resource-limits.yml overlay is rekey-family service-name-specific (services rekey-* / rekey-accept-off-* / rekey-outbound-only-*), so it does NOT constrain the nat-lan containers; the sidecar fills that gap. Only applies to the nat-lan suite; other suites ignore it.

• FIPS_MESH_LAB_TRACE_TREE — when set to any non-empty value, layers compose-trace-tree.yml over the rekey-family compose stack to bump RUST_LOG to trace level on fips::node::tree, fips::tree, fips::node::handlers::mmp, and fips::node::handlers::handshake. Distinct from FIPS_MESH_LAB_TRACE (rekey/forwarding/session/encrypted at trace); targeted at tree-partition race investigation during multi-peer startup. Mutually exclusive with FIPS_MESH_LAB_TRACE in practice — both env vars layer their overlay, but the second one's per-service environment replaces the first's. Only applies to the rekey-family suites.

• FIPS_MESH_LAB_NO_RESOURCE_LIMITS — when set to any non-empty value, omits the compose-resource-limits.yml overlay for rekey-family runs. Default behaviour keeps the overlay engaged so rekey-family lab reps stay pressure-matched to a GHA ubuntu-latest runner. Set this for unconstrained characterization where the goal is to surface a race or scheduling artefact rather than reproduce CI pressure. Only applies to the rekey-family suites; other suites ignore it.

• FIPS_MESH_LAB_RUNS_DIR — root directory for harness output (the runs/<timestamp>/ tree). When unset, the harness falls back to an in-tree path under testing/mesh-lab/ and prints a warning to stderr naming the variable and the fallback location. Set this to a path outside the source tree (e.g. /var/tmp/fips-mesh-lab or a path on a separate disk) to keep gigabyte-scale per-rep artefacts out of the checkout.

Example:

FIPS_MESH_LAB_TRACE=1 \
FIPS_MESH_LAB_RUNS_DIR=/var/tmp/fips-mesh-lab \
    bash testing/mesh-lab/run-loop.sh rekey-accept-off \
    --reps 20 --profile github-runner-equivalent

Recipes

recipes/<flake-id>.yaml files are commit-pinned reproduction recipes the harness consumes. Each recipe declares the source SHA, the suite, the pressure profile, the rep count, and the expected mechanism-match rate, so a future operator can confirm "yes the lab still reproduces this flake at the documented rate" with one command. None exist yet — they're authored as concrete reproductions surface.

How this differs from testing/ci-local.sh

ci-local.sh runs each suite exactly once in sequence (chaos scenarios in parallel up to a job slot count), produces a pass/fail matrix, and is the canonical "did anything regress" gate. The mesh-lab runs the same per-suite test scripts (it does not reimplement them) but in a loop, with deliberate host pressure applied, and with rich per-rep diagnostic capture. They share the Docker images and the test scripts.

When in doubt, debug a single suite via ci-local.sh --only <suite> first to confirm the suite is healthy on idle, then graduate to the mesh-lab when you want to chase a flake.