FIPS mesh-reliability lab

Local reproduction infrastructure for chronic CI integration-test flakiness. The goal: turn "happens occasionally on GitHub Actions" into "reproduces deterministically under controlled local pressure," then fix on bedrock instead of bumping timeouts.

Quick start

Prerequisites — same as testing/ci-local.sh:

Docker daemon reachable.
fips-test:latest and fips-test-app:latest Docker images built. The easiest way to (re)build them is to run testing/ci-local.sh --build-only once after a fresh checkout or after touching the daemon source — the lab itself does not rebuild between reps.
Python 3 with pyyaml and jinja2 installed for the chaos suites (pip3 install --user pyyaml jinja2).
stress-ng on the host for pressure profiles other than idle (sudo apt-get install stress-ng).

Simplest invocation — single rekey rep on the idle profile (no CPU pressure), output under a timestamped subdir of runs/:

bash testing/mesh-lab/run-loop.sh rekey

Twenty reps under the github-runner-equivalent pressure profile (the canonical Phase 1 acceptance-gate shape for the rekey Phase 5 flake class):

bash testing/mesh-lab/run-loop.sh rekey --reps 20 --profile github-runner-equivalent

The harness writes per-rep diagnostics to <runs-base>/runs/<timestamp>/rep-NN/ (raw logs, container state, exit codes) and a per-rep summary.json plus an aggregated <runs-base>/runs/<timestamp>/summary.json at the end. Where <runs-base> lands is controlled by the FIPS_MESH_LAB_RUNS_DIR environment variable (see below); by default it is the in-tree testing/mesh-lab/ directory. Raw artifacts are gitignored — they're big and per-developer. The summary.json shape is compact so triage doesn't require holding the raw log stream.

Suites supported

Initial target set:

rekey, rekey-accept-off, rekey-outbound-only — rekey-suite Phase 5 post-second-rekey connectivity flake class.
nat-lan — two-node NAT-traversal handshake-completion flake class.
bloom-storm — chaos scenario; logged for future opportunistic checks, not in the initial active scope.

Adding more is straightforward — see the dispatch_suite function in run-loop.sh.

Pressure profiles

Defined in pressure-profiles.sh:

idle — no pressure. Baseline; should produce zero failures on a healthy mesh.
light — placeholder. Will be calibrated.
github-runner-equivalent — placeholder. Will be calibrated to approximate the headroom an ubuntu-latest GitHub runner has while also juggling four parallel package-build workflows (estimate: 2-core / ~7 GiB total, so 1 stress-ng worker + memory ballast). The initial calibration target: this profile must reproduce the rekey Phase 5 flake class at ≥20% rate over 20 reps with mechanism-match.
heavy — placeholder. Worst-case pressure for stall-finding work.

Environment-variable knobs

The harness reads three optional environment variables that shape what each rep does, set them in the invoking shell:

FIPS_MESH_LAB_NETEM — netem argument string (e.g. "delay 10ms 5ms 25% loss 1%"). When set, the harness runs tc qdisc add dev eth0 root netem <args> inside each fips-node container after compose up. Bridge-level qdisc on the docker network does not shape inter-container traffic (Linux bridges forward port-to-port without packets traversing the bridge interface's egress qdisc), so per-container egress is the correct injection point.
FIPS_MESH_LAB_TRACE — when set to any non-empty value, the harness layers compose-trace.yml on top of the base + resource- limits compose stack. That override bumps RUST_LOG to trace level on the modules relevant to the rekey-class flake: rekey, handshake, forwarding, session, encrypted, mmp. Use only when capturing primary failure-moment evidence for mechanism investigation — log volume increases substantially. Without this knob, daemon logs only capture state transitions (rekey cutover, K-bit flip), not per-datagram forwarding decisions, which makes evidence collection for routing-state stalls effectively impossible.
FIPS_MESH_LAB_RUNS_DIR — root directory for harness output (the runs/<timestamp>/ tree). When unset, the harness falls back to an in-tree path under testing/mesh-lab/ and prints a warning to stderr naming the variable and the fallback location. Set this to a path outside the source tree (e.g. /var/tmp/fips-mesh-lab or a path on a separate disk) to keep gigabyte-scale per-rep artefacts out of the checkout.

Example:

FIPS_MESH_LAB_TRACE=1 \
FIPS_MESH_LAB_RUNS_DIR=/var/tmp/fips-mesh-lab \
    bash testing/mesh-lab/run-loop.sh rekey-accept-off \
    --reps 20 --profile github-runner-equivalent

Recipes

recipes/<flake-id>.yaml files are commit-pinned reproduction recipes the harness consumes. Each recipe declares the source SHA, the suite, the pressure profile, the rep count, and the expected mechanism-match rate, so a future operator can confirm "yes the lab still reproduces this flake at the documented rate" with one command. None exist yet — they're authored as concrete reproductions surface.

How this differs from `testing/ci-local.sh`

ci-local.sh runs each suite exactly once in sequence (chaos scenarios in parallel up to a job slot count), produces a pass/fail matrix, and is the canonical "did anything regress" gate. The mesh-lab runs the same per-suite test scripts (it does not reimplement them) but in a loop, with deliberate host pressure applied, and with rich per-rep diagnostic capture. They share the Docker images and the test scripts.

When in doubt, debug a single suite via ci-local.sh --only <suite> first to confirm the suite is healthy on idle, then graduate to the mesh-lab when you want to chase a flake.

5.9 KiB Raw Blame History