fips/testing/mesh-lab/README.md

# FIPS mesh-reliability lab

<!-- markdownlint-disable MD013 -->

Local reproduction infrastructure for chronic CI integration-test
flakiness. The goal: turn "happens occasionally on GitHub Actions" into
"reproduces deterministically under controlled local pressure," then fix
on bedrock instead of bumping timeouts.

## Quick start

Prerequisites — same as `testing/ci-local.sh`:

- Docker daemon reachable.
- `fips-test:latest` and `fips-test-app:latest` Docker images built. The
  easiest way to (re)build them is to run `testing/ci-local.sh
  --build-only` once after a fresh checkout or after touching the
  daemon source — the lab itself does not rebuild between reps.
- Python 3 with `pyyaml` and `jinja2` installed for the chaos suites
  (`pip3 install --user pyyaml jinja2`).
- `stress-ng` on the host for pressure profiles other than `idle`
  (`sudo apt-get install stress-ng`).

Simplest invocation — single rekey rep on the idle profile (no CPU
pressure), output under a timestamped subdir of `runs/`:

```bash
bash testing/mesh-lab/run-loop.sh rekey
```

Twenty reps under the github-runner-equivalent pressure profile (the
canonical Phase 1 acceptance-gate shape for the rekey Phase 5
flake class):

```bash
bash testing/mesh-lab/run-loop.sh rekey --reps 20 --profile github-runner-equivalent
```

The harness writes per-rep diagnostics to
`<runs-base>/runs/<timestamp>/rep-NN/` (raw logs, container state,
exit codes) and a per-rep `summary.json` plus an aggregated
`<runs-base>/runs/<timestamp>/summary.json` at the end. Where
`<runs-base>` lands is controlled by the `FIPS_MESH_LAB_RUNS_DIR`
environment variable (see below); by default it is the in-tree
`testing/mesh-lab/` directory. Raw artifacts are gitignored — they're
big and per-developer. The `summary.json` shape is compact so triage
doesn't require holding the raw log stream.

## Suites supported

Initial target set:

- `rekey`, `rekey-accept-off`, `rekey-outbound-only` — rekey-suite
  Phase 5 post-second-rekey connectivity flake class.
- `nat-lan` — two-node NAT-traversal handshake-completion flake class.
- `bloom-storm` — chaos scenario; logged for future opportunistic
  checks, not in the initial active scope.

Adding more is straightforward — see the `dispatch_suite` function in
[run-loop.sh](run-loop.sh).

## Pressure profiles

Defined in [pressure-profiles.sh](pressure-profiles.sh):

- `idle` — no pressure. Baseline; should produce zero failures on a
  healthy mesh.
- `light` — placeholder. Will be calibrated.
- `github-runner-equivalent` — placeholder. Will be calibrated to
  approximate the headroom an `ubuntu-latest` GitHub runner has while
  also juggling four parallel package-build workflows (estimate:
  2-core / ~7 GiB total, so 1 stress-ng worker + memory ballast). The
  initial calibration target: this profile must reproduce the rekey
  Phase 5 flake class at ≥20% rate over 20 reps with mechanism-match.
- `heavy` — placeholder. Worst-case pressure for stall-finding work.

## Environment-variable knobs

The harness reads three optional environment variables that shape what
each rep does, set them in the invoking shell:

- **`FIPS_MESH_LAB_NETEM`** — netem argument string (e.g.
  `"delay 10ms 5ms 25% loss 1%"`). When set, the harness runs
  `tc qdisc add dev eth0 root netem <args>` inside each fips-node
  container after `compose up`. Bridge-level qdisc on the docker
  network does *not* shape inter-container traffic (Linux bridges
  forward port-to-port without packets traversing the bridge
  interface's egress qdisc), so per-container egress is the correct
  injection point.

- **`FIPS_MESH_LAB_TRACE`** — when set to any non-empty value, the
  harness layers `compose-trace.yml` on top of the base + resource-
  limits compose stack. That override bumps `RUST_LOG` to trace
  level on the modules relevant to the rekey-class flake: `rekey`,
  `handshake`, `forwarding`, `session`, `encrypted`, `mmp`. Use
  only when capturing primary failure-moment evidence for mechanism
  investigation — log volume increases substantially. Without this
  knob, daemon logs only capture state transitions (rekey cutover,
  K-bit flip), not per-datagram forwarding decisions, which makes
  evidence collection for routing-state stalls effectively
  impossible.

- **`FIPS_MESH_LAB_RUNS_DIR`** — root directory for harness output
  (the `runs/<timestamp>/` tree). When unset, the harness falls back
  to an in-tree path under `testing/mesh-lab/` and prints a warning
  to stderr naming the variable and the fallback location. Set this
  to a path outside the source tree (e.g. `/var/tmp/fips-mesh-lab`
  or a path on a separate disk) to keep gigabyte-scale per-rep
  artefacts out of the checkout.

Example:

```bash
FIPS_MESH_LAB_TRACE=1 \
FIPS_MESH_LAB_RUNS_DIR=/var/tmp/fips-mesh-lab \
    bash testing/mesh-lab/run-loop.sh rekey-accept-off \
    --reps 20 --profile github-runner-equivalent
```

## Recipes

`recipes/<flake-id>.yaml` files are commit-pinned reproduction recipes
the harness consumes. Each recipe declares the source SHA, the suite,
the pressure profile, the rep count, and the expected mechanism-match
rate, so a future operator can confirm "yes the lab still reproduces
this flake at the documented rate" with one command. None exist yet —
they're authored as concrete reproductions surface.

## How this differs from `testing/ci-local.sh`

`ci-local.sh` runs each suite exactly once in sequence (chaos
scenarios in parallel up to a job slot count), produces a pass/fail
matrix, and is the canonical "did anything regress" gate. The
mesh-lab runs the *same* per-suite test scripts (it does not
reimplement them) but in a loop, with deliberate host pressure
applied, and with rich per-rep diagnostic capture. They share the
Docker images and the test scripts.

When in doubt, debug a single suite via `ci-local.sh --only <suite>`
first to confirm the suite is healthy on idle, then graduate to the
mesh-lab when you want to chase a flake.