mirror of
https://github.com/jmcorgan/fips.git
synced 2026-08-09 16:24:45 +00:00
Johnathan Corgan
434b9726aa
Reorganize the node module tree by concept rather than by
message-handling verb, as the first step of the node runtime
decomposition. Pure relocation: no wire, config, metric, or log
semantics change; the lib test count is unchanged (1577 passed).
Moves (git mv, 100% rename similarity):
- handlers/{forwarding,rx_loop,connected_udp,dispatch,encrypted}.rs
-> node/dataplane/ — the whole RX hot path (the select! run loop,
transit/local forwarding, the link-message router, the RX decrypt
path with responder K-bit cutover + roam writes, and connected-UDP
fast-path activation) now lives in one home.
- node/session.rs -> node/session/mod.rs — establishes the session
concept home for the data/state types. The message-behavior file
handlers/session.rs stays put for now (folds in with the later FSP
session step).
The IK/XX-divergent establishment files (handlers/{handshake,rekey,
timeout}.rs) and the deferred-home files (handlers/{mmp,lookup}.rs)
deliberately stay in handlers/, to move once rather than twice.
Every module is reached through impl Node methods, so no call site or
re-export shim was needed. Updated in lockstep with the moves: the
module_path!-derived tracing targets in the two mesh-lab compose-trace
overlays, a structural test's include_str! source path, doc-comments
in proto/routing and the mesh-lab docs, and the stale source-location
citations (node/handlers/{forwarding,rx_loop,encrypted}.rs and
node/session.rs) in doc-comments and the discovery design doc.
197 lines
8.9 KiB
Markdown
197 lines
8.9 KiB
Markdown
# FIPS mesh-reliability lab
|
|
|
|
<!-- markdownlint-disable MD013 -->
|
|
|
|
Local reproduction infrastructure for chronic CI integration-test
|
|
flakiness. The goal: turn "happens occasionally on GitHub Actions" into
|
|
"reproduces deterministically under controlled local pressure," then fix
|
|
on bedrock instead of bumping timeouts.
|
|
|
|
## Quick start
|
|
|
|
Prerequisites — same as `testing/ci-local.sh`:
|
|
|
|
- Docker daemon reachable.
|
|
- `fips-test:latest` and `fips-test-app:latest` Docker images built. The
|
|
easiest way to (re)build them is to run `testing/ci-local.sh
|
|
--build-only` once after a fresh checkout or after touching the
|
|
daemon source — the lab itself does not rebuild between reps.
|
|
- Python 3 with `pyyaml` and `jinja2` installed for the chaos suites
|
|
(`pip3 install --user pyyaml jinja2`).
|
|
- `stress-ng` on the host for pressure profiles other than `idle`
|
|
(`sudo apt-get install stress-ng`).
|
|
|
|
Simplest invocation — single rekey rep on the idle profile (no CPU
|
|
pressure), output under a timestamped subdir of `runs/`:
|
|
|
|
```bash
|
|
bash testing/mesh-lab/run-loop.sh rekey
|
|
```
|
|
|
|
Twenty reps under the github-runner-equivalent pressure profile (the
|
|
canonical Phase 1 acceptance-gate shape for the rekey Phase 5
|
|
flake class):
|
|
|
|
```bash
|
|
bash testing/mesh-lab/run-loop.sh rekey --reps 20 --profile github-runner-equivalent
|
|
```
|
|
|
|
The harness writes per-rep diagnostics to
|
|
`<runs-base>/runs/<timestamp>/rep-NN/` (raw logs, container state,
|
|
exit codes) and a per-rep `summary.json` plus an aggregated
|
|
`<runs-base>/runs/<timestamp>/summary.json` at the end. Where
|
|
`<runs-base>` lands is controlled by the `FIPS_MESH_LAB_RUNS_DIR`
|
|
environment variable (see below); by default it is the in-tree
|
|
`testing/mesh-lab/` directory. Raw artifacts are gitignored — they're
|
|
big and per-developer. The `summary.json` shape is compact so triage
|
|
doesn't require holding the raw log stream.
|
|
|
|
## Suites supported
|
|
|
|
Initial target set:
|
|
|
|
- `rekey`, `rekey-accept-off`, `rekey-outbound-only` — rekey-suite
|
|
Phase 5 post-second-rekey connectivity flake class.
|
|
- `nat-lan` — two-node NAT-traversal handshake-completion flake class.
|
|
- `bloom-storm` — chaos scenario; covers the `bloom_send_rate`
|
|
per-node ceiling exceedance class (single node spiking above the
|
|
ceiling while peers stay well under). Note that
|
|
chaos uses its own python sim runner (not docker-compose), so the
|
|
mesh-lab `compose-resource-limits.yml` and `compose-trace.yml`
|
|
overrides do not apply to this suite; per-rep evidence comes from
|
|
the captured `test-output.log` and the parsed `signature.json`
|
|
(which extracts the `bloom_send_rate` and `min_parent_switches`
|
|
assertion outcomes plus per-node delta distribution).
|
|
|
|
Adding more is straightforward — see the `dispatch_suite` function in
|
|
[run-loop.sh](run-loop.sh).
|
|
|
|
## Pressure profiles
|
|
|
|
Defined in [pressure-profiles.sh](pressure-profiles.sh):
|
|
|
|
- `idle` — no pressure. Baseline; should produce zero failures on a
|
|
healthy mesh.
|
|
- `light` — placeholder. Will be calibrated.
|
|
- `github-runner-equivalent` — placeholder. Will be calibrated to
|
|
approximate the headroom an `ubuntu-latest` GitHub runner has while
|
|
also juggling four parallel package-build workflows (estimate:
|
|
2-core / ~7 GiB total, so 1 stress-ng worker + memory ballast). The
|
|
initial calibration target: this profile must reproduce the rekey
|
|
Phase 5 flake class at ≥20% rate over 20 reps with mechanism-match.
|
|
- `heavy` — placeholder. Worst-case pressure for stall-finding work.
|
|
|
|
## Environment-variable knobs
|
|
|
|
The harness reads three optional environment variables that shape what
|
|
each rep does, set them in the invoking shell:
|
|
|
|
- **`FIPS_MESH_LAB_NETEM`** — netem argument string (e.g.
|
|
`"delay 10ms 5ms 25% loss 1%"`). When set, the harness runs
|
|
`tc qdisc add dev eth0 root netem <args>` inside each fips-node
|
|
container after `compose up`. Bridge-level qdisc on the docker
|
|
network does *not* shape inter-container traffic (Linux bridges
|
|
forward port-to-port without packets traversing the bridge
|
|
interface's egress qdisc), so per-container egress is the correct
|
|
injection point.
|
|
|
|
- **`FIPS_MESH_LAB_TRACE`** — when set to any non-empty value, the
|
|
harness layers a suite-specific trace-RUST_LOG compose override
|
|
on top of the base stack. Module sets are per-suite:
|
|
- rekey / rekey-accept-off / rekey-outbound-only — `rekey`,
|
|
`handshake`, `forwarding`, `session`, `encrypted`, `mmp`
|
|
(via `compose-trace.yml`).
|
|
- nat-lan — `discovery::nostr`, `transport::udp`,
|
|
`node::lifecycle`, `handlers::handshake`, `dataplane::forwarding`
|
|
(via `compose-trace-nat.yml`, picked up by
|
|
`testing/nat/scripts/nat-test.sh` through the
|
|
`FIPS_NAT_EXTRA_COMPOSE` env-var hook).
|
|
- bloom-storm — no compose override applies; chaos uses its own
|
|
python sim runner.
|
|
|
|
Use only when capturing primary failure-moment evidence for
|
|
mechanism investigation — log volume increases substantially.
|
|
Without this knob, daemon logs only capture state transitions and
|
|
not per-datagram forwarding decisions, which makes evidence
|
|
collection for routing-state stalls effectively impossible.
|
|
|
|
- **`FIPS_BLOOM_STORM_CPUSET`** — comma-separated CPU set for the
|
|
bloom-storm dispatch's container-pinning sidecar (default
|
|
`0,1`). The sidecar polls for `fips-*` containers as the chaos
|
|
sim spawns them and applies `docker update --cpuset-cpus <set>`
|
|
to each, mimicking the 2-core constraint of a GHA
|
|
`ubuntu-latest` runner. Set to a wider set (e.g. `0,1,2,3`) to
|
|
relax, or to the empty string to disable the sidecar entirely.
|
|
Only applies to the `bloom-storm` suite; other suites ignore it.
|
|
|
|
- **`FIPS_NAT_LAN_CPUSET`** — comma-separated CPU set for the
|
|
nat-lan dispatch's container-pinning sidecar (default `0,1`).
|
|
Same shape as `FIPS_BLOOM_STORM_CPUSET`, but the sidecar polls
|
|
for `fips-nat-lan-*` containers and applies the cpuset as the
|
|
compose-up creates them. The mesh-lab
|
|
`compose-resource-limits.yml` overlay is rekey-family
|
|
service-name-specific (services `rekey-*` / `rekey-accept-off-*`
|
|
/ `rekey-outbound-only-*`), so it does NOT constrain the nat-lan
|
|
containers; the sidecar fills that gap. Only applies to the
|
|
`nat-lan` suite; other suites ignore it.
|
|
|
|
- **`FIPS_MESH_LAB_TRACE_TREE`** — when set to any non-empty value,
|
|
layers `compose-trace-tree.yml` over the rekey-family compose stack
|
|
to bump `RUST_LOG` to trace level on `fips::node::tree`,
|
|
`fips::tree`, `fips::node::handlers::mmp`, and
|
|
`fips::node::handlers::handshake`. Distinct from
|
|
`FIPS_MESH_LAB_TRACE` (rekey/forwarding/session/encrypted at trace);
|
|
targeted at tree-partition race investigation during multi-peer
|
|
startup. Mutually exclusive with `FIPS_MESH_LAB_TRACE` in practice —
|
|
both env vars layer their overlay, but the second one's per-service
|
|
environment replaces the first's. Only applies to the rekey-family
|
|
suites.
|
|
|
|
- **`FIPS_MESH_LAB_NO_RESOURCE_LIMITS`** — when set to any non-empty
|
|
value, omits the `compose-resource-limits.yml` overlay for rekey-family
|
|
runs. Default behaviour keeps the overlay engaged so rekey-family lab
|
|
reps stay pressure-matched to a GHA `ubuntu-latest` runner. Set this
|
|
for unconstrained characterization where the goal is to surface a race
|
|
or scheduling artefact rather than reproduce CI pressure. Only applies
|
|
to the rekey-family suites; other suites ignore it.
|
|
|
|
- **`FIPS_MESH_LAB_RUNS_DIR`** — root directory for harness output
|
|
(the `runs/<timestamp>/` tree). When unset, the harness falls back
|
|
to an in-tree path under `testing/mesh-lab/` and prints a warning
|
|
to stderr naming the variable and the fallback location. Set this
|
|
to a path outside the source tree (e.g. `/var/tmp/fips-mesh-lab`
|
|
or a path on a separate disk) to keep gigabyte-scale per-rep
|
|
artefacts out of the checkout.
|
|
|
|
Example:
|
|
|
|
```bash
|
|
FIPS_MESH_LAB_TRACE=1 \
|
|
FIPS_MESH_LAB_RUNS_DIR=/var/tmp/fips-mesh-lab \
|
|
bash testing/mesh-lab/run-loop.sh rekey-accept-off \
|
|
--reps 20 --profile github-runner-equivalent
|
|
```
|
|
|
|
## Recipes
|
|
|
|
`recipes/<flake-id>.yaml` files are commit-pinned reproduction recipes
|
|
the harness consumes. Each recipe declares the source SHA, the suite,
|
|
the pressure profile, the rep count, and the expected mechanism-match
|
|
rate, so a future operator can confirm "yes the lab still reproduces
|
|
this flake at the documented rate" with one command. None exist yet —
|
|
they're authored as concrete reproductions surface.
|
|
|
|
## How this differs from `testing/ci-local.sh`
|
|
|
|
`ci-local.sh` runs each suite exactly once in sequence (chaos
|
|
scenarios in parallel up to a job slot count), produces a pass/fail
|
|
matrix, and is the canonical "did anything regress" gate. The
|
|
mesh-lab runs the *same* per-suite test scripts (it does not
|
|
reimplement them) but in a loop, with deliberate host pressure
|
|
applied, and with rich per-rep diagnostic capture. They share the
|
|
Docker images and the test scripts.
|
|
|
|
When in doubt, debug a single suite via `ci-local.sh --only <suite>`
|
|
first to confirm the suite is healthy on idle, then graduate to the
|
|
mesh-lab when you want to chase a flake.
|