mirror of https://github.com/jmcorgan/fips.git synced 2026-10-05 19:18:25 +00:00

Files

T

Johnathan Corgan 1cc069f1a2 Gather the host-side IPv6 plane into the ipv6tun module

The TUN adapter, the .fips DNS responder, ICMPv6 generation and TCP MSS
clamping are the host-side IPv6 plane, but the code was split between
src/upper and inline bodies in the node's session handler and lifecycle.
Gather it into one module, src/ipv6tun, and reduce its calls into the
rest of the crate to a small explicit set. This prepares the TUN adapter
to run later as a separate daemon over the native API, and gives
embedders one place to find the host-side surface. Behaviour is
unchanged apart from log target names.

- Rename src/upper to src/ipv6tun. A `pub use ipv6tun as upper;` alias
  keeps every crate::upper:: and fips::upper:: path resolving, so no
  consumer has to change.
- Move hosts.rs whole to src/hosts.rs, a top-level public module: the
  hosts file also serves peer display names, the peer ACL and fipsctl,
  so it is not host-side only. ipv6tun re-exports it for the old path.
- Move the DNS socket helpers (dual-stack bind, IPV6_RECVPKTINFO,
  interface index lookup) from Node into ipv6tun::dns, unchanged.
  Node::mesh_ifindex, which reads the live TUN device name, becomes
  Handles::mesh_ifindex.
- Move ICMPv6 Destination Unreachable and Packet Too Big sending into
  ipv6tun::icmp behind IcmpContext, which borrows the TUN channel, our
  address and the Packet Too Big rate limiter for one use. Packet Too
  Big is still rate limited and Destination Unreachable still is not.
  The discovery lookup timeout hands its queued packets over as one
  no-route report.
- Split handle_tun_outbound. The host-side half, in ipv6tun::outbound,
  validates the packet, makes both Packet Too Big decisions and sends
  the ICMPv6 replies, reaching the mesh through a small Mesh trait Node
  implements. The mesh-side half, Node::send_outbound, stays with the
  pending queue. The checks run in the same order with the same
  thresholds, and Node::handle_tun_outbound remains the entry point.
- Move the TUN and DNS child start and stop bodies into
  ipv6tun::lifecycle, and gather their nine supervisor fields and the
  node's TUN device name into one Handles struct the supervisor holds.
  The supervisor arms, their order and the child-exit reporting are
  unchanged. A TUN still counts as up when it has a device
  name, so an app-owned TUN produces no TUN teardown, and DNS counts as
  up while its task runs. The node passes a new peer-alias base to the
  running responder through Handles::publish_aliases. Node::tun_name,
  tun_tx, dns_local_addr and enable_app_owned_tun keep their behaviour;
  tests install a TUN sender through a test-only Node::install_tun.

Tracing targets follow module paths, so lines from the moved code now
log under fips::ipv6tun::* and fips::hosts instead of fips::upper::*,
fips::node::lifecycle and fips::node::handlers::session. Update the
RUST_LOG example in the MTU diagnosis guide and the test harness
filters that relied on the old targets, and note the rename in the
changelog.

2026-10-05 16:27:45 +00:00

8.9 KiB

Raw Blame History

FIPS mesh-reliability lab

Local reproduction infrastructure for chronic CI integration-test flakiness. The goal: turn "happens occasionally on GitHub Actions" into "reproduces deterministically under controlled local pressure," then fix on bedrock instead of bumping timeouts.

Quick start

Prerequisites — same as testing/ci-local.sh:

Docker daemon reachable.
fips-test:latest and fips-test-app:latest Docker images built. The easiest way to (re)build them is to run testing/ci-local.sh --build-only once after a fresh checkout or after touching the daemon source — the lab itself does not rebuild between reps.
Python 3 with pyyaml and jinja2 installed for the chaos suites (pip3 install --user pyyaml jinja2).
stress-ng on the host for pressure profiles other than idle (sudo apt-get install stress-ng).

Simplest invocation — single rekey rep on the idle profile (no CPU pressure), output under a timestamped subdir of runs/:

bash testing/mesh-lab/run-loop.sh rekey

Twenty reps under the github-runner-equivalent pressure profile (the canonical Phase 1 acceptance-gate shape for the rekey Phase 5 flake class):

bash testing/mesh-lab/run-loop.sh rekey --reps 20 --profile github-runner-equivalent

The harness writes per-rep diagnostics to <runs-base>/runs/<timestamp>/rep-NN/ (raw logs, container state, exit codes) and a per-rep summary.json plus an aggregated <runs-base>/runs/<timestamp>/summary.json at the end. Where <runs-base> lands is controlled by the FIPS_MESH_LAB_RUNS_DIR environment variable (see below); by default it is the in-tree testing/mesh-lab/ directory. Raw artifacts are gitignored — they're big and per-developer. The summary.json shape is compact so triage doesn't require holding the raw log stream.

Suites supported

Initial target set:

rekey, rekey-accept-off, rekey-outbound-only — rekey-suite Phase 5 post-second-rekey connectivity flake class.
nat-lan — two-node NAT-traversal handshake-completion flake class.
bloom-storm — chaos scenario; covers the bloom_send_rate per-node ceiling exceedance class (single node spiking above the ceiling while peers stay well under). Note that chaos uses its own python sim runner (not docker-compose), so the mesh-lab compose-resource-limits.yml and compose-trace.yml overrides do not apply to this suite; per-rep evidence comes from the captured test-output.log and the parsed signature.json (which extracts the bloom_send_rate and min_parent_switches assertion outcomes plus per-node delta distribution).

Adding more is straightforward — see the dispatch_suite function in run-loop.sh.

Pressure profiles

Defined in pressure-profiles.sh:

idle — no pressure. Baseline; should produce zero failures on a healthy mesh.
light — placeholder. Will be calibrated.
github-runner-equivalent — placeholder. Will be calibrated to approximate the headroom an ubuntu-latest GitHub runner has while also juggling four parallel package-build workflows (estimate: 2-core / ~7 GiB total, so 1 stress-ng worker + memory ballast). The initial calibration target: this profile must reproduce the rekey Phase 5 flake class at ≥20% rate over 20 reps with mechanism-match.
heavy — placeholder. Worst-case pressure for stall-finding work.

Environment-variable knobs

The harness reads optional environment variables that shape what each rep does, set them in the invoking shell:

FIPS_MESH_LAB_NETEM — netem argument string (e.g. "delay 10ms 5ms 25% loss 1%"). When set, the harness runs tc qdisc add dev eth0 root netem <args> inside each fips-node container after compose up. Bridge-level qdisc on the docker network does not shape inter-container traffic (Linux bridges forward port-to-port without packets traversing the bridge interface's egress qdisc), so per-container egress is the correct injection point.
FIPS_MESH_LAB_TRACE — when set to any non-empty value, the harness layers a suite-specific trace-RUST_LOG compose override on top of the base stack. Module sets are per-suite:
- rekey / rekey-accept-off / rekey-outbound-only — rekey, handshake, forwarding, session, encrypted, mmp (via compose-trace.yml).
- nat-lan — fips::nostr, transport::udp, node::lifecycle, ipv6tun::lifecycle, handlers::handshake, dataplane::forwarding (via compose-trace-nat.yml, picked up by testing/nat/scripts/nat-test.sh through the FIPS_NAT_EXTRA_COMPOSE env-var hook).
- bloom-storm — no compose override applies; chaos uses its own python sim runner.
Use only when capturing primary failure-moment evidence for mechanism investigation — log volume increases substantially. Without this knob, daemon logs only capture state transitions and not per-datagram forwarding decisions, which makes evidence collection for routing-state stalls effectively impossible.
FIPS_BLOOM_STORM_CPUSET — comma-separated CPU set for the bloom-storm dispatch's container-pinning sidecar (default 0,1). The sidecar polls for fips-* containers as the chaos sim spawns them and applies docker update --cpuset-cpus <set> to each, mimicking the 2-core constraint of a GHA ubuntu-latest runner. Set to a wider set (e.g. 0,1,2,3) to relax, or to the empty string to disable the sidecar entirely. Only applies to the bloom-storm suite; other suites ignore it.
FIPS_NAT_LAN_CPUSET — comma-separated CPU set for the nat-lan dispatch's container-pinning sidecar (default 0,1). Same shape as FIPS_BLOOM_STORM_CPUSET, but the sidecar polls for fips-nat-lan-* containers and applies the cpuset as the compose-up creates them. The mesh-lab compose-resource-limits.yml overlay is rekey-family service-name-specific (services rekey-* / rekey-accept-off-* / rekey-outbound-only-*), so it does NOT constrain the nat-lan containers; the sidecar fills that gap. Only applies to the nat-lan suite; other suites ignore it.
FIPS_MESH_LAB_TRACE_TREE — when set to any non-empty value, layers compose-trace-tree.yml over the rekey-family compose stack to bump RUST_LOG to trace level on fips::node::tree, fips::tree, fips::node::handlers::mmp, and fips::node::handlers::handshake. Distinct from FIPS_MESH_LAB_TRACE (rekey/forwarding/session/encrypted at trace); targeted at tree-partition race investigation during multi-peer startup. Mutually exclusive with FIPS_MESH_LAB_TRACE in practice — both env vars layer their overlay, but the second one's per-service environment replaces the first's. Only applies to the rekey-family suites.
FIPS_MESH_LAB_NO_RESOURCE_LIMITS — when set to any non-empty value, omits the compose-resource-limits.yml overlay for rekey-family runs. Default behaviour keeps the overlay engaged so rekey-family lab reps stay pressure-matched to a GHA ubuntu-latest runner. Set this for unconstrained characterization where the goal is to surface a race or scheduling artefact rather than reproduce CI pressure. Only applies to the rekey-family suites; other suites ignore it.
FIPS_MESH_LAB_RUNS_DIR — root directory for harness output (the runs/<timestamp>/ tree). When unset, the harness falls back to an in-tree path under testing/mesh-lab/ and prints a warning to stderr naming the variable and the fallback location. Set this to a path outside the source tree (e.g. /var/tmp/fips-mesh-lab or a path on a separate disk) to keep gigabyte-scale per-rep artefacts out of the checkout.

Example:

FIPS_MESH_LAB_TRACE=1 \
FIPS_MESH_LAB_RUNS_DIR=/var/tmp/fips-mesh-lab \
    bash testing/mesh-lab/run-loop.sh rekey-accept-off \
    --reps 20 --profile github-runner-equivalent

Recipes

recipes/<flake-id>.yaml files are commit-pinned reproduction recipes the harness consumes. Each recipe declares the source SHA, the suite, the pressure profile, the rep count, and the expected mechanism-match rate, so a future operator can confirm "yes the lab still reproduces this flake at the documented rate" with one command. None exist yet — they're authored as concrete reproductions surface.

How this differs from `testing/ci-local.sh`

ci-local.sh runs each suite exactly once in sequence (chaos scenarios in parallel up to a job slot count), produces a pass/fail matrix, and is the canonical "did anything regress" gate. The mesh-lab runs the same per-suite test scripts (it does not reimplement them) but in a loop, with deliberate host pressure applied, and with rich per-rep diagnostic capture. They share the Docker images and the test scripts.

When in doubt, debug a single suite via ci-local.sh --only <suite> first to confirm the suite is healthy on idle, then graduate to the mesh-lab when you want to chase a flake.

8.9 KiB Raw Blame History