4 Commits
Author SHA1 Message Date
Johnathan Corgan c412646498 testing: pre-create /run/fips/ in the unified test image
Mirrors systemd's RuntimeDirectory=fips so the daemon's
resolve_default_socket() picks /run/fips/control.sock inside
containers, matching production layout and the path that the
chaos sim harness (testing/chaos/sim/control.py) probes.

Without this, the resolver falls through to
/tmp/fips-control.sock (no /run/fips, no XDG_RUNTIME_DIR), the
daemon binds there, and the harness's hardcoded
/run/fips/control.sock probe returns FileNotFoundError on every
node. chaos-bloom-storm then fails its bloom_send_rate assertion
with start=0 nodes, end=0 nodes; other chaos scenarios pass only
because their tolerances absorb the empty samples.

Production hosts always have /run/fips materialized by the
fips.service unit's RuntimeDirectory directive before the daemon
starts, which is why the regression hit only the containerized
test path.

Verified locally with chaos-bloom-storm: max per-node delta 27
<= ceiling 30 over the trailing 30s window.
2026-05-09 18:22:22 +00:00
Johnathan Corgan 37c2973e2f Test infrastructure overhaul: gateway robustness + full CI coverage
Single combined commit covering five interlocking pieces of test and
CI work that landed during the v0.3.0-prep cycle.

## fips-gateway robustness

- src/bin/fips-gateway.rs DNS upstream probe converted from a 3-second
  hard-fail to a bounded retry loop (5 attempts × 1s timeout, 1s sleep
  between attempts; ~10s worst case). Covers the cold-boot race where
  the daemon's TUN is up but the DNS responder at [::1]:5354 is still
  binding. Each failed attempt logs at INFO. In production the binary's
  retry is the live recovery mechanism; with retry it recovers silently
  instead of relying on Restart=on-failure (~5s blip + spurious ERROR
  per cycle).
- packaging/debian/fips-gateway.service `ExecStartPre` now waits up to
  30 seconds for the daemon's `fips0` TUN to appear before exec'ing
  the gateway binary. Eliminates the cold-boot race where the gateway
  exits with `fips0 interface not found` and recovers via
  `Restart=on-failure`, producing a 5-second blip and a spurious error
  log per restart cycle.
- testing/docker/entrypoint.sh gateway-mode waits up to 30s for the
  daemon's DNS responder to bind [::1]:5354 (probes once per second
  with `dig @::1 -p 5354 ... test.fips`) before exec'ing fips-gateway.
  Belt-and-suspenders with the binary's own retry: in CI we want
  deterministic startup ordering. On timeout, fall through so the
  binary's probe reports the definitive error.

## Test infrastructure DNS bind migration to ::1

After session 359's daemon DNS-bind default flipped from `127.0.0.1`
to `::1` (the production fix for ISSUE-2026-0002), the static-test
infrastructure was carrying a stale workaround that overrode the
default back to IPv4 loopback. The fips-gateway integration test
exposed the divergence: the gateway probes its DNS upstream at
`[::1]:5354` (production default) while the daemon was binding
`127.0.0.1:5354` from the template override — IPv6-explicit sockets
do not accept v4-mapped traffic, so the upstream probe exhausted
retries and the gateway exited.

- Drop the explicit `bind_addr: "127.0.0.1"` line from every test
  config that emits it: testing/static/configs/node.template.yaml,
  testing/chaos/configs/node.template.yaml, the sidecar heredoc in
  testing/docker/entrypoint.sh, testing/acl-allowlist/generate-configs.sh
  (six per-node blocks), testing/nat/scripts/generate-configs.sh, and
  the four tor templates under testing/tor/. Daemon picks up its
  production `::1` default.
- Flip the dnsmasq forwarder for `.fips` in testing/docker/Dockerfile
  from `127.0.0.1#5354` to `::1#5354` so dnsmasq on the shared test
  image continues to reach the daemon. Template and Dockerfile must
  move together since most static suites resolve `<npub>.fips` via
  the test-image dnsmasq.

## rekey-accept-off integration variant + UDP unit test

- New `rekey-accept-off` topology and docker-compose profile under
  testing/static/. 2-node variant where node-b runs with
  `udp.accept_connections: false`. Pins the regression class that
  ISSUE-2026-0004 fixed (cross-connection winner's rekey msg1 was
  being filtered by the accept_connections gate, breaking rekey).
- testing/static/scripts/rekey-test.sh accepts REKEY_TOPOLOGY and
  REKEY_ACCEPT_OFF_NODES env vars; its inject-config subcommand
  applies the per-node `udp.accept_connections: false` edit, and
  the test asserts no sustained "Dual rekey initiation" log lines.
- New UDP variant of `should_admit_msg1` admit-rekey unit test in
  src/node/tests/handshake.rs.

## ci-local.sh full integration coverage

- New runner functions and dispatcher entries for `acl-allowlist`,
  `nat-cone` / `nat-symmetric` / `nat-lan`, `rekey-accept-off`,
  `dns-resolver`, `deb-install`. Each integrates with the existing
  summary tracking via `record`.
- New `--with-tor` flag (off by default) gates `tor-socks5-outbound`
  and `tor-directory-mode` runners. Tor stays opt-in because both
  harnesses depend on the live Tor network and would introduce a
  flake source unrelated to the FIPS code.
- New suite arrays (`ACL_SUITES`, `NAT_SUITES`, `DNS_RESOLVER_SUITES`,
  `DEB_INSTALL_SUITES`, `TOR_SUITES`) drive both the default sweep
  and `--list` output.
- `run_suite` extended to accept the new suite names for `--only`
  invocations.

## GitHub CI matrix expansions

- `gateway` matrix entry runs testing/static/scripts/gateway-test.sh
  against the existing docker-compose `gateway` profile.
- `rekey-accept-off` matrix entry exercises the new topology with
  REKEY_ACCEPT_OFF_NODES=b.
- `deb-install` matrix (debian12 + ubuntu24 + ubuntu26) runs
  testing/deb-install/test.sh with privileged systemd containers.
  ~5-7 min cold cache, ~2 min warm per distro. Self-contained: builds
  its own .deb in a Debian 12 cargo-deb builder image; does not
  depend on the build job's pre-built artifact.
- `dns-resolver` matrix entry runs the full 13-scenario harness
  (per-distro systemd resolver-backend tests + real-fips end-to-end
  scenarios) in a single job. Pins the production DNS bind path that
  ISSUE-2026-0002 lived in. ~7-12 min warm, ~12-15 min cold.

Verified locally: full `bash testing/ci-local.sh` sweep passes,
including 5/5 deb-install distros and all 13 dns-resolver scenarios.
Tor-inclusive sweep (`--with-tor`) verified in a follow-up run.
2026-04-30 10:24:32 +00:00
Johnathan Corgan 60e5fefb1f Implement outbound LAN gateway
Add fips-gateway binary: a separate daemon that allows unmodified LAN
hosts to reach FIPS mesh destinations via DNS-allocated virtual IPs
and kernel nftables NAT.

Gateway DNS resolver: forwarding proxy on [::]:53 that intercepts
.fips queries, forwards to daemon resolver (localhost:5354), allocates
virtual IPs from pool, returns AAAA records. Always sends AAAA upstream
regardless of client query type, returns proper NODATA for non-AAAA.

Virtual IP pool: fd01::/112 pool with state machine lifecycle
(Allocated → Active → Draining → Free), TTL-based reclamation,
conntrack integration for session tracking.

NAT manager: nftables DNAT/SNAT rules via rustables netlink API,
per-mapping rule lifecycle, fips0 masquerade for LAN client source
address rewriting.

Network setup: local pool route, proxy NDP for virtual IPs on LAN
interface, IPv6 forwarding validation.

Control socket at /run/fips/gateway.sock with show_gateway and
show_mappings queries. fipstop Gateway tab with pool summary gauge
and mappings table.

Gateway config section in fips.yaml with pool CIDR, LAN interface,
DNS upstream, TTL, and grace period settings.

Design doc at docs/design/fips-gateway.md.

Integration test (testing/static/scripts/gateway-test.sh): three
containers verifying DNS resolution, end-to-end HTTP, NAT state,
TTL expiration, SERVFAIL fallback, and clean shutdown.
2026-04-09 16:53:32 +00:00
Johnathan Corgan 9e63b42bd9 Consolidate Docker test harness infrastructure
Replace 4 near-identical per-harness Docker setups with unified shared
infrastructure. Net result: -463 lines across 55 files, faster CI
(8.5 min vs ~13.5 min), 14 scenarios (down from 21).

Unified Docker image (testing/docker/):
- Single Dockerfile (trixie-slim) with FIPS_TEST_MODE env var for
  mode dispatch: default, chaos, sidecar, tor-socks5, tor-directory
- Single entrypoint.sh with conditional logic per mode
- Replaces 5 Dockerfiles, 3 entrypoints, 4 resolv.conf copies

Shared build and libraries (testing/scripts/, testing/lib/):
- testing/scripts/build.sh: single build script with macOS zigbuild
  support, replaces 4 per-harness copies
- testing/lib/derive_keys.py: shared key derivation module, replaces
  3 copies of derive-keys.py
- testing/lib/log_analysis.py: shared log analysis extracted from
  chaos sim/logs.py, with CLI interface and rekey cutover tracking
- testing/lib/wait-converge.sh: shared convergence wait helpers
  (wait_for_links, wait_for_peers) using fipsctl JSON polling

Scenario consolidation:
- Remove 7 redundant scenarios: tcp-chain (subsumed by tcp-mesh),
  tcp-only (subsumed by tcp-mesh), chaos-10 (replaced by
  churn-mixed --nodes 10), churn-10/churn-20/churn-20-mixed
  (subsumed by parameterized churn-mixed), cost-mixed-7node
  (overlaps mixed-technology)
- Add churn-mixed scenario with --nodes flag for scale testing
- Reduce idle scenario durations: smoke-10 60s→30s,
  ethernet-only 90s→30s, cost-avoidance 120s→45s,
  depth-vs-cost 120s→45s, bottleneck-parent 120s→60s,
  mixed-technology 180s→90s

CI updates:
- ci-local.sh: unified image build, structured chaos suite entries
  with per-scenario flags, --skip-build for sidecar
- ci.yml: shared binary install + image build step, updated scenario
  matrix, chaos_flags support for parameterized scenarios
- Add tcp-mesh and congestion-stress to CI matrix
- Static ping test uses active peer convergence detection instead
  of hardcoded 5s sleep

Chaos infrastructure improvements (from discovery-rework branch):
- Pre-built Docker image instead of per-service build at scale
- --nodes flag in chaos.sh for runtime topology size override
2026-03-19 18:08:36 +00:00