Test infrastructure overhaul: gateway robustness + full CI coverage

Single combined commit covering five interlocking pieces of test and
CI work that landed during the v0.3.0-prep cycle.

## fips-gateway robustness

- src/bin/fips-gateway.rs DNS upstream probe converted from a 3-second
  hard-fail to a bounded retry loop (5 attempts × 1s timeout, 1s sleep
  between attempts; ~10s worst case). Covers the cold-boot race where
  the daemon's TUN is up but the DNS responder at [::1]:5354 is still
  binding. Each failed attempt logs at INFO. In production the binary's
  retry is the live recovery mechanism; with retry it recovers silently
  instead of relying on Restart=on-failure (~5s blip + spurious ERROR
  per cycle).
- packaging/debian/fips-gateway.service `ExecStartPre` now waits up to
  30 seconds for the daemon's `fips0` TUN to appear before exec'ing
  the gateway binary. Eliminates the cold-boot race where the gateway
  exits with `fips0 interface not found` and recovers via
  `Restart=on-failure`, producing a 5-second blip and a spurious error
  log per restart cycle.
- testing/docker/entrypoint.sh gateway-mode waits up to 30s for the
  daemon's DNS responder to bind [::1]:5354 (probes once per second
  with `dig @::1 -p 5354 ... test.fips`) before exec'ing fips-gateway.
  Belt-and-suspenders with the binary's own retry: in CI we want
  deterministic startup ordering. On timeout, fall through so the
  binary's probe reports the definitive error.

## Test infrastructure DNS bind migration to ::1

After session 359's daemon DNS-bind default flipped from `127.0.0.1`
to `::1` (the production fix for ISSUE-2026-0002), the static-test
infrastructure was carrying a stale workaround that overrode the
default back to IPv4 loopback. The fips-gateway integration test
exposed the divergence: the gateway probes its DNS upstream at
`[::1]:5354` (production default) while the daemon was binding
`127.0.0.1:5354` from the template override — IPv6-explicit sockets
do not accept v4-mapped traffic, so the upstream probe exhausted
retries and the gateway exited.

- Drop the explicit `bind_addr: "127.0.0.1"` line from every test
  config that emits it: testing/static/configs/node.template.yaml,
  testing/chaos/configs/node.template.yaml, the sidecar heredoc in
  testing/docker/entrypoint.sh, testing/acl-allowlist/generate-configs.sh
  (six per-node blocks), testing/nat/scripts/generate-configs.sh, and
  the four tor templates under testing/tor/. Daemon picks up its
  production `::1` default.
- Flip the dnsmasq forwarder for `.fips` in testing/docker/Dockerfile
  from `127.0.0.1#5354` to `::1#5354` so dnsmasq on the shared test
  image continues to reach the daemon. Template and Dockerfile must
  move together since most static suites resolve `<npub>.fips` via
  the test-image dnsmasq.

## rekey-accept-off integration variant + UDP unit test

- New `rekey-accept-off` topology and docker-compose profile under
  testing/static/. 2-node variant where node-b runs with
  `udp.accept_connections: false`. Pins the regression class that
  ISSUE-2026-0004 fixed (cross-connection winner's rekey msg1 was
  being filtered by the accept_connections gate, breaking rekey).
- testing/static/scripts/rekey-test.sh accepts REKEY_TOPOLOGY and
  REKEY_ACCEPT_OFF_NODES env vars; its inject-config subcommand
  applies the per-node `udp.accept_connections: false` edit, and
  the test asserts no sustained "Dual rekey initiation" log lines.
- New UDP variant of `should_admit_msg1` admit-rekey unit test in
  src/node/tests/handshake.rs.

## ci-local.sh full integration coverage

- New runner functions and dispatcher entries for `acl-allowlist`,
  `nat-cone` / `nat-symmetric` / `nat-lan`, `rekey-accept-off`,
  `dns-resolver`, `deb-install`. Each integrates with the existing
  summary tracking via `record`.
- New `--with-tor` flag (off by default) gates `tor-socks5-outbound`
  and `tor-directory-mode` runners. Tor stays opt-in because both
  harnesses depend on the live Tor network and would introduce a
  flake source unrelated to the FIPS code.
- New suite arrays (`ACL_SUITES`, `NAT_SUITES`, `DNS_RESOLVER_SUITES`,
  `DEB_INSTALL_SUITES`, `TOR_SUITES`) drive both the default sweep
  and `--list` output.
- `run_suite` extended to accept the new suite names for `--only`
  invocations.

## GitHub CI matrix expansions

- `gateway` matrix entry runs testing/static/scripts/gateway-test.sh
  against the existing docker-compose `gateway` profile.
- `rekey-accept-off` matrix entry exercises the new topology with
  REKEY_ACCEPT_OFF_NODES=b.
- `deb-install` matrix (debian12 + ubuntu24 + ubuntu26) runs
  testing/deb-install/test.sh with privileged systemd containers.
  ~5-7 min cold cache, ~2 min warm per distro. Self-contained: builds
  its own .deb in a Debian 12 cargo-deb builder image; does not
  depend on the build job's pre-built artifact.
- `dns-resolver` matrix entry runs the full 13-scenario harness
  (per-distro systemd resolver-backend tests + real-fips end-to-end
  scenarios) in a single job. Pins the production DNS bind path that
  ISSUE-2026-0002 lived in. ~7-12 min warm, ~12-15 min cold.

Verified locally: full `bash testing/ci-local.sh` sweep passes,
including 5/5 deb-install distros and all 13 dns-resolver scenarios.
Tor-inclusive sweep (`--with-tor`) verified in a follow-up run.
This commit is contained in:
Johnathan Corgan
2026-04-30 10:24:32 +00:00
parent 674c7fe1ff
commit 37c2973e2f
20 changed files with 640 additions and 51 deletions
+163 -9
View File
@@ -7,19 +7,24 @@
# --build-only Only run build + clippy
# --test-only Only run unit tests (skip build, skip integration)
# --skip-integration Skip integration tests
# --skip-chaos Skip chaos scenarios (run static + rekey + sidecar only)
# --skip-chaos Skip chaos scenarios
# --with-tor Include Tor harnesses (off by default — needs live Tor)
# --only <suite> Run a single integration suite
# -j, --jobs <N> Max parallel chaos scenarios (default: 4)
# --list List available integration suites
# -h, --help Show this help
#
# Integration suites:
# static-mesh, static-chain, rekey, gateway,
# Integration suites (default coverage):
# static-mesh, static-chain, rekey, rekey-accept-off, gateway,
# acl-allowlist, nat-cone, nat-symmetric, nat-lan,
# chaos-smoke-10, chaos-churn-mixed-10, chaos-ethernet-mesh,
# chaos-ethernet-only, chaos-tcp-mesh, chaos-bottleneck-parent,
# chaos-cost-avoidance, chaos-cost-reeval, chaos-cost-stability,
# chaos-depth-vs-cost, chaos-mixed-technology, chaos-congestion-stress,
# sidecar
# sidecar, dns-resolver, deb-install
#
# Opt-in (require --with-tor; depend on live Tor network):
# tor-socks5, tor-directory
#
# Exit codes:
# 0 — all stages passed
@@ -43,11 +48,12 @@ BUILD_ONLY=false
TEST_ONLY=false
SKIP_INTEGRATION=false
SKIP_CHAOS=false
WITH_TOR=false
ONLY_SUITE=""
# All integration suites matching ci.yml
STATIC_SUITES=(static-mesh static-chain)
REKEY_SUITES=(rekey)
REKEY_SUITES=(rekey rekey-accept-off)
# Each entry: "display-name scenario [--flag value ...]"
CHAOS_SUITES=(
"smoke-10 smoke-10"
@@ -65,6 +71,11 @@ CHAOS_SUITES=(
)
GATEWAY_SUITES=(gateway)
SIDECAR_SUITES=(sidecar)
ACL_SUITES=(acl-allowlist)
NAT_SUITES=(cone symmetric lan)
DNS_RESOLVER_SUITES=(dns-resolver)
DEB_INSTALL_SUITES=(deb-install)
TOR_SUITES=(tor-socks5 tor-directory)
# ── Colors ─────────────────────────────────────────────────────────────────
@@ -93,17 +104,32 @@ list_suites() {
echo " Rekey:"
for s in "${REKEY_SUITES[@]}"; do echo " $s"; done
echo ""
echo " Gateway:"
for s in "${GATEWAY_SUITES[@]}"; do echo " $s"; done
echo ""
echo " ACL allowlist:"
for s in "${ACL_SUITES[@]}"; do echo " $s"; done
echo ""
echo " NAT scenarios:"
for s in "${NAT_SUITES[@]}"; do echo " nat-$s"; done
echo ""
echo " Chaos scenarios:"
for entry in "${CHAOS_SUITES[@]}"; do
read -ra parts <<< "$entry"
echo " chaos-${parts[0]} (${parts[*]:1})"
done
echo ""
echo " Gateway:"
for s in "${GATEWAY_SUITES[@]}"; do echo " $s"; done
echo ""
echo " Sidecar:"
for s in "${SIDECAR_SUITES[@]}"; do echo " $s"; done
echo ""
echo " DNS resolver:"
for s in "${DNS_RESOLVER_SUITES[@]}"; do echo " $s"; done
echo ""
echo " Deb-install:"
for s in "${DEB_INSTALL_SUITES[@]}"; do echo " $s"; done
echo ""
echo " Tor (opt-in via --with-tor):"
for s in "${TOR_SUITES[@]}"; do echo " $s"; done
exit 0
}
@@ -120,6 +146,7 @@ while [[ $# -gt 0 ]]; do
--test-only) TEST_ONLY=true; shift ;;
--skip-integration) SKIP_INTEGRATION=true; shift ;;
--skip-chaos) SKIP_CHAOS=true; shift ;;
--with-tor) WITH_TOR=true; shift ;;
--only) ONLY_SUITE="$2"; shift 2 ;;
-j|--jobs) PARALLEL_JOBS="$2"; shift 2 ;;
--list) list_suites ;;
@@ -318,6 +345,98 @@ run_sidecar() {
record "sidecar" $rc
}
# Run the rekey-accept-off integration variant. Same harness as run_rekey
# but on a 2-node topology with udp.accept_connections=false on node-b.
run_rekey_accept_off() {
local compose="testing/static/docker-compose.yml"
local rc=0
info "[rekey-accept-off] Generating configs"
bash testing/static/scripts/generate-configs.sh rekey-accept-off || \
{ record "rekey-accept-off" 1; return; }
REKEY_TOPOLOGY=rekey-accept-off REKEY_ACCEPT_OFF_NODES=b \
bash testing/static/scripts/rekey-test.sh inject-config || \
{ record "rekey-accept-off" 1; return; }
info "[rekey-accept-off] Starting containers"
docker compose -f "$compose" --profile rekey-accept-off up -d || \
{ record "rekey-accept-off" 1; return; }
info "[rekey-accept-off] Running rekey test"
if REKEY_TOPOLOGY=rekey-accept-off REKEY_ACCEPT_OFF_NODES=b \
bash testing/static/scripts/rekey-test.sh; then
rc=0
else
rc=1
info "[rekey-accept-off] Collecting failure logs"
docker compose -f "$compose" --profile rekey-accept-off logs --no-color 2>&1 | tail -100
fi
docker compose -f "$compose" --profile rekey-accept-off down --volumes --remove-orphans 2>/dev/null
record "rekey-accept-off" $rc
}
# Run ACL allowlist integration test
run_acl_allowlist() {
info "[acl-allowlist] Running integration test"
if bash testing/acl-allowlist/test.sh --skip-build 2>&1; then
record "acl-allowlist" 0
else
record "acl-allowlist" 1
fi
}
# Run a NAT scenario (cone, symmetric, lan)
run_nat() {
local scenario="$1"
info "[nat-$scenario] Running NAT lab"
if bash testing/nat/scripts/nat-test.sh "$scenario" 2>&1; then
record "nat-$scenario" 0
else
record "nat-$scenario" 1
fi
}
# Run dns-resolver harness (multi-distro + e2e scenarios)
run_dns_resolver() {
info "[dns-resolver] Running multi-distro test (slow — builds per-distro images)"
if bash testing/dns-resolver/test.sh 2>&1; then
record "dns-resolver" 0
else
record "dns-resolver" 1
fi
}
# Run deb-install harness (multi-distro real-package install)
run_deb_install() {
info "[deb-install] Running multi-distro test (slow — builds .deb + per-distro install)"
if bash testing/deb-install/test.sh 2>&1; then
record "deb-install" 0
else
record "deb-install" 1
fi
}
# Run Tor SOCKS5 outbound test (live Tor network)
run_tor_socks5() {
info "[tor-socks5] Running Tor SOCKS5 outbound test (live Tor)"
if bash testing/tor/socks5-outbound/scripts/tor-test.sh 2>&1; then
record "tor-socks5" 0
else
record "tor-socks5" 1
fi
}
# Run Tor directory-mode test (live Tor network)
run_tor_directory() {
info "[tor-directory] Running Tor directory-mode test (live Tor)"
if bash testing/tor/directory-mode/scripts/directory-test.sh 2>&1; then
record "tor-directory" 0
else
record "tor-directory" 1
fi
}
# Determine which suites to run and execute them
run_integration() {
stage "Stage 3: Integration Tests"
@@ -343,12 +462,21 @@ run_integration() {
run_static "$topology"
done
# Rekey
# Rekey + rekey-accept-off variant
run_rekey
run_rekey_accept_off
# Gateway
run_gateway
# ACL allowlist
run_acl_allowlist
# NAT scenarios (sequential — each owns its compose project)
for scenario in "${NAT_SUITES[@]}"; do
run_nat "$scenario"
done
# Chaos scenarios (parallel, throttled)
if [[ "$SKIP_CHAOS" != true ]]; then
info "Running ${#CHAOS_SUITES[@]} chaos scenarios (max $PARALLEL_JOBS parallel)"
@@ -401,6 +529,18 @@ run_integration() {
# Sidecar
run_sidecar
# DNS resolver multi-distro suite (heavy — per-distro systemd images)
run_dns_resolver
# Deb-install multi-distro suite (heavy — builds .deb + per-distro install)
run_deb_install
# Tor (opt-in via --with-tor; depends on live Tor network)
if [[ "$WITH_TOR" == true ]]; then
run_tor_socks5
run_tor_directory
fi
}
# Run a single named suite
@@ -411,8 +551,14 @@ run_suite() {
run_static "${suite#static-}" ;;
rekey)
run_rekey ;;
rekey-accept-off)
run_rekey_accept_off ;;
gateway)
run_gateway ;;
acl-allowlist)
run_acl_allowlist ;;
nat-cone|nat-symmetric|nat-lan)
run_nat "${suite#nat-}" ;;
chaos-*)
local chaos_name="${suite#chaos-}"
local found=false
@@ -431,6 +577,14 @@ run_suite() {
;;
sidecar)
run_sidecar ;;
dns-resolver)
run_dns_resolver ;;
deb-install)
run_deb_install ;;
tor-socks5)
run_tor_socks5 ;;
tor-directory)
run_tor_directory ;;
*)
fail "Unknown suite: $suite"
record "$suite" 1 ;;