mirror of
https://github.com/jmcorgan/fips.git
synced 2026-07-30 19:46:15 +00:00
Test infrastructure overhaul: gateway robustness + full CI coverage
Single combined commit covering five interlocking pieces of test and CI work that landed during the v0.3.0-prep cycle. ## fips-gateway robustness - src/bin/fips-gateway.rs DNS upstream probe converted from a 3-second hard-fail to a bounded retry loop (5 attempts × 1s timeout, 1s sleep between attempts; ~10s worst case). Covers the cold-boot race where the daemon's TUN is up but the DNS responder at [::1]:5354 is still binding. Each failed attempt logs at INFO. In production the binary's retry is the live recovery mechanism; with retry it recovers silently instead of relying on Restart=on-failure (~5s blip + spurious ERROR per cycle). - packaging/debian/fips-gateway.service `ExecStartPre` now waits up to 30 seconds for the daemon's `fips0` TUN to appear before exec'ing the gateway binary. Eliminates the cold-boot race where the gateway exits with `fips0 interface not found` and recovers via `Restart=on-failure`, producing a 5-second blip and a spurious error log per restart cycle. - testing/docker/entrypoint.sh gateway-mode waits up to 30s for the daemon's DNS responder to bind [::1]:5354 (probes once per second with `dig @::1 -p 5354 ... test.fips`) before exec'ing fips-gateway. Belt-and-suspenders with the binary's own retry: in CI we want deterministic startup ordering. On timeout, fall through so the binary's probe reports the definitive error. ## Test infrastructure DNS bind migration to ::1 After session 359's daemon DNS-bind default flipped from `127.0.0.1` to `::1` (the production fix for ISSUE-2026-0002), the static-test infrastructure was carrying a stale workaround that overrode the default back to IPv4 loopback. The fips-gateway integration test exposed the divergence: the gateway probes its DNS upstream at `[::1]:5354` (production default) while the daemon was binding `127.0.0.1:5354` from the template override — IPv6-explicit sockets do not accept v4-mapped traffic, so the upstream probe exhausted retries and the gateway exited. - Drop the explicit `bind_addr: "127.0.0.1"` line from every test config that emits it: testing/static/configs/node.template.yaml, testing/chaos/configs/node.template.yaml, the sidecar heredoc in testing/docker/entrypoint.sh, testing/acl-allowlist/generate-configs.sh (six per-node blocks), testing/nat/scripts/generate-configs.sh, and the four tor templates under testing/tor/. Daemon picks up its production `::1` default. - Flip the dnsmasq forwarder for `.fips` in testing/docker/Dockerfile from `127.0.0.1#5354` to `::1#5354` so dnsmasq on the shared test image continues to reach the daemon. Template and Dockerfile must move together since most static suites resolve `<npub>.fips` via the test-image dnsmasq. ## rekey-accept-off integration variant + UDP unit test - New `rekey-accept-off` topology and docker-compose profile under testing/static/. 2-node variant where node-b runs with `udp.accept_connections: false`. Pins the regression class that ISSUE-2026-0004 fixed (cross-connection winner's rekey msg1 was being filtered by the accept_connections gate, breaking rekey). - testing/static/scripts/rekey-test.sh accepts REKEY_TOPOLOGY and REKEY_ACCEPT_OFF_NODES env vars; its inject-config subcommand applies the per-node `udp.accept_connections: false` edit, and the test asserts no sustained "Dual rekey initiation" log lines. - New UDP variant of `should_admit_msg1` admit-rekey unit test in src/node/tests/handshake.rs. ## ci-local.sh full integration coverage - New runner functions and dispatcher entries for `acl-allowlist`, `nat-cone` / `nat-symmetric` / `nat-lan`, `rekey-accept-off`, `dns-resolver`, `deb-install`. Each integrates with the existing summary tracking via `record`. - New `--with-tor` flag (off by default) gates `tor-socks5-outbound` and `tor-directory-mode` runners. Tor stays opt-in because both harnesses depend on the live Tor network and would introduce a flake source unrelated to the FIPS code. - New suite arrays (`ACL_SUITES`, `NAT_SUITES`, `DNS_RESOLVER_SUITES`, `DEB_INSTALL_SUITES`, `TOR_SUITES`) drive both the default sweep and `--list` output. - `run_suite` extended to accept the new suite names for `--only` invocations. ## GitHub CI matrix expansions - `gateway` matrix entry runs testing/static/scripts/gateway-test.sh against the existing docker-compose `gateway` profile. - `rekey-accept-off` matrix entry exercises the new topology with REKEY_ACCEPT_OFF_NODES=b. - `deb-install` matrix (debian12 + ubuntu24 + ubuntu26) runs testing/deb-install/test.sh with privileged systemd containers. ~5-7 min cold cache, ~2 min warm per distro. Self-contained: builds its own .deb in a Debian 12 cargo-deb builder image; does not depend on the build job's pre-built artifact. - `dns-resolver` matrix entry runs the full 13-scenario harness (per-distro systemd resolver-backend tests + real-fips end-to-end scenarios) in a single job. Pins the production DNS bind path that ISSUE-2026-0002 lived in. ~7-12 min warm, ~12-15 min cold. Verified locally: full `bash testing/ci-local.sh` sweep passes, including 5/5 deb-install distros and all 13 dns-resolver scenarios. Tor-inclusive sweep (`--with-tor`) verified in a follow-up run.
This commit is contained in:
+163
-9
@@ -7,19 +7,24 @@
|
||||
# --build-only Only run build + clippy
|
||||
# --test-only Only run unit tests (skip build, skip integration)
|
||||
# --skip-integration Skip integration tests
|
||||
# --skip-chaos Skip chaos scenarios (run static + rekey + sidecar only)
|
||||
# --skip-chaos Skip chaos scenarios
|
||||
# --with-tor Include Tor harnesses (off by default — needs live Tor)
|
||||
# --only <suite> Run a single integration suite
|
||||
# -j, --jobs <N> Max parallel chaos scenarios (default: 4)
|
||||
# --list List available integration suites
|
||||
# -h, --help Show this help
|
||||
#
|
||||
# Integration suites:
|
||||
# static-mesh, static-chain, rekey, gateway,
|
||||
# Integration suites (default coverage):
|
||||
# static-mesh, static-chain, rekey, rekey-accept-off, gateway,
|
||||
# acl-allowlist, nat-cone, nat-symmetric, nat-lan,
|
||||
# chaos-smoke-10, chaos-churn-mixed-10, chaos-ethernet-mesh,
|
||||
# chaos-ethernet-only, chaos-tcp-mesh, chaos-bottleneck-parent,
|
||||
# chaos-cost-avoidance, chaos-cost-reeval, chaos-cost-stability,
|
||||
# chaos-depth-vs-cost, chaos-mixed-technology, chaos-congestion-stress,
|
||||
# sidecar
|
||||
# sidecar, dns-resolver, deb-install
|
||||
#
|
||||
# Opt-in (require --with-tor; depend on live Tor network):
|
||||
# tor-socks5, tor-directory
|
||||
#
|
||||
# Exit codes:
|
||||
# 0 — all stages passed
|
||||
@@ -43,11 +48,12 @@ BUILD_ONLY=false
|
||||
TEST_ONLY=false
|
||||
SKIP_INTEGRATION=false
|
||||
SKIP_CHAOS=false
|
||||
WITH_TOR=false
|
||||
ONLY_SUITE=""
|
||||
|
||||
# All integration suites matching ci.yml
|
||||
STATIC_SUITES=(static-mesh static-chain)
|
||||
REKEY_SUITES=(rekey)
|
||||
REKEY_SUITES=(rekey rekey-accept-off)
|
||||
# Each entry: "display-name scenario [--flag value ...]"
|
||||
CHAOS_SUITES=(
|
||||
"smoke-10 smoke-10"
|
||||
@@ -65,6 +71,11 @@ CHAOS_SUITES=(
|
||||
)
|
||||
GATEWAY_SUITES=(gateway)
|
||||
SIDECAR_SUITES=(sidecar)
|
||||
ACL_SUITES=(acl-allowlist)
|
||||
NAT_SUITES=(cone symmetric lan)
|
||||
DNS_RESOLVER_SUITES=(dns-resolver)
|
||||
DEB_INSTALL_SUITES=(deb-install)
|
||||
TOR_SUITES=(tor-socks5 tor-directory)
|
||||
|
||||
# ── Colors ─────────────────────────────────────────────────────────────────
|
||||
|
||||
@@ -93,17 +104,32 @@ list_suites() {
|
||||
echo " Rekey:"
|
||||
for s in "${REKEY_SUITES[@]}"; do echo " $s"; done
|
||||
echo ""
|
||||
echo " Gateway:"
|
||||
for s in "${GATEWAY_SUITES[@]}"; do echo " $s"; done
|
||||
echo ""
|
||||
echo " ACL allowlist:"
|
||||
for s in "${ACL_SUITES[@]}"; do echo " $s"; done
|
||||
echo ""
|
||||
echo " NAT scenarios:"
|
||||
for s in "${NAT_SUITES[@]}"; do echo " nat-$s"; done
|
||||
echo ""
|
||||
echo " Chaos scenarios:"
|
||||
for entry in "${CHAOS_SUITES[@]}"; do
|
||||
read -ra parts <<< "$entry"
|
||||
echo " chaos-${parts[0]} (${parts[*]:1})"
|
||||
done
|
||||
echo ""
|
||||
echo " Gateway:"
|
||||
for s in "${GATEWAY_SUITES[@]}"; do echo " $s"; done
|
||||
echo ""
|
||||
echo " Sidecar:"
|
||||
for s in "${SIDECAR_SUITES[@]}"; do echo " $s"; done
|
||||
echo ""
|
||||
echo " DNS resolver:"
|
||||
for s in "${DNS_RESOLVER_SUITES[@]}"; do echo " $s"; done
|
||||
echo ""
|
||||
echo " Deb-install:"
|
||||
for s in "${DEB_INSTALL_SUITES[@]}"; do echo " $s"; done
|
||||
echo ""
|
||||
echo " Tor (opt-in via --with-tor):"
|
||||
for s in "${TOR_SUITES[@]}"; do echo " $s"; done
|
||||
exit 0
|
||||
}
|
||||
|
||||
@@ -120,6 +146,7 @@ while [[ $# -gt 0 ]]; do
|
||||
--test-only) TEST_ONLY=true; shift ;;
|
||||
--skip-integration) SKIP_INTEGRATION=true; shift ;;
|
||||
--skip-chaos) SKIP_CHAOS=true; shift ;;
|
||||
--with-tor) WITH_TOR=true; shift ;;
|
||||
--only) ONLY_SUITE="$2"; shift 2 ;;
|
||||
-j|--jobs) PARALLEL_JOBS="$2"; shift 2 ;;
|
||||
--list) list_suites ;;
|
||||
@@ -318,6 +345,98 @@ run_sidecar() {
|
||||
record "sidecar" $rc
|
||||
}
|
||||
|
||||
# Run the rekey-accept-off integration variant. Same harness as run_rekey
|
||||
# but on a 2-node topology with udp.accept_connections=false on node-b.
|
||||
run_rekey_accept_off() {
|
||||
local compose="testing/static/docker-compose.yml"
|
||||
local rc=0
|
||||
|
||||
info "[rekey-accept-off] Generating configs"
|
||||
bash testing/static/scripts/generate-configs.sh rekey-accept-off || \
|
||||
{ record "rekey-accept-off" 1; return; }
|
||||
REKEY_TOPOLOGY=rekey-accept-off REKEY_ACCEPT_OFF_NODES=b \
|
||||
bash testing/static/scripts/rekey-test.sh inject-config || \
|
||||
{ record "rekey-accept-off" 1; return; }
|
||||
|
||||
info "[rekey-accept-off] Starting containers"
|
||||
docker compose -f "$compose" --profile rekey-accept-off up -d || \
|
||||
{ record "rekey-accept-off" 1; return; }
|
||||
|
||||
info "[rekey-accept-off] Running rekey test"
|
||||
if REKEY_TOPOLOGY=rekey-accept-off REKEY_ACCEPT_OFF_NODES=b \
|
||||
bash testing/static/scripts/rekey-test.sh; then
|
||||
rc=0
|
||||
else
|
||||
rc=1
|
||||
info "[rekey-accept-off] Collecting failure logs"
|
||||
docker compose -f "$compose" --profile rekey-accept-off logs --no-color 2>&1 | tail -100
|
||||
fi
|
||||
|
||||
docker compose -f "$compose" --profile rekey-accept-off down --volumes --remove-orphans 2>/dev/null
|
||||
record "rekey-accept-off" $rc
|
||||
}
|
||||
|
||||
# Run ACL allowlist integration test
|
||||
run_acl_allowlist() {
|
||||
info "[acl-allowlist] Running integration test"
|
||||
if bash testing/acl-allowlist/test.sh --skip-build 2>&1; then
|
||||
record "acl-allowlist" 0
|
||||
else
|
||||
record "acl-allowlist" 1
|
||||
fi
|
||||
}
|
||||
|
||||
# Run a NAT scenario (cone, symmetric, lan)
|
||||
run_nat() {
|
||||
local scenario="$1"
|
||||
info "[nat-$scenario] Running NAT lab"
|
||||
if bash testing/nat/scripts/nat-test.sh "$scenario" 2>&1; then
|
||||
record "nat-$scenario" 0
|
||||
else
|
||||
record "nat-$scenario" 1
|
||||
fi
|
||||
}
|
||||
|
||||
# Run dns-resolver harness (multi-distro + e2e scenarios)
|
||||
run_dns_resolver() {
|
||||
info "[dns-resolver] Running multi-distro test (slow — builds per-distro images)"
|
||||
if bash testing/dns-resolver/test.sh 2>&1; then
|
||||
record "dns-resolver" 0
|
||||
else
|
||||
record "dns-resolver" 1
|
||||
fi
|
||||
}
|
||||
|
||||
# Run deb-install harness (multi-distro real-package install)
|
||||
run_deb_install() {
|
||||
info "[deb-install] Running multi-distro test (slow — builds .deb + per-distro install)"
|
||||
if bash testing/deb-install/test.sh 2>&1; then
|
||||
record "deb-install" 0
|
||||
else
|
||||
record "deb-install" 1
|
||||
fi
|
||||
}
|
||||
|
||||
# Run Tor SOCKS5 outbound test (live Tor network)
|
||||
run_tor_socks5() {
|
||||
info "[tor-socks5] Running Tor SOCKS5 outbound test (live Tor)"
|
||||
if bash testing/tor/socks5-outbound/scripts/tor-test.sh 2>&1; then
|
||||
record "tor-socks5" 0
|
||||
else
|
||||
record "tor-socks5" 1
|
||||
fi
|
||||
}
|
||||
|
||||
# Run Tor directory-mode test (live Tor network)
|
||||
run_tor_directory() {
|
||||
info "[tor-directory] Running Tor directory-mode test (live Tor)"
|
||||
if bash testing/tor/directory-mode/scripts/directory-test.sh 2>&1; then
|
||||
record "tor-directory" 0
|
||||
else
|
||||
record "tor-directory" 1
|
||||
fi
|
||||
}
|
||||
|
||||
# Determine which suites to run and execute them
|
||||
run_integration() {
|
||||
stage "Stage 3: Integration Tests"
|
||||
@@ -343,12 +462,21 @@ run_integration() {
|
||||
run_static "$topology"
|
||||
done
|
||||
|
||||
# Rekey
|
||||
# Rekey + rekey-accept-off variant
|
||||
run_rekey
|
||||
run_rekey_accept_off
|
||||
|
||||
# Gateway
|
||||
run_gateway
|
||||
|
||||
# ACL allowlist
|
||||
run_acl_allowlist
|
||||
|
||||
# NAT scenarios (sequential — each owns its compose project)
|
||||
for scenario in "${NAT_SUITES[@]}"; do
|
||||
run_nat "$scenario"
|
||||
done
|
||||
|
||||
# Chaos scenarios (parallel, throttled)
|
||||
if [[ "$SKIP_CHAOS" != true ]]; then
|
||||
info "Running ${#CHAOS_SUITES[@]} chaos scenarios (max $PARALLEL_JOBS parallel)"
|
||||
@@ -401,6 +529,18 @@ run_integration() {
|
||||
|
||||
# Sidecar
|
||||
run_sidecar
|
||||
|
||||
# DNS resolver multi-distro suite (heavy — per-distro systemd images)
|
||||
run_dns_resolver
|
||||
|
||||
# Deb-install multi-distro suite (heavy — builds .deb + per-distro install)
|
||||
run_deb_install
|
||||
|
||||
# Tor (opt-in via --with-tor; depends on live Tor network)
|
||||
if [[ "$WITH_TOR" == true ]]; then
|
||||
run_tor_socks5
|
||||
run_tor_directory
|
||||
fi
|
||||
}
|
||||
|
||||
# Run a single named suite
|
||||
@@ -411,8 +551,14 @@ run_suite() {
|
||||
run_static "${suite#static-}" ;;
|
||||
rekey)
|
||||
run_rekey ;;
|
||||
rekey-accept-off)
|
||||
run_rekey_accept_off ;;
|
||||
gateway)
|
||||
run_gateway ;;
|
||||
acl-allowlist)
|
||||
run_acl_allowlist ;;
|
||||
nat-cone|nat-symmetric|nat-lan)
|
||||
run_nat "${suite#nat-}" ;;
|
||||
chaos-*)
|
||||
local chaos_name="${suite#chaos-}"
|
||||
local found=false
|
||||
@@ -431,6 +577,14 @@ run_suite() {
|
||||
;;
|
||||
sidecar)
|
||||
run_sidecar ;;
|
||||
dns-resolver)
|
||||
run_dns_resolver ;;
|
||||
deb-install)
|
||||
run_deb_install ;;
|
||||
tor-socks5)
|
||||
run_tor_socks5 ;;
|
||||
tor-directory)
|
||||
run_tor_directory ;;
|
||||
*)
|
||||
fail "Unknown suite: $suite"
|
||||
record "$suite" 1 ;;
|
||||
|
||||
Reference in New Issue
Block a user