Test infrastructure overhaul: gateway robustness + full CI coverage

Single combined commit covering five interlocking pieces of test and
CI work that landed during the v0.3.0-prep cycle.

## fips-gateway robustness

- src/bin/fips-gateway.rs DNS upstream probe converted from a 3-second
  hard-fail to a bounded retry loop (5 attempts × 1s timeout, 1s sleep
  between attempts; ~10s worst case). Covers the cold-boot race where
  the daemon's TUN is up but the DNS responder at [::1]:5354 is still
  binding. Each failed attempt logs at INFO. In production the binary's
  retry is the live recovery mechanism; with retry it recovers silently
  instead of relying on Restart=on-failure (~5s blip + spurious ERROR
  per cycle).
- packaging/debian/fips-gateway.service `ExecStartPre` now waits up to
  30 seconds for the daemon's `fips0` TUN to appear before exec'ing
  the gateway binary. Eliminates the cold-boot race where the gateway
  exits with `fips0 interface not found` and recovers via
  `Restart=on-failure`, producing a 5-second blip and a spurious error
  log per restart cycle.
- testing/docker/entrypoint.sh gateway-mode waits up to 30s for the
  daemon's DNS responder to bind [::1]:5354 (probes once per second
  with `dig @::1 -p 5354 ... test.fips`) before exec'ing fips-gateway.
  Belt-and-suspenders with the binary's own retry: in CI we want
  deterministic startup ordering. On timeout, fall through so the
  binary's probe reports the definitive error.

## Test infrastructure DNS bind migration to ::1

After session 359's daemon DNS-bind default flipped from `127.0.0.1`
to `::1` (the production fix for ISSUE-2026-0002), the static-test
infrastructure was carrying a stale workaround that overrode the
default back to IPv4 loopback. The fips-gateway integration test
exposed the divergence: the gateway probes its DNS upstream at
`[::1]:5354` (production default) while the daemon was binding
`127.0.0.1:5354` from the template override — IPv6-explicit sockets
do not accept v4-mapped traffic, so the upstream probe exhausted
retries and the gateway exited.

- Drop the explicit `bind_addr: "127.0.0.1"` line from every test
  config that emits it: testing/static/configs/node.template.yaml,
  testing/chaos/configs/node.template.yaml, the sidecar heredoc in
  testing/docker/entrypoint.sh, testing/acl-allowlist/generate-configs.sh
  (six per-node blocks), testing/nat/scripts/generate-configs.sh, and
  the four tor templates under testing/tor/. Daemon picks up its
  production `::1` default.
- Flip the dnsmasq forwarder for `.fips` in testing/docker/Dockerfile
  from `127.0.0.1#5354` to `::1#5354` so dnsmasq on the shared test
  image continues to reach the daemon. Template and Dockerfile must
  move together since most static suites resolve `<npub>.fips` via
  the test-image dnsmasq.

## rekey-accept-off integration variant + UDP unit test

- New `rekey-accept-off` topology and docker-compose profile under
  testing/static/. 2-node variant where node-b runs with
  `udp.accept_connections: false`. Pins the regression class that
  ISSUE-2026-0004 fixed (cross-connection winner's rekey msg1 was
  being filtered by the accept_connections gate, breaking rekey).
- testing/static/scripts/rekey-test.sh accepts REKEY_TOPOLOGY and
  REKEY_ACCEPT_OFF_NODES env vars; its inject-config subcommand
  applies the per-node `udp.accept_connections: false` edit, and
  the test asserts no sustained "Dual rekey initiation" log lines.
- New UDP variant of `should_admit_msg1` admit-rekey unit test in
  src/node/tests/handshake.rs.

## ci-local.sh full integration coverage

- New runner functions and dispatcher entries for `acl-allowlist`,
  `nat-cone` / `nat-symmetric` / `nat-lan`, `rekey-accept-off`,
  `dns-resolver`, `deb-install`. Each integrates with the existing
  summary tracking via `record`.
- New `--with-tor` flag (off by default) gates `tor-socks5-outbound`
  and `tor-directory-mode` runners. Tor stays opt-in because both
  harnesses depend on the live Tor network and would introduce a
  flake source unrelated to the FIPS code.
- New suite arrays (`ACL_SUITES`, `NAT_SUITES`, `DNS_RESOLVER_SUITES`,
  `DEB_INSTALL_SUITES`, `TOR_SUITES`) drive both the default sweep
  and `--list` output.
- `run_suite` extended to accept the new suite names for `--only`
  invocations.

## GitHub CI matrix expansions

- `gateway` matrix entry runs testing/static/scripts/gateway-test.sh
  against the existing docker-compose `gateway` profile.
- `rekey-accept-off` matrix entry exercises the new topology with
  REKEY_ACCEPT_OFF_NODES=b.
- `deb-install` matrix (debian12 + ubuntu24 + ubuntu26) runs
  testing/deb-install/test.sh with privileged systemd containers.
  ~5-7 min cold cache, ~2 min warm per distro. Self-contained: builds
  its own .deb in a Debian 12 cargo-deb builder image; does not
  depend on the build job's pre-built artifact.
- `dns-resolver` matrix entry runs the full 13-scenario harness
  (per-distro systemd resolver-backend tests + real-fips end-to-end
  scenarios) in a single job. Pins the production DNS bind path that
  ISSUE-2026-0002 lived in. ~7-12 min warm, ~12-15 min cold.

Verified locally: full `bash testing/ci-local.sh` sweep passes,
including 5/5 deb-install distros and all 13 dns-resolver scenarios.
Tor-inclusive sweep (`--with-tor`) verified in a follow-up run.
This commit is contained in:
Johnathan Corgan
2026-04-30 10:24:32 +00:00
parent 674c7fe1ff
commit 37c2973e2f
20 changed files with 640 additions and 51 deletions
+158
View File
@@ -258,8 +258,15 @@ jobs:
- suite: rekey
type: rekey
topology: rekey
- suite: rekey-accept-off
type: rekey-accept-off
topology: rekey-accept-off
- suite: acl-allowlist
type: acl-allowlist
# ── Outbound LAN gateway integration test ──────────────────────
- suite: gateway
type: gateway
topology: gateway
# ── Chaos / stochastic scenarios ───────────────────────────────────
- suite: chaos-smoke-10
type: chaos
@@ -311,6 +318,33 @@ jobs:
- suite: nat-lan
type: nat
scenario: lan
# ── Real-deb install across target distros ─────────────────────
# Boots a privileged systemd container per distro, runs
# `apt install ./fips_*.deb` with the locally-built package,
# then asserts end-to-end `.fips` resolution + the
# gateway/daemon default-pairing. The most thorough single
# test surface — exercises packaging, maintainer scripts,
# systemd unit ordering, real TUN, and the DNS responder
# filter on a per-distro resolver backend.
- suite: deb-install-debian12
type: deb-install
scenario: debian12
- suite: deb-install-ubuntu24
type: deb-install
scenario: ubuntu24
- suite: deb-install-ubuntu26
type: deb-install
scenario: ubuntu26
# ── DNS resolver multi-backend coverage ────────────────────────
# Exercises every fips-dns-setup backend (resolved, dnsmasq,
# NM+dnsmasq, dns-delegate, no-resolver) across five distros,
# plus end-to-end scenarios that boot a real fips daemon with a
# real TUN and assert `dig @127.0.0.53 AAAA <npub>.fips`
# returns AAAA. Pins the production DNS bind path that
# ISSUE-2026-0002 lived in. Single matrix entry runs all 13
# scenarios sequentially; ~7-12 min warm, ~12-15 min cold.
- suite: dns-resolver
type: dns-resolver
steps:
- uses: actions/checkout@v4
@@ -391,6 +425,41 @@ jobs:
docker compose -f testing/static/docker-compose.yml \
--profile rekey down --volumes --remove-orphans
# ── Rekey + accept_connections=false variant ──────────────────────────
- name: Generate and inject configs (rekey-accept-off)
if: matrix.type == 'rekey-accept-off'
env:
REKEY_TOPOLOGY: rekey-accept-off
REKEY_ACCEPT_OFF_NODES: b
run: |
bash testing/static/scripts/generate-configs.sh rekey-accept-off
bash testing/static/scripts/rekey-test.sh inject-config
- name: Start containers (rekey-accept-off)
if: matrix.type == 'rekey-accept-off'
run: |
docker compose -f testing/static/docker-compose.yml \
--profile rekey-accept-off up -d
- name: Run rekey test (accept-off variant)
if: matrix.type == 'rekey-accept-off'
env:
REKEY_TOPOLOGY: rekey-accept-off
REKEY_ACCEPT_OFF_NODES: b
run: bash testing/static/scripts/rekey-test.sh
- name: Collect logs on failure (rekey-accept-off)
if: matrix.type == 'rekey-accept-off' && failure()
run: |
docker compose -f testing/static/docker-compose.yml \
--profile rekey-accept-off logs --no-color | tail -300
- name: Stop containers (rekey-accept-off)
if: matrix.type == 'rekey-accept-off' && always()
run: |
docker compose -f testing/static/docker-compose.yml \
--profile rekey-accept-off down --volumes --remove-orphans
# ── ACL allowlist integration test ─────────────────────────────────────
- name: Run ACL allowlist integration test
if: matrix.type == 'acl-allowlist'
@@ -454,3 +523,92 @@ jobs:
docker compose -f testing/nat/docker-compose.yml \
--profile cone --profile symmetric --profile lan \
down --volumes --remove-orphans
# ── Outbound LAN gateway integration test ──────────────────────────
- name: Generate configs (gateway)
if: matrix.type == 'gateway'
run: bash testing/static/scripts/generate-configs.sh gateway gateway-test
- name: Inject gateway config (gateway)
if: matrix.type == 'gateway'
run: bash testing/static/scripts/gateway-test.sh inject-config
- name: Start containers (gateway)
if: matrix.type == 'gateway'
run: |
docker compose -f testing/static/docker-compose.yml \
--profile gateway up -d
- name: Run gateway test
if: matrix.type == 'gateway'
run: bash testing/static/scripts/gateway-test.sh
- name: Collect logs on failure (gateway)
if: matrix.type == 'gateway' && failure()
run: |
docker compose -f testing/static/docker-compose.yml \
--profile gateway logs --no-color | tail -300
- name: Stop containers (gateway)
if: matrix.type == 'gateway' && always()
run: |
docker compose -f testing/static/docker-compose.yml \
--profile gateway down --volumes --remove-orphans
# ── Real-deb install integration ────────────────────────────────────
# The deb-install harness builds its own .deb from source in a
# cargo-deb builder image; the pre-built Linux binary from the
# build job is intentionally not used here so the test exercises
# the full packaging pipeline. ~5-7 min cold-cache on a fresh
# runner (.deb build dominates), ~1-2 min warm-cache.
- name: Run deb-install scenario
if: matrix.type == 'deb-install'
timeout-minutes: 25
run: bash testing/deb-install/test.sh ${{ matrix.scenario }}
- name: Collect logs on failure (deb-install)
if: matrix.type == 'deb-install' && failure()
run: |
docker ps -a --filter "name=fips-deb-test-${{ matrix.scenario }}" --format '{{.Names}}' | while read -r c; do
echo "--- ${c} fips.service ---"
docker exec "$c" journalctl -u fips.service --no-pager 2>&1 | tail -100 || true
echo "--- ${c} fips-dns.service ---"
docker exec "$c" journalctl -u fips-dns.service --no-pager 2>&1 | tail -100 || true
echo "--- ${c} fips-gateway.service ---"
docker exec "$c" journalctl -u fips-gateway.service --no-pager 2>&1 | tail -100 || true
done
- name: Stop containers (deb-install)
if: matrix.type == 'deb-install' && always()
run: |
docker ps -a --filter "name=fips-deb-test-${{ matrix.scenario }}" --format '{{.Names}}' | while read -r c; do
docker rm -f "$c" >/dev/null 2>&1 || true
done
# ── DNS resolver multi-backend integration ──────────────────────────
# The dns-resolver harness builds its own fips binary from source in a
# Debian 12 builder image (shared cache layout with deb-install). Runs
# all 13 scenarios in a single job: dummy-TUN backend-detection tests
# plus real-fips end-to-end queries through systemd-resolved across
# five distros. ~7-12 min warm, ~12-15 min cold.
- name: Run dns-resolver test
if: matrix.type == 'dns-resolver'
timeout-minutes: 30
run: bash testing/dns-resolver/test.sh
- name: Collect logs on failure (dns-resolver)
if: matrix.type == 'dns-resolver' && failure()
run: |
docker ps -a --filter "name=fips-dns-test-" --format '{{.Names}}' | while read -r c; do
echo "--- ${c} fips.service ---"
docker exec "$c" journalctl -u fips.service --no-pager 2>&1 | tail -100 || true
echo "--- ${c} fips-dns.service ---"
docker exec "$c" journalctl -u fips-dns.service --no-pager 2>&1 | tail -100 || true
done
- name: Stop containers (dns-resolver)
if: matrix.type == 'dns-resolver' && always()
run: |
docker ps -a --filter "name=fips-dns-test-" --format '{{.Names}}' | while read -r c; do
docker rm -f "$c" >/dev/null 2>&1 || true
done