Test infrastructure overhaul: gateway robustness + full CI coverage

Single combined commit covering five interlocking pieces of test and
CI work that landed during the v0.3.0-prep cycle.

## fips-gateway robustness

- src/bin/fips-gateway.rs DNS upstream probe converted from a 3-second
  hard-fail to a bounded retry loop (5 attempts × 1s timeout, 1s sleep
  between attempts; ~10s worst case). Covers the cold-boot race where
  the daemon's TUN is up but the DNS responder at [::1]:5354 is still
  binding. Each failed attempt logs at INFO. In production the binary's
  retry is the live recovery mechanism; with retry it recovers silently
  instead of relying on Restart=on-failure (~5s blip + spurious ERROR
  per cycle).
- packaging/debian/fips-gateway.service `ExecStartPre` now waits up to
  30 seconds for the daemon's `fips0` TUN to appear before exec'ing
  the gateway binary. Eliminates the cold-boot race where the gateway
  exits with `fips0 interface not found` and recovers via
  `Restart=on-failure`, producing a 5-second blip and a spurious error
  log per restart cycle.
- testing/docker/entrypoint.sh gateway-mode waits up to 30s for the
  daemon's DNS responder to bind [::1]:5354 (probes once per second
  with `dig @::1 -p 5354 ... test.fips`) before exec'ing fips-gateway.
  Belt-and-suspenders with the binary's own retry: in CI we want
  deterministic startup ordering. On timeout, fall through so the
  binary's probe reports the definitive error.

## Test infrastructure DNS bind migration to ::1

After session 359's daemon DNS-bind default flipped from `127.0.0.1`
to `::1` (the production fix for ISSUE-2026-0002), the static-test
infrastructure was carrying a stale workaround that overrode the
default back to IPv4 loopback. The fips-gateway integration test
exposed the divergence: the gateway probes its DNS upstream at
`[::1]:5354` (production default) while the daemon was binding
`127.0.0.1:5354` from the template override — IPv6-explicit sockets
do not accept v4-mapped traffic, so the upstream probe exhausted
retries and the gateway exited.

- Drop the explicit `bind_addr: "127.0.0.1"` line from every test
  config that emits it: testing/static/configs/node.template.yaml,
  testing/chaos/configs/node.template.yaml, the sidecar heredoc in
  testing/docker/entrypoint.sh, testing/acl-allowlist/generate-configs.sh
  (six per-node blocks), testing/nat/scripts/generate-configs.sh, and
  the four tor templates under testing/tor/. Daemon picks up its
  production `::1` default.
- Flip the dnsmasq forwarder for `.fips` in testing/docker/Dockerfile
  from `127.0.0.1#5354` to `::1#5354` so dnsmasq on the shared test
  image continues to reach the daemon. Template and Dockerfile must
  move together since most static suites resolve `<npub>.fips` via
  the test-image dnsmasq.

## rekey-accept-off integration variant + UDP unit test

- New `rekey-accept-off` topology and docker-compose profile under
  testing/static/. 2-node variant where node-b runs with
  `udp.accept_connections: false`. Pins the regression class that
  ISSUE-2026-0004 fixed (cross-connection winner's rekey msg1 was
  being filtered by the accept_connections gate, breaking rekey).
- testing/static/scripts/rekey-test.sh accepts REKEY_TOPOLOGY and
  REKEY_ACCEPT_OFF_NODES env vars; its inject-config subcommand
  applies the per-node `udp.accept_connections: false` edit, and
  the test asserts no sustained "Dual rekey initiation" log lines.
- New UDP variant of `should_admit_msg1` admit-rekey unit test in
  src/node/tests/handshake.rs.

## ci-local.sh full integration coverage

- New runner functions and dispatcher entries for `acl-allowlist`,
  `nat-cone` / `nat-symmetric` / `nat-lan`, `rekey-accept-off`,
  `dns-resolver`, `deb-install`. Each integrates with the existing
  summary tracking via `record`.
- New `--with-tor` flag (off by default) gates `tor-socks5-outbound`
  and `tor-directory-mode` runners. Tor stays opt-in because both
  harnesses depend on the live Tor network and would introduce a
  flake source unrelated to the FIPS code.
- New suite arrays (`ACL_SUITES`, `NAT_SUITES`, `DNS_RESOLVER_SUITES`,
  `DEB_INSTALL_SUITES`, `TOR_SUITES`) drive both the default sweep
  and `--list` output.
- `run_suite` extended to accept the new suite names for `--only`
  invocations.

## GitHub CI matrix expansions

- `gateway` matrix entry runs testing/static/scripts/gateway-test.sh
  against the existing docker-compose `gateway` profile.
- `rekey-accept-off` matrix entry exercises the new topology with
  REKEY_ACCEPT_OFF_NODES=b.
- `deb-install` matrix (debian12 + ubuntu24 + ubuntu26) runs
  testing/deb-install/test.sh with privileged systemd containers.
  ~5-7 min cold cache, ~2 min warm per distro. Self-contained: builds
  its own .deb in a Debian 12 cargo-deb builder image; does not
  depend on the build job's pre-built artifact.
- `dns-resolver` matrix entry runs the full 13-scenario harness
  (per-distro systemd resolver-backend tests + real-fips end-to-end
  scenarios) in a single job. Pins the production DNS bind path that
  ISSUE-2026-0002 lived in. ~7-12 min warm, ~12-15 min cold.

Verified locally: full `bash testing/ci-local.sh` sweep passes,
including 5/5 deb-install distros and all 13 dns-resolver scenarios.
Tor-inclusive sweep (`--with-tor`) verified in a follow-up run.
This commit is contained in:
Johnathan Corgan
2026-04-30 10:24:32 +00:00
parent 674c7fe1ff
commit 37c2973e2f
20 changed files with 640 additions and 51 deletions
+158
View File
@@ -258,8 +258,15 @@ jobs:
- suite: rekey
type: rekey
topology: rekey
- suite: rekey-accept-off
type: rekey-accept-off
topology: rekey-accept-off
- suite: acl-allowlist
type: acl-allowlist
# ── Outbound LAN gateway integration test ──────────────────────
- suite: gateway
type: gateway
topology: gateway
# ── Chaos / stochastic scenarios ───────────────────────────────────
- suite: chaos-smoke-10
type: chaos
@@ -311,6 +318,33 @@ jobs:
- suite: nat-lan
type: nat
scenario: lan
# ── Real-deb install across target distros ─────────────────────
# Boots a privileged systemd container per distro, runs
# `apt install ./fips_*.deb` with the locally-built package,
# then asserts end-to-end `.fips` resolution + the
# gateway/daemon default-pairing. The most thorough single
# test surface — exercises packaging, maintainer scripts,
# systemd unit ordering, real TUN, and the DNS responder
# filter on a per-distro resolver backend.
- suite: deb-install-debian12
type: deb-install
scenario: debian12
- suite: deb-install-ubuntu24
type: deb-install
scenario: ubuntu24
- suite: deb-install-ubuntu26
type: deb-install
scenario: ubuntu26
# ── DNS resolver multi-backend coverage ────────────────────────
# Exercises every fips-dns-setup backend (resolved, dnsmasq,
# NM+dnsmasq, dns-delegate, no-resolver) across five distros,
# plus end-to-end scenarios that boot a real fips daemon with a
# real TUN and assert `dig @127.0.0.53 AAAA <npub>.fips`
# returns AAAA. Pins the production DNS bind path that
# ISSUE-2026-0002 lived in. Single matrix entry runs all 13
# scenarios sequentially; ~7-12 min warm, ~12-15 min cold.
- suite: dns-resolver
type: dns-resolver
steps:
- uses: actions/checkout@v4
@@ -391,6 +425,41 @@ jobs:
docker compose -f testing/static/docker-compose.yml \
--profile rekey down --volumes --remove-orphans
# ── Rekey + accept_connections=false variant ──────────────────────────
- name: Generate and inject configs (rekey-accept-off)
if: matrix.type == 'rekey-accept-off'
env:
REKEY_TOPOLOGY: rekey-accept-off
REKEY_ACCEPT_OFF_NODES: b
run: |
bash testing/static/scripts/generate-configs.sh rekey-accept-off
bash testing/static/scripts/rekey-test.sh inject-config
- name: Start containers (rekey-accept-off)
if: matrix.type == 'rekey-accept-off'
run: |
docker compose -f testing/static/docker-compose.yml \
--profile rekey-accept-off up -d
- name: Run rekey test (accept-off variant)
if: matrix.type == 'rekey-accept-off'
env:
REKEY_TOPOLOGY: rekey-accept-off
REKEY_ACCEPT_OFF_NODES: b
run: bash testing/static/scripts/rekey-test.sh
- name: Collect logs on failure (rekey-accept-off)
if: matrix.type == 'rekey-accept-off' && failure()
run: |
docker compose -f testing/static/docker-compose.yml \
--profile rekey-accept-off logs --no-color | tail -300
- name: Stop containers (rekey-accept-off)
if: matrix.type == 'rekey-accept-off' && always()
run: |
docker compose -f testing/static/docker-compose.yml \
--profile rekey-accept-off down --volumes --remove-orphans
# ── ACL allowlist integration test ─────────────────────────────────────
- name: Run ACL allowlist integration test
if: matrix.type == 'acl-allowlist'
@@ -454,3 +523,92 @@ jobs:
docker compose -f testing/nat/docker-compose.yml \
--profile cone --profile symmetric --profile lan \
down --volumes --remove-orphans
# ── Outbound LAN gateway integration test ──────────────────────────
- name: Generate configs (gateway)
if: matrix.type == 'gateway'
run: bash testing/static/scripts/generate-configs.sh gateway gateway-test
- name: Inject gateway config (gateway)
if: matrix.type == 'gateway'
run: bash testing/static/scripts/gateway-test.sh inject-config
- name: Start containers (gateway)
if: matrix.type == 'gateway'
run: |
docker compose -f testing/static/docker-compose.yml \
--profile gateway up -d
- name: Run gateway test
if: matrix.type == 'gateway'
run: bash testing/static/scripts/gateway-test.sh
- name: Collect logs on failure (gateway)
if: matrix.type == 'gateway' && failure()
run: |
docker compose -f testing/static/docker-compose.yml \
--profile gateway logs --no-color | tail -300
- name: Stop containers (gateway)
if: matrix.type == 'gateway' && always()
run: |
docker compose -f testing/static/docker-compose.yml \
--profile gateway down --volumes --remove-orphans
# ── Real-deb install integration ────────────────────────────────────
# The deb-install harness builds its own .deb from source in a
# cargo-deb builder image; the pre-built Linux binary from the
# build job is intentionally not used here so the test exercises
# the full packaging pipeline. ~5-7 min cold-cache on a fresh
# runner (.deb build dominates), ~1-2 min warm-cache.
- name: Run deb-install scenario
if: matrix.type == 'deb-install'
timeout-minutes: 25
run: bash testing/deb-install/test.sh ${{ matrix.scenario }}
- name: Collect logs on failure (deb-install)
if: matrix.type == 'deb-install' && failure()
run: |
docker ps -a --filter "name=fips-deb-test-${{ matrix.scenario }}" --format '{{.Names}}' | while read -r c; do
echo "--- ${c} fips.service ---"
docker exec "$c" journalctl -u fips.service --no-pager 2>&1 | tail -100 || true
echo "--- ${c} fips-dns.service ---"
docker exec "$c" journalctl -u fips-dns.service --no-pager 2>&1 | tail -100 || true
echo "--- ${c} fips-gateway.service ---"
docker exec "$c" journalctl -u fips-gateway.service --no-pager 2>&1 | tail -100 || true
done
- name: Stop containers (deb-install)
if: matrix.type == 'deb-install' && always()
run: |
docker ps -a --filter "name=fips-deb-test-${{ matrix.scenario }}" --format '{{.Names}}' | while read -r c; do
docker rm -f "$c" >/dev/null 2>&1 || true
done
# ── DNS resolver multi-backend integration ──────────────────────────
# The dns-resolver harness builds its own fips binary from source in a
# Debian 12 builder image (shared cache layout with deb-install). Runs
# all 13 scenarios in a single job: dummy-TUN backend-detection tests
# plus real-fips end-to-end queries through systemd-resolved across
# five distros. ~7-12 min warm, ~12-15 min cold.
- name: Run dns-resolver test
if: matrix.type == 'dns-resolver'
timeout-minutes: 30
run: bash testing/dns-resolver/test.sh
- name: Collect logs on failure (dns-resolver)
if: matrix.type == 'dns-resolver' && failure()
run: |
docker ps -a --filter "name=fips-dns-test-" --format '{{.Names}}' | while read -r c; do
echo "--- ${c} fips.service ---"
docker exec "$c" journalctl -u fips.service --no-pager 2>&1 | tail -100 || true
echo "--- ${c} fips-dns.service ---"
docker exec "$c" journalctl -u fips-dns.service --no-pager 2>&1 | tail -100 || true
done
- name: Stop containers (dns-resolver)
if: matrix.type == 'dns-resolver' && always()
run: |
docker ps -a --filter "name=fips-dns-test-" --format '{{.Names}}' | while read -r c; do
docker rm -f "$c" >/dev/null 2>&1 || true
done
+16
View File
@@ -168,6 +168,22 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
mode where Linux's source-address routing check dropped outbound
flows from the loopback-bound socket. `outbound_only: true` is
exempt from the check (it overrides `bind_addr` to `0.0.0.0:0`)
- `fips-gateway` DNS upstream probe now retries up to 5 times with a
1-second per-attempt timeout and a 1-second delay between attempts
(~10 second worst-case wait), instead of a single 3-second hard-fail.
Covers the cold-boot race where the daemon's TUN is up (the systemd
ExecStartPre wait gates on that) but the DNS responder is still
binding `[::1]:5354`. Without retry the gateway exited and relied on
`Restart=on-failure` for recovery (5-second blip + spurious error
log line per cycle); with retry the gateway recovers gracefully
without a unit restart
- `packaging/debian/fips-gateway.service` now waits up to 30 seconds
for the daemon's `fips0` TUN to appear before exec'ing the gateway
binary (`ExecStartPre` poll loop). Eliminates the cold-boot race
where `fips-gateway` exits with `fips0 interface not found` and
recovers via `Restart=on-failure`, producing a 5-second blip and a
spurious error log line per restart cycle. If `fips0` never appears
within 30 seconds, the existing error path runs as before
- `packaging/debian/build-deb.sh` now auto-derives a per-commit Debian
Version field for dev builds (Cargo.toml version ending in `-dev`)
using the form `<base>~dev+git<YYYYMMDD>.<sha>[.dirty]-1`, e.g.
+10
View File
@@ -6,6 +6,16 @@ Requires=fips.service
[Service]
Type=simple
# Wait up to 30s for the fips daemon's TUN to come up before launching
# the gateway. fips-gateway hard-fails on a missing fips0 interface; with
# Requires=fips.service the daemon unit is started, but the TUN device
# itself is brought up some time after the daemon process starts. Without
# this wait the gateway's first ExecStart tends to lose a race on fresh
# boot, exit, and rely on Restart=on-failure for ~5s of recovery. Bounded
# wait avoids the noisy restart cycle while still surfacing real failures
# (if fips0 never appears, the wait expires and the gateway's existing
# error path runs).
ExecStartPre=/bin/sh -c 'for i in $(seq 1 30); do ip link show fips0 >/dev/null 2>&1 && exit 0; sleep 1; done; echo "fips0 did not appear within 30s" >&2; exit 1'
ExecStart=/usr/bin/fips-gateway --config /etc/fips/fips.yaml
Restart=on-failure
RestartSec=5
+51 -16
View File
@@ -202,37 +202,72 @@ async fn main() {
}
};
if let Err(e) = sock.send_to(&query, upstream_addr).await {
error!(
upstream = %upstream, error = %e,
"Failed to send DNS probe — is the FIPS daemon running?"
);
std::process::exit(1);
}
// Retry the upstream probe up to MAX_PROBE_ATTEMPTS times with a
// 1-second per-attempt timeout and a 1-second sleep between
// attempts. Total worst-case wait: ~10 seconds.
//
// Bounded retry covers the cold-boot race where this gateway and
// the fips daemon start at approximately the same time: the
// daemon's TUN may be up (the systemd ExecStartPre wait already
// gates on that) while its DNS responder is still binding
// [::1]:5354. Without this retry, the gateway hard-failed after
// a single 3-second probe and relied on Restart=on-failure for
// recovery.
const MAX_PROBE_ATTEMPTS: u32 = 5;
const PROBE_TIMEOUT_SECS: u64 = 1;
const PROBE_RETRY_DELAY_SECS: u64 = 1;
let mut buf = [0u8; 512];
match tokio::time::timeout(std::time::Duration::from_secs(3), sock.recv_from(&mut buf))
let mut last_failure: Option<String> = None;
let mut succeeded = false;
for attempt in 1..=MAX_PROBE_ATTEMPTS {
if let Err(e) = sock.send_to(&query, upstream_addr).await {
last_failure = Some(format!("send_to failed: {}", e));
} else {
match tokio::time::timeout(
std::time::Duration::from_secs(PROBE_TIMEOUT_SECS),
sock.recv_from(&mut buf),
)
.await
{
Ok(Ok(_)) => {
info!(upstream = %upstream, "DNS upstream is reachable");
info!(
upstream = %upstream, attempt = attempt,
"DNS upstream is reachable"
);
succeeded = true;
break;
}
Ok(Err(e)) => {
error!(
upstream = %upstream, error = %e,
"DNS upstream recv failed — is the FIPS daemon running?"
);
std::process::exit(1);
last_failure = Some(format!("recv_from failed: {}", e));
}
Err(_) => {
last_failure = Some(format!(
"no response within {}s",
PROBE_TIMEOUT_SECS
));
}
}
}
if attempt < MAX_PROBE_ATTEMPTS {
info!(
upstream = %upstream, attempt = attempt,
last = ?last_failure,
"DNS upstream probe attempt failed; retrying"
);
tokio::time::sleep(std::time::Duration::from_secs(PROBE_RETRY_DELAY_SECS)).await;
}
}
if !succeeded {
error!(
upstream = %upstream,
"DNS upstream did not respond within 3s — is the FIPS daemon running?"
attempts = MAX_PROBE_ATTEMPTS,
last = ?last_failure,
"DNS upstream did not become reachable after exhausting retries — is the FIPS daemon running?"
);
std::process::exit(1);
}
}
}
// --- Initialize components ---
+39
View File
@@ -1017,3 +1017,42 @@ async fn test_should_admit_msg1_admits_rekey_when_accept_off() {
assert!(node.should_admit_msg1(transport_id, &addr));
}
/// Same regression coverage as the TCP test above, but exercising the
/// UDP transport's new `accept_connections` config field (introduced
/// alongside the `outbound_only` mode). Proves the Node-level gate's
/// addr_to_link carve-out is transport-agnostic and that the new UDP
/// config knob is wired correctly through the Transport trait.
#[tokio::test]
async fn test_should_admit_msg1_admits_rekey_when_udp_accept_off() {
use crate::config::UdpConfig;
use crate::transport::udp::UdpTransport;
let mut node = make_node();
let transport_id = TransportId::new(1);
let cfg = UdpConfig {
bind_addr: Some("127.0.0.1:0".to_string()),
accept_connections: Some(false),
..Default::default()
};
let (tx, _rx) = packet_channel(64);
let udp = UdpTransport::new(transport_id, None, cfg, tx);
node.transports
.insert(transport_id, TransportHandle::Udp(udp));
let addr = TransportAddr::from_string("10.0.0.2:2121");
// Fresh msg1 (no addr_to_link entry) is rejected by the gate when
// the transport refuses inbound.
assert!(!node.should_admit_msg1(transport_id, &addr));
// Pre-populate addr_to_link as if a session were established. The
// rekey carve-out admits the msg1 even though the transport still
// says accept_connections() == false.
let link_id = node.allocate_link_id();
node.addr_to_link
.insert((transport_id, addr.clone()), link_id);
assert!(node.should_admit_msg1(transport_id, &addr));
}
@@ -38,7 +38,6 @@ tun:
dns:
enabled: true
bind_addr: "127.0.0.1"
transports:
udp:
@@ -104,7 +103,6 @@ tun:
dns:
enabled: true
bind_addr: "127.0.0.1"
transports:
udp:
@@ -158,7 +156,6 @@ tun:
dns:
enabled: true
bind_addr: "127.0.0.1"
transports:
udp:
@@ -202,7 +199,6 @@ tun:
dns:
enabled: true
bind_addr: "127.0.0.1"
transports:
udp:
@@ -246,7 +242,6 @@ tun:
dns:
enabled: true
bind_addr: "127.0.0.1"
transports:
udp:
@@ -277,7 +272,6 @@ tun:
dns:
enabled: true
bind_addr: "127.0.0.1"
transports:
udp:
-1
View File
@@ -13,7 +13,6 @@ tun:
dns:
enabled: true
bind_addr: "127.0.0.1"
transports:
udp:
+163 -9
View File
@@ -7,19 +7,24 @@
# --build-only Only run build + clippy
# --test-only Only run unit tests (skip build, skip integration)
# --skip-integration Skip integration tests
# --skip-chaos Skip chaos scenarios (run static + rekey + sidecar only)
# --skip-chaos Skip chaos scenarios
# --with-tor Include Tor harnesses (off by default — needs live Tor)
# --only <suite> Run a single integration suite
# -j, --jobs <N> Max parallel chaos scenarios (default: 4)
# --list List available integration suites
# -h, --help Show this help
#
# Integration suites:
# static-mesh, static-chain, rekey, gateway,
# Integration suites (default coverage):
# static-mesh, static-chain, rekey, rekey-accept-off, gateway,
# acl-allowlist, nat-cone, nat-symmetric, nat-lan,
# chaos-smoke-10, chaos-churn-mixed-10, chaos-ethernet-mesh,
# chaos-ethernet-only, chaos-tcp-mesh, chaos-bottleneck-parent,
# chaos-cost-avoidance, chaos-cost-reeval, chaos-cost-stability,
# chaos-depth-vs-cost, chaos-mixed-technology, chaos-congestion-stress,
# sidecar
# sidecar, dns-resolver, deb-install
#
# Opt-in (require --with-tor; depend on live Tor network):
# tor-socks5, tor-directory
#
# Exit codes:
# 0 — all stages passed
@@ -43,11 +48,12 @@ BUILD_ONLY=false
TEST_ONLY=false
SKIP_INTEGRATION=false
SKIP_CHAOS=false
WITH_TOR=false
ONLY_SUITE=""
# All integration suites matching ci.yml
STATIC_SUITES=(static-mesh static-chain)
REKEY_SUITES=(rekey)
REKEY_SUITES=(rekey rekey-accept-off)
# Each entry: "display-name scenario [--flag value ...]"
CHAOS_SUITES=(
"smoke-10 smoke-10"
@@ -65,6 +71,11 @@ CHAOS_SUITES=(
)
GATEWAY_SUITES=(gateway)
SIDECAR_SUITES=(sidecar)
ACL_SUITES=(acl-allowlist)
NAT_SUITES=(cone symmetric lan)
DNS_RESOLVER_SUITES=(dns-resolver)
DEB_INSTALL_SUITES=(deb-install)
TOR_SUITES=(tor-socks5 tor-directory)
# ── Colors ─────────────────────────────────────────────────────────────────
@@ -93,17 +104,32 @@ list_suites() {
echo " Rekey:"
for s in "${REKEY_SUITES[@]}"; do echo " $s"; done
echo ""
echo " Gateway:"
for s in "${GATEWAY_SUITES[@]}"; do echo " $s"; done
echo ""
echo " ACL allowlist:"
for s in "${ACL_SUITES[@]}"; do echo " $s"; done
echo ""
echo " NAT scenarios:"
for s in "${NAT_SUITES[@]}"; do echo " nat-$s"; done
echo ""
echo " Chaos scenarios:"
for entry in "${CHAOS_SUITES[@]}"; do
read -ra parts <<< "$entry"
echo " chaos-${parts[0]} (${parts[*]:1})"
done
echo ""
echo " Gateway:"
for s in "${GATEWAY_SUITES[@]}"; do echo " $s"; done
echo ""
echo " Sidecar:"
for s in "${SIDECAR_SUITES[@]}"; do echo " $s"; done
echo ""
echo " DNS resolver:"
for s in "${DNS_RESOLVER_SUITES[@]}"; do echo " $s"; done
echo ""
echo " Deb-install:"
for s in "${DEB_INSTALL_SUITES[@]}"; do echo " $s"; done
echo ""
echo " Tor (opt-in via --with-tor):"
for s in "${TOR_SUITES[@]}"; do echo " $s"; done
exit 0
}
@@ -120,6 +146,7 @@ while [[ $# -gt 0 ]]; do
--test-only) TEST_ONLY=true; shift ;;
--skip-integration) SKIP_INTEGRATION=true; shift ;;
--skip-chaos) SKIP_CHAOS=true; shift ;;
--with-tor) WITH_TOR=true; shift ;;
--only) ONLY_SUITE="$2"; shift 2 ;;
-j|--jobs) PARALLEL_JOBS="$2"; shift 2 ;;
--list) list_suites ;;
@@ -318,6 +345,98 @@ run_sidecar() {
record "sidecar" $rc
}
# Run the rekey-accept-off integration variant. Same harness as run_rekey
# but on a 2-node topology with udp.accept_connections=false on node-b.
run_rekey_accept_off() {
local compose="testing/static/docker-compose.yml"
local rc=0
info "[rekey-accept-off] Generating configs"
bash testing/static/scripts/generate-configs.sh rekey-accept-off || \
{ record "rekey-accept-off" 1; return; }
REKEY_TOPOLOGY=rekey-accept-off REKEY_ACCEPT_OFF_NODES=b \
bash testing/static/scripts/rekey-test.sh inject-config || \
{ record "rekey-accept-off" 1; return; }
info "[rekey-accept-off] Starting containers"
docker compose -f "$compose" --profile rekey-accept-off up -d || \
{ record "rekey-accept-off" 1; return; }
info "[rekey-accept-off] Running rekey test"
if REKEY_TOPOLOGY=rekey-accept-off REKEY_ACCEPT_OFF_NODES=b \
bash testing/static/scripts/rekey-test.sh; then
rc=0
else
rc=1
info "[rekey-accept-off] Collecting failure logs"
docker compose -f "$compose" --profile rekey-accept-off logs --no-color 2>&1 | tail -100
fi
docker compose -f "$compose" --profile rekey-accept-off down --volumes --remove-orphans 2>/dev/null
record "rekey-accept-off" $rc
}
# Run ACL allowlist integration test
run_acl_allowlist() {
info "[acl-allowlist] Running integration test"
if bash testing/acl-allowlist/test.sh --skip-build 2>&1; then
record "acl-allowlist" 0
else
record "acl-allowlist" 1
fi
}
# Run a NAT scenario (cone, symmetric, lan)
run_nat() {
local scenario="$1"
info "[nat-$scenario] Running NAT lab"
if bash testing/nat/scripts/nat-test.sh "$scenario" 2>&1; then
record "nat-$scenario" 0
else
record "nat-$scenario" 1
fi
}
# Run dns-resolver harness (multi-distro + e2e scenarios)
run_dns_resolver() {
info "[dns-resolver] Running multi-distro test (slow — builds per-distro images)"
if bash testing/dns-resolver/test.sh 2>&1; then
record "dns-resolver" 0
else
record "dns-resolver" 1
fi
}
# Run deb-install harness (multi-distro real-package install)
run_deb_install() {
info "[deb-install] Running multi-distro test (slow — builds .deb + per-distro install)"
if bash testing/deb-install/test.sh 2>&1; then
record "deb-install" 0
else
record "deb-install" 1
fi
}
# Run Tor SOCKS5 outbound test (live Tor network)
run_tor_socks5() {
info "[tor-socks5] Running Tor SOCKS5 outbound test (live Tor)"
if bash testing/tor/socks5-outbound/scripts/tor-test.sh 2>&1; then
record "tor-socks5" 0
else
record "tor-socks5" 1
fi
}
# Run Tor directory-mode test (live Tor network)
run_tor_directory() {
info "[tor-directory] Running Tor directory-mode test (live Tor)"
if bash testing/tor/directory-mode/scripts/directory-test.sh 2>&1; then
record "tor-directory" 0
else
record "tor-directory" 1
fi
}
# Determine which suites to run and execute them
run_integration() {
stage "Stage 3: Integration Tests"
@@ -343,12 +462,21 @@ run_integration() {
run_static "$topology"
done
# Rekey
# Rekey + rekey-accept-off variant
run_rekey
run_rekey_accept_off
# Gateway
run_gateway
# ACL allowlist
run_acl_allowlist
# NAT scenarios (sequential — each owns its compose project)
for scenario in "${NAT_SUITES[@]}"; do
run_nat "$scenario"
done
# Chaos scenarios (parallel, throttled)
if [[ "$SKIP_CHAOS" != true ]]; then
info "Running ${#CHAOS_SUITES[@]} chaos scenarios (max $PARALLEL_JOBS parallel)"
@@ -401,6 +529,18 @@ run_integration() {
# Sidecar
run_sidecar
# DNS resolver multi-distro suite (heavy — per-distro systemd images)
run_dns_resolver
# Deb-install multi-distro suite (heavy — builds .deb + per-distro install)
run_deb_install
# Tor (opt-in via --with-tor; depends on live Tor network)
if [[ "$WITH_TOR" == true ]]; then
run_tor_socks5
run_tor_directory
fi
}
# Run a single named suite
@@ -411,8 +551,14 @@ run_suite() {
run_static "${suite#static-}" ;;
rekey)
run_rekey ;;
rekey-accept-off)
run_rekey_accept_off ;;
gateway)
run_gateway ;;
acl-allowlist)
run_acl_allowlist ;;
nat-cone|nat-symmetric|nat-lan)
run_nat "${suite#nat-}" ;;
chaos-*)
local chaos_name="${suite#chaos-}"
local found=false
@@ -431,6 +577,14 @@ run_suite() {
;;
sidecar)
run_sidecar ;;
dns-resolver)
run_dns_resolver ;;
deb-install)
run_deb_install ;;
tor-socks5)
run_tor_socks5 ;;
tor-directory)
run_tor_directory ;;
*)
fail "Unknown suite: $suite"
record "$suite" 1 ;;
+1
View File
@@ -148,6 +148,7 @@ WORKDIR /src
COPY Cargo.toml Cargo.lock build.rs LICENSE README.md ./
COPY src ./src
COPY packaging ./packaging
COPY docs ./docs
RUN cargo build --release && cargo deb --no-build
DOCKERFILE
+1 -1
View File
@@ -31,7 +31,7 @@ RUN printf '%s\n' \
'port=53' \
'listen-address=127.0.0.1' \
'bind-interfaces' \
'server=/fips/127.0.0.1#5354' \
'server=/fips/::1#5354' \
'server=127.0.0.11' \
'no-resolv' \
>> /etc/dnsmasq.conf
+19 -1
View File
@@ -100,7 +100,6 @@ tun:
dns:
enabled: true
bind_addr: "127.0.0.1"
transports:
udp:
@@ -236,6 +235,25 @@ case "$MODE" in
echo "FATAL: fips0 did not appear within 30s"
exit 1
fi
# Wait for the daemon's DNS responder to bind [::1]:5354 before
# exec'ing fips-gateway. The gateway binary's startup probe is
# bounded (5 attempts × 1s with retry); this harness wait is the
# belt to that suspenders so we get deterministic CI behaviour
# on slow runners. Bounded to ~30 seconds; if the daemon
# really never binds DNS, the gateway's own probe will report
# the definitive error after this wait expires.
for i in $(seq 1 30); do
if dig @::1 -p 5354 +tries=1 +time=1 test.fips >/dev/null 2>&1; then
echo "Daemon DNS ready (waited ~${i}s)"
break
fi
if [ "$i" -eq 30 ]; then
echo "WARNING: daemon DNS did not respond within ~30s; proceeding with gateway startup"
fi
sleep 1
done
echo "fips0 ready, starting gateway"
exec fips-gateway --config "$CONFIG" --log-level debug
;;
-1
View File
@@ -104,7 +104,6 @@ tun:
dns:
enabled: true
bind_addr: "127.0.0.1"
port: 5354
transports:
@@ -13,7 +13,6 @@ tun:
dns:
enabled: true
bind_addr: "127.0.0.1"
transports:
udp:
@@ -0,0 +1,44 @@
# Rekey Integration Test Topology — accept_connections: false variant.
#
# Same sparse mesh as rekey.yaml. Configs are post-processed to use
# aggressive rekey timers (35s, same as the parent) AND to set
# `transports.udp.accept_connections: false` on node b — the single-peer
# node, auto-connected to c. That auto_connect-initiator-with-accept-
# off shape is what produced the pre-fix master deadlock: msg1 from c
# during a rekey arrived at b's Node-level gate, and the gate dropped
# it because its transport returned accept_connections() == false.
# The carve-out (msg1 admitted unconditionally when addr_to_link has
# the source) restores rekey. This topology pins that regression; the
# unit test `test_should_admit_msg1_admits_rekey_when_udp_accept_off`
# is the fast-feedback complement.
nodes:
a:
nsec: "0102030405060708090a0b0c0d0e0f101112131415161718191a1b1c1d1e1f20"
npub: "npub1sjlh2c3x9w7kjsqg2ay080n2lff2uvt325vpan33ke34rn8l5jcqawh57m"
docker_ip: "172.20.0.10"
peers: [d, e]
b:
nsec: "b102030405060708090a0b0c0d0e0f101112131415161718191a1b1c1d1e1fb0"
npub: "npub1tdwa4vjrjl33pcjdpf2t4p027nl86xrx24g4d3avg4vwvayr3g8qhd84le"
docker_ip: "172.20.0.11"
peers: [c]
c:
nsec: "c102030405060708090a0b0c0d0e0f101112131415161718191a1b1c1d1e1fc0"
npub: "npub1cld9yay0u24davpu6c35l4vldrhzvaq66pcqtg9a0j2cnjrn9rtsxx2pe6"
docker_ip: "172.20.0.12"
peers: [b, d, e]
d:
nsec: "d102030405060708090a0b0c0d0e0f101112131415161718191a1b1c1d1e1fd0"
npub: "npub1n9lpnv0592cc2ps6nm0ca3qls642vx7yjsv35rkxqzj2vgds52sqgpverl"
docker_ip: "172.20.0.13"
peers: [a, c, e]
e:
nsec: "e102030405060708090a0b0c0d0e0f101112131415161718191a1b1c1d1e1fe0"
npub: "npub1wf8akf8lu2zdkjkmwhl75pqvven654mpv4sz2x2tprl5265mgrzq8nhak4"
docker_ip: "172.20.0.14"
peers: [a, c, d]
+75
View File
@@ -281,6 +281,81 @@ services:
fips-net:
ipv4_address: 172.20.0.14
# ── Rekey + accept_connections=false on node b ──────────────────
# Exercises the auto_connect-initiator-with-accept-off regression
# class. Same 5-node mesh, but node b's generated config has
# `transports.udp.accept_connections: false` (injected by
# rekey-test.sh's inject-config when REKEY_ACCEPT_OFF_NODES=b).
rekey-accept-off-a:
<<: *fips-common
profiles: ["rekey-accept-off"]
container_name: fips-node-a
hostname: node-a
environment:
- RUST_LOG=info,fips::node::handlers::rekey=debug,fips::node::handlers::handshake=debug
volumes:
- ../docker/resolv.conf:/etc/resolv.conf:ro
- ./generated-configs/rekey-accept-off/node-a.yaml:/etc/fips/fips.yaml:ro
networks:
fips-net:
ipv4_address: 172.20.0.10
rekey-accept-off-b:
<<: *fips-common
profiles: ["rekey-accept-off"]
container_name: fips-node-b
hostname: node-b
environment:
- RUST_LOG=info,fips::node::handlers::rekey=debug,fips::node::handlers::handshake=debug
volumes:
- ../docker/resolv.conf:/etc/resolv.conf:ro
- ./generated-configs/rekey-accept-off/node-b.yaml:/etc/fips/fips.yaml:ro
networks:
fips-net:
ipv4_address: 172.20.0.11
rekey-accept-off-c:
<<: *fips-common
profiles: ["rekey-accept-off"]
container_name: fips-node-c
hostname: node-c
environment:
- RUST_LOG=info,fips::node::handlers::rekey=debug,fips::node::handlers::handshake=debug
volumes:
- ../docker/resolv.conf:/etc/resolv.conf:ro
- ./generated-configs/rekey-accept-off/node-c.yaml:/etc/fips/fips.yaml:ro
networks:
fips-net:
ipv4_address: 172.20.0.12
rekey-accept-off-d:
<<: *fips-common
profiles: ["rekey-accept-off"]
container_name: fips-node-d
hostname: node-d
environment:
- RUST_LOG=info,fips::node::handlers::rekey=debug,fips::node::handlers::handshake=debug
volumes:
- ../docker/resolv.conf:/etc/resolv.conf:ro
- ./generated-configs/rekey-accept-off/node-d.yaml:/etc/fips/fips.yaml:ro
networks:
fips-net:
ipv4_address: 172.20.0.13
rekey-accept-off-e:
<<: *fips-common
profiles: ["rekey-accept-off"]
container_name: fips-node-e
hostname: node-e
environment:
- RUST_LOG=info,fips::node::handlers::rekey=debug,fips::node::handlers::handshake=debug
volumes:
- ../docker/resolv.conf:/etc/resolv.conf:ro
- ./generated-configs/rekey-accept-off/node-e.yaml:/etc/fips/fips.yaml:ro
networks:
fips-net:
ipv4_address: 172.20.0.14
# ── TCP chain topology (A-B-C) ───────────────────────────────
tcp-a:
<<: *fips-common
+54 -2
View File
@@ -20,8 +20,17 @@ set -e
SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
source "$SCRIPT_DIR/../../lib/wait-converge.sh"
TOPOLOGY="rekey"
# Selectable topology — defaults to "rekey" but the rekey-accept-off
# variant exercises the auto_connect-initiator-with-accept-off
# regression class (udp.accept_connections=false on a peer that
# also auto-connects).
TOPOLOGY="${REKEY_TOPOLOGY:-rekey}"
NODES="a b c d e"
# Comma-separated list of node IDs to set udp.accept_connections=false
# on during inject-config. Empty (default) leaves all nodes accepting.
# When set, also asserted by the test that no sustained "Dual rekey
# initiation" log lines appear on the affected node.
REKEY_ACCEPT_OFF_NODES="${REKEY_ACCEPT_OFF_NODES:-}"
# Rekey timing configuration
REKEY_AFTER_SECS=35
@@ -30,13 +39,24 @@ REKEY_AFTER_SECS=35
# Inject rekey config into generated node configs. Called separately
# by CI before building Docker images.
if [ "${1:-}" = "inject-config" ]; then
echo "Injecting rekey config (after_secs=$REKEY_AFTER_SECS) into node configs..."
echo "Injecting rekey config (after_secs=$REKEY_AFTER_SECS) into node configs (topology=$TOPOLOGY)..."
if [ -n "$REKEY_ACCEPT_OFF_NODES" ]; then
echo " Setting udp.accept_connections=false on nodes: $REKEY_ACCEPT_OFF_NODES"
fi
for node in $NODES; do
cfg="$SCRIPT_DIR/../generated-configs/$TOPOLOGY/node-$node.yaml"
if [ ! -f "$cfg" ]; then
echo " Error: $cfg not found" >&2
exit 1
fi
accept_off="false"
if [ -n "$REKEY_ACCEPT_OFF_NODES" ]; then
for off_node in ${REKEY_ACCEPT_OFF_NODES//,/ }; do
if [ "$off_node" = "$node" ]; then
accept_off="true"
fi
done
fi
python3 -c "
import yaml
with open('$cfg') as f:
@@ -46,10 +66,22 @@ cfg.setdefault('node', {})['rekey'] = {
'after_secs': $REKEY_AFTER_SECS,
'after_messages': 65536,
}
if '$accept_off' == 'true':
transports = cfg.setdefault('transports', {})
udp = transports.get('udp')
if udp is None:
udp = {'bind_addr': '0.0.0.0:2121'}
transports['udp'] = udp
if isinstance(udp, dict):
udp['accept_connections'] = False
with open('$cfg', 'w') as f:
yaml.dump(cfg, f, default_flow_style=False, sort_keys=False)
"
if [ "$accept_off" = "true" ]; then
echo " ✓ node-$node (accept_connections=false)"
else
echo " ✓ node-$node"
fi
done
echo "✓ Config injection complete"
exit 0
@@ -316,6 +348,26 @@ assert_zero_count "Rekey msg2 processing failed" "Rekey msg2 failures"
assert_zero_count "Session AEAD decryption failed" \
"FSP decryption failures during rekey"
# Variant-specific: when one or more nodes have udp.accept_connections=false,
# verify the dual-init carve-out keeps the "we win, dropping their msg1"
# log line below the bug threshold. Pre-fix, a 1Hz dual-init loop produced
# ~120 occurrences over the 2-minute test; with the carve-out, the line
# fires at most a handful of times from genuine simultaneous rekeys.
if [ -n "$REKEY_ACCEPT_OFF_NODES" ]; then
DUAL_INIT_THRESHOLD=10
for off_node in ${REKEY_ACCEPT_OFF_NODES//,/ }; do
count=$(docker logs "fips-node-$off_node" 2>&1 \
| grep -cE "Dual rekey initiation: we win" || true)
if [ "${count:-0}" -le "$DUAL_INIT_THRESHOLD" ]; then
echo " PASS: node-$off_node dual-init drops below threshold ($count <= $DUAL_INIT_THRESHOLD)"
PASSED=$((PASSED + 1))
else
echo " FAIL: node-$off_node sustained dual-init drops ($count > $DUAL_INIT_THRESHOLD)"
FAILED=$((FAILED + 1))
fi
done
fi
phase_result "Log analysis"
echo ""
@@ -15,7 +15,6 @@ tun:
dns:
enabled: true
bind_addr: "127.0.0.1"
transports:
tor:
@@ -15,7 +15,6 @@ tun:
dns:
enabled: true
bind_addr: "127.0.0.1"
transports:
tor:
@@ -14,7 +14,6 @@ tun:
dns:
enabled: true
bind_addr: "127.0.0.1"
transports:
tor:
@@ -14,7 +14,6 @@ tun:
dns:
enabled: true
bind_addr: "127.0.0.1"
transports:
tor: