Fix DNS responder silent-drop on systemd-resolved deployments

The previous default configured systemd-resolved with `resolvectl dns
fips0 [<fips0_addr>]:5354`, intended to bypass an Ubuntu 22 systemd 249
interface-scoping bug. That target collides with the daemon's
mesh-interface filter on Linux: when an IPv6 packet's destination
belongs to a non-loopback interface, the kernel attributes the packet
to that interface in IPV6_PKTINFO (ipi6_ifindex == fips0) even though
loopback delivery is used (tcpdump shows lo). The mesh-interface filter
sees arrival_ifindex == mesh_ifindex and silently drops every query at
trace level — invisible to operators at the default debug level.

Net effect on stock deployments: every .fips query on systemd-resolved
hosts was silently dropped.

Daemon side
-----------

- Default `dns.bind_addr` changes from "::" to "::1" (IPv6 loopback
  only). The mesh-interface filter is then defanged on the default
  path because loopback isn't reachable from mesh peers. The filter
  remains in place defensively for operators who explicitly bind "::"
  to expose a mesh-reachable responder.

fips-dns-setup backend unification
----------------------------------

- New `try_global_drop_in` backend writes
  /etc/systemd/resolved.conf.d/fips.conf with DNS=[::1]:5354 and
  Domains=~fips. Inserted ahead of `try_resolvectl` in the dispatch
  chain. The standard loopback path has no interface scoping, so
  ipi6_ifindex reports lo and the filter passes.
- All other backends now target [::1]:5354 to match the daemon's
  default IPv6-loopback bind:
  - try_dns_delegate writes DNS=[::1]:5354
  - try_dnsmasq writes server=/fips/::1#5354
  - try_nm_dnsmasq writes server=/fips/::1#5354
- Fixed dns-delegate file path: was /etc/systemd/dns-delegate/, must
  be /etc/systemd/dns-delegate.d/ (with .d suffix). systemd-resolved
  silently ignored the previous path.
- fips-dns-teardown handles the new global-drop-in backend in cleanup.
- The legacy resolvectl per-link backend stays as a fallback,
  documented to require careful daemon bind_addr coordination.

fips-gateway upstream pairing
-----------------------------

- gateway.dns.upstream default changes from 127.0.0.1:5354 to
  [::1]:5354 to match the daemon's default bind. Linux IPv6 sockets
  bound to explicit ::1 do not accept v4-mapped traffic, so the old
  default would have caused the gateway's startup DNS reachability
  probe to time out and systemd to restart-loop the service.
- Operators who set a non-default daemon `dns.bind_addr` must also
  set `gateway.dns.upstream` to match — documented inline.

Documentation
-------------

- packaging/common/fips.yaml and packaging/openwrt-ipk fips.yaml
  examples updated; rationale for the bind_addr choice and the
  daemon/gateway pairing recorded inline.

Test coverage
-------------

- testing/dns-resolver/test.sh: real-fipsd end-to-end scenario added.
  Builds fipsd in a Debian 12 builder image (cached), runs the daemon
  with a real TUN in a privileged container, configures DNS via the
  setup script, and asserts `dig @127.0.0.53 AAAA <npub>.fips` returns
  AAAA. Refactored as a parameterized helper running across Debian
  12/13 and Ubuntu 22/24/26 (5 e2e scenarios). Backend-aware
  assertions: on systemd >= 258 the expected backend is dns-delegate;
  on older systemd it's global-drop-in. Strict content checks fail CI
  on any [::1]:5354 drift. fips-gateway also exercised in the
  debian12 scenario to lock the gateway-upstream pairing. Renamed all
  "fipsd" references to "fips" (project convention).

- testing/deb-install/ (new harness): builds the actual .deb via
  cargo-deb in a Debian 12 builder image (cached), installs via apt
  across each target distro, verifies maintainer scripts, conffile
  placement, binary placement, and end-to-end .fips resolution after
  start. Also exercises fips-gateway against the installed daemon to
  verify the gateway/daemon default pairing on a real .deb path.

- This is the test layer that was missing — the previous harness only
  verified config files were written, never that queries reached the
  daemon.

Verified: dns-resolver 78/78 assertions, deb-install 55/55 assertions
across all 5 distros (debian:12, debian:trixie, ubuntu:22.04,
ubuntu:24.04, ubuntu:26.04).
This commit is contained in:
Johnathan Corgan
2026-04-29 12:50:11 +00:00
parent 34e00b9f6e
commit bf77ececad
11 changed files with 1150 additions and 87 deletions
+1
View File
@@ -0,0 +1 @@
.cache/
+449
View File
@@ -0,0 +1,449 @@
#!/bin/bash
# Test the fips Debian package install path across target distros.
#
# Each scenario builds (or reuses) the .deb in a Debian 12 cargo-deb
# builder image (cached), boots a privileged systemd container with
# TUN access for the target distro, installs the .deb via `apt
# install ./fips_*.deb`, waits for fips.service + fips-dns.service
# to come up, and verifies that `dig @127.0.0.53 AAAA <npub>.fips`
# returns a non-empty AAAA answer through the resolver backend that
# fips-dns-setup configured. Then exercises fips-gateway against the
# same daemon to verify the gateway/daemon default-pairing.
#
# This is the most thorough test surface — it exercises:
# - cargo deb packaging (binary stripping, dependency declaration)
# - dpkg conffile placement (/etc/fips/fips.yaml)
# - postinst maintainer scripts (systemd unit enablement,
# fips-dns.service running fips-dns-setup)
# - The fips, fips-dns, and (optionally) fips-gateway systemd units
# - End-to-end .fips resolution as a real user would experience it
#
# Usage: ./test.sh [scenario ...]
# No args = run all scenarios.
# Named args = run only those (e.g., ./test.sh ubuntu26 debian12)
#
# Requirements: Docker with privileged container support, /dev/net/tun
# on the host (standard).
set -uo pipefail
SCRIPT_DIR="$(cd "$(dirname "$0")" && pwd)"
REPO_ROOT="$(cd "$SCRIPT_DIR/../.." && pwd)"
CACHE_DIR="$SCRIPT_DIR/.cache"
DEB_CACHE_DIR="$CACHE_DIR/deb"
# Timeouts
BOOT_TIMEOUT=30
SERVICE_TIMEOUT=20
DAEMON_TIMEOUT=15
PASS=0
FAIL=0
SKIP=0
# ─────────────────────────────────────────────────────────────────────
# Helpers
# ─────────────────────────────────────────────────────────────────────
log() { echo "=== $*"; }
pass() { echo " PASS: $*"; PASS=$((PASS + 1)); }
fail() { echo " FAIL: $*"; FAIL=$((FAIL + 1)); }
skip() { echo " SKIP: $*"; SKIP=$((SKIP + 1)); }
cleanup_container() {
local name="$1"
docker rm -f "$name" >/dev/null 2>&1 || true
}
build_image() {
local tag="$1"
shift
echo "$@" | docker build -t "$tag" -f - "$REPO_ROOT" >/dev/null 2>&1
}
start_systemd_container_with_tun() {
local name="$1" image="$2"
cleanup_container "$name"
docker run -d --name "$name" \
--privileged \
--cgroupns=host \
--device /dev/net/tun \
-v /sys/fs/cgroup:/sys/fs/cgroup:rw \
--tmpfs /run --tmpfs /run/lock \
"$image" >/dev/null 2>&1
}
wait_for_systemd() {
local name="$1"
for _i in $(seq 1 "$BOOT_TIMEOUT"); do
if docker exec "$name" systemctl is-system-running --wait 2>/dev/null | grep -qE 'running|degraded'; then
return 0
fi
sleep 1
done
echo " WARNING: systemd did not reach running state in ${BOOT_TIMEOUT}s (may still work)"
return 0
}
wait_for_service_active() {
local name="$1" service="$2" timeout="${3:-$SERVICE_TIMEOUT}"
for _i in $(seq 1 "$timeout"); do
if docker exec "$name" systemctl is-active --quiet "$service" 2>/dev/null; then
return 0
fi
sleep 1
done
return 1
}
container_systemd_version() {
local name="$1"
docker exec "$name" systemctl --version 2>/dev/null | head -1 \
| grep -oE '[0-9]+' | head -1
}
# ─────────────────────────────────────────────────────────────────────
# Build the .deb once in a Debian 12 cargo-deb builder image (cached
# between runs). Output cached at testing/deb-install/.cache/deb/.
# Rebuilt if any source/Cargo/packaging file is newer than the cached
# .deb, or if the .deb is missing.
# ─────────────────────────────────────────────────────────────────────
build_deb() {
mkdir -p "$DEB_CACHE_DIR"
local cached_deb
cached_deb=$(ls "$DEB_CACHE_DIR"/fips_*_amd64.deb 2>/dev/null | head -1)
if [ -n "$cached_deb" ] && [ -f "$cached_deb" ]; then
local newest_src
newest_src=$(find "$REPO_ROOT/src" "$REPO_ROOT/Cargo.toml" \
"$REPO_ROOT/Cargo.lock" "$REPO_ROOT/packaging" \
-type f -printf '%T@\n' 2>/dev/null | sort -nr | head -1)
local cached_age
cached_age=$(stat -c '%Y' "$cached_deb" 2>/dev/null || echo 0)
if awk "BEGIN { exit !($cached_age >= $newest_src) }"; then
log "Using cached .deb at $cached_deb"
return 0
fi
log "Cached .deb is stale, rebuilding"
else
log "No cached .deb, building"
fi
local builder_tag="fips-deb-test:builder"
log "Building Debian 12 cargo-deb builder image (slow on first run)"
docker build -t "$builder_tag" -f - "$REPO_ROOT" <<'DOCKERFILE' >/dev/null
FROM debian:12
ENV DEBIAN_FRONTEND=noninteractive
RUN apt-get update && apt-get install -y --no-install-recommends \
build-essential pkg-config libdbus-1-dev curl ca-certificates \
libclang-dev clang && \
apt-get clean && rm -rf /var/lib/apt/lists/*
RUN curl --proto '=https' --tlsv1.2 -sSf https://sh.rustup.rs | \
sh -s -- -y --default-toolchain stable --profile minimal
ENV PATH="/root/.cargo/bin:${PATH}"
RUN cargo install cargo-deb --version 3.6.3 --locked
WORKDIR /src
COPY Cargo.toml Cargo.lock build.rs LICENSE README.md ./
COPY src ./src
COPY packaging ./packaging
RUN cargo build --release && cargo deb --no-build
DOCKERFILE
if [ ! "$(docker images -q "$builder_tag" 2>/dev/null)" ]; then
echo " ERROR: builder image build failed"
return 1
fi
log "Extracting .deb from builder image"
rm -f "$DEB_CACHE_DIR"/*.deb
local cid
cid=$(docker create "$builder_tag")
docker cp "$cid:/src/target/debian/." "$DEB_CACHE_DIR/" >/dev/null 2>&1
docker rm "$cid" >/dev/null
# Cargo-deb leaves intermediate artifacts; keep just the .deb.
find "$DEB_CACHE_DIR" -mindepth 1 -not -name 'fips_*_amd64.deb' -delete 2>/dev/null || true
cached_deb=$(ls "$DEB_CACHE_DIR"/fips_*_amd64.deb 2>/dev/null | head -1)
if [ -n "$cached_deb" ]; then
log "Cached at $cached_deb ($(stat -c %s "$cached_deb") bytes)"
else
echo " ERROR: no .deb produced by cargo-deb"
return 1
fi
}
# ─────────────────────────────────────────────────────────────────────
# Scenario runner
#
# Args: <distro_label> <docker_base_image>
# distro_label: short tag for container/image names (e.g. "debian12")
# docker_base_image: e.g. "debian:12", "ubuntu:26.04"
# ─────────────────────────────────────────────────────────────────────
_run_deb_install_scenario() {
local distro_label="$1"
local base_image="$2"
local name="fips-deb-test-${distro_label}"
local image="fips-deb-test:${distro_label}"
log ".deb install: ${base_image}"
build_deb || { fail ".deb build failed"; return; }
local cached_deb
cached_deb=$(ls "$DEB_CACHE_DIR"/fips_*_amd64.deb 2>/dev/null | head -1)
if [ -z "$cached_deb" ] || [ ! -f "$cached_deb" ]; then
fail "no .deb available at $DEB_CACHE_DIR"
return
fi
local deb_basename
deb_basename=$(basename "$cached_deb")
# Ubuntu 22.04 bundles systemd-resolved into systemd; other
# distros require it as a separate package. Compose the apt
# package list accordingly.
local apt_packages="systemd iproute2 dbus dnsutils procps"
if [ "$base_image" != "ubuntu:22.04" ]; then
apt_packages="systemd systemd-resolved iproute2 dbus dnsutils procps"
fi
log "Building ${base_image} runtime image"
cp "$cached_deb" "$CACHE_DIR/deb-for-image"
# Place the .deb under /opt — systemd remounts /tmp during boot
# (PrivateTmp / tmpfs) which would wipe a .deb COPY'd to /tmp.
build_image "$image" "$(cat <<DOCKERFILE
FROM ${base_image}
ENV DEBIAN_FRONTEND=noninteractive
RUN apt-get update && apt-get install -y --no-install-recommends \\
${apt_packages} && \\
apt-get clean && rm -rf /var/lib/apt/lists/* && \\
systemctl enable systemd-resolved && \\
mkdir -p /opt/fips-deb
COPY testing/deb-install/.cache/deb-for-image /opt/fips-deb/${deb_basename}
CMD ["/lib/systemd/systemd"]
DOCKERFILE
)" || { fail "runtime build failed"; rm -f "$CACHE_DIR/deb-for-image"; return; }
rm -f "$CACHE_DIR/deb-for-image"
start_systemd_container_with_tun "$name" "$image"
wait_for_systemd "$name"
# Install the .deb. apt handles dependencies (libc6, systemd,
# libdbus-1-3) and runs the maintainer scripts (postinst →
# systemctl enable fips.service; fips-dns.service starts and
# runs fips-dns-setup).
log "Installing .deb (apt install /opt/fips-deb/${deb_basename})"
local install_output
install_output=$(docker exec "$name" bash -c "
apt-get update >/dev/null 2>&1
cd /opt/fips-deb && apt-get install -y --no-install-recommends ./${deb_basename} 2>&1
") || true
if echo "$install_output" | grep -qE "^E:|errors? were encountered"; then
fail "apt install reported errors"
echo "$install_output" | tail -20
cleanup_container "$name"
return
else
pass "apt install completed"
fi
# Verify shipped files landed where expected.
if docker exec "$name" test -x /usr/bin/fips; then
pass "/usr/bin/fips installed"
else
fail "/usr/bin/fips missing"
fi
if docker exec "$name" test -x /usr/bin/fips-gateway; then
pass "/usr/bin/fips-gateway installed"
else
fail "/usr/bin/fips-gateway missing"
fi
if docker exec "$name" test -f /etc/fips/fips.yaml; then
pass "/etc/fips/fips.yaml conffile installed"
else
fail "/etc/fips/fips.yaml conffile missing"
fi
# Verify fips.service is enabled (postinst enables but does not
# start on fresh install — standard Debian convention).
if docker exec "$name" systemctl is-enabled --quiet fips.service; then
pass "fips.service enabled by postinst"
else
fail "fips.service not enabled after install"
fi
if docker exec "$name" systemctl is-enabled --quiet fips-dns.service; then
pass "fips-dns.service enabled by postinst"
else
fail "fips-dns.service not enabled after install"
fi
# Start the services as a simulated boot. (On a real system,
# they'd come up on next reboot.)
docker exec "$name" systemctl start fips.service 2>&1 || true
docker exec "$name" systemctl start fips-dns.service 2>&1 || true
if wait_for_service_active "$name" fips.service; then
pass "fips.service active after explicit start"
else
fail "fips.service did not become active in ${SERVICE_TIMEOUT}s"
echo " --- fips.service status ---"
docker exec "$name" systemctl status fips.service --no-pager 2>&1 | tail -20
echo " --- fips.service journal ---"
docker exec "$name" journalctl -u fips.service --no-pager 2>&1 | tail -20
cleanup_container "$name"
return
fi
# Wait for the DNS responder to bind on the daemon's [::1]:5354.
local ready=0
for _i in $(seq 1 "$DAEMON_TIMEOUT"); do
if docker exec "$name" ss -uln 2>/dev/null | grep -q ':5354'; then
ready=1
break
fi
sleep 1
done
if [ "$ready" = "1" ]; then
pass "fips DNS listener up on port 5354"
else
fail "fips DNS listener did not appear within ${DAEMON_TIMEOUT}s"
cleanup_container "$name"
return
fi
# Wait for fips-dns.service. This should have run fips-dns-setup
# which configures the resolver backend and writes
# /run/fips/dns-backend.
sleep 2
local backend
backend=$(docker exec "$name" cat /run/fips/dns-backend 2>/dev/null || echo "(missing)")
local ver
ver=$(container_systemd_version "$name")
local expected_backend
if [ -n "$ver" ] && [ "$ver" -ge 258 ]; then
expected_backend="dns-delegate"
else
expected_backend="global-drop-in"
fi
if [ "$backend" = "$expected_backend" ]; then
pass "fips-dns.service picked $expected_backend backend (systemd $ver)"
else
fail "expected $expected_backend (systemd $ver), got: $backend"
echo " --- fips-dns.service journal ---"
docker exec "$name" journalctl -u fips-dns.service --no-pager 2>&1 | tail -20
fi
# Get the daemon's npub via fipsctl. Works for both ephemeral
# and persistent identity (no need to override the conffile).
local npub
npub=$(docker exec "$name" fipsctl show status 2>/dev/null \
| grep -oE 'npub1[a-z0-9]+' | head -1)
if [ -z "$npub" ]; then
fail "could not read npub from fipsctl show status"
cleanup_container "$name"
return
fi
echo " daemon npub: $npub"
# The actual end-to-end test: a stock dpkg-installed deployment
# must successfully resolve a .fips query through the system
# resolver. This covers the full path: maintainer scripts ran,
# service started, DNS responder bound, resolver backend
# configured, query routed and answered.
sleep 1
local stub_output
stub_output=$(docker exec "$name" dig +tries=1 +time=3 @127.0.0.53 AAAA "${npub}.fips" 2>&1)
if echo "$stub_output" | grep -qE '^[a-zA-Z0-9].*\sAAAA\s+[0-9a-f:]+'; then
pass "end-to-end dig @127.0.0.53 returns AAAA on stock .deb install"
else
fail "end-to-end dig @127.0.0.53 did not return AAAA"
echo " --- dig output ---"
echo "$stub_output" | tail -15
echo " --- resolved status ---"
docker exec "$name" resolvectl status 2>&1 | tail -25 || true
echo " --- fips journal ---"
docker exec "$name" journalctl -u fips.service --no-pager 2>&1 | tail -10
fi
# Verify fips-gateway can run against the installed daemon. Tests
# the gateway/daemon default-pairing on a real .deb install (no
# custom config). Requires enabling the unit (it's not in the
# default preset) and ipv6 forwarding (gateway checks before
# the DNS upstream check).
docker exec "$name" sysctl -w net.ipv6.conf.all.forwarding=1 >/dev/null 2>&1 || true
docker exec "$name" bash -c '
systemctl unmask fips-gateway.service 2>/dev/null
# Patch in a minimal gateway config since the shipped fips.yaml
# has gateway disabled by default.
cp /etc/fips/fips.yaml /etc/fips/fips.yaml.orig
cat >> /etc/fips/fips.yaml <<EOF
gateway:
enabled: true
pool: "fd01::/112"
lan_interface: "eth0"
EOF
systemctl restart fips.service
' >/dev/null 2>&1
sleep 3
if wait_for_service_active "$name" fips.service 5; then
:
else
fail "fips.service did not stay up after gateway-enable restart"
fi
docker exec "$name" systemctl start fips-gateway.service >/dev/null 2>&1 || true
sleep 3
if docker exec "$name" journalctl -u fips-gateway.service --no-pager 2>/dev/null \
| grep -q "DNS upstream is reachable"; then
pass "fips-gateway reaches DNS upstream at [::1]:5354 via .deb install"
else
fail "fips-gateway DNS upstream check failed against installed daemon"
echo " --- fips-gateway journal ---"
docker exec "$name" journalctl -u fips-gateway.service --no-pager 2>&1 | tail -15
fi
cleanup_container "$name"
}
# Per-distro wrappers
test_debian12() { _run_deb_install_scenario debian12 debian:12; }
test_debian13() { _run_deb_install_scenario debian13 debian:trixie; }
test_ubuntu22() { _run_deb_install_scenario ubuntu22 ubuntu:22.04; }
test_ubuntu24() { _run_deb_install_scenario ubuntu24 ubuntu:24.04; }
test_ubuntu26() { _run_deb_install_scenario ubuntu26 ubuntu:26.04; }
# ─────────────────────────────────────────────────────────────────────
# Main
# ─────────────────────────────────────────────────────────────────────
ALL_SCENARIOS="debian12 debian13 ubuntu22 ubuntu24 ubuntu26"
if [ $# -eq 0 ]; then
scenarios="$ALL_SCENARIOS"
else
scenarios="$*"
fi
for scenario in $scenarios; do
case "$scenario" in
debian12) test_debian12 ;;
debian13) test_debian13 ;;
ubuntu22) test_ubuntu22 ;;
ubuntu24) test_ubuntu24 ;;
ubuntu26) test_ubuntu26 ;;
*)
echo "Unknown scenario: $scenario"
echo "Available: $ALL_SCENARIOS"
exit 1
;;
esac
echo
done
echo "═══════════════════════════════════════"
echo "Results: $PASS passed, $FAIL failed, $SKIP skipped"
echo "═══════════════════════════════════════"
[ "$FAIL" -eq 0 ]
+1
View File
@@ -0,0 +1 @@
.cache/
+539 -38
View File
@@ -1,15 +1,25 @@
#!/bin/bash
# Test fips-dns-setup across different Linux resolver backends.
# Test fips-dns-setup across different Linux resolver backends, plus
# an end-to-end scenario that verifies a real fips answers .fips
# queries through the configured backend.
#
# Each scenario runs a systemd-based Docker container, creates a dummy
# fips0 interface, runs the setup script, verifies the detected backend
# and generated config, runs teardown, and verifies cleanup.
# fips0 interface (or a real one for the e2e scenario), runs the setup
# script, verifies the detected backend and generated config, runs
# teardown, and verifies cleanup.
#
# The end-to-end scenario additionally builds fips in a Debian 12
# builder image (cached between runs) so the binary is glibc-compatible
# across all target distros. It then starts the daemon, configures DNS
# via the script, and confirms `dig @127.0.0.53 AAAA <npub>.fips`
# returns a non-empty AAAA answer.
#
# Usage: ./test.sh [scenario ...]
# No args = run all scenarios.
# Named args = run only those (e.g., ./test.sh debian12-resolved dnsmasq)
# Named args = run only those (e.g., ./test.sh debian12-resolved e2e-debian12)
#
# Requirements: Docker with privileged container support.
# Requirements: Docker with privileged container support. The e2e
# scenario also needs /dev/net/tun on the host (standard).
set -uo pipefail
@@ -17,10 +27,16 @@ SCRIPT_DIR="$(cd "$(dirname "$0")" && pwd)"
REPO_ROOT="$(cd "$SCRIPT_DIR/../.." && pwd)"
SETUP_SCRIPT="$REPO_ROOT/packaging/common/fips-dns-setup"
TEARDOWN_SCRIPT="$REPO_ROOT/packaging/common/fips-dns-teardown"
CACHE_DIR="$SCRIPT_DIR/.cache"
FIPS_BIN_CACHE="$CACHE_DIR/fips"
FIPS_GATEWAY_BIN_CACHE="$CACHE_DIR/fips-gateway"
# Timeout for systemd boot inside container
BOOT_TIMEOUT=30
# Timeout for fips to start serving DNS in the e2e scenario
DAEMON_TIMEOUT=15
PASS=0
FAIL=0
SKIP=0
@@ -58,6 +74,19 @@ start_systemd_container() {
"$image" >/dev/null 2>&1
}
# Same, but with TUN device for the e2e scenario.
start_systemd_container_with_tun() {
local name="$1" image="$2"
cleanup_container "$name"
docker run -d --name "$name" \
--privileged \
--cgroupns=host \
--device /dev/net/tun \
-v /sys/fs/cgroup:/sys/fs/cgroup:rw \
--tmpfs /run --tmpfs /run/lock \
"$image" >/dev/null 2>&1
}
# Wait for systemd to reach a bootable state inside the container.
wait_for_systemd() {
local name="$1"
@@ -104,14 +133,158 @@ file_exists() {
docker exec "$name" test -f "$path" 2>/dev/null
}
file_contains() {
local name="$1" path="$2" needle="$3"
docker exec "$name" grep -qF "$needle" "$path" 2>/dev/null
}
# Get the major systemd version inside a container.
container_systemd_version() {
local name="$1"
docker exec "$name" systemctl --version 2>/dev/null | head -1 \
| grep -oE '[0-9]+' | head -1
}
# Verify the expected systemd-resolved-flavoured backend was picked
# and that its config file targets the new [::1]:5354 daemon bind.
# On systemd >= 258 the dns-delegate backend wins; otherwise
# global-drop-in. Either way the daemon target must be ::1 — that's
# the regression we're locking in.
verify_resolved_backend() {
local name="$1"
local ver
ver=$(container_systemd_version "$name")
local backend
backend=$(get_backend "$name")
local expected_backend expected_path
if [ -n "$ver" ] && [ "$ver" -ge 258 ]; then
expected_backend="dns-delegate"
expected_path="/etc/systemd/dns-delegate.d/fips.dns-delegate"
else
expected_backend="global-drop-in"
expected_path="/etc/systemd/resolved.conf.d/fips.conf"
fi
if [ "$backend" = "$expected_backend" ]; then
pass "detected backend: $expected_backend (systemd $ver)"
else
fail "expected $expected_backend (systemd $ver), got: $backend"
fi
if file_exists "$name" "$expected_path"; then
pass "config file written at $expected_path"
else
fail "config file missing at $expected_path"
return
fi
# All systemd-flavoured backends must target [::1]:5354 to match
# the daemon's default bind. If they don't, queries silently fail
# — Linux IPv6 sockets bound to ::1 do not accept v4 traffic.
if file_contains "$name" "$expected_path" "[::1]:5354"; then
pass "config DNS target is [::1]:5354 (matches daemon default)"
else
fail "config DNS target wrong — must be [::1]:5354"
echo " contents: $(docker exec "$name" cat "$expected_path")"
fi
# Domain forwarding line: dns-delegate uses 'Domains=fips',
# global-drop-in uses 'Domains=~fips' (wildcard prefix).
local expected_domain_line
if [ "$expected_backend" = "dns-delegate" ]; then
expected_domain_line="Domains=fips"
else
expected_domain_line="Domains=~fips"
fi
if file_contains "$name" "$expected_path" "$expected_domain_line"; then
pass "config Domains line correct ($expected_domain_line)"
else
fail "config Domains line incorrect — expected $expected_domain_line"
fi
run_teardown "$name" >/dev/null 2>&1
if ! file_exists "$name" "$expected_path"; then
pass "teardown removed config file"
else
fail "config file still exists after teardown"
fi
if ! file_exists "$name" /run/fips/dns-backend; then
pass "teardown cleaned state file"
else
fail "state file still exists after teardown"
fi
}
# ─────────────────────────────────────────────────────────────────────
# Scenarios
# Build the fips binary once in a Debian 12 builder image so it's
# glibc-compatible with every target distro (Debian 12/13, Ubuntu 22/24).
# Cached at testing/dns-resolver/.cache/fips between runs; rebuild if
# any source file is newer than the cached binary.
# ─────────────────────────────────────────────────────────────────────
build_fips_for_e2e() {
mkdir -p "$CACHE_DIR"
if [ -f "$FIPS_BIN_CACHE" ] && [ -f "$FIPS_GATEWAY_BIN_CACHE" ]; then
local newest_src
newest_src=$(find "$REPO_ROOT/src" "$REPO_ROOT/Cargo.toml" "$REPO_ROOT/Cargo.lock" \
-type f -printf '%T@\n' 2>/dev/null | sort -nr | head -1)
local cached_age
cached_age=$(stat -c '%Y' "$FIPS_BIN_CACHE" 2>/dev/null || echo 0)
local cached_gateway_age
cached_gateway_age=$(stat -c '%Y' "$FIPS_GATEWAY_BIN_CACHE" 2>/dev/null || echo 0)
local oldest_cached=$((cached_age < cached_gateway_age ? cached_age : cached_gateway_age))
if awk "BEGIN { exit !($oldest_cached >= $newest_src) }"; then
log "Using cached fips + fips-gateway binaries at $CACHE_DIR"
return 0
fi
log "Cached binaries are stale, rebuilding"
else
log "No cached binaries, building"
fi
local builder_tag="fips-dns-test:builder"
log "Building Debian 12 builder image (this may take a few minutes on first run)"
docker build -t "$builder_tag" -f - "$REPO_ROOT" <<'DOCKERFILE' >/dev/null
FROM debian:12
ENV DEBIAN_FRONTEND=noninteractive
RUN apt-get update && apt-get install -y --no-install-recommends \
build-essential pkg-config libdbus-1-dev curl ca-certificates \
libclang-dev clang && \
apt-get clean && rm -rf /var/lib/apt/lists/*
RUN curl --proto '=https' --tlsv1.2 -sSf https://sh.rustup.rs | \
sh -s -- -y --default-toolchain stable --profile minimal
ENV PATH="/root/.cargo/bin:${PATH}"
WORKDIR /src
COPY Cargo.toml Cargo.lock build.rs ./
COPY src ./src
RUN cargo build --release --bin fips --bin fips-gateway
DOCKERFILE
if [ ! "$(docker images -q "$builder_tag" 2>/dev/null)" ]; then
echo " ERROR: builder image build failed"
return 1
fi
log "Extracting fips + fips-gateway binaries from builder image"
local cid
cid=$(docker create "$builder_tag")
docker cp "$cid:/src/target/release/fips" "$FIPS_BIN_CACHE" >/dev/null 2>&1
docker cp "$cid:/src/target/release/fips-gateway" "$FIPS_GATEWAY_BIN_CACHE" >/dev/null 2>&1
docker rm "$cid" >/dev/null
chmod +x "$FIPS_BIN_CACHE" "$FIPS_GATEWAY_BIN_CACHE"
log "Cached fips ($(stat -c %s "$FIPS_BIN_CACHE") bytes) + fips-gateway ($(stat -c %s "$FIPS_GATEWAY_BIN_CACHE") bytes)"
}
# ─────────────────────────────────────────────────────────────────────
# Scenarios — script-behavior tests across distros (no daemon)
# ─────────────────────────────────────────────────────────────────────
test_debian12_resolved() {
local name="fips-dns-test-deb12-resolved"
local image="fips-dns-test:debian12-resolved"
log "Debian 12 + systemd-resolved"
log "Debian 12 + systemd-resolved (expects global-drop-in)"
build_image "$image" "$(cat <<'DOCKERFILE'
FROM debian:12
@@ -132,29 +305,14 @@ DOCKERFILE
output=$(run_setup "$name" 2>&1)
echo " output: $output"
local backend
backend=$(get_backend "$name")
if [ "$backend" = "resolvectl" ]; then
pass "detected backend: resolvectl"
else
fail "expected resolvectl, got: $backend"
fi
# Teardown
run_teardown "$name" >/dev/null 2>&1
if ! file_exists "$name" /run/fips/dns-backend; then
pass "teardown cleaned state file"
else
fail "state file still exists after teardown"
fi
verify_resolved_backend "$name"
cleanup_container "$name"
}
test_debian13_resolved() {
local name="fips-dns-test-deb13-resolved"
local image="fips-dns-test:debian13-resolved"
log "Debian 13 (trixie) + systemd-resolved"
log "Debian 13 (trixie) + systemd-resolved (expects global-drop-in)"
build_image "$image" "$(cat <<'DOCKERFILE'
FROM debian:trixie
@@ -175,21 +333,93 @@ DOCKERFILE
output=$(run_setup "$name" 2>&1)
echo " output: $output"
local backend
backend=$(get_backend "$name")
if [ "$backend" = "resolvectl" ]; then
pass "detected backend: resolvectl"
else
fail "expected resolvectl, got: $backend"
fi
verify_resolved_backend "$name"
cleanup_container "$name"
}
run_teardown "$name" >/dev/null 2>&1
if ! file_exists "$name" /run/fips/dns-backend; then
pass "teardown cleaned state file"
else
fail "state file still exists after teardown"
fi
test_ubuntu22_resolved() {
local name="fips-dns-test-u22-resolved"
local image="fips-dns-test:ubuntu22-resolved"
log "Ubuntu 22.04 + systemd-resolved (expects global-drop-in)"
# On Ubuntu 22.04 systemd-resolved is bundled with systemd (not a
# separate package). Just enable the service.
build_image "$image" "$(cat <<'DOCKERFILE'
FROM ubuntu:22.04
ENV DEBIAN_FRONTEND=noninteractive
RUN apt-get update && apt-get install -y --no-install-recommends \
systemd iproute2 dbus && \
apt-get clean && rm -rf /var/lib/apt/lists/* && \
systemctl enable systemd-resolved
CMD ["/lib/systemd/systemd"]
DOCKERFILE
)" || { fail "build failed"; return; }
start_systemd_container "$name" "$image"
wait_for_systemd "$name"
create_fips0 "$name"
local output
output=$(run_setup "$name" 2>&1)
echo " output: $output"
verify_resolved_backend "$name"
cleanup_container "$name"
}
test_ubuntu24_resolved() {
local name="fips-dns-test-u24-resolved"
local image="fips-dns-test:ubuntu24-resolved"
log "Ubuntu 24.04 + systemd-resolved (expects global-drop-in)"
build_image "$image" "$(cat <<'DOCKERFILE'
FROM ubuntu:24.04
ENV DEBIAN_FRONTEND=noninteractive
RUN apt-get update && apt-get install -y --no-install-recommends \
systemd systemd-resolved iproute2 dbus && \
apt-get clean && rm -rf /var/lib/apt/lists/* && \
systemctl enable systemd-resolved
CMD ["/lib/systemd/systemd"]
DOCKERFILE
)" || { fail "build failed"; return; }
start_systemd_container "$name" "$image"
wait_for_systemd "$name"
create_fips0 "$name"
local output
output=$(run_setup "$name" 2>&1)
echo " output: $output"
verify_resolved_backend "$name"
cleanup_container "$name"
}
test_ubuntu26_resolved() {
local name="fips-dns-test-u26-resolved"
local image="fips-dns-test:ubuntu26-resolved"
log "Ubuntu 26.04 + systemd-resolved (expects global-drop-in)"
build_image "$image" "$(cat <<'DOCKERFILE'
FROM ubuntu:26.04
ENV DEBIAN_FRONTEND=noninteractive
RUN apt-get update && apt-get install -y --no-install-recommends \
systemd systemd-resolved iproute2 dbus && \
apt-get clean && rm -rf /var/lib/apt/lists/* && \
systemctl enable systemd-resolved
CMD ["/lib/systemd/systemd"]
DOCKERFILE
)" || { fail "build failed"; return; }
start_systemd_container "$name" "$image"
wait_for_systemd "$name"
create_fips0 "$name"
local output
output=$(run_setup "$name" 2>&1)
echo " output: $output"
verify_resolved_backend "$name"
cleanup_container "$name"
}
@@ -234,6 +464,15 @@ DOCKERFILE
fail "dnsmasq config not found"
fi
# Verify config targets ::1#5354 (the daemon's default IPv6
# loopback bind). Drift from this constant would silently break
# resolution on hosts using this backend.
if file_contains "$name" /etc/dnsmasq.d/fips.conf "server=/fips/::1#5354"; then
pass "dnsmasq config targets ::1#5354 (matches daemon default)"
else
fail "dnsmasq config target wrong — must be server=/fips/::1#5354"
fi
# Teardown
run_teardown "$name" >/dev/null 2>&1
if ! file_exists "$name" /etc/dnsmasq.d/fips.conf; then
@@ -291,6 +530,12 @@ DOCKERFILE
fail "NM dnsmasq config not found"
fi
if file_contains "$name" /etc/NetworkManager/dnsmasq.d/fips.conf "server=/fips/::1#5354"; then
pass "NM dnsmasq config targets ::1#5354 (matches daemon default)"
else
fail "NM dnsmasq config target wrong — must be server=/fips/::1#5354"
fi
# Teardown
run_teardown "$name" >/dev/null 2>&1
if ! file_exists "$name" /etc/NetworkManager/dnsmasq.d/fips.conf; then
@@ -355,11 +600,259 @@ DOCKERFILE
cleanup_container "$name"
}
# ─────────────────────────────────────────────────────────────────────
# End-to-end scenarios — run a real fips + fips-gateway, configure
# DNS via the script, dig through systemd-resolved.
#
# Parameterized across Debian 12/13 and Ubuntu 22/24/26. The fips
# and fips-gateway binaries are built once in a Debian 12 builder
# image (lowest glibc target → forward-compatible with all newer
# distros) and copied into each per-distro runtime image.
# ─────────────────────────────────────────────────────────────────────
# Args: <distro_label> <docker_base_image> <apt_packages>
# distro_label: short tag for container/image names (e.g. "debian12")
# docker_base_image: e.g. "debian:12", "ubuntu:26.04"
# apt_packages: space-separated apt-get install list. Ubuntu 22.04
# bundles systemd-resolved into systemd, so the package list there
# is "systemd iproute2 dbus dnsutils libdbus-1-3 procps" (no
# separate systemd-resolved). Other distros want
# "systemd systemd-resolved iproute2 dbus dnsutils libdbus-1-3 procps".
_run_e2e_scenario() {
local distro_label="$1"
local base_image="$2"
local apt_packages="$3"
local name="fips-dns-test-e2e-${distro_label}"
local image="fips-dns-test:e2e-${distro_label}"
log "End-to-end: ${base_image} + systemd-resolved + real fips + fips-gateway + dig"
build_fips_for_e2e || { fail "fips build failed"; return; }
if [ ! -x "$FIPS_BIN_CACHE" ] || [ ! -x "$FIPS_GATEWAY_BIN_CACHE" ]; then
fail "binaries not available at $CACHE_DIR"
return
fi
log "Building e2e runtime image (${base_image})"
cp "$FIPS_BIN_CACHE" "$CACHE_DIR/fips-bin-for-image"
cp "$FIPS_GATEWAY_BIN_CACHE" "$CACHE_DIR/fips-gateway-bin-for-image"
build_image "$image" "$(cat <<DOCKERFILE
FROM ${base_image}
ENV DEBIAN_FRONTEND=noninteractive
RUN apt-get update && apt-get install -y --no-install-recommends \\
${apt_packages} && \\
apt-get clean && rm -rf /var/lib/apt/lists/* && \\
systemctl enable systemd-resolved && \\
mkdir -p /etc/fips
COPY testing/dns-resolver/.cache/fips-bin-for-image /usr/bin/fips
COPY testing/dns-resolver/.cache/fips-gateway-bin-for-image /usr/bin/fips-gateway
RUN chmod +x /usr/bin/fips /usr/bin/fips-gateway
CMD ["/lib/systemd/systemd"]
DOCKERFILE
)" || { fail "runtime build failed"; rm -f "$CACHE_DIR/fips-bin-for-image" "$CACHE_DIR/fips-gateway-bin-for-image"; return; }
rm -f "$CACHE_DIR/fips-bin-for-image" "$CACHE_DIR/fips-gateway-bin-for-image"
start_systemd_container_with_tun "$name" "$image"
wait_for_systemd "$name"
# Write a minimal fips.yaml that exercises the new defaults.
# tun.enabled: true so the daemon creates fips0 itself; identity
# persistent so /etc/fips/fips.pub gives us a stable npub to query.
docker exec "$name" bash -c 'cat > /etc/fips/fips.yaml <<EOF
node:
identity:
persistent: true
log_level: debug
tun:
enabled: true
name: fips0
dns:
enabled: true
port: 5354
EOF'
# Start fips in the background.
log "Starting fips in container"
docker exec -d "$name" bash -c '/usr/bin/fips --config /etc/fips/fips.yaml >/var/log/fips.log 2>&1'
# Wait for the DNS responder to bind.
local ready=0
for _i in $(seq 1 "$DAEMON_TIMEOUT"); do
if docker exec "$name" ss -uln 2>/dev/null | grep -q ':5354'; then
ready=1
break
fi
sleep 1
done
if [ "$ready" = "1" ]; then
pass "fips DNS listener up on port 5354"
else
fail "fips DNS listener did not appear within ${DAEMON_TIMEOUT}s"
echo " --- fips log ---"
docker exec "$name" tail -30 /var/log/fips.log 2>&1 || true
echo " --- ss -uln ---"
docker exec "$name" ss -ulnp 2>&1 || true
cleanup_container "$name"
return
fi
# Confirm the daemon picked up the new ::1 default bind. Strip
# ANSI color codes from the log line before matching since the
# tracing-subscriber default formatter wraps fields in escape codes.
local bind_line
bind_line=$(docker exec "$name" grep -m1 "DNS responder started" /var/log/fips.log 2>/dev/null \
| sed -r 's/\x1b\[[0-9;]*m//g' || echo "")
echo " daemon log: $bind_line"
if echo "$bind_line" | grep -qE "bind=\[?::1\]?:5354"; then
pass "daemon bound on [::1]:5354 (new default)"
else
fail "daemon bind line missing or wrong: $bind_line"
fi
# Run setup.
local output
output=$(run_setup "$name" 2>&1)
echo " setup output: $output"
# Pick expected backend based on systemd version: dns-delegate
# on >= 258, global-drop-in otherwise. Either way the backend
# must target [::1]:5354 for the daemon to receive queries.
local ver
ver=$(container_systemd_version "$name")
local expected_backend
if [ -n "$ver" ] && [ "$ver" -ge 258 ]; then
expected_backend="dns-delegate"
else
expected_backend="global-drop-in"
fi
local backend
backend=$(get_backend "$name")
if [ "$backend" = "$expected_backend" ]; then
pass "setup picked $expected_backend backend (systemd $ver)"
else
fail "expected $expected_backend (systemd $ver), got: $backend"
fi
# Wait briefly for systemd-resolved to apply the new config.
sleep 2
# Pull the daemon's npub from the persistent identity file.
local npub
npub=$(docker exec "$name" cat /etc/fips/fips.pub 2>/dev/null | tr -d '[:space:]')
if [ -z "$npub" ]; then
fail "no /etc/fips/fips.pub after daemon start"
echo " --- /etc/fips ---"
docker exec "$name" ls -la /etc/fips/ 2>&1 || true
cleanup_container "$name"
return
fi
echo " daemon npub: $npub"
# Direct dig to the daemon's loopback bind — must succeed.
local direct_output
direct_output=$(docker exec "$name" dig +tries=1 +time=3 @::1 -p 5354 AAAA "${npub}.fips" 2>&1)
if echo "$direct_output" | grep -qE '^[a-zA-Z0-9].*\sAAAA\s+[0-9a-f:]+'; then
pass "direct dig @::1#5354 returns AAAA"
else
fail "direct dig @::1#5354 did not return AAAA"
echo " --- dig output ---"
echo "$direct_output" | tail -15
fi
# End-to-end via systemd-resolved stub.
local stub_output
stub_output=$(docker exec "$name" dig +tries=1 +time=3 @127.0.0.53 AAAA "${npub}.fips" 2>&1)
if echo "$stub_output" | grep -qE '^[a-zA-Z0-9].*\sAAAA\s+[0-9a-f:]+'; then
pass "end-to-end dig @127.0.0.53 returns AAAA (the bug fix)"
else
fail "end-to-end dig @127.0.0.53 did not return AAAA"
echo " --- dig output ---"
echo "$stub_output" | tail -15
echo " --- resolved status ---"
docker exec "$name" resolvectl status 2>&1 | tail -20 || true
echo " --- daemon log tail ---"
docker exec "$name" tail -30 /var/log/fips.log 2>&1 || true
fi
# Verify fips-gateway's DNS upstream reachability check passes
# against the daemon's new ::1 default. This locks the regression
# class where the gateway default (was 127.0.0.1:5354) and the
# daemon default (now [::1]:5354) drift apart on Linux IPv6
# sockets that don't accept v4-mapped traffic.
docker exec "$name" bash -c 'cat > /tmp/gateway-test.yaml <<EOF
node:
identity:
persistent: true
gateway:
enabled: true
pool: "fd01::/112"
lan_interface: "eth0"
EOF'
# fips-gateway checks IPv6 forwarding before the DNS upstream
# reachability check; enable forwarding so we get to the check we
# actually want to test.
docker exec "$name" sysctl -w net.ipv6.conf.all.forwarding=1 >/dev/null 2>&1 || true
docker exec -d "$name" bash -c '/usr/bin/fips-gateway --config /tmp/gateway-test.yaml >/var/log/fips-gateway.log 2>&1 || true'
# Wait briefly for the upstream-reachability log line to appear
# one way or the other.
local gw_ok=0
for _i in $(seq 1 5); do
if docker exec "$name" grep -q "DNS upstream is reachable" /var/log/fips-gateway.log 2>/dev/null; then
gw_ok=1
break
fi
if docker exec "$name" grep -qE "DNS upstream did not respond|Failed to send DNS probe|DNS upstream recv failed" /var/log/fips-gateway.log 2>/dev/null; then
break
fi
sleep 1
done
if [ "$gw_ok" = "1" ]; then
pass "fips-gateway reaches DNS upstream at [::1]:5354 (gateway/daemon default parity)"
else
fail "fips-gateway DNS upstream check failed — defaults drifted?"
echo " --- fips-gateway log ---"
docker exec "$name" tail -20 /var/log/fips-gateway.log 2>&1 || true
fi
# Stop the gateway (it will likely have failed past the upstream
# check on something unrelated in this minimal container — we only
# care that the upstream reachability step succeeded).
docker exec "$name" pkill -f fips-gateway 2>/dev/null || true
# Teardown via the script: backend config file must be removed
# (path varies by backend selected above).
local teardown_path
if [ "$expected_backend" = "dns-delegate" ]; then
teardown_path="/etc/systemd/dns-delegate.d/fips.dns-delegate"
else
teardown_path="/etc/systemd/resolved.conf.d/fips.conf"
fi
run_teardown "$name" >/dev/null 2>&1
if ! file_exists "$name" "$teardown_path"; then
pass "teardown removed $expected_backend config at $teardown_path"
else
fail "$expected_backend config still present after teardown at $teardown_path"
fi
cleanup_container "$name"
}
# Per-distro wrappers
_pkgs_with_resolved="systemd systemd-resolved iproute2 dbus dnsutils libdbus-1-3 procps"
_pkgs_ubuntu22="systemd iproute2 dbus dnsutils libdbus-1-3 procps"
test_e2e_debian12() { _run_e2e_scenario debian12 debian:12 "$_pkgs_with_resolved"; }
test_e2e_debian13() { _run_e2e_scenario debian13 debian:trixie "$_pkgs_with_resolved"; }
test_e2e_ubuntu22() { _run_e2e_scenario ubuntu22 ubuntu:22.04 "$_pkgs_ubuntu22"; }
test_e2e_ubuntu24() { _run_e2e_scenario ubuntu24 ubuntu:24.04 "$_pkgs_with_resolved"; }
test_e2e_ubuntu26() { _run_e2e_scenario ubuntu26 ubuntu:26.04 "$_pkgs_with_resolved"; }
# ─────────────────────────────────────────────────────────────────────
# Main
# ─────────────────────────────────────────────────────────────────────
ALL_SCENARIOS="debian12-resolved debian13-resolved dnsmasq nm-dnsmasq no-resolver"
ALL_SCENARIOS="debian12-resolved debian13-resolved ubuntu22-resolved ubuntu24-resolved ubuntu26-resolved dnsmasq nm-dnsmasq no-resolver e2e-debian12 e2e-debian13 e2e-ubuntu22 e2e-ubuntu24 e2e-ubuntu26"
if [ $# -eq 0 ]; then
scenarios="$ALL_SCENARIOS"
@@ -371,9 +864,17 @@ for scenario in $scenarios; do
case "$scenario" in
debian12-resolved) test_debian12_resolved ;;
debian13-resolved) test_debian13_resolved ;;
ubuntu22-resolved) test_ubuntu22_resolved ;;
ubuntu24-resolved) test_ubuntu24_resolved ;;
ubuntu26-resolved) test_ubuntu26_resolved ;;
dnsmasq) test_dnsmasq ;;
nm-dnsmasq) test_nm_dnsmasq ;;
no-resolver) test_no_resolver ;;
e2e-debian12) test_e2e_debian12 ;;
e2e-debian13) test_e2e_debian13 ;;
e2e-ubuntu22) test_e2e_ubuntu22 ;;
e2e-ubuntu24) test_e2e_ubuntu24 ;;
e2e-ubuntu26) test_e2e_ubuntu26 ;;
*)
echo "Unknown scenario: $scenario"
echo "Available: $ALL_SCENARIOS"