A CI worker may preempt an in-flight ci-local.sh run (SIGTERM, then SIGKILL
after a grace period) to restart on a newer commit. For that kill to be safe,
the script must clean up after itself and never let a dying run collide with
its restart. It previously had no signal handling, shared the default compose
project name across runs, and tore down each suite only at the suite end.
- Derive a per-run id (honoring FIPS_CI_RUN_ID, else short-sha+random) and
namespace every docker resource to it: a fipsci_<run>_<suite> compose project
per suite and per parallel chaos child, and per-run image tags
(fips-test:<run>, fips-test-app:<run>) retagged to :latest only after both
builds succeed so :latest never points at a half-built image.
- Install a bounded, idempotent teardown trap on SIGTERM/SIGINT (+ EXIT): reap
parallel chaos children, then force-remove this run's docker resources via
the new ci-cleanup.sh, wrapped in timeout so a stuck down cannot wedge it.
- Exit 143 (SIGTERM) / 130 (SIGINT), distinct from 0 (pass) / 1 (failed), so a
preempting worker tells a cancelled run from a real failure.
- Add ci-cleanup.sh (also ci-local.sh --reap): force-removes leftover CI
resources by the com.corganlabs.fips-ci=1 label and the fipsci_ project
prefix, robust to however a prior run died.
- Label every per-suite docker resource so the label sweep reaps it after a
SIGKILL regardless of network name: direct docker run/network resources, the
sidecar compose services, and every per-suite compose network (acl-allowlist,
boringtun, firewall, nat, static, both tor suites, and the chaos generator
template). Parametrize the static/sidecar compose image refs so the per-run
tags are honored.
- Give each parallel chaos child a unique /24 from 10.30.x (a new --subnet
override on the sim CLI, assigned per-child in ci-local.sh) so parallel
children never collide on a shared docker subnet, and a chaos net can never
span a fixed-subnet suite (sidecar/static in 172.20.x). 10.30.x sits outside
docker's default-address-pool range, so an auto-assigned net cannot land on
it either; node IPs derive from the subnet, so no scenario config changes.
The drop-counter sanity check piped `nft list table inet fips`
through `awk '/counter packets/ {print $3; exit}'`. Awk's `exit`
on first match closes the pipe, the upstream `nft list` SIGPIPEs
on its next write, `set -o pipefail` makes the pipeline return
141, and the surrounding command-substitution aborts the script
before it can assign DROP_PKTS or print the section header.
Replaces the early-exit pattern with `/counter packets/ && !seen
{ print $3; seen=1 }` — same first-match output, but awk reads
the full input so nft never SIGPIPEs.
The original form had been latent for as long as the test has
existed; recent CI runs at master tip 53ad528 finally tripped
it (output shows the script dying immediately after the case-(d)
PASS, before "=== Drop counter incremented..." prints).
Verified locally: `bash testing/ci-local.sh --only firewall`
runs all six setup-and-functional steps green and prints
"PASS: drop counter = 5".
End-to-end exercise the v0.3.0 nftables firewall baseline so the
security claim — "services on fips0 are not exposed by default" — is
validated in CI rather than by operator opt-in. The packaging postinst
state is pinned by the deb-install matrix; this suite pins the actual
ruleset behavior.
testing/firewall/ — new directory mirroring the acl-allowlist
precedent (kept in its own directory, not extending testing/static):
docker-compose.yml (52 lines): two FIPS containers on bridge
172.32.0.0/24, peered over UDP/2121. node-b mounts
packaging/common/fips.nft RO at /etc/fips/fips.nft and a generated
drop-in services.nft at /etc/fips/fips.d/.
test.sh (247 lines): four-case asserter
(a) curl from node-a to node-b:8000 → DROP (port not allowlisted;
terminal counter drop fires)
(b) curl from node-b to node-a:8000 → 200 OK (reply traverses
node-b's ct state established,related accept)
(c) ping6 a→b → success (icmpv6 echo-request accept)
(d) nc -z a→b:22 → success (drop-in tcp dport 22 accept honored
via include "/etc/fips/fips.d/*.nft")
Plus drop-counter check after case (a) confirms the dropped
connection actually hit the chain's terminal counter drop.
generate-configs.sh (111 lines): mirrors acl-allowlist generator,
produces the drop-in services.nft + fips configs.
README.md (111 lines): how to run, expected output, design notes.
.gitignore: ignores generated-configs/.
testing/ci-local.sh: FIREWALL_SUITES=(firewall) array + run_firewall
runner; dispatch in run_integration and run_suite mirroring
run_acl_allowlist.
.github/workflows/ci.yml: matrix row {suite: firewall, type:
firewall} + 3 steps gated on matrix.type == 'firewall' between
acl-allowlist and gateway. Reuses fips-linux artifact + fips-test:
latest image.
Activation note: the unified test image does not run systemd, so
test.sh invokes the fips-firewall.service ExecStart
(/usr/sbin/nft -f /etc/fips/fips.nft) directly. The systemd-unit
enablement path is covered by the deb-install matrix; this suite
exercises what the unit configures, not how it gets started.
Johnathan Corgan