Johnathan Corgan
1e111fe044
Any host that can reach the LAN resolver could ask for one new .fips name after another, and each got a virtual-IP mapping until the pool's 65,535 addresses ran out. Every mapping also adds to the cost of each NAT rebuild, each pool tick and shutdown. The pool now refuses a new name once it holds 1000 live mappings, and admits new names from a token bucket of 50 that refills at 10 per second. Both checks sit after the return for a name that already has a mapping, so names in use keep resolving when new ones are refused. The ceiling is checked first, so a refusal there takes no token and is always reported as the ceiling. A token is taken only once an address has been taken from the free list, so an exhausted pool costs none. Each limit has its own PoolError variant, and the "Pool allocation failed" warning names the one that refused. VirtualIpPool::new keeps its signature and uses the compiled-in limits; with_limits and allocate_at, which takes the instant that drives the refill, let the unit tests set small limits and a clock. The limits come from a flood of the gateway suite's gateway to 500, 1000 and 2000 live mappings with no limits in place. The measurement record is kept separately. Figures: - New mappings per second, over 100 creations: 215 for the first 100, 63 at 500, 47 at 1000, 32 at 2000. - NAT rebuild, median/max over the 20 adds up to the count: 17.5/26.5 ms at 500, 22.2/34.3 ms at 1000, 32.2/38.3 ms at 2000. - Pool tick: 24 to 32 us at 500, 48 to 53 us at 1000, 105 to 115 us at 2000, beside a conntrack read of 180 to 300 us. - Shutdown at 2000 mappings: 2.4 s. These are a best case for router hardware: they come from a container on a development host, not a router, and that host has no /proc/net/nf_conntrack, so the tick figures include no conntrack parsing. The ceiling is half the largest count measured. At 1000 a rebuild took about 22 ms and shutdown about 1.2 s. The rate is below the unthrottled creation rate at every count measured, so the bucket rather than the rebuild sets how fast a flood can fill the pool. From empty that now takes about 95 s; the unthrottled run passed 1000 in about 16 s. Ten rebuilds a second at the ceiling cost about a fifth of the gateway's single runtime thread on that host. The 2000 figures exist only in the measurement, since the committed phase stops at the ceiling. To measure and to keep measuring, the "Added DNAT/SNAT rules" and "Removed DNAT/SNAT rules" debug lines now carry the mapping count after the change and the rebuild's duration in microseconds, and are also emitted, with the error, when a rebuild fails, so a failure at some count leaves a record of that count. The pool tick logs a debug line with the mapping count and the durations of the conntrack read and of the tick. The suite's gateway logs at info because RUST_LOG=info overrides the entrypoint's --log-level debug, so the gw-gateway service now enables debug for the NAT manager and the gateway binary only. The gateway suite's last phase is now the regression. It restarts the gateway with mappings that outlive the phase and reads the limits from pool.rs. It fills the pool to the ceiling with the readiness probe's mapping plus ceiling - 1 new names, retrying rate refusals, then asks once each for 20 more. It asserts all 20 get SERVFAIL, the live count equals the ceiling, at least 20 ceiling refusals are logged, the rate limit refused during the fill and not after it, the probe's name keeps its address, nothing is reclaimed, no NAT or proxy NDP failure is logged, and no 10 s window of the fill holds more than burst + 10 x rate creations. It also reports rebuild durations at 500 and 1000, ticks as they fall, and the shutdown time at the ceiling. On a tree with these assertions but no checks in allocate the phase fails five of them: 1020 mappings, 0 SERVFAIL, no ceiling or rate refusals, and 713 creations in one 10 s window. With the limits it passes in about 140 s, with rebuilds of 17.8/24.5 ms at 500 and 20.9/33.9 ms at 1000 and a 1.2 s shutdown at the ceiling. The NAT batch phase now also goes through the rate limit, so its driver retries refusals; it shares the restart, readiness gate and DNS driver with the new phase instead of carrying its own copies.
FIPS Testing
Integration and simulation test harnesses for FIPS, using Docker containers running the full protocol stack.
Test Harnesses
static/ -- Static Docker Network
Fixed topologies with manual scripts for building, config generation, connectivity tests (ping, iperf), and network impairment (netem). Useful for deterministic debugging and validating specific topology configurations.
| Topology | Nodes | Transport | Description |
|---|---|---|---|
| mesh | 5 | UDP | Sparse mesh, 6 links, multi-hop |
| chain | 5 | UDP | Linear chain, max 4-hop paths |
| rekey | 5 | UDP | Rekey integration test topology |
tor/ -- Tor Transport Integration
End-to-end Tor transport testing with Docker containers running real Tor daemons. Requires internet access for Tor bootstrapping.
| Scenario | Description |
|---|---|
| socks5-outbound | Outbound SOCKS5 connections through Tor to clearnet peer |
| directory-mode | Inbound via HiddenServiceDir onion service (co-located) |
nat/ -- NAT Traversal Lab
Real Docker NAT traversal tests for the Nostr/STUN bootstrap path,
using router containers with iptables-based NAT, a local Nostr relay,
and a local STUN responder.
| Scenario | Description |
|---|---|
| cone | Two NATed peers establish a UDP traversal path |
| symmetric | UDP traversal fails under symmetric NAT, TCP fallback wins |
| lan | Peers on the same LAN prefer local addresses over reflexive |
chaos/ -- Stochastic Simulation
Automated network testing with configurable node counts, topology algorithms (random geometric, Erdos-Renyi, chain, explicit), and fault injection (netem mutation, link flaps, traffic generation, node churn). 10 scenarios covering general stress and node churn, discovery over sparse topologies, spanning-tree and bloom-propagation regression, transport-specific validation (UDP, TCP, Ethernet), and ECN/congestion testing. Scenarios are defined in YAML and executed via a Python harness that manages the full lifecycle: topology generation, Docker orchestration, fault scheduling, log collection, and analysis.
interop/ -- Mixed-Version Interop Harness
On-demand harness that runs an N-node full mesh from a node-spec where
each node can run a different build of the FIPS daemon, then attributes
every FMP/FSP/rekey/connectivity failure to a specific version pair
(same-version vs MIXED). Used to catch interop regressions between
builds, not as a per-commit CI gate; not part of ci-local.sh.
mesh-lab/ -- Mesh Reliability Lab
On-demand harness that runs a chosen integration suite N times under a
configurable host-pressure profile (idle / light / github-runner-
equivalent / heavy via stress-ng), per-container netem impairment,
and optional trace-level RUST_LOG, capturing per-rep diagnostics and a
mechanism-match summary across the run. Used for statistical reliability
characterization of known flake classes under calibrated stress, not as
a per-commit gate; not part of ci-local.sh.
sidecar/ -- Network Sidecar Isolation
FIPS running as a sidecar container that owns the network namespace of
a companion application container, with iptables/ip6tables rules
confining the app to the mesh. scripts/test-sidecar.sh boots a
three-node chain of such pairs and asserts both connectivity and
isolation.
firewall/ -- nftables Baseline
End-to-end exercise of the production fips0 nftables baseline at
packaging/common/fips.nft, covering the default-deny, conntrack and
drop-in semantics.
acl-allowlist/ -- Peer ACL Enforcement
Six nodes with per-node allowlist files mounted at the runtime ACL paths, exercising insiders, outsiders and allowed remotes at once to check which peer pairs are admitted and which are rejected.
native-api/ -- Native Datagram API
Checks the experimental native datagram API: a client process opens a flow to a remote pubkey over a Unix socket, receives a file descriptor, and exchanges datagrams on it with no TUN device and no IPv6 emulation.
dns-resolver/ -- fips-dns-setup Backends
Runs fips-dns-setup against each supported Linux resolver backend in
systemd containers, verifying backend detection, generated config and
teardown, plus an end-to-end scenario that resolves a .fips name
through the configured backend.
deb-install/ -- Debian Package Install
Installs the built .deb in systemd containers for each
target distro and verifies unit enablement, conffile placement and
end-to-end .fips resolution as a user would meet it.
boringtun/ -- WireGuard Throughput Baseline
Two userspace WireGuard peers running Cloudflare BoringTun, measured
with iperf3, as a comparison baseline for FIPS tunnel throughput.
ble/ -- BLE L2CAP Spike
Standalone cargo project (ble_spike) that validates the bluer API
assumptions behind the BleIo trait against real adapters on two
machines. Not a Docker harness.
Running CI locally (ci-local.sh)
ci-local.sh runs the full local CI pipeline — build,
clippy, unit tests, and the integration suites (including the chaos
scenarios) — mirroring the GitHub ci.yml integration matrices. Run
./ci-local.sh --help for the full option list and --list for the
available suites. Every run starts with a parity check that verifies the
local suite set covers the same work as the GitHub matrix, per scenario for
chaos and per distro for deb-install, across every job that carries a
matrix; a divergence fails the run. GitHub
runs the same check as its own ci-parity job. --check-parity runs it
alone (see check-ci-parity.sh).
Per-run isolation and the FIPS_CI_RUN_ID override
Every invocation derives a run id and scopes all of its Docker resources to it, so two simultaneous runs on the same host (for example, one per git worktree, or an operator testing by hand while CI is in flight) never collide:
- Compose projects are named
fipsci_<run-id>_<suite>, so container, network, and volume names are all prefixed per run. - Build images are tagged
fips-test:<run-id>andfips-test-app:<run-id>, exported asFIPS_TEST_IMAGE/FIPS_TEST_APP_IMAGE, and every compose file and suite script reads those. The run does not writefips-test:latestat all: a bridge back to that shared mutable name would let a consumer that had been missed keep working while resolving whichever concurrent run wrote the tag last.:lateststays the hand-build name, produced bytesting/scripts/build.sh, and remains the default every consumer falls back to when the variables are unset. - The build context is a per-run copy at
testing/docker-<run-id>/, exported asFIPS_BUILD_CONTEXT. It is absolute because compose resolves a relative build context against the compose file's own directory rather than the working directory.testing/docker/is the hand-run context and a CI run does not write to it. Without this, two runs race on the contents of one directory and either can build a correctly-per-run-tagged image from the other's binaries. - Each parallel chaos child gets a unique, non-overlapping
/24in10.30.x(via the sim--subnetoverride).10.30.xsits outside Docker's default address pool and the fixed-subnet suites'172.xranges, so neither a sibling chaos child nor an auto-assigned network can swallow a pinned subnet.
By default the run id is <short-git-sha>-<random> — the SHA portion
records what code a container is testing, the random suffix keeps
simultaneous runs of the same SHA disjoint. Override it for a
reproducible, attach-by-name debug session:
FIPS_CI_RUN_ID=mydebug ./ci-local.sh --only static-mesh
# containers are named fipsci_mydebug_static_fips-node-a, etc.
Preemption-safety and exit codes
ci-local.sh is safe to cancel mid-run. A signal trap tears down every
compose project the run started (not just the current suite) and reaps
any in-flight parallel chaos children, bounded by a timeout so a stuck
compose down cannot wedge the trap. Exit codes distinguish a cancelled
run from a failing one:
| Code | Meaning |
|---|---|
0 |
all stages passed |
1 |
one or more stages failed |
130 |
interrupted by SIGINT — cancelled, not a failure |
143 |
terminated by SIGTERM — cancelled, not a failure |
A preempting CI worker (the push-triggered, CI-gated build pipeline that
kills an in-flight run when a newer same-branch tip arrives) maps
130/143 → cancelled (discard, do not record a failing commit), 0
→ green, any other non-zero → red.
Cleaning up leftover resources
Every CI-created container, network, and volume carries the label
com.corganlabs.fips-ci=1. If a run is hard-killed (SIGKILL, OOM, crash)
and leaves resources behind, reap them with:
./ci-local.sh --reap # or: ./ci-cleanup.sh
ci-cleanup.sh force-removes everything bearing the CI
label or a fipsci_ compose-project prefix; it is safe to run when there
is nothing to reap and safe to run repeatedly. Pass --project-prefix to
scope the sweep to a single run.
It also removes the chaos simulation's leftover host-namespace veth
interfaces (vh…a/vh…b), the one resource it touches that is neither a
docker object nor labelled — a host interface can carry neither a label
nor a compose project, so it is matched by name shape alone. That makes
the reach here asymmetric with everything above, and worth stating
plainly:
- A bare
chaos.shrun's containers survive a broad reap. Its compose project is notfipsci_, and the simulation labels only the network, not the services. - A bare
chaos.shrun's veth interfaces do not. An unscoped reap deletes them while they are in use, severing the Ethernet links of a live simulation and leaving its containers running.
So do not run a broad --reap while a bare simulation is up. Scope the
interface sweep with --veth-suffixes (which is what ci-local.sh's own
teardown passes) or wait for the simulation to finish. --project-prefix
does not help here: it scopes only the compose-project sweep.