Johnathan Corgan
1e111fe044
Any host that can reach the LAN resolver could ask for one new .fips name after another, and each got a virtual-IP mapping until the pool's 65,535 addresses ran out. Every mapping also adds to the cost of each NAT rebuild, each pool tick and shutdown. The pool now refuses a new name once it holds 1000 live mappings, and admits new names from a token bucket of 50 that refills at 10 per second. Both checks sit after the return for a name that already has a mapping, so names in use keep resolving when new ones are refused. The ceiling is checked first, so a refusal there takes no token and is always reported as the ceiling. A token is taken only once an address has been taken from the free list, so an exhausted pool costs none. Each limit has its own PoolError variant, and the "Pool allocation failed" warning names the one that refused. VirtualIpPool::new keeps its signature and uses the compiled-in limits; with_limits and allocate_at, which takes the instant that drives the refill, let the unit tests set small limits and a clock. The limits come from a flood of the gateway suite's gateway to 500, 1000 and 2000 live mappings with no limits in place. The measurement record is kept separately. Figures: - New mappings per second, over 100 creations: 215 for the first 100, 63 at 500, 47 at 1000, 32 at 2000. - NAT rebuild, median/max over the 20 adds up to the count: 17.5/26.5 ms at 500, 22.2/34.3 ms at 1000, 32.2/38.3 ms at 2000. - Pool tick: 24 to 32 us at 500, 48 to 53 us at 1000, 105 to 115 us at 2000, beside a conntrack read of 180 to 300 us. - Shutdown at 2000 mappings: 2.4 s. These are a best case for router hardware: they come from a container on a development host, not a router, and that host has no /proc/net/nf_conntrack, so the tick figures include no conntrack parsing. The ceiling is half the largest count measured. At 1000 a rebuild took about 22 ms and shutdown about 1.2 s. The rate is below the unthrottled creation rate at every count measured, so the bucket rather than the rebuild sets how fast a flood can fill the pool. From empty that now takes about 95 s; the unthrottled run passed 1000 in about 16 s. Ten rebuilds a second at the ceiling cost about a fifth of the gateway's single runtime thread on that host. The 2000 figures exist only in the measurement, since the committed phase stops at the ceiling. To measure and to keep measuring, the "Added DNAT/SNAT rules" and "Removed DNAT/SNAT rules" debug lines now carry the mapping count after the change and the rebuild's duration in microseconds, and are also emitted, with the error, when a rebuild fails, so a failure at some count leaves a record of that count. The pool tick logs a debug line with the mapping count and the durations of the conntrack read and of the tick. The suite's gateway logs at info because RUST_LOG=info overrides the entrypoint's --log-level debug, so the gw-gateway service now enables debug for the NAT manager and the gateway binary only. The gateway suite's last phase is now the regression. It restarts the gateway with mappings that outlive the phase and reads the limits from pool.rs. It fills the pool to the ceiling with the readiness probe's mapping plus ceiling - 1 new names, retrying rate refusals, then asks once each for 20 more. It asserts all 20 get SERVFAIL, the live count equals the ceiling, at least 20 ceiling refusals are logged, the rate limit refused during the fill and not after it, the probe's name keeps its address, nothing is reclaimed, no NAT or proxy NDP failure is logged, and no 10 s window of the fill holds more than burst + 10 x rate creations. It also reports rebuild durations at 500 and 1000, ticks as they fall, and the shutdown time at the ceiling. On a tree with these assertions but no checks in allocate the phase fails five of them: 1020 mappings, 0 SERVFAIL, no ceiling or rate refusals, and 713 creations in one 10 s window. With the limits it passes in about 140 s, with rebuilds of 17.8/24.5 ms at 500 and 20.9/33.9 ms at 1000 and a 1.2 s shutdown at the ceiling. The NAT batch phase now also goes through the rate limit, so its driver retries refusals; it shares the restart, readiness gate and DNS driver with the new phase instead of carrying its own copies.
Static Docker Network Test Harness
Multi-node integration test for FIPS using Docker containers with fixed topologies. Multiple topologies are provided: a sparse mesh (5 nodes, 6 links), a linear chain (5 nodes, 4 links), a gateway topology (3 nodes plus a non-FIPS LAN client), and three rekey variants. All exercise the full FIPS stack including TUN devices, DNS resolution, peer link encryption, spanning tree construction, and discovery-driven multi-hop routing.
Prerequisites
- Docker with the compose plugin
- Rust toolchain (for building the FIPS binary)
- Python 3 (for identity derivation; stdlib only, no packages required)
Quick Start
Build the binaries and images, then generate the node configs:
./testing/scripts/build.sh
./testing/static/scripts/generate-configs.sh mesh
build.sh is the shared harness builder and is run from the repo root; it
does not generate configs.
Start the mesh (default topology):
docker compose -f testing/static/docker-compose.yml up -d
./testing/static/scripts/ping-test.sh mesh # 20/20 expected
./testing/static/scripts/iperf-test.sh mesh # bandwidth test
docker compose -f testing/static/docker-compose.yml down
The mesh profile is activated by default via .env. To use a different
topology, specify the profile explicitly:
docker compose -f testing/static/docker-compose.yml --profile chain up -d
./testing/static/scripts/ping-test.sh chain
docker compose -f testing/static/docker-compose.yml --profile chain down
Topologies
Mesh
Five nodes with 6 bidirectional UDP links forming a sparse, fully connected graph. Not all nodes are direct peers -- non-adjacent pairs require discovery-driven multi-hop routing to establish end-to-end sessions.
The spanning tree is rooted at node A, which has the lexicographically
smallest NodeAddr (the first 16 bytes of SHA-256(pubkey)). Tree edges
are highlighted in blue in the diagram above.
The ping test exercises all 20 directed pairs (5 nodes x 4 targets each), covering both direct-peer and multi-hop paths.
| Link | Type |
|---|---|
| A -- D | tree edge (D's parent is A) |
| A -- E | tree edge (E's parent is A) |
| C -- D | tree edge (C's parent is D) |
| B -- C | tree edge (B's parent is C) |
| D -- E | non-tree link |
| C -- E | non-tree link |
Chain
Five nodes in a linear chain: A -- B -- C -- D -- E. Each node peers only with its immediate neighbors. Multi-hop communication (e.g., A to E) requires the discovery protocol to find routes through intermediate nodes.
The ping test covers:
- Adjacent hops: A->B, B->C (1 hop each)
- Multi-hop: A->C (2 hops), A->D (3 hops), A->E (4 hops)
- Reverse: E->A (4 hops)
Gateway
Three FIPS nodes: a gateway (a) with a LAN interface, and two mesh
destinations (b, c) directly peered with it. A non-FIPS client container
attaches to the gateway's LAN interface. Two destinations are required so the
multi-client phase of gateway-test.sh can allocate distinct virtual-IP
mappings, one per LAN client. Identities are derived deterministically from
the mesh name gateway-test.
Rekey
Same sparse mesh as the mesh topology (5 nodes, 6 links). Configs are
post-processed to use aggressive rekey timers (35s) for CI testing. The
rekey-test.sh script handles config injection and multi-phase verification.
Rekey-Accept-Off
The rekey topology with transports.udp.accept_connections: false set on
node B, the single-peer node auto-connected to C. Pins the regression where a
rekey msg1 arriving at an auto-connect initiator with accept off was dropped
by the Node-level admission gate.
Rekey-Outbound-Only
The rekey topology with transports.udp.outbound_only: true on node B, whose
peer-C address is also rewritten to the Docker hostname (node-c:2121). Pins
the regression where the hostname-versus-numeric mismatch made the
addr_to_link lookup miss and the admission carve-out fall through.
Configuration Management
File Structure
testing/static/
├── docker-compose.yml # Service definitions for all topologies
├── docker-compose.gateway-external-net.yml # Gateway on an external network
├── .env # Default compose profile
├── configs/
│ ├── node.template.yaml # Template for all node configs
│ ├── gateway-resolv.conf # LAN client resolver config
│ └── topologies/
│ ├── mesh.yaml # Mesh topology definition
│ ├── chain.yaml # Chain topology definition
│ ├── gateway.yaml # Gateway integration test (3 nodes)
│ ├── rekey.yaml # Rekey integration test (5 nodes)
│ ├── rekey-accept-off.yaml # Rekey with accept_connections off
│ └── rekey-outbound-only.yaml # Rekey with outbound_only
├── generated-configs/ # Auto-generated, run-scoped (gitignored)
│ ├── npubs.env # NPUB_A=..., NPUB_B=..., etc.
│ ├── mesh/
│ │ ├── node-a.yaml ... node-e.yaml
│ └── chain/
│ ├── node-a.yaml ... node-e.yaml
├── scripts/
│ ├── generate-configs.sh # Generate node configs from topology
│ ├── ping-test.sh # Connectivity test
│ ├── iperf-test.sh # Bandwidth test
│ ├── iperf-compare-refs.sh # Bandwidth comparison across refs
│ ├── bench-multirun.sh # Repeated benchmark runs
│ ├── gateway-test.sh # Gateway integration test
│ ├── rekey-test.sh # Rekey integration test
│ ├── admission-cap-test.sh # Peer admission cap test
│ └── netem.sh # Network impairment
├── docker-mesh-topology.svg # Mesh topology diagram
└── docker-chain-topology.svg # Chain topology diagram
The container image definition (Dockerfile), its entrypoint and the
resolv.conf that points at the FIPS resolver are shared with the other
harnesses and live in testing/docker/. The identity-derivation helper is
testing/lib/derive_keys.py.
Topology Files
Each topology file in configs/topologies/ defines:
- Node identities: nsec (hex) and npub (bech32) for each node
- Addresses:
docker_ipfor Docker-managed nodes,external_ipfor remote nodes not managed by Docker - Peer connections: which nodes peer with each other
docker_host(optional): the composehostname:this node answers to, when that is notnode-<id>. Only the gateway topology needs it
Generated peer addresses use the docker hostname, not docker_ip.
fips-net requests no subnet, so docker assigns one from its own pool and two
concurrent CI runs can bring the topology up at the same time instead of one
of them failing with Pool overlaps. docker_ip is retained as documentation
of the topology's shape and as the internal/external discriminator; an
external node keeps its external_ip in peer blocks, its address not being
ours to assign.
Example entry:
nodes:
a:
nsec: "0102030405060708..."
npub: "npub1sjlh2c3..."
docker_ip: "172.20.0.10"
peers: [d, e]
External nodes use external_ip instead of docker_ip. Config generation
skips external nodes (they run outside Docker) but includes their identity
in peer blocks and the npubs environment file.
Generating Configs
./testing/static/scripts/generate-configs.sh <topology> [mesh-name]
This reads the topology definition and generates:
- Per-node YAML config files in
generated-configs/<topology>/ generated-configs/npubs.envwith all node npubs as environment variables
Under ci-local.sh the directory is generated-configs-<run-id>, so
concurrent runs cannot overwrite each other's node configs; the compose file
and every test script read the same FIPS_CI_NAME_SUFFIX and follow it. A
bare invocation leaves the suffix unset and writes the plain path.
The npubs.env file is sourced by the test scripts and injected into
Docker containers via env_file in docker-compose.yml.
testing/scripts/build.sh compiles the binaries and builds the images; run
generate-configs.sh separately afterwards.
Adding a New Topology
- Create
configs/topologies/<name>.yamlfollowing the format ofmesh.yaml - Add corresponding service definitions to
docker-compose.ymlwithprofiles: ["<name>"] - Run
./testing/static/scripts/generate-configs.sh <name>to generate configs
Deterministic Mesh Identity Derivation
When running multiple test meshes that may peer with the same external node,
each mesh needs unique node identities to avoid key conflicts. The optional
mesh-name parameter generates deterministic per-mesh identities:
./testing/static/scripts/generate-configs.sh mesh my-mesh-1
./testing/static/scripts/generate-configs.sh chain my-mesh-1
How It Works
For each Docker node (those with docker_ip), the identity is derived as:
nsec = sha256(mesh_name + "|" + node_id) # e.g., sha256("my-mesh-1|a")
npub = bech32("npub", secp256k1_pubkey(nsec))
External nodes (those with external_ip) always keep their hardcoded
identity from the topology YAML, since they represent real nodes outside
the test environment.
Without a mesh name, the identities from the topology YAML are used as-is (the original behavior).
The derive_keys.py Script
The derivation is performed by testing/lib/derive_keys.py, a standalone tool
with no external dependencies (pure Python stdlib: hashlib for SHA-256,
manual secp256k1 scalar multiplication, and BIP-173 bech32 encoding):
$ python3 testing/lib/derive_keys.py my-mesh-1 a
nsec=<64-char-hex>
npub=npub1...
The npubs.env File
Every run of generate-configs.sh writes generated-configs/npubs.env
containing all node npubs, whether derived or from the topology YAML:
NPUB_A=npub1...
NPUB_B=npub1...
NPUB_C=npub1...
NPUB_D=npub1...
NPUB_E=npub1...
This file is:
- Sourced by test scripts (
ping-test.sh,iperf-test.sh) to resolve node identities for DNS lookups - Injected into containers via the
env_filedirective indocker-compose.yml, making$NPUB_Aetc. available as environment variables inside each container
Performance Testing
./testing/static/scripts/iperf-test.sh [mesh|chain]
./testing/static/scripts/iperf-test.sh mesh --live # show live iperf3 output
Runs iperf3 with:
- Duration: 10 seconds (
-t 10) - Parallel streams: 8 (
-P 8) - Protocol: TCP over IPv6
For before/after measurements across commits or branches:
./testing/static/scripts/iperf-compare-refs.sh origin/master HEAD mesh
The comparison script builds each ref into a separate Docker image, runs the
same topology and iperf3 settings for both images, and prints a bandwidth
summary. Override DURATION, PARALLEL, SETTLE_SECONDS, IPERF_TIMEOUT,
or RUNS in the environment when needed. RUNS is the total number of
measurements per ref; for example, RUNS=3 runs each ref three times and
prints both per-run and aggregate tables.
Network Impairment
The netem.sh script simulates adverse network conditions using tc/netem
on all running containers:
./testing/static/scripts/netem.sh [mesh|chain] <apply|remove|status> [options]
Options
| Option | Description |
|---|---|
--delay <ms> |
Fixed delay in milliseconds |
--jitter <ms> |
Delay variation (requires --delay) |
--loss <percent> |
Packet loss percentage |
--loss-corr <percent> |
Loss correlation for bursty loss |
--duplicate <percent> |
Packet duplication percentage |
--reorder <percent> |
Packet reordering probability (requires --delay) |
--corrupt <percent> |
Bit-level corruption percentage |
Presets
| Preset | Parameters |
|---|---|
lossy |
5% loss, 25% correlation |
congested |
50ms delay, 20ms jitter, 2% loss |
terrible |
100ms delay, 40ms jitter, 10% loss, 1% dup, 5% reorder |
Examples
# Apply 50ms delay with 5% packet loss
./testing/static/scripts/netem.sh mesh apply --delay 50 --loss 5
# Use a preset
./testing/static/scripts/netem.sh chain apply --preset congested
# Check current rules
./testing/static/scripts/netem.sh mesh status
# Remove all impairment
./testing/static/scripts/netem.sh mesh remove
Rules are applied to egress on each container's eth0 interface. With all
containers impaired equally, both directions of every link see the effect.
The script uses tc qdisc replace so it can be re-run safely without
removing rules first.
Container Configuration
- Base image: debian:bookworm-slim
- Capabilities:
CAP_NET_ADMIN(for TUN device creation) - Devices:
/dev/net/tunmapped into each container - DNS: FIPS built-in resolver on
127.0.0.1:53 - Transport: UDP on port 2121 (MTU 1472) or TCP on port 8443
- TUN:
fips0interface, MTU 1280
Each node resolves <npub>.fips DNS names to FIPS IPv6 addresses via its
local DNS responder, which primes the identity cache for session establishment.
Background Services
Each container runs the following services alongside FIPS:
| Service | Port | Description |
|---|---|---|
| SSH | 22 | Root login with no password (test only) |
| iperf3 | 5201 | Bandwidth testing server (-s -D) |
| HTTP | 80 | Python HTTP server serving /root/index.html |
All services bind to IPv6 (::) and are accessible over the FIPS overlay
using <npub>.fips hostnames:
# HTTP over FIPS
docker exec fips-node-b curl http://$NPUB_A.fips
# SSH over FIPS
docker exec fips-node-b ssh $NPUB_A.fips
# iperf3 over FIPS
docker exec fips-node-b iperf3 -c $NPUB_A.fips
Troubleshooting
Stale images after code changes: Docker compose may cache old layers. Force a clean rebuild:
docker compose -f testing/static/docker-compose.yml build --no-cache
Check node logs:
docker logs fips-node-a
docker logs -f fips-node-c # follow
Verify DNS resolution inside a container:
docker exec fips-node-a dig AAAA <npub>.fips @127.0.0.1
Verify binary is up to date: Compare hashes between the local build and the binary inside the container:
md5sum testing/docker/fips
docker exec fips-node-a md5sum /usr/local/bin/fips
Increase convergence time: If tests fail intermittently, the 5-second
convergence wait in ping-test.sh may be insufficient. Edit the sleep
value at the top of the script.
Missing npubs.env: If test scripts fail with "npubs.env not found", run
./testing/static/scripts/generate-configs.sh mesh (or your topology) first.