Johnathan Corgan
5a79d3fc5e
Brings up the two release lines' work: the maint harness and guard fixes, the documentation corrections, master's probe, onion and epoch fixes, and both rebuilt changelog blocks. Three conflicts. The readme conflicted on the badge pair. Resolved by taking the Rust badge that no longer asserts a version, since rust-toolchain.toml is the only place that states one, and keeping this line's own v0.6.0-dev status badge. The peer machine conflicted, and the resolution is an adaptation rather than a pick. Master deleted PeerMachine.remote_epoch on the grounds that nothing read it, and that reasoning had to be re-derived here because this line's machine is the XX rewrite and shares almost no text with it. It holds. The shadow's only production write is inbound_msg3, which is where XX crystallizes identity, so it is inbound-only exactly as the msg1 write was on the other lines; conn carries the same value written from both legs, complete_handshake on the outbound one and complete_handshake_msg3 on the inbound; the only read is the cutover action payload, whose executor arm binds nothing; and the live consumer reads conn_remote_epoch. So an initiator cutover, which runs on an outbound machine, carried a zeroed epoch here too. One thing differs and needed handling. This line has an `established` constructor the others do not, and it writes the shadow and conn from the same argument, which would have made the field direction-correct. Its only caller is in the test module and its own doc comment calls the machine inert, so it is a seam that is not wired yet rather than a production path, and it does not rescue the field. Its assignment goes with the rest; the parameter stays, because conn still needs the value. The adaptation is folded into this merge rather than left to a follow-on, because master's half of the same change reached peer_actions.rs through a clean auto-merge. Keeping this line's field while accepting that auto-merge would have left the machine emitting a payload field the executor no longer has, which is a break that only the test build shows. The changelog conflicted because both lines had rebuilt their unreleased block. The Breaking section stays at the top untouched; Unreleased now holds the ten entries that are this line's own; and the other two lines' work sits below under 0.5.0 and 0.4.2 headings, neither dated, matching how master already carries 0.4.2. Four entries existed on both sides in branch-adapted form and were merged rather than picked, so each keeps the rework's wording and this line's accuracy: the OpenWrt entry drops its IK reference, the msg1 classifier keeps the promotion-state paragraph, the SessionAck entry keeps the two XX-only exits, and the msg3 epoch entry counts six sites here against master's five.
Static Docker Network Test Harness
Multi-node integration test for FIPS using Docker containers with fixed topologies. Multiple topologies are provided: a sparse mesh (5 nodes, 6 links), a linear chain (5 nodes, 4 links), a mesh with a public external node, and a TCP chain (3 nodes). All exercise the full FIPS stack including TUN devices, DNS resolution, peer link encryption, spanning tree construction, and discovery-driven multi-hop routing.
Prerequisites
- Docker with the compose plugin
- Rust toolchain (for building the FIPS binary)
- Python 3 (for identity derivation; stdlib only, no packages required)
Quick Start
Build the binary and generate configs:
./testing/static/scripts/build.sh
Start the mesh (default topology):
docker compose -f testing/static/docker-compose.yml up -d
./testing/static/scripts/ping-test.sh mesh # 20/20 expected
./testing/static/scripts/iperf-test.sh mesh # bandwidth test
docker compose -f testing/static/docker-compose.yml down
The mesh profile is activated by default via .env. To use a different
topology, specify the profile explicitly:
docker compose -f testing/static/docker-compose.yml --profile chain up -d
./testing/static/scripts/ping-test.sh chain
docker compose -f testing/static/docker-compose.yml --profile chain down
Topologies
Mesh
Five nodes with 6 bidirectional UDP links forming a sparse, fully connected graph. Not all nodes are direct peers -- non-adjacent pairs require discovery-driven multi-hop routing to establish end-to-end sessions.
The spanning tree is rooted at node A, which has the lexicographically
smallest NodeAddr (the first 16 bytes of SHA-256(pubkey)). Tree edges
are highlighted in blue in the diagram above.
The ping test exercises all 20 directed pairs (5 nodes x 4 targets each), covering both direct-peer and multi-hop paths.
| Link | Type |
|---|---|
| A -- D | tree edge (D's parent is A) |
| A -- E | tree edge (E's parent is A) |
| C -- D | tree edge (C's parent is D) |
| B -- C | tree edge (B's parent is C) |
| D -- E | non-tree link |
| C -- E | non-tree link |
Chain
Five nodes in a linear chain: A -- B -- C -- D -- E. Each node peers only with its immediate neighbors. Multi-hop communication (e.g., A to E) requires the discovery protocol to find routes through intermediate nodes.
The ping test covers:
- Adjacent hops: A->B, B->C (1 hop each)
- Multi-hop: A->C (2 hops), A->D (3 hops), A->E (4 hops)
- Reverse: E->A (4 hops)
Mesh-Public
Same five Docker nodes as the mesh topology, plus an external public node
(pub) at a remote IP. Nodes A, B, and C peer with the public node. This
topology is for testing mixed local/remote mesh operation.
External nodes are not managed by Docker -- only their identity and address appear in the topology file so that Docker nodes can peer with them.
TCP Chain
Three nodes in a linear chain using TCP transport (port 8443) instead of UDP: A -- B -- C. Each node peers only with its immediate neighbors. Tests basic TCP transport connectivity and multi-hop routing over TCP.
The topology file sets default_transport: tcp, which causes config
generation to use TCP peer addresses (port 8443), inject the TCP transport
section, and remove the UDP transport section.
Rekey
Same sparse mesh as the mesh topology (5 nodes, 6 links). Configs are
post-processed to use aggressive rekey timers (35s) for CI testing. The
rekey-test.sh script handles config injection and multi-phase verification.
Configuration Management
File Structure
testing/static/
├── Dockerfile # Container image definition
├── docker-compose.yml # Service definitions for all topologies
├── resolv.conf # DNS config pointing to FIPS resolver
├── .env # Default compose profile
├── configs/
│ ├── node.template.yaml # Template for all node configs
│ └── topologies/
│ ├── mesh.yaml # Mesh topology definition
│ ├── chain.yaml # Chain topology definition
│ └── rekey.yaml # Rekey integration test (5 nodes)
├── generated-configs/ # Auto-generated, run-scoped (gitignored)
│ ├── npubs.env # NPUB_A=..., NPUB_B=..., etc.
│ ├── mesh/
│ │ ├── node-a.yaml ... node-e.yaml
│ └── chain/
│ ├── node-a.yaml ... node-e.yaml
├── scripts/
│ ├── build.sh # Build binary + generate configs
│ ├── generate-configs.sh # Generate node configs from topology
│ ├── derive-keys.py # Deterministic nsec/npub derivation
│ ├── ping-test.sh # Connectivity test
│ ├── iperf-test.sh # Bandwidth test
│ └── netem.sh # Network impairment
├── docker-mesh-topology.svg # Mesh topology diagram
└── docker-chain-topology.svg # Chain topology diagram
Topology Files
Each topology file in configs/topologies/ defines:
- Node identities: nsec (hex) and npub (bech32) for each node
- Addresses:
docker_ipfor Docker-managed nodes,external_ipfor remote nodes not managed by Docker - Peer connections: which nodes peer with each other
docker_host(optional): the composehostname:this node answers to, when that is notnode-<id>. Only the gateway topology needs it
Generated peer addresses use the docker hostname, not docker_ip.
fips-net requests no subnet, so docker assigns one from its own pool and two
concurrent CI runs can bring the topology up at the same time instead of one
of them failing with Pool overlaps. docker_ip is retained as documentation
of the topology's shape and as the internal/external discriminator; an
external node keeps its external_ip in peer blocks, its address not being
ours to assign.
Example entry:
nodes:
a:
nsec: "0102030405060708..."
npub: "npub1sjlh2c3..."
docker_ip: "172.20.0.10"
peers: [d, e]
External nodes use external_ip instead of docker_ip. Config generation
skips external nodes (they run outside Docker) but includes their identity
in peer blocks and the npubs environment file.
Generating Configs
./testing/static/scripts/generate-configs.sh <topology> [mesh-name]
This reads the topology definition and generates:
- Per-node YAML config files in
generated-configs/<topology>/ generated-configs/npubs.envwith all node npubs as environment variables
Under ci-local.sh the directory is generated-configs-<run-id>, so
concurrent runs cannot overwrite each other's node configs; the compose file
and every test script read the same FIPS_CI_NAME_SUFFIX and follow it. A
bare invocation leaves the suffix unset and writes the plain path.
The npubs.env file is sourced by the test scripts and injected into
Docker containers via env_file in docker-compose.yml.
The build script (scripts/build.sh) calls generate-configs.sh
automatically after compiling.
Adding a New Topology
- Create
configs/topologies/<name>.yamlfollowing the format ofmesh.yaml - Add corresponding service definitions to
docker-compose.ymlwithprofiles: ["<name>"] - Run
./testing/static/scripts/generate-configs.sh <name>to generate configs
Deterministic Mesh Identity Derivation
When running multiple test meshes that may peer with the same external node,
each mesh needs unique node identities to avoid key conflicts. The optional
mesh-name parameter generates deterministic per-mesh identities:
# Build with derived identities
./testing/static/scripts/build.sh mesh my-mesh-1
# Or generate configs directly
./testing/static/scripts/generate-configs.sh mesh my-mesh-1
./testing/static/scripts/generate-configs.sh chain my-mesh-1
How It Works
For each Docker node (those with docker_ip), the identity is derived as:
nsec = sha256(mesh_name + "|" + node_id) # e.g., sha256("my-mesh-1|a")
npub = bech32("npub", secp256k1_pubkey(nsec))
External nodes (those with external_ip) always keep their hardcoded
identity from the topology YAML, since they represent real nodes outside
the test environment.
Without a mesh name, the identities from the topology YAML are used as-is (the original behavior).
The derive-keys.py Script
The derivation is performed by scripts/derive-keys.py, a standalone tool
with no external dependencies (pure Python stdlib: hashlib for SHA-256,
manual secp256k1 scalar multiplication, and BIP-173 bech32 encoding):
$ ./testing/static/scripts/derive-keys.py my-mesh-1 a
nsec=<64-char-hex>
npub=npub1...
The npubs.env File
Every run of generate-configs.sh writes generated-configs/npubs.env
containing all node npubs, whether derived or from the topology YAML:
NPUB_A=npub1...
NPUB_B=npub1...
NPUB_C=npub1...
NPUB_D=npub1...
NPUB_E=npub1...
NPUB_PUB=npub1... # only present for topologies with a pub node
This file is:
- Sourced by test scripts (
ping-test.sh,iperf-test.sh) to resolve node identities for DNS lookups - Injected into containers via the
env_filedirective indocker-compose.yml, making$NPUB_Aetc. available as environment variables inside each container
Performance Testing
./testing/static/scripts/iperf-test.sh [mesh|chain]
./testing/static/scripts/iperf-test.sh mesh --live # show live iperf3 output
Runs iperf3 with:
- Duration: 10 seconds (
-t 10) - Parallel streams: 8 (
-P 8) - Protocol: TCP over IPv6
For before/after measurements across commits or branches:
./testing/static/scripts/iperf-compare-refs.sh origin/master HEAD mesh
The comparison script builds each ref into a separate Docker image, runs the
same topology and iperf3 settings for both images, and prints a bandwidth
summary. Override DURATION, PARALLEL, SETTLE_SECONDS, IPERF_TIMEOUT,
or RUNS in the environment when needed. RUNS is the total number of
measurements per ref; for example, RUNS=3 runs each ref three times and
prints both per-run and aggregate tables.
Network Impairment
The netem.sh script simulates adverse network conditions using tc/netem
on all running containers:
./testing/static/scripts/netem.sh [mesh|chain] <apply|remove|status> [options]
Options
| Option | Description |
|---|---|
--delay <ms> |
Fixed delay in milliseconds |
--jitter <ms> |
Delay variation (requires --delay) |
--loss <percent> |
Packet loss percentage |
--loss-corr <percent> |
Loss correlation for bursty loss |
--duplicate <percent> |
Packet duplication percentage |
--reorder <percent> |
Packet reordering probability (requires --delay) |
--corrupt <percent> |
Bit-level corruption percentage |
Presets
| Preset | Parameters |
|---|---|
lossy |
5% loss, 25% correlation |
congested |
50ms delay, 20ms jitter, 2% loss |
terrible |
100ms delay, 40ms jitter, 10% loss, 1% dup, 5% reorder |
Examples
# Apply 50ms delay with 5% packet loss
./testing/static/scripts/netem.sh mesh apply --delay 50 --loss 5
# Use a preset
./testing/static/scripts/netem.sh chain apply --preset congested
# Check current rules
./testing/static/scripts/netem.sh mesh status
# Remove all impairment
./testing/static/scripts/netem.sh mesh remove
Rules are applied to egress on each container's eth0 interface. With all
containers impaired equally, both directions of every link see the effect.
The script uses tc qdisc replace so it can be re-run safely without
removing rules first.
Container Configuration
- Base image: debian:bookworm-slim
- Capabilities:
CAP_NET_ADMIN(for TUN device creation) - Devices:
/dev/net/tunmapped into each container - DNS: FIPS built-in resolver on
127.0.0.1:53 - Transport: UDP on port 2121 (MTU 1472) or TCP on port 8443
- TUN:
fips0interface, MTU 1280
Each node resolves <npub>.fips DNS names to FIPS IPv6 addresses via its
local DNS responder, which primes the identity cache for session establishment.
Background Services
Each container runs the following services alongside FIPS:
| Service | Port | Description |
|---|---|---|
| SSH | 22 | Root login with no password (test only) |
| iperf3 | 5201 | Bandwidth testing server (-s -D) |
| HTTP | 80 | Python HTTP server serving /root/index.html |
All services bind to IPv6 (::) and are accessible over the FIPS overlay
using <npub>.fips hostnames:
# HTTP over FIPS
docker exec fips-node-b curl http://$NPUB_A.fips
# SSH over FIPS
docker exec fips-node-b ssh $NPUB_A.fips
# iperf3 over FIPS
docker exec fips-node-b iperf3 -c $NPUB_A.fips
Troubleshooting
Stale images after code changes: Docker compose may cache old layers. Force a clean rebuild:
docker compose -f testing/static/docker-compose.yml build --no-cache
Check node logs:
docker logs fips-node-a
docker logs -f fips-node-c # follow
Verify DNS resolution inside a container:
docker exec fips-node-a dig AAAA <npub>.fips @127.0.0.1
Verify binary is up to date: Compare hashes between the local build and the binary inside the container:
md5sum testing/static/fips
docker exec fips-node-a md5sum /usr/local/bin/fips
Increase convergence time: If tests fail intermittently, the 5-second
convergence wait in ping-test.sh may be insufficient. Edit the sleep
value at the top of the script.
Missing npubs.env: If test scripts fail with "npubs.env not found", run
./testing/static/scripts/generate-configs.sh mesh (or your topology) first,
or use ./testing/static/scripts/build.sh which generates configs automatically.