fips/testing/static/README.md

Static Docker Network Test Harness

Multi-node integration test for FIPS using Docker containers with fixed topologies. Multiple topologies are provided: a sparse mesh (5 nodes, 6 links), a linear chain (5 nodes, 4 links), a mesh with a public external node, and a TCP chain (3 nodes). All exercise the full FIPS stack including TUN devices, DNS resolution, peer link encryption, spanning tree construction, and discovery-driven multi-hop routing.

Prerequisites

• Docker with the compose plugin
• Rust toolchain (for building the FIPS binary)
• Python 3 (for identity derivation; stdlib only, no packages required)

Quick Start

Build the binary and generate configs:

./testing/static/scripts/build.sh

Start the mesh (default topology):

docker compose -f testing/static/docker-compose.yml up -d
./testing/static/scripts/ping-test.sh mesh      # 20/20 expected
./testing/static/scripts/iperf-test.sh mesh     # bandwidth test
docker compose -f testing/static/docker-compose.yml down

The mesh profile is activated by default via .env. To use a different topology, specify the profile explicitly:

docker compose -f testing/static/docker-compose.yml --profile chain up -d
./testing/static/scripts/ping-test.sh chain
docker compose -f testing/static/docker-compose.yml --profile chain down

Topologies

Mesh

Five nodes with 6 bidirectional UDP links forming a sparse, fully connected graph. Not all nodes are direct peers -- non-adjacent pairs require discovery-driven multi-hop routing to establish end-to-end sessions.

The spanning tree is rooted at node A, which has the lexicographically smallest NodeAddr (the first 16 bytes of SHA-256(pubkey)). Tree edges are highlighted in blue in the diagram above.

The ping test exercises all 20 directed pairs (5 nodes x 4 targets each), covering both direct-peer and multi-hop paths.

Link	Type
A -- D	tree edge (D's parent is A)
A -- E	tree edge (E's parent is A)
C -- D	tree edge (C's parent is D)
B -- C	tree edge (B's parent is C)
D -- E	non-tree link
C -- E	non-tree link

Chain

Five nodes in a linear chain: A -- B -- C -- D -- E. Each node peers only with its immediate neighbors. Multi-hop communication (e.g., A to E) requires the discovery protocol to find routes through intermediate nodes.

The ping test covers:

• Adjacent hops: A->B, B->C (1 hop each)
• Multi-hop: A->C (2 hops), A->D (3 hops), A->E (4 hops)
• Reverse: E->A (4 hops)

Mesh-Public

Same five Docker nodes as the mesh topology, plus an external public node (pub) at a remote IP. Nodes A, B, and C peer with the public node. This topology is for testing mixed local/remote mesh operation.

External nodes are not managed by Docker -- only their identity and address appear in the topology file so that Docker nodes can peer with them.

TCP Chain

Three nodes in a linear chain using TCP transport (port 8443) instead of UDP: A -- B -- C. Each node peers only with its immediate neighbors. Tests basic TCP transport connectivity and multi-hop routing over TCP.

The topology file sets default_transport: tcp, which causes config generation to use TCP peer addresses (port 8443), inject the TCP transport section, and remove the UDP transport section.

Rekey

Same sparse mesh as the mesh topology (5 nodes, 6 links). Configs are post-processed to use aggressive rekey timers (35s) for CI testing. The rekey-test.sh script handles config injection and multi-phase verification.

Configuration Management

File Structure

testing/static/
├── Dockerfile                          # Container image definition
├── docker-compose.yml                  # Service definitions for all topologies
├── resolv.conf                         # DNS config pointing to FIPS resolver
├── .env                                # Default compose profile
├── configs/
│   ├── node.template.yaml              # Template for all node configs
│   └── topologies/
│       ├── mesh.yaml                   # Mesh topology definition
│       ├── chain.yaml                  # Chain topology definition
│       ├── mesh-public.yaml            # Mesh + external public node
│       ├── tcp-chain.yaml             # TCP chain (3 nodes, port 8443)
│       └── rekey.yaml                 # Rekey integration test (5 nodes)
├── generated-configs/                  # Auto-generated, run-scoped (gitignored)
│   ├── npubs.env                       # NPUB_A=..., NPUB_B=..., etc.
│   ├── mesh/
│   │   ├── node-a.yaml ... node-e.yaml
│   ├── mesh-public/
│   │   ├── node-a.yaml ... node-e.yaml
│   ├── chain/
│   │   ├── node-a.yaml ... node-e.yaml
│   └── tcp-chain/
│       ├── node-a.yaml ... node-c.yaml
├── scripts/
│   ├── build.sh                        # Build binary + generate configs
│   ├── generate-configs.sh             # Generate node configs from topology
│   ├── derive-keys.py                  # Deterministic nsec/npub derivation
│   ├── ping-test.sh                    # Connectivity test
│   ├── iperf-test.sh                   # Bandwidth test
│   └── netem.sh                        # Network impairment
├── docker-mesh-topology.svg            # Mesh topology diagram
└── docker-chain-topology.svg           # Chain topology diagram

Topology Files

Each topology file in configs/topologies/ defines:

• Node identities: nsec (hex) and npub (bech32) for each node
• Addresses: docker_ip for Docker-managed nodes, external_ip for remote nodes not managed by Docker
• Peer connections: which nodes peer with each other
• docker_host (optional): the compose hostname: this node answers to, when that is not node-<id>. Only the gateway topology needs it

Generated peer addresses use the docker hostname, not docker_ip. fips-net requests no subnet, so docker assigns one from its own pool and two concurrent CI runs can bring the topology up at the same time instead of one of them failing with Pool overlaps. docker_ip is retained as documentation of the topology's shape and as the internal/external discriminator; an external node keeps its external_ip in peer blocks, its address not being ours to assign.

Example entry:

nodes:
  a:
    nsec: "0102030405060708..."
    npub: "npub1sjlh2c3..."
    docker_ip: "172.20.0.10"
    peers: [d, e]

External nodes use external_ip instead of docker_ip. Config generation skips external nodes (they run outside Docker) but includes their identity in peer blocks and the npubs environment file.

Generating Configs

./testing/static/scripts/generate-configs.sh <topology> [mesh-name]

This reads the topology definition and generates:

1. Per-node YAML config files in generated-configs/<topology>/
2. generated-configs/npubs.env with all node npubs as environment variables

Under ci-local.sh the directory is generated-configs-<run-id>, so concurrent runs cannot overwrite each other's node configs; the compose file and every test script read the same FIPS_CI_NAME_SUFFIX and follow it. A bare invocation leaves the suffix unset and writes the plain path.

The npubs.env file is sourced by the test scripts and injected into Docker containers via env_file in docker-compose.yml.

The build script (scripts/build.sh) calls generate-configs.sh automatically after compiling.

Adding a New Topology

1. Create configs/topologies/<name>.yaml following the format of mesh.yaml
2. Add corresponding service definitions to docker-compose.yml with profiles: ["<name>"]
3. Run ./testing/static/scripts/generate-configs.sh <name> to generate configs

Deterministic Mesh Identity Derivation

When running multiple test meshes that may peer with the same external node, each mesh needs unique node identities to avoid key conflicts. The optional mesh-name parameter generates deterministic per-mesh identities:

# Build with derived identities
./testing/static/scripts/build.sh mesh my-mesh-1

# Or generate configs directly
./testing/static/scripts/generate-configs.sh mesh my-mesh-1
./testing/static/scripts/generate-configs.sh mesh-public my-mesh-1

How It Works

For each Docker node (those with docker_ip), the identity is derived as:

nsec = sha256(mesh_name + "|" + node_id)    # e.g., sha256("my-mesh-1|a")
npub = bech32("npub", secp256k1_pubkey(nsec))

External nodes (those with external_ip) always keep their hardcoded identity from the topology YAML, since they represent real nodes outside the test environment.

Without a mesh name, the identities from the topology YAML are used as-is (the original behavior).

The derive-keys.py Script

The derivation is performed by scripts/derive-keys.py, a standalone tool with no external dependencies (pure Python stdlib: hashlib for SHA-256, manual secp256k1 scalar multiplication, and BIP-173 bech32 encoding):

$ ./testing/static/scripts/derive-keys.py my-mesh-1 a
nsec=<64-char-hex>
npub=npub1...

The npubs.env File

Every run of generate-configs.sh writes generated-configs/npubs.env containing all node npubs, whether derived or from the topology YAML:

NPUB_A=npub1...
NPUB_B=npub1...
NPUB_C=npub1...
NPUB_D=npub1...
NPUB_E=npub1...
NPUB_PUB=npub1...    # only present for topologies with a pub node

This file is:

• Sourced by test scripts (ping-test.sh, iperf-test.sh) to resolve node identities for DNS lookups
• Injected into containers via the env_file directive in docker-compose.yml, making $NPUB_A etc. available as environment variables inside each container

Performance Testing

./testing/static/scripts/iperf-test.sh [mesh|chain]
./testing/static/scripts/iperf-test.sh mesh --live   # show live iperf3 output

Runs iperf3 with:

• Duration: 10 seconds (-t 10)
• Parallel streams: 8 (-P 8)
• Protocol: TCP over IPv6

For before/after measurements across commits or branches:

./testing/static/scripts/iperf-compare-refs.sh origin/master HEAD mesh

The comparison script builds each ref into a separate Docker image, runs the same topology and iperf3 settings for both images, and prints a bandwidth summary. Override DURATION, PARALLEL, SETTLE_SECONDS, IPERF_TIMEOUT, or RUNS in the environment when needed. RUNS is the total number of measurements per ref; for example, RUNS=3 runs each ref three times and prints both per-run and aggregate tables.

Network Impairment

The netem.sh script simulates adverse network conditions using tc/netem on all running containers:

./testing/static/scripts/netem.sh [mesh|chain] <apply|remove|status> [options]

Options

Option	Description
--delay <ms>	Fixed delay in milliseconds
--jitter <ms>	Delay variation (requires --delay)
--loss <percent>	Packet loss percentage
--loss-corr <percent>	Loss correlation for bursty loss
--duplicate <percent>	Packet duplication percentage
--reorder <percent>	Packet reordering probability (requires --delay)
--corrupt <percent>	Bit-level corruption percentage

Presets

Preset	Parameters
lossy	5% loss, 25% correlation
congested	50ms delay, 20ms jitter, 2% loss
terrible	100ms delay, 40ms jitter, 10% loss, 1% dup, 5% reorder

Examples

# Apply 50ms delay with 5% packet loss
./testing/static/scripts/netem.sh mesh apply --delay 50 --loss 5

# Use a preset
./testing/static/scripts/netem.sh chain apply --preset congested

# Check current rules
./testing/static/scripts/netem.sh mesh status

# Remove all impairment
./testing/static/scripts/netem.sh mesh remove

Rules are applied to egress on each container's eth0 interface. With all containers impaired equally, both directions of every link see the effect. The script uses tc qdisc replace so it can be re-run safely without removing rules first.

Container Configuration

• Base image: debian:bookworm-slim
• Capabilities: CAP_NET_ADMIN (for TUN device creation)
• Devices: /dev/net/tun mapped into each container
• DNS: FIPS built-in resolver on 127.0.0.1:53
• Transport: UDP on port 2121 (MTU 1472) or TCP on port 8443
• TUN: fips0 interface, MTU 1280

Each node resolves <npub>.fips DNS names to FIPS IPv6 addresses via its local DNS responder, which primes the identity cache for session establishment.

Background Services

Each container runs the following services alongside FIPS:

Service	Port	Description
SSH	22	Root login with no password (test only)
iperf3	5201	Bandwidth testing server (-s -D)
HTTP	80	Python HTTP server serving /root/index.html

All services bind to IPv6 (::) and are accessible over the FIPS overlay using <npub>.fips hostnames:

# HTTP over FIPS
docker exec fips-node-b curl http://$NPUB_A.fips

# SSH over FIPS
docker exec fips-node-b ssh $NPUB_A.fips

# iperf3 over FIPS
docker exec fips-node-b iperf3 -c $NPUB_A.fips

Troubleshooting

Stale images after code changes: Docker compose may cache old layers. Force a clean rebuild:

docker compose -f testing/static/docker-compose.yml build --no-cache

Check node logs:

docker logs fips-node-a
docker logs -f fips-node-c    # follow

Verify DNS resolution inside a container:

docker exec fips-node-a dig AAAA <npub>.fips @127.0.0.1

Verify binary is up to date: Compare hashes between the local build and the binary inside the container:

md5sum testing/static/fips
docker exec fips-node-a md5sum /usr/local/bin/fips

Increase convergence time: If tests fail intermittently, the 5-second convergence wait in ping-test.sh may be insufficient. Edit the sleep value at the top of the script.

Missing npubs.env: If test scripts fail with "npubs.env not found", run ./testing/static/scripts/generate-configs.sh mesh (or your topology) first, or use ./testing/static/scripts/build.sh which generates configs automatically.