Files
fips/testing/static/docker-compose.yml
T
Johnathan Corgan 6c52b0e01e Let the static test network float so concurrent CI runs cannot collide
Two local CI runs on one host both asked docker for 172.20.0.0/24 and the
second lost its whole static family to "Pool overlaps". Docker honours a
fixed subnet request verbatim, so the only robust fix is to stop making one:
fips-net now requests no subnet and docker assigns from its own pool, which
cannot hand the same range to two runs.

That means node addresses are not known before `up`, so peers address each
other by container hostname instead. The generator emits node-<id>, or the
topology's docker_host where the compose hostname differs — only the gateway
profile, whose services are gw-*. External peers keep the address the
topology gives them, since it is not ours to assign. The resolv.conf mount
stays: dnsmasq is what forwards these names to docker's resolver and .fips
to the daemon, so removing it would take out every .fips assertion.

generated-configs is now per-run as well. A shared directory let two runs
overwrite each other's node configs, which the subnet collision had been
hiding by killing runs before that window opened. The generator, the compose
bind mounts and env_file, the six scripts that read it, and teardown all
follow FIPS_CI_NAME_SUFFIX; unset, every path renders as before. Teardown
keeps the directory after a failed run, where it is the evidence of what the
failing nodes were configured with.

Three things this exposed that were wrong independently:

admission-cap built its tcpdump patterns from the topology file's docker_ip
literals. Floating the subnet makes those match nothing, which would have
left its expect-zero "no Msg2 leaked" assertion passing because it could no
longer see anything at all. It now reads addresses from the running
containers. Restarting the denied peers together also made them swap
addresses, so each peer's counts were really the pair's total; they are
restarted one at a time now, and a check fails the suite outright if two
denied peers ever share an address, because per-peer attribution is
impossible once they do.

Attribute lookups in the generator used a fixed ten-line window and read the
next node's fields when a node omitted an attribute. An external node
followed by an internal one was classified as internal, which under hostname
peering would emit a name that resolves nowhere. Lookups are bounded to the
node's own block; generated output is byte-identical for all eight
topologies.

The rekey outbound-only variant used to rewrite peer addresses to hostnames
to set up its scenario. The generator now does that everywhere, so the
rewrite matched nothing and was silently doing no work. It asserts the
premise instead, and fails if a numeric address ever reappears.

Verified by running three instances of this compose at once — tcp-chain plus
two independent meshes — which drew 10.128.2/3/4.0/24 with no overlap while
both meshes passed ping-test 20/20 over the real .fips path. tcp-chain is
run by neither CI runner, so it was checked by hand: chain peer counts 1/2/1
and multi-hop .fips reachable both directions over TCP.

gateway-lan still pins its own IPv4 and fd02:: ranges and is unchanged here,
so the gateway profile is not yet concurrency-safe.
2026-07-23 02:05:56 +00:00

546 lines
18 KiB
YAML

networks:
# No subnet is requested: docker assigns one from the daemon's address pool,
# which is what lets two concurrent CI runs bring this topology up at the
# same time. A fixed request is honoured verbatim, so two runs asking for the
# same range collide on "Pool overlaps" — the whole reason mesh nodes now
# peer by docker hostname rather than by address.
fips-net:
driver: bridge
labels:
- "com.corganlabs.fips-ci=1"
gateway-lan:
driver: bridge
labels:
- "com.corganlabs.fips-ci=1"
enable_ipv6: true
ipam:
config:
- subnet: 172.20.1.0/24
- subnet: fd02::/64
x-fips-common: &fips-common
image: ${FIPS_TEST_IMAGE:-fips-test:latest}
cap_add:
- NET_ADMIN
devices:
- /dev/net/tun:/dev/net/tun
sysctls:
- net.ipv6.conf.all.disable_ipv6=0
restart: "no"
env_file:
- ./generated-configs${FIPS_CI_NAME_SUFFIX:-}/npubs.env
environment:
- RUST_LOG=info
# Passthrough for A/B benchmarking — set on the host shell before
# `docker compose up` to override the worker-pool defaults.
- FIPS_ENCRYPT_WORKERS=${FIPS_ENCRYPT_WORKERS:-}
- FIPS_DECRYPT_WORKERS=${FIPS_DECRYPT_WORKERS:-}
- FIPS_PERF=${FIPS_PERF:-0}
- FIPS_PERF_INTERVAL_SECS=${FIPS_PERF_INTERVAL_SECS:-5}
services:
# ── Mesh topology ──────────────────────────────────────────────
node-a:
<<: *fips-common
profiles: ["mesh"]
container_name: fips-node-a${FIPS_CI_NAME_SUFFIX:-}
hostname: node-a
volumes:
- ../docker/resolv.conf:/etc/resolv.conf:ro
- ./generated-configs${FIPS_CI_NAME_SUFFIX:-}/mesh/node-a.yaml:/etc/fips/fips.yaml:ro
networks:
fips-net:
node-b:
<<: *fips-common
profiles: ["mesh"]
container_name: fips-node-b${FIPS_CI_NAME_SUFFIX:-}
hostname: node-b
volumes:
- ../docker/resolv.conf:/etc/resolv.conf:ro
- ./generated-configs${FIPS_CI_NAME_SUFFIX:-}/mesh/node-b.yaml:/etc/fips/fips.yaml:ro
networks:
fips-net:
node-c:
<<: *fips-common
profiles: ["mesh"]
container_name: fips-node-c${FIPS_CI_NAME_SUFFIX:-}
hostname: node-c
volumes:
- ../docker/resolv.conf:/etc/resolv.conf:ro
- ./generated-configs${FIPS_CI_NAME_SUFFIX:-}/mesh/node-c.yaml:/etc/fips/fips.yaml:ro
networks:
fips-net:
node-d:
<<: *fips-common
profiles: ["mesh"]
container_name: fips-node-d${FIPS_CI_NAME_SUFFIX:-}
hostname: node-d
volumes:
- ../docker/resolv.conf:/etc/resolv.conf:ro
- ./generated-configs${FIPS_CI_NAME_SUFFIX:-}/mesh/node-d.yaml:/etc/fips/fips.yaml:ro
networks:
fips-net:
node-e:
<<: *fips-common
profiles: ["mesh"]
container_name: fips-node-e${FIPS_CI_NAME_SUFFIX:-}
hostname: node-e
volumes:
- ../docker/resolv.conf:/etc/resolv.conf:ro
- ./generated-configs${FIPS_CI_NAME_SUFFIX:-}/mesh/node-e.yaml:/etc/fips/fips.yaml:ro
networks:
fips-net:
# ── Mesh-public topology (mesh + external public node) ────────
pub-a:
<<: *fips-common
profiles: ["mesh-public"]
container_name: fips-node-a${FIPS_CI_NAME_SUFFIX:-}
hostname: node-a
volumes:
- ../docker/resolv.conf:/etc/resolv.conf:ro
- ./generated-configs${FIPS_CI_NAME_SUFFIX:-}/mesh-public/node-a.yaml:/etc/fips/fips.yaml:ro
networks:
fips-net:
pub-b:
<<: *fips-common
profiles: ["mesh-public"]
container_name: fips-node-b${FIPS_CI_NAME_SUFFIX:-}
hostname: node-b
volumes:
- ../docker/resolv.conf:/etc/resolv.conf:ro
- ./generated-configs${FIPS_CI_NAME_SUFFIX:-}/mesh-public/node-b.yaml:/etc/fips/fips.yaml:ro
networks:
fips-net:
pub-c:
<<: *fips-common
profiles: ["mesh-public"]
container_name: fips-node-c${FIPS_CI_NAME_SUFFIX:-}
hostname: node-c
volumes:
- ../docker/resolv.conf:/etc/resolv.conf:ro
- ./generated-configs${FIPS_CI_NAME_SUFFIX:-}/mesh-public/node-c.yaml:/etc/fips/fips.yaml:ro
networks:
fips-net:
pub-d:
<<: *fips-common
profiles: ["mesh-public"]
container_name: fips-node-d${FIPS_CI_NAME_SUFFIX:-}
hostname: node-d
volumes:
- ../docker/resolv.conf:/etc/resolv.conf:ro
- ./generated-configs${FIPS_CI_NAME_SUFFIX:-}/mesh-public/node-d.yaml:/etc/fips/fips.yaml:ro
networks:
fips-net:
pub-e:
<<: *fips-common
profiles: ["mesh-public"]
container_name: fips-node-e${FIPS_CI_NAME_SUFFIX:-}
hostname: node-e
volumes:
- ../docker/resolv.conf:/etc/resolv.conf:ro
- ./generated-configs${FIPS_CI_NAME_SUFFIX:-}/mesh-public/node-e.yaml:/etc/fips/fips.yaml:ro
networks:
fips-net:
# ── Chain topology (A-B-C-D-E) ────────────────────────────────
chain-a:
<<: *fips-common
profiles: ["chain"]
container_name: fips-node-a${FIPS_CI_NAME_SUFFIX:-}
hostname: node-a
volumes:
- ../docker/resolv.conf:/etc/resolv.conf:ro
- ./generated-configs${FIPS_CI_NAME_SUFFIX:-}/chain/node-a.yaml:/etc/fips/fips.yaml:ro
networks:
fips-net:
chain-b:
<<: *fips-common
profiles: ["chain"]
container_name: fips-node-b${FIPS_CI_NAME_SUFFIX:-}
hostname: node-b
volumes:
- ../docker/resolv.conf:/etc/resolv.conf:ro
- ./generated-configs${FIPS_CI_NAME_SUFFIX:-}/chain/node-b.yaml:/etc/fips/fips.yaml:ro
networks:
fips-net:
chain-c:
<<: *fips-common
profiles: ["chain"]
container_name: fips-node-c${FIPS_CI_NAME_SUFFIX:-}
hostname: node-c
volumes:
- ../docker/resolv.conf:/etc/resolv.conf:ro
- ./generated-configs${FIPS_CI_NAME_SUFFIX:-}/chain/node-c.yaml:/etc/fips/fips.yaml:ro
networks:
fips-net:
chain-d:
<<: *fips-common
profiles: ["chain"]
container_name: fips-node-d${FIPS_CI_NAME_SUFFIX:-}
hostname: node-d
volumes:
- ../docker/resolv.conf:/etc/resolv.conf:ro
- ./generated-configs${FIPS_CI_NAME_SUFFIX:-}/chain/node-d.yaml:/etc/fips/fips.yaml:ro
networks:
fips-net:
chain-e:
<<: *fips-common
profiles: ["chain"]
container_name: fips-node-e${FIPS_CI_NAME_SUFFIX:-}
hostname: node-e
volumes:
- ../docker/resolv.conf:/etc/resolv.conf:ro
- ./generated-configs${FIPS_CI_NAME_SUFFIX:-}/chain/node-e.yaml:/etc/fips/fips.yaml:ro
networks:
fips-net:
# ── Rekey integration test (mesh + aggressive rekey timers) ──
rekey-a:
<<: *fips-common
profiles: ["rekey"]
container_name: fips-node-a${FIPS_CI_NAME_SUFFIX:-}
hostname: node-a
environment:
- RUST_LOG=info,fips::node::handlers::rekey=debug
volumes:
- ../docker/resolv.conf:/etc/resolv.conf:ro
- ./generated-configs${FIPS_CI_NAME_SUFFIX:-}/rekey/node-a.yaml:/etc/fips/fips.yaml:ro
networks:
fips-net:
rekey-b:
<<: *fips-common
profiles: ["rekey"]
container_name: fips-node-b${FIPS_CI_NAME_SUFFIX:-}
hostname: node-b
environment:
- RUST_LOG=info,fips::node::handlers::rekey=debug
volumes:
- ../docker/resolv.conf:/etc/resolv.conf:ro
- ./generated-configs${FIPS_CI_NAME_SUFFIX:-}/rekey/node-b.yaml:/etc/fips/fips.yaml:ro
networks:
fips-net:
rekey-c:
<<: *fips-common
profiles: ["rekey"]
container_name: fips-node-c${FIPS_CI_NAME_SUFFIX:-}
hostname: node-c
environment:
- RUST_LOG=info,fips::node::handlers::rekey=debug
volumes:
- ../docker/resolv.conf:/etc/resolv.conf:ro
- ./generated-configs${FIPS_CI_NAME_SUFFIX:-}/rekey/node-c.yaml:/etc/fips/fips.yaml:ro
networks:
fips-net:
rekey-d:
<<: *fips-common
profiles: ["rekey"]
container_name: fips-node-d${FIPS_CI_NAME_SUFFIX:-}
hostname: node-d
environment:
- RUST_LOG=info,fips::node::handlers::rekey=debug
volumes:
- ../docker/resolv.conf:/etc/resolv.conf:ro
- ./generated-configs${FIPS_CI_NAME_SUFFIX:-}/rekey/node-d.yaml:/etc/fips/fips.yaml:ro
networks:
fips-net:
rekey-e:
<<: *fips-common
profiles: ["rekey"]
container_name: fips-node-e${FIPS_CI_NAME_SUFFIX:-}
hostname: node-e
environment:
- RUST_LOG=info,fips::node::handlers::rekey=debug
volumes:
- ../docker/resolv.conf:/etc/resolv.conf:ro
- ./generated-configs${FIPS_CI_NAME_SUFFIX:-}/rekey/node-e.yaml:/etc/fips/fips.yaml:ro
networks:
fips-net:
# ── Rekey + accept_connections=false on node b ──────────────────
# Exercises the auto_connect-initiator-with-accept-off regression
# class. Same 5-node mesh, but node b's generated config has
# `transports.udp.accept_connections: false` (injected by
# rekey-test.sh's inject-config when REKEY_ACCEPT_OFF_NODES=b).
rekey-accept-off-a:
<<: *fips-common
profiles: ["rekey-accept-off"]
container_name: fips-node-a${FIPS_CI_NAME_SUFFIX:-}
hostname: node-a
environment:
- RUST_LOG=info,fips::node::handlers::rekey=debug,fips::node::handlers::handshake=debug
volumes:
- ../docker/resolv.conf:/etc/resolv.conf:ro
- ./generated-configs${FIPS_CI_NAME_SUFFIX:-}/rekey-accept-off/node-a.yaml:/etc/fips/fips.yaml:ro
networks:
fips-net:
rekey-accept-off-b:
<<: *fips-common
profiles: ["rekey-accept-off"]
container_name: fips-node-b${FIPS_CI_NAME_SUFFIX:-}
hostname: node-b
environment:
- RUST_LOG=info,fips::node::handlers::rekey=debug,fips::node::handlers::handshake=debug
volumes:
- ../docker/resolv.conf:/etc/resolv.conf:ro
- ./generated-configs${FIPS_CI_NAME_SUFFIX:-}/rekey-accept-off/node-b.yaml:/etc/fips/fips.yaml:ro
networks:
fips-net:
rekey-accept-off-c:
<<: *fips-common
profiles: ["rekey-accept-off"]
container_name: fips-node-c${FIPS_CI_NAME_SUFFIX:-}
hostname: node-c
environment:
- RUST_LOG=info,fips::node::handlers::rekey=debug,fips::node::handlers::handshake=debug
volumes:
- ../docker/resolv.conf:/etc/resolv.conf:ro
- ./generated-configs${FIPS_CI_NAME_SUFFIX:-}/rekey-accept-off/node-c.yaml:/etc/fips/fips.yaml:ro
networks:
fips-net:
rekey-accept-off-d:
<<: *fips-common
profiles: ["rekey-accept-off"]
container_name: fips-node-d${FIPS_CI_NAME_SUFFIX:-}
hostname: node-d
environment:
- RUST_LOG=info,fips::node::handlers::rekey=debug,fips::node::handlers::handshake=debug
volumes:
- ../docker/resolv.conf:/etc/resolv.conf:ro
- ./generated-configs${FIPS_CI_NAME_SUFFIX:-}/rekey-accept-off/node-d.yaml:/etc/fips/fips.yaml:ro
networks:
fips-net:
rekey-accept-off-e:
<<: *fips-common
profiles: ["rekey-accept-off"]
container_name: fips-node-e${FIPS_CI_NAME_SUFFIX:-}
hostname: node-e
environment:
- RUST_LOG=info,fips::node::handlers::rekey=debug,fips::node::handlers::handshake=debug
volumes:
- ../docker/resolv.conf:/etc/resolv.conf:ro
- ./generated-configs${FIPS_CI_NAME_SUFFIX:-}/rekey-accept-off/node-e.yaml:/etc/fips/fips.yaml:ro
networks:
fips-net:
# ── Rekey topology with udp.outbound_only=true on node b ────
# Exercises the udp.outbound_only rekey-msg1 admission regression
# observed in production 2026-04-30. Same 5-node mesh as
# rekey-accept-off, but node-b's generated config has
# `transports.udp.outbound_only: true` (forces ephemeral bind +
# accept_connections false) AND node-b's peer-c address is
# rewritten to the docker hostname (`node-c:2121`) so the
# `addr_to_link` lookup misses on the inbound numeric source addr.
# Injected by rekey-test.sh's inject-config when
# REKEY_OUTBOUND_ONLY_NODES=b.
rekey-outbound-only-a:
<<: *fips-common
profiles: ["rekey-outbound-only"]
container_name: fips-node-a${FIPS_CI_NAME_SUFFIX:-}
hostname: node-a
environment:
- RUST_LOG=info,fips::node::handlers::rekey=debug,fips::node::handlers::handshake=debug
volumes:
- ../docker/resolv.conf:/etc/resolv.conf:ro
- ./generated-configs${FIPS_CI_NAME_SUFFIX:-}/rekey-outbound-only/node-a.yaml:/etc/fips/fips.yaml:ro
networks:
fips-net:
rekey-outbound-only-b:
<<: *fips-common
profiles: ["rekey-outbound-only"]
container_name: fips-node-b${FIPS_CI_NAME_SUFFIX:-}
hostname: node-b
environment:
- RUST_LOG=info,fips::node::handlers::rekey=debug,fips::node::handlers::handshake=debug
volumes:
- ../docker/resolv.conf:/etc/resolv.conf:ro
- ./generated-configs${FIPS_CI_NAME_SUFFIX:-}/rekey-outbound-only/node-b.yaml:/etc/fips/fips.yaml:ro
networks:
fips-net:
rekey-outbound-only-c:
<<: *fips-common
profiles: ["rekey-outbound-only"]
container_name: fips-node-c${FIPS_CI_NAME_SUFFIX:-}
hostname: node-c
environment:
- RUST_LOG=info,fips::node::handlers::rekey=debug,fips::node::handlers::handshake=debug
volumes:
- ../docker/resolv.conf:/etc/resolv.conf:ro
- ./generated-configs${FIPS_CI_NAME_SUFFIX:-}/rekey-outbound-only/node-c.yaml:/etc/fips/fips.yaml:ro
networks:
fips-net:
rekey-outbound-only-d:
<<: *fips-common
profiles: ["rekey-outbound-only"]
container_name: fips-node-d${FIPS_CI_NAME_SUFFIX:-}
hostname: node-d
environment:
- RUST_LOG=info,fips::node::handlers::rekey=debug,fips::node::handlers::handshake=debug
volumes:
- ../docker/resolv.conf:/etc/resolv.conf:ro
- ./generated-configs${FIPS_CI_NAME_SUFFIX:-}/rekey-outbound-only/node-d.yaml:/etc/fips/fips.yaml:ro
networks:
fips-net:
rekey-outbound-only-e:
<<: *fips-common
profiles: ["rekey-outbound-only"]
container_name: fips-node-e${FIPS_CI_NAME_SUFFIX:-}
hostname: node-e
environment:
- RUST_LOG=info,fips::node::handlers::rekey=debug,fips::node::handlers::handshake=debug
volumes:
- ../docker/resolv.conf:/etc/resolv.conf:ro
- ./generated-configs${FIPS_CI_NAME_SUFFIX:-}/rekey-outbound-only/node-e.yaml:/etc/fips/fips.yaml:ro
networks:
fips-net:
# ── TCP chain topology (A-B-C) ───────────────────────────────
tcp-a:
<<: *fips-common
profiles: ["tcp-chain"]
container_name: fips-node-a${FIPS_CI_NAME_SUFFIX:-}
hostname: node-a
volumes:
- ../docker/resolv.conf:/etc/resolv.conf:ro
- ./generated-configs${FIPS_CI_NAME_SUFFIX:-}/tcp-chain/node-a.yaml:/etc/fips/fips.yaml:ro
networks:
fips-net:
tcp-b:
<<: *fips-common
profiles: ["tcp-chain"]
container_name: fips-node-b${FIPS_CI_NAME_SUFFIX:-}
hostname: node-b
volumes:
- ../docker/resolv.conf:/etc/resolv.conf:ro
- ./generated-configs${FIPS_CI_NAME_SUFFIX:-}/tcp-chain/node-b.yaml:/etc/fips/fips.yaml:ro
networks:
fips-net:
tcp-c:
<<: *fips-common
profiles: ["tcp-chain"]
container_name: fips-node-c${FIPS_CI_NAME_SUFFIX:-}
hostname: node-c
volumes:
- ../docker/resolv.conf:/etc/resolv.conf:ro
- ./generated-configs${FIPS_CI_NAME_SUFFIX:-}/tcp-chain/node-c.yaml:/etc/fips/fips.yaml:ro
networks:
fips-net:
# ── Gateway integration test (gateway + server + non-FIPS client) ─
gw-gateway:
<<: *fips-common
profiles: ["gateway"]
container_name: fips-gw-gateway${FIPS_CI_NAME_SUFFIX:-}
hostname: gw-gateway
# Privileged required: gateway must enable IPv6 on eth1 (second network,
# attached after container start) and manage nftables NAT rules.
privileged: true
environment:
- RUST_LOG=info
- FIPS_TEST_MODE=gateway
sysctls:
- net.ipv6.conf.all.disable_ipv6=0
- net.ipv6.conf.default.disable_ipv6=0
- net.ipv6.conf.all.forwarding=1
- net.ipv6.conf.all.proxy_ndp=1
volumes:
- ../docker/resolv.conf:/etc/resolv.conf:ro
- ./generated-configs${FIPS_CI_NAME_SUFFIX:-}/gateway/node-a.yaml:/etc/fips/fips.yaml:ro
networks:
fips-net:
gateway-lan:
ipv4_address: 172.20.1.10
ipv6_address: fd02::10
gw-server:
<<: *fips-common
profiles: ["gateway"]
container_name: fips-gw-server${FIPS_CI_NAME_SUFFIX:-}
hostname: gw-server
volumes:
- ../docker/resolv.conf:/etc/resolv.conf:ro
- ./generated-configs${FIPS_CI_NAME_SUFFIX:-}/gateway/node-b.yaml:/etc/fips/fips.yaml:ro
networks:
fips-net:
# Second mesh destination — gives gw-client-2 a distinct npub to target
# so the gateway allocates a separate virtual-IP mapping per LAN client.
# Mirrors gw-server; not on gateway-lan.
gw-server-2:
<<: *fips-common
profiles: ["gateway"]
container_name: fips-gw-server-2${FIPS_CI_NAME_SUFFIX:-}
hostname: gw-server-2
volumes:
- ../docker/resolv.conf:/etc/resolv.conf:ro
- ./generated-configs${FIPS_CI_NAME_SUFFIX:-}/gateway/node-c.yaml:/etc/fips/fips.yaml:ro
networks:
fips-net:
gw-client:
image: ${FIPS_TEST_APP_IMAGE:-fips-test-app:latest}
profiles: ["gateway"]
container_name: fips-gw-client${FIPS_CI_NAME_SUFFIX:-}
hostname: gw-client
cap_add:
- NET_ADMIN
sysctls:
- net.ipv6.conf.all.disable_ipv6=0
volumes:
- ./configs/gateway-resolv.conf:/etc/resolv.conf:ro
networks:
gateway-lan:
ipv4_address: 172.20.1.20
ipv6_address: fd02::20
restart: "no"
env_file:
- ./generated-configs${FIPS_CI_NAME_SUFFIX:-}/npubs.env
# Second LAN client — exercises concurrent multi-client mappings.
# Same image and gateway-lan attachment as
# gw-client; the gateway must allocate a distinct virtual IP for it.
gw-client-2:
image: ${FIPS_TEST_APP_IMAGE:-fips-test-app:latest}
profiles: ["gateway"]
container_name: fips-gw-client-2${FIPS_CI_NAME_SUFFIX:-}
hostname: gw-client-2
cap_add:
- NET_ADMIN
sysctls:
- net.ipv6.conf.all.disable_ipv6=0
volumes:
- ./configs/gateway-resolv.conf:/etc/resolv.conf:ro
networks:
gateway-lan:
ipv4_address: 172.20.1.21
ipv6_address: fd02::21
restart: "no"
env_file:
- ./generated-configs${FIPS_CI_NAME_SUFFIX:-}/npubs.env