From 6c52b0e01e7ed38d87b692d41e825d2830f2ffc5 Mon Sep 17 00:00:00 2001 From: Johnathan Corgan Date: Thu, 23 Jul 2026 02:05:56 +0000 Subject: [PATCH] Let the static test network float so concurrent CI runs cannot collide MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Two local CI runs on one host both asked docker for 172.20.0.0/24 and the second lost its whole static family to "Pool overlaps". Docker honours a fixed subnet request verbatim, so the only robust fix is to stop making one: fips-net now requests no subnet and docker assigns from its own pool, which cannot hand the same range to two runs. That means node addresses are not known before `up`, so peers address each other by container hostname instead. The generator emits node-, or the topology's docker_host where the compose hostname differs — only the gateway profile, whose services are gw-*. External peers keep the address the topology gives them, since it is not ours to assign. The resolv.conf mount stays: dnsmasq is what forwards these names to docker's resolver and .fips to the daemon, so removing it would take out every .fips assertion. generated-configs is now per-run as well. A shared directory let two runs overwrite each other's node configs, which the subnet collision had been hiding by killing runs before that window opened. The generator, the compose bind mounts and env_file, the six scripts that read it, and teardown all follow FIPS_CI_NAME_SUFFIX; unset, every path renders as before. Teardown keeps the directory after a failed run, where it is the evidence of what the failing nodes were configured with. Three things this exposed that were wrong independently: admission-cap built its tcpdump patterns from the topology file's docker_ip literals. Floating the subnet makes those match nothing, which would have left its expect-zero "no Msg2 leaked" assertion passing because it could no longer see anything at all. It now reads addresses from the running containers. Restarting the denied peers together also made them swap addresses, so each peer's counts were really the pair's total; they are restarted one at a time now, and a check fails the suite outright if two denied peers ever share an address, because per-peer attribution is impossible once they do. Attribute lookups in the generator used a fixed ten-line window and read the next node's fields when a node omitted an attribute. An external node followed by an internal one was classified as internal, which under hostname peering would emit a name that resolves nowhere. Lookups are bounded to the node's own block; generated output is byte-identical for all eight topologies. The rekey outbound-only variant used to rewrite peer addresses to hostnames to set up its scenario. The generator now does that everywhere, so the rewrite matched nothing and was silently doing no work. It asserts the premise instead, and fails if a numeric address ever reappears. Verified by running three instances of this compose at once — tcp-chain plus two independent meshes — which drew 10.128.2/3/4.0/24 with no overlap while both meshes passed ping-test 20/20 over the real .fips path. tcp-chain is run by neither CI runner, so it was checked by hand: chain peer counts 1/2/1 and multi-hop .fips reachable both directions over TCP. gateway-lan still pins its own IPv4 and fd02:: ranges and is unchanged here, so the gateway profile is not yet concurrency-safe. --- testing/ci-local.sh | 16 ++- testing/static/.gitignore | 2 +- testing/static/README.md | 17 ++- .../static/configs/topologies/gateway.yaml | 7 + testing/static/docker-chain-topology.svg | 2 +- testing/static/docker-compose.yml | 122 +++++++----------- testing/static/docker-mesh-topology.svg | 2 +- testing/static/scripts/admission-cap-test.sh | 97 ++++++++++++-- testing/static/scripts/bench-multirun.sh | 2 +- testing/static/scripts/gateway-test.sh | 2 +- testing/static/scripts/generate-configs.sh | 57 +++++++- testing/static/scripts/iperf-test.sh | 2 +- testing/static/scripts/ping-test.sh | 2 +- testing/static/scripts/rekey-test.sh | 57 ++++---- 14 files changed, 253 insertions(+), 134 deletions(-) diff --git a/testing/ci-local.sh b/testing/ci-local.sh index c8a4efc..5410a34 100755 --- a/testing/ci-local.sh +++ b/testing/ci-local.sh @@ -302,9 +302,12 @@ CI_CLEANED=0 # Best-effort, BOUNDED teardown of every docker resource THIS run may have # created. Idempotent (guarded), so the signal and EXIT paths don't double-run. +# Takes the run's exit status; defaults to non-zero, which is the conservative +# reading for the signal path. ci_teardown() { [[ $CI_CLEANED -eq 1 ]] && return 0 CI_CLEANED=1 + local run_status="${1:-1}" # 1. Propagate to parallel chaos children and reap them (bounded). if [[ ${#CI_CHAOS_PIDS[@]} -gt 0 ]]; then @@ -339,6 +342,17 @@ ci_teardown() { --project-prefix "$CI_PROJECT_PREFIX" \ --images "$CI_IMAGE_TEST $CI_IMAGE_APP" \ --veth-suffixes "${_suffixes[*]}" >/dev/null || true + + # 3. The static suite's generated configs are per-run (a shared directory + # would let concurrent runs overwrite each other's node configs), so + # they are this run's to remove. Only on a green run: after a failure + # they are the evidence of what the failing nodes were actually + # configured with. Guarded on a non-empty suffix too, since without one + # the path is the unscoped working directory a developer uses by hand, + # which is not ours to delete. + if [[ $run_status -eq 0 && -n "${CI_RUN_NAME_SUFFIX:-}" ]]; then + rm -rf "$SCRIPT_DIR/static/generated-configs${CI_RUN_NAME_SUFFIX}" + fi } on_signal() { @@ -355,7 +369,7 @@ on_signal() { on_exit() { local code=$? trap '' TERM INT EXIT - ci_teardown + ci_teardown "$code" exit "$code" } diff --git a/testing/static/.gitignore b/testing/static/.gitignore index 6f7fff8..c741cc3 100644 --- a/testing/static/.gitignore +++ b/testing/static/.gitignore @@ -1,4 +1,4 @@ fips fipsctl fipstop -generated-configs +generated-configs* diff --git a/testing/static/README.md b/testing/static/README.md index d922035..bde3f05 100644 --- a/testing/static/README.md +++ b/testing/static/README.md @@ -122,7 +122,7 @@ testing/static/ │ ├── mesh-public.yaml # Mesh + external public node │ ├── tcp-chain.yaml # TCP chain (3 nodes, port 8443) │ └── rekey.yaml # Rekey integration test (5 nodes) -├── generated-configs/ # Auto-generated (gitignored) +├── generated-configs/ # Auto-generated, run-scoped (gitignored) │ ├── npubs.env # NPUB_A=..., NPUB_B=..., etc. │ ├── mesh/ │ │ ├── node-a.yaml ... node-e.yaml @@ -151,6 +151,16 @@ Each topology file in `configs/topologies/` defines: - **Addresses**: `docker_ip` for Docker-managed nodes, `external_ip` for remote nodes not managed by Docker - **Peer connections**: which nodes peer with each other +- **`docker_host`** (optional): the compose `hostname:` this node answers to, + when that is not `node-`. Only the gateway topology needs it + +Generated peer addresses use the **docker hostname**, not `docker_ip`. +`fips-net` requests no subnet, so docker assigns one from its own pool and two +concurrent CI runs can bring the topology up at the same time instead of one +of them failing with `Pool overlaps`. `docker_ip` is retained as documentation +of the topology's shape and as the internal/external discriminator; an +external node keeps its `external_ip` in peer blocks, its address not being +ours to assign. Example entry: @@ -178,6 +188,11 @@ This reads the topology definition and generates: 1. Per-node YAML config files in `generated-configs//` 2. `generated-configs/npubs.env` with all node npubs as environment variables +Under `ci-local.sh` the directory is `generated-configs-`, so +concurrent runs cannot overwrite each other's node configs; the compose file +and every test script read the same `FIPS_CI_NAME_SUFFIX` and follow it. A +bare invocation leaves the suffix unset and writes the plain path. + The `npubs.env` file is sourced by the test scripts and injected into Docker containers via `env_file` in `docker-compose.yml`. diff --git a/testing/static/configs/topologies/gateway.yaml b/testing/static/configs/topologies/gateway.yaml index b94e26b..2dda382 100644 --- a/testing/static/configs/topologies/gateway.yaml +++ b/testing/static/configs/topologies/gateway.yaml @@ -12,16 +12,23 @@ # A non-FIPS client container connects via the gateway's LAN interface. # # Uses deterministic key derivation (mesh-name: gateway-test). +# +# docker_host is the compose service's `hostname:`, which is what peers dial. +# This is the one static topology whose hostnames are not node-, so it is +# the one that has to declare them. nodes: a: docker_ip: "172.20.0.10" + docker_host: "gw-gateway" peers: [b, c] b: docker_ip: "172.20.0.11" + docker_host: "gw-server" peers: [a] c: docker_ip: "172.20.0.12" + docker_host: "gw-server-2" peers: [a] diff --git a/testing/static/docker-chain-topology.svg b/testing/static/docker-chain-topology.svg index 8c390ea..6e8ac62 100644 --- a/testing/static/docker-chain-topology.svg +++ b/testing/static/docker-chain-topology.svg @@ -33,7 +33,7 @@ - 172.20.0.0/24 (docker: fips-net) + auto-assigned subnet (docker: fips-net) - 172.20.0.0/24 (docker: fips-net) + auto-assigned subnet (docker: fips-net)