Stop concurrent local CI runs from clobbering each other's containers

Two runs on one host destroyed each other's containers, producing
mid-test "No such container" failures that look like real defects. The
automated builder runs a full local CI on the same box every few minutes,
so the machine is contended almost always and this has red-ed both a hand
run and an automated gate.

Two independent causes are fixed here. Container names were hardcoded, and
docker names are global rather than scoped by compose project, so two runs
collided on the same name; every name now takes an optional suffix that
the harness sets from the run id. And the cleanup sweep matched a label
shared by every run, so one run's teardown force-removed another's
containers; resources now also carry a per-run label and the sweep can be
narrowed to it.

A third hazard turned up that was not in the original report: the sidecar
suite passes explicit compose project names, which override the run-scoped
project and put it outside the shared prefix entirely. Its project names,
network, and derived container references are now scoped too.

Both are default-off. With the suffix unset, names render exactly as they
do today and a bare compose invocation is unchanged, which is what keeps
the hosted CI and the documentation correct. A cleanup run with no run id
still reaps everything, which is what a manual "clear the box" wants.

The literal-name sweep was not sufficient: six scripts build container
names dynamically from node labels, and two suites create their own
containers outside compose. Those are handled at their construction sites.

Verified: syntax check on all modified scripts; compose validation on
every modified file with the suffix both set and unset; and a synthetic
two-run reproduction that shows the old cleanup destroying a bystander run
and the new one leaving it alone.

Known gap: subnets are still hardcoded, so two concurrent full runs will
still collide on address-pool overlap. That fix reaches into topology
configs, chaos scenarios, diagrams and production source, so it is left
for its own change rather than half-done here.
This commit is contained in:
Johnathan Corgan
2026-07-19 06:41:25 +00:00
parent 7a97599921
commit e4a854f6b0
28 changed files with 287 additions and 251 deletions
+2 -2
View File
@@ -27,7 +27,7 @@ x-boringtun-common: &boringtun-common
services:
alice:
<<: *boringtun-common
container_name: bt-alice
container_name: bt-alice${FIPS_CI_NAME_SUFFIX:-}
hostname: alice
environment:
- ROLE=alice
@@ -38,7 +38,7 @@ services:
bob:
<<: *boringtun-common
container_name: bt-bob
container_name: bt-bob${FIPS_CI_NAME_SUFFIX:-}
hostname: bob
environment:
- ROLE=bob
+2 -2
View File
@@ -10,14 +10,14 @@ PARALLEL="${PARALLEL:-1}"
echo "=== boringtun iperf3 throughput (single TCP stream, ${DURATION}s) ==="
# Run iperf3 server on alice (background), client on bob.
docker exec -d bt-alice iperf3 -s -1 -B 10.99.0.1 -p 5201
docker exec -d bt-alice${FIPS_CI_NAME_SUFFIX:-} iperf3 -s -1 -B 10.99.0.1 -p 5201
sleep 1
# wait for tun handshake to settle (boringtun + WG keepalive)
sleep 2
# Client: bob → alice over WG (10.99.0.1)
OUT=$(docker exec bt-bob iperf3 -c 10.99.0.1 -p 5201 -t "$DURATION" -P "$PARALLEL" -J)
OUT=$(docker exec bt-bob${FIPS_CI_NAME_SUFFIX:-} iperf3 -c 10.99.0.1 -p 5201 -t "$DURATION" -P "$PARALLEL" -J)
# Pull SUM bps.
MBPS=$(echo "$OUT" | python3 -c "import json,sys; d=json.load(sys.stdin); print(f\"{d['end']['sum_received']['bits_per_second'] / 1_000_000:.2f}\")")