Files
fips/testing/nat/scripts/setup-topology.sh
T
Johnathan Corgan e4a854f6b0 Stop concurrent local CI runs from clobbering each other's containers
Two runs on one host destroyed each other's containers, producing
mid-test "No such container" failures that look like real defects. The
automated builder runs a full local CI on the same box every few minutes,
so the machine is contended almost always and this has red-ed both a hand
run and an automated gate.

Two independent causes are fixed here. Container names were hardcoded, and
docker names are global rather than scoped by compose project, so two runs
collided on the same name; every name now takes an optional suffix that
the harness sets from the run id. And the cleanup sweep matched a label
shared by every run, so one run's teardown force-removed another's
containers; resources now also carry a per-run label and the sweep can be
narrowed to it.

A third hazard turned up that was not in the original report: the sidecar
suite passes explicit compose project names, which override the run-scoped
project and put it outside the shared prefix entirely. Its project names,
network, and derived container references are now scoped too.

Both are default-off. With the suffix unset, names render exactly as they
do today and a bare compose invocation is unchanged, which is what keeps
the hosted CI and the documentation correct. A cleanup run with no run id
still reaps everything, which is what a manual "clear the box" wants.

The literal-name sweep was not sufficient: six scripts build container
names dynamically from node labels, and two suites create their own
containers outside compose. Those are handled at their construction sites.

Verified: syntax check on all modified scripts; compose validation on
every modified file with the suffix both set and unset; and a synthetic
two-run reproduction that shows the old cleanup destroying a bystander run
and the new one leaving it alone.

Known gap: subnets are still hardcoded, so two concurrent full runs will
still collide on address-pool overlap. That fix reaches into topology
configs, chaos scenarios, diagrams and production source, so it is left
for its own change rather than half-done here.
2026-07-19 06:41:25 +00:00

133 lines
3.4 KiB
Bash
Executable File

#!/bin/bash
set -euo pipefail
SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
NAT_DIR="$(cd "$SCRIPT_DIR/.." && pwd)"
SCENARIO="${1:?usage: setup-topology.sh <cone|symmetric>}"
case "$SCENARIO" in
cone)
node_a="fips-nat-cone-a${FIPS_CI_NAME_SUFFIX:-}"
node_b="fips-nat-cone-b${FIPS_CI_NAME_SUFFIX:-}"
;;
symmetric)
node_a="fips-nat-symmetric-a${FIPS_CI_NAME_SUFFIX:-}"
node_b="fips-nat-symmetric-b${FIPS_CI_NAME_SUFFIX:-}"
;;
*)
echo "Unsupported topology scenario: $SCENARIO" >&2
exit 1
;;
esac
router_a="fips-nat-router-a${FIPS_CI_NAME_SUFFIX:-}"
router_b="fips-nat-router-b${FIPS_CI_NAME_SUFFIX:-}"
helper_image() {
if [ -n "${IP_HELPER_IMAGE:-}" ]; then
echo "$IP_HELPER_IMAGE"
return 0
fi
docker inspect -f '{{.Config.Image}}' "$router_a"
}
wait_for_pid() {
local container="$1"
local timeout_secs="${2:-30}"
local deadline=$((SECONDS + timeout_secs))
local pid=""
while [ "$SECONDS" -lt "$deadline" ]; do
pid="$(docker inspect -f '{{.State.Pid}}' "$container" 2>/dev/null || true)"
if [[ "$pid" =~ ^[0-9]+$ ]] && [ "$pid" -gt 0 ]; then
echo "$pid"
return 0
fi
sleep 0.5
done
echo "Timed out waiting for container PID: $container" >&2
return 1
}
run_host_ip() {
local image="$1"
shift
docker run --rm \
--label com.corganlabs.fips-ci=1 \
--privileged \
--net=host \
--pid=host \
--entrypoint ip \
"$image" \
"$@"
}
configure_node_iface() {
local container="$1"
local current_name="$2"
local final_name="$3"
local cidr="$4"
docker exec "$container" sh -lc "
ip link set lo up &&
ip link set '$current_name' name '$final_name' &&
ip addr flush dev '$final_name' &&
ip addr add '$cidr' dev '$final_name' &&
ip link set '$final_name' up
"
}
configure_router_iface() {
local container="$1"
local current_name="$2"
local final_name="$3"
local cidr="$4"
docker exec "$container" sh -lc "
ip link set lo up &&
ip link set '$current_name' name '$final_name' &&
ip addr flush dev '$final_name' &&
ip addr add '$cidr' dev '$final_name' &&
ip link set '$final_name' up
"
}
setup_pair() {
local image="$1"
local node_container="$2"
local router_container="$3"
local host_node="$4"
local host_router="$5"
local node_cidr="$6"
local router_cidr="$7"
local node_pid router_pid
node_pid="$(wait_for_pid "$node_container")"
router_pid="$(wait_for_pid "$router_container")"
run_host_ip "$image" link delete "$host_node" >/dev/null 2>&1 || true
run_host_ip "$image" link delete "$host_router" >/dev/null 2>&1 || true
run_host_ip "$image" link add "$host_node" type veth peer name "$host_router"
run_host_ip "$image" link set "$host_node" netns "$node_pid"
run_host_ip "$image" link set "$host_router" netns "$router_pid"
configure_node_iface "$node_container" "$host_node" eth0 "$node_cidr"
configure_router_iface "$router_container" "$host_router" eth1 "$router_cidr"
}
main() {
cd "$NAT_DIR"
local image
image="$(helper_image)"
setup_pair "$image" "$node_a" "$router_a" vna0 vna1 172.31.1.10/24 172.31.1.254/24
setup_pair "$image" "$node_b" "$router_b" vnb0 vnb1 172.31.2.10/24 172.31.2.254/24
}
main "$@"