Stop concurrent local CI runs from clobbering each other's containers

Two runs on one host destroyed each other's containers, producing
mid-test "No such container" failures that look like real defects. The
automated builder runs a full local CI on the same box every few minutes,
so the machine is contended almost always and this has red-ed both a hand
run and an automated gate.

Two independent causes are fixed here. Container names were hardcoded, and
docker names are global rather than scoped by compose project, so two runs
collided on the same name; every name now takes an optional suffix that
the harness sets from the run id. And the cleanup sweep matched a label
shared by every run, so one run's teardown force-removed another's
containers; resources now also carry a per-run label and the sweep can be
narrowed to it.

A third hazard turned up that was not in the original report: the sidecar
suite passes explicit compose project names, which override the run-scoped
project and put it outside the shared prefix entirely. Its project names,
network, and derived container references are now scoped too.

Both are default-off. With the suffix unset, names render exactly as they
do today and a bare compose invocation is unchanged, which is what keeps
the hosted CI and the documentation correct. A cleanup run with no run id
still reaps everything, which is what a manual "clear the box" wants.

The literal-name sweep was not sufficient: six scripts build container
names dynamically from node labels, and two suites create their own
containers outside compose. Those are handled at their construction sites.

Verified: syntax check on all modified scripts; compose validation on
every modified file with the suffix both set and unset; and a synthetic
two-run reproduction that shows the old cleanup destroying a bystander run
and the new one leaving it alone.

Known gap: subnets are still hardcoded, so two concurrent full runs will
still collide on address-pool overlap. That fix reaches into topology
configs, chaos scenarios, diagrams and production source, so it is left
for its own change rather than half-done here.
This commit is contained in:
Johnathan Corgan
2026-07-19 06:41:25 +00:00
parent 7a97599921
commit e4a854f6b0
28 changed files with 287 additions and 251 deletions
+6 -6
View File
@@ -28,7 +28,7 @@ x-fips-common: &fips-common
services: services:
service-a: service-a:
<<: *fips-common <<: *fips-common
container_name: fips-acl-container-a container_name: fips-acl-container-a${FIPS_CI_NAME_SUFFIX:-}
hostname: host-a hostname: host-a
volumes: volumes:
- ../docker/resolv.conf:/etc/resolv.conf:ro - ../docker/resolv.conf:/etc/resolv.conf:ro
@@ -43,7 +43,7 @@ services:
service-b: service-b:
<<: *fips-common <<: *fips-common
container_name: fips-acl-container-b container_name: fips-acl-container-b${FIPS_CI_NAME_SUFFIX:-}
hostname: host-b hostname: host-b
volumes: volumes:
- ../docker/resolv.conf:/etc/resolv.conf:ro - ../docker/resolv.conf:/etc/resolv.conf:ro
@@ -58,7 +58,7 @@ services:
service-c: service-c:
<<: *fips-common <<: *fips-common
container_name: fips-acl-container-c container_name: fips-acl-container-c${FIPS_CI_NAME_SUFFIX:-}
hostname: host-c hostname: host-c
volumes: volumes:
- ../docker/resolv.conf:/etc/resolv.conf:ro - ../docker/resolv.conf:/etc/resolv.conf:ro
@@ -73,7 +73,7 @@ services:
service-d: service-d:
<<: *fips-common <<: *fips-common
container_name: fips-acl-container-d container_name: fips-acl-container-d${FIPS_CI_NAME_SUFFIX:-}
hostname: host-d hostname: host-d
volumes: volumes:
- ../docker/resolv.conf:/etc/resolv.conf:ro - ../docker/resolv.conf:/etc/resolv.conf:ro
@@ -88,7 +88,7 @@ services:
service-e: service-e:
<<: *fips-common <<: *fips-common
container_name: fips-acl-container-e container_name: fips-acl-container-e${FIPS_CI_NAME_SUFFIX:-}
hostname: host-e hostname: host-e
volumes: volumes:
- ../docker/resolv.conf:/etc/resolv.conf:ro - ../docker/resolv.conf:/etc/resolv.conf:ro
@@ -101,7 +101,7 @@ services:
service-f: service-f:
<<: *fips-common <<: *fips-common
container_name: fips-acl-container-f container_name: fips-acl-container-f${FIPS_CI_NAME_SUFFIX:-}
hostname: host-f hostname: host-f
volumes: volumes:
- ../docker/resolv.conf:/etc/resolv.conf:ro - ../docker/resolv.conf:/etc/resolv.conf:ro
+20 -20
View File
@@ -132,31 +132,31 @@ docker compose -f "$COMPOSE_FILE" down >/dev/null 2>&1 || true
docker compose -f "$COMPOSE_FILE" up -d --build docker compose -f "$COMPOSE_FILE" up -d --build
log "Waiting for expected peer convergence" log "Waiting for expected peer convergence"
wait_for_peers_exact fips-acl-container-a 3 40 wait_for_peers_exact fips-acl-container-a${FIPS_CI_NAME_SUFFIX:-} 3 40
wait_for_peers_exact fips-acl-container-b 1 40 wait_for_peers_exact fips-acl-container-b${FIPS_CI_NAME_SUFFIX:-} 1 40
wait_for_peers_exact fips-acl-container-c 0 5 wait_for_peers_exact fips-acl-container-c${FIPS_CI_NAME_SUFFIX:-} 0 5
wait_for_peers_exact fips-acl-container-d 0 5 wait_for_peers_exact fips-acl-container-d${FIPS_CI_NAME_SUFFIX:-} 0 5
wait_for_peers_exact fips-acl-container-e 1 40 wait_for_peers_exact fips-acl-container-e${FIPS_CI_NAME_SUFFIX:-} 1 40
wait_for_peers_exact fips-acl-container-f 1 40 wait_for_peers_exact fips-acl-container-f${FIPS_CI_NAME_SUFFIX:-} 1 40
log "Verifying peer sets" log "Verifying peer sets"
assert_peer_set fips-acl-container-a "npub1tdwa4vjrjl33pcjdpf2t4p027nl86xrx24g4d3avg4vwvayr3g8qhd84le npub1x5z9rwzzm26q9verutx4aajhf2zw2pyp34c6whhde2zduxqav40qgq36l6 npub1ytrut7gjncn2zfnhn56c0zgftf0w6p99gf6fu8j73hzw5603zglqc9av6c" assert_peer_set fips-acl-container-a${FIPS_CI_NAME_SUFFIX:-} "npub1tdwa4vjrjl33pcjdpf2t4p027nl86xrx24g4d3avg4vwvayr3g8qhd84le npub1x5z9rwzzm26q9verutx4aajhf2zw2pyp34c6whhde2zduxqav40qgq36l6 npub1ytrut7gjncn2zfnhn56c0zgftf0w6p99gf6fu8j73hzw5603zglqc9av6c"
assert_peer_set fips-acl-container-b "npub1sjlh2c3x9w7kjsqg2ay080n2lff2uvt325vpan33ke34rn8l5jcqawh57m" assert_peer_set fips-acl-container-b${FIPS_CI_NAME_SUFFIX:-} "npub1sjlh2c3x9w7kjsqg2ay080n2lff2uvt325vpan33ke34rn8l5jcqawh57m"
assert_peer_set fips-acl-container-c "" assert_peer_set fips-acl-container-c${FIPS_CI_NAME_SUFFIX:-} ""
assert_peer_set fips-acl-container-d "" assert_peer_set fips-acl-container-d${FIPS_CI_NAME_SUFFIX:-} ""
assert_peer_set fips-acl-container-e "npub1sjlh2c3x9w7kjsqg2ay080n2lff2uvt325vpan33ke34rn8l5jcqawh57m" assert_peer_set fips-acl-container-e${FIPS_CI_NAME_SUFFIX:-} "npub1sjlh2c3x9w7kjsqg2ay080n2lff2uvt325vpan33ke34rn8l5jcqawh57m"
assert_peer_set fips-acl-container-f "npub1sjlh2c3x9w7kjsqg2ay080n2lff2uvt325vpan33ke34rn8l5jcqawh57m" assert_peer_set fips-acl-container-f${FIPS_CI_NAME_SUFFIX:-} "npub1sjlh2c3x9w7kjsqg2ay080n2lff2uvt325vpan33ke34rn8l5jcqawh57m"
log "Checking alias-based ACL resolution" log "Checking alias-based ACL resolution"
assert_acl_field fips-acl-container-a allow_file_entries "node-a node-b node-e node-f" assert_acl_field fips-acl-container-a${FIPS_CI_NAME_SUFFIX:-} allow_file_entries "node-a node-b node-e node-f"
assert_acl_field fips-acl-container-a allow_entries "npub1sjlh2c3x9w7kjsqg2ay080n2lff2uvt325vpan33ke34rn8l5jcqawh57m npub1tdwa4vjrjl33pcjdpf2t4p027nl86xrx24g4d3avg4vwvayr3g8qhd84le npub1x5z9rwzzm26q9verutx4aajhf2zw2pyp34c6whhde2zduxqav40qgq36l6 npub1ytrut7gjncn2zfnhn56c0zgftf0w6p99gf6fu8j73hzw5603zglqc9av6c" assert_acl_field fips-acl-container-a${FIPS_CI_NAME_SUFFIX:-} allow_entries "npub1sjlh2c3x9w7kjsqg2ay080n2lff2uvt325vpan33ke34rn8l5jcqawh57m npub1tdwa4vjrjl33pcjdpf2t4p027nl86xrx24g4d3avg4vwvayr3g8qhd84le npub1x5z9rwzzm26q9verutx4aajhf2zw2pyp34c6whhde2zduxqav40qgq36l6 npub1ytrut7gjncn2zfnhn56c0zgftf0w6p99gf6fu8j73hzw5603zglqc9av6c"
assert_acl_field fips-acl-container-c allow_file_entries "node-a node-b node-c node-d node-e node-f" assert_acl_field fips-acl-container-c${FIPS_CI_NAME_SUFFIX:-} allow_file_entries "node-a node-b node-c node-d node-e node-f"
assert_acl_field fips-acl-container-c allow_entries "npub1cld9yay0u24davpu6c35l4vldrhzvaq66pcqtg9a0j2cnjrn9rtsxx2pe6 npub1n9lpnv0592cc2ps6nm0ca3qls642vx7yjsv35rkxqzj2vgds52sqgpverl npub1sjlh2c3x9w7kjsqg2ay080n2lff2uvt325vpan33ke34rn8l5jcqawh57m npub1tdwa4vjrjl33pcjdpf2t4p027nl86xrx24g4d3avg4vwvayr3g8qhd84le npub1x5z9rwzzm26q9verutx4aajhf2zw2pyp34c6whhde2zduxqav40qgq36l6 npub1ytrut7gjncn2zfnhn56c0zgftf0w6p99gf6fu8j73hzw5603zglqc9av6c" assert_acl_field fips-acl-container-c${FIPS_CI_NAME_SUFFIX:-} allow_entries "npub1cld9yay0u24davpu6c35l4vldrhzvaq66pcqtg9a0j2cnjrn9rtsxx2pe6 npub1n9lpnv0592cc2ps6nm0ca3qls642vx7yjsv35rkxqzj2vgds52sqgpverl npub1sjlh2c3x9w7kjsqg2ay080n2lff2uvt325vpan33ke34rn8l5jcqawh57m npub1tdwa4vjrjl33pcjdpf2t4p027nl86xrx24g4d3avg4vwvayr3g8qhd84le npub1x5z9rwzzm26q9verutx4aajhf2zw2pyp34c6whhde2zduxqav40qgq36l6 npub1ytrut7gjncn2zfnhn56c0zgftf0w6p99gf6fu8j73hzw5603zglqc9av6c"
log "Checking ACL rejection logs" log "Checking ACL rejection logs"
assert_log_contains fips-acl-container-a "npub1cld9yay0u24davpu6c35l4vldrhzvaq66pcqtg9a0j2cnjrn9rtsxx2pe6" assert_log_contains fips-acl-container-a${FIPS_CI_NAME_SUFFIX:-} "npub1cld9yay0u24davpu6c35l4vldrhzvaq66pcqtg9a0j2cnjrn9rtsxx2pe6"
assert_log_contains fips-acl-container-a "npub1n9lpnv0592cc2ps6nm0ca3qls642vx7yjsv35rkxqzj2vgds52sqgpverl" assert_log_contains fips-acl-container-a${FIPS_CI_NAME_SUFFIX:-} "npub1n9lpnv0592cc2ps6nm0ca3qls642vx7yjsv35rkxqzj2vgds52sqgpverl"
assert_log_contains fips-acl-container-a "context=inbound_handshake" assert_log_contains fips-acl-container-a${FIPS_CI_NAME_SUFFIX:-} "context=inbound_handshake"
assert_log_contains fips-acl-container-a "decision=denylist match" assert_log_contains fips-acl-container-a${FIPS_CI_NAME_SUFFIX:-} "decision=denylist match"
log "ACL allowlist integration test passed" log "ACL allowlist integration test passed"
+2 -2
View File
@@ -27,7 +27,7 @@ x-boringtun-common: &boringtun-common
services: services:
alice: alice:
<<: *boringtun-common <<: *boringtun-common
container_name: bt-alice container_name: bt-alice${FIPS_CI_NAME_SUFFIX:-}
hostname: alice hostname: alice
environment: environment:
- ROLE=alice - ROLE=alice
@@ -38,7 +38,7 @@ services:
bob: bob:
<<: *boringtun-common <<: *boringtun-common
container_name: bt-bob container_name: bt-bob${FIPS_CI_NAME_SUFFIX:-}
hostname: bob hostname: bob
environment: environment:
- ROLE=bob - ROLE=bob
+2 -2
View File
@@ -10,14 +10,14 @@ PARALLEL="${PARALLEL:-1}"
echo "=== boringtun iperf3 throughput (single TCP stream, ${DURATION}s) ===" echo "=== boringtun iperf3 throughput (single TCP stream, ${DURATION}s) ==="
# Run iperf3 server on alice (background), client on bob. # Run iperf3 server on alice (background), client on bob.
docker exec -d bt-alice iperf3 -s -1 -B 10.99.0.1 -p 5201 docker exec -d bt-alice${FIPS_CI_NAME_SUFFIX:-} iperf3 -s -1 -B 10.99.0.1 -p 5201
sleep 1 sleep 1
# wait for tun handshake to settle (boringtun + WG keepalive) # wait for tun handshake to settle (boringtun + WG keepalive)
sleep 2 sleep 2
# Client: bob → alice over WG (10.99.0.1) # Client: bob → alice over WG (10.99.0.1)
OUT=$(docker exec bt-bob iperf3 -c 10.99.0.1 -p 5201 -t "$DURATION" -P "$PARALLEL" -J) OUT=$(docker exec bt-bob${FIPS_CI_NAME_SUFFIX:-} iperf3 -c 10.99.0.1 -p 5201 -t "$DURATION" -P "$PARALLEL" -J)
# Pull SUM bps. # Pull SUM bps.
MBPS=$(echo "$OUT" | python3 -c "import json,sys; d=json.load(sys.stdin); print(f\"{d['end']['sum_received']['bits_per_second'] / 1_000_000:.2f}\")") MBPS=$(echo "$OUT" | python3 -c "import json,sys; d=json.load(sys.stdin); print(f\"{d['end']['sum_received']['bits_per_second'] / 1_000_000:.2f}\")")
+24 -6
View File
@@ -7,18 +7,25 @@
# prior run died: # prior run died:
# #
# 1. The CI label com.corganlabs.fips-ci=1 (attached to every direct # 1. The CI label com.corganlabs.fips-ci=1 (attached to every direct
# `docker run`/network/volume ci-local drives). # `docker run`/network/volume ci-local drives). Every run additionally
# stamps com.corganlabs.fips-ci.run=<run-id> on the same resources.
# 2. The compose project-name prefix fipsci_ (every compose project ci-local # 2. The compose project-name prefix fipsci_ (every compose project ci-local
# starts is named fipsci_<run-id>_<suite>, so its containers/networks/ # starts is named fipsci_<run-id>_<suite>, so its containers/networks/
# volumes all carry com.docker.compose.project=fipsci_... and are named # volumes all carry com.docker.compose.project=fipsci_... and are named
# with that prefix). # with that prefix).
# #
# The generic CI label is shared by every run on the host, so an unscoped label
# sweep would tear down a CONCURRENT run's resources. Pass --run-id to narrow
# the label sweep to one run; a run's own teardown always does. Without it the
# label sweep stays broad, which is what a manual "reap everything" wants.
#
# Usage: # Usage:
# ci-cleanup.sh Reap ALL fips-ci resources (any run) # ci-cleanup.sh Reap ALL fips-ci resources (any run)
# ci-cleanup.sh --project-prefix P Restrict the compose-project sweep to # ci-cleanup.sh --project-prefix P Restrict the compose-project sweep to
# names starting with P (scopes the reap # names starting with P (scopes the reap
# to a single run; the label sweep still # to a single run)
# runs) # ci-cleanup.sh --run-id ID Restrict the label sweep to the run
# labelled ID (leaves other runs alone)
# ci-cleanup.sh --label L Override the CI label (default above) # ci-cleanup.sh --label L Override the CI label (default above)
# ci-cleanup.sh --images "a b,c" Also `docker rmi -f` these image tags # ci-cleanup.sh --images "a b,c" Also `docker rmi -f` these image tags
# (space- or comma-separated) # (space- or comma-separated)
@@ -28,13 +35,16 @@
set -uo pipefail set -uo pipefail
LABEL="com.corganlabs.fips-ci=1" LABEL="com.corganlabs.fips-ci=1"
RUN_LABEL_KEY="com.corganlabs.fips-ci.run"
PROJECT_PREFIX="fipsci_" # broad default: every CI run PROJECT_PREFIX="fipsci_" # broad default: every CI run
RUN_ID="" # broad default: every CI run
IMAGES="" IMAGES=""
while [[ $# -gt 0 ]]; do while [[ $# -gt 0 ]]; do
case "$1" in case "$1" in
--label) LABEL="$2"; shift 2 ;; --label) LABEL="$2"; shift 2 ;;
--project-prefix) PROJECT_PREFIX="$2"; shift 2 ;; --project-prefix) PROJECT_PREFIX="$2"; shift 2 ;;
--run-id) RUN_ID="$2"; shift 2 ;;
--images) IMAGES="$2"; shift 2 ;; --images) IMAGES="$2"; shift 2 ;;
-h|--help) sed -n '2,/^set /{ /^set /d; s/^# \?//; p }' "$0"; exit 0 ;; -h|--help) sed -n '2,/^set /{ /^set /d; s/^# \?//; p }' "$0"; exit 0 ;;
*) echo "Unknown option: $1" >&2; exit 2 ;; *) echo "Unknown option: $1" >&2; exit 2 ;;
@@ -54,6 +64,14 @@ fi
# never wedge a caller (ci-local's signal trap relies on this being bounded). # never wedge a caller (ci-local's signal trap relies on this being bounded).
TMO=30 TMO=30
# Selector for the label sweep. With --run-id it matches only the named run, so
# a concurrent run's resources are left alone; without it, every CI run.
if [[ -n "$RUN_ID" ]]; then
SWEEP_LABEL="${RUN_LABEL_KEY}=${RUN_ID}"
else
SWEEP_LABEL="$LABEL"
fi
# Distinct compose project names (read off container labels) that start with # Distinct compose project names (read off container labels) that start with
# the configured prefix. # the configured prefix.
ci_projects() { ci_projects() {
@@ -63,7 +81,7 @@ ci_projects() {
reap_containers() { reap_containers() {
# By CI label. # By CI label.
docker ps -aq --filter "label=${LABEL}" 2>/dev/null \ docker ps -aq --filter "label=${SWEEP_LABEL}" 2>/dev/null \
| xargs -r timeout "$TMO" docker rm -f >/dev/null 2>&1 || true | xargs -r timeout "$TMO" docker rm -f >/dev/null 2>&1 || true
# By compose project (carried even when container_name is explicit). # By compose project (carried even when container_name is explicit).
local p local p
@@ -74,7 +92,7 @@ reap_containers() {
} }
reap_networks() { reap_networks() {
docker network ls -q --filter "label=${LABEL}" 2>/dev/null \ docker network ls -q --filter "label=${SWEEP_LABEL}" 2>/dev/null \
| xargs -r timeout "$TMO" docker network rm >/dev/null 2>&1 || true | xargs -r timeout "$TMO" docker network rm >/dev/null 2>&1 || true
# Compose networks are named <project>_<net> → match by name prefix so # Compose networks are named <project>_<net> → match by name prefix so
# orphaned networks (whose containers are already gone) are still caught. # orphaned networks (whose containers are already gone) are still caught.
@@ -83,7 +101,7 @@ reap_networks() {
} }
reap_volumes() { reap_volumes() {
docker volume ls -q --filter "label=${LABEL}" 2>/dev/null \ docker volume ls -q --filter "label=${SWEEP_LABEL}" 2>/dev/null \
| xargs -r timeout "$TMO" docker volume rm >/dev/null 2>&1 || true | xargs -r timeout "$TMO" docker volume rm >/dev/null 2>&1 || true
docker volume ls --format '{{.Name}}' 2>/dev/null | grep -E "^${PROJECT_PREFIX}" \ docker volume ls --format '{{.Name}}' 2>/dev/null | grep -E "^${PROJECT_PREFIX}" \
| xargs -r timeout "$TMO" docker volume rm >/dev/null 2>&1 || true | xargs -r timeout "$TMO" docker volume rm >/dev/null 2>&1 || true
+13 -2
View File
@@ -254,6 +254,10 @@ CI_PROJECT_PREFIX="fipsci_${CI_RUN_ID}"
CI_IMAGE_TEST="fips-test:${CI_RUN_ID}" CI_IMAGE_TEST="fips-test:${CI_RUN_ID}"
CI_IMAGE_APP="fips-test-app:${CI_RUN_ID}" CI_IMAGE_APP="fips-test-app:${CI_RUN_ID}"
CI_LABEL="com.corganlabs.fips-ci=1" CI_LABEL="com.corganlabs.fips-ci=1"
# Run-scoped companion to CI_LABEL. The generic label is shared by every run on
# the host, so teardown filters on this one instead — otherwise this run's reap
# would force-remove a concurrent run's containers out from under it.
CI_LABEL_RUN="com.corganlabs.fips-ci.run=${CI_RUN_ID}"
# Exported so child suite scripts + their compose/`docker run` invocations # Exported so child suite scripts + their compose/`docker run` invocations
# inherit the run identity. Compose files read FIPS_TEST_IMAGE/FIPS_TEST_APP_IMAGE # inherit the run identity. Compose files read FIPS_TEST_IMAGE/FIPS_TEST_APP_IMAGE
@@ -261,6 +265,12 @@ CI_LABEL="com.corganlabs.fips-ci=1"
export FIPS_CI_RUN_ID="$CI_RUN_ID" export FIPS_CI_RUN_ID="$CI_RUN_ID"
export FIPS_TEST_IMAGE="$CI_IMAGE_TEST" export FIPS_TEST_IMAGE="$CI_IMAGE_TEST"
export FIPS_TEST_APP_IMAGE="$CI_IMAGE_APP" export FIPS_TEST_APP_IMAGE="$CI_IMAGE_APP"
# Docker container names are GLOBAL — a compose project name does not scope
# them — so the suite compose files append this suffix to every explicit
# container_name, and the suite scripts append it wherever they address a
# container by name. Empty when unset, so a bare `docker compose up` outside
# this harness still produces today's plain names.
export FIPS_CI_NAME_SUFFIX="-${CI_RUN_ID}"
# Per-suite compose project name: ${prefix}_<suite-or-compose-basename>. Keeps # Per-suite compose project name: ${prefix}_<suite-or-compose-basename>. Keeps
# today's intra-run distinctness (one project per compose file / chaos child) # today's intra-run distinctness (one project per compose file / chaos child)
@@ -295,6 +305,7 @@ ci_teardown() {
# the whole sweep too so the trap can never wedge. # the whole sweep too so the trap can never wedge.
timeout 150 bash "$SCRIPT_DIR/ci-cleanup.sh" \ timeout 150 bash "$SCRIPT_DIR/ci-cleanup.sh" \
--label "$CI_LABEL" \ --label "$CI_LABEL" \
--run-id "$CI_RUN_ID" \
--project-prefix "$CI_PROJECT_PREFIX" \ --project-prefix "$CI_PROJECT_PREFIX" \
--images "$CI_IMAGE_TEST $CI_IMAGE_APP" >/dev/null 2>&1 || true --images "$CI_IMAGE_TEST $CI_IMAGE_APP" >/dev/null 2>&1 || true
} }
@@ -747,9 +758,9 @@ run_integration() {
# reference fips-test:latest directly; the retag happens only after BOTH # reference fips-test:latest directly; the retag happens only after BOTH
# builds succeed, so :latest never points at a half-built image. # builds succeed, so :latest never points at a half-built image.
info "Building $CI_IMAGE_TEST Docker image" info "Building $CI_IMAGE_TEST Docker image"
docker build -t "$CI_IMAGE_TEST" --label "$CI_LABEL" testing/docker --quiet \ docker build -t "$CI_IMAGE_TEST" --label "$CI_LABEL" --label "$CI_LABEL_RUN" testing/docker --quiet \
|| { record "docker-build" 1; return; } || { record "docker-build" 1; return; }
docker build -t "$CI_IMAGE_APP" --label "$CI_LABEL" \ docker build -t "$CI_IMAGE_APP" --label "$CI_LABEL" --label "$CI_LABEL_RUN" \
-f testing/docker/Dockerfile.app testing/docker --quiet \ -f testing/docker/Dockerfile.app testing/docker --quiet \
|| { record "docker-build-app" 1; return; } || { record "docker-build-app" 1; return; }
docker tag "$CI_IMAGE_TEST" fips-test:latest docker tag "$CI_IMAGE_TEST" fips-test:latest
+1 -1
View File
@@ -187,7 +187,7 @@ _run_deb_install_scenario() {
local distro_label="$1" local distro_label="$1"
local base_image="$2" local base_image="$2"
local name="fips-deb-test-${distro_label}" local name="fips-deb-test-${distro_label}${FIPS_CI_NAME_SUFFIX:-}"
local image="fips-deb-test:${distro_label}" local image="fips-deb-test:${distro_label}"
log ".deb install: ${base_image}" log ".deb install: ${base_image}"
+9 -9
View File
@@ -284,7 +284,7 @@ DOCKERFILE
# ───────────────────────────────────────────────────────────────────── # ─────────────────────────────────────────────────────────────────────
test_debian12_resolved() { test_debian12_resolved() {
local name="fips-dns-test-deb12-resolved" local name="fips-dns-test-deb12-resolved${FIPS_CI_NAME_SUFFIX:-}"
local image="fips-dns-test:debian12-resolved" local image="fips-dns-test:debian12-resolved"
log "Debian 12 + systemd-resolved (expects global-drop-in)" log "Debian 12 + systemd-resolved (expects global-drop-in)"
@@ -312,7 +312,7 @@ DOCKERFILE
} }
test_debian13_resolved() { test_debian13_resolved() {
local name="fips-dns-test-deb13-resolved" local name="fips-dns-test-deb13-resolved${FIPS_CI_NAME_SUFFIX:-}"
local image="fips-dns-test:debian13-resolved" local image="fips-dns-test:debian13-resolved"
log "Debian 13 (trixie) + systemd-resolved (expects global-drop-in)" log "Debian 13 (trixie) + systemd-resolved (expects global-drop-in)"
@@ -340,7 +340,7 @@ DOCKERFILE
} }
test_ubuntu22_resolved() { test_ubuntu22_resolved() {
local name="fips-dns-test-u22-resolved" local name="fips-dns-test-u22-resolved${FIPS_CI_NAME_SUFFIX:-}"
local image="fips-dns-test:ubuntu22-resolved" local image="fips-dns-test:ubuntu22-resolved"
log "Ubuntu 22.04 + systemd-resolved (expects global-drop-in)" log "Ubuntu 22.04 + systemd-resolved (expects global-drop-in)"
@@ -370,7 +370,7 @@ DOCKERFILE
} }
test_ubuntu24_resolved() { test_ubuntu24_resolved() {
local name="fips-dns-test-u24-resolved" local name="fips-dns-test-u24-resolved${FIPS_CI_NAME_SUFFIX:-}"
local image="fips-dns-test:ubuntu24-resolved" local image="fips-dns-test:ubuntu24-resolved"
log "Ubuntu 24.04 + systemd-resolved (expects global-drop-in)" log "Ubuntu 24.04 + systemd-resolved (expects global-drop-in)"
@@ -398,7 +398,7 @@ DOCKERFILE
} }
test_ubuntu26_resolved() { test_ubuntu26_resolved() {
local name="fips-dns-test-u26-resolved" local name="fips-dns-test-u26-resolved${FIPS_CI_NAME_SUFFIX:-}"
local image="fips-dns-test:ubuntu26-resolved" local image="fips-dns-test:ubuntu26-resolved"
log "Ubuntu 26.04 + systemd-resolved (expects global-drop-in)" log "Ubuntu 26.04 + systemd-resolved (expects global-drop-in)"
@@ -426,7 +426,7 @@ DOCKERFILE
} }
test_dnsmasq() { test_dnsmasq() {
local name="fips-dns-test-dnsmasq" local name="fips-dns-test-dnsmasq${FIPS_CI_NAME_SUFFIX:-}"
local image="fips-dns-test:dnsmasq" local image="fips-dns-test:dnsmasq"
log "Debian 12 + dnsmasq standalone" log "Debian 12 + dnsmasq standalone"
@@ -492,7 +492,7 @@ DOCKERFILE
} }
test_nm_dnsmasq() { test_nm_dnsmasq() {
local name="fips-dns-test-nm-dnsmasq" local name="fips-dns-test-nm-dnsmasq${FIPS_CI_NAME_SUFFIX:-}"
local image="fips-dns-test:nm-dnsmasq" local image="fips-dns-test:nm-dnsmasq"
log "Fedora + NetworkManager + dnsmasq plugin" log "Fedora + NetworkManager + dnsmasq plugin"
@@ -555,7 +555,7 @@ DOCKERFILE
} }
test_no_resolver() { test_no_resolver() {
local name="fips-dns-test-none" local name="fips-dns-test-none${FIPS_CI_NAME_SUFFIX:-}"
local image="fips-dns-test:none" local image="fips-dns-test:none"
log "Debian 12 bare (no resolver)" log "Debian 12 bare (no resolver)"
@@ -625,7 +625,7 @@ _run_e2e_scenario() {
local base_image="$2" local base_image="$2"
local apt_packages="$3" local apt_packages="$3"
local name="fips-dns-test-e2e-${distro_label}" local name="fips-dns-test-e2e-${distro_label}${FIPS_CI_NAME_SUFFIX:-}"
local image="fips-dns-test:e2e-${distro_label}" local image="fips-dns-test:e2e-${distro_label}"
log "End-to-end: ${base_image} + systemd-resolved + real fips + fips-gateway + dig" log "End-to-end: ${base_image} + systemd-resolved + real fips + fips-gateway + dig"
+2 -2
View File
@@ -27,7 +27,7 @@ x-fips-common: &fips-common
services: services:
service-a: service-a:
<<: *fips-common <<: *fips-common
container_name: fips-fw-container-a container_name: fips-fw-container-a${FIPS_CI_NAME_SUFFIX:-}
hostname: host-a hostname: host-a
volumes: volumes:
- ../docker/resolv.conf:/etc/resolv.conf:ro - ../docker/resolv.conf:/etc/resolv.conf:ro
@@ -40,7 +40,7 @@ services:
service-b: service-b:
<<: *fips-common <<: *fips-common
container_name: fips-fw-container-b container_name: fips-fw-container-b${FIPS_CI_NAME_SUFFIX:-}
hostname: host-b hostname: host-b
volumes: volumes:
- ../docker/resolv.conf:/etc/resolv.conf:ro - ../docker/resolv.conf:/etc/resolv.conf:ro
+2 -2
View File
@@ -22,8 +22,8 @@ TESTING_DIR="$(cd "$SCRIPT_DIR/.." && pwd)"
COMPOSE_FILE="$SCRIPT_DIR/docker-compose.yml" COMPOSE_FILE="$SCRIPT_DIR/docker-compose.yml"
GENERATE_CONFIGS="$SCRIPT_DIR/generate-configs.sh" GENERATE_CONFIGS="$SCRIPT_DIR/generate-configs.sh"
CONTAINER_A="fips-fw-container-a" CONTAINER_A="fips-fw-container-a${FIPS_CI_NAME_SUFFIX:-}"
CONTAINER_B="fips-fw-container-b" CONTAINER_B="fips-fw-container-b${FIPS_CI_NAME_SUFFIX:-}"
NPUB_A="npub1sjlh2c3x9w7kjsqg2ay080n2lff2uvt325vpan33ke34rn8l5jcqawh57m" NPUB_A="npub1sjlh2c3x9w7kjsqg2ay080n2lff2uvt325vpan33ke34rn8l5jcqawh57m"
NPUB_B="npub1tdwa4vjrjl33pcjdpf2t4p027nl86xrx24g4d3avg4vwvayr3g8qhd84le" NPUB_B="npub1tdwa4vjrjl33pcjdpf2t4p027nl86xrx24g4d3avg4vwvayr3g8qhd84le"
+15 -15
View File
@@ -34,7 +34,7 @@ services:
build: build:
context: ../.. context: ../..
dockerfile: examples/sidecar-nostr-relay/Dockerfile.app dockerfile: examples/sidecar-nostr-relay/Dockerfile.app
container_name: fips-nat-relay container_name: fips-nat-relay${FIPS_CI_NAME_SUFFIX:-}
restart: "no" restart: "no"
volumes: volumes:
- relay-data:/usr/src/app/strfry-db - relay-data:/usr/src/app/strfry-db
@@ -49,7 +49,7 @@ services:
stun: stun:
build: build:
context: ./stun context: ./stun
container_name: fips-nat-stun container_name: fips-nat-stun${FIPS_CI_NAME_SUFFIX:-}
restart: "no" restart: "no"
networks: networks:
wan: wan:
@@ -61,7 +61,7 @@ services:
build: build:
context: ./router context: ./router
profiles: ["cone", "symmetric"] profiles: ["cone", "symmetric"]
container_name: fips-nat-router-a container_name: fips-nat-router-a${FIPS_CI_NAME_SUFFIX:-}
cap_add: cap_add:
- NET_ADMIN - NET_ADMIN
sysctls: sysctls:
@@ -84,7 +84,7 @@ services:
build: build:
context: ./router context: ./router
profiles: ["cone", "symmetric"] profiles: ["cone", "symmetric"]
container_name: fips-nat-router-b container_name: fips-nat-router-b${FIPS_CI_NAME_SUFFIX:-}
cap_add: cap_add:
- NET_ADMIN - NET_ADMIN
sysctls: sysctls:
@@ -106,7 +106,7 @@ services:
cone-a: cone-a:
<<: *fips-common <<: *fips-common
profiles: ["cone"] profiles: ["cone"]
container_name: fips-nat-cone-a container_name: fips-nat-cone-a${FIPS_CI_NAME_SUFFIX:-}
hostname: fips-nat-cone-a hostname: fips-nat-cone-a
depends_on: depends_on:
- nat-a - nat-a
@@ -132,7 +132,7 @@ services:
cone-b: cone-b:
<<: *fips-common <<: *fips-common
profiles: ["cone"] profiles: ["cone"]
container_name: fips-nat-cone-b container_name: fips-nat-cone-b${FIPS_CI_NAME_SUFFIX:-}
hostname: fips-nat-cone-b hostname: fips-nat-cone-b
depends_on: depends_on:
- nat-b - nat-b
@@ -158,7 +158,7 @@ services:
symmetric-a: symmetric-a:
<<: *fips-common <<: *fips-common
profiles: ["symmetric"] profiles: ["symmetric"]
container_name: fips-nat-symmetric-a container_name: fips-nat-symmetric-a${FIPS_CI_NAME_SUFFIX:-}
hostname: fips-nat-symmetric-a hostname: fips-nat-symmetric-a
depends_on: depends_on:
- nat-a - nat-a
@@ -184,7 +184,7 @@ services:
symmetric-b: symmetric-b:
<<: *fips-common <<: *fips-common
profiles: ["symmetric"] profiles: ["symmetric"]
container_name: fips-nat-symmetric-b container_name: fips-nat-symmetric-b${FIPS_CI_NAME_SUFFIX:-}
hostname: fips-nat-symmetric-b hostname: fips-nat-symmetric-b
depends_on: depends_on:
- nat-b - nat-b
@@ -210,7 +210,7 @@ services:
lan-a: lan-a:
<<: *fips-common <<: *fips-common
profiles: ["lan"] profiles: ["lan"]
container_name: fips-nat-lan-a container_name: fips-nat-lan-a${FIPS_CI_NAME_SUFFIX:-}
hostname: fips-nat-lan-a hostname: fips-nat-lan-a
depends_on: depends_on:
- relay - relay
@@ -225,7 +225,7 @@ services:
lan-b: lan-b:
<<: *fips-common <<: *fips-common
profiles: ["lan"] profiles: ["lan"]
container_name: fips-nat-lan-b container_name: fips-nat-lan-b${FIPS_CI_NAME_SUFFIX:-}
hostname: fips-nat-lan-b hostname: fips-nat-lan-b
depends_on: depends_on:
- relay - relay
@@ -247,7 +247,7 @@ services:
nostr-pub-a: nostr-pub-a:
<<: *fips-common <<: *fips-common
profiles: ["nostr-publish-consume"] profiles: ["nostr-publish-consume"]
container_name: fips-nat-nostr-pub-a container_name: fips-nat-nostr-pub-a${FIPS_CI_NAME_SUFFIX:-}
hostname: fips-nat-nostr-pub-a hostname: fips-nat-nostr-pub-a
depends_on: depends_on:
- relay - relay
@@ -262,7 +262,7 @@ services:
nostr-pub-b: nostr-pub-b:
<<: *fips-common <<: *fips-common
profiles: ["nostr-publish-consume"] profiles: ["nostr-publish-consume"]
container_name: fips-nat-nostr-pub-b container_name: fips-nat-nostr-pub-b${FIPS_CI_NAME_SUFFIX:-}
hostname: fips-nat-nostr-pub-b hostname: fips-nat-nostr-pub-b
depends_on: depends_on:
- relay - relay
@@ -291,7 +291,7 @@ services:
stun-fault-node: stun-fault-node:
<<: *fips-common <<: *fips-common
profiles: ["stun-faults"] profiles: ["stun-faults"]
container_name: fips-nat-stun-fault-node container_name: fips-nat-stun-fault-node${FIPS_CI_NAME_SUFFIX:-}
hostname: fips-nat-stun-fault-node hostname: fips-nat-stun-fault-node
depends_on: depends_on:
- relay - relay
@@ -312,7 +312,7 @@ services:
stun-fault-peer: stun-fault-peer:
<<: *fips-common <<: *fips-common
profiles: ["stun-faults"] profiles: ["stun-faults"]
container_name: fips-nat-stun-fault-peer container_name: fips-nat-stun-fault-peer${FIPS_CI_NAME_SUFFIX:-}
hostname: fips-nat-stun-fault-peer hostname: fips-nat-stun-fault-peer
depends_on: depends_on:
- relay - relay
@@ -327,7 +327,7 @@ services:
stun-fault-shim: stun-fault-shim:
image: fips-test:latest image: fips-test:latest
profiles: ["stun-faults"] profiles: ["stun-faults"]
container_name: fips-nat-stun-fault-shim container_name: fips-nat-stun-fault-shim${FIPS_CI_NAME_SUFFIX:-}
depends_on: depends_on:
- stun-fault-node - stun-fault-node
cap_add: cap_add:
+50 -50
View File
@@ -36,7 +36,7 @@ cleanup() {
} }
helper_tcpdump_image() { helper_tcpdump_image() {
docker inspect -f '{{.Config.Image}}' fips-nat-router-a 2>/dev/null || echo nat-nat-a docker inspect -f '{{.Config.Image}}' fips-nat-router-a${FIPS_CI_NAME_SUFFIX:-} 2>/dev/null || echo nat-nat-a
} }
dump_container_state() { dump_container_state() {
@@ -202,7 +202,7 @@ dump_stun_udp_probe() {
local capture_file local capture_file
capture_file="$(mktemp)" capture_file="$(mktemp)"
docker run --rm --label com.corganlabs.fips-ci=1 --net=container:fips-nat-stun --cap-add NET_ADMIN --cap-add NET_RAW \ docker run --rm --label com.corganlabs.fips-ci=1 --label "com.corganlabs.fips-ci.run=${FIPS_CI_RUN_ID:-manual}" --net=container:fips-nat-stun${FIPS_CI_NAME_SUFFIX:-} --cap-add NET_ADMIN --cap-add NET_RAW \
--entrypoint sh "$helper_image" \ --entrypoint sh "$helper_image" \
-lc "timeout 8 tcpdump -ni any 'udp and not port 53' -c 80" \ -lc "timeout 8 tcpdump -ni any 'udp and not port 53' -c 80" \
>"$capture_file" 2>&1 & >"$capture_file" 2>&1 &
@@ -225,38 +225,38 @@ dump_stun_udp_probe() {
dump_cone_diagnostics() { dump_cone_diagnostics() {
echo "" echo ""
echo "=== cone diagnostics ===" echo "=== cone diagnostics ==="
dump_fips_state fips-nat-cone-a 172.31.254.30 7777 172.31.254.40 3478 dump_fips_state fips-nat-cone-a${FIPS_CI_NAME_SUFFIX:-} 172.31.254.30 7777 172.31.254.40 3478
dump_node_udp_probe fips-nat-cone-a dump_node_udp_probe fips-nat-cone-a${FIPS_CI_NAME_SUFFIX:-}
dump_fips_state fips-nat-cone-b 172.31.254.30 7777 172.31.254.40 3478 dump_fips_state fips-nat-cone-b${FIPS_CI_NAME_SUFFIX:-} 172.31.254.30 7777 172.31.254.40 3478
dump_node_udp_probe fips-nat-cone-b dump_node_udp_probe fips-nat-cone-b${FIPS_CI_NAME_SUFFIX:-}
dump_container_state fips-nat-router-a dump_container_state fips-nat-router-a${FIPS_CI_NAME_SUFFIX:-}
dump_router_udp_probe fips-nat-router-a fips-nat-cone-a dump_router_udp_probe fips-nat-router-a${FIPS_CI_NAME_SUFFIX:-} fips-nat-cone-a${FIPS_CI_NAME_SUFFIX:-}
dump_container_state fips-nat-router-b dump_container_state fips-nat-router-b${FIPS_CI_NAME_SUFFIX:-}
dump_router_udp_probe fips-nat-router-b fips-nat-cone-b dump_router_udp_probe fips-nat-router-b${FIPS_CI_NAME_SUFFIX:-} fips-nat-cone-b${FIPS_CI_NAME_SUFFIX:-}
dump_container_state fips-nat-relay dump_container_state fips-nat-relay${FIPS_CI_NAME_SUFFIX:-}
dump_stun_udp_probe fips-nat-cone-a dump_stun_udp_probe fips-nat-cone-a${FIPS_CI_NAME_SUFFIX:-}
dump_stun_udp_probe fips-nat-cone-b dump_stun_udp_probe fips-nat-cone-b${FIPS_CI_NAME_SUFFIX:-}
dump_container_state fips-nat-stun dump_container_state fips-nat-stun${FIPS_CI_NAME_SUFFIX:-}
} }
dump_symmetric_diagnostics() { dump_symmetric_diagnostics() {
echo "" echo ""
echo "=== symmetric diagnostics ===" echo "=== symmetric diagnostics ==="
dump_fips_state fips-nat-symmetric-a 172.31.254.30 7777 172.31.254.40 3478 dump_fips_state fips-nat-symmetric-a${FIPS_CI_NAME_SUFFIX:-} 172.31.254.30 7777 172.31.254.40 3478
dump_fips_state fips-nat-symmetric-b 172.31.254.30 7777 172.31.254.40 3478 dump_fips_state fips-nat-symmetric-b${FIPS_CI_NAME_SUFFIX:-} 172.31.254.30 7777 172.31.254.40 3478
dump_container_state fips-nat-router-a dump_container_state fips-nat-router-a${FIPS_CI_NAME_SUFFIX:-}
dump_container_state fips-nat-router-b dump_container_state fips-nat-router-b${FIPS_CI_NAME_SUFFIX:-}
dump_container_state fips-nat-relay dump_container_state fips-nat-relay${FIPS_CI_NAME_SUFFIX:-}
dump_container_state fips-nat-stun dump_container_state fips-nat-stun${FIPS_CI_NAME_SUFFIX:-}
} }
dump_lan_diagnostics() { dump_lan_diagnostics() {
echo "" echo ""
echo "=== lan diagnostics ===" echo "=== lan diagnostics ==="
dump_fips_state fips-nat-lan-a 172.31.10.30 7777 172.31.10.40 3478 dump_fips_state fips-nat-lan-a${FIPS_CI_NAME_SUFFIX:-} 172.31.10.30 7777 172.31.10.40 3478
dump_fips_state fips-nat-lan-b 172.31.10.30 7777 172.31.10.40 3478 dump_fips_state fips-nat-lan-b${FIPS_CI_NAME_SUFFIX:-} 172.31.10.30 7777 172.31.10.40 3478
dump_container_state fips-nat-relay dump_container_state fips-nat-relay${FIPS_CI_NAME_SUFFIX:-}
dump_container_state fips-nat-stun dump_container_state fips-nat-stun${FIPS_CI_NAME_SUFFIX:-}
} }
trap 'echo ""; echo "NAT test interrupted"; cleanup; exit 130' INT TERM trap 'echo ""; echo "NAT test interrupted"; cleanup; exit 130' INT TERM
@@ -334,22 +334,22 @@ run_cone() {
"$GENERATE_SCRIPT" cone "$GENERATE_SCRIPT" cone
"${COMPOSE[@]}" --profile cone up -d --build --force-recreate "${COMPOSE[@]}" --profile cone up -d --build --force-recreate
"$TOPOLOGY_SCRIPT" cone "$TOPOLOGY_SCRIPT" cone
wait_for_peers fips-nat-cone-a 1 45 || { wait_for_peers fips-nat-cone-a${FIPS_CI_NAME_SUFFIX:-} 1 45 || {
dump_cone_diagnostics dump_cone_diagnostics
return 1 return 1
} }
wait_for_peers fips-nat-cone-b 1 45 || { wait_for_peers fips-nat-cone-b${FIPS_CI_NAME_SUFFIX:-} 1 45 || {
dump_cone_diagnostics dump_cone_diagnostics
return 1 return 1
} }
assert_peer_path fips-nat-cone-a udp 172.31.254. assert_peer_path fips-nat-cone-a${FIPS_CI_NAME_SUFFIX:-} udp 172.31.254.
assert_peer_path fips-nat-cone-b udp 172.31.254. assert_peer_path fips-nat-cone-b${FIPS_CI_NAME_SUFFIX:-} udp 172.31.254.
assert_link_path fips-nat-cone-a 172.31.254. assert_link_path fips-nat-cone-a${FIPS_CI_NAME_SUFFIX:-} 172.31.254.
assert_link_path fips-nat-cone-b 172.31.254. assert_link_path fips-nat-cone-b${FIPS_CI_NAME_SUFFIX:-} 172.31.254.
# shellcheck disable=SC1090 # shellcheck disable=SC1090
source "$NAT_DIR/generated-configs/cone/npubs.env" source "$NAT_DIR/generated-configs/cone/npubs.env"
ping_peer fips-nat-cone-a "$NPUB_B" ping_peer fips-nat-cone-a${FIPS_CI_NAME_SUFFIX:-} "$NPUB_B"
ping_peer fips-nat-cone-b "$NPUB_A" ping_peer fips-nat-cone-b${FIPS_CI_NAME_SUFFIX:-} "$NPUB_A"
cleanup cleanup
} }
@@ -359,24 +359,24 @@ run_symmetric() {
NAT_MODE_A=symmetric NAT_MODE_B=symmetric "$GENERATE_SCRIPT" symmetric NAT_MODE_A=symmetric NAT_MODE_B=symmetric "$GENERATE_SCRIPT" symmetric
NAT_MODE_A=symmetric NAT_MODE_B=symmetric "${COMPOSE[@]}" --profile symmetric up -d --build --force-recreate NAT_MODE_A=symmetric NAT_MODE_B=symmetric "${COMPOSE[@]}" --profile symmetric up -d --build --force-recreate
"$TOPOLOGY_SCRIPT" symmetric "$TOPOLOGY_SCRIPT" symmetric
wait_for_peers fips-nat-symmetric-a 1 60 || { wait_for_peers fips-nat-symmetric-a${FIPS_CI_NAME_SUFFIX:-} 1 60 || {
dump_symmetric_diagnostics dump_symmetric_diagnostics
return 1 return 1
} }
wait_for_peers fips-nat-symmetric-b 1 60 || { wait_for_peers fips-nat-symmetric-b${FIPS_CI_NAME_SUFFIX:-} 1 60 || {
dump_symmetric_diagnostics dump_symmetric_diagnostics
return 1 return 1
} }
assert_peer_path fips-nat-symmetric-a tcp 172.31.254.11: assert_peer_path fips-nat-symmetric-a${FIPS_CI_NAME_SUFFIX:-} tcp 172.31.254.11:
assert_peer_path fips-nat-symmetric-b tcp 172.31.254.10: assert_peer_path fips-nat-symmetric-b${FIPS_CI_NAME_SUFFIX:-} tcp 172.31.254.10:
assert_link_path fips-nat-symmetric-a 172.31.254.11: assert_link_path fips-nat-symmetric-a${FIPS_CI_NAME_SUFFIX:-} 172.31.254.11:
assert_link_path fips-nat-symmetric-b 172.31.254.10: assert_link_path fips-nat-symmetric-b${FIPS_CI_NAME_SUFFIX:-} 172.31.254.10:
require_bootstrap_activity fips-nat-symmetric-a require_bootstrap_activity fips-nat-symmetric-a${FIPS_CI_NAME_SUFFIX:-}
require_bootstrap_activity fips-nat-symmetric-b require_bootstrap_activity fips-nat-symmetric-b${FIPS_CI_NAME_SUFFIX:-}
# shellcheck disable=SC1090 # shellcheck disable=SC1090
source "$NAT_DIR/generated-configs/symmetric/npubs.env" source "$NAT_DIR/generated-configs/symmetric/npubs.env"
ping_peer fips-nat-symmetric-a "$NPUB_B" ping_peer fips-nat-symmetric-a${FIPS_CI_NAME_SUFFIX:-} "$NPUB_B"
ping_peer fips-nat-symmetric-b "$NPUB_A" ping_peer fips-nat-symmetric-b${FIPS_CI_NAME_SUFFIX:-} "$NPUB_A"
cleanup cleanup
} }
@@ -385,22 +385,22 @@ run_lan() {
cleanup cleanup
"$GENERATE_SCRIPT" lan "$GENERATE_SCRIPT" lan
"${COMPOSE[@]}" --profile lan up -d --build --force-recreate "${COMPOSE[@]}" --profile lan up -d --build --force-recreate
wait_for_peers fips-nat-lan-a 1 45 || { wait_for_peers fips-nat-lan-a${FIPS_CI_NAME_SUFFIX:-} 1 45 || {
dump_lan_diagnostics dump_lan_diagnostics
return 1 return 1
} }
wait_for_peers fips-nat-lan-b 1 45 || { wait_for_peers fips-nat-lan-b${FIPS_CI_NAME_SUFFIX:-} 1 45 || {
dump_lan_diagnostics dump_lan_diagnostics
return 1 return 1
} }
assert_peer_path fips-nat-lan-a udp 172.31.10. assert_peer_path fips-nat-lan-a${FIPS_CI_NAME_SUFFIX:-} udp 172.31.10.
assert_peer_path fips-nat-lan-b udp 172.31.10. assert_peer_path fips-nat-lan-b${FIPS_CI_NAME_SUFFIX:-} udp 172.31.10.
assert_link_path fips-nat-lan-a 172.31.10. assert_link_path fips-nat-lan-a${FIPS_CI_NAME_SUFFIX:-} 172.31.10.
assert_link_path fips-nat-lan-b 172.31.10. assert_link_path fips-nat-lan-b${FIPS_CI_NAME_SUFFIX:-} 172.31.10.
# shellcheck disable=SC1090 # shellcheck disable=SC1090
source "$NAT_DIR/generated-configs/lan/npubs.env" source "$NAT_DIR/generated-configs/lan/npubs.env"
ping_peer fips-nat-lan-a "$NPUB_B" ping_peer fips-nat-lan-a${FIPS_CI_NAME_SUFFIX:-} "$NPUB_B"
ping_peer fips-nat-lan-b "$NPUB_A" ping_peer fips-nat-lan-b${FIPS_CI_NAME_SUFFIX:-} "$NPUB_A"
# Skip the final teardown when the mesh-lab harness wraps this # Skip the final teardown when the mesh-lab harness wraps this
# script: it needs to docker-logs the containers before teardown, # script: it needs to docker-logs the containers before teardown,
# and will run its own cleanup after capture. Failure paths above # and will run its own cleanup after capture. Failure paths above
+3 -3
View File
@@ -25,11 +25,11 @@ WAIT_LIB="$ROOT_DIR/testing/lib/wait-converge.sh"
PROFILE="nostr-publish-consume" PROFILE="nostr-publish-consume"
SCENARIO="$PROFILE" SCENARIO="$PROFILE"
COMPOSE=(docker compose -f "$NAT_DIR/docker-compose.yml") COMPOSE=(docker compose -f "$NAT_DIR/docker-compose.yml")
NODE_A="fips-nat-nostr-pub-a" NODE_A="fips-nat-nostr-pub-a${FIPS_CI_NAME_SUFFIX:-}"
NODE_B="fips-nat-nostr-pub-b" NODE_B="fips-nat-nostr-pub-b${FIPS_CI_NAME_SUFFIX:-}"
RELAY_HOST="172.31.10.30" RELAY_HOST="172.31.10.30"
RELAY_PORT=7777 RELAY_PORT=7777
RELAY_CONTAINER="fips-nat-relay" RELAY_CONTAINER="fips-nat-relay${FIPS_CI_NAME_SUFFIX:-}"
# shellcheck disable=SC1090 # shellcheck disable=SC1090
source "$WAIT_LIB" source "$WAIT_LIB"
+6 -6
View File
@@ -9,12 +9,12 @@ SCENARIO="${1:?usage: setup-topology.sh <cone|symmetric>}"
case "$SCENARIO" in case "$SCENARIO" in
cone) cone)
node_a="fips-nat-cone-a" node_a="fips-nat-cone-a${FIPS_CI_NAME_SUFFIX:-}"
node_b="fips-nat-cone-b" node_b="fips-nat-cone-b${FIPS_CI_NAME_SUFFIX:-}"
;; ;;
symmetric) symmetric)
node_a="fips-nat-symmetric-a" node_a="fips-nat-symmetric-a${FIPS_CI_NAME_SUFFIX:-}"
node_b="fips-nat-symmetric-b" node_b="fips-nat-symmetric-b${FIPS_CI_NAME_SUFFIX:-}"
;; ;;
*) *)
echo "Unsupported topology scenario: $SCENARIO" >&2 echo "Unsupported topology scenario: $SCENARIO" >&2
@@ -22,8 +22,8 @@ case "$SCENARIO" in
;; ;;
esac esac
router_a="fips-nat-router-a" router_a="fips-nat-router-a${FIPS_CI_NAME_SUFFIX:-}"
router_b="fips-nat-router-b" router_b="fips-nat-router-b${FIPS_CI_NAME_SUFFIX:-}"
helper_image() { helper_image() {
if [ -n "${IP_HELPER_IMAGE:-}" ]; then if [ -n "${IP_HELPER_IMAGE:-}" ]; then
+4 -4
View File
@@ -33,10 +33,10 @@ GENERATE_SCRIPT="$SCRIPT_DIR/generate-configs.sh"
PROFILE="stun-faults" PROFILE="stun-faults"
SCENARIO="$PROFILE" SCENARIO="$PROFILE"
COMPOSE=(docker compose -f "$NAT_DIR/docker-compose.yml") COMPOSE=(docker compose -f "$NAT_DIR/docker-compose.yml")
NODE="fips-nat-stun-fault-node" NODE="fips-nat-stun-fault-node${FIPS_CI_NAME_SUFFIX:-}"
PEER="fips-nat-stun-fault-peer" PEER="fips-nat-stun-fault-peer${FIPS_CI_NAME_SUFFIX:-}"
SHIM="fips-nat-stun-fault-shim" SHIM="fips-nat-stun-fault-shim${FIPS_CI_NAME_SUFFIX:-}"
STUN_CONTAINER="fips-nat-stun" STUN_CONTAINER="fips-nat-stun${FIPS_CI_NAME_SUFFIX:-}"
STUN_HOST="172.31.10.40" STUN_HOST="172.31.10.40"
STUN_PORT=3478 STUN_PORT=3478
DEV="eth0" DEV="eth0"
+27 -20
View File
@@ -25,7 +25,14 @@ NODE_B_NPUB="npub15h7z0ljzudqe9pgwx99cjsz2c0ennuyvkcc8zvtk3lg97xwzex9ska6g4y"
NODE_C_NSEC="15148ed0131f7da43fd13e369dfedede14fb64698f3756636b569c3a3e87438f" NODE_C_NSEC="15148ed0131f7da43fd13e369dfedede14fb64698f3756636b569c3a3e87438f"
NODE_C_NPUB="npub1zhezcykd0e34z4fxtranl45jaasgnlxv0kjqwlq2v56ggssn0w4qelcrvr" NODE_C_NPUB="npub1zhezcykd0e34z4fxtranl45jaasgnlxv0kjqwlq2v56ggssn0w4qelcrvr"
NETWORK_NAME="fips-sidecar-test" NETWORK_NAME="fips-sidecar-test${FIPS_CI_NAME_SUFFIX:-}"
# Compose project names. These are explicit (they override COMPOSE_PROJECT_NAME),
# so they carry the run suffix themselves — otherwise concurrent runs would share
# one project and tear down each other's containers.
PROJ_A="sidecar-a${FIPS_CI_NAME_SUFFIX:-}"
PROJ_B="sidecar-b${FIPS_CI_NAME_SUFFIX:-}"
PROJ_C="sidecar-c${FIPS_CI_NAME_SUFFIX:-}"
SUBNET="172.20.2.0/24" SUBNET="172.20.2.0/24"
NODE_A_IP="172.20.2.10" NODE_A_IP="172.20.2.10"
NODE_B_IP="172.20.2.11" NODE_B_IP="172.20.2.11"
@@ -48,10 +55,10 @@ fail() { echo " FAIL: $*"; FAILED=$((FAILED + 1)); }
cleanup() { cleanup() {
log "Cleaning up..." log "Cleaning up..."
# Tear down B and C first (they reference A's network as external) # Tear down B and C first (they reference A's network as external)
docker compose $COMPOSE_EXT -p sidecar-c down --volumes --remove-orphans 2>/dev/null || true docker compose $COMPOSE_EXT -p "$PROJ_C" down --volumes --remove-orphans 2>/dev/null || true
docker compose $COMPOSE_EXT -p sidecar-b down --volumes --remove-orphans 2>/dev/null || true docker compose $COMPOSE_EXT -p "$PROJ_B" down --volumes --remove-orphans 2>/dev/null || true
# Tear down A last (it owns the network) # Tear down A last (it owns the network)
docker compose $COMPOSE_BASE -p sidecar-a down --volumes --remove-orphans 2>/dev/null || true docker compose $COMPOSE_BASE -p "$PROJ_A" down --volumes --remove-orphans 2>/dev/null || true
} }
# Always clean up on exit # Always clean up on exit
@@ -84,7 +91,7 @@ FIPS_PEER_ADDR="" \
FIPS_NETWORK="$NETWORK_NAME" \ FIPS_NETWORK="$NETWORK_NAME" \
FIPS_SUBNET="$SUBNET" \ FIPS_SUBNET="$SUBNET" \
FIPS_IPV4="$NODE_A_IP" \ FIPS_IPV4="$NODE_A_IP" \
docker compose $COMPOSE_BASE -p sidecar-a up -d docker compose $COMPOSE_BASE -p "$PROJ_A" up -d
log "Starting node-b (peers with node-a, joins external network)..." log "Starting node-b (peers with node-a, joins external network)..."
FIPS_NSEC="$NODE_B_NSEC" \ FIPS_NSEC="$NODE_B_NSEC" \
@@ -94,7 +101,7 @@ FIPS_PEER_ALIAS="node-a" \
FIPS_NETWORK="$NETWORK_NAME" \ FIPS_NETWORK="$NETWORK_NAME" \
FIPS_SUBNET="$SUBNET" \ FIPS_SUBNET="$SUBNET" \
FIPS_IPV4="$NODE_B_IP" \ FIPS_IPV4="$NODE_B_IP" \
docker compose $COMPOSE_EXT -p sidecar-b up -d docker compose $COMPOSE_EXT -p "$PROJ_B" up -d
log "Starting node-c (peers with node-b, joins external network)..." log "Starting node-c (peers with node-b, joins external network)..."
FIPS_NSEC="$NODE_C_NSEC" \ FIPS_NSEC="$NODE_C_NSEC" \
@@ -104,7 +111,7 @@ FIPS_PEER_ALIAS="node-b" \
FIPS_NETWORK="$NETWORK_NAME" \ FIPS_NETWORK="$NETWORK_NAME" \
FIPS_SUBNET="$SUBNET" \ FIPS_SUBNET="$SUBNET" \
FIPS_IPV4="$NODE_C_IP" \ FIPS_IPV4="$NODE_C_IP" \
docker compose $COMPOSE_EXT -p sidecar-c up -d docker compose $COMPOSE_EXT -p "$PROJ_C" up -d
# ── Wait for convergence ────────────────────────────────────────────────── # ── Wait for convergence ──────────────────────────────────────────────────
@@ -113,7 +120,7 @@ log "Waiting for link establishment (up to ${CONVERGE_TIMEOUT}s)..."
converged=false converged=false
for i in $(seq 1 "$CONVERGE_TIMEOUT"); do for i in $(seq 1 "$CONVERGE_TIMEOUT"); do
# node-b should have 2 links (A and C) # node-b should have 2 links (A and C)
link_count=$(docker exec sidecar-b-fips-1 fipsctl show links 2>/dev/null \ link_count=$(docker exec "${PROJ_B}-fips-1" fipsctl show links 2>/dev/null \
| grep -c '"state": "connected"' || true) | grep -c '"state": "connected"' || true)
if [ "$link_count" -ge 2 ]; then if [ "$link_count" -ge 2 ]; then
converged=true converged=true
@@ -127,11 +134,11 @@ if [ "$converged" = true ]; then
else else
log "TIMEOUT: links did not converge in ${CONVERGE_TIMEOUT}s" log "TIMEOUT: links did not converge in ${CONVERGE_TIMEOUT}s"
log "node-a links:" log "node-a links:"
docker exec sidecar-a-fips-1 fipsctl show links 2>&1 || true docker exec "${PROJ_A}-fips-1" fipsctl show links 2>&1 || true
log "node-b links:" log "node-b links:"
docker exec sidecar-b-fips-1 fipsctl show links 2>&1 || true docker exec "${PROJ_B}-fips-1" fipsctl show links 2>&1 || true
log "node-c links:" log "node-c links:"
docker exec sidecar-c-fips-1 fipsctl show links 2>&1 || true docker exec "${PROJ_C}-fips-1" fipsctl show links 2>&1 || true
exit 1 exit 1
fi fi
@@ -141,9 +148,9 @@ fi
# chain. The directed pings below remain the actual assertions. # chain. The directed pings below remain the actual assertions.
_sidecar_converged() { _sidecar_converged() {
PASSED=0; FAILED=0 PASSED=0; FAILED=0
if docker exec sidecar-b-app-1 ping6 -c1 -W2 "${NODE_A_NPUB}.fips" >/dev/null 2>&1; then PASSED=$((PASSED+1)); else FAILED=$((FAILED+1)); fi if docker exec "${PROJ_B}-app-1" ping6 -c1 -W2 "${NODE_A_NPUB}.fips" >/dev/null 2>&1; then PASSED=$((PASSED+1)); else FAILED=$((FAILED+1)); fi
if docker exec sidecar-c-app-1 ping6 -c1 -W2 "${NODE_A_NPUB}.fips" >/dev/null 2>&1; then PASSED=$((PASSED+1)); else FAILED=$((FAILED+1)); fi if docker exec "${PROJ_C}-app-1" ping6 -c1 -W2 "${NODE_A_NPUB}.fips" >/dev/null 2>&1; then PASSED=$((PASSED+1)); else FAILED=$((FAILED+1)); fi
if docker exec sidecar-a-app-1 ping6 -c1 -W2 "${NODE_C_NPUB}.fips" >/dev/null 2>&1; then PASSED=$((PASSED+1)); else FAILED=$((FAILED+1)); fi if docker exec "${PROJ_A}-app-1" ping6 -c1 -W2 "${NODE_C_NPUB}.fips" >/dev/null 2>&1; then PASSED=$((PASSED+1)); else FAILED=$((FAILED+1)); fi
} }
wait_until_connected _sidecar_converged "$CONVERGE_TIMEOUT" 10 || true wait_until_connected _sidecar_converged "$CONVERGE_TIMEOUT" 10 || true
@@ -151,11 +158,11 @@ wait_until_connected _sidecar_converged "$CONVERGE_TIMEOUT" 10 || true
log "Verifying link counts..." log "Verifying link counts..."
a_links=$(docker exec sidecar-a-fips-1 fipsctl show links 2>/dev/null \ a_links=$(docker exec "${PROJ_A}-fips-1" fipsctl show links 2>/dev/null \
| grep -c '"state": "connected"' || true) | grep -c '"state": "connected"' || true)
b_links=$(docker exec sidecar-b-fips-1 fipsctl show links 2>/dev/null \ b_links=$(docker exec "${PROJ_B}-fips-1" fipsctl show links 2>/dev/null \
| grep -c '"state": "connected"' || true) | grep -c '"state": "connected"' || true)
c_links=$(docker exec sidecar-c-fips-1 fipsctl show links 2>/dev/null \ c_links=$(docker exec "${PROJ_C}-fips-1" fipsctl show links 2>/dev/null \
| grep -c '"state": "connected"' || true) | grep -c '"state": "connected"' || true)
[ "$a_links" -ge 1 ] && pass "node-a has $a_links link(s)" || fail "node-a has $a_links links (expected >= 1)" [ "$a_links" -ge 1 ] && pass "node-a has $a_links link(s)" || fail "node-a has $a_links links (expected >= 1)"
@@ -166,7 +173,7 @@ c_links=$(docker exec sidecar-c-fips-1 fipsctl show links 2>/dev/null \
log "Testing direct connectivity (B app → A via fips0)..." log "Testing direct connectivity (B app → A via fips0)..."
if docker exec sidecar-b-app-1 ping6 -c2 -W5 "${NODE_A_NPUB}.fips" >/dev/null 2>&1; then if docker exec "${PROJ_B}-app-1" ping6 -c2 -W5 "${NODE_A_NPUB}.fips" >/dev/null 2>&1; then
pass "node-b app can ping node-a via fips0" pass "node-b app can ping node-a via fips0"
else else
fail "node-b app cannot ping node-a via fips0" fail "node-b app cannot ping node-a via fips0"
@@ -176,7 +183,7 @@ fi
log "Testing multi-hop connectivity (C app → A via fips0, through B)..." log "Testing multi-hop connectivity (C app → A via fips0, through B)..."
if docker exec sidecar-c-app-1 ping6 -c2 -W10 "${NODE_A_NPUB}.fips" >/dev/null 2>&1; then if docker exec "${PROJ_C}-app-1" ping6 -c2 -W10 "${NODE_A_NPUB}.fips" >/dev/null 2>&1; then
pass "node-c app can ping node-a via fips0 (multi-hop through B)" pass "node-c app can ping node-a via fips0 (multi-hop through B)"
else else
fail "node-c app cannot ping node-a via fips0 (multi-hop through B)" fail "node-c app cannot ping node-a via fips0 (multi-hop through B)"
@@ -186,7 +193,7 @@ fi
log "Testing reverse multi-hop (A app → C via fips0, through B)..." log "Testing reverse multi-hop (A app → C via fips0, through B)..."
if docker exec sidecar-a-app-1 ping6 -c2 -W10 "${NODE_C_NPUB}.fips" >/dev/null 2>&1; then if docker exec "${PROJ_A}-app-1" ping6 -c2 -W10 "${NODE_C_NPUB}.fips" >/dev/null 2>&1; then
pass "node-a app can ping node-c via fips0 (multi-hop through B)" pass "node-a app can ping node-c via fips0 (multi-hop through B)"
else else
fail "node-a app cannot ping node-c via fips0 (multi-hop through B)" fail "node-a app cannot ping node-c via fips0 (multi-hop through B)"
+38 -38
View File
@@ -41,7 +41,7 @@ services:
node-a: node-a:
<<: *fips-common <<: *fips-common
profiles: ["mesh"] profiles: ["mesh"]
container_name: fips-node-a container_name: fips-node-a${FIPS_CI_NAME_SUFFIX:-}
hostname: node-a hostname: node-a
volumes: volumes:
- ../docker/resolv.conf:/etc/resolv.conf:ro - ../docker/resolv.conf:/etc/resolv.conf:ro
@@ -53,7 +53,7 @@ services:
node-b: node-b:
<<: *fips-common <<: *fips-common
profiles: ["mesh"] profiles: ["mesh"]
container_name: fips-node-b container_name: fips-node-b${FIPS_CI_NAME_SUFFIX:-}
hostname: node-b hostname: node-b
volumes: volumes:
- ../docker/resolv.conf:/etc/resolv.conf:ro - ../docker/resolv.conf:/etc/resolv.conf:ro
@@ -65,7 +65,7 @@ services:
node-c: node-c:
<<: *fips-common <<: *fips-common
profiles: ["mesh"] profiles: ["mesh"]
container_name: fips-node-c container_name: fips-node-c${FIPS_CI_NAME_SUFFIX:-}
hostname: node-c hostname: node-c
volumes: volumes:
- ../docker/resolv.conf:/etc/resolv.conf:ro - ../docker/resolv.conf:/etc/resolv.conf:ro
@@ -77,7 +77,7 @@ services:
node-d: node-d:
<<: *fips-common <<: *fips-common
profiles: ["mesh"] profiles: ["mesh"]
container_name: fips-node-d container_name: fips-node-d${FIPS_CI_NAME_SUFFIX:-}
hostname: node-d hostname: node-d
volumes: volumes:
- ../docker/resolv.conf:/etc/resolv.conf:ro - ../docker/resolv.conf:/etc/resolv.conf:ro
@@ -89,7 +89,7 @@ services:
node-e: node-e:
<<: *fips-common <<: *fips-common
profiles: ["mesh"] profiles: ["mesh"]
container_name: fips-node-e container_name: fips-node-e${FIPS_CI_NAME_SUFFIX:-}
hostname: node-e hostname: node-e
volumes: volumes:
- ../docker/resolv.conf:/etc/resolv.conf:ro - ../docker/resolv.conf:/etc/resolv.conf:ro
@@ -102,7 +102,7 @@ services:
pub-a: pub-a:
<<: *fips-common <<: *fips-common
profiles: ["mesh-public"] profiles: ["mesh-public"]
container_name: fips-node-a container_name: fips-node-a${FIPS_CI_NAME_SUFFIX:-}
hostname: node-a hostname: node-a
volumes: volumes:
- ../docker/resolv.conf:/etc/resolv.conf:ro - ../docker/resolv.conf:/etc/resolv.conf:ro
@@ -114,7 +114,7 @@ services:
pub-b: pub-b:
<<: *fips-common <<: *fips-common
profiles: ["mesh-public"] profiles: ["mesh-public"]
container_name: fips-node-b container_name: fips-node-b${FIPS_CI_NAME_SUFFIX:-}
hostname: node-b hostname: node-b
volumes: volumes:
- ../docker/resolv.conf:/etc/resolv.conf:ro - ../docker/resolv.conf:/etc/resolv.conf:ro
@@ -126,7 +126,7 @@ services:
pub-c: pub-c:
<<: *fips-common <<: *fips-common
profiles: ["mesh-public"] profiles: ["mesh-public"]
container_name: fips-node-c container_name: fips-node-c${FIPS_CI_NAME_SUFFIX:-}
hostname: node-c hostname: node-c
volumes: volumes:
- ../docker/resolv.conf:/etc/resolv.conf:ro - ../docker/resolv.conf:/etc/resolv.conf:ro
@@ -138,7 +138,7 @@ services:
pub-d: pub-d:
<<: *fips-common <<: *fips-common
profiles: ["mesh-public"] profiles: ["mesh-public"]
container_name: fips-node-d container_name: fips-node-d${FIPS_CI_NAME_SUFFIX:-}
hostname: node-d hostname: node-d
volumes: volumes:
- ../docker/resolv.conf:/etc/resolv.conf:ro - ../docker/resolv.conf:/etc/resolv.conf:ro
@@ -150,7 +150,7 @@ services:
pub-e: pub-e:
<<: *fips-common <<: *fips-common
profiles: ["mesh-public"] profiles: ["mesh-public"]
container_name: fips-node-e container_name: fips-node-e${FIPS_CI_NAME_SUFFIX:-}
hostname: node-e hostname: node-e
volumes: volumes:
- ../docker/resolv.conf:/etc/resolv.conf:ro - ../docker/resolv.conf:/etc/resolv.conf:ro
@@ -163,7 +163,7 @@ services:
chain-a: chain-a:
<<: *fips-common <<: *fips-common
profiles: ["chain"] profiles: ["chain"]
container_name: fips-node-a container_name: fips-node-a${FIPS_CI_NAME_SUFFIX:-}
hostname: node-a hostname: node-a
volumes: volumes:
- ../docker/resolv.conf:/etc/resolv.conf:ro - ../docker/resolv.conf:/etc/resolv.conf:ro
@@ -175,7 +175,7 @@ services:
chain-b: chain-b:
<<: *fips-common <<: *fips-common
profiles: ["chain"] profiles: ["chain"]
container_name: fips-node-b container_name: fips-node-b${FIPS_CI_NAME_SUFFIX:-}
hostname: node-b hostname: node-b
volumes: volumes:
- ../docker/resolv.conf:/etc/resolv.conf:ro - ../docker/resolv.conf:/etc/resolv.conf:ro
@@ -187,7 +187,7 @@ services:
chain-c: chain-c:
<<: *fips-common <<: *fips-common
profiles: ["chain"] profiles: ["chain"]
container_name: fips-node-c container_name: fips-node-c${FIPS_CI_NAME_SUFFIX:-}
hostname: node-c hostname: node-c
volumes: volumes:
- ../docker/resolv.conf:/etc/resolv.conf:ro - ../docker/resolv.conf:/etc/resolv.conf:ro
@@ -199,7 +199,7 @@ services:
chain-d: chain-d:
<<: *fips-common <<: *fips-common
profiles: ["chain"] profiles: ["chain"]
container_name: fips-node-d container_name: fips-node-d${FIPS_CI_NAME_SUFFIX:-}
hostname: node-d hostname: node-d
volumes: volumes:
- ../docker/resolv.conf:/etc/resolv.conf:ro - ../docker/resolv.conf:/etc/resolv.conf:ro
@@ -211,7 +211,7 @@ services:
chain-e: chain-e:
<<: *fips-common <<: *fips-common
profiles: ["chain"] profiles: ["chain"]
container_name: fips-node-e container_name: fips-node-e${FIPS_CI_NAME_SUFFIX:-}
hostname: node-e hostname: node-e
volumes: volumes:
- ../docker/resolv.conf:/etc/resolv.conf:ro - ../docker/resolv.conf:/etc/resolv.conf:ro
@@ -224,7 +224,7 @@ services:
rekey-a: rekey-a:
<<: *fips-common <<: *fips-common
profiles: ["rekey"] profiles: ["rekey"]
container_name: fips-node-a container_name: fips-node-a${FIPS_CI_NAME_SUFFIX:-}
hostname: node-a hostname: node-a
environment: environment:
- RUST_LOG=info,fips::node::handlers::rekey=debug - RUST_LOG=info,fips::node::handlers::rekey=debug
@@ -238,7 +238,7 @@ services:
rekey-b: rekey-b:
<<: *fips-common <<: *fips-common
profiles: ["rekey"] profiles: ["rekey"]
container_name: fips-node-b container_name: fips-node-b${FIPS_CI_NAME_SUFFIX:-}
hostname: node-b hostname: node-b
environment: environment:
- RUST_LOG=info,fips::node::handlers::rekey=debug - RUST_LOG=info,fips::node::handlers::rekey=debug
@@ -252,7 +252,7 @@ services:
rekey-c: rekey-c:
<<: *fips-common <<: *fips-common
profiles: ["rekey"] profiles: ["rekey"]
container_name: fips-node-c container_name: fips-node-c${FIPS_CI_NAME_SUFFIX:-}
hostname: node-c hostname: node-c
environment: environment:
- RUST_LOG=info,fips::node::handlers::rekey=debug - RUST_LOG=info,fips::node::handlers::rekey=debug
@@ -266,7 +266,7 @@ services:
rekey-d: rekey-d:
<<: *fips-common <<: *fips-common
profiles: ["rekey"] profiles: ["rekey"]
container_name: fips-node-d container_name: fips-node-d${FIPS_CI_NAME_SUFFIX:-}
hostname: node-d hostname: node-d
environment: environment:
- RUST_LOG=info,fips::node::handlers::rekey=debug - RUST_LOG=info,fips::node::handlers::rekey=debug
@@ -280,7 +280,7 @@ services:
rekey-e: rekey-e:
<<: *fips-common <<: *fips-common
profiles: ["rekey"] profiles: ["rekey"]
container_name: fips-node-e container_name: fips-node-e${FIPS_CI_NAME_SUFFIX:-}
hostname: node-e hostname: node-e
environment: environment:
- RUST_LOG=info,fips::node::handlers::rekey=debug - RUST_LOG=info,fips::node::handlers::rekey=debug
@@ -299,7 +299,7 @@ services:
rekey-accept-off-a: rekey-accept-off-a:
<<: *fips-common <<: *fips-common
profiles: ["rekey-accept-off"] profiles: ["rekey-accept-off"]
container_name: fips-node-a container_name: fips-node-a${FIPS_CI_NAME_SUFFIX:-}
hostname: node-a hostname: node-a
environment: environment:
- RUST_LOG=info,fips::node::handlers::rekey=debug,fips::node::handlers::handshake=debug - RUST_LOG=info,fips::node::handlers::rekey=debug,fips::node::handlers::handshake=debug
@@ -313,7 +313,7 @@ services:
rekey-accept-off-b: rekey-accept-off-b:
<<: *fips-common <<: *fips-common
profiles: ["rekey-accept-off"] profiles: ["rekey-accept-off"]
container_name: fips-node-b container_name: fips-node-b${FIPS_CI_NAME_SUFFIX:-}
hostname: node-b hostname: node-b
environment: environment:
- RUST_LOG=info,fips::node::handlers::rekey=debug,fips::node::handlers::handshake=debug - RUST_LOG=info,fips::node::handlers::rekey=debug,fips::node::handlers::handshake=debug
@@ -327,7 +327,7 @@ services:
rekey-accept-off-c: rekey-accept-off-c:
<<: *fips-common <<: *fips-common
profiles: ["rekey-accept-off"] profiles: ["rekey-accept-off"]
container_name: fips-node-c container_name: fips-node-c${FIPS_CI_NAME_SUFFIX:-}
hostname: node-c hostname: node-c
environment: environment:
- RUST_LOG=info,fips::node::handlers::rekey=debug,fips::node::handlers::handshake=debug - RUST_LOG=info,fips::node::handlers::rekey=debug,fips::node::handlers::handshake=debug
@@ -341,7 +341,7 @@ services:
rekey-accept-off-d: rekey-accept-off-d:
<<: *fips-common <<: *fips-common
profiles: ["rekey-accept-off"] profiles: ["rekey-accept-off"]
container_name: fips-node-d container_name: fips-node-d${FIPS_CI_NAME_SUFFIX:-}
hostname: node-d hostname: node-d
environment: environment:
- RUST_LOG=info,fips::node::handlers::rekey=debug,fips::node::handlers::handshake=debug - RUST_LOG=info,fips::node::handlers::rekey=debug,fips::node::handlers::handshake=debug
@@ -355,7 +355,7 @@ services:
rekey-accept-off-e: rekey-accept-off-e:
<<: *fips-common <<: *fips-common
profiles: ["rekey-accept-off"] profiles: ["rekey-accept-off"]
container_name: fips-node-e container_name: fips-node-e${FIPS_CI_NAME_SUFFIX:-}
hostname: node-e hostname: node-e
environment: environment:
- RUST_LOG=info,fips::node::handlers::rekey=debug,fips::node::handlers::handshake=debug - RUST_LOG=info,fips::node::handlers::rekey=debug,fips::node::handlers::handshake=debug
@@ -379,7 +379,7 @@ services:
rekey-outbound-only-a: rekey-outbound-only-a:
<<: *fips-common <<: *fips-common
profiles: ["rekey-outbound-only"] profiles: ["rekey-outbound-only"]
container_name: fips-node-a container_name: fips-node-a${FIPS_CI_NAME_SUFFIX:-}
hostname: node-a hostname: node-a
environment: environment:
- RUST_LOG=info,fips::node::handlers::rekey=debug,fips::node::handlers::handshake=debug - RUST_LOG=info,fips::node::handlers::rekey=debug,fips::node::handlers::handshake=debug
@@ -393,7 +393,7 @@ services:
rekey-outbound-only-b: rekey-outbound-only-b:
<<: *fips-common <<: *fips-common
profiles: ["rekey-outbound-only"] profiles: ["rekey-outbound-only"]
container_name: fips-node-b container_name: fips-node-b${FIPS_CI_NAME_SUFFIX:-}
hostname: node-b hostname: node-b
environment: environment:
- RUST_LOG=info,fips::node::handlers::rekey=debug,fips::node::handlers::handshake=debug - RUST_LOG=info,fips::node::handlers::rekey=debug,fips::node::handlers::handshake=debug
@@ -407,7 +407,7 @@ services:
rekey-outbound-only-c: rekey-outbound-only-c:
<<: *fips-common <<: *fips-common
profiles: ["rekey-outbound-only"] profiles: ["rekey-outbound-only"]
container_name: fips-node-c container_name: fips-node-c${FIPS_CI_NAME_SUFFIX:-}
hostname: node-c hostname: node-c
environment: environment:
- RUST_LOG=info,fips::node::handlers::rekey=debug,fips::node::handlers::handshake=debug - RUST_LOG=info,fips::node::handlers::rekey=debug,fips::node::handlers::handshake=debug
@@ -421,7 +421,7 @@ services:
rekey-outbound-only-d: rekey-outbound-only-d:
<<: *fips-common <<: *fips-common
profiles: ["rekey-outbound-only"] profiles: ["rekey-outbound-only"]
container_name: fips-node-d container_name: fips-node-d${FIPS_CI_NAME_SUFFIX:-}
hostname: node-d hostname: node-d
environment: environment:
- RUST_LOG=info,fips::node::handlers::rekey=debug,fips::node::handlers::handshake=debug - RUST_LOG=info,fips::node::handlers::rekey=debug,fips::node::handlers::handshake=debug
@@ -435,7 +435,7 @@ services:
rekey-outbound-only-e: rekey-outbound-only-e:
<<: *fips-common <<: *fips-common
profiles: ["rekey-outbound-only"] profiles: ["rekey-outbound-only"]
container_name: fips-node-e container_name: fips-node-e${FIPS_CI_NAME_SUFFIX:-}
hostname: node-e hostname: node-e
environment: environment:
- RUST_LOG=info,fips::node::handlers::rekey=debug,fips::node::handlers::handshake=debug - RUST_LOG=info,fips::node::handlers::rekey=debug,fips::node::handlers::handshake=debug
@@ -450,7 +450,7 @@ services:
tcp-a: tcp-a:
<<: *fips-common <<: *fips-common
profiles: ["tcp-chain"] profiles: ["tcp-chain"]
container_name: fips-node-a container_name: fips-node-a${FIPS_CI_NAME_SUFFIX:-}
hostname: node-a hostname: node-a
volumes: volumes:
- ../docker/resolv.conf:/etc/resolv.conf:ro - ../docker/resolv.conf:/etc/resolv.conf:ro
@@ -462,7 +462,7 @@ services:
tcp-b: tcp-b:
<<: *fips-common <<: *fips-common
profiles: ["tcp-chain"] profiles: ["tcp-chain"]
container_name: fips-node-b container_name: fips-node-b${FIPS_CI_NAME_SUFFIX:-}
hostname: node-b hostname: node-b
volumes: volumes:
- ../docker/resolv.conf:/etc/resolv.conf:ro - ../docker/resolv.conf:/etc/resolv.conf:ro
@@ -474,7 +474,7 @@ services:
tcp-c: tcp-c:
<<: *fips-common <<: *fips-common
profiles: ["tcp-chain"] profiles: ["tcp-chain"]
container_name: fips-node-c container_name: fips-node-c${FIPS_CI_NAME_SUFFIX:-}
hostname: node-c hostname: node-c
volumes: volumes:
- ../docker/resolv.conf:/etc/resolv.conf:ro - ../docker/resolv.conf:/etc/resolv.conf:ro
@@ -487,7 +487,7 @@ services:
gw-gateway: gw-gateway:
<<: *fips-common <<: *fips-common
profiles: ["gateway"] profiles: ["gateway"]
container_name: fips-gw-gateway container_name: fips-gw-gateway${FIPS_CI_NAME_SUFFIX:-}
hostname: gw-gateway hostname: gw-gateway
# Privileged required: gateway must enable IPv6 on eth1 (second network, # Privileged required: gateway must enable IPv6 on eth1 (second network,
# attached after container start) and manage nftables NAT rules. # attached after container start) and manage nftables NAT rules.
@@ -513,7 +513,7 @@ services:
gw-server: gw-server:
<<: *fips-common <<: *fips-common
profiles: ["gateway"] profiles: ["gateway"]
container_name: fips-gw-server container_name: fips-gw-server${FIPS_CI_NAME_SUFFIX:-}
hostname: gw-server hostname: gw-server
volumes: volumes:
- ../docker/resolv.conf:/etc/resolv.conf:ro - ../docker/resolv.conf:/etc/resolv.conf:ro
@@ -528,7 +528,7 @@ services:
gw-server-2: gw-server-2:
<<: *fips-common <<: *fips-common
profiles: ["gateway"] profiles: ["gateway"]
container_name: fips-gw-server-2 container_name: fips-gw-server-2${FIPS_CI_NAME_SUFFIX:-}
hostname: gw-server-2 hostname: gw-server-2
volumes: volumes:
- ../docker/resolv.conf:/etc/resolv.conf:ro - ../docker/resolv.conf:/etc/resolv.conf:ro
@@ -540,7 +540,7 @@ services:
gw-client: gw-client:
image: ${FIPS_TEST_APP_IMAGE:-fips-test-app:latest} image: ${FIPS_TEST_APP_IMAGE:-fips-test-app:latest}
profiles: ["gateway"] profiles: ["gateway"]
container_name: fips-gw-client container_name: fips-gw-client${FIPS_CI_NAME_SUFFIX:-}
hostname: gw-client hostname: gw-client
cap_add: cap_add:
- NET_ADMIN - NET_ADMIN
@@ -562,7 +562,7 @@ services:
gw-client-2: gw-client-2:
image: ${FIPS_TEST_APP_IMAGE:-fips-test-app:latest} image: ${FIPS_TEST_APP_IMAGE:-fips-test-app:latest}
profiles: ["gateway"] profiles: ["gateway"]
container_name: fips-gw-client-2 container_name: fips-gw-client-2${FIPS_CI_NAME_SUFFIX:-}
hostname: gw-client-2 hostname: gw-client-2
cap_add: cap_add:
- NET_ADMIN - NET_ADMIN
+7 -7
View File
@@ -92,7 +92,7 @@ info "phase 1: wait for node-$CAP_NODE peer_count to reach $MAX_PEERS (90s timeo
deadline=$(($(date +%s) + 90)) deadline=$(($(date +%s) + 90))
pc=0 pc=0
while [ "$(date +%s)" -lt "$deadline" ]; do while [ "$(date +%s)" -lt "$deadline" ]; do
pc=$(docker exec fips-node-$CAP_NODE fipsctl show status 2>/dev/null \ pc=$(docker exec "fips-node-${CAP_NODE}${FIPS_CI_NAME_SUFFIX:-}" fipsctl show status 2>/dev/null \
| grep -m1 peer_count | sed 's/.*: *//' | tr -d ',' || echo 0) | grep -m1 peer_count | sed 's/.*: *//' | tr -d ',' || echo 0)
[ "$pc" = "$MAX_PEERS" ] && break [ "$pc" = "$MAX_PEERS" ] && break
sleep 2 sleep 2
@@ -102,7 +102,7 @@ done
info "node-$CAP_NODE converged: peer_count=$pc" info "node-$CAP_NODE converged: peer_count=$pc"
# Identify admitted vs denied peers among configured peers # Identify admitted vs denied peers among configured peers
ADMITTED_NPUBS=$(docker exec fips-node-$CAP_NODE fipsctl show peers 2>/dev/null \ ADMITTED_NPUBS=$(docker exec "fips-node-${CAP_NODE}${FIPS_CI_NAME_SUFFIX:-}" fipsctl show peers 2>/dev/null \
| grep -oE 'npub1[a-z0-9]+' | sort -u || true) | grep -oE 'npub1[a-z0-9]+' | sort -u || true)
DENIED="" DENIED=""
ADMITTED="" ADMITTED=""
@@ -130,8 +130,8 @@ info "denied (sustained-retry): ${DENIED:-<none>}"
# with restarts every 15s we get ~30-50 firings across both denied peers. # with restarts every 15s we get ~30-50 firings across both denied peers.
info "phase 2: capture UDP/2121 on node-$CAP_NODE for ${CAPTURE_SECS}s, with denied-peer restart loop" info "phase 2: capture UDP/2121 on node-$CAP_NODE for ${CAPTURE_SECS}s, with denied-peer restart loop"
CAP_FILE=$(mktemp /tmp/admission-cap-pcap.XXXXXX.txt) CAP_FILE=$(mktemp /tmp/admission-cap-pcap.XXXXXX.txt)
HELPER_IMAGE=$(docker inspect -f '{{.Config.Image}}' fips-node-$CAP_NODE 2>/dev/null) HELPER_IMAGE=$(docker inspect -f '{{.Config.Image}}' "fips-node-${CAP_NODE}${FIPS_CI_NAME_SUFFIX:-}" 2>/dev/null)
[ -n "$HELPER_IMAGE" ] || fail "could not resolve helper image from fips-node-$CAP_NODE" [ -n "$HELPER_IMAGE" ] || fail "could not resolve helper image from fips-node-${CAP_NODE}${FIPS_CI_NAME_SUFFIX:-}"
# Background: cycle denied peers to reset their backoff and drive load. # Background: cycle denied peers to reset their backoff and drive load.
( (
@@ -140,7 +140,7 @@ HELPER_IMAGE=$(docker inspect -f '{{.Config.Image}}' fips-node-$CAP_NODE 2>/dev/
sleep 15 sleep 15
elapsed=$((elapsed + 15)) elapsed=$((elapsed + 15))
for n in $DENIED; do for n in $DENIED; do
docker restart "fips-node-$n" >/dev/null 2>&1 & docker restart "fips-node-${n}${FIPS_CI_NAME_SUFFIX:-}" >/dev/null 2>&1 &
done done
wait wait
info " [load-driver] restarted denied peers ($DENIED) at t+${elapsed}s" info " [load-driver] restarted denied peers ($DENIED) at t+${elapsed}s"
@@ -149,7 +149,7 @@ HELPER_IMAGE=$(docker inspect -f '{{.Config.Image}}' fips-node-$CAP_NODE 2>/dev/
LOAD_PID=$! LOAD_PID=$!
# Foreground: tcpdump capture for CAPTURE_SECS # Foreground: tcpdump capture for CAPTURE_SECS
docker run --rm --label com.corganlabs.fips-ci=1 --net=container:fips-node-$CAP_NODE \ docker run --rm --label com.corganlabs.fips-ci=1 --label "com.corganlabs.fips-ci.run=${FIPS_CI_RUN_ID:-manual}" --net=container:"fips-node-${CAP_NODE}${FIPS_CI_NAME_SUFFIX:-}" \
--cap-add NET_ADMIN --cap-add NET_RAW \ --cap-add NET_ADMIN --cap-add NET_RAW \
--entrypoint sh "$HELPER_IMAGE" \ --entrypoint sh "$HELPER_IMAGE" \
-c "timeout $CAPTURE_SECS tcpdump -nn -i any 'udp port 2121' -l 2>&1 || true" \ -c "timeout $CAPTURE_SECS tcpdump -nn -i any 'udp port 2121' -l 2>&1 || true" \
@@ -186,7 +186,7 @@ for n in $DENIED; do
done done
# ── Phase 4: cap'd node still at exactly max_peers ─────────────────── # ── Phase 4: cap'd node still at exactly max_peers ───────────────────
pc_final=$(docker exec fips-node-$CAP_NODE fipsctl show status 2>/dev/null \ pc_final=$(docker exec "fips-node-${CAP_NODE}${FIPS_CI_NAME_SUFFIX:-}" fipsctl show status 2>/dev/null \
| grep -m1 peer_count | sed 's/.*: *//' | tr -d ',' || echo 0) | grep -m1 peer_count | sed 's/.*: *//' | tr -d ',' || echo 0)
info "node-$CAP_NODE final peer_count=$pc_final (expected $MAX_PEERS)" info "node-$CAP_NODE final peer_count=$pc_final (expected $MAX_PEERS)"
[ "$pc_final" = "$MAX_PEERS" ] || OVERALL=1 [ "$pc_final" = "$MAX_PEERS" ] || OVERALL=1
+4 -4
View File
@@ -47,7 +47,7 @@ source "$ENV_FILE"
iperf_once() { iperf_once() {
local client="$1" dest_npub="$2" local client="$1" dest_npub="$2"
local out local out
if ! out=$(docker exec "fips-$client" iperf3 -c "${dest_npub}.fips" \ if ! out=$(docker exec "fips-${client}${FIPS_CI_NAME_SUFFIX:-}" iperf3 -c "${dest_npub}.fips" \
-t "$DURATION" -P "$PARALLEL" -J 2>&1); then -t "$DURATION" -P "$PARALLEL" -J 2>&1); then
echo FAIL echo FAIL
return return
@@ -105,7 +105,7 @@ PYEOF
ping_path() { ping_path() {
local client="$1" dest_npub="$2" local client="$1" dest_npub="$2"
local out local out
if ! out=$(docker exec "fips-$client" \ if ! out=$(docker exec "fips-${client}${FIPS_CI_NAME_SUFFIX:-}" \
ping -c "$PING_COUNT" -i 0.2 -w "$((PING_COUNT * 2))" \ ping -c "$PING_COUNT" -i 0.2 -w "$((PING_COUNT * 2))" \
-q "${dest_npub}.fips" 2>&1); then -q "${dest_npub}.fips" 2>&1); then
echo FAIL echo FAIL
@@ -203,7 +203,7 @@ CONVERGE_SECS="${FIPS_BENCH_CONVERGE_SECS:-15}"
peer_is_direct() { peer_is_direct() {
local client="$1" dest_npub="$2" local client="$1" dest_npub="$2"
docker exec "fips-$client" fipsctl show peers 2>/dev/null \ docker exec "fips-${client}${FIPS_CI_NAME_SUFFIX:-}" fipsctl show peers 2>/dev/null \
| python3 -c ' | python3 -c '
import json, sys import json, sys
target = sys.argv[1] target = sys.argv[1]
@@ -225,7 +225,7 @@ except Exception:
# multihop alternative). # multihop alternative).
peer_bytes_sent_snapshot() { peer_bytes_sent_snapshot() {
local client="$1" local client="$1"
docker exec "fips-$client" fipsctl show peers 2>/dev/null \ docker exec "fips-${client}${FIPS_CI_NAME_SUFFIX:-}" fipsctl show peers 2>/dev/null \
| python3 -c ' | python3 -c '
import json, sys import json, sys
try: try:
+5 -5
View File
@@ -20,11 +20,11 @@ source "$SCRIPT_DIR/../../lib/wait-converge.sh"
GENERATED_DIR="$SCRIPT_DIR/../generated-configs" GENERATED_DIR="$SCRIPT_DIR/../generated-configs"
ENV_FILE="$GENERATED_DIR/npubs.env" ENV_FILE="$GENERATED_DIR/npubs.env"
GATEWAY="fips-gw-gateway" GATEWAY="fips-gw-gateway${FIPS_CI_NAME_SUFFIX:-}"
SERVER="fips-gw-server" SERVER="fips-gw-server${FIPS_CI_NAME_SUFFIX:-}"
SERVER2="fips-gw-server-2" SERVER2="fips-gw-server-2${FIPS_CI_NAME_SUFFIX:-}"
CLIENT="fips-gw-client" CLIENT="fips-gw-client${FIPS_CI_NAME_SUFFIX:-}"
CLIENT2="fips-gw-client-2" CLIENT2="fips-gw-client-2${FIPS_CI_NAME_SUFFIX:-}"
# ── inject-config subcommand ───────────────────────────────────────────── # ── inject-config subcommand ─────────────────────────────────────────────
+2 -2
View File
@@ -49,7 +49,7 @@ iperf_test() {
if [ "$LIVE_OUTPUT" = true ]; then if [ "$LIVE_OUTPUT" = true ]; then
# Show live output # Show live output
echo "Running iperf3 test (live output):" echo "Running iperf3 test (live output):"
if docker exec "fips-$client_node" timeout "$IPERF_TIMEOUT" iperf3 -c "${dest_npub}.fips" -t "$DURATION" -P "$PARALLEL"; then if docker exec "fips-${client_node}${FIPS_CI_NAME_SUFFIX:-}" timeout "$IPERF_TIMEOUT" iperf3 -c "${dest_npub}.fips" -t "$DURATION" -P "$PARALLEL"; then
PASSED=$((PASSED + 1)) PASSED=$((PASSED + 1))
else else
echo "FAIL" echo "FAIL"
@@ -59,7 +59,7 @@ iperf_test() {
# Capture and summarize output # Capture and summarize output
echo -n "Running iperf3 test... " echo -n "Running iperf3 test... "
local output local output
if output=$(docker exec "fips-$client_node" timeout "$IPERF_TIMEOUT" iperf3 -c "${dest_npub}.fips" -t "$DURATION" -P "$PARALLEL" 2>&1); then if output=$(docker exec "fips-${client_node}${FIPS_CI_NAME_SUFFIX:-}" timeout "$IPERF_TIMEOUT" iperf3 -c "${dest_npub}.fips" -t "$DURATION" -P "$PARALLEL" 2>&1); then
# Check if we got valid results # Check if we got valid results
if echo "$output" | grep -q "sender"; then if echo "$output" | grep -q "sender"; then
# Extract and display results (get SUM line for aggregate bandwidth) # Extract and display results (get SUM line for aggregate bandwidth)
+3 -3
View File
@@ -180,7 +180,7 @@ do_apply() {
echo "" echo ""
for node in $NODES; do for node in $NODES; do
local container="fips-node-$node" local container="fips-node-${node}${FIPS_CI_NAME_SUFFIX:-}"
echo -n " $container ... " echo -n " $container ... "
if ! container_running "$container"; then if ! container_running "$container"; then
echo "SKIP (not running)" echo "SKIP (not running)"
@@ -199,7 +199,7 @@ do_remove() {
echo "" echo ""
for node in $NODES; do for node in $NODES; do
local container="fips-node-$node" local container="fips-node-${node}${FIPS_CI_NAME_SUFFIX:-}"
echo -n " $container ... " echo -n " $container ... "
if ! container_running "$container"; then if ! container_running "$container"; then
echo "SKIP (not running)" echo "SKIP (not running)"
@@ -219,7 +219,7 @@ do_status() {
echo "" echo ""
for node in $NODES; do for node in $NODES; do
local container="fips-node-$node" local container="fips-node-${node}${FIPS_CI_NAME_SUFFIX:-}"
echo " $container:" echo " $container:"
if ! container_running "$container"; then if ! container_running "$container"; then
echo " (not running)" echo " (not running)"
+12 -12
View File
@@ -37,7 +37,7 @@ ping_test() {
echo -n " $label ... " echo -n " $label ... "
local output local output
if output=$(docker exec "fips-$from" ping6 -c "$COUNT" -W "$TIMEOUT" "${to_npub}.fips" 2>&1); then if output=$(docker exec "fips-${from}${FIPS_CI_NAME_SUFFIX:-}" ping6 -c "$COUNT" -W "$TIMEOUT" "${to_npub}.fips" 2>&1); then
# Extract round-trip time from ping output # Extract round-trip time from ping output
local rtt local rtt
rtt=$(echo "$output" | grep -oE 'time=[0-9.]+' | cut -d= -f2) rtt=$(echo "$output" | grep -oE 'time=[0-9.]+' | cut -d= -f2)
@@ -61,7 +61,7 @@ ping_all_quiet() {
for ((i=0; i<n; i++)); do for ((i=0; i<n; i++)); do
for ((j=0; j<n; j++)); do for ((j=0; j<n; j++)); do
[ "$i" -eq "$j" ] && continue [ "$i" -eq "$j" ] && continue
if docker exec "fips-node-${LABELS[$i],,}" \ if docker exec "fips-node-${LABELS[$i],,}${FIPS_CI_NAME_SUFFIX:-}" \
ping6 -c 1 -W 1 "${NPUBS[$j]}.fips" >/dev/null 2>&1; then ping6 -c 1 -W 1 "${NPUBS[$j]}.fips" >/dev/null 2>&1; then
PASSED=$((PASSED + 1)) PASSED=$((PASSED + 1))
else else
@@ -78,18 +78,18 @@ echo ""
echo "Waiting for mesh convergence..." echo "Waiting for mesh convergence..."
if [ "$PROFILE" = "chain" ]; then if [ "$PROFILE" = "chain" ]; then
# Chain: A-B-C-D-E, each interior node has 2 peers, endpoints have 1 # Chain: A-B-C-D-E, each interior node has 2 peers, endpoints have 1
wait_for_peers fips-node-a 1 20 || true wait_for_peers fips-node-a${FIPS_CI_NAME_SUFFIX:-} 1 20 || true
wait_for_peers fips-node-b 2 20 || true wait_for_peers fips-node-b${FIPS_CI_NAME_SUFFIX:-} 2 20 || true
wait_for_peers fips-node-c 2 20 || true wait_for_peers fips-node-c${FIPS_CI_NAME_SUFFIX:-} 2 20 || true
wait_for_peers fips-node-d 2 20 || true wait_for_peers fips-node-d${FIPS_CI_NAME_SUFFIX:-} 2 20 || true
wait_for_peers fips-node-e 1 20 || true wait_for_peers fips-node-e${FIPS_CI_NAME_SUFFIX:-} 1 20 || true
elif [ "$PROFILE" = "mesh" ] || [ "$PROFILE" = "mesh-public" ]; then elif [ "$PROFILE" = "mesh" ] || [ "$PROFILE" = "mesh-public" ]; then
# Mesh: check all nodes reach their configured peer counts # Mesh: check all nodes reach their configured peer counts
wait_for_peers fips-node-a 2 20 || true wait_for_peers fips-node-a${FIPS_CI_NAME_SUFFIX:-} 2 20 || true
wait_for_peers fips-node-b 1 20 || true wait_for_peers fips-node-b${FIPS_CI_NAME_SUFFIX:-} 1 20 || true
wait_for_peers fips-node-c 3 20 || true wait_for_peers fips-node-c${FIPS_CI_NAME_SUFFIX:-} 3 20 || true
wait_for_peers fips-node-d 3 20 || true wait_for_peers fips-node-d${FIPS_CI_NAME_SUFFIX:-} 3 20 || true
wait_for_peers fips-node-e 3 20 || true wait_for_peers fips-node-e${FIPS_CI_NAME_SUFFIX:-} 3 20 || true
fi fi
# Wait for full pairwise connectivity, progress-aware: the actual pings # Wait for full pairwise connectivity, progress-aware: the actual pings
# are the convergence signal, the deadline extends while more pairs come # are the convergence signal, the deadline extends while more pairs come
+7 -7
View File
@@ -223,7 +223,7 @@ ping_one() {
if (( attempt > 1 )); then if (( attempt > 1 )); then
sleep "$PING_RETRY_DELAY" sleep "$PING_RETRY_DELAY"
fi fi
if output=$(docker exec "fips-$from" ping6 -c 1 -W "$ping_timeout" "${to_npub}.fips" 2>&1); then if output=$(docker exec "fips-${from}${FIPS_CI_NAME_SUFFIX:-}" ping6 -c 1 -W "$ping_timeout" "${to_npub}.fips" 2>&1); then
rtt=$(echo "$output" | grep -oE 'time=[0-9.]+' | cut -d= -f2) rtt=$(echo "$output" | grep -oE 'time=[0-9.]+' | cut -d= -f2)
if [ -z "$quiet" ]; then if [ -z "$quiet" ]; then
if (( attempt == 1 )); then if (( attempt == 1 )); then
@@ -289,7 +289,7 @@ count_log_pattern() {
local pattern="$1" local pattern="$1"
local total=0 local total=0
for node in $NODES; do for node in $NODES; do
local count=$(docker logs "fips-node-$node" 2>&1 | grep -c "$pattern" || true) local count=$(docker logs "fips-node-${node}${FIPS_CI_NAME_SUFFIX:-}" 2>&1 | grep -c "$pattern" || true)
total=$((total + count)) total=$((total + count))
done done
echo "$total" echo "$total"
@@ -352,7 +352,7 @@ dump_peer_connectivity() {
echo "=== Peer connectivity snapshot ===" echo "=== Peer connectivity snapshot ==="
for node in $NODES; do for node in $NODES; do
echo "--- node-$node ---" echo "--- node-$node ---"
docker exec "fips-node-$node" fipsctl show peers 2>/dev/null || true docker exec "fips-node-${node}${FIPS_CI_NAME_SUFFIX:-}" fipsctl show peers 2>/dev/null || true
echo "" echo ""
done done
} }
@@ -504,7 +504,7 @@ assert_zero_count "Session AEAD decryption failed" \
if [ -n "$REKEY_ACCEPT_OFF_NODES" ]; then if [ -n "$REKEY_ACCEPT_OFF_NODES" ]; then
DUAL_INIT_THRESHOLD=10 DUAL_INIT_THRESHOLD=10
for off_node in ${REKEY_ACCEPT_OFF_NODES//,/ }; do for off_node in ${REKEY_ACCEPT_OFF_NODES//,/ }; do
count=$(docker logs "fips-node-$off_node" 2>&1 \ count=$(docker logs "fips-node-${off_node}${FIPS_CI_NAME_SUFFIX:-}" 2>&1 \
| grep -cE "Dual rekey initiation: we win" || true) | grep -cE "Dual rekey initiation: we win" || true)
if [ "${count:-0}" -le "$DUAL_INIT_THRESHOLD" ]; then if [ "${count:-0}" -le "$DUAL_INIT_THRESHOLD" ]; then
echo " PASS: node-$off_node dual-init drops below threshold ($count <= $DUAL_INIT_THRESHOLD)" echo " PASS: node-$off_node dual-init drops below threshold ($count <= $DUAL_INIT_THRESHOLD)"
@@ -527,7 +527,7 @@ fi
if [ -n "$REKEY_OUTBOUND_ONLY_NODES" ]; then if [ -n "$REKEY_OUTBOUND_ONLY_NODES" ]; then
DUAL_INIT_THRESHOLD=10 DUAL_INIT_THRESHOLD=10
for n in $NODES; do for n in $NODES; do
count=$(docker logs "fips-node-$n" 2>&1 \ count=$(docker logs "fips-node-${n}${FIPS_CI_NAME_SUFFIX:-}" 2>&1 \
| grep -cE "Dual rekey initiation: we win" || true) | grep -cE "Dual rekey initiation: we win" || true)
if [ "${count:-0}" -le "$DUAL_INIT_THRESHOLD" ]; then if [ "${count:-0}" -le "$DUAL_INIT_THRESHOLD" ]; then
echo " PASS: node-$n dual-init drops below threshold ($count <= $DUAL_INIT_THRESHOLD)" echo " PASS: node-$n dual-init drops below threshold ($count <= $DUAL_INIT_THRESHOLD)"
@@ -562,7 +562,7 @@ else
echo "=== Node logs (rekey-related, head -200) ===" echo "=== Node logs (rekey-related, head -200) ==="
for node in $NODES; do for node in $NODES; do
echo "--- node-$node ---" echo "--- node-$node ---"
docker logs "fips-node-$node" 2>&1 | \ docker logs "fips-node-${node}${FIPS_CI_NAME_SUFFIX:-}" 2>&1 | \
grep -E "(rekey|Rekey|cross|Cross|teardown|ERROR|PANIC|K-bit|no route|next hop|TTL exhausted|MTU exceeded|Congestion|decrypt|Decrypt|AEAD|Notify|drain|Drain|promot)" | \ grep -E "(rekey|Rekey|cross|Cross|teardown|ERROR|PANIC|K-bit|no route|next hop|TTL exhausted|MTU exceeded|Congestion|decrypt|Decrypt|AEAD|Notify|drain|Drain|promot)" | \
head -200 head -200
echo "" echo ""
@@ -571,7 +571,7 @@ else
echo "=== Node logs (last 80 lines, unfiltered) ===" echo "=== Node logs (last 80 lines, unfiltered) ==="
for node in $NODES; do for node in $NODES; do
echo "--- node-$node ---" echo "--- node-$node ---"
docker logs "fips-node-$node" 2>&1 | tail -80 docker logs "fips-node-${node}${FIPS_CI_NAME_SUFFIX:-}" 2>&1 | tail -80
echo "" echo ""
done done
exit 1 exit 1
@@ -24,7 +24,7 @@ services:
sysctls: sysctls:
- net.ipv6.conf.all.disable_ipv6=0 - net.ipv6.conf.all.disable_ipv6=0
restart: "no" restart: "no"
container_name: fips-dir-a container_name: fips-dir-a${FIPS_CI_NAME_SUFFIX:-}
hostname: fips-dir-a hostname: fips-dir-a
volumes: volumes:
- ./configs/node-a.yaml:/etc/fips/fips.yaml:ro - ./configs/node-a.yaml:/etc/fips/fips.yaml:ro
@@ -44,7 +44,7 @@ services:
sysctls: sysctls:
- net.ipv6.conf.all.disable_ipv6=0 - net.ipv6.conf.all.disable_ipv6=0
restart: "no" restart: "no"
container_name: fips-dir-b container_name: fips-dir-b${FIPS_CI_NAME_SUFFIX:-}
hostname: fips-dir-b hostname: fips-dir-b
volumes: volumes:
- ./configs/node-b.yaml:/etc/fips/fips.yaml:ro - ./configs/node-b.yaml:/etc/fips/fips.yaml:ro
@@ -95,7 +95,7 @@ elapsed=0
while [ "$elapsed" -lt "$MAX_WAIT_ONION" ]; do while [ "$elapsed" -lt "$MAX_WAIT_ONION" ]; do
# Extract .onion address from structured log: onion_address=<addr>.onion # Extract .onion address from structured log: onion_address=<addr>.onion
# Strip ANSI color codes before matching (tracing emits them by default) # Strip ANSI color codes before matching (tracing emits them by default)
ONION_ADDR=$(docker logs fips-dir-a 2>&1 \ ONION_ADDR=$(docker logs fips-dir-a${FIPS_CI_NAME_SUFFIX:-} 2>&1 \
| sed 's/\x1b\[[0-9;]*m//g' \ | sed 's/\x1b\[[0-9;]*m//g' \
| grep -oE 'onion_address=[a-z2-7]{56}\.onion' \ | grep -oE 'onion_address=[a-z2-7]{56}\.onion' \
| head -1 \ | head -1 \
@@ -115,7 +115,7 @@ if [ -z "$ONION_ADDR" ]; then
echo " FAIL: Onion service not created within ${MAX_WAIT_ONION}s" echo " FAIL: Onion service not created within ${MAX_WAIT_ONION}s"
echo "" echo ""
echo "Node A logs (last 30 lines):" echo "Node A logs (last 30 lines):"
docker logs fips-dir-a 2>&1 | tail -30 docker logs fips-dir-a${FIPS_CI_NAME_SUFFIX:-} 2>&1 | tail -30
docker compose down docker compose down
exit 1 exit 1
fi fi
@@ -143,8 +143,8 @@ peers_a=0
peers_b=0 peers_b=0
elapsed=0 elapsed=0
while [ "$elapsed" -lt "$MAX_WAIT_PEER" ]; do while [ "$elapsed" -lt "$MAX_WAIT_PEER" ]; do
peers_a=$(count_connected_peers fips-dir-a) peers_a=$(count_connected_peers fips-dir-a${FIPS_CI_NAME_SUFFIX:-})
peers_b=$(count_connected_peers fips-dir-b) peers_b=$(count_connected_peers fips-dir-b${FIPS_CI_NAME_SUFFIX:-})
if [ "$peers_a" -ge 1 ] && [ "$peers_b" -ge 1 ]; then if [ "$peers_a" -ge 1 ] && [ "$peers_b" -ge 1 ]; then
echo " Both nodes connected after ${elapsed}s (A: ${peers_a}, B: ${peers_b})" echo " Both nodes connected after ${elapsed}s (A: ${peers_a}, B: ${peers_b})"
@@ -159,10 +159,10 @@ if [ "$peers_a" -lt 1 ] || [ "$peers_b" -lt 1 ]; then
echo " FAIL: Peers not established within ${MAX_WAIT_PEER}s" echo " FAIL: Peers not established within ${MAX_WAIT_PEER}s"
echo "" echo ""
echo "Node A logs (last 30 lines):" echo "Node A logs (last 30 lines):"
docker logs fips-dir-a 2>&1 | tail -30 docker logs fips-dir-a${FIPS_CI_NAME_SUFFIX:-} 2>&1 | tail -30
echo "" echo ""
echo "Node B logs (last 30 lines):" echo "Node B logs (last 30 lines):"
docker logs fips-dir-b 2>&1 | tail -30 docker logs fips-dir-b${FIPS_CI_NAME_SUFFIX:-} 2>&1 | tail -30
docker compose down docker compose down
exit 1 exit 1
fi fi
@@ -229,15 +229,15 @@ print(f'{sum(trimmed)/len(trimmed):.1f}')
echo "" echo ""
echo " Ping via Tor onion service (directory mode, Sandbox 1):" echo " Ping via Tor onion service (directory mode, Sandbox 1):"
ping_series fips-dir-a "$NPUB_B" "A → B" ping_series fips-dir-a${FIPS_CI_NAME_SUFFIX:-} "$NPUB_B" "A → B"
ping_series fips-dir-b "$NPUB_A" "B → A" ping_series fips-dir-b${FIPS_CI_NAME_SUFFIX:-} "$NPUB_A" "B → A"
echo "" echo ""
# ── Phase 6: Log analysis ──────────────────────────────────────── # ── Phase 6: Log analysis ────────────────────────────────────────
echo "Phase 6: Log analysis" echo "Phase 6: Log analysis"
for node in fips-dir-a fips-dir-b; do for node in fips-dir-a${FIPS_CI_NAME_SUFFIX:-} fips-dir-b${FIPS_CI_NAME_SUFFIX:-}; do
panics=$(docker logs "$node" 2>&1 | grep -ci "panic" || true) panics=$(docker logs "$node" 2>&1 | grep -ci "panic" || true)
errors=$(docker logs "$node" 2>&1 | grep -ci "error" || true) errors=$(docker logs "$node" 2>&1 | grep -ci "error" || true)
onion=$(docker logs "$node" 2>&1 | grep -ci "onion" || true) onion=$(docker logs "$node" 2>&1 | grep -ci "onion" || true)
@@ -16,7 +16,7 @@ networks:
services: services:
tor-daemon: tor-daemon:
image: osminogin/tor-simple:latest image: osminogin/tor-simple:latest
container_name: tor-daemon container_name: tor-daemon${FIPS_CI_NAME_SUFFIX:-}
restart: "no" restart: "no"
volumes: volumes:
- ../common/torrc:/etc/tor/torrc:ro - ../common/torrc:/etc/tor/torrc:ro
@@ -32,7 +32,7 @@ services:
sysctls: sysctls:
- net.ipv6.conf.all.disable_ipv6=0 - net.ipv6.conf.all.disable_ipv6=0
restart: "no" restart: "no"
container_name: fips-tor-a container_name: fips-tor-a${FIPS_CI_NAME_SUFFIX:-}
hostname: fips-tor-a hostname: fips-tor-a
depends_on: depends_on:
- tor-daemon - tor-daemon
@@ -53,7 +53,7 @@ services:
sysctls: sysctls:
- net.ipv6.conf.all.disable_ipv6=0 - net.ipv6.conf.all.disable_ipv6=0
restart: "no" restart: "no"
container_name: fips-tor-b container_name: fips-tor-b${FIPS_CI_NAME_SUFFIX:-}
hostname: fips-tor-b hostname: fips-tor-b
depends_on: depends_on:
- tor-daemon - tor-daemon
@@ -87,7 +87,7 @@ echo ""
echo "Phase 2: Waiting for Tor daemon to bootstrap (up to ${MAX_WAIT_TOR}s)..." echo "Phase 2: Waiting for Tor daemon to bootstrap (up to ${MAX_WAIT_TOR}s)..."
elapsed=0 elapsed=0
while [ "$elapsed" -lt "$MAX_WAIT_TOR" ]; do while [ "$elapsed" -lt "$MAX_WAIT_TOR" ]; do
if docker logs tor-daemon 2>&1 | grep -q "Bootstrapped 100%"; then if docker logs tor-daemon${FIPS_CI_NAME_SUFFIX:-} 2>&1 | grep -q "Bootstrapped 100%"; then
echo " Tor bootstrapped after ${elapsed}s" echo " Tor bootstrapped after ${elapsed}s"
break break
fi fi
@@ -100,7 +100,7 @@ if [ "$elapsed" -ge "$MAX_WAIT_TOR" ]; then
echo " FAIL: Tor daemon did not bootstrap within ${MAX_WAIT_TOR}s" echo " FAIL: Tor daemon did not bootstrap within ${MAX_WAIT_TOR}s"
echo "" echo ""
echo "Tor daemon logs:" echo "Tor daemon logs:"
docker logs tor-daemon 2>&1 | tail -20 docker logs tor-daemon${FIPS_CI_NAME_SUFFIX:-} 2>&1 | tail -20
docker compose down docker compose down
exit 1 exit 1
fi fi
@@ -114,8 +114,8 @@ peers_a=0
peers_b=0 peers_b=0
elapsed=0 elapsed=0
while [ "$elapsed" -lt "$MAX_WAIT_PEER" ]; do while [ "$elapsed" -lt "$MAX_WAIT_PEER" ]; do
peers_a=$(count_connected_peers fips-tor-a) peers_a=$(count_connected_peers fips-tor-a${FIPS_CI_NAME_SUFFIX:-})
peers_b=$(count_connected_peers fips-tor-b) peers_b=$(count_connected_peers fips-tor-b${FIPS_CI_NAME_SUFFIX:-})
if [ "$peers_a" -ge 1 ] && [ "$peers_b" -ge 1 ]; then if [ "$peers_a" -ge 1 ] && [ "$peers_b" -ge 1 ]; then
echo " Both nodes have connected peers after ${elapsed}s (A: ${peers_a}, B: ${peers_b})" echo " Both nodes have connected peers after ${elapsed}s (A: ${peers_a}, B: ${peers_b})"
@@ -130,10 +130,10 @@ if [ "$peers_a" -lt 1 ] || [ "$peers_b" -lt 1 ]; then
echo " FAIL: Peers not established within ${MAX_WAIT_PEER}s" echo " FAIL: Peers not established within ${MAX_WAIT_PEER}s"
echo "" echo ""
echo "Node A logs (last 30 lines):" echo "Node A logs (last 30 lines):"
docker logs fips-tor-a 2>&1 | tail -30 docker logs fips-tor-a${FIPS_CI_NAME_SUFFIX:-} 2>&1 | tail -30
echo "" echo ""
echo "Node B logs (last 30 lines):" echo "Node B logs (last 30 lines):"
docker logs fips-tor-b 2>&1 | tail -30 docker logs fips-tor-b${FIPS_CI_NAME_SUFFIX:-} 2>&1 | tail -30
docker compose down docker compose down
exit 1 exit 1
fi fi
@@ -200,15 +200,15 @@ print(f'{sum(trimmed)/len(trimmed):.1f}')
echo "" echo ""
echo " Ping via Tor (routed through test-us01):" echo " Ping via Tor (routed through test-us01):"
ping_series fips-tor-a "$NPUB_B" "A → B" ping_series fips-tor-a${FIPS_CI_NAME_SUFFIX:-} "$NPUB_B" "A → B"
ping_series fips-tor-b "$NPUB_A" "B → A" ping_series fips-tor-b${FIPS_CI_NAME_SUFFIX:-} "$NPUB_A" "B → A"
echo "" echo ""
# ── Phase 5: Log analysis ──────────────────────────────────────── # ── Phase 5: Log analysis ────────────────────────────────────────
echo "Phase 5: Log analysis" echo "Phase 5: Log analysis"
for node in fips-tor-a fips-tor-b; do for node in fips-tor-a${FIPS_CI_NAME_SUFFIX:-} fips-tor-b${FIPS_CI_NAME_SUFFIX:-}; do
panics=$(docker logs "$node" 2>&1 | grep -ci "panic" || true) panics=$(docker logs "$node" 2>&1 | grep -ci "panic" || true)
errors=$(docker logs "$node" 2>&1 | grep -ci "error" || true) errors=$(docker logs "$node" 2>&1 | grep -ci "error" || true)
socks5=$(docker logs "$node" 2>&1 | grep -ci "socks5\|socks" || true) socks5=$(docker logs "$node" 2>&1 | grep -ci "socks5\|socks" || true)