testing: add mixed-version interop harness

testing/README.md

@@ -54,3 +54,11 @@ testing. Scenarios are
 defined in YAML and executed via a Python harness that manages the full
 lifecycle: topology generation, Docker orchestration, fault scheduling,
 log collection, and analysis.
+
+### [interop/](interop/) -- Mixed-Version Interop Harness
+
+On-demand harness that runs an N-node full mesh from a node-spec where
+each node can run a different build of the FIPS daemon, then attributes
+every FMP/FSP/rekey/connectivity failure to a specific version pair
+(same-version vs MIXED). Used to catch interop regressions between
+builds, not as a per-commit CI gate; not part of `ci-local.sh`.

testing/interop/.gitignore

@@ -0,0 +1,10 @@
+# Generated per-node configs, generated compose, manifests
+# (generated-configs/ is ignored wholesale, so docker-compose.generated.yml
+# and nodes.env are covered).
+generated-configs/
+
+# Per-ref worktree + build-context scratch space (build-images.sh)
+.build/
+
+# Netem stress-loop artifacts (interop-stress.sh)
+.stress-runs/

testing/interop/README.md

@@ -0,0 +1,292 @@
+# Mixed-Version Interop Test Harness
+
+A CI lab harness for **mixed-version interoperability** testing. It runs an
+N-node full mesh where nodes run **different builds** of the FIPS daemon, and
+checks that every pair of versions interoperates — FMP link, FSP session,
+connectivity, and rekey survival — without a failure a same-version pair
+would not have.
+
+## What it tests
+
+The static/rekey suites bake one binary set for all nodes, so they only ever
+test a version against itself. This harness breaks that assumption: each node
+runs an image built from its own git ref. The harness then looks for
+**interop regressions** — places where two *different* versions fail to
+interoperate:
+
+- FMP handshake failures across versions
+- `unknown FMP version` drops
+- FSP / FMP AEAD decrypt failures
+- replay storms / excessive-decrypt-failure removals
+- link or session teardowns
+- asymmetric connectivity drops
+- rekey (FMP link + FSP session) that completes within a version but stalls
+  or breaks across versions
+
+Every failure is **attributed to a specific version pair**, classified
+same-version vs MIXED. The summary states whether failures are
+mixed-version-only (a genuine interop regression), both mixed and same
+(general instability), or same-version-only (a build is unstable even
+against itself).
+
+## The node-spec
+
+The harness is parameterized by a **node-spec**: a multiset of image slots,
+of size >= 2. Each slot is one of `a`, `b`, `c`, and the same slot may appear
+more than once. The slots resolve to the three images built by
+`build-images.sh`:
+
+| Slot | Role                | Intended ref                     |
+| ---- | ------------------- | -------------------------------- |
+| `a`  | version under test  | the branch tip / commit to vet   |
+| `b`  | parent / comparison | parent commit on the same branch |
+| `c`  | release baseline    | latest release tag (`v0.3.0`)    |
+
+`build-images.sh` is **unchanged** — it always builds exactly three images
+from three refs. A node-spec like `a a b c` resolves to the *same* three
+images; only the mesh topology grows.
+
+### Node identity vs image slot
+
+Node identity and image slot are **separate**:
+
+- A spec entry is a slot letter.
+- Node id = `<slot><ordinal>`, ordinal counting occurrences of that slot,
+  1-based.
+- Container name = `fips-interop-<nodeid>`.
+- IPv4 = `172.30.0.1<index>`, index = 0-based position in the spec
+  (`.10`, `.11`, `.12`, ...).
+- Each node maps to its image slot: `a1`, `a2` → `fips-interop:a`.
+- A pair is **same-version** iff the two nodes' slots resolve to the same
+  built SHA (read from `.build/refs.env`); else **MIXED**.
+
+Two same-slot nodes get **distinct identities** — `derive_keys.py` is keyed
+by node id, so `a1` and `a2` are different npubs even though they run the
+same binary.
+
+### Example specs
+
+| Node-spec | Node ids        | Pairs                                            |
+| --------- | --------------- | ------------------------------------------------ |
+| `a b c`   | `a1 b1 c1`      | 3 pairs, all MIXED — today's triangle (default). |
+| `a a b c` | `a1 a2 b1 c1`   | 6 pairs: `a1↔a2` **same** + 5 MIXED.             |
+| `a a a`   | `a1 a2 a3`      | 3 pairs, all **same** — a same-version flake rig.|
+
+### Why the `a a b c` control topology matters
+
+The triangle `a b c` has *no same-version pair* — every pair is mixed, so
+under packet loss you cannot tell an interop regression from generic loss
+noise. The `a a b c` spec adds a **control arm**: the `a1↔a2` pair runs
+identical binaries. Under a netem stress loop:
+
+- a failure on a mixed pair the control pair does **not** share → an interop
+  regression;
+- a failure both the mixed pairs and the control pair share → loss-induced
+  instability, not version-specific.
+
+That control pair is what makes a stress run *interpretable*. It is the
+default node-spec for `interop-stress.sh`.
+
+The `a a a` spec is the degenerate case: all-same-version, used purely as a
+**flake rig** — exercising a single build against itself under loss to find
+loss-induced instability with no version variable at all.
+
+## Files
+
+| File                    | Purpose                                                        |
+| ----------------------- | -------------------------------------------------------------- |
+| `build-images.sh`       | Build `fips-interop:a/b/c`, one Docker image per git ref.       |
+| `generate-configs.sh`   | Generate per-node configs, the generated compose, manifests.    |
+| `interop-test.sh`       | Test driver: bring up, converge, rekey, analyze, attribute.     |
+| `interop-stress.sh`     | Netem stress loop: N reps, pass rate, mixed-vs-same attribution.|
+| `README.md`             | This document.                                                 |
+
+Generated at runtime: `generated-configs/` (per-node configs +
+`docker-compose.generated.yml` + `nodes.env` + `npubs.env`), `.build/`,
+`.stress-runs/`. The root for these three is selected by the
+`FIPS_INTEROP_RUNS_DIR` environment variable — see
+[Scratch directory location](#scratch-directory-location) below.
+
+The static `docker-compose.yml` is gone — the compose file is now generated
+per node-spec into `generated-configs/docker-compose.generated.yml`.
+
+## How per-node images work
+
+`build-images.sh <ref-a> <ref-b> <ref-c>` (unchanged — always three refs):
+
+1. For each ref, `git worktree add --detach` a temp checkout of the repo.
+2. `cargo build --release` the four binaries (`fips`, `fipsctl`, `fipstop`,
+   `fips-gateway`) in that worktree.
+3. Copy the binaries into a build context alongside the shared
+   `testing/docker/Dockerfile` (and `entrypoint.sh`, `resolv.conf`).
+4. `docker build` it, tagging `fips-interop:<slot>` and labelling the image
+   with its ref + short SHA.
+5. Remove the temp worktree (done per-ref so peak disk stays at one worktree).
+
+It also writes `.build/refs.env`, recording each slot's ref and SHA. The
+driver reads it to know which pairs are mixed-version. (If absent, it falls
+back to the image labels.)
+
+## How to run it
+
+### Build the images (once per ref set)
+
+```bash
+cd /dpool/src/clabs/nostr/fips
+bash testing/interop/build-images.sh <ref-a> <ref-b> <ref-c>
+```
+
+Example: `A` = tip of `fix/fsp-rekey-overlapping-epoch`, `B` = `maint`,
+`C` = release tag `v0.3.0`:
+
+```bash
+bash testing/interop/build-images.sh fix/fsp-rekey-overlapping-epoch maint v0.3.0
+```
+
+### Run a single mesh
+
+```bash
+bash testing/interop/interop-test.sh [node-spec...]
+```
+
+`node-spec` defaults to `a b c` (the original triangle). Examples:
+
+```bash
+bash testing/interop/interop-test.sh                # a b c   — 3-node triangle
+bash testing/interop/interop-test.sh a a b c        # 4-node, one control pair
+bash testing/interop/interop-test.sh a a a          # 3-node same-version flake rig
+```
+
+The driver regenerates configs automatically whenever the requested
+node-spec differs from the one on disk, so changing the spec just works.
+A run takes a few minutes (driven by `REKEY_AFTER_SECS`, default 35, times
+two rekey cycles).
+
+### Run the netem stress loop
+
+```bash
+FIPS_INTEROP_NETEM="delay 10ms 5ms 25% loss 2%" \
+  bash testing/interop/interop-stress.sh [--reps N] [node-spec...]
+```
+
+- `--reps N` — repetitions (default 10).
+- `node-spec` — default `a a b c` (the control topology).
+- Reps run **serially** — the harness uses fixed container names and a fixed
+  Docker network, so two reps must never overlap.
+- If `FIPS_INTEROP_NETEM` is unset the script warns (a stress run normally
+  wants netem) but still runs a clean baseline loop.
+
+Each rep invokes `interop-test.sh` with netem set and captures its full
+output and exit code; a rep passes iff `interop-test.sh` exits 0. Artifacts
+land in `testing/interop/.stress-runs/<UTC-timestamp>/`:
+
+- `rep-NN/driver.log` — full driver output for every rep.
+- `rep-NN/docker-<container>.log` — per-container `docker logs` for **failed**
+  reps only.
+- `summary.txt` — the aggregate report.
+
+The aggregate report gives reps run, passed/failed counts and an integer
+pass rate, then tallies — across all failed reps — connectivity failures by
+pair kind (mixed vs same), and a verdict:
+
+- **mixed pairs only, never the control pair** → interop-regression signal;
+- **both mixed and same pairs** → loss-induced general instability;
+- **same-version control pair only** → the build is unstable against itself.
+
+`interop-stress.sh` exits **non-zero only** for the interop-regression
+signal. A sub-100% pass rate under loss is expected and is not by itself a
+failure, so every other outcome exits 0.
+
+### Options
+
+| Variable                     | Effect                                            |
+| ---------------------------- | ------------------------------------------------- |
+| `FIPS_INTEROP_NETEM`         | tc-netem string applied to each container's eth0, e.g. `"delay 10ms 5ms 25% loss 1%"`. Passed through `interop-stress.sh` to `interop-test.sh`. |
+| `REKEY_AFTER_SECS`           | Rekey interval for generated configs (default 35).|
+| `FIPS_INTEROP_KEEP_UP`       | `1` = leave containers running after the test.    |
+| `FIPS_INTEROP_KEEP_WORKTREES`| `1` = keep `build-images.sh` worktrees (debug).   |
+| `FIPS_INTEROP_RUNS_DIR`      | Root for the three scratch dirs — see [Scratch directory location](#scratch-directory-location). |
+
+The netem hook reuses the flake-lab mechanism (`docker exec ... tc qdisc` on
+each container's `eth0`) — host-side bridge qdisc does not shape
+inter-container port-to-port traffic, so the impairment must live inside the
+containers.
+
+### Scratch directory location
+
+The three scratch dirs the harness writes — `.build/` (per-ref build
+contexts and `refs.env`), `generated-configs/` (per-node configs +
+generated compose + manifests), and `.stress-runs/` (stress-loop
+artefacts) — are rooted under `FIPS_INTEROP_RUNS_DIR` when that
+environment variable is set. With
+
+```bash
+export FIPS_INTEROP_RUNS_DIR=/var/lib/fips-interop
+```
+
+all three land under `/var/lib/fips-interop/`, and the source tree
+stays clean.
+
+When `FIPS_INTEROP_RUNS_DIR` is unset, each harness script falls back
+to writing under `testing/interop/` itself and prints a one-line
+stderr warning naming the variable. The in-tree paths are
+`.gitignore`d, so accidentally running without the variable does not
+dirty the checkout — but pointing the variable outside the source
+tree is recommended, so lab runs do not interleave with the source
+working copy at all.
+
+## How to read the output
+
+The driver runs six phases:
+
+| Phase | Check                                                            |
+| ----- | ---------------------------------------------------------------- |
+| 0     | Bring up the mesh (+ optional netem).                            |
+| 1     | All nodes reach N-1 authenticated peers; all directed pairs ping over `fips0` (the definitive FSP-session check). |
+| 2     | First FMP rekey cutover completes within the timeout.            |
+| 3     | All pairs still ping after the first rekey.                      |
+| 4     | Wait out a second rekey cycle.                                   |
+| 5     | All pairs still ping after the second rekey.                     |
+| 6     | Per-node / per-pair interop log analysis.                        |
+
+Phase 6 is the interop-specific part. It reports:
+
+- **Global health** — panics, `ERROR` lines, `unknown FMP version` drops,
+  link teardowns, decrypt failures, handshake failures, rekey-msg2 failures.
+  Any non-zero count is broken down per node, attributed to a specific build.
+- **Rekey machinery exercised** — both FMP and FSP rekey cutovers fired.
+- **Per-pair interop summary** — each unordered pair, classified
+  same-version vs MIXED, with whether it stayed healthy through the run.
+
+The final verdict lists every failure attributed to a specific
+`x[ref@sha] <-> y[ref@sha]` pair or build, then states the attribution:
+
+- **mixed-version only** → a genuine interop regression.
+- **both mixed and same** → general instability, not version-specific.
+- **same-version only** → a build is unstable even against itself.
+
+Exit code is `0` only if every check passed and no per-pair failure was
+recorded; non-zero otherwise, with a diagnostic dump (peer/link snapshots and
+interop-relevant log tails for all nodes).
+
+## CI integration
+
+The harness is self-contained and slots into `.github/workflows/ci.yml`
+alongside the existing `integration` matrix suites. A future matrix entry
+would, per push to a PR branch:
+
+1. Resolve the three refs — e.g. `A = github.sha`,
+   `B = git rev-parse github.sha^`, `C = $(git describe --tags --abbrev=0)` or
+   a pinned release tag.
+2. `bash testing/interop/build-images.sh "$A" "$B" "$C"`.
+3. `bash testing/interop/interop-test.sh a a b c` for the control topology,
+   or `interop-stress.sh` for a loss sweep.
+4. On failure, upload the diagnostic dump (or `.stress-runs/`) as an artifact.
+
+The three `cargo build --release` passes are the cost driver; on a CI runner
+this suite is heavier than the single-image suites. Reasonable options are to
+run it only on release branches / tags, gate it behind a label, or cache the
+`fips-interop:c` (release) image since the release tag rarely moves.
+
+Until then the harness runs on demand locally — the same way the flake-lab is
+used today.

testing/interop/build-images.sh

@@ -0,0 +1,232 @@
+#!/bin/bash
+# Mixed-version interop harness: build three Docker images, one per git ref.
+#
+# Given three git refs, this script builds the FIPS daemon from each ref and
+# produces three distinct Docker images:
+#
+#   fips-interop:a   <- ref A  ("version under test")
+#   fips-interop:b   <- ref B  ("parent")
+#   fips-interop:c   <- ref C  ("release")
+#
+# It deliberately does NOT reuse the fips-test:latest tag — that tag is owned
+# by the flake-lab and the static suite, which bake a single binary set for
+# all nodes. This harness needs a different binary set per node, so each ref
+# gets its own tag.
+#
+# Mechanism:
+#   1. For each ref, `git worktree add --detach` a temp checkout of this repo.
+#   2. `cargo build --release` the four binaries in that worktree.
+#   3. Copy fips/fipsctl/fipstop/fips-gateway into a per-ref build context.
+#   4. `docker build` that context with the testing/docker Dockerfile, tagging
+#      fips-interop:<slot>.
+#   5. Remove the temp worktree.
+#
+# The harness reuses the testing/docker/Dockerfile (and its entrypoint.sh,
+# resolv.conf), so the runtime environment is identical to the static and
+# rekey suites — only the daemon binaries differ between the three images.
+#
+# Idempotent and safe to re-run: each invocation rebuilds all three images
+# from scratch and cleans up its worktrees, including stale worktrees from a
+# previous interrupted run.
+#
+# Usage:
+#   ./build-images.sh <ref-a> <ref-b> <ref-c>
+#
+# Example (the motivating run):
+#   ./build-images.sh fix/fsp-rekey-overlapping-epoch 79975d72 v0.3.0
+#
+# Environment:
+#   FIPS_INTEROP_KEEP_WORKTREES=1   Keep temp worktrees after build (debug).
+#   CARGO_BUILD_JOBS=N              Passed through to cargo if set.
+#   FIPS_INTEROP_RUNS_DIR=DIR       Root for harness scratch dirs (.build/,
+#                                   .stress-runs/, generated-configs/). When
+#                                   unset, falls back to in-tree paths under
+#                                   testing/interop/ and prints a warning to
+#                                   stderr; set it to a path outside the
+#                                   source tree to keep generated artefacts
+#                                   out of the checkout.
+
+set -euo pipefail
+
+SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
+REPO_ROOT="$(cd "$SCRIPT_DIR/../.." && pwd)"
+DOCKER_CTX_SRC="$REPO_ROOT/testing/docker"
+
+# ── Scratch-dir root ─────────────────────────────────────────────────
+#
+# FIPS_INTEROP_RUNS_DIR controls where the harness writes its scratch
+# directories (.build/, .stress-runs/, generated-configs/). When unset
+# we fall back to in-tree paths under testing/interop/ and warn the
+# operator, so the warning fires exactly once per invocation. When a
+# parent script has already warned it exports _FIPS_INTEROP_WARNED=1
+# to suppress duplicate warnings in child scripts.
+if [[ -n "${FIPS_INTEROP_RUNS_DIR:-}" ]]; then
+    RUNS_BASE="$FIPS_INTEROP_RUNS_DIR"
+    mkdir -p "$RUNS_BASE"
+else
+    RUNS_BASE="$SCRIPT_DIR"
+    if [[ -z "${_FIPS_INTEROP_WARNED:-}" ]]; then
+        echo >&2 "WARNING: FIPS_INTEROP_RUNS_DIR not set; harness output will be written under the source tree at $RUNS_BASE. Set FIPS_INTEROP_RUNS_DIR to a path outside the source tree to avoid this."
+        export _FIPS_INTEROP_WARNED=1
+    fi
+fi
+
+# Per-ref tag slots. Order matters: slot[i] <- ref[i].
+SLOTS=(a b c)
+
+# ── Args ─────────────────────────────────────────────────────────────
+
+if [ "$#" -ne 3 ]; then
+    echo "Usage: $0 <ref-a> <ref-b> <ref-c>" >&2
+    echo "" >&2
+    echo "  ref-a  version under test       -> fips-interop:a" >&2
+    echo "  ref-b  parent / comparison      -> fips-interop:b" >&2
+    echo "  ref-c  release baseline         -> fips-interop:c" >&2
+    exit 1
+fi
+
+REFS=("$1" "$2" "$3")
+
+# ── Preflight ────────────────────────────────────────────────────────
+
+if ! docker info >/dev/null 2>&1; then
+    echo "ERROR: Docker daemon is not reachable" >&2
+    exit 2
+fi
+
+if ! command -v cargo >/dev/null 2>&1; then
+    echo "ERROR: cargo not found on PATH" >&2
+    exit 2
+fi
+
+for ref in "${REFS[@]}"; do
+    if ! git -C "$REPO_ROOT" rev-parse --verify --quiet "${ref}^{commit}" >/dev/null; then
+        echo "ERROR: git ref '$ref' does not resolve to a commit" >&2
+        exit 2
+    fi
+done
+
+# ── Worktree + build-context scratch space ───────────────────────────
+
+WORK_BASE="$RUNS_BASE/.build"
+mkdir -p "$WORK_BASE"
+
+# Track worktree paths for cleanup.
+CREATED_WORKTREES=()
+
+cleanup() {
+    if [ "${FIPS_INTEROP_KEEP_WORKTREES:-}" = "1" ]; then
+        echo ""
+        echo "FIPS_INTEROP_KEEP_WORKTREES=1 — leaving worktrees in place:"
+        for wt in "${CREATED_WORKTREES[@]:-}"; do
+            [ -n "$wt" ] && echo "  $wt"
+        done
+        return
+    fi
+    for wt in "${CREATED_WORKTREES[@]:-}"; do
+        [ -n "$wt" ] || continue
+        if [ -d "$wt" ]; then
+            git -C "$REPO_ROOT" worktree remove --force "$wt" 2>/dev/null \
+                || rm -rf "$wt"
+        fi
+    done
+    git -C "$REPO_ROOT" worktree prune 2>/dev/null || true
+}
+trap cleanup EXIT
+
+# Prune any stale interop worktrees from a prior interrupted run before we
+# start, so re-running the script is clean.
+git -C "$REPO_ROOT" worktree prune 2>/dev/null || true
+
+# ── Per-ref build ────────────────────────────────────────────────────
+
+build_one() {
+    local slot="$1"
+    local ref="$2"
+    local sha
+    sha="$(git -C "$REPO_ROOT" rev-parse --short "$ref")"
+
+    echo ""
+    echo "=== Building slot '$slot'  ref='$ref'  sha=$sha ==="
+
+    local wt="$WORK_BASE/worktree-$slot"
+    local ctx="$WORK_BASE/ctx-$slot"
+
+    # Fresh worktree per build (remove a stale one first).
+    if [ -d "$wt" ]; then
+        git -C "$REPO_ROOT" worktree remove --force "$wt" 2>/dev/null \
+            || rm -rf "$wt"
+    fi
+    git -C "$REPO_ROOT" worktree add --detach "$wt" "$ref"
+    CREATED_WORKTREES+=("$wt")
+
+    # Build the four binaries from this ref.
+    local cargo_jobs_arg=()
+    if [ -n "${CARGO_BUILD_JOBS:-}" ]; then
+        cargo_jobs_arg=(--jobs "$CARGO_BUILD_JOBS")
+    fi
+    (
+        cd "$wt"
+        cargo build --release "${cargo_jobs_arg[@]}" \
+            --bin fips --bin fipsctl --bin fipstop --bin fips-gateway
+    )
+
+    # Assemble a build context: the testing/docker Dockerfile + support
+    # files from the MAIN checkout, with the per-ref binaries layered in.
+    rm -rf "$ctx"
+    mkdir -p "$ctx"
+    cp "$DOCKER_CTX_SRC/Dockerfile"    "$ctx/Dockerfile"
+    cp "$DOCKER_CTX_SRC/entrypoint.sh" "$ctx/entrypoint.sh"
+    cp "$DOCKER_CTX_SRC/resolv.conf"   "$ctx/resolv.conf"
+    for bin in fips fipsctl fipstop fips-gateway; do
+        cp "$wt/target/release/$bin" "$ctx/$bin"
+        chmod +x "$ctx/$bin"
+    done
+
+    docker build \
+        --label "fips.interop.slot=$slot" \
+        --label "fips.interop.ref=$ref" \
+        --label "fips.interop.sha=$sha" \
+        -t "fips-interop:$slot" \
+        "$ctx"
+
+    # The worktree is large (target/ dir); remove it now rather than at
+    # exit so peak disk use stays at one worktree, not three.
+    if [ "${FIPS_INTEROP_KEEP_WORKTREES:-}" != "1" ]; then
+        git -C "$REPO_ROOT" worktree remove --force "$wt" 2>/dev/null \
+            || rm -rf "$wt"
+        rm -rf "$ctx"
+    fi
+
+    echo "=== Built fips-interop:$slot  ($ref @ $sha) ==="
+}
+
+# Record the ref->sha mapping so the driver and README can report what
+# actually ran. Written before the builds so a partial failure still
+# leaves a breadcrumb.
+MANIFEST="$WORK_BASE/refs.env"
+{
+    echo "# Generated by build-images.sh on $(date -u +%Y-%m-%dT%H:%M:%SZ)"
+    for i in 0 1 2; do
+        slot="${SLOTS[$i]}"
+        ref="${REFS[$i]}"
+        sha="$(git -C "$REPO_ROOT" rev-parse --short "$ref")"
+        upper="$(echo "$slot" | tr '[:lower:]' '[:upper:]')"
+        echo "INTEROP_REF_${upper}=$ref"
+        echo "INTEROP_SHA_${upper}=$sha"
+    done
+} > "$MANIFEST"
+
+for i in 0 1 2; do
+    build_one "${SLOTS[$i]}" "${REFS[$i]}"
+done
+
+echo ""
+echo "=== All three interop images built ==="
+docker image ls --filter 'reference=fips-interop' \
+    --format 'table {{.Repository}}:{{.Tag}}\t{{.ID}}\t{{.CreatedSince}}'
+echo ""
+echo "Ref manifest: $MANIFEST"
+cat "$MANIFEST"
+echo ""
+echo "Next: bash testing/interop/interop-test.sh"

testing/interop/generate-configs.sh

@@ -0,0 +1,303 @@
+#!/bin/bash
+# Mixed-version interop harness: generate the per-node configs for a
+# node-spec.
+#
+# A NODE-SPEC is a multiset of image slots (`a`, `b`, `c`), of size >= 2.
+# Each spec entry is one node; the same slot may appear more than once.
+# The slots resolve to the three Docker images built by build-images.sh:
+#
+#   a -> fips-interop:a   ("version under test")
+#   b -> fips-interop:b   ("parent / comparison")
+#   c -> fips-interop:c   ("release baseline")
+#
+# Node identity vs image slot are SEPARATE:
+#   - A spec entry is a slot letter.
+#   - Node id = <slot><ordinal>, ordinal counting occurrences of that
+#     slot, 1-based. Spec `a a b c` -> node ids `a1 a2 b1 c1`.
+#   - Container name = fips-interop-<nodeid>.
+#   - IPv4 = 172.30.0.1<index>, index = 0-based position in the spec
+#     (10, 11, 12, 13, ...).
+#
+# Topology is a FULL MESH: every node auto-connects to every other node.
+# Node identities are derived deterministically via the shared
+# derive_keys.py helper, keyed by a unique per-node string so that two
+# nodes of the same slot (a1, a2) still get DISTINCT identities.
+#
+# Output goes to <runs-base>/generated-configs/ (see FIPS_INTEROP_RUNS_DIR
+# under Environment, below):
+#   <nodeid>.yaml                  per-node daemon config (one per node)
+#   docker-compose.generated.yml   generated compose, one service per node
+#   nodes.env                      manifest for the driver to source
+#   npubs.env                      NPUB_<NODEID> map (per-node npubs)
+#
+# Generation is deterministic and idempotent: same node-spec in -> same
+# files out.
+#
+# Usage:
+#   ./generate-configs.sh [node-spec...]
+#
+#   node-spec   space-separated slot letters, default `a b c`.
+#
+# Environment:
+#   REKEY_AFTER_SECS       FSP/FMP rekey interval (default 35, matching
+#                          the static rekey suite).
+#   FIPS_INTEROP_RUNS_DIR  Root for harness scratch dirs (.build/,
+#                          .stress-runs/, generated-configs/). When
+#                          unset, falls back to in-tree paths under
+#                          testing/interop/ and prints a warning to
+#                          stderr; set it to a path outside the source
+#                          tree to keep generated artefacts out of the
+#                          checkout.
+
+set -euo pipefail
+
+SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
+REPO_ROOT="$(cd "$SCRIPT_DIR/../.." && pwd)"
+DERIVE_KEYS="$SCRIPT_DIR/../lib/derive_keys.py"
+
+# ── Scratch-dir root ─────────────────────────────────────────────────
+#
+# FIPS_INTEROP_RUNS_DIR controls where the harness writes its scratch
+# directories (.build/, .stress-runs/, generated-configs/). When unset
+# we fall back to in-tree paths under testing/interop/ and warn the
+# operator, so the warning fires exactly once per invocation. When a
+# parent script has already warned it exports _FIPS_INTEROP_WARNED=1
+# to suppress duplicate warnings in child scripts.
+if [[ -n "${FIPS_INTEROP_RUNS_DIR:-}" ]]; then
+    RUNS_BASE="$FIPS_INTEROP_RUNS_DIR"
+    mkdir -p "$RUNS_BASE"
+else
+    RUNS_BASE="$SCRIPT_DIR"
+    if [[ -z "${_FIPS_INTEROP_WARNED:-}" ]]; then
+        echo >&2 "WARNING: FIPS_INTEROP_RUNS_DIR not set; harness output will be written under the source tree at $RUNS_BASE. Set FIPS_INTEROP_RUNS_DIR to a path outside the source tree to avoid this."
+        export _FIPS_INTEROP_WARNED=1
+    fi
+fi
+
+OUT_DIR="$RUNS_BASE/generated-configs"
+
+REKEY_AFTER_SECS="${REKEY_AFTER_SECS:-35}"
+MESH_NAME="fips-interop-mesh"
+UDP_PORT=2121
+SUBNET_PREFIX="172.30.0.1"     # node N gets ${SUBNET_PREFIX}<index>
+
+# ── Args: the node-spec ──────────────────────────────────────────────
+
+SPEC=("$@")
+if [ "${#SPEC[@]}" -eq 0 ]; then
+    SPEC=(a b c)
+fi
+
+if [ "${#SPEC[@]}" -lt 2 ]; then
+    echo "ERROR: node-spec must have at least 2 nodes (got ${#SPEC[@]})" >&2
+    echo "Usage: $0 [node-spec...]   e.g. $0 a a b c" >&2
+    exit 1
+fi
+
+for slot in "${SPEC[@]}"; do
+    case "$slot" in
+        a|b|c) ;;
+        *)
+            echo "ERROR: invalid slot '$slot' — must be one of a, b, c" >&2
+            exit 1
+            ;;
+    esac
+done
+
+# ── Derive node ids, slots, containers, IPs from the spec ────────────
+#
+# NODE_IDS preserves spec order. NODE_SLOT/NODE_IP/NODE_CTR are keyed by
+# node id.
+
+NODE_IDS=()
+declare -A NODE_SLOT NODE_IP NODE_CTR
+declare -A SLOT_COUNT=([a]=0 [b]=0 [c]=0)
+
+index=0
+for slot in "${SPEC[@]}"; do
+    SLOT_COUNT[$slot]=$(( SLOT_COUNT[$slot] + 1 ))
+    nid="${slot}${SLOT_COUNT[$slot]}"
+    NODE_IDS+=("$nid")
+    NODE_SLOT[$nid]="$slot"
+    NODE_CTR[$nid]="fips-interop-$nid"
+    NODE_IP[$nid]="${SUBNET_PREFIX}${index}"
+    index=$(( index + 1 ))
+done
+
+mkdir -p "$OUT_DIR"
+
+# Clear stale per-node configs from a previous (differently-shaped) spec
+# so generation is idempotent — a smaller spec must not leave orphans.
+rm -f "$OUT_DIR"/*.yaml
+
+# ── Phase 1: derive identities ───────────────────────────────────────
+#
+# Key the derivation by node id (a1, a2, ...) so two same-slot nodes get
+# distinct identities.
+
+declare -A NSEC NPUB
+for nid in "${NODE_IDS[@]}"; do
+    keys="$(python3 "$DERIVE_KEYS" "$MESH_NAME" "$nid")"
+    NSEC[$nid]="$(echo "$keys" | sed -n 's/^nsec=//p')"
+    NPUB[$nid]="$(echo "$keys" | sed -n 's/^npub=//p')"
+done
+
+# ── Phase 2: emit per-node configs ───────────────────────────────────
+
+emit_peer_block() {
+    # Args: peer node-id
+    local pid="$1"
+    cat <<EOF
+  - npub: "${NPUB[$pid]}"
+    alias: "$pid"
+    addresses:
+      - transport: udp
+        addr: "${NODE_IP[$pid]}:$UDP_PORT"
+    connect_policy: auto_connect
+EOF
+}
+
+for nid in "${NODE_IDS[@]}"; do
+    cfg="$OUT_DIR/$nid.yaml"
+    {
+        echo "# FIPS interop mesh — node $nid (slot ${NODE_SLOT[$nid]})"
+        echo "# Identity: ${NPUB[$nid]}"
+        echo "# Generated by testing/interop/generate-configs.sh"
+        echo ""
+        echo "node:"
+        echo "  identity:"
+        echo "    nsec: \"${NSEC[$nid]}\""
+        echo "  rekey:"
+        echo "    enabled: true"
+        echo "    after_secs: $REKEY_AFTER_SECS"
+        echo "    after_messages: 65536"
+        echo ""
+        echo "tun:"
+        echo "  enabled: true"
+        echo "  name: fips0"
+        echo "  mtu: 1280"
+        echo ""
+        echo "dns:"
+        echo "  enabled: true"
+        echo ""
+        echo "transports:"
+        echo "  udp:"
+        echo "    bind_addr: \"0.0.0.0:$UDP_PORT\""
+        echo "    mtu: 1472"
+        echo ""
+        echo "peers:"
+        for pid in "${NODE_IDS[@]}"; do
+            [ "$pid" = "$nid" ] && continue
+            emit_peer_block "$pid"
+        done
+    } > "$cfg"
+    echo "  generated $cfg"
+done
+
+# ── Phase 3: emit the generated docker-compose file ──────────────────
+#
+# One service per node, image pinned to the node's slot. This supersedes
+# the static docker-compose.yml.
+
+COMPOSE_FILE="$OUT_DIR/docker-compose.generated.yml"
+{
+    echo "# Mixed-version interop harness: generated compose, one service per node."
+    echo "#"
+    echo "# Generated by testing/interop/generate-configs.sh — DO NOT EDIT."
+    echo "# Node-spec: ${SPEC[*]}"
+    echo "#"
+    echo "# Each service is pinned to its node's image slot, so two nodes of the"
+    echo "# same slot (e.g. a1, a2) run the SAME daemon build. Netem is applied"
+    echo "# by the driver via 'docker exec ... tc qdisc' on each container eth0."
+    echo ""
+    echo "networks:"
+    echo "  fips-interop-net:"
+    echo "    driver: bridge"
+    echo "    ipam:"
+    echo "      config:"
+    echo "        - subnet: 172.30.0.0/24"
+    echo ""
+    echo "x-fips-interop-common: &fips-interop-common"
+    echo "  cap_add:"
+    echo "    - NET_ADMIN"
+    echo "  devices:"
+    echo "    - /dev/net/tun:/dev/net/tun"
+    echo "  sysctls:"
+    echo "    - net.ipv6.conf.all.disable_ipv6=0"
+    echo "  restart: \"no\""
+    echo "  environment:"
+    echo "    - RUST_LOG=info,fips::node::handlers::rekey=debug,fips::node::handlers::handshake=debug"
+    echo ""
+    echo "services:"
+    for nid in "${NODE_IDS[@]}"; do
+        slot="${NODE_SLOT[$nid]}"
+        echo "  $nid:"
+        echo "    <<: *fips-interop-common"
+        echo "    image: fips-interop:$slot"
+        echo "    container_name: ${NODE_CTR[$nid]}"
+        echo "    hostname: $nid"
+        echo "    volumes:"
+        echo "      - $REPO_ROOT/testing/docker/resolv.conf:/etc/resolv.conf:ro"
+        echo "      - ./$nid.yaml:/etc/fips/fips.yaml:ro"
+        echo "    networks:"
+        echo "      fips-interop-net:"
+        echo "        ipv4_address: ${NODE_IP[$nid]}"
+    done
+} > "$COMPOSE_FILE"
+echo "  generated $COMPOSE_FILE"
+
+# ── Phase 4: nodes.env manifest ──────────────────────────────────────
+#
+# The driver sources this. NODE_IDS is the canonical ordered node list;
+# the per-node maps are emitted as space-separated key:value pairs so the
+# driver can rebuild associative arrays without re-deriving anything.
+
+MANIFEST="$OUT_DIR/nodes.env"
+{
+    echo "# Generated by testing/interop/generate-configs.sh"
+    echo "# Mesh name: $MESH_NAME"
+    echo "# Node-spec: ${SPEC[*]}"
+    echo "INTEROP_SPEC=\"${SPEC[*]}\""
+    echo "INTEROP_NODE_IDS=\"${NODE_IDS[*]}\""
+    # Per-node maps, one var each, space-separated 'nodeid:value' tokens.
+    {
+        printf 'INTEROP_NODE_SLOTS="'
+        for nid in "${NODE_IDS[@]}"; do printf '%s:%s ' "$nid" "${NODE_SLOT[$nid]}"; done
+        printf '"\n'
+    }
+    {
+        printf 'INTEROP_NODE_CONTAINERS="'
+        for nid in "${NODE_IDS[@]}"; do printf '%s:%s ' "$nid" "${NODE_CTR[$nid]}"; done
+        printf '"\n'
+    }
+    {
+        printf 'INTEROP_NODE_IPS="'
+        for nid in "${NODE_IDS[@]}"; do printf '%s:%s ' "$nid" "${NODE_IP[$nid]}"; done
+        printf '"\n'
+    }
+    {
+        printf 'INTEROP_NODE_NPUBS="'
+        for nid in "${NODE_IDS[@]}"; do printf '%s:%s ' "$nid" "${NPUB[$nid]}"; done
+        printf '"\n'
+    }
+} > "$MANIFEST"
+echo "  generated $MANIFEST"
+
+# ── Phase 5: npubs.env — per-node npub map ───────────────────────────
+
+ENV_FILE="$OUT_DIR/npubs.env"
+{
+    echo "# Generated by testing/interop/generate-configs.sh"
+    echo "# Mesh name: $MESH_NAME"
+    for nid in "${NODE_IDS[@]}"; do
+        upper="$(echo "$nid" | tr '[:lower:]' '[:upper:]')"
+        echo "NPUB_${upper}=${NPUB[$nid]}"
+    done
+} > "$ENV_FILE"
+echo "  generated $ENV_FILE"
+
+echo ""
+echo "Mesh configs ready (spec='${SPEC[*]}', rekey.after_secs=$REKEY_AFTER_SECS):"
+for nid in "${NODE_IDS[@]}"; do
+    echo "  $nid  slot ${NODE_SLOT[$nid]}  ${NODE_IP[$nid]}  ${NPUB[$nid]}"
+done

testing/interop/interop-stress.sh

@@ -0,0 +1,299 @@
+#!/bin/bash
+# Mixed-version interop NETEM STRESS LOOP.
+#
+# Runs interop-test.sh for N repetitions under tc-netem packet loss /
+# delay, then reports a pass rate plus a mixed-vs-same-pair failure
+# attribution.
+#
+# Why repetitions: a single run under loss is a coin flip — convergence,
+# rekey, and ping retries all interact with packet loss. Only across many
+# reps does a SIGNAL separate from noise. The key is the CONTROL ARM: a
+# node-spec with a same-version pair (the default `a a b c` gives a1<->a2)
+# lets the loop distinguish:
+#
+#   - failures on MIXED pairs only, never the same-version pair
+#         -> an interop regression — versions diverge under loss.
+#   - failures on BOTH mixed and same pairs
+#         -> loss-induced general instability, not version-specific.
+#   - failures on the same-version pair only
+#         -> the version under test is unstable even against itself.
+#
+# A sub-100% pass rate under loss is EXPECTED and is not, by itself, a
+# failure. This script exits non-zero ONLY for the interop-regression
+# signal (mixed-only failures).
+#
+# Reps run SERIALLY — interop-test.sh uses fixed container names and a
+# fixed Docker network, so two reps must never overlap.
+#
+# Usage:
+#   ./interop-stress.sh [--reps N] [node-spec...]
+#
+#   --reps N    repetitions (default 10).
+#   node-spec   slot letters (a/b/c), default `a a b c` (the control
+#               topology — one same-version pair + five mixed pairs).
+#
+# Environment:
+#   FIPS_INTEROP_NETEM     tc-netem string passed through to interop-test.sh,
+#                          which applies it per-container. A stress run
+#                          normally wants this set; if unset the script
+#                          warns but still runs (a clean baseline loop).
+#   REKEY_AFTER_SECS       forwarded to interop-test.sh (default 35).
+#   FIPS_INTEROP_RUNS_DIR  Root for harness scratch dirs (.build/,
+#                          .stress-runs/, generated-configs/). When
+#                          unset, falls back to in-tree paths under
+#                          testing/interop/ and prints a warning to
+#                          stderr; set it to a path outside the source
+#                          tree to keep generated artefacts out of the
+#                          checkout.
+#
+# Artifacts (per invocation): <runs-base>/.stress-runs/<UTC-ts>/
+#   rep-NN/driver.log        full interop-test.sh output for that rep.
+#   rep-NN/docker-<node>.log per-container `docker logs` (FAILED reps only).
+#   summary.txt              the final aggregate report.
+
+set -uo pipefail
+
+SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
+DRIVER="$SCRIPT_DIR/interop-test.sh"
+
+# ── Scratch-dir root ─────────────────────────────────────────────────
+#
+# FIPS_INTEROP_RUNS_DIR controls where the harness writes its scratch
+# directories (.build/, .stress-runs/, generated-configs/). When unset
+# we fall back to in-tree paths under testing/interop/ and warn the
+# operator, so the warning fires exactly once per invocation. When a
+# parent script has already warned it exports _FIPS_INTEROP_WARNED=1
+# to suppress duplicate warnings in child scripts.
+if [[ -n "${FIPS_INTEROP_RUNS_DIR:-}" ]]; then
+    INTEROP_RUNS_BASE="$FIPS_INTEROP_RUNS_DIR"
+    mkdir -p "$INTEROP_RUNS_BASE"
+else
+    INTEROP_RUNS_BASE="$SCRIPT_DIR"
+    if [[ -z "${_FIPS_INTEROP_WARNED:-}" ]]; then
+        echo >&2 "WARNING: FIPS_INTEROP_RUNS_DIR not set; harness output will be written under the source tree at $INTEROP_RUNS_BASE. Set FIPS_INTEROP_RUNS_DIR to a path outside the source tree to avoid this."
+        export _FIPS_INTEROP_WARNED=1
+    fi
+fi
+
+RUNS_BASE="$INTEROP_RUNS_BASE/.stress-runs"
+
+# ── Args ─────────────────────────────────────────────────────────────
+
+REPS=10
+SPEC=()
+while [ "$#" -gt 0 ]; do
+    case "$1" in
+        --reps)
+            REPS="${2:-}"
+            if ! [[ "$REPS" =~ ^[0-9]+$ ]] || [ "$REPS" -lt 1 ]; then
+                echo "ERROR: --reps needs a positive integer" >&2
+                exit 1
+            fi
+            shift 2
+            ;;
+        --reps=*)
+            REPS="${1#--reps=}"
+            if ! [[ "$REPS" =~ ^[0-9]+$ ]] || [ "$REPS" -lt 1 ]; then
+                echo "ERROR: --reps needs a positive integer" >&2
+                exit 1
+            fi
+            shift
+            ;;
+        -h|--help)
+            sed -n '2,40p' "$0"
+            exit 0
+            ;;
+        --)
+            shift
+            SPEC+=("$@")
+            break
+            ;;
+        -*)
+            echo "ERROR: unknown option '$1'" >&2
+            exit 1
+            ;;
+        *)
+            SPEC+=("$1")
+            shift
+            ;;
+    esac
+done
+
+if [ "${#SPEC[@]}" -eq 0 ]; then
+    SPEC=(a a b c)
+fi
+SPEC_STR="${SPEC[*]}"
+
+# ── Preflight ────────────────────────────────────────────────────────
+
+if [ ! -x "$DRIVER" ] && [ ! -f "$DRIVER" ]; then
+    echo "ERROR: driver not found: $DRIVER" >&2
+    exit 2
+fi
+
+if [ -z "${FIPS_INTEROP_NETEM:-}" ]; then
+    echo "WARN: FIPS_INTEROP_NETEM is unset — a stress run normally wants"
+    echo "      netem packet loss/delay. Running a clean (no-impairment)"
+    echo "      loop anyway. Example:"
+    echo "        FIPS_INTEROP_NETEM=\"delay 10ms 5ms 25% loss 2%\" $0"
+    echo ""
+fi
+
+# ── Run directory ────────────────────────────────────────────────────
+
+RUN_TS="$(date -u +%Y-%m-%dT%H-%M-%SZ)"
+RUN_DIR="$RUNS_BASE/$RUN_TS"
+mkdir -p "$RUN_DIR"
+
+echo "=============================================================="
+echo " FIPS Interop Netem Stress Loop"
+echo "=============================================================="
+echo ""
+echo "Node-spec : $SPEC_STR"
+echo "Reps      : $REPS"
+echo "Netem     : ${FIPS_INTEROP_NETEM:-<none>}"
+echo "Artifacts : $RUN_DIR"
+echo ""
+
+# ── Serial rep loop ──────────────────────────────────────────────────
+
+PASS_COUNT=0
+FAIL_COUNT=0
+FAILED_REPS=()
+# Per-kind connectivity-failure tallies, summed across all failed reps.
+MIXED_FAILS=0
+SAME_FAILS=0
+
+for ((rep = 1; rep <= REPS; rep++)); do
+    rep_id="$(printf 'rep-%02d' "$rep")"
+    rep_dir="$RUN_DIR/$rep_id"
+    mkdir -p "$rep_dir"
+    driver_log="$rep_dir/driver.log"
+
+    echo "── $rep_id / $REPS ──────────────────────────────────────────"
+
+    # Run the driver, capturing full output and exit code. Netem is
+    # passed through the environment; interop-test.sh applies it.
+    FIPS_INTEROP_NETEM="${FIPS_INTEROP_NETEM:-}" \
+        bash "$DRIVER" "${SPEC[@]}" >"$driver_log" 2>&1
+    rc=$?
+
+    if [ "$rc" -eq 0 ]; then
+        PASS_COUNT=$((PASS_COUNT + 1))
+        echo "  PASS (exit 0)"
+    else
+        FAIL_COUNT=$((FAIL_COUNT + 1))
+        FAILED_REPS+=("$rep_id")
+        echo "  FAIL (exit $rc)"
+
+        # Preserve each failed container's full docker logs. The driver
+        # uses fixed container names fips-interop-<nodeid>; harvest every
+        # container matching that prefix that still exists.
+        while read -r ctr; do
+            [ -n "$ctr" ] || continue
+            docker logs "$ctr" >"$rep_dir/docker-${ctr}.log" 2>&1 || true
+        done < <(docker ps -a --filter 'name=fips-interop-' \
+                    --format '{{.Names}}' 2>/dev/null)
+
+        # Tally connectivity failures by pair kind, reusing the
+        # pair-attributed lines interop-test.sh prints. Each line is
+        # like: "  - [baseline] MIXED pair a1[...] <-> c1[...]: ping ..."
+        m="$(grep -cE '^\s*-\s*\[[^]]+\]\s+MIXED pair' "$driver_log" || true)"
+        s="$(grep -cE '^\s*-\s*\[[^]]+\]\s+same pair'  "$driver_log" || true)"
+        MIXED_FAILS=$((MIXED_FAILS + m))
+        SAME_FAILS=$((SAME_FAILS + s))
+        echo "       connectivity-failure lines: mixed=$m same=$s"
+    fi
+done
+
+echo ""
+
+# ── Aggregate report ─────────────────────────────────────────────────
+
+# Pass rate as an integer percent.
+if [ "$REPS" -gt 0 ]; then
+    PASS_RATE=$(( PASS_COUNT * 100 / REPS ))
+else
+    PASS_RATE=0
+fi
+
+# ── Verdict ──────────────────────────────────────────────────────────
+#
+# Count mixed vs same-version pairs from a completed rep's banner so the
+# verdict can normalise for the pair-count asymmetry: a spec like
+# `a a b c` has 5 mixed pairs but only 1 same-version control, so an
+# isolated loss blip lands on a mixed pair ~5x more often by pure chance.
+# A single mixed-only failure is therefore NOT a regression signal — a
+# genuine interop regression shows a concentrated, repeated mixed-only
+# pattern. Require at least REGRESSION_MIN_MIXED mixed failures before
+# calling it; below that, a mixed-only result is reported as loss noise.
+
+REGRESSION_MIN_MIXED=3
+
+MIXED_PAIRS=0
+SAME_PAIRS=0
+_banner="$RUN_DIR/rep-01/driver.log"
+if [ -f "$_banner" ]; then
+    MIXED_PAIRS=$(grep -cE '^  MIXED  ' "$_banner" || true)
+    SAME_PAIRS=$(grep -cE '^  same  ' "$_banner" || true)
+fi
+
+EXIT_CODE=0
+if [ "$MIXED_FAILS" -eq 0 ] && [ "$SAME_FAILS" -eq 0 ] && [ "$FAIL_COUNT" -eq 0 ]; then
+    VERDICT="ALL $REPS reps passed cleanly: every pair stayed healthy across"
+    VERDICT2="every rep under the applied netem profile."
+elif [ "$MIXED_FAILS" -ge "$REGRESSION_MIN_MIXED" ] && [ "$SAME_FAILS" -eq 0 ]; then
+    VERDICT="INTEROP REGRESSION: $MIXED_FAILS connectivity failures on MIXED-version"
+    VERDICT2="pairs, none on the same-version control — a concentrated mixed-only pattern above the noise threshold. Versions diverge under loss."
+    EXIT_CODE=1
+elif [ "$MIXED_FAILS" -gt 0 ] && [ "$SAME_FAILS" -gt 0 ]; then
+    VERDICT="LOSS-INDUCED INSTABILITY: failures on both mixed ($MIXED_FAILS) and"
+    VERDICT2="same-version ($SAME_FAILS) pairs — general instability under loss, not version-specific."
+elif [ "$SAME_FAILS" -gt 0 ]; then
+    VERDICT="SAME-VERSION INSTABILITY: $SAME_FAILS failure(s) on the same-version"
+    VERDICT2="control pair — a build is unstable even against itself, not an interop issue."
+elif [ "$MIXED_FAILS" -gt 0 ]; then
+    VERDICT="NO INTEROP-REGRESSION SIGNAL: $MIXED_FAILS isolated mixed-pair failure(s)"
+    VERDICT2="across $FAIL_COUNT rep(s), none on the same-version control, below the regression threshold ($REGRESSION_MIN_MIXED). With $MIXED_PAIRS mixed pairs vs $SAME_PAIRS same-version, isolated loss blips land on mixed pairs more often — consistent with loss noise, not a regression."
+else
+    VERDICT="NO connectivity-pair failures recorded, but $FAIL_COUNT rep(s) still"
+    VERDICT2="failed — on non-connectivity signatures (global-health log patterns or a missing rekey). Check the per-rep driver logs."
+fi
+
+{
+    echo "=============================================================="
+    echo " Interop Netem Stress — Aggregate Report"
+    echo "=============================================================="
+    echo ""
+    echo "Run        : $RUN_TS"
+    echo "Node-spec  : $SPEC_STR"
+    echo "Netem      : ${FIPS_INTEROP_NETEM:-<none>}"
+    echo ""
+    echo "Reps run   : $REPS"
+    echo "Passed     : $PASS_COUNT"
+    echo "Failed     : $FAIL_COUNT"
+    echo "Pass rate  : ${PASS_RATE}%"
+    if [ "${#FAILED_REPS[@]}" -gt 0 ]; then
+        echo "Failed reps: ${FAILED_REPS[*]}"
+    fi
+    echo ""
+    echo "-- Connectivity-failure attribution (summed over failed reps) --"
+    echo "  mixed-version : $MIXED_FAILS failure(s) over $MIXED_PAIRS mixed pairs x $REPS reps"
+    echo "  same-version  : $SAME_FAILS failure(s) over $SAME_PAIRS same pair(s) x $REPS reps"
+    echo "  regression threshold: >= $REGRESSION_MIN_MIXED mixed-only failures"
+    echo ""
+    echo "Verdict:"
+    echo "  $VERDICT"
+    echo "  $VERDICT2"
+    echo ""
+    if [ "$EXIT_CODE" -eq 0 ]; then
+        echo "Exit 0: no interop-regression signal (a sub-100% rate under loss"
+        echo "        is expected and is not by itself a failure)."
+    else
+        echo "Exit 1: interop-regression signal present."
+    fi
+    echo ""
+    echo "Artifacts: $RUN_DIR"
+} | tee "$RUN_DIR/summary.txt"
+
+exit "$EXIT_CODE"

testing/interop/interop-test.sh

@@ -0,0 +1,700 @@
+#!/bin/bash
+# Mixed-version interop test driver.
+#
+# Brings up an N-node full mesh from a NODE-SPEC (a multiset of image
+# slots), verifies every pair interoperates — FMP link, FSP session,
+# connectivity, rekey survival — and runs a per-node, per-pair log
+# analysis tuned to surface INTEROP problems: handshake failures,
+# FSP/FMP decrypt failures, `unknown FMP version`, replay storms,
+# link/session teardowns.
+#
+# The harness's job is NOT to test a single version. It is to find any
+# place where two DIFFERENT versions fail to interoperate in a way a
+# same-version pair would not. Every failure is attributed to a specific
+# (version-X <-> version-Y) pair, classified same-version vs MIXED.
+#
+# A node-spec like `a a b c` produces a same-version pair (a1<->a2) — the
+# CONTROL ARM — alongside the mixed pairs. The control pair is what makes
+# a netem run interpretable: a failure on a mixed pair that the control
+# pair does not share is an interop regression; a failure both share is
+# loss noise, not version-specific.
+#
+# Prerequisites:
+#   1. bash build-images.sh <ref-a> <ref-b> <ref-c>   (builds fips-interop:a/b/c)
+#   2. configs are (re)generated automatically below for the given spec.
+#
+# Usage:
+#   ./interop-test.sh [node-spec...]
+#
+#   node-spec   space-separated slot letters (a/b/c), default `a b c`.
+#               e.g. `a a b c` (4-node, one same-version control pair),
+#                    `a a a`   (3-node same-version flake rig).
+#
+# Environment:
+#   FIPS_INTEROP_NETEM     tc-netem arg string applied to every container's
+#                          eth0, e.g. "delay 10ms 5ms 25% loss 1%". Unset =
+#                          no impairment.
+#   FIPS_INTEROP_KEEP_UP   1 = leave containers running after the test (debug).
+#   REKEY_AFTER_SECS       rekey interval to generate configs with (default 35).
+#   FIPS_INTEROP_RUNS_DIR  Root for harness scratch dirs (.build/,
+#                          .stress-runs/, generated-configs/). When
+#                          unset, falls back to in-tree paths under
+#                          testing/interop/ and prints a warning to
+#                          stderr; set it to a path outside the source
+#                          tree to keep generated artefacts out of the
+#                          checkout.
+
+set -uo pipefail
+
+SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
+REPO_ROOT="$(cd "$SCRIPT_DIR/../.." && pwd)"
+source "$SCRIPT_DIR/../lib/wait-converge.sh"
+
+# ── Scratch-dir root ─────────────────────────────────────────────────
+#
+# FIPS_INTEROP_RUNS_DIR controls where the harness writes its scratch
+# directories (.build/, .stress-runs/, generated-configs/). When unset
+# we fall back to in-tree paths under testing/interop/ and warn the
+# operator, so the warning fires exactly once per invocation. When a
+# parent script has already warned it exports _FIPS_INTEROP_WARNED=1
+# to suppress duplicate warnings in child scripts.
+if [[ -n "${FIPS_INTEROP_RUNS_DIR:-}" ]]; then
+    RUNS_BASE="$FIPS_INTEROP_RUNS_DIR"
+    mkdir -p "$RUNS_BASE"
+else
+    RUNS_BASE="$SCRIPT_DIR"
+    if [[ -z "${_FIPS_INTEROP_WARNED:-}" ]]; then
+        echo >&2 "WARNING: FIPS_INTEROP_RUNS_DIR not set; harness output will be written under the source tree at $RUNS_BASE. Set FIPS_INTEROP_RUNS_DIR to a path outside the source tree to avoid this."
+        export _FIPS_INTEROP_WARNED=1
+    fi
+fi
+
+GEN_DIR="$RUNS_BASE/generated-configs"
+COMPOSE_FILE="$GEN_DIR/docker-compose.generated.yml"
+NODES_ENV="$GEN_DIR/nodes.env"
+REFS_ENV="$RUNS_BASE/.build/refs.env"
+
+REKEY_AFTER_SECS="${REKEY_AFTER_SECS:-35}"
+
+# ── Node-spec ────────────────────────────────────────────────────────
+
+SPEC=("$@")
+if [ "${#SPEC[@]}" -eq 0 ]; then
+    SPEC=(a b c)
+fi
+SPEC_STR="${SPEC[*]}"
+
+# ── Timing ───────────────────────────────────────────────────────────
+
+CONVERGENCE_TIMEOUT=60          # wait for full mesh + clean ping baseline
+PING_TIMEOUT=5
+MAX_PING_ATTEMPTS=4             # strict-ping retry (mirrors rekey-test.sh)
+PING_RETRY_DELAY=1
+# First rekey should follow shortly after the interval once converged.
+FIRST_REKEY_TIMEOUT=$((REKEY_AFTER_SECS + 20))
+SECOND_REKEY_WAIT=$((REKEY_AFTER_SECS + 5))
+REKEY_SETTLE=12                 # FSP-cutover settle budget (Phase 6)
+# Post-rekey reconvergence is polled, not fixed-slept: the mesh is given
+# up to this long to restore full connectivity after a rekey before the
+# strict assertion sweep runs. A genuinely stuck pair still fails — the
+# poll times out and the recording sweep captures it.
+POST_REKEY_TIMEOUT=45
+LOG_POLL_INTERVAL=2
+
+# ── Counters ─────────────────────────────────────────────────────────
+
+PASSED=0
+FAILED=0
+TOTAL_PASSED=0
+TOTAL_FAILED=0
+# INTEROP_FAILURES collects human-readable, pair-attributed failure lines.
+INTEROP_FAILURES=()
+
+# ── Preflight: docker + images ───────────────────────────────────────
+
+if ! docker info >/dev/null 2>&1; then
+    echo "ERROR: Docker daemon is not reachable" >&2
+    exit 2
+fi
+
+# A node-spec only ever references the three image slots, regardless of
+# how many nodes it has. Check the slots actually present in the spec.
+for slot in $(printf '%s\n' "${SPEC[@]}" | sort -u); do
+    case "$slot" in
+        a|b|c) ;;
+        *) echo "ERROR: invalid slot '$slot' in node-spec" >&2; exit 2 ;;
+    esac
+    if ! docker image inspect "fips-interop:$slot" >/dev/null 2>&1; then
+        echo "ERROR: image fips-interop:$slot not present." >&2
+        echo "Build the three images first:" >&2
+        echo "  bash $SCRIPT_DIR/build-images.sh <ref-a> <ref-b> <ref-c>" >&2
+        exit 2
+    fi
+done
+
+# ── (Re)generate configs for this node-spec ──────────────────────────
+#
+# Always regenerate when the spec on disk does not match the requested
+# spec (or no manifest exists). Generation is deterministic and cheap.
+
+need_regen=1
+if [ -f "$NODES_ENV" ]; then
+    existing_spec="$(sed -n 's/^INTEROP_SPEC="\(.*\)"$/\1/p' "$NODES_ENV")"
+    if [ "$existing_spec" = "$SPEC_STR" ]; then
+        need_regen=0
+    fi
+fi
+if [ "$need_regen" -eq 1 ]; then
+    echo "Generating mesh configs for spec '$SPEC_STR' (rekey.after_secs=$REKEY_AFTER_SECS)..."
+    REKEY_AFTER_SECS="$REKEY_AFTER_SECS" bash "$SCRIPT_DIR/generate-configs.sh" \
+        "${SPEC[@]}"
+    echo ""
+fi
+
+# ── Load the manifest ────────────────────────────────────────────────
+#
+# nodes.env gives the ordered node list and the per-node slot/container/
+# ip/npub maps. npubs.env gives NPUB_<NODEID> vars used by ping_one.
+
+# shellcheck disable=SC1090
+source "$NODES_ENV"
+# shellcheck disable=SC1091
+source "$GEN_DIR/npubs.env"
+
+read -r -a NODES <<< "$INTEROP_NODE_IDS"
+
+declare -A SLOT_OF CONTAINER NODE_IP_OF NPUB_OF
+parse_map() {
+    # parse_map <assoc-array-name> <space-separated nodeid:value tokens>
+    local -n _dst="$1"
+    local tok nid val
+    for tok in $2; do
+        nid="${tok%%:*}"
+        val="${tok#*:}"
+        _dst["$nid"]="$val"
+    done
+}
+parse_map SLOT_OF       "$INTEROP_NODE_SLOTS"
+parse_map CONTAINER     "$INTEROP_NODE_CONTAINERS"
+parse_map NODE_IP_OF    "$INTEROP_NODE_IPS"
+parse_map NPUB_OF       "$INTEROP_NODE_NPUBS"
+
+# Unordered pairs of the mesh.
+PAIRS=()
+for ((i = 0; i < ${#NODES[@]}; i++)); do
+    for ((j = i + 1; j < ${#NODES[@]}; j++)); do
+        PAIRS+=("${NODES[$i]} ${NODES[$j]}")
+    done
+done
+
+NUM_NODES="${#NODES[@]}"
+NUM_PAIRS="${#PAIRS[@]}"
+NUM_DIRECTED=$((NUM_PAIRS * 2))
+PEERS_EXPECTED=$((NUM_NODES - 1))
+
+# ── Ref / version metadata ───────────────────────────────────────────
+#
+# refs.env (written by build-images.sh) records what each SLOT was built
+# from. If absent, fall back to the image labels, then to "unknown".
+# These maps are keyed by SLOT (a/b/c), not by node id.
+
+declare -A SLOT_REF SLOT_SHA
+
+load_slot_metadata() {
+    local slot upper
+    for slot in a b c; do
+        SLOT_REF[$slot]="unknown"
+        SLOT_SHA[$slot]="unknown"
+    done
+    if [ -f "$REFS_ENV" ]; then
+        # shellcheck disable=SC1090
+        source "$REFS_ENV"
+    fi
+    for slot in a b c; do
+        upper="$(echo "$slot" | tr '[:lower:]' '[:upper:]')"
+        local ref_var="INTEROP_REF_${upper}"
+        local sha_var="INTEROP_SHA_${upper}"
+        if [ -n "${!ref_var:-}" ]; then
+            SLOT_REF[$slot]="${!ref_var}"
+            SLOT_SHA[$slot]="${!sha_var:-unknown}"
+            continue
+        fi
+        local lbl_ref lbl_sha
+        lbl_ref="$(docker image inspect "fips-interop:$slot" \
+            --format '{{ index .Config.Labels "fips.interop.ref" }}' 2>/dev/null || true)"
+        lbl_sha="$(docker image inspect "fips-interop:$slot" \
+            --format '{{ index .Config.Labels "fips.interop.sha" }}' 2>/dev/null || true)"
+        [ -n "$lbl_ref" ] && SLOT_REF[$slot]="$lbl_ref"
+        [ -n "$lbl_sha" ] && SLOT_SHA[$slot]="$lbl_sha"
+    done
+}
+
+# A pair is "mixed-version" iff the two nodes' image slots resolve to
+# different built SHAs. SHA is the precise discriminator; ref names can
+# differ yet point at the same commit.
+pair_is_mixed() {
+    local n1="$1" n2="$2"
+    local s1="${SLOT_OF[$n1]}" s2="${SLOT_OF[$n2]}"
+    if [ "${SLOT_SHA[$s1]}" = "${SLOT_SHA[$s2]}" ] \
+        && [ "${SLOT_SHA[$s1]}" != "unknown" ]; then
+        return 1   # same version
+    fi
+    return 0       # mixed (or unknown — treat as mixed for scrutiny)
+}
+
+pair_kind() {
+    if pair_is_mixed "$1" "$2"; then
+        echo "MIXED"
+    else
+        echo "same"
+    fi
+}
+
+pair_label() {
+    # e.g. "a1[A fix/...@3045212] <-> c1[C v0.3.0@b11b639]"
+    local n1="$1" n2="$2"
+    local s1="${SLOT_OF[$n1]}" s2="${SLOT_OF[$n2]}"
+    local u1 u2
+    u1="$(echo "$s1" | tr '[:lower:]' '[:upper:]')"
+    u2="$(echo "$s2" | tr '[:lower:]' '[:upper:]')"
+    echo "${n1}[$u1 ${SLOT_REF[$s1]}@${SLOT_SHA[$s1]}] <-> ${n2}[$u2 ${SLOT_REF[$s2]}@${SLOT_SHA[$s2]}]"
+}
+
+# ── Helpers ──────────────────────────────────────────────────────────
+
+# ping6 from one container to another node's mesh address (npub.fips).
+ping_one() {
+    local from_node="$1" to_node="$2" quiet="${3:-}"
+    local max_attempts="${4:-1}"
+    local from_ctr="${CONTAINER[$from_node]}"
+    local to_npub="${NPUB_OF[$to_node]}"
+    local label="$from_node -> $to_node"
+
+    local attempt=1 output rtt
+    while (( attempt <= max_attempts )); do
+        (( attempt > 1 )) && sleep "$PING_RETRY_DELAY"
+        if output=$(docker exec "$from_ctr" ping6 -c 1 -W "$PING_TIMEOUT" \
+            "${to_npub}.fips" 2>&1); then
+            rtt=$(echo "$output" | grep -oE 'time=[0-9.]+' | cut -d= -f2)
+            if [ -z "$quiet" ]; then
+                echo "  $label ... OK (${rtt:-?}ms${attempt:+, attempt $attempt})"
+            fi
+            PASSED=$((PASSED + 1))
+            return 0
+        fi
+        attempt=$((attempt + 1))
+    done
+    if [ -z "$quiet" ]; then
+        echo "  $label ... FAIL (after $max_attempts attempt(s))"
+    fi
+    FAILED=$((FAILED + 1))
+    return 1
+}
+
+# All directed pairs of the mesh. Records pair-attributed failures into
+# INTEROP_FAILURES, with the mixed/same classification.
+ping_all_pairs() {
+    local quiet="${1:-}" max_attempts="${2:-1}" context="${3:-connectivity}"
+    PASSED=0
+    FAILED=0
+    local i j
+    for i in "${NODES[@]}"; do
+        for j in "${NODES[@]}"; do
+            [ "$i" = "$j" ] && continue
+            if ! ping_one "$i" "$j" "$quiet" "$max_attempts"; then
+                # The `convergence` context is the wait_for_full_baseline
+                # detector poll, NOT an assertion: a miss there only means
+                # "not converged yet, keep polling". Recording detector
+                # misses as interop failures is wrong — it false-fails
+                # every rep that takes a moment to converge. Only the
+                # strict assertion sweeps (baseline / post-rekey-*) record.
+                if [ "$context" != "convergence" ]; then
+                    local kind
+                    kind="$(pair_kind "$i" "$j")"
+                    INTEROP_FAILURES+=("[$context] $kind pair $(pair_label "$i" "$j"): ping $i->$j FAILED")
+                fi
+            fi
+        done
+    done
+}
+
+# Poll until every directed pair pings clean, or until timeout. This is a
+# convergence DETECTOR — used at establishment (Phase 1) and after each
+# rekey (Phases 3/5). It pings with the "convergence" context, which
+# ping_all_pairs deliberately does not record as a failure; the caller
+# runs a separate strict assertion sweep afterwards.
+wait_for_full_baseline() {
+    local timeout="$1"
+    local start=$SECONDS
+    local best_passed=0 best_failed="$NUM_DIRECTED"
+    while (( SECONDS - start < timeout )); do
+        ping_all_pairs quiet 1 "convergence"
+        if [ "$PASSED" -gt "$best_passed" ]; then
+            best_passed="$PASSED"; best_failed="$FAILED"
+        fi
+        [ "$FAILED" -eq 0 ] && return 0
+        sleep 1
+    done
+    PASSED="$best_passed"; FAILED="$best_failed"
+    return 1
+}
+
+phase_result() {
+    local phase="$1"
+    TOTAL_PASSED=$((TOTAL_PASSED + PASSED))
+    TOTAL_FAILED=$((TOTAL_FAILED + FAILED))
+    if [ "$FAILED" -eq 0 ]; then
+        echo "  PASS  $phase: $PASSED/$((PASSED + FAILED))"
+    else
+        echo "  FAIL  $phase: $PASSED passed, $FAILED FAILED"
+    fi
+}
+
+# Count a pattern across all node logs.
+count_log_pattern() {
+    local pattern="$1" total=0 n count
+    for n in "${NODES[@]}"; do
+        count=$(docker logs "${CONTAINER[$n]}" 2>&1 | grep -cE "$pattern" || true)
+        total=$((total + count))
+    done
+    echo "$total"
+}
+
+# Per-node count of a pattern.
+count_node_pattern() {
+    local node="$1" pattern="$2"
+    docker logs "${CONTAINER[$node]}" 2>&1 | grep -cE "$pattern" || true
+}
+
+wait_for_log_pattern_count() {
+    local pattern="$1" min_count="$2" timeout="$3"
+    local start=$SECONDS
+    while (( SECONDS - start < timeout )); do
+        [ "$(count_log_pattern "$pattern")" -ge "$min_count" ] && return 0
+        sleep "$LOG_POLL_INTERVAL"
+    done
+    [ "$(count_log_pattern "$pattern")" -ge "$min_count" ]
+}
+
+dump_diagnostics() {
+    echo ""
+    echo "=== Peer / link snapshot ==="
+    local n s
+    for n in "${NODES[@]}"; do
+        s="${SLOT_OF[$n]}"
+        echo "--- $n (slot $s, ${SLOT_REF[$s]}@${SLOT_SHA[$s]}) ---"
+        docker exec "${CONTAINER[$n]}" fipsctl show peers 2>/dev/null || true
+        docker exec "${CONTAINER[$n]}" fipsctl show links 2>/dev/null || true
+        echo ""
+    done
+    echo "=== Recent node logs (interop-relevant lines) ==="
+    for n in "${NODES[@]}"; do
+        echo "--- $n ---"
+        docker logs "${CONTAINER[$n]}" 2>&1 \
+            | grep -E "(ERROR|PANIC|panicked|FMP version|handshake|Handshake|decrypt|Decrypt|teardown|rekey|Rekey|K-bit|established|Established)" \
+            | tail -40
+        echo ""
+    done
+}
+
+compose() {
+    docker compose -f "$COMPOSE_FILE" "$@"
+}
+
+cleanup() {
+    if [ "${FIPS_INTEROP_KEEP_UP:-}" = "1" ]; then
+        echo ""
+        echo "FIPS_INTEROP_KEEP_UP=1 — leaving mesh running. Tear down with:"
+        echo "  docker compose -f $COMPOSE_FILE down --volumes --remove-orphans"
+        return
+    fi
+    compose down --volumes --remove-orphans >/dev/null 2>&1 || true
+}
+trap cleanup EXIT
+trap 'echo ""; echo "Interrupted"; exit 130' INT
+
+load_slot_metadata
+
+# ── Banner ───────────────────────────────────────────────────────────
+
+echo "=============================================================="
+echo " FIPS Mixed-Version Interop Test"
+echo "=============================================================="
+echo ""
+echo "Node-spec: $SPEC_STR   ($NUM_NODES nodes, $NUM_PAIRS pairs, $NUM_DIRECTED directed)"
+echo ""
+echo "Mesh nodes:"
+for n in "${NODES[@]}"; do
+    s="${SLOT_OF[$n]}"
+    u="$(echo "$s" | tr '[:lower:]' '[:upper:]')"
+    echo "  $n  slot $u  ${SLOT_REF[$s]} @ ${SLOT_SHA[$s]}  (${NODE_IP_OF[$n]})"
+done
+echo ""
+echo "Mesh pairs:"
+for p in "${PAIRS[@]}"; do
+    read -r n1 n2 <<< "$p"
+    echo "  $(pair_kind "$n1" "$n2")  $(pair_label "$n1" "$n2")"
+done
+echo ""
+if [ -n "${FIPS_INTEROP_NETEM:-}" ]; then
+    echo "Netem impairment: $FIPS_INTEROP_NETEM"
+    echo ""
+fi
+echo "rekey.after_secs=$REKEY_AFTER_SECS"
+echo ""
+
+# ── Phase 0: bring up the mesh ───────────────────────────────────────
+
+echo "Phase 0: Starting mesh"
+compose down --volumes --remove-orphans >/dev/null 2>&1 || true
+if ! compose up -d; then
+    echo "  FAIL  compose up failed"
+    exit 1
+fi
+
+# Optional netem: applied via `docker exec ... tc qdisc` on each
+# container's eth0 — host bridge qdisc does NOT shape inter-container
+# port-to-port traffic, so the impairment must live inside each
+# container. Same mechanism as testing/flake-lab/run-loop.sh.
+if [ -n "${FIPS_INTEROP_NETEM:-}" ]; then
+    echo "  Applying netem to each container eth0: $FIPS_INTEROP_NETEM"
+    for n in "${NODES[@]}"; do
+        if docker exec "${CONTAINER[$n]}" tc qdisc add dev eth0 root \
+            netem ${FIPS_INTEROP_NETEM} >/dev/null 2>&1; then
+            echo "    $n: netem applied"
+        else
+            echo "    $n: WARN netem apply failed"
+        fi
+    done
+fi
+echo ""
+
+# ── Phase 1: FMP link + FSP session establishment ────────────────────
+
+echo "Phase 1: Link/session establishment + connectivity baseline"
+PASSED=0; FAILED=0
+
+# Every node must reach N-1 authenticated peers. A peer in `show peers`
+# is an authenticated FMP-link peer.
+for n in "${NODES[@]}"; do
+    if ! wait_for_peers "${CONTAINER[$n]}" "$PEERS_EXPECTED" "$CONVERGENCE_TIMEOUT"; then
+        echo "  $n did not reach $PEERS_EXPECTED authenticated peers"
+    fi
+done
+
+# The all-pairs ping over fips0 is the definitive FSP-session check: in
+# a full mesh every pair is a direct neighbor, so a successful ping
+# proves that pair's FSP session carries traffic in that direction.
+if wait_for_full_baseline "$CONVERGENCE_TIMEOUT"; then
+    ping_all_pairs "" "$MAX_PING_ATTEMPTS" "baseline"
+    phase_result "Establishment baseline (all $NUM_DIRECTED directed pairs)"
+else
+    echo "  Best baseline before timeout: $PASSED/$((PASSED + FAILED))"
+    ping_all_pairs "" "$MAX_PING_ATTEMPTS" "baseline"
+    phase_result "Establishment baseline (all $NUM_DIRECTED directed pairs)"
+    dump_diagnostics
+    echo ""
+    echo "=== Results: $TOTAL_PASSED passed, $TOTAL_FAILED failed ==="
+    echo "FAIL: mesh did not converge — see pair attribution above."
+    exit 1
+fi
+echo ""
+
+# ── Phase 2: first rekey cycle ───────────────────────────────────────
+
+echo "Phase 2: First rekey cycle (waiting up to ${FIRST_REKEY_TIMEOUT}s)"
+PASSED=0; FAILED=0
+wait_for_log_pattern_count \
+    "Rekey cutover complete \(initiator\), K-bit flipped" 1 \
+    "$FIRST_REKEY_TIMEOUT" || true
+fmp_cutovers="$(count_log_pattern 'Rekey cutover complete \(initiator\), K-bit flipped')"
+if [ "$fmp_cutovers" -ge 1 ]; then
+    echo "  PASS  FMP rekey initiator cutovers: $fmp_cutovers"
+    PASSED=$((PASSED + 1))
+else
+    echo "  FAIL  no FMP rekey cutover observed within ${FIRST_REKEY_TIMEOUT}s"
+    FAILED=$((FAILED + 1))
+    INTEROP_FAILURES+=("[rekey] no FMP rekey cutover completed across the mesh")
+fi
+phase_result "First rekey cycle"
+echo ""
+
+# ── Phase 3: post-first-rekey connectivity ───────────────────────────
+
+echo "Phase 3: Post-first-rekey connectivity (reconverge within ${POST_REKEY_TIMEOUT}s)"
+PASSED=0; FAILED=0
+# Poll until the mesh has reconverged after the rekey, then assert once.
+# The detector poll records nothing (see ping_all_pairs); only the strict
+# sweep below records, so a brief rekey-induced disruption is not a fail.
+wait_for_full_baseline "$POST_REKEY_TIMEOUT" || true
+ping_all_pairs "" "$MAX_PING_ATTEMPTS" "post-rekey-1"
+phase_result "Post-first-rekey (all $NUM_DIRECTED directed pairs)"
+echo ""
+
+# ── Phase 4: second rekey cycle ──────────────────────────────────────
+
+echo "Phase 4: Second rekey cycle (waiting ${SECOND_REKEY_WAIT}s)"
+sleep "$SECOND_REKEY_WAIT"
+echo ""
+
+echo "Phase 5: Post-second-rekey connectivity (reconverge within ${POST_REKEY_TIMEOUT}s)"
+PASSED=0; FAILED=0
+wait_for_full_baseline "$POST_REKEY_TIMEOUT" || true
+ping_all_pairs "" "$MAX_PING_ATTEMPTS" "post-rekey-2"
+phase_result "Post-second-rekey (all $NUM_DIRECTED directed pairs)"
+echo ""
+
+# ── Phase 6: per-node, per-pair interop log analysis ─────────────────
+#
+# This is the interop-specific analysis. For each unordered pair it
+# reports same vs mixed-version, then scans BOTH endpoints' logs for
+# interop failure signatures. A signature firing on a mixed pair that
+# does not fire on a same-version pair is the harness's primary signal.
+
+echo "Phase 6: Interop log analysis"
+PASSED=0; FAILED=0
+
+# Give FSP rekey (which trails FMP rekey) a bounded chance to complete
+# at least one cutover before the final assertions.
+wait_for_log_pattern_count "FSP rekey cutover complete" 1 "$REKEY_SETTLE" || true
+
+# Global negative checks — these must be zero on EVERY node regardless
+# of version pairing. A non-zero count is attributed to the node and,
+# where the count is asymmetric across versions, flagged as interop.
+echo ""
+echo "  -- Global health (all $NUM_NODES nodes) --"
+
+declare -A GLOBAL_PATTERNS=(
+    ["PANIC|panicked"]="panics"
+    ["ERROR"]="error-level log lines"
+    ["unknown FMP version|Unknown FMP version"]="unknown-FMP-version drops"
+    ["MMP link teardown"]="MMP link teardowns"
+    ["Excessive decrypt failures"]="excessive-decrypt-failure removals"
+    ["Session AEAD decryption failed"]="FSP AEAD decrypt failures"
+    ["Rekey msg2 processing failed"]="rekey msg2 failures"
+    ["Handshake failed|handshake failed|Handshake error"]="handshake failures"
+)
+
+for pat in "${!GLOBAL_PATTERNS[@]}"; do
+    desc="${GLOBAL_PATTERNS[$pat]}"
+    total="$(count_log_pattern "$pat")"
+    if [ "$total" -eq 0 ]; then
+        echo "    PASS  $desc: 0"
+        PASSED=$((PASSED + 1))
+    else
+        echo "    FAIL  $desc: $total (expected 0)"
+        FAILED=$((FAILED + 1))
+        # Per-node breakdown so the count can be attributed to a build.
+        for n in "${NODES[@]}"; do
+            c="$(count_node_pattern "$n" "$pat")"
+            if [ "$c" -gt 0 ]; then
+                s="${SLOT_OF[$n]}"
+                u="$(echo "$s" | tr '[:lower:]' '[:upper:]')"
+                echo "          $n [$u ${SLOT_REF[$s]}@${SLOT_SHA[$s]}]: $c"
+                INTEROP_FAILURES+=("[log] node $n ($u ${SLOT_REF[$s]}@${SLOT_SHA[$s]}): $c x '$desc'")
+            fi
+        done
+    fi
+done
+
+# Positive checks — the rekey machinery actually exercised both layers.
+echo ""
+echo "  -- Rekey machinery exercised --"
+fmp_total="$(count_log_pattern 'Rekey cutover complete \(initiator\), K-bit flipped')"
+fsp_total="$(count_log_pattern 'FSP rekey cutover complete')"
+if [ "$fmp_total" -ge 1 ]; then
+    echo "    PASS  FMP rekey cutovers across mesh: $fmp_total"
+    PASSED=$((PASSED + 1))
+else
+    echo "    FAIL  FMP rekey cutovers: $fmp_total (expected >= 1)"
+    FAILED=$((FAILED + 1))
+fi
+if [ "$fsp_total" -ge 1 ]; then
+    echo "    PASS  FSP rekey cutovers across mesh: $fsp_total"
+    PASSED=$((PASSED + 1))
+else
+    # FSP rekey not completing is a soft signal: report it but only
+    # hard-fail if a pair also lost connectivity. The connectivity
+    # phases already cover the user-visible impact.
+    echo "    WARN  FSP rekey cutovers: $fsp_total (expected >= 1)"
+fi
+
+# Per-pair summary: classify each unordered pair and report whether it
+# stayed healthy through the run. "Healthy" = no connectivity failure
+# recorded for either direction of the pair.
+echo ""
+echo "  -- Per-pair interop summary --"
+for p in "${PAIRS[@]}"; do
+    read -r n1 n2 <<< "$p"
+    kind="$(pair_kind "$n1" "$n2")"
+    label="$(pair_label "$n1" "$n2")"
+    pair_failed=0
+    if [ "${#INTEROP_FAILURES[@]}" -gt 0 ]; then
+        for f in "${INTEROP_FAILURES[@]}"; do
+            case "$f" in
+                *"ping $n1->$n2 FAILED"*|*"ping $n2->$n1 FAILED"*)
+                    pair_failed=1 ;;
+            esac
+        done
+    fi
+    if [ "$pair_failed" -eq 0 ]; then
+        echo "    PASS  $kind pair $label: stayed healthy"
+        PASSED=$((PASSED + 1))
+    else
+        echo "    FAIL  $kind pair $label: connectivity failed during the run"
+        FAILED=$((FAILED + 1))
+    fi
+done
+
+phase_result "Interop log analysis"
+echo ""
+
+# ── Summary ──────────────────────────────────────────────────────────
+
+echo "=============================================================="
+echo " Results: $TOTAL_PASSED checks passed, $TOTAL_FAILED failed"
+echo "=============================================================="
+
+# `${#arr[@]}` cannot be combined with `:-`; count into a plain var.
+INTEROP_FAILURE_COUNT="${#INTEROP_FAILURES[@]}"
+
+if [ "$TOTAL_FAILED" -eq 0 ] && [ "$INTEROP_FAILURE_COUNT" -eq 0 ]; then
+    echo ""
+    echo "PASS: all versions interoperate cleanly across the mesh (spec '$SPEC_STR')."
+    exit 0
+fi
+
+echo ""
+echo "INTEROP FAILURES (attributed to specific version pairs / builds):"
+if [ "$INTEROP_FAILURE_COUNT" -gt 0 ]; then
+    printf '%s\n' "${INTEROP_FAILURES[@]}" | sort -u | sed 's/^/  - /'
+else
+    echo "  (no per-pair failure recorded; see phase output above)"
+fi
+echo ""
+
+# Highlight whether failures concentrate on mixed-version pairs — the
+# defining interop signal.
+mixed_hits=0
+same_hits=0
+if [ "$INTEROP_FAILURE_COUNT" -gt 0 ]; then
+    for f in "${INTEROP_FAILURES[@]}"; do
+        case "$f" in
+            *"MIXED pair"*) mixed_hits=$((mixed_hits + 1)) ;;
+            *"same pair"*)  same_hits=$((same_hits + 1)) ;;
+        esac
+    done
+fi
+echo "Connectivity-failure attribution: mixed-version=$mixed_hits  same-version=$same_hits"
+if [ "$mixed_hits" -gt 0 ] && [ "$same_hits" -eq 0 ]; then
+    echo "=> Failures are MIXED-VERSION ONLY: a genuine interop regression."
+elif [ "$mixed_hits" -gt 0 ] && [ "$same_hits" -gt 0 ]; then
+    echo "=> Failures hit both mixed and same-version pairs: likely a general"
+    echo "   instability, not version-specific — investigate the common cause."
+elif [ "$same_hits" -gt 0 ]; then
+    echo "=> Failures are SAME-VERSION ONLY: not an interop issue per se;"
+    echo "   a build is unstable even against itself."
+fi
+
+dump_diagnostics
+exit 1