Merge maint into master, bringing the chaos suite's failure reporting with it

The chaos simulation could not report a failure on this line either: scenarios collided on globally-scoped container names and a shared config directory, the runner swallowed whatever they raised, and the harness discarded their exit codes before anything read them. The five commits scope the names and directories per scenario, give an aborted run its own exit code and let that code reach the summary, stop a failed scenario harvesting whichever containers happen to hold its names, and keep the daemon's reason for a failure instead of throwing it away.

Testing only, and the merge is textual: every chaos file matches maint byte for byte, and the ci-local divergence on this line does not touch the changed regions.

testing/README.md

@@ -144,3 +144,22 @@ and leaves resources behind, reap them with:
 label or a `fipsci_` compose-project prefix; it is safe to run when there
 is nothing to reap and safe to run repeatedly. Pass `--project-prefix` to
 scope the sweep to a single run.
+
+It also removes the chaos simulation's leftover host-namespace veth
+interfaces (`vh…a`/`vh…b`), the one resource it touches that is neither a
+docker object nor labelled — a host interface can carry neither a label
+nor a compose project, so it is matched by name shape alone. That makes
+the reach here asymmetric with everything above, and worth stating
+plainly:
+
+- A bare `chaos.sh` run's **containers** survive a broad reap. Its
+  compose project is not `fipsci_`, and the simulation labels only the
+  network, not the services.
+- A bare `chaos.sh` run's **veth interfaces do not.** An unscoped reap
+  deletes them while they are in use, severing the Ethernet links of a
+  live simulation and leaving its containers running.
+
+So do not run a broad `--reap` while a bare simulation is up. Scope the
+interface sweep with `--veth-suffixes` (which is what `ci-local.sh`'s own
+teardown passes) or wait for the simulation to finish. `--project-prefix`
+does not help here: it scopes only the compose-project sweep.

testing/chaos/README.md

@@ -257,12 +257,51 @@ since they use beacon discovery.
 Results written to `sim-results/` (configurable via
 `logging.output_dir`):
 
+- `status.txt` -- How the run ended, plus the scenario, the seed and the
+  container names it used; one `key=value` per line
 - `analysis.txt` -- Summary: panics, errors, sessions, metrics
 - `metadata.txt` -- Seed, node count, edges, adjacency list
 - `runner.log` -- Orchestration events (topology, netem, churn, traffic) with timestamps
 - `fips-node-nXX.log` -- Per-node log output
 
-Exit code 0 on success, 2 if panics detected.
+The `status` field reads:
+
+- `completed` -- ran for its configured duration
+- `interrupted` -- a signal cut the run short, so the artifacts are real
+  but describe less time than the scenario asked for
+- `aborted` -- the run raised part way through; same caveat, and
+  `runner.log` carries the traceback
+- `setup-failed` -- the containers never started
+- `teardown-failed` -- the mesh ran but its logs or analysis could not be
+  produced
+
+A `setup-failed` directory holds `runner.log` and `status.txt` and nothing
+else. Nothing is harvested, because container names are global to the host
+and reading them after a failed setup describes whichever run holds them
+now. So `analysis.txt` in a result directory is proof that this scenario's
+own mesh existed. A directory with no `status.txt` was written before this
+was the case and says nothing either way.
+
+Exit codes:
+
+- `0` -- Ran to completion, no panics, every assertion passed
+- `1` -- The scenario file could not be loaded, or a second interrupt
+  arrived while the first was being handled
+- `2` -- Panics found in the collected node logs. Also what the argument
+  parser exits with when it rejects the command line, before any run starts
+- `3` -- A post-run assertion failed
+- `4` -- Setup, warmup, the simulation loop or teardown raised, so the run
+  did not complete; `runner.log` carries the traceback
+
+Codes 2 and 3 describe what a mesh that ran did. Code 4 says there is
+nothing to describe, and takes precedence over both. Code 2 is dual-use:
+a run that never started cannot have panicked, so read it together with
+whether `runner.log` exists.
+
+A run stopped by a signal exits on this same ladder rather than one of its
+own: what it collected before stopping is still worth reporting, and
+`status.txt` says it was cut short. `chaos.sh` reports 130 for a Ctrl-C of
+its own accord.
 
 ## Creating Custom Scenarios
 

testing/chaos/sim/__main__.py

@@ -60,6 +60,11 @@ def main():
     runner = SimRunner(scenario)
     result = runner.run()
 
+    # Liveness before content. A simulation that raised on its way up, or part
+    # way through, has no panic count or assertion outcome worth reporting, and
+    # saying so is more useful than whatever partial content it did produce.
+    if runner.aborted:
+        sys.exit(4)
     if result and result.panics:
         sys.exit(2)
     if runner.assertions_failed:

testing/chaos/sim/compose.py

@@ -47,7 +47,7 @@ services:
 {% for node in nodes %}
   {{ node.node_id }}:
     <<: *fips-common
-    container_name: fips-node-{{ node.node_id }}
+    container_name: {{ topology.container_name(node.node_id) }}
     hostname: {{ node.node_id }}
     volumes:
       - ./{{ node.node_id }}.yaml:/etc/fips/fips.yaml:ro
@@ -82,6 +82,7 @@ def generate_compose(
         image=FIPS_SIM_IMAGE,
         nodes=nodes,
         resolv_conf=resolv_conf,
+        topology=topology,
     )
 
     path = os.path.join(output_dir, "docker-compose.yml")

testing/chaos/sim/docker_exec.py

@@ -7,6 +7,34 @@ import logging
 
 log = logging.getLogger(__name__)
 
+# `docker compose` renders its progress UI on stderr, so a failed command can
+# have hundreds of lines of captured output behind it. Log the tail: whatever
+# the daemon refused is the last thing it wrote.
+OUTPUT_TAIL_CHARS = 2000
+
+
+def _tail(text: str) -> str:
+    """Trim captured output to its last few KB for logging."""
+    text = text.strip()
+    if not text:
+        return "(empty)"
+    if len(text) <= OUTPUT_TAIL_CHARS:
+        return text
+    return "...\n" + text[-OUTPUT_TAIL_CHARS:]
+
+
+def _decode(output) -> str:
+    """Render captured output as text.
+
+    A timed-out command hands back what it had written as bytes, even when
+    the call asked for text, so the timeout path cannot assume either.
+    """
+    if output is None:
+        return ""
+    if isinstance(output, bytes):
+        return output.decode(errors="replace")
+    return output
+
 
 class DockerExecError(Exception):
     def __init__(self, container, cmd, returncode, stderr):
@@ -47,16 +75,50 @@ def docker_compose(
     timeout: int = 300,
     check: bool = True,
 ) -> subprocess.CompletedProcess:
-    """Run a docker compose command with the given compose file."""
+    """Run a docker compose command with the given compose file.
+
+    Output is captured, so a failure's stderr is logged here before anything
+    else sees it. `CalledProcessError` reports only the argv and the exit
+    status, and a `check=False` caller reads neither, so without this the one
+    place the daemon says what it objected to -- a container name already in
+    use, an unusable subnet, a missing image -- is captured and then thrown
+    away. Nothing about the success path changes.
+    """
     cmd = ["docker", "compose", "-f", compose_file] + args
     log.info("Running: %s", " ".join(cmd))
-    return subprocess.run(
-        cmd,
-        capture_output=True,
-        text=True,
-        timeout=timeout,
-        check=check,
-    )
+    try:
+        result = subprocess.run(
+            cmd,
+            capture_output=True,
+            text=True,
+            timeout=timeout,
+        )
+    except subprocess.TimeoutExpired as e:
+        # A timeout raises from inside communicate(), so the return-code
+        # branch below never runs and this is the only chance to say what
+        # the command had managed to emit. The partials arrive as bytes
+        # even under text=True.
+        log.error(
+            "%s timed out after %ds\nstderr: %s\nstdout: %s",
+            " ".join(cmd),
+            timeout,
+            _tail(_decode(e.stderr)),
+            _tail(_decode(e.stdout)),
+        )
+        raise
+    if result.returncode != 0:
+        log.error(
+            "%s exited %d\nstderr: %s\nstdout: %s",
+            " ".join(cmd),
+            result.returncode,
+            _tail(result.stderr),
+            _tail(result.stdout),
+        )
+        if check:
+            raise subprocess.CalledProcessError(
+                result.returncode, cmd, result.stdout, result.stderr
+            )
+    return result
 
 
 def is_container_running(container: str) -> bool:

testing/chaos/sim/logs.py

@@ -29,9 +29,18 @@ __all__ = ["AnalysisResult", "analyze_logs", "collect_logs", "write_sim_metadata
 
 
 def collect_logs(container_names: list[str], output_dir: str) -> dict[str, str]:
-    """Collect all output (stdout + stderr) from all containers."""
+    """Collect all output (stdout + stderr) from all containers.
+
+    Raises RuntimeError if any container's output could not be read, or if
+    there was nothing to read. `docker logs` writes its own failures to
+    stderr and exits non-zero, so without the returncode check the daemon's
+    "No such container" reply was stored as though it were the node's log
+    and analysed as a mesh with no panics, no errors and no sessions --
+    which reads exactly like a clean run.
+    """
     os.makedirs(output_dir, exist_ok=True)
     logs = {}
+    failed = []
 
     for name in container_names:
         try:
@@ -41,17 +50,35 @@ def collect_logs(container_names: list[str], output_dir: str) -> dict[str, str]:
                 text=True,
                 timeout=30,
             )
-            raw = result.stdout + result.stderr
-            log_text = strip_ansi(raw)
-            logs[name] = log_text
-
-            path = os.path.join(output_dir, f"{name}.log")
-            with open(path, "w") as f:
-                f.write(log_text)
-
-        except (subprocess.TimeoutExpired, Exception) as e:
+        except Exception as e:
             log.warning("Failed to collect logs from %s: %s", name, e)
-            logs[name] = ""
+            failed.append(name)
+            continue
+
+        if result.returncode != 0:
+            log.warning(
+                "docker logs %s exited %d: %s",
+                name, result.returncode, result.stderr.strip(),
+            )
+            failed.append(name)
+            continue
+
+        # A container's own stderr comes back on our stderr, so both
+        # streams are log content on the success path.
+        log_text = strip_ansi(result.stdout + result.stderr)
+        logs[name] = log_text
+
+        path = os.path.join(output_dir, f"{name}.log")
+        with open(path, "w") as f:
+            f.write(log_text)
+
+    if failed:
+        raise RuntimeError(
+            f"could not collect logs from {len(failed)}/{len(container_names)} "
+            f"containers: {', '.join(failed)}"
+        )
+    if not logs:
+        raise RuntimeError("no container logs were collected")
 
     return logs
 

testing/chaos/sim/naming.py

@@ -0,0 +1,41 @@
+"""Scoping suffix for names that live in a global namespace."""
+
+from __future__ import annotations
+
+import hashlib
+import os
+import sys
+
+
+def name_suffix() -> str:
+    """Return the suffix appended to globally-scoped names.
+
+    Docker container names and the generated-config directory are shared
+    across every simulation running on the host, so the harness exports
+    ``FIPS_CI_NAME_SUFFIX`` to keep concurrent runs and concurrent
+    scenarios apart. The suffix is empty when the variable is unset, so a
+    bare ``chaos.sh`` run renders exactly the same names as it always has.
+    """
+    return os.environ.get("FIPS_CI_NAME_SUFFIX", "")
+
+
+def veth_token(suffix: str) -> str:
+    """Shorten a name suffix to four hex characters.
+
+    Host interface names have only 15 characters to work with, far fewer
+    than the suffix needs, so concurrent scenarios are told apart by a
+    hash of it instead. Empty for an empty suffix, so a bare run's
+    interface names are unchanged.
+    """
+    if not suffix:
+        return ""
+    return hashlib.sha1(suffix.encode()).hexdigest()[:4]
+
+
+if __name__ == "__main__":
+    # Print the token for each suffix given on the command line, one per
+    # line. ci-cleanup.sh reaps host interfaces by token and calls this
+    # rather than re-deriving the hash, so widening the token here cannot
+    # leave the reaper matching the old width.
+    for arg in sys.argv[1:]:
+        print(veth_token(arg))

testing/chaos/sim/runner.py

@@ -20,6 +20,7 @@ from .docker_exec import docker_compose
 from .link_swap import LinkSwapManager
 from .links import LinkManager
 from .logs import AnalysisResult, analyze_logs, collect_logs, write_sim_metadata
+from .naming import name_suffix
 from .netem import NetemManager
 from .nodes import NodeManager
 from .peer_churn import PeerChurnManager
@@ -40,6 +41,18 @@ class SimRunner:
         self.output_dir: str = self._resolve_output_dir(scenario)
         self._interrupted = False
 
+        # Set when setup, warmup or the simulation loop raises. The exception
+        # is logged rather than propagated, so nothing else on the return path
+        # of run() carries the fact that the simulation did not complete.
+        self.aborted = False
+
+        # Whether this run ever owned containers. Teardown runs from a
+        # finally, so it runs after a failed setup too, and container names
+        # are global: without this it would harvest whatever currently
+        # answers to them. topology and compose_file are both set well
+        # before the containers start and so cannot stand in for it.
+        self._containers_started = False
+
         # Shared set of currently-down node IDs (updated by NodeManager,
         # read by NetemManager, LinkManager, TrafficManager)
         self._down_nodes: set[str] = set()
@@ -82,7 +95,12 @@ class SimRunner:
             self._warmup()
             self._simulation_loop()
         except Exception:
+            # Log rather than re-raise: the default excepthook writes to stderr
+            # and would not reach the file handler installed in _setup(), so a
+            # re-raise would lose the traceback from runner.log, which is the
+            # artifact that survives into CI. The flag carries the abort out.
             log.exception("Simulation failed")
+            self.aborted = True
         finally:
             result = self._teardown()
 
@@ -133,9 +151,17 @@ class SimRunner:
             log.info("  %s: peers=%s", nid, ",".join(peers))
 
         # 2. Generate configs
+        #
+        # The directory carries the same suffix as the container names, so
+        # parallel scenarios cannot overwrite each other's compose file
+        # between writing it and starting containers from it.
         docker_network_dir = os.path.join(os.path.dirname(__file__), "..")
         config_dir = os.path.normpath(
-            os.path.join(docker_network_dir, "generated-configs", "sim")
+            os.path.join(
+                docker_network_dir,
+                "generated-configs",
+                f"sim{self.topology.name_suffix}",
+            )
         )
         # Select ephemeral identity nodes (if peer churn enabled)
         self._ephemeral_nodes: set[str] = set()
@@ -172,6 +198,7 @@ class SimRunner:
         # 5. Start containers
         log.info("Starting %d containers...", len(self.topology.nodes))
         docker_compose(self.compose_file, ["up", "-d"])
+        self._containers_started = True
 
         # 6. Set up veth pairs for Ethernet edges (before netem)
         #
@@ -409,11 +436,57 @@ class SimRunner:
     def assertions_failed(self) -> bool:
         return any(not o.passed for o in self.assertion_outcomes)
 
+    def _run_status(self) -> str:
+        """Name for how the run itself ended, before teardown is considered."""
+        if self.aborted:
+            return "aborted"
+        if self._interrupted:
+            return "interrupted"
+        return "completed"
+
+    def _write_status(self, status: str) -> None:
+        """Record how the run ended, for a reader who has only this directory.
+
+        One field per line so it can be grepped. The container names are
+        included because they are the handle on everything else the run
+        touched, and because a directory that names them cannot be mistaken
+        for a directory assembled from some other scenario's mesh.
+
+        Never raises. Both callers run this immediately before stopping the
+        containers, so letting a full disk or a vanished output directory
+        out of here would leak the mesh it was about to tear down.
+        """
+        try:
+            os.makedirs(self.output_dir, exist_ok=True)
+            path = os.path.join(self.output_dir, "status.txt")
+            with open(path, "w") as f:
+                f.write(f"status={status}\n")
+                f.write(f"scenario={self.scenario.name}\n")
+                f.write(f"seed={self.scenario.seed}\n")
+                f.write(f"containers=fips-node-nNN{name_suffix()}\n")
+        except Exception:
+            log.exception("Could not write status file")
+
     def _teardown(self) -> AnalysisResult | None:
         """Stop dynamic elements, collect logs, analyze, stop containers."""
-        result = None
+        if not self._containers_started:
+            # There is no mesh to harvest. Snapshots, logs, analysis and
+            # assertions all address containers by a name that is global to
+            # the host, so running them here would describe whoever holds
+            # those names now and leave a plausible report of a run that
+            # never happened. Leaving them out makes the presence of
+            # analysis.txt proof that this scenario's own mesh existed.
+            self._write_status("setup-failed")
+            if self.compose_file:
+                # `up -d` can fail part way through, so this run may own
+                # containers or a network even with no mesh to speak of.
+                log.info("Stopping containers...")
+                docker_compose(self.compose_file, ["down"], check=False)
+            return None
 
-        if self.topology and self.compose_file:
+        result = None
+        status = self._run_status()
+        try:
             # Evaluate post-run assertions before doing any teardown so
             # control sockets are still reachable.
             self._evaluate_assertions()
@@ -498,8 +571,19 @@ class SimRunner:
             if self.veth_mgr:
                 log.info("Cleaning up veth pairs...")
                 self.veth_mgr.teardown_all()
-
-            # Stop containers
+        except Exception:
+            # Same reasoning as run(): logging keeps the traceback in
+            # runner.log, where re-raising would send it to stderr instead
+            # and past the exit ladder, which would report a harvest that
+            # fell over as a bad command line.
+            log.exception("Teardown failed")
+            self.aborted = True
+            status = "teardown-failed"
+            result = None
+        finally:
+            # Status first: it is the one artifact that must exist whatever
+            # else happens, and stopping containers can still time out.
+            self._write_status(status)
             log.info("Stopping containers...")
             docker_compose(
                 self.compose_file,

testing/chaos/sim/topology.py

@@ -8,6 +8,7 @@ from collections import deque
 from dataclasses import dataclass, field
 
 from .keys import derive
+from .naming import name_suffix, veth_token
 from .scenario import TopologyConfig
 
 
@@ -28,6 +29,18 @@ class SimTopology:
     edges: set[tuple[str, str]] = field(default_factory=set)
     # Per-edge transport type; edges not in this dict default to "udp"
     edge_transport: dict[tuple[str, str], str] = field(default_factory=dict)
+    # Suffix scoping globally-visible names to this run and scenario; empty
+    # outside the CI harness, which keeps a bare run's names unchanged.
+    name_suffix: str = ""
+
+    @property
+    def veth_token(self) -> str:
+        """Short stand-in for the suffix, for names bound by IFNAMSIZ.
+
+        Derived rather than stored so no caller can build a topology whose
+        host names are scoped differently from its container names.
+        """
+        return veth_token(self.name_suffix)
 
     def transport_for_edge(self, a: str, b: str) -> str:
         """Get the transport type for an edge (defaults to 'udp')."""
@@ -107,7 +120,27 @@ class SimTopology:
         return not connected
 
     def container_name(self, node_id: str) -> str:
-        return f"fips-node-{node_id}"
+        return f"fips-node-{node_id}{self.name_suffix}"
+
+    def veth_host_name(self, node_a: str, node_b: str, end: str) -> str:
+        """Generate the host-namespace veth name for one end of an edge.
+
+        Format: ``vh{token}{NN}{MM}{end}`` (max 15 chars for IFNAMSIZ).
+        Host interfaces are global, so the token keeps a scenario from
+        deleting a concurrent scenario's pair; it is empty outside the CI
+        harness, yielding the same "vh0104a" this has always produced.
+
+        ``node_a`` and ``node_b`` must be in canonical edge order. Unlike
+        ``veth_interface_name()`` this is not symmetric: the far end is
+        ``end="b"`` on the same ordering, so swapping the arguments names
+        an interface that does not exist.
+        """
+        nn_local = node_a.replace("n", "")
+        nn_peer = node_b.replace("n", "")
+        name = f"vh{self.veth_token}{nn_local}{nn_peer}{end}"
+        if len(name) > 15:
+            raise ValueError(f"veth host name too long: {name!r} ({len(name)} > 15)")
+        return name
 
     def directed_outbound(self) -> dict[str, list[str]]:
         """Assign each static-config edge to exactly one node for outbound connection.
@@ -219,7 +252,14 @@ def generate_topology(
         nodes[a].peers.append(b)
         nodes[b].peers.append(a)
 
-    topo = SimTopology(nodes=nodes, edges=edges, edge_transport=edge_transport)
+    # Read the environment once, here, so every name a run produces comes
+    # from the same value.
+    topo = SimTopology(
+        nodes=nodes,
+        edges=edges,
+        edge_transport=edge_transport,
+        name_suffix=name_suffix(),
+    )
 
     # Connectivity check with retry
     if config.ensure_connected:

testing/chaos/sim/veth.py

@@ -4,7 +4,8 @@ Creates veth pairs between Docker containers for Ethernet-transport
 edges. Each Ethernet edge gets a veth pair with one end moved into
 each container's network namespace. Naming:
 
-  Host (temporary):  vh{NN}{MM}a / vh{NN}{MM}b
+  Host (temporary):  vh{token}{NN}{MM}a / vh{token}{NN}{MM}b
+                     (via SimTopology.veth_host_name())
   Container:         ve-{local}-{peer}  (via veth_interface_name())
 
 After creation, the container-side MAC addresses are queried and
@@ -113,9 +114,7 @@ class VethManager:
             if a != node_id and b != node_id:
                 continue
             # Remove existing pair if any (host-side might still exist)
-            nn_a = a.replace("n", "")
-            nn_b = b.replace("n", "")
-            host_a = f"vh{nn_a}{nn_b}a"
+            host_a = self.topology.veth_host_name(a, b, "a")
             _run_host(["ip", "link", "delete", host_a], image, check=False)
             # Re-create
             self._create_veth_pair(a, b, image)
@@ -142,14 +141,14 @@ class VethManager:
             return
 
         # Generate names
-        nn_a = node_a.replace("n", "")
-        nn_b = node_b.replace("n", "")
-        host_a = f"vh{nn_a}{nn_b}a"
-        host_b = f"vh{nn_a}{nn_b}b"
+        host_a = self.topology.veth_host_name(node_a, node_b, "a")
+        host_b = self.topology.veth_host_name(node_a, node_b, "b")
         final_a = veth_interface_name(node_a, node_b)
         final_b = veth_interface_name(node_b, node_a)
 
-        # Clean up any stale pair
+        # Clean up a stale pair left by this scenario. The token makes the
+        # name unique to this run, so a pair orphaned by an earlier run is
+        # no longer reclaimed here — `ci-cleanup.sh` reaps those instead.
         _run_host(["ip", "link", "delete", host_a], image, check=False)
 
         # Create veth pair on host
@@ -253,7 +252,11 @@ def _run_host(cmd: list[str], image: str, check: bool = True) -> bool:
             timeout=30,
         )
         if check and result.returncode != 0:
-            log.debug(
+            # Warning, not debug: the runner logs at INFO unless asked for
+            # -v, so at debug this never reached runner.log and the callers
+            # below report only that a pair could not be created. The
+            # check=False deletes are expected to fail and stay silent.
+            log.warning(
                 "ip cmd failed: %s -> %s",
                 " ".join(cmd),
                 result.stderr.strip(),

testing/ci-cleanup.sh

@@ -1,5 +1,5 @@
 #!/bin/bash
-# Reap FIPS CI docker resources: containers, networks, volumes, images.
+# Reap FIPS CI resources: containers, networks, volumes, images, veth pairs.
 #
 # Force-removes everything created by ci-local.sh that is still around —
 # whether a run finished cleanly, was preempted (SIGTERM/SIGKILL), OOM-killed,
@@ -19,26 +19,56 @@
 # the label sweep to one run; a run's own teardown always does. Without it the
 # label sweep stays broad, which is what a manual "reap everything" wants.
 #
+# Host-namespace veth interfaces are the one non-docker resource reaped here.
+# The chaos simulation creates each pair in the host namespace and then moves
+# the two ends into container namespaces, so a run killed in between leaves the
+# pair behind, and nothing else on the box removes it. Their names carry a
+# short token derived from the scenario's name suffix, so --veth-suffixes takes
+# the suffixes a run used and reaps only the tokens those could produce. Only a
+# reap with neither --run-id nor --veth-suffixes matches every simulation veth
+# name, in step with the broad label sweep. --project-prefix does not scope the
+# veth sweep at all: a host interface carries no compose project.
+#
+# These interfaces are also the one resource reaped WITHOUT regard to the CI
+# label — they cannot carry one — so an unscoped reap takes the host ends of a
+# bare `chaos.sh` simulation's veth pairs too, even though it leaves that
+# simulation's unlabelled containers running. Scope with --veth-suffixes (or
+# just don't reap) while a bare simulation is up.
+#
 # Usage:
 #   ci-cleanup.sh                       Reap ALL fips-ci resources (any run)
 #   ci-cleanup.sh --project-prefix P    Restrict the compose-project sweep to
 #                                       names starting with P (scopes the reap
-#                                       to a single run)
+#                                       to a single run). Does NOT scope the
+#                                       host veth sweep — see --veth-suffixes
 #   ci-cleanup.sh --run-id ID           Restrict the label sweep to the run
 #                                       labelled ID (leaves other runs alone)
 #   ci-cleanup.sh --label L             Override the CI label (default above)
 #   ci-cleanup.sh --images "a b,c"      Also `docker rmi -f` these image tags
 #                                       (space- or comma-separated)
+#   ci-cleanup.sh --veth-suffixes "a b" Restrict the host veth sweep to the
+#                                       name suffixes a single run used
+#                                       (space- or comma-separated). Without
+#                                       it the sweep reaches every simulation
+#                                       veth name on the host, including a
+#                                       bare `chaos.sh` run's live ones
 #
 # Safe to run when there is nothing to reap, and safe to run repeatedly.
 # Also reachable as `ci-local.sh --reap`.
 set -uo pipefail
 
+SCRIPT_DIR="$(cd "$(dirname "$0")" && pwd)"
+
 LABEL="com.corganlabs.fips-ci=1"
 RUN_LABEL_KEY="com.corganlabs.fips-ci.run"
 PROJECT_PREFIX="fipsci_"   # broad default: every CI run
 RUN_ID=""                  # broad default: every CI run
 IMAGES=""
+VETH_SUFFIXES=""           # empty AND no --run-id: every simulation veth name
+# ip(8) runs inside this image, the same way the simulation creates the
+# interfaces, so the reap works wherever the simulation does. The chaos
+# simulation builds it, and it carries iproute2.
+VETH_IMAGE="fips-test:latest"
 
 while [[ $# -gt 0 ]]; do
     case "$1" in
@@ -46,6 +76,7 @@ while [[ $# -gt 0 ]]; do
         --project-prefix) PROJECT_PREFIX="$2"; shift 2 ;;
         --run-id)         RUN_ID="$2"; shift 2 ;;
         --images)         IMAGES="$2"; shift 2 ;;
+        --veth-suffixes)  VETH_SUFFIXES="$2"; shift 2 ;;
         -h|--help)        sed -n '2,/^set /{ /^set /d; s/^# \?//; p }' "$0"; exit 0 ;;
         *)                echo "Unknown option: $1" >&2; exit 2 ;;
     esac
@@ -107,6 +138,107 @@ reap_volumes() {
         | xargs -r timeout "$TMO" docker volume rm >/dev/null 2>&1 || true
 }
 
+# Every failure below leaves interfaces behind rather than widening the sweep,
+# so each one is silent by construction. Say so on stderr instead, or a reap
+# that reclaimed nothing looks exactly like a reap that had nothing to reclaim.
+veth_warn() { echo "ci-cleanup: host veth sweep skipped: $*" >&2; }
+
+# One node id, exactly as chaos/sim/topology.py renders it (f"n{i+1:02d}"):
+# zero-padded to two digits, and never zero-padded beyond that. Spelling it out
+# keeps shapes the simulation cannot emit (vh01020a) out of the sweep.
+VETH_NODE_ID='(0[0-9]|[1-9][0-9]+)'
+
+# Regex matching the host veth names to remove. With --veth-suffixes it covers
+# only the tokens those suffixes hash to, so a concurrent run's interfaces —
+# which carry a different token — cannot match. The token derivation is read
+# from the simulation itself rather than repeated here, so widening it cannot
+# leave this matching the old width. Empty output means "reap nothing".
+veth_pattern() {
+    # vh{token}{NN}{MM}{a,b}: the token is 4 hex or wholly absent — never a
+    # part of one — and the two node ids follow. Anchored and shaped this
+    # tightly so the unscoped sweep cannot reach an interface the simulation
+    # never made. Broad only for an unscoped reap: once --run-id names a
+    # single run, no missing or empty suffix list may widen this back out to
+    # every run.
+    if [[ -z "$RUN_ID" && -z "$VETH_SUFFIXES" ]]; then
+        printf '^vh([0-9a-f]{4})?%s%s[ab]$' "$VETH_NODE_ID" "$VETH_NODE_ID"
+        return 0
+    fi
+    if [[ -z "$VETH_SUFFIXES" ]]; then
+        veth_warn "--run-id given with no --veth-suffixes"
+        return 0
+    fi
+    local sfx=() tok alt="" out rc
+    read -ra sfx <<< "${VETH_SUFFIXES//,/ }"
+    if [[ ${#sfx[@]} -eq 0 ]]; then
+        veth_warn "--veth-suffixes is empty"
+        return 0
+    fi
+    if ! command -v python3 >/dev/null 2>&1; then
+        veth_warn "python3 not found, cannot derive interface tokens"
+        return 0
+    fi
+    # SCRIPT_DIR comes from $0, so invoking this script through a symlink or a
+    # copy on $PATH points PYTHONPATH at a tree with no simulation in it. Say
+    # which tree was tried when the derivation fails, rather than no-opping.
+    out="$(PYTHONPATH="$SCRIPT_DIR/chaos" python3 -m sim.naming "${sfx[@]}" 2>&1)"
+    rc=$?
+    if [[ $rc -ne 0 ]]; then
+        veth_warn "token derivation failed under $SCRIPT_DIR/chaos: $out"
+        return 0
+    fi
+    for tok in $out; do
+        [[ "$tok" =~ ^[0-9a-f]{4}$ ]] && alt="${alt:+$alt|}$tok"
+    done
+    if [[ -z "$alt" ]]; then
+        veth_warn "no interface tokens derived from: ${sfx[*]}"
+        return 0
+    fi
+    printf '^vh(%s)%s%s[ab]$' "$alt" "$VETH_NODE_ID" "$VETH_NODE_ID"
+}
+
+# ip(8) in a privileged --net=host container, matching how the simulation
+# creates these interfaces (see chaos/sim/veth.py).
+veth_ip() {
+    timeout "$TMO" docker run --rm --privileged --net=host \
+        --entrypoint ip "$VETH_IMAGE" "$@" 2>/dev/null
+}
+
+# The same, taking `ip -batch` commands on stdin so any number of deletes costs
+# one container. -force keeps ip going past an interface that vanished under us.
+veth_ip_batch() {
+    timeout "$TMO" docker run --rm -i --privileged --net=host \
+        --entrypoint ip "$VETH_IMAGE" -force -batch - >/dev/null 2>&1 || true
+}
+
+reap_veths() {
+    local pattern
+    pattern="$(veth_pattern)"
+    [[ -z "$pattern" ]] && return 0
+    # Without the image there is no way to run ip(8). Orphans can outlive it —
+    # `docker image prune -a`, a build host that prunes between runs, or a run
+    # aborted before ci-local.sh retags :latest all remove it while interfaces
+    # are still up — so this is a real skip, not "nothing was ever run here".
+    if ! docker image inspect "$VETH_IMAGE" >/dev/null 2>&1; then
+        veth_warn "image $VETH_IMAGE not present, cannot run ip(8)"
+        return 0
+    fi
+    # Both ends of a pair match, but deleting either removes both, so collapse
+    # each pair to one delete and issue the lot in a single container. The
+    # whole script runs under a caller-imposed timeout, and a container spawn
+    # per name would put a large orphan set at risk of exhausting it. `sort -u`
+    # orders `...a` before `...b`, so the surviving end of a pair the
+    # simulation was killed part-way through moving is still the one picked.
+    local names
+    names="$(veth_ip -o link show \
+        | sed -E 's/^[0-9]+: ([^:@]+).*/\1/' \
+        | grep -E "$pattern" | sort -u \
+        | awk '{ k = substr($0, 1, length($0) - 1)
+                 if (!(k in seen)) { seen[k] = 1; print } }')"
+    [[ -z "$names" ]] && return 0
+    sed 's/^/link delete /' <<< "$names" | veth_ip_batch
+}
+
 reap_images() {
     [[ -z "$IMAGES" ]] && return 0
     local imgs
@@ -115,10 +247,12 @@ reap_images() {
     timeout "$TMO" docker rmi -f "${imgs[@]}" >/dev/null 2>&1 || true
 }
 
-# Order matters: containers reference networks/volumes, so drop them first.
+# Order matters: containers reference networks/volumes, so drop them first, and
+# the veth sweep needs an image to run ip(8) in, so it precedes the image reap.
 reap_containers
 reap_networks
 reap_volumes
+reap_veths
 reap_images
 
 exit 0

testing/ci-local.sh

@@ -14,9 +14,14 @@
 #   --list               List available integration suites
 #   --check-parity       Verify this suite set matches ci.yml's integration
 #                        matrix (see testing/check-ci-parity.sh), then exit
-#   --reap               Force-remove all leftover FIPS CI docker resources
+#   --reap               Force-remove all leftover FIPS CI resources
 #                        (containers/networks/volumes carrying the CI label or a
-#                        fipsci_ compose project), then exit. See ci-cleanup.sh.
+#                        fipsci_ compose project, plus every chaos-simulation
+#                        host veth interface), then exit. See ci-cleanup.sh.
+#                        Host interfaces carry no label, so they are matched by
+#                        name shape: this reaps a bare chaos.sh run's live
+#                        interfaces too, though not its containers. Don't run
+#                        it while a bare simulation is up.
 #   -h, --help           Show this help
 #
 # Integration suites (default coverage):
@@ -271,12 +276,25 @@ export FIPS_TEST_APP_IMAGE="$CI_IMAGE_APP"
 # container by name. Empty when unset, so a bare `docker compose up` outside
 # this harness still produces today's plain names.
 export FIPS_CI_NAME_SUFFIX="-${CI_RUN_ID}"
+# run_chaos narrows that export to one scenario, and bash scopes a `local`
+# dynamically: a function called while run_chaos is on the stack — the signal
+# trap, on the `--only chaos-*` path where run_chaos is not a subshell — reads
+# the narrowed value, not this one. Snapshot the run-wide value so the rebuild
+# below always sees it, and make it readonly so no scope can shadow it either.
+CI_RUN_NAME_SUFFIX="$FIPS_CI_NAME_SUFFIX"
+readonly CI_RUN_NAME_SUFFIX
 
 # Per-suite compose project name: ${prefix}_<suite-or-compose-basename>. Keeps
 # today's intra-run distinctness (one project per compose file / chaos child)
 # while adding the cross-run prefix that scopes the reap.
 ci_project() { printf '%s_%s' "$CI_PROJECT_PREFIX" "$1"; }
 
+# The name suffix one chaos scenario runs under. Its container names, its
+# generated-config directory and the token in its host veth names all derive
+# from this, so teardown recomputes it to know which interfaces are ours. Reads
+# the snapshot rather than the export for the reason given above.
+ci_chaos_suffix() { printf -- '-%s%s' "$1" "$CI_RUN_NAME_SUFFIX"; }
+
 # PIDs of in-flight parallel chaos children (subshells). The trap signals these.
 CI_CHAOS_PIDS=()
 CI_CLEANED=0
@@ -301,13 +319,25 @@ ci_teardown() {
     fi
 
     # 2. Remove all compose projects + direct-run resources + per-run images
-    #    for this run. ci-cleanup.sh wraps each docker op in `timeout`; bound
-    #    the whole sweep too so the trap can never wedge.
+    #    for this run, plus any host veth interface a chaos scenario was
+    #    killed part-way through creating. Host interfaces carry no docker
+    #    label, so hand over the suffixes this run's scenarios used and let
+    #    the reap derive their names — a blind sweep would take a concurrent
+    #    run's live interfaces with it. ci-cleanup.sh wraps each docker op in
+    #    `timeout`; bound the whole sweep too so the trap can never wedge.
+    #    Its stdout is routine progress and goes nowhere, but stderr carries
+    #    only a skipped sweep or a bad option, and a sweep that quietly stops
+    #    reaping is how interfaces would accumulate unnoticed. Let it through.
+    local _suffixes=() _entry
+    for _entry in "${CHAOS_SUITES[@]}"; do
+        _suffixes+=("$(ci_chaos_suffix "${_entry%% *}")")
+    done
     timeout 150 bash "$SCRIPT_DIR/ci-cleanup.sh" \
         --label "$CI_LABEL" \
         --run-id "$CI_RUN_ID" \
         --project-prefix "$CI_PROJECT_PREFIX" \
-        --images "$CI_IMAGE_TEST $CI_IMAGE_APP" >/dev/null 2>&1 || true
+        --images "$CI_IMAGE_TEST $CI_IMAGE_APP" \
+        --veth-suffixes "${_suffixes[*]}" >/dev/null || true
 }
 
 on_signal() {
@@ -516,9 +546,17 @@ run_chaos() {
     local name="$1"
     shift
     local rc=0
-    # Distinct project per scenario (chaos children run in parallel). When
-    # invoked from a background subshell this export is local to that child.
-    export COMPOSE_PROJECT_NAME="$(ci_project "chaos-$name")"
+    # Distinct project per scenario (chaos children run in parallel). Scoped to
+    # this function so the --only path cannot leak it into a later suite.
+    local -x COMPOSE_PROJECT_NAME="$(ci_project "chaos-$name")"
+
+    # Container names and the generated-config directory are GLOBAL and are not
+    # scoped by the compose project, so narrow the run-wide suffix to this
+    # scenario. Parallel children then cannot claim each other's names or
+    # overwrite each other's compose file.
+    local suffix
+    suffix="$(ci_chaos_suffix "$name")"
+    local -x FIPS_CI_NAME_SUFFIX="$suffix"
 
     info "[chaos/$name] Running simulation"
     if bash testing/chaos/scripts/chaos.sh "$@" 2>&1; then
@@ -528,6 +566,17 @@ run_chaos() {
     fi
 
     record "chaos-$name" $rc
+
+    # record() ends in pass()/fail(), which are echoes, so it returns 0 for any
+    # rc — and without this return so does run_chaos. On the parallel path the
+    # function's status is the child subshell's status, and that is the only
+    # thing the waiting parent can see: the child's own RESULTS entry and its
+    # OVERALL=1 die with its process, and its FAIL line goes to a logfile only
+    # the unreachable branch reads. So a bare record() left every scenario
+    # recorded as a pass whatever the simulation reported. On the --only path
+    # record() already wrote the true rc into the parent's own RESULTS, and
+    # returning it as well is inert: this script does not set -e.
+    return $rc
 }
 
 # Run gateway integration test
@@ -930,7 +979,12 @@ run_suite() {
                 fi
             done
             if [[ "$found" != true ]]; then
-                # Fall back to using the name as the scenario directly
+                # Fall back to using the name as the scenario directly. Note
+                # teardown rebuilds veth suffixes from CHAOS_SUITES only, so a
+                # scenario named here but absent from that list is outside the
+                # host-interface reap. Harmless while every ethernet-transport
+                # scenario is declared there; add one that isn't and its
+                # orphaned pairs will need an unscoped `--reap` to clear.
                 run_chaos "$chaos_name" "$chaos_name"
             fi
             ;;