Retire the ECN A/B test framing and record why suites are excluded

ecn-ab-test.sh was a -test.sh with no path to a failing result: it
asserts nothing, applies no threshold, and nothing invokes it. It also
could not have worked. It read a fixed sim-results/ecn-ab-on/ path while
the runner has written timestamped directories since 2026-03-20, and not
one ecn-ab result directory exists on disk, so it has found neither input
for months and the "+10.2% recv throughput" figure the README carried is
not reproducible from anything available.

Rename it to ecn-ab-compare.sh, fix the path resolution to glob the
timestamped directories, and stop swallowing a failed simulation with
|| true so a broken run cannot feed the comparison.

Deliberately not adding the threshold assertion that would make it
gateable. That needs a calibration corpus, and none exists precisely
because the tool has never produced a kept result; a number invented now
would assert a guess. The prerequisite is a calibration run set and a
decision about what ECN is expected to deliver, which is a protocol
question. Written in the script header rather than left implied.

Add the exclusion block to ci-local.sh naming every suite and scenario
neither runner runs, with the reason for each: interop, boringtun,
iperf-test, this comparison tool, mesh-lab, and the four chaos scenarios
outside CHAOS_SUITES. Until now "not in the suite list" was
indistinguishable from "forgotten".

testing/chaos/README.md

@@ -124,8 +124,14 @@ detection.
 - **ecn-ab-on / ecn-ab-off**: Paired scenarios with identical conditions
   (6-node tree, 10 Mbps egress, 1000 kbps ingress policing, 10ms link
   delay, 8 KB recv buffer) differing only in `ecn.enabled`.
-  `ecn-ab-test.sh` runs both and compares throughput and congestion
-  counters. Initial results: +10.2% recv throughput with ECN enabled.
+  `ecn-ab-compare.sh` runs both and prints a side-by-side of throughput
+  and congestion counters. It is a manual tool, not a test: it asserts
+  nothing and no runner invokes it. The "+10.2% recv throughput with ECN
+  enabled" figure once recorded here is not reproducible from anything on
+  disk — the script read a fixed `sim-results/ecn-ab-on/` path while the
+  runner has written timestamped directories since 2026-03-20, and no
+  ecn-ab result directory survives. The path bug is fixed; the figure is
+  left out until a run produces one.
 
 ### Ingress Traffic Control
 

testing/chaos/ecn-ab-compare.sh

@@ -1,10 +1,27 @@
 #!/usr/bin/env bash
-# ECN A/B Throughput Test
+# ECN A/B Throughput Comparison  (a manual tool, NOT a test)
 #
 # Runs two identical chaos scenarios — one with ECN enabled, one disabled —
-# and compares iperf3 throughput and congestion counter results.
+# and prints a side-by-side of iperf3 throughput and congestion counters.
 #
-# Usage: ./ecn-ab-test.sh [--seed N] [--duration N]
+# Renamed from ecn-ab-test.sh on 2026-07-23. The old name claimed a verdict
+# this has never produced: it asserts nothing, applies no threshold, and no
+# runner invokes it. Naming it a test made it look like coverage.
+#
+# It also could not have worked. It read sim-results/ecn-ab-on/... while the
+# runner has written sim-results/<timestamp>-<scenario>/ since 2026-03-20, so
+# it has found neither input for at least four months, and there is not one
+# archived ecn-ab result directory on disk. That path bug is fixed below.
+#
+# WHAT IS STILL MISSING, and why it was not added: turning this into a real
+# test needs a threshold — how much throughput ECN should buy, or how much
+# lower the congestion counters should run — and there is no corpus to derive
+# one from, precisely because the tool has never produced a kept result. A
+# number invented here would assert the author's guess. The prerequisite is a
+# calibration run set, and that is a protocol question about what ECN is
+# expected to deliver, not a harness one.
+#
+# Usage: ./ecn-ab-compare.sh [--seed N] [--duration N]
 
 set -euo pipefail
 cd "$(dirname "$0")"
@@ -24,12 +41,12 @@ echo ""
 
 # --- Run A: ECN ON ---
 echo "--- Phase A: ECN ENABLED ---"
-sudo python3 -m sim scenarios/ecn-ab-on.yaml "${EXTRA_ARGS[@]}" || true
+sudo python3 -m sim scenarios/ecn-ab-on.yaml "${EXTRA_ARGS[@]}"
 echo ""
 
 # --- Run B: ECN OFF ---
 echo "--- Phase B: ECN DISABLED ---"
-sudo python3 -m sim scenarios/ecn-ab-off.yaml "${EXTRA_ARGS[@]}" || true
+sudo python3 -m sim scenarios/ecn-ab-off.yaml "${EXTRA_ARGS[@]}"
 echo ""
 
 # --- Compare results ---
@@ -37,7 +54,27 @@ echo "=== Results ==="
 echo ""
 
 python3 - <<'PYEOF'
+import glob
 import json
+
+
+def latest(scenario, filename):
+    """Newest run directory for a scenario, or None.
+
+    The runner writes sim-results/<timestamp>-<scenario>/, so a fixed path
+    such as sim-results/ecn-ab-on/ has never matched anything. Sorting the
+    glob works because the timestamp is the leading, fixed-width component.
+    """
+    dirs = sorted(glob.glob(f"sim-results/*-{scenario}"))
+    if not dirs:
+        print(f"  no run directory found for {scenario}")
+        return None
+    path = f"{dirs[-1]}/{filename}"
+    if not glob.glob(path):
+        print(f"  {scenario}: {filename} missing from {dirs[-1]}")
+        return None
+    return path
+
 import os
 import sys
 
@@ -92,8 +129,10 @@ def print_sessions(label, sessions):
     print(f"    {'':>14}  completed={n}  incomplete={incomplete}")
     return total_recv / n
 
-on_results = load_results("sim-results/ecn-ab-on/iperf3-results.json")
-off_results = load_results("sim-results/ecn-ab-off/iperf3-results.json")
+on_path = latest("ecn-ab-on", "iperf3-results.json")
+off_path = latest("ecn-ab-off", "iperf3-results.json")
+on_results = load_results(on_path) if on_path else []
+off_results = load_results(off_path) if off_path else []
 
 on_sessions = extract_throughput(on_results)
 off_sessions = extract_throughput(off_results)
@@ -111,8 +150,10 @@ if avg_on and avg_off:
 
 # Congestion counters
 print("Congestion Counters (final snapshot):")
-for label, path in [("ECN ON", "sim-results/ecn-ab-on/congestion-snapshot-final.json"),
-                     ("ECN OFF", "sim-results/ecn-ab-off/congestion-snapshot-final.json")]:
+for label, scenario in [("ECN ON", "ecn-ab-on"), ("ECN OFF", "ecn-ab-off")]:
+    path = latest(scenario, "congestion-snapshot-final.json")
+    if path is None:
+        continue
     snap = load_congestion(path)
     if not snap:
         print(f"  {label}: no snapshot")

testing/ci-local.sh

@@ -40,6 +40,29 @@
 # Opt-in (require --with-tor; depend on live Tor network):
 #   tor-socks5, tor-directory
 #
+# Deliberately not run by either runner, with the reason for each. Recorded
+# here so that "not in the suite list" stops being indistinguishable from
+# "forgotten", which is what it was until 2026-07-23:
+#   interop/         Manual. Driven by interop-stress.sh, which runs N
+#                    repetitions serially under netem and takes far longer
+#                    than a CI slot. No CI-sized entry point exists yet;
+#                    writing one is the work, not adding a line here.
+#   boringtun/       Comparative benchmark against a non-FIPS implementation.
+#                    Measures rather than asserts, and needs a boringtun
+#                    build CI does not have.
+#   iperf-test.sh    Bandwidth measurement, no pass/fail. Driven manually by
+#                    iperf-compare-refs.sh.
+#   ecn-ab-compare.sh  Manual A/B comparison, renamed from ecn-ab-test.sh
+#                    because it asserts nothing. Making it gateable needs a
+#                    calibration corpus that does not exist; see its header.
+#   mesh-lab/        Long-running multi-host lab, not a suite.
+#   ecn-ab-on, ecn-ab-off, maelstrom, maelstrom-sparse
+#                    Chaos scenarios excluded from CHAOS_SUITES. The two
+#                    ecn-ab ones are halves of the manual comparison above.
+#                    The two maelstrom ones are 600 s stress runs kept for
+#                    manual investigation. All four still load-check and
+#                    carry the default max_errors ceiling if run by hand.
+#
 # Exit codes:
 #   0   — all stages passed
 #   1   — one or more stages failed