diff --git a/testing/chaos/README.md b/testing/chaos/README.md index 401c40e..3fb31d6 100644 --- a/testing/chaos/README.md +++ b/testing/chaos/README.md @@ -124,8 +124,14 @@ detection. - **ecn-ab-on / ecn-ab-off**: Paired scenarios with identical conditions (6-node tree, 10 Mbps egress, 1000 kbps ingress policing, 10ms link delay, 8 KB recv buffer) differing only in `ecn.enabled`. - `ecn-ab-test.sh` runs both and compares throughput and congestion - counters. Initial results: +10.2% recv throughput with ECN enabled. + `ecn-ab-compare.sh` runs both and prints a side-by-side of throughput + and congestion counters. It is a manual tool, not a test: it asserts + nothing and no runner invokes it. The "+10.2% recv throughput with ECN + enabled" figure once recorded here is not reproducible from anything on + disk — the script read a fixed `sim-results/ecn-ab-on/` path while the + runner has written timestamped directories since 2026-03-20, and no + ecn-ab result directory survives. The path bug is fixed; the figure is + left out until a run produces one. ### Ingress Traffic Control diff --git a/testing/chaos/ecn-ab-test.sh b/testing/chaos/ecn-ab-compare.sh similarity index 60% rename from testing/chaos/ecn-ab-test.sh rename to testing/chaos/ecn-ab-compare.sh index 1aa47e5..29fe0ed 100755 --- a/testing/chaos/ecn-ab-test.sh +++ b/testing/chaos/ecn-ab-compare.sh @@ -1,10 +1,27 @@ #!/usr/bin/env bash -# ECN A/B Throughput Test +# ECN A/B Throughput Comparison (a manual tool, NOT a test) # # Runs two identical chaos scenarios — one with ECN enabled, one disabled — -# and compares iperf3 throughput and congestion counter results. +# and prints a side-by-side of iperf3 throughput and congestion counters. # -# Usage: ./ecn-ab-test.sh [--seed N] [--duration N] +# Renamed from ecn-ab-test.sh on 2026-07-23. The old name claimed a verdict +# this has never produced: it asserts nothing, applies no threshold, and no +# runner invokes it. Naming it a test made it look like coverage. +# +# It also could not have worked. It read sim-results/ecn-ab-on/... while the +# runner has written sim-results/-/ since 2026-03-20, so +# it has found neither input for at least four months, and there is not one +# archived ecn-ab result directory on disk. That path bug is fixed below. +# +# WHAT IS STILL MISSING, and why it was not added: turning this into a real +# test needs a threshold — how much throughput ECN should buy, or how much +# lower the congestion counters should run — and there is no corpus to derive +# one from, precisely because the tool has never produced a kept result. A +# number invented here would assert the author's guess. The prerequisite is a +# calibration run set, and that is a protocol question about what ECN is +# expected to deliver, not a harness one. +# +# Usage: ./ecn-ab-compare.sh [--seed N] [--duration N] set -euo pipefail cd "$(dirname "$0")" @@ -24,12 +41,12 @@ echo "" # --- Run A: ECN ON --- echo "--- Phase A: ECN ENABLED ---" -sudo python3 -m sim scenarios/ecn-ab-on.yaml "${EXTRA_ARGS[@]}" || true +sudo python3 -m sim scenarios/ecn-ab-on.yaml "${EXTRA_ARGS[@]}" echo "" # --- Run B: ECN OFF --- echo "--- Phase B: ECN DISABLED ---" -sudo python3 -m sim scenarios/ecn-ab-off.yaml "${EXTRA_ARGS[@]}" || true +sudo python3 -m sim scenarios/ecn-ab-off.yaml "${EXTRA_ARGS[@]}" echo "" # --- Compare results --- @@ -37,7 +54,27 @@ echo "=== Results ===" echo "" python3 - <<'PYEOF' +import glob import json + + +def latest(scenario, filename): + """Newest run directory for a scenario, or None. + + The runner writes sim-results/-/, so a fixed path + such as sim-results/ecn-ab-on/ has never matched anything. Sorting the + glob works because the timestamp is the leading, fixed-width component. + """ + dirs = sorted(glob.glob(f"sim-results/*-{scenario}")) + if not dirs: + print(f" no run directory found for {scenario}") + return None + path = f"{dirs[-1]}/{filename}" + if not glob.glob(path): + print(f" {scenario}: {filename} missing from {dirs[-1]}") + return None + return path + import os import sys @@ -92,8 +129,10 @@ def print_sessions(label, sessions): print(f" {'':>14} completed={n} incomplete={incomplete}") return total_recv / n -on_results = load_results("sim-results/ecn-ab-on/iperf3-results.json") -off_results = load_results("sim-results/ecn-ab-off/iperf3-results.json") +on_path = latest("ecn-ab-on", "iperf3-results.json") +off_path = latest("ecn-ab-off", "iperf3-results.json") +on_results = load_results(on_path) if on_path else [] +off_results = load_results(off_path) if off_path else [] on_sessions = extract_throughput(on_results) off_sessions = extract_throughput(off_results) @@ -111,8 +150,10 @@ if avg_on and avg_off: # Congestion counters print("Congestion Counters (final snapshot):") -for label, path in [("ECN ON", "sim-results/ecn-ab-on/congestion-snapshot-final.json"), - ("ECN OFF", "sim-results/ecn-ab-off/congestion-snapshot-final.json")]: +for label, scenario in [("ECN ON", "ecn-ab-on"), ("ECN OFF", "ecn-ab-off")]: + path = latest(scenario, "congestion-snapshot-final.json") + if path is None: + continue snap = load_congestion(path) if not snap: print(f" {label}: no snapshot") diff --git a/testing/ci-local.sh b/testing/ci-local.sh index f0137fb..fc1fae6 100755 --- a/testing/ci-local.sh +++ b/testing/ci-local.sh @@ -40,6 +40,29 @@ # Opt-in (require --with-tor; depend on live Tor network): # tor-socks5, tor-directory # +# Deliberately not run by either runner, with the reason for each. Recorded +# here so that "not in the suite list" stops being indistinguishable from +# "forgotten", which is what it was until 2026-07-23: +# interop/ Manual. Driven by interop-stress.sh, which runs N +# repetitions serially under netem and takes far longer +# than a CI slot. No CI-sized entry point exists yet; +# writing one is the work, not adding a line here. +# boringtun/ Comparative benchmark against a non-FIPS implementation. +# Measures rather than asserts, and needs a boringtun +# build CI does not have. +# iperf-test.sh Bandwidth measurement, no pass/fail. Driven manually by +# iperf-compare-refs.sh. +# ecn-ab-compare.sh Manual A/B comparison, renamed from ecn-ab-test.sh +# because it asserts nothing. Making it gateable needs a +# calibration corpus that does not exist; see its header. +# mesh-lab/ Long-running multi-host lab, not a suite. +# ecn-ab-on, ecn-ab-off, maelstrom, maelstrom-sparse +# Chaos scenarios excluded from CHAOS_SUITES. The two +# ecn-ab ones are halves of the manual comparison above. +# The two maelstrom ones are 600 s stress runs kept for +# manual investigation. All four still load-check and +# carry the default max_errors ceiling if run by hand. +# # Exit codes: # 0 — all stages passed # 1 — one or more stages failed