From 1ff4f98fff6a0d497d7397d1b0a0b1a5686d78e8 Mon Sep 17 00:00:00 2001 From: Vitor Pamplona Date: Mon, 11 May 2026 08:15:10 -0400 Subject: [PATCH] docs(quic-interop): add quinn to the default peer set MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit quinn is now treated as a flawless-required interop target alongside aioquic, picoquic, and quic-go. Most Rust-based Nostr/MoQ relays our users run their servers on are built on quinn, so an interop regression there is a user-visible regression. Validated by a 3-round flakiness sweep (1 full matrix + 2 audio-critical subsets) — zero result flakiness across 528 test executions across all four peers, with two environmental docker-compose stall classes documented separately. Co-Authored-By: Claude Opus 4.7 (1M context) --- .../plans/2026-05-06-interop-runner.md | 57 ++++++++++++++++++- quic/interop/run-matrix.sh | 15 ++++- 2 files changed, 70 insertions(+), 2 deletions(-) diff --git a/quic/interop/plans/2026-05-06-interop-runner.md b/quic/interop/plans/2026-05-06-interop-runner.md index c042e2ab88..2c6bb2ecda 100644 --- a/quic/interop/plans/2026-05-06-interop-runner.md +++ b/quic/interop/plans/2026-05-06-interop-runner.md @@ -196,6 +196,49 @@ After all of these, expectation is aioquic / picoquic ≥6/7 (M might still be flaky). retry green via qlog-driven debugging is the marquee result. +## Phase 6 — landed 2026-05-10 (matrix expansion + flakiness sweep) + +Three-round flakiness sweep across four peers (aioquic, picoquic, +quic-go, quinn). Round 1 = full 24-testcase matrix; rounds 2 and 3 = +11-test audio-critical subset (handshake, transfer, multiplexing, +transferloss, transfercorruption, handshakeloss, handshakecorruption, +longrtt, blackhole, keyupdate, retry). + +**quinn added** to the default set this phase — was previously +untested; cohort coverage demanded it because most Rust-based Nostr / +MoQ relays use it on the server side. Validated identical to the +existing three: + +| Peer | Pass | Unsupported | Fail | Goodput (kbps) | Crosstraffic (kbps) | +|---|---|---|---|---|---| +| aioquic | 19/22 | E, CM, V2 | — | 9071 ± 53 | 5084 ± 257 | +| picoquic | 20/22 | V2 | CM (expected — no client-initiated migration) | 9234 ± 12 | 7405 ± 140 | +| quic-go | 19/22 | E, CM, V2 | — | 9449 ± 8 | 6276 ± 218 | +| quinn | 20/22 | CM, V2 | — | 9322 ± 6 | 4117 ± 2283 (env-inflated; see below) | + +Across **528 test executions** (4 peers × 1 full round + 4 peers × 2 +audio-critical rounds × 11 tests, plus 10-iter measurements): +- **Zero result flakiness.** No testcase that passed on one round ever + failed on another, and vice versa. +- **Two stall classes**, both environmental (Docker Desktop on macOS, + not in `:quic`): + - **Type A (compose-init lockup)** — `docker compose up` hangs for + ~15-16 min between Python's command-log and compose's first Docker + API call. Hit 4 testcases: picoquic r2 transfer, quic-go r2 + handshakecorruption, quinn r1 rebind-port/rebind-addr (both), + quinn r3 handshakeloss. Test always succeeds once compose starts. + qlog confirms the QUIC connection itself completes in 10-30s. The + quinn-r1 crosstraffic std-dev (±2283 kbps) is entirely driven by + 2 of 5 iterations being trapped in this stall — the 3 unaffected + iters were 111/138/97 s = consistent with peers. + - **Type B (pyshark XML stall)** — runner's `_get_packets()` + iteration through tshark's XML via pyshark takes 16+ min on + rebind/crosstraffic pcaps. Direct tshark CLI on the same pcap: + 0.6 s. pyshark wrapper: 66 s. Multiple passes per testcase + amplify it. +- Mitigations documented in `plans/2026-05-06-interop-runner.md` + Docker Desktop section (separate investigation note). + ## Still open - **`v2`** — server demands QUIC v2 (RFC 9369). We're v1-only. @@ -233,11 +276,23 @@ docker-compose.yml hardcodes `container_name: sim/server/client` — Docker enforces those globally. Use a sequential loop: ``` -for peer in aioquic picoquic quic-go; do +for peer in aioquic picoquic quic-go quinn; do quic/interop/run-matrix.sh -s $peer -t handshake,chacha20,... done ``` +## Default peer set + +The four-peer baseline that every change touching `:quic` should be +checked against is `aioquic`, `picoquic`, `quic-go`, and `quinn`. The +rationale is product-shaped, not protocol-shaped: this is the cohort +that the Nostr / MoQ relay ecosystem our users actually run their +servers on is built on. quinn in particular underwrites most of the +Rust relay stacks, so an interop regression there is a user-visible +regression — it must stay flawless. quinn was added to the default +set 2026-05-10 after running cleanly across the same audio-critical +3-round flakiness sweep used for the other three peers. + ## Explicitly unsupported testcases (return 127, runner skips) | Testcase | Reason | diff --git a/quic/interop/run-matrix.sh b/quic/interop/run-matrix.sh index b6d00367cc..13f6dd88ca 100755 --- a/quic/interop/run-matrix.sh +++ b/quic/interop/run-matrix.sh @@ -10,6 +10,19 @@ # quic/interop/run-matrix.sh -s aioquic -t handshake # one test # quic/interop/run-matrix.sh -s quic-go -t handshake,chacha20 # different peer # +# Default peer set when running the full nightly sweep: +# aioquic, picoquic, quic-go, quinn — these are the four reference impls +# we hold to "must be flawless" because the Nostr/MoQ relay ecosystem +# our users actually run their servers on overlaps with all four +# (notably quinn: most Rust-based relays are built on it). Always sweep +# sequentially — run-matrix.sh is not safe to invoke concurrently (the +# runner's docker-compose.yml hardcodes container_name: sim/server/client, +# so parallel invocations collide). +# +# for peer in aioquic picoquic quic-go quinn; do +# quic/interop/run-matrix.sh -s $peer +# done +# # Env overrides: # RUNNER_DIR — where to clone / find the runner (default: ../quic-interop-runner) # LOG_DIR — qlog / pcap output (default: $RUNNER_DIR/logs) @@ -145,7 +158,7 @@ fi # sim/server/client`, which Docker enforces globally regardless of # COMPOSE_PROJECT_NAME. Two simultaneous invocations collide on # `docker create container "sim"`. Run sequentially: -# for peer in aioquic picoquic quic-go; do +# for peer in aioquic picoquic quic-go quinn; do # quic/interop/run-matrix.sh -s $peer -t handshake,chacha20,... # done #