From 56f00058d40729b396443446f14e2646d3902c44 Mon Sep 17 00:00:00 2001 From: Johnathan Corgan Date: Wed, 22 Jul 2026 22:03:08 +0000 Subject: [PATCH] Run the CI parity guard in both runners The guard that keeps "local green" and "GitHub green" meaning the same thing was invoked by nothing. Its only entry point was an exec behind an explicit --check-parity flag, which by exec-ing could not be combined with an actual run, and no workflow step called it at all. A guard nobody runs is a guard that does not exist, which is how the mixed-profile divergence on next survived unnoticed. It now runs as the first stage of every local run and as its own job on GitHub. Locally it reports through the same result-recording path as every other stage, so a divergence sets the run's exit status instead of scrolling past. It is deliberately placed above the mode branches, so a divergence also fails --only, --test-only and --build-only runs: whichever subset was asked for, the claim that a local result means what a GitHub result means is what has broken. The GitHub side installs pyyaml explicitly rather than assuming the runner image carries it. --check-parity keeps working exactly as before. Three comment blocks that described the old folded comparison are corrected here, in the workflow, the local runner and the testing README; they were spread across three files and only two mention the guard by name, so the third is best found by searching for what it claims rather than for the script. --- .github/workflows/ci.yml | 22 ++++++++++++++++------ testing/README.md | 7 +++++-- testing/ci-local.sh | 32 ++++++++++++++++++++++++-------- 3 files changed, 45 insertions(+), 16 deletions(-) diff --git a/.github/workflows/ci.yml b/.github/workflows/ci.yml index 0b33deb..b84e3f3 100644 --- a/.github/workflows/ci.yml +++ b/.github/workflows/ci.yml @@ -38,12 +38,12 @@ env: # unreliable on GitHub-hosted runners. # tor-directory — same; live Tor dependency. # -# Granularity-only differences (same coverage, different matrix shape — -# NOT a divergence): -# deb-install — split here into per-distro legs (debian12/debian13/ -# ubuntu22/ubuntu24/ubuntu26) for parallelism; local runs the -# same distro set in one suite. -# dns-resolver — single leg here; runs all scenarios (same as local). +# The two runners express the same work in different matrix shapes, and the +# parity guard compares through that shape rather than around it: chaos legs +# are compared per scenario (and per flag) via their `scenario:` field, +# deb-install legs per distro. The one leg still compared at leg granularity +# is dns-resolver — a single leg here, running all of its scenarios +# internally, exactly as the local suite does. # ───────────────────────────────────────────────────────────────────────────── # ───────────────────────────────────────────────────────────────────────────── @@ -52,6 +52,16 @@ env: # Builds on Linux x86_64, Linux aarch64, and macOS. # ───────────────────────────────────────────────────────────────────────────── jobs: + ci-parity: + name: CI parity + runs-on: ubuntu-latest + steps: + - uses: actions/checkout@v6 + - name: Install Python deps + run: pip3 install --quiet pyyaml + - name: Check local and GitHub runners cover the same work + run: bash testing/check-ci-parity.sh + fmt: name: Format check runs-on: ubuntu-latest diff --git a/testing/README.md b/testing/README.md index 2dd2760..6933a43 100644 --- a/testing/README.md +++ b/testing/README.md @@ -79,8 +79,11 @@ a per-commit gate; not part of `ci-local.sh`. clippy, unit tests, and the integration suites (including the chaos scenarios) — mirroring the GitHub `ci.yml` integration matrix. Run `./ci-local.sh --help` for the full option list and `--list` for the -available suites. `--check-parity` verifies the local suite set matches -the GitHub matrix (see [check-ci-parity.sh](check-ci-parity.sh)). +available suites. Every run starts with a parity check that verifies the +local suite set covers the same work as the GitHub matrix, per scenario for +chaos and per distro for deb-install; a divergence fails the run. GitHub +runs the same check as its own `ci-parity` job. `--check-parity` runs it +alone (see [check-ci-parity.sh](check-ci-parity.sh)). ### Per-run isolation and the `FIPS_CI_RUN_ID` override diff --git a/testing/ci-local.sh b/testing/ci-local.sh index ffdb60d..c8a4efc 100755 --- a/testing/ci-local.sh +++ b/testing/ci-local.sh @@ -1,5 +1,6 @@ #!/bin/bash -# Run the CI pipeline locally: build, unit tests, integration tests. +# Run the CI pipeline locally: CI parity check, build, unit tests, +# integration tests. # # Usage: ./ci-local.sh [options] # @@ -53,19 +54,19 @@ # (.github/workflows/ci.yml) MUST run the same integration suites, EXCEPT for # the deliberate local-only entries below. Adding a suite to one runner # without the other means "local green" and "GitHub green" stop being -# equivalent. testing/check-ci-parity.sh enforces this and fails on drift. +# equivalent. testing/check-ci-parity.sh enforces this and fails on drift; it +# runs as the first stage of every local run, before the build. # # Deliberate local-only (NOT on the GitHub gate), with reason: # tor-socks5 — requires live Tor network; opt-in via --with-tor, # unreliable on GitHub-hosted runners. # tor-directory — same; live Tor dependency. # -# Granularity-only differences (same coverage, different matrix shape — -# NOT a divergence): -# deb-install — local runs all distros sequentially in one suite; GitHub -# splits into per-distro legs (debian12/debian13/ubuntu22/ -# ubuntu24/ubuntu26 — the same distro set). -# dns-resolver — single suite both sides; runs all scenarios. +# The two runners express the same work in different matrix shapes, and the +# guard compares through that shape rather than around it: chaos legs are +# compared per scenario (and per flag), deb-install legs per distro. The one +# leg still compared at leg granularity is dns-resolver — a single suite on +# both sides that runs all of its scenarios internally. # ───────────────────────────────────────────────────────────────────────────── set -uo pipefail @@ -1034,6 +1035,16 @@ print_summary() { echo "" } +# Verify the local default suite set and the GitHub matrix still cover the +# same work. Runs first: it takes about a second, and a divergence should be +# reported before a half-hour suite rather than after it. +run_ci_parity() { + local rc=0 + info "[ci-parity] Comparing the local suite set against the GitHub matrix" + "$SCRIPT_DIR/check-ci-parity.sh" || rc=$? + record "ci-parity" $rc +} + # ── Main ─────────────────────────────────────────────────────────────────── main() { @@ -1042,6 +1053,11 @@ main() { stage "FIPS Local CI" info "Project root: $PROJECT_ROOT" + # Above the mode branches deliberately, so --only, --test-only and + # --build-only are gated too: a divergence invalidates any claim that a + # local run means what a GitHub run means, whichever subset was asked for. + run_ci_parity + if [[ "$TEST_ONLY" == true ]]; then run_tests elif [[ "$BUILD_ONLY" == true ]]; then