mesh-lab: parse rekey Phase 1 baseline failure; add no-resource-limits gate

Extend parse_rekey to emit phase1_status / phase1_baseline_passed /
phase1_baseline_total fields in signature.json by scraping rekey-test.sh's
"Best observed baseline before timeout: N/M passed" line (the timeout
path) and "Pre-rekey baseline (all 20 pairs): N/M passed" (the success
path). Phase-5-shape parsing is unchanged.

Extend mechanism_match_rekey to also fire on the Phase 1 characteristic
12/20 split — the multi-hop-routing-failure shape where direct-peer pairs
pass and the four multi-hop pairs (x 2 directions) fail. The Phase 5
predicate remains intact for the pre-existing flake class.

Add FIPS_MESH_LAB_NO_RESOURCE_LIMITS=1 to run_rekey_family for
unconstrained characterization runs where the goal is to surface a race
or scheduling artefact rather than reproduce GHA pressure. Default
behaviour (variable unset) keeps the compose-resource-limits.yml overlay
engaged.

Documented in README.md and the in-script env-var header block.
This commit is contained in:
Johnathan Corgan
2026-05-26 15:10:04 +00:00
parent 00bd849ee1
commit c4c3fdd94b
2 changed files with 84 additions and 23 deletions
+11 -3
View File
@@ -53,9 +53,9 @@ Initial target set:
- `rekey`, `rekey-accept-off`, `rekey-outbound-only` — rekey-suite
Phase 5 post-second-rekey connectivity flake class.
- `nat-lan` — two-node NAT-traversal handshake-completion flake class.
- `bloom-storm` — chaos scenario; covers the
[ISSUE-2026-0026](../../../prj/fips/issues/ISSUE-2026-0026.md)
`bloom_send_rate` per-node ceiling exceedance class. Note that
- `bloom-storm` — chaos scenario; covers the `bloom_send_rate`
per-node ceiling exceedance class (single node spiking above the
ceiling while peers stay well under). Note that
chaos uses its own python sim runner (not docker-compose), so the
mesh-lab `compose-resource-limits.yml` and `compose-trace.yml`
overrides do not apply to this suite; per-rep evidence comes from
@@ -135,6 +135,14 @@ each rep does, set them in the invoking shell:
containers; the sidecar fills that gap. Only applies to the
`nat-lan` suite; other suites ignore it.
- **`FIPS_MESH_LAB_NO_RESOURCE_LIMITS`** — when set to any non-empty
value, omits the `compose-resource-limits.yml` overlay for rekey-family
runs. Default behaviour keeps the overlay engaged so rekey-family lab
reps stay pressure-matched to a GHA `ubuntu-latest` runner. Set this
for unconstrained characterization where the goal is to surface a race
or scheduling artefact rather than reproduce CI pressure. Only applies
to the rekey-family suites; other suites ignore it.
- **`FIPS_MESH_LAB_RUNS_DIR`** — root directory for harness output
(the `runs/<timestamp>/` tree). When unset, the harness falls back
to an in-tree path under `testing/mesh-lab/` and prints a warning
+73 -20
View File
@@ -25,6 +25,12 @@
# of the base + resource-limits stack to
# bump RUST_LOG to trace on rekey/handshake/
# forwarding/session/encrypted/mmp modules.
# FIPS_MESH_LAB_NO_RESOURCE_LIMITS
# when set, omits the compose-resource-limits.yml
# overlay for rekey-family runs. Use for
# unconstrained characterization (e.g. surfacing
# a race without GHA-pressure shaping). Does not
# affect other suites.
# FIPS_MESH_LAB_RUNS_DIR Root for harness output (runs/ and any
# other scratch). When unset, falls back to
# an in-tree path under testing/mesh-lab/
@@ -163,6 +169,10 @@ run_rekey_family() {
# ubuntu-latest runner imposes. Base compose is unmodified so
# ci-local.sh stays unconstrained for day-to-day developer runs.
#
# FIPS_MESH_LAB_NO_RESOURCE_LIMITS=1 omits the resource-limits overlay
# for unconstrained characterization runs where the goal is to expose
# a race or scheduling artifact rather than reproduce GHA pressure.
#
# Trace-logging override: set FIPS_MESH_LAB_TRACE=1 in the environment
# to bump RUST_LOG to trace level on the modules relevant to the
# rekey-class flake (rekey, handshake, forwarding, session, encrypted,
@@ -170,17 +180,14 @@ run_rekey_family() {
# primary failure-moment evidence for mechanism investigation.
local compose_args=(
-f testing/static/docker-compose.yml
-f testing/mesh-lab/compose-resource-limits.yml
--profile "$compose_profile"
)
if [ -n "${FIPS_MESH_LAB_TRACE:-}" ]; then
compose_args=(
-f testing/static/docker-compose.yml
-f testing/mesh-lab/compose-resource-limits.yml
-f testing/mesh-lab/compose-trace.yml
--profile "$compose_profile"
)
if [ -z "${FIPS_MESH_LAB_NO_RESOURCE_LIMITS:-}" ]; then
compose_args+=(-f testing/mesh-lab/compose-resource-limits.yml)
fi
if [ -n "${FIPS_MESH_LAB_TRACE:-}" ]; then
compose_args+=(-f testing/mesh-lab/compose-trace.yml)
fi
compose_args+=(--profile "$compose_profile")
(
cd "$REPO_ROOT" || exit 1
@@ -265,6 +272,34 @@ parse_rekey() {
phase6_status="errors-observed"
fi
# Phase 1 baseline convergence — captures the failure shape where the
# pre-rekey baseline never reaches 20/20 within the convergence
# timeout. rekey-test.sh prints the line
# Best observed baseline before timeout: N/M passed
# on the timeout path (else-branch) and exits 1 before any later
# phase runs. phase1_status reports `ok` (the if-branch ran a verbose
# ping_all and phase_result), `timeout` (the else-branch fired), or
# `unknown` (neither line scraped, e.g. test never reached Phase 1).
local phase1_status="unknown"
local phase1_passed=""
local phase1_total=""
local phase1_line
phase1_line=$(grep -m1 'Best observed baseline before timeout:' "$log" 2>/dev/null || true)
if [ -n "$phase1_line" ]; then
phase1_status="timeout"
phase1_passed=$(echo "$phase1_line" | grep -oE '[0-9]+/[0-9]+' | head -1 | cut -d/ -f1)
phase1_total=$(echo "$phase1_line" | grep -oE '[0-9]+/[0-9]+' | head -1 | cut -d/ -f2)
elif grep -q '✓ Pre-rekey baseline (all 20 pairs):' "$log" 2>/dev/null; then
phase1_status="ok"
local phase1_ok
phase1_ok=$(grep -m1 '✓ Pre-rekey baseline (all 20 pairs):' "$log" \
| grep -oE '[0-9]+/[0-9]+' | head -1)
phase1_passed="${phase1_ok%/*}"
phase1_total="${phase1_ok#*/}"
fi
phase1_passed="${phase1_passed:-0}"
phase1_total="${phase1_total:-0}"
# Late FSP K-bit cutover detection — scan node logs for cutover-
# complete events occurring within or after the Phase 5 settle
# window. The settle window is the 12 s before Phase 5's first
@@ -290,14 +325,26 @@ parse_rekey() {
--arg pairs "$phase5_failures" \
--arg phase6 "$phase6_status" \
--arg events "$late_fsp_events" \
'{phase5_failing_pairs: $pairs, phase6_log_analysis: $phase6, late_fsp_events_per_node_tail: $events}' \
--arg p1_status "$phase1_status" \
--argjson p1_passed "$phase1_passed" \
--argjson p1_total "$phase1_total" \
'{phase5_failing_pairs: $pairs,
phase6_log_analysis: $phase6,
late_fsp_events_per_node_tail: $events,
phase1_status: $p1_status,
phase1_baseline_passed: $p1_passed,
phase1_baseline_total: $p1_total}' \
> "$REP_DIR/signature.json"
}
# Heuristic mechanism-match check for the rekey Phase 5 flake class.
# True iff:
# - At least one Phase 5 ping fails
# - Phase 6 log analysis is all-green (no ERROR/PANIC noise)
# Heuristic mechanism-match check for the rekey-family flake classes.
# True iff EITHER:
# - Phase 5 mechanism: at least one Phase 5 ping fails AND Phase 6 log
# analysis is all-green (the post-rekey reconvergence-flake shape)
# - Phase 1 mechanism: Phase 1 baseline timed out with the
# characteristic 12/20 multi-hop-only split (multi-hop routing not
# converging while direct-link forwarding works in the 5-node
# sparse mesh).
mechanism_match_rekey() {
local REP_DIR="$1"
local sig="$REP_DIR/signature.json"
@@ -314,14 +361,20 @@ mechanism_match_rekey() {
echo "false"
return
fi
local pairs phase6
local pairs phase6 p1_status p1_passed
pairs=$(jq -r '.phase5_failing_pairs' "$sig")
phase6=$(jq -r '.phase6_log_analysis' "$sig")
p1_status=$(jq -r '.phase1_status' "$sig")
p1_passed=$(jq -r '.phase1_baseline_passed' "$sig")
if [ -n "$pairs" ] && [ "$phase6" = "all-green" ]; then
echo "true"
else
echo "false"
return
fi
if [ "$p1_status" = "timeout" ] && [ "$p1_passed" = "12" ]; then
echo "true"
return
fi
echo "false"
}
run_nat_lan() {
@@ -363,7 +416,7 @@ run_nat_lan() {
# bumps RUST_LOG to trace on discovery::nostr, transport::udp,
# node::lifecycle, handlers::handshake, handlers::forwarding —
# the modules covering the cross-init / adoption / handshake
# path that ISSUE-2026-0027 exhibits. Path is repo-relative.
# path that the NAT-traversal flake exhibits. Path is repo-relative.
local -a env_args=(FIPS_NAT_SKIP_FINAL_CLEANUP=1)
if [ -n "${FIPS_MESH_LAB_TRACE:-}" ]; then
env_args+=(FIPS_NAT_EXTRA_COMPOSE=testing/mesh-lab/compose-trace-nat.yml)
@@ -603,7 +656,7 @@ mechanism_match_nat_lan() {
# on `stats.bloom.sent` deltas over the trailing 30s window of the
# 180s run. The flake class tracked here is a single node spiking
# above the ceiling while peers stay well under (asymmetric
# distribution), seen on master CI as ISSUE-2026-0026.
# distribution), as seen on master CI.
#
# Unlike rekey and nat-lan, chaos doesn't use docker-compose — the
# python sim runner under `python3 -m sim` owns the container
@@ -747,7 +800,7 @@ EOF
mechanism_match_bloom_storm() {
local REP_DIR="$1"
local log="$REP_DIR/test-output.log"
# The ISSUE-2026-0026 mechanism is a bloom_send_rate FAIL with
# The bloom-storm mechanism is a bloom_send_rate FAIL with
# at least one named offender (i.e., not the "failed to sample
# window endpoints" sub-failure mode, which is harness rather
# than mechanism). The asymmetric-distribution check (one node