testing/mesh-lab: nat-lan cpu-pinning sidecar + trace overlay + stall_signature

Three deltas to the mesh-lab nat-lan suite for stall characterization:

- FIPS_NAT_LAN_CPUSET env-var-driven CPU-pinning sidecar in
  run_nat_lan, mirroring the bloom-storm pattern. Pinning is needed
  because the mesh-lab compose-resource-limits.yml override is
  rekey-family service-name specific (rekey-* / rekey-accept-off-* /
  rekey-outbound-only-*), so it does not constrain the nat-lan
  containers. Default cpuset 0,1 mimics a GHA 2-core runner; empty
  disables the sidecar.

- New compose-trace-nat.yml overlay that bumps RUST_LOG to trace on
  discovery::nostr, transport::udp, node::lifecycle,
  handlers::handshake, handlers::forwarding — the modules covering
  the cross-init / adoption / handshake path. Picked up by the
  nat-test.sh COMPOSE array via a new FIPS_NAT_EXTRA_COMPOSE
  colon-separated env-var hook. run_nat_lan sets this hook when
  FIPS_MESH_LAB_TRACE is non-empty; the README env-var section
  updated to reflect that FIPS_MESH_LAB_TRACE now applies to nat-lan
  in addition to the rekey-family.

- parse_nat_lan extended with a per-node stall_signature emitting
  last-occurrence timestamps for eight event categories (startup,
  discovery, adoption, handshake_init, msg2_sent, cross_init_ignore_*,
  handshake_failed) plus derived last_meaningful_event_ts,
  last_event_category, silent_gap_s. Top-level stall_class binned as
  no_timeout / silent / localized / distributed / incomplete from
  the per-node categories. Aggregation phase consumes the per-rep
  signatures across a characterization run to classify stall
  mechanism.

Wired support in nat-test.sh: FIPS_NAT_EXTRA_COMPOSE colon-separated
list of repo-relative or absolute compose files layered onto the
base via the COMPOSE array; FIPS_NAT_SKIP_FINAL_CLEANUP gates the
success-path teardown so the mesh-lab harness can capture docker
logs before tearing down (failure paths already returned without
cleanup, leaving stall-state containers intact for capture).

Smoke-tested on idle profile with TRACE on: 1 rep PASS, 32/36 TRACE
lines per node, signature.json events all populated with the
expected category timestamps.
This commit is contained in:
Johnathan Corgan
2026-05-24 17:49:37 +00:00
parent dae33d4fd1
commit cc7f967128
4 changed files with 321 additions and 18 deletions
+23 -1
View File
@@ -13,6 +13,21 @@ WAIT_LIB="$ROOT_DIR/testing/lib/wait-converge.sh"
SCENARIO="${1:-all}"
COMPOSE=(docker compose -f "$NAT_DIR/docker-compose.yml")
# Optional extra compose-file overlay chain (colon-separated paths), e.g.
# trace-RUST_LOG overrides supplied by the mesh-lab harness when
# FIPS_MESH_LAB_TRACE=1. Paths are interpreted relative to the repo root
# (ROOT_DIR) unless absolute. The mesh-lab harness sets this env var to
# `testing/mesh-lab/compose-trace-nat.yml` for nat-lan trace runs.
if [ -n "${FIPS_NAT_EXTRA_COMPOSE:-}" ]; then
IFS=':' read -ra _NAT_EXTRA <<< "${FIPS_NAT_EXTRA_COMPOSE}"
for _f in "${_NAT_EXTRA[@]}"; do
case "$_f" in
/*) COMPOSE+=(-f "$_f") ;;
*) COMPOSE+=(-f "$ROOT_DIR/$_f") ;;
esac
done
fi
source "$WAIT_LIB"
cleanup() {
@@ -386,7 +401,14 @@ run_lan() {
source "$NAT_DIR/generated-configs/lan/npubs.env"
ping_peer fips-nat-lan-a "$NPUB_B"
ping_peer fips-nat-lan-b "$NPUB_A"
cleanup
# Skip the final teardown when the mesh-lab harness wraps this
# script: it needs to docker-logs the containers before teardown,
# and will run its own cleanup after capture. Failure paths above
# already leave containers up via the bare `return 1` so the
# harness can capture stall-state evidence.
if [ -z "${FIPS_NAT_SKIP_FINAL_CLEANUP:-}" ]; then
cleanup
fi
}
main() {