mirror of
https://github.com/jmcorgan/fips.git
synced 2026-07-30 19:46:15 +00:00
d21f818659369a6c1aa39dd3c41390facc40a71c
20
Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
 Johnathan Corgan
|
d21f818659 |
Check that log strings tests match on are still ones the daemon emits
A test that greps the daemon's log for a message the daemon stopped emitting does not fail. It stops observing, and an assertion built on it — especially one expecting a count of zero — then passes because nothing can be seen rather than because nothing happened. Several findings have come from that one class, so it is now checked mechanically. testing/check-log-strings.py extracts the strings test code matches against daemon log text and requires each to exist in src/. It reads four shapes: python `"..." in line` tests, the first argument of the shell log helpers, bash associative-array pattern tables, and greps whose input is daemon log output. Patterns carry regex syntax, so each is reduced to the longest literal run of every alternation branch, with escaping resolved in the same pass — `\.` is a literal dot and `.` is a wildcard, and conflating them would let a pattern match text that is not there. Scoping is by what a grep READS, not by what its file mentions. A suite greps its own analyzer output, fipsctl JSON, Tor's log and ping output, and none of those have to correspond to a string in src/; scoping on the file produced 34 false positives. Strings that legitimately do not come from src/ are named in ALLOWED with a reason, so the exceptions are reviewable rather than invisible. It found six dead strings, three of which were not previously known: a second copy of "Excessive decrypt failures" in the interop pattern table, a dead "Handshake error" branch inside an alternation whose other branches still matched, and "bootstrap failed" in the NAT suite. All six are corrected to what the daemon emits, or dropped where the surviving branch already covers the case. Verified by breaking it two ways and confirming it reports. Also here, since it is the same failure shape one layer down: the shared log library returned an empty string for a container whose logs it could not read, so analysing a missing container reported no panics, no errors and no sessions, and exited 0 — indistinguishable from a clean run. It now reports which containers could not be read and raises, matching the fix already landed in the chaos copy. |
||
|
Johnathan Corgan
|
6c52b0e01e |
Let the static test network float so concurrent CI runs cannot collide
Two local CI runs on one host both asked docker for 172.20.0.0/24 and the second lost its whole static family to "Pool overlaps". Docker honours a fixed subnet request verbatim, so the only robust fix is to stop making one: fips-net now requests no subnet and docker assigns from its own pool, which cannot hand the same range to two runs. That means node addresses are not known before `up`, so peers address each other by container hostname instead. The generator emits node-<id>, or the topology's docker_host where the compose hostname differs — only the gateway profile, whose services are gw-*. External peers keep the address the topology gives them, since it is not ours to assign. The resolv.conf mount stays: dnsmasq is what forwards these names to docker's resolver and .fips to the daemon, so removing it would take out every .fips assertion. generated-configs is now per-run as well. A shared directory let two runs overwrite each other's node configs, which the subnet collision had been hiding by killing runs before that window opened. The generator, the compose bind mounts and env_file, the six scripts that read it, and teardown all follow FIPS_CI_NAME_SUFFIX; unset, every path renders as before. Teardown keeps the directory after a failed run, where it is the evidence of what the failing nodes were configured with. Three things this exposed that were wrong independently: admission-cap built its tcpdump patterns from the topology file's docker_ip literals. Floating the subnet makes those match nothing, which would have left its expect-zero "no Msg2 leaked" assertion passing because it could no longer see anything at all. It now reads addresses from the running containers. Restarting the denied peers together also made them swap addresses, so each peer's counts were really the pair's total; they are restarted one at a time now, and a check fails the suite outright if two denied peers ever share an address, because per-peer attribution is impossible once they do. Attribute lookups in the generator used a fixed ten-line window and read the next node's fields when a node omitted an attribute. An external node followed by an internal one was classified as internal, which under hostname peering would emit a name that resolves nowhere. Lookups are bounded to the node's own block; generated output is byte-identical for all eight topologies. The rekey outbound-only variant used to rewrite peer addresses to hostnames to set up its scenario. The generator now does that everywhere, so the rewrite matched nothing and was silently doing no work. It asserts the premise instead, and fails if a numeric address ever reappears. Verified by running three instances of this compose at once — tcp-chain plus two independent meshes — which drew 10.128.2/3/4.0/24 with no overlap while both meshes passed ping-test 20/20 over the real .fips path. tcp-chain is run by neither CI runner, so it was checked by hand: chain peer counts 1/2/1 and multi-hop .fips reachable both directions over TCP. gateway-lan still pins its own IPv4 and fd02:: ranges and is unchanged here, so the gateway profile is not yet concurrency-safe. |
||
|
Johnathan Corgan
|
e4a854f6b0 |
Stop concurrent local CI runs from clobbering each other's containers
Two runs on one host destroyed each other's containers, producing mid-test "No such container" failures that look like real defects. The automated builder runs a full local CI on the same box every few minutes, so the machine is contended almost always and this has red-ed both a hand run and an automated gate. Two independent causes are fixed here. Container names were hardcoded, and docker names are global rather than scoped by compose project, so two runs collided on the same name; every name now takes an optional suffix that the harness sets from the run id. And the cleanup sweep matched a label shared by every run, so one run's teardown force-removed another's containers; resources now also carry a per-run label and the sweep can be narrowed to it. A third hazard turned up that was not in the original report: the sidecar suite passes explicit compose project names, which override the run-scoped project and put it outside the shared prefix entirely. Its project names, network, and derived container references are now scoped too. Both are default-off. With the suffix unset, names render exactly as they do today and a bare compose invocation is unchanged, which is what keeps the hosted CI and the documentation correct. A cleanup run with no run id still reaps everything, which is what a manual "clear the box" wants. The literal-name sweep was not sufficient: six scripts build container names dynamically from node labels, and two suites create their own containers outside compose. Those are handled at their construction sites. Verified: syntax check on all modified scripts; compose validation on every modified file with the suffix both set and unset; and a synthetic two-run reproduction that shows the old cleanup destroying a bystander run and the new one leaving it alone. Known gap: subnets are still hardcoded, so two concurrent full runs will still collide on address-pool overlap. That fix reaches into topology configs, chaos scenarios, diagrams and production source, so it is left for its own change rather than half-done here. |
||
|
Johnathan Corgan
|
7a97599921 |
testing: floor the rekey second-cutover wait at the log-analysis threshold
The rekey suites' Phase 4 polled until the initiator-cutover count reached its value at phase entry plus two, while Phase 6 asserts an absolute minimum of four. On an unloaded host the ping phases can outrun the jittered first rekey cycle, so Phase 4 can begin with a single cutover on the books, release at three, and leave Phase 6 to lose a sub-second race against the remaining first-cycle cutovers. A rekey-outbound-only run failed exactly this way: five counted cutover lines by t=32.6s of container life, with the Phase-6 count snapshot at t~32.4s seeing three. Floor the Phase-4 wait target at four so a released wait guarantees the Phase-6 assertion — the docker-logs count is monotone over a container's lifetime. A genuine rekey stall still times out the bounded wait and fails Phase 6 with the real count. Also drop the second-cycle framing from the comment, the phase banner, and the assertion description: with rekey timers resetting at each cutover a second cycle cannot begin inside this test's window, so the counted events are first-cycle cutovers spread across the topology's links, emitted once or twice per completed link rekey depending on which side's promotion path logs them. |
||
|
Johnathan Corgan
|
fbb4fb8879 |
testing: wait for both second-cycle rekey cutovers before asserting the count
Phase 4 of the rekey suite waited for only one more initiator cutover (guaranteeing three) while Phase 6 asserts at least four. The fourth cutover then had to land in the brief window between the wait returning and the Phase 6 log snapshot, so host load could push it past the window and fail the run even though every cutover completed correctly. Wait for two more cutovers, the full second rekey cycle, matching the Phase 6 threshold, so the asserted count is guaranteed before it is checked. |
||
|
Johnathan Corgan
|
ab915d0479 |
testing: convert remaining fixed-sleep settles to progress-aware polling
Replace the fixed post-first-rekey and post-second-rekey settle sleeps in the rekey integration test, and the fixed-deadline baseline wait in the interop test, with the deterministic wait_until_connected progress-aware polling helper already used for the initial baseline convergence. Each converted site fails fast with structured diagnostics when the mesh is stuck and extends its deadline only while pairwise reachability is still climbing, removing the wall-clock settle windows that produced intermittent connectivity failures after a rekey. |
||
|
Johnathan Corgan
|
f29c2e65fa |
test: replace fixed convergence timeouts with progress-aware connectivity wait
Add wait_until_connected to the shared convergence helpers: it polls a suite's own pairwise pings (the signal it actually asserts on), returns as soon as every pair is reachable, extends its deadline while the reachable-pair count is still climbing, and gives up only when progress stalls. Use it in the rekey, static-mesh, and sidecar suites in place of the fixed wall-clock baseline timeout and the blind sleep, which timed out under concurrent CI load while the mesh was still converging. |
||
|
Johnathan Corgan
|
ffd78440a8 |
node: periodically re-broadcast TreeAnnounce on no-change in check_periodic_parent_reeval
Closes the eventually-consistent gap in spanning-tree state distribution. Every existing send_tree_announce_to_all call site gates on a local state-change event (parent switch, self-root promotion, ancestry change, peer promotion, parent loss). Once a partition latches — for example a parent-switch announce stranded in the brief cross-init handshake swap window, where the announce arrives on a session-index whose decrypt-worker entry has been unregistered — neither side's state changes again, so neither side ever re-broadcasts. The existing 60 s check_periodic_parent_reeval was a re-evaluation, not a re-broadcast: it short-circuited silently on no-change. Production-side healing depended on incidental link churn; lab harnesses with stable docker-bridge links had no equivalent path. Add a final else branch that fires send_tree_announce_to_all unconditionally on the no-change path, alongside the existing switch and self-promote arms. Receivers coalesce by sequence comparison (ParentDeclaration::is_fresher_than) and short-circuit at the `if !updated` gate in handle_tree_announce; same-sequence repeats drop silently with no cascade. The per-peer 500 ms rate-limiter is well below this 60 s cadence and does not suppress the heartbeat broadcast. The fix is a general protocol-robustness improvement: it addresses any in-flight TreeAnnounce loss class, not only the specific cross-init swap-window drop site. testing/static/scripts/rekey-test.sh BASELINE_CONVERGENCE_TIMEOUT 60 -> 65 so a partition healed by the periodic broadcast at T+60 lands inside the convergence window. wait_for_full_baseline early-exits on PASS, so successful reps see no extra wall-clock. |
||
|
Johnathan Corgan
|
6e5cb8965f |
Make FSP session rekey hitless under packet loss and reordering
An FSP session rekey could leave the two endpoints holding different key sets for a brief window: if a handshake message was lost in transit, one side rotated to the new keys while the other did not. Traffic sealed in one key epoch then reached a peer still on the other epoch and failed to decrypt, producing bursts of AEAD decryption failures and dropped connectivity until a later rekey cycle reconverged the pair. Choreographing the cutover order cannot close this window: any fixed ordering still leaves a skew that packet reordering widens. Make rekey correctness independent of cutover timing by overlapping the key epochs on the receive path. During a rekey transition the receiver trial-decrypts each frame against every live session it holds: current, the not-yet-promoted pending session, and the draining previous session. The K-bit becomes a hint that orders the trial-decrypt cascade rather than a hard gate, and a frame that authenticates against the pending session is itself the cutover signal. No rotation ordering and no packet reordering can then cause a decryption failure. The pre-rekey Noise session is held in the `previous` slot until the peer has demonstrably moved off it. Its drain deadline is anchored on the most recent frame the peer authenticated against that slot, refreshed each time the trial-decrypt cascade lands there, rather than on a fixed wall-clock timer started unilaterally at the local cutover. A peer that never received the new keys keeps authenticating against `previous` and the slot stays live; without this, a fixed timer would erase the only key set that could decrypt the peer's frames, producing a permanent silent decrypt failure on a live data path. A peer that never catches up is handled by the existing FSP session liveness path rather than by silent decrypt failure. The lost-handshake liveness gap is closed separately by retransmitting the third rekey handshake message until the peer is confirmed on the new keys, with a bounded retry budget after which the rekey cycle is cleanly abandoned and retried on the next timer. Adds unit tests covering the trial-decrypt cascade (epoch selection, promotion on pending decrypt, reordered old-epoch stragglers after cutover, per-slot replay-window integrity), the msg3 retransmission lifecycle, and the peer-progress-aware drain retirement. |
||
|
Johnathan Corgan
|
306e455513 |
rekey-test: retry strict-ping asserts on failure
The Phase 1, Phase 3, and Phase 5 strict asserts each fire a single ping per directed pair. Under low-level packet loss (e.g. 1% i.i.d. per-direction loss from a CI runner under pressure), a single-shot round-trip fails at ~2% per pair, so a 20-pair strict assert misses with probability 1 - (0.98)^20 = ~33% per phase from ICMP noise alone, well above the routing-state signal the asserts are meant to catch. ping_one gains a max_attempts parameter (default 1, preserving existing call sites). On failure it retries up to MAX_PING_ATTEMPTS-1 additional times with PING_RETRY_DELAY seconds between attempts. Per-pair worst case under the defaults (4 attempts, 1 s spacing, 5 s ping6 -W timeout) is 4*5 + 3 = 23 s; per-rep worst case scales with the failing-pair count. Successful retries log "OK (RTT, attempt N)"; exhausted retries log "FAIL (after N attempts)". The retry budget is wired into all three strict asserts: - Phase 1 final ping_all (after wait_for_full_baseline converges) - Phase 3 ping_all (post-first-rekey) - Phase 5 ping_all (post-second-rekey) The wait_for_full_baseline convergence loop itself stays single-shot. Its job is to detect when the mesh first sees a fully clean 20-pair batch, and retries inside the loop would conflate transient ping loss with still-converging routing state. No daemon code changes. |
||
|
Johnathan Corgan
|
80fb086071 |
test(rekey): add Phase 5 settle window for post-second-rekey convergence
Phase 5's per-pair connectivity check ran immediately after the second rekey cycle, with no settle for routing reconvergence. Under GitHub-runner CPU contention, post-rekey parent-switches and coord-cache flushes can take longer than the per-ping 5s timeout for a small fraction of pairs (1-3 of 20 typically), even though the rekey mechanism itself completes cleanly. Phase 6 log analysis stays all-green on these failed runs; the failure is purely connectivity timing. Mirror Phase 3's existing 12-second settle pattern: reuse REKEY_SETTLE and emit the same banner shape. Two-line change. Cost on the success path is a fixed 12s per suite run. |
||
|
Johnathan Corgan
|
6533276eda |
test(rekey): bump Phase 1 baseline-convergence headroom 36s → 60s
The Phase 1 pre-rekey baseline in `wait_for_full_baseline` occasionally times out on GitHub-hosted runners with one ping pair failing to converge inside the BASELINE_CONVERGENCE_TIMEOUT window. Phases 2–6 always pass cleanly when this happens — the rekey itself is fine, the mesh just hasn't finished spanning-tree + bloom-filter convergence by the time Phase 1 starts pinging. The wait loop returns as soon as all 20 pairs converge, so the cost on the success path is unchanged (typical local CI returns well under the old 36s). The bump only adds headroom on the failure path. Operators previously worked around this with `gh run rerun --failed`; this aims to retire that workaround for the IK/XK lines. Distinct from the next-branch XX rekey dual-init race, which is a real protocol bug tracked separately. |
||
|
Johnathan Corgan
|
96c6b7dea8 |
Admit rekey msg1 from established peers when addr forms differ
Companion to the ethernet `accept_connections: false` rekey-deadlock fix from earlier this release: the same dual-init failure mode shows up over UDP when peers register by hostname, and the existing addr_to_link-only carve-out in `should_admit_msg1` doesn't cover it. The carve-out's first predicate keys `addr_to_link` by the literal `TransportAddr` that `initiate_connection` inserted, which is the hostname-form when a peer config carries a hostname (e.g., `core-vm.tail65015.ts.net:2121`). Inbound packets always arrive with numeric source addrs because `udp_receive_loop` builds the `TransportAddr` from the `SocketAddr` the kernel reports via `recvfrom`. `TransportAddr` equality is byte-exact, so the two forms don't match and the lookup misses. With `udp.accept_connections: false` (or `udp.outbound_only: true`, which forces it false) the gate then rejects the rekey msg1 from an established peer. The dual-init tie-breaker stalls because the loser side never produces msg2; both sides retry indefinitely and the winner side keeps logging "Dual rekey initiation: we win, dropping their msg1" at 1Hz. The earlier ethernet fix didn't generalize to this variant because ethernet TransportAddrs are always numeric MAC bytes — both the config-time form and the inbound-arrival form match identically. Add a second predicate to `should_admit_msg1`: an active peer's `current_addr()` matching `(transport_id, remote_addr)`. `current_addr` is updated and refreshed from inbound encrypted-frame source addrs (`handlers/encrypted.rs`), which are always numeric `SocketAddr`-form, so this catches the established peer regardless of how its `addr_to_link` key was originally inserted. The fast `addr_to_link` check stays first; the iteration over peers is bounded by peer count and only runs when the first predicate misses. Regression coverage in this commit: - Unit test `test_should_admit_msg1_admits_rekey_when_addr_form_differs` in `src/node/tests/handshake.rs`. Constructs the failing scenario in-process: `addr_to_link` populated with hostname-form key, peer's `current_addr` at the resolved numeric form, query with numeric form. Without the new predicate this fails immediately. - New integration topology `rekey-outbound-only` plus matching docker-compose profile. Same 5-node mesh shape as `rekey-accept-off` but `inject-config` sets `udp.outbound_only: true` on node-b and rewrites node-b's peer-c address from the numeric docker IP to the docker hostname (`node-c:2121`), reproducing the production hostname-vs-numeric mismatch. The test asserts no sustained "Dual rekey initiation: we win" log lines on any node (>10 = bug) and the existing rekey health checks catch the connectivity loss the loop produces. - `testing/ci-local.sh` and `.github/workflows/ci.yml` extended to run the new variant in the local sweep and the GitHub CI integration matrix alongside `rekey` and `rekey-accept-off`. Verified locally: full `bash testing/ci-local.sh` sweep passes 29/29 suites (23m 12s) with the new variant green; 1084 unit tests pass. |
||
|
Johnathan Corgan
|
37c2973e2f |
Test infrastructure overhaul: gateway robustness + full CI coverage
Single combined commit covering five interlocking pieces of test and CI work that landed during the v0.3.0-prep cycle. ## fips-gateway robustness - src/bin/fips-gateway.rs DNS upstream probe converted from a 3-second hard-fail to a bounded retry loop (5 attempts × 1s timeout, 1s sleep between attempts; ~10s worst case). Covers the cold-boot race where the daemon's TUN is up but the DNS responder at [::1]:5354 is still binding. Each failed attempt logs at INFO. In production the binary's retry is the live recovery mechanism; with retry it recovers silently instead of relying on Restart=on-failure (~5s blip + spurious ERROR per cycle). - packaging/debian/fips-gateway.service `ExecStartPre` now waits up to 30 seconds for the daemon's `fips0` TUN to appear before exec'ing the gateway binary. Eliminates the cold-boot race where the gateway exits with `fips0 interface not found` and recovers via `Restart=on-failure`, producing a 5-second blip and a spurious error log per restart cycle. - testing/docker/entrypoint.sh gateway-mode waits up to 30s for the daemon's DNS responder to bind [::1]:5354 (probes once per second with `dig @::1 -p 5354 ... test.fips`) before exec'ing fips-gateway. Belt-and-suspenders with the binary's own retry: in CI we want deterministic startup ordering. On timeout, fall through so the binary's probe reports the definitive error. ## Test infrastructure DNS bind migration to ::1 After session 359's daemon DNS-bind default flipped from `127.0.0.1` to `::1` (the production fix for ISSUE-2026-0002), the static-test infrastructure was carrying a stale workaround that overrode the default back to IPv4 loopback. The fips-gateway integration test exposed the divergence: the gateway probes its DNS upstream at `[::1]:5354` (production default) while the daemon was binding `127.0.0.1:5354` from the template override — IPv6-explicit sockets do not accept v4-mapped traffic, so the upstream probe exhausted retries and the gateway exited. - Drop the explicit `bind_addr: "127.0.0.1"` line from every test config that emits it: testing/static/configs/node.template.yaml, testing/chaos/configs/node.template.yaml, the sidecar heredoc in testing/docker/entrypoint.sh, testing/acl-allowlist/generate-configs.sh (six per-node blocks), testing/nat/scripts/generate-configs.sh, and the four tor templates under testing/tor/. Daemon picks up its production `::1` default. - Flip the dnsmasq forwarder for `.fips` in testing/docker/Dockerfile from `127.0.0.1#5354` to `::1#5354` so dnsmasq on the shared test image continues to reach the daemon. Template and Dockerfile must move together since most static suites resolve `<npub>.fips` via the test-image dnsmasq. ## rekey-accept-off integration variant + UDP unit test - New `rekey-accept-off` topology and docker-compose profile under testing/static/. 2-node variant where node-b runs with `udp.accept_connections: false`. Pins the regression class that ISSUE-2026-0004 fixed (cross-connection winner's rekey msg1 was being filtered by the accept_connections gate, breaking rekey). - testing/static/scripts/rekey-test.sh accepts REKEY_TOPOLOGY and REKEY_ACCEPT_OFF_NODES env vars; its inject-config subcommand applies the per-node `udp.accept_connections: false` edit, and the test asserts no sustained "Dual rekey initiation" log lines. - New UDP variant of `should_admit_msg1` admit-rekey unit test in src/node/tests/handshake.rs. ## ci-local.sh full integration coverage - New runner functions and dispatcher entries for `acl-allowlist`, `nat-cone` / `nat-symmetric` / `nat-lan`, `rekey-accept-off`, `dns-resolver`, `deb-install`. Each integrates with the existing summary tracking via `record`. - New `--with-tor` flag (off by default) gates `tor-socks5-outbound` and `tor-directory-mode` runners. Tor stays opt-in because both harnesses depend on the live Tor network and would introduce a flake source unrelated to the FIPS code. - New suite arrays (`ACL_SUITES`, `NAT_SUITES`, `DNS_RESOLVER_SUITES`, `DEB_INSTALL_SUITES`, `TOR_SUITES`) drive both the default sweep and `--list` output. - `run_suite` extended to accept the new suite names for `--only` invocations. ## GitHub CI matrix expansions - `gateway` matrix entry runs testing/static/scripts/gateway-test.sh against the existing docker-compose `gateway` profile. - `rekey-accept-off` matrix entry exercises the new topology with REKEY_ACCEPT_OFF_NODES=b. - `deb-install` matrix (debian12 + ubuntu24 + ubuntu26) runs testing/deb-install/test.sh with privileged systemd containers. ~5-7 min cold cache, ~2 min warm per distro. Self-contained: builds its own .deb in a Debian 12 cargo-deb builder image; does not depend on the build job's pre-built artifact. - `dns-resolver` matrix entry runs the full 13-scenario harness (per-distro systemd resolver-backend tests + real-fips end-to-end scenarios) in a single job. Pins the production DNS bind path that ISSUE-2026-0002 lived in. ~7-12 min warm, ~12-15 min cold. Verified locally: full `bash testing/ci-local.sh` sweep passes, including 5/5 deb-install distros and all 13 dns-resolver scenarios. Tor-inclusive sweep (`--with-tor`) verified in a follow-up run. |
||
|
Johnathan Corgan
|
1e3b2c319e |
testing: rekey settle past drain window + stale tooling references
- testing/static/scripts/rekey-test.sh: bump REKEY_SETTLE from 5s to 12s so post-rekey ping_all samples are taken after the 10s old- session drain window has closed (DRAIN_WINDOW_SECS in handlers/rekey.rs). The 5s sample window straddled the drain, occasionally catching old-session decrypts mid-cutover and producing spurious ping failures. - testing/lib/log_analysis.py: discovery-succeeded match string is "proof verified, route cached" (the wording in handlers/discovery.rs); the regex looked for the older "caching route" wording, so the chaos analysis summary always reported `Succeeded: 0` regardless of how many successes the raw node logs actually showed. - testing/scripts/build.sh: drop the second cargo invocation that passed `--features gateway --bin fips-gateway`. PR #79 removed the `gateway` cargo feature in favor of platform cfg gates, so this invocation now fails with `the package 'fips' does not contain this feature: gateway`. The first cargo build already produces fips-gateway as a workspace binary on every supported target. |
||
 joandJohnathan Corgan
|
8e38d889fa | testing: stabilize rekey integration timing | ||
|
Johnathan Corgan
|
fc8c0dce15 |
Increase rekey test convergence wait from 5s to 15s
Phase 1 pre-rekey baseline was failing intermittently on CI runners because 5s wasn't enough for multi-hop discovery (B→D requires B→C→D). The mesh always converged by Phase 3, confirming this was purely a timing issue. |
||
|
Johnathan Corgan
|
959134657f |
Fix FSP rekey cutover race and MMP metric discontinuity
The FSP XK rekey handshake has a race condition where the initiator can cut over (K-bit flip) and send data encrypted with the new session before msg3 reaches the responder. The responder has no pending session yet, so K-bit detection fails and packets are dropped. Defer FSP initiator cutover by 2 seconds after handshake completion (FSP_CUTOVER_DELAY_MS) to give msg3 time to traverse the mesh. FMP (IK, 2 messages) is unaffected since the responder completes during msg1 processing. Also fix MMP metric corruption after rekey cutover: the new session starts with counter 0 but MMP state carries highest_counter and GapTracker.expected_next from the old session, producing false reorder counts, jitter spikes, and invalid OWD trends. Add reset_for_rekey() methods that clear counter-dependent state while preserving RTT estimates. Additional fixes: - Remove stale peers_by_index entry on abandon_rekey error path - Replace redundant peers_by_index inserts with debug assertions verifying the pre-registration invariant - Tighten rekey integration test to zero tolerance (was 4 failures) |
||
|
Johnathan Corgan
|
31c0b2c969 |
Tolerate transient rekey cutover failures in integration test
Phase 3 (post-first-rekey connectivity) can see brief disruptions while links are mid-cutover. Add a 5s settle time after the rekey wait and allow up to 4 transient pair failures in Phase 3. Phase 5 remains strict, ensuring full recovery after the second rekey cycle. |
||
|
Johnathan Corgan
|
bf117df0ca |
Add periodic Noise rekey with fresh DH for forward secrecy (FMP + FSP)
Implement periodic full rekey at both protocol layers using fresh DH key exchanges. Uses the existing K-bit flag (FLAG_KEY_EPOCH / FSP_FLAG_K) to coordinate cutover between peers. FMP layer (IK pattern): - ActivePeer gains rekey state: pending/previous sessions, K-bit epoch tracking, drain window, dampening timer - Handshake state stored on ActivePeer with msg1 sent on existing link - Encrypted frame handler detects K-bit flips, promotes pending sessions, falls back to previous session during drain - Handshake handlers distinguish rekey from new connections using addr_to_link lookup with identity-based fallback - Free all session indices (current, rekey, pending, previous) on peer removal FSP layer (XK pattern): - SessionEntry gains parallel rekey fields with XK-specific state for the 3-message handshake - Route availability check before FSP rekey initiation - Encrypted session handler adds K-bit flip detection and dual-session decrypt fallback - SessionSetup/Ack/Msg3 handlers extended for rekey paths Defense-in-depth: - Consecutive decryption failure detector (threshold=20) triggers forced peer removal instead of waiting for link-dead timeout - Identity-based rekey detection as fallback when addr_to_link doesn't match (e.g., TCP ephemeral ports) Configuration: RekeyConfig with enabled flag, after_secs (default 120), and after_messages (default 65536) thresholds. Logging: info for successful K-bit cutover completions, warn for failures, debug for intermediate handshake steps, trace for routine operations (resends, drain cleanup). Rekey lifecycle: 1. Timer/counter fires -> initiator starts new handshake 2. Old session continues handling traffic during handshake 3. Handshake completes -> initiator cuts over, flips K-bit 4. Responder sees flipped K-bit -> promotes new session 5. Both keep old session for 10s drain window 6. After drain, old session discarded Integration test: Docker-based multi-phase test exercising both FMP and FSP rekey with aggressive timers (35s). Verifies connectivity across all 20 directed pairs survives two consecutive rekey cycles. Includes rekey topology, docker-compose profile, and CI matrix entry. Increase ping test convergence wait from 3s to 5s for CI reliability. |