The connection leg's stored msg1/msg2 handshake-resend buffers duplicated the
peer machine's own connection-state copy. Write the machine copy at the outbound
msg1-prep and the two inbound authorize sites, then source the retransmit
resend-bytes reader, send_stored_msg1, and the pending tier of find_stored_msg2
from the machine. The post-promote active-peer msg2 copy is written from the same
wire bytes, so the pending tier now matching after promotion is value-identical.
Byte-identical resend behavior.
The connection leg's HandshakeState field duplicated the peer machine's
handshake phase. Delete it and derive the displayed handshake state string
from the machine's PeerState, looked up by link id (mirroring the resend
count). Move the failure signal onto the machine: a send_failed flag that
preserves retransmit eligibility (the machine stays in its handshake phase),
alongside the existing Failed state. The leg's crypto self-gates now guard on
Noise handle presence instead of the deleted phase field, and mark_failed
only drops the handle. Telemetry strings, wire bytes, index allocation, and
the stale-connection reaping are byte-identical for all normal paths.
The pending-handshake PeerConnection map and the per-peer control
machine map were parallel LinkId-keyed structures whose keysets must
stay coherent by hand. With every leg now born with a machine, the leg
becomes storage inside its machine (leg: Option<PeerConnection>, pure
storage the machine never reads or drives) and Node loses the
connections field; every access routes through the machine.
The non-mechanical lowerings, each argued at the site: the
rekey-vs-establish gate in handle_msg2 tests leg-absence (an established
peer's machine stays keyed by its link, so machine-presence would
misclassify every rekey msg2 as a fresh establish); the
connecting-predicates, peering observation, and handshake-slot budget
iterate machines-with-legs so connect-window machines (leg not yet
born) are excluded exactly as before and never double-counted against
their pending-connect slot; the cross-connection extract takes the leg
before disposing the machine; the stale reaper takes the leg and leaves
the machine untouched when none is present, matching the old early
return. The map-coherence debug check keeps its machine-has-carrier
direction with the embedded leg as a carrier; the leg-to-machine
direction is now true by construction and its gate const is gone.
connection_count() counts machines with legs; the connections()
iterator, the test seams, and the control-socket connection rows are
re-implemented over the embedded legs with unchanged output.
PeerSlot (and its entire impl surface) was referenced only by its own
module tests and the lib.rs re-export; peer storage has always used the
separate connections/peers maps. PeerConnection's resend_count/
next_resend_at_ms/record_resend delegations had no production callers:
the live resend counter is machine-sourced (connection_resend_count reads
the per-peer machine), and the FSP session layer uses SessionEntry's own
methods. ConnectionState::next_resend_at_ms is now test-only (its
remaining callers are the fmp state unit tests) and marked cfg(test).
The PeerSlot unit tests go with the enum; test_resend_count_tracking is
dropped because the delegation target's schedule arithmetic is already
covered by the fmp resend_bookkeeping test.
Move the outbound handshake-timeout reap off the unconditional check_timeouts
scan onto the machine-armed HandshakeTimeout timer. A new drive_handshake_timeouts
(run before the retransmit drive, so a timed-out leg is reaped rather than resent
on the same tick) reaps the outbound legs that carry a HandshakeTimeout timer and
have idle-timed-out this tick.
The timer's presence selects the leg (only outbound legs arm one; IK inbound arms
none); the reap threshold is the shell is_timed_out(now, config) predicate, not
the timer's stored deadline. The machine arms the timer from a hardcoded constant
at dial, which is not authoritative for an operator-tuned handshake_timeout_secs,
so reading the threshold from config each tick keeps the reap neutral for any
timeout value and on the last_activity clock exactly as before.
check_timeouts keeps reaping everything else: failed connections (all of them,
promptly) and the idle-timeout of legs without a machine timer (inbound legs, and
any machine-less connection). Total coverage is unchanged. check_timeouts runs
before the timer drive, so a failed outbound leg is reaped there first, its timers
dropped, and the timeout drive never double-fires on it.
Split drive_peer_timers into the timeout drive and the (byte-identical) retransmit
drive. The dormant machine timeout handler is not dispatched, so the session index
is freed once, by cleanup_stale_connection. A regression test drives a timed-out
outbound leg to reap; the existing check_timeouts tests continue to anchor the
residual sweep. Test count 1631 to 1632.
Move the outbound msg1 resend off the unconditional tick function onto the
machine-armed retransmit timer. The per-peer machine already arms a
HandshakeRetransmit deadline at dial; a new drive_peer_timers fires the due ones
(kind-filtered to the retransmit timer) and homes the resend counter on the
machine, where the operator-visible count now reads from.
The resend decision reads the same operator config as before (interval, backoff,
max), the wire bytes and transport target still come from the shell connection,
and the pure core computes the backoff schedule. As with the deleted
resend_pending_handshakes, the count and reschedule advance only on a successful
send: a failed send neither advances the count nor marks the connection failed,
it just retries on the next tick. The handshake-timeout timer stays on the legacy
check_timeouts path, and the rekey/liveness timers keep their own shell drivers,
so drive_peer_timers deliberately fires only the retransmit kind.
The show_connections resend count is relocated to read from the machine (the
counter's new home) via connection_resend_count; machine-less and inbound
connections report 0, matching what the shell connection reported before. Delete
resend_pending_handshakes and resend_candidates (the latter also from the
LifecycleView trait, its only user). The resend unit test is re-expressed against
the machine + timer path, covering the due check and the record-on-success
semantics.
Known limitation: the first resend interval is armed from the machine's hardcoded
1000ms constant, which equals the config default. Under an operator override of
handshake_resend_interval_ms the first resend diverges from the pre-change
dial+interval; subsequent resends and the cap remain config-driven. Neutralizing
the first-resend override needs the interval threaded into the sync core (the
shell arm would be clobbered by the machine's own timer arming at dial), deferred
to when the connection and machine entities merge. Test count unchanged at 1631.
Introduce a sans-IO lifecycle supervisor: a synchronous step(event) ->
[action] state machine (SupervisorFsm) that authors the substrate-child
spawn and teardown order, plus an owner struct (Supervisor) holding the
substrate runtime fields and embedding the FSM. The substrate-lifecycle
fields leave Node's flat list into the owner - state, packet_tx, the TUN
reader/writer handles + shutdown fd + channels, the DNS task + identity
channel, the Nostr/LAN rendezvous drivers, and the encrypt/decrypt worker
pools; the dataplane keeps packet_rx.
start()/stop() become the driver executing the FSM's SpawnChild/StopChild
actions: same children, same order, same warn/debug-and-continue on
optional failures, same logs, same NodeState transitions. Behavior- and
wire-neutral. The only determinism change is that transports now tear
down in ascending-id order, previously nondeterministic HashMap iteration.
The bounded Draining phase and the Running{Full|Degraded} health split
land as separate follow-on commits.
Reorganize the node module tree by concept rather than by
message-handling verb, as the first step of the node runtime
decomposition. Pure relocation: no wire, config, metric, or log
semantics change; the lib test count is unchanged (1577 passed).
Moves (git mv, 100% rename similarity):
- handlers/{forwarding,rx_loop,connected_udp,dispatch,encrypted}.rs
-> node/dataplane/ — the whole RX hot path (the select! run loop,
transit/local forwarding, the link-message router, the RX decrypt
path with responder K-bit cutover + roam writes, and connected-UDP
fast-path activation) now lives in one home.
- node/session.rs -> node/session/mod.rs — establishes the session
concept home for the data/state types. The message-behavior file
handlers/session.rs stays put for now (folds in with the later FSP
session step).
The IK/XX-divergent establishment files (handlers/{handshake,rekey,
timeout}.rs) and the deferred-home files (handlers/{mmp,lookup}.rs)
deliberately stay in handlers/, to move once rather than twice.
Every module is reached through impl Node methods, so no call site or
re-export shim was needed. Updated in lockstep with the moves: the
module_path!-derived tracing targets in the two mesh-lab compose-trace
overlays, a structural test's include_str! source path, doc-comments
in proto/routing and the mesh-lab docs, and the stale source-location
citations (node/handlers/{forwarding,rx_loop,encrypted}.rs and
node/session.rs) in doc-comments and the discovery design doc.
Move the FMP mesh-layer wire format (common prefix, encrypted/msg1/msg2
headers, and the build_*/inner-header codec fns) out of node/wire.rs and
into proto/fmp/wire.rs, so the whole FMP wire surface lives with its
subsystem, matching the proto/fsp/wire.rs layout. The wire module becomes
pub(crate) mod wire; callers reach it via crate::proto::fmp::wire.
Behavior-neutral: pure relocation plus import-path rewrites across the
node/peer consumers; no logic change. Full lib suite green at baseline.
Remove the duplicated immutable fields (config, identity, startup_epoch,
started_at, is_leaf_only, max_connections/peers/links) from the Node
struct so the Arc<NodeContext> bundle is the single source of truth.
Previously Node owned these fields and a parallel context copy, kept in
lockstep by rebuild_context() at every mutation site — pure overhead that
existed only because of the duplication.
- Replace rebuild_context() with replace_context(): a clone-edit-swap of
the whole Arc. The per-instance context stays immutable; mutation swaps
the Arc. This is the sole runtime mutation path (constructors, leaf_only,
update_peers).
- Add Copy-returning accessors startup_epoch() and max_connections()/
max_peers()/max_links(); migrate the remaining direct field readers onto
the accessors. node_addr()/npub()/Debug now read identity/is_leaf_only
from the context.
- update_peers reads the pre-update peer set from the live context Arc
before building a fresh Config + context and swapping — preserving the
read-before-write ordering its mutation-window test depends on.
- Remove the test-only set_max_* setters; tests set the limits on Config at
construction instead (new make_node_with_max_peers/links helpers).
- Add a ci-local guard that fails if the Node struct re-declares a bundled
field, so the single-store invariant can't silently regress.
cargo test --lib 1291/0; clippy -D warnings and release build clean.
Companion to the ethernet `accept_connections: false` rekey-deadlock
fix from earlier this release: the same dual-init failure mode shows
up over UDP when peers register by hostname, and the existing
addr_to_link-only carve-out in `should_admit_msg1` doesn't cover it.
The carve-out's first predicate keys `addr_to_link` by the literal
`TransportAddr` that `initiate_connection` inserted, which is the
hostname-form when a peer config carries a hostname (e.g.,
`core-vm.tail65015.ts.net:2121`). Inbound packets always arrive with
numeric source addrs because `udp_receive_loop` builds the
`TransportAddr` from the `SocketAddr` the kernel reports via
`recvfrom`. `TransportAddr` equality is byte-exact, so the two forms
don't match and the lookup misses. With `udp.accept_connections:
false` (or `udp.outbound_only: true`, which forces it false) the
gate then rejects the rekey msg1 from an established peer. The
dual-init tie-breaker stalls because the loser side never produces
msg2; both sides retry indefinitely and the winner side keeps
logging "Dual rekey initiation: we win, dropping their msg1" at 1Hz.
The earlier ethernet fix didn't generalize to this variant because
ethernet TransportAddrs are always numeric MAC bytes — both the
config-time form and the inbound-arrival form match identically.
Add a second predicate to `should_admit_msg1`: an active peer's
`current_addr()` matching `(transport_id, remote_addr)`.
`current_addr` is updated and refreshed from inbound encrypted-frame
source addrs (`handlers/encrypted.rs`), which are always numeric
`SocketAddr`-form, so this catches the established peer regardless
of how its `addr_to_link` key was originally inserted. The fast
`addr_to_link` check stays first; the iteration over peers is
bounded by peer count and only runs when the first predicate misses.
Regression coverage in this commit:
- Unit test `test_should_admit_msg1_admits_rekey_when_addr_form_differs`
in `src/node/tests/handshake.rs`. Constructs the failing scenario
in-process: `addr_to_link` populated with hostname-form key, peer's
`current_addr` at the resolved numeric form, query with numeric form.
Without the new predicate this fails immediately.
- New integration topology `rekey-outbound-only` plus matching
docker-compose profile. Same 5-node mesh shape as `rekey-accept-off`
but `inject-config` sets `udp.outbound_only: true` on node-b and
rewrites node-b's peer-c address from the numeric docker IP to the
docker hostname (`node-c:2121`), reproducing the production
hostname-vs-numeric mismatch. The test asserts no sustained
"Dual rekey initiation: we win" log lines on any node (>10 = bug)
and the existing rekey health checks catch the connectivity loss
the loop produces.
- `testing/ci-local.sh` and `.github/workflows/ci.yml` extended to
run the new variant in the local sweep and the GitHub CI integration
matrix alongside `rekey` and `rekey-accept-off`.
Verified locally: full `bash testing/ci-local.sh` sweep passes 29/29
suites (23m 12s) with the new variant green; 1084 unit tests pass.
Single combined commit covering five interlocking pieces of test and
CI work that landed during the v0.3.0-prep cycle.
## fips-gateway robustness
- src/bin/fips-gateway.rs DNS upstream probe converted from a 3-second
hard-fail to a bounded retry loop (5 attempts × 1s timeout, 1s sleep
between attempts; ~10s worst case). Covers the cold-boot race where
the daemon's TUN is up but the DNS responder at [::1]:5354 is still
binding. Each failed attempt logs at INFO. In production the binary's
retry is the live recovery mechanism; with retry it recovers silently
instead of relying on Restart=on-failure (~5s blip + spurious ERROR
per cycle).
- packaging/debian/fips-gateway.service `ExecStartPre` now waits up to
30 seconds for the daemon's `fips0` TUN to appear before exec'ing
the gateway binary. Eliminates the cold-boot race where the gateway
exits with `fips0 interface not found` and recovers via
`Restart=on-failure`, producing a 5-second blip and a spurious error
log per restart cycle.
- testing/docker/entrypoint.sh gateway-mode waits up to 30s for the
daemon's DNS responder to bind [::1]:5354 (probes once per second
with `dig @::1 -p 5354 ... test.fips`) before exec'ing fips-gateway.
Belt-and-suspenders with the binary's own retry: in CI we want
deterministic startup ordering. On timeout, fall through so the
binary's probe reports the definitive error.
## Test infrastructure DNS bind migration to ::1
After session 359's daemon DNS-bind default flipped from `127.0.0.1`
to `::1` (the production fix for ISSUE-2026-0002), the static-test
infrastructure was carrying a stale workaround that overrode the
default back to IPv4 loopback. The fips-gateway integration test
exposed the divergence: the gateway probes its DNS upstream at
`[::1]:5354` (production default) while the daemon was binding
`127.0.0.1:5354` from the template override — IPv6-explicit sockets
do not accept v4-mapped traffic, so the upstream probe exhausted
retries and the gateway exited.
- Drop the explicit `bind_addr: "127.0.0.1"` line from every test
config that emits it: testing/static/configs/node.template.yaml,
testing/chaos/configs/node.template.yaml, the sidecar heredoc in
testing/docker/entrypoint.sh, testing/acl-allowlist/generate-configs.sh
(six per-node blocks), testing/nat/scripts/generate-configs.sh, and
the four tor templates under testing/tor/. Daemon picks up its
production `::1` default.
- Flip the dnsmasq forwarder for `.fips` in testing/docker/Dockerfile
from `127.0.0.1#5354` to `::1#5354` so dnsmasq on the shared test
image continues to reach the daemon. Template and Dockerfile must
move together since most static suites resolve `<npub>.fips` via
the test-image dnsmasq.
## rekey-accept-off integration variant + UDP unit test
- New `rekey-accept-off` topology and docker-compose profile under
testing/static/. 2-node variant where node-b runs with
`udp.accept_connections: false`. Pins the regression class that
ISSUE-2026-0004 fixed (cross-connection winner's rekey msg1 was
being filtered by the accept_connections gate, breaking rekey).
- testing/static/scripts/rekey-test.sh accepts REKEY_TOPOLOGY and
REKEY_ACCEPT_OFF_NODES env vars; its inject-config subcommand
applies the per-node `udp.accept_connections: false` edit, and
the test asserts no sustained "Dual rekey initiation" log lines.
- New UDP variant of `should_admit_msg1` admit-rekey unit test in
src/node/tests/handshake.rs.
## ci-local.sh full integration coverage
- New runner functions and dispatcher entries for `acl-allowlist`,
`nat-cone` / `nat-symmetric` / `nat-lan`, `rekey-accept-off`,
`dns-resolver`, `deb-install`. Each integrates with the existing
summary tracking via `record`.
- New `--with-tor` flag (off by default) gates `tor-socks5-outbound`
and `tor-directory-mode` runners. Tor stays opt-in because both
harnesses depend on the live Tor network and would introduce a
flake source unrelated to the FIPS code.
- New suite arrays (`ACL_SUITES`, `NAT_SUITES`, `DNS_RESOLVER_SUITES`,
`DEB_INSTALL_SUITES`, `TOR_SUITES`) drive both the default sweep
and `--list` output.
- `run_suite` extended to accept the new suite names for `--only`
invocations.
## GitHub CI matrix expansions
- `gateway` matrix entry runs testing/static/scripts/gateway-test.sh
against the existing docker-compose `gateway` profile.
- `rekey-accept-off` matrix entry exercises the new topology with
REKEY_ACCEPT_OFF_NODES=b.
- `deb-install` matrix (debian12 + ubuntu24 + ubuntu26) runs
testing/deb-install/test.sh with privileged systemd containers.
~5-7 min cold cache, ~2 min warm per distro. Self-contained: builds
its own .deb in a Debian 12 cargo-deb builder image; does not
depend on the build job's pre-built artifact.
- `dns-resolver` matrix entry runs the full 13-scenario harness
(per-distro systemd resolver-backend tests + real-fips end-to-end
scenarios) in a single job. Pins the production DNS bind path that
ISSUE-2026-0002 lived in. ~7-12 min warm, ~12-15 min cold.
Verified locally: full `bash testing/ci-local.sh` sweep passes,
including 5/5 deb-install distros and all 13 dns-resolver scenarios.
Tor-inclusive sweep (`--with-tor`) verified in a follow-up run.
The accept_connections gate at the top of handle_msg1 was applied
unconditionally, so rekey msg1 from a peer with whom an established
link already existed was dropped on the same path as fresh handshakes
from strangers. Combined with the dual-init tie-breaker, this
deadlocked at ~25 minutes when both sides' rekey timers fired
near-simultaneously: the smaller-NodeAddr side wins as initiator and
expects the larger side to consume its rekey msg1, but if the larger
side has accept_connections=false the gate dropped it. Both sides
retried at 1 Hz indefinitely; the affected peer fell out of MMP-active
rotation.
Extract the gate decision into Node::should_admit_msg1, which admits
unconditionally when addr_to_link already has an entry for the
(transport_id, remote_addr) pair (rekey/restart on an established
session) and otherwise consults the transport's accept_connections().
Fresh msg1 from strangers is still rejected before any Noise crypto.
Three unit tests pin the truth table: no transport (admit), accept_off
no-link (reject, behavior unchanged), accept_off with-link (admit, the
carve-out).
The fix generalizes for free to BLE, which has the same Node-level gate.
TCP and Tor were never subject to this deadlock because their accept
condition is runtime state (bind_addr.is_some() / onion_address.is_some()),
not a config flag.
Update the default UDP bind port from 4000 to 2121 (decimal) and the
default Ethernet EtherType from 0x88B5 to 0x2121 across all source
code, documentation, configuration templates, test fixtures, and
scripts. Remove references to "IEEE 802 experimental range" since
0x2121 is not in that range.
Each node generates a random 8-byte startup epoch, encrypted inside
both Noise IK handshake messages (msg1 and msg2). When a peer's msg1
arrives with a different epoch than the stored value, the node tears
down the stale session and processes the msg1 as a new connection,
enabling near-instant restart detection instead of the 30-second
dead timeout.
Wire format impact:
- msg1: 82 -> 106 bytes (added 24-byte encrypted epoch after ss DH)
- msg2: 33 -> 57 bytes (added 24-byte encrypted epoch after se DH)
- Wire msg1: 90 -> 114 bytes, wire msg2: 45 -> 69 bytes
Add message-level retry for Noise IK handshake within the 30s timeout
window. Previously, a lost msg1 or msg2 required the full timeout to
expire before cleanup and retry. Under 10% bidirectional loss (~19%
per attempt), this made connection establishment unreliable.
Initiator resends stored msg1 bytes with exponential backoff (1s, 2s,
4s, 8s, 16s — 5 resends). Responder stores msg2 and resends on
duplicate msg1 receipt. Duplicate msg2 at initiator drops silently
via existing pending_outbound cleanup.
Config: handshake_resend_interval_ms (1000), handshake_resend_backoff
(2.0), handshake_max_resends (5) on node.rate_limit.
P(all 6 attempts fail) under 19% loss = 0.19^6 ≈ 0.005%.
Under high-throughput forwarding, the kernel default 212KB receive
buffer overflows, silently dropping ~11% of UDP datagrams at transit
nodes. Add configurable recv_buf_size and send_buf_size to UdpConfig
(default 2MB each) using socket2 for pre-bind buffer configuration.
Startup log now reports actual buffer sizes granted by the kernel.
Requires net.core.rmem_max >= 2097152 on the host for the full 2MB
to take effect; otherwise the kernel silently clamps.
Split handlers.rs (986 lines) into handlers/ with 5 subfiles organized
by responsibility: rx_loop, encrypted, handshake, dispatch, timeout.
Split tests.rs (2350 lines) into tests/ with 4 subfiles: unit tests,
handshake integration, spanning tree convergence, and bloom filter tests.
Shared test helpers extracted to tests/mod.rs.
Visibility adjusted from pub(super) to pub(in crate::node) for handler
methods now two levels deep. Unused imports cleaned up in node/mod.rs.
All 316 tests pass, zero warnings.
Johnathan Corgan