An FSP session rekey could leave the two endpoints holding different
key sets for a brief window: if a handshake message was lost in
transit, one side rotated to the new keys while the other did not.
Traffic sealed in one key epoch then reached a peer still on the
other epoch and failed to decrypt, producing bursts of AEAD
decryption failures and dropped connectivity until a later rekey
cycle reconverged the pair. Choreographing the cutover order cannot
close this window: any fixed ordering still leaves a skew that
packet reordering widens.
Make rekey correctness independent of cutover timing by overlapping
the key epochs on the receive path. During a rekey transition the
receiver trial-decrypts each frame against every live session it
holds: current, the not-yet-promoted pending session, and the
draining previous session. The K-bit becomes a hint that orders the
trial-decrypt cascade rather than a hard gate, and a frame that
authenticates against the pending session is itself the cutover
signal. No rotation ordering and no packet reordering can then cause
a decryption failure.
The pre-rekey Noise session is held in the `previous` slot until the
peer has demonstrably moved off it. Its drain deadline is anchored
on the most recent frame the peer authenticated against that slot,
refreshed each time the trial-decrypt cascade lands there, rather
than on a fixed wall-clock timer started unilaterally at the local
cutover. A peer that never received the new keys keeps authenticating
against `previous` and the slot stays live; without this, a fixed
timer would erase the only key set that could decrypt the peer's
frames, producing a permanent silent decrypt failure on a live data
path. A peer that never catches up is handled by the existing FSP
session liveness path rather than by silent decrypt failure.
The lost-handshake liveness gap is closed separately by retransmitting
the third rekey handshake message until the peer is confirmed on the
new keys, with a bounded retry budget after which the rekey cycle is
cleanly abandoned and retried on the next timer.
Adds unit tests covering the trial-decrypt cascade (epoch selection,
promotion on pending decrypt, reordered old-epoch stragglers after
cutover, per-slot replay-window integrity), the msg3 retransmission
lifecycle, and the peer-progress-aware drain retirement.
Add a per-session signed jitter offset (uniform [-15, +15] seconds)
to the rekey timer triggers in check_rekey (FMP) and check_session_rekey
(FSP). The configured `node.rekey.after_secs` becomes the nominal
interval rather than a floor; mean is preserved. Desynchronizes
both endpoints in symmetric-start meshes so the dual-initiation
race stops occurring rather than being resolved after the fact by
the smaller-NodeAddr tie-breaker.
Per-session storage means each rekey cutover reconstructs the
session and redraws the jitter naturally — successive cycles get
independent offsets, preventing drift back into sync.
The chacha20 crate (RustCrypto) ships SSE2 + soft backends only — on
aarch64 (Apple Silicon, ARM Linux servers, Docker on M-series Macs) it
falls through to a portable software impl at ~600–800 MB/s/core. ring
0.17 wraps BoringSSL's hand-tuned ChaCha20-Poly1305, which dispatches
to NEON on aarch64 and AVX2/AVX-512 on x86_64 — typically 3-5 GB/s/core
on the same hardware.
Same wire format. ChaCha20-Poly1305 is byte-deterministic for a given
(key, nonce, plaintext, aad), so any correct AEAD implementation
produces identical ciphertext. The full noise test suite covers this
implicitly: IK and XK roundtrip handshakes, replay window correctness,
multi-message nonce sequencing, and 100-message stress all pass at
1129/1129 (the lib's full `cargo test` count) — these only succeed if
ring's output matches what the receiver's existing replay-window
decrypt path expects.
Implementation notes:
* `LessSafeKey` (and `UnboundKey`) deliberately do not implement
Clone for safety. `CipherState`'s manual Clone impl rebuilds it
from the retained 32-byte key — cheap for ChaCha20-Poly1305 since
construction is essentially a key copy + a constant-time check.
* The keyed AEAD is now cached in `CipherState.cipher` instead of
being re-derived per packet. This was already a perf win for the
chacha20poly1305 backend (`new_from_slice` per packet was hot in
profiles); for ring it's a bigger win because `LessSafeKey`
construction also derives the Poly1305 key.
* Public `Vec<u8>`-returning API preserved. New module-private
`seal`/`open` helpers wrap ring's `seal_in_place_append_tag` /
`open_in_place` so the per-packet allocation pattern is local to
one place.
* `EndToEndState::Established` triggers `clippy::large_enum_variant`
after the swap (`NoiseSession` grew from ~600 to ~1.5 KB because
ring precomputes the Poly1305 key state at construction). That
precomputation is the win — boxing the variant would re-add an
indirection per packet and work against it. `#[allow]`'d at the
enum decl with a justifying comment.
ring is widely deployed (rustls, hyper-rustls, AWS SDK, …) and a
pure-Rust crate (uses BoringSSL's asm via a vendored build). It
introduces no new C toolchain requirements that aren't already there
for any rustls user.
Bench data from a downstream consumer of this crate (Docker e2e,
DURATION=10, identical hardware before/after, aarch64 Linux on
Apple Silicon):
2-node direct (A↔B):
TCP 1-stream 437 → 1097 Mbps (2.51×)
TCP 4-stream 439 → 1109 Mbps (2.53×)
TCP 8-stream 445 → 1069 Mbps (2.40×)
UDP @1000 Mbit 599/40% loss → 1000 Mbps lossless
ping under load ~0.6 ms (unchanged)
3-node forced transit (A → C → B):
TCP 1-stream 438 → 1019 Mbps (2.33×)
TCP 4-stream 421 → 982 Mbps (2.33×)
TCP 8-stream 443 → 1031 Mbps (2.33×)
UDP @1000 Mbit 475/52% loss → 1000 Mbps lossless
ping under load 7.68 ms / 215 ms max → 0.72 ms / 3.6 ms max
The relay-path lift is the cleanest tell on the bottleneck: the
transit node was crypto-bound (single-threaded soft chacha couldn't
keep up with offered rate), so the queue accumulated under load. With
NEON the relay isn't crypto-bound and the queue stops accumulating —
the 215ms ping-tail collapses to 3.6ms.
After rekey cutover, frames encrypted with the old session can still
arrive during the 10s drain window. These carry large sender timestamps
from the old session, producing enormous transit deltas that spike the
EWMA jitter estimator (observed 2,000-7,000ms spikes on UDP links).
Add a time-based grace period (DRAIN_WINDOW_SECS + 5s = 15s) after
rekey that suppresses jitter updates until drain-window frames have
flushed. Baselines still update every frame so calculation resumes
cleanly after grace expires.
Fixes#10
The FSP XK rekey handshake has a race condition where the initiator
can cut over (K-bit flip) and send data encrypted with the new session
before msg3 reaches the responder. The responder has no pending session
yet, so K-bit detection fails and packets are dropped.
Defer FSP initiator cutover by 2 seconds after handshake completion
(FSP_CUTOVER_DELAY_MS) to give msg3 time to traverse the mesh. FMP
(IK, 2 messages) is unaffected since the responder completes during
msg1 processing.
Also fix MMP metric corruption after rekey cutover: the new session
starts with counter 0 but MMP state carries highest_counter and
GapTracker.expected_next from the old session, producing false
reorder counts, jitter spikes, and invalid OWD trends. Add
reset_for_rekey() methods that clear counter-dependent state while
preserving RTT estimates.
Additional fixes:
- Remove stale peers_by_index entry on abandon_rekey error path
- Replace redundant peers_by_index inserts with debug assertions
verifying the pre-registration invariant
- Tighten rekey integration test to zero tolerance (was 4 failures)
Implement periodic full rekey at both protocol layers using fresh DH
key exchanges. Uses the existing K-bit flag (FLAG_KEY_EPOCH /
FSP_FLAG_K) to coordinate cutover between peers.
FMP layer (IK pattern):
- ActivePeer gains rekey state: pending/previous sessions, K-bit epoch
tracking, drain window, dampening timer
- Handshake state stored on ActivePeer with msg1 sent on existing link
- Encrypted frame handler detects K-bit flips, promotes pending
sessions, falls back to previous session during drain
- Handshake handlers distinguish rekey from new connections using
addr_to_link lookup with identity-based fallback
- Free all session indices (current, rekey, pending, previous) on
peer removal
FSP layer (XK pattern):
- SessionEntry gains parallel rekey fields with XK-specific state
for the 3-message handshake
- Route availability check before FSP rekey initiation
- Encrypted session handler adds K-bit flip detection and dual-session
decrypt fallback
- SessionSetup/Ack/Msg3 handlers extended for rekey paths
Defense-in-depth:
- Consecutive decryption failure detector (threshold=20) triggers
forced peer removal instead of waiting for link-dead timeout
- Identity-based rekey detection as fallback when addr_to_link
doesn't match (e.g., TCP ephemeral ports)
Configuration: RekeyConfig with enabled flag, after_secs (default 120),
and after_messages (default 65536) thresholds.
Logging: info for successful K-bit cutover completions, warn for
failures, debug for intermediate handshake steps, trace for routine
operations (resends, drain cleanup).
Rekey lifecycle:
1. Timer/counter fires -> initiator starts new handshake
2. Old session continues handling traffic during handshake
3. Handshake completes -> initiator cuts over, flips K-bit
4. Responder sees flipped K-bit -> promotes new session
5. Both keep old session for 10s drain window
6. After drain, old session discarded
Integration test: Docker-based multi-phase test exercising both FMP
and FSP rekey with aggressive timers (35s). Verifies connectivity
across all 20 directed pairs survives two consecutive rekey cycles.
Includes rekey topology, docker-compose profile, and CI matrix entry.
Increase ping test convergence wait from 3s to 5s for CI reliability.
The session-layer handshake now uses the 3-message XK pattern instead
of the 2-message IK pattern, providing stronger initiator identity
hiding. The initiator static key is deferred to msg3 and encrypted
under the es+ee DH chain, so eavesdroppers cannot identify the
initiator from the handshake.
XK pattern: -> e, es (msg1) / <- e, ee + epoch (msg2) / -> s, se + epoch (msg3)
Key changes:
- Add XK handshake methods alongside existing IK methods in noise module
- Add SessionMsg3 wire format and FSP_PHASE_MSG3 (0x03) prefix
- Replace Responding state with AwaitingMsg3 in session state machine
- Rewrite session handlers: handle_session_setup defers identity to msg3,
handle_session_ack processes msg2 and sends msg3, new handle_session_msg3
completes the responder handshake and registers identity
- Link-layer (FMP) continues to use Noise IK unchanged
- Add comprehensive XK unit tests and update all integration tests
handle_session_ack() had a bug where take_state() followed by
a non-Initiating match would re-insert the entry with state=None,
causing panics on subsequent .state()/.state_mut() calls. Fix by
checking is_initiating() before take_state().
Add safe is_initiating()/is_responding() to SessionEntry (matching
existing is_established()) and convert all production .state() callers
to use these None-safe methods. Gate .state() with #[cfg(test)].
Add resend logic for SessionSetup/SessionAck messages routed through
the mesh. Stores the encoded payload on SessionEntry for resend in a
fresh SessionDatagram (so routing can adapt to topology changes).
Uses the same config parameters as link-layer retry.
Also fixes a latent bug: Initiating/Responding sessions previously
had no timeout — a stuck handshake would live forever. Now cleaned up
after handshake_timeout_secs (default 30s).
Responder idempotency: duplicate SessionSetup triggers resend of
stored SessionAck instead of being silently dropped. Initiator-side
duplicate SessionAck already handled safely (entry.take_state() sees
Established, puts it back and returns).
Handshake payload cleared on Established transition at both initiator
(handle_session_ack) and responder (handle_encrypted_session_msg).
Replace raw NodeAddr hex strings in log output with human-readable
identifiers using a four-tier lookup: configured alias, active peer
short npub, session endpoint short npub, or truncated hex fallback.
- Add PeerIdentity::short_npub() for compact npub display (npub1xxxx...yyyy)
- Add NodeAddr::short_hex() for compact hex fallback (first 4 bytes + ...)
- Add peer_aliases map populated at startup from peer config
- Add Node::peer_display_name() with four-tier resolution
- Add SessionEntry::remote_pubkey() accessor for session-layer lookups
- Update all 120 log field occurrences across 11 handler/node files
- Pass pre-computed display names to MMP static metric/teardown methods
to work around borrow checker constraints in iterator loops
MMP reports (SenderReport, ReceiverReport, PathMtuNotification) were
resetting the idle timer on both RX and TX paths, preventing sessions
from ever timing out when MMP traffic kept flowing. Changed touch() to
only be called for DataPacket send/receive and session establishment,
so sessions with no application data tear down after the idle timeout
even with active MMP measurement traffic.
Include source and destination coordinates on the first N DataPackets
of each session (default 5, configurable via
node.session.coords_warmup_packets). This warms transit node
coord_caches so multi-hop forwarding and error signal routing work
after SessionSetup cache entries expire.
On CoordsRequired receipt, reset the counter to re-enable coordinate
inclusion for the next N packets, handling mid-session cache expiry
and path changes.
Changes:
- SessionConfig: add coords_warmup_packets field (default 5)
- SessionEntry: add coords_warmup_remaining counter, initialized on
Established transition (both initiator and responder paths)
- send_session_data(): attach coords via DataPacket::with_coords()
while counter > 0, decrement per send
- handle_coords_required(): reset counter for affected session
- 4 new unit tests for counter lifecycle and config default
Sessions in the Established state that have no activity for 90 seconds
are now automatically removed. This ensures idle sessions are torn down
before transit node coord_cache entries expire (300s TTL), so that when
traffic resumes a fresh SessionSetup re-warms transit node caches with
current coordinates.
The identity cache now stores registration timestamps and expires
entries after 60 seconds via lazy expiry on lookup. This prevents
unbounded growth while allowing natural repopulation through DNS
resolution on next use.
Timer ordering: identity (60s) < session (90s) < coord_cache (300s).
Both timeouts are configurable: node.session.idle_timeout_secs and
node.cache.identity_ttl_secs. Setting idle_timeout_secs to 0 disables
session idle purging.
Changes:
- Add idle_timeout_secs (default 90) to SessionConfig
- Add identity_ttl_secs (default 60) to CacheConfig
- Add timestamp to identity_cache entries, lazy expiry on lookup
- Add purge_idle_sessions() called from tick loop
- Remove #[cfg(test)] from SessionEntry::last_activity()
- 7 new tests covering timeout behavior and edge cases
Implement Noise IK session handshake between arbitrary endpoints, carried
inside SessionDatagram envelopes through the mesh. Sessions use a three-state
machine (Initiating → Established for initiator, Responding → Established
for responder on first DataPacket). Includes session initiation API,
encrypted data transfer, simultaneous initiation tie-break, error signal
handlers (CoordsRequired, PathBroken), and local delivery wiring in the
forwarding handler.
New files: node/session.rs (state types), handlers/session.rs (~500 lines,
all session message handlers + send path), tests/session.rs (11 tests).
100-node integration test establishes sessions across random topology,
sends bidirectional encrypted datagrams through injected TUN channels,
and verifies 200/200 deliveries with 100% session establishment. Reports
routing statistics including avg 4.1 link hops per datagram.
404 tests pass (up from 393).
Johnathan Corgan
Martti MalmiandJohnathan Corgan