Store node counters in an atomic metric registry read through &self, and
introduce a shared NodeContext bundle holding the effectively-immutable
fields (config, identity, startup epoch, capability limits). Source the
immutable config and identity reads across the receive hot path, the
handshake/session/mmp/encrypted state machines, and the discovery, tree,
bloom, retry, and lifecycle modules through the context accessors rather
than direct field reads. The Node fields and the context are rebuilt in
lockstep at every mutation site.
The discovery request dedup cache (recent_requests) silently dropped
LookupRequests once it reached MAX_RECENT_DISCOVERY_REQUESTS, with no
counter to surface the condition. Add a DiscoveryReject::ReqDedupCacheFull
reject reason backed by a req_dedup_cache_full counter on DiscoveryStats,
mirroring the existing duplicate-request counter, and record it at the
drop site so the rejection is visible in show_routing.
Bring the runtime peer-list refresh and opt-in mDNS LAN discovery work
on master into the receive-path RejectReason / reloadable-config
integration branch. Code files auto-merge clean; the only conflict is
the CHANGELOG Unreleased section, resolved as the union of both sets of
entries.
Add Node::update_peers for runtime peer-list refresh. It re-derives the
active peer connections from a new peer configuration, adding newly
configured peers and removing those no longer present, while keeping
links to peers that remain in the set rather than tearing every
connection down. The call returns an UpdatePeersOutcome summarizing the
added, removed, and retained peers.
PeerAddress gains a seen_at_ms recency field (with_seen_at_ms). Active
path selection now sorts address candidates by recency so the most
recently observed address wins when concurrent path probes race.
complete_rekey_msg2 now returns the remote peer's startup epoch
alongside the new Noise session, letting the rekey path detect a peer
restart and clear stale session state. A stale FSP session is cleared
when a peer restart is detected during FMP rekey or cross-connection
promotion, so the session-layer map no longer lingers out of sync with
the freshly promoted peer.
Per-tick work budgets bound the connection churn in a single node tick
(MAX_DISCOVERY_CONNECTS_PER_TICK, MAX_RETRY_CONNECTIONS_PER_TICK,
MAX_PARALLEL_PATH_CANDIDATES_PER_PEER); work beyond a tick's budget is
deferred to the next tick rather than discarded.
Co-authored-by: Johnathan Corgan <johnathan@corganlabs.com>
Introduce a typed RejectReason enum and a NodeStats::record_reject
dispatch so every receive-path rejection-and-return site bumps a
machine-readable per-subsystem counter while keeping its operator-facing
log line. The top-level variants mirror the existing NodeStats subsystem
split (Tree, Bloom, Discovery, Forwarding) and add Handshake, Session,
Mmp, and Transport categories; HandshakeStats, SessionStats, and MmpStats
are new sub-stats.
Wired clusters: tree and MMP outbound sign-failure; the FSP session
unknown-session and state-machine cluster; the Noise IK handshake
state-machine cluster (msg1/msg2); and the decode / crypto / cap /
semantic tail across bloom, discovery, forwarding, mmp, and tree. The
TreeStats::ancestry_invalid counter, present since the scaffold but never
incremented, is now bumped from the validate_semantics ancestry rejection.
Several handshake, MMP, tree, and discovery paths that previously had no
counter at all are now counted, including the send_lookup_response
no-route drop (DiscoveryStats::resp_no_route).
Existing direct counters at the bloom / discovery / forwarding sites are
retained alongside the new dispatch while the rollout is in progress (the
bloom_poison tests expect the transitional +2 delta); a later change
collapses the duplicate increment.
Adds source-side TCP MSS clamping informed by per-destination path
MTU learned via discovery, with a conservative IPv6-minimum-derived
ceiling for cold flows where discovery has not yet completed. Closes
the multi-hop default-config TCP wedges observed in production where
a sender's local-floor MSS exceeds what some intermediate forwarder
hop is willing to carry: silent drops, no PTB feedback through the
userspace TUN to the kernel TCP stack, retransmits at the same too-
large MSS, application connection times out.
## Architecture
A new `Arc<RwLock<HashMap<FipsAddress, u16>>>` field
`path_mtu_lookup` on Node mirrors the per-destination path MTU in a
form accessible from sync TUN reader/writer threads. A new
`per_flow_max_mss` helper in `src/upper/tun.rs` reads the lookup at
SYN-clamp time and returns the appropriate ceiling for the flow.
Three write sites populate `path_mtu_lookup`:
1. **Discovery originator branch** of `handle_lookup_response`:
the path MTU bottleneck accumulated through the reverse path
lands here when a LookupResponse arrives at the originator.
Same value also lands in `coord_cache` per the existing
`insert_with_path_mtu` API.
2. **FMP peer-promotion seed** (`seed_path_mtu_for_link_peer`):
when an FMP link-layer peer is promoted to active, the local
outgoing-link MTU on the peer's transport seeds the lookup.
Tighter existing values (learned via discovery) are preserved;
the seed only writes when no entry exists or the existing
value is looser than the link MTU. Without this seed,
directly-configured peers (auto_connect / static peer config)
would leave `path_mtu_lookup` empty for their FipsAddress
because the FSP session establishes without ever issuing a
LookupRequest.
3. **Target-edge fold at `send_lookup_response`**: when a node is
the discovery target, it folds its own outgoing-link MTU to
the response's next-hop into `path_mtu` before sending.
Without this fold, the response leaves the target with
`path_mtu = u16::MAX` and only intermediate transits min-fold;
the target's first reverse-path hop is never represented in
the bottleneck calculation. Refactored the existing transit-
side min-fold into a shared `apply_outgoing_link_mtu_to_response`
helper called from both sites.
## Read-side: per_flow_max_mss
Two TUN call sites consume the lookup:
- Outbound `handle_tun_packet` clamps SYN MSS using packet[24..40]
(IPv6 destination) as the lookup key.
- Inbound `TunWriter::run` clamps SYN-ACK MSS using packet[8..24]
(IPv6 source).
When the lookup contains a learned value, the helper computes
`min(global_max_mss, effective_ipv6_mtu(path_mtu) - 60)` where 60
is IPv6 (40) + TCP (20) headers and `effective_ipv6_mtu` accounts
for the FIPS encapsulation overhead.
When the lookup is empty for a destination — the cold-flow case —
the helper returns `min(global_max_mss, IPv6-minimum-derived
ceiling)`. RFC 8200 mandates every IPv6 path accept ≥1280-byte
packets, so the IPv6-minimum-derived MSS (1280 - 77 - 60 = 1143)
fits any compliant path. Without this conservative ceiling, the
first SYN to a destination with no learned path MTU exits the TUN
at the kernel-natural MSS (TUN MTU - 60), and the application
connection wedges silently before discovery completes for a
corrected second SYN to fire. The fix is provably safe: the
ceiling is taken with `min` against the local global so operators
with even tighter local floors are never loosened upward.
Subsequent flows pick up the actual learned per-destination value
once discovery (or the FMP-promotion seed for direct peers)
populates the lookup.
## Diagnostic logging
All write and read sites emit instrumentation suitable for
operators bisecting a wedged path:
- `debug!` log on every `path_mtu_lookup` write (discovery
originator path and FMP-promotion seed path), showing the
FipsAddress, written value, prior value, and post-write map
size. `warn!` on poisoned-lock failure path.
- `trace!` log per `per_flow_max_mss` call covering every
fall-through branch (wrong addr_bytes length, non-fd::/8
prefix, lookup poisoned, no entry for destination, empty-lookup
conservative ceiling) and the success path. trace level filters
out under normal log settings; capture with
`RUST_LOG=info,fips::node::handlers::discovery=debug,fips::upper::tun=trace`.
## Tests
15 new unit tests across 3 files:
- `per_flow_max_mss` (8 tests in `src/upper/tun.rs::tests`):
empty-lookup conservative ceiling, empty-lookup global-smaller
floor, learned-value-overrides-conservative, per-destination
smaller, per-destination larger capped by global, non-fips
addr, short addr slice, per-destination independence.
- `seed_path_mtu_for_link_peer` (4 tests in
`src/node/tests/unit.rs`): seed when empty, keep tighter
existing, tighten looser existing, no-op for unknown
transport.
- Discovery integration (3 tests in `src/node/tests/discovery.rs`):
apply_outgoing_link_mtu_to_response on unknown peer no-op,
two-node target-edge fold (path_mtu reflects target-edge link),
three-node chain transit min-fold (existing test, updated for
target-edge inclusion).
Two pre-existing discovery tests had assertions updated to
account for the target-edge fold:
- `test_response_path_mtu_two_node`: previously asserted
`u16::MAX` (no transit to min-fold); now asserts 1280 (the
test transport MTU, folded in by send_lookup_response).
- `test_response_path_mtu_four_node_chain`: previously asserted
1350 (transit MTUs only); now asserts 1280 (target-edge MTU
is the bottleneck).
- `test_transit_forwards_when_mtu_sufficient`: previously
asserted 1400 (transit MTU only); now asserts 1280 (target-
edge MTU is the bottleneck).
## Verification
Local CI on this commit: 29/29 suites pass, 1105 lib tests pass,
clippy --all-targets --all-features -D warnings clean, cargo fmt
clean. Production deploy verified via trace capture across the
managed fleet: cold-flow conservative ceiling branch fires on
first SYN, learned-lookup branch takes over once discovery
completes, both behaviors observable end-to-end at the SYN MSS
on the wire.
No wire-format change. No config-format change.
The default 30s post-failure backoff (300s cap, doubling per
consecutive failure) was set to bound traffic from chatty apps
looking up unreachable targets, but in practice it dominates
cold-start mesh convergence: a single timed-out lookup during
initial bloom-filter propagation suppresses any retry for 30s, and
the existing reset triggers (parent change, new peer, first RTT,
reconnection) don't fire on a stable post-handshake topology. The
suppression window winds up dictating the protocol's effective
time-to-converge instead of bounding repeat traffic.
Replaces the single-lookup-with-internal-retry model
(`timeout_secs`/`retry_interval_secs`/`max_attempts`) with a
per-attempt timeout sequence in `node.discovery.attempt_timeouts_secs`,
defaulting to `[1, 2, 4, 8]`. Each attempt sends a fresh LookupRequest
with a new random request_id so successive attempts can take different
forwarding paths as the bloom and tree state evolve. The destination
is declared unreachable only after the sequence is exhausted (15s
total at the default).
Disables post-failure suppression by default (`backoff_base_secs`/
`backoff_max_secs` now `0`/`0`). The `DiscoveryBackoff` machinery
stays in tree (inert at zero base/cap); operators with chatty apps
generating repeat lookups against unreachable destinations can opt
back in.
`PendingLookup` field shape unchanged so the control-socket
`show_routing` JSON (`pending_lookups[].attempt`/`initiated_ms`/
`last_sent_ms`) keeps the same schema for fipstop and external
consumers; `last_sent_ms` now means "current-attempt start" under
the new state machine.
Extend the fipsctl control query interface with visibility into internal
state critical for protocol security auditing, mesh troubleshooting, and
operational monitoring.
New command:
fipsctl show identity-cache
Lists every node identity cached by the daemon (learned from DNS
resolution, peer handshakes, sessions, and static config). Shows
npub, IPv6 address, display name, and LRU age alongside the
configured cache capacity.
Extended queries:
show peers — Noise session counters (send_counter, highest received
counter) for rekey urgency assessment. Per-peer replay suppression
and consecutive decrypt failure counts for active attack detection.
Session index visibility for hijack analysis. Rekey lifecycle
state (in_progress, draining, K-bit epoch).
show sessions — Handshake resend count during establishment for
connectivity debugging. Rekey and session health fields
(session_start, K-bit, coords warmup, drain state) when
established.
show cache — Individual coordinate cache entries with tree
coordinates, depth, path MTU, and age. Enables route-level
debugging by showing exactly which destinations have cached
routes and via what tree path. Renames the top-level count
field from "entries" to "count" for clarity.
show routing — Pending discovery lookups expanded from count to
per-target detail (attempt number, age, last sent). Pending
TUN packet queue depth for backpressure visibility. Connection
retry state per peer (retry count, next attempt, auto-reconnect
flag).
Updates fipstop to match the revised show_cache and show_routing
response schemas. Updates README monitoring section with the complete
fipsctl command list.
Co-authored-by: Johnathan Corgan <johnathan@corganlabs.com>
Replace discovery flooding with bloom-filter-guided tree routing:
lookups sent only to tree peers (parent + children) whose bloom
filter contains the target. If no tree peer matches, fall back to
non-tree peers with bloom matches before dropping the request. This
produces single-path forwarding through the spanning tree (90%
traffic reduction) while recovering from dead ends caused by stale
bloom filters, tree restructuring, or transit node failures.
Remove visited bloom filter from LookupRequest wire format (-257
bytes per request). Tree routing is inherently loop-free; request_id
dedup handles edge cases during tree restructuring. Add response-
forwarded flag to prevent response routing loops from convergent
request paths.
Add originator-side exponential backoff (30s base, 300s cap) after
lookup timeouts and bloom misses. Backoff resets on topology changes
(parent switch, new peer, first RTT, reconnection).
Add transit-side per-target rate limiting (2s minimum interval) for
forwarded lookups as defense-in-depth.
Add discovery retry within the timeout window (default: send at T=0,
retry at T=5s, fail at T=10s) to compensate for single-path fragility.
Lookups with zero eligible tree peers fail immediately.
Improve discovery logging: promote key events to info (initiation,
success, timeout with failure count). Add debug logging for dedup,
pending packet retry, backoff suppression, forward rate limiting,
and backoff reset.
New config: discovery.backoff_base_secs, backoff_max_secs,
forward_min_interval_secs, retry_interval_secs, max_attempts.
New stats: req_backoff_suppressed, req_forward_rate_limited,
req_bloom_miss, req_no_tree_peer, req_fallback_forwarded.
Removed: req_already_visited, visited bloom filter.
Guard discovery triggers in handle_path_broken() and
handle_coords_required() with has_cached_identity() check. When the
XK responder receives an error signal before msg3 completes, the
initiator's identity is unknown, making LookupResponse proof
verification impossible. Skip discovery in this case — the handshake
retry mechanism handles recovery.
Downgrade the identity_cache miss log from ERROR to WARN since it's a
known race condition, not a bug.
Add min_mtu (u16) to LookupRequest and path_mtu (u16) to
LookupResponse, enabling the discovery system to report transport
MTU capability along the lookup path.
LookupRequest carries min_mtu (origin's minimum MTU requirement,
default 0 = no requirement). LookupResponse carries path_mtu
(initialized to u16::MAX by the target, reduced by transit nodes
via min(path_mtu, outgoing_link_mtu) on the reverse path).
path_mtu is a transit annotation like SessionDatagram.path_mtu and
is NOT included in the proof signature. The originator stores the
discovered path_mtu in CacheEntry alongside cached coordinates.
Wire format: +2 bytes each for LookupRequest and LookupResponse.
The target node's send_lookup_response() was using greedy tree routing
(find_next_hop) as the primary method to route responses back to the
origin. When the target had the origin's coords cached from a prior
lookup, find_next_hop would route the response to whichever peer was
closest to the origin in tree space -- which might not have been on the
request's forward path. That peer would have no recent_requests entry
for the request_id, causing it to treat the response as if it were the
originator, fail identity_cache lookup, and discard the response.
Fix: prefer the reverse-path (recent_requests.from_peer) as the
primary routing method for the first hop, falling back to
find_next_hop only if no recent_request entry exists.
Also adds diagnostic output on failure to aid future debugging.
Include target_coords in proof_bytes() signed data to prevent transit
nodes from substituting fake coordinates. Add mandatory signature
verification at the originator using the target's public key from
identity_cache (guaranteed available since lookups are only initiated
from contexts where the key is already cached).
Verification failure discards the response. Identity cache miss (should
never happen) logs an error and discards. Add four new tests covering
verification success, failure, cache miss, and coordinate substitution
detection.
Replace raw NodeAddr hex strings in log output with human-readable
identifiers using a four-tier lookup: configured alias, active peer
short npub, session endpoint short npub, or truncated hex fallback.
- Add PeerIdentity::short_npub() for compact npub display (npub1xxxx...yyyy)
- Add NodeAddr::short_hex() for compact hex fallback (first 4 bytes + ...)
- Add peer_aliases map populated at startup from peer config
- Add Node::peer_display_name() with four-tier resolution
- Add SessionEntry::remote_pubkey() accessor for session-layer lookups
- Update all 120 log field occurrences across 11 handler/node files
- Pass pre-computed display names to MMP static metric/teardown methods
to work around borrow checker constraints in iterator loops
- PathBroken handler: convert to async, trigger re-discovery via
maybe_initiate_lookup(), reset COORDS_PRESENT warmup counter
(was a stub that only invalidated coord_cache)
- CoordsRequired recovery timing: reset warmup counter in
handle_lookup_response() when discovery completes for an
established session, so COORDS_PRESENT packets fire after
fresh coords are available (not just on CoordsRequired receipt)
- Routing error rate limiting: add RoutingErrorRateLimiter
(100ms per-destination, matching ICMP PTB pattern) to gate
send_routing_error() at transit nodes
- Remove root refresh dead code: the 1800s periodic root
re-announcement in check_tree_state() only propagated to
depth 1 (sequence-only changes don't cascade). Root loss
detection relies on link failure propagation which works
correctly.
Identity cache: remove TTL-based expiry (60s TTL broke active sessions
after expiry since handle_tun_outbound checks identity_cache before
session table). Replace with LRU-only eviction bounded by configurable
identity_size (default 10K). Lookup now touches timestamp for LRU
freshness.
Cache merge: unify coord_cache and route_cache into single coordinate
cache. Both stored NodeAddr→TreeCoordinate; the layer distinction was
conceptual, not functional. Discovery-sourced entries now get the same
TTL+refresh treatment as session-sourced entries. Simplifies
find_next_hop() to single cache lookup.
Parent-change flush: clear coord_cache after recompute_coords() in both
parent-switch paths of handle_tree_announce(). Stale coordinates after
tree reconvergence cause dead-end routing that's more expensive than
re-discovery.
Tested: 493 unit tests passed, clippy clean, Docker mesh 20/20,
Docker chain 6/6.
Wire the discovery protocol into the data plane path so that
find_next_hop() consults route_cache as a fallback when coord_cache
has no entry. When session initiation fails due to missing routes,
trigger discovery (initiate_lookup) instead of immediately sending
ICMPv6 Destination Unreachable. On discovery completion, retry
session initiation for any pending TUN packets.
Changes:
- find_next_hop() falls back to route_cache when coord_cache misses
- handle_tun_outbound() triggers discovery on session failure
- handle_lookup_response() retries session after discovery completes
- handle_coords_required() triggers discovery for missing coordinates
- Add pending_lookups deduplication with 10-second timeout
- Periodic cleanup of stale lookups in tick handler
- New test: route_cache fallback verification in find_next_hop
Implement Noise IK session handshake between arbitrary endpoints, carried
inside SessionDatagram envelopes through the mesh. Sessions use a three-state
machine (Initiating → Established for initiator, Responding → Established
for responder on first DataPacket). Includes session initiation API,
encrypted data transfer, simultaneous initiation tie-break, error signal
handlers (CoordsRequired, PathBroken), and local delivery wiring in the
forwarding handler.
New files: node/session.rs (state types), handlers/session.rs (~500 lines,
all session message handlers + send path), tests/session.rs (11 tests).
100-node integration test establishes sessions across random topology,
sends bidirectional encrypted datagrams through injected TUN channels,
and verifies 200/200 deliveries with 100% session establishment. Reports
routing statistics including avg 4.1 link hops per datagram.
404 tests pass (up from 393).
Johnathan Corgan
Martti MalmiandJohnathan Corgan
Tim O'SheaandJohnathan Corgan