mirror of
https://github.com/jmcorgan/fips.git
synced 2026-08-09 16:24:45 +00:00
13a98ae7025def95ec594e12897ded6e62819d18
27
Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
 Johnathan Corgan
|
bf2e7a0892 |
Derive each peer's npub once instead of once per tick
The per-tick stats snapshot ran a bech32 encode for every tracked peer, and for the common mesh peer — one with no hosts-file entry and no configured alias — it ran a second one, because the display-name fallback chain bottoms out in the same encode. At 240 peers that was 14.1 ms per tick, a third of the tick body and its second largest cost, all of it recomputing values that cannot change. Cache the npub and the shortened npub on the peer at construction. An npub is a pure function of the peer's public key, and the identity is never mutated after construction: there is no setter, no identity_mut, and no assignment to the field anywhere in the tree, so the cache cannot go stale. The display name itself is deliberately NOT cached. Two of its inputs do mutate at runtime — the alias map and the host map, the latter reloaded on this same tick — so a resolved name stored on the peer would go stale on an alias change or a hosts reload. Only the immutable component is memoized. Tests cover both constructors, that the cached npub matches the identity, and that the display name still tracks an alias change. The memoization itself is asserted by pointer stability rather than by timing, so it is deterministic under load. Three deliberate breaks were each caught by exactly one test: re-deriving instead of memoizing, populating one constructor's cache from the wrong source, and reordering the display-name fallback so it stops honoring aliases. |
||
|
Johnathan Corgan
|
d6457fa74f |
Stop awaiting the advert refetch on the retry tick
process_pending_retries runs inline on the node's 1-second rx-loop tick. For each due peer it awaited a Nostr relay fetch carrying a 2-second timeout, and discarded the result. With up to sixteen due peers in one tick body, the timeouts stack: field profiling measured single 2.00 s stalls as the common case and a worst tick of 12.4 s against a 1 s period, with every other rx-loop arm delayed behind it by as much as 4.2 s. Spawn the refetch instead of awaiting it, matching the pattern the failure arm of the same loop already uses thirty lines below. The dial now uses whatever advert is cached at that moment and the refreshed one lands for the next retry of that peer. Since retries are backoff-paced, that defers the benefit by one backoff interval rather than losing it, and the result was already being discarded, so nothing downstream read it. The test drives four due peers whose refetches all hang against a local listener that accepts and never speaks, so the fetch burns its full timeout with no network egress. Awaited, the call takes 8.0 s; spawned, it returns in milliseconds. It also asserts every due peer was still attempted and rescheduled, so a version that skipped the dial entirely cannot pass it. |
||
|
Johnathan Corgan
|
08a226fb63 |
Test cost-based parent selection and kernel-drop detection as unit tests
The cost-selection chaos scenarios (cost-reeval, cost-avoidance, cost-stability, depth-vs-cost, mixed-technology, bottleneck-parent) tested TreeState::evaluate_parent's decision logic through a Docker mesh that could not exercise it reliably: the tree roots at whichever node holds the smallest NodeAddr, MMP link costs take several measurement windows to settle, and the parent hold-down plus hysteresis timing all confound the outcome. A deterministic link-cost flap still produced zero periodic parent switches in a full run. Replace those six scenarios with deterministic unit tests in src/tree/tests.rs that drive evaluate_parent directly: cheaper-link selection at equal depth, switch-on-cost-change, hysteresis suppressing a marginal change while allowing a significant one, and the depth-versus-cost effective-depth tradeoff. Each is constructed so that breaking the cost or hysteresis logic makes it fail. The congestion kernel-drop signal (SO_RXQ_OVFL) cannot be provoked deterministically in Docker: a fresh daemon reader keeps up with container-speed traffic, so the socket receive queue never overflows (an unshaped run with a 4 KB buffer and heavy traffic recorded zero drops on every node). Extract the drop-detection edge -- read the cumulative counter, fire an event only on the transition into a new drop burst -- into TransportDropState::observe_drops and unit-test it directly. congestion-stress keeps its ECN and MMP congestion-signal assertions, which do need the real shaped bottleneck queue. Remove the retired scenarios from both CI runners and update the chaos README. |
||
|
Johnathan Corgan
|
d9a4a7807c |
node: make the shared context the sole store of immutable state
Remove the duplicated immutable fields (config, identity, startup_epoch, started_at, is_leaf_only, max_connections/peers/links) from the Node struct so the Arc<NodeContext> bundle is the single source of truth. Previously Node owned these fields and a parallel context copy, kept in lockstep by rebuild_context() at every mutation site — pure overhead that existed only because of the duplication. - Replace rebuild_context() with replace_context(): a clone-edit-swap of the whole Arc. The per-instance context stays immutable; mutation swaps the Arc. This is the sole runtime mutation path (constructors, leaf_only, update_peers). - Add Copy-returning accessors startup_epoch() and max_connections()/ max_peers()/max_links(); migrate the remaining direct field readers onto the accessors. node_addr()/npub()/Debug now read identity/is_leaf_only from the context. - update_peers reads the pre-update peer set from the live context Arc before building a fresh Config + context and swapping — preserving the read-before-write ordering its mutation-window test depends on. - Remove the test-only set_max_* setters; tests set the limits on Config at construction instead (new make_node_with_max_peers/links helpers). - Add a ci-local guard that fails if the Node struct re-declares a bundled field, so the single-store invariant can't silently regress. cargo test --lib 1291/0; clippy -D warnings and release build clean. |
||
|
Johnathan Corgan
|
08b8b3908e |
node: extract immutable state into a shared context and atomic metric registry
Store node counters in an atomic metric registry read through &self, and introduce a shared NodeContext bundle holding the effectively-immutable fields (config, identity, startup epoch, capability limits). Source the immutable config and identity reads across the receive hot path, the handshake/session/mmp/encrypted state machines, and the discovery, tree, bloom, retry, and lifecycle modules through the context accessors rather than direct field reads. The Node fields and the context are rebuilt in lockstep at every mutation site. |
||
 Martti MalmiandJohnathan Corgan
|
da0d9d39a0 |
node: refresh active peer paths without dropping links
Add Node::update_peers for runtime peer-list refresh. It re-derives the active peer connections from a new peer configuration, adding newly configured peers and removing those no longer present, while keeping links to peers that remain in the set rather than tearing every connection down. The call returns an UpdatePeersOutcome summarizing the added, removed, and retained peers. PeerAddress gains a seen_at_ms recency field (with_seen_at_ms). Active path selection now sorts address candidates by recency so the most recently observed address wins when concurrent path probes race. complete_rekey_msg2 now returns the remote peer's startup epoch alongside the new Noise session, letting the rekey path detect a peer restart and clear stale session state. A stale FSP session is cleared when a peer restart is detected during FMP rekey or cross-connection promotion, so the session-layer map no longer lingers out of sync with the freshly promoted peer. Per-tick work budgets bound the connection churn in a single node tick (MAX_DISCOVERY_CONNECTS_PER_TICK, MAX_RETRY_CONNECTIONS_PER_TICK, MAX_PARALLEL_PATH_CANDIDATES_PER_PEER); work beyond a tick's budget is deferred to the next tick rather than discarded. Co-authored-by: Johnathan Corgan <johnathan@corganlabs.com> |
||
|
Johnathan Corgan
|
f6429c19d2 | Merge maint into master (admission-gate Msg2 silent-drop + integration suite) | ||
|
Johnathan Corgan
|
5b229c03bf |
node: skip Msg1 → Msg2 reply when at max_peers cap
Move the max_peers cap check in handle_msg1 forward, from the late check inside promote_connection (which fires after Msg2 has already been built and put on the wire) to an early position after identity verification but before index allocation and the Msg2 send. When the gate fires for a net-new identity, the Msg1 is silent-dropped — no response goes back to the peer, no AEAD compute or wire bytes are spent. Bypass preserved for known peers (reconnect / cross-connection): if the sender's NodeAddr is already in self.peers, or if a pending outbound connection is in flight to the same identity, the gate is skipped so legitimate maintenance traffic continues to work. The late check inside promote_connection is intentionally retained as defense-in-depth against future call sites or a disconnect racing between the early-gate decision and promotion. Wire-cost rationale: a 45 s tcpdump at saturation observed ~3.6 cap-denials/s steady-state, each previously paying the full Noise IK responder crypto + Msg2 (~104 B) on the wire before being rejected. The bigger value is cleaner peer-side semantics — the peer no longer sees a fake-completed handshake whose data frames subsequently fail decryption locally. Two new unit tests cover the cases: - handle_msg1_silent_drops_at_cap_for_new_peer drives a wire-pumped Msg1 from a fresh identity into a saturated node and asserts no Msg2 reaches the sender socket. Stash-verifies as FAIL on the pre-fix tree (Msg2 hits the wire) and PASS post-fix. - handle_msg1_admits_existing_peer_at_cap drives a Msg1 from an identity already in self.peers and asserts the gate does not evict it. This is a regression check (the no-gate tree behaves the same way here, but the test guards against an accidental future gate that breaks known-peer admit). |
||
|
Johnathan Corgan
|
6991a152e6 |
Merge maint into master (outbound admission gate + mesh-size parent skip)
Brings two structural fixes landed on maint: - compute_mesh_size: explicit parent skip in the children loop, so the disjoint-subtree invariant no longer depends on peer_declaration cache freshness. - max_peers: outbound connection-initiation gated on the cap (auto-reconnect retries, Nostr-mediated discovery established adoption, and both sides of the NAT-traversal punch sequence). Inbound msg1 admission gate unchanged. |
||
|
Johnathan Corgan
|
d4687e5d30 |
node: gate outbound connection initiation on max_peers
node.limits.max_peers was honored only on inbound msg1 admission (handshake.rs handle_msg1 returns PeerLimitExceeded when peers.len is at the cap). Four outbound initiation paths proceeded unconditionally at capacity: auto-reconnect retries (process_pending_retries), Nostr-mediated discovery's BootstrapEvent::Established adoption (poll_nostr_discovery), NAT-traversal punch initiation (the outgoing side of the offer/answer/punch sequence in the Nostr discovery runtime), and NAT-traversal punch response (the incoming side of the same sequence). A saturated node burned CPU, UDP probes, STUN observations, and Nostr relay traffic on connections that the inbound gate would reject the moment they reached msg1. Introduce Node::outbound_admission_check (peers.len < max_peers, or true when max_peers == 0 as the no-cap sentinel) and gate the four paths. The discovery runtime lives in a separate task and does not hold a Node reference; bridge via an Arc<AtomicBool> the runtime reads and Node refreshes once per tick from outbound_admission_check. The atomic granularity is intentionally loose: one-tick lag is acceptable because the inbound msg1 gate continues to be the authoritative cap, and in-flight handshakes started below the cap are allowed to complete. Inbound gate at handshake.rs is unchanged. |
||
|
Martti MalmiandJohnathan Corgan
|
c0ccedb491 | nostr: start discovery without blocking node startup | ||
|
Martti MalmiandJohnathan Corgan
|
87d1af0269 |
nostr: ignore stale traversal for active peers
Skip BootstrapEvent::Established and BootstrapEvent::Failed dispatch in poll_nostr_discovery for peers that are already connected or actively handshaking. Without these guards, stale traversal events arriving after a peer connected through a different path would either attempt to adopt a redundant socket against the live connection (Established) or poison the per-peer failure-state cooldown and trigger redundant retraversal via schedule_retry / try_peer_addresses (Failed). The four guard sites use a new is_connecting_to_peer helper extracted from the existing closure inside initiate_peer_connection; the helper checks for an in-flight outbound handshake state. adopt_established_traversal gains a defense-in-depth check returning PeerAlreadyExists when called against an already-promoted peer, so the invariant holds if a future caller bypasses the outer dispatch guard. Side benefit: narrows a cooldown-poisoning vector previously available to an attacker injecting stale failure events for an active peer. Test coverage for the new behavior: - test_try_peer_addresses_skips_connected_peer - test_try_peer_addresses_skips_connecting_peer - test_nostr_traversal_failure_skips_connected_peer (Failed-arm event injection) - test_nostr_traversal_established_skips_connected_peer (Established-arm event injection, mirror of the Failed test) - test_adopted_traversal_skips_already_connected_peer (adopt_established_traversal defense-in-depth) CHANGELOG entry under [Unreleased] / Fixed. Closes #87 |
||
|
64cc30df12 |
Schedule retry on startup peer-init failure
When initiate_peer_connections() runs at boot, address resolution can fail for an entire peer (no operational transport for the configured transport types, all addresses unreachable, NAT rebind invalidated cached endpoints, etc.). Before this change the failure was logged and silently forgotten — the peer entry stayed in a dead state forever, accepting incoming pings but unable to answer them, until the daemon was manually restarted. The retry plumbing (schedule_retry / process_pending_retries with exponential backoff) already exists and is wired into the post-handshake failure paths (BootstrapEvent::Failed, MMP dead-link timeout, handshake timeout). The startup loop just wasn't calling it. Mirror the BootstrapEvent::Failed path: on a startup peer-init error, parse the peer's npub and call schedule_retry so the peer recovers without operator intervention. Includes a regression test that asserts retry_pending is populated when initiate_peer_connections() fails for a peer with no operational transport. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> |
||
|
Johnathan Corgan
|
ae607431eb |
Per-destination TCP MSS clamping at the TUN boundary
Adds source-side TCP MSS clamping informed by per-destination path MTU learned via discovery, with a conservative IPv6-minimum-derived ceiling for cold flows where discovery has not yet completed. Closes the multi-hop default-config TCP wedges observed in production where a sender's local-floor MSS exceeds what some intermediate forwarder hop is willing to carry: silent drops, no PTB feedback through the userspace TUN to the kernel TCP stack, retransmits at the same too- large MSS, application connection times out. ## Architecture A new `Arc<RwLock<HashMap<FipsAddress, u16>>>` field `path_mtu_lookup` on Node mirrors the per-destination path MTU in a form accessible from sync TUN reader/writer threads. A new `per_flow_max_mss` helper in `src/upper/tun.rs` reads the lookup at SYN-clamp time and returns the appropriate ceiling for the flow. Three write sites populate `path_mtu_lookup`: 1. **Discovery originator branch** of `handle_lookup_response`: the path MTU bottleneck accumulated through the reverse path lands here when a LookupResponse arrives at the originator. Same value also lands in `coord_cache` per the existing `insert_with_path_mtu` API. 2. **FMP peer-promotion seed** (`seed_path_mtu_for_link_peer`): when an FMP link-layer peer is promoted to active, the local outgoing-link MTU on the peer's transport seeds the lookup. Tighter existing values (learned via discovery) are preserved; the seed only writes when no entry exists or the existing value is looser than the link MTU. Without this seed, directly-configured peers (auto_connect / static peer config) would leave `path_mtu_lookup` empty for their FipsAddress because the FSP session establishes without ever issuing a LookupRequest. 3. **Target-edge fold at `send_lookup_response`**: when a node is the discovery target, it folds its own outgoing-link MTU to the response's next-hop into `path_mtu` before sending. Without this fold, the response leaves the target with `path_mtu = u16::MAX` and only intermediate transits min-fold; the target's first reverse-path hop is never represented in the bottleneck calculation. Refactored the existing transit- side min-fold into a shared `apply_outgoing_link_mtu_to_response` helper called from both sites. ## Read-side: per_flow_max_mss Two TUN call sites consume the lookup: - Outbound `handle_tun_packet` clamps SYN MSS using packet[24..40] (IPv6 destination) as the lookup key. - Inbound `TunWriter::run` clamps SYN-ACK MSS using packet[8..24] (IPv6 source). When the lookup contains a learned value, the helper computes `min(global_max_mss, effective_ipv6_mtu(path_mtu) - 60)` where 60 is IPv6 (40) + TCP (20) headers and `effective_ipv6_mtu` accounts for the FIPS encapsulation overhead. When the lookup is empty for a destination — the cold-flow case — the helper returns `min(global_max_mss, IPv6-minimum-derived ceiling)`. RFC 8200 mandates every IPv6 path accept ≥1280-byte packets, so the IPv6-minimum-derived MSS (1280 - 77 - 60 = 1143) fits any compliant path. Without this conservative ceiling, the first SYN to a destination with no learned path MTU exits the TUN at the kernel-natural MSS (TUN MTU - 60), and the application connection wedges silently before discovery completes for a corrected second SYN to fire. The fix is provably safe: the ceiling is taken with `min` against the local global so operators with even tighter local floors are never loosened upward. Subsequent flows pick up the actual learned per-destination value once discovery (or the FMP-promotion seed for direct peers) populates the lookup. ## Diagnostic logging All write and read sites emit instrumentation suitable for operators bisecting a wedged path: - `debug!` log on every `path_mtu_lookup` write (discovery originator path and FMP-promotion seed path), showing the FipsAddress, written value, prior value, and post-write map size. `warn!` on poisoned-lock failure path. - `trace!` log per `per_flow_max_mss` call covering every fall-through branch (wrong addr_bytes length, non-fd::/8 prefix, lookup poisoned, no entry for destination, empty-lookup conservative ceiling) and the success path. trace level filters out under normal log settings; capture with `RUST_LOG=info,fips::node::handlers::discovery=debug,fips::upper::tun=trace`. ## Tests 15 new unit tests across 3 files: - `per_flow_max_mss` (8 tests in `src/upper/tun.rs::tests`): empty-lookup conservative ceiling, empty-lookup global-smaller floor, learned-value-overrides-conservative, per-destination smaller, per-destination larger capped by global, non-fips addr, short addr slice, per-destination independence. - `seed_path_mtu_for_link_peer` (4 tests in `src/node/tests/unit.rs`): seed when empty, keep tighter existing, tighten looser existing, no-op for unknown transport. - Discovery integration (3 tests in `src/node/tests/discovery.rs`): apply_outgoing_link_mtu_to_response on unknown peer no-op, two-node target-edge fold (path_mtu reflects target-edge link), three-node chain transit min-fold (existing test, updated for target-edge inclusion). Two pre-existing discovery tests had assertions updated to account for the target-edge fold: - `test_response_path_mtu_two_node`: previously asserted `u16::MAX` (no transit to min-fold); now asserts 1280 (the test transport MTU, folded in by send_lookup_response). - `test_response_path_mtu_four_node_chain`: previously asserted 1350 (transit MTUs only); now asserts 1280 (target-edge MTU is the bottleneck). - `test_transit_forwards_when_mtu_sufficient`: previously asserted 1400 (transit MTU only); now asserts 1280 (target- edge MTU is the bottleneck). ## Verification Local CI on this commit: 29/29 suites pass, 1105 lib tests pass, clippy --all-targets --all-features -D warnings clean, cargo fmt clean. Production deploy verified via trace capture across the managed fleet: cold-flow conservative ceiling branch fires on first SYN, learned-lookup branch takes over once discovery completes, both behaviors observable end-to-end at the SYN MSS on the wire. No wire-format change. No config-format change. |
||
|
Johnathan Corgan
|
8448e38510 |
Make Node::transport_mtu() deterministic across restarts (TCP black hole fix)
Default-config TCP flows between fips peers were stalling completely (cwnd-pinned, 0 bps for 10s+) on a non-trivial fraction of restarts. Reproducible with iperf3 between any two peers. Root cause: `Node::transport_mtu()` iterated `self.transports.values()` (HashMap with default RandomState hasher) and returned `handle.mtu()` of the first one whose `is_operational()` returned true. Two stacked sources of non-determinism stacked on each other: HashMap iteration order is randomized per-process via RandomState, and async transport `.start()` completion order races each daemon restart. The returned value drives the TCP MSS clamp ceiling computed once at TUN init (src/upper/tun.rs:501-524) and stored as immutable max_mss in the reader/writer thread state. When the picker landed on a transport with MTU > 1357 (any non-UDP-1280 in the standard fleet defaults), `max_mss > 1220` (kernel's natural fips0-MTU-derived MSS), the daemon's clamp was silently a no-op, and the kernel emitted 1220-byte segments. Those wrap into 1280-byte IPv6 → 1357-byte fips datagrams that exceed UDP-1280 transports at any forwarding hop, causing silent drops with no PTB feedback to the kernel TCP stack. Fix: return min across operational transports instead of first-iterated. With UDP-1280 in the configured set (the common case), `transport_mtu = 1280`, `max_mss = 1143 < 1220`, daemon's clamp engages, MSS=1143 reaches the wire, packets fit, throughput recovers. Empirical green light from a single-UDP-config end-to-end test: iperf3-without-`-M` recovered to ~21 Mbps with no operator-side nft TCPMSS rules. Adds three unit tests: - transport_mtu_returns_min_across_operational: pin selection to smallest MTU when multiple operational transports differ. - transport_mtu_fallback_when_no_operational_transports: 1280 fallback. - transport_mtu_min_with_single_operational: trivial single-transport case. The `effective_ipv6_mtu` field reported by `fipsctl show status` was also racy (consequence of the same bug); fixed by this change as a side effect. |
||
 
|
34e00b9f6e |
Add Nostr-mediated overlay discovery and UDP NAT traversal (#53)
Optional peer discovery and NAT hole-punching path gated behind a new
`nostr-discovery` cargo feature. Nodes publish signed overlay endpoint
adverts to public Nostr relays, consume peer adverts to populate
fallback dial addresses, and use STUN-assisted UDP hole punching with
NIP-59 gift-wrap offer/answer signaling to establish direct UDP paths
between NATed peers. Once a punched socket is up, it is handed into
the existing FIPS UDP transport and the standard Noise/FMP session
stack takes over unchanged.
The cargo feature is in the default feature set
(`default = ["nostr-discovery"]`) so stock builds include it; a
build that explicitly disables default features (or selects a
feature set without `nostr-discovery`) does not link the nostr /
nostr-sdk crates and does not emit a no-op poll in the tick loop.
Runtime behavior is independently gated by
`node.discovery.nostr.enabled`, which defaults to false; if the
config enables Nostr on a non-feature build, startup logs a
warning and continues without it.
== Cargo feature and dependencies
- New cargo feature `nostr-discovery = ["dep:nostr", "dep:nostr-sdk"]`.
Not in the default feature set.
- New optional Linux-only dependencies: `nostr 0.44` (features: std,
nip59) and `nostr-sdk 0.44`. Gift-wrap unwrap is hand-rolled in
`src/discovery/nostr/signal.rs` rather than relying on the SDK's
rumor-author check, which FIPS sidesteps by trusting `seal.pubkey`
exclusively.
== Wire format
Overlay advert event: `kind 37195`, parameterized replaceable
(NIP-01 application-defined replaceable range 30000-39999), with
`d = "fips-overlay-v1"`. The digits visually spell FIPS (7=F, 1=I,
9=P, 5=S); a relay survey confirmed the kind is unused.
Advert content carries the version tag, endpoint list
(`udp|tcp|tor` + addr), optional signal-relay and stun-server
metadata, and `issuedAt` / `expiresAt` timestamps. Endpoint
`addr: "nat"` is the sentinel that triggers traversal on the peer
side. NIP-40 `expiration` tag bounds staleness on permanent
shutdown. Lifecycle relies on parameterized-replaceable
supersession; the daemon does not emit NIP-09 kind-5 deletes —
strict relays (Damus, Primal) race delete-against-replace and can
silently drop the replacement.
Gift-wrapped signal event: `kind 21059`. Punch packets carry magic
values `PUNCH_MAGIC` / `PUNCH_ACK_MAGIC`, a sequence number, and a
16-byte session hash.
== Discovery surface
- `src/discovery.rs` (always compiled)
- `EstablishedTraversal`: bound UDP socket + selected remote +
peer npub + optional transport name/config tuning overrides.
- `BootstrapHandoffResult`: returned on successful handoff —
allocated transport id, local/remote addrs, peer NodeAddr,
session id.
- `src/discovery/nostr/` (`#![cfg(feature = "nostr-discovery")]`)
- `types.rs`: wire and control types described above. `ADVERT_KIND`
constant. `BootstrapError` enumerates failure modes (disabled,
missing advert, missing NAT endpoint, no usable relays, invalid
advert, invalid npub, signal timeout, punch timeout, replay,
STUN failure, protocol, nostr, io, serde, event-parse).
- `runtime.rs`: `NostrDiscovery` coordinator. Owns the shared
nostr-sdk `Client`, subscribes to advert + signal event kinds,
maintains a bounded advert cache and a bounded seen-sessions
replay set, drains `BootstrapEvent::{Established, Failed}` for
the node to consume, exposes `update_local_advert`,
`request_connect`, `advert_endpoints_for_peer`,
`cached_open_discovery_candidates`, and `shutdown`.
- `signal.rs`: NIP-59 gift-wrap encode/decode. Outbound wraps are
built against per-attempt ephemeral keys; inbound events are
unwrapped against the node identity.
- `stun.rs`: RFC 5389/8489 Binding Request client with
XOR-MAPPED-ADDRESS parsing for both IPv4 and IPv6; used only to
observe the initiator's own reflexive address against its
locally configured STUN list (peer-advertised STUN is
informational, never an egress target).
- `traversal.rs`: per-attempt candidate-pair punch planner.
Allocates a fresh `0.0.0.0:0` UDP socket per attempt, enumerates
LAN-private and ULA interface addresses alongside the STUN
reflexive address, schedules probe/ack exchanges at the
configured interval for the configured duration, and picks the
first candidate pair that authenticates end-to-end.
Strategy ordering is Reflexive↔Reflexive first, then LAN, then
Mixed. The STUN-observed pair is the only candidate that's reliable
across arbitrary network topologies; trying it first prevents the
planner from latching onto a misleading host-candidate path before
the reflexive path gets a chance. There is no catch-all
Local↔Local strategy: a previous design that paired every local
host candidate from one side with every local host candidate from
the other could declare success on a one-way reachable asymmetric
L3 path (corporate VPN, Tailscale subnet route, overlapping private
address space), only for the FMP handshake to stall because the
return path didn't match. The legitimate `Lan` strategy still pairs
candidates that share a subnet.
== Configuration surface
`node.discovery.nostr.*` (`NostrDiscoveryConfig`), all `serde(default)`
with `deny_unknown_fields`:
- `enabled` (default false), `advertise` (default true)
- `advert_relays`, `dm_relays`, `stun_servers`: defaults are
`wss://relay.damus.io`, `wss://nos.lol`, `wss://offchain.pub`
for both relay lists, and Google / Cloudflare / Twilio for STUN.
Operators are expected to override for production. Other
verified-working public relays for reference:
`nostr.bitcoiner.social`, `nostr-pub.wellorder.net`,
`nostr.oxtr.dev`, `nostr.mom`.
- `app` (default `"fips-overlay-v1"`), `signal_ttl_secs` (120)
- `policy`: `NostrDiscoveryPolicy::{Disabled, ConfiguredOnly (default),
Open}` — controls whether advert-derived endpoints are consumed
only for peers carrying `via_nostr = true`, or also for
non-configured peers within a budget cap.
- `share_local_candidates` (default false) — when false, the offer's
`local_addresses` list is empty and peers see only the reflexive
address. Enable per-node only for genuinely same-LAN deployments;
off-by-default eliminates the misleading-path failure mode for
the common case where peers are not on the same broadcast domain.
- `open_discovery_max_pending` (64) — caps queued open-discovery
retries; bounded by available outbound slots.
- `max_concurrent_incoming_offers` (16) — semaphore against offer
spam; excess offers are debug-logged and dropped.
- `advert_cache_max_entries` (2048) and `seen_sessions_max_entries`
(2048) — bound memory under ambient relay volume; overflow
evictions are debug-logged.
- `attempt_timeout_secs` (10), `replay_window_secs` (300)
- `punch_start_delay_ms` (2000), `punch_interval_ms` (200),
`punch_duration_ms` (10000)
- `advert_ttl_secs` (3600), `advert_refresh_secs` (1800)
Per-peer and per-transport flags:
- `PeerConfig.via_nostr: bool` — when true (and Nostr is enabled),
advert-derived addresses are appended as fallback dial candidates
after static addresses for that peer.
- `PeerConfig.addresses` is now `serde(default)` and may be empty
when `via_nostr: true`; validation requires at least one of the
two to be present per peer, and the error message names the
peer's npub.
- `UdpConfig.advertise_on_nostr: Option<bool>` and
`UdpConfig.public: Option<bool>` — UDP transports can be
advertised either as direct `host:port` (public = true) or as the
`addr: "nat"` sentinel that triggers rendezvous on the peer side.
- `TcpConfig.advertise_on_nostr` and `TorConfig.advertise_on_nostr`
— TCP and Tor onion endpoints can be advertised as directly
reachable.
- A reserved peer address `transport: udp, addr: "nat"` parses without
special-casing in YAML and routes through the bootstrap runtime.
Cross-field validation (`Config::validate`, called from `Node::new`
and `Node::with_identity`):
- Any transport with `advertise_on_nostr = true` requires
`node.discovery.nostr.enabled = true`.
- Any peer with `via_nostr = true` requires
`node.discovery.nostr.enabled = true`.
- A non-public UDP advert (`advertise_on_nostr = true`,
`public = false` — i.e. `udp:nat`) additionally requires at least
one `dm_relay` and at least one `stun_server`.
Surfaced as `ConfigError::Validation`.
== Node integration
`src/node/lifecycle.rs` is the main integration point.
- At node start (after transports are up, before TUN), if Nostr is
enabled and the feature is compiled in, `NostrDiscovery::start` is
invoked, the initial local overlay advert is built from the live
transport set and published, and the runtime handle is stored.
- The rx tick loop calls `poll_nostr_discovery` (feature-gated both
at method definition and call site), which refreshes the local
advert, drains bootstrap events, adopts established traversals,
schedules retries for failed traversals, and — under `policy:
open` — enqueues outbound retries for non-configured peers
visible in the advert cache, bounded by
`open_discovery_max_pending` and the remaining outbound slots.
- Outbound peer dialing is refactored to `try_peer_addresses`, which
first exhausts the static address list in priority order and only
then appends advert-derived fallback addresses; both lists run
through the same `attempt_peer_address_list` code path. The
`udp:nat` sentinel address triggers `NostrDiscovery::request_connect`
for the peer instead of a direct dial and returns `Ok(())`.
- `build_overlay_advert` walks operational transports, consults
per-instance `UdpConfig` / `TcpConfig` / `TorConfig` (matching by
optional transport instance name), and emits an `OverlayAdvert`
including `signalRelays` and `stunServers` when any UDP endpoint
is advertised as NAT.
- `adopt_established_traversal` is the bootstrap handoff API:
allocates a new `TransportId`, constructs a `UdpTransport` with
the user-supplied (or default) `UdpConfig`, calls the new
`adopt_socket_async` to reuse the punched socket verbatim,
registers the transport in the normal transport map, records it
in `bootstrap_transports`, and calls `initiate_connection` so the
normal handshake path runs. On failure, the transport is stopped
and removed cleanly and the set membership is rolled back.
- On clean shutdown, `NostrDiscovery::shutdown` is awaited so
background tasks stop before transports are torn down. (The
advert is not explicitly retracted; NIP-40 expiration plus the
next refresh from any live publisher supersedes it.)
New `Node` fields:
- `nostr_discovery: Option<Arc<NostrDiscovery>>` (feature-gated).
- `bootstrap_transports: HashSet<TransportId>` — per-peer UDP
transports adopted from NAT traversal, cleaned up via
`cleanup_bootstrap_transport_if_unused` whenever the link,
connection, peer, or pending-connect referencing them is removed.
Retry and error surface:
- `RetryState.expires_at_ms: Option<u64>` — optional absolute expiry
for a retry entry. `pump_retries` drops expired entries with an
info log. Used for open-discovery retries, which expire at two
times the advert TTL.
- New `NodeError::BootstrapHandoff(String)` returned from
`adopt_established_traversal` when the underlying transport
adoption fails or local address discovery fails.
- New `ConfigError::Validation(String)`.
- A small refactor extracts `Node::now_ms()` and reuses it across
lifecycle, rx-loop tick, and timeout bookkeeping.
== UDP transport
`src/transport/udp/`:
- `UdpRawSocket::adopt(std::net::UdpSocket, recv_buf, send_buf)`:
adopts an externally bound socket, makes it non-blocking, applies
the configured buffer sizes (warning if the kernel clamps), and
reports the resulting local address. Preserves the NAT mapping —
no rebind.
- `UdpTransport::adopt_socket_async(std::net::UdpSocket)`: the
`start_async` analogue for an already-bound socket, wiring the
async socket and recv task exactly as the fresh-bind path would.
- `Drop` impl for `UdpTransport`: if a transport is dropped while
still holding a recv task or socket (for example on error
teardown), aborts the task, clears the socket, and emits a debug
log so the cleanup is visible in tracing rather than silent.
== Logging and observability
Default `EnvFilter` demotes third-party relay-pool DEBUG output to
TRACE-only: `nostr_relay_pool`, `nostr_sdk`, and `nostr` are pinned
at INFO when our level is anything below TRACE, and at TRACE when
our level is TRACE — so the raw frames are still reachable when
explicitly asked for. RUST_LOG continues to override completely.
Concise one-line DEBUG events are emitted at the meaningful points
in the discovery / hole-punch sequence:
- `advert: published` (event id, relay count, endpoints, ttl)
- `advert: peer cached` (notify-loop ingress for non-self)
- `advert: resolved` (cache hit / relay fetch outcome)
- `traversal: initiator starting`
- `traversal: initiator STUN observed` (reflexive, local count)
- `traversal: offer sent` (session id, relay count, event id)
- `traversal: answer received` (accepted, reflexive, local)
- `traversal: initiator punch succeeded` (remote addr)
- `traversal: offer received` (responder side)
- `traversal: responder STUN observed`
- `traversal: answer sent`
- `traversal: responder punch succeeded`
Npubs are shortened to `npub1<4>..<4>` and event/session ids to
their first 8 hex characters.
Other operator-facing logs:
- `UdpTransport` adoption and drop paths log at info / debug.
- `adopt_established_traversal` logs at debug on entry and info on
successful return, tagged with peer npub, session id, transport
id, and both socket endpoints, so the bootstrap handoff is
traceable end-to-end alongside the `UdpTransport::drop` log.
- `cleanup_bootstrap_transport_if_unused` logs at debug when the
reference-count check drops an adopted transport.
- `connect_peer` tags its entry `debug!` with `peer_npub` so
downstream STUN, punch, and handshake logs for the same peer
correlate for operators.
- Advert-cache and seen-sessions overflow evictions log at debug so
mis-sized caps are visible under ambient relay volume.
- Gift-wrap unwrap failures on `SIGNAL_KIND` events log at trace
(hot path: fires for every unrelated signal event on the same
relay).
- Traversal-offer handler failures log at debug. Expected conditions
such as punch timeout on symmetric NAT are covered there; real
problems are reported upstream via `BootstrapEvent::Failed`.
- Inbound-offer rate-limit messages name the governing config field
(`max_concurrent_incoming_offers`) and state that the offer was
rate-limited rather than failing.
== Tests
- 18 new unit tests in `src/discovery/nostr/tests.rs` covering advert
encoding, signal envelope round-trip, STUN parsing, punch-packet
codec, and replay-window enforcement. Run under the
`nostr-discovery` feature.
- Config-validation tests in `src/config/mod.rs` covering the three
cross-field invariants and YAML parsing of the full
`node.discovery.nostr` block plus `peers[].via_nostr`, empty
`addresses` with `via_nostr: true`, and a `udp: nat` address.
- `src/node/tests/bootstrap.rs` integration tests that drive a
synthetic traversal (bound UDP socket pair + synthetic peer
identity) through `adopt_established_traversal` and assert the
Noise handshake completes over the adopted socket.
- Punch-planner tests assert reflexive-before-LAN ordering and that
same-LAN scenarios still include the LAN target in the plan.
- `testing/nat/` Docker NAT lab harness:
- Local `strfry` relay, local STUN responder, and one or two
router containers performing `iptables` NAT.
- Node LAN interfaces are provisioned with explicit `veth` pairs
injected into the node and router namespaces so every packet
traverses the router namespace (plain Docker bridges are not
used for the LAN).
- `cone` scenario: both peers behind full-cone-emulation NAT
(SNAT with source-port preservation, inbound DNAT back to the
single LAN host regardless of remote source); asserts UDP
traversal succeeds and link remote addresses are on the router
WAN subnet.
- `symmetric` scenario: `MASQUERADE --random-fully`; asserts UDP
traversal fails and TCP fallback converges over router-
published WAN addresses.
- `lan` scenario: both peers share a LAN subnet; asserts LAN
addresses are preferred over reflexive ones.
- Cleanup tears down all profile-gated services
(`--profile cone --profile symmetric --profile lan`) so no
orphan containers survive a run.
- `testing/scripts/build.sh` builds the Docker test image with
`--features "tui nostr-discovery"` by default so NAT-harness
binaries include bootstrap support.
== CI
- Linux release build and nextest unit-test job both use
`--features "gateway nostr-discovery"` so the feature-gated code
and its unit tests compile and run in CI.
- Three new integration matrix entries (`nat-cone`, `nat-symmetric`,
`nat-lan`) invoke `testing/nat/scripts/nat-test.sh`, collect
`docker compose logs` on failure, and always stop containers.
== Packaging and operations
- `packaging/common/fips.yaml` ships a fully commented
`node.discovery.nostr.*` block, plus documented
`advertise_on_nostr` / `public` examples under the UDP transport,
an `advertise_on_nostr` example under TCP, and a `via_nostr: true`
example under the static peer section with both a direct
`host:port` UDP address and a `udp: nat` fallback.
- `.github/workflows/package-openwrt.yml`: NIP-94 release event
publishes target the new default relay set.
== Documentation
- `README.md`: overlay discovery + NAT traversal moved from
"Near-term priorities" into "What works today".
- `docs/design/fips-intro.md`: rewrites the paragraphs that
previously described Nostr discovery and NAT traversal as future
work; describes the shipped mechanism and the feature gate.
- `docs/design/fips-transport-layer.md`: drops the "(future
direction)" qualifier from the Nostr Relay Discovery section,
expands with the `udp:nat` advertisement and bootstrap handoff
description, and updates the Current State callout.
- `docs/design/fips-mesh-layer.md`: notes that mid-session NAT
rebinding (roaming) and initial NAT traversal (Nostr path) are
distinct mechanisms.
- `docs/design/fips-configuration.md`: documents the full
`node.discovery.nostr.*` surface, including the three resource
caps and `share_local_candidates`.
- `docs/design/fips-nostr-discovery.md`: design and configuration
reference for the shipped mechanism, including the empty-
`addresses`-with-`via_nostr` shorthand.
- `docs/proposals/nostr-udp-hole-punch-protocol.md`: adds an
Implemented status callout, clarifies that the punch socket is
per-peer and per-attempt rather than shared with the application
listener, aligns field names with the shipped JSON
(`sessionId`, `issuedAt` / `expiresAt`, `reflexiveAddress`,
`localAddresses`, `stunServer`), sets the `d`-tag to
`fips-overlay-v1`, names the kind as 37195, and notes that
advertised STUN entries are informational.
- `docs/proposals/README.md`: adds a Status column and marks the
hole-punching proposal Implemented.
- `CHANGELOG.md`: Unreleased > Added entry covering the discovery
path, STUN/punch path, configuration surface, and Docker NAT lab.
Co-authored-by: Johnathan Corgan <johnathan@corganlabs.com>
|
||
|
Johnathan Corgan
|
42834b8008 |
Fix auto-connect reconnect on graceful peer disconnect
handle_disconnect() called remove_active_peer without scheduling a reconnect, orphaning auto-connect peers on a clean upstream shutdown. Mirror the pattern from the other three peer-removal paths (link-dead, decrypt failure, peer restart) which all schedule reconnect after removal. Adds test_disconnect_schedules_reconnect regression test that verifies handle_disconnect populates retry_pending for an auto-connect peer. Visibility of handle_disconnect bumped to pub(in crate::node) for direct unit-test access. Fixes #60. |
||
|
Johnathan Corgan
|
13c0b70dc3 |
Add rustfmt formatting policy and reformat codebase
Add rustfmt.toml with stable defaults and apply cargo fmt to all source files. This establishes a consistent formatting baseline for CI enforcement. |
||
 KieranandJohnathan Corgan
|
e8ef15acb7 |
Fix stale session cleanup and identity cache pre-seeding
Cherry-picked from v0l PR #6 (issue #5): 1. remove_active_peer() now removes the end-to-end session from self.sessions when evicting a peer. The stale Established entry caused initiate_session() to silently return Ok(()) via the is_established() guard, preventing session re-establishment after link-layer reconnection. 2. initiate_peer_connections() pre-seeds the identity cache from configured peer npubs at startup, so TUN packets can be dispatched immediately without waiting for handshake completion. 3. schedule_reconnect() preserves accumulated backoff when a retry entry already exists, preventing exponential backoff reset on repeated link-dead cycles. Includes regression tests for all three fixes. |
||
|
Johnathan Corgan
|
324535e76d |
Make auto-connect peers retry indefinitely on initial connection failure
Previously, static peers configured with AutoConnect gave up after 6 attempts (1 initial + 5 retries). If the remote peer was offline at startup, the node permanently abandoned the connection. The reconnect path (after MMP link-dead) already retried indefinitely but only activated after a peer had been previously connected. Remove the reconnect parameter from schedule_retry() and always set reconnect=true when creating retry entries, since only auto-connect peers reach this code path. The 300s backoff cap prevents resource waste. The max_retries=0 config still works as an explicit kill switch. |
||
|
Johnathan Corgan
|
daf1e629df |
Change default UDP port to 2121 and EtherType to 0x2121
Update the default UDP bind port from 4000 to 2121 (decimal) and the default Ethernet EtherType from 0x88B5 to 0x2121 across all source code, documentation, configuration templates, test fixtures, and scripts. Remove references to "IEEE 802 experimental range" since 0x2121 is not in that range. |
||
|
Johnathan Corgan
|
58664c7c77 |
Update dependencies: rand 0.10, rtnetlink 0.20, tun 0.8, and others
Bump rand (0.8→0.10), rtnetlink (0.14→0.20), tun (0.7→0.8), simple-dns (0.9→0.11), socket2 (0.5→0.6), and criterion (0.5→0.8). Migrate all rand call sites: thread_rng()→rng(), gen()→random(), gen_range()→random_range(), RngCore→Rng trait. Work around secp256k1 0.30 requiring rand 0.8 by generating random bytes directly and constructing SecretKey from slice. Migrate rtnetlink to builder-based API: LinkSetRequest replaced with LinkUnspec builder + change(), RouteAddRequest replaced with RouteMessageBuilder. Remove bloom benchmark (criterion 0.8 incompatible with old harness config). |
||
|
Johnathan Corgan
|
f920526ece |
Add epoch-based peer restart detection to Noise IK handshake
Each node generates a random 8-byte startup epoch, encrypted inside both Noise IK handshake messages (msg1 and msg2). When a peer's msg1 arrives with a different epoch than the stored value, the node tears down the stale session and processes the msg1 as a new connection, enabling near-instant restart detection instead of the 30-second dead timeout. Wire format impact: - msg1: 82 -> 106 bytes (added 24-byte encrypted epoch after ss DH) - msg2: 33 -> 57 bytes (added 24-byte encrypted epoch after se DH) - Wire msg1: 90 -> 114 bytes, wire msg2: 45 -> 69 bytes |
||
|
Johnathan Corgan
|
78a73e1749 |
Auto-reconnect after MMP peer removal, directed outbound configs, sim improvements
Auto-reconnect: - Add per-peer auto_reconnect config (default true) to PeerConfig - schedule_reconnect() feeds removed peers back into retry system with unlimited retries and exponential backoff after MMP dead timeout - RetryState gains reconnect flag to distinguish startup retries (max_retries-limited) from auto-reconnect (unlimited) Retry re-fire fix: - process_pending_retries() now pushes retry_after_ms past the handshake timeout window after successful initiate_peer_connection(), preventing retries from firing every tick with no backoff Chaos sim improvements: - Directed outbound configs: BFS spanning tree + lower-ID-first assignment eliminates dual-connect race conditions in simulation - Save runner log (runner.log) alongside per-node logs for event correlation - Increase churn-20 traffic aggressiveness and node churn (max_down_nodes 3→5, traffic interval min 0s, duration max 90s, concurrent flows 5→10) |
||
|
Johnathan Corgan
|
b8a1f322c2 |
Module reorganization and clippy cleanup
Move single-consumer modules into node/:
- rate_limit.rs, wire.rs, dns.rs — exclusively used by node subsystem
- Reduces top-level lib.rs from 16 to 13 modules
Split large files into focused subdirectories:
- noise.rs (1475 lines) → noise/{mod, handshake, session, replay, tests}.rs
- tree.rs (1479 lines) → tree/{mod, coordinate, declaration, state, tests}.rs
- bloom.rs (849 lines) → bloom/{mod, filter, state, tests}.rs
- All public APIs re-exported from mod.rs, no external import changes
Remove unused rate_limit defaults:
- HANDSHAKE_TIMEOUT_SECS, MAX_PENDING_INBOUND constants
- Default constructor eliminated in favor of with_params() taking config values
Fix all clippy warnings across codebase:
- Remove .clone() on Copy types, collapse nested ifs, replace match-return-None
with ?, remove/gate unused code, fix loop indexing, remove unnecessary casts
- Box large PeerSlot enum variants to reduce size disparity
- cargo clippy --all-targets now reports zero warnings
|
||
|
Johnathan Corgan
|
7463d8799a |
Promote 27 hardcoded constants to configurable parameters
Add 9 config subsection structs (LimitsConfig, RateLimitConfig, RetryConfig, CacheConfig, DiscoveryConfig, TreeConfig, BloomConfig, SessionConfig, BuffersConfig) under node.* with serde defaults. Wire all configurable values through to consuming code: - Resource limits (max_connections, max_peers, max_links, max_pending_inbound) - Rate limiting (handshake_burst, handshake_rate, handshake_timeout_secs) - Retry/backoff (consolidate max_retries, base_interval_secs under node.retry.*, add max_backoff_secs) - Cache sizes/TTL (coord_size, coord_ttl_secs, route_size) - Discovery (ttl, timeout_secs, recent_expiry_secs) - Spanning tree (root_refresh_secs, announce_min_interval_ms, parent_switch_threshold) - Bloom filter (update_debounce_ms) - Session/data plane (default_hop_limit, pending_packets_per_dest, pending_max_destinations) - Internal buffers (packet_channel, tun_channel, dns_channel) - Network internals (base_rtt_ms, tick_interval_secs) - DNS responder TTL (dns.ttl) REPLAY_WINDOW_SIZE kept as compile-time constant (array sizing). Disable flaky test_discovery_100_nodes (run with --ignored). |
||
|
Johnathan Corgan
|
cc29c51cac |
Refactor node/handlers.rs and node/tests.rs into subdirectories
Split handlers.rs (986 lines) into handlers/ with 5 subfiles organized by responsibility: rx_loop, encrypted, handshake, dispatch, timeout. Split tests.rs (2350 lines) into tests/ with 4 subfiles: unit tests, handshake integration, spanning tree convergence, and bloom filter tests. Shared test helpers extracted to tests/mod.rs. Visibility adjusted from pub(super) to pub(in crate::node) for handler methods now two levels deep. Unused imports cleaned up in node/mod.rs. All 316 tests pass, zero warnings. |