Two independent rendering glitches, both most visible over SSH and inside
tmux:
Startup: ratatui::try_init() enters the alternate screen but never clears
it, and the first terminal.draw() only emits cells that differ from an
assumed-blank internal buffer. On terminals that don't hand back a cleared
alternate buffer (notably tmux, and amplified by SSH latency) the prior
contents show through. Force a full repaint with terminal.clear() before
the first draw.
Quit: the input EventHandler spawned a detached thread that polled stdin in
a loop outliving the main loop, so at quit it kept reading after raw mode
was disabled and stray bytes (a keystroke or a terminal query response)
echoed onto the restored screen. Give the thread a stop flag and join it
before restoring the terminal; poll on a short fixed interval (decoupled
from the refresh tick) so quit stays responsive.
git log --oneline -1
When an FMP msg1 or FSP msg3 rekey retransmission budget is exhausted, the
cycle is abandoned and retried on the next timer. On lossy or high-latency
links this is an expected, self-limiting outcome: the existing session stays
valid and keeps carrying traffic, so the abort is not a failure that warrants
operator-level visibility. Demote both abandon-cycle messages from warn to
debug to cut steady-state log noise on nodes with many flapping peers.
A manual disconnect tore down only the local side and sent the peer nothing, so
the peer kept its session and never re-emitted its tree and filter
announcements; on reconnect it was never re-adopted as a child and its bloom
filter was never recorded. Send the disconnected peer a scoped Disconnect, the
same message graceful shutdown sends to all peers, so both sides tear down and
re-handshake cleanly on the next connection.
The pool's TTL clock (VirtualIpMapping.last_referenced) advanced only on
DNS re-query, never on traffic, and the mapping-TTL is wired equal to the
DNS TTL, so an in-use mapping was forced to drain at TTL and reclaimed at
the first zero-conntrack tick (a stale drain_start gave no grace effective
protection), breaking long-lived, bursty, or DNS-cached clients.
In tick(), refresh last_referenced whenever conntrack reports sessions > 0
so an actively used mapping never ages out, and recover a Draining mapping
to Active (clearing drain_start) when traffic resumes, so a later drain
gets a fresh grace window instead of a stale one. The Active arm now only
drains an idle mapping. The DNS-TTL / idle-reclaim-TTL wiring is unchanged.
Adds regression tests for continuous-traffic-survives-past-TTL, bursty
drain-then-recover, and fresh-grace-on-redrain.
The per-transport TCP inbound cap was hardwired to 256 and never read
node.limits.max_connections, so raising max_connections was a silent
no-op for inbound TCP. Resolve the effective cap with precedence:
explicit per-transport max_inbound_connections, then node-wide
max_connections, then the built-in default of 256. Established peers
remain bounded node-wide by add_connection, so deriving the per-transport
raw-accept ceiling from max_connections does not admit more real peers
across multiple transports.
Add effective_max_inbound on the TCP transport with a node_max_connections
setter wired from create_transports, plus a precedence unit test.
Estimate the OR-union cardinality over self plus every connected peer's
inbound filter, dropping the parent/child tree gating in
compute_mesh_size. Filter propagation is split-horizon, so cross-links
advertise near-complete mesh views; unioning all peers yields the same
set as the tree-only union in steady state (OR dedups overlap and no
filter can over-count) while damping the node-count flap on parent
switches, since dropping the parent no longer collapses the upward leg.
This also removes the estimate's dependence on tree-declaration cache
freshness.
Rename the debug-log child_count to contributor_count, adapt the two
membership-invariant tests to all-peers semantics, and add a test that
the estimate stays stable across a parent drop when a healthy cross-link
is present.
The inbound FilterAnnounce FPR cap rejects filters whose false-positive
rate (fill^k) exceeds the configured maximum. On the fixed 1 KB / k=5
filter, 0.05 corresponds to fill 0.549 (~1,300 reachable entries), and
the busiest nodes' aggregates were beginning to hit that ceiling as the
mesh grew. Raise the default to 0.10 (fill 0.631, ~1,630 entries) to
restore headroom toward the fixed-filter capacity limit. A saturated or
poisoned filter is ~100% FPR and remains rejected, so the antipoison
gate is not materially weakened.
Updates the config default, the config-reference and bloom-filter
design docs, and the changelog.
A node with a single tree peer has its periodic parent re-evaluation
disabled (it needs at least two peers for a meaningful comparison), so
it depends entirely on its peer pushing a TreeAnnounce for it to attach.
That push happens once at promotion time plus on the parent's slow
periodic no-change re-broadcast. If the one-shot attaching announce is
lost, the single-uplink node falls back to self-root and cannot recover
until the next periodic re-broadcast (reeval_interval_secs later),
stranding it out of the tree and unreachable end-to-end in the interim.
Make tree-position exchange self-healing on the receive path: when an
accepted TreeAnnounce advertises a root strictly worse (higher NodeAddr,
since election is smallest-wins) than our own, echo our current
declaration back to that peer. A stranded self-root node's announce now
provokes its better-rooted peer to re-push its real position immediately,
so the node re-attaches within a round-trip instead of waiting for the
periodic cadence.
Echo only in that one direction. If the peer's root is lower (better)
than ours, we are the stale side: the peer would ignore our worse root
anyway and we converge via the parent re-evaluation that follows, so
echoing back is pure waste and would double announce traffic in the
learning direction during a root change or partition merge. Equal roots
are already converged. The echo is bounded by the existing per-peer
500 ms tree-announce rate limiter and is a no-op once the peer adopts our
root, so it adds no traffic in a converged mesh.
Add a spanning-tree unit test that drives a converged child back to
self-root with its peer-ancestry view of the root cleared (modelling the
lost attaching announce), and asserts the root re-pushes on the
resulting root disagreement and the child re-attaches.
Ignore duplicate or counter-regressed ReceiverReports before updating
RTT, loss, goodput, or ETX, so a delayed or reordered report can no
longer poison link metrics. Compute the RTT-from-echo sample with
checked timestamp arithmetic and reject zero, negative, or out-of-range
results instead of risking wrap or underflow on untrusted wire values.
On the sender side, when receiver dwell time overflows the u16 wire
field, suppress the timestamp echo (send 0) and saturate dwell to
u16::MAX rather than truncating, so a bogus small RTT cannot be formed.
Adds duplicate, out-of-order, wrapped-add, and future-dated (checked_sub)
sample tests, asserts loss and goodput stay unchanged on a dropped
duplicate, and covers the dwell-overflow echo suppression. Documents the
behavior in the MMP design note and CHANGELOG.
Co-authored-by: Johnathan Corgan <johnathan@corganlabs.com>
On a saturated public-mesh node the connection-lifecycle and capacity-cap
events fire continuously and drown out the genuinely notable INFO/WARN
lines. Demote them to debug and drop a redundant duplicate:
- FMP K-bit cutover promotion (encrypted): info -> debug
- "Connection promoted to active peer" (handshake): info -> debug, and
remove the duplicate "Inbound peer promoted to active" line that
shadowed it on the inbound path
- "Peer restart detected" (handshake): info -> debug
- "Peer removed and state cleaned up" (dispatch): info -> debug
- "Rejecting inbound TCP connection (max_inbound_connections reached)"
(tcp): warn -> debug
- "Congestion detected, CE flag set on forwarded packet" (forwarding):
warn -> debug
- "Removing peer: link dead timeout" (mmp): warn -> debug
These are expected, high-frequency conditions on a busy public node (new
and reconnecting peers, ECN CE marking, the inbound connection cap, and
link-dead churn), not operator-actionable signals.
The mesh-size estimator summed the per-filter cardinality of the parent
filter and each child filter, which assumes those filters are perfectly
disjoint. When they overlap -- a stale or oversized parent filter, or a
routing loop -- the sum over-counts and inflates the reported mesh size
to as much as several times the true size.
Estimate the cardinality of the OR-union of the contributing filters
(self + parent + children) once instead. OR is idempotent, so any
overlap is deduplicated: the result equals the old sum in the disjoint
case and stays correct under overlap. The union is seeded from a clone
of a contributing filter so it keeps that filter's size class, and a
filter whose size class does not match is skipped rather than panicking.
The refuse-to-estimate behavior on a saturated or above-cap filter is
preserved.
Add a regression test with overlapping parent and child filters where
the naive sum over-counts and the union estimate tracks the distinct
member count.
The transport layer used Mutex::lock().unwrap() at ten sites across the
UDP, BLE, and Ethernet code. A std mutex poisons if a thread panics
while holding it, after which every lock().unwrap() on that same mutex
also panics, turning one fault into a cascade. These critical sections
only perform short HashMap/Vec operations on locally constructed values
and are not reachable from peer input, but the idiom is fragile against
any future in-section panic. Replace each with
lock().unwrap_or_else(|e| e.into_inner()), which recovers the guarded
data and removes the cascade with no new dependency and no call-graph
change.
Also replace four self.local_addr.unwrap() calls in the UDP start and
adopt paths with a sentinel fallback. The value is provably set just
above each log line today, but the unwrap is brittle against a future
reordering; logging an unbound sentinel is harmless and cannot panic.
When two peers each auto_connect to the other, each runs both an
initiator and a responder NAT-traversal session and binds a separate
UDP socket per session. Each side adopts only the first Established
event and drops the loser session's socket; when the two sides adopt
mismatched sessions, each sends its Noise msg1 to a peer port the peer
has already stopped draining, and both handshakes stall.
Deterministically keep the session initiated by the smaller NodeAddr,
decided on the responder path: decline an incoming offer only when we
also have an in-flight outbound initiator for the same peer and our
NodeAddr is smaller. The peer's redundant initiator then times out,
leaving a single matching socket pair on both ends. Asymmetric
(one-sided) auto_connect has no co-active initiator and is never
suppressed, so connectivity is preserved; an undecidable NodeAddr falls
through to answering.
Reuses the NodeAddr tie-breaker convention already used by the
cross-connection and rekey dual-init paths. Adds a unit test for the
election helper.
Add a Loopback variant to TransportHandle backed by an unbounded
in-process channel and a shared address-to-receiver registry, so
node-level multi-node tests deliver packets directly between nodes
instead of over real localhost UDP sockets. This removes the kernel
UDP receive-buffer overflow that dropped handshake packets when many
tests ran in parallel under CPU contention, and lets the large-network
convergence tests run reliably in the default suite again (their
parallel-load ignore markers are removed).
The new transport and its enum variant are cfg(test)-gated, so the
daemon build is unaffected.
The FMP rekey msg1 resend driver retransmitted indefinitely with no cap
and no abandon, so a rekey that never completed kept resending msg1
forever. Give it a retransmission budget: cap resends at
handshake_max_resends with exponential backoff and abandon the rekey
cycle cleanly once the budget is exhausted, mirroring the FSP session
rekey msg3 driver.
With the cap in place the link-dead heartbeat can safely become
rekey-aware: check_link_heartbeats now suppresses teardown while a rekey
is in progress with msg1 budget remaining, instead of reaping a link
that is still actively carrying rekey-handshake traffic. The suppression
terminates deterministically (the budget abandons on exhaustion, cutover
clears the in-progress flag), so a genuinely dead link is still reaped on
the next cycle.
Adds a rekey_msg1_resend_count counter on ActivePeer reset at every
rekey-clear and cutover site, msg1 resend-budget unit tests, and two-node
heartbeat suppression/resume/regression integration tests.
Raise the in-process backpressure headroom in make_test_node_with_mtu
(request an 8 MiB recv_buf_size on UdpConfig and grow packet_channel from
256 to 8192) to reduce localhost-UDP receive overflow under parallel-CPU
scheduler contention, and mark the large-network convergence tests
#[ignore] so cargo test --lib stays green by default. The ignored tests
remain runnable on demand with --ignored or --test-threads=1.
Add per-direction pool_inbound/pool_outbound counters to TcpStats and
TorStats, updated at every pool-insert, receive-loop-exit, transport-stop,
and send-failure removal site. Compare the max_inbound_connections cap
against pool_inbound rather than the combined pool length, so outbound
connect-on-send connections no longer consume the operator-facing inbound
budget. The configuration field name and operator semantics are preserved;
only the cap-check comparison and accounting change.
Move the max_peers cap check in handle_msg1 forward, from the late
check inside promote_connection (which fires after Msg2 has already
been built and put on the wire) to an early position after identity
verification but before index allocation and the Msg2 send. When the
gate fires for a net-new identity, the Msg1 is silent-dropped — no
response goes back to the peer, no AEAD compute or wire bytes are
spent.
Bypass preserved for known peers (reconnect / cross-connection): if
the sender's NodeAddr is already in self.peers, or if a pending
outbound connection is in flight to the same identity, the gate is
skipped so legitimate maintenance traffic continues to work. The
late check inside promote_connection is intentionally retained as
defense-in-depth against future call sites or a disconnect racing
between the early-gate decision and promotion.
Wire-cost rationale: a 45 s tcpdump at saturation observed ~3.6
cap-denials/s steady-state, each previously paying the full Noise IK
responder crypto + Msg2 (~104 B) on the wire before being rejected.
The bigger value is cleaner peer-side semantics — the peer no longer
sees a fake-completed handshake whose data frames subsequently fail
decryption locally.
Two new unit tests cover the cases:
- handle_msg1_silent_drops_at_cap_for_new_peer drives a wire-pumped
Msg1 from a fresh identity into a saturated node and asserts no
Msg2 reaches the sender socket. Stash-verifies as FAIL on the
pre-fix tree (Msg2 hits the wire) and PASS post-fix.
- handle_msg1_admits_existing_peer_at_cap drives a Msg1 from an
identity already in self.peers and asserts the gate does not evict
it. This is a regression check (the no-gate tree behaves the same
way here, but the test guards against an accidental future gate
that breaks known-peer admit).
node.limits.max_peers was honored only on inbound msg1 admission
(handshake.rs handle_msg1 returns PeerLimitExceeded when peers.len
is at the cap). Four outbound initiation paths proceeded unconditionally
at capacity: auto-reconnect retries (process_pending_retries),
Nostr-mediated discovery's BootstrapEvent::Established adoption
(poll_nostr_discovery), NAT-traversal punch initiation (the outgoing
side of the offer/answer/punch sequence in the Nostr discovery
runtime), and NAT-traversal punch response (the incoming side of the
same sequence). A saturated node burned CPU, UDP probes, STUN
observations, and Nostr relay traffic on connections that the inbound
gate would reject the moment they reached msg1.
Introduce Node::outbound_admission_check (peers.len < max_peers, or
true when max_peers == 0 as the no-cap sentinel) and gate the four
paths. The discovery runtime lives in a separate task and does not
hold a Node reference; bridge via an Arc<AtomicBool> the runtime
reads and Node refreshes once per tick from outbound_admission_check.
The atomic granularity is intentionally loose: one-tick lag is
acceptable because the inbound msg1 gate continues to be the
authoritative cap, and in-flight handshakes started below the cap
are allowed to complete.
Inbound gate at handshake.rs is unchanged.
The mesh-size estimator's children loop relied on the cached
peer_declaration(parent_id).parent_id() != my_addr check to exclude
the parent. That cached view briefly disagrees with our own latest
my_declaration().parent_id() during the window between a local
parent-switch and the new parent's next inbound TreeAnnounce: the
peer-declaration cache still names us as the parent's parent, so the
parent is iterated as if it were a child and its (typically dominant)
bloom cardinality is added a second time. Symptom: estimated mesh size
displayed in fipsctl show status and fipstop nearly-but-not-exactly
doubles during tree rebalancing.
Make the invariant structural with an explicit peer_addr == parent_id
skip at the head of the children loop. Per-peer 500 ms rate-limiter
and overall recompute cadence are unchanged.
Adds a regression test that constructs the stale-peer-declaration
scenario directly and asserts the parent is not double-counted.
Closes the eventually-consistent gap in spanning-tree state
distribution. Every existing send_tree_announce_to_all call site
gates on a local state-change event (parent switch, self-root
promotion, ancestry change, peer promotion, parent loss). Once a
partition latches — for example a parent-switch announce stranded
in the brief cross-init handshake swap window, where the announce
arrives on a session-index whose decrypt-worker entry has been
unregistered — neither side's state changes again, so neither
side ever re-broadcasts. The existing 60 s check_periodic_parent_reeval
was a re-evaluation, not a re-broadcast: it short-circuited
silently on no-change. Production-side healing depended on
incidental link churn; lab harnesses with stable docker-bridge
links had no equivalent path.
Add a final else branch that fires send_tree_announce_to_all
unconditionally on the no-change path, alongside the existing
switch and self-promote arms. Receivers coalesce by sequence
comparison (ParentDeclaration::is_fresher_than) and short-circuit
at the `if !updated` gate in handle_tree_announce; same-sequence
repeats drop silently with no cascade. The per-peer 500 ms
rate-limiter is well below this 60 s cadence and does not suppress
the heartbeat broadcast.
The fix is a general protocol-robustness improvement: it addresses
any in-flight TreeAnnounce loss class, not only the specific
cross-init swap-window drop site.
testing/static/scripts/rekey-test.sh BASELINE_CONVERGENCE_TIMEOUT
60 -> 65 so a partition healed by the periodic broadcast at T+60
lands inside the convergence window. wait_for_full_baseline
early-exits on PASS, so successful reps see no extra wall-clock.
The tick body's per-peer check_* loops (heartbeats, bloom
announces, MMP reports, tree announces) called transport.send
for every active peer, which on TCP/Tor fell through to a 5 s
connect-on-send wait for any peer whose pool entry was not yet
established. That wedged the entire tick body for the full
connect_timeout_ms per unreachable peer; under post-restart
convergence on a high-peer mesh, this cascaded into multi-
second tick stalls. On master, the same mechanism also starved
the per-tick control-snapshot republish and pushed fipsctl
queries onto an mpsc fallback that was itself queued behind
the wedged rx_loop, producing the 5-second fipsctl head-of-line
pattern operators observed on loaded nodes.
Gate send_encrypted_link_message_with_ce on
transport.connection_state before the send: proceed only when
Connected; on None, kick off a non-blocking background connect
(idempotent — TransportHandle::connect dedupes against the
connecting pool and spawns the timeout-bounded TcpStream::connect
inside its own tokio task) and fail this send fast with a
clear "transport connection not ready" error. A subsequent
tick retries once the pool has an entry. The reconnect
lifecycle (check_link_heartbeats, process_pending_retries,
poll_pending_connects) is unchanged. The connect-on-send
branch in transport.send_async itself remains in place for
code paths that legitimately need synchronous connect (e.g.,
explicit operator-driven fipsctl connect).
When both peers' Nostr-mediated UDP punches complete within the
same scheduling window, each side's `BootstrapEvent::Established`
event arrives with `is_connecting_to_peer` already true: each side
received an inbound msg1 from the peer's pre-punch outbound
attempt, which created a connecting-state record. The deduplication
skip then fires on both sides, neither installs the fresh
traversal socket as canonical, and the peer-adoption budget
(45 s) expires. Cross-node wall-clock alignment of the skip log
line in observed failures was within ~1 ms — simultaneous dual-
fire under contention, the dual-initiation pattern.
Apply the deterministic NodeAddr tie-breaker already used at
`handlers/handshake.rs:269` for rekey dual-initiation and in
`peer::cross_connection_winner` for cross-connection resolution.
Smaller NodeAddr wins as adopter: enumerate the in-flight
connections whose `expected_identity` points at this peer, tear
them down via the canonical `cleanup_stale_connection` helper, and
fall through to `adopt_established_traversal`. Larger NodeAddr
loses and keeps the existing `continue` semantics; the loser's
in-flight outbound is reconciled by `handle_msg1`'s cross-
connection logic when the winner's fresh msg1 arrives over the
adopted socket.
`cleanup_stale_connection` visibility bumped from module-private
to `pub(in crate::node)` so it is callable from `lifecycle.rs`.
The defensive re-check inside `adopt_established_traversal`
itself is left as-is — after the outer cleanup the winner reaches
it with `is_connecting_to_peer == false`, so the inner skip
won't trip. The `BootstrapEvent::Failed` arm is unchanged: there
is no winning outcome on dual failure, and the existing skip +
retry-schedule semantics are correct.
An FSP session rekey could leave the two endpoints holding different
key sets for a brief window: if a handshake message was lost in
transit, one side rotated to the new keys while the other did not.
Traffic sealed in one key epoch then reached a peer still on the
other epoch and failed to decrypt, producing bursts of AEAD
decryption failures and dropped connectivity until a later rekey
cycle reconverged the pair. Choreographing the cutover order cannot
close this window: any fixed ordering still leaves a skew that
packet reordering widens.
Make rekey correctness independent of cutover timing by overlapping
the key epochs on the receive path. During a rekey transition the
receiver trial-decrypts each frame against every live session it
holds: current, the not-yet-promoted pending session, and the
draining previous session. The K-bit becomes a hint that orders the
trial-decrypt cascade rather than a hard gate, and a frame that
authenticates against the pending session is itself the cutover
signal. No rotation ordering and no packet reordering can then cause
a decryption failure.
The pre-rekey Noise session is held in the `previous` slot until the
peer has demonstrably moved off it. Its drain deadline is anchored
on the most recent frame the peer authenticated against that slot,
refreshed each time the trial-decrypt cascade lands there, rather
than on a fixed wall-clock timer started unilaterally at the local
cutover. A peer that never received the new keys keeps authenticating
against `previous` and the slot stays live; without this, a fixed
timer would erase the only key set that could decrypt the peer's
frames, producing a permanent silent decrypt failure on a live data
path. A peer that never catches up is handled by the existing FSP
session liveness path rather than by silent decrypt failure.
The lost-handshake liveness gap is closed separately by retransmitting
the third rekey handshake message until the peer is confirmed on the
new keys, with a bounded retry budget after which the rekey cycle is
cleanly abandoned and retried on the next timer.
Adds unit tests covering the trial-decrypt cascade (epoch selection,
promotion on pending decrypt, reordered old-epoch stragglers after
cutover, per-slot replay-window integrity), the msg3 retransmission
lifecycle, and the peer-progress-aware drain retirement.
Replaces the unconditional `CoordCache::clear()` calls at parent-switch,
become-root, and loop-detection sites with two targeted invalidation
methods scoped to what actually makes an entry stale:
- `invalidate_via_node(node_addr)`: drop entries whose cached
destination ancestry contains `node_addr`. Used at parent-position-
change sites — our prefix changed, so destinations downstream of
us have stale-prefix coords.
- `invalidate_other_roots(current_root)`: drop entries rooted under
a different root than the current one. Used at root-change sites.
Under the previous global flush, parent switches blanked the cache
across the board, leaving `find_next_hop` returning `None` for every
non-direct-peer destination until the cache passively re-warmed via
incoming TreeAnnounces / SessionSetup. Surgical invalidation
preserves entries that remain correct after the topology change.
The cached coord describes a destination's tree position; that
position only goes stale relative to our own routing decisions when
our own prefix changes (entries we are downstream of) or the root
changes (entries in a different tree). Peer removal does not
invalidate cached coords: `Node::find_next_hop` recomputes the
next-hop decision on every call against the current peer set, bloom
filters, and tree state, and Discovery already triggers on
`no route to destination` errors when a destination becomes
unroutable through us. The peer-removal site retains the original
"no cache invalidation" behavior.
Each method returns the count of entries removed for observability.
Unit tests cover each method against the cases enumerated in the
acceptance criterion.
Skip BootstrapEvent::Established and BootstrapEvent::Failed dispatch
in poll_nostr_discovery for peers that are already connected or
actively handshaking. Without these guards, stale traversal events
arriving after a peer connected through a different path would
either attempt to adopt a redundant socket against the live
connection (Established) or poison the per-peer failure-state
cooldown and trigger redundant retraversal via schedule_retry /
try_peer_addresses (Failed).
The four guard sites use a new is_connecting_to_peer helper extracted
from the existing closure inside initiate_peer_connection; the helper
checks for an in-flight outbound handshake state. adopt_established_traversal
gains a defense-in-depth check returning PeerAlreadyExists when called
against an already-promoted peer, so the invariant holds if a future
caller bypasses the outer dispatch guard.
Side benefit: narrows a cooldown-poisoning vector previously available
to an attacker injecting stale failure events for an active peer.
Test coverage for the new behavior:
- test_try_peer_addresses_skips_connected_peer
- test_try_peer_addresses_skips_connecting_peer
- test_nostr_traversal_failure_skips_connected_peer (Failed-arm event
injection)
- test_nostr_traversal_established_skips_connected_peer (Established-arm
event injection, mirror of the Failed test)
- test_adopted_traversal_skips_already_connected_peer
(adopt_established_traversal defense-in-depth)
CHANGELOG entry under [Unreleased] / Fixed.
Closes#87
Add a per-session signed jitter offset (uniform [-15, +15] seconds)
to the rekey timer triggers in check_rekey (FMP) and check_session_rekey
(FSP). The configured `node.rekey.after_secs` becomes the nominal
interval rather than a floor; mean is preserved. Desynchronizes
both endpoints in symmetric-start meshes so the dual-initiation
race stops occurring rather than being resolved after the fact by
the smaller-NodeAddr tie-breaker.
Per-session storage means each rekey cutover reconstructs the
session and redraws the jitter naturally — successive cycles get
independent offsets, preventing drift back into sync.
Operator-facing IPs in user-visible configs/docs (examples, tutorials,
packaging, sidecar templates) are now the resolvable hostnames of the
public test fleet (test-us01.fips.network, etc.) so they keep working
without baking specific addresses into examples.
Doc-comment and test fixtures in src/config/transport.rs use RFC 5737
TEST-NET-2 (198.51.100.1) so they cannot accidentally point at a real
host.
Also resyncs the openwrt-ipk fips.yaml with the common reference
(merge from master) and applies the same DNS-name swap there.
The chacha20 crate (RustCrypto) ships SSE2 + soft backends only — on
aarch64 (Apple Silicon, ARM Linux servers, Docker on M-series Macs) it
falls through to a portable software impl at ~600–800 MB/s/core. ring
0.17 wraps BoringSSL's hand-tuned ChaCha20-Poly1305, which dispatches
to NEON on aarch64 and AVX2/AVX-512 on x86_64 — typically 3-5 GB/s/core
on the same hardware.
Same wire format. ChaCha20-Poly1305 is byte-deterministic for a given
(key, nonce, plaintext, aad), so any correct AEAD implementation
produces identical ciphertext. The full noise test suite covers this
implicitly: IK and XK roundtrip handshakes, replay window correctness,
multi-message nonce sequencing, and 100-message stress all pass at
1129/1129 (the lib's full `cargo test` count) — these only succeed if
ring's output matches what the receiver's existing replay-window
decrypt path expects.
Implementation notes:
* `LessSafeKey` (and `UnboundKey`) deliberately do not implement
Clone for safety. `CipherState`'s manual Clone impl rebuilds it
from the retained 32-byte key — cheap for ChaCha20-Poly1305 since
construction is essentially a key copy + a constant-time check.
* The keyed AEAD is now cached in `CipherState.cipher` instead of
being re-derived per packet. This was already a perf win for the
chacha20poly1305 backend (`new_from_slice` per packet was hot in
profiles); for ring it's a bigger win because `LessSafeKey`
construction also derives the Poly1305 key.
* Public `Vec<u8>`-returning API preserved. New module-private
`seal`/`open` helpers wrap ring's `seal_in_place_append_tag` /
`open_in_place` so the per-packet allocation pattern is local to
one place.
* `EndToEndState::Established` triggers `clippy::large_enum_variant`
after the swap (`NoiseSession` grew from ~600 to ~1.5 KB because
ring precomputes the Poly1305 key state at construction). That
precomputation is the win — boxing the variant would re-add an
indirection per packet and work against it. `#[allow]`'d at the
enum decl with a justifying comment.
ring is widely deployed (rustls, hyper-rustls, AWS SDK, …) and a
pure-Rust crate (uses BoringSSL's asm via a vendored build). It
introduces no new C toolchain requirements that aren't already there
for any rustls user.
Bench data from a downstream consumer of this crate (Docker e2e,
DURATION=10, identical hardware before/after, aarch64 Linux on
Apple Silicon):
2-node direct (A↔B):
TCP 1-stream 437 → 1097 Mbps (2.51×)
TCP 4-stream 439 → 1109 Mbps (2.53×)
TCP 8-stream 445 → 1069 Mbps (2.40×)
UDP @1000 Mbit 599/40% loss → 1000 Mbps lossless
ping under load ~0.6 ms (unchanged)
3-node forced transit (A → C → B):
TCP 1-stream 438 → 1019 Mbps (2.33×)
TCP 4-stream 421 → 982 Mbps (2.33×)
TCP 8-stream 443 → 1031 Mbps (2.33×)
UDP @1000 Mbit 475/52% loss → 1000 Mbps lossless
ping under load 7.68 ms / 215 ms max → 0.72 ms / 3.6 ms max
The relay-path lift is the cleanest tell on the bottleneck: the
transit node was crypto-bound (single-threaded soft chacha couldn't
keep up with offered rate), so the queue accumulated under load. With
NEON the relay isn't crypto-bound and the queue stops accumulating —
the 215ms ping-tail collapses to 3.6ms.
Freshly-restarted nodes with policy: open silently lost the historical
event replay that relays send in response to subscribe(). The
broadcast::Receiver was created INSIDE spawn_notify_loop, which the
tokio runtime starts at some indeterminate point after subscribe()
returns. tokio's broadcast channel only delivers messages sent after
the receiver is created; messages dispatched in the gap between
subscribe() issuing the REQ and the spawned task calling
client.notifications() were dropped by external_notification_sender.send
returning Err(SendError) with no subscribers attached.
Symptom on a node with policy: open: non-configured peers were not
discovered until they next re-published their advert (default
advert_refresh_secs = 1800s = 30 min). Configured peers were unaffected
because fetch_advert (relay-fetch path) caches them at startup-sweep
time. The bug has been latent since 34e00b9 added Nostr discovery —
relay-fetch covered the common case for configured-peer setups.
Fix: create the broadcast::Receiver in start() before subscribe() and
pass it into spawn_notify_loop. The receiver now exists when the REQ
replay arrives, so historical events flow through the cache path.
Also handle broadcast::error::RecvError::Lagged separately from
::Closed. The previous `while let Ok(...) = recv().await` exited the
loop on any Err, so a single lag event would silently kill the entire
subscription consumer with no recovery. Lagged now logs a warn (with
the skipped count) and continues; only Closed exits the loop.
Add two info-level log lines for in-field observability of the loop's
liveness. "nostr notify loop entered" fires once at task start; "nostr
notify loop received first event" fires once after the first
successful recv() with elapsed_ms since loop entry. Together these
turn the previous silent-failure shape (zero advert: peer cached
log lines indistinguishable between dead loop and idle channel) into
an immediately greppable startup signal — operators can confirm the
loop is alive and see how long it took to receive its first event,
catching any future regression in the subscription codepath in
seconds rather than waiting one advert_refresh_secs interval.
No public API change; the test fixture (NostrDiscovery::new_for_test)
does not call spawn_notify_loop and is unaffected.
The advert cache inside fetch_advert is read-only on hit — once a peer's
overlay advert is cached, every subsequent lookup returns the same
endpoints regardless of whether they still work. So when a peer rebinds
its NAT (or its STUN-discovered port flaps), connection retries to that
peer dial the same dead address forever, even with exponential backoff
firing at the right cadence.
Observed in deployment: macOS daemon's view of a Linux peer would
"regress" — peer marked rch=False after a brief link-dead window, then
hours of "Retry connection initiation failed: no operational transport
for any of <npub>'s addresses" with no recovery. Manual pause+resume of
the daemon (which restarts the FIPS endpoint and forces fresh advert
fetches) was the only way out.
When initiate_peer_connection / a retry tick returns
NodeError::NoTransportForType, fire-and-forget refetch_advert_for_stale_check
on the peer's npub. This re-fetches kind 37195 from advert_relays; if
the relay has a newer advert it replaces the cached entry, if it has
nothing it evicts the cached entry. Either way the next retry tick goes
to fresh data instead of looping on the same dead endpoint.
Mirrors the existing stale-advert sweep that runs from the
BootstrapEvent::Failed (NAT-traversal-streak) path, but covers the
direct-UDP-retry path which never crosses that streak threshold.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
When initiate_peer_connections() runs at boot, address resolution can
fail for an entire peer (no operational transport for the configured
transport types, all addresses unreachable, NAT rebind invalidated cached
endpoints, etc.). Before this change the failure was logged and silently
forgotten — the peer entry stayed in a dead state forever, accepting
incoming pings but unable to answer them, until the daemon was manually
restarted.
The retry plumbing (schedule_retry / process_pending_retries with
exponential backoff) already exists and is wired into the post-handshake
failure paths (BootstrapEvent::Failed, MMP dead-link timeout, handshake
timeout). The startup loop just wasn't calling it. Mirror the
BootstrapEvent::Failed path: on a startup peer-init error, parse the
peer's npub and call schedule_retry so the peer recovers without
operator intervention.
Includes a regression test that asserts retry_pending is populated when
initiate_peer_connections() fails for a peer with no operational
transport.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
The previous fix (6ebca3e) only refetched the advert when retry returned
NodeError::NoTransportForType (cache returned no addresses at all). But
the much more common stale-cache failure mode is: cache returns an
endpoint that LOOKS valid (the address it had last week, before the
peer's NAT rebound), the dial succeeds at the IP layer, the handshake
times out, MMP fires, schedule_reconnect adds the entry back to
retry_pending, next retry hits the same cached endpoint, dials it
again, times out again. Loop forever — no NoTransportForType ever
fires because the cache has data, just dead data.
Move refetch_advert_for_stale_check to before each retry attempt
unconditionally. Cheap (one Filter query against advert_relays with
a 2s timeout, bounded by the retry backoff cadence), and replaces the
cache only if the relay has a newer advert or evicts if the relay has
nothing. Keeps the retry loop pinned to relay ground truth instead of
whatever the cache happened to learn at startup.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
When a node was the smallest-NodeAddr peer it could see (no smaller
neighbor available as a parent), the spanning-tree state was promoting
it to root. But the ancestry it advertised on the next TreeAnnounce
still referenced its previous parent's path, so receiving peers
rejected the announce with `invalid ancestry: advertised root X is
not the minimum path entry Y`, blocking mesh transit on any path that
needed to traverse this node.
Detect the self-root transition explicitly in `TreeState::become_root`
and rebuild the advertised ancestry to start from self. Also surface
the same path through the MMP receive handler so a stale ancestry
inherited across reconnect is corrected eagerly rather than waiting
for the next observation tick.
Adds 80 unit tests in `tree::tests` covering self-root transitions,
mid-chain ancestor disappearance, and ancestry validation against the
new root, plus a regression in `node::tests::spanning_tree` for a
3-node chain where the middle node's only parent (the smallest-addr
peer) goes away — previously it would advertise an ancestry rejected
by both endpoints; now it self-roots cleanly.
The run_rx_loop's `tokio::select!` was costing one full scheduler hop
+ futex per inbound packet and per outbound TUN packet. Under
sustained load that capped throughput at one event per scheduler
quantum — independent of CPU (which sat near-idle) because every
iteration parked the worker, woke it via futex, processed one event,
then parked again.
After the await on `packet_rx.recv()` / `tun_outbound_rx.recv()`
fires, drain up to 256 additional ready items via `try_recv()` in a
tight inner loop before yielding back to `select!`. `biased` ordering
gives the data-plane branches priority over tick / control / DNS
under sustained load.
The 256 cap is empirically tuned to keep the worker on a busy stream
between yield points (a contiguous burst of ~256 MTU-sized packets
≈ 400 KB of contiguous traffic) while still bounding the inner loop
so a flood on one branch can't starve the periodic tick or control
socket. Lower caps (64) left perf on the table; higher caps (1024+)
delayed tick handling visibly under stress.
Pairs with the recvmmsg(2) change in the previous commit: the kernel
UDP queue now hands packets to `packet_rx` in 32-batches, and the
rx_loop drains them without a per-packet scheduler hop.
The UDP recv loop drained the kernel queue one packet per recvmsg(2).
Each call paid full per-syscall + per-task-wakeup overhead (~50us avg
including a futex-based scheduler hop), so under sustained load the
loop ran at one rx event per scheduler quantum — the dominant cap on
inbound packet rate.
On Linux, switch the steady-state path to recvmmsg(2) with a 32-packet
batch. A single readable() wakeup drains up to 32 datagrams in one
syscall before yielding back to the reactor. Stack-allocated mmsghdr
arrays sized to a module-level `BATCH_SIZE` constant.
`SO_RXQ_OVFL` is sampled once per batch off the cmsg chain of `msgs[0]`
and plumbed through `AsyncUdpSocket::recv_batch` as `(count, drops)`.
The counter is socket-wide and monotonic, so a single sample per batch
gives the 1Hz `sample_transport_congestion()` detector ample fresh
values under load (one batch = up to 32 datagrams). Cost is one
stack-allocated CMSG_SPACE(4) buffer + one CMSG_FIRSTHDR walk per
batch syscall.
macOS / Windows fall through to the per-packet recv_from loop —
recvmmsg is Linux-specific and the per-packet API is fast enough on
those platforms for now (recvmsg_x for Darwin can be added later).
The slice-array build also drops the `MaybeUninit::uninit().assume_init()`
+ `transmute` pair for `std::array::from_fn` over a single shared
`backing.iter_mut()` — same disjoint mutable borrows, no `unsafe`.
`PeerIdentity::pubkey_full()` falls through to
`self.pubkey.public_key(Parity::Even)` whenever the parity-aware full
key wasn't passed at construction (i.e. for every peer constructed
from an npub or x-only key). Underneath, that runs a secp256k1 EC
point parse — `fe_sqrt` + `fe_mul` + `ge_set_xo_var` — which is ~6%
of per-packet CPU on the bulk-data send path for a value that never
changes after construction.
Compute it eagerly. The same EC point parse already runs at
construction inside `NodeAddr::from_pubkey`, so the cost is paid once
where it would be paid anyway.
`Node::adopt_established_traversal` was constructing the adopted UDP
transport with `UdpConfig::default()` — MTU 1280, default recv/send
buffer sizes, default accept/advertise flags. If the operator had
configured a higher MTU on the primary `[transports.udp]` listener
(e.g. 1500 on a path where larger frames are known viable), full-sized
tunnel datagrams sent over the NAT-traversed link would exceed the
adopted socket's MTU and get dropped at the socket layer with no
visibility into why throughput collapsed.
Inherit the primary UDP config (MTU + recv/send buffer sizes + accept
/ advertise flags) and clear the bind / external-address fields since
the adopted socket is already bound. Lookup tries `transport_name`
first so operators with multiple named `[transports.udp.<name>]`
listeners pick up inheritance from the matching listener, and falls
back to the unnamed `Single` listener so single-instance configs work
unchanged.
The previous default of MTU 1280 was deliberately the IPv6 minimum,
the only value guaranteed to survive arbitrary middlebox paths. With
this change, operators who set their primary listener higher (based
on known-clean LAN topology) will have NAT-traversed flows initially
attempting that higher MTU and possibly black-holing on tighter paths
until reactive `MtuExceeded` recovery kicks in. Documented in the
adoption call-site comment so future readers understand why the
conservative default went away.
Discovered in a downstream consumer where a `MESH_TUNNEL_MTU=1320` /
encrypted wire ~1426B produced silent packet drop on every session
that had been promoted onto a NAT-traversed link.
Adds two sibling tests in `src/node/tests/bootstrap.rs` pinning the
new behaviour for the `Single` and `Named` config variants.
Surfaces local services reachable from the mesh, paired with their
current `inet fips` baseline filter classification. Lands to the
right of the existing TUN section in the Traffic block.
A new daemon control query `show_listening_sockets` returns IPv6
listeners bound to either `::` (wildcard) or the node's fd00::/8
address, each classified as Accept / Drop / Unknown / NoFirewall
against the running inbound chain. fipstop renders the result as a
table beside the Traffic counters: Accept rows in default White,
Drop / Unknown in DarkGray, a yellow banner above the table when
`fips-firewall.service` is inactive, and a trailing `*` on
wildcard binds to remind the operator the bind is not
fips0-specific.
Daemon side:
- `src/control/listening.rs` walks `/proc/net/tcp6` and
`/proc/net/udp6` via the procfs crate (LISTEN state for TCP,
wildcard remote for UDP), filters to fips0-reachable binds, and
resolves inodes to PID / comm via `/proc/<pid>/fd`.
- `src/control/firewall_state.rs` shells out to
`nft -j list table inet fips` and walks the inbound chain.
Recognises canonical accepts (`tcp/udp dport N accept`,
`dport { ... } accept`, `dport A-B accept`), the iifname-scoping
line, conntrack and icmpv6 lines (skipped). Any rule with
unrecognised matchers (saddr filters, jumps, daddr filters) or
non-terminal verdicts forces Unknown classification for the
ports it references. Eleven unit tests cover the classification
logic; the listening enumerator carries a /proc-parsing test of
its own.
- `show_listening_sockets` emits
`{fips0_addr, firewall_active, sockets[]}` with per-row
`{proto, local_addr, port, pid, process, filter, wildcard_bind}`.
fipstop side:
- `src/bin/fipstop/ui/dashboard.rs` splits the Traffic block into
a 50/50 horizontal layout; the existing TUN + Forwarded panel
occupies the left half.
- `src/bin/fipstop/ui/listening.rs` renders the right half.
- `main.rs` fetches the new query each tick when the Node tab is
active. Errors are non-fatal: an old daemon without the query
leaves the payload at None and the panel renders "loading...".
`Cargo.toml` gains `procfs = "0.18"` on the Linux target. IPv4
listeners are not enumerated — fips0 is IPv6-only.
Folded in: revert the default-socket lookup from writability-probe
back to existence-based selection. The previous tempfile-probe on
`/run/fips` silently steered fipstop / fipsctl onto an XDG path
the daemon never bound for any user in the `fips` group whose
shell session had not yet picked up the supplementary group (no
re-login after `usermod -aG`). `XDG_RUNTIME_DIR` is set on every
modern systemd-managed user session, so this hit the common case.
The kernel checks actual group membership at `connect(2)`, so a
user who genuinely cannot connect now gets a clear `EACCES`
rather than a silent path mismatch. Drops the now-unused
`is_writable_dir` helper. `XDG_RUNTIME_DIR` existence validation
is preserved.
Documentation:
- `docs/reference/cli-fipstop.md` — Node-tab row updated, new
"Listening on fips0 panel" section.
- `docs/reference/control-socket.md` — `show_listening_sockets`
added to the read-only queries table.
- `docs/how-to/enable-mesh-firewall.md` — new "Verify with
fipstop" section.
- `docs/tutorials/host-a-service.md` — fipstop callouts at
Steps 3, 5, 6 + Troubleshooting bullet + wildcard-bind reminder
under "What you've learned".
- `CHANGELOG.md` — new bullet under `Added / Operator Tooling`,
resolver `Fixed` entry rewritten to describe the
existence-based final shape.
Both variants of SessionMessageType were never emitted anywhere
in src/, and the production from_byte dispatch sites lacked
Some-arms for them — any 0x00/0x01 byte that reached either
dispatcher would log "Unknown..." and drop. The matching rustdoc
tables described an Offset 0 msg_type byte that the encode() path
has never written; the actual wire format is the FSP common prefix
[ver_phase][flags][payload_len:2 LE] with body keyed by phase
nibble, as documented in docs/reference/wire-formats.md.
Drop the variants, drop their from_byte/to_byte/Display arms, fix
the two stale rustdoc tables to describe the real wire shape, and
trim the variant-iteration unit test that enumerated them.
Zero on-wire behaviour change.
The Ethernet data frame format is `[type:1][length:2 LE][payload]`,
so the per-link payload MTU is the interface MTU minus 3 bytes,
not minus 1. The 2-byte length field is required to trim NIC
minimum-frame padding before AEAD verification.
The implementation in src/transport/ethernet/mod.rs already uses
saturating_sub(3) correctly; only the rustdoc on the effective_mtu
field and the EthernetConfig.mtu field's documentation lagged behind.
No behaviour change.
The gateway is designed for systems already serving DHCP and DNS to
a LAN segment (canonically an OpenWrt AP). On those systems port 53
is already taken by the existing resolver, so the prior `[::]:53`
default conflicted with the gateway's intended deployment target out
of the box.
The OpenWrt ipk previously overrode this in its packaged config as a
workaround; matching the source default to what the canonical
deployment actually wants makes the override redundant and removes a
foot-gun for fresh manual Linux-host installs. The redundant
`dns.listen` line in `packaging/openwrt-ipk/files/etc/fips/fips.yaml`
is dropped along with this change.
Operators on a host without a pre-existing resolver on port 53 can
opt back into the wildcard bind by setting `dns.listen: "[::]:53"`
explicitly. The new default binds IPv6 loopback only — Linux IPv6
sockets bound to explicit `::1` do not accept v4-mapped traffic, so
forwarders that reach the gateway over IPv4 loopback need to be
pointed at an explicit IPv4 listen address instead.
Touches the gateway config struct and its default-value test, the
commented-out gateway example in the Debian common fips.yaml, the
OpenWrt ipk config (override removed), the gateway reference /
how-to / design / tutorial / troubleshoot docs, and a CHANGELOG
entry under [Unreleased] -> Changed.
Daemon and client tools previously evaluated the same three locations
(`/run/fips`, `XDG_RUNTIME_DIR`, `/tmp`) in different orders, allowing
fipsctl/fipstop to connect to a socket the daemon never bound when
neither side set `node.control.socket_path` explicitly.
Collapse the three call sites (`default_control_path`,
`default_gateway_path`, `ControlConfig::default_socket_path`) into a
shared `resolve_default_socket` helper. Canonical order is
`/run/fips` -> `$XDG_RUNTIME_DIR/fips/` -> `/tmp/fips-<name>`. Two
hardening fixes folded in: writability is probed via tempfile create
rather than mode bits (ACL- and group-aware), and `XDG_RUNTIME_DIR`
is validated as an existing directory before being used (avoids
stale post-logout values).
The deployed fleet is unaffected -- packaged configs set
`node.control.socket_path` explicitly. The fix surfaces for dev
runs and the binary-install getting-started path.
A leaf node's my_coords could go stale after an upstream
mid-chain ancestor swap, leaving non-parent destinations with
100% loss until either the parent or the depth also changed.
The broadcast gate in handle_tree_announce's
`else if !is_root && parent_id == from` branch compared only
(root, depth). A swap that altered an interior ancestor without
changing root or depth (e.g. A->B->C reorganizing to A->D->C
while keeping (A, depth=2)) was silently dropped one hop below
the swap node. Downstream nodes' coords paths then drifted from
the real tree topology, defeating greedy distance routing for
any destination whose path crossed the unrepresented section.
Widen the gate to compare the full my_coords.node_addrs() so
mid-chain swaps propagate to leaves the same way root/depth
changes already did. The gate body's bloom-marking is adjusted
in step so the wider gate doesn't generate empty/redundant
FilterAnnounces to every peer on every mid-chain swap
propagation: mark_changed_peers replaces
mark_all_updates_needed in the gate body (parent_id is
unchanged in this branch, so outgoing filter content is
typically unchanged, and mark_changed_peers correctly marks
zero peers in that case), and the unconditional
mark_update_needed(*from) at the top of handle_tree_announce
is removed (bloom exchange initiation is already handled at
handshake completion, and ongoing content changes are picked
up naturally by mark_changed_peers in handle_filter_announce
when peers send their next filter).
Required surface change: peer_inbound_filters in
src/node/bloom.rs upgraded from private to pub(super) so the
gate body can call it.
Verified in a 6-node depth-4 docker reproduction under
tc/netem-induced parent flapping: a depth-4 leaf's
ancestry_changed counter advances with upstream parent
switches while bloom_sent stays at zero matching the
pre-change steady-state baseline.
Open-discovery NAT traversal succeeds at the UDP layer regardless
of what FMP-protocol version the peer speaks. When the daemon
discovers a peer running a different FMP version (e.g. a v0/v1 mix
during a mid-rollout window, or a misconfigured peer in the same
advert namespace), the punch sequence completes, the socket is
adopted via `Node::adopt_established_traversal`, and we initiate
an FMP handshake. The peer drops our msg1 at its own version-gate
and we drop their msg1/msg2 at `Unknown FMP version, dropping`.
Neither side advances the handshake.
Today the bootstrap transport sits idle until the 31s stale-
handshake timeout, drops, and the open-discovery sweep ~30s later
fires the full STUN+offer+answer+punch sequence again — every
minute, indefinitely, against peers the handshake literally cannot
complete with.
Add a `Node::bootstrap_transport_npubs` map populated alongside
`bootstrap_transports` at adopt time. The rx loop reverse-maps the
transport_id → npub on version-mismatch and bumps the discovery
layer's `failure_state` to a long structural cooldown via the new
`NostrDiscovery::record_protocol_mismatch` API. The next sweep
skips the npub for `protocol_mismatch_cooldown_secs` (default
86400 = 24h, separate from the 30-min transient-failure
`extended_cooldown_secs`).
One-shot WARN per fresh observation. Repeat mismatches inside the
cooldown window are silent (the failure_state method returns false
when an existing comparable cooldown is already in place). The
handshake/transport teardown chain is unchanged — the fix is
specifically about preventing the *next* sweep cycle from
re-traversing.
Cleared on `cleanup_bootstrap_transport_if_unused` and on the
adopt-failure rollback path so completed handshakes don't leave
stale entries behind.
Four new unit tests in `failure_state.rs` cover fresh-entry
signaling, repeat-suppression inside the window, streak-pin
behavior for `show_peers` rendering, and post-cooldown re-arming.
The TUN-side TCP MSS clamp consults `path_mtu_lookup` (FipsAddress-
keyed) when sizing outbound TCP flows. Until now, only the reactive
`MtuExceeded` handler mirrored the bottleneck MTU into that store;
the proactive end-to-end `PathMtuNotification` echoed by the
destination updated only `MmpSessionState.path_mtu`, leaving the TUN
mirror stale.
On stable long-lived paths, the proactive echo can tighten the
session-canonical MTU well before any transit router fires a
`MtuExceeded` for those flows (since all current traffic is already
sized by the tighter session value). New TCP flows opened during
that window get clamped by the discovery-time value rather than the
session-canonical one, leading to PMTU-D loss until the reactive
path eventually fires.
Mirror the post-apply MTU into `path_mtu_lookup` whenever
`apply_notification` returns true, with the same tighter-only
semantics as the reactive mirror — never loosen the clamp. Gated on
the bool return so spurious writes don't happen on rejected
increases or no-op same-value notifications.
Four new unit tests exercise the empty-lookup write, tighten-
existing, keep-tighter-existing, and no-session-no-op paths,
parallel to the existing reactive-mirror test trio.
When a UDP transport had `advertise_on_nostr: true` + `public: true`
+ `bind_addr: 0.0.0.0:NNNN`, the advert builder previously read the
kernel's `local_addr()`, found `0.0.0.0`, filtered it out (correctly
— wildcard isn't a valid advertised endpoint), and silently emitted
no UDP endpoint in the published advert. Operators on AWS EIP / GCP
/ Azure setups (where binding to the public IP directly is impossible
because 1:1 NAT does the address translation off-host) had no way to
advertise UDP without binding to a specific local IP — and no log
explaining what was happening. TCP had the same shape, with no
`public: true` precondition.
Three pieces, layered. UDP gets zero-config autodiscovery via STUN;
both UDP and TCP get an explicit operator-supplied override; the
fall-through path now logs loudly instead of silently skipping.
UDP public-IP autodiscovery (STUN)
----------------------------------
In the UDP `is_public()` + wildcard-bind branch, run a one-shot
STUN observation against an ephemeral UDP socket on the daemon's
configured `stun_servers`. Take the reflexive IPv4 (the
STUN-reported port is the ephemeral source port and is discarded),
combine with the configured listener port for the advert
(`udp:<reflexive-ip>:<port>`). Works on AWS EIP / GCP / Azure
1:1-NAT setups because STUN sees the public-Internet egress IP and
the bind port is preserved through 1:1 NAT.
Result is cached per-transport on a new `public_udp_addr_cache`
field on `NostrDiscovery` (keyed by `TransportId.as_u32()`).
Asymmetric cache TTL: a successful observation is cached for
`advert_refresh_secs` (default 30 min) so we don't STUN every
refresh tick. A failed observation is cached for only 60s
(`PUBLIC_UDP_ADDR_FAILURE_TTL`) so a transient STUN flake at
startup retries within ~a minute and the advert grows its UDP
endpoint as soon as STUN starts working — rather than waiting the
full 30-min cycle.
The shared `observe_traversal_addresses` STUN helper had a
hard-coded 2s per-server response wait, right for the
per-traversal flow (latency-sensitive — 3 STUN servers worst-case
= 6s) but too short for the one-shot advert-publish startup
discovery. Parameterized `per_server_timeout` on the helper, with
two named constants in `stun.rs`: `TRAVERSAL_STUN_TIMEOUT = 2s`
(existing call sites) and `ADVERT_STUN_TIMEOUT = 5s` (new
public-UDP discovery path). Both use `tokio::time::timeout_at`
under the hood, so success returns immediately — the timeout is
only the worst case.
`external_addr` override (UDP + TCP)
------------------------------------
New `external_addr: Option<String>` field on `transports.udp.*`
and `transports.tcp.*` for explicit advertise-as override. Takes
precedence over both the bound `local_addr` and (for UDP) the
STUN-derived autodiscovery.
Required for TCP on cloud-NAT setups (AWS EIP, GCP/Azure external
IPs) where binding to the public IP directly fails with
`EADDRNOTAVAIL` because the public IP isn't on a host interface —
the network fabric does 1:1 NAT off-host. Without this field the
operator's only TCP path was "leave advert off" or "find a way to
make the public IP locally bindable."
For UDP, `external_addr` is optional but useful as a deterministic
alternative to STUN. Operators who want to skip STUN egress, whose
STUN servers are blocked, or who want the daemon to not depend on
external services for advert content can specify it explicitly.
The accessor parses two shapes:
- Bare IP (`"54.183.70.180"` or `"2001:db8::1"`): combines with
the configured `bind_addr` port.
- Full host:port (`"54.183.70.180:8443"` or `"[2001:db8::1]:443"`):
used verbatim — useful for port-forward setups where the
externally-visible port differs from the bind port.
Final precedence in `Node::build_overlay_advert` (now async, only
caller `refresh_overlay_advert` was already async):
- UDP: `external_addr` → non-wildcard `local_addr` → STUN → loud warn
- TCP: `external_addr` → non-wildcard `local_addr` → loud warn
Loud warns instead of silent skips
----------------------------------
The wildcard-bind fall-through paths now log a `warn!` pointing at
the operator-side fixes:
- UDP: "set transports.udp.external_addr, bind to a specific
public IP, or ensure node.discovery.nostr.stun_servers is
reachable"
- TCP: "Either set external_addr to the public IP (recommended for
cloud 1:1-NAT setups) or bind explicitly to the public IP"
Replaces the silent skip that previously cost operators a
debugging session when the advert mysteriously contained only the
Tor onion endpoint.
Tests
-----
11 new unit tests in `src/config/transport.rs` covering the
parser (IPv4/IPv6, bare/full, malformed) and the accessor (UDP
with default bind, UDP with explicit port override, UDP unset,
TCP without bind_addr, TCP with bind_addr, TCP with full
socket-addr override, parse_bind_port for IPv4/IPv6/malformed).
The 38-test nostr suite still passes.
CHANGELOG entries under `[Unreleased]` Fixed.
Public-test daemons with populous open-discovery caches generate
sustained NAT-traversal-failure WARN volume (~140/hour, ~3500/day)
against cache-learned peers that have gone offline — their adverts
are absent from major Nostr relays but cached entries persist until
their advertised `valid_until` expires. The daemon kept publishing
offers indefinitely under exponential backoff with no per-peer
suppression, drowning operator signal and hammering relays. A
parallel concern: the strict freshness check at signal.rs silently
rejected offers under modest clock skew (now_ms() anchors to
SystemTime once at startup, so post-startup NTP step adjustments
don't propagate on long-uptime daemons), indistinguishable from
"peer is offline."
Six independent improvements layered on the existing retry logic.
Per-npub WARN log rate-limit
----------------------------
New `FailureState` struct on `NostrDiscovery` records per-npub
`last_warn_at_ms`. Subsequent failures inside `warn_log_interval_secs`
(default 5 min) emit DEBUG instead of WARN. Each WARN now also
carries `consecutive_failures` and remaining `cooldown_secs` so
operators can read the trajectory without grepping multiple lines.
Per-npub consecutive-failure counter + extended cooldown
--------------------------------------------------------
After `failure_streak_threshold` (default 5) consecutive failures
against a peer, the next `extended_cooldown_secs` (default 1800)
of attempts are suppressed by pushing
`retry_pending[npub].retry_after_ms` past the cooldown wall. The
open-discovery sweep also consults `cooldown_until` and increments
a new `skipped_cooldown` counter so a peer whose `retry_pending`
was cleared by max_retries doesn't get re-enqueued during the
cooldown window. Caps offer-publish rate per dead peer regardless
of how often the sweep tries to re-enqueue.
Stale-advert eviction on streak-threshold transition
----------------------------------------------------
On the threshold-crossing transition (one-shot, not every
subsequent failure), `tokio::spawn` an active re-fetch of the
peer's Kind 37195 advert from `advert_relays`. Three outcomes:
- absent on relays → cache evicted; sweep won't re-enqueue
(peer is genuinely gone).
- newer `created_at` → cache refreshed + streak reset
(peer republished; allowed to retry immediately).
- same → cache untouched; cooldown stands.
Cost: ~one fetch per dead peer per 30-min cooldown cycle, vs
hundreds of offer publishes/hour today.
Clock-skew tolerance on freshness check
---------------------------------------
`signal.rs` `validate_offer_freshness` and
`validate_traversal_answer_for_offer` now allow ±60s grace beyond
strict TTL. Both return a new `FreshnessOutcome` enum so callers
can DEBUG-log when an offer/answer was only accepted via the grace
window. `FRESHNESS_SKEW_TOLERANCE_MS` is hard-coded — loosening
this past minutes erodes the freshness/replay security boundary
and operators tend to tune in the wrong direction.
NTP-style skew estimate (offer_received_at echo)
------------------------------------------------
Added optional `offerReceivedAt: Option<u64>` field to
`TraversalAnswer` payload. Responder fills it with `now_ms()` at
offer-receipt time. Initiator computes the standard NTP offset
formula `((T2-T1) + (T3-T4)) / 2` against the round-trip and
DEBUG-logs when `|skew| ≥ 30s`. Skew is also stashed in
`FailureState` and surfaced in `show_peers`. Non-breaking — older
responders that don't fill the field still produce valid answers,
and `estimate_clock_skew` returns `None`.
Per-peer state in `show_peers` JSON
-----------------------------------
Each peer entry in `show_peers` now carries:
"nostr_traversal": {
"consecutive_failures": <u32>,
"in_cooldown": <bool>,
"cooldown_until_ms": <u64 | null>,
"last_observed_skew_ms": <i64 | null>
}
Always emitted (schema-stable); values populated when discovery is
enabled and the npub has a recorded entry. Required a new public
`Node::nostr_discovery_handle()` accessor and refactored
`FailureState`'s internal Mutex from `tokio::sync` to `std::sync`
(operations never hold across await), which lets the synchronous
`show_peers` handler call `snapshot()` directly without the
dispatcher becoming async.
New config knobs (under `node.discovery.nostr`)
-----------------------------------------------
failure_streak_threshold: 5
extended_cooldown_secs: 1800
warn_log_interval_secs: 300
failure_state_max_entries: 4096
Tests
-----
12 new unit tests:
- 5 in `tests.rs` covering freshness strict / tolerated / rejected
outcomes, NTP skew estimation, and the backward-compat None case
when the responder didn't fill `offer_received_at`.
- 7 in `failure_state.rs` covering streak/warn-rate-limit state
transitions, cooldown active vs expired semantics,
success-resets-streak, observed-skew records, and size-cap
eviction by oldest `last_failure_at`.
CHANGELOG entries added under `[Unreleased]` Fixed.
Johnathan Corgan
Martti MalmiandJohnathan Corgan
Arjen
