The chacha20 crate (RustCrypto) ships SSE2 + soft backends only — on
aarch64 (Apple Silicon, ARM Linux servers, Docker on M-series Macs) it
falls through to a portable software impl at ~600–800 MB/s/core. ring
0.17 wraps BoringSSL's hand-tuned ChaCha20-Poly1305, which dispatches
to NEON on aarch64 and AVX2/AVX-512 on x86_64 — typically 3-5 GB/s/core
on the same hardware.
Same wire format. ChaCha20-Poly1305 is byte-deterministic for a given
(key, nonce, plaintext, aad), so any correct AEAD implementation
produces identical ciphertext. The full noise test suite covers this
implicitly: IK and XK roundtrip handshakes, replay window correctness,
multi-message nonce sequencing, and 100-message stress all pass at
1129/1129 (the lib's full `cargo test` count) — these only succeed if
ring's output matches what the receiver's existing replay-window
decrypt path expects.
Implementation notes:
* `LessSafeKey` (and `UnboundKey`) deliberately do not implement
Clone for safety. `CipherState`'s manual Clone impl rebuilds it
from the retained 32-byte key — cheap for ChaCha20-Poly1305 since
construction is essentially a key copy + a constant-time check.
* The keyed AEAD is now cached in `CipherState.cipher` instead of
being re-derived per packet. This was already a perf win for the
chacha20poly1305 backend (`new_from_slice` per packet was hot in
profiles); for ring it's a bigger win because `LessSafeKey`
construction also derives the Poly1305 key.
* Public `Vec<u8>`-returning API preserved. New module-private
`seal`/`open` helpers wrap ring's `seal_in_place_append_tag` /
`open_in_place` so the per-packet allocation pattern is local to
one place.
* `EndToEndState::Established` triggers `clippy::large_enum_variant`
after the swap (`NoiseSession` grew from ~600 to ~1.5 KB because
ring precomputes the Poly1305 key state at construction). That
precomputation is the win — boxing the variant would re-add an
indirection per packet and work against it. `#[allow]`'d at the
enum decl with a justifying comment.
ring is widely deployed (rustls, hyper-rustls, AWS SDK, …) and a
pure-Rust crate (uses BoringSSL's asm via a vendored build). It
introduces no new C toolchain requirements that aren't already there
for any rustls user.
Bench data from a downstream consumer of this crate (Docker e2e,
DURATION=10, identical hardware before/after, aarch64 Linux on
Apple Silicon):
2-node direct (A↔B):
TCP 1-stream 437 → 1097 Mbps (2.51×)
TCP 4-stream 439 → 1109 Mbps (2.53×)
TCP 8-stream 445 → 1069 Mbps (2.40×)
UDP @1000 Mbit 599/40% loss → 1000 Mbps lossless
ping under load ~0.6 ms (unchanged)
3-node forced transit (A → C → B):
TCP 1-stream 438 → 1019 Mbps (2.33×)
TCP 4-stream 421 → 982 Mbps (2.33×)
TCP 8-stream 443 → 1031 Mbps (2.33×)
UDP @1000 Mbit 475/52% loss → 1000 Mbps lossless
ping under load 7.68 ms / 215 ms max → 0.72 ms / 3.6 ms max
The relay-path lift is the cleanest tell on the bottleneck: the
transit node was crypto-bound (single-threaded soft chacha couldn't
keep up with offered rate), so the queue accumulated under load. With
NEON the relay isn't crypto-bound and the queue stops accumulating —
the 215ms ping-tail collapses to 3.6ms.
Freshly-restarted nodes with policy: open silently lost the historical
event replay that relays send in response to subscribe(). The
broadcast::Receiver was created INSIDE spawn_notify_loop, which the
tokio runtime starts at some indeterminate point after subscribe()
returns. tokio's broadcast channel only delivers messages sent after
the receiver is created; messages dispatched in the gap between
subscribe() issuing the REQ and the spawned task calling
client.notifications() were dropped by external_notification_sender.send
returning Err(SendError) with no subscribers attached.
Symptom on a node with policy: open: non-configured peers were not
discovered until they next re-published their advert (default
advert_refresh_secs = 1800s = 30 min). Configured peers were unaffected
because fetch_advert (relay-fetch path) caches them at startup-sweep
time. The bug has been latent since 34e00b9 added Nostr discovery —
relay-fetch covered the common case for configured-peer setups.
Fix: create the broadcast::Receiver in start() before subscribe() and
pass it into spawn_notify_loop. The receiver now exists when the REQ
replay arrives, so historical events flow through the cache path.
Also handle broadcast::error::RecvError::Lagged separately from
::Closed. The previous `while let Ok(...) = recv().await` exited the
loop on any Err, so a single lag event would silently kill the entire
subscription consumer with no recovery. Lagged now logs a warn (with
the skipped count) and continues; only Closed exits the loop.
Add two info-level log lines for in-field observability of the loop's
liveness. "nostr notify loop entered" fires once at task start; "nostr
notify loop received first event" fires once after the first
successful recv() with elapsed_ms since loop entry. Together these
turn the previous silent-failure shape (zero advert: peer cached
log lines indistinguishable between dead loop and idle channel) into
an immediately greppable startup signal — operators can confirm the
loop is alive and see how long it took to receive its first event,
catching any future regression in the subscription codepath in
seconds rather than waiting one advert_refresh_secs interval.
No public API change; the test fixture (NostrDiscovery::new_for_test)
does not call spawn_notify_loop and is unaffected.
The advert cache inside fetch_advert is read-only on hit — once a peer's
overlay advert is cached, every subsequent lookup returns the same
endpoints regardless of whether they still work. So when a peer rebinds
its NAT (or its STUN-discovered port flaps), connection retries to that
peer dial the same dead address forever, even with exponential backoff
firing at the right cadence.
Observed in deployment: macOS daemon's view of a Linux peer would
"regress" — peer marked rch=False after a brief link-dead window, then
hours of "Retry connection initiation failed: no operational transport
for any of <npub>'s addresses" with no recovery. Manual pause+resume of
the daemon (which restarts the FIPS endpoint and forces fresh advert
fetches) was the only way out.
When initiate_peer_connection / a retry tick returns
NodeError::NoTransportForType, fire-and-forget refetch_advert_for_stale_check
on the peer's npub. This re-fetches kind 37195 from advert_relays; if
the relay has a newer advert it replaces the cached entry, if it has
nothing it evicts the cached entry. Either way the next retry tick goes
to fresh data instead of looping on the same dead endpoint.
Mirrors the existing stale-advert sweep that runs from the
BootstrapEvent::Failed (NAT-traversal-streak) path, but covers the
direct-UDP-retry path which never crosses that streak threshold.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
When initiate_peer_connections() runs at boot, address resolution can
fail for an entire peer (no operational transport for the configured
transport types, all addresses unreachable, NAT rebind invalidated cached
endpoints, etc.). Before this change the failure was logged and silently
forgotten — the peer entry stayed in a dead state forever, accepting
incoming pings but unable to answer them, until the daemon was manually
restarted.
The retry plumbing (schedule_retry / process_pending_retries with
exponential backoff) already exists and is wired into the post-handshake
failure paths (BootstrapEvent::Failed, MMP dead-link timeout, handshake
timeout). The startup loop just wasn't calling it. Mirror the
BootstrapEvent::Failed path: on a startup peer-init error, parse the
peer's npub and call schedule_retry so the peer recovers without
operator intervention.
Includes a regression test that asserts retry_pending is populated when
initiate_peer_connections() fails for a peer with no operational
transport.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
The previous fix (6ebca3e) only refetched the advert when retry returned
NodeError::NoTransportForType (cache returned no addresses at all). But
the much more common stale-cache failure mode is: cache returns an
endpoint that LOOKS valid (the address it had last week, before the
peer's NAT rebound), the dial succeeds at the IP layer, the handshake
times out, MMP fires, schedule_reconnect adds the entry back to
retry_pending, next retry hits the same cached endpoint, dials it
again, times out again. Loop forever — no NoTransportForType ever
fires because the cache has data, just dead data.
Move refetch_advert_for_stale_check to before each retry attempt
unconditionally. Cheap (one Filter query against advert_relays with
a 2s timeout, bounded by the retry backoff cadence), and replaces the
cache only if the relay has a newer advert or evicts if the relay has
nothing. Keeps the retry loop pinned to relay ground truth instead of
whatever the cache happened to learn at startup.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
When a node was the smallest-NodeAddr peer it could see (no smaller
neighbor available as a parent), the spanning-tree state was promoting
it to root. But the ancestry it advertised on the next TreeAnnounce
still referenced its previous parent's path, so receiving peers
rejected the announce with `invalid ancestry: advertised root X is
not the minimum path entry Y`, blocking mesh transit on any path that
needed to traverse this node.
Detect the self-root transition explicitly in `TreeState::become_root`
and rebuild the advertised ancestry to start from self. Also surface
the same path through the MMP receive handler so a stale ancestry
inherited across reconnect is corrected eagerly rather than waiting
for the next observation tick.
Adds 80 unit tests in `tree::tests` covering self-root transitions,
mid-chain ancestor disappearance, and ancestry validation against the
new root, plus a regression in `node::tests::spanning_tree` for a
3-node chain where the middle node's only parent (the smallest-addr
peer) goes away — previously it would advertise an ancestry rejected
by both endpoints; now it self-roots cleanly.
The run_rx_loop's `tokio::select!` was costing one full scheduler hop
+ futex per inbound packet and per outbound TUN packet. Under
sustained load that capped throughput at one event per scheduler
quantum — independent of CPU (which sat near-idle) because every
iteration parked the worker, woke it via futex, processed one event,
then parked again.
After the await on `packet_rx.recv()` / `tun_outbound_rx.recv()`
fires, drain up to 256 additional ready items via `try_recv()` in a
tight inner loop before yielding back to `select!`. `biased` ordering
gives the data-plane branches priority over tick / control / DNS
under sustained load.
The 256 cap is empirically tuned to keep the worker on a busy stream
between yield points (a contiguous burst of ~256 MTU-sized packets
≈ 400 KB of contiguous traffic) while still bounding the inner loop
so a flood on one branch can't starve the periodic tick or control
socket. Lower caps (64) left perf on the table; higher caps (1024+)
delayed tick handling visibly under stress.
Pairs with the recvmmsg(2) change in the previous commit: the kernel
UDP queue now hands packets to `packet_rx` in 32-batches, and the
rx_loop drains them without a per-packet scheduler hop.
The UDP recv loop drained the kernel queue one packet per recvmsg(2).
Each call paid full per-syscall + per-task-wakeup overhead (~50us avg
including a futex-based scheduler hop), so under sustained load the
loop ran at one rx event per scheduler quantum — the dominant cap on
inbound packet rate.
On Linux, switch the steady-state path to recvmmsg(2) with a 32-packet
batch. A single readable() wakeup drains up to 32 datagrams in one
syscall before yielding back to the reactor. Stack-allocated mmsghdr
arrays sized to a module-level `BATCH_SIZE` constant.
`SO_RXQ_OVFL` is sampled once per batch off the cmsg chain of `msgs[0]`
and plumbed through `AsyncUdpSocket::recv_batch` as `(count, drops)`.
The counter is socket-wide and monotonic, so a single sample per batch
gives the 1Hz `sample_transport_congestion()` detector ample fresh
values under load (one batch = up to 32 datagrams). Cost is one
stack-allocated CMSG_SPACE(4) buffer + one CMSG_FIRSTHDR walk per
batch syscall.
macOS / Windows fall through to the per-packet recv_from loop —
recvmmsg is Linux-specific and the per-packet API is fast enough on
those platforms for now (recvmsg_x for Darwin can be added later).
The slice-array build also drops the `MaybeUninit::uninit().assume_init()`
+ `transmute` pair for `std::array::from_fn` over a single shared
`backing.iter_mut()` — same disjoint mutable borrows, no `unsafe`.
`PeerIdentity::pubkey_full()` falls through to
`self.pubkey.public_key(Parity::Even)` whenever the parity-aware full
key wasn't passed at construction (i.e. for every peer constructed
from an npub or x-only key). Underneath, that runs a secp256k1 EC
point parse — `fe_sqrt` + `fe_mul` + `ge_set_xo_var` — which is ~6%
of per-packet CPU on the bulk-data send path for a value that never
changes after construction.
Compute it eagerly. The same EC point parse already runs at
construction inside `NodeAddr::from_pubkey`, so the cost is paid once
where it would be paid anyway.
`Node::adopt_established_traversal` was constructing the adopted UDP
transport with `UdpConfig::default()` — MTU 1280, default recv/send
buffer sizes, default accept/advertise flags. If the operator had
configured a higher MTU on the primary `[transports.udp]` listener
(e.g. 1500 on a path where larger frames are known viable), full-sized
tunnel datagrams sent over the NAT-traversed link would exceed the
adopted socket's MTU and get dropped at the socket layer with no
visibility into why throughput collapsed.
Inherit the primary UDP config (MTU + recv/send buffer sizes + accept
/ advertise flags) and clear the bind / external-address fields since
the adopted socket is already bound. Lookup tries `transport_name`
first so operators with multiple named `[transports.udp.<name>]`
listeners pick up inheritance from the matching listener, and falls
back to the unnamed `Single` listener so single-instance configs work
unchanged.
The previous default of MTU 1280 was deliberately the IPv6 minimum,
the only value guaranteed to survive arbitrary middlebox paths. With
this change, operators who set their primary listener higher (based
on known-clean LAN topology) will have NAT-traversed flows initially
attempting that higher MTU and possibly black-holing on tighter paths
until reactive `MtuExceeded` recovery kicks in. Documented in the
adoption call-site comment so future readers understand why the
conservative default went away.
Discovered in a downstream consumer where a `MESH_TUNNEL_MTU=1320` /
encrypted wire ~1426B produced silent packet drop on every session
that had been promoted onto a NAT-traversed link.
Adds two sibling tests in `src/node/tests/bootstrap.rs` pinning the
new behaviour for the `Single` and `Named` config variants.
Surfaces local services reachable from the mesh, paired with their
current `inet fips` baseline filter classification. Lands to the
right of the existing TUN section in the Traffic block.
A new daemon control query `show_listening_sockets` returns IPv6
listeners bound to either `::` (wildcard) or the node's fd00::/8
address, each classified as Accept / Drop / Unknown / NoFirewall
against the running inbound chain. fipstop renders the result as a
table beside the Traffic counters: Accept rows in default White,
Drop / Unknown in DarkGray, a yellow banner above the table when
`fips-firewall.service` is inactive, and a trailing `*` on
wildcard binds to remind the operator the bind is not
fips0-specific.
Daemon side:
- `src/control/listening.rs` walks `/proc/net/tcp6` and
`/proc/net/udp6` via the procfs crate (LISTEN state for TCP,
wildcard remote for UDP), filters to fips0-reachable binds, and
resolves inodes to PID / comm via `/proc/<pid>/fd`.
- `src/control/firewall_state.rs` shells out to
`nft -j list table inet fips` and walks the inbound chain.
Recognises canonical accepts (`tcp/udp dport N accept`,
`dport { ... } accept`, `dport A-B accept`), the iifname-scoping
line, conntrack and icmpv6 lines (skipped). Any rule with
unrecognised matchers (saddr filters, jumps, daddr filters) or
non-terminal verdicts forces Unknown classification for the
ports it references. Eleven unit tests cover the classification
logic; the listening enumerator carries a /proc-parsing test of
its own.
- `show_listening_sockets` emits
`{fips0_addr, firewall_active, sockets[]}` with per-row
`{proto, local_addr, port, pid, process, filter, wildcard_bind}`.
fipstop side:
- `src/bin/fipstop/ui/dashboard.rs` splits the Traffic block into
a 50/50 horizontal layout; the existing TUN + Forwarded panel
occupies the left half.
- `src/bin/fipstop/ui/listening.rs` renders the right half.
- `main.rs` fetches the new query each tick when the Node tab is
active. Errors are non-fatal: an old daemon without the query
leaves the payload at None and the panel renders "loading...".
`Cargo.toml` gains `procfs = "0.18"` on the Linux target. IPv4
listeners are not enumerated — fips0 is IPv6-only.
Folded in: revert the default-socket lookup from writability-probe
back to existence-based selection. The previous tempfile-probe on
`/run/fips` silently steered fipstop / fipsctl onto an XDG path
the daemon never bound for any user in the `fips` group whose
shell session had not yet picked up the supplementary group (no
re-login after `usermod -aG`). `XDG_RUNTIME_DIR` is set on every
modern systemd-managed user session, so this hit the common case.
The kernel checks actual group membership at `connect(2)`, so a
user who genuinely cannot connect now gets a clear `EACCES`
rather than a silent path mismatch. Drops the now-unused
`is_writable_dir` helper. `XDG_RUNTIME_DIR` existence validation
is preserved.
Documentation:
- `docs/reference/cli-fipstop.md` — Node-tab row updated, new
"Listening on fips0 panel" section.
- `docs/reference/control-socket.md` — `show_listening_sockets`
added to the read-only queries table.
- `docs/how-to/enable-mesh-firewall.md` — new "Verify with
fipstop" section.
- `docs/tutorials/host-a-service.md` — fipstop callouts at
Steps 3, 5, 6 + Troubleshooting bullet + wildcard-bind reminder
under "What you've learned".
- `CHANGELOG.md` — new bullet under `Added / Operator Tooling`,
resolver `Fixed` entry rewritten to describe the
existence-based final shape.
Forward-merge of 12 master commits past the previous merge
(823b830, master @ 18019bb): dep-audit bumps (rand,
clap, tun, rtnetlink, windows-service, plus the bump-safe
lockfile batch), bloom-storm chaos scenario, control-socket
resolver consolidation, gateway dns.listen default change,
OpenWrt ipk README refresh, gateway tutorial review, Ethernet
MTU rustdoc fix, dead session-variant drop, CHANGELOG prep.
Conflict resolution:
- CHANGELOG.md: both bullets kept under [Unreleased] / Fixed.
Master's spanning-tree internal-path-propagation fix precedes
next's tree-ancestry-test determinism entry and the
responder-Disconnect XX-handshake entry.
- src/protocol/session.rs: kept next's SessionSetup/SessionAck
variants and rustdoc. Master's drop of those variants suits
v0.3.0's FSP phase-byte dispatch but is undone by next's
v0.4.0 wire format, which retains the variants and uses the
inner msg_type byte for handshake identification.
- src/transport/ethernet/mod.rs: kept next's "interface MTU - 4"
comment. Master corrected the v0.3.0 3-byte rustdoc; next
redesigned the framing to a 4-byte header (type/flags/length)
for shared-media beacons, so master's correction does not
apply to next's format.
Auto-merged cleanly: Cargo.toml (next's 0.4.0-dev + the new dep
pins from master), Cargo.lock, all gateway docs,
docs/reference/configuration.md, packaging files,
.github/workflows/ci.yml, testing/chaos/sim/* and
testing/ci-local.sh (bloom-storm additions), src/config/*.
Local verification: cargo build --release, cargo test (1252
passed, 4 ignored), cargo clippy -D warnings, cargo fmt --check
all green.
Both variants of SessionMessageType were never emitted anywhere
in src/, and the production from_byte dispatch sites lacked
Some-arms for them — any 0x00/0x01 byte that reached either
dispatcher would log "Unknown..." and drop. The matching rustdoc
tables described an Offset 0 msg_type byte that the encode() path
has never written; the actual wire format is the FSP common prefix
[ver_phase][flags][payload_len:2 LE] with body keyed by phase
nibble, as documented in docs/reference/wire-formats.md.
Drop the variants, drop their from_byte/to_byte/Display arms, fix
the two stale rustdoc tables to describe the real wire shape, and
trim the variant-iteration unit test that enumerated them.
Zero on-wire behaviour change.
The Ethernet data frame format is `[type:1][length:2 LE][payload]`,
so the per-link payload MTU is the interface MTU minus 3 bytes,
not minus 1. The 2-byte length field is required to trim NIC
minimum-frame padding before AEAD verification.
The implementation in src/transport/ethernet/mod.rs already uses
saturating_sub(3) correctly; only the rustdoc on the effective_mtu
field and the EthernetConfig.mtu field's documentation lagged behind.
No behaviour change.
The gateway is designed for systems already serving DHCP and DNS to
a LAN segment (canonically an OpenWrt AP). On those systems port 53
is already taken by the existing resolver, so the prior `[::]:53`
default conflicted with the gateway's intended deployment target out
of the box.
The OpenWrt ipk previously overrode this in its packaged config as a
workaround; matching the source default to what the canonical
deployment actually wants makes the override redundant and removes a
foot-gun for fresh manual Linux-host installs. The redundant
`dns.listen` line in `packaging/openwrt-ipk/files/etc/fips/fips.yaml`
is dropped along with this change.
Operators on a host without a pre-existing resolver on port 53 can
opt back into the wildcard bind by setting `dns.listen: "[::]:53"`
explicitly. The new default binds IPv6 loopback only — Linux IPv6
sockets bound to explicit `::1` do not accept v4-mapped traffic, so
forwarders that reach the gateway over IPv4 loopback need to be
pointed at an explicit IPv4 listen address instead.
Touches the gateway config struct and its default-value test, the
commented-out gateway example in the Debian common fips.yaml, the
OpenWrt ipk config (override removed), the gateway reference /
how-to / design / tutorial / troubleshoot docs, and a CHANGELOG
entry under [Unreleased] -> Changed.
Daemon and client tools previously evaluated the same three locations
(`/run/fips`, `XDG_RUNTIME_DIR`, `/tmp`) in different orders, allowing
fipsctl/fipstop to connect to a socket the daemon never bound when
neither side set `node.control.socket_path` explicitly.
Collapse the three call sites (`default_control_path`,
`default_gateway_path`, `ControlConfig::default_socket_path`) into a
shared `resolve_default_socket` helper. Canonical order is
`/run/fips` -> `$XDG_RUNTIME_DIR/fips/` -> `/tmp/fips-<name>`. Two
hardening fixes folded in: writability is probed via tempfile create
rather than mode bits (ACL- and group-aware), and `XDG_RUNTIME_DIR`
is validated as an existing directory before being used (avoids
stale post-logout values).
The deployed fleet is unaffected -- packaged configs set
`node.control.socket_path` explicitly. The fix surfaces for dev
runs and the binary-install getting-started path.
A leaf node's my_coords could go stale after an upstream
mid-chain ancestor swap, leaving non-parent destinations with
100% loss until either the parent or the depth also changed.
The broadcast gate in handle_tree_announce's
`else if !is_root && parent_id == from` branch compared only
(root, depth). A swap that altered an interior ancestor without
changing root or depth (e.g. A->B->C reorganizing to A->D->C
while keeping (A, depth=2)) was silently dropped one hop below
the swap node. Downstream nodes' coords paths then drifted from
the real tree topology, defeating greedy distance routing for
any destination whose path crossed the unrepresented section.
Widen the gate to compare the full my_coords.node_addrs() so
mid-chain swaps propagate to leaves the same way root/depth
changes already did. The gate body's bloom-marking is adjusted
in step so the wider gate doesn't generate empty/redundant
FilterAnnounces to every peer on every mid-chain swap
propagation: mark_changed_peers replaces
mark_all_updates_needed in the gate body (parent_id is
unchanged in this branch, so outgoing filter content is
typically unchanged, and mark_changed_peers correctly marks
zero peers in that case), and the unconditional
mark_update_needed(*from) at the top of handle_tree_announce
is removed (bloom exchange initiation is already handled at
handshake completion, and ongoing content changes are picked
up naturally by mark_changed_peers in handle_filter_announce
when peers send their next filter).
Required surface change: peer_inbound_filters in
src/node/bloom.rs upgraded from private to pub(super) so the
gate body can call it.
Verified in a 6-node depth-4 docker reproduction under
tc/netem-induced parent flapping: a depth-4 leaf's
ancestry_changed counter advances with upstream parent
switches while bloom_sent stays at zero matching the
pre-change steady-state baseline.
Bump the default Nostr-discovery advert namespace from
`fips-overlay-v1` to `fips-overlay-v1-next` on next. Master
continues to publish under `fips-overlay-v1`.
Background: next runs FMP-v1 (Noise XX, msg1 33 bytes) which is
wire-incompatible with master's FMP-v0. Until now both branches
defaulted to the same Nostr advert namespace, so a stock
next-branch daemon's open-discovery sweep would happily pick up
master peers' adverts (and vice versa), succeed at the UDP punch,
adopt the socket, and fail every FMP handshake at the version-gate.
The per-peer-mismatch cooldown introduced on master is the safety
net for any case that slips past this default; the namespace
separation is the structural answer.
Three sites updated:
- `src/discovery/nostr/types.rs` `ADVERT_IDENTIFIER` const
documents why the value is branch-specific.
- `src/config/node.rs` `default_app()` matches.
- `src/discovery/nostr/tests.rs` and the
`testing/nat/scripts/nostr-relay-test.sh` malformed-advert
fixture publish under the new namespace so test harnesses see
the same adverts a real daemon would.
Operators who need cross-branch discovery during a coordinated
rolling upgrade can override `node.discovery.nostr.app` in
fips.yaml back to `fips-overlay-v1`.
Open-discovery NAT traversal succeeds at the UDP layer regardless
of what FMP-protocol version the peer speaks. When the daemon
discovers a peer running a different FMP version (e.g. a v0/v1 mix
during a mid-rollout window, or a misconfigured peer in the same
advert namespace), the punch sequence completes, the socket is
adopted via `Node::adopt_established_traversal`, and we initiate
an FMP handshake. The peer drops our msg1 at its own version-gate
and we drop their msg1/msg2 at `Unknown FMP version, dropping`.
Neither side advances the handshake.
Today the bootstrap transport sits idle until the 31s stale-
handshake timeout, drops, and the open-discovery sweep ~30s later
fires the full STUN+offer+answer+punch sequence again — every
minute, indefinitely, against peers the handshake literally cannot
complete with.
Add a `Node::bootstrap_transport_npubs` map populated alongside
`bootstrap_transports` at adopt time. The rx loop reverse-maps the
transport_id → npub on version-mismatch and bumps the discovery
layer's `failure_state` to a long structural cooldown via the new
`NostrDiscovery::record_protocol_mismatch` API. The next sweep
skips the npub for `protocol_mismatch_cooldown_secs` (default
86400 = 24h, separate from the 30-min transient-failure
`extended_cooldown_secs`).
One-shot WARN per fresh observation. Repeat mismatches inside the
cooldown window are silent (the failure_state method returns false
when an existing comparable cooldown is already in place). The
handshake/transport teardown chain is unchanged — the fix is
specifically about preventing the *next* sweep cycle from
re-traversing.
Cleared on `cleanup_bootstrap_transport_if_unused` and on the
adopt-failure rollback path so completed handshakes don't leave
stale entries behind.
Four new unit tests in `failure_state.rs` cover fresh-entry
signaling, repeat-suppression inside the window, streak-pin
behavior for `show_peers` rendering, and post-cooldown re-arming.
The TUN-side TCP MSS clamp consults `path_mtu_lookup` (FipsAddress-
keyed) when sizing outbound TCP flows. Until now, only the reactive
`MtuExceeded` handler mirrored the bottleneck MTU into that store;
the proactive end-to-end `PathMtuNotification` echoed by the
destination updated only `MmpSessionState.path_mtu`, leaving the TUN
mirror stale.
On stable long-lived paths, the proactive echo can tighten the
session-canonical MTU well before any transit router fires a
`MtuExceeded` for those flows (since all current traffic is already
sized by the tighter session value). New TCP flows opened during
that window get clamped by the discovery-time value rather than the
session-canonical one, leading to PMTU-D loss until the reactive
path eventually fires.
Mirror the post-apply MTU into `path_mtu_lookup` whenever
`apply_notification` returns true, with the same tighter-only
semantics as the reactive mirror — never loosen the clamp. Gated on
the bool return so spurious writes don't happen on rejected
increases or no-op same-value notifications.
Four new unit tests exercise the empty-lookup write, tighten-
existing, keep-tighter-existing, and no-session-no-op paths,
parallel to the existing reactive-mirror test trio.
When a UDP transport had `advertise_on_nostr: true` + `public: true`
+ `bind_addr: 0.0.0.0:NNNN`, the advert builder previously read the
kernel's `local_addr()`, found `0.0.0.0`, filtered it out (correctly
— wildcard isn't a valid advertised endpoint), and silently emitted
no UDP endpoint in the published advert. Operators on AWS EIP / GCP
/ Azure setups (where binding to the public IP directly is impossible
because 1:1 NAT does the address translation off-host) had no way to
advertise UDP without binding to a specific local IP — and no log
explaining what was happening. TCP had the same shape, with no
`public: true` precondition.
Three pieces, layered. UDP gets zero-config autodiscovery via STUN;
both UDP and TCP get an explicit operator-supplied override; the
fall-through path now logs loudly instead of silently skipping.
UDP public-IP autodiscovery (STUN)
----------------------------------
In the UDP `is_public()` + wildcard-bind branch, run a one-shot
STUN observation against an ephemeral UDP socket on the daemon's
configured `stun_servers`. Take the reflexive IPv4 (the
STUN-reported port is the ephemeral source port and is discarded),
combine with the configured listener port for the advert
(`udp:<reflexive-ip>:<port>`). Works on AWS EIP / GCP / Azure
1:1-NAT setups because STUN sees the public-Internet egress IP and
the bind port is preserved through 1:1 NAT.
Result is cached per-transport on a new `public_udp_addr_cache`
field on `NostrDiscovery` (keyed by `TransportId.as_u32()`).
Asymmetric cache TTL: a successful observation is cached for
`advert_refresh_secs` (default 30 min) so we don't STUN every
refresh tick. A failed observation is cached for only 60s
(`PUBLIC_UDP_ADDR_FAILURE_TTL`) so a transient STUN flake at
startup retries within ~a minute and the advert grows its UDP
endpoint as soon as STUN starts working — rather than waiting the
full 30-min cycle.
The shared `observe_traversal_addresses` STUN helper had a
hard-coded 2s per-server response wait, right for the
per-traversal flow (latency-sensitive — 3 STUN servers worst-case
= 6s) but too short for the one-shot advert-publish startup
discovery. Parameterized `per_server_timeout` on the helper, with
two named constants in `stun.rs`: `TRAVERSAL_STUN_TIMEOUT = 2s`
(existing call sites) and `ADVERT_STUN_TIMEOUT = 5s` (new
public-UDP discovery path). Both use `tokio::time::timeout_at`
under the hood, so success returns immediately — the timeout is
only the worst case.
`external_addr` override (UDP + TCP)
------------------------------------
New `external_addr: Option<String>` field on `transports.udp.*`
and `transports.tcp.*` for explicit advertise-as override. Takes
precedence over both the bound `local_addr` and (for UDP) the
STUN-derived autodiscovery.
Required for TCP on cloud-NAT setups (AWS EIP, GCP/Azure external
IPs) where binding to the public IP directly fails with
`EADDRNOTAVAIL` because the public IP isn't on a host interface —
the network fabric does 1:1 NAT off-host. Without this field the
operator's only TCP path was "leave advert off" or "find a way to
make the public IP locally bindable."
For UDP, `external_addr` is optional but useful as a deterministic
alternative to STUN. Operators who want to skip STUN egress, whose
STUN servers are blocked, or who want the daemon to not depend on
external services for advert content can specify it explicitly.
The accessor parses two shapes:
- Bare IP (`"54.183.70.180"` or `"2001:db8::1"`): combines with
the configured `bind_addr` port.
- Full host:port (`"54.183.70.180:8443"` or `"[2001:db8::1]:443"`):
used verbatim — useful for port-forward setups where the
externally-visible port differs from the bind port.
Final precedence in `Node::build_overlay_advert` (now async, only
caller `refresh_overlay_advert` was already async):
- UDP: `external_addr` → non-wildcard `local_addr` → STUN → loud warn
- TCP: `external_addr` → non-wildcard `local_addr` → loud warn
Loud warns instead of silent skips
----------------------------------
The wildcard-bind fall-through paths now log a `warn!` pointing at
the operator-side fixes:
- UDP: "set transports.udp.external_addr, bind to a specific
public IP, or ensure node.discovery.nostr.stun_servers is
reachable"
- TCP: "Either set external_addr to the public IP (recommended for
cloud 1:1-NAT setups) or bind explicitly to the public IP"
Replaces the silent skip that previously cost operators a
debugging session when the advert mysteriously contained only the
Tor onion endpoint.
Tests
-----
11 new unit tests in `src/config/transport.rs` covering the
parser (IPv4/IPv6, bare/full, malformed) and the accessor (UDP
with default bind, UDP with explicit port override, UDP unset,
TCP without bind_addr, TCP with bind_addr, TCP with full
socket-addr override, parse_bind_port for IPv4/IPv6/malformed).
The 38-test nostr suite still passes.
CHANGELOG entries under `[Unreleased]` Fixed.
Public-test daemons with populous open-discovery caches generate
sustained NAT-traversal-failure WARN volume (~140/hour, ~3500/day)
against cache-learned peers that have gone offline — their adverts
are absent from major Nostr relays but cached entries persist until
their advertised `valid_until` expires. The daemon kept publishing
offers indefinitely under exponential backoff with no per-peer
suppression, drowning operator signal and hammering relays. A
parallel concern: the strict freshness check at signal.rs silently
rejected offers under modest clock skew (now_ms() anchors to
SystemTime once at startup, so post-startup NTP step adjustments
don't propagate on long-uptime daemons), indistinguishable from
"peer is offline."
Six independent improvements layered on the existing retry logic.
Per-npub WARN log rate-limit
----------------------------
New `FailureState` struct on `NostrDiscovery` records per-npub
`last_warn_at_ms`. Subsequent failures inside `warn_log_interval_secs`
(default 5 min) emit DEBUG instead of WARN. Each WARN now also
carries `consecutive_failures` and remaining `cooldown_secs` so
operators can read the trajectory without grepping multiple lines.
Per-npub consecutive-failure counter + extended cooldown
--------------------------------------------------------
After `failure_streak_threshold` (default 5) consecutive failures
against a peer, the next `extended_cooldown_secs` (default 1800)
of attempts are suppressed by pushing
`retry_pending[npub].retry_after_ms` past the cooldown wall. The
open-discovery sweep also consults `cooldown_until` and increments
a new `skipped_cooldown` counter so a peer whose `retry_pending`
was cleared by max_retries doesn't get re-enqueued during the
cooldown window. Caps offer-publish rate per dead peer regardless
of how often the sweep tries to re-enqueue.
Stale-advert eviction on streak-threshold transition
----------------------------------------------------
On the threshold-crossing transition (one-shot, not every
subsequent failure), `tokio::spawn` an active re-fetch of the
peer's Kind 37195 advert from `advert_relays`. Three outcomes:
- absent on relays → cache evicted; sweep won't re-enqueue
(peer is genuinely gone).
- newer `created_at` → cache refreshed + streak reset
(peer republished; allowed to retry immediately).
- same → cache untouched; cooldown stands.
Cost: ~one fetch per dead peer per 30-min cooldown cycle, vs
hundreds of offer publishes/hour today.
Clock-skew tolerance on freshness check
---------------------------------------
`signal.rs` `validate_offer_freshness` and
`validate_traversal_answer_for_offer` now allow ±60s grace beyond
strict TTL. Both return a new `FreshnessOutcome` enum so callers
can DEBUG-log when an offer/answer was only accepted via the grace
window. `FRESHNESS_SKEW_TOLERANCE_MS` is hard-coded — loosening
this past minutes erodes the freshness/replay security boundary
and operators tend to tune in the wrong direction.
NTP-style skew estimate (offer_received_at echo)
------------------------------------------------
Added optional `offerReceivedAt: Option<u64>` field to
`TraversalAnswer` payload. Responder fills it with `now_ms()` at
offer-receipt time. Initiator computes the standard NTP offset
formula `((T2-T1) + (T3-T4)) / 2` against the round-trip and
DEBUG-logs when `|skew| ≥ 30s`. Skew is also stashed in
`FailureState` and surfaced in `show_peers`. Non-breaking — older
responders that don't fill the field still produce valid answers,
and `estimate_clock_skew` returns `None`.
Per-peer state in `show_peers` JSON
-----------------------------------
Each peer entry in `show_peers` now carries:
"nostr_traversal": {
"consecutive_failures": <u32>,
"in_cooldown": <bool>,
"cooldown_until_ms": <u64 | null>,
"last_observed_skew_ms": <i64 | null>
}
Always emitted (schema-stable); values populated when discovery is
enabled and the npub has a recorded entry. Required a new public
`Node::nostr_discovery_handle()` accessor and refactored
`FailureState`'s internal Mutex from `tokio::sync` to `std::sync`
(operations never hold across await), which lets the synchronous
`show_peers` handler call `snapshot()` directly without the
dispatcher becoming async.
New config knobs (under `node.discovery.nostr`)
-----------------------------------------------
failure_streak_threshold: 5
extended_cooldown_secs: 1800
warn_log_interval_secs: 300
failure_state_max_entries: 4096
Tests
-----
12 new unit tests:
- 5 in `tests.rs` covering freshness strict / tolerated / rejected
outcomes, NTP skew estimation, and the backward-compat None case
when the responder didn't fill `offer_received_at`.
- 7 in `failure_state.rs` covering streak/warn-rate-limit state
transitions, cooldown active vs expired semantics,
success-resets-streak, observed-skew records, and size-cap
eviction by oldest `last_failure_at`.
CHANGELOG entries added under `[Unreleased]` Fixed.
The control-query show_bloom snapshot, baselined on master, mismatches
the next-branch output because the bloom filter v2 work on next added
six new stats fields (deltas_sent, full_sends, nacks_received,
nacks_sent, size_changes, total_compressed_bytes, total_raw_bytes)
and dropped non_v1. The snapshot test correctly flagged this as
schema drift.
The drift is intentional (next's bloom v2 is a deliberate schema
expansion, not a regression), so the right action per the test's own
guidance is to delete the fixture and let the test regenerate it
against next's actual output.
No production-code change.
The control-query snapshot tests panicked on the Windows GitHub runner
because git's default core.autocrlf=true converted fixture files
(src/control/snapshots/*.json) to CRLF on checkout, while the
in-memory JSON output is LF. trim_end() only strips trailing newlines,
not interior \r, so every snapshot comparison mismatched.
Two defenses:
1. .gitattributes: pin src/control/snapshots/*.json to text eol=lf so
future Windows checkouts keep the fixtures LF-only regardless of
local git config.
2. src/control/queries.rs (assert_snapshot): replace \r\n with \n in
the expected text before comparison, so any future
re-introduction of CRLF (a contributor with non-LF editor settings,
a different runner, etc.) doesn't surface as a snapshot mismatch.
Cover two adjacent runtime behaviors in the discovery state machine
that were previously unpinned at the test level.
1. Open-discovery startup sweep iterate-filter-queue contract.
Cover the runtime sweep behavior: iterate advert cache, apply
skip-filters (own-pubkey, already-connected peers), queue eligible
entries to retry_pending. The config layer was tested but the sweep's
own filtering logic was unpinned.
src/discovery/nostr/runtime.rs: add #[cfg(test)] impl block with
three pub(crate) helpers — new_for_test() builds a minimal
NostrDiscovery with empty cache and no relays/background tasks (uses
fresh nostr::Keys signer + Client::builder().autoconnect(false));
cached_advert_for_test() wraps an OverlayEndpointAdvert into a
CachedOverlayAdvert valid for 1h; insert_advert_for_test() writes
direct to the advert_cache RwLock. All three vanish from release
builds via cfg-gating.
src/node/lifecycle.rs: visibility-only widen on
run_open_discovery_sweep from private async fn to
pub(in crate::node) async fn so the in-tree test can drive it
directly. Same pattern as already-pub(in crate::node) handlers in
src/node/handlers/.
src/node/tests/discovery.rs: add #[tokio::test]
test_open_discovery_sweep_queues_eligible_skips_filtered. Builds
Node + Arc<NostrDiscovery>, injects 3 adverts (eligible, already-
connected peer, own-pubkey), invokes the sweep, asserts retry_pending
contains exactly the eligible entry with matching peer_config npub
and the two filtered entries do NOT appear.
2. Per-attempt timeout state machine in check_pending_lookups.
Cover the central new behavior of f16b837: the [1, 2, 4, 8] retry
sequence (cumulative deadlines 1100/3100/7100/15100ms), one fresh
LookupRequest per attempt, and final-timeout reaching the unreachable
state. The opt-in DiscoveryBackoff machinery was well-tested but inert
at default config; this pins the state machine that runs by default.
Add test_check_pending_lookups_default_sequence_unreachable to
src/node/tests/discovery.rs. Constructs a Node with a peer that has
the target in its bloom but cannot respond (no Noise session — the
state-machine bookkeeping is independent of wire-send success).
Drives check_pending_lookups deterministically through:
t=1100 → second attempt; entry.attempt advances; req_initiated++
t=3100 → third attempt; entry.attempt advances; req_initiated++
t=7100 → fourth attempt; entry.attempt advances; req_initiated++
t=15099 → no-op (one ms before final deadline)
t=15100 → final timeout
At t=15100 asserts: pending_lookups[target] removed; resp_timed_out
counter +1 (this is the actual counter name); pending_tun_packets
[target] removed (queued packet dropped); a frame on the TUN sender
with IPv6 + next_header=58 + ICMPv6 type=1 (Destination Unreachable).
Fresh-request_id-per-attempt is structurally guaranteed: LookupRequest
::generate() unconditionally calls rand::random::<u64>(), and the
test asserts req_initiated increments by exactly 1 per retry (proving
initiate_lookup runs fresh each time, not a resend of cached state).
The originator's request_id isn't stored on the originator side
(deliberately omitted from recent_requests so the response is
recognized as "ours"), so direct request_id capture is not feasible
and counter-tick is the load-bearing observable.
No production logic touched; no visibility widening needed
(check_pending_lookups was already pub(in crate::node)).
Cover the security-relevant defensive signal: sustained decrypt
failures indicate key drift or active probing; threshold-trip force-
removes the peer.
src/peer/active.rs (+43): two unit tests on the counter struct itself
- test_increment_decrypt_failures_monotonic asserts each
increment_decrypt_failures() call returns count+1 for at least 25
iterations
- test_reset_decrypt_failures_zeroes_counter asserts the reset
helper zeroes a non-zero counter and is idempotent
src/node/tests/decrypt_failure.rs (new, 93 lines): end-to-end test
- Builds a Node + connected peer via existing make_completed_connection
/ add_connection / promote_connection harness so peers_by_index is
exercised, not just peers
- Drives the peer to threshold-20 by calling handle_decrypt_failure 20
times; asserts iterations 1..20 leave the peer registered with
monotonically increasing counter, then iteration 20 evicts from both
peers and peers_by_index
src/node/handlers/encrypted.rs: visibility-only widen on
handle_decrypt_failure from private to pub(in crate::node) so the
in-tree test can drive the threshold logic without re-implementing
it. Same pattern as the already-pub(in crate::node)
handle_encrypted_frame in the same file.
Threshold pinned: DECRYPT_FAILURE_THRESHOLD = 20 at
src/node/handlers/encrypted.rs:11.
Add hand-rolled JSON snapshot harness in src/control/queries.rs to
detect silent schema drift in operator-facing control-socket
responses. Builds a Node with deterministic identity
(Identity::from_secret_bytes(&[0xAB; 32])), invokes each of the 18
show_* handlers, redacts 17 volatile fields (version, pid, exe_path,
control_socket, tun_name, allow_file, deny_file, *_ms / *_secs_ago /
uptime_secs), sorts object keys recursively, and compares against a
fixture in src/control/snapshots/.
First run writes snapshots and passes; subsequent runs enforce.
Future schema changes show as a snapshot diff that operators update
intentionally — not a stability contract, just a tripwire so drift
is never silent.
A 19th meta-test dispatch_covers_all_snapshotted_handlers walks every
name through dispatch() to confirm each returns status: ok and trips
if a 19th handler is added without a matching snapshot.
No new dependencies (insta deliberately not added; Cargo.toml
[dev-dependencies] keeps tempfile + criterion only). 18 fixture
files added, ~544 lines combined; harness is 367 added lines, all
inside #[cfg(test)] mod tests.
Add #[cfg(target_os = "macos")] unit tests catching macOS-specific
regressions before they reach the macos-latest GitHub runner.
src/upper/tun.rs: surgical refactor extracts the inline
AF_INET6_HEADER constant into module-scope helpers
utun_af_inet6_header() (encode) and parse_utun_af_prefix() (decode
inverse for round-trip testability). TunWriter::run now calls the
helper instead of the inline const; behavior unchanged. Six new
tests pin the AF=30 constant matching Darwin, big-endian byte order,
encode/parse round-trip, short-buffer rejection, minimum header
acceptance with trailing payload, and no-panic on garbage bytes.
src/transport/ethernet/socket_macos.rs: existing test mod already
covered bpf_wordalign and 5 parse_next_frame cases. Three new tests
fill genuine gaps: struct layout pin against kernel ABI, caplen-
overrun rejection, full Ethernet header round-trip via parse.
Existing tests pre-exist; only adding to the same #[cfg(test)] block.
Add test_routing_bloom_hit_not_closer_falls_through_to_tree to
src/node/tests/routing.rs covering the regression class fixed in
a859da7: bloom candidates exist BUT none are strictly closer to
destination than the tree parent. Pre-fix find_next_hop returned
None (NoRoute); post-fix it falls through to greedy tree routing.
Existing tests covered bloom-hit-closer, bloom-preferred-over-tree,
no-bloom-hit-tree-fallback, and bloom-hit-without-coords — but the
in-between case was unpinned.
Topology: self is tree root with two children (tree_peer at distance
1, bloom_peer at distance 3); destination is one hop below tree_peer
(self distance 2). Only bloom_peer advertises dest in its filter.
Distance of the only bloom candidate (3) is not strictly less than
self's (2), so select_best_candidate returns None and the call must
fall through to greedy tree routing returning tree_peer. Asserts
explicit identity (assert_eq! tree_peer_addr) and rules out the
wrong-peer regression (assert_ne! bloom_peer_addr).
Add table-driven unit test test_log_level_parser to src/config/node.rs
covering all 5 explicit match arms (trace, debug, warn|warning, error),
the implicit None-and-unknown → INFO default, case-insensitivity via
to_lowercase (TRACE / Debug / Warning / WARN / ERROR / INFO), and
edge cases (empty string, "verbose"). Pins observed behavior: there
is no explicit "info" arm — it falls through the wildcard to INFO,
identical to unknown strings.
Add 6 negative-input unit tests to src/discovery/nostr/stun.rs covering
truncated header (all lengths 0..20), bad magic cookie, unknown attribute
type (skip-not-error), truncated XOR-MAPPED-ADDRESS, length-overflow
attribute, and transaction-ID mismatch. The happy path was exercised by
the 3 NAT scenarios but the parser had no negative-input coverage.
Tests pin observed behavior: parse_stun_binding_success returns
Option<SocketAddr>, so all malformed-response cases assert None rather
than an error variant. Unknown TLVs are silently skipped via the loop's
default arm; length-overflow triggers the value_end > packet.len() guard
and breaks out of the loop without panicking.
When a transit forwarder drops an oversized data packet and reports
the bottleneck back via MtuExceeded, the receive-side handler
already updates per-session MmpSessionState::path_mtu (used by PTB
synthesis to feed kernel TCP). It did not, however, update
path_mtu_lookup — the per-destination map the TUN reader/writer
consult at TCP MSS clamp time. So forward-path-asymmetry flows kept
clamping at the discovery reverse-path value (too generous for the
actual forward-path budget) on every subsequent SYN.
Add the missing write at the same point apply_notification runs.
Keep the tighter of existing-or-new — the clamp must never loosen.
Same write-shape as seed_path_mtu_for_link_peer.
Tests:
- Three focused unit tests on handle_mtu_exceeded for the empty,
tighten, and keep-tighter cases.
- Extended test_multihop_pmtud_heterogeneous_mtu to assert the
lookup tightens after the wire-level MtuExceeded propagation,
alongside its existing PathMtuState assertion.
Adds two #[cfg(test)] accessors on Node (path_mtu_lookup_get /
path_mtu_lookup_insert) and bumps handle_mtu_exceeded to
pub(in crate::node) for direct test invocation.
The B3 path_mtu_lookup plumbing landed without updating the
windows_tun::run_tun_reader signature or its inner handle_tun_packet
call, breaking the Windows build. Linux/macOS variants and
TunWriter (Windows) were already plumbed; this brings the Windows
reader into line.
No behavioral change on any platform.
Forward-merges the squashed per-destination TCP MSS clamping work
(master `ae60743`) into next.
Auto-merge of source files clean. Two pre-existing test assertions
on the next-side discovery tests were updated as part of merge
resolution to reflect the target-edge MTU fold introduced by the
merged-in commit:
- test_transit_forwards_when_mtu_sufficient: 1400 → 1280 = min(target-edge 1280, transit 1400)
- test_response_path_mtu_four_node_chain: 1350 → 1280 = min(target-edge 1280, transits 1350+1500)
Resulting next tree is bit-for-bit identical to the pre-squash next
state previously verified by full local CI sweep (29 + 1 next-only
suites pass on `6a8b519` in 23m 41s).
Adds source-side TCP MSS clamping informed by per-destination path
MTU learned via discovery, with a conservative IPv6-minimum-derived
ceiling for cold flows where discovery has not yet completed. Closes
the multi-hop default-config TCP wedges observed in production where
a sender's local-floor MSS exceeds what some intermediate forwarder
hop is willing to carry: silent drops, no PTB feedback through the
userspace TUN to the kernel TCP stack, retransmits at the same too-
large MSS, application connection times out.
## Architecture
A new `Arc<RwLock<HashMap<FipsAddress, u16>>>` field
`path_mtu_lookup` on Node mirrors the per-destination path MTU in a
form accessible from sync TUN reader/writer threads. A new
`per_flow_max_mss` helper in `src/upper/tun.rs` reads the lookup at
SYN-clamp time and returns the appropriate ceiling for the flow.
Three write sites populate `path_mtu_lookup`:
1. **Discovery originator branch** of `handle_lookup_response`:
the path MTU bottleneck accumulated through the reverse path
lands here when a LookupResponse arrives at the originator.
Same value also lands in `coord_cache` per the existing
`insert_with_path_mtu` API.
2. **FMP peer-promotion seed** (`seed_path_mtu_for_link_peer`):
when an FMP link-layer peer is promoted to active, the local
outgoing-link MTU on the peer's transport seeds the lookup.
Tighter existing values (learned via discovery) are preserved;
the seed only writes when no entry exists or the existing
value is looser than the link MTU. Without this seed,
directly-configured peers (auto_connect / static peer config)
would leave `path_mtu_lookup` empty for their FipsAddress
because the FSP session establishes without ever issuing a
LookupRequest.
3. **Target-edge fold at `send_lookup_response`**: when a node is
the discovery target, it folds its own outgoing-link MTU to
the response's next-hop into `path_mtu` before sending.
Without this fold, the response leaves the target with
`path_mtu = u16::MAX` and only intermediate transits min-fold;
the target's first reverse-path hop is never represented in
the bottleneck calculation. Refactored the existing transit-
side min-fold into a shared `apply_outgoing_link_mtu_to_response`
helper called from both sites.
## Read-side: per_flow_max_mss
Two TUN call sites consume the lookup:
- Outbound `handle_tun_packet` clamps SYN MSS using packet[24..40]
(IPv6 destination) as the lookup key.
- Inbound `TunWriter::run` clamps SYN-ACK MSS using packet[8..24]
(IPv6 source).
When the lookup contains a learned value, the helper computes
`min(global_max_mss, effective_ipv6_mtu(path_mtu) - 60)` where 60
is IPv6 (40) + TCP (20) headers and `effective_ipv6_mtu` accounts
for the FIPS encapsulation overhead.
When the lookup is empty for a destination — the cold-flow case —
the helper returns `min(global_max_mss, IPv6-minimum-derived
ceiling)`. RFC 8200 mandates every IPv6 path accept ≥1280-byte
packets, so the IPv6-minimum-derived MSS (1280 - 77 - 60 = 1143)
fits any compliant path. Without this conservative ceiling, the
first SYN to a destination with no learned path MTU exits the TUN
at the kernel-natural MSS (TUN MTU - 60), and the application
connection wedges silently before discovery completes for a
corrected second SYN to fire. The fix is provably safe: the
ceiling is taken with `min` against the local global so operators
with even tighter local floors are never loosened upward.
Subsequent flows pick up the actual learned per-destination value
once discovery (or the FMP-promotion seed for direct peers)
populates the lookup.
## Diagnostic logging
All write and read sites emit instrumentation suitable for
operators bisecting a wedged path:
- `debug!` log on every `path_mtu_lookup` write (discovery
originator path and FMP-promotion seed path), showing the
FipsAddress, written value, prior value, and post-write map
size. `warn!` on poisoned-lock failure path.
- `trace!` log per `per_flow_max_mss` call covering every
fall-through branch (wrong addr_bytes length, non-fd::/8
prefix, lookup poisoned, no entry for destination, empty-lookup
conservative ceiling) and the success path. trace level filters
out under normal log settings; capture with
`RUST_LOG=info,fips::node::handlers::discovery=debug,fips::upper::tun=trace`.
## Tests
15 new unit tests across 3 files:
- `per_flow_max_mss` (8 tests in `src/upper/tun.rs::tests`):
empty-lookup conservative ceiling, empty-lookup global-smaller
floor, learned-value-overrides-conservative, per-destination
smaller, per-destination larger capped by global, non-fips
addr, short addr slice, per-destination independence.
- `seed_path_mtu_for_link_peer` (4 tests in
`src/node/tests/unit.rs`): seed when empty, keep tighter
existing, tighten looser existing, no-op for unknown
transport.
- Discovery integration (3 tests in `src/node/tests/discovery.rs`):
apply_outgoing_link_mtu_to_response on unknown peer no-op,
two-node target-edge fold (path_mtu reflects target-edge link),
three-node chain transit min-fold (existing test, updated for
target-edge inclusion).
Two pre-existing discovery tests had assertions updated to
account for the target-edge fold:
- `test_response_path_mtu_two_node`: previously asserted
`u16::MAX` (no transit to min-fold); now asserts 1280 (the
test transport MTU, folded in by send_lookup_response).
- `test_response_path_mtu_four_node_chain`: previously asserted
1350 (transit MTUs only); now asserts 1280 (target-edge MTU
is the bottleneck).
- `test_transit_forwards_when_mtu_sufficient`: previously
asserted 1400 (transit MTU only); now asserts 1280 (target-
edge MTU is the bottleneck).
## Verification
Local CI on this commit: 29/29 suites pass, 1105 lib tests pass,
clippy --all-targets --all-features -D warnings clean, cargo fmt
clean. Production deploy verified via trace capture across the
managed fleet: cold-flow conservative ceiling branch fires on
first SYN, learned-lookup branch takes over once discovery
completes, both behaviors observable end-to-end at the SYN MSS
on the wire.
No wire-format change. No config-format change.
Adopted ephemeral UDP transports created by adopt_established_traversal()
default to UdpConfig::default() (MTU=1280, IPv6 minimum) when the
bootstrap runtime hands a socket without an explicit transport_config
override. This is by design: NAT-traversal middlebox MTU is unpredictable
and the IPv6 minimum is the only value guaranteed by spec to survive
arbitrary paths.
Add an explanatory comment at the call site so future readers find the
rationale without spelunking through ISSUE-2026-0013, and so any future
change to the inheritance behavior is a deliberate decision rather than
an accidental refactor.
No behavior change.
Default-config TCP flows between fips peers were stalling completely
(cwnd-pinned, 0 bps for 10s+) on a non-trivial fraction of restarts.
Reproducible with iperf3 between any two peers.
Root cause: `Node::transport_mtu()` iterated `self.transports.values()`
(HashMap with default RandomState hasher) and returned `handle.mtu()`
of the first one whose `is_operational()` returned true. Two stacked
sources of non-determinism stacked on each other: HashMap iteration
order is randomized per-process via RandomState, and async transport
`.start()` completion order races each daemon restart.
The returned value drives the TCP MSS clamp ceiling computed once at
TUN init (src/upper/tun.rs:501-524) and stored as immutable max_mss
in the reader/writer thread state. When the picker landed on a
transport with MTU > 1357 (any non-UDP-1280 in the standard fleet
defaults), `max_mss > 1220` (kernel's natural fips0-MTU-derived MSS),
the daemon's clamp was silently a no-op, and the kernel emitted
1220-byte segments. Those wrap into 1280-byte IPv6 → 1357-byte fips
datagrams that exceed UDP-1280 transports at any forwarding hop,
causing silent drops with no PTB feedback to the kernel TCP stack.
Fix: return min across operational transports instead of first-iterated.
With UDP-1280 in the configured set (the common case),
`transport_mtu = 1280`, `max_mss = 1143 < 1220`, daemon's clamp
engages, MSS=1143 reaches the wire, packets fit, throughput recovers.
Empirical green light from a single-UDP-config end-to-end test:
iperf3-without-`-M` recovered to ~21 Mbps with no operator-side nft
TCPMSS rules.
Adds three unit tests:
- transport_mtu_returns_min_across_operational: pin selection to
smallest MTU when multiple operational transports differ.
- transport_mtu_fallback_when_no_operational_transports: 1280 fallback.
- transport_mtu_min_with_single_operational: trivial single-transport
case.
The `effective_ipv6_mtu` field reported by `fipsctl show status` was
also racy (consequence of the same bug); fixed by this change as a
side effect.
The local ci-local.sh and the GitHub CI clippy invocations both used
`cargo clippy --all -- -D warnings`, which only checks lib + bin
targets. Test code, integration tests, and benches were not lint-gated.
Three pre-existing clippy errors lurked in test modules as a result
(two field_reassign_with_default in config tests, one
items_after_test_module in stun.rs).
Tighten both invocations to `cargo clippy --all-targets --all-features
-- -D warnings` so the gate covers everything cargo can build, and
fix the three exposed errors:
- src/config/mod.rs: rewrite two test-only `Config::default()` +
field-reassign sites to struct-update syntax.
- src/discovery/nostr/stun.rs: move helper `random_txn_id` above the
`#[cfg(test)] mod tests` block.
Also adds a dedicated Clippy job to the GitHub CI workflow so the
strict gate runs on every PR (the workflow had no clippy job before;
clippy ran only via testing/ci-local.sh on operator machines).
No behavior changes; lint hygiene + CI hardening only.
Under `node.discovery.nostr.policy: open`, the per-tick auto-dial in
`queue_open_discovery_retries` was supposed to pick up adverts cached
from the relay subscription backlog at startup, but in practice only
adverts arriving live (after the daemon was up) were being dialed.
Backlog adverts sat in the in-memory cache until they aged out.
Adds a one-shot startup sweep that runs once per daemon start, gated
identically to the per-tick sweep (`enabled` && `policy == open`),
after a configurable settle delay so the relay subscription backlog
has time to populate the advert cache. The sweep iterates the cache
with the same skip-filters as the per-tick path (statically-configured
peers, already-connected, retry-pending, connecting) plus a tighter
age filter: only adverts whose `created_at` is within
`startup_sweep_max_age_secs` of now are queued.
Two new config fields under `node.discovery.nostr`:
- `startup_sweep_delay_secs` (default 5)
- `startup_sweep_max_age_secs` (default 3600 = one hour)
Both are only consulted when `policy == open`; under any other
policy the sweep is a no-op.
Adds diagnostic logging to the open-discovery sweep so operators can
verify what the auto-dial path is doing on each daemon bring-up:
info-level on each retry-queued enqueue (with peer short-npub and
advert age), and a one-line summary on every startup sweep and on
any per-tick sweep that queues at least one retry. The summary
breaks down skipped candidates by reason (age, configured, self,
already-connected, retry-pending, connecting, no-endpoints,
invalid-npub) — currently the path was silent so there was no
operator-visible signal that the cache iteration was running.
Refactors the existing `queue_open_discovery_retries` body into a
shared `run_open_discovery_sweep(max_age_secs, caller)` helper so
the per-tick and startup paths share filter/queue logic and only
differ in the age filter and log label. Surfaces `created_at` from
`NostrDiscovery::cached_open_discovery_candidates` (return tuple
extended) so the age filter has the data it needs.
Three new unit tests in `config::node::tests` cover the new defaults,
YAML override round-trip, and partial-YAML default fallback.
The Nostr overlay advert publisher serialized `transport: tor`
endpoints as a bare `<onion>.onion` hostname with no port. The Tor
address parser requires `<host>:<port>` form and rejected the bare
shape with `expected host:port`. Any peer receiving a Tor-only
advert went into a persistent retry-fail loop on jittered backoff
until the advert aged out of the discovery cache. The bug had been
latent for as long as Tor adverts have been published on Nostr, and
was masked in deployments where every node also advertised a
non-Tor transport (peers fell through to the working endpoint).
Surfaced first on a deployment where Tor was the only advert path.
Publisher now emits `<onion>.onion:<port>` using a new
`transports.tor.advertised_port` config field that defaults to 443,
matching the Tor `HiddenServicePort 443 127.0.0.1:<bind_port>`
convention. Operators whose torrc uses a non-default virtual port
can override.
Adds a unit test that pins the publisher/parser contract: formats
the advert exactly as the publisher does and asserts `parse_tor_addr`
accepts the result; asserts the bare-onion form (the bug) does not
parse, catching any future regression that drops the port again.
Parser is unchanged (already correct).
Martti MalmiandJohnathan Corgan
