The identifier "discovery" named three unrelated subsystems; the FMP
overlay coordinate-lookup subsystem is now consistently "lookup". This
finishes the concept-#1 rename across the shell, config, and metric
layers left after the earlier proto-layer rename:
- Handler module node::handlers::discovery -> node::handlers::lookup, and
reset_discovery_backoff -> reset_lookup_backoff.
- The Node lookup-engine field Node.discovery -> Node.lookup, renamed by
resolved binding so the metrics().discovery and node.discovery config
paths are left untouched.
- The lookup metric types DiscoveryMetrics -> LookupMetrics,
DiscoveryStatsSnapshot -> LookupStatsSnapshot, and Metrics.discovery ->
Metrics.lookup.
Two surfaces cross a stability boundary and ship behind a compatibility
window, both marked in-code for removal at the v2 cutover:
- The control-socket metric family is dual-emitted under both "discovery"
(deprecated alias) and "lookup" so existing dashboards keep working.
- The node.discovery.* config table is split into node.lookup.* (mesh
lookup scalars) and node.rendezvous.* (nostr/lan peer rendezvous).
NodeConfig does not deny unknown fields, so a naive rename would make a
deployed node.discovery: block deserialize into nothing and silently
revert every setting to default. A deprecated all-Option
DiscoveryConfigCompat field captures a legacy block and a new post-parse
Config::normalize_deprecated_keys pass folds it into the new tables with
a one-time deprecation warning.
Flip the packaged fips.yaml templates to the new keys, add legacy/new/
scalar compat parse tests, and record the split and deprecations in the
CHANGELOG. Behavior-neutral; fmt/clippy clean, lib suite green.
The shared fips.yaml ships ethernet.wan.interface: "eth0", the default
WAN port on OpenWrt 24 and earlier. OpenWrt 25 (DSA) boards, which the
.apk package targets, name the WAN port "wan" instead. Rewrite the
staged copy at build time so the as-installed config binds the Ethernet
transport to the right port out of the box, without maintaining a second
copy of the config file. The .ipk package keeps "eth0".
Update the openwrt README to document eth0 (24) vs wan (25/DSA) and add
a CHANGELOG entry.
Prepare the source tree for the v0.4.0 release cut, leaving the version
at 0.4.0-dev (the version bump rides a separate release-candidate
commit).
- CHANGELOG: backfill the missing entry for the route-class transit
counters and fipstop routing-tab reorg, then reorganize the Unreleased
block from a flat per-commit list into topic-grouped subsections
mirroring the 0.3.0 entry (coalescing interim fixes into net-effect
descriptions without dropping technical detail), and stamp it as
[0.4.0] - 2026-06-21 with a fresh empty Unreleased block. The date is
provisional and reconfirmed at the final tag.
- Release notes: refresh both RELEASE-NOTES.md and the versioned archive
(kept byte-identical) to cover the OpenWrt .apk packaging, the Nix
flake, the macOS self-traffic checksum fix (#117), and the route-class
transit counters.
- README: bump the status badge to v0.4.0 so it matches the prose.
Claude-Session: https://claude.ai/code/session_01A2pYfSypNmmG4HyHwZuLex
Add OpenWrt .apk packaging for OpenWrt 25+, where apk-tools is the
mandatory package manager. The existing .ipk continues to cover OpenWrt
24.x and earlier. Built SDK-free like the .ipk: it reuses the
cargo-zigbuild cross-compile and the shared installed-filesystem payload
under openwrt-ipk/files, and assembles the ADB container with the
official `apk mkpkg` applet from apk-tools 3.0.5 built from source, so no
OpenWrt SDK image is needed.
The package-openwrt workflow is refactored so a single compile-binaries
job cross-compiles and strips each arch once; both the .ipk and .apk
packagers consume the binaries via a new --bin-dir flag instead of each
recompiling. A build-apk job (aarch64, x86_64) builds apk-tools,
packages, and structurally verifies the .apk with `apk adbdump`. Releases
now publish .apk artifacts and checksums alongside .ipk; the release
download is scoped to fips_* so the shared raw-binary artifacts are not
swept into the published release. apk-version.sh maps a release tag or
commit height to an apk-tools-valid version, covered by a case-table test.
Packages are unsigned, installed with `apk add --allow-untrusted`,
matching the .ipk posture.
Also fix the OpenWrt control socket: the init script now pre-creates
/run/fips before starting the daemon, the procd equivalent of the systemd
unit's RuntimeDirectory=fips. Without it, on a fresh boot the daemon
resolves its control socket to /tmp (since /run/fips does not yet exist),
while fips-gateway later creates /run/fips for its own gateway.sock,
leaving fipsctl/fipstop resolving a /run/fips/control.sock the daemon
never bound.
Self-addressed TCP/UDP connections to a node's own <npub>.fips address
half-opened and hung on macOS. macOS routes self-traffic as loopback (a
LOCAL route via lo0), which defers the transport TX checksum, but the
point-to-point utun then egresses the packet into the daemon with only
the pseudo-header partial checksum present. The hairpin path added in
9a9e90a re-injected these verbatim, so the local stack dropped every
segment whose checksum MSS clamping didn't happen to rewrite: the
SYN/SYN-ACK got through (clamping recomputes them) but the bare ACK,
data, and FIN were dropped for a bad checksum, leaving the listener
stuck in SYN_RCVD.
Recompute the TCP/UDP checksum for self-addressed packets on the hairpin
path before re-injection, completing the self-delivery 9a9e90a started
(which only covered ICMP and the TCP handshake). Linux is unaffected: it
loops self-traffic via lo before the TUN, so the hairpin branch never
fires and checksums are already valid.
Confirmed on macOS: a self-connect that previously timed out now
completes in ~9ms with payload delivered.
Add a flake that builds all four binaries (fips, fipsctl, fips-gateway,
fipstop) on Linux and macOS, pinning the exact toolchain from
rust-toolchain.toml (1.94.1 + rustfmt/clippy) via fenix so Nix builds
match CI and the AUR/Debian packaging.
Wire up the build-time native deps the source tree needs: pkg-config and
bindgenHook (libclang) for the rustables/libdbus-sys bindgen step, and
dbus for bluer's BLE support. autoPatchelfHook rewrites the binary RPATHs
so the daemon resolves libdbus-1.so.3 and libgcc_s.so.1 from the Nix store
at runtime — without it `fips` fails to load on NixOS, which has no global
/usr/lib.
Outputs: packages.{default,fips}, apps for each binary, checks.fips, and a
devShell with the pinned toolchain plus cargo-edit. Tests are skipped in
the package build since they exercise TUN devices, raw sockets, and mDNS
that aren't available in the sandbox, mirroring the AUR/Debian packaging.
Verified end-to-end in a pure Nix store (nixos/nix container): nix build,
nix flake check, and running all four binaries succeed against the
committed flake.lock.
Documented across the install and developer docs: a Nix / NixOS section in
packaging/README.md, the from-source guide in docs/getting-started.md
(noting the flake produces binaries only, with NixOS system integration
through the system configuration rather than the installer), the CHANGELOG,
README, CONTRIBUTING, and the v0.4.0 release notes.
Co-authored-by: Johnathan Corgan <johnathan@corganlabs.com>
A packet destined for our own mesh address reached the TUN reader on
macOS (utun egresses self-traffic into the daemon despite the lo0 host
route) and was pushed onto the mesh outbound path, where it was dropped
for lack of a session/route to self. Hairpin self-addressed packets back
to the TUN writer instead, so ping6 and connections to our own
<npub>.fips address are delivered locally.
On Linux the kernel already loops self-traffic via `lo` before it reaches
the TUN, so the branch never fires there; the check is kept unconditional
as a daemon-level delivery invariant and to keep it covered by Linux CI.
Bring the source tree to its finished v0.4.0 content state ahead of the
release candidate. Documentation, changelog, release notes, and
packaging metadata only; no code or version-string changes.
CHANGELOG.md: backfill the operator-visible changes that landed since
v0.3.0 (the show_metrics scraper query and fipsctl stats metrics, the
discovery dedup-cache-full counter, the off-rx_loop control read
surface and its new daemon-resolved fields, the fipstop TUI overhaul,
the TCP inbound cap now honoring max_connections, host-map hot-reload,
log-noise demotions, and the net bug fixes), topic-grouped rather than
replayed per commit; intra-cycle fixes folded into their feature
entries; the duplicate Unreleased Fixed section merged into one.
Release notes: author docs/releases/release-notes-v0.4.0.md to the
operator-upgrade bar and mirror it byte-for-byte into the root
RELEASE-NOTES.md. Attribute each feature to its author in Contributors
(Nym transport and the mixnet demo to @oleksky, opt-in mDNS LAN
discovery to @mmalmi).
README.md: add the Nym transport to the support matrix (Linux, macOS,
Windows; OpenWrt pending verification), the multi-transport bullet, and
the "What works today" list; fold mDNS LAN discovery into the existing
Nostr-discovery bullets; refresh the status narrative to v0.4.0 over a
global, public test mesh.
packaging/common/fips.yaml: add commented Nym transport and mDNS LAN
discovery example stanzas with verified field names and defaults.
Cargo.toml: add homepage, keywords, and categories crate metadata.
docs/reference: update the cli-fips version example to the released
form; document the new control-socket output fields and the typed
RejectReason families in control-socket.md; sync cli-fipstop.md with the
overhauled TUI keybindings.
The connect_refused stat counter (the Refused line in fipstop) was
defined but never incremented: every SOCKS5 connect failure recorded
socks5_errors instead, so the counter sat at zero and the operator-facing
gauge was permanently misleading. Both the synchronous connect path and
the background connect_async task now count a genuine SOCKS5 REP=0x05
refusal as connect_refused and every other failure as a socks5_error,
distinguishing them precisely via tokio_socks::Error::ConnectionRefused.
Extends the mock SOCKS5 server with a configurable reply code and adds
two tests covering the refused and general-failure paths.
Add an outbound-only Nym mixnet transport that tunnels FMP peer links
through a local nym-socks5-client SOCKS5 proxy into the Nym mixnet. It
structurally mirrors the Tor SOCKS5 transport (connection pool,
connect-on-send background promotion, FMP-v0 framing reused from TCP)
with the onion, inbound-listener, and control-port machinery removed.
Wires the transport through the full TransportHandle dispatch, NymConfig
(standard transport-instance pattern), and node instantiation, and
surfaces its counters in fipstop. Includes a mock SOCKS5 harness and unit
coverage for the address-parsing paths.
Also adds an isolated single-container example
(examples/sidecar-nostr-mixnet-relay/) demonstrating FIPS peering across
the mixnet end to end. No new crate dependencies: tokio_socks, socket2,
and futures are already pulled in by the Tor transport.
The inbound FilterAnnounce FPR cap rejects filters whose false-positive
rate (fill^k) exceeds the configured maximum. On the fixed 1 KB / k=5
filter, 0.05 corresponds to fill 0.549 (~1,300 reachable entries), and
the busiest nodes' aggregates were beginning to hit that ceiling as the
mesh grew. Raise the default to 0.10 (fill 0.631, ~1,630 entries) to
restore headroom toward the fixed-filter capacity limit. A saturated or
poisoned filter is ~100% FPR and remains rejected, so the antipoison
gate is not materially weakened.
Updates the config default, the config-reference and bloom-filter
design docs, and the changelog.
Ignore duplicate or counter-regressed ReceiverReports before updating
RTT, loss, goodput, or ETX, so a delayed or reordered report can no
longer poison link metrics. Compute the RTT-from-echo sample with
checked timestamp arithmetic and reject zero, negative, or out-of-range
results instead of risking wrap or underflow on untrusted wire values.
On the sender side, when receiver dwell time overflows the u16 wire
field, suppress the timestamp echo (send 0) and saturate dwell to
u16::MAX rather than truncating, so a bogus small RTT cannot be formed.
Adds duplicate, out-of-order, wrapped-add, and future-dated (checked_sub)
sample tests, asserts loss and goodput stay unchanged on a dropped
duplicate, and covers the dwell-overflow echo suppression. Documents the
behavior in the MMP design note and CHANGELOG.
Co-authored-by: Johnathan Corgan <johnathan@corganlabs.com>
Record the changes that landed since the last changelog update: the
dual-auto_connect traversal-session election, the FMP link-layer rekey
reliability fixes (bounded msg1 retransmission with rekey-aware
heartbeat, and authenticate-before-cutover), the in-process loopback
test transport and progress-aware convergence wait that remove CI flake
classes, the local/GitHub CI suite-parity and single-source toolchain
selection, and the packaging change shipping the config as an example
that postinst seeds when absent.
Store node counters in an atomic metric registry read through &self, and
introduce a shared NodeContext bundle holding the effectively-immutable
fields (config, identity, startup epoch, capability limits). Source the
immutable config and identity reads across the receive hot path, the
handshake/session/mmp/encrypted state machines, and the discovery, tree,
bloom, retry, and lifecycle modules through the context accessors rather
than direct field reads. The Node fields and the context are rebuilt in
lockstep at every mutation site.
PeerRecvDrain::drop previously called std::thread::join on the worker
thread synchronously. The worker uses packet_tx.blocking_send on a
tokio mpsc Sender, which internally parks the worker via
tokio::block_on on the same current_thread runtime that drives
rx_loop. Calling join from inside remove_active_peer (which runs on
the runtime thread, the runtime's sole driver) created a circular
wait:
- rx_loop blocks in libc futex via Thread::join
- the worker being joined cannot observe the stop flag because the
runtime that polls it is the very thread now blocked joining it
- all other PeerRecvDrain workers park on the same runtime via
block_on, so a single peer's removal wedges every worker on the
daemon
The /proc snapshot from a production wedge showed exactly this
shape: 107 of 108 threads in futex_do_wait, 101 of them named
fips-peer-drain. fipsctl became unresponsive (EAGAIN on control
socket), SIGTERM was ignored, and Docker SIGKILLed the container
after the 10 s grace period. Two confirmed wedges on the public
test deployment (52 min and 23 min uptime), plus a third on the
admission-gate-Msg2-silent-drop build at 2 min 21 sec — all ending
with the identical "Peer removed and state cleaned up
tree_changed=false" final log line preceding total silence.
Fix: detach the std::thread instead of joining. The stop flag plus
self-pipe write already signal the worker to exit; the worker's
kernel-level libc::poll inside the drain loop sees the wake, checks
the flag, exits, and the OS reclaims the thread state independently
of the JoinHandle being dropped.
The trigger was statistically amplified by aggressive multi-npub-
from-one-NAT peer reconnect patterns at the moment of the 30 s
link-dead-timeout peer-removal, but not bounded to them. Any
peer whose disconnect happens with the per-peer drain worker
parked in block_on can fire the bug. The admission-gate work
that landed earlier in this branch line compressed more handshake
work per rx_loop tick, increasing the rate at which workers are
parked in block_on and so reducing time-to-wedge — but the
underlying bug pre-dated the admission gate and pre-dated this
fix branch.
The deployed wedged daemon is mitigated operationally by blocking
the trigger IP at the host firewall; this commit removes the bug
class entirely.
Bring the runtime peer-list refresh and opt-in mDNS LAN discovery work
on master into the receive-path RejectReason / reloadable-config
integration branch. Code files auto-merge clean; the only conflict is
the CHANGELOG Unreleased section, resolved as the union of both sets of
entries.
Add scoped mDNS / DNS-SD discovery for peers on the same local link,
giving sub-second pairing without a relay or NAT-traversal roundtrip.
A node advertises its npub, protocol version, and an optional network
scope over link-local multicast, and browses for matching adverts to
initiate Noise handshakes against same-LAN peers.
LAN discovery is disabled by default; operators enable it with
node.discovery.lan.enabled: true. Default-off avoids reintroducing a
per-LAN identity broadcast on nodes that have deliberately disabled
other discovery channels, and avoids any multicast surprise on upgrade.
The startup advertised-port picker now excludes bootstrap transports
and selects a non-bootstrap operational UDP transport with a stable
lowest-id selector, so the advertised port is deterministic across
restarts rather than dependent on HashMap iteration order. This
matches the per-dial transport selection used for discovered peers.
Co-authored-by: Johnathan Corgan <johnathan@corganlabs.com>
Add Node::update_peers for runtime peer-list refresh. It re-derives the
active peer connections from a new peer configuration, adding newly
configured peers and removing those no longer present, while keeping
links to peers that remain in the set rather than tearing every
connection down. The call returns an UpdatePeersOutcome summarizing the
added, removed, and retained peers.
PeerAddress gains a seen_at_ms recency field (with_seen_at_ms). Active
path selection now sorts address candidates by recency so the most
recently observed address wins when concurrent path probes race.
complete_rekey_msg2 now returns the remote peer's startup epoch
alongside the new Noise session, letting the rekey path detect a peer
restart and clear stale session state. A stale FSP session is cleared
when a peer restart is detected during FMP rekey or cross-connection
promotion, so the session-layer map no longer lingers out of sync with
the freshly promoted peer.
Per-tick work budgets bound the connection churn in a single node tick
(MAX_DISCOVERY_CONNECTS_PER_TICK, MAX_RETRY_CONNECTIONS_PER_TICK,
MAX_PARALLEL_PATH_CANDIDATES_PER_PEER); work beyond a tick's budget is
deferred to the next tick rather than discarded.
Co-authored-by: Johnathan Corgan <johnathan@corganlabs.com>
Introduce a typed RejectReason enum and a NodeStats::record_reject
dispatch so every receive-path rejection-and-return site bumps a
machine-readable per-subsystem counter while keeping its operator-facing
log line. The top-level variants mirror the existing NodeStats subsystem
split (Tree, Bloom, Discovery, Forwarding) and add Handshake, Session,
Mmp, and Transport categories; HandshakeStats, SessionStats, and MmpStats
are new sub-stats.
Wired clusters: tree and MMP outbound sign-failure; the FSP session
unknown-session and state-machine cluster; the Noise IK handshake
state-machine cluster (msg1/msg2); and the decode / crypto / cap /
semantic tail across bloom, discovery, forwarding, mmp, and tree. The
TreeStats::ancestry_invalid counter, present since the scaffold but never
incremented, is now bumped from the validate_semantics ancestry rejection.
Several handshake, MMP, tree, and discovery paths that previously had no
counter at all are now counted, including the send_lookup_response
no-route drop (DiscoveryStats::resp_no_route).
Existing direct counters at the bloom / discovery / forwarding sites are
retained alongside the new dispatch while the rollout is in progress (the
bloom_poison tests expect the transitional +2 delta); a later change
collapses the duplicate increment.
Add per-direction pool_inbound/pool_outbound counters to TcpStats and
TorStats, updated at every pool-insert, receive-loop-exit, transport-stop,
and send-failure removal site. Compare the max_inbound_connections cap
against pool_inbound rather than the combined pool length, so outbound
connect-on-send connections no longer consume the operator-facing inbound
budget. The configuration field name and operator semantics are preserved;
only the cap-check comparison and accounting change.
Move the max_peers cap check in handle_msg1 forward, from the late
check inside promote_connection (which fires after Msg2 has already
been built and put on the wire) to an early position after identity
verification but before index allocation and the Msg2 send. When the
gate fires for a net-new identity, the Msg1 is silent-dropped — no
response goes back to the peer, no AEAD compute or wire bytes are
spent.
Bypass preserved for known peers (reconnect / cross-connection): if
the sender's NodeAddr is already in self.peers, or if a pending
outbound connection is in flight to the same identity, the gate is
skipped so legitimate maintenance traffic continues to work. The
late check inside promote_connection is intentionally retained as
defense-in-depth against future call sites or a disconnect racing
between the early-gate decision and promotion.
Wire-cost rationale: a 45 s tcpdump at saturation observed ~3.6
cap-denials/s steady-state, each previously paying the full Noise IK
responder crypto + Msg2 (~104 B) on the wire before being rejected.
The bigger value is cleaner peer-side semantics — the peer no longer
sees a fake-completed handshake whose data frames subsequently fail
decryption locally.
Two new unit tests cover the cases:
- handle_msg1_silent_drops_at_cap_for_new_peer drives a wire-pumped
Msg1 from a fresh identity into a saturated node and asserts no
Msg2 reaches the sender socket. Stash-verifies as FAIL on the
pre-fix tree (Msg2 hits the wire) and PASS post-fix.
- handle_msg1_admits_existing_peer_at_cap drives a Msg1 from an
identity already in self.peers and asserts the gate does not evict
it. This is a regression check (the no-gate tree behaves the same
way here, but the test guards against an accidental future gate
that breaks known-peer admit).
Brings two structural fixes landed on maint:
- compute_mesh_size: explicit parent skip in the children loop, so the
disjoint-subtree invariant no longer depends on peer_declaration cache
freshness.
- max_peers: outbound connection-initiation gated on the cap (auto-reconnect
retries, Nostr-mediated discovery established adoption, and both sides
of the NAT-traversal punch sequence). Inbound msg1 admission gate
unchanged.
node.limits.max_peers was honored only on inbound msg1 admission
(handshake.rs handle_msg1 returns PeerLimitExceeded when peers.len
is at the cap). Four outbound initiation paths proceeded unconditionally
at capacity: auto-reconnect retries (process_pending_retries),
Nostr-mediated discovery's BootstrapEvent::Established adoption
(poll_nostr_discovery), NAT-traversal punch initiation (the outgoing
side of the offer/answer/punch sequence in the Nostr discovery
runtime), and NAT-traversal punch response (the incoming side of the
same sequence). A saturated node burned CPU, UDP probes, STUN
observations, and Nostr relay traffic on connections that the inbound
gate would reject the moment they reached msg1.
Introduce Node::outbound_admission_check (peers.len < max_peers, or
true when max_peers == 0 as the no-cap sentinel) and gate the four
paths. The discovery runtime lives in a separate task and does not
hold a Node reference; bridge via an Arc<AtomicBool> the runtime
reads and Node refreshes once per tick from outbound_admission_check.
The atomic granularity is intentionally loose: one-tick lag is
acceptable because the inbound msg1 gate continues to be the
authoritative cap, and in-flight handshakes started below the cap
are allowed to complete.
Inbound gate at handshake.rs is unchanged.
The mesh-size estimator's children loop relied on the cached
peer_declaration(parent_id).parent_id() != my_addr check to exclude
the parent. That cached view briefly disagrees with our own latest
my_declaration().parent_id() during the window between a local
parent-switch and the new parent's next inbound TreeAnnounce: the
peer-declaration cache still names us as the parent's parent, so the
parent is iterated as if it were a child and its (typically dominant)
bloom cardinality is added a second time. Symptom: estimated mesh size
displayed in fipsctl show status and fipstop nearly-but-not-exactly
doubles during tree rebalancing.
Make the invariant structural with an explicit peer_addr == parent_id
skip at the head of the children loop. Per-peer 500 ms rate-limiter
and overall recompute cadence are unchanged.
Adds a regression test that constructs the stale-peer-declaration
scenario directly and asserts the parent is not double-counted.
Closes the eventually-consistent gap in spanning-tree state
distribution. Every existing send_tree_announce_to_all call site
gates on a local state-change event (parent switch, self-root
promotion, ancestry change, peer promotion, parent loss). Once a
partition latches — for example a parent-switch announce stranded
in the brief cross-init handshake swap window, where the announce
arrives on a session-index whose decrypt-worker entry has been
unregistered — neither side's state changes again, so neither
side ever re-broadcasts. The existing 60 s check_periodic_parent_reeval
was a re-evaluation, not a re-broadcast: it short-circuited
silently on no-change. Production-side healing depended on
incidental link churn; lab harnesses with stable docker-bridge
links had no equivalent path.
Add a final else branch that fires send_tree_announce_to_all
unconditionally on the no-change path, alongside the existing
switch and self-promote arms. Receivers coalesce by sequence
comparison (ParentDeclaration::is_fresher_than) and short-circuit
at the `if !updated` gate in handle_tree_announce; same-sequence
repeats drop silently with no cascade. The per-peer 500 ms
rate-limiter is well below this 60 s cadence and does not suppress
the heartbeat broadcast.
The fix is a general protocol-robustness improvement: it addresses
any in-flight TreeAnnounce loss class, not only the specific
cross-init swap-window drop site.
testing/static/scripts/rekey-test.sh BASELINE_CONVERGENCE_TIMEOUT
60 -> 65 so a partition healed by the periodic broadcast at T+60
lands inside the convergence window. wait_for_full_baseline
early-exits on PASS, so successful reps see no extra wall-clock.
The tick body's per-peer check_* loops (heartbeats, bloom
announces, MMP reports, tree announces) called transport.send
for every active peer, which on TCP/Tor fell through to a 5 s
connect-on-send wait for any peer whose pool entry was not yet
established. That wedged the entire tick body for the full
connect_timeout_ms per unreachable peer; under post-restart
convergence on a high-peer mesh, this cascaded into multi-
second tick stalls. On master, the same mechanism also starved
the per-tick control-snapshot republish and pushed fipsctl
queries onto an mpsc fallback that was itself queued behind
the wedged rx_loop, producing the 5-second fipsctl head-of-line
pattern operators observed on loaded nodes.
Gate send_encrypted_link_message_with_ce on
transport.connection_state before the send: proceed only when
Connected; on None, kick off a non-blocking background connect
(idempotent — TransportHandle::connect dedupes against the
connecting pool and spawns the timeout-bounded TcpStream::connect
inside its own tokio task) and fail this send fast with a
clear "transport connection not ready" error. A subsequent
tick retries once the pool has an entry. The reconnect
lifecycle (check_link_heartbeats, process_pending_retries,
poll_pending_connects) is unchanged. The connect-on-send
branch in transport.send_async itself remains in place for
code paths that legitimately need synchronous connect (e.g.,
explicit operator-driven fipsctl connect).
When both peers' Nostr-mediated UDP punches complete within the
same scheduling window, each side's `BootstrapEvent::Established`
event arrives with `is_connecting_to_peer` already true: each side
received an inbound msg1 from the peer's pre-punch outbound
attempt, which created a connecting-state record. The deduplication
skip then fires on both sides, neither installs the fresh
traversal socket as canonical, and the peer-adoption budget
(45 s) expires. Cross-node wall-clock alignment of the skip log
line in observed failures was within ~1 ms — simultaneous dual-
fire under contention, the dual-initiation pattern.
Apply the deterministic NodeAddr tie-breaker already used at
`handlers/handshake.rs:269` for rekey dual-initiation and in
`peer::cross_connection_winner` for cross-connection resolution.
Smaller NodeAddr wins as adopter: enumerate the in-flight
connections whose `expected_identity` points at this peer, tear
them down via the canonical `cleanup_stale_connection` helper, and
fall through to `adopt_established_traversal`. Larger NodeAddr
loses and keeps the existing `continue` semantics; the loser's
in-flight outbound is reconciled by `handle_msg1`'s cross-
connection logic when the winner's fresh msg1 arrives over the
adopted socket.
`cleanup_stale_connection` visibility bumped from module-private
to `pub(in crate::node)` so it is callable from `lifecycle.rs`.
The defensive re-check inside `adopt_established_traversal`
itself is left as-is — after the outer cleanup the winner reaches
it with `is_connecting_to_peer == false`, so the inner skip
won't trip. The `BootstrapEvent::Failed` arm is unchanged: there
is no winning outcome on dual failure, and the existing skip +
retry-schedule semantics are correct.
An FSP session rekey could leave the two endpoints holding different
key sets for a brief window: if a handshake message was lost in
transit, one side rotated to the new keys while the other did not.
Traffic sealed in one key epoch then reached a peer still on the
other epoch and failed to decrypt, producing bursts of AEAD
decryption failures and dropped connectivity until a later rekey
cycle reconverged the pair. Choreographing the cutover order cannot
close this window: any fixed ordering still leaves a skew that
packet reordering widens.
Make rekey correctness independent of cutover timing by overlapping
the key epochs on the receive path. During a rekey transition the
receiver trial-decrypts each frame against every live session it
holds: current, the not-yet-promoted pending session, and the
draining previous session. The K-bit becomes a hint that orders the
trial-decrypt cascade rather than a hard gate, and a frame that
authenticates against the pending session is itself the cutover
signal. No rotation ordering and no packet reordering can then cause
a decryption failure.
The pre-rekey Noise session is held in the `previous` slot until the
peer has demonstrably moved off it. Its drain deadline is anchored
on the most recent frame the peer authenticated against that slot,
refreshed each time the trial-decrypt cascade lands there, rather
than on a fixed wall-clock timer started unilaterally at the local
cutover. A peer that never received the new keys keeps authenticating
against `previous` and the slot stays live; without this, a fixed
timer would erase the only key set that could decrypt the peer's
frames, producing a permanent silent decrypt failure on a live data
path. A peer that never catches up is handled by the existing FSP
session liveness path rather than by silent decrypt failure.
The lost-handshake liveness gap is closed separately by retransmitting
the third rekey handshake message until the peer is confirmed on the
new keys, with a bounded retry budget after which the rekey cycle is
cleanly abandoned and retried on the next timer.
Adds unit tests covering the trial-decrypt cascade (epoch selection,
promotion on pending decrypt, reordered old-epoch stragglers after
cutover, per-slot replay-window integrity), the msg3 retransmission
lifecycle, and the peer-progress-aware drain retirement.
Commit 57a089f6 (the GitHub #102 fix) landed without a CHANGELOG
entry. Add the `[Unreleased]` / `### Fixed` line so the macOS
package-integrity fix is on record before the v0.3.1 cut.
The AUR `fips` and `fips-git` packages did not install the
`fips-dns-setup` and `fips-dns-teardown` helper scripts that
`fips-dns.service` runs. The Debian package ships them to
`/usr/lib/fips/` through the `[package.metadata.deb]` assets, but the
AUR `package()` functions never replicated those install steps, so
`fips-dns.service` failed to start on Arch with "Unable to locate
executable /usr/lib/fips/fips-dns-setup".
Add the two `install -Dm0755` lines to both PKGBUILDs so the AUR
packages match the Debian layout.
Also harden the transition between the `fips` and `fips-git`
packages: each PKGBUILD now declares the other variant's `-debug`
split package as a conflict and opts out of the debug split, so a
stale debug build cannot retain ownership of installed files when
switching between the release and VCS packages. The `aur-publish`
workflow gains a validated `pkgrel` dispatch input so corrected
packaging can be republished against an existing release tag without
retagging.
Fixes#98
(cherry picked from commit 4cf550e23d)
Moves both AEAD layers (ChaCha20-Poly1305, one round per layer per
packet) plus the sendmsg syscall off the rx_loop task onto a per-shard
worker pool, adds per-peer connect(2)-ed UDP with SO_REUSEPORT, and
uses Linux UDP GSO (sendmsg+UDP_SEGMENT — kernel splits one super-skb
into N on-the-wire datagrams in a single TX-stack walk) when packets
in a batch are uniform-size. Same kernel primitive WireGuard's
in-kernel module and BoringTun use to hit 2.5–3.2 Gbps single-stream.
Single TCP stream on a 5-node docker-bridge mesh, 5 x 15 s x P=1:
A→D: 1379 → 2708 Mbps (1.96x, RTT +0.12 ms)
A→E: 1394 → 2663 Mbps (1.91x, RTT +0.11 ms)
E→A: 1406 → 2624 Mbps (1.87x, RTT +0.19 ms)
Static-peer pairs only — every CoV under 3%, 0 outliers, 0% ICMP
loss. The ~+100 µs RTT is the worker queue handoff cost; AEAD +
sendmmsg now run on a separate core in exchange.
What lands:
- src/node/encrypt_worker.rs: std::thread + crossbeam_channel
workers; hash-by-destination dispatch pins a TCP flow to one
worker so wire ordering is preserved; per-worker sendmmsg(2)
batching up to 32; Linux uses sendmsg(2)+UDP_SEGMENT when
packets in a group are uniform-size.
- src/node/decrypt_worker.rs: receive-side mirror. Each shard owns
its session's recv cipher + replay window in a thread-local
HashMap (no shared RwLock/Mutex). Sessions are handed off at
promote_connection and re-registered on K-bit flip / rekey
cutover.
- src/node/handlers/session.rs try_send_session_data_pipelined:
FSP+FMP both seal in-place in the worker on one wire-buffer
alloc; no intermediate inner_plaintext / fsp_payload Vecs.
- src/transport/udp/connected_peer.rs + peer_drain.rs: per-peer
connect(2)-ed UDP socket with SO_REUSEPORT (set on the listen
socket too — without that, EADDRINUSE on activation and every
packet falls back to the wildcard path); the worker sends with
msg_name=NULL and the kernel uses its cached 5-tuple. Tick-
driven activation in handlers/connected_udp.rs, idempotent.
- src/transport/udp/mod.rs: mem::replace the recvmmsg backing buffer
instead of buf.to_vec() per packet — single pointer swap, no
MTU-sized memcpy.
- src/protocol/link.rs SessionDatagramRef: zero-copy borrowed view
used by handle_session_datagram for the bulk local-delivery
path; handle_session_payload takes the borrowed payload
directly (no payload[35..].to_vec()).
- src/transport/mod.rs TransportAddr::from_socket_addr: collapses
the two-alloc from_string(addr.to_string()) pattern to one.
- src/node/handlers/rx_loop.rs: decrypt-fallback drain promoted
ahead of packet_rx in the select! (TCP ACK starvation fix);
interleaved fallback drain every 32 packets inside the rx burst
loop.
- noise::Session: send_cipher_clone / recv_cipher_clone /
recv_replay_snapshot_owned / take_send_counter / accept_replay
so off-task workers can hold a cloned cipher + reserved counter
while the dispatcher keeps replay/counter sequencing serial.
CipherState::cipher_clone returns a refcount-bumped LessSafeKey.
AsyncUdpSocket: AsRawFd so workers issue raw sendmmsg / sendmsg
without going through the tokio reactor.
- Worker pool sizing: both default to num_cpus, overridable via
FIPS_ENCRYPT_WORKERS=N / FIPS_DECRYPT_WORKERS=N. Per-peer
connected UDP can be disabled via FIPS_CONNECTED_UDP=0.
- src/perf_profile.rs: optional per-stage timing reporter under
FIPS_PERF=1 (or FIPS_PIPELINE_TRACE=1). Off by default; zero
overhead when disabled.
- All cfg(unix)-gated. Windows continues on the existing tokio-
based send/recv.
Decrypt worker session lifecycle:
- Node::unregister_decrypt_worker_session mirrors the existing
register helper. Wired at the two natural sites that already
iterate peers_by_index: the rekey drain-completion block in
handlers/rekey.rs (drops the worker entry for the old our_index
once the drain window has expired and the cache_key is
unreachable to any in-flight OLD-K packet), and remove_active_peer
in handlers/dispatch.rs (drops the worker entry for each of the
four index slots: current, rekey, pending, previous). Only
our_index is normally registered; unregister_session is fire-
and-forget for missing entries, so calling unconditionally on
all four slots is correct and bounds the cleanup without per-
slot accounting. Without these callers the per-worker sessions
HashMap and the Node's decrypt_registered_sessions set would
grow monotonically per rekey on long-lived peers.
Testing:
- testing/static/scripts/bench-multirun.sh: multi-run iperf3 +
ping bench. N reruns (default 5), median / min / max / CoV % /
per-run outlier flag, avg ping RTT, ICMP loss %, TCP retransmit
total. Plain client→dest labels + topology header. Pre-bench
peer-convergence check (FIPS_BENCH_CONVERGE_SECS, default 15);
per-path route verification via stats.bytes_sent deltas — fails
fast if traffic exits via a non-static-peer link.
- testing/static/docker-compose.yml: passes FIPS_ENCRYPT_WORKERS /
FIPS_DECRYPT_WORKERS / FIPS_PERF through to containers for A/B
benchmarking without rebuilds.
- testing/static/scripts/iperf-test.sh: same plain client→dest
labels + topology header (was multihop/direct/N hop, which
conflated topology distance with on-wire path).
- .config/nextest.toml: synthetic UDP node tests serialized
through a max-threads=1 test group. Localhost handshakes drop
on shared CI runners under parallel load; one-at-a-time keeps
assertions reliable.
- src/node/tests/spanning_tree.rs: repair_missing_edge_handshakes
— retries up to 5 times for synthetic edges whose msg1 was
dropped, with a drain after each edge retry instead of after
each attempt's full burst.
- src/node/decrypt_worker.rs::tests: two unit tests asserting
WorkerMsg::UnregisterSession removes the worker-thread session
HashMap entry (handle_msg_unregister_session_removes_entry) and
is a no-op for never-seen cache_keys
(handle_msg_unregister_session_idempotent_on_unknown_key), which
is the safety invariant the unconditional unregister calls at
the four index slots in remove_active_peer rely on.
- src/node/encrypt_worker.rs::unix_tests
pipelined_send_wire_layout_roundtrips_canonical_decoders: mirrors
the encoder geometry of try_send_session_data_pipelined (no
coords, the common established-session path), runs the worker's
real seal + send via flush_direct_batch_sync, and decodes the
resulting wire packet using only canonical receive-side decoders
(EncryptedHeader::parse, SessionDatagramRef::decode, FSP header
parse, noise::open). Any divergence between the hand-rolled
encoder offsets (fsp_aad_offset, fsp_plaintext_offset) and the
decoders fails at one of the parse / open / decode steps before
the inner-plaintext assertion fires. Complements the existing
fsp_preseal_runs_before_outer_fmp_seal test which covers the
seal-ordering invariant with synthetic headers but does not
exercise the wire-layout invariant.
CHANGELOG.md [Unreleased] # Changed entry added describing the
worker-pool threading model, hash-by-destination dispatch,
sendmmsg/UDP_GSO, per-peer connected UDP, the operator-facing env
vars, and the bench numbers above.
Cherry-picks from mmalmi/master (paths translated from
crates/fips-core/src/ to src/): 9b7c723, 0deb5cb, 13f7339, e036c0e,
3740a68, 3792f83, 8510193, 4910b07, e53f545, e4e2896, 5fe4af5,
1d01ada, 8c37008, e12469e, 6eb2860.
Co-authored-by: Johnathan Corgan <johnathan@corganlabs.com>
Publish the 13-criteria PR review checklist the maintainer runs on
every incoming PR so contributors (and their coding agents) can run
the same pass before opening, surfacing problems before the review
round trip. CONTRIBUTING.md gets a new 'Self-review against the
project review checklist' subsection under 'Submitting pull requests'
and a Further Reading entry. CHANGELOG [Unreleased] gets an Added
entry.
Two [Unreleased] / Changed entries that should have landed alongside
the originating commits but didn't:
- macOS recvmsg_x batched receive (originally 59225ccf): completes
the Linux-equivalent inbound batching shape on Apple builds. Now
sequenced before the rx zero-copy entry so the section reads as a
coherent receive-path progression.
- Platform-specific test-build warning cleanup (originally 6bd40640,
PR #93): non-behavioral; documents the gating decisions that keep
cross-platform builds warning-clean.
Two Fixed entries appended to [Unreleased]:
- The coord cache surgical invalidation (49bd2104): replaces the
global CoordCache::clear() at parent-switch / become-root /
loop-detection / root-change sites with two targeted methods
(invalidate_via_node, invalidate_other_roots). Preserves cache
entries that remain correct after the topology change.
- The rekey-test strict-ping retry (306e4555): Phase 1 / 3 / 5
per-pair pings now retry up to 4 attempts. Brings the ICMP-noise
miss-floor from ~33% per phase to ~3.2e-6 at 1% loss without
changing the failure-shape signal the asserts target. Test
scaffold only, no daemon code changes.
Johnathan Corgan
ArjenandJohnathan Corgan
sandwichandJohnathan Corgan
olekskyandJohnathan Corgan
Martti MalmiandJohnathan Corgan