laantungir/fips
1
0
Fork 0
mirror of https://github.com/jmcorgan/fips.git synced 2026-07-31 03:56:15 +00:00
Code Issues Packages Projects Releases Wiki Activity

Compare commits

base: laantungir/fips:maint
Branches Tags
laantungir/fips:master laantungir/fips:feat/platform-peer-queue laantungir/fips:next laantungir/fips:maint laantungir/fips:ble-v2
laantungir/fips:v0.4.1 laantungir/fips:v0.4.0 laantungir/fips:v0.4.0-rc2 laantungir/fips:v0.4.0-rc1 laantungir/fips:v0.3.0 laantungir/fips:v0.2.1-rel laantungir/fips:v0.2.1 laantungir/fips:v0.2.0-rel laantungir/fips:untagged-837413e700b3aaa62052 laantungir/fips:v0.1.0
...
compare: laantungir/fips:master
Branches Tags
laantungir/fips:feat/platform-peer-queue laantungir/fips:next laantungir/fips:master laantungir/fips:maint laantungir/fips:ble-v2
laantungir/fips:v0.4.1 laantungir/fips:v0.4.0 laantungir/fips:v0.4.0-rc2 laantungir/fips:v0.4.0-rc1 laantungir/fips:v0.3.0 laantungir/fips:v0.2.1-rel laantungir/fips:v0.2.1 laantungir/fips:v0.2.0-rel laantungir/fips:untagged-837413e700b3aaa62052 laantungir/fips:v0.1.0
152 Commits
maint ... master
Author SHA1 Message Date
Johnathan Corgan 83c4e800a5 Merge branch 'maint' 
# Conflicts:
#	src/node/retry.rs
 2026-07-29 02:17:34 +00:00
Johnathan Corgan 2045b51f5d Watch the Nostr loops that never return, not the one that does 
Every node reported state: degraded permanently. The liveness probe
polled connect_task, which wraps a single Client::connect() call. That
call only spawns a per-relay background connection task and returns,
so the handle finished moments after start on a perfectly healthy
node, the supervisor saw a child exit, and the node latched Degraded
for the rest of its life.

Nothing behaved differently, because every consumer of NodeState
treats Degraded the same as Running. What was lost is the signal: a
genuine degradation was indistinguishable from the permanent false
one.

Watch the three service loops that cannot return by design instead —
the inbound notify loop, the advert publisher, and the refresh ticker.
Each is an unconditional loop, so a finished handle means a panic or
an abort, which is unrecoverable and matches the one-way ChildExited
latch in the supervisor. connect_task and relay_startup_task stay
deliberately unwatched, and the doc comment now says why, since
watching either reproduces this bug exactly.

The tests install task handles directly and check both directions:
that a finished connect_task alongside three live loops reports
healthy, which is the production configuration a few hundred
milliseconds after start, and that each loop dying on its own reports
degraded. Reverting to the old predicate reds four of five; replacing
the predicate with a constant false also reds four of five, so the
fix cannot pass by never reporting degraded at all.
 2026-07-29 01:31:13 +00:00
Johnathan Corgan 3c4f003c91 Merge branch 'maint' 
# Conflicts:
#	src/nostr/runtime.rs
#	src/nostr/tests.rs
 2026-07-29 00:36:19 +00:00
Johnathan Corgan 2e8fb60970 Let build-deb.sh build with Cargo features, and mark the version when it does 
A measurement run needs the .deb built with a non-default feature, and the
script had no way to express that: its argument loop took only --target,
--version and --no-build, and nothing forwarded a feature list or read one
from the environment. --features <list> now forwards to cargo-deb's native
-F.

The marking is the half that is easy to skip and matters more. The
auto-derived dev Version is built from the crate version, the commit date
and the sha, none of which change when a feature is enabled, so an
instrumented package and a default package of the same commit carried
byte-identical versions. Two consequences, and the second is worse than
the first: the node offers no way to tell which one it is running, and
reverting is an install of a version already present, which no-ops
silently and leaves the instrumented binary in place. That is precisely
the failure the per-commit version was introduced to prevent, reappearing
one level down. The Version now carries a +<features> marker, folded to
dots since underscores and commas are not legal there.

The marker sorts above the unmarked build, checked with dpkg
--compare-versions rather than assumed, so installing a feature build is
an upgrade and reverting is a downgrade: revert with dpkg -i, not apt
install. The ~dev ordering below a tagged release is preserved.

--features is refused together with --no-build, which would stamp the
marker onto whatever binaries happened to be sitting in target/ while
claiming the features had reached them.

Verified end to end rather than by reading: a real --features profiling
build produces a package whose fipsctl accepts the profile subcommand and
whose daemon carries the capture-file header text that only exists under
the feature gate. The release packaging path is untouched, shown by
diffing the cargo invocation the old and new scripts produce across five
argument shapes including the --version/--no-build pair the packaging
workflow uses; all five are identical.

Authored on master rather than maint, which takes bug fixes and CI or
tooling changes but not new capability, so the three copies of this
script now differ by design.
 2026-07-28 17:40:28 +00:00
Johnathan Corgan 8162b7d9fc Cover the capture writer's failure path and split its flush cycle 
The mesh rehearsal could not exercise the write-error stop at all, and
that is not a gap in the rehearsal. Removing the sink file leaves the
writer's descriptor valid, so writes keep succeeding into the unlinked
inode and the capture runs on; mounting a tiny filesystem inside the test
container is refused outright. So the terminal state added for a failing
sink had no coverage from either direction.

Split one flush cycle out of the writer loop as a function returning what
it decided, so the loop owns the waiting and the state transition while
the decision is testable on its own. A sink that fails every write now
drives the error outcome directly, and a working sink is asserted to
continue so the first test cannot pass for a writer that always stops.
The residual gap is recorded at the test: what is covered is that an
error from the sink produces the error outcome rather than the cap
outcome, not that a real full disk reaches that branch.

The rehearsal itself was otherwise clean on a live three-node mesh. The
header carries node, build, platform, tick period and the reading
caveats; every step reports every interval; stopping returns in 53 ms
rather than waiting out the flush interval; an idle status reports no
path and no bytes; arming twice is refused naming the active file; an
unwritable directory fails the command; and the node kept serving
throughout. On an idle node the entry gap sits at one tick period and
arm starvation reads about 1.4 ms, which is small but not zero, so the
measurement is live rather than degenerate.
 2026-07-28 01:30:11 +00:00
Johnathan Corgan 7493153a89 Add a feature-gated profiler for the rx-loop maintenance tick 
The tick arm runs twenty-six housekeeping steps in sequence on the one
runtime thread, and is polled last, so anything slow in it holds up
inbound packets, TUN traffic and control commands behind it. Field
evidence says that happens for over a second at a time, but the
attribution behind that is two months old and predates the
connect-on-send gate, the control read isolation, and the peer lifecycle
rework. This measures it rather than continuing to reason about it.

Per step it records exact count, max and total into fixed static
counters; a dedicated writer thread drains them every ten seconds to a
TSV under /var/log/fips, one file per capture, capped at 32 MB. Nothing
accumulates: the counters are swapped to zero each interval and the
thread holds no history. Arming is `fipsctl profile tick on`, served in
the control accept task so the toggle cannot queue behind the very
behaviour it measures, and it does not survive a restart.

The whole thing is behind a Cargo feature that is off by default,
because the risk worth eliminating is the twenty-six edited call sites
in the hot loop. With the feature off the macro expands to the bare
expression, which makes the default build's neutrality something you
read off the generated code rather than something a benchmark fails to
disprove.

The measurement that matters is how late each tick is against the
deadline it was scheduled for, since the arm is polled last and that
lateness is the delay. Two earlier designs derived it from the interval
between entries and both under-reported: the schedule is fixed, so a
steady delay leaves every gap exactly one period and any gap-derived
figure reads zero under precisely the sustained overload this is meant
to find. The interval hands back its own deadline, so the delay is now a
subtraction with no model behind it, and a test drives three late ticks
at a constant gap to keep it that way.

CI gains a default-features clippy and a feature-on build and test on
both runners, closing the gap left by clippy already running with all
features.
 2026-07-28 00:56:54 +00:00
Johnathan Corgan 52dc21726a Merge branch 'maint' 2026-07-27 17:57:21 +00:00
Johnathan Corgan 76d7e43d60 Merge branch 'maint' 
Carries today's maint batch: the local CI image-scoping work, which gives
each run its own build context and test image tag instead of writing the
shared mutable one, and the retirement of the bloom-storm chaos scenario.

Both apply unchanged here — the compose files, suite lists and matrix legs
they touch are identical on the two lines, so no branch adaptation was
needed. Parity stays symmetric across runners at 21 legs a side on this
branch and 24 on maint, the gap being the three rekey Docker suites maint
keeps by design.
 2026-07-26 17:16:26 +00:00
Johnathan Corgan d94300a4f4 Merge branch 'maint' 2026-07-25 19:15:54 +00:00
Johnathan Corgan 66d5be2bfb Merge branch 'maint' into master 
Both maint changes carry over unchanged. The hop-limit helpers follow the
module rename from src/protocol/link.rs to src/proto/link.rs, applied by
rename detection and verified at the renamed path rather than assumed; the old
path is not recreated. The rekey config validation and its jitter-constant
import apply as written, the node module exposing the constant on this branch
as well.

The only conflict was the unreleased changelog, where master's own Changed
entries and the incoming hop-limit entry were both correct for their branch.
Resolved as the union, with the incoming entry appended to master's Changed
list.
 2026-07-25 17:14:30 +00:00
Johnathan Corgan 7505b86a5d Merge branch 'maint' 
Carries the address-to-link map's corrected doc comment. The only
conflict was the adjacent line naming the discovery config keys, which
this line renamed to node.lookup.* and the maintenance line did not; kept
this branch's names.
 2026-07-25 01:53:15 +00:00
Johnathan Corgan b29908ba8a Pin the rekey counter arm's boundary against the real trigger 
The send-counter arm of the rekey trigger had no test that could fail for
the reason it existed. The integration suites never reached it: they
inject after_messages = 65536 against a ping-driven workload, so every
rekey they observe is time-triggered. The unit test that looked like the
guard reproduced the trigger's OR predicate as a local closure and
asserted against its own copy, so deleting the arm from the real
predicate left it green.

That is the coverage hole that let the data-plane drop across a
message-count-triggered rekey survive: the defect is precisely a rekey of
a young session, which only the counter arm can produce.

Replaces the existing at-threshold check with a boundary test driving
poll_rekey, holding the time arm off so the counter is provably what
decides. The silence-below half is the part that is new, and it earns its
place by measurement: widening the bound to `p.counter >= 1` reds exactly
one test in the library suite, this one. Deletion, narrowing to `>` and
inversion are caught here too, though the time arm's negative tests
already catch a bound removed altogether.

Removes the closure-based test rather than repairing it. Its one
assertion over real code, that a fresh session entry's jitter lies within
the symmetric bound, is already covered over 100 samples elsewhere.

Note this leaves the deployed maintenance line untested on the same arm:
its trigger is inline in an async method that reads the clock directly,
with no pure function to drive, so the same test cannot be written there
without the sans-IO seam this line already has.
 2026-07-25 00:35:16 +00:00
Johnathan Corgan 9213cce6c4 Retire the rekey Docker suites; timing, continuity and variants are in-process 
The three rekey integration suites (rekey, rekey-accept-off,
rekey-outbound-only) are dropped from both runners. Their coverage now
lives in fast, deterministic in-process tests:

- rekey timing and choreography (trigger, K-bit cutover, drain, jitter,
  guards) in the sans-IO poll_rekey tests (proto/fsp, proto/fmp);
- data-plane continuity across a real cutover in
  rekey_cutover_preserves_data_plane (a real IK rekey over loopback);
- the accept-off dual-init regression and the udp.outbound_only rekey
  loop in the should_admit_msg1 and dual-init characterization tests.

Drop them from both runners in lockstep so the parity guard stays green,
and record the retirement in the deliberately-not-run block with a pointer
to the standalone runner. The rekey-test.sh script stays on disk.
 2026-07-24 03:51:13 +00:00
Johnathan Corgan f44de56fd8 Add an in-process test that a rekey cutover preserves the data plane 
Drives a real IK rekey handshake to K-bit cutover between two loopback
nodes and asserts an encrypted datagram sent after the cutover decodes on
the new session, with no spurious peer teardown. The rekey is forced
deterministically: rekey.after_messages = 1 crosses the initiator's
trigger on the first sent datagram, and both sessions are backdated past
the responder's 30s rekey-acceptance gate, so no wall clock is read.

The cutover is proven to actually occur (the live session index changes),
which is what makes the continuity assertion meaningful rather than
vacuous. Adds a make_test_node_with_config loopback helper for setting the
rekey thresholds. The rekey timing and choreography decisions themselves
are already covered exhaustively by the sans-IO poll_rekey tests.
 2026-07-24 03:39:47 +00:00
Johnathan Corgan 768df453b7 Merge branch 'maint' 2026-07-24 02:40:19 +00:00
Johnathan Corgan 08547e8dff Merge maint: retire cost/congestion chaos scenarios 
Bring up the retirement of the six cost-based parent-selection chaos
scenarios (cost-reeval, cost-avoidance, cost-stability, depth-vs-cost,
mixed-technology, bottleneck-parent), which tested a decision the Docker
harness could not exercise reliably.

master already carries the equivalent sans-IO coverage in
src/proto/stp/tests (effective-depth cost selection, hysteresis, cost
degradation) and its transport-drop tests, so this keeps master's src
unchanged and takes only the scenario removals and the CI-list, README and
scenario-comment updates from maint.
 2026-07-23 23:44:31 +00:00
Johnathan Corgan abf2f9ba48 Merge branch 'maint' 2026-07-23 19:51:58 +00:00
Johnathan Corgan c58149b0b5 Merge branch 'maint' 2026-07-23 14:09:43 +00:00
Johnathan Corgan bc9bef717e Merge branch 'maint' 2026-07-23 05:29:07 +00:00
Johnathan Corgan 8d7053fa72 Merge branch 'maint' 2026-07-23 04:16:51 +00:00
Johnathan Corgan fc8ebd5a06 Merge branch 'maint' 2026-07-23 03:23:18 +00:00
Johnathan Corgan a1222d6d75 Merge the open !FIPS access SSID layer from GitHub 2026-07-23 03:02:54 +00:00
ArjenandGitHub f624013b83 feat(openwrt): open !FIPS access SSID layer (#126) 
Access layer for phones and laptops to reach FIPS routers on OpenWrt,
stacked on the 802.11s mesh backhaul from #123. Squashed from five
commits by Arjen (Origami74); their original messages follow.

* feat(openwrt): open !FIPS access SSID — fips-ap-setup helper, default transport binding, how-to

  Client access layer for phones and laptops: every FIPS router
  broadcasts the same open SSID ('!FIPS' — the leading '!' sorts it to
  the top of alphabetically ordered network pickers), forming one
  standard ESS. Clients save it once and roam between all FIPS routers
  natively, with FIPS's Noise IK handshake as the only security layer:

  - fips-ap-setup: opt-in UCI helper that creates the 'fips-ap0' open AP
    (encryption none — security type must be uniform across routers or
    clients treat the ESS as different saved networks), an isolated
    network with a static ULA /64, RA-only odhcpd addressing (stateless
    SLAAC, no DHCP — the minimum that satisfies Android's provisioning
    check; no internet by design, so phones keep cellular as default
    route), and a locked-down fips_ap firewall zone (no path to br-lan
    or the WAN; only ICMPv6, mDNS, and the FIPS transports reachable).
    'remove' subcommand undoes it. Radio setup stays opt-in; a package
    must not commandeer radios on install.
  - fips.yaml: ship 'ap0'/'ap1' Ethernet-transport entries commented out
    (matching the 802.11s mesh backhaul) so a stock install that never
    creates fips-ap* logs no per-boot "interface missing" bind warning;
    fips-ap-setup uncomments the matching block when it creates the
    interface and re-comments it on remove.
  - Regression test: extend shipped_openwrt_config_parses to assert the
    ap0/ap1 entries ship commented out and still parse once uncommented,
    alongside mesh0/mesh1.
  - Packaging: install the helper in ipk/apk/buildroot (three synced
    copies), extend CI structural checks and shellcheck targets.
  - docs/how-to/set-up-open-access-ssid.md: full guide, including the
    one-time 'no internet, stay connected' acceptance (stored per SSID,
    covers every FIPS router) and the security-type-uniformity
    constraint.

* docs(openwrt): correct open-SSID/mesh security framing — peering is open by design

  The fips-ap-setup/fips-mesh-setup comments and both how-tos claimed a
  stranger "cannot pass the FIPS handshake" / "their frames die at the
  handshake" / the handshake surface "drops them". That is wrong: FIPS
  peer admission is open. An inbound handshake from any net-new identity
  is promoted (node::handlers::handshake::promote_connection), gated only
  by the daemon's max-peers cap — there is no allowlist, no PSK, and the
  AuthChallenge path is not wired to admission. The Noise IK handshake
  provides authentication (no impersonation of another identity, no MITM),
  not authorization.

  Restate the model accurately in all five places: a stranger on the open
  SSID (or the open mesh) can associate AND form a FIPS peer link — that
  is the point of open access. Containment is the isolated fips_ap zone
  (no path to br-lan or the WAN) plus the max-peers cap, not the
  handshake. Clarify that AP client isolation is an L2 control only: a
  peered stranger is an overlay peer like any other, so the FIPS overlay,
  not L2, is the trust boundary between clients.

* feat(openwrt): serve DHCPv4 on the access SSID from a fixed roamable subnet

  RA-only addressing satisfied Android's provisioning check but left
  anything expecting IPv4 with a self-assigned address and a "no IP"
  complaint. dnsmasq now leases out of 10.21.<N>.0/24 (N = radio index;
  prefix echoes FIPS port 2121), deliberately identical on every router:
  a roaming client keeps its lease across the ESS, and dnsmasq's
  authoritative mode — the OpenWrt default, pinned by the helper — ACKs
  the renew a foreign router never issued. Lease collisions across
  routers surface as a NAK on renew and the client re-DHCPs.

  The dhcp section's 'dhcpv4 server' is read by both dnsmasq (default
  images) and odhcpd (only with maindhcp), so either arrangement serves.
  A DHCPv4/udp-67 accept rule joins the fips_ap zone; DHCPv6 stays off,
  the ULA RA stays as-is, and nothing depends on an upstream. The zone
  remains isolated — no forwardings, no internet.

* feat(openwrt): enable mDNS rendezvous from fips-ap-setup

  Phone FIPS apps cannot open raw-Ethernet sockets, so DNS-SD is how they
  find the router's daemon — but node.rendezvous.lan defaults to off. Ship
  the lan block commented in fips.yaml (consistent with the apN transport
  entries) and have fips-ap-setup uncomment it when creating the access
  SSID. The awk match is scoped to node.rendezvous because
  transports.ethernet carries a 'lan' entry at the same indent. The switch
  is daemon-wide, so 'remove' deliberately leaves it on rather than guess
  whether other transports rely on it.

* fix(openwrt): bind UDP dual-stack [::]:2121 so access-SSID clients reach it

  The shipped router config bound the UDP transport "0.0.0.0:2121" (IPv4
  wildcard) while the mDNS LAN advert announces every interface address,
  including the router's IPv6 link-local — which phones on the !FIPS
  access SSID rightly prefer (their cellular default route swallows v4,
  and fd00::/8 is captured by the Myco mesh TUN). Result: the client's
  Noise msg1 arrives on an unbound v6 port and is silently lost; the
  handshake resends and times out.

  Symptom chain (observed on-device): mDNS resolve OK, platform push OK,
  "Sent Noise handshake message 1" to [fe80::…%N]:2121, four resends, no
  reply, 30 s stale-timeout.

  OpenWrt is Linux (bindv6only=0), so "[::]" accepts IPv4 via v4-mapped
  addresses too — nothing is lost. packaging/common is deliberately left
  on "0.0.0.0" for now: Windows defaults IPV6_V6ONLY=1, where "[::]"
  would drop v4 instead.
 2026-07-22 20:02:20 -07:00
Johnathan Corgan 16b1bc2c5c Merge branch 'maint' 2026-07-23 02:42:48 +00:00
Johnathan Corgan fbd0dba870 Merge branch 'maint' 2026-07-23 02:06:06 +00:00
Johnathan Corgan c2d6283bd6 Merge branch 'maint' 2026-07-22 22:57:33 +00:00
Johnathan Corgan 946df54d0f Merge maint into master, bringing the chaos suite's failure reporting with it 
The chaos simulation could not report a failure on this line either: scenarios collided on globally-scoped container names and a shared config directory, the runner swallowed whatever they raised, and the harness discarded their exit codes before anything read them. The five commits scope the names and directories per scenario, give an aborted run its own exit code and let that code reach the summary, stop a failed scenario harvesting whichever containers happen to hold its names, and keep the daemon's reason for a failure instead of throwing it away.

Testing only, and the merge is textual: every chaos file matches maint byte for byte, and the ci-local divergence on this line does not touch the changed regions.
 2026-07-22 09:33:58 +00:00
Johnathan Corgan 29923cf676 Merge maint into master, reconciling the two hop limit implementations 
Both branches carry the same correction, authored separately because the
sans-IO relocation moved the logic into a core that returns an outcome rather
than acting inline. Git resolved the file move as a rename and offered to merge
the two versions textually, which would have been wrong in both files.

Resolved by keeping master's re-expression wholesale for the handler and its
tests, and taking the changelog entry from maint. Verified rather than assumed:
the resulting tree is byte identical to the reviewed and CI-green master commit
across every code path, and the merge contributes nothing but the fifteen line
changelog entry. Since the code is unchanged from a tree that already passed the
full local suite, that result still stands and no re-run is owed.

Nothing was lost from maint's side. Its tests are written against a file layout
that does not exist here, and master carries equivalent coverage written against
its own shape, including the multi-node chain and the cache warming cases that
the maint version does not have.
 2026-07-22 02:59:44 +00:00
Johnathan Corgan 34431209f5 Follow IP semantics for the session datagram hop limit on the refactored path 
The same correction as on the maintenance line, re-expressed rather than merged,
because the sans-IO relocation moved this logic into a synchronous core that
returns an outcome instead of acting inline.

Delivery to the addressed node is no longer gated on the hop limit, and the
decrement happens before the drop decision, so a datagram that would leave with
zero is not sent. The reachable radius does not change: the old behavior refused
to deliver at zero but was willing to send at zero, and the new behavior is the
mirror of that, so the two cancel.

The shell above the core needed changing too, and that was the real work. It
carried two hop-limit predicates of its own, both justified by comments citing the
ordering that existed before the refactor. The first gated coordinate cache
warming, which this fix requires to be unconditional; left alone it would have
suppressed warming twice over, once for a datagram addressed here and now
delivered, and once for a transit datagram dropped for hop limit whose plaintext
coordinates are still perfectly usable. The second gated next hop resolution,
which carries a cache touch side effect, and now mirrors the core's
would-leave-zero rule instead of the old one. Neither divergence would have
surfaced as a test failure.

That last point is worth stating plainly, because it nearly went the other way.
Restoring the warming gate on top of the corrected core passes the entire suite
without a single failure. The tests could not see the seam between the core rule
and the shell's copy of it. So this adds coverage that can: a three node chain
pinning the decrement across a real hop, which is the only thing exercising the
shell predicates composed with the core, and two warming tests at a zero hop limit
covering both the delivered and the dropped case. Each was checked by putting the
specific defect it targets back and confirming the test fails.

One existing test that claimed to cover hop limit behavior asserted nothing
whatsoever. It asserts now, but those assertions do not discriminate this change,
since the case it exercises behaves identically under both rules. They guard
against the test going vacuous again, not against this defect, and the comment
says so.
 2026-07-22 02:32:34 +00:00
Johnathan Corgan 80c956a6fd Merge maint into master after the v0.4.1 rollover 
maint carries only its 0.4.2-dev version bump; master keeps its own
0.5.0-dev development version. Recorded as a linkage merge so later
forward-merges of real fixes land cleanly.
 2026-07-19 19:06:08 +00:00
Johnathan Corgan a47ddbd5a5 Merge maint into master after the v0.4.1 release 
Carries the v0.4.1 content up the one-way branch flow. The release itself
belongs to maint, so the parts of this merge that identify a version are
resolved in master's favor and the parts that describe the project's state
are taken from maint.

Kept master's: Cargo.toml and Cargo.lock at 0.5.0-dev, the README status
badge, and the "FIPS is at v0.5.0-dev on the master branch" line. A patch
release consumes no minor version and master's development line is
unaffected by it.

Took from maint: the bloom antipoison FPR default change and the
duplicate-definition fix it rests on, the two docs that describe them, the
new v0.4.1 release notes, the correction to the v0.4.0 release date, and the
[0.4.1] changelog section, which slots below master's own [Unreleased] and
above [0.4.0]. The root RELEASE-NOTES.md mirror moves to v0.4.1 because it
tracks the latest shipped release, which also clears the stale provisional
date it had been carrying.

One line needed splitting rather than choosing: the README said "v0.4.0 has
shipped" inside the paragraph that identifies master as 0.5.0-dev. The
version identity stays master's and the shipped-release pointer moves to
v0.4.1.

Quartet green on the result: 1645 tests passed, clippy clean with -D warnings.
 2026-07-19 18:57:52 +00:00
Johnathan Corgan ea74cd7e58 Merge branch 'maint' 2026-07-19 07:40:07 +00:00
Johnathan Corgan 37adb13d5b Merge branch 'maint' 2026-07-19 07:17:26 +00:00
Johnathan Corgan 93a5b71728 Merge branch 'maint' 2026-07-19 06:46:55 +00:00
Johnathan Corgan 4ff7de4d81 Merge branch 'maint' 2026-07-19 06:45:18 +00:00
Johnathan Corgan 0c3d9a0b73 Correct stale view-trait docs and drop internal shorthand from comments 
The lifecycle view's doc described the shell as implementing it over a
`connections` map that no longer exists. The establish view named that map
too, and was doubly wrong: both of its snapshot builders read only
`peers`. Rewrite both to describe the maps actually read.

Several comments in the node module carried shorthand from private
working notes - bare parenthetical labels, and one reference to an
internal tracker item - none of which mean anything to a reader of this
repo. Rewrite them so each clause states its own reasoning inline. The
paragraph above the tracker reference already carried the rationale, so
that one is simply dropped.

Comment-only; no behavior change.
 2026-07-19 06:11:40 +00:00
Johnathan Corgan 281ed132f1 Merge branch 'master' into refactor-node 2026-07-19 04:25:36 +00:00
Johnathan Corgan c5492f4572 Merge branch 'maint' 2026-07-19 04:25:25 +00:00
Johnathan Corgan 7fe1d75637 peer: delete the pending-connection type, leaving the control machine whole 
The per-peer control machine has absorbed every field the pending
connection carried. What remained was a struct holding two Noise
handles beside a duplicate copy of bookkeeping nobody read. Replace it
with a small carrier for the two handles and delete the type.

Presence of that carrier, not the state of the handles inside it, is
what marks a machine as mid-handshake. The distinction is essential
rather than stylistic: a failed handshake drops its initiation handle
and is deliberately retained so the stale sweep can reclaim it, and a
completed one has its session taken before disposal. Deriving presence
from the handles would make both invisible to the sweep, the
connection count, and the peering budget at once, leaking the slot
permanently. A test drives an empty carrier past every presence
predicate and then detaches it, so a future edit cannot quietly couple
the two.

The remote startup epoch now comes from the surviving carrier, which
the handshake operations already wrote at the same two points with the
same value. The paired writes onto the pending connection's own
bookkeeping had no readers left and are gone.

The handshake-phase surface leaves the public API: it was public by
accident rather than design, and the machine behind it is crate
internal. Callers outside the crate that need a view of pending
handshakes go through the operator queries, which are unchanged.

ConnectionState::inbound_with_transport loses its last non-test caller
with the inbound seed and is marked test-only.
 2026-07-19 01:18:05 +00:00
Johnathan Corgan e21e09d7e6 peer: source the handshake identity and index family from the control machine 
The pending connection and the per-peer control machine have carried
duplicate copies of the handshake-phase fields since the machine gained
its own connection state. Read them from the machine and drop the
connection's projections.

The peer identity is the sharp one. The connection learned it from msg1
and the machine's carrier did not, so the two views genuinely disagreed
for inbound connections until the Noise operations moved onto the
machine and began recording each result on both. That is now in place,
so every reader can take the machine's copy unchanged: the establish
snapshot, the promotion sweep for competing connections, the dial and
path in-progress checks, the peering observations and per-peer in-flight
budget, and the stale-connection sweep's retry address.

Also repointed: our_index, their_index, transport_id, started_at, the
stored handshake message bytes, and the idle-timeout check. Two
duplicate index writes on the connection are dropped, both immediately
preceded by the machine-side write of the same value. Reads that
previously came off a connection detached just before its machine was
disposed now capture the machine's value first.

Test seeding follows the establish paths: the seed builders write
our_index to the carrier, which promotion now reads. A new test pins
the inbound identity learn on the carrier and the retry address a
failed inbound connection reports to the sweep, so a silent regression
to a blank identity cannot pass.

ConnectionState::duration loses its last non-test caller with the
connection accessor and is marked test-only.
 2026-07-19 00:25:17 +00:00
Johnathan Corgan e7537929ba node: move link, direction, and peer address onto the control machine 
These three had no counterpart on the control machine, so readers still
reached them through the pending connection. Add machine-side
accessors and repoint every reader, then drop the connection's.

The peer address needed its writes lifted, not just its reads
repointed: the machine's copy was never written. It is now written at
each of the points the connection's copy was — the inbound seed, the
dial, message-2 completion, and the two paths that seed a machine from
a pre-built connection — so promotion and the resend path read a value
with the same provenance at the same time as before.

Link and direction need no lift. Both machine constructors already
seed them from the same arguments the connection is built with, so an
outbound machine carries outbound state and an inbound one inbound,
and the machine's link always equals the connection's. The handshake
operations' direction guards read the machine's copy for the same
reason.

The stale-connection sweep's teardown log and its resend path both now
take the transport and address from the machine, and each keeps its own
check that a pending connection is still attached rather than relying
on the caller to have established it.

Add a test pinning link, direction, and address on the two shapes that
seed a carrier independently — the dial and an accepted message 1. The
cross-connection winner reads both values from a carrier one of those
two already seeded.
 2026-07-18 23:17:21 +00:00
Johnathan Corgan 347cbe60bd peer: move the Noise handshake operations onto the control machine 
The pending connection drove its own Noise handshake while the control
machine held it, so the crypto and the state it produces lived on the
carrier that is going away. Move the six operations onto the machine:
starting and completing an initiation, processing an inbound
initiation, taking the session, testing for one, and dropping the
handle on failure. Bodies are unchanged apart from reaching the
handles through the attached connection.

Each operation now records its results on both carriers. The learned
identity, the remote epoch, and the activity stamp are written to the
machine's own bookkeeping at the same point and with the same value as
they are written to the connection's. The connection's copies still
have readers until those reads are repointed, so both have to be
written; the machine's copy of the learned identity was previously
never populated for an inbound connection, which is why an inbound
pending row showed no expected peer.

Driving the handshake from the machine means the machine has to exist
before the crypto runs. On the inbound path it is built above the
message-1 processing and still kept local, so a rejected message
leaves no registry entry and allocates no index. On the outbound path
it already existed from the dial, so it simply takes the connection
before the index allocation, and both failure arms unwind it as they
did.

Completing message 2 no longer mirrors the activity stamp separately,
since the completion itself now writes both carriers at the point the
mirror was approximating.

Three tests cover what the compiler cannot. A connection whose
handshake failed holds neither Noise handle yet must stay visible to
the stale-connection sweep, or every failed connection would leak and
hold a peering-budget slot forever. A message 1 rejected by the crypto
or by the ACL must leave no machine, no index, and the same rejection
count as before. A dial whose message-1 preparation fails must unwind
the machine registered at dial time; that test drives the
index-allocation failure rather than a crypto failure, which is the
arm this change actually widens, since the allocation now happens with
the connection already attached.
 2026-07-18 22:55:38 +00:00
Johnathan Corgan 11ec16777c node: yield the control machine when iterating pending connections 
`Node::connections()` yielded the pending connection itself, so its
consumers reached the handshake-phase fields through that value. The
pending connection is being folded into the control machine, and the
fields will move off it, so yield the machine (keyed by its link) and
let each consumer reach what it needs from there.

Membership is unchanged: the iterator still selects machines that
carry a pending connection, which is the predicate `connection_count`
and the stale-connection sweep already use. Every consumer takes the
same value from the same place, one hop further out, and no field
changes carrier here.

The method is now internal to the crate. It yields the control
machine, which is not part of the published surface, and nothing
outside the crate iterates pending connections — operator views of
the handshake phase go through the query surface instead.
 2026-07-18 22:06:46 +00:00
Johnathan Corgan 5b09e22956 node: read promotion's carrier fields in one borrow 
`promote_connection` detached the pending connection and then
interleaved reads off that detached value with three separate lookups
of the same control machine. Gather the machine-side fields
(`their_index`, `transport_id`, link stats) in the single `get_mut`
that takes the connection, so the machine is borrowed once.

Behaviour is unchanged. The connection is still taken before anything
is validated, so a rejected promotion leaves the machine with no
pending connection, and the prelude only gathers options: the checks
below it still report the first missing field in the same order
(`our_index`, `their_index`, `transport_id`, `source_addr`).

Link stats move to the prelude, ahead of those checks. The value is
the same either way: they live on the surviving carrier rather than
the detached connection, and nothing between the two points touches
the machine map. The lookup could not fail at the old site either,
since the function had already reached it through a successful lookup
on the same key, so dropping the defaulting arm changes nothing.

Add a test covering the error order and the detach, driving promotion
with connections missing each required field in turn plus every later
one, so an implementation that validated during the prelude would
report the wrong field. Test seeding grows a variant that lets the
caller shape the seed.
 2026-07-18 21:45:09 +00:00
Johnathan Corgan a382b17931 node: seed control machines directly in tests, ahead of removing the leg 
Tests built a free-standing PeerConnection, mutated it, and handed it to
Node::add_connection by value. None of those sites survives the removal of
PeerConnection, so converting them afterwards would mean one enormous commit
that cannot be reviewed honestly. Convert them now, while the struct still
exists and the conversion can be validated against a green tree.

Adds a cfg(test) Node::seed_handshake_machine plus a HandshakeSeed builder,
and rewrites make_completed_connection (now seed_completed_connection) and
the twenty inline builders onto it. add_connection keeps its body and loses
its test callers.

The builder's carrier seeding is a verbatim copy of add_connection's: the two
conditional writes for their_index and transport_id, then set_leg, through the
same entry().or_insert_with() so an existing leg-less machine keeps its
constructor-side fields. Nothing else reaches the carrier -- our_index,
source_addr, post-construction started_at and the stored handshake bytes stay
leg-only. Seeding more than that would let these tests observe a carrier
richer than production's and keep passing even if a later production
write-lift were missed.

The Noise exchange now runs on the already-seeded leg rather than before the
hand-over. That is neutral because the only read of expected_identity is
guarded by is_outbound, and no crypto method allocates a session index.

Also adds a compile-time check that PeerAction is Clone + Eq, which is what
keeps a runtime handle from being smuggled into an action payload.
 2026-07-18 21:24:03 +00:00
Johnathan Corgan a90049d3a1 Merge branch 'master' into refactor-node 
Bring the Android-ready core (target_os gating and app-owned TUN seam)
onto this branch so subsequent work builds on the current mainline tree.
 2026-07-18 20:38:56 +00:00
Johnathan Corgan 791b35c221 node: make the peer machine's connection state the sole session-index carrier 
The session index was carried three ways: on the leg, on the peer machine's own
connection-state copy, and on a separate machine-owned shadow field. Populate the
machine's connection copy on the outbound path (it was written only on the inbound
and cross-connection-swap paths before), then delete the shadow field and route its
readers, the decrypt-registration lifecycle, and the rekey cutover through the
single carrier. The shadow and the connection copy diverged only on paths that are
unreachable or dormant in production (a rekey cutover with no newly allocated index,
the swap winner's later reads, and the never-dispatched timeout/disconnect events),
so the merge is byte-identical for every reachable path. Rewrote the outbound-promote
index tests and the dial-time comment for the single-carrier contract.
 2026-07-18 17:19:44 +00:00
Johnathan Corgan 6a80790742 node: source connection index/transport/stats from the peer machine 
The connection leg's their_index, transport_id and link_stats duplicated the peer
machine's own connection-state copy. Route every production reader through the
machine copy: promotion, the stale-connection reaper, the outbound msg1 resend, and
the in-use / connecting-path checks. Seed the inbound machine's transport_id from the
msg1 packet (the same source the leg used), matching the outbound dial seed, so the
hard-required promote read never sees a missing transport. link_stats is a
never-mutated zero seed, so its leg accessors are removed outright; the leg
their_index/transport_id getters/setters remain only for the test-only connection
builder. Also delete the write-only next_resend_at_ms field from the connection
state (the resend deadline is carried by the machine-armed retransmit timer, not this
field). Byte-identical.
 2026-07-18 16:43:22 +00:00
Johnathan Corgan 60b8acf716 node: source handshake resend buffers from the peer machine 
The connection leg's stored msg1/msg2 handshake-resend buffers duplicated the
peer machine's own connection-state copy. Write the machine copy at the outbound
msg1-prep and the two inbound authorize sites, then source the retransmit
resend-bytes reader, send_stored_msg1, and the pending tier of find_stored_msg2
from the machine. The post-promote active-peer msg2 copy is written from the same
wire bytes, so the pending tier now matching after promotion is value-identical.
Byte-identical resend behavior.
 2026-07-18 16:00:15 +00:00
Johnathan Corgan 4fc295d90a node: source connection peer-identity telemetry from the peer machine 
The expected_peer connection-row field read the peer identity off the leg;
source it from the machine's own connection-state copy instead, matching the
started_at/last_activity treatment. The machine carrier holds the same identity
as the leg for every connection that appears in the view: outbound legs seed
both from the dialed identity, and inbound legs never rest in the connections
view (the machine takes the leg at promotion), so their machine copy is
unobserved. The leg accessor stays for the crypto-extraction path and retry
decisions. Byte-identical.
 2026-07-18 15:32:04 +00:00
Johnathan Corgan b3f2018fce node: source connection start/activity timestamps from the peer machine 
The connection leg's started_at and last_activity duplicated the peer
machine's own connection-state copy. Make the machine copy the sole live
telemetry and timeout carrier: re-stamp it with the leg's provenance when
the leg is attached at handshake start (the msg1-prep clock, not the earlier
dial-time constructor value) and mirror the completion touch, then delete the
leg's last_activity/touch accessors and source the connection-row projection,
show_connections, and the timeout readers from the machine. Byte-identical
for all normal paths.
 2026-07-18 15:11:43 +00:00
Johnathan Corgan 56bbc81a40 node: derive handshake state from the peer machine, delete the leg field 
The connection leg's HandshakeState field duplicated the peer machine's
handshake phase. Delete it and derive the displayed handshake state string
from the machine's PeerState, looked up by link id (mirroring the resend
count). Move the failure signal onto the machine: a send_failed flag that
preserves retransmit eligibility (the machine stays in its handshake phase),
alongside the existing Failed state. The leg's crypto self-gates now guard on
Noise handle presence instead of the deleted phase field, and mark_failed
only drops the handle. Telemetry strings, wire bytes, index allocation, and
the stale-connection reaping are byte-identical for all normal paths.
 2026-07-18 02:26:39 +00:00
Johnathan Corgan b38f8c6ffb node: classify inbound msg1 via a single machine decision site 
handle_msg1 now creates one local machine before classification and routes
on the InboundDecision it returns, instead of calling establish_inbound in
the shell and re-deriving the decision inside each arm. The promote and
restart arms drop their own per-arm machine construction and the redundant
InboundMsg1 step; their phase-1 actions come from the single call. The
effect-bearing arm bodies (rekey-respond, duplicate resend, reject
bookkeeping, and the promote/restart tails) are unchanged.

Behavior-neutral: the same establish_inbound evaluation over the same
snapshot and wire, done once in the machine method rather than once in the
shell, with byte-identical arm bodies and no change to index allocation,
wire sends, machine state, or telemetry.
 2026-07-17 19:17:25 +00:00
ArjenandJohnathan Corgan cf62cff5f4 feat: Android-ready core: target_os gating and app-owned TUN seam 
Make the FIPS core build and run as an embedded Android library. The host
app owns the TUN (e.g. an Android VpnService) and FIPS performs no
system-TUN or CAP_NET_ADMIN operations.

Squashed from the following changes:

- gate desktop transports/TUN by target_os, not features: a plain
  `cargo build` now compiles for every target with no flags. Ethernet (raw
  AF_PACKET / BPF) is gated to linux/macos, so Android (target_os =
  "android", not "linux") self-excludes it as Windows already did; real
  system-TUN ops are gated per linux/macos and Android gets a no-op stub;
  the ipi6_ifindex cast handles it being i32 on Android vs u32 on macOS. No
  Cargo features are introduced; desktop builds are unchanged.

- app-owned TUN seam: Node::enable_app_owned_tun() lets an embedder that
  owns the TUN fd exchange IPv6 packet bytes with FIPS over channels
  instead of FIPS creating a system TUN device. It returns (app_outbound_tx,
  app_inbound_rx): the embedder pushes packets read from its fd into the
  outbound sender (app -> mesh) and pulls packets destined for its fd from
  the inbound receiver (mesh -> app). start() gates system-TUN creation on
  tun_tx being unset, so with the channels pre-installed it skips device
  creation and does no system-TUN ops; both directions reuse the existing
  inbound-shim and run_rx_loop wiring. Packets entering via app_outbound_tx
  bypass handle_tun_packet, so the embedder must push only fd00::/8-destined
  packets and clamp TCP MSS on outbound SYNs; the rustdoc and the
  IPv6-adapter design doc spell this out.

- keep the android target warning-clean so the cross-compile check passes
  clippy -D warnings.

- add an Android cross-compile CI check: cross-compile the library for
  aarch64-linux-android via cargo-ndk and run clippy -D warnings. Android
  ships as an embedded library (the host app owns the TUN), so there is no
  daemon binary to package; this is a check job, not a packaging one.

- docs: list Android as a supported platform.

Tests: app_owned_tun_seam_wires_channels covers the channel round-trip and
the Active state; start_skips_system_tun_when_app_owned runs start() and
asserts no named system device is created.
 2026-07-17 18:25:55 +00:00
Johnathan Corgan 7fe3388f2f peer: return the inbound establish decision from the machine arm 
The inbound msg1 machine arm re-derived the establish decision for
its own action selection while the shell matched on a separately
computed copy. Expose the arm as a method returning the decision
alongside the actions so a driver can route on it directly; the
step dispatch delegates and keeps its action-only shape.

The dead rekey-respond arm is stripped to decision-only and its
helper deleted: it allocated from the real index allocator, wrote
the rekey shadow index, emitted a wrong-framed rekey send, stamped
the dampening clock, and flipped state, none of which happens on
the live path, where the inline respond body owns all effects. The
resend-msg2 arm's send emission is stripped the same way; the
decision already carries the stored bytes.
 2026-07-17 13:47:50 +00:00
Johnathan Corgan 252d16fab9 node: feed msg1 send failure to the peer machine as an event 
send_stored_msg1 marked the embedded leg failed by writing it
directly from the shell. Route the write through a new
HandshakeSendFailed machine event instead: the machine marks its
leg so the stale-connection sweep reclaims it, without leaving the
handshaking state, so retransmit eligibility survives the window
between the failed send and the sweep exactly as before.

send_stored_msg1 gains a now_ms parameter threaded from the action
executor for the step call; the failure arm itself ignores it.
 2026-07-17 07:12:40 +00:00
Johnathan Corgan 94d7b91244 node: route the outbound msg2 establish decision through the machine 
handle_msg2 no longer pre-computes establish_outbound alongside the
machine's own evaluation of the same snapshot. The shell now builds
the snapshot, steps the persistent outbound machine once at the
decision point, and routes on what comes back: the promote action
vector drives promotion through the executor as before, and the
ResolveCrossConnection decision selects the inline swap/keep
resolution bodies, which are unchanged.

The machine step and its defensive transient-rebuild move up from
the promote arm to the decision point; the cross-connection path
keeps its take-leg-then-dispose ordering with the step preceding
both. Comment prose at the touched sites refreshed to describe the
single-decision-site shape.
 2026-07-17 06:59:30 +00:00
Johnathan Corgan 7790eb86bd peer: make the outbound msg2 swap/keep machine arms decision-only 
The machine's on_msg2 cross-connection arms crystallized state and
emitted FreeIndex/RegisterDecryptSession actions that duplicate the
inline shell resolution, but they are unreachable on the live path
(the shell removes the machine before running the swap/keep bodies).
Making them live in that shape would double-free the outbound index
through the executor.

Strip both arms to a single new ResolveCrossConnection { swap }
action: a decision conveyed to the driver, not an effect. The shell
intercepts it and runs the inline resolution, which owns all effects
permanently. The action executor gets a defensive unreachable arm.
The Promote arm and set_their_index are unchanged.
 2026-07-17 06:44:09 +00:00
Johnathan Corgan cf1c957336 node: embed the handshake leg in the peer machine, drop the connections map 
The pending-handshake PeerConnection map and the per-peer control
machine map were parallel LinkId-keyed structures whose keysets must
stay coherent by hand. With every leg now born with a machine, the leg
becomes storage inside its machine (leg: Option<PeerConnection>, pure
storage the machine never reads or drives) and Node loses the
connections field; every access routes through the machine.

The non-mechanical lowerings, each argued at the site: the
rekey-vs-establish gate in handle_msg2 tests leg-absence (an established
peer's machine stays keyed by its link, so machine-presence would
misclassify every rekey msg2 as a fresh establish); the
connecting-predicates, peering observation, and handshake-slot budget
iterate machines-with-legs so connect-window machines (leg not yet
born) are excluded exactly as before and never double-counted against
their pending-connect slot; the cross-connection extract takes the leg
before disposing the machine; the stale reaper takes the leg and leaves
the machine untouched when none is present, matching the old early
return. The map-coherence debug check keeps its machine-has-carrier
direction with the embedded leg as a carrier; the leg-to-machine
direction is now true by construction and its gate const is gone.

connection_count() counts machines with legs; the connections()
iterator, the test seams, and the control-socket connection rows are
re-implemented over the embedded legs with unchanged output.
 2026-07-17 04:27:53 +00:00
Johnathan Corgan 5ccd95cf3f node: add a debug-build peer-map coherence check 
Assert, once per tick in debug builds, that the peer control-machine map
and its carriers stay coherent: every control machine has a live
handshake leg, an active peer, or a pending connect (a machine with none
is a leak the stale reaper can never see), and every handshake leg has a
machine. The second direction sits behind a file-local const so a branch
where handshake-window legs legitimately run machine-less can gate it
off without weakening the leak tripwire.

Teach the add_connection test seam to seed a control machine for the leg
it inserts (derived from the connection's direction and identity), and
remove_connection to dispose it, so the seam-built topologies satisfy
the check. Three unit tests: seam coherence, coherence through a real
promotion, and the orphaned-machine panic.
 2026-07-17 02:09:28 +00:00
Johnathan Corgan 119b85d28e node: correct stale liveness claims in machine, executor, and timer docs 
Several module and field docs still described the per-peer machine as
unwired shadow scaffolding. It has been live for some time: machines are
inserted at dial and inbound msg1, stepped by the handshake handlers and
the rekey-cadence and liveness-reap routers, and the executor's
SwapSendState/CompleteDrain/InvalidateSendState arms are the authoritative
paths (the inline bodies survive only as debug-assert release fallbacks).
Rewrite those docs to the current truth while keeping the still-true
dormancy facts: PeerEvent::Timeout and PeerEvent::Tick are never
dispatched in production, retransmit fires on the machine-armed deadline
while the timeout reaper keys on timer presence with the config
threshold, and the remaining inert executor stubs are SendRekey,
SendLinkMessage, and the connected-UDP arms. Drop the stale
allow(dead_code) on the peer_machines field.
 2026-07-17 01:33:06 +00:00
Johnathan Corgan 74245e80ac peer: remove the dead PeerSlot enum and PeerConnection resend API 
PeerSlot (and its entire impl surface) was referenced only by its own
module tests and the lib.rs re-export; peer storage has always used the
separate connections/peers maps. PeerConnection's resend_count/
next_resend_at_ms/record_resend delegations had no production callers:
the live resend counter is machine-sourced (connection_resend_count reads
the per-peer machine), and the FSP session layer uses SessionEntry's own
methods. ConnectionState::next_resend_at_ms is now test-only (its
remaining callers are the fmp state unit tests) and marked cfg(test).

The PeerSlot unit tests go with the enum; test_resend_count_tracking is
dropped because the delegation target's schedule arithmetic is already
covered by the fmp resend_bookkeeping test.
 2026-07-17 01:32:49 +00:00
Johnathan Corgan e42598a86e Merge branch 'maint' into master 2026-07-16 05:55:20 +00:00
ArjenandJohnathan Corgan 054d17aac5 feat(openwrt): 802.11s open-mesh backhaul support 
Router-to-router radio backhaul over an open 802.11s mesh interface,
with FIPS providing all encryption (Noise IK), authentication, and
routing on top of bare L2 neighbor links. The mesh runs OPEN with
mesh_fwding 0 — SAE would duplicate the Noise layer and force ath10k
raw mode, and FIPS is the routing layer — so the Noise handshake is the
real auth/encryption boundary and FIPS's spanning tree does the routing.

fips-mesh-setup: an opt-in UCI helper that creates a per-radio
mesh-point interface (radio0 -> fips-mesh0, radio1 -> fips-mesh1;
trailing-digit derivation with a free-index fallback and a collision
guard). Radio setup stays opt-in — a package must not commandeer radios
on install. 'remove' takes an optional radio and otherwise removes all
instances. Dual-band routers get one instance per radio; FIPS treats the
two backhaul paths as failover, not multipath: it keeps one active link
per peer (cross-connection resolution picks a single winner), and the
second band stands by, re-establishing the peer after keepalive timeout —
traffic never uses both bands at once.

fips.yaml ships the mesh0/mesh1 Ethernet-transport entries commented
out, so a stock install that never creates fips-mesh* logs no per-boot
"interface missing" bind warning. fips-mesh-setup uncomments the matching
meshN block when it creates the interface and re-comments it on remove,
so the flash-and-drop-in flow needs no manual config edit. The file is
rewritten 0600-first (it may hold an inline nsec) via an atomic replace.

Two field-found silent non-peering causes are surfaced by the helper and
the guide:

- Same channel: mesh points only peer on a shared channel, and 'auto'
  lets each radio pick its own. The helper prints the radio's
  band/channel and warns loudly on 'auto' with the exact uci command to
  pin one; the how-to gains an ordered no-peers triage (channel mismatch,
  on-air scan check, DFS CAC wait, regdomain).
- STA channel capture: a client (sta) interface drags the whole radio to
  its upstream AP's channel, so a mesh pinned elsewhere never joins and
  does not recover until the STA disconnects. The helper warns when the
  target radio carries a STA; the guide documents the incompatibility of a
  roaming uplink with a fixed-channel mesh on the same radio.

Both the create and remove paths run 'wifi reload', which briefly drops
every client AP on all radios; the how-to sets that expectation.

Regression test: the shipped OpenWrt fips.yaml must parse via the real
Config deserializer in both states — as shipped (mesh inactive) and after
the uncomment the helper performs.

Packaging: the helper is installed across the ipk/apk/buildroot paths
(three synced copies), with the CI structural checks and shellcheck
targets extended to cover it. Full guide in
docs/how-to/set-up-80211s-mesh-backhaul.md.
 2026-07-16 05:48:04 +00:00
Johnathan Corgan 87399795f8 node: observe rekey-msg2 and cross-connection resolution in the per-peer machine 
Feed the per-peer machine two observation events so its control/rekey
shadow state stays coherent with the inline crypto-session installs,
without moving those installs. The crypto effect bodies are unchanged.

On rekey-msg2 completion, feed the machine the responder index on the
success path only (after the pending session is installed); the abandon
path feeds nothing.

On cross-connection resolution, feed the machine the swap outcome: on a
swap it records the new outbound and responder indices to mirror the
session replacement, and on a keep it does nothing. The observation
targets the promoted peer's machine, not the outbound leg whose machine
is removed as the block is entered.

Both observation handlers emit no action; nothing consumes the updated
shadow state yet.
 2026-07-15 23:50:18 +00:00
Johnathan Corgan 765819f52b node: reap the handshake timeout through the per-peer machine timer 
Move the outbound handshake-timeout reap off the unconditional check_timeouts
scan onto the machine-armed HandshakeTimeout timer. A new drive_handshake_timeouts
(run before the retransmit drive, so a timed-out leg is reaped rather than resent
on the same tick) reaps the outbound legs that carry a HandshakeTimeout timer and
have idle-timed-out this tick.

The timer's presence selects the leg (only outbound legs arm one; IK inbound arms
none); the reap threshold is the shell is_timed_out(now, config) predicate, not
the timer's stored deadline. The machine arms the timer from a hardcoded constant
at dial, which is not authoritative for an operator-tuned handshake_timeout_secs,
so reading the threshold from config each tick keeps the reap neutral for any
timeout value and on the last_activity clock exactly as before.

check_timeouts keeps reaping everything else: failed connections (all of them,
promptly) and the idle-timeout of legs without a machine timer (inbound legs, and
any machine-less connection). Total coverage is unchanged. check_timeouts runs
before the timer drive, so a failed outbound leg is reaped there first, its timers
dropped, and the timeout drive never double-fires on it.

Split drive_peer_timers into the timeout drive and the (byte-identical) retransmit
drive. The dormant machine timeout handler is not dispatched, so the session index
is freed once, by cleanup_stale_connection. A regression test drives a timed-out
outbound leg to reap; the existing check_timeouts tests continue to anchor the
residual sweep. Test count 1631 to 1632.
 2026-07-15 20:23:49 +00:00
Johnathan Corgan 5021197f5c node: drive the handshake msg1 resend from the per-peer machine timer 
Move the outbound msg1 resend off the unconditional tick function onto the
machine-armed retransmit timer. The per-peer machine already arms a
HandshakeRetransmit deadline at dial; a new drive_peer_timers fires the due ones
(kind-filtered to the retransmit timer) and homes the resend counter on the
machine, where the operator-visible count now reads from.

The resend decision reads the same operator config as before (interval, backoff,
max), the wire bytes and transport target still come from the shell connection,
and the pure core computes the backoff schedule. As with the deleted
resend_pending_handshakes, the count and reschedule advance only on a successful
send: a failed send neither advances the count nor marks the connection failed,
it just retries on the next tick. The handshake-timeout timer stays on the legacy
check_timeouts path, and the rekey/liveness timers keep their own shell drivers,
so drive_peer_timers deliberately fires only the retransmit kind.

The show_connections resend count is relocated to read from the machine (the
counter's new home) via connection_resend_count; machine-less and inbound
connections report 0, matching what the shell connection reported before. Delete
resend_pending_handshakes and resend_candidates (the latter also from the
LifecycleView trait, its only user). The resend unit test is re-expressed against
the machine + timer path, covering the due check and the record-on-success
semantics.

Known limitation: the first resend interval is armed from the machine's hardcoded
1000ms constant, which equals the config default. Under an operator override of
handshake_resend_interval_ms the first resend diverges from the pre-change
dial+interval; subsequent resends and the cap remain config-driven. Neutralizing
the first-resend override needs the interval threaded into the sync core (the
shell arm would be clobbered by the machine's own timer arming at dial), deferred
to when the connection and machine entities merge. Test count unchanged at 1631.
 2026-07-15 19:38:49 +00:00
Johnathan Corgan 31f5a8c1b7 peer: add a per-peer timer store populated by the machine timer actions 
The control machine already emits SetTimer/CancelTimer actions when it arms the
handshake retransmit and timeout deadlines, but their executor arms were a
single no-op stub and the machine's Timeout event was never dispatched -- the
real work still runs on the legacy tick (check_timeouts,
resend_pending_handshakes).

Add the storage the time-as-input driver will read: a peer_timers map keyed by
LinkId then TimerKind, holding each armed timer's absolute deadline. The SetTimer
arm now inserts (overwrite = reschedule) and CancelTimer removes; the outbound
msg2 promote cancels the two dial-armed handshake timers, since the machine
survives promotion and the entries would otherwise linger in the store.

This store is a shadow: it is written and cleared but no driver reads it yet, so
behavior is unchanged. The legacy tick stays authoritative until the driver that
feeds Timeout and deletes the overlapping tick paths lands.

Route every machine removal through a new remove_peer_machine(link) choke-point
that drops the timer store alongside the machine, replacing the twelve direct
peer_machines.remove sites so no armed timer outlives its machine. TimerKind
gains Hash (to key the store) and Ord (for deterministic driver collection); it
is internal and never serialized. Test count unchanged at 1631.
 2026-07-15 18:49:50 +00:00
Johnathan Corgan 3e7ca90212 peer: drive the connection-oriented outbound dial through the machine 
Route the connection-oriented (TCP/Tor) outbound path through the peer state
machine, matching how the connectionless path already works.
initiate_connection's oriented branch now drives PeerEvent::Dial with
connection_oriented=true; the machine parks in Connecting and emits
OpenTransport, whose executor arm performs the non-blocking transport.connect
and pushes the PendingConnect. When the connect resolves, poll_pending_connects
prepares msg1 in the shell and then drives PeerEvent::TransportConnected, which
sends msg1 via the machine's SendHandshake arm.

The msg1 prepare (index allocation, Noise leaf, wire arming) MUST run in the
shell before the TransportConnected drive: send_stored_msg1 only transmits an
already-armed wire, so a drive-only path would silently send nothing. The
machine's our_index stays unset; the connect-failure path keeps its direct
handshake-timeout handling (TransportFailed stays dormant). The now-unused
Node::start_handshake helper is removed.

Behavior-neutral: same transport.connect, same PendingConnect, same msg1 send
and failure teardown as the removed inline path -- only the driver changes from
inline shell code to the state machine.
 2026-07-15 14:40:58 +00:00
Johnathan Corgan 9588c50063 peer: persist the outbound control machine at dial, not at msg1 prepare 
Move the peer_machines insert for an outbound leg out of the tail of
prepare_outbound_msg1 to a single shared site in initiate_connection, before
the connection-oriented / connectionless fork. This lets the connection-oriented
path find the machine at dial time (so it can be driven through the connect
handshake) without prepare_outbound_msg1 -- which for that path runs after the
connect completes -- clobbering an in-progress machine back to Discovered.

Because the machine now exists before the fallible dial steps, add
peer_machines.remove to every failure path in the widened dial window: the
index-allocation and Noise-leaf failures in prepare_outbound_msg1, the oriented
transport.connect failure, and both poll_pending_connects teardown arms
(handshake-start failure and async connect failure). remove_link does not touch
peer_machines, so these explicit removes are required.

Behavior-neutral: the connectionless path still drives the machine to
Handshaking and sends msg1 identically; the machine's our_index stays unset
(no spurious UnregisterDecryptSession on a later inbound restart); a failed dial
leaves peer_machines empty for that link exactly as before; and no
connection-oriented machine drive is wired here.
 2026-07-15 14:12:49 +00:00
Johnathan Corgan f698da50b6 peer: add unit coverage for the connection-oriented outbound dial path 
The connection-oriented (TCP/Tor) outbound connect->handshake path had no
`cargo test --lib` coverage; it was exercised only by the opt-in Tor
integration suites, and the TCP node tests bypass it via a manual
connectionless handshake helper. Add three unit tests:

- a machine-level test driving Dial{connection_oriented:true} -> Connecting
  (emitting only OpenTransport, no msg1) -> TransportConnected ->
  Handshaking{SentMsg1}, asserting the exact OpenTransport and
  SendHandshake+SetTimer action vectors that start_outbound_handshake emits
  (its oriented reach via on_transport_connected was previously untested; the
  connectionless reach via on_dial was already covered);

- two node-level tests over a real loopback TcpTransport: a successful connect
  that reaches start_handshake (observed via a pending_outbound entry), and a
  connect to a closed port that routes through the failure arm (link torn down,
  no msg1 dispatched).

Tests only; no production change.
 2026-07-15 13:52:34 +00:00
Johnathan Corgan 1f765cfd8f peer: drive the connectionless outbound msg1 send through the machine 
Split the outbound handshake setup out of start_handshake into
prepare_outbound_msg1 (allocate the index, run the Noise leaf, frame and arm
msg1 -- the fallible steps, returning an error the caller propagates) and
send_stored_msg1 (transmit the armed wire). A connectionless dial now runs
prepare in the shell, then drives the control machine, whose SendHandshake
action sends the wire via the executor. Connection-oriented dials keep calling
start_handshake, now prepare followed by the send inline.

The dial event gains a connection_oriented flag so the machine's on_dial sends
msg1 immediately for connectionless transports (no connect step) instead of
opening a transport first. Behavior is unchanged: the same index, Noise leaf,
wire bytes, maps, and send-error handling as before, with index-allocation and
Noise failures still propagated synchronously before the send. A regression
test covers that a promote from the post-dial handshaking state is identical to
the former discovered-state promote.
 2026-07-15 06:15:21 +00:00
Johnathan Corgan 3b99a416ad peer: persist the outbound control machine at dial 
Create and persist the per-peer control machine when an outbound handshake is
dialed, keyed by its link, instead of building a transient at msg2. The msg2
completion path now looks up that persisted machine to drive the promote,
falling back to a transient only if none is present (e.g. a direct-seeded test).

The machine parks in the Discovered state until promotion and is inert to the
liveness reap and rekey cadence while unpromoted, since it is absent from the
peers map. It is removed on every path that ends the outbound leg without
promoting -- the stale-connection reaper, the msg2 authorization-failure arm,
and the cross-connection resolution block -- mirroring the connection's own
lifetime so no dangling machine survives.

Its session index is deliberately left unset on the machine (the shell owns the
index on its connection), so a later inbound restart does not emit a spurious
decrypt-session unregister. A regression test covers that invariant.
 2026-07-15 04:06:44 +00:00
Johnathan Corgan e064c96df3 peer: route peer-loss reports by kind 
Add a LostKind discriminator to the ReportLost action so the executor can
route an un-promoted handshake failure to the connected-guarded reconnect
reflex (note_handshake_timeout) and an established peer's link-death to the
unconditional one (note_link_dead), instead of collapsing every loss to
note_link_dead.

The two loss producers dispatched today, the liveness reap and the
inbound-restart-then-promote arm, both keep the link-dead routing, so
behavior is unchanged. The handshake-timeout and dial-failure producers are
tagged accordingly but stay dormant until their events are dispatched.
 2026-07-15 03:37:31 +00:00
Johnathan Corgan a70c725e48 node: drop stale dead-code allows and a redundant clock read 
Remove the now-stale `#[allow(dead_code)]` on `advance_peer_machine`
and `execute_peer_actions`; both are live (called from the link-dead
reap, rekey cadence routing, and the handshake establish sites).

In `route_rekey_cadence`, reuse the ambient context timestamp for the
machine step instead of taking a second clock sample; the extra
sample's only consumer was the currently-inert cutover drain timer, so
the change is behavior-neutral.
 2026-07-15 00:56:58 +00:00
Johnathan Corgan c8077967cd peer: add a round-trip contract test for the action message type 
Prove the per-peer machine's action type is a runtime-agnostic message
contract: it is Send + Sync + 'static, and every variant round-trips
unchanged through a single-threaded async channel (a current-thread runtime,
sender and receiver on one thread). A wildcard-free match over the variants
makes any future action a compile error here until it is added to the sample,
so a variant that embedded a runtime handle could not slip through unproven.
 2026-07-14 23:36:18 +00:00
Johnathan Corgan 5dfa571908 node: pin promote-failure warnings to the handshake tracing target 
The two promote-failure warn! sites in the peer-action executor emit under
this module's own tracing target rather than the handshake target the rest of
the establish-path logging uses. Pin both (outbound "Failed to promote
connection" and inbound "Failed to promote inbound connection") to
fips::node::handlers::handshake, matching the sibling max-peers reject log and
keeping all establish-path diagnostics grouped under one target regardless of
which module physically emits them.
 2026-07-14 23:23:54 +00:00
Johnathan Corgan 6bebca88ac node: rewrite planning-phase locators out of source comments 
Comments across the per-peer machine, executor, lifecycle supervisor,
and peering reconciler carried internal rollout labels and design-note
section references. Rewrite them to describe the code's behavior
directly. Comment-text only; no code changes.
 2026-07-14 01:38:52 +00:00
Johnathan Corgan 5d5da69a5b node: drive the link-dead peer reap through the per-peer machine 
Route each link-dead peer that the tick sweep's plan_heartbeats decides
to reap through the per-peer machine and executor, replacing the inline
reap body in check_link_heartbeats. The batch decision, the liveness
snapshots (read from the hot-path-written receive clock), and the
heartbeat-send arm stay shell-side and byte-unchanged; the machine only
consumes the decided LinkDeadSuspected, tearing the peer down via
remove_active_peer and reporting the loss to the reconciler exactly as
before, on the same tick with the same wall-clock timestamp. The reap
log stays shell-side.

The machine's link-dead handler no longer emits a decrypt-session
unregister keyed by its shadow index (the full peer teardown already
unregisters the real index; the shadow could have drifted to a reused
index), and its guard now covers the Established state a freshly
promoted peer sits in.

Handshake-timeout, retransmit, and stale-connection cleanup stay inline:
they act on pre-promotion legs that have no machine, and the loss reflex
they use differs from the link-dead one.
 2026-07-13 17:08:35 +00:00
Johnathan Corgan e05b868cf8 node: drive rekey cadence cutover and drain through the peer machine 
Route each Cutover and Drain that the shell-side batch poll_rekey decides
through the per-peer machine and the executor, replacing the inline
effect bodies in check_rekey. The batch decision and its per-peer
snapshots stay shell-side and byte-unchanged: poll_rekey phase-groups all
cutovers, then all drains, then all initiations across the peer set, and
that ordering governs the shared index allocator's free-then-allocate
sequence that appears on the wire, so the machine only consumes the
already-decided actions (a new RekeyConsume event) without re-deciding.
InitiateRekey stays inline (its Noise msg1 build is a shell-side leaf)
with a RekeyInitiated observation feeding the machine so its control
state stays coherent for the next tick's cutover.

The cutover and drain logs, which relocated into the executor in the
prior commit, are pinned back to the fips::node::handlers::rekey tracing
target so they stay visible under the operator's module log filter.

Also clears the machine's shadow draining_index on drain so a later
cross-connection resolution cannot double-free the already-freed index.
 2026-07-13 16:20:45 +00:00
Johnathan Corgan 0ebd1b44c0 node: add the initiator-cutover and drain executor actions (unwired) 
Prepare the establish executor to drive FMP rekey by activating the
SwapSendState action (initiator K-bit cutover via cutover_to_new_session,
with the gated decrypt-worker re-registration) and adding a CompleteDrain
action (erase the drained previous session: free its index, drop its
peers_by_index entry, unregister its decrypt session). Both reproduce the
current inline rekey.rs cutover and drain bodies. The machine's drain
mapping now emits CompleteDrain, using the real drained index rather than
a shadow copy.

Unwired: nothing drives the machine's rekey path yet (live rekey still
runs inline), so these arms are unreachable and the change is
behavior-neutral. The cadence fold that routes cutover and drain through
the machine follows.
 2026-07-13 14:51:11 +00:00
Johnathan Corgan 800cfb23e3 node: relocate decrypt-session registration to the establish executor 
Move register_decrypt_worker_session out of promote_connection into the
executor's PromoteToActive handler, gated on a promoted or
cross-connection-won result. Every live promote now flows through that
one executor path, so registration still fires exactly once at the same
synchronous point; the direct test callers of promote_connection spawn
no worker pool, so the call was already a no-op for them.

Add the cross-connection loser-link teardown (close the losing
transport, remove its link, re-point addr_to_link at the winner) to the
executor as a guarded follow-up. It is unreachable on the current driven
establish paths, which only promote net-new peers, and asserts so, but
keeps the executor complete for when that case is driven.

Remove the now-dead drive_promote_to_active and ConnAction::PromoteToActive.
 2026-07-13 14:20:56 +00:00
Johnathan Corgan e9112cc1bb node: drive net-new outbound establish through the per-peer machine 
Cut the net-new outbound handshake completion (a received msg2 that
promotes a fresh outbound leg to a new peer) over to the per-peer
control machine, mirroring the inbound cutover. handle_msg2 still runs
the msg2 prologue, the ACL check, and the cross-connection swap/keep
arms inline; for the net-new promote it now builds a transient machine,
steps it, and drives promote_connection through the executor. The
session index was already allocated at dial, so there is no two-phase
authorize here.

The wire, index sequence, and peer registry state after promote are
byte-neutral. To keep the promote-failure path neutral, the executor's
cleanup now distinguishes inbound from outbound: an outbound promote
failure records the reject only, matching the prior handler, rather
than the inbound path's link and index teardown.

Cross-connection swap/keep, rekey-msg2, and the dial path stay inline;
the loser-link surgery for the currently unreachable driven
cross-connection case lands with the register relocation next.
 2026-07-13 13:42:08 +00:00
Johnathan Corgan c80a7fdea5 node: drive restart inbound establish through the per-peer machine 
Cut the restart handshake (an inbound msg1 from a peer that reconnected
with a new epoch) over to the per-peer control machine, completing the
inbound establish cutover. The old peer is torn down and the fresh leg
promotes through the same two-step authorize-then-allocate path as the
net-new case: the machine's first step emits the old-peer teardown
(invalidate send-state, report loss), the shell interposes the ACL
check, and the second step allocates the new index and sends msg2. The
old index is freed before the new one is allocated and the msg2 wire
bytes are unchanged, so the sequence stays byte-neutral.

With restart driven through the machine, the shared inline establish
tail that only the restart arm reached is deleted.

Also bounds the peer_machines map (remove_active_peer now drops the
peer's machine entry) and restores the msg2-send, promote, and
index-allocation failure warnings the cutover had dropped.
 2026-07-13 12:46:23 +00:00
Johnathan Corgan 0bf031dd32 node: drive net-new inbound establish through the per-peer machine 
Cut the net-new inbound handshake (a fresh msg1 that promotes to a new
peer, plus the at-capacity reject) over from the inline handle_msg1 logic
to the per-peer control machine. handle_msg1 still classifies via
establish_inbound and still owns the Noise wire step, the late ACL check,
and the promote_connection registry surgery; for the net-new path it now
builds the machine, steps it, and executes the returned actions.

Authorization is interposed between two machine steps so the session
index is allocated only after the ACL check passes: a rejected or
unauthorized msg1 consumes no index, matching the prior order exactly.
The msg2 wire bytes, the index-allocation sequence, and the reject
metrics are all byte-neutral. Restart, resend, rekey-respond, and the
other reject arms stay inline unchanged; they move to the machine once
outbound establish is cut over and every promoted peer has a machine.

Also fills in the executor's send-failure and promote-failure cleanup so
a mid-establish error tears the leg down and frees its index exactly as
before.
 2026-07-13 12:05:17 +00:00
Johnathan Corgan 4a0584a5e9 node/dataplane: add the per-peer machine home and action executor (unwired) 
Add Node.peer_machines (a LinkId-keyed map from the stable link handle to
the per-peer control machine) as the home for the machines, and a new
dataplane/peer_actions.rs holding execute_peer_actions / advance_peer_machine:
the executor that maps each PeerAction the machine emits to its shell call —
frame and send a handshake via build_msg2, drive promote_connection and feed
the PromotionResult back through the machine, tear a peer down via
remove_active_peer, free session indices, report loss via note_link_dead.
Actions for the rekey, connected-UDP, and timer paths are stubbed with notes
for the commits that fold those mechanisms in.

Unwired: nothing drives the machine yet — no live handler path calls the
executor and peer_machines is never populated — so this is behavior-neutral;
the inbound and outbound establish paths still run their existing inline
logic. The executor is cut over path-by-path in the following commits.
 2026-07-13 11:11:20 +00:00
Johnathan Corgan 59155df4e3 peer: split ActivePeer send-state into PeerSendState (two-tier boundary) 
Draw the control/published-send-state boundary inside ActivePeer by
grouping the send-critical fields — the three epoch session slots
{current, previous, pending}, the K-bit flag and session-start, the
transport target, the connected-UDP handles, and the hot counters —
into a new co-located PeerSendState struct. The control-tier fields
(identity, connectivity, declaration/ancestry, filter and tree-announce
groups, remote_epoch, the rekey-negotiation sub-machine, and the rest)
stay on ActivePeer.

Behavior-neutral: a pure field regrouping. Every accessor signature is
unchanged (bodies now read/write self.send.*), so the hot path and the
handlers are byte-untouched; both K-bit cutovers still rotate the three
slots atomically with the same control-tier updates. No Arc/ArcSwap —
the fields are co-located and read by plain borrow; publishing behind a
shared cell is later plumbing for a sharded data plane.
 2026-07-13 10:36:15 +00:00
Johnathan Corgan fcaee74ec0 peer: add the per-peer FMP control state machine (unwired) 
Introduce src/peer/machine.rs: a sans-IO per-peer control FSM that
consolidates the scattered handshake/rekey/timeout driver logic now
spread across node/handlers. The machine is a pure reducer —
step(event, now, index_allocator) -> [action] — that reuses the
existing FMP decision cores (establish_inbound/establish_outbound/
cross_connection_winner/poll_*) rather than reimplementing any
decision, and returns runtime-agnostic actions the driver executes.

Control-tier state only; the published send-state boundary and the
driver wiring land in following commits. The machine is terminal at
Closed — re-dial is the reconciler's, so it holds no cross-attempt
retry state.

Includes eight unit tests: inbound and outbound establish, N:1
identity crystallization, the dual-initiation tie-break,
restart-override, rekey initiator cutover, the data-plane-owned
responder cutover boundary, and liveness -> link-dead -> report-lost.
Unwired — nothing calls it yet.
 2026-07-13 09:58:45 +00:00
Johnathan Corgan 56e3d56c25 peer/connected_udp: own the connected-socket fd with OwnedFd 
Replace the hand-rolled RawFd field and the unsafe Drop on
ConnectedPeerSocket with an OwnedFd, whose own drop glue closes the
fd. from_fd now stores the OwnedFd it already receives instead of
stripping ownership through into_raw_fd, and the manual libc::close is
gone, shrinking the unsafe surface.

Behavior-neutral: the fd still closes exactly once at last-Arc-drop and
as_raw_fd returns the same underlying fd while the socket is alive.
 2026-07-13 09:20:04 +00:00
Johnathan Corgan 7b0590f70e node/lifecycle: wire the runtime child-exit producer 
Wire exit detection for the four directly-observable optional children so
the supervisor FSM's ChildExited edge fires at runtime. A runtime
child-liveness mpsc channel carries a Child on exit: the DNS task and the
two TUN threads self-report when their body returns, and a 2s poll monitor
reports mDNS and Nostr via new is_finished accessors. The rx_loop gains a
select arm that steps the FSM and republishes health (Degraded, since a
running node always has at least one transport up). Transports and worker
pools expose no runtime-exit signal and are left for a follow-up.

Adds LanRendezvous::is_finished and NostrRendezvous::is_finished; the
latter treats a shutdown-taken connect_task as finished so the monitor
terminates after a stop instead of polling forever.
 2026-07-13 08:48:51 +00:00
Johnathan Corgan b93a127623 node/lifecycle: add runtime ChildExited health routing to the supervisor FSM 
Add the sans-IO half of runtime child-liveness monitoring: a ChildExited
event and an on_child_exited handler that routes a runtime task or thread
exit the same way a start failure does. An optional child exiting degrades
the node (Degraded, with the child recorded in the health reasons); the
last transport exiting publishes Failed. There is no restart, and the
handler is inert outside Running (startup, drain, and teardown own their
own child bookkeeping via the pending/up sets). Failed here is a published
health signal only, not a teardown.

The start and runtime paths share the health classification, extracted
from resolve_start_health into classify_health.

The producer that emits the event (the exit-detection wiring) lands in a
following commit; the event variant carries a temporary allow(dead_code)
until then.
 2026-07-13 08:25:45 +00:00
Johnathan Corgan 85a4983dbe node/peering: restore the open-discovery sweep summary log 
The overlay-discovery cutover to the sans-IO reconciler dropped the
operator-facing "open-discovery sweep complete" summary, including its
per-reason skip breakdown, because the sweep logic moved into the
log-free core. Have the core accumulate the enqueue/skip tally as it
reconciles and return it as data; the driver adds the two values only it
holds (the raw cache size and the self-advert filter) and emits the
summary, reproducing the old startup-vs-per-tick summarize gate and the
budget-zero debug path.

The self-advert precedence matches the pre-cutover sweep: a stale or
self-configured own advert is attributed to the age/configured buckets,
not skipped_self.

Behavior-neutral: the tally is pure side-counting; no dial or enqueue
decision changes.
 2026-07-13 08:00:46 +00:00
Johnathan Corgan 5090ab7851 node/peering: drive opportunistic transport-neighbor growth through the reconciler 
Cut the last scattered peering mechanism — opportunistic growth from
transport-neighbor beacons and LAN mDNS — over to the reconciler's
opportunistic layer via a gate-checked reconcile_opportunistic wrapper,
one call per tick slot (transport and LAN stay separate slots so their
per-tick budget and per-peer cap are not shared). The driver keeps the
beacon/mDNS I/O and the path-granular prefilters that read live state
(self, fresh-enough-to-skip, connecting-on-path) and executes the emitted
Connect intents; the core owns the connected/budget/per-peer-cap
decisions. A first-wins per-peer dedup on the LAN path reproduces the old
inline once-per-peer dial now that the snapshot core cannot observe the
intra-tick connecting feedback.

Delete the now-dead discovery_connect_budget helper. With this, all three
scattered peering mechanisms (auto-connect and retry, overlay discovery,
neighbor growth) are unified in one sans-IO reconciler. Behavior-neutral;
unit test count unchanged (1607).
 2026-07-13 06:02:21 +00:00
Johnathan Corgan 03ced618ce node/peering: drive Nostr open-discovery through the reconciler overlay layer 
Cut the overlay (Nostr open-discovery) enqueue over to the sans-IO
reconciler. A gate-checked reconcile_overlay wrapper runs the overlay
layer alone at the discovery tick slot; the monolithic reconcile would
re-fire the always-on retry-dial and double the per-tick dial cap, so the
overlay slot must call only its layer. The driver builds the candidate
pool from the overlay advert cache (self excluded), the cooldown and
configured-npub sets, and the startup/steady max-age, then feeds the
reconciler; the emitted enqueue set drives the per-enqueue identity-cache
and alias pre-seed exactly as before. Enqueued entries are dialed at the
retry slot, preserving the two-phase cadence.

Delete the now-dead enqueue-budget and expiry helpers. Behavior-neutral:
same candidates enqueued in the same order with the same budget, cap, and
expiry. Opportunistic transport-neighbor growth remains imperative and is
cut over next. Unit test count unchanged (1607).
 2026-07-13 05:34:28 +00:00
Johnathan Corgan bf81f422ea node/peering: drive the mandatory-floor and retry mechanism through the reconciler 
Route the auto-connect floor and the connection-retry mechanism through
the sans-IO reconciler instead of the imperative Node methods. A thin
peering driver (note_handshake_timeout / note_link_dead) centralizes the
reflex path with the gate guard and the already-connected check; the
per-tick retry-dial and the startup floor build reconciler inputs and
execute the emitted Connect intents. The three imperative retry methods
are removed and their call sites rerouted.

Wire the drain and startup gates. Entering a bounded drain now clears the
retry schedule and suppresses the peer-loss reconnect reflex (the drain
window runs under the Suspended gate), so the drain no longer fights a
reconnect for the peers it just closed. The startup floor runs under an
explicit Reconciling gate at the existing peer-connect seam, preserving
the current dial position.

Behavior-neutral except the intended drain change. Overlay discovery and
opportunistic transport-neighbor growth remain imperative for now; they
observe the relocated retry schedule and are cut over next. Unit tests
migrated to the driver API with assertions unchanged (1607).
 2026-07-13 05:00:10 +00:00
Johnathan Corgan a0cf593580 node: relocate the peering fields off Node into the Peering owner 
Move retry_pending and pending_connects from Node into the Peering owner
struct (Node.peering), the home introduced with the reconciler core.
Pure mechanical relocation: the two collections now live behind
self.peering, with roughly seventy accessor sites repointed. No decision
logic is touched and the unit-test count is unchanged (1607). The
imperative peering methods still run; the reconciler is wired in the
following commit.
 2026-07-13 04:24:34 +00:00
Johnathan Corgan 5d08d27d3c node/peering: add the sans-IO peering reconciler core (unwired) 
Introduce PeeringReconciler, the synchronous decision core for peer
desired-state, with its input/action vocabulary (Gate, Budget, Observed,
Candidate, DiscoveryPools, Policy, PeeringAction) and the Peering owner
struct. reconcile() computes connect/retry intents across four layers:
the auto-connect mandatory floor, the ceiling-only overlay enqueue, the
opportunistic transport-neighbor growth, and the node.limits ceiling
enforced inline. It mirrors the supervisor sans-IO shape: no I/O, no
clock reads (time enters as a parameter), no runtime handles.

The core owns its own cross-attempt retry schedule so escalating backoff
survives the fresh connection created per re-dial. Gate-guarded reflexes
(on_handshake_timeout / on_link_dead) reproduce the current backoff math
and no-op while draining. Overlay is strictly ceiling-only (no peer-count
set-point). Disconnect is defined but never emitted (no shedding).

Unwired here: the driver still runs the imperative peering methods and
Node holds the live retry map. 11 unit tests drive the pure core with
synthetic inputs. No behavior change.
 2026-07-13 04:10:36 +00:00
Johnathan Corgan b676c9d83a node: rewrap over-width retry_state_iter signature 
Homing the retry schedule under peering/ lengthened the RetryState
module path, pushing the retry_state_iter return-type line past the
100-column max_width. Rewrap to satisfy rustfmt. No behavior change.
 2026-07-13 04:00:41 +00:00
Johnathan Corgan a45eefb58a node: home the peering concept; relocate the connection-retry schedule 
Create src/node/peering/ as the home for the peer desired-state
(homeostatic reconciler) concept and move the cross-attempt connection
retry schedule into it: RetryState plus schedule_retry /
schedule_reconnect / process_pending_retries.

Mechanical relocation only. The methods remain inherent on Node; the
retry_count must persist across re-dials (a fresh connection is created
each attempt), which is why the schedule belongs in the peering home
rather than on a per-connection type. The one-level-deeper module path
requires pub(super) -> pub(in crate::node) to preserve the prior scope,
plus module-path fixups at the reference sites. No behavior change; unit
test count unchanged (1596).
 2026-07-13 03:30:52 +00:00
Johnathan Corgan d61d189572 node: split Running into Full/Degraded, add Failed health state 
Determine node health at start completion instead of unconditionally
reaching Running. Zero transports up is now Failed (fatal): start()
tears down cleanly and returns an error, and the daemon exits. Any
configured optional child that failed to start - a transport beyond the
first, Nostr, mDNS, TUN, DNS, or a worker pool - leaves the node
Degraded but serving, with an operator warning naming what failed. All
configured children up is Full. A child the node was never asked to run
does not count against health.

The published NodeState gains Degraded and Failed variants, both visible
via control queries; Degraded is operational, Failed is not. The
lifecycle FSM gains the health states plus the PublishState action that
drives them - a health fork cannot be a single direct state write, which
is why the earlier commits deferred it to here.

Runtime child-exit health re-evaluation (a running child dying) is a
separate liveness-monitoring mechanism left for a follow-up; this commit
is start-time health only.
 2026-07-13 00:46:31 +00:00
Johnathan Corgan d6ca632251 node: add bounded graceful-shutdown drain phase 
Add an operator-visible Draining phase on daemon shutdown. On the
shutdown signal the node broadcasts Disconnect to all peers, then keeps
serving for a bounded window - up to node.drain_timeout_secs (default 2s),
exiting early once all peers are gone - before tearing down. This lets
in-flight traffic settle and peers observe the disconnect before the
transports close, rather than the previous immediate teardown.

The lifecycle FSM gains a Draining state plus Drain/DrainDeadlineElapsed
events; the run loop observes the shutdown signal and transitions to
draining in place - one continuous loop, so the channel receivers are
never destructively cancelled. The published NodeState gains a Draining
variant, visible via control queries during the window. The immediate
stop() path used by tests and non-daemon callers is unchanged: it still
tears down immediately with no drain wait.

The reconciler-gate actions the drain emits are no-ops until the peering
reconciler lands and consumes them.
 2026-07-12 23:11:28 +00:00
Johnathan Corgan 6c5fd3f4b0 node: extract lifecycle supervisor FSM, migrate substrate fields (behavior-neutral) 
Introduce a sans-IO lifecycle supervisor: a synchronous step(event) ->
[action] state machine (SupervisorFsm) that authors the substrate-child
spawn and teardown order, plus an owner struct (Supervisor) holding the
substrate runtime fields and embedding the FSM. The substrate-lifecycle
fields leave Node's flat list into the owner - state, packet_tx, the TUN
reader/writer handles + shutdown fd + channels, the DNS task + identity
channel, the Nostr/LAN rendezvous drivers, and the encrypt/decrypt worker
pools; the dataplane keeps packet_rx.

start()/stop() become the driver executing the FSM's SpawnChild/StopChild
actions: same children, same order, same warn/debug-and-continue on
optional failures, same logs, same NodeState transitions. Behavior- and
wire-neutral. The only determinism change is that transports now tear
down in ascending-id order, previously nondeterministic HashMap iteration.

The bounded Draining phase and the Running{Full|Degraded} health split
land as separate follow-on commits.
 2026-07-12 21:03:46 +00:00
Johnathan Corgan 434b9726aa node: establish dataplane/ and session/ concept homes (behavior-neutral) 
Reorganize the node module tree by concept rather than by
message-handling verb, as the first step of the node runtime
decomposition. Pure relocation: no wire, config, metric, or log
semantics change; the lib test count is unchanged (1577 passed).

Moves (git mv, 100% rename similarity):
- handlers/{forwarding,rx_loop,connected_udp,dispatch,encrypted}.rs
  -> node/dataplane/ — the whole RX hot path (the select! run loop,
  transit/local forwarding, the link-message router, the RX decrypt
  path with responder K-bit cutover + roam writes, and connected-UDP
  fast-path activation) now lives in one home.
- node/session.rs -> node/session/mod.rs — establishes the session
  concept home for the data/state types. The message-behavior file
  handlers/session.rs stays put for now (folds in with the later FSP
  session step).

The IK/XX-divergent establishment files (handlers/{handshake,rekey,
timeout}.rs) and the deferred-home files (handlers/{mmp,lookup}.rs)
deliberately stay in handlers/, to move once rather than twice.

Every module is reached through impl Node methods, so no call site or
re-export shim was needed. Updated in lockstep with the moves: the
module_path!-derived tracing targets in the two mesh-lab compose-trace
overlays, a structural test's include_str! source path, doc-comments
in proto/routing and the mesh-lab docs, and the stale source-location
citations (node/handlers/{forwarding,rx_loop,encrypted}.rs and
node/session.rs) in doc-comments and the discovery design doc.
 2026-07-12 19:22:08 +00:00
Johnathan Corgan 26d70ebb59 Merge branch 'maint' into master 
Forward-merge three maint fixes, hand-relocated into master's
post-sans-IO / discovery-to-lookup structure:
- drop the redundant TreeMetrics parent_switched counter (keep
  parent_switches), reconciled across the refactored tree/mmp sites and
  the spanning-tree test reads
- keep the tighter path_mtu when applying a LookupResponse, now in
  handlers/lookup.rs after the discovery module rename
- reuse one shared secp256k1 context in the identity module
 2026-07-12 16:51:01 +00:00
Johnathan Corgan 9b46b6fa85 fipstop: rename routing-stats "Discovery" labels to "Lookup" 
The routing-stats pane's Discovery Requests/Responses sections show the FMP
overlay coordinate-lookup counters. Rename the section labels and the nested
JSON key they read from "discovery" to "lookup" to match the metric family's
canonical name. The daemon dual-emits both keys, so this reads the current
name and no longer depends on the deprecated "discovery" alias.
 2026-07-12 01:52:54 +00:00
Johnathan Corgan cbc089b820 transport/ethernet: rename config discovery flag to listen 
Rename the Ethernet per-interface config flag from discovery to listen,
so the receive/transmit toggle pair reads as the symmetric announce
(transmit) / listen (receive) neighbor-beacon vocabulary. The old
discovery: key is still accepted via a serde alias, so deployed configs
load unchanged; to_yaml re-emits it under the canonical listen: name.
Marked deprecated for removal at the v2 cutover.

Updates the config field + accessor, the transport listen_enabled local,
the one struct-literal test consumer, the chaos sim config generator,
packaged fips.yaml examples, and the classified operator-facing docs
(ethernet neighbor-beacon subsystem prose; the generic Transport
discovery capability prose is left unchanged). Adds a compat parse test
asserting the legacy alias, the new key, and that deny_unknown_fields
still rejects unknown keys. Behavior-neutral.
 2026-07-11 23:13:28 +00:00
Johnathan Corgan 4c95be0000 transport/ble: rename discovery module to neighbor 
Mirror the ethernet neighbor rename in the BLE transport: the internal
peer-detection buffer becomes NeighborBuffer (from DiscoveryBuffer) and
the module is renamed discovery -> neighbor. The BlueR/bluez API terms
(DiscoveryFilter, set_discovery_filter) and the BLE advertise/scan
mechanism vocabulary are unchanged, as is BleConfig. Behavior-neutral.
 2026-07-11 23:01:48 +00:00
Johnathan Corgan e362ab67a6 transport/ethernet: rename discovery module to neighbor 
Rename the ethernet link-local neighbor-beacon subsystem from the
overloaded "discovery" vocabulary to a neighbor umbrella. The beacon
frame vocabulary is retained (build_beacon/parse_beacon/BEACON_SIZE/
FRAME_TYPE_BEACON/FRAME_TYPE_DATA); only the subsystem and buffer
identifiers change: DiscoveryBuffer becomes NeighborBuffer,
discovery_buffer becomes neighbor_buffer, and DISCOVERY_VERSION becomes
BEACON_VERSION (wire value 0x01 unchanged). Config toggles are left for
a follow-up commit. Behavior-neutral.
 2026-07-11 22:56:05 +00:00
Johnathan Corgan 6c9f55ea80 transport: rename darwin_sockopts to sockopts_macos 
The module is gated target_os="macos" (not the broader Darwin/iOS family),
so the name now tracks the cfg and matches the *_macos.rs file convention
(io_macos.rs). Drops the redundant inner #![cfg(target_os="macos")] (the
decl gate already covers it) and corrects a stale top comment that claimed
the module stays visible on Linux — it is macos-decl-gated, so it never was.
Mechanical rename; no logic change.
 2026-07-11 21:37:00 +00:00
Johnathan Corgan 89a31fd555 transport: move the connected-UDP fast-path handles into the peer module 
ConnectedPeerSocket and PeerRecvDrain are node/peer-side logic: the Transport
trait never touches them, ActivePeer stores them, and node's encrypt worker
drives them. They only happened to live under transport/udp. Relocate the
handle types to a new src/peer/connected_udp/ module (socket.rs + drain.rs),
which the node handler and encrypt worker reach via node -> peer (no new edge;
a node home would have forced a peer -> node cycle).

The udp transport keeps only the kernel-construction seam: open_connected_fd,
now folded into udp/io.rs (the byte-layer home) behind a linux/macos-gated
submodule and re-exported as transport::udp::open_connected_fd. The node
handler builds the fd through it and adopts it via ConnectedPeerSocket::from_fd.

Behavior-neutral: no wire/config/metric/log change; the per-packet hot path
(bare-RawFd send_batch_gso/raw) is untouched. Preserves the Arc multi-owner
contract, the drop-drain-before-socket ordering, and the drain's detach-on-Drop
deadlock avoidance. Reconciles the old cfg(unix)/any(linux,macos) double-gate
onto the single any(linux,macos) predicate.
 2026-07-11 21:30:36 +00:00
Johnathan Corgan ab0a46f2c0 transport: extract udp open_connected_fd syscall body into a standalone fn 
Pulls the connected-UDP socket construction (socket/REUSEADDR/REUSEPORT/
BUFFORCE/bind/connect + darwin tuning) out of ConnectedPeerSocket::open into
a pub(crate) open_connected_fd returning an OwnedFd; open() now delegates and
adopts the fd. Error paths still close the fd via the temporary's Drop; the
success tail transfers ownership through OwnedFd. Behavior identical. Prepares
the fast-path handle types to move to the peer module while the socket
construction stays with the udp transport.
 2026-07-11 21:08:29 +00:00
Johnathan Corgan e839aead7a transport: extract tcp connection-pool types into pool.rs 
Moves the inline TcpConnection/ConnectingEntry structs, the Direction
enum, and the ConnectionPool/ConnectingPool type aliases out of the
1147-line tcp/mod.rs into a dedicated pool.rs, matching the canonical
per-transport layout. Struct fields are pub(crate) so mod.rs can still
construct and read them across the module boundary. Pure relocation; no
logic change.
 2026-07-11 20:26:42 +00:00
Johnathan Corgan 6d6889d0f6 transport: give ethernet a canonical addr.rs home for MAC parsing 
Moves parse_mac_string into ethernet/addr.rs and re-exports it at the
ethernet module root so the ethernet::parse_mac_string path stays
byte-identical for its one external consumer (zero churn there). Its
unit tests stay in mod.rs, calling through the re-export. Pure
relocation; no logic change.
 2026-07-11 20:19:08 +00:00
Johnathan Corgan 196d9492da transport: rename ethernet byte-layer module from socket to io 
Normalizes the ethernet transport onto the canonical byte-layer name
(io.rs), including the per-OS files io_linux.rs/io_macos.rs and their
#[path] wiring. Pure file rename plus module-path and doc-comment
updates; no logic, wire, config, metric, or log change.
 2026-07-11 20:14:37 +00:00
Johnathan Corgan 0f2e91b479 transport: rename udp byte-layer module from socket to io 
Normalizes the udp transport onto the canonical per-transport module
layout (ble is the reference: io/pool/addr/stats). Pure file rename plus
module-path updates at the use sites; no logic, wire, config, metric, or
log change. Behavior identical.
 2026-07-11 20:10:19 +00:00
Johnathan Corgan 32475d859e transport: consolidate tor and nym SOCKS5 dialing into a shared module 
The Nym transport was a near-clone of the Tor transport's outbound
SOCKS5 path. Extract the shared logic into src/transport/socks5/ so both
transports drive one implementation instead of two maintained copies:

- share one SOCKS5 mock server between the tor and nym tests
- extract the common send/receive/connect counters into a shared
  ProxiedStatsBase (snapshot structs and emitted metrics unchanged)
- add a shared Socks5Dialer that collapses all six connect variants;
  the sole dialing difference (tor's per-destination circuit-isolation
  auth vs nym's no-auth) is an enum on the dialer
- route both tor and nym dialing through the shared dialer
- share the proxied connection pool, generic over a per-connection meta
  type that carries tor's inbound/outbound direction counting (nym uses
  the unit type)
- share the proxied receive loop

Behavior-neutral: no wire-format, config-key, emitted-metric, log, or
error-variant change. Tor keeps its control-port, inbound/onion, and
directory surface and its own address validation.
 2026-07-11 18:45:41 +00:00
Johnathan Corgan f2e6b8befb transport: promote FMP stream framing to transport::framing 
The FMP frame-boundary reader lived in transport/tcp/stream.rs, but the
Tor and Nym transports both reached across module boundaries to
'use crate::transport::tcp::stream::read_fmp_packet', a layering smell:
the reader is a shared stream-framing utility, not a TCP-private one.
Move the file to transport/framing.rs and repoint the tcp/tor/nym use
sites, removing the tor->tcp and nym->tcp dependencies. Behavior
unchanged.
 2026-07-11 04:32:38 +00:00
Johnathan Corgan 1aacdfa086 transport: share connection-pool counters via PoolCounters 
TcpStats and TorStats each carried an identical pair of pool_inbound/
pool_outbound atomics plus the same five record_pool_* / pool_inbound_count
methods over them. Move the counter logic into a shared PoolCounters
struct (new transport/stats_common.rs) embedded as a 'pool' field in both.
The public record_pool_* methods stay as thin delegators so all call
sites and the flat pool_inbound/pool_outbound snapshot fields are
unchanged; only the duplicated atomic bookkeeping is now single-sourced.
 2026-07-11 04:30:19 +00:00
Johnathan Corgan a7dfe47663 transport: derive Serialize for EthernetStatsSnapshot 
EthernetStatsSnapshot derived only Clone/Debug/Default, unlike every
other transport's *StatsSnapshot which also derives Serialize. That gap
forced a hand-rolled serde_json::json!{} arm in TransportHandle::
transport_stats() that re-listed all ten fields by hand. Add the
Serialize derive and collapse the arm to the same one-line
serde_json::to_value(...) form the other transports use. The emitted
JSON keys and values are unchanged.
 2026-07-11 04:27:50 +00:00
Johnathan Corgan 8aab71af86 bench: add routing next-hop microbench 
Measures the per-forwarded-packet cost of routing candidate assembly
(routing_candidates over a synthetic RoutingView) against a zero-alloc
reference across 8/32/128/256 peers, with per-call allocation counts via
a counting allocator. Criterion harness; no production code change.
 2026-07-11 01:48:34 +00:00
Johnathan Corgan 2cffc10520 Merge branch 'maint' (bloom single-digest double-hashing) 
Forward-merges the bloom SHA-256-once fix from maint. The bloom filter was
relocated to proto/bloom/ by the sans-IO refactor, so the fix applied cleanly
to proto/bloom/core.rs; the behavior-neutral test was re-homed into
proto/bloom/tests/core.rs (maint carried it in the pre-split bloom/tests.rs).
 2026-07-11 01:32:48 +00:00
Johnathan Corgan 1c1ed0d939 Relocate PromotionResult into proto/fmp 
Move the PromotionResult enum (and its impl) out of peer::mod and into
proto/fmp/core.rs, alongside the cross_connection_winner tie-break helper
that was relocated the same way. This is FMP connection-lifecycle result
vocabulary, so it belongs in the FMP subsystem home rather than the peer
module.

Behavior-neutral pure type relocation: consumers import it from
crate::proto::fmp, and the crate-root public path crate::PromotionResult
is preserved via a re-export in lib.rs (mirroring cross_connection_winner).
Full lib suite green at baseline.
 2026-07-10 16:52:18 +00:00
Johnathan Corgan 1c41f73931 Relocate FMP link wire codec into proto/fmp 
Move the FMP mesh-layer wire format (common prefix, encrypted/msg1/msg2
headers, and the build_*/inner-header codec fns) out of node/wire.rs and
into proto/fmp/wire.rs, so the whole FMP wire surface lives with its
subsystem, matching the proto/fsp/wire.rs layout. The wire module becomes
pub(crate) mod wire; callers reach it via crate::proto::fmp::wire.

Behavior-neutral: pure relocation plus import-path rewrites across the
node/peer consumers; no logic change. Full lib suite green at baseline.
 2026-07-10 16:42:17 +00:00
Johnathan Corgan 39ad4d2e67 Extract the nostr rendezvous decision logic into sans-IO state machines 
Carve the nostr rendezvous engine's decision logic out of the async
NostrRendezvous driver into synchronous, clock-injected cores, following
the failure_state.rs pattern: sync state behind std::sync::Mutex, time
passed in as now_ms, no .await and no I/O in the core. The driver performs
all relay/socket I/O and executes the returned plans.

- AdvertMachine (src/nostr/advert.rs): advert publish/cache/fetch/prune
  decision logic; the driver executes a returned PublishPlan.
- TraversalMachine (src/nostr/traversal_machine.rs): the engine-scoped
  cross-session decision state -- in-flight-initiator dedup, the dual-init
  responder suppression election, and the replay/seen-session cache.
- classify_punch_packet (src/nostr/traversal.rs): a pure classifier for
  the punch recv loop's packet branch table.

The driver keeps all I/O, the NAT punch send cadence, the offer-slot
admission semaphore, and the pending-answer oneshot routing. Behavior-
neutral: no wire, config-key, or metric change. Adds unit coverage for
the advert plans, election ordering, replay eviction, initiator dedup,
and punch classification.
 2026-07-10 03:28:11 +00:00
Johnathan Corgan 4d2504f59d Update trace-target references for the relocated rendezvous modules 
Moving the nostr rendezvous engine to src/nostr/ (and mDNS to src/mdns/)
changed the module-path-derived tracing targets from
fips::discovery::nostr::* to fips::nostr::*. Update the RUST_LOG filters
in the NAT and mesh-lab test compose files and the resolve-peers-via-nostr
tutorial's debug recipe to the new targets so trace configs and the
documented journal-watch command keep emitting the intended lines.

Without this, the stun-faults suite's Phase-0 pre-flight (which greps the
daemon journal for the debug-level "STUN observation succeeded" /
"traversal: initiator STUN observed" lines) saw those lines suppressed —
the daemon behaved correctly, but the stale RUST_LOG target hid the
evidence. Test-harness and docs only; no source or behavior change.
 2026-07-09 23:20:28 +00:00
Johnathan Corgan 1208f6a5c2 Consolidate the nostr rendezvous driver into src/nostr 
Pull the rendezvous driver state and the movable driver logic out of the
Node struct into the src/nostr home. A new RendezvousDriver owns the
engine handle and the four bookkeeping fields (traversal start time,
startup-sweep latch, adopted bootstrap-transport set and their npubs)
that previously sat loose on Node; Node holds a single driver field and
reaches the same data through thin accessors, including the rx-loop
hot-path protocol-mismatch hook.

The advert build/refresh, the via_nostr fallback-address resolve, the
overlay-endpoint-to-PeerAddress mapping, and the bootstrap request move
onto the driver, taking their Node inputs explicitly (a transport-
endpoint snapshot for advert building) rather than reading Node fields
directly. Transport/connection-table-bound work (traversal adoption,
bootstrap-transport cleanup, the open-discovery sweep, and the outbound
budget calculators) stays on Node as thin glue that calls into the
driver.

Behavior-neutral relocation: statements moved verbatim, no logic, wire,
config-key, or metric changes. lifecycle.rs shrinks ~230 lines. cargo
fmt/build/clippy clean; lib suite 1547 passing (baseline unchanged).
 2026-07-09 22:13:56 +00:00
Johnathan Corgan 3f80530cc5 Move nostr peer rendezvous and mDNS into dedicated module homes 
Relocate the overlay peer-rendezvous subsystem out of the overloaded
src/discovery/ tree into two focused, independent homes: src/nostr/
(relay-mediated overlay endpoint advertise/resolve/auto-mesh plus NAT
traversal) and src/mdns/ (link-local DNS-SD rendezvous). The two
subsystems are independent, so they get separate homes rather than
sharing one.

Drop the ambiguous "Discovery" stem from their identifiers in favor of
"Rendezvous": NostrDiscovery -> NostrRendezvous, LanDiscovery ->
LanRendezvous, and the matching config, policy, field, and method names.
The former src/discovery.rs handoff types (EstablishedTraversal,
BootstrapHandoffResult, the punch-packet helpers) fold into
src/nostr/handoff and stay reachable via the crate-root re-exports.

Pure relocation and rename: no logic, wire-format, config-key, metric,
or tracing-target changes. The operator-facing node.rendezvous.nostr.*
and node.rendezvous.lan.* config keys and the fips-overlay-v1 advert
namespace are byte-identical. cargo fmt/build/clippy clean; lib test
suite 1547 passing (baseline unchanged).
 2026-07-09 21:51:54 +00:00
Johnathan Corgan 3b401a0cbd Disambiguate the mesh-lookup "discovery" name across the source tree 
The identifier "discovery" named three unrelated subsystems; the FMP
overlay coordinate-lookup subsystem is now consistently "lookup". This
finishes the concept-#1 rename across the shell, config, and metric
layers left after the earlier proto-layer rename:

- Handler module node::handlers::discovery -> node::handlers::lookup, and
  reset_discovery_backoff -> reset_lookup_backoff.
- The Node lookup-engine field Node.discovery -> Node.lookup, renamed by
  resolved binding so the metrics().discovery and node.discovery config
  paths are left untouched.
- The lookup metric types DiscoveryMetrics -> LookupMetrics,
  DiscoveryStatsSnapshot -> LookupStatsSnapshot, and Metrics.discovery ->
  Metrics.lookup.

Two surfaces cross a stability boundary and ship behind a compatibility
window, both marked in-code for removal at the v2 cutover:

- The control-socket metric family is dual-emitted under both "discovery"
  (deprecated alias) and "lookup" so existing dashboards keep working.
- The node.discovery.* config table is split into node.lookup.* (mesh
  lookup scalars) and node.rendezvous.* (nostr/lan peer rendezvous).
  NodeConfig does not deny unknown fields, so a naive rename would make a
  deployed node.discovery: block deserialize into nothing and silently
  revert every setting to default. A deprecated all-Option
  DiscoveryConfigCompat field captures a legacy block and a new post-parse
  Config::normalize_deprecated_keys pass folds it into the new tables with
  a one-time deprecation warning.

Flip the packaged fips.yaml templates to the new keys, add legacy/new/
scalar compat parse tests, and record the split and deprecations in the
CHANGELOG. Behavior-neutral; fmt/clippy clean, lib suite green.
 2026-07-09 15:43:31 +00:00
Johnathan Corgan 0b2212e1e8 proto: bound powi test drift by relative tolerance, not ULP 
The previous portability fix bounded the powi test's drift from std::powi
at an absolute ULP count, but Windows/MSVC drift grows with the exponent
(3 ULP by exp=14), so no fixed ULP bound is portable.

Replace the ULP oracle with a loose relative-tolerance sanity sweep
(1e-11), which any sane libm clears by ~1000x regardless of exponent
while still catching a grossly wrong impl. Add a golden-bit pin for the
exp=14 case that exposed the growth. The powi function is unchanged; the
golden-bit determinism pins already passed on Windows.
 2026-07-09 01:30:18 +00:00
Johnathan Corgan b2ce7cd3c8 proto: make the powi guard test portable across platforms 
The powi guard test asserted our square-and-multiply result was
bit-for-bit equal to f64::powi, which failed the Windows unit-test job by
one ULP. f64::powi is not portably bit-stable: Linux and macOS lower it
to compiler-rt's __powidf2, but Windows/MSVC rounds differently.

Our square-and-multiply is pure IEEE-754 f64 multiplication and is
therefore deterministic across every platform, which is the property that
actually matters for mesh nodes to agree on bloom FPR and backoff timing
regardless of OS. Replace the std bit-equality assertion with golden-bit
pins on our own output (locking that cross-platform determinism) plus a
bounded-drift sanity sweep against std::powi. The powi function itself is
unchanged.
 2026-07-09 01:19:49 +00:00
Johnathan Corgan e3e03f6a5d proto: rename the mesh discovery subsystem module from discovery to lookup 
Rename src/proto/discovery to src/proto/lookup and bring the module's naming
onto the lookup stem, matching its concept: a mesh lookup of a node's
coordinates from its pubkey, sent into the mesh as a bloom-filter-guided
multicast request that returns a unicast response with the coordinates.

- the module directory and its declaration (proto::discovery -> proto::lookup)
- exported types: DiscoveryAction -> LookupAction, DiscoveryBackoff ->
  LookupBackoff, DiscoveryForwardRateLimiter -> LookupForwardRateLimiter,
  MAX_RECENT_DISCOVERY_REQUESTS -> MAX_RECENT_LOOKUP_REQUESTS, and the Discovery
  state struct -> Lookup
- internal terminology: doc comments, the "overlay-lookup" phrasing, the
  empty_discovery/suppressing_discovery test helpers, and the disc
  parameter/variable all take the lookup names
- the already-lookup-named wire types (LookupRequest/LookupResponse) are unchanged

Behavior-neutral: no wire bytes or decision logic change. Two references are
intentionally kept as "discovery": the still-named node::handlers::discovery
shell module and the node.discovery.* config keys, which belong to the broader
disambiguation of the shell, config, and metric surfaces still to come.
 2026-07-08 21:54:24 +00:00
Johnathan Corgan 2b009196b5 proto: no_std hygiene for the fmt/alloc imports and the filter math 
Bring the proto tree closer to a std+alloc-only posture, behavior unchanged on
every decision path:

- Sweep the remaining std::fmt and std::collections imports over to core::fmt
  and alloc::collections across the wire codecs and their tests. Subsystem files
  that shadow the core name with a child core module use the leading-colon
  ::core::fmt form. Imports only.

- Replace the two f64::powi calls (bloom false-positive-rate, FMP backoff timer)
  with a shared core-only square-and-multiply helper in proto/math, bit-identical
  to std::powi (a guard test pins this against every exponent the codecs reach),
  and route the diagnostic estimated-count natural log through libm::log. The FPR
  reject decision and the backoff timer are bit-for-bit unchanged; only the
  debug-only count estimate may differ by at most one ULP.
 2026-07-08 19:00:25 +00:00
Johnathan Corgan 5d13090d8f proto: add a bounds-checked byte reader/writer and adopt it across the wire codecs 
Introduce a shared proto/codec module with a cursor Reader (short reads fail with
MessageTooShort { expected: position + needed, got: total }) and an append Writer,
and adopt them across the seven subsystem wire codecs, replacing the repetitive
manual slicing, try_into, and from_le_bytes extraction. Each existing length check
maps to the reader (an up-front minimum becomes require at position zero, so the
per-field expected values, including the tree-announce expected 99, are reproduced
exactly); the bloom exact-length check stays explicit since it also rejects
over-long payloads. Encoded bytes and decode decisions are unchanged.
 2026-07-08 19:00:25 +00:00
Johnathan Corgan 6538731176 proto: replace the string Malformed error with typed variants and drop thiserror 
Replace Malformed(String) with a no_std-clean set: Malformed(&static str)
for the static decode diagnostics, BadSizeClass { got, max } for the two
bloom size-class checks, and typed sources BadCoord(CoordError) /
BadBloom(BloomError) for the coordinate and bloom construction failures. Add a
dedicated CoordError so proto/coord no longer depends upward on stp TreeError,
resolving the temporary inversion from the coordinate relocation. Drop the
thiserror derive from the four proto error types (Error, TreeError, BloomError,
CoordError) in favor of hand-rolled core::fmt::Display and core::error::Error
impls. thiserror remains in use elsewhere in the crate. Wire decode decisions
are unchanged; only the error type and its diagnostic text change.
 2026-07-08 19:00:25 +00:00
Johnathan Corgan 9697026c81 proto: share a per-address rate limiter and backoff helper across subsystems 
Hoist the two line-for-line-identical per-destination minimum-interval limiters
(discovery forward, routing error) into a shared PerAddrRateLimiter, and fold
the duplicated exponential-backoff math (discovery originator, FMP retry) into a
shared backoff_ms helper, both in a new proto/rate_limit module. The subsystem
limiters become thin delegating newtypes so should_forward/should_send and the
backoff call conventions are preserved. Behavior is byte-identical.
 2026-07-08 19:00:25 +00:00
Johnathan Corgan a2400d823f proto/stp: extract ParentDeclaration and relocate the coordinate type to a shared proto/coord module 
Two behavior-neutral relocations, wire bytes unchanged:

- Move the ParentDeclaration type and its impls out of state.rs into a dedicated
  declaration.rs, re-exported at the same path, and relocate the inline wire.rs
  test module into tests/wire.rs alongside the other stp unit tests (reusing the
  shared test helpers). wire.rs drops to non-test code only.

- Move TreeCoordinate, CoordEntry, and the coordinate wire codec out of
  proto/stp into a shared proto/coord module (a peer of proto/link), re-exported
  from proto::stp so every existing import path keeps resolving unchanged.
  Coordinate is a shared addressing primitive with many non-stp consumers, so it
  no longer belongs under the spanning-tree subsystem. The codec carries a
  documented temporary dependency on stp::TreeError until a dedicated CoordError
  is introduced.
 2026-07-08 19:00:25 +00:00
Johnathan Corgan dc9334e725 proto/stp: hoist the parent flap/hold-down veto to the shell for a clock-free classify core 
evaluate_parent no longer reads the injected clock: it returns a ParentEval of
Mandatory, Discretionary, or None, and the flap/hold-down veto is applied at the
edge via a new TreeState::is_switch_suppressed(now_ms), gating only the
discretionary arm. classify_announce/classify_periodic take the pre-computed
switch_suppressed bool instead of now_ms, so the whole classify ladder is
clock-free; the shell callers (tree announce/periodic re-eval, MMP first-RTT
re-eval, handle_parent_lost) compute the veto verdict. The no-coords parent
case stays discretionary (veto-gated) exactly as before. Behavior unchanged;
the veto tests now assert the moved responsibility.
 2026-07-08 19:00:24 +00:00
Johnathan Corgan 309a91d293 proto/mmp: split state into per-role modules and dissolve the vestigial mmp shell 
Reorganize the MMP subsystem, behavior unchanged throughout:

- Restore the pre-migration role split: move SenderState into sender.rs,
  ReceiverState (with its GapTracker helper) into receiver.rs, MmpMetrics and
  RrLog into metrics.rs, and PathMtuState into path_mtu.rs, leaving the Mmp
  aggregate plus the peer/session state in state.rs (1339 to 196 lines). Home
  the module constants in a new limits.rs and re-export them at the same paths.
  Pure code-motion with module rewiring; visibility unchanged.

- Dissolve the 97-line src/mmp shell, which held only MmpConfig and the
  monotonic mono_ms() clock: move MmpConfig into config/node.rs (re-exported as
  crate::config::MmpConfig), move mono_ms() into a new top-level src/time.rs
  shell time seam, repoint all callers, and delete src/mmp/. Also correct stale
  src/mmp/ doc-comment path labels to proto/mmp/ where they name the protocol
  primitives.
 2026-07-08 19:00:24 +00:00
Johnathan Corgan c1ddbf053c proto/discovery: home the recent-request cap and inject request_id from the shell 
Two behavior-neutral discovery cleanups:

- Move MAX_RECENT_DISCOVERY_REQUESTS out of the node discovery handler into the
  discovery subsystem limits module and re-export it, keeping the two use sites
  unchanged. Same value and semantics; only its home moves.

- Remove the ambient rand draw from LookupRequest by deleting generate() and
  having the shell draw the random request_id and pass it into the existing
  new() constructor. Same per-request u64 draw, now at the shell, leaving the
  discovery codec free of ambient RNG reads.
 2026-07-08 19:00:24 +00:00
Johnathan Corgan b53db662c3 proto/bloom: add BloomFilter::fpr() and use it for the false-positive-rate 
The false-positive-rate primitive (fill ratio raised to the hash count) was
inlined at three sites. Hoist it into a BloomFilter::fpr() method and call it
from all three; the threshold/policy logic stays at the call sites. Behavior
and result bytes are unchanged.
 2026-07-08 19:00:24 +00:00
Johnathan Corgan 4ad5940114 proto/fsp: extract FSP session protocol into sans-IO layout, retire src/protocol 
Migrate the FSP end-to-end session subsystem into src/proto/fsp/ following the
established sans-IO shape, and retire the src/protocol grab-bag now that FSP was
its last occupant.

Relocate the FSP session wire (node/session_wire.rs plus the FSP message types
from protocol/session.rs) into proto/fsp/wire.rs. Hoist the pure decision logic
into proto/fsp/core.rs over plain-data SessionSnapshots returning an ordered
FspAction list the shell drives: session-rekey policy, msg3-resend
classification, post-decrypt epoch reaction, setup/dual-init tie-break,
coords/path-MTU emit-policy, bounded pending-queue, and IPv6 ECN. The
crypto-owning SessionEntry stays shell-side in node/session.rs (matching the FMP
ActivePeer pattern); proto/fsp is wire + core + limits only, with no proto->noise
dependency and no crypto.

Move the coords helpers to proto/stp/ (they serialize TreeCoordinate), and split
SessionMessageType: the encrypted-inner 0x10-0x1F variants stay in proto/fsp/wire.rs
while the 0x20-0x2F routing signals become a new RoutingSignalType in
proto/routing/wire.rs. Migrate the session-MMP shell adapter, which continues to
drive proto/mmp/.

Retire src/protocol: LinkMessageType and SessionDatagram move to a new shared
proto/link.rs, ProtocolError becomes proto::Error (relocated verbatim), the
deprecated MessageType alias and the unimported PROTOCOL_VERSION are dropped, and
src/protocol/ is deleted along with its lib.rs module declaration.

Behavior-neutral: wire bytes unchanged, oracle tests pass unedited except
mod-path relocation; adds rekey/epoch characterization tests and pure
poll/emit-policy core tests.
 2026-07-08 04:57:00 +00:00
Johnathan Corgan 4ed674ea8b proto/bloom: relocate bloom filter into sans-IO layout 
Migrate the v1 bloom filter subsystem into src/proto/bloom/, matching the
discovery/routing/fmp/mmp/stp reference layout and completing the proto/
relocation series for the data/wire subsystems.

- wire.rs: the FilterAnnounce (0x20) codec, moved from protocol/filter.rs
- core.rs: the pure BloomFilter algorithm (hash/insert/contains/merge/
  as_bytes/from_bytes/estimated_count), moved from bloom/filter.rs
- state.rs: BloomState (per-peer inbound store, compute_outgoing_filter,
  the injected-clock send debounce), moved from bloom/state.rs
- limits.rs: the v1 sizing constants
- mod.rs: module wiring + the BloomError enum
- tests/: the unit suite split by target (core/state/wire), no inline tests

no_std+alloc hygiene: core::fmt over std::fmt, the tracing dependency
dropped from the pure filter, and std collections replaced with
BTreeMap/BTreeSet (NodeAddr: Ord) for deterministic iteration. The pure
filter combination stays a BloomState method; the two irreducible shell
gathers (peer_inbound_filters, build_filter_announce) remain in the async
shell. Wire bytes and observable behavior are unchanged; full local CI
green (36/36) including the bloom-storm chaos gate.
 2026-07-07 23:10:12 +00:00
Johnathan Corgan a67801099d proto/stp: sans-IO spanning-tree state machine 
Migrate the full non-async spanning-tree surface into proto/stp/, mirroring the
discovery/routing/fmp/mmp conversions. The classification ladder (parent-switch /
self-root / loop-drop / ancestry-update / periodic-rebroadcast / parent-lost) moves
out of the async node handlers into a pure Stp classify layer returning a
TreeDecision the shell drives, with effect ordering and per-arm invalidation
preserved verbatim. src/tree/ relocates wholesale: TreeState + ParentDeclaration data
+ coordinates into proto/stp/{state,coordinate}, the flap-dampening / hold-down
cluster into a FlapDampener in limits.rs, and the wire codec into wire.rs. The clock
is injected as u64 (wall-clock secs for the escaping declaration timestamp, monotonic
ms for the dampening timers via mmp::mono_ms); declaration crypto is field-partitioned
so sign/verify/hash run in the shell while the in-core modules carry data +
signing_bytes only. Peer maps/sets move to BTree; core/state/coordinate/limits are
core+alloc clean, with wire.rs the one std-tethered file. Behavior-neutral:
characterization tests added for the handler decision arms; convergence suite and
ci-local (36/36) green.
 2026-07-07 17:07:33 +00:00
Johnathan Corgan 50a595a0ed proto/mmp: sans-IO metrics-reporting state machine 2026-07-07 09:24:17 +00:00
Johnathan Corgan 4802792e38 proto/fmp: sans-IO connection-lifecycle state machine 2026-07-07 06:20:02 +00:00
Johnathan Corgan 9ea57b483a proto/routing: sans-IO transit + hop-selection state machine 2026-07-07 04:12:05 +00:00
Johnathan Corgan e03b206f62 proto/discovery: sans-IO state-machine migration + no_std reductions 
Migrate the FMP discovery decision logic out of the async handlers into
synchronous, runtime-agnostic sans-IO state machines owned by the protocol
structs, with I/O pushed to the edges. Pulls the full decision surface into a
pure core (backoff, rate-limit, planners, response routing), consolidates the
tests into a per-module tree with a shared crate testutil, and injects a u64
wall-clock so the core is free of Instant and std time.

Also brings the module toward no_std+alloc: the four discovery maps use
alloc::collections::BTreeMap (HashMap's RandomState is std-only), Arc is spelled
alloc::sync::Arc, the backoff-reset log lives in the shell (the core returns the
cleared count so observability stays out of the pure core), and the crate root
names alloc directly. The one remaining tether is ProtocolError's
std::error::Error coupling in the wire codec.

First subsystem of the broader sans-IO refactor; establishes the extraction
patterns and conventions carried forward to the remaining protocols.
 2026-07-05 21:59:21 +00:00
Johnathan Corgan 1dbfefc9d0 Merge branch 'maint' 2026-07-02 22:03:41 +00:00
Johnathan Corgan 793f844448 Merge branch 'maint' 2026-06-29 16:47:43 +00:00
Johnathan Corgan 243bd7985a Merge branch 'maint' 2026-06-29 14:23:57 +00:00
Johnathan Corgan 30c5808e09 Merge maint into master after the v0.4.0 rollover 
maint carries only its 0.4.1-dev version bump; master keeps its own
0.5.0-dev development version. Recorded as a linkage merge so later
forward-merges of real fixes land cleanly.
 2026-06-27 18:35:41 +00:00
Johnathan Corgan 3c9a629ad4 Open 0.5.0-dev cycle on master after v0.4.0 release 
Bump the version to 0.5.0-dev and update the status badge.
 2026-06-27 18:28:52 +00:00
257 changed files with 41689 additions and 19207 deletions
Expand all files Collapse all files
.github/workflows/ci.yml
+59 -109
View File
@@ -110,6 +110,59 @@ jobs:
restore-keys: |
${{ runner.os }}-cargo-
- run: cargo clippy --all-targets --all-features -- -D warnings
# An optional feature means two source trees, and --all-features lints
# only one of them. The default build is what ships, so lint it
# explicitly: without this stage, code that compiles only with
# `profiling` enabled would pass CI while breaking every release build.
# Mirrored in testing/ci-local.sh — check-ci-parity.sh compares
# integration suites only and will not catch a stage added to one runner
# and not the other.
- name: Clippy (default features)
run: cargo clippy --all-targets -- -D warnings
- name: Build with the tick-body profiler enabled
run: cargo build --workspace --features profiling
# ───────────────────────────────────────────────────────────────────────────
# Android cross-check
#
# FIPS runs on Android as an embedded library — the host app owns the TUN
# (an Android VpnService), so there are no daemon binaries to package, unlike
# the desktop targets. This job only cross-compiles the library for the
# android target to guard the android-only cfg paths (and the `not(android)`
# exclusions) from silently bit-rotting; nothing else in CI compiles them.
# cargo-ndk wires the NDK toolchain, which is required even for a check
# because `ring` compiles C at build time.
# ───────────────────────────────────────────────────────────────────────────
android-check:
name: Android cross-check (aarch64)
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v6
- name: Install Rust toolchain (+ Android target)
uses: actions-rust-lang/setup-rust-toolchain@v1
with:
target: aarch64-linux-android
components: clippy
cache: false
rustflags: ''
- name: Cache Cargo registry + build
uses: actions/cache@v5
with:
path: |
~/.cargo/registry
~/.cargo/git
target
key: ${{ runner.os }}-cargo-android-${{ hashFiles('**/Cargo.lock') }}
restore-keys: |
${{ runner.os }}-cargo-
- name: Install cargo-ndk
uses: taiki-e/install-action@v2
with:
tool: cargo-ndk
- name: Clippy the library for Android
run: |
export ANDROID_NDK_HOME="${ANDROID_NDK_HOME:-$ANDROID_NDK_LATEST_HOME}"
cargo ndk -t arm64-v8a clippy --lib -- -D warnings
build:
name: Build (${{ matrix.os }})
@@ -251,6 +304,12 @@ jobs:
check_name: Unit Tests Summary
fail_on_failure: false
# The `profiling` feature adds a module, a recorder and a writer thread
# that the default-feature run above never compiles, so its own tests do
# not execute there. Mirrored in testing/ci-local.sh.
- name: Run library tests with the tick-body profiler enabled
run: cargo test --lib --features profiling
# ─────────────────────────────────────────────────────────────────────────────
# Job 2b – Unit tests (macOS)
# ─────────────────────────────────────────────────────────────────────────────
@@ -374,16 +433,6 @@ jobs:
- suite: static-chain
type: static
topology: chain
# ── Rekey integration test ──────────────────────────────────────────
- suite: rekey
type: rekey
topology: rekey
- suite: rekey-accept-off
type: rekey-accept-off
topology: rekey-accept-off
- suite: rekey-outbound-only
type: rekey-outbound-only
topology: rekey-outbound-only
# ── Firewall baseline (fips0 nftables default-deny) ────────────
- suite: firewall
type: firewall
@@ -520,105 +569,6 @@ jobs:
docker compose -f testing/static/docker-compose.yml \
--profile ${{ matrix.topology }} down --volumes --remove-orphans
# ── Rekey integration test ──────────────────────────────────────────────
- name: Generate and inject configs (rekey)
if: matrix.type == 'rekey'
run: |
bash testing/static/scripts/generate-configs.sh rekey
bash testing/static/scripts/rekey-test.sh inject-config
- name: Start containers (rekey)
if: matrix.type == 'rekey'
run: |
docker compose -f testing/static/docker-compose.yml \
--profile rekey up -d
- name: Run rekey test
if: matrix.type == 'rekey'
run: bash testing/static/scripts/rekey-test.sh
- name: Collect logs on failure (rekey)
if: matrix.type == 'rekey' && failure()
run: |
docker compose -f testing/static/docker-compose.yml \
--profile rekey logs --no-color
- name: Stop containers (rekey)
if: matrix.type == 'rekey' && always()
run: |
docker compose -f testing/static/docker-compose.yml \
--profile rekey down --volumes --remove-orphans
# ── Rekey + accept_connections=false variant ──────────────────────────
- name: Generate and inject configs (rekey-accept-off)
if: matrix.type == 'rekey-accept-off'
env:
REKEY_TOPOLOGY: rekey-accept-off
REKEY_ACCEPT_OFF_NODES: b
run: |
bash testing/static/scripts/generate-configs.sh rekey-accept-off
bash testing/static/scripts/rekey-test.sh inject-config
- name: Start containers (rekey-accept-off)
if: matrix.type == 'rekey-accept-off'
run: |
docker compose -f testing/static/docker-compose.yml \
--profile rekey-accept-off up -d
- name: Run rekey test (accept-off variant)
if: matrix.type == 'rekey-accept-off'
env:
REKEY_TOPOLOGY: rekey-accept-off
REKEY_ACCEPT_OFF_NODES: b
run: bash testing/static/scripts/rekey-test.sh
- name: Collect logs on failure (rekey-accept-off)
if: matrix.type == 'rekey-accept-off' && failure()
run: |
docker compose -f testing/static/docker-compose.yml \
--profile rekey-accept-off logs --no-color | tail -300
- name: Stop containers (rekey-accept-off)
if: matrix.type == 'rekey-accept-off' && always()
run: |
docker compose -f testing/static/docker-compose.yml \
--profile rekey-accept-off down --volumes --remove-orphans
# ── Rekey + udp.outbound_only=true variant ─────────────────────────────
- name: Generate and inject configs (rekey-outbound-only)
if: matrix.type == 'rekey-outbound-only'
env:
REKEY_TOPOLOGY: rekey-outbound-only
REKEY_OUTBOUND_ONLY_NODES: b
run: |
bash testing/static/scripts/generate-configs.sh rekey-outbound-only
bash testing/static/scripts/rekey-test.sh inject-config
- name: Start containers (rekey-outbound-only)
if: matrix.type == 'rekey-outbound-only'
run: |
docker compose -f testing/static/docker-compose.yml \
--profile rekey-outbound-only up -d
- name: Run rekey test (outbound-only variant)
if: matrix.type == 'rekey-outbound-only'
env:
REKEY_TOPOLOGY: rekey-outbound-only
REKEY_OUTBOUND_ONLY_NODES: b
run: bash testing/static/scripts/rekey-test.sh
- name: Collect logs on failure (rekey-outbound-only)
if: matrix.type == 'rekey-outbound-only' && failure()
run: |
docker compose -f testing/static/docker-compose.yml \
--profile rekey-outbound-only logs --no-color | tail -300
- name: Stop containers (rekey-outbound-only)
if: matrix.type == 'rekey-outbound-only' && always()
run: |
docker compose -f testing/static/docker-compose.yml \
--profile rekey-outbound-only down --volumes --remove-orphans
# ── Firewall baseline integration test ─────────────────────────────────
- name: Run firewall baseline integration test
if: matrix.type == 'firewall'
.github/workflows/package-openwrt.yml
+5
View File
@@ -285,6 +285,8 @@ jobs:
"$FILES_DIR/etc/fips/firewall.sh"
"$FILES_DIR/etc/hotplug.d/net/99-fips"
"$FILES_DIR/etc/uci-defaults/90-fips-setup"
"$FILES_DIR/usr/bin/fips-mesh-setup"
"$FILES_DIR/usr/bin/fips-ap-setup"
)
fail=0
for f in "${TARGETS[@]}"; do
@@ -404,6 +406,8 @@ jobs:
./usr/bin/fipsctl
./usr/bin/fipstop
./usr/bin/fips-gateway
./usr/bin/fips-mesh-setup
./usr/bin/fips-ap-setup
./etc/init.d/fips
./etc/init.d/fips-gateway
./etc/fips/fips.yaml
@@ -717,6 +721,7 @@ jobs:
for path in \
usr/bin/fips usr/bin/fipsctl usr/bin/fipstop usr/bin/fips-gateway \
usr/bin/fips-mesh-setup usr/bin/fips-ap-setup \
etc/init.d/fips etc/init.d/fips-gateway \
etc/fips/fips.yaml etc/fips/firewall.sh etc/dnsmasq.d/fips.conf \
etc/sysctl.d/fips-gateway.conf etc/sysctl.d/fips-bridge.conf \
CHANGELOG.md
+56
View File
@@ -9,13 +9,69 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
### Added
- An optional tick-body profiler behind the new `profiling` Cargo feature,
**off by default**. When enabled, `fipsctl profile tick on [--dir PATH]` /
`off` / `status` starts and stops a capture at runtime with no restart. Each
capture writes one tab-separated file (default `/var/log/fips`, capped at
32 MB) carrying, per ten-second interval, the exact count, max and total for
every step of the rx-loop tick arm, the whole-tick span, and gauges for ticks,
peer count, the gap between successive tick-arm entries and the resulting
arm-starvation delay. With the feature off the instrumentation macro is a pure
pass-through, so a default build contains no timing code on the tick path.
`LogsDirectory=fips` was added to the packaged systemd units so the capture
directory is created and cleaned up declaratively.
- `packaging/debian/build-deb.sh --features <list>` builds the `.deb` with a
Cargo feature list, which is how an instrumented package is produced for a
measurement run. The auto-derived dev Version gains a matching `+<features>`
marker, so a feature build and a default build of the same commit are no
longer indistinguishable: without it the two carry byte-identical versions,
an install of one over the other is an apt no-op, and the running node offers
no way to tell which one it has. The marker sorts above the unmarked build, so
installing a feature build is an upgrade and reverting to the default build is
a downgrade — revert with `dpkg -i` rather than `apt install`. `--features` is
refused together with `--no-build`, which would stamp the marker onto binaries
the features never reached.
### Changed
- The Ethernet transport's per-interface `discovery` flag was renamed to
`listen` (`transports.ethernet.*`) to match the symmetric `announce`
(transmit) / `listen` (receive) neighbor-beacon vocabulary. The old
`discovery:` key is still accepted via a serde alias, so deployed configs
continue to load unchanged; `Config::to_yaml()` re-emits it under the
canonical `listen:` name. Update your `fips.yaml` to `listen:`.
- The mesh-lookup control-metrics family is now emitted under the key
`lookup` in `fipsctl stats metrics` and `show routing`. The former key
`discovery` is still emitted as a deprecated alias carrying identical
counters; update dashboards and alerts to read `lookup`.
- The overloaded `node.discovery.*` config table was split into
`node.lookup.*` (mesh-lookup scalars: `ttl`, `attempt_timeouts_secs`,
`recent_expiry_secs`, `backoff_base_secs`, `backoff_max_secs`,
`forward_min_interval_secs`) and `node.rendezvous.*` (peer rendezvous:
`nostr.*`, `lan.*`). A deployed `node.discovery:` block still loads and is
folded into the new tables with a one-time deprecation warning; migrate your
`fips.yaml` to the new keys.
- `SessionDatagram::decrement_ttl` and `SessionDatagram::can_forward` now match
the forwarder's IP hop-limit semantics: `decrement_ttl` decrements first and
reports false when the result is zero, and `can_forward` is true only at a
TTL of 2 or more.
### Deprecated
- The `discovery` metric-family key (control-socket JSON). It is dual-emitted
alongside the new `lookup` key during a migration window and will be removed.
Migrate dashboards/alerts from `discovery.*` to `lookup.*`.
- The `node.discovery.*` config table. Its keys were split into `node.lookup.*`
(mesh-lookup) and `node.rendezvous.*` (peer rendezvous). A legacy
`node.discovery:` block still applies for now with a deprecation warning and
will be removed; migrate to `node.lookup.*` / `node.rendezvous.*`.
- The Ethernet `transports.ethernet.discovery` flag, renamed to
`transports.ethernet.listen`. The old key is still accepted via a serde
alias and will be removed at the v2 cutover; migrate to `listen`.
### Fixed
- Nostr NAT traversal no longer breaks after the host suspends. The traversal
Cargo.lock
Generated
+2 -1
View File
@@ -1074,7 +1074,7 @@ checksum = "9844ddc3a6e533d62bba727eb6c28b5d360921d5175e9ff0f1e621a5c590a4d5"
[[package]]
name = "fips"
version = "0.4.2-dev"
version = "0.5.0-dev"
dependencies = [
"arc-swap",
"bech32",
@@ -1087,6 +1087,7 @@ dependencies = [
"hex",
"hkdf",
"libc",
"libm",
"mdns-sd",
"nostr",
"nostr-sdk",
Cargo.toml
+15 -1
View File
@@ -1,6 +1,6 @@
[package]
name = "fips"
version = "0.4.2-dev"
version = "0.5.0-dev"
edition = "2024"
description = "A distributed, decentralized network routing protocol for mesh nodes connecting over arbitrary transports"
license = "MIT"
@@ -11,12 +11,21 @@ readme = "README.md"
keywords = ["mesh", "p2p", "decentralized", "overlay-network", "nostr"]
categories = ["network-programming", "command-line-utilities", "cryptography"]
[features]
default = []
# Tick-body profiler (`src/instr`). Off by default: enabling it edits 26 call
# sites in the rx loop's hot tick arm, and only a compile-time gate makes the
# default build's neutrality a property of the generated code rather than of a
# runtime check. Build with `--features profiling` for a measurement run.
profiling = []
[dependencies]
ratatui = "0.30"
secp256k1 = { version = "0.30", features = ["rand", "global-context"] }
sha2 = "0.10"
hkdf = "0.12"
ring = "0.17"
libm = "0.2"
rand = "0.10.1"
crossbeam-channel = "0.5"
thiserror = "2.0"
@@ -108,3 +117,8 @@ path = "src/bin/fips-gateway.rs"
[[bin]]
name = "fipstop"
path = "src/bin/fipstop/main.rs"
[[bench]]
name = "routing_next_hop"
path = "benches/routing_next_hop.rs"
harness = false
README.md
+16 -14
View File
@@ -3,7 +3,7 @@
![banner](docs/logos/fips_banner.png)
[![License: MIT](https://img.shields.io/badge/license-MIT-blue.svg)](LICENSE)
[![Rust](https://img.shields.io/badge/rust-1.85%2B-orange.svg)](https://www.rust-lang.org/)
[![Status](https://img.shields.io/badge/status-v0.4.2--dev-green.svg)](#status--roadmap)
[![Status](https://img.shields.io/badge/status-v0.5.0--dev-green.svg)](#status--roadmap)
A self-organizing encrypted mesh network built on Nostr identities,
capable of operating over arbitrary transports without central
@@ -112,17 +112,19 @@ tutorial progression starting at
cargo build --release
```
Requires Rust 1.94.1+ (edition 2024). Linux, macOS, and Windows are
supported; transport availability varies by platform.
Requires Rust 1.94.1+ (edition 2024). Linux, macOS, and Windows run as
standalone daemons; Android is supported as an embedded library (the host
app owns the TUN, e.g. a `VpnService`). Transport availability varies by
platform.
| Transport | Linux | macOS | Windows | OpenWrt |
|-----------|:-----:|:-----:|:-------:|:-------:|
| UDP | ✅ | ✅ | ✅ | ✅ |
| TCP | ✅ | ✅ | ✅ | ✅ |
| Ethernet | ✅ | ✅ | ❌ | ✅ |
| Tor | ✅ | ✅ | ✅ | ✅ |
| Nym | ✅ | ✅ | ✅ | ❌ |
| BLE | ✅ | ❌ | ❌ | ❌ |
| Transport | Linux | macOS | Windows | Android | OpenWrt |
|-----------|:-----:|:-----:|:-------:|:-------:|:-------:|
| UDP | ✅ | ✅ | ✅ | ✅ | ✅ |
| TCP | ✅ | ✅ | ✅ | ✅ | ✅ |
| Ethernet | ✅ | ✅ | ❌ | ❌ | ✅ |
| Tor | ✅ | ✅ | ✅ | ❌ | ✅ |
| Nym | ✅ | ✅ | ✅ | ❌ | ❌ |
| BLE | ✅ | ❌ | ❌ | ❌ | ❌ |
On Linux, a source build requires `libclang` — the LAN gateway's
nftables bindings are generated by `bindgen` at build time, which
@@ -210,10 +212,10 @@ testing/ Docker-based integration test harnesses + chaos simulation
## Status & roadmap
FIPS is at **v0.4.2-dev** on the `maint` branch.
FIPS is at **v0.5.0-dev** on the `master` branch.
[v0.4.1](https://github.com/jmcorgan/fips/releases/tag/v0.4.1) has
shipped; this line carries patch-level fixes for the 0.4.x series. The
core protocol works end-to-end over
shipped; this development line continues the testing-and-polishing
track toward v0.5.0. The core protocol works end-to-end over
UDP, TCP, Ethernet, Tor, Nym, and Bluetooth on a global, public test
mesh of thousands of nodes. v0.4.0 added the Nym mixnet transport and
mDNS LAN discovery alongside the existing Nostr-mediated peer discovery,
benches/routing_next_hop.rs
+365
View File
@@ -0,0 +1,365 @@
//! Micro-benchmark quantifying the per-forwarded-packet heap-allocation cost
//! of the routing next-hop candidate-assembly path.
//!
//! `find_next_hop` runs once per forwarded data packet. Its sans-IO core
//! assembles a `Vec<Candidate>` by enumerating every peer through the
//! `RoutingView` seam: `peer_addrs()` materializes a `Vec<NodeAddr>` of all
//! peers, the survivors are snapshotted (each cloning its `TreeCoordinate`),
//! and the result is collected into a second `Vec`. This bench measures that
//! per-call allocation against a fused zero-alloc reference that iterates the
//! peer map directly and borrows coordinates instead of cloning.
//!
//! Visibility caveat: the production `routing_candidates` / `select_best_candidate`
//! / `RoutingView` / `Candidate` are `pub(crate)` (src/proto/routing/core.rs)
//! and are not re-exported at the crate root, so an external bench crate cannot
//! name them. Rather than change production visibility, this file reproduces
//! that path verbatim over the real public `NodeAddr` / `TreeCoordinate` /
//! `CoordEntry` / `BloomFilter` types with the same iterator chain and the same
//! `HashMap`-backed view the shell uses (src/node/mod.rs NodeRoutingView). The
//! allocation behavior is therefore identical to production by construction;
//! only the symbol identity differs.
use std::alloc::{GlobalAlloc, Layout, System};
use std::collections::HashMap;
use std::hint::black_box;
use std::sync::atomic::{AtomicUsize, Ordering};
use criterion::{BenchmarkId, Criterion, criterion_group, criterion_main};
use fips::{BloomFilter, NodeAddr, TreeCoordinate};
// ---------------------------------------------------------------------------
// Counting global allocator: bumps a process-global counter on every heap
// allocation operation (alloc / alloc_zeroed / realloc). Sampled tightly and
// single-threaded in `report_allocs` so no unrelated allocations are captured.
// ---------------------------------------------------------------------------
struct CountingAlloc;
static ALLOCS: AtomicUsize = AtomicUsize::new(0);
unsafe impl GlobalAlloc for CountingAlloc {
unsafe fn alloc(&self, layout: Layout) -> *mut u8 {
ALLOCS.fetch_add(1, Ordering::Relaxed);
unsafe { System.alloc(layout) }
}
unsafe fn dealloc(&self, ptr: *mut u8, layout: Layout) {
unsafe { System.dealloc(ptr, layout) }
}
unsafe fn alloc_zeroed(&self, layout: Layout) -> *mut u8 {
ALLOCS.fetch_add(1, Ordering::Relaxed);
unsafe { System.alloc_zeroed(layout) }
}
unsafe fn realloc(&self, ptr: *mut u8, layout: Layout, new_size: usize) -> *mut u8 {
ALLOCS.fetch_add(1, Ordering::Relaxed);
unsafe { System.realloc(ptr, layout, new_size) }
}
}
#[global_allocator]
static GLOBAL: CountingAlloc = CountingAlloc;
const PEER_COUNTS: [usize; 4] = [8, 32, 128, 256];
/// Fraction of peers whose bloom filter reports the destination reachable.
const REACH_NUMERATOR: usize = 1;
const REACH_DENOMINATOR: usize = 2;
/// Tree depth for synthetic coordinates (self..root), a realistic mesh depth.
const COORD_DEPTH: usize = 8;
// ---------------------------------------------------------------------------
// Reproduction of the pub(crate) routing seam (src/proto/routing/core.rs).
// ---------------------------------------------------------------------------
trait RoutingView {
fn peer_addrs(&self) -> Vec<NodeAddr>;
fn peer_may_reach(&self, peer: &NodeAddr, dest: &NodeAddr) -> bool;
fn peer_can_send(&self, peer: &NodeAddr) -> bool;
fn peer_link_cost(&self, peer: &NodeAddr) -> f64;
fn peer_coords(&self, peer: &NodeAddr) -> Option<TreeCoordinate>;
}
struct Candidate {
addr: NodeAddr,
can_send: bool,
link_cost: f64,
coords: Option<TreeCoordinate>,
}
/// Verbatim from `routing::routing_candidates` (core.rs). Allocates the
/// `peer_addrs` Vec, clones each survivor's coords, and collects into a Vec.
fn routing_candidates(rv: &impl RoutingView, dest: &NodeAddr) -> Vec<Candidate> {
rv.peer_addrs()
.into_iter()
.filter(|peer| rv.peer_may_reach(peer, dest))
.map(|peer| Candidate {
can_send: rv.peer_can_send(&peer),
link_cost: rv.peer_link_cost(&peer),
coords: rv.peer_coords(&peer),
addr: peer,
})
.collect()
}
/// Verbatim from `routing::select_best_candidate` (core.rs). Pure, no alloc.
fn select_best_candidate(
candidates: &[Candidate],
dest_coords: &TreeCoordinate,
my_coords: &TreeCoordinate,
) -> Option<NodeAddr> {
let my_distance = my_coords.distance_to(dest_coords);
let mut best: Option<(&Candidate, f64, usize)> = None;
for candidate in candidates {
if !candidate.can_send {
continue;
}
let cost = candidate.link_cost;
let dist = candidate
.coords
.as_ref()
.map(|pc| pc.distance_to(dest_coords))
.unwrap_or(usize::MAX);
if dist >= my_distance {
continue;
}
let dominated = match &best {
None => true,
Some((_, best_cost, best_dist)) => {
cost < *best_cost
|| (cost == *best_cost && dist < *best_dist)
|| (cost == *best_cost
&& dist == *best_dist
&& candidate.addr < best.as_ref().unwrap().0.addr)
}
};
if dominated {
best = Some((candidate, cost, dist));
}
}
best.map(|(candidate, _, _)| candidate.addr)
}
// ---------------------------------------------------------------------------
// Bench-local view, HashMap-backed exactly like src/node/mod.rs NodeRoutingView.
// ---------------------------------------------------------------------------
struct BenchPeer {
bloom: BloomFilter,
can_send: bool,
link_cost: f64,
}
struct BenchView {
peers: HashMap<NodeAddr, BenchPeer>,
coords: HashMap<NodeAddr, TreeCoordinate>,
}
impl RoutingView for BenchView {
fn peer_addrs(&self) -> Vec<NodeAddr> {
self.peers.keys().copied().collect()
}
fn peer_may_reach(&self, peer: &NodeAddr, dest: &NodeAddr) -> bool {
self.peers.get(peer).is_some_and(|p| p.bloom.contains(dest))
}
fn peer_can_send(&self, peer: &NodeAddr) -> bool {
self.peers.get(peer).is_some_and(|p| p.can_send)
}
fn peer_link_cost(&self, peer: &NodeAddr) -> f64 {
self.peers.get(peer).map_or(f64::INFINITY, |p| p.link_cost)
}
fn peer_coords(&self, peer: &NodeAddr) -> Option<TreeCoordinate> {
self.coords.get(peer).cloned()
}
}
/// Zero-alloc reference: what an iterator/visitor seam would do. Iterates the
/// peer map directly, fuses the may_reach + can_send filters, borrows coords
/// instead of cloning, and tracks the best hop inline. No Vec, no coord clone.
fn resolve_next_hop_zeroalloc(
view: &BenchView,
dest: &NodeAddr,
dest_coords: &TreeCoordinate,
my_coords: &TreeCoordinate,
) -> Option<NodeAddr> {
let my_distance = my_coords.distance_to(dest_coords);
let mut best: Option<(NodeAddr, f64, usize)> = None;
for (addr, peer) in &view.peers {
if !peer.bloom.contains(dest) {
continue;
}
if !peer.can_send {
continue;
}
let cost = peer.link_cost;
let dist = view
.coords
.get(addr)
.map(|pc| pc.distance_to(dest_coords))
.unwrap_or(usize::MAX);
if dist >= my_distance {
continue;
}
let dominated = match &best {
None => true,
Some((best_addr, best_cost, best_dist)) => {
cost < *best_cost
|| (cost == *best_cost && dist < *best_dist)
|| (cost == *best_cost && dist == *best_dist && *addr < *best_addr)
}
};
if dominated {
best = Some((*addr, cost, dist));
}
}
best.map(|(addr, _, _)| addr)
}
// ---------------------------------------------------------------------------
// Scenario construction.
// ---------------------------------------------------------------------------
fn addr(tag: u8, i: u16) -> NodeAddr {
let mut b = [0u8; 16];
b[0] = tag;
b[1..3].copy_from_slice(&i.to_le_bytes());
NodeAddr::from_bytes(b)
}
/// A depth-`COORD_DEPTH` coordinate whose leaf is `leaf`, sharing a fixed
/// interior path and root with `shared_tag`. Peers built with the dest's
/// shared_tag sit close to the destination (distance 2); a distinct shared_tag
/// sits far (near the root), modeling our own position.
fn coord(leaf: NodeAddr, shared_tag: u8) -> TreeCoordinate {
let mut path = Vec::with_capacity(COORD_DEPTH);
path.push(leaf);
for level in 1..(COORD_DEPTH - 1) {
path.push(addr(shared_tag, level as u16));
}
path.push(addr(9, 0)); // common root
TreeCoordinate::from_addrs(path).expect("valid coord path")
}
struct Scenario {
view: BenchView,
dest: NodeAddr,
dest_coords: TreeCoordinate,
my_coords: TreeCoordinate,
}
impl Scenario {
fn new(n: usize) -> Self {
let dest = addr(2, 0);
// Destination path uses interior tag 4; peers reuse tag 4 so survivors
// are close to the destination. Our own coords use tag 5 (far).
let dest_coords = coord(dest, 4);
let my_coords = coord(addr(6, 0), 5);
let mut peers = HashMap::new();
let mut coords = HashMap::new();
for i in 0..n {
let paddr = addr(1, i as u16);
let mut bloom = BloomFilter::new();
// Realistic fill: a handful of unrelated reachable addrs.
for f in 0..4u16 {
bloom.insert(&addr(7, i as u16 * 4 + f));
}
// A controlled fraction advertise the destination as reachable.
if (i % REACH_DENOMINATOR) < REACH_NUMERATOR {
bloom.insert(&dest);
}
peers.insert(
paddr,
BenchPeer {
bloom,
can_send: true,
link_cost: 1.0 + (i as f64) * 0.01,
},
);
// Peers share the destination's interior path (tag 4) → close.
coords.insert(paddr, coord(paddr, 4));
}
Self {
view: BenchView { peers, coords },
dest,
dest_coords,
my_coords,
}
}
fn survivors(&self) -> usize {
self.view
.peers
.values()
.filter(|p| p.bloom.contains(&self.dest))
.count()
}
}
// ---------------------------------------------------------------------------
// Allocation-per-call report (printed once, before criterion timing).
// ---------------------------------------------------------------------------
fn count_allocs<T>(iters: usize, mut f: impl FnMut() -> T) -> f64 {
for _ in 0..8 {
black_box(f());
}
let start = ALLOCS.load(Ordering::Relaxed);
for _ in 0..iters {
black_box(f());
}
let end = ALLOCS.load(Ordering::Relaxed);
(end - start) as f64 / iters as f64
}
fn report_allocs() {
const ITERS: usize = 2000;
println!("\n=== allocations per call (heap alloc ops: alloc+alloc_zeroed+realloc) ===");
println!(
"{:>6} {:>10} {:>16} {:>16}",
"peers", "survivors", "current/call", "zero-alloc/call"
);
for &n in &PEER_COUNTS {
let s = Scenario::new(n);
let survivors = s.survivors();
let current = count_allocs(ITERS, || {
let cands = routing_candidates(&s.view, &s.dest);
select_best_candidate(&cands, &s.dest_coords, &s.my_coords)
});
let zero = count_allocs(ITERS, || {
resolve_next_hop_zeroalloc(&s.view, &s.dest, &s.dest_coords, &s.my_coords)
});
println!("{n:>6} {survivors:>10} {current:>16.2} {zero:>16.2}");
}
println!();
}
fn bench_next_hop(c: &mut Criterion) {
report_allocs();
let mut group = c.benchmark_group("find_next_hop");
for &n in &PEER_COUNTS {
let scenario = Scenario::new(n);
group.bench_with_input(BenchmarkId::new("current_alloc", n), &n, |b, _| {
b.iter(|| {
let cands = routing_candidates(&scenario.view, &scenario.dest);
black_box(select_best_candidate(
&cands,
&scenario.dest_coords,
&scenario.my_coords,
))
});
});
group.bench_with_input(BenchmarkId::new("zero_alloc_ref", n), &n, |b, _| {
b.iter(|| {
black_box(resolve_next_hop_zeroalloc(
&scenario.view,
&scenario.dest,
&scenario.dest_coords,
&scenario.my_coords,
))
});
});
}
group.finish();
}
criterion_group! {
name = benches;
config = Criterion::default().sample_size(50);
targets = bench_next_hop
}
criterion_main!(benches);
docs/design/fips-ipv6-adapter.md
+28
View File
@@ -302,6 +302,34 @@ alternative — running under a dedicated unprivileged service
account with the capability granted on the binary — see
[../how-to/run-as-unprivileged-user.md](../how-to/run-as-unprivileged-user.md).
### App-Owned TUN (embedded hosts)
On platforms where FIPS is embedded rather than run as a daemon — notably
Android, where the `VpnService` owns the TUN fd and the app has no
`CAP_NET_ADMIN` — FIPS does not create `fips0` itself. Instead the embedder owns
the fd and exchanges IPv6 packet bytes with FIPS over channels.
`Node::enable_app_owned_tun()` sets this up. It is called after `Node::new` and
before `start()` (and before the node is moved into a background task), mirroring
`control_read_handle()`, and returns two app-side channel ends:
- **app → mesh** — the embedder pushes IPv6 packets read from its fd into
`app_outbound_tx`. These are drained by `run_rx_loop` into `handle_tun_outbound`
and routed exactly as the Reader Thread's output would be.
- **mesh → app** — inbound mesh traffic on port 256 is reconstructed and written
to the node's `tun_tx` (the same sink the Writer Thread reads); the embedder
pulls from `app_inbound_rx` and writes to its fd.
With the channels installed, `start()` skips system-TUN creation (it gates on
`tun_tx` being unset), so FIPS does no `CAP_NET_ADMIN` operations.
Because packets enter via `app_outbound_tx` rather than the Reader Thread, they
**bypass `handle_tun_packet`** — the `fd00::/8` destination filter, the ICMPv6
Destination Unreachable for off-mesh dests (see [Reader Thread](#reader-thread)),
and the [TUN-Side TCP MSS Clamping](#tun-side-tcp-mss-clamping). The embedder is
therefore responsible for routing only `fd00::/8` to its TUN (so only mesh-bound
packets arrive) and for clamping TCP MSS on outbound SYNs.
## Implementation Status
| Feature | Status |
docs/design/fips-nostr-discovery.md
+1 -1
View File
@@ -477,7 +477,7 @@ The TXT record carries three keys (`src/discovery/lan/mod.rs:47-55`):
Once per node tick, the node drains browser events and acts on them in
`poll_lan_discovery()` (`src/node/lifecycle.rs:907`, called from
`src/node/handlers/rx_loop.rs:266`). For each discovered peer it finds
`src/node/dataplane/rx_loop.rs:266`). For each discovered peer it finds
a UDP transport whose family matches the peer address, parses the
`npub` into a `PeerIdentity`, skips peers it is already connected to or
currently connecting to, and otherwise initiates a connection.
docs/design/fips-transport-layer.md
+6 -6
View File
@@ -262,7 +262,7 @@ UDP (1500 vs 1472 MTU).
- **No IP dependency**: Operates below the IP layer. Nodes on the same
Ethernet segment can communicate without IP addresses or routing
infrastructure
- **Broadcast discovery**: Nodes discover each other via periodic beacon
- **Broadcast neighbor detection**: Nodes discover each other via periodic beacon
broadcasts on the shared medium, with no static peer configuration required
- **Higher MTU**: Standard Ethernet frames carry 1500 bytes of payload,
yielding an effective FIPS MTU of 1499 after the frame type prefix
@@ -293,7 +293,7 @@ socket.
| Addressing | 6-byte MAC address |
| Platform | Linux only (`CAP_NET_RAW` required) |
### Beacon Discovery
### Neighbor Beacons
Ethernet nodes discover peers via broadcast beacons sent to
ff:ff:ff:ff:ff:ff. Each beacon is a 34-byte frame containing the sender's
@@ -301,7 +301,7 @@ x-only public key. Receiving nodes extract the MAC source address from the
frame and the public key from the payload, then report the discovered peer
to FMP.
Four configuration flags control discovery behavior — `discovery`
Four configuration flags control neighbor behavior — `listen`
(listen for beacons), `announce` (broadcast beacons), `auto_connect`
(initiate handshakes to discovered peers), and `accept_connections`
(accept inbound handshakes). The flag table and per-flag defaults
@@ -310,13 +310,13 @@ under `transports.ethernet.*`.
A typical discoverable node sets `announce`, `auto_connect`, and
`accept_connections` all true. A passive listener uses just
`discovery: true` to observe the network without announcing itself.
`listen: true` to observe the network without announcing itself.
### WiFi Compatibility
WiFi interfaces in infrastructure (managed) mode work transparently for
unicast — the mac80211 subsystem handles frame translation between 802.11
and 802.3. Broadcast beacon discovery is unreliable in managed mode because
and 802.3. Broadcast neighbor detection is unreliable in managed mode because
access points commonly isolate clients from each other's broadcast traffic.
Startup logging:
@@ -895,7 +895,7 @@ transitions through `Starting` to `Up` (operational). `stop()` moves to
| --------- | ------ | ----- |
| UDP/IP | **Implemented** | Primary transport, AsyncFd/recvmsg, SO_RXQ_OVFL kernel drop detection |
| TCP/IP | **Implemented** | FMP header-based framing, non-blocking connect, per-connection MSS MTU |
| Ethernet | **Implemented** | AF_PACKET SOCK_DGRAM, EtherType 0x2121, beacon discovery, Linux only |
| Ethernet | **Implemented** | AF_PACKET SOCK_DGRAM, EtherType 0x2121, neighbor beacons, Linux only |
| WiFi | **Implemented** (via Ethernet transport, infrastructure mode) | mac80211 translates 802.11↔802.3; broadcast beacons unreliable through APs |
| Tor | **Implemented** | Outbound SOCKS5, inbound via onion service, .onion and clearnet addressing |
| Nym | **Implemented** | Outbound-only SOCKS5 through nym-socks5-client, mixnet anonymity, IP/hostname addressing |
docs/how-to/README.md
+2
View File
@@ -25,4 +25,6 @@ X" to "X is done".
| [persistent-identity.md](persistent-identity.md) | Provision a stable Nostr keypair so the node keeps the same npub across restarts |
| [host-aliases.md](host-aliases.md) | Use shortnames (`test-us01.fips`, `my-laptop.fips`) instead of full npubs by editing `/etc/fips/hosts` or setting peer aliases |
| [set-up-bluetooth-peer.md](set-up-bluetooth-peer.md) | Configure a Bluetooth Low Energy peer link |
| [set-up-80211s-mesh-backhaul.md](set-up-80211s-mesh-backhaul.md) | Link OpenWrt FIPS routers over an open 802.11s radio backhaul (FIPS provides encryption, authentication, and routing) |
| [set-up-open-access-ssid.md](set-up-open-access-ssid.md) | Broadcast the open `!FIPS` access SSID so phones and laptops roam onto the mesh (one ESS: save once, roam every FIPS router) |
| [diagnose-mtu-issues.md](diagnose-mtu-issues.md) | Triage MTU-shaped failures and rule out their imposters (bufferbloat, transport saturation) |
docs/how-to/set-up-80211s-mesh-backhaul.md
+249
View File
@@ -0,0 +1,249 @@
# Set Up an 802.11s Mesh Backhaul (OpenWrt)
Link FIPS routers over radio — no cables, no APs, no shared
infrastructure — by running the Ethernet transport on an open 802.11s
mesh interface. The radio layer provides nothing but L2 frames to
direct neighbors; FIPS provides everything else: encryption and
authentication (Noise IK), peer discovery (Ethernet beacons), and
routing (the spanning tree).
For the transport design, see
[../design/fips-transport-layer.md](../design/fips-transport-layer.md).
For all `transports.ethernet.*` configuration keys, see
[../reference/configuration.md](../reference/configuration.md).
## Why open, why forwarding off
Two deliberate choices distinguish this from a stock 802.11s setup:
- **`encryption none`** — the mesh is open on purpose. Every FIPS peer
link is already authenticated and encrypted by the Noise IK
handshake, so SAE at L2 would duplicate that work, add a shared
credential to provision across routers, and (on ath10k) force the
firmware into its slower raw Tx/Rx mode. A stranger can form an
802.11s peering with your router *and* a FIPS peer link on top of it —
the same open model as mDNS and BLE discovery, where the advert is
only a hint and the handshake authenticates each link (no
impersonation, no MITM) rather than gating who may peer. Admission is
open up to the daemon's max-peers cap. What you concede: any nearby
radio can peer and reach the FIPS overlay surface; L2 metadata (MAC
addresses, frame sizes) is visible in the air; a hostile radio can
burn airtime — all inherent to an open radio link.
- **`mesh_fwding 0`** — disables 802.11s's own HWMP routing so each
mesh link is a plain neighbor link. FIPS is the routing layer; two
routing layers would fight, and broadcast discovery beacons would
flood the whole mesh instead of reaching direct neighbors only.
The interface is **not** bridged into `br-lan` — the FIPS Ethernet
transport binds it directly.
## When to use
- Two or more OpenWrt FIPS routers within radio range of each other,
where running cable is impractical.
- You want the mesh segment to keep working with zero shared
credentials or per-site configuration ("flash and drop in").
It is **not** for connecting phones or laptops — client devices
cannot join an 802.11s mesh. They enter the mesh through a normal AP
on the same router (see constraints below), or over BLE.
## Requirements
- OpenWrt 22.03+ with the FIPS package installed.
- A radio whose driver supports mesh point interfaces. Check with:
```sh
iw list | grep -A 10 "Supported interface modes" | grep "mesh point"
```
The mainstream OpenWrt chips (ath9k, ath10k, mt76) all qualify.
- Ideally a dual- or tri-band router, so one band can be dedicated to
the backhaul (see constraints).
## Step 1 — create the mesh interface(s)
On **each** router, run the helper once per radio you want in the
backhaul:
```sh
fips-mesh-setup radio1
```
This creates an open 802.11s interface with mesh ID `fips-mesh` and
HWMP forwarding off, attaches it to an unmanaged netifd interface (no
IP configuration — none is needed), uncomments the matching `meshN`
transport entry in `/etc/fips/fips.yaml` (see Step 2), and reloads the
radio. Interfaces are named by radio index: `radio0` → `fips-mesh0`,
`radio1` → `fips-mesh1`. Pass a second argument to use a different
mesh ID.
Note: the helper runs `wifi reload`, which re-applies the whole
wireless config and so briefly drops every client AP on all radios for
a few seconds. `fips-mesh-setup remove` reloads the same way. Expect
the blip if clients are connected.
On dual-band routers, meshing **both** bands is worth it: 2.4 GHz
reaches further at lower rates, 5 GHz carries more over shorter
links. Note this is **failover, not multipath**: FIPS keeps one
active link per peer, so traffic uses one band at a time — the other
is a standby that re-establishes the peer if the active link dies
(detection via keepalive timeout, so a cutover takes seconds, not
milliseconds):
```sh
fips-mesh-setup radio0
fips-mesh-setup radio1
```
**Pin the same channel on every backhaul router, per band.** Mesh
points only peer on the same channel, and the mesh inherits whatever
the radio is set to — with `channel 'auto'` (the default on many
devices) each router picks its own and the mesh silently never forms.
The script prints the radio's current band and channel and warns on
`auto`:
```sh
uci set wireless.radio1.channel='36'
uci commit wireless && wifi reload
```
Prefer a non-DFS channel (36–48 on 5 GHz): on DFS channels the radio
must wait ~60 s in CAC before transmitting after every reload.
Equivalent manual UCI (per radio), if you prefer to see what it does:
```sh
uci batch <<'EOF'
set wireless.fips_mesh_radio1=wifi-iface
set wireless.fips_mesh_radio1.device='radio1'
set wireless.fips_mesh_radio1.mode='mesh'
set wireless.fips_mesh_radio1.mesh_id='fips-mesh'
set wireless.fips_mesh_radio1.encryption='none'
set wireless.fips_mesh_radio1.mesh_fwding='0'
set wireless.fips_mesh_radio1.ifname='fips-mesh1'
set wireless.fips_mesh_radio1.network='fips_mesh_radio1'
set network.fips_mesh_radio1=interface
set network.fips_mesh_radio1.proto='none'
EOF
uci commit
wifi reload
```
## Step 2 — check the FIPS transport binding
The `fips.yaml` shipped in the OpenWrt package carries one transport
entry per radio, but **commented out** — so a stock install that never
runs this helper logs no per-boot "interface missing" warning.
`fips-mesh-setup` uncommented the matching `meshN` entry in Step 1, so
there is normally nothing to do here. If you maintain your own config
(or ran the manual UCI above instead of the helper), make sure the
entries are present and uncommented:
```yaml
transports:
ethernet:
mesh0:
interface: "fips-mesh0"
discovery: true
announce: true
auto_connect: true
accept_connections: true
mesh1:
interface: "fips-mesh1"
discovery: true
announce: true
auto_connect: true
accept_connections: true
```
## Step 3 — restart the daemon (order matters)
```sh
/etc/init.d/fips restart
```
Restart fips **after** the mesh interface is up. A transport whose
interface is missing at startup is logged and skipped, not retried —
so if the daemon comes up before the radio, the mesh transport stays
dead until the next restart. (An interface that *vanishes and
returns* after startup is recovered automatically; only the missing-
at-startup case needs this ordering.)
## Verify
L2 first — the 802.11s peering, with a second configured router in
range:
```sh
iw dev fips-mesh0 station dump
```
You should see one station entry per neighbor router, with signal
levels. No entries means a radio problem, not a FIPS problem — triage
in this order:
1. **Channel mismatch** (the most common cause): compare
`iw dev fips-mesh0 info` on both routers — mesh ID *and* channel
must match exactly.
2. **The mesh interface never joined** — `iw dev fips-meshX info`
shows `type mesh point` but **no channel line**, and `station dump`
is empty. Usual cause: a client (`sta`) interface on the same
radio. A STA must follow its upstream AP's channel, the whole
radio follows the STA, and a mesh pinned to a different channel
silently stays down. Check for a STA sharing the radio
(`iw dev`, look for `type managed` on the same phy), compare
`iw dev <sta-iface> info | grep channel`, and re-pin the mesh
channel to match — on every backhaul router.
3. **Is the other router transmitting at all?**
```sh
iw dev fips-mesh0 scan | grep -i -B4 "MESH ID"
```
Its mesh ID visible → transmission works, peering is failing
(mesh ID typo, or one side has encryption set). Nothing visible →
check `wifi status` on the other router, remember the ~60 s DFS
CAC wait, and confirm the country code is set
(`uci get wireless.radio1.country`) — an unset regdomain can
block channels entirely.
4. `logread | grep -iE "mesh|fips-mesh0"` on both sides.
Then the FIPS layer on top:
```sh
logread | grep -i beacon # beacons flowing on the new transport
fipsctl show peers # neighbor authenticated and connected
fipsctl show links # link on the 'ethernet' transport
```
Discovery is automatic: each node beacons its pubkey every few
seconds, and `auto_connect` initiates the Noise handshake on first
sight.
## Constraints
- **Airtime is shared per radio.** All virtual interfaces on one
radio (AP + mesh) share one channel, and multi-hop forwarding on a
single radio roughly halves throughput per hop. On dual/tri-band
hardware, dedicate one band to `fips-mesh0` and serve clients on
the others.
- **AP + mesh coexistence is driver-dependent.** It works on the
mainstream chips (this is the standard Freifunk/Gluon setup), but
check `iw list` under "valid interface combinations" for your
hardware.
- **Clients can't join.** Phones and laptops reach the mesh through
the router's normal AP or via BLE — never through the 802.11s
interface.
- **Radio links are lossy.** A neighbor at the edge of range will
form an 802.11s peering yet deliver a fraction of its frames.
Expect link-quality effects that don't exist on wired Ethernet.
- **A client (STA) uplink on the same radio owns the channel.** The
STA must follow whatever channel its upstream AP uses; every other
interface on that radio follows the STA. A mesh pinned to a
different channel silently never joins, and it does **not** recover
when the STA disconnects — a `wifi reload` (plus a fips restart) is
needed. A *roaming* uplink (travel-router / hotspot-chasing setups)
is fundamentally incompatible with a fixed-channel mesh on the same
radio: dedicate the mesh to the radio the STA never uses, and treat
any mesh sharing a STA radio as best-effort.
docs/how-to/set-up-open-access-ssid.md
+267
View File
@@ -0,0 +1,267 @@
# Set Up the Open FIPS Access SSID (OpenWrt)
Give phones and laptops a way in: every FIPS router broadcasts the
same open SSID — `!FIPS` — from its access radio. Same SSID + unique
BSSIDs is one standard ESS, so a client saves the network once and
roams between all FIPS routers natively, with no per-router setup and
no shared credentials (the Freifunk model). The leading `!` sorts the
network to the top of alphabetically ordered pickers (iOS, desktop
OSes — Android sorts by signal strength) and is part of the name:
SSIDs match byte-for-byte or not at all. The radio layer provides
nothing but open L2 to the nearest router; FIPS provides everything
else: encryption and authentication (Noise IK), discovery
(mDNS/Ethernet beacons), and mobility (the overlay identity survives
roaming, so no 802.11r or L2 tricks are needed).
This is the *access* layer — how clients reach FIPS routers. For the
router-to-router *backhaul*, see
[set-up-80211s-mesh-backhaul.md](set-up-80211s-mesh-backhaul.md).
For all `transports.ethernet.*` configuration keys, see
[../reference/configuration.md](../reference/configuration.md).
## Why open, why this addressing
Three deliberate choices distinguish this from a stock guest network:
- **`encryption none`** — the SSID is open on purpose, and it *must*
be. Clients key a saved network on SSID **plus security type**: if
one router used a PSK and another OWE, the same `FIPS` name would be
three different saved networks and roaming would break. Open is the
only security type that needs zero provisioning, and OWE is left out
for now for exactly this uniformity reason (OWE-transition mode is
inconsistent across client vendors). Every FIPS peer link is already
authenticated and encrypted by the Noise IK handshake. A stranger
can associate *and* form a FIPS peer link — that is the point of open
access; the handshake authenticates each link (no impersonation, no
MITM) but does not gate who may peer, and admission is open up to the
daemon's max-peers cap. What confines a hostile peer is the isolated
`fips_ap` zone (no path to br-lan or the WAN — see below), not the
handshake. What you concede: any nearby device can reach the FIPS
overlay surface (handshake, discovery, lookup, routing) and peer with
the router; L2 metadata is visible in the air; a hostile radio can
burn airtime — all inherent to an open radio link.
- **DHCPv4 from a fixed subnet, plus IPv6 router advertisements.**
dnsmasq leases IPv4 out of `10.21.<N>.0/24` (`N` = the radio index;
the prefix echoes FIPS port 2121). The subnet is deliberately
**identical on every router**: a roaming phone keeps its lease
across routers, and dnsmasq's authoritative mode (the OpenWrt
default, pinned by the helper) ACKs a renew the new router never
issued. odhcpd additionally announces a ULA prefix (`fd..`-range)
for stateless SLAAC; DHCPv6 stays off. FIPS itself only needs
link-local + mDNS, but Android's provisioning check requires an RA
or a DHCP offer and *disconnects* with neither, and plain laptops
expect a real IPv4 address. Works with or without an upstream —
nothing here depends on the WAN. The IPv6 side stays per-router and
disposable; in all cases the FIPS overlay identity, not the IP, is
the mobility anchor.
- **Isolated interface** — its own network and firewall zone, with no
path to `br-lan` and no forwarding to the WAN. Inbound traffic is
rejected except DHCPv4, ICMPv6 (SLAAC itself), mDNS, and the FIPS
transport ports; the raw-Ethernet transport (EtherType 0x2121) is
not IP and never traverses the firewall. AP client isolation is on, so clients
cannot reach each other at L2 — two FIPS phones on one router still
reach each other through the router at the overlay layer.
## The "no internet" behavior (expected, one-time acceptance)
The network intentionally provides **no internet**. On first connect,
a phone's validation probe fails and it asks whether to stay on a
network without internet access — choose **stay connected** and
**don't ask again**. That choice is stored per SSID, so accepting it
once covers every FIPS router anywhere.
After that, the network is marked "connected, no internet"
(unvalidated) and the phone keeps **cellular as its default route**
while staying associated — normal apps never notice the FIPS network
exists. FIPS apps bind their sockets to the Wi-Fi network explicitly,
so mesh traffic flows over Wi-Fi while everything else uses cellular.
## When to use
- Any FIPS router that should serve phones and laptops directly, not
just peer with other routers.
- You want clients to roam between FIPS routers with zero per-router
or per-site configuration.
It is the complement of the 802.11s backhaul: the backhaul links
routers (clients cannot join it), the access SSID admits clients.
Both can share a radio, at an airtime cost (see constraints).
## Requirements
- OpenWrt 22.03+ with the FIPS package installed (fw4; dnsmasq and
odhcpd are part of the default images).
- Any radio — AP mode needs no special driver support.
## Step 1 — create the access point(s)
On **each** router, run the helper once per radio that should serve
clients:
```sh
fips-ap-setup radio0
```
This creates an open AP with SSID `!FIPS` and client isolation, an
isolated network with `10.21.<N>.1/24` and a static ULA `/64`, a
DHCPv4 + RA dhcp config (dnsmasq leases, SLAAC, no DHCPv6), and a
locked-down `fips_ap` firewall zone — then reloads the radio. Interfaces are named by radio index: `radio0` →
`fips-ap0`, `radio1` → `fips-ap1`. Pass a second argument to use a
different SSID — but the SSID, like the security type, must be
identical on **all** routers or clients will treat them as separate
networks and stop roaming.
On dual-band routers, run it for both radios so clients can pick
either band:
```sh
fips-ap-setup radio0
fips-ap-setup radio1
```
**Channels are free per router.** Unlike the mesh backhaul, there is
no same-channel constraint — clients scan when they roam — so leave
each router on whatever channel suits its RF environment.
Equivalent manual UCI (per radio), if you prefer to see what it does
(`fdxx:...` stands for a `/64` out of the router's ULA prefix):
```sh
uci batch <<'EOF'
set wireless.fips_ap_radio0=wifi-iface
set wireless.fips_ap_radio0.device='radio0'
set wireless.fips_ap_radio0.mode='ap'
set wireless.fips_ap_radio0.ssid='!FIPS'
set wireless.fips_ap_radio0.encryption='none'
set wireless.fips_ap_radio0.isolate='1'
set wireless.fips_ap_radio0.ifname='fips-ap0'
set wireless.fips_ap_radio0.network='fips_ap_radio0'
set network.fips_ap_radio0=interface
set network.fips_ap_radio0.proto='static'
set network.fips_ap_radio0.ipaddr='10.21.0.1'
set network.fips_ap_radio0.netmask='255.255.255.0'
set network.fips_ap_radio0.ip6addr='fdxx:xxxx:xxxx:fa00::1/64'
set dhcp.fips_ap_radio0=dhcp
set dhcp.fips_ap_radio0.interface='fips_ap_radio0'
set dhcp.fips_ap_radio0.ra='server'
set dhcp.fips_ap_radio0.ra_default='2'
set dhcp.fips_ap_radio0.dhcpv6='disabled'
set dhcp.fips_ap_radio0.dhcpv4='server'
set dhcp.fips_ap_radio0.start='10'
set dhcp.fips_ap_radio0.limit='200'
EOF
uci commit
wifi reload
```
plus the `fips_ap` firewall zone (input/forward REJECT, no
forwardings, ACCEPT rules for DHCPv4/UDP 67, ICMPv6, UDP 5353/2121,
TCP 8443).
## Step 2 — check the FIPS transport binding
The `fips.yaml` shipped in the OpenWrt package carries one transport
entry per access interface, but **commented out** — so a stock install
that never runs this helper logs no per-boot "interface missing"
warning. `fips-ap-setup` uncommented the matching `apN` entry in Step 1,
and also enabled `node.rendezvous.lan` (the daemon's mDNS/DNS-SD
rendezvous — phone FIPS apps cannot see raw-Ethernet beacons, so mDNS
is how they find the daemon; the switch is daemon-wide and stays on if
you later remove the AP). So there is normally nothing to do here. If
you maintain your own config (or ran the manual UCI above instead of
the helper), make sure both are present and uncommented:
```yaml
node:
rendezvous:
lan:
enabled: true
```
```yaml
transports:
ethernet:
ap0:
interface: "fips-ap0"
discovery: true
announce: true
auto_connect: true
accept_connections: true
ap1:
interface: "fips-ap1"
discovery: true
announce: true
auto_connect: true
accept_connections: true
```
## Step 3 — restart the daemon (order matters)
```sh
/etc/init.d/fips restart
```
Restart fips **after** the AP interface is up. A transport whose
interface is missing at startup is logged and skipped, not retried —
so if the daemon comes up before the radio, the access transport
stays dead until the next restart. (An interface that *vanishes and
returns* after startup is recovered automatically; only the missing-
at-startup case needs this ordering.)
## Verify
L2 and addressing first, with a phone or laptop connected to `!FIPS`:
```sh
iw dev fips-ap0 station dump # one entry per associated client
ip addr show dev fips-ap0 # 10.21.0.1/24 and the fd..::1/64
cat /tmp/dhcp.leases # one lease per connected client
```
No station entries means a radio problem; an association that drops
after ~30 s usually means the client never got an address — check
`logread | grep -e dnsmasq -e odhcpd` and that the
`dhcp.fips_ap_radio0` section survived
(`uci show dhcp | grep fips_ap`).
Then the FIPS layer on top, for a client running FIPS:
```sh
logread | grep -i beacon # beacons flowing on the new transport
fipsctl show peers # client authenticated and connected
```
On the phone itself: the network shows "connected, no internet" and
stays associated — that is the designed steady state, not an error.
## Constraints
- **SSID and security type must be uniform across ALL routers.**
One router with a PSK (or OWE) under the same name splits the ESS
into different saved networks and silently breaks roaming. Never
"harden" a single router.
- **Airtime is shared per radio.** An access AP and a mesh backhaul
on the same radio share one channel. On dual/tri-band hardware,
dedicate a band to the backhaul and serve clients on the others.
- **Strangers can associate and peer — by design.** Open access means
any nearby device can complete the Noise handshake and become a FIPS
peer (up to the max-peers cap); the handshake authenticates each link,
it does not restrict who joins. They reach only the FIPS overlay
surface — the isolated zone gives no path to br-lan or the WAN. Do not
add forwardings to the `fips_ap` zone: that would turn the open SSID
into a hotspot and hand the isolation away.
- **Roaming is client-driven.** Clients decide when to hop BSSIDs
(standard ESS behavior); the IPv4 lease survives the hop (same
subnet everywhere), the SLAAC address renumbers, and FIPS sessions
ride through because the overlay identity is the anchor. Expect a
brief L2 gap during the hop, as on any ESS without 802.11r.
- **The `10.21.<N>.0/24` convention must hold everywhere.** Lease
survival depends on every router serving the same subnet from the
same radio index — the helper guarantees this; don't hand-pick
per-router subnets. Two routers can lease the same address to two
different clients; after a roam the conflict is caught (dnsmasq
NAKs a renew for an address in use) and the client re-DHCPs. If a
laptop is *also* wired to a LAN that really uses `10.21.<N>.0/24`,
its routing table will conflict — a corner case worth knowing, not
designing around: the zone forwards nowhere, so the FIPS side never
reaches beyond the router either way.
docs/reference/cli-fipsctl.md
+52
View File
@@ -115,6 +115,58 @@ Tell the daemon to drop a peer link.
| -------- | ----------- |
| `peer` | npub (bech32) or hostname from `/etc/fips/hosts`. |
### `profile tick <on|off|status>`
> **Reading the output.** Step durations are wall clock measured across `await`
> points, not CPU time: a step that waits on I/O accrues that wait, and other
> tasks may run inside the span. That is the intended measure for head-of-line
> delay, and it means a large step is not necessarily an expensive one.
> `arm_starvation` is measured directly as the entry time minus the deadline
> the interval scheduled that tick for. It is not derived from
> `tick_entry_gap`, which carries no starvation signal on its own: under a
> steady delay every gap is exactly one tick period.
Start, stop and inspect a capture of the rx-loop tick body. **Present
only when both `fipsctl` and the daemon are built with
`--features profiling`**; the feature is off by default, so a stock
package does not carry this subcommand and a stock daemon reports
`profile_tick_*` as an unknown command.
| Subcommand | Control-socket command | Description |
| ---------- | ---------------------- | ----------- |
| `profile tick on` | `profile_tick_on` | Create the capture file and start recording. Fails if a capture is already running (naming the active file) or if the directory cannot be written. |
| `profile tick off` | `profile_tick_off` | Stop the capture. The writer is woken immediately, drains once more and is joined, so the command returns promptly. Succeeds, reporting nothing active, when no capture is running. |
| `profile tick status` | `profile_tick_status` | Report `idle`, `running`, `stopped_by_cap` or `stopped_by_error`, plus the active path, bytes written, flush interval and byte cap. |
`profile tick on` options:
| Flag | Argument | Default | Description |
| ---- | -------- | ------- | ----------- |
| `--dir` | directory path | `/var/log/fips` | Where to write the capture. Created if absent. Use it to profile a non-root `cargo run`, or on a platform whose log root differs. |
One file is written per capture, named `profile-<UTC timestamp>.tsv`.
It opens with a `#`-prefixed header block (node npub, build version,
platform, configured tick period, flush interval, byte cap, start
time), then a tab-separated column header, then one row per measured
step per flush interval:
```text
ts_unix kind domain name count max total unit
```
`kind` is `step` for a timed span and `gauge` for a sampled scalar, so
a gauge value never lands under a duration column; `unit` names the
unit of `max` and `total` for that row. Every step present in the build
gets a row every interval, including zero-count rows. Gauges cover
ticks per interval, peer count, the wall gap between successive
tick-arm entries, and the arm-starvation delay, which is measured
against the deadline the tick was scheduled for rather than derived
from the gap.
A capture stops itself on reaching 32 MB, appending a `#` line saying
so; `profile tick status` then reports `stopped_by_cap` until the next
`on` or `off` clears it.
## Exit Codes
| Code | Meaning |
docs/reference/configuration.md
+6 -6
View File
@@ -436,7 +436,7 @@ Requires `CAP_NET_RAW` or running as root. Linux only.
| `mtu` | u16 | *(auto)* | Override MTU. Default: interface MTU minus 3 (for frame type + length prefix) |
| `recv_buf_size` | usize | `2097152` | Socket receive buffer size in bytes (2 MB) |
| `send_buf_size` | usize | `2097152` | Socket send buffer size in bytes (2 MB) |
| `discovery` | bool | `true` | Listen for discovery beacons from other nodes |
| `listen` | bool | `true` | Listen for neighbor beacons from other nodes |
| `announce` | bool | `false` | Broadcast announcement beacons on the LAN |
| `auto_connect` | bool | `false` | Auto-connect to discovered peers |
| `accept_connections` | bool | `false` | Accept incoming connection attempts from discovered peers |
@@ -450,7 +450,7 @@ transports:
ethernet:
lan:
interface: "eth0"
discovery: true
listen: true
announce: true
backbone:
interface: "eth1"
@@ -458,7 +458,7 @@ transports:
```
Each named instance operates independently with its own socket and
discovery state. The instance name is used in log messages and the
neighbor state. The instance name is used in log messages and the
`name()` method on the Transport trait.
### TCP (`transports.tcp.*`)
@@ -840,7 +840,7 @@ peers:
### Mixed UDP + Ethernet Example
A node bridging internet peers (UDP) and a local Ethernet segment with
beacon discovery:
neighbor beacons:
```yaml
node:
@@ -856,7 +856,7 @@ transports:
mtu: 1472
ethernet:
interface: "eth0"
discovery: true
listen: true
announce: true
auto_connect: true
accept_connections: true
@@ -999,7 +999,7 @@ transports:
# mtu: null # null = interface MTU - 3 (typically 1497)
# recv_buf_size: 2097152 # 2 MB
# send_buf_size: 2097152 # 2 MB
# discovery: true # listen for beacons
# listen: true # listen for beacons
# announce: false # broadcast beacons
# auto_connect: false # connect to discovered peers
# accept_connections: false # accept inbound handshakes
docs/reference/control-socket.md
+15
View File
@@ -159,6 +159,21 @@ not reproduced here to avoid duplicating the source.
Both commands run on the daemon's main task and may block briefly
while the node mutates its state.
#### Profiler toggle (`--features profiling` builds only)
| Command | Params | Behaviour |
| ------- | ------ | --------- |
| `profile_tick_on` | `dir` (optional directory path; default `/var/log/fips`) | Creates the capture file, publishes its path, and starts the writer thread. `data`: `state`, `path`, `interval_secs`, `byte_cap`. Errors if a capture is already running (naming the active file) or the directory is unwritable. |
| `profile_tick_off` | — | Stops the capture, drains once more, joins the writer. `data`: `state`, `stopped`, `stopped_by_cap`, `stopped_by_error`, `path`, `bytes`. |
| `profile_tick_status` | — | `data`: `state` (`idle` / `running` / `stopped_by_cap` / `stopped_by_error`), `path`, `bytes`, `byte_cap`, `interval_secs`. |
Unlike `connect` and `disconnect`, these three are served in the
control accept task rather than on the daemon's main task. All of their
state is process statics and none of them needs `&mut Node`, so
routing them through the main loop would only make the toggle queue
behind the tick body it exists to measure. They are absent from a
default build, where the daemon answers them as unknown commands.
## Gateway Command Catalog
`fips-gateway` exposes a separate control socket with its own command
docs/reference/security.md
+1 -1
View File
@@ -209,7 +209,7 @@ for the metadata-privacy model and the rejection of onion routing.
| --------- | --------------- | ------------ | ------ |
| UDP | None until `bind_addr` set | `0.0.0.0:2121` typical | Operator sets `transports.udp.bind_addr` |
| TCP | None until `bind_addr` set | None — outbound-only without bind | Operator sets `transports.tcp.bind_addr` |
| Ethernet | Listens on configured interface (raw `AF_PACKET`) | EtherType 0x2121 on selected interface | Per-flag `discovery`, `announce`, `auto_connect`, `accept_connections` |
| Ethernet | Listens on configured interface (raw `AF_PACKET`) | EtherType 0x2121 on selected interface | Per-flag `listen`, `announce`, `auto_connect`, `accept_connections` |
| Tor | None until `directory_service` configured | `127.0.0.1:8443` (loopback only) | Operator sets `transports.tor.directory_service` and configures `HiddenServiceDir` in `torrc` |
| BLE | Off by default | n/a | Operator enables `transports.ble.*` |
| Nostr discovery | Off by default | n/a (relay client, not a listener) | Operator sets `node.discovery.nostr.enabled: true` |
docs/tutorials/ground-up-mesh.md
+9 -9
View File
@@ -71,7 +71,7 @@ supplies the rest:
- **Addressing**: the `fips0` adapter takes an `fd97:...` ULA
derived from the npub. No DHCP. No SLAAC. The address is
cryptographically tied to the identity.
- **Discovery**: each daemon broadcasts a small beacon on the
- **Neighbor detection**: each daemon broadcasts a small beacon on the
link advertising its npub; the other daemon's listener picks
it up and dials in over the same link.
- **Routing**: the FIPS mesh layer builds its own spanning tree
@@ -177,7 +177,7 @@ different interface names — that is normal.
Edit `/etc/fips/fips.yaml` on **both** nodes. Under
`transports:`, add an `ethernet:` block. The key settings are
the four discovery flags — both nodes must opt in to all four,
the four neighbor flags — both nodes must opt in to all four,
and they default to off:
```yaml
@@ -185,7 +185,7 @@ transports:
ethernet:
interface: "<eth>" # the name from Step 1
announce: true # broadcast our beacon on the link
discovery: true # listen for beacons (default; shown for clarity)
listen: true # listen for beacons (default; shown for clarity)
auto_connect: true # dial peers we discover
accept_connections: true # accept dial-ins from peers we discover
```
@@ -194,7 +194,7 @@ Each flag does one thing:
- `announce: true` — emit a small beacon every
`beacon_interval_secs` (default 30s) carrying our npub.
- `discovery: true` — listen for incoming beacons; populate a
- `listen: true` — listen for incoming beacons; populate a
candidate-peer list keyed by source MAC and observed npub.
- `auto_connect: true` — when we see a beacon from an npub
we have not yet peered with, initiate the outbound Noise
@@ -218,7 +218,7 @@ is "all four flags on both ends."
> lan:
> interface: "eth0"
> announce: true
> discovery: true
> listen: true
> auto_connect: true
> accept_connections: true
> dongle:
@@ -227,7 +227,7 @@ is "all four flags on both ends."
> # ...
> ```
>
> Each named instance runs its own socket and discovery state.
> Each named instance runs its own socket and neighbor state.
> A single ground-up link only needs the flat form shown
> first; named instances become useful when the same node
> bridges multiple physical segments.
@@ -390,7 +390,7 @@ What you do need on the AP side:
networks and "secure" enterprise APs ship with it on.
When client isolation is on, the AP refuses to forward
station-to-station frames — the broadcast beacons never
arrive at the other node, and discovery fails silently.
arrive at the other node, and neighbor detection fails silently.
If beacons aren't crossing, this is the first thing to
check.
@@ -401,7 +401,7 @@ adapter name.
### Bluetooth LE (experimental but works)
BLE is a separate transport (`transports.ble.*`) with its own
discovery model — L2CAP advertisements rather than raw L2
neighbor-detection model — L2CAP advertisements rather than raw L2
broadcasts. The shape of the tutorial is the same (advertise +
scan + auto-connect + accept), but the prerequisites are
different: BlueZ, `bluetoothd`, an HCI adapter, and the
@@ -424,7 +424,7 @@ Windows builds skip it.
a radio link), `CAP_NET_RAW`, and a few config flags on each
end are sufficient. The mesh supplies its own identity,
addressing, discovery, and routing.
- **Discovery is a four-flag opt-in.** `announce`, `discovery`,
- **Neighbor detection is a four-flag opt-in.** `announce`, `listen`,
`auto_connect`, and `accept_connections` each control one
thing; both ends must agree before a link will form.
- **The two modes coexist.** Overlay peers and ground-up peers
docs/tutorials/resolve-peers-via-nostr.md
+1 -1
View File
@@ -178,7 +178,7 @@ The resolution itself happens at debug-log level, so you will
not see it in the default-level journal. The user-facing way to
confirm everything worked is `fipsctl show peers` in the next
step. (To watch the resolution in the journal, run the daemon
manually with `RUST_LOG=fips::discovery::nostr=debug`; not
manually with `RUST_LOG=fips::nostr=debug`; not
necessary for this tutorial.)
## Step 5: Verify the resolved endpoint
packaging/common/fips.yaml
+18 -9
View File
@@ -10,12 +10,21 @@ node:
#
# Or set an explicit key (overrides persistent):
# nsec: "nsec1..."
discovery:
# Optional Nostr-mediated overlay endpoint discovery.
# Mesh-lookup protocol (node.lookup.*): the overlay coordinate-lookup engine
# (mesh address -> coordinates). Defaults shown; uncomment to override.
# lookup:
# ttl: 64
# attempt_timeouts_secs: [1, 2, 4, 8]
# recent_expiry_secs: 10
# backoff_base_secs: 0
# backoff_max_secs: 0
# forward_min_interval_secs: 2
rendezvous:
# Optional Nostr-mediated overlay endpoint rendezvous.
# nostr:
# enabled: true
# policy: configured_only # disabled | configured_only | open
# open_discovery_max_pending: 64 # caps queued open-discovery retries
# open_discovery_max_pending: 64 # caps queued open-rendezvous retries
# app: "fips-overlay-v1"
# advertise: true
# advert_relays:
@@ -34,17 +43,17 @@ node:
# - "stun:stun.cloudflare.com:3478"
# - "stun:global.stun.twilio.com:3478"
#
# Optional mDNS-based LAN discovery for sub-second same-LAN pairing.
# Optional mDNS-based LAN rendezvous for sub-second same-LAN pairing.
# Opt-in (default false): default-off avoids a per-LAN identity
# broadcast on nodes that have deliberately disabled other discovery
# broadcast on nodes that have deliberately disabled other rendezvous
# channels, and avoids any multicast surprise on upgrade. Requires an
# operational UDP transport (the advertised port is the one peers dial).
# lan:
# enabled: false
# # Optional application/network scope carried in the LAN-only TXT
# # record. Browsers that set a scope ignore adverts for other scopes.
# # Kept separate from the Nostr discovery `app` tag so relay-visible
# # adverts can stay generic while LAN discovery stays per-private-network.
# # Kept separate from the Nostr rendezvous `app` tag so relay-visible
# # adverts can stay generic while LAN rendezvous stays per-private-network.
# # scope: "lab-floor-3"
# # Advanced: overrides the mDNS service type. Leave unset in normal
# # use — only needed to run multiple isolated services on one
@@ -89,7 +98,7 @@ transports:
# Ethernet transport — uncomment and set your interface name.
# ethernet:
# interface: "eth0"
# discovery: true
# listen: true
# announce: true
# auto_connect: true
# accept_connections: true
@@ -147,5 +156,5 @@ peers: []
# - transport: udp
# addr: "test-us01.fips.network:2121" # IP or hostname (e.g., "peer.example.com:2121")
# - transport: udp
# addr: "nat" # Use node.discovery.nostr for Nostr/STUN hole punching
# addr: "nat" # Use node.rendezvous.nostr for Nostr/STUN hole punching
# connect_policy: auto_connect
packaging/debian/build-deb.sh
+44 -2
View File
@@ -2,6 +2,7 @@
# Build a .deb package for FIPS using cargo-deb.
#
# Usage: ./build-deb.sh [--target <triple>] [--version <version>] [--no-build]
# [--features <list>]
#
# Prerequisites: cargo-deb (install with: cargo install cargo-deb)
# Output: deploy/fips_<version>_<arch>.deb
@@ -19,6 +20,9 @@ Options:
--target <triple> Rust target triple to build/package
--version <version> Override Debian package version
--no-build Package existing binaries without running cargo build
--features <list> Cargo features to build with (comma-separated). Marks the
auto-derived Version so the package is distinguishable
from a default build of the same commit.
-h, --help Show this help
EOF
}
@@ -26,6 +30,7 @@ EOF
TARGET_TRIPLE=""
VERSION_OVERRIDE=""
NO_BUILD=0
FEATURES=""
while [[ $# -gt 0 ]]; do
case "$1" in
@@ -41,6 +46,10 @@ while [[ $# -gt 0 ]]; do
NO_BUILD=1
shift
;;
--features)
FEATURES="${2:?missing value for --features}"
shift 2
;;
-h|--help)
usage
exit 0
@@ -53,6 +62,16 @@ while [[ $# -gt 0 ]]; do
esac
done
# A feature build that skips the build step would stamp a feature-marked Version
# onto whatever binaries already sit in target/, which is the one outcome the
# marking exists to prevent. Refuse rather than emit a package that misdescribes
# itself.
if [[ -n "${FEATURES}" && "${NO_BUILD}" -eq 1 ]]; then
echo "--features cannot be combined with --no-build: the features would not" >&2
echo "reach the binaries, but the Version would claim they had." >&2
exit 1
fi
cd "${PROJECT_ROOT}"
# Ensure cargo-deb is available
@@ -81,13 +100,33 @@ if [[ -z "${VERSION_OVERRIDE}" ]]; then
if [[ -n "$(git status --porcelain 2>/dev/null)" ]]; then
DIRTY_SUFFIX=".dirty"
fi
# Debian Version: <upstream>~dev+git<YYYYMMDD>.<sha>[.dirty]-1
# A feature build of a given commit is a different package from the
# default build of that same commit, but nothing else in this version
# says so: the crate version, the date and the sha are all identical.
# Without a marker the two are byte-identical versions, so installing
# one over the other is an apt no-op (the very failure the per-commit
# version above exists to prevent) and the node offers no way to tell
# which one it is running. Underscores and commas are not legal in a
# Debian version, so the feature list is folded to dots.
FEATURE_SUFFIX=""
if [[ -n "${FEATURES}" ]]; then
FEATURE_SUFFIX="+$(printf '%s' "${FEATURES}" | tr -c 'a-zA-Z0-9.' '.')"
fi
# Debian Version: <upstream>~dev+git<YYYYMMDD>.<sha>[.dirty][+<features>]-1
# The "~" makes every dev build sort BEFORE the eventual tagged
# release; the date+sha makes consecutive dev builds compare as
# different versions; the trailing "-1" is the Debian revision.
VERSION_OVERRIDE="${BASE_VERSION}~dev+git${GIT_DATE}.${GIT_SHA}${DIRTY_SUFFIX}-1"
# The feature suffix sorts ABOVE the unsuffixed build, so installing a
# feature build is an upgrade and reverting to the default build is a
# downgrade — which apt refuses without being told to, and `dpkg -i`
# performs. Revert with `dpkg -i`, not `apt install`.
VERSION_OVERRIDE="${BASE_VERSION}~dev+git${GIT_DATE}.${GIT_SHA}${DIRTY_SUFFIX}${FEATURE_SUFFIX}-1"
echo "Auto-derived dev Version: ${VERSION_OVERRIDE}"
fi
elif [[ -n "${FEATURES}" ]]; then
echo "Warning: --version was given with --features, so the Version carries no" >&2
echo "feature marker and this package is indistinguishable from a default" >&2
echo "build of the same commit. Mark it yourself if that matters." >&2
fi
# Build the .deb package
@@ -105,6 +144,9 @@ fi
if [[ "${NO_BUILD}" -eq 1 ]]; then
cargo_args+=(--no-build)
fi
if [[ -n "${FEATURES}" ]]; then
cargo_args+=(--features "${FEATURES}")
fi
cargo "${cargo_args[@]}"
# Move output to deploy/
packaging/debian/fips.service
+5
View File
@@ -19,6 +19,11 @@ RestartSec=5
RuntimeDirectory=fips
RuntimeDirectoryMode=0750
# Log directory (/var/log/fips/), where the built-in tick-body profiler writes
# its capture files. Declared so systemd creates it on start and removes it on
# purge; the daemon runs as root and already has access without it.
LogsDirectory=fips
# Security hardening (daemon runs as root for TUN and raw sockets)
ProtectHome=yes
PrivateTmp=yes
packaging/openwrt-apk/build-apk.sh
+2
View File
@@ -182,6 +182,8 @@ install -m 0755 "$RELEASE_DIR/fips" "$STAGE_DIR/usr/bin/fips"
install -m 0755 "$RELEASE_DIR/fipsctl" "$STAGE_DIR/usr/bin/fipsctl"
install -m 0755 "$RELEASE_DIR/fipstop" "$STAGE_DIR/usr/bin/fipstop"
install -m 0755 "$RELEASE_DIR/fips-gateway" "$STAGE_DIR/usr/bin/fips-gateway"
install -m 0755 "$FILES_DIR/usr/bin/fips-mesh-setup" "$STAGE_DIR/usr/bin/fips-mesh-setup"
install -m 0755 "$FILES_DIR/usr/bin/fips-ap-setup" "$STAGE_DIR/usr/bin/fips-ap-setup"
install -d "$STAGE_DIR/etc/init.d"
install -m 0755 "$FILES_DIR/etc/init.d/fips" "$STAGE_DIR/etc/init.d/fips"
packaging/openwrt-ipk/Makefile
+6
View File
@@ -96,6 +96,12 @@ define Package/fips/install
$(INSTALL_BIN) $(RUST_RELEASE_DIR)/fipstop $(1)/usr/bin/fipstop
$(INSTALL_BIN) $(RUST_RELEASE_DIR)/fips-gateway $(1)/usr/bin/fips-gateway
# 802.11s mesh backhaul setup helper
$(INSTALL_BIN) $(CURDIR)/files/usr/bin/fips-mesh-setup $(1)/usr/bin/fips-mesh-setup
# Open "FIPS" access SSID setup helper
$(INSTALL_BIN) $(CURDIR)/files/usr/bin/fips-ap-setup $(1)/usr/bin/fips-ap-setup
# procd init script
$(INSTALL_DIR) $(1)/etc/init.d
$(INSTALL_BIN) $(CURDIR)/files/etc/init.d/fips $(1)/etc/init.d/fips
packaging/openwrt-ipk/README.md
+1
View File
@@ -14,6 +14,7 @@ For ad-hoc deployment without the build system, see
| `/usr/bin/fipsctl` | CLI control tool (`fipsctl show peers`, `fipsctl show links`, …) |
| `/usr/bin/fipstop` | Live TUI dashboard |
| `/usr/bin/fips-gateway` | Outbound LAN gateway service (not started by default) |
| `/usr/bin/fips-mesh-setup` | Opt-in helper — creates an open 802.11s mesh interface for router↔router backhaul |
| `/etc/init.d/fips` | procd service for the daemon (auto-start, crash respawn) |
| `/etc/init.d/fips-gateway` | procd service for the gateway (disabled by default) |
| `/etc/fips/fips.yaml` | Node configuration (edit before first start) |
packaging/openwrt-ipk/build-ipk.sh
+2
View File
@@ -161,6 +161,8 @@ install -m 0755 "$RELEASE_DIR/fips" "$DATA_DIR/usr/bin/fips"
install -m 0755 "$RELEASE_DIR/fipsctl" "$DATA_DIR/usr/bin/fipsctl"
install -m 0755 "$RELEASE_DIR/fipstop" "$DATA_DIR/usr/bin/fipstop"
install -m 0755 "$RELEASE_DIR/fips-gateway" "$DATA_DIR/usr/bin/fips-gateway"
install -m 0755 "$FILES_DIR/usr/bin/fips-mesh-setup" "$DATA_DIR/usr/bin/fips-mesh-setup"
install -m 0755 "$FILES_DIR/usr/bin/fips-ap-setup" "$DATA_DIR/usr/bin/fips-ap-setup"
install -d "$DATA_DIR/etc/init.d"
install -m 0755 "$FILES_DIR/etc/init.d/fips" "$DATA_DIR/etc/init.d/fips"
packaging/openwrt-ipk/files/etc/fips/fips.yaml
+79 -8
View File
@@ -10,12 +10,21 @@ node:
#
# Or set an explicit key (overrides persistent):
# nsec: "nsec1..."
discovery:
# Optional Nostr-mediated overlay endpoint discovery.
# Mesh-lookup protocol (node.lookup.*): the overlay coordinate-lookup engine
# (mesh address -> coordinates). Defaults shown; uncomment to override.
# lookup:
# ttl: 64
# attempt_timeouts_secs: [1, 2, 4, 8]
# recent_expiry_secs: 10
# backoff_base_secs: 0
# backoff_max_secs: 0
# forward_min_interval_secs: 2
rendezvous:
# Optional Nostr-mediated overlay endpoint rendezvous.
# nostr:
# enabled: true
# policy: configured_only # disabled | configured_only | open
# open_discovery_max_pending: 64 # caps queued open-discovery retries
# open_discovery_max_pending: 64 # caps queued open-rendezvous retries
# app: "fips-overlay-v1"
# advertise: true
# advert_relays:
@@ -34,6 +43,14 @@ node:
# - "stun:stun.cloudflare.com:3478"
# - "stun:global.stun.twilio.com:3478"
# mDNS/DNS-SD peer rendezvous on the local link. Ships commented (the
# daemon default is off); 'fips-ap-setup' uncomments it when creating
# the access SSID — phone FIPS apps cannot see raw-Ethernet beacons,
# so mDNS is how they find this router's daemon. Daemon-wide switch,
# left enabled on 'fips-ap-setup remove'.
# lan:
# enabled: true
tun:
enabled: true
name: fips0
@@ -55,7 +72,11 @@ dns:
transports:
udp:
bind_addr: "0.0.0.0:2121"
# Dual-stack wildcard, not "0.0.0.0": access-SSID clients (phones) learn
# this node's addresses from the mDNS advert and prefer the IPv6
# link-local — a v4-only bind silently drops their Noise msg1.
# OpenWrt is Linux (bindv6only=0), so "[::]" accepts v4 too.
bind_addr: "[::]:2121"
# advertise_on_nostr: true
# public: false # false => advertise udp:nat; true => advertise bound host:port
# accept_connections: true # default; refuse inbound msg1 when false
@@ -74,23 +95,73 @@ transports:
ethernet:
wan:
interface: "eth0"
discovery: true
listen: true
announce: true
auto_connect: true
accept_connections: true
wwan:
interface: "phy0-sta0"
discovery: true
listen: true
announce: true
auto_connect: true
accept_connections: true
lan:
interface: "br-lan"
discovery: true
listen: true
announce: true
auto_connect: true
accept_connections: true
# 802.11s mesh backhaul between FIPS routers. These entries ship
# commented out so a stock install that never creates fips-mesh*
# logs no per-boot "interface missing" bind warning. Running
# 'fips-mesh-setup <radio>' creates the interface AND uncomments the
# matching block here (once per radio; radio0 -> fips-mesh0, radio1 ->
# fips-mesh1); 'fips-mesh-setup remove' re-comments it. Restart fips
# after — a transport whose interface is missing at startup is skipped,
# not retried. Dual-band routers can mesh on both bands at once —
# failover, not multipath: FIPS keeps one active link per peer, the
# other band stands by. The mesh runs OPEN (no SAE) with 802.11s
# forwarding off: FIPS's Noise handshake is the encryption and
# authentication, and FIPS is the routing layer. See
# docs/how-to/set-up-80211s-mesh-backhaul.md.
# mesh0:
# interface: "fips-mesh0"
# discovery: true
# announce: true
# auto_connect: true
# accept_connections: true
# mesh1:
# interface: "fips-mesh1"
# discovery: true
# announce: true
# auto_connect: true
# accept_connections: true
# Open "!FIPS" access SSID for phones and laptops running FIPS. These
# entries ship commented out so a stock install that never creates
# fips-ap* logs no per-boot "interface missing" bind warning. Running
# 'fips-ap-setup <radio>' creates the interface AND uncomments the
# matching block here (once per radio; radio0 -> fips-ap0, radio1 ->
# fips-ap1); 'fips-ap-setup remove' re-comments it. Restart fips after
# — a transport whose interface is missing at startup is skipped, not
# retried. The SSID is OPEN and isolated on purpose: FIPS's Noise
# handshake is the only security layer, and associated clients reach
# nothing but the FIPS handshake surface. See
# docs/how-to/set-up-open-access-ssid.md.
# ap0:
# interface: "fips-ap0"
# discovery: true
# announce: true
# auto_connect: true
# accept_connections: true
# ap1:
# interface: "fips-ap1"
# discovery: true
# announce: true
# auto_connect: true
# accept_connections: true
# Bluetooth Low Energy transport — requires BlueZ and the 'ble' feature.
# ble:
# adapter: "hci0"
@@ -121,5 +192,5 @@ peers: []
# - transport: udp
# addr: "test-us01.fips.network:2121" # IP or hostname (e.g., "peer.example.com:2121")
# - transport: udp
# addr: "nat" # Use node.discovery.nostr for Nostr/STUN hole punching
# addr: "nat" # Use node.rendezvous.nostr for Nostr/STUN hole punching
# connect_policy: auto_connect
packaging/openwrt-ipk/files/usr/bin/fips-ap-setup
Executable
+412
View File
@@ -0,0 +1,412 @@
#!/bin/sh
# fips-ap-setup — configure the open "FIPS" access SSID for phones/laptops.
#
# Usage:
# fips-ap-setup <radio> [ssid] e.g. fips-ap-setup radio0
# fips-ap-setup remove [radio] no radio: remove all instances
#
# Creates an open AP on the given radio so client devices running FIPS can
# reach the router. Every FIPS router broadcasts the SAME SSID ("!FIPS" by
# default — the leading '!' sorts it to the top of alphabetically ordered
# network pickers): same SSID + unique BSSIDs is one standard ESS, so a
# phone saves the network once and roams between all FIPS routers natively.
#
# - encryption 'none' — the AP is OPEN on purpose. FIPS's Noise IK
# handshake authenticates and encrypts everything above the radio, and
# the security type must be uniform across ALL routers anyway: clients
# key a saved network on SSID + security type, so one router with a PSK
# splits the ESS into a different saved network. A stranger can
# associate AND form a FIPS peer link — that is the point of open
# access. The Noise handshake authenticates each link (no
# impersonation of another identity, no MITM); it does NOT gate who
# may peer. Admission is open up to the daemon's max-peers cap; the
# firewall zone below is what confines every client to the FIPS
# overlay (no path to br-lan or the WAN).
# - DHCPv4 + RA IPv6 — dnsmasq serves DHCPv4 from a FIXED subnet,
# 10.21.<N>.0/24 (echoes FIPS port 2121), identical on every router:
# a roaming phone keeps its lease across routers, and dnsmasq's
# authoritative mode (the OpenWrt default) ACKs the renew a foreign
# router never issued. odhcpd additionally announces a ULA prefix in
# router advertisements (stateless SLAAC); DHCPv6 stays off. FIPS
# itself only needs link-local + mDNS, but client provisioning checks
# (Android disconnects without an RA or a DHCP offer) and plain
# laptops both want a real address. The network provides no internet,
# so phones mark it unvalidated and keep cellular as the default
# route while staying associated.
# - ISOLATED — own network and firewall zone: no path to
# br-lan, no forwarding to the WAN, and AP client isolation on.
# Associated clients reach only the FIPS handshake surface.
#
# Interfaces are named per radio index (radio0 -> fips-ap0, radio1 ->
# fips-ap1). Unlike the 802.11s backhaul there is NO same-channel
# constraint — clients scan when they roam, so every router picks its
# access channels freely.
#
# The shipped /etc/fips/fips.yaml carries 'ap0' and 'ap1' entries under
# 'transports.ethernet' bound to these names, but commented out — a stock
# install that never creates fips-ap* then logs no bind warning. This
# helper uncomments the matching entry when it creates an interface and
# re-comments it on remove, so the daemon binds the transport without a
# manual config edit. It also uncomments the node.rendezvous.lan block
# (mDNS/DNS-SD — how phone FIPS apps discover the daemon); that switch is
# daemon-wide and stays on at remove. After an interface is up, restart
# fips.
# See docs/how-to/set-up-open-access-ssid.md for the full guide.
DEFAULT_SSID="!FIPS"
CONFIG="/etc/fips/fips.yaml"
# Replace $CONFIG with the rewritten $CONFIG.tmp. Force mode 0600 first: the
# package installs fips.yaml 0600 (it may hold an inline 'nsec' private key),
# and a fresh tmp file would otherwise land world-readable after the move.
ap_config_write() {
chmod 600 "$CONFIG.tmp" && mv "$CONFIG.tmp" "$CONFIG"
}
# Uncomment the 'ap<idx>' transports.ethernet block in $CONFIG (created by
# 'fips-ap-setup'). Reversible with ap_config_disable. Returns:
# 0 enabled (or already active) 1 no config file 2 no such block
ap_config_enable() {
idx="$1"
[ -f "$CONFIG" ] || return 1
grep -q "^ ap$idx:" "$CONFIG" && return 0
grep -q "^ # ap$idx:" "$CONFIG" || return 2
awk -v idx="$idx" '
$0 ~ ("^ # ap" idx ":[ \t]*$") { blk = 1; sub(/^ # /, " "); print; next }
blk && /^ # / { sub(/^ # /, " "); print; next }
{ blk = 0; print }
' "$CONFIG" > "$CONFIG.tmp" && ap_config_write
}
# Uncomment the 'lan' block under node.rendezvous in $CONFIG — the daemon's
# mDNS/DNS-SD responder+browser. Phone FIPS apps cannot open raw-Ethernet
# sockets, so mDNS is how they find this router's daemon. The match is
# scoped to node.rendezvous: transports.ethernet also has a 'lan' entry at
# the same indent. Daemon-wide switch — enabled here, deliberately NOT
# re-commented on remove (other transports use it once on). Returns:
# 0 enabled (or already active) 1 no config file 2 no such block
lan_rendezvous_enable() {
[ -f "$CONFIG" ] || return 1
state="$(awk '
/^[A-Za-z_]/ { top = $1 }
top == "node:" && /^ [A-Za-z_]/ { sec = $1 }
top == "node:" && sec == "rendezvous:" && /^ lan:[ \t]*$/ { print "active"; exit }
top == "node:" && sec == "rendezvous:" && /^ # lan:[ \t]*$/ { print "commented"; exit }
' "$CONFIG")"
case "$state" in
active) return 0 ;;
commented) ;;
*) return 2 ;;
esac
awk '
/^[A-Za-z_]/ { top = $1 }
top == "node:" && /^ [A-Za-z_]/ { sec = $1 }
top == "node:" && sec == "rendezvous:" && $0 ~ /^ # lan:[ \t]*$/ { blk = 1; sub(/^ # /, " "); print; next }
blk && /^ # / { sub(/^ # /, " "); print; next }
{ blk = 0; print }
' "$CONFIG" > "$CONFIG.tmp" && ap_config_write
}
# Re-comment the 'ap<idx>' block so the daemon stops binding it (and stops
# warning about the now-missing interface). Inverse of ap_config_enable.
ap_config_disable() {
idx="$1"
[ -f "$CONFIG" ] || return 1
grep -q "^ ap$idx:" "$CONFIG" || return 0
awk -v idx="$idx" '
$0 ~ ("^ ap" idx ":[ \t]*$") { blk = 1; sub(/^ /, " # "); print; next }
blk && /^ / { sub(/^ /, " # "); print; next }
{ blk = 0; print }
' "$CONFIG" > "$CONFIG.tmp" && ap_config_write
}
usage() {
echo "Usage: fips-ap-setup <radio> [ssid]" >&2
echo " fips-ap-setup remove [radio]" >&2
echo "Radios on this device:" >&2
uci show wireless 2>/dev/null | sed -n "s/^wireless\.\([^.]*\)=wifi-device$/ \1/p" >&2
exit 1
}
# List the UCI section names of fips-managed access-point wifi-ifaces.
ap_sections() {
uci show wireless 2>/dev/null | sed -n "s/^wireless\.\(fips_ap[^.=]*\)=wifi-iface$/\1/p"
}
# ---------------------------------------------------------------------------
# remove [radio] — delete the wireless, network, dhcp, and firewall sections
# created below
# ---------------------------------------------------------------------------
if [ "$1" = "remove" ]; then
if [ -n "$2" ]; then
SECTIONS="fips_ap_$(printf '%s' "$2" | tr -c 'a-zA-Z0-9_' '_')"
else
SECTIONS="$(ap_sections)"
fi
[ -n "$SECTIONS" ] || {
echo "No fips access-point instances configured."
exit 0
}
for section in $SECTIONS; do
ifname="$(uci -q get "wireless.$section.ifname")"
uci -q delete "wireless.$section"
uci -q delete "network.$section"
uci -q delete "dhcp.$section"
uci -q del_list "firewall.fips_ap.network=$section"
# Re-comment the matching ap<N> transport in fips.yaml so the
# daemon stops warning about the interface we just removed.
idx="$(printf '%s' "$ifname" | sed -n 's/.*[^0-9]\([0-9]\{1,\}\)$/\1/p')"
[ -n "$idx" ] && ap_config_disable "$idx"
echo "Removed ${ifname:-$section}."
done
# Drop the shared zone and its rules once the last instance is gone.
if [ -z "$(uci -q get firewall.fips_ap.network)" ]; then
uci -q delete firewall.fips_ap
uci -q delete firewall.fips_ap_icmpv6
uci -q delete firewall.fips_ap_dhcpv4
uci -q delete firewall.fips_ap_mdns
uci -q delete firewall.fips_ap_fips_udp
uci -q delete firewall.fips_ap_fips_tcp
fi
uci commit wireless
uci commit network
uci commit dhcp
uci commit firewall
wifi reload
/etc/init.d/dnsmasq reload
/etc/init.d/odhcpd reload
/etc/init.d/firewall reload
echo "Restart fips: /etc/init.d/fips restart"
exit 0
fi
RADIO="$1"
SSID="${2:-$DEFAULT_SSID}"
[ -n "$RADIO" ] || usage
if [ "$(uci -q get "wireless.$RADIO")" != "wifi-device" ]; then
echo "Error: '$RADIO' is not a wifi-device in /etc/config/wireless." >&2
usage
fi
# One instance per radio: section fips_ap_<radio>, netdev fips-ap<N>
# where N is the radio's trailing index (radio0 -> fips-ap0). For radios
# named without a trailing number, fall back to the first free index.
SECTION="fips_ap_$(printf '%s' "$RADIO" | tr -c 'a-zA-Z0-9_' '_')"
IDX="$(printf '%s' "$RADIO" | sed -n 's/.*[^0-9]\([0-9]\{1,\}\)$/\1/p')"
[ -n "$IDX" ] || IDX="$(printf '%s' "$RADIO" | sed -n 's/^\([0-9]\{1,\}\)$/\1/p')"
if [ -z "$IDX" ]; then
IDX=0
while uci show wireless 2>/dev/null | grep -q "\.ifname='fips-ap$IDX'"; do
IDX=$((IDX + 1))
done
fi
AP_IFNAME="fips-ap$IDX"
# Refuse a name collision from another radio's instance (e.g. two radios
# whose names end in the same digit) rather than silently hijacking it.
OWNER="$(uci show wireless 2>/dev/null \
| sed -n "s/^wireless\.\(fips_ap[^.=]*\)\.ifname='$AP_IFNAME'$/\1/p")"
if [ -n "$OWNER" ] && [ "$OWNER" != "$SECTION" ]; then
echo "Error: $AP_IFNAME is already used by section '$OWNER'." >&2
echo "Remove it first: fips-ap-setup remove" >&2
exit 1
fi
# ---------------------------------------------------------------------------
# Wireless: open AP with client isolation. Clients of the same AP cannot
# exchange L2 frames directly — two FIPS phones on one router still reach
# each other through the router at the overlay layer. Isolation is an L2
# control only: a stranger who peers is an overlay peer like any other, so
# the FIPS overlay (not L2) is the trust boundary between clients.
# ---------------------------------------------------------------------------
uci -q delete "wireless.$SECTION"
uci set "wireless.$SECTION=wifi-iface"
uci set "wireless.$SECTION.device=$RADIO"
uci set "wireless.$SECTION.mode=ap"
uci set "wireless.$SECTION.ssid=$SSID"
uci set "wireless.$SECTION.encryption=none"
uci set "wireless.$SECTION.isolate=1"
uci set "wireless.$SECTION.ifname=$AP_IFNAME"
uci set "wireless.$SECTION.network=$SECTION"
# Radios ship disabled on fresh OpenWrt installs; a disabled radio would
# leave the AP down with no error anywhere visible.
if [ "$(uci -q get "wireless.$RADIO.disabled")" = "1" ]; then
echo "Note: enabling $RADIO (was disabled)."
uci -q delete "wireless.$RADIO.disabled"
fi
CHANNEL="$(uci -q get "wireless.$RADIO.channel")"
BAND="$(uci -q get "wireless.$RADIO.band")"
# ---------------------------------------------------------------------------
# Network: IPv4 from the fixed convention 10.21.<IDX>.1/24 — deterministic,
# so every router serving the same radio index lands on the same subnet and
# a roaming client's lease stays valid. IPv6 is a static ULA /64 so odhcpd
# has a prefix to announce; that space is per-router and disposable — a
# roaming phone SLAACs a fresh address on each router, and the FIPS overlay
# identity (not the IP) is the mobility anchor. The ULA is derived from the
# router's global ULA prefix; a re-run keeps the address already configured.
# ---------------------------------------------------------------------------
AP_ADDR="$(uci -q get "network.$SECTION.ip6addr")"
case "$AP_ADDR" in
fd*) ;; # keep the existing address on re-run
*)
ULA_BASE=""
ULA_PREFIX="$(uci -q get network.globals.ula_prefix)"
case "$ULA_PREFIX" in
fd*::/48) ULA_BASE="${ULA_PREFIX%::/48}" ;;
esac
if [ -z "$ULA_BASE" ]; then
HEX="$(head -c 5 /dev/urandom | hexdump -e '5/1 "%02x"')"
ULA_BASE="fd$(printf '%s' "$HEX" | cut -c1-2):$(printf '%s' "$HEX" | cut -c3-6):$(printf '%s' "$HEX" | cut -c7-10)"
echo "Note: no usable ULA prefix in network.globals — generated $ULA_BASE::/48 for this AP."
fi
# 64000 = 0xfa00 — high subnet IDs keep clear of br-lan's low
# ip6assign allocations from the same ULA prefix.
AP_ADDR="$ULA_BASE:$(printf '%04x' $((64000 + IDX)))::1/64"
;;
esac
AP_ADDR4="10.21.$IDX.1"
uci -q delete "network.$SECTION"
uci set "network.$SECTION=interface"
uci set "network.$SECTION.proto=static"
uci set "network.$SECTION.ipaddr=$AP_ADDR4"
uci set "network.$SECTION.netmask=255.255.255.0"
uci set "network.$SECTION.ip6addr=$AP_ADDR"
# ---------------------------------------------------------------------------
# DHCP/RA: dnsmasq DHCPv4 leases out of 10.21.<IDX>.0/24, plus router
# advertisements for the ULA (stateless SLAAC, no DHCPv6). ra_default '2'
# announces a default router even without an upstream default route:
# Android's provisioning wants address + route + DNS, and its validation
# probe then fails by design (no internet), so the phone keeps cellular as
# the default route. 'dhcpv4 server' is read by BOTH dnsmasq (the default
# DHCPv4 server) and odhcpd (serves v4 only when odhcpd.maindhcp is set),
# so either arrangement hands out leases.
# ---------------------------------------------------------------------------
uci -q delete "dhcp.$SECTION"
uci set "dhcp.$SECTION=dhcp"
uci set "dhcp.$SECTION.interface=$SECTION"
uci set "dhcp.$SECTION.ra=server"
uci set "dhcp.$SECTION.ra_default=2"
uci set "dhcp.$SECTION.dhcpv6=disabled"
uci set "dhcp.$SECTION.dhcpv4=server"
uci set "dhcp.$SECTION.start=10"
uci set "dhcp.$SECTION.limit=200"
# Authoritative is the OpenWrt default, but roaming correctness depends on
# it (a foreign router must ACK a lease it never issued), so pin it.
[ -n "$(uci -q get dhcp.@dnsmasq[0])" ] && uci set dhcp.@dnsmasq[0].authoritative=1
# ---------------------------------------------------------------------------
# Firewall: one shared 'fips_ap' zone for all instances. Everything is
# rejected except what a FIPS client needs — DHCPv4 (addressing), ICMPv6
# (SLAAC itself), mDNS (discovery), and the FIPS UDP/TCP transports (the
# handshake surface).
# The raw-Ethernet transport (EtherType 0x2121) is not IP and never
# traverses the firewall. No forwardings exist, so there is no path to
# br-lan or the WAN.
# ---------------------------------------------------------------------------
if [ "$(uci -q get firewall.fips_ap)" != "zone" ]; then
uci set firewall.fips_ap=zone
fi
uci set firewall.fips_ap.name=fips_ap
uci set firewall.fips_ap.input=REJECT
uci set firewall.fips_ap.output=ACCEPT
uci set firewall.fips_ap.forward=REJECT
uci -q del_list "firewall.fips_ap.network=$SECTION"
uci add_list "firewall.fips_ap.network=$SECTION"
# ap_rule <section-suffix> <name> <proto> [dest_port]
ap_rule() {
rule="firewall.fips_ap_$1"
uci -q delete "$rule"
uci set "$rule=rule"
uci set "$rule.name=$2"
uci set "$rule.src=fips_ap"
uci set "$rule.proto=$3"
uci set "$rule.target=ACCEPT"
[ -z "${4:-}" ] || uci set "$rule.dest_port=$4"
}
ap_rule icmpv6 "FIPS-AP-ICMPv6" icmp
uci set firewall.fips_ap_icmpv6.family=ipv6
ap_rule dhcpv4 "FIPS-AP-DHCPv4" udp 67
uci set firewall.fips_ap_dhcpv4.family=ipv4
ap_rule mdns "FIPS-AP-mDNS" udp 5353
ap_rule fips_udp "FIPS-AP-FIPS-UDP" udp 2121
ap_rule fips_tcp "FIPS-AP-FIPS-TCP" tcp 8443
uci commit wireless
uci commit network
uci commit dhcp
uci commit firewall
wifi reload
/etc/init.d/dnsmasq reload
/etc/init.d/odhcpd reload
/etc/init.d/firewall reload
# Enable the matching ap<N> transport in the shipped fips.yaml (it ships
# commented out). Tailor the restart hint to what we could do.
ap_config_enable "$IDX"
case $? in
0) TRANSPORT_NOTE="The ap$IDX transport in $CONFIG that binds '$AP_IFNAME' is
now uncommented and enabled." ;;
1) TRANSPORT_NOTE="No $CONFIG found — add a transports.ethernet entry binding
interface '$AP_IFNAME' by hand." ;;
*) TRANSPORT_NOTE="No 'ap$IDX' entry in $CONFIG — add a transports.ethernet
entry binding interface '$AP_IFNAME' by hand (copy the ap0 block)." ;;
esac
# Phones discover the daemon via mDNS, not raw-Ethernet beacons — make sure
# the daemon-wide mDNS rendezvous is on.
if lan_rendezvous_enable; then
MDNS_NOTE="node.rendezvous.lan (mDNS) is enabled — phone FIPS apps
discover this router via DNS-SD."
else
MDNS_NOTE="Could not enable mDNS in $CONFIG — set
'node.rendezvous.lan.enabled: true' by hand; phone FIPS apps rely
on it to discover this router."
fi
cat <<EOF
Created open access SSID '$SSID' as $AP_IFNAME on $RADIO \
(band ${BAND:-?}, channel ${CHANNEL:-auto}).
DHCPv4 on $AP_ADDR4/24 and RA IPv6 on $AP_ADDR — no internet,
isolated from br-lan and the WAN.
ALL FIPS routers must broadcast this SSID with the same security type
(open) — phones then save it once and roam between routers as one
network. The 10.21.$IDX.0/24 subnet is the same on every router on
purpose: leases survive roaming. Unlike the mesh backhaul, channels
are free per router. On a dual-band router, run fips-ap-setup for the
other radio too so clients can pick either band.
On first connect a phone warns that the network has no internet —
choose "stay connected" and "don't ask again". That choice is stored
per SSID, so it covers every FIPS router.
Next steps:
1. $TRANSPORT_NOTE
2. $MDNS_NOTE
Restart the daemon AFTER the interface is up — a transport whose
interface is missing at startup is skipped, not retried:
/etc/init.d/fips restart
3. Associate a phone or laptop running FIPS and verify:
iw dev $AP_IFNAME station dump
and the FIPS link on top of it:
fipsctl show peers
Run 'fips-ap-setup remove' to undo all instances, or
'fips-ap-setup remove $RADIO' for just this one.
EOF
packaging/openwrt-ipk/files/usr/bin/fips-mesh-setup
Executable
+261
View File
@@ -0,0 +1,261 @@
#!/bin/sh
# fips-mesh-setup — configure open 802.11s mesh interfaces for FIPS backhaul.
#
# Usage:
# fips-mesh-setup <radio> [mesh-id] e.g. fips-mesh-setup radio1
# fips-mesh-setup remove [radio] no radio: remove all instances
#
# Creates a mesh-point interface on the given radio and leaves everything
# above L2 to FIPS. Run once per radio: dual-band routers can mesh on both
# bands at once (2.4 GHz reaches further, 5 GHz carries more). Note this is
# failover, not multipath — FIPS keeps one active link per peer; the other
# band stands by and reconnects the peer if the active link dies.
#
# - encryption 'none' — the mesh is OPEN on purpose. FIPS's Noise IK
# handshake authenticates and encrypts every peer link, so SAE would
# only duplicate that (and on ath10k it forces the slower raw Tx/Rx
# firmware mode). A stranger can form an 802.11s peering AND a FIPS
# peer link — the Noise handshake authenticates each link (no
# impersonation of another identity, no MITM), it does not gate who
# may peer. Admission is open up to the daemon's max-peers cap.
# - mesh_fwding '0' — disables 802.11s HWMP forwarding so each mesh
# link is a plain L2 neighbor link. FIPS is the routing layer; two
# routing layers would fight.
#
# Interfaces are named per radio index (radio0 -> fips-mesh0, radio1 ->
# fips-mesh1) and are intentionally NOT bridged into br-lan: the FIPS
# Ethernet transport binds each directly and runs discovery beacons over it.
#
# The shipped /etc/fips/fips.yaml carries 'mesh0' and 'mesh1' entries under
# 'transports.ethernet' bound to these names, but commented out — a stock
# install that never creates fips-mesh* then logs no bind warning. This
# helper uncomments the matching entry when it creates an interface and
# re-comments it on remove, so the daemon binds the transport without a
# manual config edit. After an interface is up, restart fips.
# See docs/how-to/set-up-80211s-mesh-backhaul.md for the full guide.
DEFAULT_MESH_ID="fips-mesh"
CONFIG="/etc/fips/fips.yaml"
# Replace $CONFIG with the rewritten $CONFIG.tmp. Force mode 0600 first: the
# package installs fips.yaml 0600 (it may hold an inline 'nsec' private key),
# and a fresh tmp file would otherwise land world-readable after the move.
mesh_config_write() {
chmod 600 "$CONFIG.tmp" && mv "$CONFIG.tmp" "$CONFIG"
}
# Uncomment the 'mesh<idx>' transports.ethernet block in $CONFIG (created by
# 'fips-mesh-setup'). Reversible with mesh_config_disable. Returns:
# 0 enabled (or already active) 1 no config file 2 no such block
mesh_config_enable() {
idx="$1"
[ -f "$CONFIG" ] || return 1
grep -q "^ mesh$idx:" "$CONFIG" && return 0
grep -q "^ # mesh$idx:" "$CONFIG" || return 2
awk -v idx="$idx" '
$0 ~ ("^ # mesh" idx ":[ \t]*$") { blk = 1; sub(/^ # /, " "); print; next }
blk && /^ # / { sub(/^ # /, " "); print; next }
{ blk = 0; print }
' "$CONFIG" > "$CONFIG.tmp" && mesh_config_write
}
# Re-comment the 'mesh<idx>' block so the daemon stops binding it (and stops
# warning about the now-missing interface). Inverse of mesh_config_enable.
mesh_config_disable() {
idx="$1"
[ -f "$CONFIG" ] || return 1
grep -q "^ mesh$idx:" "$CONFIG" || return 0
awk -v idx="$idx" '
$0 ~ ("^ mesh" idx ":[ \t]*$") { blk = 1; sub(/^ /, " # "); print; next }
blk && /^ / { sub(/^ /, " # "); print; next }
{ blk = 0; print }
' "$CONFIG" > "$CONFIG.tmp" && mesh_config_write
}
usage() {
echo "Usage: fips-mesh-setup <radio> [mesh-id]" >&2
echo " fips-mesh-setup remove [radio]" >&2
echo "Radios on this device:" >&2
uci show wireless 2>/dev/null | sed -n "s/^wireless\.\([^.]*\)=wifi-device$/ \1/p" >&2
exit 1
}
# List the UCI section names of fips-managed mesh wifi-ifaces.
mesh_sections() {
uci show wireless 2>/dev/null | sed -n "s/^wireless\.\(fips_mesh[^.=]*\)=wifi-iface$/\1/p"
}
# ---------------------------------------------------------------------------
# remove [radio] — delete the wireless and network sections created below
# ---------------------------------------------------------------------------
if [ "$1" = "remove" ]; then
if [ -n "$2" ]; then
SECTIONS="fips_mesh_$(printf '%s' "$2" | tr -c 'a-zA-Z0-9_' '_')"
else
SECTIONS="$(mesh_sections)"
fi
[ -n "$SECTIONS" ] || {
echo "No fips mesh instances configured."
exit 0
}
for section in $SECTIONS; do
ifname="$(uci -q get "wireless.$section.ifname")"
uci -q delete "wireless.$section"
uci -q delete "network.$section"
# Re-comment the matching mesh<N> transport in fips.yaml so the
# daemon stops warning about the interface we just removed.
idx="$(printf '%s' "$ifname" | sed -n 's/.*[^0-9]\([0-9]\{1,\}\)$/\1/p')"
[ -n "$idx" ] && mesh_config_disable "$idx"
echo "Removed ${ifname:-$section}."
done
uci commit wireless
uci commit network
# 'wifi reload' re-applies the whole wireless config, so it briefly drops
# every client AP on all radios (a few seconds) — expected on remove.
wifi reload
echo "Restart fips: /etc/init.d/fips restart"
exit 0
fi
RADIO="$1"
MESH_ID="${2:-$DEFAULT_MESH_ID}"
[ -n "$RADIO" ] || usage
if [ "$(uci -q get "wireless.$RADIO")" != "wifi-device" ]; then
echo "Error: '$RADIO' is not a wifi-device in /etc/config/wireless." >&2
usage
fi
# One instance per radio: section fips_mesh_<radio>, netdev fips-mesh<N>
# where N is the radio's trailing index (radio0 -> fips-mesh0). For radios
# named without a trailing number, fall back to the first free index.
SECTION="fips_mesh_$(printf '%s' "$RADIO" | tr -c 'a-zA-Z0-9_' '_')"
IDX="$(printf '%s' "$RADIO" | sed -n 's/.*[^0-9]\([0-9]\{1,\}\)$/\1/p')"
[ -n "$IDX" ] || IDX="$(printf '%s' "$RADIO" | sed -n 's/^\([0-9]\{1,\}\)$/\1/p')"
if [ -z "$IDX" ]; then
IDX=0
while uci show wireless 2>/dev/null | grep -q "\.ifname='fips-mesh$IDX'"; do
IDX=$((IDX + 1))
done
fi
MESH_IFNAME="fips-mesh$IDX"
# Refuse a name collision from another radio's instance (e.g. two radios
# whose names end in the same digit) rather than silently hijacking it.
OWNER="$(uci show wireless 2>/dev/null \
| sed -n "s/^wireless\.\(fips_mesh[^.=]*\)\.ifname='$MESH_IFNAME'$/\1/p")"
if [ -n "$OWNER" ] && [ "$OWNER" != "$SECTION" ]; then
echo "Error: $MESH_IFNAME is already used by section '$OWNER'." >&2
echo "Remove it first: fips-mesh-setup remove" >&2
exit 1
fi
# ---------------------------------------------------------------------------
# Driver capability check (advisory — config below is harmless either way)
# ---------------------------------------------------------------------------
if command -v iw >/dev/null 2>&1; then
if ! iw list 2>/dev/null | grep -q "\* mesh point"; then
echo "Warning: no radio on this device advertises 'mesh point' support" >&2
echo "(iw list | grep 'mesh point'). The interface may fail to come up." >&2
fi
fi
# ---------------------------------------------------------------------------
# Wireless: open 802.11s mesh point, HWMP forwarding off
# ---------------------------------------------------------------------------
uci -q delete "wireless.$SECTION"
uci set "wireless.$SECTION=wifi-iface"
uci set "wireless.$SECTION.device=$RADIO"
uci set "wireless.$SECTION.mode=mesh"
uci set "wireless.$SECTION.mesh_id=$MESH_ID"
uci set "wireless.$SECTION.encryption=none"
uci set "wireless.$SECTION.mesh_fwding=0"
uci set "wireless.$SECTION.ifname=$MESH_IFNAME"
uci set "wireless.$SECTION.network=$SECTION"
# Radios ship disabled on fresh OpenWrt installs; a disabled radio would
# leave the mesh interface down with no error anywhere visible.
if [ "$(uci -q get "wireless.$RADIO.disabled")" = "1" ]; then
echo "Note: enabling $RADIO (was disabled)."
uci -q delete "wireless.$RADIO.disabled"
fi
# The mesh inherits the radio's channel, and mesh points only peer on the
# same channel. 'auto' lets each router pick its own — the classic silent
# non-peering cause — so surface the setting loudly.
CHANNEL="$(uci -q get "wireless.$RADIO.channel")"
BAND="$(uci -q get "wireless.$RADIO.band")"
if [ -z "$CHANNEL" ] || [ "$CHANNEL" = "auto" ]; then
echo "Warning: $RADIO channel is '${CHANNEL:-unset}' — each router may" >&2
echo "auto-select a different channel and mesh points only peer on the" >&2
echo "same one. Pin the same channel on every backhaul router, e.g.:" >&2
echo " uci set wireless.$RADIO.channel='36' && uci commit wireless && wifi reload" >&2
fi
# A client (sta) interface on the same radio follows its upstream AP's
# channel and drags every other interface with it — a mesh pinned to a
# different channel silently never joins, and does not recover when the
# STA disconnects.
for s in $(uci show wireless 2>/dev/null | sed -n "s/^wireless\.\([^.]*\)\.mode='sta'$/\1/p"); do
if [ "$(uci -q get "wireless.$s.device")" = "$RADIO" ]; then
echo "Warning: $RADIO also carries client interface '$s' (mode 'sta')." >&2
echo "The whole radio follows that STA's upstream channel — a mesh" >&2
echo "pinned to a different channel stays down silently. Align the" >&2
echo "mesh channel with the upstream AP, or put the mesh on a radio" >&2
echo "without a STA (a roaming uplink is incompatible with a" >&2
echo "fixed-channel mesh on the same radio)." >&2
fi
done
# ---------------------------------------------------------------------------
# Network: unmanaged interface so netifd brings the netdev up. No IP config —
# the FIPS Ethernet transport speaks raw frames on it.
# ---------------------------------------------------------------------------
uci -q delete "network.$SECTION"
uci set "network.$SECTION=interface"
uci set "network.$SECTION.proto=none"
uci commit wireless
uci commit network
# 'wifi reload' re-applies the whole wireless config, so it briefly drops
# every client AP on all radios (a few seconds) — expected when adding a mesh.
wifi reload
# Enable the matching mesh<N> transport in the shipped fips.yaml (it ships
# commented out). Tailor the restart hint to what we could do.
mesh_config_enable "$IDX"
case $? in
0) TRANSPORT_NOTE="The mesh$IDX transport in $CONFIG that binds '$MESH_IFNAME' is
now uncommented and enabled." ;;
1) TRANSPORT_NOTE="No $CONFIG found — add a transports.ethernet entry binding
interface '$MESH_IFNAME' by hand." ;;
*) TRANSPORT_NOTE="No 'mesh$IDX' entry in $CONFIG — add a transports.ethernet
entry binding interface '$MESH_IFNAME' by hand (copy the mesh0 block)." ;;
esac
cat <<EOF
Created open 802.11s mesh '$MESH_ID' as $MESH_IFNAME on $RADIO \
(band ${BAND:-?}, channel ${CHANNEL:-auto}).
ALL routers in this backhaul must share this mesh ID AND channel
(per band). On a dual-band router, run fips-mesh-setup for the other
radio too — second band is a standby path (failover, not multipath).
Next steps:
1. $TRANSPORT_NOTE
Restart the daemon AFTER the interface is up — a transport whose
interface is missing at startup is skipped, not retried:
/etc/init.d/fips restart
2. Verify L2 peering with a second FIPS router in range:
iw dev $MESH_IFNAME station dump
and the FIPS link on top of it:
fipsctl show peers
Run 'fips-mesh-setup remove' to undo all instances, or
'fips-mesh-setup remove $RADIO' for just this one.
EOF
packaging/systemd/README.install.md
+1 -1
View File
@@ -68,7 +68,7 @@ and set the interface name:
transports:
ethernet:
interface: "eth0"
discovery: true
listen: true
announce: true
auto_connect: true
accept_connections: true
packaging/systemd/fips.service
+5
View File
@@ -14,6 +14,11 @@ RestartSec=5
RuntimeDirectory=fips
RuntimeDirectoryMode=0750
# Log directory (/var/log/fips/), where the built-in tick-body profiler writes
# its capture files. Declared so systemd creates it on start and removes it on
# purge; the daemon runs as root and already has access without it.
LogsDirectory=fips
# Security hardening (daemon runs as root for TUN and raw sockets)
ProtectHome=yes
PrivateTmp=yes
src/bin/fips.rs
+14 -17
View File
@@ -8,7 +8,7 @@ use fips::config::{IdentitySource, resolve_identity};
use fips::version;
use fips::{Config, Node};
use std::path::PathBuf;
use tracing::{debug, error, info, warn};
use tracing::{debug, error, info};
use tracing_subscriber::{EnvFilter, fmt};
/// FIPS mesh network daemon
@@ -157,26 +157,23 @@ async fn run_daemon(
info!("FIPS running");
// Run the RX event loop until shutdown signal.
// stop() drops the packet channel, causing run_rx_loop to exit.
tokio::select! {
result = node.run_rx_loop() => {
match result {
Ok(()) => info!("RX loop exited"),
Err(e) => error!("RX loop error: {}", e),
}
}
_ = shutdown_signal => {
info!("Shutdown signal received");
}
// Serve until the shutdown signal, then drain in place before returning.
// The rx loop observes the signal directly, so its channels are never
// destructively cancelled — they live in the loop's locals across serve and
// drain, and are dropped only on clean exit (after which teardown does not
// need them). On the signal the loop broadcasts a shutdown Disconnect and
// waits (bounded by node.drain_timeout_secs) for peers to clear.
match node.run_rx_loop_with_shutdown(shutdown_signal).await {
Ok(()) => info!("RX loop exited"),
Err(e) => error!("RX loop error: {}", e),
}
info!("FIPS shutting down");
// Stop the node (shuts down transports, TUN, I/O threads)
if let Err(e) = node.stop().await {
warn!("Error during shutdown: {}", e);
}
// Close the drain window (if the loop drained) and tear down. A drained
// loop tears down without re-broadcasting; a loop that exited some other
// way falls back to the immediate stop().
node.finish_shutdown().await;
info!("FIPS shutdown complete");
}
src/bin/fipsctl.rs
+45
View File
@@ -76,6 +76,37 @@ enum Commands {
#[command(subcommand)]
what: StatsCommands,
},
/// Control the built-in profiler (requires a `--features profiling` build)
#[cfg(feature = "profiling")]
Profile {
#[command(subcommand)]
what: ProfileCommands,
},
}
#[cfg(feature = "profiling")]
#[derive(Subcommand, Debug)]
enum ProfileCommands {
/// Profile the rx-loop tick body
Tick {
#[command(subcommand)]
action: ProfileTickAction,
},
}
#[cfg(feature = "profiling")]
#[derive(Subcommand, Debug)]
enum ProfileTickAction {
/// Start a capture
On {
/// Directory for the capture file (default /var/log/fips)
#[arg(long)]
dir: Option<PathBuf>,
},
/// Stop the running capture
Off,
/// Report capture state
Status,
}
#[derive(Subcommand, Debug)]
@@ -471,6 +502,20 @@ fn main() {
build_command("show_stats_history", params)
}
},
#[cfg(feature = "profiling")]
Commands::Profile { what } => match what {
ProfileCommands::Tick { action } => match action {
ProfileTickAction::On { dir } => match dir {
Some(dir) => build_command(
"profile_tick_on",
serde_json::json!({"dir": dir.display().to_string()}),
),
None => build_query("profile_tick_on"),
},
ProfileTickAction::Off => build_query("profile_tick_off"),
ProfileTickAction::Status => build_query("profile_tick_status"),
},
},
Commands::Keygen { .. } => unreachable!(),
};
src/bin/fipstop/ui/routing.rs
+22 -22
View File
@@ -134,8 +134,8 @@ fn draw_routing_stats(
let cols =
Layout::horizontal([Constraint::Percentage(50), Constraint::Percentage(50)]).split(inner);
// Shorthand for a nested counter value (e.g. discovery.req_received).
let disc = |key: &str| helpers::nested_u64(data, "discovery", key);
// Shorthand for a nested counter value (e.g. lookup.req_received).
let lookup = |key: &str| helpers::nested_u64(data, "lookup", key);
let err = |key: &str| helpers::nested_u64(data, "error_signals", key);
let cong = |key: &str| helpers::nested_u64(data, "congestion", key);
@@ -174,32 +174,32 @@ fn draw_routing_stats(
));
left.push(Line::from(""));
left.extend(section(
"Discovery Requests",
"Lookup Requests",
&[
("Received", disc("req_received")),
("Forwarded", disc("req_forwarded")),
("Initiated", disc("req_initiated")),
("Deduplicated", disc("req_deduplicated")),
("Target Is Us", disc("req_target_is_us")),
("Duplicate", disc("req_duplicate")),
("Bloom Miss", disc("req_bloom_miss")),
("Backoff Suppressed", disc("req_backoff_suppressed")),
("Fwd Rate Limited", disc("req_forward_rate_limited")),
("TTL Exhausted", disc("req_ttl_exhausted")),
("Decode Error", disc("req_decode_error")),
("Received", lookup("req_received")),
("Forwarded", lookup("req_forwarded")),
("Initiated", lookup("req_initiated")),
("Deduplicated", lookup("req_deduplicated")),
("Target Is Us", lookup("req_target_is_us")),
("Duplicate", lookup("req_duplicate")),
("Bloom Miss", lookup("req_bloom_miss")),
("Backoff Suppressed", lookup("req_backoff_suppressed")),
("Fwd Rate Limited", lookup("req_forward_rate_limited")),
("TTL Exhausted", lookup("req_ttl_exhausted")),
("Decode Error", lookup("req_decode_error")),
],
));
left.push(Line::from(""));
left.extend(section(
"Discovery Responses",
"Lookup Responses",
&[
("Received", disc("resp_received")),
("Accepted", disc("resp_accepted")),
("Forwarded", disc("resp_forwarded")),
("Timed Out", disc("resp_timed_out")),
("Identity Miss", disc("resp_identity_miss")),
("Proof Failed", disc("resp_proof_failed")),
("Decode Error", disc("resp_decode_error")),
("Received", lookup("resp_received")),
("Accepted", lookup("resp_accepted")),
("Forwarded", lookup("resp_forwarded")),
("Timed Out", lookup("resp_timed_out")),
("Identity Miss", lookup("resp_identity_miss")),
("Proof Failed", lookup("resp_proof_failed")),
("Decode Error", lookup("resp_decode_error")),
],
));
src/bloom/mod.rs
-60
View File
@@ -1,60 +0,0 @@
//! Bloom Filter Implementation
//!
//! 1KB Bloom filters for reachability in FIPS routing. Each node
//! maintains filters that summarize which destinations are reachable
//! through each peer, enabling efficient routing decisions without
//! global network knowledge.
//!
//! ## v1 Parameters
//!
//! - Size: 1 KB (8,192 bits) - sized for actual ~400-800 entry occupancy
//! - Hash functions: k=5 - optimal at ~1,200 entries, good for 800-1,600
//! - Bandwidth: 1 KB/announce (75% reduction from original 4KB design)
//!
//! These parameters are right-sized for typical network occupancy of
//! ~250-800 entries per node.
mod filter;
mod state;
use thiserror::Error;
pub use filter::BloomFilter;
pub use state::BloomState;
/// Default filter size in bits (1KB = 8,192 bits).
///
/// Sized for ~800-1,600 entries. FPR ~0.05% at 400 entries, ~0.9% at 800.
/// This is v1 protocol default (size_class=1).
pub const DEFAULT_FILTER_SIZE_BITS: usize = 8192;
/// Default filter size in bytes (1KB).
pub const DEFAULT_FILTER_SIZE_BYTES: usize = DEFAULT_FILTER_SIZE_BITS / 8;
/// Default number of hash functions.
///
/// k=5 is optimal at ~1,200 entries and a good compromise for 800-1,600.
/// At 400 entries: FPR ~0.05%. At 800 entries: FPR ~0.9%.
pub const DEFAULT_HASH_COUNT: u8 = 5;
/// Size class for v1 protocol (1 KB filters).
pub const V1_SIZE_CLASS: u8 = 1;
/// Filter sizes by size_class: bytes = 512 << size_class
pub const SIZE_CLASS_BYTES: [usize; 4] = [512, 1024, 2048, 4096];
/// Errors related to Bloom filter operations.
#[derive(Debug, Error)]
pub enum BloomError {
#[error("invalid filter size: expected {expected} bits, got {got}")]
InvalidSize { expected: usize, got: usize },
#[error("filter size must be a multiple of 8, got {0}")]
SizeNotByteAligned(usize),
#[error("hash count must be positive")]
ZeroHashCount,
}
#[cfg(test)]
mod tests;
src/cache/coord_cache.rs
Vendored
+1 -1
View File
@@ -9,7 +9,7 @@ use std::collections::HashMap;
use super::CacheStats;
use super::entry::CacheEntry;
use crate::NodeAddr;
use crate::tree::TreeCoordinate;
use crate::proto::stp::TreeCoordinate;
/// Default maximum entries in coordinate cache.
pub const DEFAULT_COORD_CACHE_SIZE: usize = 50_000;
src/cache/entry.rs
Vendored
+1 -1
View File
@@ -1,6 +1,6 @@
//! Cache entry with TTL and LRU tracking.
use crate::tree::TreeCoordinate;
use crate::proto::stp::TreeCoordinate;
/// A cached coordinate entry.
#[derive(Clone, Debug)]
src/config/mod.rs
+210 -31
View File
@@ -34,9 +34,9 @@ use thiserror::Error;
#[cfg(target_os = "linux")]
pub use gateway::{ConntrackConfig, GatewayConfig, GatewayDnsConfig, PortForward, Proto};
pub use node::{
BloomConfig, BuffersConfig, CacheConfig, ControlConfig, DiscoveryConfig, LimitsConfig,
NodeConfig, NostrDiscoveryConfig, NostrDiscoveryPolicy, RateLimitConfig, RekeyConfig,
RetryConfig, SessionConfig, SessionMmpConfig, TreeConfig,
BloomConfig, BuffersConfig, CacheConfig, ControlConfig, LimitsConfig, LookupConfig, MmpConfig,
NodeConfig, NostrRendezvousConfig, NostrRendezvousPolicy, RateLimitConfig, RekeyConfig,
RendezvousConfig, RetryConfig, SessionConfig, SessionMmpConfig, TreeConfig,
};
pub use peer::{ConnectPolicy, PeerAddress, PeerConfig};
pub use transport::{
@@ -490,10 +490,59 @@ impl Config {
source: e,
})?;
serde_yaml::from_str(&contents).map_err(|e| ConfigError::ParseYaml {
path: path.to_path_buf(),
source: e,
})
let mut config: Config =
serde_yaml::from_str(&contents).map_err(|e| ConfigError::ParseYaml {
path: path.to_path_buf(),
source: e,
})?;
config.normalize_deprecated_keys();
Ok(config)
}
/// COMPAT (drop at the v2 cutover): fold a deprecated `node.discovery:`
/// block into the `node.lookup.*` (mesh-lookup scalars) and
/// `node.rendezvous.*` (nostr/LAN peer rendezvous) tables that replaced it.
///
/// Runs at every deserialize boundary (see `load_file`). A present legacy
/// field fills the corresponding new-table field, so a config that predates
/// the split keeps behaving identically. When a legacy block is seen, a
/// one-time deprecation warning names the old→new key moves. Exposed to the
/// crate so config tests that deserialize directly can invoke it.
pub(crate) fn normalize_deprecated_keys(&mut self) {
let Some(compat) = self.node.discovery.take() else {
return;
};
tracing::warn!(
target: "fips::config",
"`node.discovery.*` is deprecated and will be removed: mesh-lookup \
scalars moved to `node.lookup.*`, and peer-rendezvous keys moved to \
`node.rendezvous.nostr.*` / `node.rendezvous.lan.*`. Please migrate; \
a legacy `node.discovery` block still applies for now."
);
if let Some(v) = compat.ttl {
self.node.lookup.ttl = v;
}
if let Some(v) = compat.attempt_timeouts_secs {
self.node.lookup.attempt_timeouts_secs = v;
}
if let Some(v) = compat.recent_expiry_secs {
self.node.lookup.recent_expiry_secs = v;
}
if let Some(v) = compat.backoff_base_secs {
self.node.lookup.backoff_base_secs = v;
}
if let Some(v) = compat.backoff_max_secs {
self.node.lookup.backoff_max_secs = v;
}
if let Some(v) = compat.forward_min_interval_secs {
self.node.lookup.forward_min_interval_secs = v;
}
if let Some(v) = compat.nostr {
self.node.rendezvous.nostr = v;
}
if let Some(v) = compat.lan {
self.node.rendezvous.lan = v;
}
}
/// Get the standard search paths in priority order (lowest to highest).
@@ -601,7 +650,7 @@ impl Config {
/// Validate cross-field configuration invariants.
pub fn validate(&self) -> Result<(), ConfigError> {
let nostr = &self.node.discovery.nostr;
let nostr = &self.node.rendezvous.nostr;
let any_transport_advertises_on_nostr = self
.transports
@@ -621,13 +670,13 @@ impl Config {
if any_transport_advertises_on_nostr && !nostr.enabled {
return Err(ConfigError::Validation(
"at least one transport has `advertise_on_nostr = true`, but `node.discovery.nostr.enabled` is false".to_string(),
"at least one transport has `advertise_on_nostr = true`, but `node.rendezvous.nostr.enabled` is false".to_string(),
));
}
if self.peers.iter().any(|peer| peer.via_nostr) && !nostr.enabled {
return Err(ConfigError::Validation(
"at least one peer has `via_nostr = true`, but `node.discovery.nostr.enabled` is false".to_string(),
"at least one peer has `via_nostr = true`, but `node.rendezvous.nostr.enabled` is false".to_string(),
));
}
@@ -649,12 +698,12 @@ impl Config {
if nostr.enabled && has_nat_udp_advert {
if nostr.dm_relays.is_empty() {
return Err(ConfigError::Validation(
"NAT UDP advert publishing requires `node.discovery.nostr.dm_relays` to be non-empty".to_string(),
"NAT UDP advert publishing requires `node.rendezvous.nostr.dm_relays` to be non-empty".to_string(),
));
}
if nostr.stun_servers.is_empty() {
return Err(ConfigError::Validation(
"NAT UDP advert publishing requires `node.discovery.nostr.stun_servers` to be non-empty".to_string(),
"NAT UDP advert publishing requires `node.rendezvous.nostr.stun_servers` to be non-empty".to_string(),
));
}
}
@@ -748,6 +797,84 @@ node:
assert!(config.has_identity());
}
/// The fips.yaml shipped in the OpenWrt package must keep parsing as the
/// config schema evolves. Both the 802.11s mesh backhaul entries
/// (docs/how-to/set-up-80211s-mesh-backhaul.md) and the open-access SSID
/// entries (docs/how-to/set-up-open-access-ssid.md) ship commented out —
/// one per radio, so dual-band routers can run either on both bands — so
/// a stock install that never creates fips-mesh*/fips-ap* logs no
/// per-boot bind warning; `fips-mesh-setup`/`fips-ap-setup` uncomment the
/// matching block when they create the interface. Verify both states
/// parse: as shipped (both inactive), and after the uncomment the helpers
/// perform.
#[test]
fn shipped_openwrt_config_parses() {
let yaml = include_str!("../../packaging/openwrt-ipk/files/etc/fips/fips.yaml");
// As shipped: parses, and the mesh/ap entries are commented out (a
// running daemon binds no fips-mesh*/fips-ap* transport, no warning).
let config: Config = serde_yaml::from_str(yaml).expect("shipped OpenWrt fips.yaml");
for name in ["mesh0", "mesh1", "ap0", "ap1"] {
assert!(
!config
.transports
.ethernet
.iter()
.any(|(n, _)| n == Some(name)),
"{name} must ship commented out, not active, in fips.yaml"
);
}
// What `fips-mesh-setup`/`fips-ap-setup` produce: uncomment each
// block, which must still parse into a transport bound to the right
// netdev.
let uncommented =
uncomment_transport_blocks(&uncomment_transport_blocks(yaml, "mesh"), "ap");
let config: Config = serde_yaml::from_str(&uncommented)
.expect("fips.yaml with mesh and ap transports uncommented");
for (name, interface) in [
("mesh0", "fips-mesh0"),
("mesh1", "fips-mesh1"),
("ap0", "fips-ap0"),
("ap1", "fips-ap1"),
] {
assert!(
config
.transports
.ethernet
.iter()
.any(|(n, eth)| n == Some(name) && eth.interface == interface),
"{name} entry missing after uncommenting shipped fips.yaml"
);
}
}
/// Mirror the setup helpers' block uncomment: strip the ` # ` prefix
/// from each `# <prefix><N>:` header and its ` # ` continuation
/// lines, leaving every other comment untouched.
fn uncomment_transport_blocks(yaml: &str, prefix: &str) -> String {
let header = format!(" # {prefix}");
let mut out = String::new();
let mut in_block = false;
for line in yaml.lines() {
let is_header = line
.strip_prefix(&header)
.and_then(|r| r.strip_suffix(':'))
.is_some_and(|n| !n.is_empty() && n.bytes().all(|b| b.is_ascii_digit()));
if is_header {
in_block = true;
out.push_str(&line.replacen(" # ", " ", 1));
} else if in_block && line.starts_with(" # ") {
out.push_str(&line.replacen(" # ", " ", 1));
} else {
in_block = false;
out.push_str(line);
}
out.push('\n');
}
out
}
#[test]
fn test_parse_yaml_with_hex() {
let yaml = r#"
@@ -1288,7 +1415,9 @@ peers:
}
#[test]
fn test_parse_nostr_discovery_config() {
fn test_parse_legacy_discovery_nostr_config_compat() {
// COMPAT (drop at the v2 cutover): a deprecated `node.discovery.nostr`
// block must fold into `node.rendezvous.nostr` via normalize.
let yaml = r#"
node:
discovery:
@@ -1312,26 +1441,27 @@ peers:
- transport: udp
addr: "nat"
"#;
let config: Config = serde_yaml::from_str(yaml).unwrap();
assert!(config.node.discovery.nostr.enabled);
assert!(!config.node.discovery.nostr.advertise);
assert_eq!(config.node.discovery.nostr.app, "fips.nat.test.v1");
assert_eq!(config.node.discovery.nostr.signal_ttl_secs, 45);
let mut config: Config = serde_yaml::from_str(yaml).unwrap();
config.normalize_deprecated_keys();
assert!(config.node.rendezvous.nostr.enabled);
assert!(!config.node.rendezvous.nostr.advertise);
assert_eq!(config.node.rendezvous.nostr.app, "fips.nat.test.v1");
assert_eq!(config.node.rendezvous.nostr.signal_ttl_secs, 45);
assert_eq!(
config.node.discovery.nostr.policy,
NostrDiscoveryPolicy::ConfiguredOnly
config.node.rendezvous.nostr.policy,
NostrRendezvousPolicy::ConfiguredOnly
);
assert_eq!(config.node.discovery.nostr.open_discovery_max_pending, 12);
assert_eq!(config.node.rendezvous.nostr.open_discovery_max_pending, 12);
assert_eq!(
config.node.discovery.nostr.advert_relays,
config.node.rendezvous.nostr.advert_relays,
vec!["wss://relay-a.example".to_string()]
);
assert_eq!(
config.node.discovery.nostr.dm_relays,
config.node.rendezvous.nostr.dm_relays,
vec!["wss://relay-b.example".to_string()]
);
assert_eq!(
config.node.discovery.nostr.stun_servers,
config.node.rendezvous.nostr.stun_servers,
vec!["stun:stun.example.org:3478".to_string()]
);
assert_eq!(
@@ -1341,6 +1471,55 @@ peers:
assert!(config.peers[0].via_nostr);
}
#[test]
fn test_parse_lookup_and_rendezvous_new_keys() {
// The post-split keys parse directly, with no deprecated block and no
// normalize warning.
let yaml = r#"
node:
lookup:
ttl: 7
attempt_timeouts_secs: [3, 6]
forward_min_interval_secs: 9
rendezvous:
nostr:
enabled: true
app: "fips.new.keys.v1"
"#;
let mut config: Config = serde_yaml::from_str(yaml).unwrap();
config.normalize_deprecated_keys();
assert_eq!(config.node.lookup.ttl, 7);
assert_eq!(config.node.lookup.attempt_timeouts_secs, vec![3, 6]);
assert_eq!(config.node.lookup.forward_min_interval_secs, 9);
// Unset scalar keeps its default.
assert_eq!(config.node.lookup.recent_expiry_secs, 10);
assert!(config.node.rendezvous.nostr.enabled);
assert_eq!(config.node.rendezvous.nostr.app, "fips.new.keys.v1");
assert!(config.node.discovery.is_none());
}
#[test]
fn test_legacy_discovery_lookup_scalars_compat() {
// COMPAT (drop at the v2 cutover): legacy `node.discovery` mesh-lookup
// scalars must fold into `node.lookup`; unset keys keep their defaults.
let yaml = r#"
node:
discovery:
ttl: 5
backoff_base_secs: 4
backoff_max_secs: 30
"#;
let mut config: Config = serde_yaml::from_str(yaml).unwrap();
config.normalize_deprecated_keys();
assert_eq!(config.node.lookup.ttl, 5);
assert_eq!(config.node.lookup.backoff_base_secs, 4);
assert_eq!(config.node.lookup.backoff_max_secs, 30);
// Unset legacy scalar leaves the new-table default intact.
assert_eq!(config.node.lookup.attempt_timeouts_secs, vec![1, 2, 4, 8]);
// The compat block is consumed by normalize.
assert!(config.node.discovery.is_none());
}
#[test]
fn test_validate_transport_advert_requires_nostr_enabled() {
let mut config = Config::default();
@@ -1348,7 +1527,7 @@ peers:
advertise_on_nostr: Some(true),
..Default::default()
});
config.node.discovery.nostr.enabled = false;
config.node.rendezvous.nostr.enabled = false;
let err = config.validate().expect_err("validation should fail");
assert!(err.to_string().contains("advertise_on_nostr"));
@@ -1364,7 +1543,7 @@ peers:
}],
..Default::default()
};
config.node.discovery.nostr.enabled = false;
config.node.rendezvous.nostr.enabled = false;
let err = config.validate().expect_err("validation should fail");
assert!(err.to_string().contains("via_nostr"));
@@ -1385,7 +1564,7 @@ peers:
// Empty addresses + via_nostr=true + nostr.enabled=true → ok.
config.peers[0].via_nostr = true;
config.node.discovery.nostr.enabled = true;
config.node.rendezvous.nostr.enabled = true;
config
.validate()
.expect("via_nostr should allow empty addresses");
@@ -1394,8 +1573,8 @@ peers:
#[test]
fn test_validate_nat_udp_advert_requires_relays_and_stun() {
let mut config = Config::default();
config.node.discovery.nostr.enabled = true;
config.node.discovery.nostr.dm_relays.clear();
config.node.rendezvous.nostr.enabled = true;
config.node.rendezvous.nostr.dm_relays.clear();
config.transports.udp = TransportInstances::Single(UdpConfig {
advertise_on_nostr: Some(true),
public: Some(false),
@@ -1405,8 +1584,8 @@ peers:
let err = config.validate().expect_err("validation should fail");
assert!(err.to_string().contains("dm_relays"));
config.node.discovery.nostr.dm_relays = vec!["wss://relay.example".to_string()];
config.node.discovery.nostr.stun_servers.clear();
config.node.rendezvous.nostr.dm_relays = vec!["wss://relay.example".to_string()];
config.node.rendezvous.nostr.stun_servers.clear();
let err = config.validate().expect_err("validation should fail");
assert!(err.to_string().contains("stun_servers"));
}
src/config/node.rs
+221 -75
View File
@@ -7,7 +7,7 @@
use serde::{Deserialize, Serialize};
use super::IdentityConfig;
use crate::mmp::{DEFAULT_LOG_INTERVAL_SECS, DEFAULT_OWD_WINDOW_SIZE, MmpConfig, MmpMode};
use crate::proto::mmp::{DEFAULT_LOG_INTERVAL_SECS, DEFAULT_OWD_WINDOW_SIZE, MmpMode};
// ============================================================================
// Node Configuration Subsections
@@ -186,48 +186,42 @@ impl CacheConfig {
}
}
/// Discovery protocol (`node.discovery.*`).
/// Mesh-lookup protocol (`node.lookup.*`): the overlay coordinate-lookup
/// engine (address → coordinates). The peer-rendezvous keys that used to
/// share this table (`nostr`/`lan`) now live under [`RendezvousConfig`]
/// (`node.rendezvous.*`).
#[derive(Debug, Clone, Serialize, Deserialize)]
pub struct DiscoveryConfig {
/// Hop limit for LookupRequest flood (`node.discovery.ttl`).
#[serde(default = "DiscoveryConfig::default_ttl")]
pub struct LookupConfig {
/// Hop limit for LookupRequest flood (`node.lookup.ttl`).
#[serde(default = "LookupConfig::default_ttl")]
pub ttl: u8,
/// Per-attempt timeouts in seconds (`node.discovery.attempt_timeouts_secs`).
/// Per-attempt timeouts in seconds (`node.lookup.attempt_timeouts_secs`).
/// Each entry is the time to wait for a response before sending the next
/// LookupRequest (with a fresh request_id). Sequence length determines the
/// total number of attempts before declaring the destination unreachable.
/// Default `[1, 2, 4, 8]` gives 4 attempts and a 15s total budget.
#[serde(default = "DiscoveryConfig::default_attempt_timeouts_secs")]
#[serde(default = "LookupConfig::default_attempt_timeouts_secs")]
pub attempt_timeouts_secs: Vec<u64>,
/// Dedup cache expiry in seconds (`node.discovery.recent_expiry_secs`).
#[serde(default = "DiscoveryConfig::default_recent_expiry_secs")]
/// Dedup cache expiry in seconds (`node.lookup.recent_expiry_secs`).
#[serde(default = "LookupConfig::default_recent_expiry_secs")]
pub recent_expiry_secs: u64,
/// Base backoff after lookup failure in seconds (`node.discovery.backoff_base_secs`).
/// Base backoff after lookup failure in seconds (`node.lookup.backoff_base_secs`).
/// Doubles per consecutive failure up to `backoff_max_secs`. Defaults to 0
/// (no post-failure suppression); the per-attempt sequence in
/// `attempt_timeouts_secs` provides the only retry pacing.
#[serde(default = "DiscoveryConfig::default_backoff_base_secs")]
#[serde(default = "LookupConfig::default_backoff_base_secs")]
pub backoff_base_secs: u64,
/// Maximum backoff cap in seconds (`node.discovery.backoff_max_secs`).
#[serde(default = "DiscoveryConfig::default_backoff_max_secs")]
/// Maximum backoff cap in seconds (`node.lookup.backoff_max_secs`).
#[serde(default = "LookupConfig::default_backoff_max_secs")]
pub backoff_max_secs: u64,
/// Minimum interval between forwarded lookups for the same target in seconds
/// (`node.discovery.forward_min_interval_secs`).
/// (`node.lookup.forward_min_interval_secs`).
/// Defense-in-depth against misbehaving nodes.
#[serde(default = "DiscoveryConfig::default_forward_min_interval_secs")]
#[serde(default = "LookupConfig::default_forward_min_interval_secs")]
pub forward_min_interval_secs: u64,
/// Nostr-mediated overlay endpoint discovery.
#[serde(default = "DiscoveryConfig::default_nostr")]
pub nostr: NostrDiscoveryConfig,
/// mDNS / DNS-SD peer discovery on the local link. Identity surface
/// is a strict subset of what `nostr.advertise` already publishes
/// publicly, so there's no marginal privacy cost; the latency win
/// for same-LAN peers is large (sub-second pairing, no relay).
#[serde(default = "DiscoveryConfig::default_lan")]
pub lan: crate::discovery::lan::LanDiscoveryConfig,
}
impl Default for DiscoveryConfig {
impl Default for LookupConfig {
fn default() -> Self {
Self {
ttl: 64,
@@ -236,13 +230,11 @@ impl Default for DiscoveryConfig {
backoff_base_secs: 0,
backoff_max_secs: 0,
forward_min_interval_secs: 2,
nostr: NostrDiscoveryConfig::default(),
lan: crate::discovery::lan::LanDiscoveryConfig::default(),
}
}
}
impl DiscoveryConfig {
impl LookupConfig {
fn default_ttl() -> u8 {
64
}
@@ -261,12 +253,45 @@ impl DiscoveryConfig {
fn default_forward_min_interval_secs() -> u64 {
2
}
fn default_nostr() -> NostrDiscoveryConfig {
NostrDiscoveryConfig::default()
}
fn default_lan() -> crate::discovery::lan::LanDiscoveryConfig {
crate::discovery::lan::LanDiscoveryConfig::default()
}
}
/// Peer rendezvous (`node.rendezvous.*`): how the node finds peers to connect
/// to at all — Nostr-mediated overlay endpoints and mDNS/DNS-SD on the local
/// link. Distinct from mesh lookup ([`LookupConfig`]), which finds coordinates
/// for an already-known mesh address.
#[derive(Debug, Clone, Default, Serialize, Deserialize)]
pub struct RendezvousConfig {
/// Nostr-mediated overlay endpoint rendezvous (`node.rendezvous.nostr.*`).
#[serde(default)]
pub nostr: NostrRendezvousConfig,
/// mDNS / DNS-SD peer rendezvous on the local link (`node.rendezvous.lan.*`).
/// Identity surface is a strict subset of what `nostr.advertise` already
/// publishes publicly, so there's no marginal privacy cost; the latency
/// win for same-LAN peers is large (sub-second pairing, no relay).
#[serde(default)]
pub lan: crate::mdns::LanRendezvousConfig,
}
/// COMPAT (drop at the v2 cutover): a deprecated legacy `node.discovery:` block.
///
/// The `node.discovery.*` table was split into `node.lookup.*` (mesh-lookup
/// scalars) and `node.rendezvous.*` (nostr/LAN peer rendezvous). Because
/// `NodeConfig` does not deny unknown fields, a still-deployed `node.discovery:`
/// block would otherwise deserialize into nothing and silently revert every
/// lookup/rendezvous setting to its default. This all-`Option` mirror captures
/// it so [`Config::normalize_deprecated_keys`] can fold it into the new tables
/// with a one-time deprecation warning; unset legacy keys stay `None` and leave
/// the new-table defaults intact.
#[derive(Debug, Clone, Deserialize)]
pub(crate) struct DiscoveryConfigCompat {
pub ttl: Option<u8>,
pub attempt_timeouts_secs: Option<Vec<u64>>,
pub recent_expiry_secs: Option<u64>,
pub backoff_base_secs: Option<u64>,
pub backoff_max_secs: Option<u64>,
pub forward_min_interval_secs: Option<u64>,
pub nostr: Option<NostrRendezvousConfig>,
pub lan: Option<crate::mdns::LanRendezvousConfig>,
}
/// Nostr advert discovery policy.
@@ -278,33 +303,33 @@ impl DiscoveryConfig {
/// - `open`: also consider adverts for non-configured peers
#[derive(Debug, Clone, Copy, Default, PartialEq, Eq, Serialize, Deserialize)]
#[serde(rename_all = "snake_case")]
pub enum NostrDiscoveryPolicy {
pub enum NostrRendezvousPolicy {
Disabled,
#[default]
ConfiguredOnly,
Open,
}
/// Nostr-mediated overlay endpoint discovery (`node.discovery.nostr.*`).
/// Nostr-mediated overlay endpoint discovery (`node.rendezvous.nostr.*`).
#[derive(Debug, Clone, Serialize, Deserialize)]
#[serde(deny_unknown_fields)]
pub struct NostrDiscoveryConfig {
pub struct NostrRendezvousConfig {
/// Enable Nostr-signaled traversal bootstrap.
#[serde(default)]
pub enabled: bool,
/// Publish service advertisements so remote peers can bootstrap inbound.
#[serde(default = "NostrDiscoveryConfig::default_advertise")]
#[serde(default = "NostrRendezvousConfig::default_advertise")]
pub advertise: bool,
/// Relay URLs used for service advertisements.
#[serde(default = "NostrDiscoveryConfig::default_advert_relays")]
#[serde(default = "NostrRendezvousConfig::default_advert_relays")]
pub advert_relays: Vec<String>,
/// Relay URLs used for encrypted signaling events.
#[serde(default = "NostrDiscoveryConfig::default_dm_relays")]
#[serde(default = "NostrRendezvousConfig::default_dm_relays")]
pub dm_relays: Vec<String>,
/// STUN servers used for local reflexive address discovery.
/// Outbound observation uses only this local list; peer-advertised STUN
/// values are informational and are not treated as egress targets.
#[serde(default = "NostrDiscoveryConfig::default_stun_servers")]
#[serde(default = "NostrRendezvousConfig::default_stun_servers")]
pub stun_servers: Vec<String>,
/// Whether to advertise local (RFC 1918 / ULA) interface addresses as
/// host candidates in the traversal offer.
@@ -318,85 +343,85 @@ pub struct NostrDiscoveryConfig {
#[serde(default)]
pub share_local_candidates: bool,
/// Traversal application namespace and advert identifier suffix.
#[serde(default = "NostrDiscoveryConfig::default_app")]
#[serde(default = "NostrRendezvousConfig::default_app")]
pub app: String,
/// Signaling TTL in seconds.
#[serde(default = "NostrDiscoveryConfig::default_signal_ttl_secs")]
#[serde(default = "NostrRendezvousConfig::default_signal_ttl_secs")]
pub signal_ttl_secs: u64,
/// Policy for advert-derived endpoint discovery.
#[serde(default)]
pub policy: NostrDiscoveryPolicy,
pub policy: NostrRendezvousPolicy,
/// Max number of open-discovery peers queued for outbound retry/connection
/// at once. Prevents unbounded queue growth from ambient advert traffic.
#[serde(default = "NostrDiscoveryConfig::default_open_discovery_max_pending")]
#[serde(default = "NostrRendezvousConfig::default_open_discovery_max_pending")]
pub open_discovery_max_pending: usize,
/// Max concurrent inbound traversal offers processed at once.
/// Acts as a rate limit against offer spam from relays.
#[serde(default = "NostrDiscoveryConfig::default_max_concurrent_incoming_offers")]
#[serde(default = "NostrRendezvousConfig::default_max_concurrent_incoming_offers")]
pub max_concurrent_incoming_offers: usize,
/// Max cached overlay adverts retained from relay traffic.
/// Bounds memory under ambient advert volume.
#[serde(default = "NostrDiscoveryConfig::default_advert_cache_max_entries")]
#[serde(default = "NostrRendezvousConfig::default_advert_cache_max_entries")]
pub advert_cache_max_entries: usize,
/// Max seen-session IDs retained for replay detection.
/// Oldest entries are evicted when the cap is exceeded.
#[serde(default = "NostrDiscoveryConfig::default_seen_sessions_max_entries")]
#[serde(default = "NostrRendezvousConfig::default_seen_sessions_max_entries")]
pub seen_sessions_max_entries: usize,
/// Overall punch attempt timeout in seconds.
#[serde(default = "NostrDiscoveryConfig::default_attempt_timeout_secs")]
#[serde(default = "NostrRendezvousConfig::default_attempt_timeout_secs")]
pub attempt_timeout_secs: u64,
/// Replay tracking retention window in seconds.
#[serde(default = "NostrDiscoveryConfig::default_replay_window_secs")]
#[serde(default = "NostrRendezvousConfig::default_replay_window_secs")]
pub replay_window_secs: u64,
/// Delay before punch traffic starts.
#[serde(default = "NostrDiscoveryConfig::default_punch_start_delay_ms")]
#[serde(default = "NostrRendezvousConfig::default_punch_start_delay_ms")]
pub punch_start_delay_ms: u64,
/// Interval between punch packets.
#[serde(default = "NostrDiscoveryConfig::default_punch_interval_ms")]
#[serde(default = "NostrRendezvousConfig::default_punch_interval_ms")]
pub punch_interval_ms: u64,
/// How long to keep punching before failure.
#[serde(default = "NostrDiscoveryConfig::default_punch_duration_ms")]
#[serde(default = "NostrRendezvousConfig::default_punch_duration_ms")]
pub punch_duration_ms: u64,
/// Advert TTL in seconds.
#[serde(default = "NostrDiscoveryConfig::default_advert_ttl_secs")]
#[serde(default = "NostrRendezvousConfig::default_advert_ttl_secs")]
pub advert_ttl_secs: u64,
/// How often adverts are refreshed in seconds.
#[serde(default = "NostrDiscoveryConfig::default_advert_refresh_secs")]
#[serde(default = "NostrRendezvousConfig::default_advert_refresh_secs")]
pub advert_refresh_secs: u64,
/// Settle delay in seconds after Nostr discovery starts before the
/// one-shot startup sweep of cached adverts runs. Allows the relay
/// subscription backlog to populate the in-memory advert cache.
/// Only used under `policy: open`. Default: 5.
#[serde(default = "NostrDiscoveryConfig::default_startup_sweep_delay_secs")]
#[serde(default = "NostrRendezvousConfig::default_startup_sweep_delay_secs")]
pub startup_sweep_delay_secs: u64,
/// Maximum age in seconds for cached adverts considered by the
/// one-shot startup sweep. Adverts whose `created_at` is older than
/// `now - startup_sweep_max_age_secs` are skipped. Only used under
/// `policy: open`. Default: 3600 (1 hour).
#[serde(default = "NostrDiscoveryConfig::default_startup_sweep_max_age_secs")]
#[serde(default = "NostrRendezvousConfig::default_startup_sweep_max_age_secs")]
pub startup_sweep_max_age_secs: u64,
/// Number of consecutive NAT-traversal failures against a peer before
/// an extended cooldown is applied to throttle further offer publishes.
/// At this threshold the daemon also actively re-fetches the peer's
/// advert from `advert_relays` to evict cache entries for peers that
/// have gone away. Default: 5.
#[serde(default = "NostrDiscoveryConfig::default_failure_streak_threshold")]
#[serde(default = "NostrRendezvousConfig::default_failure_streak_threshold")]
pub failure_streak_threshold: u32,
/// Cooldown applied to a peer once `failure_streak_threshold` is hit.
/// Suppresses both open-discovery sweep enqueues and per-attempt
/// retry firings until elapsed. Default: 1800 (30 minutes).
#[serde(default = "NostrDiscoveryConfig::default_extended_cooldown_secs")]
#[serde(default = "NostrRendezvousConfig::default_extended_cooldown_secs")]
pub extended_cooldown_secs: u64,
/// Minimum interval between `NAT traversal failed` WARN log lines for
/// the same peer. Subsequent failures inside the window log at DEBUG.
/// Reduces log spam on public-test nodes with many cache-learned
/// peers. Default: 300 (5 minutes).
#[serde(default = "NostrDiscoveryConfig::default_warn_log_interval_secs")]
#[serde(default = "NostrRendezvousConfig::default_warn_log_interval_secs")]
pub warn_log_interval_secs: u64,
/// Maximum entries retained in the per-npub failure-state map.
/// Bounds memory under high cache turnover. Oldest entries (by last
/// failure time) evicted when the cap is exceeded. Default: 4096.
#[serde(default = "NostrDiscoveryConfig::default_failure_state_max_entries")]
#[serde(default = "NostrRendezvousConfig::default_failure_state_max_entries")]
pub failure_state_max_entries: usize,
/// Cooldown applied after observing a fatal protocol mismatch on a
/// Nostr-adopted bootstrap transport (e.g. `Unknown FMP version`
@@ -404,11 +429,11 @@ pub struct NostrDiscoveryConfig {
/// of `extended_cooldown_secs` and much longer because the mismatch
/// is structural — re-traversing the peer is wasted effort until one
/// side upgrades. Default: 86400 (24 hours).
#[serde(default = "NostrDiscoveryConfig::default_protocol_mismatch_cooldown_secs")]
#[serde(default = "NostrRendezvousConfig::default_protocol_mismatch_cooldown_secs")]
pub protocol_mismatch_cooldown_secs: u64,
}
impl Default for NostrDiscoveryConfig {
impl Default for NostrRendezvousConfig {
fn default() -> Self {
Self {
enabled: false,
@@ -419,7 +444,7 @@ impl Default for NostrDiscoveryConfig {
share_local_candidates: false,
app: Self::default_app(),
signal_ttl_secs: Self::default_signal_ttl_secs(),
policy: NostrDiscoveryPolicy::default(),
policy: NostrRendezvousPolicy::default(),
open_discovery_max_pending: Self::default_open_discovery_max_pending(),
max_concurrent_incoming_offers: Self::default_max_concurrent_incoming_offers(),
advert_cache_max_entries: Self::default_advert_cache_max_entries(),
@@ -442,7 +467,7 @@ impl Default for NostrDiscoveryConfig {
}
}
impl NostrDiscoveryConfig {
impl NostrRendezvousConfig {
fn default_advertise() -> bool {
true
}
@@ -726,6 +751,41 @@ impl SessionConfig {
}
}
/// MMP configuration (`node.mmp.*`).
#[derive(Debug, Clone, Serialize, Deserialize)]
pub struct MmpConfig {
/// Operating mode (`node.mmp.mode`).
#[serde(default)]
pub mode: MmpMode,
/// Periodic operator log interval in seconds (`node.mmp.log_interval_secs`).
#[serde(default = "MmpConfig::default_log_interval_secs")]
pub log_interval_secs: u64,
/// OWD trend ring buffer size (`node.mmp.owd_window_size`).
#[serde(default = "MmpConfig::default_owd_window_size")]
pub owd_window_size: usize,
}
impl Default for MmpConfig {
fn default() -> Self {
Self {
mode: MmpMode::default(),
log_interval_secs: DEFAULT_LOG_INTERVAL_SECS,
owd_window_size: DEFAULT_OWD_WINDOW_SIZE,
}
}
}
impl MmpConfig {
fn default_log_interval_secs() -> u64 {
DEFAULT_LOG_INTERVAL_SECS
}
fn default_owd_window_size() -> usize {
DEFAULT_OWD_WINDOW_SIZE
}
}
/// Session-layer Metrics Measurement Protocol (`node.session_mmp.*`).
///
/// Separate from link-layer `node.mmp.*` to allow independent mode/interval
@@ -970,6 +1030,19 @@ pub struct NodeConfig {
#[serde(default = "NodeConfig::default_link_dead_timeout_secs")]
pub link_dead_timeout_secs: u64,
/// Graceful-shutdown drain deadline in seconds (`node.drain_timeout_secs`).
/// The bounded `Draining` phase broadcasts a shutdown `Disconnect` and then
/// waits up to this long for peers to clear before tearing down, early-
/// exiting as soon as all peers are gone. `None` selects the 2-second
/// default (see [`NodeConfig::drain_timeout`]).
///
/// Kept `Option` deliberately: `NodeConfig` has no `deny_unknown_fields`, so
/// a naive non-`Option` add with a `default` fn would silently rewrite the
/// value into deployed configs on the next serialize. The `Option` +
/// `skip_serializing_if` keeps absent configs absent.
#[serde(default, skip_serializing_if = "Option::is_none")]
pub drain_timeout_secs: Option<u64>,
/// Resource limits (`node.limits.*`).
#[serde(default)]
pub limits: LimitsConfig,
@@ -986,9 +1059,19 @@ pub struct NodeConfig {
#[serde(default)]
pub cache: CacheConfig,
/// Discovery protocol (`node.discovery.*`).
/// Mesh-lookup protocol (`node.lookup.*`).
#[serde(default)]
pub discovery: DiscoveryConfig,
pub lookup: LookupConfig,
/// Peer rendezvous (`node.rendezvous.*`).
#[serde(default)]
pub rendezvous: RendezvousConfig,
/// COMPAT (drop at the v2 cutover): a deprecated legacy `node.discovery:`
/// block, folded into `lookup`/`rendezvous` by
/// [`Config::normalize_deprecated_keys`]. Never re-serialized.
#[serde(default, skip_serializing)]
pub(crate) discovery: Option<DiscoveryConfigCompat>,
/// Spanning tree (`node.tree.*`).
#[serde(default)]
@@ -1041,11 +1124,14 @@ impl Default for NodeConfig {
base_rtt_ms: 100,
heartbeat_interval_secs: 10,
link_dead_timeout_secs: 30,
drain_timeout_secs: None,
limits: LimitsConfig::default(),
rate_limit: RateLimitConfig::default(),
retry: RetryConfig::default(),
cache: CacheConfig::default(),
discovery: DiscoveryConfig::default(),
lookup: LookupConfig::default(),
rendezvous: RendezvousConfig::default(),
discovery: None,
tree: TreeConfig::default(),
bloom: BloomConfig::default(),
session: SessionConfig::default(),
@@ -1089,12 +1175,72 @@ impl NodeConfig {
fn default_link_dead_timeout_secs() -> u64 {
30
}
/// Graceful-shutdown drain deadline as a `Duration`.
///
/// Returns the configured `drain_timeout_secs`, or the 2-second default
/// when unset. Used by the daemon's bounded `Draining` phase.
pub fn drain_timeout(&self) -> std::time::Duration {
std::time::Duration::from_secs(self.drain_timeout_secs.unwrap_or(2))
}
}
#[cfg(test)]
mod tests {
use super::*;
#[test]
fn test_config_default() {
let config = MmpConfig::default();
assert_eq!(config.mode, MmpMode::Full);
assert_eq!(config.log_interval_secs, 30);
assert_eq!(config.owd_window_size, 32);
}
#[test]
fn test_config_yaml_parse() {
let yaml = r#"
mode: lightweight
log_interval_secs: 60
owd_window_size: 48
"#;
let config: MmpConfig = serde_yaml::from_str(yaml).unwrap();
assert_eq!(config.mode, MmpMode::Lightweight);
assert_eq!(config.log_interval_secs, 60);
assert_eq!(config.owd_window_size, 48);
}
#[test]
fn test_config_yaml_partial() {
let yaml = "mode: minimal";
let config: MmpConfig = serde_yaml::from_str(yaml).unwrap();
assert_eq!(config.mode, MmpMode::Minimal);
assert_eq!(config.log_interval_secs, DEFAULT_LOG_INTERVAL_SECS);
assert_eq!(config.owd_window_size, DEFAULT_OWD_WINDOW_SIZE);
}
#[test]
fn test_drain_timeout_default_and_override() {
// Unset → the 2-second default.
let c = NodeConfig::default();
assert_eq!(c.drain_timeout_secs, None);
assert_eq!(c.drain_timeout(), std::time::Duration::from_secs(2));
// Explicit override is honored.
let c2 = NodeConfig {
drain_timeout_secs: Some(10),
..NodeConfig::default()
};
assert_eq!(c2.drain_timeout(), std::time::Duration::from_secs(10));
// A zero override is a valid (immediate) drain, not the default.
let c3 = NodeConfig {
drain_timeout_secs: Some(0),
..NodeConfig::default()
};
assert_eq!(c3.drain_timeout(), std::time::Duration::from_secs(0));
}
#[test]
fn test_ecn_config_defaults() {
let c = EcnConfig::default();
@@ -1123,27 +1269,27 @@ mod tests {
}
#[test]
fn test_nostr_discovery_startup_sweep_defaults() {
let c = NostrDiscoveryConfig::default();
fn test_nostr_rendezvous_startup_sweep_defaults() {
let c = NostrRendezvousConfig::default();
assert_eq!(c.startup_sweep_delay_secs, 5);
assert_eq!(c.startup_sweep_max_age_secs, 3_600);
}
#[test]
fn test_nostr_discovery_startup_sweep_yaml_override() {
fn test_nostr_rendezvous_startup_sweep_yaml_override() {
let yaml = "enabled: true\npolicy: open\nstartup_sweep_delay_secs: 10\nstartup_sweep_max_age_secs: 1800\n";
let c: NostrDiscoveryConfig = serde_yaml::from_str(yaml).unwrap();
let c: NostrRendezvousConfig = serde_yaml::from_str(yaml).unwrap();
assert!(c.enabled);
assert_eq!(c.policy, NostrDiscoveryPolicy::Open);
assert_eq!(c.policy, NostrRendezvousPolicy::Open);
assert_eq!(c.startup_sweep_delay_secs, 10);
assert_eq!(c.startup_sweep_max_age_secs, 1_800);
}
#[test]
fn test_nostr_discovery_startup_sweep_partial_yaml_uses_defaults() {
fn test_nostr_rendezvous_startup_sweep_partial_yaml_uses_defaults() {
// Only override delay; max_age should fall back to default.
let yaml = "enabled: true\nstartup_sweep_delay_secs: 30\n";
let c: NostrDiscoveryConfig = serde_yaml::from_str(yaml).unwrap();
let c: NostrRendezvousConfig = serde_yaml::from_str(yaml).unwrap();
assert_eq!(c.startup_sweep_delay_secs, 30);
assert_eq!(c.startup_sweep_max_age_secs, 3_600);
}
src/config/transport.rs
+25 -6
View File
@@ -282,9 +282,10 @@ pub struct EthernetConfig {
#[serde(default, skip_serializing_if = "Option::is_none")]
pub send_buf_size: Option<usize>,
/// Listen for discovery beacons from other nodes. Default: true.
#[serde(default, skip_serializing_if = "Option::is_none")]
pub discovery: Option<bool>,
/// Listen for neighbor beacons from other nodes. Default: true.
/// (Renamed from `discovery`; the old key is still accepted.)
#[serde(default, alias = "discovery", skip_serializing_if = "Option::is_none")]
pub listen: Option<bool>,
/// Broadcast announcement beacons on the LAN. Default: false.
#[serde(default, skip_serializing_if = "Option::is_none")]
@@ -319,9 +320,9 @@ impl EthernetConfig {
self.send_buf_size.unwrap_or(DEFAULT_ETHERNET_SEND_BUF)
}
/// Whether to listen for discovery beacons. Default: true.
pub fn discovery(&self) -> bool {
self.discovery.unwrap_or(true)
/// Whether to listen for neighbor beacons. Default: true.
pub fn listen(&self) -> bool {
self.listen.unwrap_or(true)
}
/// Whether to broadcast announcement beacons. Default: false.
@@ -1046,4 +1047,22 @@ mod tests {
assert_eq!(parse_bind_port("[::]:443"), Some(443));
assert_eq!(parse_bind_port("not-a-socket-addr"), None);
}
#[test]
fn ethernet_listen_accepts_legacy_discovery_alias_and_rejects_unknown() {
// (a) The legacy `discovery:` key is still accepted via serde alias.
let legacy: EthernetConfig =
serde_yaml::from_str("interface: eth0\ndiscovery: true\n").unwrap();
assert_eq!(legacy.listen, Some(true));
// (b) The new canonical `listen:` key parses into the renamed field.
let renamed: EthernetConfig =
serde_yaml::from_str("interface: eth0\nlisten: true\n").unwrap();
assert_eq!(renamed.listen, Some(true));
// (c) `deny_unknown_fields` still rejects an unknown ethernet key.
let bogus: Result<EthernetConfig, _> =
serde_yaml::from_str("interface: eth0\nbogus: true\n");
assert!(bogus.is_err());
}
}
src/control/queries.rs
+23 -14
View File
@@ -236,7 +236,7 @@ pub fn show_peers(node: &Node) -> Value {
// Per-npub Nostr-traversal failure-state snapshot, indexed by npub
// for O(1) per-peer lookup. Empty if Nostr discovery is disabled.
let nostr_state: std::collections::HashMap<String, _> = node
.nostr_discovery_handle()
.nostr_rendezvous_handle()
.map(|d| {
d.failure_state_snapshot()
.into_iter()
@@ -1303,17 +1303,18 @@ pub fn show_connections(node: &Node) -> Value {
let now = now_ms();
let connections: Vec<Value> = node
.connections()
.map(|conn| {
.map(|(_, machine)| {
let link_id = machine.link_id();
let mut conn_json = json!({
"link_id": conn.link_id().as_u64(),
"direction": format!("{}", conn.direction()),
"handshake_state": format!("{}", conn.handshake_state()),
"started_at_ms": conn.started_at(),
"idle_ms": now.saturating_sub(conn.last_activity()),
"resend_count": conn.resend_count(),
"link_id": link_id.as_u64(),
"direction": format!("{}", machine.conn_direction()),
"handshake_state": node.connection_handshake_state(link_id),
"started_at_ms": node.connection_started_at(link_id),
"idle_ms": now.saturating_sub(node.connection_last_activity(link_id)),
"resend_count": node.connection_resend_count(link_id),
});
if let Some(identity) = conn.expected_identity() {
if let Some(identity) = node.connection_expected_identity(link_id) {
conn_json["expected_peer"] = json!(identity.npub());
}
@@ -1487,7 +1488,9 @@ pub fn show_routing(node: &Node) -> Value {
"recent_requests": node.recent_request_count(),
"retries": retries,
"forwarding": serde_json::to_value(metrics.forwarding.snapshot()).unwrap_or_default(),
"discovery": serde_json::to_value(metrics.discovery.snapshot()).unwrap_or_default(),
// COMPAT: `discovery` is the deprecated alias for `lookup`; drop at the v2 cutover.
"discovery": serde_json::to_value(metrics.lookup.snapshot()).unwrap_or_default(),
"lookup": serde_json::to_value(metrics.lookup.snapshot()).unwrap_or_default(),
"error_signals": serde_json::to_value(metrics.errors.snapshot()).unwrap_or_default(),
"congestion": serde_json::to_value(metrics.congestion.snapshot()).unwrap_or_default(),
})
@@ -1543,7 +1546,9 @@ pub(crate) fn show_routing_from_handle(handle: &super::read_handle::ControlReadH
"recent_requests": view.recent_requests,
"retries": retries,
"forwarding": serde_json::to_value(metrics.forwarding.snapshot()).unwrap_or_default(),
"discovery": serde_json::to_value(metrics.discovery.snapshot()).unwrap_or_default(),
// COMPAT: `discovery` is the deprecated alias for `lookup`; drop at the v2 cutover.
"discovery": serde_json::to_value(metrics.lookup.snapshot()).unwrap_or_default(),
"lookup": serde_json::to_value(metrics.lookup.snapshot()).unwrap_or_default(),
"error_signals": serde_json::to_value(metrics.errors.snapshot()).unwrap_or_default(),
"congestion": serde_json::to_value(metrics.congestion.snapshot()).unwrap_or_default(),
})
@@ -2328,7 +2333,9 @@ pub(crate) fn show_metrics_from_handle(handle: &super::read_handle::ControlReadH
let m = handle.metrics();
json!({
"forwarding": m.forwarding.snapshot(),
"discovery": m.discovery.snapshot(),
// COMPAT: `discovery` is the deprecated alias for `lookup`; drop at the v2 cutover.
"discovery": m.lookup.snapshot(),
"lookup": m.lookup.snapshot(),
"tree": m.tree.snapshot(),
"bloom": m.bloom.snapshot(),
"congestion": m.congestion.snapshot(),
@@ -2761,12 +2768,12 @@ mod tests {
/// Structural confirmation that the rx_loop no longer dispatches `show_*`:
/// the rx_loop source carries no `queries::dispatch` call and no
/// `starts_with("show_")` routing branch. Reads the committed source of
/// `src/node/handlers/rx_loop.rs` and asserts both markers are absent. This
/// `src/node/dataplane/rx_loop.rs` and asserts both markers are absent. This
/// is the milestone's "remove `show_*` from the data-plane dispatch path"
/// invariant, guarded against regression.
#[test]
fn rx_loop_has_no_show_dispatch() {
let src = include_str!("../node/handlers/rx_loop.rs");
let src = include_str!("../node/dataplane/rx_loop.rs");
assert!(
!src.contains("queries::dispatch"),
"rx_loop must not call queries::dispatch (show_* served off-loop)"
@@ -2794,7 +2801,9 @@ mod tests {
let expected_families = [
("forwarding", "received_packets"),
// `discovery` is the deprecated dual-emit alias for `lookup`; drop at the v2 cutover.
("discovery", "req_received"),
("lookup", "req_received"),
("tree", "accepted"),
("bloom", "accepted"),
("congestion", "ce_forwarded"),
src/control/read_handle.rs
+31
View File
@@ -118,10 +118,41 @@ impl ControlReadHandle {
/// Cutover queries (R1) read only `NodeContext` / `MetricsRegistry` (the state
/// the read handle already bundles) plus host-OS facts (`/proc`, nftables), so
/// they render entirely in the control task without touching `Node`.
///
/// **It now also carries mutating commands**, namely the `profile_tick_*`
/// family under the `profiling` feature. They are served here rather than on
/// the rx_loop deliberately: all of their state is process statics, they need
/// no `&mut Node`, and routing them through the loop would make the toggle
/// queue behind the very behavior it exists to measure.
pub(crate) fn snapshot_dispatch(request: &Request, handle: &ControlReadHandle) -> Option<Response> {
use crate::control::queries;
match request.command.as_str() {
// Tick-body profiler toggle. Present only in a `--features profiling`
// build; otherwise these fall through to the rx_loop dispatch, which
// reports them as unknown commands.
#[cfg(feature = "profiling")]
"profile_tick_on" => {
let dir = request
.params
.as_ref()
.and_then(|p| p.get("dir"))
.and_then(|v| v.as_str());
let context = handle.context();
let npub = context.identity.npub();
let period = context.config.node.tick_interval_secs;
Some(match crate::instr::capture::start(dir, &npub, period) {
Ok(value) => Response::ok(value),
Err(e) => Response::error(e),
})
}
#[cfg(feature = "profiling")]
"profile_tick_off" => Some(match crate::instr::capture::stop() {
Ok(value) => Response::ok(value),
Err(e) => Response::error(e),
}),
#[cfg(feature = "profiling")]
"profile_tick_status" => Some(Response::ok(crate::instr::capture::status())),
"show_listening_sockets" => Some(Response::ok(
queries::show_listening_sockets_from_handle(handle),
)),
src/control/snapshots/show_routing.json
+24
View File
@@ -63,6 +63,30 @@
"ttl_exhausted_packets": 0
},
"identity_cache_entries": 0,
"lookup": {
"req_backoff_suppressed": 0,
"req_bloom_miss": 0,
"req_decode_error": 0,
"req_dedup_cache_full": 0,
"req_deduplicated": 0,
"req_duplicate": 0,
"req_fallback_forwarded": 0,
"req_forward_rate_limited": 0,
"req_forwarded": 0,
"req_initiated": 0,
"req_no_tree_peer": 0,
"req_received": 0,
"req_target_is_us": 0,
"req_ttl_exhausted": 0,
"resp_accepted": 0,
"resp_decode_error": 0,
"resp_forwarded": 0,
"resp_identity_miss": 0,
"resp_no_route": 0,
"resp_proof_failed": 0,
"resp_received": 0,
"resp_timed_out": 0
},
"pending_lookups": [],
"pending_tun_destinations": 0,
"pending_tun_packets": 0,
src/instr/capture.rs
+418
View File
@@ -0,0 +1,418 @@
//! Capture lifecycle: the arm/disarm state machine, the sink file, and the
//! `fipsctl`-facing operations.
//!
//! The toggle — not the writer — creates and opens the sink and publishes its
//! path, so an unwritable directory fails the `on` command loudly instead of
//! being discovered later by a background thread with nobody to report to.
//!
//! Capture state is a single atomic state machine (`Idle`, `Running`,
//! `StoppedByCap`) transitioned by `compare_exchange`. Every accepted control
//! connection is served by its own spawned task, so two simultaneous `on`
//! requests are genuinely concurrent and must not both create a writer.
use std::fs::File;
use std::io::Write;
use std::path::PathBuf;
use std::sync::Mutex;
use std::sync::atomic::{AtomicBool, AtomicU8, AtomicU64, Ordering};
use std::time::{Duration, SystemTime, UNIX_EPOCH};
use super::recorder;
use super::writer;
/// Default sink directory. Overridable per capture with `--dir`.
pub(crate) const DEFAULT_DIR: &str = "/var/log/fips";
/// Writer flush interval.
pub(crate) const INTERVAL: Duration = Duration::from_secs(10);
/// Size at which a capture stops itself. Reaching it stops the capture rather
/// than rotating: the point of a capture is a bounded, self-describing window.
pub(crate) const BYTE_CAP: u64 = 32 * 1024 * 1024;
pub(crate) const IDLE: u8 = 0;
pub(crate) const RUNNING: u8 = 1;
pub(crate) const STOPPED_BY_CAP: u8 = 2;
/// The writer could not write and stopped itself. Distinct from a cap stop:
/// a capture that died on a full disk produced a truncated window, and calling
/// that "stopped_by_cap" tells the operator it ran to its limit when it did
/// not. The trailer line explaining it goes to the same failing file, so the
/// state is the only signal that survives.
pub(crate) const STOPPED_BY_ERROR: u8 = 3;
static STATE: AtomicU8 = AtomicU8::new(IDLE);
static GATE: AtomicBool = AtomicBool::new(false);
static BYTES: AtomicU64 = AtomicU64::new(0);
static ACTIVE_PATH: Mutex<Option<PathBuf>> = Mutex::new(None);
static WRITER: Mutex<Option<writer::Handle>> = Mutex::new(None);
/// The per-tick gate. One relaxed load per tick when the feature is compiled in
/// and no capture is running.
#[inline]
pub(crate) fn gate() -> bool {
GATE.load(Ordering::Relaxed)
}
pub(crate) fn bytes_written() -> u64 {
BYTES.load(Ordering::Relaxed)
}
pub(crate) fn add_bytes(n: u64) -> u64 {
BYTES.fetch_add(n, Ordering::Relaxed) + n
}
fn active_path() -> Option<PathBuf> {
ACTIVE_PATH
.lock()
.unwrap_or_else(|e| e.into_inner())
.clone()
}
fn path_display() -> String {
active_path()
.map(|p| p.display().to_string())
.unwrap_or_else(|| "<none>".to_string())
}
fn state_name(state: u8) -> &'static str {
match state {
RUNNING => "running",
STOPPED_BY_CAP => "stopped_by_cap",
STOPPED_BY_ERROR => "stopped_by_error",
_ => "idle",
}
}
/// Called by the writer when it stops itself. `terminal` is `STOPPED_BY_CAP`
/// or `STOPPED_BY_ERROR`. Returns true if this call is the one that stopped it.
pub(crate) fn mark_stopped(terminal: u8) -> bool {
debug_assert!(terminal == STOPPED_BY_CAP || terminal == STOPPED_BY_ERROR);
GATE.store(false, Ordering::Relaxed);
STATE
.compare_exchange(RUNNING, terminal, Ordering::AcqRel, Ordering::Acquire)
.is_ok()
}
/// Join the writer thread, if one exists. Never called while holding another
/// lock the writer might want.
fn reap() {
let handle = WRITER.lock().unwrap_or_else(|e| e.into_inner()).take();
if let Some(handle) = handle {
handle.stop_and_join();
}
}
/// Arm a capture.
///
/// Opens the sink first and only then starts the writer, so a bad `--dir` is
/// reported to the caller rather than logged into the void.
pub(crate) fn start(
dir: Option<&str>,
node_npub: &str,
tick_period_secs: u64,
) -> Result<serde_json::Value, String> {
claim()?;
match open_sink(dir, node_npub, tick_period_secs) {
Ok((file, path, header_len)) => {
recorder::reset();
BYTES.store(header_len, Ordering::Relaxed);
match writer::spawn(file) {
Ok(handle) => {
*WRITER.lock().unwrap_or_else(|e| e.into_inner()) = Some(handle);
*ACTIVE_PATH.lock().unwrap_or_else(|e| e.into_inner()) = Some(path.clone());
GATE.store(true, Ordering::Release);
Ok(serde_json::json!({
"state": "running",
"path": path.display().to_string(),
"interval_secs": INTERVAL.as_secs(),
"byte_cap": BYTE_CAP,
}))
}
Err(e) => {
let _ = std::fs::remove_file(&path);
BYTES.store(0, Ordering::Relaxed);
STATE.store(IDLE, Ordering::Release);
Err(format!("cannot start profile writer thread: {e}"))
}
}
}
Err(e) => {
BYTES.store(0, Ordering::Relaxed);
STATE.store(IDLE, Ordering::Release);
Err(e)
}
}
}
/// Take the capture slot, reaping a cap-stopped predecessor if that is what is
/// in the way.
fn claim() -> Result<(), String> {
match STATE.compare_exchange(IDLE, RUNNING, Ordering::AcqRel, Ordering::Acquire) {
Ok(_) => Ok(()),
Err(RUNNING) => Err(format!("capture already running: {}", path_display())),
Err(stopped @ (STOPPED_BY_CAP | STOPPED_BY_ERROR)) => {
reap();
*ACTIVE_PATH.lock().unwrap_or_else(|e| e.into_inner()) = None;
STATE
.compare_exchange(stopped, RUNNING, Ordering::AcqRel, Ordering::Acquire)
.map(|_| ())
.map_err(|_| "capture state changed concurrently; retry".to_string())
}
Err(_) => Err("capture in an unexpected state".to_string()),
}
}
/// Disarm the capture. Succeeds when nothing is running, reporting so.
pub(crate) fn stop() -> Result<serde_json::Value, String> {
let previous = STATE.load(Ordering::Acquire);
if previous == IDLE {
return Ok(serde_json::json!({"state": "idle", "stopped": false}));
}
GATE.store(false, Ordering::Release);
// The writer wakes on the stop message rather than after the interval, so
// this join returns promptly instead of parking the caller for up to one
// flush interval.
reap();
let path = path_display();
*ACTIVE_PATH.lock().unwrap_or_else(|e| e.into_inner()) = None;
let bytes = bytes_written();
// Clear the counter with the slot: a later `status` while idle must not
// report the previous capture's byte total as though a capture were live.
BYTES.store(0, Ordering::Relaxed);
STATE.store(IDLE, Ordering::Release);
Ok(serde_json::json!({
"state": "idle",
"stopped": true,
"stopped_by_cap": previous == STOPPED_BY_CAP,
"stopped_by_error": previous == STOPPED_BY_ERROR,
"path": path,
"bytes": bytes,
}))
}
/// Report capture state. Distinguishes all four states.
pub(crate) fn status() -> serde_json::Value {
let state = STATE.load(Ordering::Acquire);
serde_json::json!({
"state": state_name(state),
"path": active_path().map(|p| p.display().to_string()),
"bytes": bytes_written(),
"byte_cap": BYTE_CAP,
"interval_secs": INTERVAL.as_secs(),
})
}
/// Stop and reap at daemon teardown. Idempotent.
pub(crate) fn shutdown() {
if STATE.load(Ordering::Acquire) != IDLE {
let _ = stop();
}
}
/// Create the sink file and write its header block. Returns the open file, its
/// path, and the number of header bytes written.
fn open_sink(
dir: Option<&str>,
node_npub: &str,
tick_period_secs: u64,
) -> Result<(File, PathBuf, u64), String> {
let dir = PathBuf::from(dir.unwrap_or(DEFAULT_DIR));
std::fs::create_dir_all(&dir)
.map_err(|e| format!("cannot use profile directory {}: {e}", dir.display()))?;
let start_unix = SystemTime::now()
.duration_since(UNIX_EPOCH)
.map(|d| d.as_secs())
.unwrap_or(0);
let path = dir.join(format!("profile-{}.tsv", compact_utc(start_unix)));
let mut file = File::create(&path)
.map_err(|e| format!("cannot create profile file {}: {e}", path.display()))?;
let header = format!(
"# fips tick profile\n\
# node\t{node}\n\
# build\t{build}\n\
# platform\t{platform}\n\
# tick_period_secs\t{period}\n\
# interval_secs\t{interval}\n\
# byte_cap\t{cap}\n\
# start_utc\t{start_utc}\n\
# start_unix\t{start_unix}\n\
# NOTE\tstep durations are WALL CLOCK across await points, not CPU time:\n\
# NOTE\ta step that awaits I/O accrues the wait, and other tasks may run\n\
# NOTE\tinside that span. That is the intended measure for head-of-line\n\
# NOTE\tdelay; do not read a large step as CPU cost.\n\
# NOTE\tarm_starvation is measured directly as (entry time - the deadline\n\
# NOTE\tthe interval scheduled the tick for). It is NOT derived from\n\
# NOTE\ttick_entry_gap, which carries no starvation signal by itself:\n\
# NOTE\tunder a steady delay every gap is exactly one tick period.\n\
ts_unix\tkind\tdomain\tname\tcount\tmax\ttotal\tunit\n",
node = node_npub,
build = crate::version::short_version(),
platform = std::env::consts::OS,
period = tick_period_secs,
interval = INTERVAL.as_secs(),
cap = BYTE_CAP,
start_utc = iso_utc(start_unix),
start_unix = start_unix,
);
file.write_all(header.as_bytes())
.map_err(|e| format!("cannot write profile header to {}: {e}", path.display()))?;
Ok((file, path, header.len() as u64))
}
/// Break a Unix timestamp into UTC `(year, month, day, hour, minute, second)`.
///
/// Hinnant's `civil_from_days`, era-based. No date crate is in the dependency
/// set and one filename stamp does not justify adding one.
fn utc_parts(unix: u64) -> (i64, u32, u32, u32, u32, u32) {
let days = (unix / 86_400) as i64;
let secs = unix % 86_400;
let z = days + 719_468;
let era = z.div_euclid(146_097);
let doe = z.rem_euclid(146_097);
let yoe = (doe - doe / 1_460 + doe / 36_524 - doe / 146_096) / 365;
let y = yoe + era * 400;
let doy = doe - (365 * yoe + yoe / 4 - yoe / 100);
let mp = (5 * doy + 2) / 153;
let d = (doy - (153 * mp + 2) / 5 + 1) as u32;
let m = (if mp < 10 { mp + 3 } else { mp - 9 }) as u32;
let y = if m <= 2 { y + 1 } else { y };
(
y,
m,
d,
(secs / 3_600) as u32,
((secs % 3_600) / 60) as u32,
(secs % 60) as u32,
)
}
/// `20260727T191500Z` — filename-safe.
fn compact_utc(unix: u64) -> String {
let (y, mo, d, h, mi, s) = utc_parts(unix);
format!("{y:04}{mo:02}{d:02}T{h:02}{mi:02}{s:02}Z")
}
/// `2026-07-27T19:15:00Z` — for the header block.
fn iso_utc(unix: u64) -> String {
let (y, mo, d, h, mi, s) = utc_parts(unix);
format!("{y:04}-{mo:02}-{d:02}T{h:02}:{mi:02}:{s:02}Z")
}
#[cfg(test)]
mod tests {
use super::*;
#[test]
fn utc_parts_matches_known_instants() {
assert_eq!(utc_parts(0), (1970, 1, 1, 0, 0, 0));
assert_eq!(utc_parts(946_684_800), (2000, 1, 1, 0, 0, 0));
// 2026-07-27T19:15:00Z
assert_eq!(utc_parts(1_785_179_700), (2026, 7, 27, 19, 15, 0));
// Leap day.
assert_eq!(utc_parts(1_709_164_800), (2024, 2, 29, 0, 0, 0));
}
#[test]
fn stamps_render_expected_shapes() {
assert_eq!(compact_utc(1_785_179_700), "20260727T191500Z");
assert_eq!(iso_utc(1_785_179_700), "2026-07-27T19:15:00Z");
}
// The lock these tests take is shared with the recorder tests, which
// mutate the same statics. See `crate::instr::test_serial`.
#[test]
fn capture_round_trip_writes_header_and_rows() {
let _guard = crate::instr::test_serial();
let dir = tempfile::tempdir().expect("tempdir");
let dir_str = dir.path().to_str().unwrap().to_string();
let started = start(Some(&dir_str), "npub1test", 1).expect("start");
assert_eq!(started["state"], "running");
assert!(gate(), "gate must be armed while running");
let path = PathBuf::from(started["path"].as_str().unwrap());
// A second `on` is refused while one is running, and names the file.
let refused = start(Some(&dir_str), "npub1test", 1).unwrap_err();
assert!(refused.contains(&path.display().to_string()), "{refused}");
// Feed one observation so the drained rows are not all zero.
recorder::record(
recorder::Domain::Tick,
recorder::Step::WholeTick,
Duration::from_millis(7),
);
// Stopping wakes the writer immediately; it drains once more and joins.
let stopped = stop().expect("stop");
assert_eq!(stopped["stopped"], true);
assert_eq!(stopped["stopped_by_cap"], false);
assert!(!gate(), "gate must be clear after stop");
let text = std::fs::read_to_string(&path).expect("read capture");
assert!(text.starts_with("# fips tick profile\n"), "{text}");
assert!(text.contains("# node\tnpub1test\n"), "{text}");
assert!(
text.contains("ts_unix\tkind\tdomain\tname\tcount\tmax\ttotal\tunit\n"),
"{text}"
);
// The final drain emitted one row per emitted step, plus the gauges.
let rows: Vec<&str> = text
.lines()
.filter(|l| l.starts_with(|c: char| c.is_ascii_digit()))
.collect();
let expected_steps = recorder::STEPS.iter().filter(|s| s.emitted()).count();
assert_eq!(rows.len(), expected_steps + recorder::N_GAUGES);
// The 7 ms observation above is in the whole-tick row, converted to
// microseconds. Bounds rather than equality: the gate is process-wide,
// so a node under test elsewhere in this binary may have ticked into
// the same capture window.
let whole_tick = rows
.iter()
.find(|r| r.contains("\tstep\ttick\twhole_tick\t"))
.expect("whole_tick row");
let fields: Vec<&str> = whole_tick.split('\t').collect();
assert_eq!(fields.last(), Some(&"us"), "{whole_tick}");
assert!(
fields[4].parse::<u64>().unwrap() >= 1,
"count: {whole_tick}"
);
assert!(
fields[5].parse::<u64>().unwrap() >= 7_000,
"max: {whole_tick}"
);
assert!(
rows.iter()
.any(|r| r.contains("\tgauge\ttick\tarm_starvation\t")),
"{text}"
);
// A stop with nothing running is not an error.
let again = stop().expect("second stop");
assert_eq!(again["stopped"], false);
}
#[test]
fn start_fails_loudly_on_an_unwritable_directory() {
let _guard = crate::instr::test_serial();
let err = start(Some("/proc/fips-profile-should-not-exist"), "npub1test", 1)
.expect_err("must fail");
assert!(err.contains("profile directory"), "{err}");
// The failed attempt must leave the slot free for the next try.
assert_eq!(STATE.load(Ordering::Acquire), IDLE);
assert!(!gate());
}
#[test]
fn status_reports_the_bounds_it_is_enforcing() {
let _guard = crate::instr::test_serial();
let value = status();
assert_eq!(value["byte_cap"], BYTE_CAP);
assert_eq!(value["interval_secs"], INTERVAL.as_secs());
}
}
src/instr/mod.rs
+171
View File
@@ -0,0 +1,171 @@
//! Tick-body instrumentation.
//!
//! A purpose-built, feature-gated profiler for the rx-loop tick arm. It exists
//! to answer one question with field data: which subsystem step dominates the
//! tick body, and how long does the tick arm wait behind the other `select!`
//! arms before it runs at all.
//!
//! # Shape
//!
//! - Everything that costs anything at runtime is behind the `profiling` Cargo
//! feature, which is **off by default**. The default build's neutrality is a
//! property of the generated code, not of a runtime check.
//! - The instrumentation macro is defined twice, once per feature state. The
//! feature-off definition is a pure pass-through: it expands to the measured
//! expression and nothing else, so no timing code exists in a default build.
//! - The always-present surface — [`gate`], [`tick_entry`], [`tick_gauges`],
//! [`shutdown`] — exists in both feature states because the call sites in
//! `rx_loop.rs` and the lifecycle teardown must compile either way. Their
//! feature-off forms are empty (and [`gate`] is a `const fn` returning
//! `false`), so they cost nothing.
//! - The module is named `instr` rather than `profiling` so that it sorts
//! before `node` in `lib.rs`'s alphabetical module list: a `#[macro_use]`
//! module must be declared before the modules that use its macros.
//!
//! # Data model
//!
//! Domain above step: [`Domain`] carries exactly one variant today
//! (`Domain::Tick`). The primitive, the recorder, the writer and the `fipsctl`
//! surface all take a domain, so adding a data-path domain later is additive.
//! No second domain is declared until something records into it.
//!
//! Per (domain, step) the recorder keeps an exact count, max and total in fixed
//! static `AtomicU64` arrays — no histogram, no accumulation, a fixed footprint
//! regardless of run length. Gauges (ticks per interval, peer count, and the
//! arm-starvation figures) live in a parallel array and are emitted with an
//! explicit row kind so a gauge value never lands under a duration column.
#[cfg(feature = "profiling")]
pub(crate) mod capture;
#[cfg(feature = "profiling")]
mod recorder;
#[cfg(feature = "profiling")]
mod writer;
#[cfg(feature = "profiling")]
pub(crate) use recorder::{Domain, Step, now, record};
// ---------------------------------------------------------------------------
// The macro pair.
//
// Every path in the body is `$crate::`-qualified. `macro_rules!` bodies are not
// path-hygienic: an unqualified `Instant::now()` or `record(..)` would resolve
// at the *call site* (`rx_loop.rs`), where neither name is in scope. Importing
// them there is worse still, because the imports would be unused in the
// feature-off build and red it under `-D warnings`.
//
// `$e` is evaluated exactly once in both forms, which is what makes nesting the
// whole-tick span around the per-step spans safe.
// ---------------------------------------------------------------------------
/// Time `$e` as one step of `$domain`, when `$on` is true.
///
/// `$on` is the per-tick gate hoist: the enable flag is read once at the top of
/// the tick arm into a local, and that local is passed explicitly to every
/// invocation, because macro hygiene makes a call-site local invisible inside
/// the macro body.
#[cfg(feature = "profiling")]
macro_rules! instr_step {
($on:expr, $domain:expr, $step:expr, $e:expr) => {{
let t0 = if $on {
Some($crate::instr::now())
} else {
None
};
let r = $e;
if let Some(t) = t0 {
$crate::instr::record($domain, $step, t.elapsed());
}
r
}};
}
/// Feature-off form: a pure pass-through. The expansion contains no clock read,
/// no counter update and no reference to the recorder — only the measured
/// expression, plus a discard of the gate local so it is not unused.
#[cfg(not(feature = "profiling"))]
macro_rules! instr_step {
($on:expr, $domain:expr, $step:expr, $e:expr) => {{
let _ = &$on;
$e
}};
}
// ---------------------------------------------------------------------------
// Always-present surface.
// ---------------------------------------------------------------------------
/// Whether a capture is armed. Read **once per tick** into a local that is then
/// passed to each `instr_step!` invocation, so the feature-on-but-idle cost of
/// the whole tick arm is a single relaxed load.
#[cfg(feature = "profiling")]
#[inline]
pub(crate) fn gate() -> bool {
capture::gate()
}
/// Feature-off gate: a `const fn` returning `false`, so the whole tick arm
/// folds to the uninstrumented sequence at compile time.
#[cfg(not(feature = "profiling"))]
#[inline]
pub(crate) const fn gate() -> bool {
false
}
/// Record how late this tick-arm entry is against its scheduled deadline.
///
/// The arm is polled **last** under `biased;`, so its lateness is the time it
/// spent waiting behind the packet, TUN and control arms. `tokio::time::
/// interval::tick` returns the deadline it was scheduled for, so this is a
/// direct subtraction rather than a model. Two earlier designs derived it from
/// the inter-entry gap instead and both under-reported: one by the previous
/// body, the other by reporting only the first difference of the delay, so a
/// sustained stall read as zero. The inter-entry gap is still recorded as its
/// own gauge, but it carries no starvation signal on its own.
#[cfg(feature = "profiling")]
#[inline]
pub(crate) fn tick_entry(on: bool, deadline: std::time::Instant, now: std::time::Instant) {
recorder::tick_entry(on, deadline, now);
}
#[cfg(not(feature = "profiling"))]
#[inline]
pub(crate) fn tick_entry(_on: bool, _deadline: std::time::Instant, _now: std::time::Instant) {}
/// Sample the per-tick gauges taken from node state.
#[cfg(feature = "profiling")]
#[inline]
pub(crate) fn tick_gauges(on: bool, peers: u64) {
recorder::tick_gauges(on, peers);
}
#[cfg(not(feature = "profiling"))]
#[inline]
pub(crate) fn tick_gauges(_on: bool, _peers: u64) {}
/// Stop and reap any running capture at daemon teardown. Idempotent.
#[cfg(feature = "profiling")]
pub(crate) fn shutdown() {
capture::shutdown();
}
#[cfg(not(feature = "profiling"))]
pub(crate) fn shutdown() {}
/// One serialization lock for every test in this module tree.
///
/// The recorder counters and the capture state machine are the *same* process
/// statics: `capture::start` calls `recorder::reset`, and `capture::stop`
/// drains every slot. Two suites with their own locks therefore do not
/// serialize against each other, and the feature-on stage runs tests as
/// threads in one process, so a capture round-trip can zero the counters a
/// recorder test is mid-way through asserting on. One lock for both.
#[cfg(all(test, feature = "profiling"))]
pub(crate) static TEST_SERIAL: std::sync::Mutex<()> = std::sync::Mutex::new(());
/// Take the shared test lock, recovering from a poisoned mutex so one failing
/// test does not cascade into every other one.
#[cfg(all(test, feature = "profiling"))]
pub(crate) fn test_serial() -> std::sync::MutexGuard<'static, ()> {
TEST_SERIAL.lock().unwrap_or_else(|e| e.into_inner())
}
src/instr/recorder.rs
+478
View File
@@ -0,0 +1,478 @@
//! Fixed-footprint recorder: exact count / max / total per (domain, step).
//!
//! All state is process statics, not `Node` state, because the `fipsctl`
//! handler that arms and disarms a capture runs in the control accept task and
//! has no `&Node` — that is the whole point of serving it off-loop, so it
//! cannot queue behind the behavior it is measuring.
//!
//! The writer thread is the only reader. It takes each interval's figures with
//! `swap(0)`, so there are no "previous value" arrays to carry and the counters
//! are per-interval by construction.
use std::sync::LazyLock;
use std::sync::atomic::{AtomicU64, Ordering::Relaxed};
use std::time::{Duration, Instant};
/// Measurement domain. Structural only: one variant today.
///
/// A data-path domain is deliberately **not** declared until something records
/// into it. What generalizes here is the enum, the counter table and the
/// writer; the per-tick gate hoist does not, so a data-path domain will need
/// its own gate strategy.
#[derive(Copy, Clone, Debug, PartialEq, Eq)]
#[repr(usize)]
pub(crate) enum Domain {
Tick = 0,
}
pub(crate) const N_DOMAINS: usize = 1;
pub(crate) const DOMAINS: [Domain; N_DOMAINS] = [Domain::Tick];
impl Domain {
pub(crate) const fn name(self) -> &'static str {
match self {
Domain::Tick => "tick",
}
}
}
/// One measured step of the rx-loop tick arm, in call order, plus the
/// whole-body span.
///
/// `as usize` indexes the counter arrays, so the discriminants are dense and
/// `WholeTick` is last (it defines `N_STEPS`). Variants are declared
/// unconditionally — see [`Step::emitted`] for how the two platform- and
/// profile-conditional steps are kept out of the emitted table.
#[derive(Copy, Clone, Debug, PartialEq, Eq)]
#[repr(usize)]
pub(crate) enum Step {
CheckTimeouts = 0,
ReloadPeerAcl,
ReloadHostMap,
PollPendingConnects,
PollNostrRendezvous,
PollLanRendezvous,
DrivePeerTimers,
ResendPendingRekeys,
ResendPendingSessionHandshakes,
ResendPendingSessionMsg3,
PurgeIdleSessions,
ProcessPendingRetries,
CheckTreeState,
CheckBloomState,
ComputeMeshSize,
RecordStatsHistory,
CheckMmpReports,
CheckSessionMmpReports,
CheckLinkHeartbeats,
CheckRekey,
CheckSessionRekey,
CheckPendingLookups,
PollTransportDiscovery,
SampleTransportCongestion,
ActivateConnectedUdpSessions,
DebugAssertPeerMapsCoherent,
/// The whole tick-arm body, from before `check_timeouts` to after the last
/// step. Composes safely with the per-step spans because the macro
/// evaluates its measured expression exactly once.
WholeTick,
}
pub(crate) const N_STEPS: usize = Step::WholeTick as usize + 1;
/// Every step, in emission order. Index `i` of this table is `STEPS[i] as
/// usize`; `steps_table_is_dense` asserts it.
pub(crate) const STEPS: [Step; N_STEPS] = [
Step::CheckTimeouts,
Step::ReloadPeerAcl,
Step::ReloadHostMap,
Step::PollPendingConnects,
Step::PollNostrRendezvous,
Step::PollLanRendezvous,
Step::DrivePeerTimers,
Step::ResendPendingRekeys,
Step::ResendPendingSessionHandshakes,
Step::ResendPendingSessionMsg3,
Step::PurgeIdleSessions,
Step::ProcessPendingRetries,
Step::CheckTreeState,
Step::CheckBloomState,
Step::ComputeMeshSize,
Step::RecordStatsHistory,
Step::CheckMmpReports,
Step::CheckSessionMmpReports,
Step::CheckLinkHeartbeats,
Step::CheckRekey,
Step::CheckSessionRekey,
Step::CheckPendingLookups,
Step::PollTransportDiscovery,
Step::SampleTransportCongestion,
Step::ActivateConnectedUdpSessions,
Step::DebugAssertPeerMapsCoherent,
Step::WholeTick,
];
impl Step {
pub(crate) const fn name(self) -> &'static str {
match self {
Step::CheckTimeouts => "check_timeouts",
Step::ReloadPeerAcl => "reload_peer_acl",
Step::ReloadHostMap => "reload_host_map",
Step::PollPendingConnects => "poll_pending_connects",
Step::PollNostrRendezvous => "poll_nostr_rendezvous",
Step::PollLanRendezvous => "poll_lan_rendezvous",
Step::DrivePeerTimers => "drive_peer_timers",
Step::ResendPendingRekeys => "resend_pending_rekeys",
Step::ResendPendingSessionHandshakes => "resend_pending_session_handshakes",
Step::ResendPendingSessionMsg3 => "resend_pending_session_msg3",
Step::PurgeIdleSessions => "purge_idle_sessions",
Step::ProcessPendingRetries => "process_pending_retries",
Step::CheckTreeState => "check_tree_state",
Step::CheckBloomState => "check_bloom_state",
Step::ComputeMeshSize => "compute_mesh_size",
Step::RecordStatsHistory => "record_stats_history",
Step::CheckMmpReports => "check_mmp_reports",
Step::CheckSessionMmpReports => "check_session_mmp_reports",
Step::CheckLinkHeartbeats => "check_link_heartbeats",
Step::CheckRekey => "check_rekey",
Step::CheckSessionRekey => "check_session_rekey",
Step::CheckPendingLookups => "check_pending_lookups",
Step::PollTransportDiscovery => "poll_transport_discovery",
Step::SampleTransportCongestion => "sample_transport_congestion",
Step::ActivateConnectedUdpSessions => "activate_connected_udp_sessions",
Step::DebugAssertPeerMapsCoherent => "debug_assert_peer_maps_coherent",
Step::WholeTick => "whole_tick",
}
}
/// Whether this step gets a row in this build.
///
/// Two steps are conditionally compiled at their call sites. Emitting a row
/// for them in a build where the call site does not exist would publish a
/// count that is structurally zero forever, which reads as "this step never
/// runs" rather than "this step is not in this build". The predicates below
/// are the same `cfg` expressions that gate the call sites in
/// `node::dataplane::rx_loop`; keep them in step.
pub(crate) const fn emitted(self) -> bool {
match self {
Step::ActivateConnectedUdpSessions => {
cfg!(any(target_os = "linux", target_os = "macos"))
}
Step::DebugAssertPeerMapsCoherent => cfg!(debug_assertions),
_ => true,
}
}
}
/// A scalar sampled once per tick, as opposed to a duration.
///
/// Gauges carry their own row kind and their own unit in the output so a gauge
/// value can never be read as a duration.
#[derive(Copy, Clone, Debug, PartialEq, Eq)]
#[repr(usize)]
pub(crate) enum Gauge {
Ticks = 0,
Peers,
TickGap,
ArmStarvation,
}
pub(crate) const N_GAUGES: usize = Gauge::ArmStarvation as usize + 1;
pub(crate) const GAUGES: [Gauge; N_GAUGES] = [
Gauge::Ticks,
Gauge::Peers,
Gauge::TickGap,
Gauge::ArmStarvation,
];
impl Gauge {
pub(crate) const fn name(self) -> &'static str {
match self {
Gauge::Ticks => "ticks",
Gauge::Peers => "peers",
Gauge::TickGap => "tick_entry_gap",
Gauge::ArmStarvation => "arm_starvation",
}
}
/// Unit of the `max` and `total` columns for this gauge.
pub(crate) const fn unit(self) -> &'static str {
match self {
Gauge::Ticks => "ticks",
Gauge::Peers => "peers",
Gauge::TickGap | Gauge::ArmStarvation => "us",
}
}
/// Whether the gauge's stored values are nanosecond durations that the
/// writer converts to microseconds.
pub(crate) const fn is_duration(self) -> bool {
matches!(self, Gauge::TickGap | Gauge::ArmStarvation)
}
}
const N_SLOTS: usize = N_DOMAINS * N_STEPS;
static COUNT: [AtomicU64; N_SLOTS] = [const { AtomicU64::new(0) }; N_SLOTS];
static MAX_NS: [AtomicU64; N_SLOTS] = [const { AtomicU64::new(0) }; N_SLOTS];
static TOTAL_NS: [AtomicU64; N_SLOTS] = [const { AtomicU64::new(0) }; N_SLOTS];
static G_COUNT: [AtomicU64; N_GAUGES] = [const { AtomicU64::new(0) }; N_GAUGES];
static G_MAX: [AtomicU64; N_GAUGES] = [const { AtomicU64::new(0) }; N_GAUGES];
static G_TOTAL: [AtomicU64; N_GAUGES] = [const { AtomicU64::new(0) }; N_GAUGES];
/// Monotonic baseline so tick-arm entry times fit in an atomic. Offset by one
/// on store so that zero can mean "no previous entry".
static BASE: LazyLock<Instant> = LazyLock::new(Instant::now);
static PREV_ENTRY_NS: AtomicU64 = AtomicU64::new(0);
#[inline]
const fn slot(domain: Domain, step: Step) -> usize {
(domain as usize * N_STEPS) + step as usize
}
/// Read the clock for a step span.
#[inline]
pub(crate) fn now() -> Instant {
Instant::now()
}
/// Record one observation of `step`.
#[inline]
pub(crate) fn record(domain: Domain, step: Step, elapsed: Duration) {
let ns = elapsed.as_nanos() as u64;
let idx = slot(domain, step);
COUNT[idx].fetch_add(1, Relaxed);
TOTAL_NS[idx].fetch_add(ns, Relaxed);
MAX_NS[idx].fetch_max(ns, Relaxed);
}
#[inline]
fn record_gauge(gauge: Gauge, value: u64) {
let idx = gauge as usize;
G_COUNT[idx].fetch_add(1, Relaxed);
G_TOTAL[idx].fetch_add(value, Relaxed);
G_MAX[idx].fetch_max(value, Relaxed);
}
/// Sample the inter-entry gap and the measured arm-starvation delay.
pub(crate) fn tick_entry(on: bool, deadline: Instant, now: Instant) {
if !on {
return;
}
// Offset by one so that a stored zero unambiguously means "no previous
// entry", even for an entry that lands on the baseline instant.
let stamp = BASE.elapsed().as_nanos() as u64 + 1;
let late = now.saturating_duration_since(deadline).as_nanos() as u64;
tick_entry_at(stamp, late);
}
/// The clock-free half of [`tick_entry`]: both times are inputs, so the
/// arithmetic can be driven with synthetic stamps in a test.
///
/// `late_ns` is how far past its scheduled deadline this entry was, measured
/// directly rather than derived. Two earlier designs derived it from the
/// inter-entry gap and were both wrong: subtracting the previous body
/// understated it by exactly the body, and subtracting `max(period, body)`
/// reported the *first difference* of the delay, so a sustained stall — the
/// overload regime this measurement exists to characterize — read as zero
/// forever. `tokio::time::interval::tick` hands back the deadline it was
/// scheduled for, so the delay is a subtraction with no model behind it.
fn tick_entry_at(stamp: u64, late_ns: u64) {
let prev = PREV_ENTRY_NS.swap(stamp, Relaxed);
record_gauge(Gauge::Ticks, 1);
record_gauge(Gauge::ArmStarvation, late_ns);
if prev == 0 {
// First entry of this capture: there is no previous entry to measure a
// gap against, and the idle interval before arming is not a gap. The
// lateness above does not depend on a previous entry, so it still counts.
return;
}
record_gauge(Gauge::TickGap, stamp.saturating_sub(prev));
}
/// Sample the gauges that come from node state.
pub(crate) fn tick_gauges(on: bool, peers: u64) {
if !on {
return;
}
record_gauge(Gauge::Peers, peers);
}
/// Take (and clear) this interval's figures for one step: count, max ns, total
/// ns. Called only by the writer thread.
pub(crate) fn take_step(domain: Domain, step: Step) -> (u64, u64, u64) {
let idx = slot(domain, step);
(
COUNT[idx].swap(0, Relaxed),
MAX_NS[idx].swap(0, Relaxed),
TOTAL_NS[idx].swap(0, Relaxed),
)
}
/// Take (and clear) this interval's figures for one gauge.
pub(crate) fn take_gauge(gauge: Gauge) -> (u64, u64, u64) {
let idx = gauge as usize;
(
G_COUNT[idx].swap(0, Relaxed),
G_MAX[idx].swap(0, Relaxed),
G_TOTAL[idx].swap(0, Relaxed),
)
}
/// Zero every counter so a capture starts from a clean slate.
pub(crate) fn reset() {
for i in 0..N_SLOTS {
COUNT[i].store(0, Relaxed);
MAX_NS[i].store(0, Relaxed);
TOTAL_NS[i].store(0, Relaxed);
}
for i in 0..N_GAUGES {
G_COUNT[i].store(0, Relaxed);
G_MAX[i].store(0, Relaxed);
G_TOTAL[i].store(0, Relaxed);
}
PREV_ENTRY_NS.store(0, Relaxed);
}
#[cfg(test)]
mod tests {
use super::*;
use std::sync::MutexGuard;
/// Shared with the capture tests: they mutate the same statics. See
/// `crate::instr::test_serial`.
fn serial() -> MutexGuard<'static, ()> {
crate::instr::test_serial()
}
#[test]
fn steps_table_is_dense() {
for (i, step) in STEPS.iter().enumerate() {
assert_eq!(*step as usize, i, "step {} is out of order", step.name());
}
assert_eq!(STEPS.len(), N_STEPS);
}
#[test]
fn gauges_table_is_dense() {
for (i, gauge) in GAUGES.iter().enumerate() {
assert_eq!(*gauge as usize, i, "gauge {} is out of order", gauge.name());
}
assert_eq!(GAUGES.len(), N_GAUGES);
}
#[test]
fn step_names_are_unique() {
let mut names: Vec<&str> = STEPS.iter().map(|s| s.name()).collect();
names.sort_unstable();
let before = names.len();
names.dedup();
assert_eq!(before, names.len(), "duplicate step name");
}
#[test]
fn emitted_row_count_matches_build() {
let emitted = STEPS.iter().filter(|s| s.emitted()).count();
// 24 unconditional subsystem steps + the whole-tick span, plus the two
// conditionally-compiled steps where this build has them.
let mut expected = 25;
if cfg!(any(target_os = "linux", target_os = "macos")) {
expected += 1;
}
if cfg!(debug_assertions) {
expected += 1;
}
assert_eq!(emitted, expected);
}
#[test]
fn record_accumulates_count_max_and_total() {
let _guard = serial();
reset();
record(Domain::Tick, Step::CheckRekey, Duration::from_nanos(10));
record(Domain::Tick, Step::CheckRekey, Duration::from_nanos(30));
let (count, max, total) = take_step(Domain::Tick, Step::CheckRekey);
assert_eq!((count, max, total), (2, 30, 40));
// Taking clears the slot.
assert_eq!(take_step(Domain::Tick, Step::CheckRekey), (0, 0, 0));
}
#[test]
fn starvation_is_the_measured_lateness_of_the_entry() {
let _guard = serial();
reset();
// The interval hands back the deadline it was scheduled for, so the
// delay is `now - deadline` and nothing is derived from the period, the
// previous entry, or the previous body.
PREV_ENTRY_NS.store(1_000_000_000, Relaxed);
tick_entry_at(1_100_000_000, 50_000_000);
assert_eq!(take_gauge(Gauge::TickGap), (1, 100_000_000, 100_000_000));
assert_eq!(
take_gauge(Gauge::ArmStarvation),
(1, 50_000_000, 50_000_000)
);
assert_eq!(take_gauge(Gauge::Ticks).0, 1);
reset();
}
#[test]
fn sustained_lateness_is_reported_on_every_tick() {
let _guard = serial();
reset();
// The regime the two earlier designs both hid. Three consecutive entries
// each 50 ms past their deadline, one period apart, i.e. the arm waiting
// a constant amount behind the other select arms every round. The gaps
// are all exactly one period, so any formula derived from the
// inter-entry gap reports zero here; measured lateness reports 50 ms
// three times, which is the truth.
PREV_ENTRY_NS.store(1_000_000_000, Relaxed);
tick_entry_at(1_050_000_000, 50_000_000);
tick_entry_at(1_100_000_000, 50_000_000);
tick_entry_at(1_150_000_000, 50_000_000);
let (count, max, total) = take_gauge(Gauge::ArmStarvation);
assert_eq!(count, 3);
assert_eq!(max, 50_000_000);
assert_eq!(total, 150_000_000);
// ...and the gap alone carries no signal about it: every gap is one
// period, exactly as it would be on a perfectly healthy node.
assert_eq!(take_gauge(Gauge::TickGap), (3, 50_000_000, 150_000_000));
reset();
}
#[test]
fn first_entry_of_a_capture_records_no_gap_but_still_records_lateness() {
let _guard = serial();
reset();
tick_entry_at(500, 7_000_000);
assert_eq!(take_gauge(Gauge::Ticks).0, 1);
assert_eq!(take_gauge(Gauge::TickGap), (0, 0, 0));
// Lateness does not depend on a previous entry, so the first tick of a
// capture still contributes one.
assert_eq!(take_gauge(Gauge::ArmStarvation), (1, 7_000_000, 7_000_000));
reset();
}
#[test]
fn an_on_schedule_entry_reports_no_starvation() {
let _guard = serial();
reset();
PREV_ENTRY_NS.store(1_000_000_000, Relaxed);
tick_entry_at(1_050_000_000, 0);
assert_eq!(take_gauge(Gauge::ArmStarvation), (1, 0, 0));
assert_eq!(take_gauge(Gauge::TickGap), (1, 50_000_000, 50_000_000));
reset();
}
#[test]
fn gate_off_records_nothing() {
let _guard = serial();
reset();
let t = Instant::now();
tick_entry(false, t, t);
tick_gauges(false, 42);
assert_eq!(take_gauge(Gauge::Ticks), (0, 0, 0));
assert_eq!(take_gauge(Gauge::Peers), (0, 0, 0));
}
}
src/instr/writer.rs
+218
View File
@@ -0,0 +1,218 @@
//! The capture writer: a dedicated, named OS thread with an explicit
//! lifecycle.
//!
//! There is no worker-thread lifecycle in this codebase to copy — the crypto
//! worker pools are never torn down and drop their join handles at spawn — so
//! this one is designed here.
//!
//! Two properties matter:
//!
//! - **It is an OS thread, not a spawned task.** The runtime is
//! `current_thread`, so file I/O on a task would run on the rx loop's own
//! thread and stall every `select!` arm, including the one being measured.
//! - **It waits on a channel with a timeout, not on a sleep.** `recv_timeout`
//! returns immediately when the toggle sends stop, so `off` performs a final
//! drain and joins promptly instead of parking the caller for up to a full
//! flush interval.
//!
//! The thread is created lazily when a capture starts, so a node that never
//! arms one never has the thread.
use std::fs::File;
use std::io::Write;
use std::sync::mpsc::{self, Receiver, RecvTimeoutError, Sender};
use std::thread::{self, JoinHandle};
use std::time::{SystemTime, UNIX_EPOCH};
use super::capture::{self, BYTE_CAP, INTERVAL};
use super::recorder::{self, DOMAINS, GAUGES, STEPS};
/// Owner-side handle to the writer thread.
pub(crate) struct Handle {
stop_tx: Sender<()>,
join: JoinHandle<()>,
}
impl Handle {
/// Wake the writer, let it drain once more, and join it.
pub(crate) fn stop_and_join(self) {
// A send error means the thread already exited (cap stop); joining is
// still correct and returns at once.
let _ = self.stop_tx.send(());
let _ = self.join.join();
}
}
/// Start the writer thread on an already-open sink.
pub(crate) fn spawn(file: File) -> std::io::Result<Handle> {
let (stop_tx, stop_rx) = mpsc::channel();
let join = thread::Builder::new()
.name("fips-profile".to_string())
.spawn(move || run(file, stop_rx))?;
Ok(Handle { stop_tx, join })
}
/// What one flush cycle decided. Separated from [`run`] so the terminal paths
/// can be driven in a test with a failing sink: the loop below owns the waiting
/// and the state transition, this owns the decision.
#[derive(Debug, PartialEq, Eq)]
enum Cycle {
Continue,
CapReached,
WriteFailed,
}
/// Drain one interval into the sink and decide whether the capture goes on.
fn flush_cycle<W: Write>(file: &mut W) -> Cycle {
if flush(file).is_err() {
// The sink is gone or full; stop rather than spinning on a broken file
// for the rest of the run.
let _ = note(file, "capture stopped: write error");
return Cycle::WriteFailed;
}
if capture::bytes_written() >= BYTE_CAP {
let _ = note(
file,
&format!("capture stopped: byte cap {BYTE_CAP} reached"),
);
let _ = file.flush();
return Cycle::CapReached;
}
Cycle::Continue
}
fn run<W: Write>(mut file: W, stop_rx: Receiver<()>) {
loop {
match stop_rx.recv_timeout(INTERVAL) {
// Stop requested, or the owner went away: final drain, then exit.
Ok(()) | Err(RecvTimeoutError::Disconnected) => {
let _ = flush(&mut file);
let _ = file.flush();
return;
}
Err(RecvTimeoutError::Timeout) => match flush_cycle(&mut file) {
Cycle::Continue => {}
Cycle::WriteFailed => {
// The trailer went to the same failing file, so it is not a
// signal that survives. `stop` and a subsequent `on` both
// clear the state without surfacing it, so an operator would
// otherwise never learn the window was truncated.
tracing::warn!(
target: "fips::instr",
"profile capture stopped: write error on the sink"
);
capture::mark_stopped(capture::STOPPED_BY_ERROR);
return;
}
Cycle::CapReached => {
capture::mark_stopped(capture::STOPPED_BY_CAP);
return;
}
},
}
}
}
/// Append a `#`-prefixed trailer line.
fn note<W: Write>(file: &mut W, text: &str) -> std::io::Result<()> {
let line = format!("# {text}\n");
file.write_all(line.as_bytes())?;
capture::add_bytes(line.len() as u64);
Ok(())
}
/// Emit one interval: every step of every domain, then the gauges.
///
/// Every emitted step gets a row every interval, including zero-count rows, so
/// "this step did not run" is visible rather than absent. The two steps whose
/// call sites are conditionally compiled are excluded in builds that do not
/// have them, so no row is structurally zero forever.
fn flush<W: Write>(file: &mut W) -> std::io::Result<()> {
let ts = SystemTime::now()
.duration_since(UNIX_EPOCH)
.map(|d| d.as_secs())
.unwrap_or(0);
let mut out = String::with_capacity(4096);
for domain in DOMAINS {
for step in STEPS {
if !step.emitted() {
continue;
}
let (count, max_ns, total_ns) = recorder::take_step(domain, step);
out.push_str(&format!(
"{ts}\tstep\t{domain}\t{name}\t{count}\t{max}\t{total}\tus\n",
domain = domain.name(),
name = step.name(),
max = max_ns / 1_000,
total = total_ns / 1_000,
));
}
}
// Gauges carry the tick domain today; the row kind and the unit column keep
// them distinguishable from the duration rows above.
for gauge in GAUGES {
let (count, mut max, mut total) = recorder::take_gauge(gauge);
if gauge.is_duration() {
max /= 1_000;
total /= 1_000;
}
out.push_str(&format!(
"{ts}\tgauge\t{domain}\t{name}\t{count}\t{max}\t{total}\t{unit}\n",
domain = recorder::Domain::Tick.name(),
name = gauge.name(),
unit = gauge.unit(),
));
}
file.write_all(out.as_bytes())?;
capture::add_bytes(out.len() as u64);
Ok(())
}
#[cfg(test)]
mod tests {
use super::*;
/// A sink that fails every write, so the writer's error path is driven by a
/// real `Err` rather than asserted about.
struct AlwaysFails;
impl Write for AlwaysFails {
fn write(&mut self, _buf: &[u8]) -> std::io::Result<usize> {
Err(std::io::Error::new(
std::io::ErrorKind::StorageFull,
"no space left on device",
))
}
fn flush(&mut self) -> std::io::Result<()> {
Ok(())
}
}
/// A failing sink ends the capture as a write error, which `run` turns into
/// `STOPPED_BY_ERROR` — not into a byte-cap stop. The distinction is what
/// tells an operator that a window is truncated rather than complete.
///
/// **Coverage note.** This drives the decision, not the filesystem
/// condition. The mesh rehearsal cannot produce one: removing the file
/// leaves the writer's descriptor valid and writes keep succeeding into the
/// unlinked inode, and mounting a tiny filesystem inside the test container
/// is refused. So "a real ENOSPC reaches this branch" stays unexercised;
/// what is covered is that an `Err` from the sink produces the error
/// outcome and not the cap outcome.
#[test]
fn a_failing_sink_ends_the_cycle_as_an_error_not_a_cap() {
let _guard = crate::instr::test_serial();
assert_eq!(flush_cycle(&mut AlwaysFails), Cycle::WriteFailed);
}
/// The healthy path must not be reported as either terminal state, or the
/// test above would pass for a writer that always stops.
#[test]
fn a_working_sink_continues() {
let _guard = crate::instr::test_serial();
let mut sink: Vec<u8> = Vec::new();
assert_eq!(flush_cycle(&mut sink), Cycle::Continue);
}
}
src/lib.rs
+53 -20
View File
@@ -3,22 +3,34 @@
//! A distributed, decentralized network routing protocol for mesh nodes
//! connecting over arbitrary transports.
pub mod bloom;
// Name the `alloc` crate directly so the sans-IO protocol cores can spell their
// heap-type imports in `no_std`-forward form (`alloc::sync::Arc`,
// `alloc::collections::BTreeMap`). The crate remains `std`; this only reduces the
// distance to extracting the pure cores into a `no_std` crate later.
extern crate alloc;
pub mod cache;
pub mod config;
pub mod control;
pub mod discovery;
#[cfg(target_os = "linux")]
pub mod gateway;
pub mod identity;
pub mod mmp;
// Declared before `node` (and named to sort there) because it carries
// `#[macro_use]`: the tick instrumentation macro must be in scope for the
// modules that follow.
#[macro_use]
pub(crate) mod instr;
pub mod mdns;
pub mod node;
pub mod noise;
pub mod nostr;
pub mod peer;
pub mod perf_profile;
pub mod protocol;
pub(crate) mod proto;
#[cfg(test)]
pub(crate) mod testutil;
mod time;
pub mod transport;
pub mod tree;
pub mod upper;
pub mod utils;
pub mod version;
@@ -33,14 +45,16 @@ pub use identity::{
pub use config::{Config, ConfigError, IdentityConfig, NymConfig, TorConfig, UdpConfig};
pub use upper::config::{DnsConfig, TunConfig};
// Re-export discovery types
pub use discovery::{BootstrapHandoffResult, EstablishedTraversal};
// Re-export nostr rendezvous handoff types
pub use nostr::{BootstrapHandoffResult, EstablishedTraversal, is_punch_packet};
// Re-export tree types
pub use tree::{CoordEntry, ParentDeclaration, TreeCoordinate, TreeError, TreeState};
// Re-export tree types (relocated from tree:: to proto::stp)
pub use proto::stp::{
CoordEntry, CoordError, ParentDeclaration, TreeCoordinate, TreeError, TreeState,
};
// Re-export bloom filter types
pub use bloom::{BloomError, BloomFilter, BloomState};
// Re-export bloom filter types (relocated from bloom:: to proto::bloom)
pub use proto::bloom::{BloomError, BloomFilter, BloomState};
// Re-export transport types
pub use transport::udp::UdpTransport;
@@ -50,21 +64,40 @@ pub use transport::{
TransportState, TransportType, packet_channel,
};
// Re-export protocol types
pub use protocol::{
CoordsRequired, FilterAnnounce, HandshakeMessageType, LinkMessageType, LookupRequest,
LookupResponse, PathBroken, ProtocolError, SessionAck, SessionDatagram, SessionFlags,
SessionMessageType, SessionSetup, TreeAnnounce,
// Re-export link-layer types (relocated from protocol:: to proto::link)
pub use proto::link::{LinkMessageType, SessionDatagram};
// Re-export the shared protocol error (relocated from protocol:: to proto::Error)
pub use proto::Error;
// Re-export FSP session wire types (relocated from protocol:: to proto::fsp)
pub use proto::fsp::{SessionAck, SessionFlags, SessionMessageType, SessionSetup};
// Re-export STP wire types (relocated from protocol:: to proto::stp)
pub use proto::stp::TreeAnnounce;
// Re-export bloom wire types (relocated from protocol:: to proto::bloom)
pub use proto::bloom::FilterAnnounce;
// Re-export discovery wire types (relocated from protocol:: to proto::lookup)
pub use proto::lookup::{LookupRequest, LookupResponse};
// Re-export routing wire types (relocated from protocol:: to proto::routing)
pub use proto::routing::{
COORDS_REQUIRED_SIZE, CoordsRequired, MTU_EXCEEDED_SIZE, MtuExceeded, PathBroken,
};
// Re-export FMP link-framing wire type (relocated from protocol:: to proto::fmp)
pub use proto::fmp::HandshakeMessageType;
// Re-export cache types
pub use cache::{CacheEntry, CacheError, CacheStats, CoordCache};
// Re-export FMP tie-break helper and promotion result (relocated from peer:: to proto::fmp)
pub use proto::fmp::{PromotionResult, cross_connection_winner};
// Re-export peer types
pub use peer::{
ActivePeer, ConnectivityState, HandshakeState, PeerConnection, PeerError, PeerSlot,
PromotionResult, cross_connection_winner,
};
pub use peer::{ActivePeer, ConnectivityState, PeerError};
// Re-export node types
pub use node::{Node, NodeError, NodeState, UpdatePeersOutcome};
src/discovery/lan/mod.rs → src/mdns/mod.rs
+28 -19
View File
@@ -54,8 +54,12 @@ pub const TXT_KEY_SCOPE: &str = "scope";
/// `PROTOCOL_VERSION`).
pub const TXT_KEY_VERSION: &str = "v";
/// FIPS protocol version advertised in the mDNS TXT `v` key. Kept in sync
/// with the Nostr rendezvous `PROTOCOL_VERSION` (same value, `"1"`).
const TXT_PROTOCOL_VERSION: &str = "1";
#[derive(Debug, Error)]
pub enum LanDiscoveryError {
pub enum LanRendezvousError {
#[error("mDNS daemon init failed: {0}")]
Daemon(String),
#[error("mDNS register failed: {0}")]
@@ -79,7 +83,7 @@ pub struct LanDiscoveredPeer {
pub observed_at: Instant,
}
/// Browser-side events surfaced by `LanDiscovery::drain_events`.
/// Browser-side events surfaced by `LanRendezvous::drain_events`.
#[derive(Debug, Clone)]
pub enum LanEvent {
Discovered(LanDiscoveredPeer),
@@ -87,17 +91,17 @@ pub enum LanEvent {
/// Runtime configuration for the mDNS responder + browser.
#[derive(Debug, Clone, serde::Serialize, serde::Deserialize)]
pub struct LanDiscoveryConfig {
pub struct LanRendezvousConfig {
/// Master switch. Default: `false` — LAN discovery is opt-in. Operators
/// who want sub-second same-LAN pairing enable it via
/// `node.discovery.lan.enabled: true`. Default-off avoids reintroducing
/// `node.rendezvous.lan.enabled: true`. Default-off avoids reintroducing
/// a per-LAN identity broadcast on nodes that have deliberately disabled
/// other discovery channels, and avoids any multicast surprise on upgrade.
#[serde(default = "LanDiscoveryConfig::default_enabled")]
#[serde(default = "LanRendezvousConfig::default_enabled")]
pub enabled: bool,
/// Overridable service type, primarily so integration tests can run
/// multiple isolated services on the same loopback interface.
#[serde(default = "LanDiscoveryConfig::default_service_type")]
#[serde(default = "LanRendezvousConfig::default_service_type")]
pub service_type: String,
/// Optional application/network scope carried in the LAN-only TXT
/// record. Browsers that set a scope ignore adverts for other scopes.
@@ -109,7 +113,7 @@ pub struct LanDiscoveryConfig {
pub scope: Option<String>,
}
impl Default for LanDiscoveryConfig {
impl Default for LanRendezvousConfig {
fn default() -> Self {
Self {
enabled: Self::default_enabled(),
@@ -119,7 +123,7 @@ impl Default for LanDiscoveryConfig {
}
}
impl LanDiscoveryConfig {
impl LanRendezvousConfig {
fn default_enabled() -> bool {
false
}
@@ -129,7 +133,7 @@ impl LanDiscoveryConfig {
}
/// Running mDNS responder + browser bound to the node's UDP advert port.
pub struct LanDiscovery {
pub struct LanRendezvous {
daemon: ServiceDaemon,
own_npub: String,
instance_fullname: String,
@@ -137,7 +141,12 @@ pub struct LanDiscovery {
event_pump: tokio::task::JoinHandle<()>,
}
impl LanDiscovery {
impl LanRendezvous {
/// Whether the mDNS event-pump task has exited (runtime liveness).
pub fn is_finished(&self) -> bool {
self.event_pump.is_finished()
}
/// Start the mDNS responder and browser.
///
/// `advertised_port` is the UDP port the operational UDP transport
@@ -148,16 +157,16 @@ impl LanDiscovery {
identity: &Identity,
scope: Option<String>,
advertised_port: u16,
config: LanDiscoveryConfig,
) -> Result<Arc<Self>, LanDiscoveryError> {
config: LanRendezvousConfig,
) -> Result<Arc<Self>, LanRendezvousError> {
if !config.enabled {
return Err(LanDiscoveryError::Disabled);
return Err(LanRendezvousError::Disabled);
}
if advertised_port == 0 {
return Err(LanDiscoveryError::NoAdvertisedPort);
return Err(LanRendezvousError::NoAdvertisedPort);
}
let daemon = ServiceDaemon::new().map_err(|e| LanDiscoveryError::Daemon(e.to_string()))?;
let daemon = ServiceDaemon::new().map_err(|e| LanRendezvousError::Daemon(e.to_string()))?;
let npub = identity.npub();
// mDNS DNS labels are capped at 63 bytes. 16 bech32 chars of npub
@@ -176,7 +185,7 @@ impl LanDiscovery {
}
props.insert(
TXT_KEY_VERSION.to_string(),
super::nostr::PROTOCOL_VERSION.to_string(),
TXT_PROTOCOL_VERSION.to_string(),
);
// host_ipv4 is set to "127.0.0.1" *and* enable_addr_auto() is
@@ -193,18 +202,18 @@ impl LanDiscovery {
advertised_port,
Some(props),
)
.map_err(|e| LanDiscoveryError::Register(e.to_string()))?
.map_err(|e| LanRendezvousError::Register(e.to_string()))?
.enable_addr_auto();
let instance_fullname = service_info.get_fullname().to_string();
daemon
.register(service_info)
.map_err(|e| LanDiscoveryError::Register(e.to_string()))?;
.map_err(|e| LanRendezvousError::Register(e.to_string()))?;
let browse_rx = daemon
.browse(&config.service_type)
.map_err(|e| LanDiscoveryError::Browse(e.to_string()))?;
.map_err(|e| LanRendezvousError::Browse(e.to_string()))?;
let (events_tx, events_rx) = tokio::sync::mpsc::unbounded_channel();
let own_npub = npub.clone();
src/discovery/lan/tests.rs → src/mdns/tests.rs
+12 -12
View File
@@ -4,7 +4,7 @@ use std::time::Duration;
use crate::Identity;
use mdns_sd::ScopedIp;
use super::{LanDiscovery, LanDiscoveryConfig, LanEvent};
use super::{LanEvent, LanRendezvous, LanRendezvousConfig};
/// Distinct service type per test run so concurrent cargo-test workers
/// on the same machine don't cross-feed each other's adverts via the
@@ -15,8 +15,8 @@ fn isolated_service_type(tag: &str) -> String {
format!("_fipstest-{tag}-{rand:08x}._udp.local.")
}
fn config_for(service_type: String) -> LanDiscoveryConfig {
LanDiscoveryConfig {
fn config_for(service_type: String) -> LanRendezvousConfig {
LanRendezvousConfig {
enabled: true,
service_type,
scope: None,
@@ -47,7 +47,7 @@ fn non_link_local_ipv6_advert_is_preserved() {
}
async fn wait_for_peer(
discovery: &LanDiscovery,
discovery: &LanRendezvous,
expected_npub: &str,
timeout: Duration,
) -> Option<super::LanDiscoveredPeer> {
@@ -64,7 +64,7 @@ async fn wait_for_peer(
None
}
/// Two LanDiscovery instances on isolated service types — `a` browses
/// Two LanRendezvous instances on isolated service types — `a` browses
/// only its own type and never sees `b`, and vice versa. Sanity check
/// that the scope-isolation defense works (we'd lose isolation if mdns-
/// sd ever leaked across service types).
@@ -76,7 +76,7 @@ async fn isolated_service_types_do_not_cross_feed() {
let service_a = isolated_service_type("isolated-a");
let service_b = isolated_service_type("isolated-b");
let lan_a = LanDiscovery::start(
let lan_a = LanRendezvous::start(
&identity_a,
Some("scope-x".to_string()),
61001,
@@ -84,7 +84,7 @@ async fn isolated_service_types_do_not_cross_feed() {
)
.await
.expect("start a");
let lan_b = LanDiscovery::start(
let lan_b = LanRendezvous::start(
&identity_b,
Some("scope-x".to_string()),
61002,
@@ -118,7 +118,7 @@ async fn isolated_service_types_do_not_cross_feed() {
assert!(!saw_a_from_b, "isolated service types must not cross-feed");
}
/// Two LanDiscovery instances on the same service type and the same
/// Two LanRendezvous instances on the same service type and the same
/// scope: each should observe the other's advert within a few seconds.
/// Exercises the responder + browser + TXT plumbing end-to-end.
///
@@ -136,7 +136,7 @@ async fn matched_scope_peers_observe_each_other() {
let service = isolated_service_type("matched");
let lan_a = LanDiscovery::start(
let lan_a = LanRendezvous::start(
&identity_a,
Some("scope-shared".to_string()),
61101,
@@ -144,7 +144,7 @@ async fn matched_scope_peers_observe_each_other() {
)
.await
.expect("start a");
let lan_b = LanDiscovery::start(
let lan_b = LanRendezvous::start(
&identity_b,
Some("scope-shared".to_string()),
61102,
@@ -181,7 +181,7 @@ async fn cross_scope_advert_is_filtered() {
let service = isolated_service_type("cross-scope");
let lan_a = LanDiscovery::start(
let lan_a = LanRendezvous::start(
&identity_a,
Some("scope-a".to_string()),
61201,
@@ -189,7 +189,7 @@ async fn cross_scope_advert_is_filtered() {
)
.await
.expect("start a");
let lan_b = LanDiscovery::start(
let lan_b = LanRendezvous::start(
&identity_b,
Some("scope-b".to_string()),
61202,
src/mmp/metrics.rs
-556
View File
@@ -1,556 +0,0 @@
//! MMP derived metrics.
//!
//! `MmpMetrics` processes incoming ReceiverReports (from our peer) and
//! maintains derived metrics: SRTT, loss rate, goodput, ETX, and dual
//! EWMA trend indicators. Updated by the sender side when it receives
//! a ReceiverReport about its own traffic.
use crate::mmp::algorithms::{DualEwma, SrttEstimator, compute_etx};
use crate::mmp::report::ReceiverReport;
use std::time::Instant;
use tracing::trace;
/// Derived MMP metrics, updated from incoming ReceiverReports.
///
/// This lives on the sender side: when we receive a ReceiverReport from
/// our peer describing what they observed about our traffic, we process
/// it here to compute RTT, loss, goodput, and trend indicators.
pub struct MmpMetrics {
/// Smoothed RTT from timestamp echo.
pub srtt: SrttEstimator,
/// Dual EWMA trend detectors.
pub rtt_trend: DualEwma,
pub loss_trend: DualEwma,
pub goodput_trend: DualEwma,
pub jitter_trend: DualEwma,
pub etx_trend: DualEwma,
/// Forward delivery ratio (what fraction of our frames the peer received).
pub delivery_ratio_forward: f64,
/// Reverse delivery ratio (set when we compute from our own receiver state).
pub delivery_ratio_reverse: f64,
/// ETX computed from bidirectional delivery ratios.
pub etx: f64,
/// Smoothed goodput in bytes/sec (forward direction: what the peer received from us).
pub goodput_bps: f64,
// --- State for delta computation ---
/// Previous ReceiverReport's cumulative counters (for computing interval deltas).
prev_rr_cum_packets: u64,
prev_rr_cum_bytes: u64,
prev_rr_highest_counter: u64,
prev_rr_ecn_ce: u32,
prev_rr_reorder: u32,
/// Time of previous ReceiverReport (for goodput rate computation).
prev_rr_time: Option<Instant>,
/// Whether we have a previous ReceiverReport for delta computation.
has_prev_rr: bool,
// --- State for reverse delivery ratio delta computation ---
/// Previous reverse-side cumulative packets received (our receiver state).
prev_reverse_packets: u64,
/// Previous reverse-side highest counter (our receiver state).
prev_reverse_highest: u64,
/// Whether we have a previous reverse-side snapshot for delta computation.
has_prev_reverse: bool,
}
impl MmpMetrics {
/// Reset state derived from ReceiverReport counters for rekey cutover.
///
/// The new session starts with counter 0, so the prev_rr deltas must
/// be reset to avoid computing bogus loss/goodput from the counter
/// discontinuity. RTT (SRTT) is preserved since it remains valid.
pub fn reset_for_rekey(&mut self) {
self.prev_rr_cum_packets = 0;
self.prev_rr_cum_bytes = 0;
self.prev_rr_highest_counter = 0;
self.prev_rr_ecn_ce = 0;
self.prev_rr_reorder = 0;
self.prev_rr_time = None;
self.has_prev_rr = false;
self.delivery_ratio_forward = 1.0;
self.prev_reverse_packets = 0;
self.prev_reverse_highest = 0;
self.has_prev_reverse = false;
// Keep srtt, etx, trends, goodput_bps — they'll refresh from data
}
pub fn new() -> Self {
Self {
srtt: SrttEstimator::new(),
rtt_trend: DualEwma::new(),
loss_trend: DualEwma::new(),
goodput_trend: DualEwma::new(),
jitter_trend: DualEwma::new(),
etx_trend: DualEwma::new(),
delivery_ratio_forward: 1.0,
delivery_ratio_reverse: 1.0,
etx: 1.0,
goodput_bps: 0.0,
prev_rr_cum_packets: 0,
prev_rr_cum_bytes: 0,
prev_rr_highest_counter: 0,
prev_rr_ecn_ce: 0,
prev_rr_reorder: 0,
prev_rr_time: None,
has_prev_rr: false,
prev_reverse_packets: 0,
prev_reverse_highest: 0,
has_prev_reverse: false,
}
}
/// Process an incoming ReceiverReport (from the peer about our traffic).
///
/// `our_timestamp_ms` is the current session-relative time in ms (for RTT).
/// `now` is the current monotonic time (for goodput rate computation).
///
/// Returns `true` if this report produced the first SRTT measurement
/// (transition from uninitialized to initialized).
pub fn process_receiver_report(
&mut self,
rr: &ReceiverReport,
our_timestamp_ms: u32,
now: Instant,
) -> bool {
let had_srtt = self.srtt.initialized();
if self.has_prev_rr {
let counters_regressed = rr.highest_counter < self.prev_rr_highest_counter
|| rr.cumulative_packets_recv < self.prev_rr_cum_packets
|| rr.cumulative_bytes_recv < self.prev_rr_cum_bytes
|| rr.ecn_ce_count < self.prev_rr_ecn_ce
|| rr.cumulative_reorder_count < self.prev_rr_reorder;
let duplicate_counters = rr.highest_counter == self.prev_rr_highest_counter
&& rr.cumulative_packets_recv == self.prev_rr_cum_packets
&& rr.cumulative_bytes_recv == self.prev_rr_cum_bytes
&& rr.ecn_ce_count == self.prev_rr_ecn_ce
&& rr.cumulative_reorder_count == self.prev_rr_reorder;
// Safe to drop: reports are only built after interval data, so
// a fresh report always advances at least one cumulative counter.
if counters_regressed || duplicate_counters {
trace!(
highest_counter = rr.highest_counter,
prev_highest_counter = self.prev_rr_highest_counter,
cumulative_packets_recv = rr.cumulative_packets_recv,
prev_cumulative_packets_recv = self.prev_rr_cum_packets,
cumulative_bytes_recv = rr.cumulative_bytes_recv,
prev_cumulative_bytes_recv = self.prev_rr_cum_bytes,
"Ignoring stale MMP ReceiverReport"
);
return false;
}
}
// --- RTT from timestamp echo ---
// RTT = now - echoed_timestamp - dwell_time
if rr.timestamp_echo > 0 {
let echo_ms = rr.timestamp_echo;
let dwell_ms = u32::from(rr.dwell_time);
let rtt_sample_ms = echo_ms
.checked_add(dwell_ms)
.and_then(|send_done_ms| our_timestamp_ms.checked_sub(send_done_ms));
match rtt_sample_ms {
Some(rtt_ms) if rtt_ms > 0 => {
let rtt_us = (rtt_ms as i64) * 1000;
trace!(
our_ts = our_timestamp_ms,
echo = echo_ms,
dwell = dwell_ms,
rtt_ms = rtt_ms,
srtt_ms = self.srtt.srtt_us() as f64 / 1000.0,
"RTT sample from timestamp echo"
);
self.srtt.update(rtt_us);
self.rtt_trend.update(rtt_us as f64);
}
_ => {
trace!(
our_ts = our_timestamp_ms,
echo = echo_ms,
dwell = dwell_ms,
"Ignoring invalid MMP RTT sample"
);
}
}
}
// --- Loss rate from cumulative counters ---
// Delta: frames the peer should have received vs. actually received
if self.has_prev_rr {
let counter_span = rr
.highest_counter
.saturating_sub(self.prev_rr_highest_counter);
let packets_delta = rr
.cumulative_packets_recv
.saturating_sub(self.prev_rr_cum_packets);
if counter_span > 0 {
let delivery = (packets_delta as f64) / (counter_span as f64);
self.delivery_ratio_forward = delivery.clamp(0.0, 1.0);
let loss_rate = 1.0 - self.delivery_ratio_forward;
self.loss_trend.update(loss_rate);
self.etx = compute_etx(self.delivery_ratio_forward, self.delivery_ratio_reverse);
self.etx_trend.update(self.etx);
}
}
// --- Goodput from cumulative bytes + time delta ---
if self.has_prev_rr {
let bytes_delta = rr
.cumulative_bytes_recv
.saturating_sub(self.prev_rr_cum_bytes);
self.goodput_trend.update(bytes_delta as f64);
// Compute bytes/sec if we have a time reference
if let Some(prev_time) = self.prev_rr_time {
let elapsed = now.duration_since(prev_time);
let secs = elapsed.as_secs_f64();
if secs > 0.0 {
let bps = bytes_delta as f64 / secs;
// EWMA smoothing: α = 1/4
if self.goodput_bps == 0.0 {
self.goodput_bps = bps;
} else {
self.goodput_bps += (bps - self.goodput_bps) * 0.25;
}
}
}
}
// --- Jitter trend ---
self.jitter_trend.update(rr.jitter as f64);
// --- Save for next delta ---
self.prev_rr_cum_packets = rr.cumulative_packets_recv;
self.prev_rr_cum_bytes = rr.cumulative_bytes_recv;
self.prev_rr_highest_counter = rr.highest_counter;
self.prev_rr_ecn_ce = rr.ecn_ce_count;
self.prev_rr_reorder = rr.cumulative_reorder_count;
self.prev_rr_time = Some(now);
self.has_prev_rr = true;
!had_srtt && self.srtt.initialized()
}
/// Update the reverse delivery ratio from our own receiver state.
///
/// Computes a per-interval delta (same as forward ratio) rather than
/// a lifetime cumulative ratio, so ETX responds to recent conditions.
pub fn update_reverse_delivery(&mut self, our_recv_packets: u64, peer_highest: u64) {
if self.has_prev_reverse {
let counter_span = peer_highest.saturating_sub(self.prev_reverse_highest);
let packets_delta = our_recv_packets.saturating_sub(self.prev_reverse_packets);
if counter_span > 0 {
let delivery = (packets_delta as f64) / (counter_span as f64);
self.delivery_ratio_reverse = delivery.clamp(0.0, 1.0);
self.etx = compute_etx(self.delivery_ratio_forward, self.delivery_ratio_reverse);
self.etx_trend.update(self.etx);
}
}
self.prev_reverse_packets = our_recv_packets;
self.prev_reverse_highest = peer_highest;
self.has_prev_reverse = true;
}
/// Current smoothed RTT in milliseconds, or `None` if not yet measured.
pub fn srtt_ms(&self) -> Option<f64> {
if self.srtt.initialized() {
Some(self.srtt.srtt_us() as f64 / 1000.0)
} else {
None
}
}
/// Current loss rate (0.0 = no loss, 1.0 = total loss).
pub fn loss_rate(&self) -> f64 {
1.0 - self.delivery_ratio_forward
}
/// Smoothed loss rate (long-term EWMA), or `None` if not yet initialized.
pub fn smoothed_loss(&self) -> Option<f64> {
if self.loss_trend.initialized() {
Some(self.loss_trend.long())
} else {
None
}
}
/// Smoothed ETX (long-term EWMA), or `None` if not yet initialized.
pub fn smoothed_etx(&self) -> Option<f64> {
if self.etx_trend.initialized() {
Some(self.etx_trend.long())
} else {
None
}
}
/// Current smoothed goodput in bytes/sec, or 0 if not yet measured.
pub fn goodput_bps(&self) -> f64 {
self.goodput_bps
}
/// Cumulative ECN CE count from the most recent ReceiverReport.
pub fn last_ecn_ce_count(&self) -> u32 {
self.prev_rr_ecn_ce
}
}
impl Default for MmpMetrics {
fn default() -> Self {
Self::new()
}
}
// ============================================================================
// Tests
// ============================================================================
#[cfg(test)]
mod tests {
use super::*;
use std::time::Duration;
fn make_rr(
highest_counter: u64,
cum_packets: u64,
cum_bytes: u64,
timestamp_echo: u32,
dwell: u16,
jitter: u32,
) -> ReceiverReport {
ReceiverReport {
highest_counter,
cumulative_packets_recv: cum_packets,
cumulative_bytes_recv: cum_bytes,
timestamp_echo,
dwell_time: dwell,
max_burst_loss: 0,
mean_burst_loss: 0,
jitter,
ecn_ce_count: 0,
owd_trend: 0,
burst_loss_count: 0,
cumulative_reorder_count: 0,
interval_packets_recv: 0,
interval_bytes_recv: 0,
}
}
#[test]
fn test_rtt_from_echo() {
let mut m = MmpMetrics::new();
let now = Instant::now();
// Peer echoes timestamp 1000ms, dwell=5ms, our current time=1050ms
let rr = make_rr(10, 10, 5000, 1000, 5, 0);
m.process_receiver_report(&rr, 1050, now);
assert!(m.srtt.initialized());
// RTT = 1050 - 1000 - 5 = 45ms
let srtt_ms = m.srtt_ms().unwrap();
assert!((srtt_ms - 45.0).abs() < 1.0, "srtt={srtt_ms}, expected ~45");
}
#[test]
fn test_ignores_duplicate_receiver_report_after_valid_sample() {
let mut m = MmpMetrics::new();
let t0 = Instant::now();
let rr1 = make_rr(10, 10, 5_000, 1_000, 5, 0);
m.process_receiver_report(&rr1, 1_050, t0);
let rr2 = make_rr(20, 18, 14_000, 1_100, 5, 0);
m.process_receiver_report(&rr2, 1_150, t0 + Duration::from_secs(1));
let baseline_srtt_ms = m.srtt_ms().unwrap();
let baseline_loss = m.loss_rate();
let baseline_goodput = m.goodput_bps();
assert!(baseline_loss > 0.0);
assert!(baseline_goodput > 0.0);
// A duplicate of the same counters arriving later would be a 4.895s
// RTT sample if accepted. It is stale and must not move metrics.
m.process_receiver_report(&rr2, 6_000, t0 + Duration::from_secs(5));
assert_eq!(m.srtt_ms().unwrap(), baseline_srtt_ms);
assert_eq!(m.loss_rate(), baseline_loss);
assert_eq!(m.goodput_bps(), baseline_goodput);
}
#[test]
fn test_ignores_out_of_order_receiver_report_after_valid_sample() {
let mut m = MmpMetrics::new();
let now = Instant::now();
let valid_rr = make_rr(20, 20, 10000, 1000, 5, 0);
m.process_receiver_report(&valid_rr, 1050, now);
let baseline_srtt_ms = m.srtt_ms().unwrap();
let old_rr = make_rr(10, 10, 5000, 1000, 0, 0);
m.process_receiver_report(&old_rr, 6000, now + Duration::from_secs(5));
let srtt_ms = m.srtt_ms().unwrap();
assert_eq!(srtt_ms, baseline_srtt_ms);
}
#[test]
fn test_ignores_wrapped_rtt_sample() {
let mut m = MmpMetrics::new();
let now = Instant::now();
let wrapped_rr = make_rr(10, 10, 5000, u32::MAX - 10, 20, 0);
m.process_receiver_report(&wrapped_rr, 15, now);
assert!(m.srtt_ms().is_none());
}
#[test]
fn test_ignores_future_rtt_sample() {
let mut m = MmpMetrics::new();
let now = Instant::now();
let future_rr = make_rr(10, 10, 5_000, 2_000, 5, 0);
m.process_receiver_report(&future_rr, 1_000, now);
assert!(m.srtt_ms().is_none());
}
#[test]
fn test_loss_rate_computation() {
let mut m = MmpMetrics::new();
let t0 = Instant::now();
// First report: baseline
let rr1 = make_rr(100, 100, 50000, 0, 0, 0);
m.process_receiver_report(&rr1, 0, t0);
// Second report: 200 counters sent, 190 received (5% loss)
let rr2 = make_rr(300, 290, 145000, 0, 0, 0);
m.process_receiver_report(&rr2, 0, t0 + Duration::from_secs(1));
let loss = m.loss_rate();
assert!((loss - 0.05).abs() < 0.01, "loss={loss}, expected ~0.05");
}
#[test]
fn test_etx_updates() {
let mut m = MmpMetrics::new();
assert_eq!(m.etx, 1.0); // initial: perfect
// Simulate some loss via forward ratio
m.delivery_ratio_forward = 0.9;
// First call establishes the baseline (no ETX update yet)
m.update_reverse_delivery(100, 100);
assert_eq!(m.etx, 1.0); // still perfect — baseline only
// Second call: 190 of 200 frames received (5% loss)
m.update_reverse_delivery(290, 300);
assert!(m.etx > 1.0);
assert!(m.etx < 2.0);
}
#[test]
fn test_no_rtt_without_echo() {
let mut m = MmpMetrics::new();
let now = Instant::now();
let rr = make_rr(10, 10, 5000, 0, 0, 0);
m.process_receiver_report(&rr, 1000, now);
assert!(m.srtt_ms().is_none());
}
#[test]
fn test_jitter_trend() {
let mut m = MmpMetrics::new();
let t0 = Instant::now();
let rr1 = make_rr(10, 10, 5000, 0, 0, 100);
m.process_receiver_report(&rr1, 0, t0);
let rr2 = make_rr(20, 20, 10000, 0, 0, 500);
m.process_receiver_report(&rr2, 0, t0 + Duration::from_secs(1));
assert!(m.jitter_trend.initialized());
// Short-term should be closer to 500 than long-term
assert!(m.jitter_trend.short() > m.jitter_trend.long());
}
#[test]
fn test_goodput_bps() {
let mut m = MmpMetrics::new();
let t0 = Instant::now();
// First report: baseline (50KB received)
let rr1 = make_rr(100, 100, 50_000, 0, 0, 0);
m.process_receiver_report(&rr1, 0, t0);
assert_eq!(m.goodput_bps(), 0.0); // no rate yet (first report)
// Second report 1s later: 150KB total (100KB delta in 1s = 100KB/s)
let rr2 = make_rr(300, 290, 150_000, 0, 0, 0);
m.process_receiver_report(&rr2, 0, t0 + Duration::from_secs(1));
assert!(
m.goodput_bps() > 90_000.0,
"goodput={}, expected ~100000",
m.goodput_bps()
);
assert!(
m.goodput_bps() < 110_000.0,
"goodput={}, expected ~100000",
m.goodput_bps()
);
}
#[test]
fn test_reverse_delivery_delta() {
let mut m = MmpMetrics::new();
// First call: baseline only, no ratio update
m.update_reverse_delivery(100, 100);
assert_eq!(m.delivery_ratio_reverse, 1.0); // unchanged from default
// Second call: perfect delivery (200 new frames, all received)
m.update_reverse_delivery(300, 300);
assert!((m.delivery_ratio_reverse - 1.0).abs() < 0.001);
// Third call: 50% loss (100 frames sent, 50 received)
m.update_reverse_delivery(350, 400);
assert!(
(m.delivery_ratio_reverse - 0.5).abs() < 0.001,
"reverse={}, expected 0.5",
m.delivery_ratio_reverse
);
}
#[test]
fn test_reverse_delivery_rekey_reset() {
let mut m = MmpMetrics::new();
// Establish baseline and one measurement
m.update_reverse_delivery(100, 100);
m.update_reverse_delivery(300, 300);
assert!((m.delivery_ratio_reverse - 1.0).abs() < 0.001);
// Rekey resets reverse state
m.reset_for_rekey();
// First call after rekey: baseline only
m.update_reverse_delivery(50, 50);
// delivery_ratio_reverse was reset to 1.0 by reset_for_rekey's
// clearing of delivery_ratio_forward; reverse is not explicitly
// reset — but the delta state is, so next call computes fresh.
assert_eq!(m.delivery_ratio_reverse, 1.0);
// Second call after rekey: 80% delivery
m.update_reverse_delivery(90, 100);
assert!(
(m.delivery_ratio_reverse - 0.8).abs() < 0.001,
"reverse={}, expected 0.8",
m.delivery_ratio_reverse
);
}
}
src/mmp/mod.rs
-555
View File
@@ -1,555 +0,0 @@
//! Metrics Measurement Protocol (MMP) — link-layer instantiation.
//!
//! Measures link quality between adjacent peers: RTT, loss, jitter,
//! throughput, one-way delay trend, and ETX. Operates on the per-frame
//! hooks (counter, timestamp, flags) introduced by the FMP wire format
//! revision.
//!
//! Three operating modes trade measurement fidelity for overhead:
//! - **Full**: sender + receiver reports at RTT-adaptive intervals
//! - **Lightweight**: receiver reports only (infer loss from counters)
//! - **Minimal**: spin bit + CE echo only, no reports
use serde::{Deserialize, Serialize};
use std::fmt::{self, Debug};
use std::time::{Duration, Instant};
// Sub-modules
pub mod algorithms;
pub mod metrics;
pub mod receiver;
pub mod report;
pub mod sender;
// Re-exports
pub use algorithms::{
DualEwma, JitterEstimator, OwdTrendDetector, SpinBitState, SrttEstimator, compute_etx,
};
pub use metrics::MmpMetrics;
pub use receiver::ReceiverState;
pub use report::{ReceiverReport, SenderReport};
pub use sender::SenderState;
// Session-layer re-exports
// MmpSessionState and PathMtuState are defined in this file
// ============================================================================
// Constants
// ============================================================================
/// SenderReport body size (after msg_type byte): 3 reserved + 44 payload = 47.
pub const SENDER_REPORT_BODY_SIZE: usize = 47;
/// ReceiverReport body size (after msg_type byte): 3 reserved + 64 payload = 67.
pub const RECEIVER_REPORT_BODY_SIZE: usize = 67;
/// SenderReport total wire size including inner header: 5 + 47 = 52.
pub const SENDER_REPORT_WIRE_SIZE: usize = 52;
/// ReceiverReport total wire size including inner header: 5 + 67 = 72.
pub const RECEIVER_REPORT_WIRE_SIZE: usize = 72;
// --- EWMA parameters (as shift amounts for integer arithmetic) ---
/// Jitter EWMA: α = 1/16 (RFC 3550 §6.4.1).
pub const JITTER_ALPHA_SHIFT: u32 = 4;
/// SRTT: α = 1/8 (Jacobson, RFC 6298).
pub const SRTT_ALPHA_SHIFT: u32 = 3;
/// RTTVAR: β = 1/4 (Jacobson, RFC 6298).
pub const RTTVAR_BETA_SHIFT: u32 = 2;
/// Dual EWMA short-term: α = 1/4.
pub const EWMA_SHORT_ALPHA: f64 = 0.25;
/// Dual EWMA long-term: α = 1/32.
pub const EWMA_LONG_ALPHA: f64 = 1.0 / 32.0;
// --- Timing defaults (milliseconds) ---
/// Default report interval before SRTT is available (cold start).
pub const DEFAULT_COLD_START_INTERVAL_MS: u64 = 200;
/// Minimum report interval (SRTT clamp floor).
///
/// Raised from 100ms to 1000ms: parent re-evaluation runs every 60s,
/// so 60 samples/cycle is more than sufficient for EWMA convergence (~10).
/// The cold-start phase uses `DEFAULT_COLD_START_INTERVAL_MS` (200ms) for
/// fast initial SRTT convergence before transitioning to this floor.
pub const MIN_REPORT_INTERVAL_MS: u64 = 1_000;
/// Maximum report interval (SRTT clamp ceiling).
pub const MAX_REPORT_INTERVAL_MS: u64 = 5_000;
/// Number of SRTT samples before transitioning from cold-start to normal floor.
///
/// During cold-start, report intervals use `DEFAULT_COLD_START_INTERVAL_MS` as
/// the floor to gather SRTT samples quickly. After this many updates, the floor
/// switches to `MIN_REPORT_INTERVAL_MS`.
pub const COLD_START_SAMPLES: u32 = 5;
/// Default OWD ring buffer capacity.
pub const DEFAULT_OWD_WINDOW_SIZE: usize = 32;
/// Default operator log interval in seconds.
pub const DEFAULT_LOG_INTERVAL_SECS: u64 = 30;
// --- Session-layer timing defaults ---
// Session reports are routed end-to-end (bandwidth cost on every transit link),
// so intervals are higher than link-layer.
/// Session-layer minimum report interval.
pub const MIN_SESSION_REPORT_INTERVAL_MS: u64 = 500;
/// Session-layer maximum report interval.
pub const MAX_SESSION_REPORT_INTERVAL_MS: u64 = 10_000;
/// Session-layer cold-start report interval (before SRTT is available).
pub const SESSION_COLD_START_INTERVAL_MS: u64 = 1_000;
// ============================================================================
// Operating Mode
// ============================================================================
/// MMP operating mode.
#[derive(Debug, Default, Clone, Copy, PartialEq, Eq, Serialize, Deserialize)]
#[serde(rename_all = "lowercase")]
pub enum MmpMode {
/// Sender + receiver reports at RTT-adaptive intervals. Maximum fidelity.
#[default]
Full,
/// Receiver reports only. Loss inferred from counter gaps.
Lightweight,
/// Spin bit + CE echo only. No reports exchanged.
Minimal,
}
impl fmt::Display for MmpMode {
fn fmt(&self, f: &mut fmt::Formatter<'_>) -> fmt::Result {
match self {
MmpMode::Full => write!(f, "full"),
MmpMode::Lightweight => write!(f, "lightweight"),
MmpMode::Minimal => write!(f, "minimal"),
}
}
}
// ============================================================================
// Configuration
// ============================================================================
/// MMP configuration (`node.mmp.*`).
#[derive(Debug, Clone, Serialize, Deserialize)]
pub struct MmpConfig {
/// Operating mode (`node.mmp.mode`).
#[serde(default)]
pub mode: MmpMode,
/// Periodic operator log interval in seconds (`node.mmp.log_interval_secs`).
#[serde(default = "MmpConfig::default_log_interval_secs")]
pub log_interval_secs: u64,
/// OWD trend ring buffer size (`node.mmp.owd_window_size`).
#[serde(default = "MmpConfig::default_owd_window_size")]
pub owd_window_size: usize,
}
impl Default for MmpConfig {
fn default() -> Self {
Self {
mode: MmpMode::default(),
log_interval_secs: DEFAULT_LOG_INTERVAL_SECS,
owd_window_size: DEFAULT_OWD_WINDOW_SIZE,
}
}
}
impl MmpConfig {
fn default_log_interval_secs() -> u64 {
DEFAULT_LOG_INTERVAL_SECS
}
fn default_owd_window_size() -> usize {
DEFAULT_OWD_WINDOW_SIZE
}
}
// ============================================================================
// Per-Peer MMP State
// ============================================================================
/// Combined MMP state for a single peer link.
///
/// Wraps sender, receiver, metrics, and spin bit state. One instance
/// per `ActivePeer`.
pub struct MmpPeerState {
pub sender: SenderState,
pub receiver: ReceiverState,
pub metrics: MmpMetrics,
pub spin_bit: SpinBitState,
mode: MmpMode,
log_interval: Duration,
last_log_time: Option<Instant>,
}
impl MmpPeerState {
/// Create MMP state for a new peer link.
///
/// `is_initiator`: true if this node initiated the Noise handshake
/// (determines spin bit role).
pub fn new(config: &MmpConfig, is_initiator: bool) -> Self {
Self {
sender: SenderState::new(),
receiver: ReceiverState::new(config.owd_window_size),
metrics: MmpMetrics::new(),
spin_bit: SpinBitState::new(is_initiator),
mode: config.mode,
log_interval: Duration::from_secs(config.log_interval_secs),
last_log_time: None,
}
}
/// Reset counter-dependent state for rekey cutover.
pub fn reset_for_rekey(&mut self, now: Instant) {
self.receiver.reset_for_rekey(now);
self.metrics.reset_for_rekey();
}
/// Current operating mode.
pub fn mode(&self) -> MmpMode {
self.mode
}
/// Check if it's time to emit a periodic metrics log.
pub fn should_log(&self, now: Instant) -> bool {
match self.last_log_time {
None => true,
Some(last) => now.duration_since(last) >= self.log_interval,
}
}
/// Mark that a periodic log was emitted.
pub fn mark_logged(&mut self, now: Instant) {
self.last_log_time = Some(now);
}
}
// ============================================================================
// Per-Session MMP State (session-layer instantiation)
// ============================================================================
/// Combined MMP state for a single end-to-end session.
///
/// Wraps sender, receiver, metrics, spin bit, and path MTU state.
/// One instance per established `SessionEntry`.
pub struct MmpSessionState {
pub sender: SenderState,
pub receiver: ReceiverState,
pub metrics: MmpMetrics,
pub spin_bit: SpinBitState,
mode: MmpMode,
log_interval: Duration,
last_log_time: Option<Instant>,
pub path_mtu: PathMtuState,
}
impl MmpSessionState {
/// Create MMP state for a new session.
///
/// `is_initiator`: true if this node initiated the Noise handshake
/// (determines spin bit role).
pub fn new(config: &crate::config::SessionMmpConfig, is_initiator: bool) -> Self {
Self {
sender: SenderState::new_with_cold_start(SESSION_COLD_START_INTERVAL_MS),
receiver: ReceiverState::new_with_cold_start(
config.owd_window_size,
SESSION_COLD_START_INTERVAL_MS,
),
metrics: MmpMetrics::new(),
spin_bit: SpinBitState::new(is_initiator),
mode: config.mode,
log_interval: Duration::from_secs(config.log_interval_secs),
last_log_time: None,
path_mtu: PathMtuState::new(),
}
}
/// Reset counter-dependent state for rekey cutover.
pub fn reset_for_rekey(&mut self, now: Instant) {
self.receiver.reset_for_rekey(now);
self.metrics.reset_for_rekey();
}
/// Current operating mode.
pub fn mode(&self) -> MmpMode {
self.mode
}
/// Check if it's time to emit a periodic metrics log.
pub fn should_log(&self, now: Instant) -> bool {
match self.last_log_time {
None => true,
Some(last) => now.duration_since(last) >= self.log_interval,
}
}
/// Mark that a periodic log was emitted.
pub fn mark_logged(&mut self, now: Instant) {
self.last_log_time = Some(now);
}
}
impl Debug for MmpSessionState {
fn fmt(&self, f: &mut fmt::Formatter<'_>) -> fmt::Result {
f.debug_struct("MmpSessionState")
.field("mode", &self.mode)
.field("path_mtu", &self.path_mtu.current_mtu())
.finish_non_exhaustive()
}
}
// ============================================================================
// Path MTU State (session-layer only)
// ============================================================================
/// Path MTU tracking for a single session.
///
/// Destination side: observes `path_mtu` from incoming SessionDatagram envelopes
/// and generates PathMtuNotification messages back to the source.
///
/// Source side: applies received PathMtuNotification to limit outbound datagram
/// size. Decrease is immediate; increase requires 3 consecutive notifications.
pub struct PathMtuState {
/// Current effective path MTU (what we use for sending).
current_mtu: u16,
/// Last observed path MTU from incoming datagrams (destination-side).
last_observed_mtu: u16,
/// Whether the observed MTU has changed since the last notification.
observed_changed: bool,
/// Last time a PathMtuNotification was sent.
last_notification_time: Option<Instant>,
/// Notification interval: max(10s, 5 * SRTT). Default 10s.
notification_interval: Duration,
/// For source-side increase tracking: consecutive higher-value notifications.
consecutive_increase_count: u8,
/// Time of the first notification in the current increase sequence.
first_increase_time: Option<Instant>,
/// The MTU value being proposed for increase.
pending_increase_mtu: u16,
}
impl PathMtuState {
/// Create path MTU state with no initial measurement.
pub fn new() -> Self {
Self {
current_mtu: u16::MAX,
last_observed_mtu: u16::MAX,
observed_changed: false,
last_notification_time: None,
notification_interval: Duration::from_secs(10),
consecutive_increase_count: 0,
first_increase_time: None,
pending_increase_mtu: 0,
}
}
/// Current effective path MTU (source-side, for sending).
pub fn current_mtu(&self) -> u16 {
self.current_mtu
}
/// Last observed incoming path MTU (destination-side).
pub fn last_observed_mtu(&self) -> u16 {
self.last_observed_mtu
}
/// Update notification interval from SRTT: max(10s, 5 * SRTT).
pub fn update_interval_from_srtt(&mut self, srtt_ms: f64) {
let five_srtt = Duration::from_millis((srtt_ms * 5.0) as u64);
self.notification_interval = five_srtt.max(Duration::from_secs(10));
}
/// Seed source-side current_mtu from outbound transport MTU.
///
/// Called on each send. Only decreases (never increases) the current_mtu
/// so the destination's PathMtuNotification can still raise it later.
/// Ensures current_mtu doesn't stay at u16::MAX before any notification
/// arrives from the destination.
pub fn seed_source_mtu(&mut self, outbound_mtu: u16) {
if outbound_mtu < self.current_mtu {
self.current_mtu = outbound_mtu;
}
}
// --- Destination side ---
/// Observe the path_mtu from an incoming SessionDatagram envelope.
///
/// Called on the destination (receiver) side for every session message.
pub fn observe_incoming_mtu(&mut self, path_mtu: u16) {
if path_mtu != self.last_observed_mtu {
self.observed_changed = true;
self.last_observed_mtu = path_mtu;
}
}
/// Check if a PathMtuNotification should be sent.
///
/// Send on first measurement, on decrease (immediate), or periodic
/// confirmation at the notification interval.
pub fn should_send_notification(&self, now: Instant) -> bool {
if self.last_observed_mtu == u16::MAX {
return false; // No measurement yet
}
match self.last_notification_time {
None => true, // First measurement
Some(last) => {
// Immediate on decrease
if self.observed_changed && self.last_observed_mtu < self.current_mtu {
return true;
}
// Periodic confirmation
now.duration_since(last) >= self.notification_interval
}
}
}
/// Build a PathMtuNotification from current state.
///
/// Returns the path_mtu value to send. Caller handles encoding.
pub fn build_notification(&mut self, now: Instant) -> Option<u16> {
if self.last_observed_mtu == u16::MAX {
return None;
}
self.last_notification_time = Some(now);
self.observed_changed = false;
Some(self.last_observed_mtu)
}
// --- Source side ---
/// Apply a received PathMtuNotification.
///
/// - Decrease: immediate (take the lower value).
/// - Increase: require 3 consecutive notifications with the same higher
/// value, spanning at least 2 * notification_interval.
///
/// Returns `true` if the effective MTU changed.
pub fn apply_notification(&mut self, reported_mtu: u16, now: Instant) -> bool {
if reported_mtu < self.current_mtu {
// Decrease: immediate
self.current_mtu = reported_mtu;
self.consecutive_increase_count = 0;
self.first_increase_time = None;
return true;
}
if reported_mtu > self.current_mtu {
// Increase: track consecutive notifications
if reported_mtu == self.pending_increase_mtu {
self.consecutive_increase_count += 1;
} else {
// Different value: reset sequence
self.pending_increase_mtu = reported_mtu;
self.consecutive_increase_count = 1;
self.first_increase_time = Some(now);
}
// Accept increase after 3 consecutive spanning 2 * interval
if self.consecutive_increase_count >= 3
&& let Some(first_time) = self.first_increase_time
{
let required = self.notification_interval * 2;
if now.duration_since(first_time) >= required {
self.current_mtu = reported_mtu;
self.consecutive_increase_count = 0;
self.first_increase_time = None;
return true;
}
}
}
// No change (equal or increase not yet confirmed)
false
}
}
impl Default for PathMtuState {
fn default() -> Self {
Self::new()
}
}
impl Debug for MmpPeerState {
fn fmt(&self, f: &mut fmt::Formatter<'_>) -> fmt::Result {
f.debug_struct("MmpPeerState")
.field("mode", &self.mode)
.finish_non_exhaustive()
}
}
// ============================================================================
// Tests
// ============================================================================
#[cfg(test)]
mod tests {
use super::*;
#[test]
fn test_mode_default() {
assert_eq!(MmpMode::default(), MmpMode::Full);
}
#[test]
fn test_mode_display() {
assert_eq!(MmpMode::Full.to_string(), "full");
assert_eq!(MmpMode::Lightweight.to_string(), "lightweight");
assert_eq!(MmpMode::Minimal.to_string(), "minimal");
}
#[test]
fn test_mode_serde_roundtrip() {
let yaml = "full";
let mode: MmpMode = serde_yaml::from_str(yaml).unwrap();
assert_eq!(mode, MmpMode::Full);
let yaml = "lightweight";
let mode: MmpMode = serde_yaml::from_str(yaml).unwrap();
assert_eq!(mode, MmpMode::Lightweight);
let yaml = "minimal";
let mode: MmpMode = serde_yaml::from_str(yaml).unwrap();
assert_eq!(mode, MmpMode::Minimal);
}
#[test]
fn test_config_default() {
let config = MmpConfig::default();
assert_eq!(config.mode, MmpMode::Full);
assert_eq!(config.log_interval_secs, 30);
assert_eq!(config.owd_window_size, 32);
}
#[test]
fn test_config_yaml_parse() {
let yaml = r#"
mode: lightweight
log_interval_secs: 60
owd_window_size: 48
"#;
let config: MmpConfig = serde_yaml::from_str(yaml).unwrap();
assert_eq!(config.mode, MmpMode::Lightweight);
assert_eq!(config.log_interval_secs, 60);
assert_eq!(config.owd_window_size, 48);
}
#[test]
fn test_config_yaml_partial() {
let yaml = "mode: minimal";
let config: MmpConfig = serde_yaml::from_str(yaml).unwrap();
assert_eq!(config.mode, MmpMode::Minimal);
assert_eq!(config.log_interval_secs, DEFAULT_LOG_INTERVAL_SECS);
assert_eq!(config.owd_window_size, DEFAULT_OWD_WINDOW_SIZE);
}
}
src/mmp/report.rs
-385
View File
@@ -1,385 +0,0 @@
//! MMP report wire format: SenderReport and ReceiverReport.
//!
//! Serialization and deserialization for the two report types exchanged
//! between link-layer peers. Wire format follows the MMP design doc.
use crate::protocol::ProtocolError;
// ============================================================================
// SenderReport (msg_type 0x01, 48-byte body including type byte)
// ============================================================================
/// Link-layer sender report.
///
/// Wire layout (48 bytes total, sent as link message):
/// ```text
/// [0] msg_type = 0x01
/// [1-3] reserved (zero)
/// [4-11] interval_start_counter: u64 LE
/// [12-19] interval_end_counter: u64 LE
/// [20-23] interval_start_timestamp: u32 LE
/// [24-27] interval_end_timestamp: u32 LE
/// [28-31] interval_bytes_sent: u32 LE
/// [32-39] cumulative_packets_sent: u64 LE
/// [40-47] cumulative_bytes_sent: u64 LE
/// ```
#[derive(Debug, Clone, PartialEq, Eq)]
pub struct SenderReport {
pub interval_start_counter: u64,
pub interval_end_counter: u64,
pub interval_start_timestamp: u32,
pub interval_end_timestamp: u32,
pub interval_bytes_sent: u32,
pub cumulative_packets_sent: u64,
pub cumulative_bytes_sent: u64,
}
/// ReceiverReport (msg_type 0x02, 68-byte body including type byte)
///
/// Wire layout (68 bytes total, sent as link message):
/// ```text
/// [0] msg_type = 0x02
/// [1-3] reserved (zero)
/// [4-11] highest_counter: u64 LE
/// [12-19] cumulative_packets_recv: u64 LE
/// [20-27] cumulative_bytes_recv: u64 LE
/// [28-31] timestamp_echo: u32 LE
/// [32-33] dwell_time: u16 LE
/// [34-35] max_burst_loss: u16 LE
/// [36-37] mean_burst_loss: u16 LE (u8.8 fixed-point)
/// [38-39] reserved: u16 LE
/// [40-43] jitter: u32 LE (microseconds)
/// [44-47] ecn_ce_count: u32 LE
/// [48-51] owd_trend: i32 LE (µs/s)
/// [52-55] burst_loss_count: u32 LE
/// [56-59] cumulative_reorder_count: u32 LE
/// [60-63] interval_packets_recv: u32 LE
/// [64-67] interval_bytes_recv: u32 LE
/// ```
#[derive(Debug, Clone, PartialEq, Eq)]
pub struct ReceiverReport {
pub highest_counter: u64,
pub cumulative_packets_recv: u64,
pub cumulative_bytes_recv: u64,
pub timestamp_echo: u32,
pub dwell_time: u16,
pub max_burst_loss: u16,
pub mean_burst_loss: u16,
pub jitter: u32,
pub ecn_ce_count: u32,
pub owd_trend: i32,
pub burst_loss_count: u32,
pub cumulative_reorder_count: u32,
pub interval_packets_recv: u32,
pub interval_bytes_recv: u32,
}
// Encode/decode will be implemented in Step 2.
impl SenderReport {
/// Encode to wire format (48 bytes: msg_type + 3 reserved + 44 payload).
pub fn encode(&self) -> Vec<u8> {
let mut buf = Vec::with_capacity(48);
buf.push(0x01); // msg_type
buf.extend_from_slice(&[0u8; 3]); // reserved
buf.extend_from_slice(&self.interval_start_counter.to_le_bytes());
buf.extend_from_slice(&self.interval_end_counter.to_le_bytes());
buf.extend_from_slice(&self.interval_start_timestamp.to_le_bytes());
buf.extend_from_slice(&self.interval_end_timestamp.to_le_bytes());
buf.extend_from_slice(&self.interval_bytes_sent.to_le_bytes());
buf.extend_from_slice(&self.cumulative_packets_sent.to_le_bytes());
buf.extend_from_slice(&self.cumulative_bytes_sent.to_le_bytes());
buf
}
/// Decode from payload after msg_type byte has been consumed.
///
/// `payload` starts at the reserved bytes (offset 1 in the wire format).
pub fn decode(payload: &[u8]) -> Result<Self, ProtocolError> {
if payload.len() < 47 {
return Err(ProtocolError::MessageTooShort {
expected: 47,
got: payload.len(),
});
}
// Skip 3 reserved bytes
let p = &payload[3..];
Ok(Self {
interval_start_counter: u64::from_le_bytes(p[0..8].try_into().unwrap()),
interval_end_counter: u64::from_le_bytes(p[8..16].try_into().unwrap()),
interval_start_timestamp: u32::from_le_bytes(p[16..20].try_into().unwrap()),
interval_end_timestamp: u32::from_le_bytes(p[20..24].try_into().unwrap()),
interval_bytes_sent: u32::from_le_bytes(p[24..28].try_into().unwrap()),
cumulative_packets_sent: u64::from_le_bytes(p[28..36].try_into().unwrap()),
cumulative_bytes_sent: u64::from_le_bytes(p[36..44].try_into().unwrap()),
})
}
}
impl ReceiverReport {
/// Encode to wire format (68 bytes: msg_type + 3 reserved + 64 payload).
pub fn encode(&self) -> Vec<u8> {
let mut buf = Vec::with_capacity(68);
buf.push(0x02); // msg_type
buf.extend_from_slice(&[0u8; 3]); // reserved
buf.extend_from_slice(&self.highest_counter.to_le_bytes());
buf.extend_from_slice(&self.cumulative_packets_recv.to_le_bytes());
buf.extend_from_slice(&self.cumulative_bytes_recv.to_le_bytes());
buf.extend_from_slice(&self.timestamp_echo.to_le_bytes());
buf.extend_from_slice(&self.dwell_time.to_le_bytes());
buf.extend_from_slice(&self.max_burst_loss.to_le_bytes());
buf.extend_from_slice(&self.mean_burst_loss.to_le_bytes());
buf.extend_from_slice(&[0u8; 2]); // reserved
buf.extend_from_slice(&self.jitter.to_le_bytes());
buf.extend_from_slice(&self.ecn_ce_count.to_le_bytes());
buf.extend_from_slice(&self.owd_trend.to_le_bytes());
buf.extend_from_slice(&self.burst_loss_count.to_le_bytes());
buf.extend_from_slice(&self.cumulative_reorder_count.to_le_bytes());
buf.extend_from_slice(&self.interval_packets_recv.to_le_bytes());
buf.extend_from_slice(&self.interval_bytes_recv.to_le_bytes());
buf
}
/// Decode from payload after msg_type byte has been consumed.
///
/// `payload` starts at the reserved bytes (offset 1 in the wire format).
pub fn decode(payload: &[u8]) -> Result<Self, ProtocolError> {
if payload.len() < 67 {
return Err(ProtocolError::MessageTooShort {
expected: 67,
got: payload.len(),
});
}
// Skip 3 reserved bytes
let p = &payload[3..];
Ok(Self {
highest_counter: u64::from_le_bytes(p[0..8].try_into().unwrap()),
cumulative_packets_recv: u64::from_le_bytes(p[8..16].try_into().unwrap()),
cumulative_bytes_recv: u64::from_le_bytes(p[16..24].try_into().unwrap()),
timestamp_echo: u32::from_le_bytes(p[24..28].try_into().unwrap()),
dwell_time: u16::from_le_bytes(p[28..30].try_into().unwrap()),
max_burst_loss: u16::from_le_bytes(p[30..32].try_into().unwrap()),
mean_burst_loss: u16::from_le_bytes(p[32..34].try_into().unwrap()),
// skip 2 reserved bytes at p[34..36]
jitter: u32::from_le_bytes(p[36..40].try_into().unwrap()),
ecn_ce_count: u32::from_le_bytes(p[40..44].try_into().unwrap()),
owd_trend: i32::from_le_bytes(p[44..48].try_into().unwrap()),
burst_loss_count: u32::from_le_bytes(p[48..52].try_into().unwrap()),
cumulative_reorder_count: u32::from_le_bytes(p[52..56].try_into().unwrap()),
interval_packets_recv: u32::from_le_bytes(p[56..60].try_into().unwrap()),
interval_bytes_recv: u32::from_le_bytes(p[60..64].try_into().unwrap()),
})
}
}
// ============================================================================
// Conversions between link-layer and session-layer report types
// ============================================================================
use crate::protocol::{SessionReceiverReport, SessionSenderReport};
impl From<&SenderReport> for SessionSenderReport {
fn from(r: &SenderReport) -> Self {
Self {
interval_start_counter: r.interval_start_counter,
interval_end_counter: r.interval_end_counter,
interval_start_timestamp: r.interval_start_timestamp,
interval_end_timestamp: r.interval_end_timestamp,
interval_bytes_sent: r.interval_bytes_sent,
cumulative_packets_sent: r.cumulative_packets_sent,
cumulative_bytes_sent: r.cumulative_bytes_sent,
}
}
}
impl From<&SessionSenderReport> for SenderReport {
fn from(r: &SessionSenderReport) -> Self {
Self {
interval_start_counter: r.interval_start_counter,
interval_end_counter: r.interval_end_counter,
interval_start_timestamp: r.interval_start_timestamp,
interval_end_timestamp: r.interval_end_timestamp,
interval_bytes_sent: r.interval_bytes_sent,
cumulative_packets_sent: r.cumulative_packets_sent,
cumulative_bytes_sent: r.cumulative_bytes_sent,
}
}
}
impl From<&ReceiverReport> for SessionReceiverReport {
fn from(r: &ReceiverReport) -> Self {
Self {
highest_counter: r.highest_counter,
cumulative_packets_recv: r.cumulative_packets_recv,
cumulative_bytes_recv: r.cumulative_bytes_recv,
timestamp_echo: r.timestamp_echo,
dwell_time: r.dwell_time,
max_burst_loss: r.max_burst_loss,
mean_burst_loss: r.mean_burst_loss,
jitter: r.jitter,
ecn_ce_count: r.ecn_ce_count,
owd_trend: r.owd_trend,
burst_loss_count: r.burst_loss_count,
cumulative_reorder_count: r.cumulative_reorder_count,
interval_packets_recv: r.interval_packets_recv,
interval_bytes_recv: r.interval_bytes_recv,
}
}
}
impl From<&SessionReceiverReport> for ReceiverReport {
fn from(r: &SessionReceiverReport) -> Self {
Self {
highest_counter: r.highest_counter,
cumulative_packets_recv: r.cumulative_packets_recv,
cumulative_bytes_recv: r.cumulative_bytes_recv,
timestamp_echo: r.timestamp_echo,
dwell_time: r.dwell_time,
max_burst_loss: r.max_burst_loss,
mean_burst_loss: r.mean_burst_loss,
jitter: r.jitter,
ecn_ce_count: r.ecn_ce_count,
owd_trend: r.owd_trend,
burst_loss_count: r.burst_loss_count,
cumulative_reorder_count: r.cumulative_reorder_count,
interval_packets_recv: r.interval_packets_recv,
interval_bytes_recv: r.interval_bytes_recv,
}
}
}
// ============================================================================
// Tests
// ============================================================================
#[cfg(test)]
mod tests {
use super::*;
fn sample_sender_report() -> SenderReport {
SenderReport {
interval_start_counter: 100,
interval_end_counter: 200,
interval_start_timestamp: 5000,
interval_end_timestamp: 6000,
interval_bytes_sent: 50_000,
cumulative_packets_sent: 10_000,
cumulative_bytes_sent: 5_000_000,
}
}
fn sample_receiver_report() -> ReceiverReport {
ReceiverReport {
highest_counter: 195,
cumulative_packets_recv: 9_500,
cumulative_bytes_recv: 4_750_000,
timestamp_echo: 5900,
dwell_time: 5,
max_burst_loss: 3,
mean_burst_loss: 384, // 1.5 in u8.8
jitter: 1200,
ecn_ce_count: 0,
owd_trend: -50,
burst_loss_count: 2,
cumulative_reorder_count: 10,
interval_packets_recv: 95,
interval_bytes_recv: 47_500,
}
}
#[test]
fn test_sender_report_encode_size() {
let sr = sample_sender_report();
let encoded = sr.encode();
assert_eq!(encoded.len(), 48);
assert_eq!(encoded[0], 0x01); // msg_type
}
#[test]
fn test_sender_report_roundtrip() {
let sr = sample_sender_report();
let encoded = sr.encode();
// decode expects payload after msg_type
let decoded = SenderReport::decode(&encoded[1..]).unwrap();
assert_eq!(sr, decoded);
}
#[test]
fn test_sender_report_too_short() {
let result = SenderReport::decode(&[0u8; 10]);
assert!(result.is_err());
}
#[test]
fn test_receiver_report_encode_size() {
let rr = sample_receiver_report();
let encoded = rr.encode();
assert_eq!(encoded.len(), 68);
assert_eq!(encoded[0], 0x02); // msg_type
}
#[test]
fn test_receiver_report_roundtrip() {
let rr = sample_receiver_report();
let encoded = rr.encode();
// decode expects payload after msg_type
let decoded = ReceiverReport::decode(&encoded[1..]).unwrap();
assert_eq!(rr, decoded);
}
#[test]
fn test_receiver_report_too_short() {
let result = ReceiverReport::decode(&[0u8; 10]);
assert!(result.is_err());
}
#[test]
fn test_sender_report_zero_values() {
let sr = SenderReport {
interval_start_counter: 0,
interval_end_counter: 0,
interval_start_timestamp: 0,
interval_end_timestamp: 0,
interval_bytes_sent: 0,
cumulative_packets_sent: 0,
cumulative_bytes_sent: 0,
};
let encoded = sr.encode();
let decoded = SenderReport::decode(&encoded[1..]).unwrap();
assert_eq!(sr, decoded);
}
#[test]
fn test_receiver_report_max_values() {
let rr = ReceiverReport {
highest_counter: u64::MAX,
cumulative_packets_recv: u64::MAX,
cumulative_bytes_recv: u64::MAX,
timestamp_echo: u32::MAX,
dwell_time: u16::MAX,
max_burst_loss: u16::MAX,
mean_burst_loss: u16::MAX,
jitter: u32::MAX,
ecn_ce_count: u32::MAX,
owd_trend: i32::MAX,
burst_loss_count: u32::MAX,
cumulative_reorder_count: u32::MAX,
interval_packets_recv: u32::MAX,
interval_bytes_recv: u32::MAX,
};
let encoded = rr.encode();
let decoded = ReceiverReport::decode(&encoded[1..]).unwrap();
assert_eq!(rr, decoded);
}
#[test]
fn test_receiver_report_negative_owd_trend() {
let rr = ReceiverReport {
owd_trend: -12345,
..sample_receiver_report()
};
let encoded = rr.encode();
let decoded = ReceiverReport::decode(&encoded[1..]).unwrap();
assert_eq!(decoded.owd_trend, -12345);
}
}
src/mmp/sender.rs
-418
View File
@@ -1,418 +0,0 @@
//! MMP sender state machine.
//!
//! Tracks what this node has sent to a specific peer and produces
//! SenderReport messages on demand. One `SenderState` per active peer.
use std::time::{Duration, Instant};
use crate::mmp::report::SenderReport;
use crate::mmp::{
COLD_START_SAMPLES, DEFAULT_COLD_START_INTERVAL_MS, MAX_REPORT_INTERVAL_MS,
MIN_REPORT_INTERVAL_MS,
};
/// Per-peer sender-side MMP state.
///
/// Records cumulative and interval counters for every frame transmitted
/// to this peer. Produces `SenderReport` snapshots on demand.
pub struct SenderState {
// --- Cumulative (lifetime) ---
cumulative_packets_sent: u64,
cumulative_bytes_sent: u64,
// --- Current interval ---
interval_start_counter: u64,
interval_start_timestamp: u32,
interval_bytes_sent: u32,
/// Counter of the most recently sent frame.
last_counter: u64,
/// Timestamp of the most recently sent frame.
last_timestamp: u32,
/// Whether any frames have been sent in the current interval.
interval_has_data: bool,
// --- Report timing ---
last_report_time: Option<Instant>,
report_interval: Duration,
// --- Send failure backoff ---
/// Consecutive send failure count for backoff calculation.
consecutive_send_failures: u32,
// --- Cold-start tracking ---
/// Number of SRTT-based interval updates received.
srtt_sample_count: u32,
}
impl SenderState {
pub fn new() -> Self {
Self::new_with_cold_start(DEFAULT_COLD_START_INTERVAL_MS)
}
/// Create with a custom cold-start interval (ms).
///
/// Used by session-layer MMP which needs a longer initial interval
/// since reports consume bandwidth on every transit link.
pub fn new_with_cold_start(cold_start_ms: u64) -> Self {
Self {
cumulative_packets_sent: 0,
cumulative_bytes_sent: 0,
interval_start_counter: 0,
interval_start_timestamp: 0,
interval_bytes_sent: 0,
last_counter: 0,
last_timestamp: 0,
interval_has_data: false,
last_report_time: None,
report_interval: Duration::from_millis(cold_start_ms),
consecutive_send_failures: 0,
srtt_sample_count: 0,
}
}
/// Record a frame sent to this peer.
///
/// Called on the TX path for every encrypted link message.
/// `counter` is the AEAD nonce/counter, `timestamp` is the inner header
/// session-relative timestamp (ms), `bytes` is the wire payload size.
pub fn record_sent(&mut self, counter: u64, timestamp: u32, bytes: usize) {
if !self.interval_has_data {
self.interval_start_counter = counter;
self.interval_start_timestamp = timestamp;
self.interval_has_data = true;
}
self.last_counter = counter;
self.last_timestamp = timestamp;
self.interval_bytes_sent = self.interval_bytes_sent.saturating_add(bytes as u32);
self.cumulative_packets_sent += 1;
self.cumulative_bytes_sent += bytes as u64;
}
/// Build a SenderReport from current state and reset the interval.
///
/// Returns `None` if no frames have been sent since the last report.
pub fn build_report(&mut self, now: Instant) -> Option<SenderReport> {
if !self.interval_has_data {
return None;
}
let report = SenderReport {
interval_start_counter: self.interval_start_counter,
interval_end_counter: self.last_counter,
interval_start_timestamp: self.interval_start_timestamp,
interval_end_timestamp: self.last_timestamp,
interval_bytes_sent: self.interval_bytes_sent,
cumulative_packets_sent: self.cumulative_packets_sent,
cumulative_bytes_sent: self.cumulative_bytes_sent,
};
// Reset interval
self.interval_has_data = false;
self.interval_bytes_sent = 0;
self.last_report_time = Some(now);
Some(report)
}
/// Check if it's time to send a report.
///
/// When consecutive send failures have occurred, the effective interval
/// is multiplied by an exponential backoff factor (2^failures, capped at 32×).
pub fn should_send_report(&self, now: Instant) -> bool {
if !self.interval_has_data {
return false;
}
match self.last_report_time {
None => true, // Never sent a report — send immediately
Some(last) => {
let effective = self
.report_interval
.mul_f64(self.send_failure_backoff_multiplier());
now.duration_since(last) >= effective
}
}
}
/// Record a send failure. Returns the new consecutive failure count.
pub fn record_send_failure(&mut self) -> u32 {
self.consecutive_send_failures += 1;
self.consecutive_send_failures
}
/// Record a successful send. Returns the previous failure count (for summary logging).
pub fn record_send_success(&mut self) -> u32 {
let prev = self.consecutive_send_failures;
self.consecutive_send_failures = 0;
prev
}
/// Get the backoff multiplier based on consecutive failures.
///
/// Returns 1.0 for no failures, 2.0 for 1 failure, 4.0 for 2, ...
/// capped at 32.0 (5 failures).
pub fn send_failure_backoff_multiplier(&self) -> f64 {
if self.consecutive_send_failures == 0 {
1.0
} else {
2.0_f64.powi(self.consecutive_send_failures.min(5) as i32)
}
}
/// Update the report interval based on SRTT (link-layer defaults).
///
/// Sender reports at 2× SRTT clamped to [floor, MAX]. During cold-start
/// (first `COLD_START_SAMPLES` updates), the floor is the cold-start
/// interval (200ms) for fast SRTT convergence. After that, it rises to
/// `MIN_REPORT_INTERVAL_MS` (1000ms) for steady-state efficiency.
pub fn update_report_interval_from_srtt(&mut self, srtt_us: i64) {
self.srtt_sample_count = self.srtt_sample_count.saturating_add(1);
let floor = if self.srtt_sample_count <= COLD_START_SAMPLES {
DEFAULT_COLD_START_INTERVAL_MS
} else {
MIN_REPORT_INTERVAL_MS
};
self.update_report_interval_with_bounds(srtt_us, floor, MAX_REPORT_INTERVAL_MS);
}
/// Update the report interval based on SRTT with custom bounds.
///
/// Used by session-layer MMP which needs higher clamp values since
/// each report consumes bandwidth on every transit link.
pub fn update_report_interval_with_bounds(&mut self, srtt_us: i64, min_ms: u64, max_ms: u64) {
if srtt_us <= 0 {
return;
}
let interval_us = (srtt_us * 2) as u64;
let interval_ms = (interval_us / 1000).clamp(min_ms, max_ms);
self.report_interval = Duration::from_millis(interval_ms);
}
// --- Accessors ---
pub fn cumulative_packets_sent(&self) -> u64 {
self.cumulative_packets_sent
}
pub fn cumulative_bytes_sent(&self) -> u64 {
self.cumulative_bytes_sent
}
pub fn report_interval(&self) -> Duration {
self.report_interval
}
pub fn consecutive_send_failures(&self) -> u32 {
self.consecutive_send_failures
}
}
impl Default for SenderState {
fn default() -> Self {
Self::new()
}
}
// ============================================================================
// Tests
// ============================================================================
#[cfg(test)]
mod tests {
use super::*;
#[test]
fn test_new_sender_state() {
let s = SenderState::new();
assert_eq!(s.cumulative_packets_sent(), 0);
assert_eq!(s.cumulative_bytes_sent(), 0);
}
#[test]
fn test_record_sent() {
let mut s = SenderState::new();
s.record_sent(1, 100, 500);
s.record_sent(2, 200, 600);
assert_eq!(s.cumulative_packets_sent(), 2);
assert_eq!(s.cumulative_bytes_sent(), 1100);
}
#[test]
fn test_build_report_empty() {
let mut s = SenderState::new();
assert!(s.build_report(Instant::now()).is_none());
}
#[test]
fn test_build_report() {
let mut s = SenderState::new();
s.record_sent(10, 1000, 500);
s.record_sent(11, 1100, 600);
s.record_sent(12, 1200, 400);
let report = s.build_report(Instant::now()).unwrap();
assert_eq!(report.interval_start_counter, 10);
assert_eq!(report.interval_end_counter, 12);
assert_eq!(report.interval_start_timestamp, 1000);
assert_eq!(report.interval_end_timestamp, 1200);
assert_eq!(report.interval_bytes_sent, 1500);
assert_eq!(report.cumulative_packets_sent, 3);
assert_eq!(report.cumulative_bytes_sent, 1500);
}
#[test]
fn test_build_report_resets_interval() {
let mut s = SenderState::new();
s.record_sent(1, 100, 500);
let _ = s.build_report(Instant::now());
// Second report with no new data returns None
assert!(s.build_report(Instant::now()).is_none());
// New data starts a fresh interval
s.record_sent(2, 200, 300);
let report = s.build_report(Instant::now()).unwrap();
assert_eq!(report.interval_start_counter, 2);
assert_eq!(report.interval_bytes_sent, 300);
// Cumulative continues
assert_eq!(report.cumulative_packets_sent, 2);
assert_eq!(report.cumulative_bytes_sent, 800);
}
#[test]
fn test_should_send_report_no_data() {
let s = SenderState::new();
assert!(!s.should_send_report(Instant::now()));
}
#[test]
fn test_should_send_report_first_time() {
let mut s = SenderState::new();
s.record_sent(1, 100, 500);
assert!(s.should_send_report(Instant::now()));
}
#[test]
fn test_should_send_report_respects_interval() {
let mut s = SenderState::new();
let t0 = Instant::now();
s.record_sent(1, 100, 500);
let _ = s.build_report(t0);
s.record_sent(2, 200, 500);
// Immediately after report — should not send
assert!(!s.should_send_report(t0));
// After interval elapses
let t1 = t0 + s.report_interval() + Duration::from_millis(1);
assert!(s.should_send_report(t1));
}
#[test]
fn test_update_report_interval_cold_start() {
let mut s = SenderState::new();
// During cold-start, floor is 200ms (DEFAULT_COLD_START_INTERVAL_MS)
// 50ms RTT → 100ms sender interval (2× SRTT), clamped to cold-start floor 200ms
s.update_report_interval_from_srtt(50_000);
assert_eq!(s.report_interval(), Duration::from_millis(200));
// 500ms RTT → 1000ms sender interval (above cold-start floor)
s.update_report_interval_from_srtt(500_000);
assert_eq!(s.report_interval(), Duration::from_millis(1000));
}
#[test]
fn test_update_report_interval_after_cold_start() {
let mut s = SenderState::new();
// Burn through cold-start samples (COLD_START_SAMPLES = 5)
for _ in 0..COLD_START_SAMPLES {
s.update_report_interval_from_srtt(500_000);
}
// 6th sample: now in steady state, floor is MIN_REPORT_INTERVAL_MS (1000ms)
// 50ms RTT → 100ms sender interval (2× SRTT), clamped to 1000ms
s.update_report_interval_from_srtt(50_000);
assert_eq!(
s.report_interval(),
Duration::from_millis(MIN_REPORT_INTERVAL_MS)
);
// 3s RTT → 6s, clamped to max 5s
s.update_report_interval_from_srtt(3_000_000);
assert_eq!(
s.report_interval(),
Duration::from_millis(MAX_REPORT_INTERVAL_MS)
);
}
#[test]
fn test_backoff_multiplier_progression() {
let mut s = SenderState::new();
// No failures → multiplier 1.0
assert_eq!(s.send_failure_backoff_multiplier(), 1.0);
assert_eq!(s.consecutive_send_failures(), 0);
// Progressive failures: 2^1, 2^2, 2^3, 2^4, 2^5
let expected = [2.0, 4.0, 8.0, 16.0, 32.0];
for (i, &exp) in expected.iter().enumerate() {
let count = s.record_send_failure();
assert_eq!(count, (i + 1) as u32);
assert_eq!(s.send_failure_backoff_multiplier(), exp);
}
// Beyond 5 failures: stays capped at 32.0
s.record_send_failure(); // 6th
assert_eq!(s.send_failure_backoff_multiplier(), 32.0);
s.record_send_failure(); // 7th
assert_eq!(s.send_failure_backoff_multiplier(), 32.0);
}
#[test]
fn test_backoff_reset_on_success() {
let mut s = SenderState::new();
// Accumulate failures
s.record_send_failure();
s.record_send_failure();
s.record_send_failure();
assert_eq!(s.consecutive_send_failures(), 3);
assert_eq!(s.send_failure_backoff_multiplier(), 8.0);
// Success resets and returns previous count
let prev = s.record_send_success();
assert_eq!(prev, 3);
assert_eq!(s.consecutive_send_failures(), 0);
assert_eq!(s.send_failure_backoff_multiplier(), 1.0);
}
#[test]
fn test_backoff_success_with_no_prior_failures() {
let mut s = SenderState::new();
// Success with no failures returns 0
let prev = s.record_send_success();
assert_eq!(prev, 0);
assert_eq!(s.consecutive_send_failures(), 0);
}
#[test]
fn test_should_send_report_respects_backoff() {
let mut s = SenderState::new();
let t0 = Instant::now();
s.record_sent(1, 100, 500);
let _ = s.build_report(t0);
// Record a failure: multiplier becomes 2.0
s.record_send_failure();
s.record_sent(2, 200, 500);
// At 1× interval: should NOT send (backoff requires 2×)
let t1 = t0 + s.report_interval() + Duration::from_millis(1);
assert!(!s.should_send_report(t1));
// At 2× interval: should send
let t2 = t0 + s.report_interval() * 2 + Duration::from_millis(1);
assert!(s.should_send_report(t2));
}
}
src/node/bloom.rs
+7 -7
View File
@@ -4,12 +4,12 @@
//! including debounced propagation to peers.
use crate::NodeAddr;
use crate::bloom::BloomFilter;
use crate::protocol::FilterAnnounce;
use crate::proto::bloom::BloomFilter;
use crate::proto::bloom::FilterAnnounce;
use super::reject::BloomReject;
use super::{Node, NodeError};
use std::collections::HashMap;
use std::collections::BTreeMap;
use tracing::{debug, warn};
impl Node {
@@ -17,8 +17,8 @@ impl Node {
///
/// Returns a map of (peer_node_addr -> filter) for peers that
/// have sent us a FilterAnnounce.
pub(super) fn peer_inbound_filters(&self) -> HashMap<NodeAddr, BloomFilter> {
let mut filters = HashMap::new();
pub(super) fn peer_inbound_filters(&self) -> BTreeMap<NodeAddr, BloomFilter> {
let mut filters = BTreeMap::new();
for (addr, peer) in &self.peers {
if self.is_tree_peer(addr)
&& let Some(filter) = peer.inbound_filter()
@@ -79,7 +79,7 @@ impl Node {
// operator to see one clear message, not spam.
let max_fpr = self.config().node.bloom.max_inbound_fpr;
let out_fill = sent_filter.fill_ratio();
let out_fpr = out_fill.powi(sent_filter.hash_count() as i32);
let out_fpr = sent_filter.fpr();
if out_fpr > max_fpr {
let now = std::time::Instant::now();
let should_warn = self
@@ -221,7 +221,7 @@ impl Node {
// to wipe a victim's contribution to aggregation.
let max_fpr = self.config().node.bloom.max_inbound_fpr;
let fill = announce.filter.fill_ratio();
let fpr = fill.powi(announce.filter.hash_count() as i32);
let fpr = announce.filter.fpr();
if fpr > max_fpr {
self.metrics()
.bloom
src/node/handlers/connected_udp.rs → src/node/dataplane/connected_udp.rs
+18 -16
View File
@@ -45,12 +45,10 @@ impl Node {
/// (e.g. only non-UDP transports). Enabled on Linux and macOS:
/// both kernels route a matching peer 5-tuple to the connected
/// socket when it shares the wildcard listen port via SO_REUSEPORT.
/// Only compiled on Linux/macOS — the sole caller (the rx_loop tick) is
/// gated the same way, so on other targets (android) there is nothing to do.
#[cfg(any(target_os = "linux", target_os = "macos"))]
pub(in crate::node) async fn activate_connected_udp_sessions(&mut self) {
#[cfg(not(any(target_os = "linux", target_os = "macos")))]
{
// No-op on platforms without the connected-UDP fast path.
}
#[cfg(any(target_os = "linux", target_os = "macos"))]
{
if !connected_udp_enabled() {
return;
@@ -142,20 +140,24 @@ impl Node {
(peer_sa, local, recv_buf, send_buf, tx)
};
// Open the connected socket on the kernel side.
let socket = std::sync::Arc::new(
crate::transport::udp::connected_peer::ConnectedPeerSocket::open(
local_addr,
peer_socket_addr,
recv_buf,
send_buf,
)
.map_err(|e| format!("ConnectedPeerSocket::open: {e}"))?,
);
// Open the connected socket on the kernel side, then adopt the
// fd into the owning handle.
let owned = crate::transport::udp::open_connected_fd(
local_addr,
peer_socket_addr,
recv_buf,
send_buf,
)
.map_err(|e| format!("open_connected_fd: {e}"))?;
let socket = std::sync::Arc::new(crate::peer::connected_udp::ConnectedPeerSocket::from_fd(
owned,
peer_socket_addr,
local_addr,
));
// Spawn the drain thread. It feeds `packet_tx` exactly like
// the wildcard listen socket — rx_loop dispatches identically.
let drain = crate::transport::udp::peer_drain::PeerRecvDrain::spawn(
let drain = crate::peer::connected_udp::PeerRecvDrain::spawn(
socket.clone(),
transport_id,
peer_socket_addr,
src/node/handlers/dispatch.rs → src/node/dataplane/dispatch.rs
+11 -2
View File
@@ -73,7 +73,7 @@ impl Node {
/// entries — other removal paths (link-dead, decrypt failure, peer
/// restart) all schedule reconnect.
pub(in crate::node) fn handle_disconnect(&mut self, from: &NodeAddr, payload: &[u8]) {
let disconnect = match crate::protocol::Disconnect::decode(payload) {
let disconnect = match crate::proto::fmp::Disconnect::decode(payload) {
Ok(msg) => msg,
Err(e) => {
debug!(from = %self.peer_display_name(from), error = %e, "Malformed disconnect message");
@@ -93,7 +93,7 @@ impl Node {
.duration_since(std::time::UNIX_EPOCH)
.map(|d| d.as_millis() as u64)
.unwrap_or(0);
self.schedule_reconnect(addr, now_ms);
self.note_link_dead(addr, now_ms);
}
/// Remove an active peer and clean up all associated state.
@@ -187,6 +187,15 @@ impl Node {
// Remove link and address mapping
self.remove_link(&link_id);
// Bound `peer_machines`: drop this peer's machine
// entry, keyed by the `link_id` derived above BEFORE the `peers` removal.
// This cleans up the OLD peer's machine on an inbound restart and prevents
// unbounded growth on the establish success path. NEUTRAL: nothing on the
// live path reads `peer_machines` except the establish executor, which only
// ever touches the in-flight establish's (distinct) `link_id`; no reader
// depends on a stale entry, so removal changes no behavior — it only bounds
// the map.
self.remove_peer_machine(link_id);
if let Some(transport_id) = transport_id {
self.cleanup_bootstrap_transport_if_unused(transport_id);
}
src/node/handlers/encrypted.rs → src/node/dataplane/encrypted.rs
+13 -12
View File
@@ -1,10 +1,11 @@
//! Encrypted frame handling (hot path).
use crate::node::Node;
use crate::node::wire::{EncryptedHeader, FLAG_CE, FLAG_KEY_EPOCH, FLAG_SP, strip_inner_header};
use crate::noise::NoiseError;
use crate::proto::fmp::wire::{
EncryptedHeader, FLAG_CE, FLAG_KEY_EPOCH, FLAG_SP, strip_inner_header,
};
use crate::transport::ReceivedPacket;
use std::time::Instant;
use tracing::{debug, trace, warn};
/// Force-remove a peer after this many consecutive decryption failures.
@@ -171,7 +172,7 @@ impl Node {
#[cfg(unix)]
{
let cache_key = (packet.transport_id, header.receiver_idx.as_u32());
if let Some(workers) = self.decrypt_workers.as_ref().cloned()
if let Some(workers) = self.supervisor.decrypt_workers.as_ref().cloned()
&& self.decrypt_registered_sessions.contains(&cache_key)
{
let job = crate::node::decrypt_worker::DecryptJob {
@@ -259,7 +260,7 @@ impl Node {
};
// MMP per-frame processing and statistics
let now = Instant::now();
let now_ms = crate::time::mono_ms();
let ce_flag = header.flags & FLAG_CE != 0;
let sp_flag = header.flags & FLAG_SP != 0;
@@ -270,9 +271,9 @@ impl Node {
timestamp,
packet.data.len(),
ce_flag,
now,
now_ms,
);
let _spin_rtt = mmp.spin_bit.rx_observe(sp_flag, header.counter, now);
let _spin_rtt = mmp.spin_bit.rx_observe(sp_flag, header.counter, now_ms);
}
peer.set_current_addr(packet.transport_id, packet.remote_addr.clone());
peer.link_stats_mut()
@@ -355,7 +356,7 @@ impl Node {
} else {
return;
};
let now = Instant::now();
let now_ms = crate::time::mono_ms();
let mut address_changed = false;
if let Some(peer) = self.peers.get_mut(node_addr) {
peer.reset_decrypt_failures();
@@ -365,8 +366,8 @@ impl Node {
peer.touch(packet_timestamp_ms);
if let Some(mmp) = peer.mmp_mut() {
mmp.receiver
.record_recv(fmp_counter, inner_ts, packet_len, ce_flag, now);
let _spin_rtt = mmp.spin_bit.rx_observe(sp_flag, fmp_counter, now);
.record_recv(fmp_counter, inner_ts, packet_len, ce_flag, now_ms);
let _spin_rtt = mmp.spin_bit.rx_observe(sp_flag, fmp_counter, now_ms);
}
}
// Address rotation invalidates the per-peer connect()-ed UDP
@@ -450,7 +451,7 @@ impl Node {
/// black-hole the session.
#[cfg(unix)]
pub(in crate::node) fn register_decrypt_worker_session(&mut self, node_addr: &crate::NodeAddr) {
let Some(workers) = self.decrypt_workers.as_ref().cloned() else {
let Some(workers) = self.supervisor.decrypt_workers.as_ref().cloned() else {
return;
};
let (cache_key, state) = {
@@ -491,7 +492,7 @@ impl Node {
&mut self,
cache_key: (crate::transport::TransportId, u32),
) {
if let Some(workers) = self.decrypt_workers.as_ref() {
if let Some(workers) = self.supervisor.decrypt_workers.as_ref() {
workers.unregister_session(cache_key);
}
self.decrypt_registered_sessions.remove(&cache_key);
@@ -533,7 +534,7 @@ impl Node {
.duration_since(std::time::UNIX_EPOCH)
.map(|d| d.as_millis() as u64)
.unwrap_or(0);
self.schedule_reconnect(addr, now_ms);
self.note_link_dead(addr, now_ms);
}
}
}
src/node/handlers/forwarding.rs → src/node/dataplane/forwarding.rs
+221 -162
View File
@@ -2,21 +2,21 @@
//!
//! Handles incoming SessionDatagram (0x00) link messages: decodes the
//! envelope, performs coordinate cache warming from plaintext session-layer
//! headers, delivers locally when the datagram is addressed to this node,
//! otherwise enforces the transit hop limit and routes to the next hop, and
//! generates error signals on routing failure.
//! headers, pre-resolves the next hop for forwardable transit datagrams, and
//! drives the routing core's outcome — local delivery when the datagram is
//! addressed to this node, the transit hop-limit drop, the forward, or the
//! error signal generated on routing failure.
use crate::NodeAddr;
use crate::node::reject::ForwardingReject;
use crate::node::session_wire::{
use crate::node::{Node, NodeError, NodeRoutingView};
use crate::proto::fsp::wire::{
FSP_COMMON_PREFIX_SIZE, FSP_HEADER_SIZE, FSP_PHASE_ESTABLISHED, FSP_PHASE_MSG1, FSP_PHASE_MSG2,
FspCommonPrefix, parse_encrypted_coords,
};
use crate::node::{Node, NodeError};
use crate::protocol::{
CoordsRequired, MtuExceeded, PathBroken, SessionAck, SessionDatagram, SessionDatagramRef,
SessionSetup,
};
use crate::proto::fsp::{SessionAck, SessionSetup};
use crate::proto::link::{SessionDatagram, SessionDatagramRef};
use crate::proto::routing::{DropReason, NextHop, RouteAction, RouteOutcome};
use std::time::{Duration, Instant};
use tracing::{debug, warn};
@@ -44,131 +44,169 @@ impl Node {
}
};
// Coordinate cache warming from plaintext session-layer headers.
// Runs ahead of both the delivery and the TTL decisions: the coords
// a peer put on the wire are equally valid whichever way those go.
let my_addr = *self.node_addr();
// Coordinate cache warming from plaintext session-layer headers. Runs
// ahead of both the delivery and the TTL decisions the core makes: the
// coords a peer put on the wire are equally valid whichever way those
// go, and the only arrivals this newly warms from are those with an
// exhausted TTL, whose every insert is already achievable at TTL 1.
self.try_warm_coord_cache_ref(&datagram_ref);
// Local delivery: dispatch to session layer handlers without
// materializing an owned SessionDatagram payload Vec. Delivery to
// the addressed node is *not* TTL-gated — under IP semantics the
// TTL governs forwarding, not delivery to the addressed host — so
// this test precedes the TTL gate below.
if datagram_ref.dest_addr == *self.node_addr() {
self.metrics().forwarding.record_delivered(payload.len());
self.handle_session_payload(
&datagram_ref.src_addr,
datagram_ref.payload,
datagram_ref.path_mtu,
incoming_ce,
)
.await;
return;
}
// Pre-resolve the next hop only for datagrams the core can actually
// forward: not locally destined, and carrying a TTL that survives the
// decrement (`ttl > 1` — the shell-side mirror of the core's
// would-leave-zero drop). This keeps `find_next_hop`'s coord-cache
// LRU-touch side effect scoped to genuine forwards, as it was when the
// TTL test ran inline ahead of it. Warming above has already run, so
// the resolution observes freshly cached coords.
let next_hop = if datagram_ref.dest_addr != my_addr && datagram_ref.ttl > 1 {
self.resolve_next_hop(&datagram_ref.dest_addr)
} else {
None
};
// TTL enforcement on the transit path: decrement first, then drop if
// the datagram would leave with a TTL of zero. `saturating_sub` folds
// the already-exhausted arrival (ttl=0) into the same test as the
// last-hop arrival (ttl=1); neither is transmitted.
let forwarded_ttl = datagram_ref.ttl.saturating_sub(1);
if forwarded_ttl == 0 {
self.metrics()
.forwarding
.record_reject_bytes(ForwardingReject::TtlExhausted, payload.len());
debug!(
src = %datagram_ref.src_addr,
dest = %datagram_ref.dest_addr,
ttl = datagram_ref.ttl,
"SessionDatagram TTL exhausted, dropping"
);
return;
}
// Read local congestion once and reuse it for both the CE decision
// (via the view) and the congestion metric/log below, keeping
// `detect_congestion` the single source of truth.
let congested = next_hop
.as_ref()
.map(|nh| self.detect_congestion(&nh.addr))
.unwrap_or(false);
let mut datagram = datagram_ref.into_owned();
datagram.ttl = forwarded_ttl;
// Borrow the routing tables disjointly from `&mut self.routing` for
// the pure decision, then release both before driving the outcome.
let outcome = {
let view = NodeRoutingView {
coord_cache: &self.coord_cache,
peers: &self.peers,
tree_state: &self.tree_state,
congested,
};
self.routing
.route(&datagram_ref, &my_addr, incoming_ce, next_hop, &view)
};
// Find next hop toward destination
let next_hop_addr = match self.find_next_hop(&datagram.dest_addr) {
Some(peer) => *peer.node_addr(),
None => {
match outcome {
RouteOutcome::Drop {
reason: DropReason::TtlExhausted,
} => {
self.metrics()
.forwarding
.record_reject_bytes(ForwardingReject::TtlExhausted, payload.len());
debug!(
src = %datagram_ref.src_addr,
dest = %datagram_ref.dest_addr,
ttl = datagram_ref.ttl,
"SessionDatagram TTL exhausted, dropping"
);
}
RouteOutcome::DeliverLocal => {
// Local delivery: dispatch to session layer handlers without
// materializing an owned SessionDatagram payload Vec.
self.metrics().forwarding.record_delivered(payload.len());
self.handle_session_payload(
&datagram_ref.src_addr,
datagram_ref.payload,
datagram_ref.path_mtu,
incoming_ce,
)
.await;
}
RouteOutcome::NoRoute => {
self.metrics()
.forwarding
.record_reject_bytes(ForwardingReject::NoRoute, payload.len());
let original = datagram_ref.into_owned();
debug!(
src = %self.peer_display_name(&datagram.src_addr),
dest = %self.peer_display_name(&datagram.dest_addr),
src = %self.peer_display_name(&original.src_addr),
dest = %self.peer_display_name(&original.dest_addr),
bytes = payload.len(),
"Dropping transit SessionDatagram: no route to destination"
);
self.send_routing_error(&datagram).await;
return;
self.send_routing_error(&original).await;
}
};
RouteOutcome::Forward {
next_hop,
bytes,
outgoing_ce,
} => {
let dest = datagram_ref.dest_addr;
// Apply path_mtu min() from the outgoing link's transport MTU
if let Some(peer) = self.peers.get(&next_hop_addr)
// ECN CE relay: congestion was detected locally above; emit the
// metric and rate-limited log at the transit chokepoint.
if congested {
self.metrics().congestion.congestion_detected.inc();
let now = Instant::now();
let should_log = self
.last_congestion_log
.map(|t| now.duration_since(t) >= Duration::from_secs(5))
.unwrap_or(true);
if should_log {
self.last_congestion_log = Some(now);
debug!(next_hop = %next_hop, "Congestion detected, CE flag set on forwarded packet");
}
}
match self
.send_encrypted_link_message_with_ce(&next_hop, &bytes, outgoing_ce)
.await
{
Err(NodeError::MtuExceeded { mtu, .. }) => {
self.metrics()
.forwarding
.record_reject_bytes(ForwardingReject::MtuExceeded, payload.len());
self.send_mtu_exceeded_error(dest, datagram_ref.src_addr, mtu)
.await;
}
Err(e) => {
self.metrics()
.forwarding
.record_reject_bytes(ForwardingReject::SendError, payload.len());
debug!(
next_hop = %next_hop,
dest = %dest,
error = %e,
"Failed to forward SessionDatagram"
);
}
Ok(()) => {
self.metrics().forwarding.record_forwarded(bytes.len());
// Classify this transit forward by route class (partition
// of forwarded_packets). Done here, at the data-plane
// chokepoint, so the error-signal routing callers of
// find_next_hop are excluded.
let class = self.classify_forward(&dest, &next_hop);
self.metrics().forwarding.record_route_class(class);
if outgoing_ce {
self.metrics().congestion.ce_forwarded.inc();
}
}
}
}
}
}
/// Resolve the next hop toward `dest` into its address plus the outgoing
/// link's transport MTU. Returns `None` when there is no route.
///
/// The MTU defaults to `u16::MAX` (a no-op min-fold) when the peer's
/// transport is not resolvable, matching the pre-refactor inline behavior
/// where the MTU `if let` chain simply did not fire.
fn resolve_next_hop(&mut self, dest: &NodeAddr) -> Option<NextHop> {
let addr = *self.find_next_hop(dest)?.node_addr();
let link_mtu = if let Some(peer) = self.peers.get(&addr)
&& let Some(tid) = peer.transport_id()
&& let Some(transport) = self.transports.get(&tid)
{
if let Some(addr) = peer.current_addr() {
datagram.path_mtu = datagram.path_mtu.min(transport.link_mtu(addr));
} else {
datagram.path_mtu = datagram.path_mtu.min(transport.mtu());
}
}
// ECN CE relay: propagate incoming CE and detect local congestion
let local_congestion = self.detect_congestion(&next_hop_addr);
let outgoing_ce = incoming_ce || local_congestion;
if local_congestion {
self.metrics().congestion.congestion_detected.inc();
let now = Instant::now();
let should_log = self
.last_congestion_log
.map(|t| now.duration_since(t) >= Duration::from_secs(5))
.unwrap_or(true);
if should_log {
self.last_congestion_log = Some(now);
debug!(next_hop = %next_hop_addr, "Congestion detected, CE flag set on forwarded packet");
}
}
// Forward: re-encode (includes 0x00 type byte) and send
let encoded = datagram.encode();
if let Err(e) = self
.send_encrypted_link_message_with_ce(&next_hop_addr, &encoded, outgoing_ce)
.await
{
match e {
NodeError::MtuExceeded { mtu, .. } => {
self.metrics()
.forwarding
.record_reject_bytes(ForwardingReject::MtuExceeded, payload.len());
self.send_mtu_exceeded_error(&datagram, mtu).await;
}
_ => {
self.metrics()
.forwarding
.record_reject_bytes(ForwardingReject::SendError, payload.len());
debug!(
next_hop = %next_hop_addr,
dest = %datagram.dest_addr,
error = %e,
"Failed to forward SessionDatagram"
);
}
match peer.current_addr() {
Some(link_addr) => transport.link_mtu(link_addr),
None => transport.mtu(),
}
} else {
self.metrics().forwarding.record_forwarded(encoded.len());
// Classify this transit forward by route class (partition of
// forwarded_packets). Done here, at the data-plane chokepoint, so
// the error-signal routing callers of find_next_hop are excluded.
let class = self.classify_forward(&datagram.dest_addr, &next_hop_addr);
self.metrics().forwarding.record_route_class(class);
if outgoing_ce {
self.metrics().congestion.ce_forwarded.inc();
}
}
u16::MAX
};
Some(NextHop { addr, link_mtu })
}
/// Attempt to warm the coordinate cache from session-layer payload headers.
@@ -269,35 +307,41 @@ impl Node {
/// If we can't route the error back to the source either, drop silently.
/// No cascading errors.
async fn send_routing_error(&mut self, original: &SessionDatagram) {
// Rate limit: one error signal per destination per 100ms
if !self
.routing_error_rate_limiter
.should_send(&original.dest_addr)
{
return;
}
let my_addr = *self.node_addr();
let now_ms = std::time::SystemTime::now()
.duration_since(std::time::UNIX_EPOCH)
.map(|d| d.as_millis() as u64)
.unwrap_or(0);
let default_ttl = self.config().node.session.default_ttl;
let error_payload =
if let Some(coords) = self.coord_cache().get(&original.dest_addr, now_ms) {
let coords = coords.clone();
PathBroken::new(original.dest_addr, my_addr)
.with_last_coords(coords)
.encode()
} else {
CoordsRequired::new(original.dest_addr, my_addr).encode()
// Pure decision: rate-limit gate + PathBroken/CoordsRequired choice +
// error-PDU encode. Borrow the routing tables disjointly from
// `&mut self.routing`, then release them before the reverse-hop lookup.
let action = {
let view = NodeRoutingView {
coord_cache: &self.coord_cache,
peers: &self.peers,
tree_state: &self.tree_state,
congested: false,
};
self.routing.synth_routing_error(
&original.dest_addr,
&original.src_addr,
&my_addr,
&view,
now_ms,
default_ttl,
)
};
let RouteAction::SendError { toward, bytes } = match action {
Some(action) => action,
// Rate limited: drop silently. No cascading errors.
None => return,
};
let error_dg = SessionDatagram::new(my_addr, original.src_addr, error_payload)
.with_ttl(self.config().node.session.default_ttl);
let next_hop_addr = match self.find_next_hop(&original.src_addr) {
// Resolve the reverse link hop only now, after the gate passed, so
// `find_next_hop`'s coord-cache touch keeps its pre-refactor scope.
let next_hop_addr = match self.find_next_hop(&toward) {
Some(peer) => *peer.node_addr(),
None => {
debug!(
@@ -309,9 +353,8 @@ impl Node {
}
};
let encoded = error_dg.encode();
if let Err(e) = self
.send_encrypted_link_message(&next_hop_addr, &encoded)
.send_encrypted_link_message(&next_hop_addr, &bytes)
.await
{
debug!(
@@ -333,37 +376,50 @@ impl Node {
/// Called when `send_encrypted_link_message()` fails with
/// `NodeError::MtuExceeded` during forwarding. The signal tells the
/// source the bottleneck MTU so it can immediately reduce its path MTU.
async fn send_mtu_exceeded_error(&mut self, original: &SessionDatagram, bottleneck_mtu: u16) {
// Rate limit: reuse routing_error_rate_limiter keyed on dest_addr
if !self
.routing_error_rate_limiter
.should_send(&original.dest_addr)
{
return;
}
///
/// `dest` is the failed datagram's destination (rate-limit key); `toward`
/// is its source, where the signal is routed back.
async fn send_mtu_exceeded_error(
&mut self,
dest: NodeAddr,
toward: NodeAddr,
bottleneck_mtu: u16,
) {
let my_addr = *self.node_addr();
let now_ms = Self::now_ms();
let default_ttl = self.config().node.session.default_ttl;
let error_payload = MtuExceeded::new(original.dest_addr, my_addr, bottleneck_mtu).encode();
// Pure decision: rate-limit gate + MtuExceeded PDU + encode.
let action = self.routing.synth_mtu_exceeded(
&dest,
&toward,
&my_addr,
bottleneck_mtu,
now_ms,
default_ttl,
);
let RouteAction::SendError { toward, bytes } = match action {
Some(action) => action,
// Rate limited: drop silently. No cascading errors.
None => return,
};
let error_dg = SessionDatagram::new(my_addr, original.src_addr, error_payload)
.with_ttl(self.config().node.session.default_ttl);
let next_hop_addr = match self.find_next_hop(&original.src_addr) {
// Resolve the reverse link hop only now, after the gate passed, so
// `find_next_hop`'s coord-cache touch keeps its pre-refactor scope.
let next_hop_addr = match self.find_next_hop(&toward) {
Some(peer) => *peer.node_addr(),
None => {
debug!(
src = %original.src_addr,
dest = %original.dest_addr,
src = %toward,
dest = %dest,
"Cannot route MtuExceeded signal back to source, dropping"
);
return;
}
};
let encoded = error_dg.encode();
if let Err(e) = self
.send_encrypted_link_message(&next_hop_addr, &encoded)
.send_encrypted_link_message(&next_hop_addr, &bytes)
.await
{
debug!(
@@ -373,8 +429,8 @@ impl Node {
);
} else {
debug!(
original_dest = %original.dest_addr,
error_dest = %original.src_addr,
original_dest = %dest,
error_dest = %toward,
bottleneck_mtu,
"Sent MtuExceeded error signal"
);
@@ -417,10 +473,13 @@ impl Node {
for (&tid, transport) in &self.transports {
let congestion = transport.congestion();
let state = self.transport_drops.entry(tid).or_default();
if let Some(current) = congestion.recv_drops
&& state.observe_drops(current)
{
new_drop_events.push(tid);
if let Some(current) = congestion.recv_drops {
let new_drops = current > state.prev_drops;
if new_drops && !state.dropping {
new_drop_events.push(tid);
}
state.dropping = new_drops;
state.prev_drops = current;
}
}
for tid in new_drop_events {
src/node/dataplane/mod.rs
+18
View File
@@ -0,0 +1,18 @@
//! Data plane: the RX `select!` loop and the per-packet forwarding path.
//!
//! Holds the whole hot path in one home: the `select!` run loop
//! (`rx_loop`), transit/local datagram forwarding (`forwarding`), the
//! link-message router (`dispatch`), the RX decrypt path including responder
//! K-bit cutover and address-roam writes (`encrypted`), and the per-peer
//! connected-UDP fast-path socket activation (`connected_udp`). Each module
//! contributes `impl Node` methods driven by the run loop.
#[cfg(unix)]
pub(crate) mod connected_udp;
mod dispatch;
mod encrypted;
mod forwarding;
mod peer_actions;
mod rx_loop;
pub(in crate::node) use peer_actions::PeerActionCtx;
src/node/dataplane/peer_actions.rs
+557
View File
@@ -0,0 +1,557 @@
//! Executor for the per-peer control machine's [`PeerAction`]s.
//!
//! The per-peer FSM in [`crate::peer::machine`] is a sans-IO reducer: it decides
//! *what* must happen and returns a `Vec<PeerAction>`; this module is the *doing*
//! half — the thin driver that maps each action onto the exact shell call it
//! stands for (`build_msg2` + `transport.send`, `promote_connection`,
//! `remove_active_peer`, `index_allocator.free`, `note_link_dead`, …).
//!
//! ## Progressive cutover
//!
//! The executor is wired incrementally. Live today: the inbound establish
//! (`handle_msg1` → `step(InboundMsg1)`), the outbound msg2 promote
//! (`handle_msg2` looks up the dial-persisted machine), the connectionless
//! outbound msg1 send (`SendHandshake` with `their_index == None` →
//! `send_stored_msg1`, driven from `initiate_connection`), the
//! connection-oriented dial (`OpenTransport` performs the non-blocking
//! `transport.connect`; `TransportConnected` drives the connect-resolution msg1
//! send from `poll_pending_connects`), the rekey cadence (`check_rekey` →
//! `route_rekey_cadence` → `RekeyConsume`, driving the `SwapSendState` and
//! `CompleteDrain` arms), and the liveness reap (`route_link_dead` →
//! `LinkDeadSuspected`, driving `InvalidateSendState` → `remove_active_peer`).
//!
//! The genuine inert stubs remaining are `SendRekey`, `SendLinkMessage`, and
//! the connected-UDP arms. `RegisterDecryptSession` is a deliberate no-op —
//! see its arm for the note.
//!
//! The timer arms (`SetTimer`/`CancelTimer`) populate/clear the per-peer timer
//! store (`peer_timers`). The `HandshakeRetransmit` and `HandshakeTimeout`
//! deadlines are read and fired by `drive_peer_timers` (the handshake resend +
//! reap home). The rekey/liveness kinds are still SHADOW — driven by their own
//! shell drivers — so populating them stays behavior-neutral.
use crate::PeerIdentity;
use crate::node::Node;
use crate::node::reject::{HandshakeReject, RejectReason};
use crate::peer::machine::{LostKind, PeerAction, PeerEvent};
use crate::proto::fmp::PromotionResult;
use crate::proto::fmp::wire::build_msg2;
use crate::transport::{LinkId, TransportAddr, TransportId};
use crate::utils::index::SessionIndex;
use std::collections::VecDeque;
use tracing::{debug, trace, warn};
/// Ambient shell facts a [`PeerAction`] executor needs that the machine's
/// runtime-agnostic action payloads deliberately omit (verified identity,
/// transport target, the msg2 framing indices, the promotion timestamp).
///
/// Unlike a machine event/action payload this is **executor-side**, so it may
/// hold real values resolved from the wire context (cf. `handle_msg1`'s
/// `wire`/`packet` locals and `promote_connection`'s ambient args). It is
/// built fresh per driven step by the caller at cutover time.
#[allow(dead_code)]
pub(in crate::node) struct PeerActionCtx {
/// The authenticated peer identity: `PromoteToActive` /
/// `InvalidateSendState` resolve their `NodeAddr` from this.
pub(in crate::node) verified_identity: PeerIdentity,
/// The transport the exchange is happening over (msg2 send target, decrypt
/// cache-key transport half).
pub(in crate::node) transport_id: TransportId,
/// The peer's wire address (msg2 send target).
pub(in crate::node) remote_addr: TransportAddr,
/// Our session index for this exchange (msg2 framing sender_idx).
pub(in crate::node) our_index: Option<SessionIndex>,
/// The peer's session index for this exchange (msg2 framing
/// receiver_idx).
pub(in crate::node) their_index: Option<SessionIndex>,
/// The wire timestamp driving this step (promotion ts / loss-report clock).
pub(in crate::node) now_ms: u64,
/// Establish direction for this exchange. Discriminates the
/// `PromoteToActive` failure cleanup: the pre-refactor inbound
/// (`handle_msg1`) and outbound (`handle_msg2`) promote-Err arms were NOT
/// byte-identical, so the executor must reproduce each. `false` = inbound
/// (drop link + reverse map + free index), `true` = outbound (record the
/// reject only; leave the dead link/`addr_to_link` for the stale-connection
/// reaper, matching old `handle_msg2`).
pub(in crate::node) is_outbound: bool,
}
impl Node {
/// Advance the machine for `link` by one event and execute the resulting
/// actions.
///
/// The borrow structure the whole seam turns on: the machine
/// needs `&mut IndexAllocator` as a synchronous capability *while it is
/// itself borrowed mutably out of `peer_machines`*. `peer_machines` and
/// `index_allocator` are **distinct `Node` fields**, so the collect below is
/// a disjoint two-field borrow the checker accepts; once the actions are
/// collected both borrows drop and the executor runs against `&mut self`.
pub(in crate::node) async fn advance_peer_machine(
&mut self,
link: LinkId,
event: PeerEvent,
now: u64,
ambient: &PeerActionCtx,
) {
let actions = match self.peer_machines.get_mut(&link) {
// Disjoint field borrow: `self.peer_machines` (the map entry) and
// `self.index_allocator` (the capability) are separate fields.
Some(machine) => machine.step(event, now, &mut self.index_allocator),
None => return,
};
self.execute_peer_actions(link, ambient, actions).await;
}
/// Map each [`PeerAction`] onto its shell call.
///
/// `PromoteToActive` feeds its [`PromotionResult`](crate::proto::fmp::PromotionResult)
/// back into the machine and appends the follow-up actions to the same
/// worklist — a queue rather than self-recursion so the async executor stays a
/// single flat future (no boxing) and the emitted order is preserved (the
/// establish sequences always end in `PromoteToActive`, so its follow-ups run
/// after any siblings).
pub(in crate::node) async fn execute_peer_actions(
&mut self,
link: LinkId,
ambient: &PeerActionCtx,
actions: Vec<PeerAction>,
) {
let mut queue: VecDeque<PeerAction> = actions.into();
while let Some(action) = queue.pop_front() {
match action {
PeerAction::OpenTransport {
transport_id,
remote_addr,
} => {
// Outbound connection-oriented dial. `initiate_connection`'s
// oriented branch drove the machine to `Connecting`, which
// emitted this action. Perform the non-blocking
// `transport.connect` and, on success, push the
// `PendingConnect` for `poll_pending_connects` to resolve. On
// connect error, tear down the dial-window state (link,
// reverse map, control machine) and abort the queue — the
// executor-local mirror of the old inline
// `initiate_connection` connect+push.
if let Some(transport) = self.transports.get(&transport_id) {
match transport.connect(&remote_addr).await {
Ok(()) => {
debug!(
transport_id = %transport_id,
remote_addr = %remote_addr,
link_id = %link,
"Transport connect initiated (non-blocking)"
);
self.peering
.pending_connects
.push(crate::node::PendingConnect {
link_id: link,
transport_id,
remote_addr,
peer_identity: ambient.verified_identity,
});
}
Err(_e) => {
self.links.remove(&link);
self.addr_to_link.remove(&(transport_id, remote_addr));
self.remove_peer_machine(link);
return;
}
}
}
}
PeerAction::SendHandshake { bytes } => {
// Two outbound directions share this action, discriminated by
// `their_index`:
// msg2 (`their_index == Some`): the machine payload is the
// UNFRAMED Noise msg2; frame it with our/their index
// (`build_msg2`) and send.
// msg1 (`their_index == None`): a fresh outbound handshake;
// the machine's empty payload is ignored — the shell already
// allocated the index, ran the Noise leaf, and armed the
// wire at dial (`prepare_outbound_msg1`); this just sends the
// stored wire (see `send_stored_msg1`).
if let (Some(sender_idx), Some(receiver_idx)) =
(ambient.our_index, ambient.their_index)
{
let frame = build_msg2(sender_idx, receiver_idx, &bytes);
// Surface the send Result. A missing transport skips
// the send and continues (mirrors `handle_msg1`'s
// `if let Some(transport)` guard); a send *error* runs the
// pre-refactor msg2-send-failure cleanup (`handle_msg1`
// L494-503) and ABORTS the remaining queue so the queued
// `PromoteToActive` never runs.
let send_err = match self.transports.get(&ambient.transport_id) {
Some(transport) => {
transport.send(&ambient.remote_addr, &frame).await.err()
}
None => None,
};
if let Some(e) = send_err {
// Restored pre-refactor msg2-send-failure warn!
// (`handle_msg1` L665): the send error text is surfaced
// at the executor point where the failure is now handled.
warn!(link_id = %link, error = %e, "Failed to send msg2");
self.links.remove(&link);
self.addr_to_link
.remove(&(ambient.transport_id, ambient.remote_addr.clone()));
if let Some(idx) = ambient.our_index {
let _ = self.index_allocator.free(idx);
}
self.remove_peer_machine(link);
self.stats_mut()
.record_reject(RejectReason::Handshake(HandshakeReject::BadState));
return;
}
} else {
// msg1: the shell already allocated the index, ran the
// Noise leaf, and armed the wire on the connection at dial
// (`prepare_outbound_msg1`); send the stored wire. The
// machine's empty payload is ignored.
let _ = bytes;
self.send_stored_msg1(
link,
ambient.transport_id,
&ambient.remote_addr,
ambient.now_ms,
)
.await;
}
}
PeerAction::SendRekey { .. } => {
// Rekey msg2 framing (`build_msg2(our_new_index, …)`,
// `handshake.rs:365`) + send. Rekey fold is not yet wired.
}
PeerAction::SendLinkMessage { .. } => {
// Encrypt + send a link-control frame (heartbeat / filter
// / tree / disconnect). Data-plane-owned; not yet wired.
}
PeerAction::PromoteToActive { link: promote_link } => {
// Ambient supplies the verified identity + promotion ts
// that `promote_connection` needs (resolved from the wire ctx).
match self.promote_connection(
promote_link,
ambient.verified_identity,
ambient.now_ms,
) {
Ok(result) => {
// The decrypt-worker registration relocated
// OUT of `promote_connection` into THIS single executor
// arm — the one live caller of `promote_connection` (both
// the inbound `handle_msg1` and outbound `handle_msg2`
// net-new establish paths reach it here). Register iff the
// promotion actually created or replaced a peer
// (`Promoted | CrossConnectionWon`), NEVER on
// `CrossConnectionLost`. Run synchronously right after
// `promote_connection` returns, before feeding
// `PromotionResolved` and before any await — the exact
// synchronous point (and Promoted/Won gating) of the
// pre-refactor in-`promote_connection` call. No-op when
// the worker pool isn't spawned (`register_...` early-
// returns), so the direct `promote_connection` test
// callers (which bypass this executor) are unaffected.
#[cfg(unix)]
match result {
PromotionResult::Promoted(node_addr)
| PromotionResult::CrossConnectionWon { node_addr, .. } => {
self.register_decrypt_worker_session(&node_addr);
}
PromotionResult::CrossConnectionLost { .. } => {}
}
// Feed the outcome back into the machine and fold the
// follow-up actions (RegisterDecryptSession — now a
// redundant no-op, see its arm — and the cross-conn index
// frees) into the worklist. Disjoint field borrow again.
let follow = match self.peer_machines.get_mut(&promote_link) {
Some(machine) => machine.step(
PeerEvent::PromotionResolved { result },
ambient.now_ms,
&mut self.index_allocator,
),
None => Vec::new(),
};
queue.extend(follow);
// Defensive cross-connection loser-link surgery.
// LINK-ONLY: close the losing transport connection, drop
// its link, and re-point `addr_to_link`, reproducing the
// pre-refactor inline `handle_msg2`/`handle_msg1` per-arm
// order EXACTLY. The index-plane frees/unregisters are
// owned by the machine's `PromotionResolved{Won/Lost}`
// follow-up (queued just above), so NOTHING here touches
// an index — no double-free.
//
// UNREACHABLE on every current driven path: the inbound
// and outbound net-new establish arms only route to the
// machine when no promoted peer exists for the node_addr
// (and `RestartThenPromote` removes the old peer first),
// so `promote_connection` always returns `Promoted`. The
// `debug_assert!(false, ..)` catches any future path that
// drives a cross-connection through the executor without
// the matching send-state handling.
match result {
PromotionResult::CrossConnectionWon { loser_link_id, .. } => {
debug_assert!(
false,
"executor CrossConnectionWon is unreachable on \
driven net-new establish paths"
);
// Close the losing transport connection (no-op for
// connectionless) via the LOSER link's own
// transport/addr, then drop the losing link.
if let Some(loser_link) = self.links.get(&loser_link_id) {
let loser_tid = loser_link.transport_id();
let loser_addr = loser_link.remote_addr().clone();
if let Some(transport) = self.transports.get(&loser_tid) {
transport.close_connection(&loser_addr).await;
}
}
self.remove_link(&loser_link_id);
// Point `addr_to_link` at the winning (current)
// link.
self.addr_to_link.insert(
(ambient.transport_id, ambient.remote_addr.clone()),
promote_link,
);
}
PromotionResult::CrossConnectionLost { winner_link_id } => {
debug_assert!(
false,
"executor CrossConnectionLost is unreachable on \
driven net-new establish paths"
);
// Close this (losing) connection, drop its link,
// and restore `addr_to_link` to the winner.
if let Some(transport) =
self.transports.get(&ambient.transport_id)
{
transport.close_connection(&ambient.remote_addr).await;
}
self.remove_link(&promote_link);
self.addr_to_link.insert(
(ambient.transport_id, ambient.remote_addr.clone()),
winner_link_id,
);
}
PromotionResult::Promoted(_) => {}
}
}
Err(e) => {
// Promotion failed. `promote_connection` already
// removed `connections[link]` and (on error) handled its
// own index internally. The pre-refactor inbound and
// outbound promote-Err arms were NOT byte-identical, so
// discriminate on `ambient.is_outbound`. The queue is
// drained (PromoteToActive is the last establish action),
// so no explicit abort.
if ambient.is_outbound {
// OLD outbound (`handle_msg2` promote-Err): warn +
// record_reject ONLY. NO `remove_link`, NO
// `index_allocator.free`, NO `addr_to_link` removal —
// the dead link/addr_to_link/pending_outbound were
// left for the 30s stale-connection reaper
// (`promote_connection` already handled
// `connections[link]`/its index on error). Restored
// pre-refactor outbound warn! ("Failed to promote
// connection").
//
// The outbound machine was persisted at dial; it is
// additive state that did not exist pre-refactor, so
// removing it on promote failure is neutral vs old and
// prevents a leak.
warn!(
target: "fips::node::handlers::handshake",
link_id = %promote_link,
error = %e,
"Failed to promote connection"
);
self.stats_mut().record_reject(RejectReason::Handshake(
HandshakeReject::BadState,
));
self.remove_peer_machine(promote_link);
} else {
// OLD inbound (`handle_msg1` L587-591): drop the link
// + reverse map, free our index, discard the machine,
// and record the reject. Restored pre-refactor inbound
// promote-failure warn! (`handle_msg1` L757).
warn!(
target: "fips::node::handlers::handshake",
link_id = %promote_link,
error = %e,
"Failed to promote inbound connection"
);
self.remove_link(&promote_link);
if let Some(idx) = ambient.our_index {
let _ = self.index_allocator.free(idx);
}
self.remove_peer_machine(promote_link);
self.stats_mut().record_reject(RejectReason::Handshake(
HandshakeReject::BadState,
));
}
}
}
}
PeerAction::ResolveCrossConnection { .. } => {
// A decision token, not an effect: the outbound msg2
// handler intercepts it and runs the inline swap/keep
// resolution itself, so it must never reach the executor.
debug_assert!(
false,
"ResolveCrossConnection is intercepted by the msg2 \
handler and must never reach the executor"
);
}
PeerAction::SwapSendState { .. } => {
// Initiator cutover: the live authoritative rekey-cadence
// path, routed here from `check_rekey` via
// `route_rekey_cadence` → `PeerEvent::RekeyConsume`; the
// inline body survives only as `cutover_peer_inline`, a
// debug-assert release fallback. `addr` is resolved
// from the ambient verified identity (as `InvalidateSendState`
// does). The decrypt re-register folds HERE, gated on
// `did_cutover` — the generic `RegisterDecryptSession` arm stays a
// no-op so a promote never double-registers.
let node_addr = *ambient.verified_identity.node_addr();
let did_cutover = if let Some(peer) = self.peers.get_mut(&node_addr) {
if let Some(_old_our_index) = peer.cutover_to_new_session() {
// New index was pre-registered in peers_by_index
// during msg2 handling (handshake.rs).
debug_assert!(
peer.transport_id().is_some()
&& peer.our_index().is_some()
&& self.peers_by_index.contains_key(&(
peer.transport_id().unwrap(),
peer.our_index().unwrap().as_u32()
)),
"peers_by_index should contain pre-registered new index after cutover"
);
debug!(
// Pin the target to the pre-refactor module: this
// cutover log relocated from handlers/rekey.rs into
// the executor, but operators (and the test harness)
// filter it under fips::node::handlers::rekey. Keeping
// the target preserves the observable log contract.
target: "fips::node::handlers::rekey",
peer = %self.peer_display_name(&node_addr),
"Rekey cutover complete (initiator), K-bit flipped"
);
true
} else {
false
}
} else {
false
};
// Re-register the new session with the decrypt worker — the
// cache_key (transport_id, our_index) just changed, so the
// old worker entry is stale and every packet on the new
// session would miss the worker's HashMap lookup.
#[cfg(unix)]
if did_cutover {
self.register_decrypt_worker_session(&node_addr);
}
#[cfg(not(unix))]
let _ = did_cutover;
}
PeerAction::CompleteDrain { peer: node_addr } => {
// Initiator drain completion: the live authoritative
// rekey-cadence path, routed here from `check_rekey` via
// `route_rekey_cadence` → `PeerEvent::RekeyConsume`; the
// inline body survives only as `drain_peer_inline`, a
// debug-assert release fallback. Extract the real previous
// index + transport_id under the peer borrow, drop the
// borrow, then run the cache_key cleanup (which takes
// &mut self for unregister_decrypt_worker_session).
let drained = self.peers.get_mut(&node_addr).and_then(|peer| {
peer.complete_drain().map(|idx| (idx, peer.transport_id()))
});
if let Some((old_our_index, transport_id)) = drained {
if let Some(tid) = transport_id {
let cache_key = (tid, old_our_index.as_u32());
self.peers_by_index.remove(&cache_key);
#[cfg(unix)]
self.unregister_decrypt_worker_session(cache_key);
}
let _ = self.index_allocator.free(old_our_index);
trace!(
// Pin to the pre-refactor module (see the cutover log
// above) so the relocated drain log stays visible under
// the operator's fips::node::handlers::rekey filter.
target: "fips::node::handlers::rekey",
peer = %self.peer_display_name(&node_addr),
old_index = %old_our_index,
"Drain complete, previous session erased"
);
}
}
PeerAction::InvalidateSendState => {
// The FULL teardown. `remove_active_peer`
// (`dispatch.rs:107`) frees the four index slots
// (current/rekey/pending/previous), drops `peers_by_index`,
// unregisters the decrypt worker, removes the FSP `sessions`
// entry and `pending_tun_packets`. The machine emits NO
// `FreeIndex` for those slots, so there is no double-free.
self.remove_active_peer(ambient.verified_identity.node_addr());
}
PeerAction::RegisterDecryptSession { index } => {
let _ = index;
// No-op by design. The decrypt-worker
// registration relocated into the `PromoteToActive` Ok arm above, gated on
// the returned `PromotionResult`, so it runs once per live
// promote (Promoted/Won) at the pre-refactor synchronous point.
// This machine-emitted action is now redundant with that arm;
// kept as an inert no-op (rather than removing the emission) so
// the machine's action sequence and its unit tests stay
// unchanged. The keyed-by-NodeAddr register does not need the
// machine's `index` payload.
}
PeerAction::UnregisterDecryptSession { index } => {
// Executor supplies `transport_id` from ambient; keyed by
// (tid, index) like `remove_active_peer` / the rekey drain path.
#[cfg(unix)]
self.unregister_decrypt_worker_session((ambient.transport_id, index.as_u32()));
#[cfg(not(unix))]
let _ = index;
}
PeerAction::FreeIndex { index } => {
let _ = self.index_allocator.free(index);
}
PeerAction::ActivateConnectedUdp | PeerAction::TeardownConnectedUdp => {
// Connected-UDP plane ownership (`connected_udp.rs`).
}
PeerAction::SetTimer { kind, at_ms } => {
// Populate the per-peer timer store (overwrite = reschedule).
// The `HandshakeRetransmit` and `HandshakeTimeout` deadlines
// are read + fired by `drive_peer_timers`. Rekey/liveness kinds
// are still SHADOW here — they keep their own shell drivers —
// so populating them stays behavior-neutral.
self.peer_timers
.entry(link)
.or_default()
.insert(kind, at_ms);
}
PeerAction::CancelTimer { kind } => {
if let Some(timers) = self.peer_timers.get_mut(&link) {
timers.remove(&kind);
}
}
PeerAction::ReportLost { peer, kind } => {
// The single loss token, routed to the reconciler reflex the
// `kind` names: an un-promoted handshake attempt takes the
// connected-guarded `note_handshake_timeout` (`driver.rs:28`),
// an established peer's link-death takes the unconditional
// `note_link_dead` (`driver.rs:48`).
match kind {
LostKind::HandshakeTimeout => {
self.note_handshake_timeout(peer, ambient.now_ms);
}
LostKind::LinkDead => {
self.note_link_dead(peer, ambient.now_ms);
}
}
}
}
}
}
}
src/node/handlers/rx_loop.rs → src/node/dataplane/rx_loop.rs
+192 -45
View File
@@ -1,10 +1,10 @@
//! RX event loop and packet dispatch.
use crate::control::{ControlSocket, commands};
use crate::node::wire::{
use crate::node::{Node, NodeError};
use crate::proto::fmp::wire::{
COMMON_PREFIX_SIZE, CommonPrefix, FMP_VERSION, PHASE_ESTABLISHED, PHASE_MSG1, PHASE_MSG2,
};
use crate::node::{Node, NodeError};
use crate::transport::ReceivedPacket;
use std::time::Duration;
use tracing::{debug, info, warn};
@@ -42,12 +42,42 @@ impl Node {
/// This method takes ownership of the packet_rx channel and runs
/// until the channel is closed (typically when stop() is called).
pub async fn run_rx_loop(&mut self) -> Result<(), NodeError> {
// No shutdown observer → today's infinite loop, byte-identical. All
// existing callers/tests use this; `pending()` never fires, so the
// shutdown/deadline arms below stay permanently disabled.
self.run_rx_loop_with_shutdown(std::future::pending()).await
}
/// The rx event loop, which serves until `shutdown` fires and then drains
/// **in place** before returning.
///
/// The channel receivers are moved into this frame's locals and live across
/// both serve and drain, so — unlike a `select!`-cancelled loop — they are
/// never destructively dropped mid-flight; they are released only on clean
/// exit, after which teardown does not need them.
///
/// - While serving (`drain_deadline == None`) the loop is behaviorally
/// identical to before: the shutdown arm, the deadline arm, and the
/// peers-empty early-exit are all guarded off, so the hot per-packet path
/// and the `biased` order of the real arms are unchanged.
/// - When `shutdown` fires, the loop calls [`Node::enter_drain`] once
/// (broadcast Disconnect, gate the reconciler off) and arms the bounded
/// deadline, then keeps servicing inbound/tick/peer-removal until all
/// peers clear or the deadline elapses, then returns. The caller
/// ([`Node::finish_shutdown`]) closes the window and tears down.
pub async fn run_rx_loop_with_shutdown(
&mut self,
shutdown: impl std::future::Future<Output = ()>,
) -> Result<(), NodeError> {
tokio::pin!(shutdown);
// `None` = serving; `Some(deadline)` = draining (bounded window).
let mut drain_deadline: Option<tokio::time::Instant> = None;
let mut packet_rx = self.packet_rx.take().ok_or(NodeError::NotStarted)?;
// Take the TUN outbound receiver, or create a dummy channel that never
// produces messages (when TUN is disabled). Holding the sender prevents
// the channel from closing.
let (mut tun_outbound_rx, _tun_guard) = match self.tun_outbound_rx.take() {
let (mut tun_outbound_rx, _tun_guard) = match self.supervisor.tun_outbound_rx.take() {
Some(rx) => (rx, None),
None => {
let (tx, rx) = tokio::sync::mpsc::channel(1);
@@ -57,7 +87,7 @@ impl Node {
// Take the DNS identity receiver, or create a dummy channel (when DNS
// is disabled). Same pattern as TUN outbound.
let (mut dns_identity_rx, _dns_guard) = match self.dns_identity_rx.take() {
let (mut dns_identity_rx, _dns_guard) = match self.supervisor.dns_identity_rx.take() {
Some(rx) => (rx, None),
None => {
let (tx, rx) = tokio::sync::mpsc::channel(1);
@@ -65,8 +95,20 @@ impl Node {
}
};
let mut tick =
tokio::time::interval(Duration::from_secs(self.config().node.tick_interval_secs));
// Take the runtime child-liveness receiver, or a dummy channel (when the
// node was seeded straight into Running without a start()). Holding the
// dummy sender in the guard keeps the channel open. Same pattern as TUN
// outbound / DNS identity.
let (mut child_exit_rx, _child_exit_guard) = match self.child_exit_rx.take() {
Some(rx) => (rx, None),
None => {
let (tx, rx) = tokio::sync::mpsc::channel(1);
(rx, Some(tx))
}
};
let tick_period = Duration::from_secs(self.config().node.tick_interval_secs);
let mut tick = tokio::time::interval(tick_period);
// Set up control socket channel
let (control_tx, mut control_rx) =
@@ -122,6 +164,13 @@ impl Node {
crate::perf_profile::maybe_spawn_reporter();
loop {
// Bounded drain mode: break as soon as all peers have cleared. In
// normal mode (`None`) this short-circuits before touching
// `self.peers`, so the loop is byte-identical.
if drain_deadline.is_some() && self.peers.is_empty() {
info!("Drain complete: all peers cleared, ending drain loop");
break;
}
tokio::select! {
biased;
// Decrypt-worker fallback drains FIRST. Under sustained
@@ -214,6 +263,27 @@ impl Node {
}
}
}
// Runtime child-liveness. Placed AFTER `packet_rx` so the hot
// inbound path keeps its `biased` priority. A directly-observable
// child (TUN threads, DNS/mDNS/Nostr) exited on its own; feed the
// FSM, which republishes health (Degraded here — a Running node
// always has ≥1 transport up). `on_child_exited` only ever emits
// `PublishState`; other variants are ignored defensively.
maybe_child = child_exit_rx.recv() => {
if let Some(child) = maybe_child {
let actions = self
.supervisor
.fsm
.step(crate::node::lifecycle::supervisor::Event::ChildExited { child });
for action in actions {
if let crate::node::lifecycle::supervisor::Action::PublishState(ns) =
action
{
self.supervisor.state = ns;
}
}
}
}
Some(ipv6_packet) = tun_outbound_rx.recv() => {
self.handle_tun_outbound(ipv6_packet).await;
let mut drained = 0;
@@ -249,41 +319,114 @@ impl Node {
).await;
let _ = response_tx.send(response);
}
_ = tick.tick() => {
self.check_timeouts();
let now_ms = Self::now_ms();
self.reload_peer_acl().await;
// The host map hot-reloads on the same tick as the ACL. It
// is polled separately from `reload_peer_acl` because the
// ACL's embedded alias reloader and this snapshot are
// distinct resources; the `path_mtu_lookup` cache and the
// `nostr_discovery` subsystem are deliberately excluded
// from `Reloadable` since neither reloads from a backing
// file (see `node::reloadable`).
self.reload_host_map().await;
self.poll_pending_connects().await;
self.poll_nostr_discovery().await;
self.poll_lan_discovery().await;
self.resend_pending_handshakes(now_ms).await;
self.resend_pending_rekeys(now_ms).await;
self.resend_pending_session_handshakes(now_ms).await;
self.resend_pending_session_msg3(now_ms).await;
self.purge_idle_sessions(now_ms);
self.process_pending_retries(now_ms).await;
self.check_tree_state().await;
self.check_bloom_state().await;
self.compute_mesh_size();
self.record_stats_history();
self.check_mmp_reports().await;
self.check_session_mmp_reports().await;
self.check_link_heartbeats().await;
self.check_rekey().await;
self.check_session_rekey().await;
self.check_pending_lookups(now_ms).await;
self.poll_transport_discovery().await;
self.sample_transport_congestion();
#[cfg(any(target_os = "linux", target_os = "macos"))]
self.activate_connected_udp_sessions().await;
deadline = tick.tick() => {
// Tick-body instrumentation. The gate is read ONCE per tick
// into `instr_on`, which is then passed explicitly to every
// `instr_step!` invocation — macro hygiene makes a call-site
// local invisible inside the macro body. With the
// `profiling` feature off, `gate()` is a `const fn`
// returning false and the macro is a pure pass-through, so
// the whole arm compiles to the uninstrumented sequence.
//
// `tick_entry` records how late this entry is against the
// deadline the interval scheduled it for. That is the
// measurement this instrumentation exists for: the arm is
// polled LAST under `biased;`, so the
// lateness IS the time it spent waiting behind the packet,
// TUN and control arms. `tick()` hands back its scheduled
// deadline, so this is a subtraction rather than a model.
// The whole-tick span below measures the body alone.
let instr_on = crate::instr::gate();
crate::instr::tick_entry(instr_on, deadline.into_std(), std::time::Instant::now());
instr_step!(instr_on, crate::instr::Domain::Tick, crate::instr::Step::WholeTick, {
instr_step!(instr_on, crate::instr::Domain::Tick, crate::instr::Step::CheckTimeouts,
self.check_timeouts());
let now_ms = Self::now_ms();
instr_step!(instr_on, crate::instr::Domain::Tick, crate::instr::Step::ReloadPeerAcl,
self.reload_peer_acl().await);
// The host map hot-reloads on the same tick as the ACL. It
// is polled separately from `reload_peer_acl` because the
// ACL's embedded alias reloader and this snapshot are
// distinct resources; the `path_mtu_lookup` cache and the
// `nostr_rendezvous` subsystem are deliberately excluded
// from `Reloadable` since neither reloads from a backing
// file (see `node::reloadable`).
instr_step!(instr_on, crate::instr::Domain::Tick, crate::instr::Step::ReloadHostMap,
self.reload_host_map().await);
instr_step!(instr_on, crate::instr::Domain::Tick, crate::instr::Step::PollPendingConnects,
self.poll_pending_connects().await);
instr_step!(instr_on, crate::instr::Domain::Tick, crate::instr::Step::PollNostrRendezvous,
self.poll_nostr_rendezvous().await);
instr_step!(instr_on, crate::instr::Domain::Tick, crate::instr::Step::PollLanRendezvous,
self.poll_lan_rendezvous().await);
instr_step!(instr_on, crate::instr::Domain::Tick, crate::instr::Step::DrivePeerTimers,
self.drive_peer_timers(now_ms).await);
instr_step!(instr_on, crate::instr::Domain::Tick, crate::instr::Step::ResendPendingRekeys,
self.resend_pending_rekeys(now_ms).await);
instr_step!(instr_on, crate::instr::Domain::Tick, crate::instr::Step::ResendPendingSessionHandshakes,
self.resend_pending_session_handshakes(now_ms).await);
instr_step!(instr_on, crate::instr::Domain::Tick, crate::instr::Step::ResendPendingSessionMsg3,
self.resend_pending_session_msg3(now_ms).await);
instr_step!(instr_on, crate::instr::Domain::Tick, crate::instr::Step::PurgeIdleSessions,
self.purge_idle_sessions(now_ms));
instr_step!(instr_on, crate::instr::Domain::Tick, crate::instr::Step::ProcessPendingRetries,
self.process_pending_retries(now_ms).await);
instr_step!(instr_on, crate::instr::Domain::Tick, crate::instr::Step::CheckTreeState,
self.check_tree_state().await);
instr_step!(instr_on, crate::instr::Domain::Tick, crate::instr::Step::CheckBloomState,
self.check_bloom_state().await);
instr_step!(instr_on, crate::instr::Domain::Tick, crate::instr::Step::ComputeMeshSize,
self.compute_mesh_size());
instr_step!(instr_on, crate::instr::Domain::Tick, crate::instr::Step::RecordStatsHistory,
self.record_stats_history());
instr_step!(instr_on, crate::instr::Domain::Tick, crate::instr::Step::CheckMmpReports,
self.check_mmp_reports().await);
instr_step!(instr_on, crate::instr::Domain::Tick, crate::instr::Step::CheckSessionMmpReports,
self.check_session_mmp_reports().await);
instr_step!(instr_on, crate::instr::Domain::Tick, crate::instr::Step::CheckLinkHeartbeats,
self.check_link_heartbeats().await);
instr_step!(instr_on, crate::instr::Domain::Tick, crate::instr::Step::CheckRekey,
self.check_rekey().await);
instr_step!(instr_on, crate::instr::Domain::Tick, crate::instr::Step::CheckSessionRekey,
self.check_session_rekey().await);
instr_step!(instr_on, crate::instr::Domain::Tick, crate::instr::Step::CheckPendingLookups,
self.check_pending_lookups(now_ms).await);
instr_step!(instr_on, crate::instr::Domain::Tick, crate::instr::Step::PollTransportDiscovery,
self.poll_transport_discovery().await);
instr_step!(instr_on, crate::instr::Domain::Tick, crate::instr::Step::SampleTransportCongestion,
self.sample_transport_congestion());
#[cfg(any(target_os = "linux", target_os = "macos"))]
instr_step!(instr_on, crate::instr::Domain::Tick, crate::instr::Step::ActivateConnectedUdpSessions,
self.activate_connected_udp_sessions().await);
// Debug-build sweep of the peer-lifecycle map invariant
// (leaked machines / machine-less legs); two map scans,
// compiled out of release builds.
#[cfg(debug_assertions)]
instr_step!(instr_on, crate::instr::Domain::Tick, crate::instr::Step::DebugAssertPeerMapsCoherent,
self.debug_assert_peer_maps_coherent());
});
crate::instr::tick_gauges(instr_on, self.peers.len() as u64);
}
// Shutdown signal → enter the bounded drain in place, ONCE.
// Gated on `is_none()` so it only fires while serving; after
// entering drain the arm is disabled (the completed signal is
// never polled again) and the deadline arm below bounds the
// window. Placed after the real arms so their `biased` priority
// is unchanged, and inert while serving with `pending()`.
_ = &mut shutdown, if drain_deadline.is_none() => {
self.enter_drain().await;
drain_deadline =
Some(tokio::time::Instant::now() + self.config().node.drain_timeout());
}
// Bounded drain deadline (drain mode only). Placed LAST so the
// `biased` priority of the normal arms is unchanged, and gated
// on `is_some()` so in normal mode the branch is disabled — the
// future is created but never polled and never fires.
_ = tokio::time::sleep_until(
drain_deadline.unwrap_or_else(tokio::time::Instant::now)
), if drain_deadline.is_some() => {
info!("Drain deadline elapsed, ending drain loop");
break;
}
}
}
@@ -319,12 +462,16 @@ impl Node {
// though no msg1/msg2 exchange can ever succeed. Bump the
// discovery-layer cooldown to the long protocol-mismatch
// window and emit a single WARN per fresh observation.
if self.bootstrap_transports.contains(&packet.transport_id)
if self
.supervisor
.nostr_rendezvous
.is_bootstrap_transport(&packet.transport_id)
&& let Some(npub) = self
.bootstrap_transport_npubs
.get(&packet.transport_id)
.supervisor
.nostr_rendezvous
.bootstrap_transport_npub(&packet.transport_id)
.cloned()
&& let Some(handle) = self.nostr_discovery_handle()
&& let Some(handle) = self.nostr_rendezvous_handle()
{
let now_ms = Self::now_ms();
let cooldown_secs = handle.protocol_mismatch_cooldown_secs();
src/node/decrypt_worker.rs
+5 -5
View File
@@ -561,7 +561,7 @@ mod tests {
let open_cipher = LessSafeKey::new(unbound2);
let counter: u64 = 7;
const HDR: usize = crate::node::wire::ESTABLISHED_HEADER_SIZE;
const HDR: usize = crate::proto::fmp::wire::ESTABLISHED_HEADER_SIZE;
// Build a wire packet `[16-byte header][4-byte inner ts][1 byte link msg]`
// with capacity for the trailing AEAD tag. Header bytes
// double as AAD and as the on-wire prefix.
@@ -569,7 +569,7 @@ mod tests {
// Header: fill the flags byte (the second byte) with both
// FLAG_CE and FLAG_SP set; the rest is uninterpreted by the
// worker (it just AADs the whole 16 bytes).
let flags_byte = crate::node::wire::FLAG_CE | crate::node::wire::FLAG_SP;
let flags_byte = crate::proto::fmp::wire::FLAG_CE | crate::proto::fmp::wire::FLAG_SP;
let mut header = [0u8; HDR];
header[1] = flags_byte;
wire.extend_from_slice(&header);
@@ -626,11 +626,11 @@ mod tests {
"fmp_flags must round-trip from DecryptJob to DecryptFallback"
);
assert!(
fallback.fmp_flags & crate::node::wire::FLAG_CE != 0,
fallback.fmp_flags & crate::proto::fmp::wire::FLAG_CE != 0,
"FLAG_CE bit lost on worker path"
);
assert!(
fallback.fmp_flags & crate::node::wire::FLAG_SP != 0,
fallback.fmp_flags & crate::proto::fmp::wire::FLAG_SP != 0,
"FLAG_SP bit lost on worker path"
);
}
@@ -724,7 +724,7 @@ mod tests {
let open_cipher = LessSafeKey::new(unbound);
let counter: u64 = 11;
const HDR: usize = crate::node::wire::ESTABLISHED_HEADER_SIZE;
const HDR: usize = crate::proto::fmp::wire::ESTABLISHED_HEADER_SIZE;
let header = [0u8; HDR];
let mut wire = Vec::with_capacity(HDR + 4 + 1 + 16);
wire.extend_from_slice(&header);
src/node/discovery_rate_limit.rs
-376
View File
@@ -1,376 +0,0 @@
//! Discovery protocol rate limiting and backoff.
//!
//! Two complementary mechanisms:
//!
//! - **`DiscoveryBackoff`** (originator-side, optional): Exponential
//! suppression of fresh lookups after the per-attempt sequence in
//! `node.discovery.attempt_timeouts_secs` has been exhausted.
//! **Disabled by default** (base/cap = 0); the per-attempt sequence
//! is the only retry pacing in the standard configuration. Reset on
//! topology changes (parent change, new peer, first RTT, reconnection).
//!
//! - **`DiscoveryForwardRateLimiter`** (transit-side): Per-target minimum
//! interval for forwarded requests. Defense-in-depth against misbehaving
//! nodes generating fresh request_ids at high rate.
use crate::NodeAddr;
use std::collections::HashMap;
use std::time::{Duration, Instant};
// ============================================================================
// Originator-side: Discovery Backoff
// ============================================================================
/// Default base backoff after first lookup failure. `0` = disabled.
const DEFAULT_BACKOFF_BASE_SECS: u64 = 0;
/// Default maximum backoff cap. `0` = disabled.
const DEFAULT_BACKOFF_MAX_SECS: u64 = 0;
/// Backoff multiplier per consecutive failure.
const BACKOFF_MULTIPLIER: u64 = 2;
/// Exponential backoff for failed discovery lookups.
///
/// Tracks targets whose lookups have timed out and suppresses
/// re-initiation with increasing delays. Cleared on topology changes.
pub struct DiscoveryBackoff {
/// Maps target → (suppress_until, consecutive_failures).
entries: HashMap<NodeAddr, BackoffEntry>,
/// Base backoff duration (first failure).
base: Duration,
/// Maximum backoff cap.
max: Duration,
}
struct BackoffEntry {
/// Don't re-initiate until this instant.
suppress_until: Instant,
/// Consecutive failures (drives exponential backoff).
failures: u32,
}
impl DiscoveryBackoff {
/// Create with default parameters (disabled — base/cap = 0).
pub fn new() -> Self {
Self::with_params(DEFAULT_BACKOFF_BASE_SECS, DEFAULT_BACKOFF_MAX_SECS)
}
/// Create with custom base and max backoff in seconds.
pub fn with_params(base_secs: u64, max_secs: u64) -> Self {
Self {
entries: HashMap::new(),
base: Duration::from_secs(base_secs),
max: Duration::from_secs(max_secs),
}
}
/// Check if a lookup for this target is suppressed.
///
/// Returns true if the target is in backoff and should not be
/// looked up yet.
pub fn is_suppressed(&self, target: &NodeAddr) -> bool {
if let Some(entry) = self.entries.get(target) {
Instant::now() < entry.suppress_until
} else {
false
}
}
/// Record a lookup failure (timeout) for a target.
///
/// Increments the failure count and sets the next suppression
/// window using exponential backoff.
pub fn record_failure(&mut self, target: &NodeAddr) {
let now = Instant::now();
let failures = self.entries.get(target).map_or(0, |e| e.failures) + 1;
let backoff_secs = self
.base
.as_secs()
.saturating_mul(BACKOFF_MULTIPLIER.saturating_pow(failures.saturating_sub(1)));
let backoff = Duration::from_secs(backoff_secs.min(self.max.as_secs()));
self.entries.insert(
*target,
BackoffEntry {
suppress_until: now + backoff,
failures,
},
);
}
/// Record a successful lookup — remove backoff for this target.
pub fn record_success(&mut self, target: &NodeAddr) {
self.entries.remove(target);
}
/// Clear all backoff entries.
///
/// Called on topology changes that might make previously-unreachable
/// targets reachable (parent change, new peer, first RTT, reconnection).
pub fn reset_all(&mut self) {
self.entries.clear();
}
/// Whether any entries exist.
pub fn is_empty(&self) -> bool {
self.entries.is_empty()
}
/// Current number of entries.
pub fn entry_count(&self) -> usize {
self.entries.len()
}
/// Get the failure count for a target (for logging).
pub fn failure_count(&self, target: &NodeAddr) -> u32 {
self.entries.get(target).map_or(0, |e| e.failures)
}
#[cfg(test)]
pub fn len(&self) -> usize {
self.entries.len()
}
}
impl Default for DiscoveryBackoff {
fn default() -> Self {
Self::new()
}
}
// ============================================================================
// Transit-side: Discovery Forward Rate Limiter
// ============================================================================
/// Default minimum interval between forwarded lookups for the same target.
const DEFAULT_FORWARD_MIN_INTERVAL: Duration = Duration::from_secs(2);
/// Maximum age of entries before cleanup.
const FORWARD_MAX_AGE: Duration = Duration::from_secs(60);
/// Rate limiter for forwarded discovery requests.
///
/// Tracks the last time a LookupRequest was forwarded for each target
/// and enforces a minimum interval to prevent floods from misbehaving
/// nodes generating fresh request_ids.
pub struct DiscoveryForwardRateLimiter {
last_forwarded: HashMap<NodeAddr, Instant>,
min_interval: Duration,
max_age: Duration,
}
impl DiscoveryForwardRateLimiter {
/// Create with default parameters (2s interval).
pub fn new() -> Self {
Self {
last_forwarded: HashMap::new(),
min_interval: DEFAULT_FORWARD_MIN_INTERVAL,
max_age: FORWARD_MAX_AGE,
}
}
/// Create with a custom minimum interval.
pub fn with_interval(min_interval: Duration) -> Self {
Self {
last_forwarded: HashMap::new(),
min_interval,
max_age: FORWARD_MAX_AGE,
}
}
/// Check if we should forward a lookup for this target.
///
/// Returns true if enough time has passed since the last forward
/// for this target. Updates internal state when returning true.
pub fn should_forward(&mut self, target: &NodeAddr) -> bool {
let now = Instant::now();
if let Some(&last) = self.last_forwarded.get(target)
&& now.duration_since(last) < self.min_interval
{
return false;
}
self.last_forwarded.insert(*target, now);
self.cleanup(now);
true
}
/// Replace the minimum interval (e.g., set to zero to disable).
#[cfg(test)]
pub fn set_interval(&mut self, interval: Duration) {
self.min_interval = interval;
}
/// Remove entries older than max_age.
fn cleanup(&mut self, now: Instant) {
self.last_forwarded
.retain(|_, &mut last| now.duration_since(last) < self.max_age);
}
#[cfg(test)]
pub fn len(&self) -> usize {
self.last_forwarded.len()
}
}
impl Default for DiscoveryForwardRateLimiter {
fn default() -> Self {
Self::new()
}
}
// ============================================================================
// Tests
// ============================================================================
#[cfg(test)]
mod tests {
use super::*;
use std::thread;
fn addr(val: u8) -> NodeAddr {
let mut bytes = [0u8; 16];
bytes[0] = val;
NodeAddr::from_bytes(bytes)
}
// --- DiscoveryBackoff tests ---
#[test]
fn test_backoff_not_suppressed_initially() {
let backoff = DiscoveryBackoff::new();
assert!(!backoff.is_suppressed(&addr(1)));
}
#[test]
fn test_backoff_suppressed_after_failure() {
// Backoff is opt-in; exercise the suppression path with explicit params.
let mut backoff = DiscoveryBackoff::with_params(30, 300);
backoff.record_failure(&addr(1));
assert!(backoff.is_suppressed(&addr(1)));
// Different target not affected
assert!(!backoff.is_suppressed(&addr(2)));
}
#[test]
fn test_backoff_cleared_on_success() {
let mut backoff = DiscoveryBackoff::with_params(30, 300);
backoff.record_failure(&addr(1));
assert!(backoff.is_suppressed(&addr(1)));
backoff.record_success(&addr(1));
assert!(!backoff.is_suppressed(&addr(1)));
}
#[test]
fn test_backoff_reset_all() {
let mut backoff = DiscoveryBackoff::new();
backoff.record_failure(&addr(1));
backoff.record_failure(&addr(2));
assert_eq!(backoff.len(), 2);
backoff.reset_all();
assert_eq!(backoff.len(), 0);
assert!(!backoff.is_suppressed(&addr(1)));
}
#[test]
fn test_backoff_exponential() {
let mut backoff = DiscoveryBackoff::with_params(1, 300);
// First failure: 1s backoff
backoff.record_failure(&addr(1));
assert_eq!(backoff.failure_count(&addr(1)), 1);
// Second failure: 2s backoff
backoff.record_failure(&addr(1));
assert_eq!(backoff.failure_count(&addr(1)), 2);
// Third failure: 4s backoff
backoff.record_failure(&addr(1));
assert_eq!(backoff.failure_count(&addr(1)), 3);
}
#[test]
fn test_backoff_expires() {
let mut backoff = DiscoveryBackoff::with_params(0, 0);
backoff.record_failure(&addr(1));
// With 0s backoff, should not be suppressed
assert!(!backoff.is_suppressed(&addr(1)));
}
#[test]
fn test_backoff_capped() {
let mut backoff = DiscoveryBackoff::with_params(1, 10);
// Record many failures
for _ in 0..20 {
backoff.record_failure(&addr(1));
}
// Backoff should be capped at max (10s), not overflow
let entry = backoff.entries.get(&addr(1)).unwrap();
let remaining = entry.suppress_until.duration_since(Instant::now());
assert!(remaining <= Duration::from_secs(11));
}
// --- DiscoveryForwardRateLimiter tests ---
#[test]
fn test_forward_first_allowed() {
let mut limiter = DiscoveryForwardRateLimiter::new();
assert!(limiter.should_forward(&addr(1)));
}
#[test]
fn test_forward_rapid_rate_limited() {
let mut limiter = DiscoveryForwardRateLimiter::new();
assert!(limiter.should_forward(&addr(1)));
assert!(!limiter.should_forward(&addr(1)));
assert!(!limiter.should_forward(&addr(1)));
}
#[test]
fn test_forward_different_targets_independent() {
let mut limiter = DiscoveryForwardRateLimiter::new();
assert!(limiter.should_forward(&addr(1)));
assert!(limiter.should_forward(&addr(2)));
assert!(!limiter.should_forward(&addr(1)));
assert!(!limiter.should_forward(&addr(2)));
}
#[test]
fn test_forward_allowed_after_interval() {
let mut limiter = DiscoveryForwardRateLimiter::with_interval(Duration::from_millis(100));
assert!(limiter.should_forward(&addr(1)));
thread::sleep(Duration::from_millis(110));
assert!(limiter.should_forward(&addr(1)));
}
#[test]
fn test_forward_cleanup_removes_old() {
let mut limiter = DiscoveryForwardRateLimiter::new();
assert!(limiter.should_forward(&addr(1)));
assert!(limiter.should_forward(&addr(2)));
assert_eq!(limiter.len(), 2);
let future = Instant::now() + Duration::from_secs(61);
limiter.cleanup(future);
assert_eq!(limiter.len(), 0);
}
#[test]
fn test_forward_cleanup_preserves_recent() {
let mut limiter = DiscoveryForwardRateLimiter::new();
assert!(limiter.should_forward(&addr(1)));
assert_eq!(limiter.len(), 1);
limiter.cleanup(Instant::now());
assert_eq!(limiter.len(), 1);
}
}
src/node/encrypt_worker.rs
+16 -19
View File
@@ -50,9 +50,9 @@
// warnings rather than gate every function individually.
#![cfg_attr(not(unix), allow(dead_code))]
use crate::node::session_wire::FSP_HEADER_SIZE;
use crate::node::wire::ESTABLISHED_HEADER_SIZE;
use crate::transport::udp::socket::AsyncUdpSocket;
use crate::proto::fmp::wire::ESTABLISHED_HEADER_SIZE;
use crate::proto::fsp::wire::FSP_HEADER_SIZE;
use crate::transport::udp::io::AsyncUdpSocket;
#[cfg(not(target_os = "macos"))]
use crossbeam_channel::{Receiver, SendError, Sender, TrySendError, bounded};
use ring::aead::{Aad, LessSafeKey, Nonce};
@@ -132,8 +132,7 @@ pub(crate) struct FmpSendJob {
/// the job completes and the worker drops it, only the peer's
/// strong ref remains.
#[cfg(any(target_os = "linux", target_os = "macos"))]
pub connected_socket:
Option<std::sync::Arc<crate::transport::udp::connected_peer::ConnectedPeerSocket>>,
pub connected_socket: Option<std::sync::Arc<crate::peer::connected_udp::ConnectedPeerSocket>>,
/// Bulk endpoint data may be dropped when the kernel reports UDP
/// send-queue exhaustion. Control/rekey frames keep retrying so
/// congestion cannot strand the session.
@@ -716,8 +715,7 @@ fn mac_now_ms() -> u64 {
struct MacSequencedSendFlow {
key: MacSendFlowKey,
socket: AsyncUdpSocket,
connected_socket:
Option<std::sync::Arc<crate::transport::udp::connected_peer::ConnectedPeerSocket>>,
connected_socket: Option<std::sync::Arc<crate::peer::connected_udp::ConnectedPeerSocket>>,
dest_addr: SocketAddr,
next_seq: std::sync::atomic::AtomicU64,
last_used_ms: std::sync::atomic::AtomicU64,
@@ -754,9 +752,7 @@ impl MacSequencedSendFlow {
fn spawn(
key: MacSendFlowKey,
socket: AsyncUdpSocket,
connected_socket: Option<
std::sync::Arc<crate::transport::udp::connected_peer::ConnectedPeerSocket>,
>,
connected_socket: Option<std::sync::Arc<crate::peer::connected_udp::ConnectedPeerSocket>>,
dest_addr: SocketAddr,
now_ms: u64,
) -> Arc<Self> {
@@ -1024,8 +1020,7 @@ fn flush_batch_sync(
struct EncryptedGroup {
socket: AsyncUdpSocket,
#[cfg(any(target_os = "linux", target_os = "macos"))]
connected_socket:
Option<std::sync::Arc<crate::transport::udp::connected_peer::ConnectedPeerSocket>>,
connected_socket: Option<std::sync::Arc<crate::peer::connected_udp::ConnectedPeerSocket>>,
dest_addr: SocketAddr,
wire_packets: Vec<Vec<u8>>,
drop_on_backpressure: bool,
@@ -1710,7 +1705,7 @@ fn send_batch_gso(
}
/// Direct `sendmmsg(2)` wrapper for the sync worker. The
/// `transport::udp::socket` module's existing `send_batch` is
/// `transport::udp::io` module's existing `send_batch` is
/// pub(crate) on `UdpRawSocket`, but we don't have a handle to the
/// raw socket from here — we just have the FD. Re-implementing
/// inline is ~15 lines and avoids tunnelling the inner socket
@@ -1780,7 +1775,7 @@ fn send_batch_raw(
#[cfg(all(test, unix))]
mod unix_tests {
use super::*;
use crate::transport::udp::socket::UdpRawSocket;
use crate::transport::udp::io::UdpRawSocket;
use ring::aead::{LessSafeKey, UnboundKey};
use std::net::UdpSocket;
@@ -1904,10 +1899,12 @@ mod unix_tests {
#[test]
fn pipelined_send_wire_layout_roundtrips_canonical_decoders() {
use crate::NodeAddr;
use crate::node::session_wire::build_fsp_header;
use crate::node::wire::{EncryptedHeader, FLAG_KEY_EPOCH, build_established_header};
use crate::noise::TAG_SIZE;
use crate::protocol::{LinkMessageType, SESSION_DATAGRAM_HEADER_SIZE, SessionDatagramRef};
use crate::proto::fmp::wire::{EncryptedHeader, FLAG_KEY_EPOCH, build_established_header};
use crate::proto::fsp::wire::build_fsp_header;
use crate::proto::link::{
LinkMessageType, SESSION_DATAGRAM_HEADER_SIZE, SessionDatagramRef,
};
use crate::utils::index::SessionIndex;
let rt = tokio::runtime::Builder::new_current_thread()
@@ -2218,7 +2215,7 @@ mod tests {
/// AsRawFd impl.
#[test]
fn flush_batch_routes_each_target_separately() {
use crate::transport::udp::socket::UdpRawSocket;
use crate::transport::udp::io::UdpRawSocket;
use ring::aead::{LessSafeKey, UnboundKey};
use std::net::UdpSocket;
@@ -2264,7 +2261,7 @@ mod tests {
const B_WIRE: usize = 16 + B_PLAINTEXT + 16; // 96
fn make_job(
socket: crate::transport::udp::socket::AsyncUdpSocket,
socket: crate::transport::udp::io::AsyncUdpSocket,
cipher: &LessSafeKey,
counter: u64,
dest: SocketAddr,
src/node/handlers/discovery.rs
-753
View File
@@ -1,753 +0,0 @@
//! LookupRequest/LookupResponse discovery protocol handlers.
//!
//! Handles coordinate discovery via bloom-filter-guided tree routing.
//! Requests are forwarded only to tree peers (parent + children) whose
//! bloom filter contains the target. TTL and request_id dedup provide
//! safety bounds.
use crate::node::reject::DiscoveryReject;
use crate::node::{Node, RecentRequest};
use crate::protocol::{LookupRequest, LookupResponse};
use crate::transport::{TransportAddr, TransportId};
use crate::{NodeAddr, PeerIdentity};
use tracing::{debug, info, trace, warn};
const MAX_RECENT_DISCOVERY_REQUESTS: usize = 4096;
impl Node {
/// Handle an incoming LookupRequest from a peer.
///
/// Processing steps:
/// 1. Decode and validate
/// 2. Check request_id for duplicates (dedup / reverse-path routing)
/// 3. Record request for reverse-path forwarding
/// 4. Lazy purge expired entries
/// 5. If we're the target, generate and send response
/// 6. If TTL > 0, forward to tree peers whose bloom filter matches
pub(in crate::node) async fn handle_lookup_request(&mut self, from: &NodeAddr, payload: &[u8]) {
self.metrics().discovery.req_received.inc();
let request = match LookupRequest::decode(payload) {
Ok(req) => req,
Err(e) => {
self.metrics()
.discovery
.record_reject(DiscoveryReject::ReqDecodeError);
debug!(from = %self.peer_display_name(from), error = %e, "Malformed LookupRequest");
return;
}
};
let now_ms = Self::now_ms();
self.purge_expired_requests(now_ms);
// Dedup: drop if we've already seen this request_id.
// Also serves as loop protection — tree routing is loop-free,
// but request_id dedup catches edge cases during tree restructuring.
if self.recent_requests.contains_key(&request.request_id) {
self.metrics()
.discovery
.record_reject(DiscoveryReject::ReqDuplicate);
debug!(
request_id = request.request_id,
from = %self.peer_display_name(from),
"Duplicate LookupRequest, dropping"
);
return;
}
if self.recent_requests.len() >= MAX_RECENT_DISCOVERY_REQUESTS {
self.metrics()
.discovery
.record_reject(DiscoveryReject::ReqDedupCacheFull);
debug!(
request_id = request.request_id,
from = %self.peer_display_name(from),
recent_requests = self.recent_requests.len(),
max_recent_requests = MAX_RECENT_DISCOVERY_REQUESTS,
"Discovery request dedup cache full, dropping LookupRequest"
);
return;
}
// Record for reverse-path forwarding and dedup
self.recent_requests
.insert(request.request_id, RecentRequest::new(*from, now_ms));
// Are we the target?
if request.target == *self.node_addr() {
self.metrics().discovery.req_target_is_us.inc();
debug!(
request_id = request.request_id,
origin = %self.peer_display_name(&request.origin),
"We are the lookup target, generating response"
);
self.send_lookup_response(&request).await;
return;
}
// Forward if TTL permits
if request.can_forward() {
// Transit-side rate limit: collapse rapid-fire lookups for the
// same target from misbehaving nodes generating fresh request_ids.
if !self
.discovery_forward_limiter
.should_forward(&request.target)
{
self.metrics().discovery.req_forward_rate_limited.inc();
debug!(
request_id = request.request_id,
target = %self.peer_display_name(&request.target),
"Forward rate limited, suppressing LookupRequest"
);
return;
}
self.metrics().discovery.req_forwarded.inc();
self.forward_lookup_request(request).await;
} else {
self.metrics()
.discovery
.record_reject(DiscoveryReject::ReqTtlExhausted);
debug!(
request_id = request.request_id,
target = %self.peer_display_name(&request.target),
"LookupRequest TTL exhausted"
);
}
}
/// Handle an incoming LookupResponse from a peer.
///
/// Processing steps:
/// 1. Decode and validate
/// 2. Check recent_requests to determine if we originated or are forwarding
/// 3. If originator: verify proof signature, then cache target_coords and path_mtu in coord_cache
/// 4. If transit: apply path_mtu min(outgoing_link_mtu), reverse-path forward to from_peer
pub(in crate::node) async fn handle_lookup_response(
&mut self,
from: &NodeAddr,
payload: &[u8],
) {
self.metrics().discovery.resp_received.inc();
let mut response = match LookupResponse::decode(payload) {
Ok(resp) => resp,
Err(e) => {
self.metrics()
.discovery
.record_reject(DiscoveryReject::RespDecodeError);
debug!(from = %self.peer_display_name(from), error = %e, "Malformed LookupResponse");
return;
}
};
let now_ms = Self::now_ms();
// Check if we forwarded this request (transit node) or originated it
if let Some(recent) = self.recent_requests.get_mut(&response.request_id) {
// Already forwarded a response for this request — drop to
// prevent response routing loops.
if recent.response_forwarded {
debug!(
request_id = response.request_id,
target = %self.peer_display_name(&response.target),
"Response already forwarded for this request, dropping"
);
return;
}
recent.response_forwarded = true;
// Transit node: reverse-path forward
let from_peer = recent.from_peer;
self.metrics().discovery.resp_forwarded.inc();
// Apply path_mtu min() from the outgoing link's transport MTU
self.apply_outgoing_link_mtu_to_response(&mut response, &from_peer);
debug!(
request_id = response.request_id,
target = %self.peer_display_name(&response.target),
next_hop = %self.peer_display_name(&from_peer),
path_mtu = response.path_mtu,
"Reverse-path forwarding LookupResponse"
);
let encoded = response.encode();
if let Err(e) = self.send_encrypted_link_message(&from_peer, &encoded).await {
debug!(
next_hop = %self.peer_display_name(&from_peer),
error = %e,
"Failed to forward LookupResponse"
);
}
} else {
// We originated this request — verify proof before caching
let target = response.target;
let path_mtu = response.path_mtu;
// Look up the target's public key from identity_cache
let mut prefix = [0u8; 15];
prefix.copy_from_slice(&target.as_bytes()[0..15]);
let target_pubkey = match self.lookup_by_fips_prefix(&prefix) {
Some((_addr, pubkey)) => pubkey,
None => {
self.metrics()
.discovery
.record_reject(DiscoveryReject::RespIdentityMiss);
warn!(
request_id = response.request_id,
target = %self.peer_display_name(&target),
"identity_cache miss for lookup target, cannot verify proof"
);
return;
}
};
// Verify the proof signature
let (xonly, _parity) = target_pubkey.x_only_public_key();
let peer_id = PeerIdentity::from_pubkey(xonly);
let proof_data =
LookupResponse::proof_bytes(response.request_id, &target, &response.target_coords);
if !peer_id.verify(&proof_data, &response.proof) {
self.metrics()
.discovery
.record_reject(DiscoveryReject::RespProofFailed);
warn!(
request_id = response.request_id,
target = %self.peer_display_name(&target),
"LookupResponse proof verification failed, discarding"
);
return;
}
self.metrics().discovery.resp_accepted.inc();
// Clear backoff on success — target is reachable
self.discovery_backoff.record_success(&target);
info!(
request_id = response.request_id,
target = %self.peer_display_name(&target),
depth = response.target_coords.depth(),
path_mtu = path_mtu,
"Discovery succeeded, proof verified, route cached"
);
self.coord_cache
.insert_with_path_mtu(target, response.target_coords, now_ms, path_mtu);
// Mirror path_mtu into the FipsAddress-keyed read-only lookup
// map used by the TUN reader/writer at TCP MSS clamp time.
let fips_addr = crate::FipsAddress::from_node_addr(&target);
match self.path_mtu_lookup.write() {
Ok(mut map) => match map.get(&fips_addr).copied() {
Some(existing) if existing <= path_mtu => {
// Keep the tighter learned value; never loosen the
// clamp. A reactive MtuExceeded or PathMtuNotification
// tighten takes precedence over a looser discovery
// estimate (cross-carrier keep-tighter).
debug!(
target = %self.peer_display_name(&target),
fips_addr = %fips_addr,
path_mtu = path_mtu,
existing = existing,
"LookupResponse: keeping tighter existing path_mtu_lookup value"
);
}
other => {
map.insert(fips_addr, path_mtu);
debug!(
target = %self.peer_display_name(&target),
fips_addr = %fips_addr,
path_mtu = path_mtu,
prior = ?other,
map_len = map.len(),
"Wrote path_mtu_lookup from discovery LookupResponse"
);
}
},
Err(e) => {
warn!(
target = %self.peer_display_name(&target),
fips_addr = %fips_addr,
path_mtu = path_mtu,
error = %e,
"path_mtu_lookup write lock poisoned; clamp will not see this update"
);
}
}
// Clean up pending lookup tracking
self.pending_lookups.remove(&target);
// If an established session exists, reset the warmup counter.
let n = self.config().node.session.coords_warmup_packets;
if let Some(entry) = self.sessions.get_mut(&target)
&& entry.is_established()
{
entry.set_coords_warmup_remaining(n);
debug!(
dest = %self.peer_display_name(&target),
warmup_packets = n,
"Reset coords warmup after discovery for existing session"
);
}
// If we have pending TUN packets for this target, retry session
// initiation. The coord_cache now has coords, so find_next_hop()
// should succeed.
if let Some(packets) = self.pending_tun_packets.get(&target) {
debug!(
dest = %self.peer_display_name(&target),
queued_packets = packets.len(),
"Retrying queued packets after discovery"
);
self.retry_session_after_discovery(target).await;
}
}
}
/// Generate and send a LookupResponse when we are the target.
async fn send_lookup_response(&mut self, request: &LookupRequest) {
let our_coords = self.tree_state().my_coords().clone();
// Sign proof: Identity::sign hashes with SHA-256 internally
let proof_data =
LookupResponse::proof_bytes(request.request_id, &request.target, &our_coords);
let proof = self.identity().sign(&proof_data);
let mut response =
LookupResponse::new(request.request_id, request.target, our_coords, proof);
// Route toward origin via reverse path.
let next_hop_addr = if let Some(recent) = self.recent_requests.get(&request.request_id) {
recent.from_peer
} else {
// Fallback: try greedy tree routing toward origin
match self.find_next_hop(&request.origin) {
Some(peer) => *peer.node_addr(),
None => {
debug!(
origin = %self.peer_display_name(&request.origin),
"Cannot route LookupResponse: no reverse path or tree route to origin"
);
self.metrics()
.discovery
.record_reject(DiscoveryReject::RespNoRoute);
return;
}
}
};
// Fold our outgoing-link MTU into path_mtu so the target-edge link
// appears in the bottleneck calculation. Without this, the response
// leaves the target with path_mtu = u16::MAX and only intermediate
// transits min-fold; the target's first reverse-path hop is missed.
self.apply_outgoing_link_mtu_to_response(&mut response, &next_hop_addr);
debug!(
request_id = request.request_id,
origin = %self.peer_display_name(&request.origin),
next_hop = %self.peer_display_name(&next_hop_addr),
path_mtu = response.path_mtu,
"Sending LookupResponse"
);
let encoded = response.encode();
if let Err(e) = self
.send_encrypted_link_message(&next_hop_addr, &encoded)
.await
{
debug!(
next_hop = %self.peer_display_name(&next_hop_addr),
error = %e,
"Failed to send LookupResponse"
);
}
}
/// Forward a LookupRequest to eligible peers.
///
/// Primary path: tree peers (parent + children) whose bloom filter
/// contains the target. Restricting to tree peers follows the spanning
/// tree partition, producing a single directed path.
///
/// Fallback: if no tree peer's bloom matches, try non-tree peers whose
/// bloom contains the target. This recovers from dead ends caused by
/// stale bloom filters, tree restructuring, or transit node failures.
async fn forward_lookup_request(&mut self, mut request: LookupRequest) {
if !request.forward() {
return;
}
// Collect tree peers whose bloom filter contains the target
let forward_to: Vec<NodeAddr> = self
.peers
.iter()
.filter(|(addr, peer)| self.is_tree_peer(addr) && peer.may_reach(&request.target))
.map(|(addr, _)| *addr)
.collect();
// Fallback: if no tree peer matches, try non-tree bloom-matching peers
let (forward_to, used_fallback) = if forward_to.is_empty() {
let fallback: Vec<NodeAddr> = self
.peers
.iter()
.filter(|(addr, peer)| !self.is_tree_peer(addr) && peer.may_reach(&request.target))
.map(|(addr, _)| *addr)
.collect();
if fallback.is_empty() {
self.metrics().discovery.req_no_tree_peer.inc();
trace!(
request_id = request.request_id,
"No eligible peers to forward LookupRequest"
);
return;
}
(fallback, true)
} else {
(forward_to, false)
};
if used_fallback {
self.metrics().discovery.req_fallback_forwarded.inc();
debug!(
request_id = request.request_id,
target = %self.peer_display_name(&request.target),
ttl = request.ttl,
peer_count = forward_to.len(),
"Forwarding LookupRequest via non-tree fallback"
);
} else {
debug!(
request_id = request.request_id,
target = %self.peer_display_name(&request.target),
ttl = request.ttl,
peer_count = forward_to.len(),
"Forwarding LookupRequest"
);
}
let encoded = request.encode();
for peer_addr in forward_to {
if let Err(e) = self.send_encrypted_link_message(&peer_addr, &encoded).await {
debug!(
peer = %self.peer_display_name(&peer_addr),
error = %e,
"Failed to forward LookupRequest to peer"
);
}
}
}
/// Initiate a discovery lookup for a target node.
///
/// Creates a LookupRequest and sends it to tree peers whose bloom
/// filters contain the target. Returns the number of peers sent to.
/// The originator does NOT record the request_id in recent_requests,
/// so when the response arrives, it's recognized as "our request".
pub(in crate::node) async fn initiate_lookup(&mut self, target: &NodeAddr, ttl: u8) -> usize {
self.metrics().discovery.req_initiated.inc();
let origin = *self.node_addr();
let origin_coords = self.tree_state().my_coords().clone();
let request = LookupRequest::generate(*target, origin, origin_coords, ttl, 0);
// Send only to tree peers whose bloom filter contains the target
let peer_addrs: Vec<NodeAddr> = self
.peers
.iter()
.filter(|(addr, peer)| self.is_tree_peer(addr) && peer.may_reach(target))
.map(|(addr, _)| *addr)
.collect();
let peer_count = peer_addrs.len();
debug!(
request_id = request.request_id,
target = %self.peer_display_name(target),
ttl = ttl,
peer_count = peer_count,
total_peers = self.peers.len(),
"Discovery lookup initiated"
);
if peer_count == 0 {
return 0;
}
let encoded = request.encode();
for peer_addr in peer_addrs {
if let Err(e) = self.send_encrypted_link_message(&peer_addr, &encoded).await {
debug!(
peer = %self.peer_display_name(&peer_addr),
error = %e,
"Failed to send LookupRequest to peer"
);
}
}
peer_count
}
/// Initiate a discovery lookup if one is not already pending for this target.
///
/// Checks: pending dedup, post-failure backoff (off by default), bloom
/// filter pre-check. If all pass, sends the first attempt's LookupRequest.
/// Subsequent attempts (with fresh request_ids) are scheduled by
/// [`Self::check_pending_lookups`] when each attempt's per-attempt timeout
/// expires, using the sequence in `node.discovery.attempt_timeouts_secs`.
pub(in crate::node) async fn maybe_initiate_lookup(&mut self, dest: &NodeAddr) {
let now_ms = Self::now_ms();
// Dedup: any pending lookup means we are already trying.
if self.pending_lookups.contains_key(dest) {
self.metrics().discovery.req_deduplicated.inc();
debug!(
target_node = %self.peer_display_name(dest),
"Discovery lookup deduplicated, already pending"
);
return;
}
// Optional post-failure suppression. Defaults are 0/0 (inert);
// operators can opt in by setting `node.discovery.backoff_*_secs`.
if self.discovery_backoff.is_suppressed(dest) {
self.metrics().discovery.req_backoff_suppressed.inc();
debug!(
target_node = %self.peer_display_name(dest),
failures = self.discovery_backoff.failure_count(dest),
"Discovery lookup suppressed by backoff"
);
return;
}
// Bloom filter pre-check: if no peer's filter contains the target,
// it's not in the mesh — skip the lookup and record as failure.
let reachable = self.peers.values().any(|peer| peer.may_reach(dest));
if !reachable {
self.metrics().discovery.req_bloom_miss.inc();
self.discovery_backoff.record_failure(dest);
debug!(
target_node = %self.peer_display_name(dest),
"Discovery skipped, target not in any peer bloom filter"
);
return;
}
self.pending_lookups
.insert(*dest, PendingLookup::new(now_ms));
let ttl = self.config().node.discovery.ttl;
let sent = self.initiate_lookup(dest, ttl).await;
// If no tree peers had the target, fail immediately
if sent == 0 {
self.pending_lookups.remove(dest);
self.discovery_backoff.record_failure(dest);
debug!(
target_node = %self.peer_display_name(dest),
"Discovery failed, no tree peers with bloom match"
);
}
}
/// Check pending lookups for next-attempt or final timeout.
///
/// Called periodically from the tick handler. The lookup state machine
/// runs through `node.discovery.attempt_timeouts_secs` (default
/// `[1, 2, 4, 8]`): each entry is the deadline for one attempt. When the
/// current attempt's deadline elapses:
/// - If more entries remain: send the next attempt with a fresh
/// `request_id`.
/// - Otherwise: declare the destination unreachable, drop queued packets,
/// and emit ICMPv6 destination-unreachable for each.
pub(in crate::node) async fn check_pending_lookups(&mut self, now_ms: u64) {
let timeouts = self.config().node.discovery.attempt_timeouts_secs.clone();
let max_attempts = timeouts.len() as u8;
// Collect targets needing action
let mut to_retry: Vec<NodeAddr> = Vec::new();
let mut to_timeout: Vec<NodeAddr> = Vec::new();
for (&target, entry) in &self.pending_lookups {
let attempt_idx = (entry.attempt as usize).saturating_sub(1);
let attempt_timeout_ms = timeouts.get(attempt_idx).copied().unwrap_or(0) * 1000;
if now_ms.saturating_sub(entry.last_sent_ms) >= attempt_timeout_ms {
if entry.attempt >= max_attempts {
to_timeout.push(target);
} else {
to_retry.push(target);
}
}
}
// Process retries
for target in to_retry {
if let Some(entry) = self.pending_lookups.get_mut(&target) {
entry.attempt += 1;
entry.last_sent_ms = now_ms;
let attempt = entry.attempt;
let ttl = self.config().node.discovery.ttl;
let sent = self.initiate_lookup(&target, ttl).await;
if sent > 0 {
debug!(
target_node = %self.peer_display_name(&target),
attempt = attempt,
"Discovery retry sent"
);
}
}
}
// Process timeouts
for addr in to_timeout {
self.metrics().discovery.resp_timed_out.inc();
self.pending_lookups.remove(&addr);
// Record failure for optional backoff
self.discovery_backoff.record_failure(&addr);
let failures = self.discovery_backoff.failure_count(&addr);
let queued = self.pending_tun_packets.remove(&addr);
let pkt_count = queued.as_ref().map_or(0, |p| p.len());
info!(
target_node = %self.peer_display_name(&addr),
queued_packets = pkt_count,
failures = failures,
"Discovery lookup timed out, destination unreachable"
);
if let Some(packets) = queued {
for pkt in &packets {
self.send_icmpv6_dest_unreachable(pkt);
}
}
}
}
/// Reset discovery backoff on topology changes.
pub(in crate::node) fn reset_discovery_backoff(&mut self) {
if !self.discovery_backoff.is_empty() {
debug!(
entries = self.discovery_backoff.entry_count(),
"Resetting discovery backoff on topology change"
);
self.discovery_backoff.reset_all();
}
}
/// Remove expired entries from the recent_requests cache.
fn purge_expired_requests(&mut self, current_time_ms: u64) {
let expiry_ms = self.config().node.discovery.recent_expiry_secs * 1000;
self.recent_requests
.retain(|_, entry| !entry.is_expired(current_time_ms, expiry_ms));
}
/// Min-fold our outgoing-link MTU into a LookupResponse's `path_mtu`.
///
/// Used at both transit-side reverse-path forward and at the target's
/// own send_lookup_response. The link MTU we apply is the MTU of the
/// transport+addr we'll use to deliver the response toward `next_hop`.
/// No-op when `next_hop` is not a directly-connected peer or its
/// transport is not registered.
pub(in crate::node) fn apply_outgoing_link_mtu_to_response(
&self,
response: &mut LookupResponse,
next_hop: &NodeAddr,
) {
if let Some(peer) = self.peers.get(next_hop)
&& let Some(tid) = peer.transport_id()
&& let Some(transport) = self.transports.get(&tid)
{
let link_mtu = if let Some(addr) = peer.current_addr() {
transport.link_mtu(addr)
} else {
transport.mtu()
};
response.path_mtu = response.path_mtu.min(link_mtu);
}
}
/// Seed `path_mtu_lookup` for a directly-connected peer.
///
/// Called when an FMP link-layer peer is promoted to active. The seed
/// value is the local outgoing-link MTU on the peer's transport, which
/// is the actual link constraint for direct-link traffic. Stored only
/// when no tighter value exists: discovery's reverse-path bottleneck
/// or MMP `MtuExceeded` reactive learning take precedence when smaller.
///
/// Without this seed, configured/auto-connect peers (which establish
/// sessions without going through the discovery Lookup flow) leave
/// `path_mtu_lookup` empty for their FipsAddress, causing
/// `per_flow_max_mss` to fall back to the global ceiling and the
/// SYN-time TCP MSS clamp to over-estimate the effective path.
pub(in crate::node) fn seed_path_mtu_for_link_peer(
&self,
peer_addr: &NodeAddr,
transport_id: TransportId,
addr: &TransportAddr,
) {
let Some(transport) = self.transports.get(&transport_id) else {
debug!(
peer = %self.peer_display_name(peer_addr),
transport_id = %transport_id,
"seed_path_mtu_for_link_peer: transport not registered, skipping seed"
);
return;
};
let link_mtu = transport.link_mtu(addr);
let fips_addr = crate::FipsAddress::from_node_addr(peer_addr);
let Ok(mut map) = self.path_mtu_lookup.write() else {
warn!(
peer = %self.peer_display_name(peer_addr),
"seed_path_mtu_for_link_peer: path_mtu_lookup write lock poisoned"
);
return;
};
match map.get(&fips_addr).copied() {
Some(existing) if existing <= link_mtu => {
// Keep the tighter learned value; never loosen the clamp.
debug!(
peer = %self.peer_display_name(peer_addr),
fips_addr = %fips_addr,
link_mtu = link_mtu,
existing = existing,
"seed_path_mtu_for_link_peer: keeping tighter existing value"
);
}
other => {
map.insert(fips_addr, link_mtu);
debug!(
peer = %self.peer_display_name(peer_addr),
fips_addr = %fips_addr,
link_mtu = link_mtu,
prior = ?other,
map_len = map.len(),
"seed_path_mtu_for_link_peer: wrote link MTU"
);
}
}
}
}
/// Tracks a pending discovery lookup with retry state.
pub struct PendingLookup {
/// When the lookup was first initiated.
pub initiated_ms: u64,
/// When the last attempt was sent.
pub last_sent_ms: u64,
/// Current attempt number (1 = initial, 2 = first retry, ...).
pub attempt: u8,
}
impl PendingLookup {
pub fn new(now_ms: u64) -> Self {
Self {
initiated_ms: now_ms,
last_sent_ms: now_ms,
attempt: 1,
}
}
}
src/node/handlers/handshake.rs
+887 -562
View File
File diff suppressed because it is too large Load Diff
src/node/handlers/lookup.rs
+739
View File
@@ -0,0 +1,739 @@
//! LookupRequest/LookupResponse mesh lookup protocol handlers.
//!
//! Handles coordinate lookup via bloom-filter-guided tree routing.
//! Requests are forwarded only to tree peers (parent + children) whose
//! bloom filter contains the target. TTL and request_id dedup provide
//! safety bounds.
use crate::node::Node;
use crate::node::reject::DiscoveryReject;
use crate::proto::lookup::{
LookupAction, LookupRequest, LookupResponse, MAX_RECENT_LOOKUP_REQUESTS,
};
use crate::transport::{TransportAddr, TransportId};
use crate::{NodeAddr, PeerIdentity};
use tracing::{debug, info, trace, warn};
/// Shell adapter exposing the live routing tables to the sans-IO discovery
/// core's `RoutingView` read seam. Lives in `node` so it can read `Node`'s
/// private `peers` map and call the crate-private tree/bloom predicates.
///
/// Holding `&Node` whole is fine for the forward path because it does not
/// also need `&mut self.lookup` concurrently. A later commit whose core
/// step needs `&mut discovery` while reading routing state should narrow this
/// to borrow only `peers` + `tree_state` instead of the whole node.
struct NodeRoutingView<'a> {
node: &'a Node,
}
impl crate::proto::lookup::RoutingView for NodeRoutingView<'_> {
fn is_tree_peer(&self, addr: &NodeAddr) -> bool {
self.node.is_tree_peer(addr)
}
fn peers_reaching(&self, target: &NodeAddr) -> Vec<NodeAddr> {
self.node
.peers
.iter()
.filter(|(_, peer)| peer.may_reach(target))
.map(|(addr, _)| *addr)
.collect()
}
}
impl Node {
/// Handle an incoming LookupRequest from a peer.
///
/// Processing steps:
/// 1. Decode and validate
/// 2. Check request_id for duplicates (dedup / reverse-path routing)
/// 3. Record request for reverse-path forwarding
/// 4. Lazy purge expired entries
/// 5. If we're the target, generate and send response
/// 6. If TTL > 0, forward to tree peers whose bloom filter matches
pub(in crate::node) async fn handle_lookup_request(&mut self, from: &NodeAddr, payload: &[u8]) {
self.metrics().lookup.req_received.inc();
let request = match LookupRequest::decode(payload) {
Ok(req) => req,
Err(e) => {
self.metrics()
.lookup
.record_reject(DiscoveryReject::ReqDecodeError);
debug!(from = %self.peer_display_name(from), error = %e, "Malformed LookupRequest");
return;
}
};
let now_ms = Self::now_ms();
let recent_expiry_ms = self.config().node.lookup.recent_expiry_secs * 1000;
let my_addr = *self.node_addr();
use crate::proto::lookup::RequestOutcome;
match crate::proto::lookup::classify_request(
&mut self.lookup,
&request,
from,
&my_addr,
now_ms,
recent_expiry_ms,
MAX_RECENT_LOOKUP_REQUESTS,
) {
RequestOutcome::Duplicate => {
self.metrics()
.lookup
.record_reject(DiscoveryReject::ReqDuplicate);
debug!(
request_id = request.request_id,
from = %self.peer_display_name(from),
"Duplicate LookupRequest, dropping"
);
}
RequestOutcome::DedupCacheFull { len } => {
self.metrics()
.lookup
.record_reject(DiscoveryReject::ReqDedupCacheFull);
debug!(
request_id = request.request_id,
from = %self.peer_display_name(from),
recent_requests = len,
max_recent_requests = MAX_RECENT_LOOKUP_REQUESTS,
"Discovery request dedup cache full, dropping LookupRequest"
);
}
RequestOutcome::RespondAsTarget => {
self.metrics().lookup.req_target_is_us.inc();
debug!(
request_id = request.request_id,
origin = %self.peer_display_name(&request.origin),
"We are the lookup target, generating response"
);
self.send_lookup_response(&request).await;
}
RequestOutcome::Forward => {
self.metrics().lookup.req_forwarded.inc();
self.forward_lookup_request(request).await;
}
RequestOutcome::ForwardRateLimited => {
self.metrics().lookup.req_forward_rate_limited.inc();
debug!(
request_id = request.request_id,
target = %self.peer_display_name(&request.target),
"Forward rate limited, suppressing LookupRequest"
);
}
RequestOutcome::TtlExhausted => {
self.metrics()
.lookup
.record_reject(DiscoveryReject::ReqTtlExhausted);
debug!(
request_id = request.request_id,
target = %self.peer_display_name(&request.target),
"LookupRequest TTL exhausted"
);
}
}
}
/// Handle an incoming LookupResponse from a peer.
///
/// Processing steps:
/// 1. Decode and validate
/// 2. Check recent_requests to determine if we originated or are forwarding
/// 3. If originator: verify proof signature, then cache target_coords and path_mtu in coord_cache
/// 4. If transit: apply path_mtu min(outgoing_link_mtu), reverse-path forward to from_peer
pub(in crate::node) async fn handle_lookup_response(
&mut self,
from: &NodeAddr,
payload: &[u8],
) {
self.metrics().lookup.resp_received.inc();
let mut response = match LookupResponse::decode(payload) {
Ok(resp) => resp,
Err(e) => {
self.metrics()
.lookup
.record_reject(DiscoveryReject::RespDecodeError);
debug!(from = %self.peer_display_name(from), error = %e, "Malformed LookupResponse");
return;
}
};
let now_ms = Self::now_ms();
// Check if we forwarded this request (transit node) or originated it
match crate::proto::lookup::classify_response(&mut self.lookup, response.request_id) {
crate::proto::lookup::ResponseRoute::AlreadyForwarded => {
// Already forwarded a response for this request — drop to
// prevent response routing loops.
debug!(
request_id = response.request_id,
target = %self.peer_display_name(&response.target),
"Response already forwarded for this request, dropping"
);
}
crate::proto::lookup::ResponseRoute::Transit { from_peer } => {
// Transit node: reverse-path forward
self.metrics().lookup.resp_forwarded.inc();
// Apply path_mtu min() from the outgoing link's transport MTU
self.apply_outgoing_link_mtu_to_response(&mut response, &from_peer);
debug!(
request_id = response.request_id,
target = %self.peer_display_name(&response.target),
next_hop = %self.peer_display_name(&from_peer),
path_mtu = response.path_mtu,
"Reverse-path forwarding LookupResponse"
);
let encoded = response.encode();
if let Err(e) = self.send_encrypted_link_message(&from_peer, &encoded).await {
debug!(
next_hop = %self.peer_display_name(&from_peer),
error = %e,
"Failed to forward LookupResponse"
);
}
}
crate::proto::lookup::ResponseRoute::Originator => {
// We originated this request — verify proof before caching
let target = response.target;
let path_mtu = response.path_mtu;
// Look up the target's public key from identity_cache
let mut prefix = [0u8; 15];
prefix.copy_from_slice(&target.as_bytes()[0..15]);
let target_pubkey = match self.lookup_by_fips_prefix(&prefix) {
Some((_addr, pubkey)) => pubkey,
None => {
self.metrics()
.lookup
.record_reject(DiscoveryReject::RespIdentityMiss);
warn!(
request_id = response.request_id,
target = %self.peer_display_name(&target),
"identity_cache miss for lookup target, cannot verify proof"
);
return;
}
};
// Verify the proof signature
let (xonly, _parity) = target_pubkey.x_only_public_key();
let peer_id = PeerIdentity::from_pubkey(xonly);
let proof_data = LookupResponse::proof_bytes(
response.request_id,
&target,
&response.target_coords,
);
if !peer_id.verify(&proof_data, &response.proof) {
self.metrics()
.lookup
.record_reject(DiscoveryReject::RespProofFailed);
warn!(
request_id = response.request_id,
target = %self.peer_display_name(&target),
"LookupResponse proof verification failed, discarding"
);
return;
}
self.metrics().lookup.resp_accepted.inc();
info!(
request_id = response.request_id,
target = %self.peer_display_name(&target),
depth = response.target_coords.depth(),
path_mtu = path_mtu,
"Discovery succeeded, proof verified, route cached"
);
// Apply the accept-side effects: the core clears the success
// state (backoff + pending lookup) and returns the
// cross-subsystem effects for us to drive.
let actions = crate::proto::lookup::on_response_accepted(
&mut self.lookup,
&target,
response.target_coords,
now_ms,
path_mtu,
);
self.drive_response_actions(actions).await;
}
}
}
/// Drive the cross-subsystem effects returned by the discovery core's
/// accept-side planning. Each arm reproduces the original inline effect
/// exactly (same metrics/logs/writes, same order).
async fn drive_response_actions(&mut self, actions: Vec<LookupAction>) {
for action in actions {
match action {
LookupAction::CacheCoords {
target,
coords,
now_ms,
path_mtu,
} => {
self.coord_cache
.insert_with_path_mtu(target, coords, now_ms, path_mtu);
}
LookupAction::WritePathMtu { target, path_mtu } => {
// Mirror path_mtu into the FipsAddress-keyed read-only lookup
// map used by the TUN reader/writer at TCP MSS clamp time.
let fips_addr = crate::FipsAddress::from_node_addr(&target);
match self.path_mtu_lookup.write() {
Ok(mut map) => match map.get(&fips_addr).copied() {
Some(existing) if existing <= path_mtu => {
// Keep the tighter learned value; never loosen
// the clamp. A reactive MtuExceeded or
// PathMtuNotification tighten takes precedence
// over a looser discovery estimate
// (cross-carrier keep-tighter).
debug!(
target = %self.peer_display_name(&target),
fips_addr = %fips_addr,
path_mtu = path_mtu,
existing = existing,
"LookupResponse: keeping tighter existing path_mtu_lookup value"
);
}
other => {
map.insert(fips_addr, path_mtu);
debug!(
target = %self.peer_display_name(&target),
fips_addr = %fips_addr,
path_mtu = path_mtu,
prior = ?other,
map_len = map.len(),
"Wrote path_mtu_lookup from discovery LookupResponse"
);
}
},
Err(e) => {
warn!(
target = %self.peer_display_name(&target),
fips_addr = %fips_addr,
path_mtu = path_mtu,
error = %e,
"path_mtu_lookup write lock poisoned; clamp will not see this update"
);
}
}
}
LookupAction::ResetWarmupIfEstablished { target } => {
// If an established session exists, reset the warmup counter.
let n = self.config().node.session.coords_warmup_packets;
if let Some(entry) = self.sessions.get_mut(&target)
&& entry.is_established()
{
entry.set_coords_warmup_remaining(n);
debug!(
dest = %self.peer_display_name(&target),
warmup_packets = n,
"Reset coords warmup after discovery for existing session"
);
}
}
LookupAction::RetryQueuedPackets { target } => {
// If we have pending TUN packets for this target, retry session
// initiation. The coord_cache now has coords, so find_next_hop()
// should succeed.
if let Some(packets) = self.pending_tun_packets.get(&target) {
debug!(
dest = %self.peer_display_name(&target),
queued_packets = packets.len(),
"Retrying queued packets after discovery"
);
self.retry_session_after_discovery(target).await;
}
}
LookupAction::SendLink { peer, bytes } => {
if let Err(e) = self.send_encrypted_link_message(&peer, &bytes).await {
debug!(
peer = %self.peer_display_name(&peer),
error = %e,
"Failed to send discovery link message"
);
}
}
}
}
}
/// Generate and send a LookupResponse when we are the target.
async fn send_lookup_response(&mut self, request: &LookupRequest) {
let our_coords = self.tree_state().my_coords().clone();
// Sign proof: Identity::sign hashes with SHA-256 internally
let proof_data =
LookupResponse::proof_bytes(request.request_id, &request.target, &our_coords);
let proof = self.identity().sign(&proof_data);
let mut response =
LookupResponse::new(request.request_id, request.target, our_coords, proof);
// Route toward origin. The reverse-path decision (the peer the request
// arrived from, recorded in recent_requests) is the sans-IO core's; the
// greedy tree-route fallback is a &mut coord-cache op kept in the shell.
use crate::proto::lookup::ResponseRouteDecision;
let next_hop_addr = match crate::proto::lookup::plan_response_route(
&self.lookup,
request.request_id,
) {
ResponseRouteDecision::ReversePath(peer) => peer,
ResponseRouteDecision::NeedsTreeRoute => match self.find_next_hop(&request.origin) {
Some(peer) => *peer.node_addr(),
None => {
debug!(
origin = %self.peer_display_name(&request.origin),
"Cannot route LookupResponse: no reverse path or tree route to origin"
);
self.metrics()
.lookup
.record_reject(DiscoveryReject::RespNoRoute);
return;
}
},
};
// Fold our outgoing-link MTU into path_mtu so the target-edge link
// appears in the bottleneck calculation. Without this, the response
// leaves the target with path_mtu = u16::MAX and only intermediate
// transits min-fold; the target's first reverse-path hop is missed.
self.apply_outgoing_link_mtu_to_response(&mut response, &next_hop_addr);
debug!(
request_id = request.request_id,
origin = %self.peer_display_name(&request.origin),
next_hop = %self.peer_display_name(&next_hop_addr),
path_mtu = response.path_mtu,
"Sending LookupResponse"
);
let encoded = response.encode();
if let Err(e) = self
.send_encrypted_link_message(&next_hop_addr, &encoded)
.await
{
debug!(
next_hop = %self.peer_display_name(&next_hop_addr),
error = %e,
"Failed to send LookupResponse"
);
}
}
/// Forward a LookupRequest to eligible peers.
///
/// Primary path: tree peers (parent + children) whose bloom filter
/// contains the target. Restricting to tree peers follows the spanning
/// tree partition, producing a single directed path.
///
/// Fallback: if no tree peer's bloom matches, try non-tree peers whose
/// bloom contains the target. This recovers from dead ends caused by
/// stale bloom filters, tree restructuring, or transit node failures.
async fn forward_lookup_request(&mut self, mut request: LookupRequest) {
// Plan the forward with the sans-IO decision core. The core owns the
// TTL decrement, tree/fallback peer selection, and single-encode
// fan-out; the shell keeps all metrics/logging and drives the sends.
let outcome = {
let rv = NodeRoutingView { node: self };
crate::proto::lookup::plan_forward(&mut request, &rv)
};
match outcome {
crate::proto::lookup::ForwardOutcome::TtlExhausted => {}
crate::proto::lookup::ForwardOutcome::NoPeers => {
self.metrics().lookup.req_no_tree_peer.inc();
trace!(
request_id = request.request_id,
"No eligible peers to forward LookupRequest"
);
}
crate::proto::lookup::ForwardOutcome::Forward {
actions,
used_fallback,
} => {
let peer_count = actions.len();
if used_fallback {
self.metrics().lookup.req_fallback_forwarded.inc();
debug!(
request_id = request.request_id,
target = %self.peer_display_name(&request.target),
ttl = request.ttl,
peer_count,
"Forwarding LookupRequest via non-tree fallback"
);
} else {
debug!(
request_id = request.request_id,
target = %self.peer_display_name(&request.target),
ttl = request.ttl,
peer_count,
"Forwarding LookupRequest"
);
}
for action in actions {
if let LookupAction::SendLink { peer, bytes } = action
&& let Err(e) = self.send_encrypted_link_message(&peer, &bytes).await
{
debug!(
peer = %self.peer_display_name(&peer),
error = %e,
"Failed to forward LookupRequest to peer"
);
}
}
}
}
}
/// Initiate a discovery lookup for a target node.
///
/// Creates a LookupRequest and sends it to tree peers whose bloom
/// filters contain the target. Returns the number of peers sent to.
/// The originator does NOT record the request_id in recent_requests,
/// so when the response arrives, it's recognized as "our request".
pub(in crate::node) async fn initiate_lookup(&mut self, target: &NodeAddr, ttl: u8) -> usize {
self.metrics().lookup.req_initiated.inc();
let origin = *self.node_addr();
let origin_coords = self.tree_state().my_coords().clone();
let request_id = {
use rand::RngExt;
rand::rng().random()
};
let request = LookupRequest::new(request_id, *target, origin, origin_coords, ttl, 0);
// Tree-peer bloom-match selection + single encode live in the sans-IO
// core. The core keeps the tree-only (no non-tree fallback) behavior;
// the shell drives the sends and keeps all metrics/logging.
let actions = {
let rv = NodeRoutingView { node: self };
crate::proto::lookup::plan_initiate(&request, &rv)
};
let peer_count = actions.len();
debug!(
request_id = request.request_id,
target = %self.peer_display_name(target),
ttl = ttl,
peer_count = peer_count,
total_peers = self.peers.len(),
"Discovery lookup initiated"
);
for action in actions {
if let LookupAction::SendLink { peer, bytes } = action
&& let Err(e) = self.send_encrypted_link_message(&peer, &bytes).await
{
debug!(
peer = %self.peer_display_name(&peer),
error = %e,
"Failed to send LookupRequest to peer"
);
}
}
peer_count
}
/// Initiate a discovery lookup if one is not already pending for this target.
///
/// Checks: pending dedup, post-failure backoff (off by default), bloom
/// filter pre-check. If all pass, sends the first attempt's LookupRequest.
/// Subsequent attempts (with fresh request_ids) are scheduled by
/// [`Self::check_pending_lookups`] when each attempt's per-attempt timeout
/// expires, using the sequence in `node.lookup.attempt_timeouts_secs`.
pub(in crate::node) async fn maybe_initiate_lookup(&mut self, dest: &NodeAddr) {
let now_ms = Self::now_ms();
// Bloom filter pre-check (view read) BEFORE the core call: if no peer's
// filter contains the target, it's not in the mesh. Reading `self.peers`
// here keeps the `&mut self.lookup` borrow in `initiate_gate` from
// overlapping the immutable peer-table read.
let reachable = self.peers.values().any(|peer| peer.may_reach(dest));
use crate::proto::lookup::InitiateDecision;
match crate::proto::lookup::initiate_gate(&mut self.lookup, dest, now_ms, reachable) {
InitiateDecision::Deduplicated => {
self.metrics().lookup.req_deduplicated.inc();
debug!(
target_node = %self.peer_display_name(dest),
"Discovery lookup deduplicated, already pending"
);
}
InitiateDecision::Suppressed { failures } => {
self.metrics().lookup.req_backoff_suppressed.inc();
debug!(
target_node = %self.peer_display_name(dest),
failures = failures,
"Discovery lookup suppressed by backoff"
);
}
InitiateDecision::BloomMiss => {
self.metrics().lookup.req_bloom_miss.inc();
debug!(
target_node = %self.peer_display_name(dest),
"Discovery skipped, target not in any peer bloom filter"
);
}
InitiateDecision::Proceed => {
let ttl = self.config().node.lookup.ttl;
let sent = self.initiate_lookup(dest, ttl).await;
// If no tree peers had the target, fail immediately
if sent == 0 {
crate::proto::lookup::initiate_failed(&mut self.lookup, dest, now_ms);
debug!(
target_node = %self.peer_display_name(dest),
"Discovery failed, no tree peers with bloom match"
);
}
}
}
}
/// Check pending lookups for next-attempt or final timeout.
///
/// Called periodically from the tick handler. The lookup state machine
/// runs through `node.lookup.attempt_timeouts_secs` (default
/// `[1, 2, 4, 8]`): each entry is the deadline for one attempt. When the
/// current attempt's deadline elapses:
/// - If more entries remain: send the next attempt with a fresh
/// `request_id`.
/// - Otherwise: declare the destination unreachable, drop queued packets,
/// and emit ICMPv6 destination-unreachable for each.
pub(in crate::node) async fn check_pending_lookups(&mut self, now_ms: u64) {
let attempt_timeouts = self.config().node.lookup.attempt_timeouts_secs.clone();
let outcome =
crate::proto::lookup::poll_pending(&mut self.lookup, now_ms, &attempt_timeouts);
for (target, attempt) in outcome.retries {
let ttl = self.config().node.lookup.ttl;
let sent = self.initiate_lookup(&target, ttl).await;
if sent > 0 {
debug!(
target_node = %self.peer_display_name(&target),
attempt = attempt,
"Discovery retry sent"
);
}
}
for (addr, failures) in outcome.timeouts {
self.metrics().lookup.resp_timed_out.inc();
let queued = self.pending_tun_packets.remove(&addr);
let pkt_count = queued.as_ref().map_or(0, |p| p.len());
info!(
target_node = %self.peer_display_name(&addr),
queued_packets = pkt_count,
failures = failures,
"Discovery lookup timed out, destination unreachable"
);
if let Some(packets) = queued {
for pkt in &packets {
self.send_icmpv6_dest_unreachable(pkt);
}
}
}
}
/// Reset discovery backoff on topology changes.
pub(in crate::node) fn reset_lookup_backoff(&mut self) {
let cleared = self.lookup.reset_backoff();
if cleared > 0 {
debug!(
entries = cleared,
"Resetting discovery backoff on topology change"
);
}
}
/// Min-fold our outgoing-link MTU into a LookupResponse's `path_mtu`.
///
/// Used at both transit-side reverse-path forward and at the target's
/// own send_lookup_response. The link MTU we apply is the MTU of the
/// transport+addr we'll use to deliver the response toward `next_hop`.
/// No-op when `next_hop` is not a directly-connected peer or its
/// transport is not registered.
pub(in crate::node) fn apply_outgoing_link_mtu_to_response(
&self,
response: &mut LookupResponse,
next_hop: &NodeAddr,
) {
if let Some(peer) = self.peers.get(next_hop)
&& let Some(tid) = peer.transport_id()
&& let Some(transport) = self.transports.get(&tid)
{
let link_mtu = if let Some(addr) = peer.current_addr() {
transport.link_mtu(addr)
} else {
transport.mtu()
};
response.path_mtu = response.path_mtu.min(link_mtu);
}
}
/// Seed `path_mtu_lookup` for a directly-connected peer.
///
/// Called when an FMP link-layer peer is promoted to active. The seed
/// value is the local outgoing-link MTU on the peer's transport, which
/// is the actual link constraint for direct-link traffic. Stored only
/// when no tighter value exists: discovery's reverse-path bottleneck
/// or MMP `MtuExceeded` reactive learning take precedence when smaller.
///
/// Without this seed, configured/auto-connect peers (which establish
/// sessions without going through the discovery Lookup flow) leave
/// `path_mtu_lookup` empty for their FipsAddress, causing
/// `per_flow_max_mss` to fall back to the global ceiling and the
/// SYN-time TCP MSS clamp to over-estimate the effective path.
pub(in crate::node) fn seed_path_mtu_for_link_peer(
&self,
peer_addr: &NodeAddr,
transport_id: TransportId,
addr: &TransportAddr,
) {
let Some(transport) = self.transports.get(&transport_id) else {
debug!(
peer = %self.peer_display_name(peer_addr),
transport_id = %transport_id,
"seed_path_mtu_for_link_peer: transport not registered, skipping seed"
);
return;
};
let link_mtu = transport.link_mtu(addr);
let fips_addr = crate::FipsAddress::from_node_addr(peer_addr);
let Ok(mut map) = self.path_mtu_lookup.write() else {
warn!(
peer = %self.peer_display_name(peer_addr),
"seed_path_mtu_for_link_peer: path_mtu_lookup write lock poisoned"
);
return;
};
match map.get(&fips_addr).copied() {
Some(existing) if existing <= link_mtu => {
// Keep the tighter learned value; never loosen the clamp.
debug!(
peer = %self.peer_display_name(peer_addr),
fips_addr = %fips_addr,
link_mtu = link_mtu,
existing = existing,
"seed_path_mtu_for_link_peer: keeping tighter existing value"
);
}
other => {
map.insert(fips_addr, link_mtu);
debug!(
peer = %self.peer_display_name(peer_addr),
fips_addr = %fips_addr,
link_mtu = link_mtu,
prior = ?other,
map_len = map.len(),
"seed_path_mtu_for_link_peer: wrote link MTU"
);
}
}
}
}
src/node/handlers/mmp.rs
+306 -335
View File
@@ -5,20 +5,63 @@
//! and teardown metric logs.
use crate::NodeAddr;
use crate::mmp::MmpMode;
use crate::mmp::MmpSessionState;
use crate::mmp::report::{ReceiverReport, SenderReport};
use crate::node::Node;
use crate::node::dataplane::PeerActionCtx;
use crate::node::reject::{MmpReject, RejectReason, TreeReject};
use crate::protocol::{
LinkMessageType, PathMtuNotification, SessionMessageType, SessionReceiverReport,
SessionSenderReport,
use crate::node::tree::sign_declaration;
use crate::peer::machine::PeerEvent;
use crate::proto::link::LinkMessageType;
use crate::proto::mmp::{
LinkReportKind, LinkReportSnapshot, MmpAction, PeerLivenessSnapshot, ReceiverReport, RrLog,
SenderReport,
};
use crate::proto::stp::ParentEval;
use crate::transport::{TransportAddr, TransportId};
use std::time::{Duration, Instant};
use tracing::{debug, info, trace, warn};
/// Emit the operator `trace!` point for a processed ReceiverReport outcome.
///
/// These log points used to live inside `MmpMetrics::process_receiver_report`;
/// the sans-IO migration returns the outcome as an [`RrLog`] and re-emits it
/// here, shell-side, preserving the original field set, content, and (relative
/// to the surrounding handler logs) ordering. The original traces carried no
/// peer identifier, so none is added here.
pub(super) fn log_rr_outcome(rr: &ReceiverReport, our_timestamp_ms: u32, log: RrLog) {
match log {
RrLog::Stale {
prev_highest,
prev_packets,
prev_bytes,
} => trace!(
highest_counter = rr.highest_counter,
prev_highest_counter = prev_highest,
cumulative_packets_recv = rr.cumulative_packets_recv,
prev_cumulative_packets_recv = prev_packets,
cumulative_bytes_recv = rr.cumulative_bytes_recv,
prev_cumulative_bytes_recv = prev_bytes,
"Ignoring stale MMP ReceiverReport"
),
RrLog::RttSample { rtt_ms, srtt_ms } => trace!(
our_ts = our_timestamp_ms,
echo = rr.timestamp_echo,
dwell = u32::from(rr.dwell_time),
rtt_ms = rtt_ms,
srtt_ms = srtt_ms,
"RTT sample from timestamp echo"
),
RrLog::InvalidRtt => trace!(
our_ts = our_timestamp_ms,
echo = rr.timestamp_echo,
dwell = u32::from(rr.dwell_time),
"Ignoring invalid MMP RTT sample"
),
RrLog::None => {}
}
}
/// Format bytes/sec as human-readable throughput.
fn format_throughput(bps: f64) -> String {
pub(in crate::node) fn format_throughput(bps: f64) -> String {
if bps == 0.0 {
"n/a".to_string()
} else if bps >= 1_000_000.0 {
@@ -113,10 +156,12 @@ impl Node {
// Process the report: computes RTT from timestamp echo, updates
// loss rate, goodput rate, jitter trend, and ETX.
let now = Instant::now();
let first_rtt = mmp
.metrics
.process_receiver_report(&rr, our_timestamp_ms, now);
let now_ms = crate::time::mono_ms();
let (first_rtt, rr_log) =
mmp.metrics
.process_receiver_report(&rr, our_timestamp_ms, now_ms);
// Re-emit the operator trace the core used to log mid-decision.
log_rr_outcome(&rr, our_timestamp_ms, rr_log);
// Feed SRTT back to sender/receiver report interval tuning
if let Some(srtt_ms) = mmp.metrics.srtt_ms() {
@@ -144,25 +189,43 @@ impl Node {
// Trigger re-evaluation so the node doesn't wait for the next
// periodic tick or TreeAnnounce.
if first_rtt {
let peer_costs: std::collections::HashMap<crate::NodeAddr, f64> = self
let peer_costs: std::collections::BTreeMap<crate::NodeAddr, f64> = self
.peers
.iter()
.filter(|(_, p)| p.has_srtt())
.map(|(a, p)| (*a, p.link_cost()))
.collect();
if let Some(new_parent) = self.tree_state.evaluate_parent(&peer_costs) {
// Wall-clock seconds for the escaping declaration timestamp;
// monotonic ms for the flap-dampening / hold-down timers.
let now_secs = std::time::SystemTime::now()
.duration_since(std::time::UNIX_EPOCH)
.map(|d| d.as_secs())
.unwrap_or(0);
let mono_now_ms = crate::time::mono_ms();
// Compute the flap-dampening / hold-down veto at the edge; a mandatory
// switch bypasses it, a discretionary one is taken only if not suppressed.
let switch_suppressed = self.tree_state.is_switch_suppressed(mono_now_ms);
let new_parent = match self
.tree_state
.evaluate_parent(&peer_costs, &std::collections::BTreeSet::new())
{
ParentEval::Mandatory(p) => Some(p),
ParentEval::Discretionary(p) if !switch_suppressed => Some(p),
ParentEval::Discretionary(_) | ParentEval::None => None,
};
if let Some(new_parent) = new_parent {
let new_seq = self.tree_state.my_declaration().sequence() + 1;
let timestamp = std::time::SystemTime::now()
.duration_since(std::time::UNIX_EPOCH)
.map(|d| d.as_secs())
.unwrap_or(0);
let flap_dampened = self.tree_state.set_parent(new_parent, new_seq, timestamp);
let flap_dampened =
self.tree_state
.set_parent(new_parent, new_seq, now_secs, mono_now_ms);
self.tree_state.recompute_coords();
// Clone identity once: sign_declaration borrows &mut tree_state while
// the identity() accessor borrows all of &self, so an owned copy avoids
// the split-borrow conflict on this infrequent parent-switch path.
let our_identity = self.identity().clone();
if let Err(e) = self.tree_state.sign_declaration(&our_identity) {
if let Err(e) =
sign_declaration(self.tree_state.my_declaration_mut(), &our_identity)
{
warn!(error = %e, "Failed to sign declaration after first-RTT parent eval");
self.metrics()
.tree
@@ -172,7 +235,7 @@ impl Node {
// Surgical invalidation — see CoordCache::invalidate_via_node doc.
self.coord_cache
.invalidate_via_node(our_identity.node_addr());
self.reset_discovery_backoff();
self.reset_lookup_backoff();
self.metrics().tree.parent_switches.inc();
info!(
new_parent = %self.peer_display_name(&new_parent),
@@ -190,10 +253,12 @@ impl Node {
let all_peers: Vec<crate::NodeAddr> = self.peers.keys().copied().collect();
self.bloom_state.mark_all_updates_needed(all_peers);
} else if !self.tree_state.is_root() && self.tree_state.should_be_root() {
self.tree_state.become_root();
self.tree_state.become_root(now_secs);
// Clone identity once (see the parent-switch branch above for why).
let our_identity = self.identity().clone();
if let Err(e) = self.tree_state.sign_declaration(&our_identity) {
if let Err(e) =
sign_declaration(self.tree_state.my_declaration_mut(), &our_identity)
{
warn!(error = %e, "Failed to sign self-root declaration after first-RTT");
self.metrics()
.tree
@@ -203,7 +268,7 @@ impl Node {
// Surgical invalidation — see CoordCache::invalidate_other_roots doc.
self.coord_cache
.invalidate_other_roots(our_identity.node_addr());
self.reset_discovery_backoff();
self.reset_lookup_backoff();
self.metrics().tree.parent_switches.inc();
info!(
new_root = %self.tree_state.root(),
@@ -221,65 +286,91 @@ impl Node {
///
/// Called from the tick handler. Also emits periodic operator logs.
pub(in crate::node) async fn check_mmp_reports(&mut self) {
let now = Instant::now();
let now_ms = crate::time::mono_ms();
// Collect peers that need reports (can't borrow self mutably while iterating)
let mut sender_reports: Vec<(NodeAddr, Vec<u8>)> = Vec::new();
let mut receiver_reports: Vec<(NodeAddr, Vec<u8>)> = Vec::new();
// Build one report-gating snapshot per peer, resolving every timing read
// shell-side into a `bool`. `send_sr`/`send_rr` are `true` on the master
// (IK) line — there is no profile negotiation here; the forward-merge to
// `-next` wires them to `peer.send_sr()`/`peer.send_rr()` (plan spot c).
// The snapshots own only `NodeAddr`/`MmpMode`/`bool`, so the
// peer-iteration borrow is released before the pure decision runs and the
// driving loop mutates the reporting state.
let snapshots: Vec<LinkReportSnapshot> = self
.peers
.iter()
.filter_map(|(node_addr, peer)| {
let mmp = peer.mmp()?;
Some(LinkReportSnapshot {
peer: *node_addr,
mode: mmp.mode(),
send_sr: true,
send_rr: true,
sr_due: mmp.sender.should_send_report(now_ms),
rr_due: mmp.receiver.should_send_report(now_ms),
log_due: mmp.should_log(now_ms),
})
})
.collect();
for (node_addr, peer) in self.peers.iter_mut() {
// Compute display name before taking mutable MMP borrow
let peer_name = self
.peer_aliases
.get(node_addr)
.cloned()
.unwrap_or_else(|| peer.identity().short_npub());
let actions = self.mmp.plan_link_reports(&snapshots);
let Some(mmp) = peer.mmp_mut() else {
continue;
};
let mode = mmp.mode();
// Sender reports: Full mode only
if mode == MmpMode::Full
&& mmp.sender.should_send_report(now)
&& let Some(sr) = mmp.sender.build_report(now)
{
sender_reports.push((*node_addr, sr.encode()));
}
// Receiver reports: Full and Lightweight modes
if mode != MmpMode::Minimal
&& mmp.receiver.should_send_report(now)
&& let Some(rr) = mmp.receiver.build_report(now)
{
receiver_reports.push((*node_addr, rr.encode()));
}
// Periodic operator logging
if mmp.should_log(now) {
Self::log_mmp_metrics(&peer_name, mmp);
mmp.mark_logged(now);
}
}
// Send collected reports
for (node_addr, encoded) in sender_reports {
if let Err(e) = self.send_encrypted_link_message(&node_addr, &encoded).await {
debug!(peer = %self.peer_display_name(&node_addr), error = %e, "Failed to send SenderReport");
}
}
for (node_addr, encoded) in receiver_reports {
if let Err(e) = self.send_encrypted_link_message(&node_addr, &encoded).await {
debug!(peer = %self.peer_display_name(&node_addr), error = %e, "Failed to send ReceiverReport");
// Drive the planned actions in their phase-grouped order (all logs, then
// all SenderReports, then all ReceiverReports). Logs run first because the
// operator log reads cumulative_packets_sent, which each report send
// advances (send_encrypted_link_message -> sender.record_sent); the
// pre-refactor handler logged during its collect pass, before any send.
// `build_report` (which advances the interval state) is called only on a
// SendLinkReport action, exactly as the pre-refactor gate did.
for action in actions {
match action {
MmpAction::SendLinkReport { peer, kind } => {
let encoded = self
.peers
.get_mut(&peer)
.and_then(|p| p.mmp_mut())
.and_then(|mmp| match kind {
LinkReportKind::Sender => {
mmp.sender.build_report(now_ms).map(|sr| sr.encode())
}
LinkReportKind::Receiver => {
mmp.receiver.build_report(now_ms).map(|rr| rr.encode())
}
});
if let Some(encoded) = encoded
&& let Err(e) = self.send_encrypted_link_message(&peer, &encoded).await
{
let label = match kind {
LinkReportKind::Sender => "Failed to send SenderReport",
LinkReportKind::Receiver => "Failed to send ReceiverReport",
};
debug!(peer = %self.peer_display_name(&peer), error = %e, "{}", label);
}
}
MmpAction::LogLink { peer } => {
// Resolve the display name exactly as the pre-refactor loop
// did (alias, else short_npub) — not `peer_display_name`,
// which also consults the host map.
let peer_name = self.peer_aliases.get(&peer).cloned().unwrap_or_else(|| {
self.peers
.get(&peer)
.map(|p| p.identity().short_npub())
.unwrap_or_default()
});
if let Some(mmp) = self.peers.get_mut(&peer).and_then(|p| p.mmp_mut()) {
Self::log_mmp_metrics(&peer_name, mmp);
mmp.mark_logged(now_ms);
}
}
MmpAction::ReapPeer { .. }
| MmpAction::Heartbeat { .. }
| MmpAction::SendSessionReport { .. }
| MmpAction::LogSession { .. } => {}
}
}
}
/// Emit periodic MMP metrics for a peer.
fn log_mmp_metrics(peer_name: &str, mmp: &crate::mmp::MmpPeerState) {
fn log_mmp_metrics(peer_name: &str, mmp: &crate::proto::mmp::MmpPeerState) {
let m = &mmp.metrics;
let rtt_str = if m.rtt_trend.initialized() {
@@ -307,7 +398,10 @@ impl Node {
}
/// Emit a teardown log summarizing lifetime MMP metrics for a removed peer.
pub(in crate::node) fn log_mmp_teardown(peer_name: &str, mmp: &crate::mmp::MmpPeerState) {
pub(in crate::node) fn log_mmp_teardown(
peer_name: &str,
mmp: &crate::proto::mmp::MmpPeerState,
) {
let m = &mmp.metrics;
let jitter_ms = mmp.receiver.jitter_us() as f64 / 1000.0;
@@ -332,205 +426,6 @@ impl Node {
);
}
// === Session-layer MMP ===
/// Check all sessions for pending MMP reports and send them.
///
/// Called from the tick handler. Also emits periodic session MMP logs.
/// Uses the collect-then-send pattern to avoid borrowing conflicts.
pub(in crate::node) async fn check_session_mmp_reports(&mut self) {
let now = Instant::now();
// Collect reports to send: (dest_addr, msg_type, encoded_body)
let mut reports: Vec<(NodeAddr, u8, Vec<u8>)> = Vec::new();
for (dest_addr, entry) in self.sessions.iter_mut() {
// Compute display name before taking mutable MMP borrow
let session_name = self
.peer_aliases
.get(dest_addr)
.cloned()
.unwrap_or_else(|| {
let (xonly, _) = entry.remote_pubkey().x_only_public_key();
crate::PeerIdentity::from_pubkey(xonly).short_npub()
});
let Some(mmp) = entry.mmp_mut() else {
continue;
};
let mode = mmp.mode();
// Sender reports: Full mode only
if mode == MmpMode::Full
&& mmp.sender.should_send_report(now)
&& let Some(sr) = mmp.sender.build_report(now)
{
let session_sr: SessionSenderReport = SessionSenderReport::from(&sr);
reports.push((
*dest_addr,
SessionMessageType::SenderReport.to_byte(),
session_sr.encode(),
));
}
// Receiver reports: Full and Lightweight modes
if mode != MmpMode::Minimal
&& mmp.receiver.should_send_report(now)
&& let Some(rr) = mmp.receiver.build_report(now)
{
let session_rr: SessionReceiverReport = SessionReceiverReport::from(&rr);
reports.push((
*dest_addr,
SessionMessageType::ReceiverReport.to_byte(),
session_rr.encode(),
));
}
// PathMtu notifications (all modes)
if mmp.path_mtu.should_send_notification(now)
&& let Some(mtu_value) = mmp.path_mtu.build_notification(now)
{
let notif = PathMtuNotification::new(mtu_value);
reports.push((
*dest_addr,
SessionMessageType::PathMtuNotification.to_byte(),
notif.encode(),
));
}
// Periodic operator logging
if mmp.should_log(now) {
Self::log_session_mmp_metrics(&session_name, mmp);
mmp.mark_logged(now);
}
}
// Send collected reports via session-layer encryption.
// Track per-destination success/failure for backoff and log suppression.
let mut send_results: Vec<(NodeAddr, bool)> = Vec::new();
for (dest_addr, msg_type, body) in reports {
match self.send_session_msg(&dest_addr, msg_type, &body).await {
Ok(()) => {
send_results.push((dest_addr, true));
}
Err(e) => {
// Peek at current failure count for log suppression
let failures = self
.sessions
.get(&dest_addr)
.and_then(|entry| entry.mmp())
.map(|mmp| mmp.sender.consecutive_send_failures())
.unwrap_or(0);
if failures < 3 {
debug!(
dest = %self.peer_display_name(&dest_addr),
msg_type,
error = %e,
"Failed to send session MMP report"
);
} else if failures == 3 {
debug!(
dest = %self.peer_display_name(&dest_addr),
"Suppressing further session MMP send failure logs"
);
}
// failures > 3: silently suppressed
send_results.push((dest_addr, false));
}
}
}
// Update backoff state from send results.
// Deduplicate: a destination counts as success if ANY report succeeded,
// failure only if ALL reports for that destination failed.
let mut dest_success: std::collections::HashMap<NodeAddr, bool> =
std::collections::HashMap::new();
for (dest, ok) in &send_results {
let entry = dest_success.entry(*dest).or_insert(false);
if *ok {
*entry = true;
}
}
for (dest_addr, success) in dest_success {
if let Some(entry) = self.sessions.get_mut(&dest_addr)
&& let Some(mmp) = entry.mmp_mut()
{
if success {
let prev = mmp.sender.record_send_success();
if prev > 3 {
debug!(
dest = %self.peer_display_name(&dest_addr),
consecutive_failures = prev,
"Resumed session MMP reporting"
);
}
} else {
mmp.sender.record_send_failure();
}
}
}
}
/// Emit periodic session MMP metrics.
fn log_session_mmp_metrics(session_name: &str, mmp: &MmpSessionState) {
let m = &mmp.metrics;
let rtt_str = if m.rtt_trend.initialized() {
format!("{:.1}ms", m.rtt_trend.long() / 1000.0)
} else {
"n/a".to_string()
};
let loss_str = if m.loss_trend.initialized() {
format!("{:.1}%", m.loss_trend.long() * 100.0)
} else {
"n/a".to_string()
};
let jitter_ms = mmp.receiver.jitter_us() as f64 / 1000.0;
debug!(
session = %session_name,
rtt = %rtt_str,
loss = %loss_str,
jitter = format_args!("{:.1}ms", jitter_ms),
goodput = %format_throughput(m.goodput_bps()),
mtu = mmp.path_mtu.last_observed_mtu(),
tx_pkts = mmp.sender.cumulative_packets_sent(),
rx_pkts = mmp.receiver.cumulative_packets_recv(),
"MMP session metrics"
);
}
/// Emit a teardown log summarizing lifetime session MMP metrics.
pub(in crate::node) fn log_session_mmp_teardown(session_name: &str, mmp: &MmpSessionState) {
let m = &mmp.metrics;
let jitter_ms = mmp.receiver.jitter_us() as f64 / 1000.0;
let rtt_str = match m.srtt_ms() {
Some(rtt) => format!("{:.1}ms", rtt),
None => "n/a".to_string(),
};
let loss_str = format!("{:.1}%", m.loss_rate() * 100.0);
debug!(
session = %session_name,
rtt = %rtt_str,
loss = %loss_str,
jitter = format_args!("{:.1}ms", jitter_ms),
etx = format_args!("{:.2}", m.etx),
goodput = %format_throughput(m.goodput_bps()),
send_mtu = mmp.path_mtu.current_mtu(),
observed_mtu = mmp.path_mtu.last_observed_mtu(),
tx_pkts = mmp.sender.cumulative_packets_sent(),
tx_bytes = mmp.sender.cumulative_bytes_sent(),
rx_pkts = mmp.receiver.cumulative_packets_recv(),
rx_bytes = mmp.receiver.cumulative_bytes_recv(),
"MMP session teardown"
);
}
/// Send heartbeats and remove dead peers.
///
/// Called from the tick handler. Sends a 1-byte heartbeat to each peer
@@ -538,83 +433,159 @@ impl Node {
/// hasn't sent us a frame within the link dead timeout.
pub(in crate::node) async fn check_link_heartbeats(&mut self) {
let now = Instant::now();
// Monotonic ms for the MMP receiver's injected-`u64` liveness clock; the
// Instant `now` is still used for the shell-owned heartbeat timing and
// the session-start fallback (both `ActivePeer` Instants).
let now_ms = crate::time::mono_ms();
let heartbeat_interval = Duration::from_secs(self.config().node.heartbeat_interval_secs);
let dead_timeout = Duration::from_secs(self.config().node.link_dead_timeout_secs);
let dead_timeout_ms = dead_timeout.as_millis() as u64;
let max_resends = self.config().node.rate_limit.handshake_max_resends;
let heartbeat_msg = [LinkMessageType::Heartbeat.to_byte()];
// Collect heartbeats to send and dead peers to remove
let mut heartbeats: Vec<NodeAddr> = Vec::new();
let mut dead_peers: Vec<NodeAddr> = Vec::new();
// Build one liveness snapshot per peer, resolving every clock read and
// the rekey-suppression predicate shell-side. The snapshots own only
// `NodeAddr`/`bool`, so the peer-iteration borrow is released before the
// pure decision runs and the driving loop mutates the registry.
let snapshots: Vec<PeerLivenessSnapshot> = self
.peers
.iter()
.map(|(node_addr, peer)| {
// Check liveness via the MMP receiver's last-received monotonic
// ms. Fall back to session_start (an `ActivePeer` Instant) for
// peers that never sent data, keeping that branch in Instant
// space so no monotonic-ms epoch conversion is needed.
let time_dead = if let Some(mmp) = peer.mmp() {
match mmp.receiver.last_recv_ms() {
Some(last_ms) => now_ms.saturating_sub(last_ms) >= dead_timeout_ms,
None => now.duration_since(peer.session_start()) >= dead_timeout,
}
} else {
false
};
for (node_addr, peer) in self.peers.iter() {
// Check liveness via MMP receiver last_recv_time.
// Fall back to session_start for peers that never sent data.
let time_dead = if let Some(mmp) = peer.mmp() {
let reference_time = mmp
.receiver
.last_recv_time()
.unwrap_or(peer.session_start());
now.duration_since(reference_time) >= dead_timeout
} else {
false
};
// Suppress teardown while an FMP rekey is genuinely in flight
// with budget left: a rekey-handshake link is not silent. The
// msg1 resend cap guarantees this terminates (abandon on
// exhaustion or cutover on completion clears
// `rekey_in_progress`), so a truly dead link is reaped on the
// next cycle.
let rekey_active = peer.rekey_in_progress()
&& peer.rekey_msg1_resend_count() < max_resends
&& peer.rekey_msg1().is_some();
// Suppress teardown while an FMP rekey is genuinely in flight with
// budget left: a rekey-handshake link is not silent. The msg1
// resend cap guarantees this terminates (abandon on exhaustion or
// cutover on completion clears `rekey_in_progress`), so a truly
// dead link is reaped on the next cycle.
let rekey_active = peer.rekey_in_progress()
&& peer.rekey_msg1_resend_count() < max_resends
&& peer.rekey_msg1().is_some();
// Check if heartbeat is due.
let heartbeat_due = match peer.last_heartbeat_sent() {
None => true,
Some(last) => now.duration_since(last) >= heartbeat_interval,
};
let is_dead = time_dead && !rekey_active;
if is_dead {
dead_peers.push(*node_addr);
continue;
}
PeerLivenessSnapshot {
peer: *node_addr,
time_dead,
rekey_active,
heartbeat_due,
}
})
.collect();
// Check if heartbeat is due
let needs_heartbeat = match peer.last_heartbeat_sent() {
None => true,
Some(last) => now.duration_since(last) >= heartbeat_interval,
};
if needs_heartbeat {
heartbeats.push(*node_addr);
}
}
let actions = self.mmp.plan_heartbeats(&snapshots);
// Remove dead peers and schedule auto-reconnect
// Wall-clock basis for reconnect scheduling, sourced once (as before).
let now_ms = std::time::SystemTime::now()
.duration_since(std::time::UNIX_EPOCH)
.map(|d| d.as_millis() as u64)
.unwrap_or(0);
for addr in &dead_peers {
debug!(
peer = %self.peer_display_name(addr),
timeout_secs = self.config().node.link_dead_timeout_secs,
"Removing peer: link dead timeout"
);
self.remove_active_peer(addr);
self.schedule_reconnect(*addr, now_ms);
}
// Send heartbeats (skip peers we just removed)
for addr in heartbeats {
if dead_peers.contains(&addr) {
continue;
}
if let Some(peer) = self.peers.get_mut(&addr) {
peer.mark_heartbeat_sent(now);
}
if let Err(e) = self
.send_encrypted_link_message(&addr, &heartbeat_msg)
.await
{
trace!(peer = %self.peer_display_name(&addr), error = %e, "Failed to send heartbeat");
// Drive the planned actions: all reaps first (each removed +
// reconnect-scheduled), then all heartbeats (a just-reaped peer is never
// heartbeated — the core never emits both for the same peer).
for action in actions {
match action {
MmpAction::ReapPeer { peer } => {
// Log SHELL-SIDE before routing so the reap keeps the
// `fips::node::handlers::mmp` tracing target (no relocation into
// the executor, no target pin needed).
debug!(
peer = %self.peer_display_name(&peer),
timeout_secs = self.config().node.link_dead_timeout_secs,
"Removing peer: link dead timeout"
);
self.route_link_dead(peer, now_ms).await;
}
MmpAction::Heartbeat { peer } => {
if let Some(p) = self.peers.get_mut(&peer) {
p.mark_heartbeat_sent(now);
}
if let Err(e) = self
.send_encrypted_link_message(&peer, &heartbeat_msg)
.await
{
trace!(peer = %self.peer_display_name(&peer), error = %e, "Failed to send heartbeat");
}
}
MmpAction::SendLinkReport { .. }
| MmpAction::LogLink { .. }
| MmpAction::SendSessionReport { .. }
| MmpAction::LogSession { .. } => {}
}
}
}
/// Route a link-dead liveness reap through the peer machine + executor.
/// Mirrors [`route_rekey_cadence`](Node::route_rekey_cadence): the
/// shell already decided (the tick sweep's `plan_heartbeats` batch emitted
/// this `ReapPeer` in phase order), so the machine only CONSUMES the decision
/// via [`PeerEvent::LinkDeadSuspected`]. The resulting executor arms
/// (`InvalidateSendState` → `remove_active_peer`, `ReportLost` →
/// `note_link_dead`) reproduce the pre-refactor inline reap body exactly.
///
/// An established peer always has a `peer_machine`. If the peer vanished
/// between snapshot and effect, the old inline body was already a
/// no-op, so we return; if the machine is absent (which should be impossible)
/// we fall back to the byte-identical inline body under a `debug_assert`.
///
/// `now_ms` is the sweep's hoisted wall-clock ms (the same value the old reap
/// fed `note_link_dead`); it flows to the executor `ReportLost` arm via
/// `ambient.now_ms`.
async fn route_link_dead(&mut self, node_addr: NodeAddr, now_ms: u64) {
let link = match self.peers.get(&node_addr) {
Some(peer) => peer.link_id(),
None => return,
};
if !self.peer_machines.contains_key(&link) {
debug_assert!(false, "peer machine present for every established peer");
self.remove_active_peer(&node_addr);
self.note_link_dead(node_addr, now_ms);
return;
}
let ambient = self.link_dead_ctx(&node_addr, now_ms);
self.advance_peer_machine(link, PeerEvent::LinkDeadSuspected, Self::now_ms(), &ambient)
.await;
}
/// Ambient shell facts for the routed liveness reap. Mirrors
/// [`rekey_cadence_ctx`](Node::rekey_cadence_ctx). The executor reads only
/// `verified_identity` (`InvalidateSendState` → `remove_active_peer` resolves
/// its `NodeAddr` from it, so it must equal `node_addr`) and `now_ms`
/// (`ReportLost` → `note_link_dead`, the wall-clock reconnect basis). The
/// transport/index/direction fields are unused by these two arms and are
/// populated best-effort for coherence. `now_ms` is threaded in (rather than
/// re-read) so the value fed to `note_link_dead` is byte-identical to the old
/// reap's hoisted wall-clock for every peer in the sweep.
fn link_dead_ctx(&self, node_addr: &NodeAddr, now_ms: u64) -> PeerActionCtx {
let peer = &self.peers[node_addr];
PeerActionCtx {
verified_identity: *peer.identity(),
transport_id: peer.transport_id().unwrap_or_else(|| TransportId::new(0)),
remote_addr: peer
.current_addr()
.cloned()
.unwrap_or_else(|| TransportAddr::new(Vec::new())),
our_index: peer.our_index(),
their_index: peer.their_index(),
now_ms,
is_outbound: false,
}
}
}
src/node/handlers/mod.rs
+2 -8
View File
@@ -1,14 +1,8 @@
//! RX event loop and message handlers.
//! Message handlers: per-message-type behavior on `impl Node`.
#[cfg(unix)]
pub(crate) mod connected_udp;
pub(crate) mod discovery;
mod dispatch;
mod encrypted;
mod forwarding;
mod handshake;
pub(crate) mod lookup;
mod mmp;
mod rekey;
mod rx_loop;
pub(in crate::node) mod session;
mod timeout;
src/node/handlers/rekey.rs
+407 -307
View File
@@ -7,28 +7,28 @@
use crate::NodeAddr;
use crate::node::Node;
use crate::node::wire::build_msg1;
use crate::node::dataplane::PeerActionCtx;
use crate::noise::HandshakeState;
use crate::protocol::{SessionDatagram, SessionSetup};
use crate::peer::machine::PeerEvent;
use crate::proto::fmp::wire::build_msg1;
use crate::proto::fmp::{ConnAction, LifecycleView, PeerSnapshot, RekeyCfg, RekeyResendSnapshot};
use crate::proto::fsp::{
FspAction, RekeyMsg3ResendSnapshot, SessionSetup, SessionSnapshot, cutover_timer_elapsed,
};
use crate::proto::link::SessionDatagram;
use crate::transport::{TransportAddr, TransportId};
use tracing::{debug, trace, warn};
/// Keep previous session alive for this long after cutover.
///
/// FMP-scoped copy for `check_rekey`; the FSP session-rekey timing bounds live
/// in `crate::proto::fsp::limits`.
const DRAIN_WINDOW_SECS: u64 = 10;
/// Suppress local rekey initiation for this long after receiving
/// a peer's rekey msg1.
/// a peer's rekey msg1. FMP-scoped copy for `check_rekey`.
const REKEY_DAMPENING_SECS: u64 = 30;
/// Liveness bound on how long the FSP rekey initiator holds the
/// `current` + `pending` state before cutting over to the new epoch.
///
/// This is NOT safety-critical: overlapping-epoch trial-decrypt covers
/// any skew between the two endpoints' cutovers. The timer only bounds
/// how long the initiator advertises the old K-bit. An opportunistic
/// early cutover also fires if the initiator authenticates a peer frame
/// against its own `pending` session (the responder cut over first).
const FSP_CUTOVER_DELAY_MS: u64 = 2000;
impl Node {
/// Periodic rekey check. Called from the tick loop.
///
@@ -41,121 +41,225 @@ impl Node {
return;
}
let rekey_after_secs = self.config().node.rekey.after_secs;
let rekey_after_messages = self.config().node.rekey.after_messages;
let cfg = RekeyCfg {
after_secs: self.config().node.rekey.after_secs,
after_messages: self.config().node.rekey.after_messages,
};
// Collect peers that need action (to avoid borrow conflicts)
let mut peers_to_cutover: Vec<NodeAddr> = Vec::new();
let mut peers_to_drain: Vec<NodeAddr> = Vec::new();
let mut peers_to_rekey: Vec<NodeAddr> = Vec::new();
for (node_addr, peer) in &self.peers {
if !peer.has_session() || !peer.is_healthy() {
continue;
}
// 1. Initiator-side cutover: we completed a rekey and have
// a pending session ready. Cut over on the next tick.
if peer.pending_new_session().is_some() && !peer.rekey_in_progress() {
peers_to_cutover.push(*node_addr);
continue;
}
// 2. Drain window expiry
if peer.is_draining() && peer.drain_expired(DRAIN_WINDOW_SECS) {
peers_to_drain.push(*node_addr);
}
// 3. Rekey trigger
if peer.rekey_in_progress() {
continue;
}
if peer.is_rekey_dampened(REKEY_DAMPENING_SECS) {
continue;
}
let elapsed = peer.session_established_at().elapsed().as_secs();
let counter = peer
.noise_session()
.map(|s| s.current_send_counter())
.unwrap_or(0);
// Apply per-session symmetric jitter to desynchronize
// dual-initiation in symmetric-start meshes.
let effective_after_secs =
rekey_after_secs.saturating_add_signed(peer.rekey_jitter_secs());
if elapsed >= effective_after_secs || counter >= rekey_after_messages {
peers_to_rekey.push(*node_addr);
// The shell snapshots each healthy peer's rekey ages/flags (every clock
// read resolved here); the core decides cutover/drain/trigger with no
// clock, phase-grouped to preserve the pre-refactor execution order.
// The batch `poll_rekey` + snapshots STAY SHELL-SIDE and BYTE-UNCHANGED:
// the cross-peer phase-grouping (all Cutover → all Drain →
// all InitiateRekey) governs the shared `index_allocator` free-then-alloc
// SEQUENCE that appears on the wire. The machine must NOT re-poll; it
// CONSUMES each decided `ConnAction` in the same order the batch returned.
let snapshots = self.rekey_peers();
for action in self.fmp.poll_rekey(snapshots, &cfg) {
match action {
// Initiator cutover: route the decided action through the peer
// machine + executor. The executor's `SwapSendState` arm
// reproduces the pre-refactor cutover body EXACTLY.
ConnAction::Cutover { peer: node_addr } => {
self.route_rekey_cadence(node_addr, ConnAction::Cutover { peer: node_addr })
.await;
}
// Drain completion: route through the machine + executor. The
// executor's `CompleteDrain` arm reads the REAL previous index
// from `complete_drain()` and frees it at the same point the old
// inline body did (index-order preserving).
ConnAction::Drain { peer: node_addr } => {
self.route_rekey_cadence(node_addr, ConnAction::Drain { peer: node_addr })
.await;
}
// Initiate a new rekey: STAYS INLINE (the Noise msg1 build +
// index allocation are a shell-side leaf, byte-unchanged). Feed
// the machine a `RekeyInitiated` observation afterward so its
// control state stays coherent for the next tick's Cutover/Drain.
ConnAction::InitiateRekey { peer: node_addr } => {
self.initiate_rekey(&node_addr).await;
self.observe_rekey_initiated(&node_addr);
}
#[allow(unreachable_patterns)]
_ => {}
}
}
}
// Execute cutover for initiator side
for node_addr in peers_to_cutover {
let did_cutover = if let Some(peer) = self.peers.get_mut(&node_addr) {
if let Some(_old_our_index) = peer.cutover_to_new_session() {
// New index was pre-registered in peers_by_index
// during msg2 handling (handshake.rs).
debug_assert!(
peer.transport_id().is_some()
&& peer.our_index().is_some()
&& self.peers_by_index.contains_key(&(
peer.transport_id().unwrap(),
peer.our_index().unwrap().as_u32()
)),
"peers_by_index should contain pre-registered new index after cutover"
);
debug!(
peer = %self.peer_display_name(&node_addr),
"Rekey cutover complete (initiator), K-bit flipped"
);
true
} else {
false
}
/// Route a cadence-decided `Cutover`/`Drain` `ConnAction` through the peer
/// machine + executor. The shell already decided (batch `poll_rekey`);
/// the machine consumes via [`PeerEvent::RekeyConsume`] WITHOUT re-polling,
/// preserving the phase order. The `SwapSendState`/`CompleteDrain` executor
/// arms reproduce the pre-refactor inline effect bodies exactly.
///
/// An established peer always has a `peer_machine`. If the peer vanished
/// between snapshot and effect, the old inline body was a no-op, so we do
/// nothing; if the machine is absent (which should be impossible) we fall
/// back to the byte-identical inline body under a `debug_assert`.
async fn route_rekey_cadence(&mut self, node_addr: NodeAddr, action: ConnAction) {
let link = match self.peers.get(&node_addr) {
Some(peer) => peer.link_id(),
None => return,
};
if !self.peer_machines.contains_key(&link) {
debug_assert!(
false,
"peer machine present for every established rekey peer"
);
match action {
ConnAction::Cutover { peer } => self.cutover_peer_inline(&peer),
ConnAction::Drain { peer } => self.drain_peer_inline(&peer),
_ => {}
}
return;
}
let ambient = self.rekey_cadence_ctx(&node_addr);
self.advance_peer_machine(
link,
PeerEvent::RekeyConsume { action },
ambient.now_ms,
&ambient,
)
.await;
}
/// Feed the machine the `RekeyInitiated` observation after the inline
/// `initiate_rekey`. The obs emits no action, so there is no executor
/// pass — a bare `step` keeps the machine's control state coherent.
fn observe_rekey_initiated(&mut self, node_addr: &NodeAddr) {
let link = match self.peers.get(node_addr) {
Some(peer) => peer.link_id(),
None => return,
};
if let Some(machine) = self.peer_machines.get_mut(&link) {
let acts = machine.step(
PeerEvent::RekeyInitiated,
Self::now_ms(),
&mut self.index_allocator,
);
debug_assert!(acts.is_empty(), "RekeyInitiated is a pure observation");
} else {
debug_assert!(
false,
"peer machine present for every established rekey peer"
);
}
}
/// Ambient shell facts for the routed cadence Cutover/Drain step. Only
/// `verified_identity` is read by the `SwapSendState`/`CompleteDrain`
/// executor arms — `SwapSendState` resolves its `NodeAddr` from it (so it must
/// equal `node_addr`), and `CompleteDrain` carries its peer in the action
/// payload. The transport/index/direction fields are unused by these two arms
/// (they matter only to `PromoteToActive`, never emitted on this path) and are
/// populated best-effort for coherence.
fn rekey_cadence_ctx(&self, node_addr: &NodeAddr) -> PeerActionCtx {
let peer = &self.peers[node_addr];
PeerActionCtx {
verified_identity: *peer.identity(),
transport_id: peer.transport_id().unwrap_or_else(|| TransportId::new(0)),
remote_addr: peer
.current_addr()
.cloned()
.unwrap_or_else(|| TransportAddr::new(Vec::new())),
our_index: peer.our_index(),
their_index: peer.their_index(),
now_ms: Self::now_ms(),
is_outbound: false,
}
}
/// Pre-refactor initiator cutover body, retained as the release fallback for
/// the (should-be-impossible) missing-machine case. Byte-identical to the old
/// inline `ConnAction::Cutover` arm and to the executor's `SwapSendState` arm.
fn cutover_peer_inline(&mut self, node_addr: &NodeAddr) {
let did_cutover = if let Some(peer) = self.peers.get_mut(node_addr) {
if let Some(_old_our_index) = peer.cutover_to_new_session() {
// New index was pre-registered in peers_by_index during msg2
// handling (handshake.rs).
debug_assert!(
peer.transport_id().is_some()
&& peer.our_index().is_some()
&& self.peers_by_index.contains_key(&(
peer.transport_id().unwrap(),
peer.our_index().unwrap().as_u32()
)),
"peers_by_index should contain pre-registered new index after cutover"
);
debug!(
peer = %self.peer_display_name(node_addr),
"Rekey cutover complete (initiator), K-bit flipped"
);
true
} else {
false
};
// Re-register the new session with the decrypt worker — the
// cache_key (transport_id, our_index) just changed, so the
// old worker entry is stale and every packet on the new
// session would miss the worker's HashMap lookup.
#[cfg(unix)]
if did_cutover {
self.register_decrypt_worker_session(&node_addr);
}
#[cfg(not(unix))]
let _ = did_cutover;
} else {
false
};
// Re-register the new session with the decrypt worker — the cache_key
// (transport_id, our_index) just changed, so the old worker entry is
// stale and every packet on the new session would miss the lookup.
#[cfg(unix)]
if did_cutover {
self.register_decrypt_worker_session(node_addr);
}
#[cfg(not(unix))]
let _ = did_cutover;
}
// Execute drain completion
for node_addr in peers_to_drain {
// Extract the old index and transport_id under the peer
// borrow, then drop the borrow so the cache_key cleanup
// below can take &mut self for unregister_decrypt_worker_session.
let drained = self
.peers
.get_mut(&node_addr)
.and_then(|peer| peer.complete_drain().map(|idx| (idx, peer.transport_id())));
if let Some((old_our_index, transport_id)) = drained {
if let Some(tid) = transport_id {
let cache_key = (tid, old_our_index.as_u32());
self.peers_by_index.remove(&cache_key);
#[cfg(unix)]
self.unregister_decrypt_worker_session(cache_key);
}
let _ = self.index_allocator.free(old_our_index);
trace!(
peer = %self.peer_display_name(&node_addr),
old_index = %old_our_index,
"Drain complete, previous session erased"
);
/// Pre-refactor drain-completion body, retained as the release fallback for
/// the (should-be-impossible) missing-machine case. Byte-identical to the old
/// inline `ConnAction::Drain` arm and to the executor's `CompleteDrain` arm.
fn drain_peer_inline(&mut self, node_addr: &NodeAddr) {
// Extract the old index and transport_id under the peer borrow, then drop
// the borrow so the cache_key cleanup below can take &mut self for
// unregister_decrypt_worker_session.
let drained = self
.peers
.get_mut(node_addr)
.and_then(|peer| peer.complete_drain().map(|idx| (idx, peer.transport_id())));
if let Some((old_our_index, transport_id)) = drained {
if let Some(tid) = transport_id {
let cache_key = (tid, old_our_index.as_u32());
self.peers_by_index.remove(&cache_key);
#[cfg(unix)]
self.unregister_decrypt_worker_session(cache_key);
}
let _ = self.index_allocator.free(old_our_index);
trace!(
peer = %self.peer_display_name(node_addr),
old_index = %old_our_index,
"Drain complete, previous session erased"
);
}
}
// Initiate new rekeys
for node_addr in peers_to_rekey {
self.initiate_rekey(&node_addr).await;
}
/// Snapshot every healthy peer with a session for the rekey decision,
/// pre-computing its monotonic ages and timer predicates so the pure core
/// applies the thresholds without reading a clock (see [`PeerSnapshot`]).
///
/// Lives here, beside the drain/dampening constants and the FSP analog, so
/// the forward-merge onto `next` reconciles rekey timing in one place.
pub(in crate::node) fn rekey_peer_snapshots(&self) -> Vec<PeerSnapshot> {
self.peers
.iter()
.filter(|(_, peer)| peer.has_session() && peer.is_healthy())
.map(|(node_addr, peer)| PeerSnapshot {
addr: *node_addr,
has_pending: peer.pending_new_session().is_some(),
rekey_in_progress: peer.rekey_in_progress(),
is_draining: peer.is_draining(),
drain_expired: peer.drain_expired(DRAIN_WINDOW_SECS),
is_dampened: peer.is_rekey_dampened(REKEY_DAMPENING_SECS),
elapsed_secs: peer.session_established_at().elapsed().as_secs(),
counter: peer
.noise_session()
.map(|s| s.current_send_counter())
.unwrap_or(0),
jitter_secs: peer.rekey_jitter_secs(),
})
.collect()
}
/// Initiate an outbound rekey to a peer.
@@ -260,60 +364,74 @@ impl Node {
let backoff = self.config().node.rate_limit.handshake_resend_backoff;
let max_resends = self.config().node.rate_limit.handshake_max_resends;
// Collect peers needing action
let mut to_resend: Vec<(NodeAddr, Vec<u8>)> = Vec::new();
let mut to_abandon: Vec<NodeAddr> = Vec::new();
// The shell snapshots each in-flight rekey (resend-due predicate
// resolved here); the core classifies abandon-vs-resend and computes
// the backoff, abandons first.
let candidates = self.rekey_resend_candidates(now_ms);
for action in
self.fmp
.poll_rekey_resends(candidates, now_ms, interval_ms, backoff, max_resends)
{
match action {
// Abandon rekey cycles that exhausted their retransmission budget.
ConnAction::AbandonRekey { peer: node_addr } => {
if let Some(peer) = self.peers.get_mut(&node_addr) {
peer.abandon_rekey();
}
debug!(
peer = %self.peer_display_name(&node_addr),
"FMP rekey aborted: msg1 unconfirmed after max retransmissions, abandoning cycle"
);
}
ConnAction::ResendRekeyMsg1 {
peer: node_addr,
bytes,
next_resend_at_ms,
} => {
let (transport_id, remote_addr) = match self.peers.get(&node_addr) {
Some(p) => match (p.transport_id(), p.current_addr()) {
(Some(tid), Some(addr)) => (tid, addr.clone()),
_ => continue,
},
None => continue,
};
for (node_addr, peer) in &self.peers {
if !peer.rekey_in_progress() || peer.rekey_msg1().is_none() {
continue;
}
if peer.rekey_msg1_resend_count() >= max_resends {
to_abandon.push(*node_addr);
continue;
}
if peer.needs_msg1_resend(now_ms) {
to_resend.push((*node_addr, peer.rekey_msg1().unwrap().to_vec()));
let sent = if let Some(transport) = self.transports.get(&transport_id) {
transport.send(&remote_addr, &bytes).await.is_ok()
} else {
false
};
if sent && let Some(peer) = self.peers.get_mut(&node_addr) {
peer.record_rekey_msg1_resend(next_resend_at_ms);
let count = peer.rekey_msg1_resend_count();
trace!(
peer = %self.peer_display_name(&node_addr),
resend = count,
"Resent rekey msg1"
);
}
}
#[allow(unreachable_patterns)]
_ => {}
}
}
}
// Abandon rekey cycles that exhausted their retransmission budget.
for node_addr in to_abandon {
if let Some(peer) = self.peers.get_mut(&node_addr) {
peer.abandon_rekey();
}
debug!(
peer = %self.peer_display_name(&node_addr),
"FMP rekey aborted: msg1 unconfirmed after max retransmissions, abandoning cycle"
);
}
for (node_addr, msg1_bytes) in to_resend {
let (transport_id, remote_addr) = match self.peers.get(&node_addr) {
Some(p) => match (p.transport_id(), p.current_addr()) {
(Some(tid), Some(addr)) => (tid, addr.clone()),
_ => continue,
},
None => continue,
};
let sent = if let Some(transport) = self.transports.get(&transport_id) {
transport.send(&remote_addr, &msg1_bytes).await.is_ok()
} else {
false
};
if sent && let Some(peer) = self.peers.get_mut(&node_addr) {
let count = peer.rekey_msg1_resend_count() + 1;
let next = now_ms + (interval_ms as f64 * backoff.powi(count as i32)) as u64;
peer.record_rekey_msg1_resend(next);
trace!(
peer = %self.peer_display_name(&node_addr),
resend = count,
"Resent rekey msg1"
);
}
}
/// Snapshot every peer with a rekey handshake in flight (and a stored
/// msg1) for the retransmission decision, pre-evaluating the resend-due
/// predicate against `now_ms` so the core reads no clock.
pub(in crate::node) fn rekey_resend_snapshots(&self, now_ms: u64) -> Vec<RekeyResendSnapshot> {
self.peers
.iter()
.filter(|(_, peer)| peer.rekey_in_progress() && peer.rekey_msg1().is_some())
.map(|(node_addr, peer)| RekeyResendSnapshot {
peer: *node_addr,
resend_count: peer.rekey_msg1_resend_count(),
needs_resend: peer.needs_msg1_resend(now_ms),
msg1: peer.rekey_msg1().unwrap().to_vec(),
})
.collect()
}
/// Retransmit FSP rekey msg3 until the responder is confirmed on the
@@ -352,66 +470,77 @@ impl Node {
let ttl = self.config().node.session.default_ttl;
let my_addr = *self.node_addr();
// Collect rekey initiators whose msg3 retransmission is due.
let mut to_resend: Vec<(NodeAddr, Vec<u8>)> = Vec::new();
let mut to_abandon: Vec<NodeAddr> = Vec::new();
for (node_addr, entry) in &self.sessions {
// Only the rekey initiator retains a msg3 payload.
let payload = match entry.rekey_msg3_payload() {
Some(p) => p,
None => continue,
};
if entry.rekey_msg3_next_resend_ms() == 0 || now_ms < entry.rekey_msg3_next_resend_ms()
{
continue;
}
if entry.rekey_msg3_resend_count() >= max_resends {
to_abandon.push(*node_addr);
continue;
}
to_resend.push((*node_addr, payload.to_vec()));
}
// Abandon rekey cycles that exhausted their retransmission budget.
for node_addr in to_abandon {
if let Some(entry) = self.sessions.get_mut(&node_addr) {
entry.abandon_rekey();
}
debug!(
peer = %self.peer_display_name(&node_addr),
"FSP rekey aborted: msg3 unconfirmed after max retransmissions, abandoning cycle"
);
}
// Retransmit msg3 for cycles still within budget.
for (node_addr, payload) in to_resend {
let mut datagram = SessionDatagram::new(my_addr, node_addr, payload).with_ttl(ttl);
let sent = match self.send_session_datagram(&mut datagram).await {
Ok(_) => true,
Err(e) => {
// The shell snapshots each session retaining a msg3 payload (resend-due
// predicate resolved here); the core classifies abandon-vs-resend,
// abandons first.
let candidates = self.rekey_msg3_resend_snapshots(now_ms);
for action in self.fsp.poll_rekey_msg3_resends(candidates, max_resends) {
match action {
FspAction::AbandonRekey { addr } => {
if let Some(entry) = self.sessions.get_mut(&addr) {
entry.abandon_rekey();
}
debug!(
peer = %self.peer_display_name(&node_addr),
error = %e,
"FSP rekey msg3 retransmission failed"
peer = %self.peer_display_name(&addr),
"FSP rekey aborted: msg3 unconfirmed after max retransmissions, abandoning cycle"
);
false
}
};
FspAction::ResendSessionMsg3 { addr } => {
let payload = match self
.sessions
.get(&addr)
.and_then(|e| e.rekey_msg3_payload())
{
Some(p) => p.to_vec(),
None => continue,
};
let mut datagram = SessionDatagram::new(my_addr, addr, payload).with_ttl(ttl);
let sent = match self.send_session_datagram(&mut datagram).await {
Ok(_) => true,
Err(e) => {
debug!(
peer = %self.peer_display_name(&addr),
error = %e,
"FSP rekey msg3 retransmission failed"
);
false
}
};
if sent && let Some(entry) = self.sessions.get_mut(&node_addr) {
let count = entry.rekey_msg3_resend_count() + 1;
let next = now_ms + (interval_ms as f64 * backoff.powi(count as i32)) as u64;
entry.record_rekey_msg3_resend(next);
trace!(
peer = %self.peer_display_name(&node_addr),
resend = count,
"Resent FSP rekey msg3"
);
if sent && let Some(entry) = self.sessions.get_mut(&addr) {
let count = entry.rekey_msg3_resend_count() + 1;
let next =
now_ms + (interval_ms as f64 * backoff.powi(count as i32)) as u64;
entry.record_rekey_msg3_resend(next);
trace!(
peer = %self.peer_display_name(&addr),
resend = count,
"Resent FSP rekey msg3"
);
}
}
#[allow(unreachable_patterns)]
_ => {}
}
}
}
/// Snapshot every session retaining a rekey-msg3 payload for the
/// retransmission decision, pre-evaluating the resend-due predicate against
/// `now_ms` so the core reads no clock.
fn rekey_msg3_resend_snapshots(&self, now_ms: u64) -> Vec<RekeyMsg3ResendSnapshot> {
self.sessions
.iter()
.filter(|(_, entry)| entry.rekey_msg3_payload().is_some())
.map(|(node_addr, entry)| RekeyMsg3ResendSnapshot {
addr: *node_addr,
resend_count: entry.rekey_msg3_resend_count(),
resend_due: entry.rekey_msg3_next_resend_ms() != 0
&& now_ms >= entry.rekey_msg3_next_resend_ms(),
})
.collect()
}
/// Periodic session (FSP) rekey check. Called from the tick loop.
///
/// For each established session:
@@ -429,100 +558,71 @@ impl Node {
return;
}
let rekey_after_secs = self.config().node.rekey.after_secs;
let rekey_after_messages = self.config().node.rekey.after_messages;
let cfg = crate::proto::fsp::RekeyCfg {
after_secs: self.config().node.rekey.after_secs,
after_messages: self.config().node.rekey.after_messages,
};
let now_ms = Self::now_ms();
let drain_ms = DRAIN_WINDOW_SECS * 1000;
let dampening_ms = REKEY_DAMPENING_SECS * 1000;
let mut sessions_to_cutover: Vec<NodeAddr> = Vec::new();
let mut sessions_to_drain: Vec<NodeAddr> = Vec::new();
let mut sessions_to_rekey: Vec<NodeAddr> = Vec::new();
for (node_addr, entry) in &self.sessions {
if !entry.is_established() {
continue;
}
// 1. Initiator-side cutover (option A): completed rekey,
// pending session ready, liveness timer elapsed. This is
// an unconditional timer, NOT gated on responder progress —
// overlapping-epoch trial-decrypt covers the cutover skew,
// so flipping the K-bit here is always safe. An
// opportunistic early cutover also happens in
// `handle_encrypted_session_msg` if the initiator
// authenticates a peer frame against its own `pending`.
if entry.pending_new_session().is_some()
&& !entry.has_rekey_in_progress()
&& entry.is_rekey_initiator()
&& now_ms.saturating_sub(entry.rekey_completed_ms()) >= FSP_CUTOVER_DELAY_MS
{
sessions_to_cutover.push(*node_addr);
continue;
}
// 2. Drain window expiry
if entry.is_draining() && entry.drain_expired(now_ms, drain_ms) {
sessions_to_drain.push(*node_addr);
}
// 3. Rekey trigger
if entry.has_rekey_in_progress() {
continue;
}
if entry.pending_new_session().is_some() {
continue; // Pending session present, awaiting cutover
}
if entry.rekey_msg3_payload().is_some() {
// Initiator already cut over on its liveness timer but is
// still retransmitting msg3 to a responder not yet
// confirmed on the new epoch. Don't start another rekey
// until the current cycle's msg3 is delivered or abandoned.
continue;
}
if entry.is_rekey_dampened(now_ms, dampening_ms) {
continue;
}
let elapsed_secs = now_ms.saturating_sub(entry.session_start_ms()) / 1000;
let counter = entry.send_counter();
// Apply per-session symmetric jitter to desynchronize
// dual-initiation in symmetric-start meshes.
let effective_after_secs =
rekey_after_secs.saturating_add_signed(entry.rekey_jitter_secs());
if elapsed_secs >= effective_after_secs || counter >= rekey_after_messages {
sessions_to_rekey.push(*node_addr);
// The shell snapshots each established session's rekey ages/flags
// (every clock read resolved here); the core decides
// cutover/drain/trigger with no clock, phase-grouped to preserve the
// pre-refactor execution order.
let snapshots = self.session_rekey_snapshots(now_ms);
for action in self.fsp.poll_rekey(snapshots, &cfg) {
match action {
FspAction::CutOver { addr } => {
if let Some(entry) = self.sessions.get_mut(&addr)
&& entry.cutover_to_new_session(now_ms)
{
debug!(
peer = %self.peer_display_name(&addr),
"FSP rekey cutover complete (initiator), K-bit flipped"
);
}
}
FspAction::CompleteDrain { addr } => {
if let Some(entry) = self.sessions.get_mut(&addr) {
entry.complete_drain();
trace!(
peer = %self.peer_display_name(&addr),
"FSP drain complete, previous session erased"
);
}
}
FspAction::InitiateRekey { addr } => {
self.initiate_session_rekey(&addr).await;
}
#[allow(unreachable_patterns)]
_ => {}
}
}
}
// Execute cutover for initiator side
for node_addr in sessions_to_cutover {
if let Some(entry) = self.sessions.get_mut(&node_addr)
&& entry.cutover_to_new_session(now_ms)
{
debug!(
peer = %self.peer_display_name(&node_addr),
"FSP rekey cutover complete (initiator), K-bit flipped"
);
}
}
// Execute drain completion
for node_addr in sessions_to_drain {
if let Some(entry) = self.sessions.get_mut(&node_addr) {
entry.complete_drain();
trace!(
peer = %self.peer_display_name(&node_addr),
"FSP drain complete, previous session erased"
);
}
}
// Initiate new rekeys
for node_addr in sessions_to_rekey {
self.initiate_session_rekey(&node_addr).await;
}
/// Snapshot every established session for the FSP rekey decision,
/// pre-computing its monotonic age and timer predicates so the pure core
/// applies the thresholds without reading a clock (see [`SessionSnapshot`]).
fn session_rekey_snapshots(&self, now_ms: u64) -> Vec<SessionSnapshot> {
let drain_ms = crate::proto::fsp::limits::DRAIN_WINDOW_SECS * 1000;
let dampening_ms = crate::proto::fsp::limits::REKEY_DAMPENING_SECS * 1000;
self.sessions
.iter()
.filter(|(_, entry)| entry.is_established())
.map(|(node_addr, entry)| SessionSnapshot {
addr: *node_addr,
has_pending: entry.pending_new_session().is_some(),
rekey_in_progress: entry.has_rekey_in_progress(),
is_rekey_initiator: entry.is_rekey_initiator(),
cutover_timer_elapsed: cutover_timer_elapsed(now_ms, entry.rekey_completed_ms()),
is_draining: entry.is_draining(),
drain_expired: entry.drain_expired(now_ms, drain_ms),
has_rekey_msg3_payload: entry.rekey_msg3_payload().is_some(),
is_dampened: entry.is_rekey_dampened(now_ms, dampening_ms),
elapsed_secs: now_ms.saturating_sub(entry.session_start_ms()) / 1000,
counter: entry.send_counter(),
jitter_secs: entry.rekey_jitter_secs(),
})
.collect()
}
/// Initiate an FSP session rekey.
src/node/handlers/session.rs
+397 -154
View File
@@ -6,34 +6,39 @@
//! encrypted data, and error signals (CoordsRequired, PathBroken).
use crate::NodeAddr;
use crate::mmp::report::ReceiverReport;
use crate::mmp::{MAX_SESSION_REPORT_INTERVAL_MS, MIN_SESSION_REPORT_INTERVAL_MS};
use crate::node::handlers::mmp::format_throughput;
use crate::node::reject::{RejectReason, SessionReject};
use crate::node::session::{EndToEndState, EpochSlot, SessionEntry};
use crate::node::session_wire::{
FSP_COMMON_PREFIX_SIZE, FSP_FLAG_CP, FSP_FLAG_K, FSP_HEADER_SIZE, FSP_PHASE_ESTABLISHED,
FSP_PHASE_MSG1, FSP_PHASE_MSG2, FSP_PHASE_MSG3, FSP_PORT_HEADER_SIZE, FSP_PORT_IPV6_SHIM,
FspCommonPrefix, FspEncryptedHeader, build_fsp_header, fsp_prepend_inner_header,
fsp_strip_inner_header, parse_encrypted_coords,
};
#[cfg(unix)]
use crate::node::wire::{
ESTABLISHED_HEADER_SIZE, FLAG_KEY_EPOCH, FLAG_SP, build_established_header,
};
use crate::node::{Node, NodeError};
use crate::noise::{
HandshakeState, XK_HANDSHAKE_MSG1_SIZE, XK_HANDSHAKE_MSG2_SIZE, XK_HANDSHAKE_MSG3_SIZE,
};
#[cfg(unix)]
use crate::protocol::LinkMessageType;
#[cfg(unix)]
use crate::protocol::SESSION_DATAGRAM_HEADER_SIZE;
use crate::protocol::{
CoordsRequired, FspInnerFlags, MtuExceeded, PathBroken, PathMtuNotification, SessionAck,
SessionDatagram, SessionMessageType, SessionMsg3, SessionReceiverReport, SessionSenderReport,
SessionSetup,
use crate::proto::fmp::wire::{
ESTABLISHED_HEADER_SIZE, FLAG_KEY_EPOCH, FLAG_SP, build_established_header,
};
use crate::protocol::{coords_wire_size, encode_coords};
use crate::proto::fsp::wire::{
FSP_COMMON_PREFIX_SIZE, FSP_FLAG_CP, FSP_FLAG_K, FSP_HEADER_SIZE, FSP_PHASE_ESTABLISHED,
FSP_PHASE_MSG1, FSP_PHASE_MSG2, FSP_PHASE_MSG3, FSP_PORT_HEADER_SIZE, FSP_PORT_IPV6_SHIM,
FspCommonPrefix, FspEncryptedHeader, build_fsp_header, fsp_prepend_inner_header,
fsp_strip_inner_header, parse_encrypted_coords,
};
use crate::proto::fsp::{
DecryptSlot, EpochReaction, FspAction, FspInnerFlags, SessionAck, SessionMessageType,
SessionMsg3, SessionSetup, mark_ipv6_ecn_ce,
};
#[cfg(unix)]
use crate::proto::link::LinkMessageType;
#[cfg(unix)]
use crate::proto::link::SESSION_DATAGRAM_HEADER_SIZE;
use crate::proto::link::SessionDatagram;
use crate::proto::mmp::{
BackoffUpdate, MmpAction, MmpSessionState, PathMtuNotification, ReceiverReport, SendResult,
SessionReceiverReport, SessionReportKind, SessionReportSnapshot, SessionSenderReport,
};
use crate::proto::mmp::{MAX_SESSION_REPORT_INTERVAL_MS, MIN_SESSION_REPORT_INTERVAL_MS};
use crate::proto::routing::{CoordsRequired, MtuExceeded, PathBroken, RoutingSignalType};
use crate::proto::stp::{coords_wire_size, encode_coords};
#[cfg(unix)]
use crate::transport::TransportHandle;
use crate::upper::icmp::FIPS_OVERHEAD;
@@ -51,8 +56,8 @@ struct PipelinedSend<'a> {
timestamp: u32,
fsp_flags: u8,
inner_plaintext: &'a [u8],
my_coords: Option<&'a crate::tree::TreeCoordinate>,
dest_coords: Option<&'a crate::tree::TreeCoordinate>,
my_coords: Option<&'a crate::proto::stp::TreeCoordinate>,
dest_coords: Option<&'a crate::proto::stp::TreeCoordinate>,
}
impl Node {
@@ -104,14 +109,14 @@ impl Node {
}
let error_type = inner[0];
let error_body = &inner[1..];
match SessionMessageType::from_byte(error_type) {
Some(SessionMessageType::CoordsRequired) => {
match RoutingSignalType::from_byte(error_type) {
Some(RoutingSignalType::CoordsRequired) => {
self.handle_coords_required(error_body).await;
}
Some(SessionMessageType::PathBroken) => {
Some(RoutingSignalType::PathBroken) => {
self.handle_path_broken(error_body).await;
}
Some(SessionMessageType::MtuExceeded) => {
Some(RoutingSignalType::MtuExceeded) => {
self.handle_mtu_exceeded(error_body).await;
}
_ => {
@@ -166,11 +171,14 @@ impl Node {
match parse_encrypted_coords(coord_data) {
Ok((src_coords, dest_coords, bytes_consumed)) => {
let now_ms = Self::now_ms();
if let Some(coords) = src_coords {
self.coord_cache.insert(*src_addr, coords, now_ms);
}
if let Some(coords) = dest_coords {
self.coord_cache.insert(*self.node_addr(), coords, now_ms);
let my_addr = *self.node_addr();
for action in
self.fsp
.plan_cache_coords(*src_addr, my_addr, src_coords, dest_coords)
{
if let FspAction::CacheCoords { addr, coords } = action {
self.coord_cache.insert(addr, coords, now_ms);
}
}
ciphertext_offset += bytes_consumed;
}
@@ -248,44 +256,55 @@ impl Node {
}
};
// React to the epoch the frame decrypted against.
match slot {
EpochSlot::Pending => {
// A frame that authenticates against `pending` is itself
// the cutover signal — proof the peer derived the new
// session and moved to it. Promote now: current →
// previous, pending → current, flip the K-bit. The
// header K-bit is no longer the gating event; the
// authenticated decrypt is.
// React to the epoch the frame decrypted against. The shell opened
// the frame; the core classifies the post-decrypt reaction over the
// plain-data slot + session flags, and the shell applies the
// `SessionEntry` mutation.
let decrypt_slot = match slot {
EpochSlot::Current => DecryptSlot::Current,
EpochSlot::Pending => DecryptSlot::Pending,
EpochSlot::Previous => DecryptSlot::Previous,
};
match self.fsp.classify_epoch(
decrypt_slot,
entry.rekey_msg3_payload().is_some(),
entry.pending_new_session().is_some(),
) {
EpochReaction::PromoteConfirming => {
// A frame that authenticates against `pending` is itself the
// cutover signal — proof the peer derived the new session and
// moved to it. The peer received msg3, so confirm it on the new
// epoch (stop retransmitting) before `handle_peer_kbit_flip`
// consumes the pending session, then promote.
info!(
peer = %self.peer_display_name(src_addr),
"Peer FSP new-epoch frame authenticated, FSP rekey cutover complete, promoting new session"
);
// The peer derived the new session, so it received msg3:
// confirm it on the new epoch and stop retransmitting.
// `handle_peer_kbit_flip` consumes the pending session,
// so confirm first.
if entry.rekey_msg3_payload().is_some() {
entry.confirm_peer_new_epoch();
}
entry.confirm_peer_new_epoch();
entry.handle_peer_kbit_flip(now_ms);
}
EpochSlot::Current => {
// If we still retain a msg3 retransmission payload but no
// longer hold a `pending` session, we are the rekey
// initiator that already cut over on its own timer:
// `current` is now the new epoch, so a frame decrypting
// against it confirms the responder reached the new
// epoch. Stop retransmitting msg3.
if entry.rekey_msg3_payload().is_some() && entry.pending_new_session().is_none() {
entry.confirm_peer_new_epoch();
}
EpochReaction::Promote => {
// Promote now: current → previous, pending → current, flip the
// K-bit. The header K-bit is only a hint; the authenticated
// decrypt is the gating event.
info!(
peer = %self.peer_display_name(src_addr),
"Peer FSP new-epoch frame authenticated, FSP rekey cutover complete, promoting new session"
);
entry.handle_peer_kbit_flip(now_ms);
}
EpochSlot::Previous => {
// The peer is still on the old epoch. `fsp_trial_decrypt`
// already refreshed the drain deadline so the `previous`
// slot is not retired while the peer keeps using it —
// no further state change here, just deliver.
EpochReaction::ConfirmResponder => {
// We are the rekey initiator that already cut over on its own
// timer: `current` is now the new epoch, so a frame decrypting
// against it confirms the responder reached it. Stop
// retransmitting msg3.
entry.confirm_peer_new_epoch();
}
EpochReaction::None => {
// Steady-state `current`, or an old-epoch `previous` straggler:
// `fsp_trial_decrypt` already refreshed the drain deadline so
// the `previous` slot is not retired while the peer keeps using
// it — no further state change, just deliver.
}
}
@@ -305,16 +324,16 @@ impl Node {
if let Some(entry) = self.sessions.get_mut(src_addr)
&& let Some(mmp) = entry.mmp_mut()
{
let now = std::time::Instant::now();
let now_ms = crate::time::mono_ms();
mmp.receiver
.record_recv(header.counter, timestamp, plaintext.len(), ce_flag, now);
.record_recv(header.counter, timestamp, plaintext.len(), ce_flag, now_ms);
// Spin bit: advance state machine for correct TX reflection.
// RTT samples not fed into SRTT — timestamp-echo provides
// accurate RTT; spin bit includes variable inter-frame delays.
let inner_flags = FspInnerFlags::from_byte(inner_flags_byte);
let _spin_rtt = mmp
.spin_bit
.rx_observe(inner_flags.spin_bit, header.counter, now);
.rx_observe(inner_flags.spin_bit, header.counter, now_ms);
}
// Feed path_mtu from datagram envelope to MMP path MTU tracking.
@@ -355,7 +374,7 @@ impl Node {
mark_ipv6_ecn_ce(&mut packet);
self.metrics().congestion.ce_received.inc();
}
if let Some(tun_tx) = &self.tun_tx {
if let Some(tun_tx) = &self.supervisor.tun_tx {
if let Err(e) = tun_tx.send(packet) {
debug!(error = %e, "Failed to deliver decompressed IPv6 packet to TUN");
}
@@ -444,7 +463,7 @@ impl Node {
if let Some(existing) = self.sessions.get(src_addr) {
if existing.is_initiating() {
// Simultaneous initiation: smaller NodeAddr wins as initiator
if self.identity().node_addr() < src_addr {
if crate::proto::fsp::initiation_winner(self.identity().node_addr(), src_addr) {
// We win — drop their setup, they'll process ours
debug!(
src = %self.peer_display_name(src_addr),
@@ -482,7 +501,10 @@ impl Node {
// simultaneously. Apply tie-breaker — smaller NodeAddr
// wins as initiator (same as initial session setup).
if rekey_in_progress {
if self.identity().node_addr() < src_addr {
if crate::proto::fsp::initiation_winner(
self.identity().node_addr(),
src_addr,
) {
// We win as initiator — drop their msg1.
debug!(
src = %self.peer_display_name(src_addr),
@@ -919,6 +941,221 @@ impl Node {
// === Session-layer MMP report handlers ===
/// Check all sessions for pending MMP reports and send them.
///
/// Called from the tick handler. Also emits periodic session MMP logs.
/// Uses the collect-then-send pattern to avoid borrowing conflicts.
pub(in crate::node) async fn check_session_mmp_reports(&mut self) {
let now_ms = crate::time::mono_ms();
// Build one report-gating snapshot per session, resolving every timing
// read shell-side into a `bool`. The snapshots own only
// `NodeAddr`/`MmpMode`/`bool`, so the session-iteration borrow is released
// before the pure decision runs and the driving loop mutates the
// reporting state / performs the sends.
let snapshots: Vec<SessionReportSnapshot> = self
.sessions
.iter()
.filter_map(|(dest_addr, entry)| {
let mmp = entry.mmp()?;
Some(SessionReportSnapshot {
dest: *dest_addr,
mode: mmp.mode(),
sr_due: mmp.sender.should_send_report(now_ms),
rr_due: mmp.receiver.should_send_report(now_ms),
mtu_due: mmp.path_mtu.should_send_notification(now_ms),
log_due: mmp.should_log(now_ms),
})
})
.collect();
let actions = self.mmp.plan_session_reports(&snapshots);
// Drive the planned actions in phase-grouped order (all logs, then the
// sends in per-session SR/RR/MTU order). Logs run first because the
// session operator log reads cumulative_packets_sent, which each send
// advances (send_session_msg -> sender.record_sent); the pre-refactor
// handler logged during its collect pass, before any send. Each build
// (`build_report`/`build_notification`, which advance interval/
// notification state) runs only on its SendSessionReport action, exactly
// as the pre-refactor collect pass did. Per-destination success/failure
// is collected for the backoff dedup + failure-log suppression.
let mut send_results: Vec<SendResult> = Vec::new();
for action in actions {
match action {
MmpAction::LogSession { dest } => {
// Resolve the display name exactly as the pre-refactor loop
// did (alias, else short_npub from the session's remote key).
let session_name = self.peer_aliases.get(&dest).cloned().unwrap_or_else(|| {
self.sessions
.get(&dest)
.map(|entry| {
let (xonly, _) = entry.remote_pubkey().x_only_public_key();
crate::PeerIdentity::from_pubkey(xonly).short_npub()
})
.unwrap_or_default()
});
if let Some(mmp) = self.sessions.get_mut(&dest).and_then(|e| e.mmp_mut()) {
Self::log_session_mmp_metrics(&session_name, mmp);
mmp.mark_logged(now_ms);
}
}
MmpAction::SendSessionReport { dest, kind } => {
let built = self
.sessions
.get_mut(&dest)
.and_then(|entry| entry.mmp_mut())
.and_then(|mmp| match kind {
SessionReportKind::Sender => {
mmp.sender.build_report(now_ms).map(|sr| {
(
SessionMessageType::SenderReport.to_byte(),
SessionSenderReport::from(&sr).encode(),
)
})
}
SessionReportKind::Receiver => {
mmp.receiver.build_report(now_ms).map(|rr| {
(
SessionMessageType::ReceiverReport.to_byte(),
SessionReceiverReport::from(&rr).encode(),
)
})
}
SessionReportKind::PathMtu => {
mmp.path_mtu.build_notification(now_ms).map(|mtu_value| {
(
SessionMessageType::PathMtuNotification.to_byte(),
PathMtuNotification::new(mtu_value).encode(),
)
})
}
});
let Some((msg_type, body)) = built else {
continue;
};
match self.send_session_msg(&dest, msg_type, &body).await {
Ok(()) => send_results.push(SendResult { dest, ok: true }),
Err(e) => {
// Peek at current failure count for log suppression
// (unchanged by the backoff apply, which runs later).
let failures = self
.sessions
.get(&dest)
.and_then(|entry| entry.mmp())
.map(|mmp| mmp.sender.consecutive_send_failures())
.unwrap_or(0);
if failures < 3 {
debug!(
dest = %self.peer_display_name(&dest),
msg_type,
error = %e,
"Failed to send session MMP report"
);
} else if failures == 3 {
debug!(
dest = %self.peer_display_name(&dest),
"Suppressing further session MMP send failure logs"
);
}
// failures > 3: silently suppressed
send_results.push(SendResult { dest, ok: false });
}
}
}
MmpAction::ReapPeer { .. }
| MmpAction::Heartbeat { .. }
| MmpAction::SendLinkReport { .. }
| MmpAction::LogLink { .. } => {}
}
}
// Deduplicate send results per destination (any-ok -> success, all-fail
// -> failure) and apply the backoff state transition for each dest.
for update in self.mmp.plan_backoff(&send_results) {
match update {
BackoffUpdate::Success { dest } => {
if let Some(mmp) = self.sessions.get_mut(&dest).and_then(|e| e.mmp_mut()) {
let prev = mmp.sender.record_send_success();
if prev > 3 {
debug!(
dest = %self.peer_display_name(&dest),
consecutive_failures = prev,
"Resumed session MMP reporting"
);
}
}
}
BackoffUpdate::Failure { dest } => {
if let Some(mmp) = self.sessions.get_mut(&dest).and_then(|e| e.mmp_mut()) {
mmp.sender.record_send_failure();
}
}
}
}
}
/// Emit periodic session MMP metrics.
fn log_session_mmp_metrics(session_name: &str, mmp: &MmpSessionState) {
let m = &mmp.metrics;
let rtt_str = if m.rtt_trend.initialized() {
format!("{:.1}ms", m.rtt_trend.long() / 1000.0)
} else {
"n/a".to_string()
};
let loss_str = if m.loss_trend.initialized() {
format!("{:.1}%", m.loss_trend.long() * 100.0)
} else {
"n/a".to_string()
};
let jitter_ms = mmp.receiver.jitter_us() as f64 / 1000.0;
debug!(
session = %session_name,
rtt = %rtt_str,
loss = %loss_str,
jitter = format_args!("{:.1}ms", jitter_ms),
goodput = %format_throughput(m.goodput_bps()),
mtu = mmp.path_mtu.last_observed_mtu(),
tx_pkts = mmp.sender.cumulative_packets_sent(),
rx_pkts = mmp.receiver.cumulative_packets_recv(),
"MMP session metrics"
);
}
/// Emit a teardown log summarizing lifetime session MMP metrics.
pub(in crate::node) fn log_session_mmp_teardown(session_name: &str, mmp: &MmpSessionState) {
let m = &mmp.metrics;
let jitter_ms = mmp.receiver.jitter_us() as f64 / 1000.0;
let rtt_str = match m.srtt_ms() {
Some(rtt) => format!("{:.1}ms", rtt),
None => "n/a".to_string(),
};
let loss_str = format!("{:.1}%", m.loss_rate() * 100.0);
debug!(
session = %session_name,
rtt = %rtt_str,
loss = %loss_str,
jitter = format_args!("{:.1}ms", jitter_ms),
etx = format_args!("{:.2}", m.etx),
goodput = %format_throughput(m.goodput_bps()),
send_mtu = mmp.path_mtu.current_mtu(),
observed_mtu = mmp.path_mtu.last_observed_mtu(),
tx_pkts = mmp.sender.cumulative_packets_sent(),
tx_bytes = mmp.sender.cumulative_bytes_sent(),
rx_pkts = mmp.receiver.cumulative_packets_recv(),
rx_bytes = mmp.receiver.cumulative_bytes_recv(),
"MMP session teardown"
);
}
/// Handle an incoming session-layer SenderReport (msg_type 0x11).
///
/// Informational only — the peer is telling us about what they sent.
@@ -974,9 +1211,11 @@ impl Node {
return;
};
let now = std::time::Instant::now();
mmp.metrics
.process_receiver_report(&rr, our_timestamp_ms, now);
let (_first_rtt, rr_log) =
mmp.metrics
.process_receiver_report(&rr, our_timestamp_ms, crate::time::mono_ms());
// Re-emit the operator trace the core used to log mid-decision.
super::mmp::log_rr_outcome(&rr, our_timestamp_ms, rr_log);
// Feed SRTT back to sender/receiver report interval tuning (session-layer bounds)
if let Some(srtt_ms) = mmp.metrics.srtt_ms() {
@@ -1042,8 +1281,9 @@ impl Node {
};
let old_mtu = mmp.path_mtu.current_mtu();
let now = std::time::Instant::now();
let changed = mmp.path_mtu.apply_notification(notif.path_mtu, now);
let changed = mmp
.path_mtu
.apply_notification(notif.path_mtu, crate::time::mono_ms());
let new_mtu = mmp.path_mtu.current_mtu();
if !changed {
@@ -1065,28 +1305,34 @@ impl Node {
// tighter of existing-or-new — never loosen the clamp.
let fips_addr = crate::FipsAddress::from_node_addr(src_addr);
match self.path_mtu_lookup.write() {
Ok(mut map) => match map.get(&fips_addr).copied() {
Some(existing) if existing <= new_mtu => {
Ok(mut map) => {
// Read existing, decide, and apply the write under one guard so
// the keep-tighter update stays atomic.
let prior = map.get(&fips_addr).copied();
let actions = self.fsp.plan_path_mtu_tighten(fips_addr, prior, new_mtu);
if actions.is_empty() {
debug!(
dest = %peer_name,
fips_addr = %fips_addr,
new_mtu,
existing,
existing = prior.unwrap_or(new_mtu),
"PathMtuNotification: keeping tighter existing path_mtu_lookup value"
);
}
other => {
map.insert(fips_addr, new_mtu);
debug!(
dest = %peer_name,
fips_addr = %fips_addr,
new_mtu,
prior = ?other,
map_len = map.len(),
"PathMtuNotification: tightened path_mtu_lookup"
);
for action in actions {
if let FspAction::TightenPathMtuLookup { fips_addr, mtu } = action {
map.insert(fips_addr, mtu);
debug!(
dest = %peer_name,
fips_addr = %fips_addr,
new_mtu,
prior = ?prior,
map_len = map.len(),
"PathMtuNotification: tightened path_mtu_lookup"
);
}
}
},
}
Err(e) => {
warn!(
dest = %peer_name,
@@ -1125,7 +1371,7 @@ impl Node {
// Send standalone CoordsWarmup immediately (rate-limited)
if self
.coords_response_rate_limiter
.should_send(&msg.dest_addr)
.should_send(&msg.dest_addr, Self::now_ms())
{
if let Some(entry) = self.sessions.get(&msg.dest_addr)
&& entry.is_established()
@@ -1141,12 +1387,19 @@ impl Node {
// Only trigger discovery if we have the target's identity cached —
// otherwise we can't verify the LookupResponse proof.
if self.has_cached_identity(&msg.dest_addr) {
self.maybe_initiate_lookup(&msg.dest_addr).await;
} else {
let has_cached_identity = self.has_cached_identity(&msg.dest_addr);
let actions = self
.fsp
.plan_coords_required_lookup(msg.dest_addr, has_cached_identity);
if actions.is_empty() {
debug!(dest = %msg.dest_addr,
"Skipping discovery after CoordsRequired: no cached identity for target");
}
for action in actions {
if let FspAction::InitiateLookup { dest } = action {
self.maybe_initiate_lookup(&dest).await;
}
}
// Reset coords warmup counter so the next N packets also include
// COORDS_PRESENT, re-warming transit caches along the path.
@@ -1186,7 +1439,7 @@ impl Node {
// Send standalone CoordsWarmup immediately (rate-limited)
if self
.coords_response_rate_limiter
.should_send(&msg.dest_addr)
.should_send(&msg.dest_addr, Self::now_ms())
{
if let Some(entry) = self.sessions.get(&msg.dest_addr)
&& entry.is_established()
@@ -1200,16 +1453,26 @@ impl Node {
"PathBroken response rate-limited, skipping standalone CoordsWarmup");
}
// Invalidate stale cached coordinates
self.coord_cache.remove(&msg.dest_addr);
// Trigger re-discovery to get fresh coordinates, but only if we have
// the target's identity cached — otherwise we can't verify the
// LookupResponse proof. This avoids a race when the XK responder
// receives PathBroken before msg3 completes (identity unknown).
if self.has_cached_identity(&msg.dest_addr) {
self.maybe_initiate_lookup(&msg.dest_addr).await;
} else {
// Invalidate stale cached coordinates, then (only if the target's
// identity is cached — else the LookupResponse proof cannot be verified,
// e.g. when the XK responder receives PathBroken before msg3 completes)
// trigger re-discovery. The core emits invalidate-then-lookup in order.
let has_cached_identity = self.has_cached_identity(&msg.dest_addr);
let actions = self
.fsp
.plan_path_broken(msg.dest_addr, has_cached_identity);
for action in actions {
match action {
FspAction::InvalidateCoords { addr } => {
self.coord_cache.remove(&addr);
}
FspAction::InitiateLookup { dest } => {
self.maybe_initiate_lookup(&dest).await;
}
_ => {}
}
}
if !has_cached_identity {
debug!(dest = %msg.dest_addr,
"Skipping discovery after PathBroken: no cached identity for target");
}
@@ -1256,8 +1519,10 @@ impl Node {
&& let Some(mmp) = entry.mmp_mut()
{
let old_mtu = mmp.path_mtu.current_mtu();
let now = std::time::Instant::now();
if mmp.path_mtu.apply_notification(msg.mtu, now) {
if mmp
.path_mtu
.apply_notification(msg.mtu, crate::time::mono_ms())
{
let new_mtu = mmp.path_mtu.current_mtu();
info!(
dest = %peer_name,
@@ -1277,28 +1542,34 @@ impl Node {
// tighter of existing-or-new — never loosen the clamp.
let fips_addr = crate::FipsAddress::from_node_addr(&msg.dest_addr);
match self.path_mtu_lookup.write() {
Ok(mut map) => match map.get(&fips_addr).copied() {
Some(existing) if existing <= msg.mtu => {
Ok(mut map) => {
// Read existing, decide, and apply the write under one guard so
// the keep-tighter update stays atomic.
let prior = map.get(&fips_addr).copied();
let actions = self.fsp.plan_path_mtu_tighten(fips_addr, prior, msg.mtu);
if actions.is_empty() {
debug!(
dest = %peer_name,
fips_addr = %fips_addr,
bottleneck_mtu = msg.mtu,
existing,
existing = prior.unwrap_or(msg.mtu),
"Reactive MtuExceeded: keeping tighter existing path_mtu_lookup value"
);
}
other => {
map.insert(fips_addr, msg.mtu);
debug!(
dest = %peer_name,
fips_addr = %fips_addr,
bottleneck_mtu = msg.mtu,
prior = ?other,
map_len = map.len(),
"Reactive MtuExceeded: tightened path_mtu_lookup"
);
for action in actions {
if let FspAction::TightenPathMtuLookup { fips_addr, mtu } = action {
map.insert(fips_addr, mtu);
debug!(
dest = %peer_name,
fips_addr = %fips_addr,
bottleneck_mtu = msg.mtu,
prior = ?prior,
map_len = map.len(),
"Reactive MtuExceeded: tightened path_mtu_lookup"
);
}
}
},
}
Err(e) => {
warn!(
dest = %peer_name,
@@ -1561,7 +1832,7 @@ impl Node {
send: PipelinedSend<'_>,
) -> Result<bool, NodeError> {
let dest_addr = send.dest_addr;
let Some(workers) = self.encrypt_workers.as_ref().cloned() else {
let Some(workers) = self.supervisor.encrypt_workers.as_ref().cloned() else {
return Ok(false);
};
@@ -2065,7 +2336,10 @@ impl Node {
/// Returns our own coordinates as a fallback (the SessionSetup will
/// carry src_coords for return path routing; empty dest_coords
/// would fail wire encoding since TreeCoordinate requires ≥1 entry).
pub(in crate::node) fn get_dest_coords(&self, dest: &NodeAddr) -> crate::tree::TreeCoordinate {
pub(in crate::node) fn get_dest_coords(
&self,
dest: &NodeAddr,
) -> crate::proto::stp::TreeCoordinate {
let now_ms = Self::now_ms();
if let Some(coords) = self.coord_cache.get(dest, now_ms) {
return coords.clone();
@@ -2174,7 +2448,7 @@ impl Node {
let our_ipv6 = FipsAddress::from_node_addr(self.node_addr()).to_ipv6();
if let Some(response) =
build_dest_unreachable(original_packet, DestUnreachableCode::NoRoute, our_ipv6)
&& let Some(tun_tx) = &self.tun_tx
&& let Some(tun_tx) = &self.supervisor.tun_tx
{
let _ = tun_tx.send(response);
}
@@ -2209,7 +2483,7 @@ impl Node {
// causes a PMTUD blackhole when both src and ICMP-src are local.
let dest_addr = Ipv6Addr::from(<[u8; 16]>::try_from(&original_packet[24..40]).unwrap());
if let Some(response) = build_packet_too_big(original_packet, mtu, dest_addr)
&& let Some(tun_tx) = &self.tun_tx
&& let Some(tun_tx) = &self.supervisor.tun_tx
{
debug!(
original_src = %src_addr,
@@ -2234,10 +2508,7 @@ impl Node {
let per_dest = self.config().node.session.pending_packets_per_dest;
let queue = self.pending_tun_packets.entry(dest_addr).or_default();
if queue.len() >= per_dest {
queue.pop_front(); // Drop oldest
}
queue.push_back(packet);
crate::proto::fsp::push_bounded_pending(queue, packet, per_dest);
}
/// Flush pending packets for a destination whose session just reached Established.
@@ -2288,31 +2559,3 @@ impl Node {
}
}
}
/// Mark ECN-CE in an IPv6 packet's Traffic Class field.
///
/// IPv6 Traffic Class occupies bits across bytes 0 and 1:
/// byte[0] bits[3:0] = TC[7:4]
/// byte[1] bits[7:4] = TC[3:0]
/// ECN is TC[1:0]. Only marks CE (0b11) if the packet is ECN-capable
/// (ECT(0) or ECT(1)). Packets with ECN=0b00 (Not-ECT) are never marked
/// per RFC 3168.
///
/// No checksum update needed: IPv6 has no header checksum, and the Traffic
/// Class field is not part of the TCP/UDP pseudo-header.
pub(in crate::node) fn mark_ipv6_ecn_ce(packet: &mut [u8]) {
if packet.len() < 2 {
return;
}
// Extract 8-bit Traffic Class from IPv6 header bytes 0-1
let tc = ((packet[0] & 0x0F) << 4) | (packet[1] >> 4);
let ecn = tc & 0x03;
// Only mark CE on ECN-capable packets (ECT(0)=0b10 or ECT(1)=0b01)
if ecn == 0 {
return;
}
// Set both ECN bits to 1 (CE = 0b11)
let new_tc = tc | 0x03;
packet[0] = (packet[0] & 0xF0) | (new_tc >> 4);
packet[1] = (new_tc << 4) | (packet[1] & 0x0F);
}
src/node/handlers/timeout.rs
+298 -77
View File
@@ -2,58 +2,108 @@
//! and handshake message resend scheduling.
use crate::node::Node;
use crate::peer::HandshakeState;
use crate::peer::machine::TimerKind;
use crate::proto::fmp::{
ConnAction, ConnSnapshot, LifecycleView, PeerSnapshot, RekeyResendSnapshot,
};
use crate::transport::LinkId;
use tracing::{debug, info};
impl LifecycleView for Node {
fn stale_connections(&self, now_ms: u64, timeout_ms: u64) -> Vec<ConnSnapshot> {
// `is_failed()` legs are always reaped here (~1s), as before. The
// idle-timeout is reaped here ONLY for legs whose timeout is not already
// driven by a machine `HandshakeTimeout` timer — i.e. inbound legs (IK
// inbound arms none). Outbound legs with an armed timer are reaped by
// `drive_handshake_timeouts`, so excluding them here avoids a double
// reap.
self.peer_machines
.iter()
.filter(|(_, machine)| machine.leg().is_some())
.filter(|(link_id, machine)| {
machine.is_failed()
|| (machine.conn_is_timed_out(now_ms, timeout_ms)
&& !self.peer_timers.get(*link_id).is_some_and(|timers| {
timers.contains_key(&TimerKind::HandshakeTimeout)
}))
})
.map(|(link_id, machine)| ConnSnapshot {
link: *link_id,
is_outbound: machine.conn_is_outbound(),
retry_addr: machine.conn_expected_identity().map(|id| *id.node_addr()),
resend_count: 0,
msg1: Vec::new(),
})
.collect()
}
fn rekey_peers(&self) -> Vec<PeerSnapshot> {
// The snapshot builder lives in `rekey` beside its drain/dampening
// constants; the read-seam unifies here.
self.rekey_peer_snapshots()
}
fn rekey_resend_candidates(&self, now_ms: u64) -> Vec<RekeyResendSnapshot> {
self.rekey_resend_snapshots(now_ms)
}
}
impl Node {
/// Check for timed-out handshake connections and clean them up.
///
/// Called periodically by the RX event loop. Removes connections that have
/// been idle longer than the configured handshake timeout or are in Failed state.
///
/// The stale/failed predicate and every registry mutation stay shell-side;
/// the retry-then-teardown choreography is the pure
/// [`Fmp::poll_timeouts`](crate::proto::fmp::Fmp::poll_timeouts) decision.
pub(in crate::node) fn check_timeouts(&mut self) {
if self.connections.is_empty() {
if self.connection_count() == 0 {
return;
}
let now_ms = Self::now_ms();
let timeout_ms = self.config().node.rate_limit.handshake_timeout_secs * 1000;
let stale: Vec<LinkId> = self
.connections
.iter()
.filter(|(_, conn)| conn.is_timed_out(now_ms, timeout_ms) || conn.is_failed())
.map(|(link_id, _)| *link_id)
.collect();
for link_id in stale {
// Log and schedule retry before cleanup (need connection state)
if let Some(conn) = self.connections.get(&link_id) {
let direction = conn.direction();
let idle_ms = conn.idle_time(now_ms);
if conn.is_failed() {
debug!(
link_id = %link_id,
direction = %direction,
"Failed handshake connection cleaned up"
);
} else {
debug!(
link_id = %link_id,
direction = %direction,
idle_secs = idle_ms / 1000,
"Stale handshake connection timed out"
);
}
// Schedule retry for failed outbound auto-connect peers
if conn.is_outbound()
&& let Some(identity) = conn.expected_identity()
{
self.schedule_retry(*identity.node_addr(), now_ms);
let stale = self.stale_connections(now_ms, timeout_ms);
for action in self.fmp.poll_timeouts(stale) {
match action {
ConnAction::ScheduleRetry { peer } => self.note_handshake_timeout(peer, now_ms),
ConnAction::Teardown { link } => {
// Log before cleanup (needs live connection state). The
// failure signal is now read from the control machine; the
// leg still carries direction/idle for the log fields.
let is_failed = self
.peer_machines
.get(&link)
.is_some_and(|machine| machine.is_failed());
if let Some(machine) = self
.peer_machines
.get(&link)
.filter(|machine| machine.leg().is_some())
{
let direction = machine.conn_direction();
if is_failed {
debug!(
link_id = %link,
direction = %direction,
"Failed handshake connection cleaned up"
);
} else {
debug!(
link_id = %link,
direction = %direction,
idle_secs =
now_ms.saturating_sub(self.connection_last_activity(link)) / 1000,
"Stale handshake connection timed out"
);
}
}
self.cleanup_stale_connection(link, now_ms);
}
#[allow(unreachable_patterns)]
_ => {}
}
self.cleanup_stale_connection(link_id, now_ms);
}
}
@@ -63,15 +113,32 @@ impl Node {
/// the link and address mapping. Does not log — callers provide context-appropriate
/// log messages.
pub(in crate::node) fn cleanup_stale_connection(&mut self, link_id: LinkId, _now_ms: u64) {
let conn = match self.connections.remove(&link_id) {
// Take the connection off its machine BEFORE disposing the machine
// (the machine owns it), keeping it readable for the index/link
// cleanup below. The machine shares the connection's `link_id` and
// lifetime; dropping it here means a reaped handshake leg leaves no
// dangling machine. A no-op for promoted peers — `promote_connection`
// already consumed their connection, so this reaper never runs for
// them.
let _detached_leg = match self
.peer_machines
.get_mut(&link_id)
.and_then(|machine| machine.take_leg())
{
Some(c) => c,
None => return,
};
let transport_id = conn.transport_id();
// Read the transport ID and session index off the surviving carrier
// before disposing the machine (the leg no longer projects them).
let (transport_id, our_index) = match self.peer_machines.get(&link_id) {
Some(machine) => (machine.conn_transport_id(), machine.our_index()),
None => (None, None),
};
self.remove_peer_machine(link_id);
// Free session index and pending_outbound if allocated
if let Some(idx) = conn.our_index() {
if let Some(tid) = conn.transport_id() {
if let Some(idx) = our_index {
if let Some(tid) = transport_id {
self.pending_outbound.remove(&(tid, idx.as_u32()));
}
let _ = self.index_allocator.free(idx);
@@ -84,53 +151,196 @@ impl Node {
}
}
/// Resend handshake messages for pending connections.
/// Act on the per-peer machine timers this tick.
///
/// For outbound connections in SentMsg1 state, resends the stored msg1
/// with exponential backoff. Called periodically from the RX event loop.
pub(in crate::node) async fn resend_pending_handshakes(&mut self, now_ms: u64) {
if self.connections.is_empty() {
/// The sans-IO machine arms `SetTimer`/`CancelTimer` actions into
/// [`peer_timers`](Node::peer_timers); this is the shell driver that acts on
/// them: timeout reaps idle-timed-out outbound legs, retransmit resends the
/// due msg1s. Handshake-TIMEOUT is driven before handshake-RETRANSMIT so a
/// timed-out leg is reaped rather than resent on the same tick. The
/// rekey/liveness kinds keep their own shell drivers, so only the two
/// handshake kinds are driven here.
pub(in crate::node) async fn drive_peer_timers(&mut self, now_ms: u64) {
if self.peer_timers.is_empty() {
return;
}
self.drive_handshake_timeouts(now_ms);
self.drive_handshake_retransmits(now_ms).await;
}
/// Reap the outbound legs whose machine `HandshakeTimeout` timer marks them
/// as machine-timeout-owned and which have idle-timed-out this tick.
///
/// The timer's PRESENCE selects the leg (only OUTBOUND legs arm one — IK
/// inbound arms none); the reap THRESHOLD is the shell `is_timed_out(now,
/// config)` predicate, NOT the timer's stored deadline. This matters because
/// the machine arms the timer from a hardcoded constant at dial, which is not
/// authoritative for an operator-tuned `handshake_timeout_secs` — reading the
/// threshold from config each tick keeps the reap neutral for any config, and
/// off the `last_activity` clock exactly as the old `check_timeouts` did. A
/// timed-out leg is reaped by the old Teardown path: the outbound retry
/// reflex, then `cleanup_stale_connection` (which drops the machine + timers).
///
/// `check_timeouts` keeps reaping everything else — `is_failed()` legs and the
/// idle-timeout of legs without a machine timer (inbound legs).
fn drive_handshake_timeouts(&mut self, now_ms: u64) {
let timeout_ms = self.config().node.rate_limit.handshake_timeout_secs * 1000;
let timer_links: Vec<LinkId> = self
.peer_timers
.iter()
.filter(|(_, timers)| timers.contains_key(&TimerKind::HandshakeTimeout))
.map(|(link, _)| *link)
.collect();
for link in timer_links {
// The idle-timeout threshold reads the survivor carrier's
// last-activity; presence of a pending handshake is what decides
// between reaping and dropping an orphan timer.
let timed_out = self
.peer_machines
.get(&link)
.is_some_and(|machine| machine.conn_is_timed_out(now_ms, timeout_ms));
let (reap, retry_peer) = match self.has_pending_leg(&link) {
true if timed_out => {
let retry_peer = if self
.peer_machines
.get(&link)
.is_some_and(|machine| machine.conn_is_outbound())
{
self.peer_machines
.get(&link)
.and_then(|machine| machine.conn_expected_identity())
.map(|id| *id.node_addr())
} else {
None
};
(true, retry_peer)
}
// Not yet idle-timed-out: leave the timer for a later tick.
true => (false, None),
false => {
// Orphan timer (connection already reaped elsewhere) — drop it.
if let Some(timers) = self.peer_timers.get_mut(&link) {
timers.remove(&TimerKind::HandshakeTimeout);
}
(false, None)
}
};
if reap {
if let Some(peer) = retry_peer {
self.note_handshake_timeout(peer, now_ms);
}
debug!(link_id = %link, "Handshake connection timed out");
self.cleanup_stale_connection(link, now_ms);
}
}
}
/// Fire due handshake-retransmit timers: resend the stored msg1.
///
/// The pre-fold `resend_pending_handshakes` logic, re-homed: the *due* signal
/// is the machine-armed timer (not the connection's `next_resend_at_ms`), and
/// the resend counter lives on the machine (the operator-visible count reads
/// from there). The wire bytes and transport target still come from the shell
/// connection, the pure core computes the backoff schedule, and — matching the
/// old shell exactly — the count and reschedule advance only on a successful
/// send; a failed send neither advances the count nor marks the connection
/// failed, it just retries next tick.
async fn drive_handshake_retransmits(&mut self, now_ms: u64) {
let max_resends = self.config().node.rate_limit.handshake_max_resends;
let interval_ms = self.config().node.rate_limit.handshake_resend_interval_ms;
let backoff = self.config().node.rate_limit.handshake_resend_backoff;
// Collect resend candidates: outbound, in SentMsg1, with stored msg1,
// under max resends, and past the scheduled time.
let candidates: Vec<(LinkId, Vec<u8>)> = self
.connections
// Collect due retransmit timers (kind-filtered).
let due: Vec<LinkId> = self
.peer_timers
.iter()
.filter(|(_, conn)| {
conn.is_outbound()
&& conn.handshake_state() == HandshakeState::SentMsg1
&& conn.resend_count() < max_resends
&& conn.next_resend_at_ms() > 0
&& now_ms >= conn.next_resend_at_ms()
})
.filter_map(|(link_id, conn)| {
conn.handshake_msg1().map(|msg1| (*link_id, msg1.to_vec()))
.filter(|(_, timers)| {
timers
.get(&TimerKind::HandshakeRetransmit)
.is_some_and(|&at_ms| now_ms >= at_ms)
})
.map(|(link, _)| *link)
.collect();
if due.is_empty() {
return;
}
for (link_id, msg1_bytes) in candidates {
// Get transport and address info from the connection
let (transport_id, remote_addr) = match self.connections.get(&link_id) {
Some(conn) => match (conn.transport_id(), conn.source_addr()) {
(Some(tid), Some(addr)) => (tid, addr.clone()),
_ => continue,
},
// Classify each due link against the machine + connection. A timer whose
// machine has left `SentMsg1` (promoted/gone) or has hit the resend cap
// is dropped — no more resends, exactly as the old shell stopped
// selecting a capped/settled connection; the handshake-timeout reaper
// takes it from there.
let mut candidates: Vec<ConnSnapshot> = Vec::new();
let mut drop_timers: Vec<LinkId> = Vec::new();
for link in due {
let armed = match self.peer_machines.get(&link) {
Some(machine)
if machine.is_handshaking_sent_msg1()
&& machine.resend_count() < max_resends =>
{
machine.resend_count()
}
Some(_) => {
drop_timers.push(link);
continue;
}
None => {
drop_timers.push(link);
continue;
}
};
match self
.peer_machines
.get(&link)
.and_then(|machine| machine.conn_handshake_msg1())
{
// Armed but the stored wire isn't there yet — leave the timer and
// retry next tick (matches the old candidate filter skipping it).
None => continue,
Some(msg1) => candidates.push(ConnSnapshot {
link,
is_outbound: true,
retry_addr: None,
resend_count: armed,
msg1: msg1.to_vec(),
}),
}
}
for link in drop_timers {
if let Some(timers) = self.peer_timers.get_mut(&link) {
timers.remove(&TimerKind::HandshakeRetransmit);
}
}
for action in self
.fmp
.poll_resends(candidates, now_ms, interval_ms, backoff)
{
let ConnAction::ResendMsg1 {
link,
bytes,
next_resend_at_ms,
} = action
else {
continue;
};
let (transport_id, remote_addr) = match self.peer_machines.get(&link) {
Some(machine) if machine.leg().is_some() => {
match (machine.conn_transport_id(), machine.conn_source_addr()) {
(Some(tid), Some(addr)) => (tid, addr.clone()),
_ => continue,
}
}
_ => continue,
};
// Send the stored msg1
let sent = if let Some(transport) = self.transports.get(&transport_id) {
match transport.send(&remote_addr, &msg1_bytes).await {
match transport.send(&remote_addr, &bytes).await {
Ok(_) => true,
Err(e) => {
debug!(
link_id = %link_id,
link_id = %link,
error = %e,
"Handshake msg1 resend failed"
);
@@ -141,15 +351,26 @@ impl Node {
false
};
if sent && let Some(conn) = self.connections.get_mut(&link_id) {
let count = conn.resend_count() + 1;
let next = now_ms + (interval_ms as f64 * backoff.powi(count as i32)) as u64;
conn.record_resend(next);
debug!(
link_id = %link_id,
resend = count,
"Resent handshake msg1"
);
if sent {
if let Some(machine) = self.peer_machines.get_mut(&link) {
machine.record_resend(next_resend_at_ms);
debug!(
link_id = %link,
resend = machine.resend_count(),
"Resent handshake msg1"
);
}
self.peer_timers
.entry(link)
.or_default()
.insert(TimerKind::HandshakeRetransmit, next_resend_at_ms);
} else {
// Failed send: keep retrying at the tick cadence (the old shell
// left next_resend_at_ms unchanged so the connection stayed due).
self.peer_timers
.entry(link)
.or_default()
.insert(TimerKind::HandshakeRetransmit, now_ms);
}
}
}
@@ -204,7 +425,7 @@ impl Node {
.collect();
for (dest_addr, payload) in candidates {
use crate::protocol::SessionDatagram;
use crate::proto::link::SessionDatagram;
let mut datagram = SessionDatagram::new(my_addr, dest_addr, payload).with_ttl(ttl);
let sent = match self.send_session_datagram(&mut datagram).await {
src/node/lifecycle.rs
-2761
View File
File diff suppressed because it is too large Load Diff
src/node/lifecycle/mod.rs
+3321
View File
File diff suppressed because it is too large Load Diff
src/node/lifecycle/supervisor.rs
+1364
View File
File diff suppressed because it is too large Load Diff
src/node/metrics.rs
+15 -48
View File
@@ -15,8 +15,8 @@ use std::sync::atomic::{AtomicU64, Ordering};
use crate::node::reject::{BloomReject, DiscoveryReject, ForwardingReject, TreeReject};
use crate::node::stats::{
BloomStatsSnapshot, CongestionStatsSnapshot, DiscoveryStatsSnapshot, ErrorSignalStatsSnapshot,
ForwardingStatsSnapshot, TreeStatsSnapshot,
BloomStatsSnapshot, CongestionStatsSnapshot, ErrorSignalStatsSnapshot, ForwardingStatsSnapshot,
LookupStatsSnapshot, TreeStatsSnapshot,
};
/// An atomic counter.
@@ -90,43 +90,10 @@ pub struct ForwardingMetrics {
}
/// Route class of a transit-forwarded packet, classified from tree
/// coordinates at the forwarding decision point. The six variants
/// partition `forwarded_packets` exactly.
///
/// Two variants are up-and-over forwards (destination not in the chosen
/// peer's subtree); they differ in whether they depend on a child
/// advertising cross-link reach *upward* to its parent:
/// - `TreeDownCross`: the chosen peer is our tree descendant, but the
/// destination is *not* in that child's subtree. The forward only fired
/// because the child advertised cross-link reach upward to us, beyond its
/// own subtree. If children advertised only their subtree upward, this
/// forward would route up instead, so its count measures how much
/// forwarding depends on the upward cross-link advertisement — the
/// dive-to-tree-child cut-through.
/// - `CrosslinkAscend`: the chosen peer is lateral (neither ancestor nor
/// descendant) and the destination is not in its subtree. This is a node
/// using its *own* cross-link, learned via the peer's split-horizon
/// advertisement to its neighbors, so it does not depend on any upward
/// advertisement. Tracked alongside `TreeDownCross` as the lateral
/// up-and-over contrast.
#[derive(Debug, Clone, Copy, PartialEq, Eq)]
pub enum RouteClass {
/// Chosen peer is our ancestor (tree-up).
TreeUp,
/// Chosen peer is our descendant and dest is in its subtree (canonical
/// tree-down).
TreeDown,
/// Chosen peer is our descendant but dest is *not* in its subtree: the
/// dive-to-tree-child cut-through enabled by upward cross-link
/// advertisement.
TreeDownCross,
/// Chosen peer is lateral and dest is in its subtree (subtree entry).
CrosslinkDescend,
/// Chosen peer is lateral and dest is not in its subtree (up-and-over).
CrosslinkAscend,
/// Chosen peer is the destination itself (degenerate direct hop).
DirectPeer,
}
/// coordinates at the forwarding decision point. Defined by the sans-IO
/// routing core and re-exported here for the forwarding-metrics counters
/// ([`ForwardingMetrics::record_route_class`]).
pub(crate) use crate::proto::routing::RouteClass;
impl ForwardingMetrics {
/// Record a received packet of `bytes` payload (packets and bytes).
@@ -235,7 +202,7 @@ impl ForwardingMetrics {
/// Discovery metric counters.
#[derive(Default)]
pub struct DiscoveryMetrics {
pub struct LookupMetrics {
pub req_received: Padded<Counter>,
pub req_decode_error: Counter,
pub req_duplicate: Counter,
@@ -260,7 +227,7 @@ pub struct DiscoveryMetrics {
pub resp_timed_out: Counter,
}
impl DiscoveryMetrics {
impl LookupMetrics {
/// Mirror of `DiscoveryStats::record_reject`: route a typed discovery
/// rejection to its counter.
#[inline]
@@ -278,8 +245,8 @@ impl DiscoveryMetrics {
}
/// Sample every counter into a serializable snapshot.
pub fn snapshot(&self) -> DiscoveryStatsSnapshot {
DiscoveryStatsSnapshot {
pub fn snapshot(&self) -> LookupStatsSnapshot {
LookupStatsSnapshot {
req_received: self.req_received.get(),
req_decode_error: self.req_decode_error.get(),
req_duplicate: self.req_duplicate.get(),
@@ -460,7 +427,7 @@ impl ErrorMetrics {
#[derive(Default)]
pub struct MetricsRegistry {
pub forwarding: ForwardingMetrics,
pub discovery: DiscoveryMetrics,
pub lookup: LookupMetrics,
pub tree: TreeMetrics,
pub bloom: BloomMetrics,
pub congestion: CongestionMetrics,
@@ -488,7 +455,7 @@ mod tests {
#[test]
fn discovery_record_reject_routes_to_field() {
let m = DiscoveryMetrics::default();
let m = LookupMetrics::default();
m.record_reject(DiscoveryReject::ReqDuplicate);
m.record_reject(DiscoveryReject::ReqDuplicate);
m.record_reject(DiscoveryReject::RespNoRoute);
@@ -499,7 +466,7 @@ mod tests {
#[test]
fn discovery_direct_counters_increment() {
let m = DiscoveryMetrics::default();
let m = LookupMetrics::default();
m.req_received.inc();
m.req_forwarded.inc();
m.req_forwarded.inc();
@@ -532,9 +499,9 @@ mod tests {
fn registry_subcounters_are_independent() {
let r = MetricsRegistry::new();
r.forwarding.record_received(10);
r.discovery.req_received.inc();
r.lookup.req_received.inc();
assert_eq!(r.forwarding.received_packets.get(), 1);
assert_eq!(r.forwarding.received_bytes.get(), 10);
assert_eq!(r.discovery.req_received.get(), 1);
assert_eq!(r.lookup.req_received.get(), 1);
}
}
src/node/mod.rs
+589 -471
View File
File diff suppressed because it is too large Load Diff
src/node/peering/driver.rs
+156
View File
@@ -0,0 +1,156 @@
//! Thin async driver for the peering reconciler.
//!
//! These `impl Node` methods are the I/O edge of the sans-IO
//! [`super::reconcile::PeeringReconciler`]: they snapshot the live dataplane
//! maps into the reconciler's plain-data inputs, invoke the pure core, and
//! perform the dial / advert-refetch I/O each [`PeeringAction`] names. They also
//! host the two gate-guarded reflex wrappers every peer-loss call site routes
//! through, so drain suppression and the connected-guard live in one place.
//!
//! The `Policy` / `Observed` / `Budget` builders these methods consume live in
//! [`crate::node::lifecycle`] next to the surviving budget helpers and limit
//! constants they wrap.
use crate::identity::NodeAddr;
use crate::node::{Node, NodeError};
use tracing::warn;
use super::reconcile::{DiscoveryPools, Gate, PeeringAction};
impl Node {
/// Reflex: an outbound handshake timed out (replaces the old
/// `Node::schedule_retry` call sites).
///
/// Replicates `schedule_retry`'s connected-guard — the pure core cannot
/// observe the peers map, so the driver drops the event when the peer is
/// already connected — then feeds the gate-guarded reconciler reflex with
/// the gate derived from the live published state.
pub(in crate::node) fn note_handshake_timeout(&mut self, node_addr: NodeAddr, now_ms: u64) {
if self.peers.contains_key(&node_addr) {
return;
}
let policy =
self.build_peering_policy(self.config().auto_connect_peers().cloned().collect());
let gate = Gate::from_state(self.supervisor.state);
let _ = self
.peering
.reconciler
.on_handshake_timeout(node_addr, now_ms, &policy, gate);
}
/// Reflex: a link went dead / a peer was lost (replaces the old
/// `Node::schedule_reconnect` call sites).
///
/// No connected-guard — the peer is already gone by the time a link-dead /
/// disconnect event fires (`schedule_reconnect` had none). The gate is
/// derived from the live published state so a drain self-suppresses the
/// reconnect.
pub(in crate::node) fn note_link_dead(&mut self, node_addr: NodeAddr, now_ms: u64) {
let policy =
self.build_peering_policy(self.config().auto_connect_peers().cloned().collect());
let gate = Gate::from_state(self.supervisor.state);
let _ = self
.peering
.reconciler
.on_link_dead(node_addr, now_ms, &policy, gate);
}
/// Process pending retries whose time has arrived (replaces the old
/// `Node::process_pending_retries` body).
///
/// The pure retry-dial phase owns the decision — drop expired entries, refuse
/// to grow when admission binds, dial the first `retry_per_tick` due entries
/// (bumping their `retry_after_ms` past the handshake window). This driver
/// performs the advert-refetch + dial I/O each emitted `Connect` names, and
/// on an immediate dial error feeds the `on_handshake_timeout` reflex so the
/// optimistic re-fire suppression is overwritten by proper backoff. During a
/// drain the gate is `Suspended`, so the reconcile clears the schedule and
/// emits nothing.
pub(in crate::node) async fn process_pending_retries(&mut self, now_ms: u64) {
if self.peering.reconciler.retry_pending.is_empty() {
return;
}
// Retry-dial cadence slot: empty config floor and empty discovery
// pools, so only the retry-dial phase acts.
let policy = self.build_peering_policy(Vec::new());
let observed = self.observe_peering();
let budget = self.build_peering_budget();
let gate = Gate::from_state(self.supervisor.state);
let actions = self.peering.reconciler.reconcile(
&policy,
&observed,
&budget,
&DiscoveryPools::default(),
now_ms,
gate,
);
for action in actions {
let PeeringAction::Connect(candidate) = action else {
continue;
};
let Some(identity) = candidate.identity else {
continue;
};
let node_addr = *identity.node_addr();
let Some(peer_config) = self
.peering
.reconciler
.retry_pending
.get(&node_addr)
.map(|state| state.peer_config.clone())
else {
continue;
};
// Kick off a refresh of the peer's overlay advert. The cache is
// read-only on hit, so a retry without a refetch dials the same
// cached endpoint — and the most common reason a peer landed in the
// retry schedule is that endpoint just stopped working (NAT rebind,
// port change, peer restart).
//
// Fire-and-forget, NOT awaited: this runs inline on the 1s rx-loop
// tick, and the fetch carries a 2s relay timeout that would stall
// the tick — and every other rx-loop arm with it — by up to 2s per
// due peer. So the dial below uses whatever advert is cached now
// and the refreshed one lands for the *next* retry of this peer.
// Retries are backoff-paced, so that defers the benefit by one
// backoff interval rather than losing it.
if let Some(bootstrap) = self.supervisor.nostr_rendezvous.engine_arc() {
let npub = peer_config.npub.clone();
tokio::spawn(async move {
let _ = bootstrap.refetch_advert_for_stale_check(&npub).await;
});
}
match self.initiate_peer_connection(&peer_config).await {
// The core already pushed `retry_after_ms` past the handshake
// window; a successful promotion clears the entry, a later
// timeout re-fires the reflex with proper backoff.
Ok(()) => {}
Err(e) => {
warn!(
peer = %self.peer_display_name(&node_addr),
error = %e,
"Retry connection initiation failed"
);
// No-transport failures usually mean the cached overlay
// advert is stale; force a re-fetch so the next tick picks up
// fresh endpoints.
if matches!(e, NodeError::NoTransportForType(_))
&& let Some(bootstrap) = self.supervisor.nostr_rendezvous.engine_arc()
{
let npub = peer_config.npub.clone();
tokio::spawn(async move {
let _ = bootstrap.refetch_advert_for_stale_check(&npub).await;
});
}
// Immediate failure counts as an attempt: overwrite the
// optimistic re-fire suppression with backoff.
self.note_handshake_timeout(node_addr, now_ms);
}
}
}
}
}
src/node/peering/mod.rs
+14
View File
@@ -0,0 +1,14 @@
//! Peering homeostasis — the desired-state controller for the node's peer set.
//!
//! This module is the home for the peering-reconciler concept: config defines a
//! desired peer set; the reconciler converges the observed set toward it
//! (auto-connect floor, overlay pool, transport-neighbor growth) under the
//! `node.limits` ceiling. Startup and steady-state are the same loop.
//!
//! The cross-attempt retry schedule (`retry.rs`) lives here because a fresh
//! connection is created per re-dial, so the escalating backoff count must
//! persist in the reconciler, not per-connection.
pub(in crate::node) mod driver;
pub(in crate::node) mod reconcile;
pub(in crate::node) mod retry;
src/node/peering/reconcile.rs
+1447
View File
File diff suppressed because it is too large Load Diff
src/node/peering/retry.rs
+49
View File
@@ -0,0 +1,49 @@
//! Cross-attempt retry state for auto-connect peers.
//!
//! [`RetryState`] is the durable per-peer schedule entry the peering reconciler
//! owns (it lives in [`crate::node::peering::reconcile::PeeringReconciler`], not
//! on a per-connection object, because a fresh connection is created per re-dial
//! so the escalating backoff count must persist across attempts). The decision
//! logic that reads and mutates it — the retry-dial phase and the
//! `on_handshake_timeout` / `on_link_dead` reflexes — lives in the sans-IO
//! reconciler core; the driver wrappers that feed it (retry-dial I/O, the
//! gate-guarded reflex call sites) live in [`super::driver`].
use crate::config::PeerConfig;
/// Per-tick cap on retry-dial connection attempts (ceiling).
pub(in crate::node) const MAX_RETRY_CONNECTIONS_PER_TICK: usize = 16;
/// Tracks retry state for a peer across connection attempts.
pub struct RetryState {
/// The peer config to use for initiating retries.
pub peer_config: PeerConfig,
/// Number of retries attempted so far.
pub retry_count: u32,
/// Timestamp (Unix ms) when the next retry should be attempted.
pub retry_after_ms: u64,
/// Whether this is an auto-reconnect (unlimited retries, ignores max_retries).
pub reconnect: bool,
/// Optional absolute expiry for this retry entry (Unix ms).
///
/// When set, retries are dropped after this point even if reconnect logic
/// would otherwise continue.
pub expires_at_ms: Option<u64>,
}
impl RetryState {
/// Create a new retry state for a peer.
pub fn new(peer_config: PeerConfig) -> Self {
Self {
peer_config,
retry_count: 0,
retry_after_ms: 0,
reconnect: false,
expires_at_ms: None,
}
}
}
src/node/reject.rs
+1 -1
View File
@@ -222,7 +222,7 @@ pub enum MmpReject {
/// Forwarding-path rejection reasons.
///
/// Each variant corresponds to a silent-rejection path in
/// `src/node/handlers/forwarding.rs::handle_session_datagram`. Matching
/// `src/node/dataplane/forwarding.rs::handle_session_datagram`. Matching
/// `ForwardingStats` counters already track packets and bytes for each
/// outcome; `record_reject` mirrors the packet-count side of the bump
/// for parity with the other rejection clusters.
src/node/reloadable.rs
+1 -1
View File
@@ -41,7 +41,7 @@
//! file. There is nothing to poll. (Its read side could adopt the same
//! lock-free `ArcSwap` shape in the future, but that is an optimization, not
//! a reload.)
//! - `nostr_discovery` is an async spawned subsystem, not a snapshot of disk
//! - `nostr_rendezvous` is an async spawned subsystem, not a snapshot of disk
//! state.
//!
//! Both [`HostMapReloadable`] and the peer ACL reloader currently stat
src/node/retry.rs
-425
View File
@@ -1,425 +0,0 @@
//! Connection retry logic for auto-connect peers.
//!
//! When an outbound handshake fails (timeout or send error), the node can
//! automatically retry with exponential backoff. Retry state lives on Node
//! (not PeerConnection) because each retry creates a fresh connection.
use super::{Node, NodeError};
use crate::PeerIdentity;
use crate::config::PeerConfig;
use crate::identity::NodeAddr;
use tracing::{debug, info, warn};
// MAX_BACKOFF_MS is now derived from config: node.retry.max_backoff_secs * 1000
const MAX_RETRY_CONNECTIONS_PER_TICK: usize = 16;
/// Tracks retry state for a peer across connection attempts.
pub struct RetryState {
/// The peer config to use for initiating retries.
pub peer_config: PeerConfig,
/// Number of retries attempted so far.
pub retry_count: u32,
/// Timestamp (Unix ms) when the next retry should be attempted.
pub retry_after_ms: u64,
/// Whether this is an auto-reconnect (unlimited retries, ignores max_retries).
pub reconnect: bool,
/// Optional absolute expiry for this retry entry (Unix ms).
///
/// When set, retries are dropped after this point even if reconnect logic
/// would otherwise continue.
pub expires_at_ms: Option<u64>,
}
impl RetryState {
/// Create a new retry state for a peer.
pub fn new(peer_config: PeerConfig) -> Self {
Self {
peer_config,
retry_count: 0,
retry_after_ms: 0,
reconnect: false,
expires_at_ms: None,
}
}
/// Calculate the backoff delay in milliseconds for the current retry count.
///
/// Uses exponential backoff: `base_interval_ms * 2^retry_count`,
/// capped at `MAX_BACKOFF_MS`.
pub fn backoff_ms(&self, base_interval_ms: u64, max_backoff_ms: u64) -> u64 {
let multiplier = 1u64.checked_shl(self.retry_count).unwrap_or(u64::MAX);
base_interval_ms
.saturating_mul(multiplier)
.min(max_backoff_ms)
}
}
impl Node {
/// Schedule a retry for a failed outbound connection, if applicable.
///
/// Only schedules if the peer is an auto-connect peer and max retries
/// have not been exhausted (unless `reconnect` is true, which retries
/// indefinitely). Does nothing if the peer is already connected or has
/// a connection in progress.
pub(super) fn schedule_retry(&mut self, node_addr: NodeAddr, now_ms: u64) {
let retry_cfg = &self.config().node.retry;
let max_retries = retry_cfg.max_retries;
if max_retries == 0 {
return;
}
// Don't retry if peer is already connected
if self.peers.contains_key(&node_addr) {
return;
}
let base_interval_ms = retry_cfg.base_interval_secs * 1000;
let max_backoff_ms = retry_cfg.max_backoff_secs * 1000;
let peer_name = self.peer_display_name(&node_addr);
if let Some(state) = self.retry_pending.get_mut(&node_addr) {
// Already tracking — increment
state.retry_count += 1;
if !state.reconnect && state.retry_count > max_retries {
info!(
peer = %peer_name,
attempts = state.retry_count,
"Max retries exhausted, giving up on peer"
);
self.retry_pending.remove(&node_addr);
return;
}
let delay = state.backoff_ms(base_interval_ms, max_backoff_ms);
state.retry_after_ms = now_ms + delay;
debug!(
peer = %peer_name,
retry = state.retry_count,
reconnect = state.reconnect,
delay_secs = delay / 1000,
"Scheduling connection retry"
);
} else {
// First failure — find the matching PeerConfig
let peer_config = self
.config()
.auto_connect_peers()
.find(|pc| {
PeerIdentity::from_npub(&pc.npub)
.map(|id| *id.node_addr() == node_addr)
.unwrap_or(false)
})
.cloned();
if let Some(pc) = peer_config {
let mut state = RetryState::new(pc);
state.retry_count = 1;
state.reconnect = true;
let delay = state.backoff_ms(base_interval_ms, max_backoff_ms);
state.retry_after_ms = now_ms + delay;
debug!(
peer = %self.peer_display_name(&node_addr),
delay_secs = delay / 1000,
"First connection attempt failed, scheduling retry"
);
self.retry_pending.insert(node_addr, state);
}
// If not found in auto_connect_peers, no retry (one-shot connection)
}
}
/// Schedule auto-reconnect for a peer removed by MMP dead timeout.
///
/// Looks up the peer in auto-connect config and checks `auto_reconnect`.
/// If enabled, feeds the peer into the retry system with unlimited retries.
///
/// If a retry entry already exists (e.g. from a previous failed handshake
/// attempt during an earlier reconnect cycle), the existing retry count is
/// preserved and incremented rather than reset to zero. This ensures
/// exponential backoff accumulates across repeated link-dead events instead
/// of resetting to the base interval on every peer removal.
pub(super) fn schedule_reconnect(&mut self, node_addr: NodeAddr, now_ms: u64) {
// Find peer in auto-connect config
let peer_config = self
.config()
.auto_connect_peers()
.find(|pc| {
PeerIdentity::from_npub(&pc.npub)
.map(|id| *id.node_addr() == node_addr)
.unwrap_or(false)
})
.cloned();
let Some(pc) = peer_config else {
return; // Not an auto-connect peer, no reconnect
};
if !pc.auto_reconnect {
debug!(
peer = %self.peer_display_name(&node_addr),
"Auto-reconnect disabled for peer, skipping"
);
return;
}
let base_interval_ms = self.config().node.retry.base_interval_secs * 1000;
let max_backoff_ms = self.config().node.retry.max_backoff_secs * 1000;
let peer_name = self.peer_display_name(&node_addr);
// If we already have accumulated backoff from previous failed attempts,
// preserve and bump it rather than resetting to zero. This prevents the
// exponential backoff from being discarded on each link-dead cycle.
if let Some(state) = self.retry_pending.get_mut(&node_addr) {
state.reconnect = true;
state.retry_count += 1;
let delay = state.backoff_ms(base_interval_ms, max_backoff_ms);
state.retry_after_ms = now_ms + delay;
debug!(
peer = %peer_name,
retry = state.retry_count,
delay_secs = delay / 1000,
"Scheduling auto-reconnect after link-dead removal (backoff preserved)"
);
return;
}
let mut state = RetryState::new(pc);
state.reconnect = true;
let delay = state.backoff_ms(base_interval_ms, max_backoff_ms);
state.retry_after_ms = now_ms + delay;
debug!(
peer = %peer_name,
delay_secs = delay / 1000,
"Scheduling auto-reconnect after link-dead removal"
);
self.retry_pending.insert(node_addr, state);
}
/// Process pending retries whose time has arrived.
///
/// For each due retry, initiates a fresh connection attempt. The retry
/// entry stays in `retry_pending` until the connection succeeds (cleared
/// in `promote_connection`) or max retries are exhausted (cleared in
/// `schedule_retry`).
pub(super) async fn process_pending_retries(&mut self, now_ms: u64) {
if self.retry_pending.is_empty() {
return;
}
let expired: Vec<NodeAddr> = self
.retry_pending
.iter()
.filter_map(|(addr, state)| {
state
.expires_at_ms
.filter(|expires_at_ms| now_ms >= *expires_at_ms)
.map(|_| *addr)
})
.collect();
for node_addr in expired {
self.retry_pending.remove(&node_addr);
info!(
peer = %self.peer_display_name(&node_addr),
"Retry window expired, dropping pending retry state"
);
}
if self.retry_pending.is_empty() {
return;
}
if !self.outbound_admission_check() {
debug!(
peers = self.peers.len(),
max_peers = self.max_peers(),
retry_pending = self.retry_pending.len(),
"Suppressing auto-reconnect retries: at capacity"
);
return;
}
// Collect retries that are due
let due: Vec<NodeAddr> = self
.retry_pending
.iter()
.filter(|(_, state)| now_ms >= state.retry_after_ms)
.map(|(addr, _)| *addr)
.collect();
let deferred = due.len().saturating_sub(MAX_RETRY_CONNECTIONS_PER_TICK);
if deferred > 0 {
debug!(
due = due.len(),
processing = MAX_RETRY_CONNECTIONS_PER_TICK,
deferred,
"Retry processing budget exhausted; deferring remaining peers"
);
}
for node_addr in due.into_iter().take(MAX_RETRY_CONNECTIONS_PER_TICK) {
// Peer may have connected inbound while we waited
if self.peers.contains_key(&node_addr) {
self.retry_pending.remove(&node_addr);
continue;
}
let state = match self.retry_pending.get(&node_addr) {
Some(s) => s,
None => continue,
};
debug!(
peer = %self.peer_display_name(&node_addr),
retry = state.retry_count,
"Attempting connection retry"
);
let peer_config = state.peer_config.clone();
// Kick off a refresh of the peer's overlay advert. The cache is
// read-only on hit (see fetch_advert), so every retry without a
// refetch dials the same cached endpoint — and the most common
// reason a peer ended up in retry_pending is that the cached
// endpoint just stopped working (NAT rebind, port change, peer
// restart on a different port). Without this refresh the retry
// loop dials the same dead address forever.
//
// refetch_advert_for_stale_check uses the relay's advert as
// ground truth: replaces the cache if there's a newer one,
// evicts if the relay has nothing, otherwise leaves it.
//
// Fire-and-forget, NOT awaited: this runs inline on the 1s
// rx-loop tick, and the fetch carries a 2s relay timeout that
// would stall the tick — and every other rx-loop arm with it —
// by up to 2s per due peer, MAX_RETRY_CONNECTIONS_PER_TICK times
// over. So the dial below uses whatever advert is cached now and
// the refreshed one lands for the *next* retry of this peer.
// Retries are backoff-paced, so that defers the benefit by one
// backoff interval rather than losing it.
if let Some(bootstrap) = self.nostr_discovery.clone() {
let npub = peer_config.npub.clone();
tokio::spawn(async move {
let _ = bootstrap.refetch_advert_for_stale_check(&npub).await;
});
}
match self.initiate_peer_connection(&peer_config).await {
Ok(()) => {
// Push retry_after_ms past the handshake timeout window so
// we don't re-fire on the next tick. If the handshake
// succeeds, promote_connection() clears retry_pending. If
// it times out, check_timeouts() calls schedule_retry()
// which bumps the counter and applies proper backoff.
let hs_timeout_ms = self.config().node.rate_limit.handshake_timeout_secs * 1000;
if let Some(state) = self.retry_pending.get_mut(&node_addr) {
state.retry_after_ms = now_ms + hs_timeout_ms;
}
debug!(
peer = %self.peer_display_name(&node_addr),
"Retry connection initiated, suppressing re-fire for {}s",
self.config().node.rate_limit.handshake_timeout_secs,
);
}
Err(e) => {
warn!(
peer = %self.peer_display_name(&node_addr),
error = %e,
"Retry connection initiation failed"
);
// No-transport failures usually mean the cached overlay
// advert is stale (peer rebound NAT, switched relay, etc.).
// The advert cache is read-only inside fetch_advert, so
// every retry returns the same dead address until the
// entry expires. Force a re-fetch so the next retry tick
// picks up fresh endpoints.
if matches!(e, NodeError::NoTransportForType(_))
&& let Some(bootstrap) = self.nostr_discovery.clone()
{
let npub = peer_config.npub.clone();
tokio::spawn(async move {
let _ = bootstrap.refetch_advert_for_stale_check(&npub).await;
});
}
// Immediate failure counts as an attempt — schedule next retry
// (reconnect flag is preserved on existing retry_pending entry)
self.schedule_retry(node_addr, now_ms);
}
}
}
}
}
#[cfg(test)]
mod tests {
use super::*;
use crate::config::PeerConfig;
const TEST_MAX_BACKOFF_MS: u64 = 300_000;
#[test]
fn test_backoff_exponential() {
let state = RetryState {
peer_config: PeerConfig::default(),
retry_count: 0,
retry_after_ms: 0,
reconnect: false,
expires_at_ms: None,
};
// base = 5000ms
assert_eq!(state.backoff_ms(5000, TEST_MAX_BACKOFF_MS), 5000); // 5s * 2^0
let state = RetryState {
retry_count: 1,
..state
};
assert_eq!(state.backoff_ms(5000, TEST_MAX_BACKOFF_MS), 10_000); // 5s * 2^1
let state = RetryState {
retry_count: 2,
..state
};
assert_eq!(state.backoff_ms(5000, TEST_MAX_BACKOFF_MS), 20_000); // 5s * 2^2
let state = RetryState {
retry_count: 3,
..state
};
assert_eq!(state.backoff_ms(5000, TEST_MAX_BACKOFF_MS), 40_000); // 5s * 2^3
let state = RetryState {
retry_count: 4,
..state
};
assert_eq!(state.backoff_ms(5000, TEST_MAX_BACKOFF_MS), 80_000); // 5s * 2^4
}
#[test]
fn test_backoff_cap() {
let state = RetryState {
peer_config: PeerConfig::default(),
retry_count: 20, // 2^20 * 5000 would be huge
retry_after_ms: 0,
reconnect: false,
expires_at_ms: None,
};
assert_eq!(
state.backoff_ms(5000, TEST_MAX_BACKOFF_MS),
TEST_MAX_BACKOFF_MS
);
}
#[test]
fn test_backoff_zero_base() {
let state = RetryState {
peer_config: PeerConfig::default(),
retry_count: 3,
retry_after_ms: 0,
reconnect: false,
expires_at_ms: None,
};
assert_eq!(state.backoff_ms(0, TEST_MAX_BACKOFF_MS), 0);
}
}
src/node/routing_error_rate_limit.rs
-161
View File
@@ -1,161 +0,0 @@
//! Routing error signal rate limiting.
//!
//! Prevents routing error floods (CoordsRequired / PathBroken) by
//! rate-limiting error signals per destination address at transit nodes.
use crate::NodeAddr;
use std::collections::HashMap;
use std::time::{Duration, Instant};
/// Rate limiter for routing error signals (CoordsRequired / PathBroken).
///
/// Tracks the last time a routing error was sent for each destination
/// address and enforces a minimum interval to prevent floods.
pub struct RoutingErrorRateLimiter {
/// Maps destination NodeAddr to the last time we sent an error about it.
last_sent: HashMap<NodeAddr, Instant>,
/// Minimum interval between error signals for the same destination.
min_interval: Duration,
/// Maximum age of entries before cleanup.
max_age: Duration,
}
impl RoutingErrorRateLimiter {
/// Create a new rate limiter.
///
/// Default: max 10 errors/sec per destination (100ms interval).
pub fn new() -> Self {
Self {
last_sent: HashMap::new(),
min_interval: Duration::from_millis(100),
max_age: Duration::from_secs(10),
}
}
/// Create a rate limiter with a custom minimum interval.
pub fn with_interval(min_interval: Duration) -> Self {
Self {
last_sent: HashMap::new(),
min_interval,
max_age: Duration::from_secs(10),
}
}
/// Check if we should send a routing error for this destination.
///
/// Returns true if enough time has passed since the last error for
/// this destination, or if this is the first error. Updates internal
/// state when returning true.
pub fn should_send(&mut self, dest_addr: &NodeAddr) -> bool {
let now = Instant::now();
if let Some(&last) = self.last_sent.get(dest_addr)
&& now.duration_since(last) < self.min_interval
{
return false;
}
self.last_sent.insert(*dest_addr, now);
self.cleanup(now);
true
}
/// Remove entries older than max_age.
fn cleanup(&mut self, now: Instant) {
self.last_sent
.retain(|_, &mut last| now.duration_since(last) < self.max_age);
}
#[cfg(test)]
pub fn len(&self) -> usize {
self.last_sent.len()
}
}
impl Default for RoutingErrorRateLimiter {
fn default() -> Self {
Self::new()
}
}
#[cfg(test)]
mod tests {
use super::*;
use std::thread;
fn addr(val: u8) -> NodeAddr {
let mut bytes = [0u8; 16];
bytes[0] = val;
NodeAddr::from_bytes(bytes)
}
#[test]
fn test_first_send_allowed() {
let mut limiter = RoutingErrorRateLimiter::new();
assert!(limiter.should_send(&addr(1)));
}
#[test]
fn test_rapid_sends_rate_limited() {
let mut limiter = RoutingErrorRateLimiter::new();
assert!(limiter.should_send(&addr(1)));
assert!(!limiter.should_send(&addr(1)));
assert!(!limiter.should_send(&addr(1)));
}
#[test]
fn test_different_destinations_independent() {
let mut limiter = RoutingErrorRateLimiter::new();
assert!(limiter.should_send(&addr(1)));
assert!(limiter.should_send(&addr(2)));
assert!(!limiter.should_send(&addr(1)));
assert!(!limiter.should_send(&addr(2)));
}
#[test]
fn test_send_allowed_after_interval() {
let mut limiter = RoutingErrorRateLimiter::new();
assert!(limiter.should_send(&addr(1)));
thread::sleep(Duration::from_millis(110));
assert!(limiter.should_send(&addr(1)));
}
#[test]
fn test_cleanup_removes_old_entries() {
let mut limiter = RoutingErrorRateLimiter::new();
assert!(limiter.should_send(&addr(1)));
assert!(limiter.should_send(&addr(2)));
assert_eq!(limiter.len(), 2);
let future = Instant::now() + Duration::from_secs(11);
limiter.cleanup(future);
assert_eq!(limiter.len(), 0);
}
#[test]
fn test_cleanup_preserves_recent_entries() {
let mut limiter = RoutingErrorRateLimiter::new();
assert!(limiter.should_send(&addr(1)));
assert_eq!(limiter.len(), 1);
limiter.cleanup(Instant::now());
assert_eq!(limiter.len(), 1);
}
#[test]
fn test_with_interval_custom_rate() {
let mut limiter = RoutingErrorRateLimiter::with_interval(Duration::from_millis(500));
assert!(limiter.should_send(&addr(1)));
assert!(!limiter.should_send(&addr(1)));
// Still rate-limited after 200ms (would pass with default 100ms)
thread::sleep(Duration::from_millis(200));
assert!(!limiter.should_send(&addr(1)));
// Allowed after 500ms total
thread::sleep(Duration::from_millis(350));
assert!(limiter.should_send(&addr(1)));
}
}
src/node/session.rs → src/node/session/mod.rs
+139 -7
View File
@@ -5,13 +5,11 @@
//! (SessionSetup/SessionAck/SessionMsg3) carried inside SessionDatagram
//! envelopes through the mesh.
use std::time::Instant;
use crate::NodeAddr;
use crate::config::SessionMmpConfig;
use crate::mmp::MmpSessionState;
use crate::node::REKEY_JITTER_SECS;
use crate::noise::{HandshakeState, NoiseSession};
use crate::proto::mmp::MmpSessionState;
use rand::RngExt;
use secp256k1::PublicKey;
@@ -338,7 +336,12 @@ impl SessionEntry {
/// Initialize session-layer MMP state (called on Established transition).
pub(crate) fn init_mmp(&mut self, config: &SessionMmpConfig) {
self.mmp = Some(MmpSessionState::new(config, self.is_initiator));
self.mmp = Some(MmpSessionState::new(
config.mode,
config.log_interval_secs,
config.owd_window_size,
self.is_initiator,
));
}
// === Traffic Counters ===
@@ -665,9 +668,9 @@ impl SessionEntry {
self.rekey_jitter_secs = draw_rekey_jitter();
// Reset MMP counters to avoid metric discontinuity
let now = Instant::now();
let now_ms = crate::time::mono_ms();
if let Some(mmp) = &mut self.mmp {
mmp.reset_for_rekey(now);
mmp.reset_for_rekey(now_ms);
}
true
}
@@ -798,8 +801,8 @@ impl SessionEntry {
#[cfg(test)]
mod overlapping_epoch_tests {
use super::*;
use crate::node::session_wire::{FSP_FLAG_K, build_fsp_header};
use crate::noise::HandshakeState;
use crate::proto::fsp::wire::{FSP_FLAG_K, build_fsp_header};
use secp256k1::{Keypair, Secp256k1, SecretKey};
/// Deterministic keypair from a single seed byte.
@@ -1262,4 +1265,133 @@ mod overlapping_epoch_tests {
"window must expire on the plain wall-clock timer when peer is off the old epoch"
);
}
// ========================================================================
// Rekey-policy characterization (pins `check_session_rekey`'s decision
// boundaries before the `Fsp::poll_rekey` hoist — these thresholds have no
// other test module).
// ========================================================================
/// The initiator liveness-cutover delay used by `check_session_rekey`
/// (`FSP_CUTOVER_DELAY_MS`). Mirrored here as the characterization anchor.
const CUTOVER_DELAY_MS: u64 = 2000;
/// Build an established entry that has completed a rekey as initiator and
/// holds a pending session awaiting the K-bit cutover.
fn entry_pending_cutover(rekey_completed_ms: u64) -> SessionEntry {
let (_cur_send, cur_recv) = xk_pair(1, 2);
let (_new_send, new_recv) = xk_pair(3, 4);
let mut entry = entry_with_current(cur_recv);
// Mark ourselves the rekey initiator, then land the completed session
// as pending (clears rekey_state, so has_rekey_in_progress() == false).
entry.set_rekey_state(HandshakeState::new_xk_responder(keypair(7)), true);
entry.set_pending_session(new_recv);
entry.set_rekey_completed_ms(rekey_completed_ms);
entry
}
// The initiator-side cutover predicate: pending session present, no rekey
// in progress, we are the initiator, and the liveness timer has elapsed.
#[test]
fn rekey_cutover_predicate_boundary() {
let completed = 1_000u64;
let entry = entry_pending_cutover(completed);
assert!(entry.pending_new_session().is_some());
assert!(!entry.has_rekey_in_progress());
assert!(entry.is_rekey_initiator());
// Not yet eligible one ms before the delay elapses.
let just_before = completed + CUTOVER_DELAY_MS - 1;
assert!(
just_before.saturating_sub(entry.rekey_completed_ms()) < CUTOVER_DELAY_MS,
"cutover must not fire before the liveness delay"
);
// Eligible exactly at the delay.
let at = completed + CUTOVER_DELAY_MS;
assert!(
at.saturating_sub(entry.rekey_completed_ms()) >= CUTOVER_DELAY_MS,
"cutover fires once the liveness delay has elapsed"
);
}
// The rekey trigger's own threshold arithmetic is tested against the real
// predicate in src/proto/fmp/tests/core.rs, which drives poll_rekey. A test
// here previously reproduced that OR predicate as a local closure and
// asserted against its own copy, so it could not fail for the reason it
// existed: deleting the counter arm from the trigger left it green. Its one
// assertion over real code, that a fresh entry's jitter lies within the
// symmetric bound, is covered over 100 samples by
// test_session_entry_rekey_jitter_in_range in src/node/tests/session.rs.
// Dampening boundary: within `dampening_ms` of the peer's rekey msg1, local
// initiation is suppressed; at/after the window it is not.
#[test]
fn rekey_dampening_boundary() {
let (_s, recv) = xk_pair(1, 2);
let mut entry = entry_with_current(recv);
const DAMP_MS: u64 = 30_000;
// No peer rekey recorded → never dampened.
assert!(!entry.is_rekey_dampened(50_000, DAMP_MS));
entry.record_peer_rekey(10_000);
assert!(
entry.is_rekey_dampened(10_000 + DAMP_MS - 1, DAMP_MS),
"dampened within the window"
);
assert!(
!entry.is_rekey_dampened(10_000 + DAMP_MS, DAMP_MS),
"not dampened once the window has elapsed"
);
}
// Epoch-reaction: a frame authenticating against `pending` while a msg3
// retransmission is retained confirms the peer on the new epoch (clears the
// msg3 payload) and then promotes.
#[test]
fn epoch_reaction_pending_confirms_then_promotes() {
let (mut p_send, p_recv) = xk_pair(3, 4);
let (_cur_send, cur_recv) = xk_pair(1, 2);
let mut entry = entry_with_current(cur_recv);
let k_before = entry.current_k_bit();
entry.set_pending_session(p_recv);
entry.set_rekey_msg3_payload(vec![0xAB; 8], 5_000);
assert!(entry.rekey_msg3_payload().is_some());
let (ct, counter, hdr) = seal(&mut p_send, b"new-epoch", !k_before);
let (_pt, slot) = entry
.fsp_trial_decrypt(&ct, counter, &hdr, !k_before, 2_000)
.expect("pending frame decrypts");
assert_eq!(slot, EpochSlot::Pending);
// Reaction order: confirm (while pending still held) then promote.
entry.confirm_peer_new_epoch();
assert!(entry.rekey_msg3_payload().is_none());
entry.handle_peer_kbit_flip(2_000);
assert!(entry.pending_new_session().is_none());
assert_ne!(entry.current_k_bit(), k_before);
}
// Epoch-reaction: as the initiator that already cut over on its own timer
// (msg3 retained, no pending), a frame authenticating against `current`
// confirms the responder reached the new epoch.
#[test]
fn epoch_reaction_current_confirms_responder() {
let (mut cur_send, cur_recv) = xk_pair(1, 2);
let mut entry = entry_with_current(cur_recv);
entry.set_rekey_msg3_payload(vec![0xCD; 8], 5_000);
assert!(entry.pending_new_session().is_none());
assert!(entry.rekey_msg3_payload().is_some());
let (ct, counter, hdr) = seal(&mut cur_send, b"steady", false);
let (_pt, slot) = entry
.fsp_trial_decrypt(&ct, counter, &hdr, false, 2_000)
.expect("current frame decrypts");
assert_eq!(slot, EpochSlot::Current);
// The Current-with-retained-msg3-and-no-pending arm confirms.
entry.confirm_peer_new_epoch();
assert!(entry.rekey_msg3_payload().is_none());
}
}
src/node/session_wire.rs
-618
View File
@@ -1,618 +0,0 @@
//! FSP Wire Format Parsing and Serialization
//!
//! Defines the FIPS session-layer wire format (FSP) for packet dispatch.
//! All FSP messages begin with a 4-byte common prefix followed by phase-specific
//! fields. Encrypted messages use a 12-byte cleartext header as AAD for AEAD,
//! and a 6-byte encrypted inner header containing timestamps and message type.
//!
//! ## Common Prefix (4 bytes)
//!
//! ```text
//! [ver+phase:1][flags:1][payload_len:2 LE]
//! ```
//!
//! ## DataPacket Port Multiplexing
//!
//! DataPacket (msg_type 0x10) payloads inside the AEAD envelope carry a 4-byte
//! port header for service dispatch:
//!
//! ```text
//! [src_port:2 LE][dst_port:2 LE][service payload...]
//! ```
//!
//! Port 256 (0x100) = IPv6 shim with header compression.
//!
//! ## Message Classes
//!
//! | Phase | U Flag | Type | Description |
//! |-------|--------|------------------|-----------------------------------|
//! | 0x0 | 0 | Encrypted | Post-handshake encrypted data |
//! | 0x0 | 1 | Plaintext error | CoordsRequired, PathBroken |
//! | 0x1 | - | Handshake msg1 | SessionSetup (Noise XK msg1) |
//! | 0x2 | - | Handshake msg2 | SessionAck (Noise XK msg2) |
//! | 0x3 | - | Handshake msg3 | SessionMsg3 (Noise XK msg3) |
use crate::protocol::{ProtocolError, decode_optional_coords};
use crate::tree::TreeCoordinate;
// ============================================================================
// Constants
// ============================================================================
/// FSP protocol version (4 high bits of byte 0).
pub const FSP_VERSION: u8 = 0;
/// Phase value for established (encrypted or plaintext error) messages.
pub const FSP_PHASE_ESTABLISHED: u8 = 0x0;
/// Phase value for SessionSetup (Noise IK message 1).
pub const FSP_PHASE_MSG1: u8 = 0x1;
/// Phase value for SessionAck (Noise handshake message 2).
pub const FSP_PHASE_MSG2: u8 = 0x2;
/// Phase value for XK message 3 (initiator's encrypted static).
pub const FSP_PHASE_MSG3: u8 = 0x3;
/// Size of the common packet prefix (all FSP message types).
pub const FSP_COMMON_PREFIX_SIZE: usize = 4;
/// Size of the full encrypted message header (prefix + counter).
pub const FSP_HEADER_SIZE: usize = 12;
/// Size of the encrypted inner header (timestamp + msg_type + inner_flags).
pub const FSP_INNER_HEADER_SIZE: usize = 6;
/// AEAD authentication tag size (ChaCha20-Poly1305).
const TAG_SIZE: usize = 16;
/// Minimum size for an encrypted FSP message: header + tag (no plaintext).
pub const FSP_ENCRYPTED_MIN_SIZE: usize = FSP_HEADER_SIZE + TAG_SIZE; // 28 bytes
// FSP DataPacket port header constants.
/// Size of the FSP DataPacket port header (src_port + dst_port).
pub const FSP_PORT_HEADER_SIZE: usize = 4;
/// FSP port: IPv6 shim service.
pub const FSP_PORT_IPV6_SHIM: u16 = 256;
// Cleartext flag bit constants (byte 1 of common prefix, phase 0x0 only).
/// Coords Present — source and destination coordinates follow the header.
pub const FSP_FLAG_CP: u8 = 0x01;
/// Key Epoch — selects active key during rekeying.
#[allow(dead_code)]
pub const FSP_FLAG_K: u8 = 0x02;
/// Unencrypted — payload is plaintext (error signals).
pub const FSP_FLAG_U: u8 = 0x04;
// Inner flag bit constants (byte 5 of decrypted inner header).
/// Spin bit for end-to-end RTT measurement (inside AEAD).
#[allow(dead_code)]
pub const FSP_INNER_FLAG_SP: u8 = 0x01;
// ============================================================================
// Common Prefix
// ============================================================================
/// Parsed FSP common packet prefix (first 4 bytes of every FSP message).
///
/// Wire format:
/// ```text
/// [ver(4bits)+phase(4bits)][flags:1][payload_len:2 LE]
/// ```
#[derive(Clone, Debug)]
pub struct FspCommonPrefix {
/// Protocol version (high nibble of byte 0).
#[cfg_attr(not(test), allow(dead_code))]
pub version: u8,
/// Session lifecycle phase (low nibble of byte 0).
pub phase: u8,
/// Per-message signal flags.
pub flags: u8,
/// Length of payload following the phase-specific header.
#[cfg_attr(not(test), allow(dead_code))]
pub payload_len: u16,
}
impl FspCommonPrefix {
/// Parse a common prefix from the first 4 bytes of FSP message data.
pub fn parse(data: &[u8]) -> Option<Self> {
if data.len() < FSP_COMMON_PREFIX_SIZE {
return None;
}
let version = data[0] >> 4;
let phase = data[0] & 0x0F;
let flags = data[1];
let payload_len = u16::from_le_bytes([data[2], data[3]]);
Some(Self {
version,
phase,
flags,
payload_len,
})
}
/// Check if the Unencrypted flag is set.
pub fn is_unencrypted(&self) -> bool {
self.flags & FSP_FLAG_U != 0
}
/// Check if the Coords Present flag is set.
pub fn has_coords(&self) -> bool {
self.flags & FSP_FLAG_CP != 0
}
/// Encode the ver+phase byte.
fn ver_phase_byte(version: u8, phase: u8) -> u8 {
(version << 4) | (phase & 0x0F)
}
}
// ============================================================================
// Encrypted Message Header
// ============================================================================
/// Parsed FSP encrypted message header (phase 0x0, U flag clear).
///
/// Wire format (12 bytes):
/// ```text
/// [ver+phase:1][flags:1][payload_len:2 LE][counter:8 LE]
/// ```
///
/// The full 12-byte header is used as AAD for the AEAD construction.
/// No receiver_idx — unlike FMP, FSP is end-to-end (dispatched by src_addr
/// from the SessionDatagram envelope, not by index).
#[derive(Clone, Debug)]
pub struct FspEncryptedHeader {
/// Per-message flags (CP, K).
pub flags: u8,
/// Length of encrypted payload (excluding AEAD tag).
#[cfg_attr(not(test), allow(dead_code))]
pub payload_len: u16,
/// Monotonic counter used as AEAD nonce.
pub counter: u64,
/// Raw 12-byte header for use as AEAD AAD.
pub header_bytes: [u8; FSP_HEADER_SIZE],
}
impl FspEncryptedHeader {
/// Parse an encrypted message header from FSP message data.
///
/// Returns None if the data is too short or has wrong version/phase,
/// or if the U flag is set (plaintext messages use a different path).
pub fn parse(data: &[u8]) -> Option<Self> {
if data.len() < FSP_ENCRYPTED_MIN_SIZE {
return None;
}
let version = data[0] >> 4;
let phase = data[0] & 0x0F;
if version != FSP_VERSION || phase != FSP_PHASE_ESTABLISHED {
return None;
}
let flags = data[1];
// U flag means plaintext — not an encrypted message
if flags & FSP_FLAG_U != 0 {
return None;
}
let payload_len = u16::from_le_bytes([data[2], data[3]]);
let counter = u64::from_le_bytes([
data[4], data[5], data[6], data[7], data[8], data[9], data[10], data[11],
]);
let mut header_bytes = [0u8; FSP_HEADER_SIZE];
header_bytes.copy_from_slice(&data[..FSP_HEADER_SIZE]);
Some(Self {
flags,
payload_len,
counter,
header_bytes,
})
}
/// Check if the Coords Present flag is set.
pub fn has_coords(&self) -> bool {
self.flags & FSP_FLAG_CP != 0
}
/// Offset where ciphertext (or coords if CP) begins in the original data.
#[cfg_attr(not(test), allow(dead_code))]
pub fn data_offset(&self) -> usize {
FSP_HEADER_SIZE
}
}
// ============================================================================
// Serialization Helpers
// ============================================================================
/// Build the 12-byte cleartext header for an encrypted FSP message.
///
/// Returns the header bytes for use as AEAD AAD.
pub fn build_fsp_header(counter: u64, flags: u8, payload_len: u16) -> [u8; FSP_HEADER_SIZE] {
let mut header = [0u8; FSP_HEADER_SIZE];
header[0] = FspCommonPrefix::ver_phase_byte(FSP_VERSION, FSP_PHASE_ESTABLISHED);
header[1] = flags;
header[2..4].copy_from_slice(&payload_len.to_le_bytes());
header[4..12].copy_from_slice(&counter.to_le_bytes());
header
}
/// Assemble a wire-format encrypted FSP message.
///
/// Format: `[header:12][ciphertext+tag]`
#[cfg_attr(not(test), allow(dead_code))]
pub fn build_fsp_encrypted(header: &[u8; FSP_HEADER_SIZE], ciphertext: &[u8]) -> Vec<u8> {
let mut packet = Vec::with_capacity(FSP_HEADER_SIZE + ciphertext.len());
packet.extend_from_slice(header);
packet.extend_from_slice(ciphertext);
packet
}
/// Build a 4-byte common prefix for a handshake message.
///
/// `phase` should be `FSP_PHASE_MSG1`, `FSP_PHASE_MSG2`, or `FSP_PHASE_MSG3`.
/// Flags are zero during handshake.
#[cfg_attr(not(test), allow(dead_code))]
pub fn build_fsp_handshake_prefix(phase: u8, payload_len: u16) -> [u8; FSP_COMMON_PREFIX_SIZE] {
let mut prefix = [0u8; FSP_COMMON_PREFIX_SIZE];
prefix[0] = FspCommonPrefix::ver_phase_byte(FSP_VERSION, phase);
prefix[1] = 0x00; // flags must be zero during handshake
prefix[2..4].copy_from_slice(&payload_len.to_le_bytes());
prefix
}
/// Build a 4-byte common prefix for a plaintext error signal.
///
/// Sets phase 0x0 and U flag.
#[cfg_attr(not(test), allow(dead_code))]
pub fn build_fsp_error_prefix(payload_len: u16) -> [u8; FSP_COMMON_PREFIX_SIZE] {
let mut prefix = [0u8; FSP_COMMON_PREFIX_SIZE];
prefix[0] = FspCommonPrefix::ver_phase_byte(FSP_VERSION, FSP_PHASE_ESTABLISHED);
prefix[1] = FSP_FLAG_U;
prefix[2..4].copy_from_slice(&payload_len.to_le_bytes());
prefix
}
// ============================================================================
// Inner Header Helpers
// ============================================================================
/// Prepend the 6-byte FSP inner header to a message payload.
///
/// Inner header: `[timestamp:4 LE][msg_type:1][inner_flags:1]`
///
/// The caller provides the message-type-specific payload (e.g., application
/// data for msg_type 0x10, report fields for SenderReport). This function
/// prepends the inner header.
pub fn fsp_prepend_inner_header(
timestamp_ms: u32,
msg_type: u8,
inner_flags: u8,
payload: &[u8],
) -> Vec<u8> {
let mut buf = Vec::with_capacity(FSP_INNER_HEADER_SIZE + payload.len());
buf.extend_from_slice(&timestamp_ms.to_le_bytes());
buf.push(msg_type);
buf.push(inner_flags);
buf.extend_from_slice(payload);
buf
}
/// Strip the 6-byte FSP inner header from a decrypted payload.
///
/// Returns `(timestamp, msg_type, inner_flags, &rest)` or None if too short.
pub fn fsp_strip_inner_header(plaintext: &[u8]) -> Option<(u32, u8, u8, &[u8])> {
if plaintext.len() < FSP_INNER_HEADER_SIZE {
return None;
}
let timestamp = u32::from_le_bytes([plaintext[0], plaintext[1], plaintext[2], plaintext[3]]);
let msg_type = plaintext[4];
let inner_flags = plaintext[5];
Some((
timestamp,
msg_type,
inner_flags,
&plaintext[FSP_INNER_HEADER_SIZE..],
))
}
// ============================================================================
// Coordinate Parsing (for transit nodes and receive path)
// ============================================================================
/// Parse source and destination coordinates from the cleartext section
/// of an encrypted FSP message when the CP flag is set.
///
/// Coordinates appear between the 12-byte header and the ciphertext:
/// `[src_coords_count:2 LE][src_coords:16×n][dest_coords_count:2 LE][dest_coords:16×m]`
///
/// Returns `(src_coords, dest_coords, bytes_consumed)`.
pub fn parse_encrypted_coords(
data: &[u8],
) -> Result<(Option<TreeCoordinate>, Option<TreeCoordinate>, usize), ProtocolError> {
let (src_coords, src_consumed) = decode_optional_coords(data)?;
let (dest_coords, dest_consumed) = decode_optional_coords(&data[src_consumed..])?;
Ok((src_coords, dest_coords, src_consumed + dest_consumed))
}
// ============================================================================
// Tests
// ============================================================================
#[cfg(test)]
mod tests {
use super::*;
// ===== Size Constant Tests =====
#[test]
fn test_wire_sizes() {
assert_eq!(FSP_COMMON_PREFIX_SIZE, 4);
assert_eq!(FSP_HEADER_SIZE, 12);
assert_eq!(FSP_INNER_HEADER_SIZE, 6);
assert_eq!(FSP_ENCRYPTED_MIN_SIZE, 28); // 12 + 16
}
// ===== Common Prefix Tests =====
#[test]
fn test_common_prefix_parse_established() {
let data = [0x00, 0x01, 0x40, 0x00]; // ver=0, phase=0, flags=CP, payload_len=64
let prefix = FspCommonPrefix::parse(&data).unwrap();
assert_eq!(prefix.version, 0);
assert_eq!(prefix.phase, FSP_PHASE_ESTABLISHED);
assert_eq!(prefix.flags, FSP_FLAG_CP);
assert_eq!(prefix.payload_len, 64);
assert!(prefix.has_coords());
assert!(!prefix.is_unencrypted());
}
#[test]
fn test_common_prefix_parse_handshake() {
let data = [0x01, 0x00, 0x50, 0x00]; // ver=0, phase=1, flags=0, payload_len=80
let prefix = FspCommonPrefix::parse(&data).unwrap();
assert_eq!(prefix.version, 0);
assert_eq!(prefix.phase, FSP_PHASE_MSG1);
assert_eq!(prefix.flags, 0);
assert_eq!(prefix.payload_len, 80);
}
#[test]
fn test_common_prefix_parse_error_signal() {
let data = [0x00, FSP_FLAG_U, 0x22, 0x00]; // ver=0, phase=0, U flag, payload_len=34
let prefix = FspCommonPrefix::parse(&data).unwrap();
assert_eq!(prefix.phase, FSP_PHASE_ESTABLISHED);
assert!(prefix.is_unencrypted());
assert_eq!(prefix.payload_len, 34);
}
#[test]
fn test_common_prefix_too_short() {
assert!(FspCommonPrefix::parse(&[0, 0, 0]).is_none());
}
// ===== Encrypted Header Tests =====
#[test]
fn test_encrypted_header_parse() {
let counter = 42u64;
let flags = FSP_FLAG_CP;
let payload_len = 100u16;
let header = build_fsp_header(counter, flags, payload_len);
// Build a minimal packet: header + 16 bytes of fake ciphertext (tag)
let mut packet = Vec::from(header);
packet.extend_from_slice(&[0xaa; TAG_SIZE]);
let parsed = FspEncryptedHeader::parse(&packet).unwrap();
assert_eq!(parsed.counter, 42);
assert_eq!(parsed.flags, FSP_FLAG_CP);
assert_eq!(parsed.payload_len, 100);
assert!(parsed.has_coords());
assert_eq!(parsed.header_bytes, header);
assert_eq!(parsed.data_offset(), FSP_HEADER_SIZE);
}
#[test]
fn test_encrypted_header_too_short() {
let packet = vec![0x00; FSP_ENCRYPTED_MIN_SIZE - 1];
assert!(FspEncryptedHeader::parse(&packet).is_none());
}
#[test]
fn test_encrypted_header_wrong_phase() {
let mut packet = vec![0x00; FSP_ENCRYPTED_MIN_SIZE];
packet[0] = 0x01; // phase 1 (msg1), not established
assert!(FspEncryptedHeader::parse(&packet).is_none());
}
#[test]
fn test_encrypted_header_wrong_version() {
let mut packet = vec![0x00; FSP_ENCRYPTED_MIN_SIZE];
packet[0] = 0x10; // version 1, phase 0
assert!(FspEncryptedHeader::parse(&packet).is_none());
}
#[test]
fn test_encrypted_header_u_flag_rejected() {
let mut packet = vec![0x00; FSP_ENCRYPTED_MIN_SIZE];
packet[1] = FSP_FLAG_U; // U flag set → not encrypted
assert!(FspEncryptedHeader::parse(&packet).is_none());
}
// ===== Build Header Tests =====
#[test]
fn test_build_fsp_header() {
let header = build_fsp_header(1000, FSP_FLAG_CP, 200);
assert_eq!(header[0], 0x00); // ver=0, phase=0
assert_eq!(header[1], FSP_FLAG_CP);
assert_eq!(u16::from_le_bytes([header[2], header[3]]), 200);
assert_eq!(
u64::from_le_bytes([
header[4], header[5], header[6], header[7], header[8], header[9], header[10],
header[11],
]),
1000
);
}
#[test]
fn test_build_fsp_encrypted() {
let header = build_fsp_header(0, 0, 10);
let ciphertext = vec![0xCC; 26]; // 10 payload + 16 tag
let packet = build_fsp_encrypted(&header, &ciphertext);
assert_eq!(packet.len(), FSP_HEADER_SIZE + 26);
assert_eq!(&packet[..FSP_HEADER_SIZE], &header);
assert_eq!(&packet[FSP_HEADER_SIZE..], &ciphertext[..]);
}
// ===== Handshake Prefix Tests =====
#[test]
fn test_build_fsp_handshake_prefix_msg1() {
let prefix = build_fsp_handshake_prefix(FSP_PHASE_MSG1, 100);
assert_eq!(prefix[0], 0x01); // ver=0, phase=1
assert_eq!(prefix[1], 0x00); // flags zero
assert_eq!(u16::from_le_bytes([prefix[2], prefix[3]]), 100);
let parsed = FspCommonPrefix::parse(&prefix).unwrap();
assert_eq!(parsed.phase, FSP_PHASE_MSG1);
}
#[test]
fn test_build_fsp_handshake_prefix_msg2() {
let prefix = build_fsp_handshake_prefix(FSP_PHASE_MSG2, 50);
assert_eq!(prefix[0], 0x02); // ver=0, phase=2
assert_eq!(prefix[1], 0x00);
assert_eq!(u16::from_le_bytes([prefix[2], prefix[3]]), 50);
}
#[test]
fn test_build_fsp_handshake_prefix_msg3() {
let prefix = build_fsp_handshake_prefix(FSP_PHASE_MSG3, 73);
assert_eq!(prefix[0], 0x03); // ver=0, phase=3
assert_eq!(prefix[1], 0x00); // flags zero
assert_eq!(u16::from_le_bytes([prefix[2], prefix[3]]), 73);
let parsed = FspCommonPrefix::parse(&prefix).unwrap();
assert_eq!(parsed.phase, FSP_PHASE_MSG3);
}
// ===== Error Prefix Tests =====
#[test]
fn test_build_fsp_error_prefix() {
let prefix = build_fsp_error_prefix(34);
assert_eq!(prefix[0], 0x00); // ver=0, phase=0
assert_eq!(prefix[1], FSP_FLAG_U);
assert_eq!(u16::from_le_bytes([prefix[2], prefix[3]]), 34);
let parsed = FspCommonPrefix::parse(&prefix).unwrap();
assert!(parsed.is_unencrypted());
assert_eq!(parsed.phase, FSP_PHASE_ESTABLISHED);
}
// ===== Inner Header Tests =====
#[test]
fn test_inner_header_prepend_strip() {
let timestamp: u32 = 12345;
let msg_type: u8 = 0x10;
let inner_flags: u8 = 0x01; // SP bit
let payload = vec![0xAA, 0xBB, 0xCC];
let with_header = fsp_prepend_inner_header(timestamp, msg_type, inner_flags, &payload);
assert_eq!(with_header.len(), FSP_INNER_HEADER_SIZE + 3);
let (ts, mt, flags, rest) = fsp_strip_inner_header(&with_header).unwrap();
assert_eq!(ts, 12345);
assert_eq!(mt, 0x10);
assert_eq!(flags, 0x01);
assert_eq!(rest, &payload[..]);
}
#[test]
fn test_inner_header_empty_payload() {
let with_header = fsp_prepend_inner_header(0, 0x13, 0, &[]);
assert_eq!(with_header.len(), FSP_INNER_HEADER_SIZE);
let (ts, mt, flags, rest) = fsp_strip_inner_header(&with_header).unwrap();
assert_eq!(ts, 0);
assert_eq!(mt, 0x13);
assert_eq!(flags, 0);
assert!(rest.is_empty());
}
#[test]
fn test_inner_header_too_short() {
assert!(fsp_strip_inner_header(&[0, 0, 0, 0, 0]).is_none()); // needs 6 bytes
assert!(fsp_strip_inner_header(&[]).is_none());
}
// ===== Flag Constants Tests =====
#[test]
fn test_flag_bits_distinct() {
// Cleartext flags don't overlap
assert_eq!(FSP_FLAG_CP & FSP_FLAG_K, 0);
assert_eq!(FSP_FLAG_CP & FSP_FLAG_U, 0);
assert_eq!(FSP_FLAG_K & FSP_FLAG_U, 0);
}
#[test]
fn test_header_roundtrip() {
let counter = 0xDEADBEEF_12345678u64;
let flags = FSP_FLAG_CP | FSP_FLAG_K;
let payload_len = 1234u16;
let header = build_fsp_header(counter, flags, payload_len);
let ciphertext = vec![0xFF; payload_len as usize + TAG_SIZE];
let packet = build_fsp_encrypted(&header, &ciphertext);
let parsed = FspEncryptedHeader::parse(&packet).unwrap();
assert_eq!(parsed.counter, counter);
assert_eq!(parsed.flags, flags);
assert_eq!(parsed.payload_len, payload_len);
assert!(parsed.has_coords());
assert_eq!(parsed.header_bytes, header);
}
#[test]
fn test_all_message_types_through_prefix() {
// Encrypted (phase 0, no U)
let prefix = FspCommonPrefix::parse(&[0x00, 0x00, 0x10, 0x00]).unwrap();
assert_eq!(prefix.phase, 0);
assert!(!prefix.is_unencrypted());
// Error signal (phase 0, U set)
let prefix = FspCommonPrefix::parse(&[0x00, FSP_FLAG_U, 0x22, 0x00]).unwrap();
assert_eq!(prefix.phase, 0);
assert!(prefix.is_unencrypted());
// SessionSetup (phase 1)
let prefix = FspCommonPrefix::parse(&[0x01, 0x00, 0x50, 0x00]).unwrap();
assert_eq!(prefix.phase, 1);
// SessionAck (phase 2)
let prefix = FspCommonPrefix::parse(&[0x02, 0x00, 0x21, 0x00]).unwrap();
assert_eq!(prefix.phase, 2);
// SessionMsg3 (phase 3)
let prefix = FspCommonPrefix::parse(&[0x03, 0x00, 0x49, 0x00]).unwrap();
assert_eq!(prefix.phase, 3);
}
}
src/node/stats.rs
+1 -1
View File
@@ -238,7 +238,7 @@ pub struct ForwardingStatsSnapshot {
}
#[derive(Clone, Debug, Default, Serialize)]
pub struct DiscoveryStatsSnapshot {
pub struct LookupStatsSnapshot {
pub req_received: u64,
pub req_decode_error: u64,
pub req_duplicate: u64,
src/node/tests/acl.rs
+64 -10
View File
@@ -2,7 +2,7 @@ use super::*;
use crate::ReceivedPacket;
use crate::node::acl::PeerAclReloader;
use crate::node::reloadable::HostMapReloadable;
use crate::node::wire::{build_msg1, build_msg2};
use crate::proto::fmp::wire::{build_msg1, build_msg2};
use crate::upper::hosts::HostMap;
use crate::utils::index::SessionIndex;
use std::path::PathBuf;
@@ -56,7 +56,7 @@ async fn test_inbound_msg1_denied_by_acl() {
node_b.reload_peer_acl().await;
let peer_b_identity = PeerIdentity::from_pubkey_full(node_b.identity().pubkey_full());
let mut conn_a = PeerConnection::outbound(LinkId::new(1), peer_b_identity, 1000);
let mut conn_a = outbound_leg(LinkId::new(1), peer_b_identity, 1000);
let noise_msg1 = conn_a
.start_handshake(node_a.identity().keypair(), node_a.startup_epoch(), 1000)
.unwrap();
@@ -84,14 +84,23 @@ async fn test_outbound_msg2_denied_after_acl_reload() {
let peer_b_identity = PeerIdentity::from_pubkey_full(node_b.identity().pubkey_full());
let link_id_a = node_a.allocate_link_id();
let mut conn_a = PeerConnection::outbound(link_id_a, peer_b_identity, 1000);
let our_index_a = node_a.index_allocator.allocate().unwrap();
let noise_msg1 = conn_a
.start_handshake(node_a.identity().keypair(), node_a.startup_epoch(), 1000)
node_a
.seed_handshake_machine(
HandshakeSeed::outbound(link_id_a, peer_b_identity, 1000)
.with_our_index(our_index_a)
.with_transport_id(transport_id)
.with_source_addr(remote_addr.clone()),
)
.unwrap();
let keypair_a = node_a.identity().keypair();
let epoch_a = node_a.startup_epoch();
let noise_msg1 = node_a
.peer_machines
.get_mut(&link_id_a)
.unwrap()
.start_handshake(keypair_a, epoch_a, 1000)
.unwrap();
conn_a.set_our_index(our_index_a);
conn_a.set_transport_id(transport_id);
conn_a.set_source_addr(remote_addr.clone());
let link_a = Link::connectionless(
link_id_a,
@@ -104,12 +113,11 @@ async fn test_outbound_msg2_denied_after_acl_reload() {
node_a
.addr_to_link
.insert((transport_id, remote_addr.clone()), link_id_a);
node_a.connections.insert(link_id_a, conn_a);
node_a
.pending_outbound
.insert((transport_id, our_index_a.as_u32()), link_id_a);
let mut conn_b = PeerConnection::inbound(LinkId::new(2), 1000);
let mut conn_b = inbound_leg(LinkId::new(2), 1000);
let responder_epoch = [0x11; 8];
let noise_msg2 = conn_b
.receive_handshake_init(
@@ -178,3 +186,49 @@ async fn test_outbound_connect_not_denied_by_allowlist_miss() {
assert!(!matches!(result, Err(NodeError::AccessDenied(_))));
}
/// The ACL-rejected arm of the same property the Noise-failure arm pins in
/// `unit.rs`: a msg1 that is admitted by the crypto but denied by the ACL
/// still leaves nothing behind. The control machine is built above the crypto
/// so it can drive the handshake, but it stays a local until a promote tail
/// inserts it, so a denial drops it.
#[tokio::test]
async fn test_acl_rejected_msg1_leaves_no_registry_trace() {
let (dir, mut node_b) = make_acl_node();
let node_a = make_node();
std::fs::write(deny_path(&dir), format!("{}\n", node_a.npub())).unwrap();
node_b.reload_peer_acl().await;
let peer_b_identity = PeerIdentity::from_pubkey_full(node_b.identity().pubkey_full());
let mut conn_a = outbound_leg(LinkId::new(1), peer_b_identity, 1000);
let noise_msg1 = conn_a
.start_handshake(node_a.identity().keypair(), node_a.startup_epoch(), 1000)
.unwrap();
let wire_msg1 = build_msg1(SessionIndex::new(7), &noise_msg1);
let packet = ReceivedPacket::with_timestamp(
TransportId::new(1),
TransportAddr::from_string("127.0.0.1:5000"),
wire_msg1,
1000,
);
node_b.handle_msg1(packet).await;
assert!(
node_b.peer_machines.is_empty(),
"an ACL-denied msg1 must leave no control machine behind"
);
assert_eq!(node_b.connection_count(), 0);
assert_eq!(node_b.peer_count(), 0);
assert_eq!(node_b.link_count(), 0);
assert!(
node_b.peers_by_index.is_empty(),
"an ACL-denied msg1 must allocate no session index"
);
assert_eq!(
node_b.stats().handshake.bad_state,
1,
"the denial is attributed to the handshake state-machine counter"
);
}
src/node/tests/bloom.rs
+7 -7
View File
@@ -455,9 +455,9 @@ async fn test_bloom_filter_split_horizon() {
/// counted once, not the double-count fingerprint.
#[test]
fn compute_mesh_size_counts_each_peer_filter_once() {
use crate::bloom::BloomFilter;
use crate::peer::ActivePeer;
use crate::tree::ParentDeclaration;
use crate::proto::bloom::BloomFilter;
use crate::proto::stp::ParentDeclaration;
let mut node = make_node();
let my_addr = *node.tree_state().my_node_addr();
@@ -498,8 +498,8 @@ fn compute_mesh_size_counts_each_peer_filter_once() {
// Seed parent ancestry first so recompute_coords can extend it and
// flip is_root() to false; child ancestry is for completeness.
let parent_ancestry = crate::tree::TreeCoordinate::root_with_meta(parent_addr, 1, 1);
let child_ancestry = crate::tree::TreeCoordinate::root_with_meta(child_addr, 1, 1);
let parent_ancestry = crate::proto::stp::TreeCoordinate::root_with_meta(parent_addr, 1, 1);
let child_ancestry = crate::proto::stp::TreeCoordinate::root_with_meta(child_addr, 1, 1);
// Inject the stale-cache scenario: peer_declaration(P) still names
// US (M) as P's parent (the pre-switch advert that the cache hasn't
// refreshed yet). Q is a legitimate child also naming M as parent.
@@ -513,7 +513,7 @@ fn compute_mesh_size_counts_each_peer_filter_once() {
.update_peer(child_decl, child_ancestry);
// Switch our parent to P and recompute coords so root flips off self.
node.tree_state_mut().set_parent(parent_addr, 2, 1);
node.tree_state_mut().set_parent(parent_addr, 2, 1, 1);
node.tree_state_mut().recompute_coords();
assert!(
!node.tree_state().is_root(),
@@ -550,8 +550,8 @@ fn compute_mesh_size_counts_each_peer_filter_once() {
/// `estimated_mesh_size` carries.
#[test]
fn compute_mesh_size_unions_overlapping_filters() {
use crate::bloom::BloomFilter;
use crate::peer::ActivePeer;
use crate::proto::bloom::BloomFilter;
let mut node = make_node();
@@ -632,8 +632,8 @@ fn compute_mesh_size_unions_overlapping_filters() {
/// removes the parent, and asserts the estimate does not collapse.
#[test]
fn compute_mesh_size_stable_across_parent_drop_with_cross_link() {
use crate::bloom::BloomFilter;
use crate::peer::ActivePeer;
use crate::proto::bloom::BloomFilter;
let mut node = make_node();
src/node/tests/bloom_poison.rs
+2 -2
View File
@@ -7,9 +7,9 @@
//! bloom.rs.
use super::*;
use crate::bloom::{BloomFilter, DEFAULT_FILTER_SIZE_BITS, DEFAULT_HASH_COUNT};
use crate::peer::ActivePeer;
use crate::protocol::FilterAnnounce;
use crate::proto::bloom::FilterAnnounce;
use crate::proto::bloom::{BloomFilter, DEFAULT_FILTER_SIZE_BITS, DEFAULT_HASH_COUNT};
/// Inject a synthetic active peer into the node with a known NodeAddr.
/// Returns the peer's NodeAddr.
src/node/tests/bootstrap.rs
+20 -21
View File
@@ -3,7 +3,7 @@
use super::*;
use crate::EstablishedTraversal;
use crate::config::{TransportInstances, UdpConfig};
use crate::node::wire::{PHASE_MSG1, PHASE_MSG2};
use crate::proto::fmp::wire::{PHASE_MSG1, PHASE_MSG2};
use crate::transport::udp::UdpTransport;
use crate::utils::index::IndexAllocator;
use std::collections::HashMap;
@@ -24,17 +24,17 @@ async fn test_adopted_udp_traversal_completes_handshake() {
let (packet_tx_a, packet_rx_a) = packet_channel(64);
let (packet_tx_b, packet_rx_b) = packet_channel(64);
node_a.packet_tx = Some(packet_tx_a.clone());
node_a.supervisor.packet_tx = Some(packet_tx_a.clone());
node_a.packet_rx = Some(packet_rx_a);
node_a.state = NodeState::Running;
node_a.supervisor.state = NodeState::Running;
let mut transport_b = UdpTransport::new(transport_id_b, None, udp_config, packet_tx_b.clone());
transport_b.start_async().await.unwrap();
let addr_b = transport_b.local_addr().unwrap();
node_b.packet_tx = Some(packet_tx_b.clone());
node_b.supervisor.packet_tx = Some(packet_tx_b.clone());
node_b.packet_rx = Some(packet_rx_b);
node_b.state = NodeState::Running;
node_b.supervisor.state = NodeState::Running;
node_b
.transports
.insert(transport_id_b, TransportHandle::Udp(transport_b));
@@ -91,9 +91,9 @@ async fn test_adopted_udp_traversal_completes_handshake() {
async fn test_failed_adopted_traversal_cleans_up_transport() {
let mut node = make_node();
let (packet_tx, packet_rx) = packet_channel(64);
node.packet_tx = Some(packet_tx);
node.supervisor.packet_tx = Some(packet_tx);
node.packet_rx = Some(packet_rx);
node.state = NodeState::Running;
node.supervisor.state = NodeState::Running;
node.index_allocator = IndexAllocator::with_max_attempts(0);
let peer = make_node();
@@ -121,15 +121,14 @@ async fn test_failed_adopted_traversal_cleans_up_transport() {
async fn test_adopted_traversal_skips_already_connected_peer() {
let mut node = make_node();
let (packet_tx, packet_rx) = packet_channel(64);
node.packet_tx = Some(packet_tx);
node.supervisor.packet_tx = Some(packet_tx);
node.packet_rx = Some(packet_rx);
node.state = NodeState::Running;
node.supervisor.state = NodeState::Running;
let transport_id = TransportId::new(1);
let link_id = LinkId::new(1);
let (conn, peer_identity) = make_completed_connection(&mut node, link_id, transport_id, 1_000);
let peer_identity = seed_completed_connection(&mut node, link_id, transport_id, 1_000);
let peer_node_addr = *peer_identity.node_addr();
node.add_connection(conn).unwrap();
node.promote_connection(link_id, peer_identity, 2_000)
.unwrap();
@@ -180,17 +179,17 @@ async fn test_third_peer_can_handshake_via_adopted_transport_socket() {
let (packet_tx_b, packet_rx_b) = packet_channel(64);
let (packet_tx_c, packet_rx_c) = packet_channel(64);
node_a.packet_tx = Some(packet_tx_a.clone());
node_a.supervisor.packet_tx = Some(packet_tx_a.clone());
node_a.packet_rx = Some(packet_rx_a);
node_a.state = NodeState::Running;
node_a.supervisor.state = NodeState::Running;
node_b.packet_tx = Some(packet_tx_b.clone());
node_b.supervisor.packet_tx = Some(packet_tx_b.clone());
node_b.packet_rx = Some(packet_rx_b);
node_b.state = NodeState::Running;
node_b.supervisor.state = NodeState::Running;
node_c.packet_tx = Some(packet_tx_c.clone());
node_c.supervisor.packet_tx = Some(packet_tx_c.clone());
node_c.packet_rx = Some(packet_rx_c);
node_c.state = NodeState::Running;
node_c.supervisor.state = NodeState::Running;
let mut transport_a = UdpTransport::new(transport_id_a, None, udp_config.clone(), packet_tx_a);
transport_a.start_async().await.unwrap();
@@ -300,9 +299,9 @@ async fn test_adopted_udp_inherits_mtu_from_single_primary_config() {
let mut node = make_node_with(config);
let (packet_tx, packet_rx) = packet_channel(64);
node.packet_tx = Some(packet_tx);
node.supervisor.packet_tx = Some(packet_tx);
node.packet_rx = Some(packet_rx);
node.state = NodeState::Running;
node.supervisor.state = NodeState::Running;
let peer = make_node();
let adopted_socket = std::net::UdpSocket::bind("127.0.0.1:0").unwrap();
@@ -350,9 +349,9 @@ async fn test_adopted_udp_inherits_mtu_from_named_primary_config() {
let mut node = make_node_with(config);
let (packet_tx, packet_rx) = packet_channel(64);
node.packet_tx = Some(packet_tx);
node.supervisor.packet_tx = Some(packet_tx);
node.packet_rx = Some(packet_rx);
node.state = NodeState::Running;
node.supervisor.state = NodeState::Running;
let peer = make_node();
let adopted_socket = std::net::UdpSocket::bind("127.0.0.1:0").unwrap();
src/node/tests/decrypt_failure.rs
+3 -4
View File
@@ -1,6 +1,6 @@
//! Tests for the consecutive-decrypt-failure threshold force-removal path.
//!
//! Covers `Node::handle_decrypt_failure` (in `node/handlers/encrypted.rs`),
//! Covers `Node::handle_decrypt_failure` (in `node/dataplane/encrypted.rs`),
//! which increments `ActivePeer::increment_decrypt_failures` on each AEAD
//! verification failure and force-removes the peer once
//! `DECRYPT_FAILURE_THRESHOLD` consecutive failures are observed. The
@@ -18,7 +18,7 @@ use super::*;
/// the full `peers_by_index` cleanup path (not just the bare `peers` table).
#[test]
fn test_decrypt_failure_threshold_removes_peer() {
// Threshold constant in node/handlers/encrypted.rs (kept in sync with
// Threshold constant in node/dataplane/encrypted.rs (kept in sync with
// production code; see DECRYPT_FAILURE_THRESHOLD).
const THRESHOLD: u32 = 20;
@@ -28,10 +28,9 @@ fn test_decrypt_failure_threshold_removes_peer() {
// Build a fully-promoted active peer with our_index/transport_id set
// so peers_by_index is populated by promote_connection.
let (conn, identity) = make_completed_connection(&mut node, link_id, transport_id, 1_000);
let identity = seed_completed_connection(&mut node, link_id, transport_id, 1_000);
let node_addr = *identity.node_addr();
node.add_connection(conn).unwrap();
node.promote_connection(link_id, identity, 2_000).unwrap();
// Sanity: peer is registered and indexed.
src/node/tests/disconnect.rs
+2 -2
View File
@@ -6,7 +6,7 @@
use super::spanning_tree::*;
use super::*;
use crate::protocol::{Disconnect, DisconnectReason};
use crate::proto::fmp::{Disconnect, DisconnectReason};
/// 3-node chain: middle node disconnects one peer.
///
@@ -295,7 +295,7 @@ async fn test_disconnect_clears_session() {
);
// Node 0 sends Disconnect to node 1.
let disconnect = crate::protocol::Disconnect::new(DisconnectReason::Shutdown);
let disconnect = crate::proto::fmp::Disconnect::new(DisconnectReason::Shutdown);
nodes[0]
.node
.send_encrypted_link_message(&node1_addr, &disconnect.encode())
src/node/tests/discovery.rs
+86 -59
View File
@@ -5,9 +5,8 @@
//! response routing.
use super::*;
use crate::node::RecentRequest;
use crate::protocol::{LookupRequest, LookupResponse};
use crate::tree::TreeCoordinate;
use crate::proto::lookup::{LookupRequest, LookupResponse, RecentRequest};
use crate::proto::stp::TreeCoordinate;
use spanning_tree::{
cleanup_nodes, generate_random_edges, lock_large_network_test, process_available_packets,
run_tree_test, verify_tree_convergence,
@@ -23,7 +22,7 @@ async fn test_request_decode_error() {
let from = make_node_addr(0xAA);
// Too-short payload: should log error and return without panic
node.handle_lookup_request(&from, &[0x00; 5]).await;
assert!(node.recent_requests.is_empty());
assert!(node.lookup.recent_requests.is_empty());
}
#[tokio::test]
@@ -39,11 +38,11 @@ async fn test_request_dedup() {
// First request: accepted
node.handle_lookup_request(&from, payload).await;
assert_eq!(node.recent_requests.len(), 1);
assert_eq!(node.lookup.recent_requests.len(), 1);
// Duplicate request: dropped
node.handle_lookup_request(&from, payload).await;
assert_eq!(node.recent_requests.len(), 1);
assert_eq!(node.lookup.recent_requests.len(), 1);
}
#[tokio::test]
@@ -61,7 +60,7 @@ async fn test_request_target_is_self() {
// Should succeed without panic (response send will fail silently
// since we have no peers to route toward origin)
node.handle_lookup_request(&from, payload).await;
assert!(node.recent_requests.contains_key(&777));
assert!(node.lookup.recent_requests.contains_key(&777));
}
#[tokio::test]
@@ -77,7 +76,7 @@ async fn test_request_ttl_zero_not_forwarded() {
node.handle_lookup_request(&from, payload).await;
// Request recorded, but not forwarded (TTL=0, and no peers anyway)
assert!(node.recent_requests.contains_key(&666));
assert!(node.lookup.recent_requests.contains_key(&666));
}
// ============================================================================
@@ -115,7 +114,7 @@ async fn test_response_originator_caches_route() {
let payload = &response.encode()[1..]; // skip msg_type
// No entry in recent_requests for 555 → we're the originator
assert!(!node.recent_requests.contains_key(&555));
assert!(!node.lookup.recent_requests.contains_key(&555));
node.handle_lookup_response(&from, payload).await;
@@ -149,7 +148,8 @@ async fn test_response_transit_needs_recent_request() {
.duration_since(std::time::UNIX_EPOCH)
.unwrap()
.as_millis() as u64;
node.recent_requests
node.lookup
.recent_requests
.insert(444, RecentRequest::new(make_node_addr(0xDD), now_ms));
// Handle response — should try to reverse-path forward to 0xDD
@@ -319,14 +319,16 @@ async fn test_recent_request_expiry() {
.as_millis() as u64;
// Insert an old request (11 seconds ago)
node.recent_requests
node.lookup
.recent_requests
.insert(123, RecentRequest::new(make_node_addr(1), now_ms - 11_000));
// Insert a recent request
node.recent_requests
node.lookup
.recent_requests
.insert(456, RecentRequest::new(make_node_addr(2), now_ms));
assert_eq!(node.recent_requests.len(), 2);
assert_eq!(node.lookup.recent_requests.len(), 2);
// Trigger purge via a new lookup request
let target = make_node_addr(0xBB);
@@ -338,9 +340,9 @@ async fn test_recent_request_expiry() {
.await;
// Old entry (123) should be purged, recent entry (456) and new entry (789) kept
assert!(!node.recent_requests.contains_key(&123));
assert!(node.recent_requests.contains_key(&456));
assert!(node.recent_requests.contains_key(&789));
assert!(!node.lookup.recent_requests.contains_key(&123));
assert!(node.lookup.recent_requests.contains_key(&456));
assert!(node.lookup.recent_requests.contains_key(&789));
}
// ============================================================================
@@ -379,7 +381,7 @@ async fn test_request_forwarding_two_node() {
// Node1 should have recorded the request
assert!(
nodes[1].node.recent_requests.contains_key(&42),
nodes[1].node.lookup.recent_requests.contains_key(&42),
"Node 1 should have recorded the forwarded request"
);
@@ -447,13 +449,13 @@ async fn test_request_three_node_chain() {
// Node1 should have been a transit node (has the request_id in recent_requests)
assert!(
!nodes[1].node.recent_requests.is_empty(),
!nodes[1].node.lookup.recent_requests.is_empty(),
"Node 1 should have recorded the forwarded request"
);
// Node2 should have received the request (it's the target)
assert!(
!nodes[2].node.recent_requests.is_empty(),
!nodes[2].node.lookup.recent_requests.is_empty(),
"Node 2 should have received the request"
);
@@ -502,7 +504,7 @@ async fn test_request_dedup_convergent_paths() {
// Node2 (the target) must have received the request
assert!(
nodes[2].node.recent_requests.contains_key(&300),
nodes[2].node.lookup.recent_requests.contains_key(&300),
"Node 2 (target) should have received the request"
);
@@ -958,28 +960,28 @@ async fn test_originator_lookup_response_keeps_tighter_path_mtu_lookup() {
/// Pin the iterate-filter-queue contract of `run_open_discovery_sweep`.
///
/// Builds a `Node` with `nostr.policy = Open` and an empty peer list,
/// then injects three cached adverts into a test `NostrDiscovery` and
/// then injects three cached adverts into a test `NostrRendezvous` and
/// asserts the sweep:
/// - queues a retry for an eligible (unknown, not-self) advert,
/// - skips the advert whose author is our own node identity, and
/// - skips the advert whose author is an already-connected peer.
///
/// Uses `NostrDiscovery::new_for_test()` and `insert_advert_for_test()`
/// Uses `NostrRendezvous::new_for_test()` and `insert_advert_for_test()`
/// (both `#[cfg(test)]`-gated test escape hatches in
/// `src/discovery/nostr/runtime.rs`) to populate the cache without
/// requiring live relay subscriptions.
#[tokio::test]
async fn test_open_discovery_sweep_queues_eligible_skips_filtered() {
use crate::config::NostrDiscoveryPolicy;
use crate::discovery::nostr::{NostrDiscovery, OverlayEndpointAdvert, OverlayTransportKind};
use crate::config::NostrRendezvousPolicy;
use crate::nostr::{NostrRendezvous, OverlayEndpointAdvert, OverlayTransportKind};
use crate::peer::ActivePeer;
use crate::transport::LinkId;
use std::sync::Arc;
// Build node with open-discovery enabled.
let mut config = crate::Config::new();
config.node.discovery.nostr.enabled = true;
config.node.discovery.nostr.policy = NostrDiscoveryPolicy::Open;
config.node.rendezvous.nostr.enabled = true;
config.node.rendezvous.nostr.policy = NostrRendezvousPolicy::Open;
let mut node = crate::Node::new(config).unwrap();
// Identity of an already-connected peer; insert into node.peers
@@ -1002,8 +1004,8 @@ async fn test_open_discovery_sweep_queues_eligible_skips_filtered() {
let self_npub = crate::encode_npub(&node.identity().pubkey());
let self_node_addr = *node.identity().node_addr();
// Build a NostrDiscovery test instance and inject the three adverts.
let bootstrap = Arc::new(NostrDiscovery::new_for_test());
// Build a NostrRendezvous test instance and inject the three adverts.
let bootstrap = Arc::new(NostrRendezvous::new_for_test());
let endpoint = OverlayEndpointAdvert {
transport: OverlayTransportKind::Udp,
addr: "203.0.113.7:2121".to_string(),
@@ -1014,36 +1016,57 @@ async fn test_open_discovery_sweep_queues_eligible_skips_filtered() {
.unwrap_or(0);
for npub in [&eligible_npub, &connected_npub, &self_npub] {
let advert =
NostrDiscovery::cached_advert_for_test(npub.clone(), endpoint.clone(), now_secs);
NostrRendezvous::cached_advert_for_test(npub.clone(), endpoint.clone(), now_secs);
bootstrap.insert_advert_for_test(npub.clone(), advert).await;
}
// The sweep now runs through the gate-checked reconciler overlay layer,
// which is inert unless the node is Running/Degraded. In production the
// sweep fires only from the rx_loop tick (which spins after `start()`
// returns Running), so drive the node into `Running` to reflect that.
node.supervisor.state = crate::node::NodeState::Running;
// Run the sweep.
node.run_open_discovery_sweep(&bootstrap, Some(3_600), "test")
.await;
node.run_open_discovery_sweep(&bootstrap, Some(3_600)).await;
// Eligible peer was queued.
assert!(
node.retry_pending.contains_key(&eligible_node_addr),
node.peering
.reconciler
.retry_pending
.contains_key(&eligible_node_addr),
"eligible advert should be queued for retry"
);
let queued = node.retry_pending.get(&eligible_node_addr).unwrap();
let queued = node
.peering
.reconciler
.retry_pending
.get(&eligible_node_addr)
.unwrap();
assert_eq!(queued.peer_config.npub, eligible_npub);
// Connected-peer skip filter held.
assert!(
!node.retry_pending.contains_key(&connected_node_addr),
!node
.peering
.reconciler
.retry_pending
.contains_key(&connected_node_addr),
"advert for already-connected peer must not be queued"
);
// Self skip filter held.
assert!(
!node.retry_pending.contains_key(&self_node_addr),
!node
.peering
.reconciler
.retry_pending
.contains_key(&self_node_addr),
"advert authored by own node must not be queued"
);
// Exactly one queued entry from the three injected adverts.
assert_eq!(node.retry_pending.len(), 1);
assert_eq!(node.peering.reconciler.retry_pending.len(), 1);
}
// ============================================================================
@@ -1053,17 +1076,17 @@ async fn test_open_discovery_sweep_queues_eligible_skips_filtered() {
/// Pin the per-attempt timeout sequence in `check_pending_lookups`.
///
/// Drives the state machine deterministically through the default
/// `node.discovery.attempt_timeouts_secs = [1, 2, 4, 8]` sequence.
/// `node.lookup.attempt_timeouts_secs = [1, 2, 4, 8]` sequence.
/// Asserts:
/// 1. **Sequence timing** — retries fire at the cumulative deadlines
/// (t=1100ms, 3100ms, 7100ms) and unreachable at t=15100ms.
/// 2. **Fresh `initiate_lookup` per attempt** — `req_initiated` counter
/// increments by exactly one on each retry. The actual `request_id`
/// is generated by `LookupRequest::generate(...)` via `rand::random()`
/// inside `initiate_lookup` and is not stored on the originator
/// side, so per-attempt freshness is verified indirectly: each
/// `req_initiated` increment corresponds to one fresh
/// `LookupRequest::generate` call.
/// is drawn via `rand::rng().random()` at the shell inside
/// `initiate_lookup` and passed to `LookupRequest::new(...)`; it is
/// not stored on the originator side, so per-attempt freshness is
/// verified indirectly: each `req_initiated` increment corresponds
/// to one fresh `initiate_lookup` call.
/// 3. **Final-timeout state transitions** — `pending_lookups` entry is
/// removed, `discovery.resp_timed_out` counter ticks, queued packet
/// is drained, and an ICMPv6 Destination Unreachable frame is
@@ -1075,9 +1098,9 @@ async fn test_open_discovery_sweep_queues_eligible_skips_filtered() {
/// that `initiate_lookup` ran fresh on each attempt.
#[tokio::test]
async fn test_check_pending_lookups_default_sequence_unreachable() {
use crate::bloom::BloomFilter;
use crate::node::handlers::discovery::PendingLookup;
use crate::peer::ActivePeer;
use crate::proto::bloom::BloomFilter;
use crate::proto::lookup::PendingLookup;
use crate::transport::LinkId;
use std::sync::mpsc;
@@ -1086,14 +1109,14 @@ async fn test_check_pending_lookups_default_sequence_unreachable() {
// Default attempt_timeouts_secs is [1, 2, 4, 8]. Confirm so the test
// cannot silently drift if the default changes.
assert_eq!(
node.config().node.discovery.attempt_timeouts_secs,
node.config().node.lookup.attempt_timeouts_secs,
vec![1, 2, 4, 8],
"test pins the [1,2,4,8] default; update the test if the default changes"
);
// Inject a TUN sender so `send_icmpv6_dest_unreachable` is observable.
let (tun_tx, tun_rx) = mpsc::channel::<Vec<u8>>();
node.tun_tx = Some(tun_tx);
node.supervisor.tun_tx = Some(tun_tx);
// Build a target identity (the unreachable destination).
let target_identity = Identity::generate();
@@ -1119,7 +1142,7 @@ async fn test_check_pending_lookups_default_sequence_unreachable() {
// as its parent. `is_tree_peer` checks both directions — the child
// direction (peer.parent_id == self.node_addr) is what we exercise.
let our_addr = *node.node_addr();
let peer_decl = crate::tree::ParentDeclaration::new(peer_addr, our_addr, 1, 0);
let peer_decl = crate::proto::stp::ParentDeclaration::new(peer_addr, our_addr, 1, 0);
let peer_coords = TreeCoordinate::from_addrs(vec![peer_addr, our_addr]).unwrap();
node.tree_state_mut().update_peer(peer_decl, peer_coords);
assert!(node.is_tree_peer(&peer_addr), "peer must be a tree peer");
@@ -1145,16 +1168,18 @@ async fn test_check_pending_lookups_default_sequence_unreachable() {
// Inject a PendingLookup directly: attempt=1, last_sent_ms=0. This
// mirrors the post-condition of a successful `maybe_initiate_lookup`
// at t=0 without depending on wall-clock-derived `Self::now_ms()`.
node.pending_lookups
node.lookup
.pending_lookups
.insert(target_addr, PendingLookup::new(0));
let baseline_initiated = node.metrics().discovery.req_initiated.get();
let baseline_timed_out = node.metrics().discovery.resp_timed_out.get();
let baseline_initiated = node.metrics().lookup.req_initiated.get();
let baseline_timed_out = node.metrics().lookup.resp_timed_out.get();
// --- t = 1100ms: first retry deadline (1*1000) ---
node.check_pending_lookups(1100).await;
{
let entry = node
.lookup
.pending_lookups
.get(&target_addr)
.expect("still pending");
@@ -1162,7 +1187,7 @@ async fn test_check_pending_lookups_default_sequence_unreachable() {
assert_eq!(entry.last_sent_ms, 1100);
}
assert_eq!(
node.metrics().discovery.req_initiated.get(),
node.metrics().lookup.req_initiated.get(),
baseline_initiated + 1,
"retry #1 must invoke initiate_lookup exactly once"
);
@@ -1171,6 +1196,7 @@ async fn test_check_pending_lookups_default_sequence_unreachable() {
node.check_pending_lookups(3100).await;
{
let entry = node
.lookup
.pending_lookups
.get(&target_addr)
.expect("still pending");
@@ -1178,7 +1204,7 @@ async fn test_check_pending_lookups_default_sequence_unreachable() {
assert_eq!(entry.last_sent_ms, 3100);
}
assert_eq!(
node.metrics().discovery.req_initiated.get(),
node.metrics().lookup.req_initiated.get(),
baseline_initiated + 2,
"retry #2 must invoke initiate_lookup exactly once more"
);
@@ -1187,6 +1213,7 @@ async fn test_check_pending_lookups_default_sequence_unreachable() {
node.check_pending_lookups(7100).await;
{
let entry = node
.lookup
.pending_lookups
.get(&target_addr)
.expect("still pending");
@@ -1194,7 +1221,7 @@ async fn test_check_pending_lookups_default_sequence_unreachable() {
assert_eq!(entry.last_sent_ms, 7100);
}
assert_eq!(
node.metrics().discovery.req_initiated.get(),
node.metrics().lookup.req_initiated.get(),
baseline_initiated + 3,
"retry #3 must invoke initiate_lookup exactly once more"
);
@@ -1202,16 +1229,16 @@ async fn test_check_pending_lookups_default_sequence_unreachable() {
// --- Just-before-final: at t=15099ms the 8s window is not yet reached ---
node.check_pending_lookups(15_099).await;
assert!(
node.pending_lookups.contains_key(&target_addr),
node.lookup.pending_lookups.contains_key(&target_addr),
"8s window not yet expired: pending_lookup must persist"
);
assert_eq!(
node.metrics().discovery.req_initiated.get(),
node.metrics().lookup.req_initiated.get(),
baseline_initiated + 3,
"no new attempt before final deadline"
);
assert_eq!(
node.metrics().discovery.resp_timed_out.get(),
node.metrics().lookup.resp_timed_out.get(),
baseline_timed_out,
"no timeout before final deadline"
);
@@ -1225,18 +1252,18 @@ async fn test_check_pending_lookups_default_sequence_unreachable() {
// Pending lookup is dropped.
assert!(
!node.pending_lookups.contains_key(&target_addr),
!node.lookup.pending_lookups.contains_key(&target_addr),
"final timeout must remove the pending_lookups entry"
);
// resp_timed_out counter ticked.
assert_eq!(
node.metrics().discovery.resp_timed_out.get(),
node.metrics().lookup.resp_timed_out.get(),
baseline_timed_out + 1,
"final timeout must increment discovery.resp_timed_out"
);
// No additional initiate_lookup on the timeout step.
assert_eq!(
node.metrics().discovery.req_initiated.get(),
node.metrics().lookup.req_initiated.get(),
baseline_initiated + 3,
"the final-timeout step must NOT call initiate_lookup"
);
src/node/tests/establish_chartests.rs
+1034
View File
File diff suppressed because it is too large Load Diff
src/node/tests/forwarding.rs
+208 -96
View File
@@ -5,9 +5,11 @@
//! multi-hop forwarding through live node topologies.
use super::*;
use crate::node::session_wire::{FSP_FLAG_CP, build_fsp_header};
use crate::protocol::{SessionAck, SessionDatagram, SessionSetup, encode_coords};
use crate::tree::TreeCoordinate;
use crate::proto::fsp::wire::{FSP_FLAG_CP, build_fsp_header};
use crate::proto::fsp::{SessionAck, SessionSetup};
use crate::proto::link::SessionDatagram;
use crate::proto::stp::TreeCoordinate;
use crate::proto::stp::encode_coords;
use spanning_tree::{
TestNode, cleanup_nodes, process_available_packets, run_tree_test, verify_tree_convergence,
};
@@ -70,10 +72,10 @@ async fn test_forwarding_ttl_one_local_delivery_is_not_gated() {
assert_eq!(fwd.ttl_exhausted_packets.get(), 0);
}
/// Acceptance: a datagram addressed to this node with ttl=0 is delivered
/// locally. The TTL governs forwarding, not delivery to the addressed host,
/// so the gate must sit after the local-delivery test — and the
/// `TtlExhausted` reject must not be charged for a delivered datagram.
/// The shell's half of the acceptance: a datagram addressed to this node with
/// ttl=0 reaches the session layer and is charged to `delivered`, not to the
/// `TtlExhausted` reject. The shell has its own TTL-shaped gates ahead of the
/// core, so the core-level test alone would not pin this.
#[tokio::test]
async fn test_forwarding_ttl_zero_local_delivery_is_not_gated() {
let mut node = make_node();
@@ -98,9 +100,10 @@ async fn test_forwarding_ttl_zero_local_delivery_is_not_gated() {
assert_eq!(fwd.drop_no_route_packets.get(), 0);
}
/// Acceptance: a transit datagram arriving with ttl=1 would leave with ttl=0,
/// so it is dropped here rather than transmitted. Reaching the routing step at
/// all (`drop_no_route`) would mean it had been handed to the forwarder.
/// The shell's half of the transit boundary: ttl=1 is charged to
/// `TtlExhausted` and never reaches the routing step, and ttl=2 clears the
/// gate — visible here as the no-route charge this peerless node produces
/// once the datagram gets that far.
#[tokio::test]
async fn test_forwarding_ttl_one_transit_dropped_before_routing() {
let mut node = make_node();
@@ -126,9 +129,6 @@ async fn test_forwarding_ttl_one_transit_dropped_before_routing() {
assert_eq!(fwd.delivered_packets.get(), 0);
}
/// The other side of the same boundary: ttl=2 clears the gate. This node has
/// no peers, so it fails at the routing step instead — which is the evidence
/// that the TTL gate passed it through.
#[tokio::test]
async fn test_forwarding_ttl_two_transit_clears_the_gate() {
let mut node = make_node();
@@ -366,6 +366,104 @@ async fn test_coord_cache_warming_encrypted_msg_no_coords() {
);
}
/// Cache warming is not gated on the TTL, case 1 of 2: a datagram addressed
/// to this node that arrives already exhausted is delivered, and its
/// plaintext coordinates still reach the cache.
///
/// The warming call sits ahead of the routing decision precisely so that it
/// is unconditional. The other warming tests all run at the default TTL of 64
/// and so cannot see a TTL-shaped gate around it; this one runs at zero.
/// `SessionSetup` is used rather than a CP-flagged encrypted message because
/// the local-delivery path caches coords from the latter itself, which would
/// mask a suppressed warming call.
#[tokio::test]
async fn test_coord_cache_warming_ttl_zero_local_delivery() {
let mut node = make_node();
let from = make_node_addr(0xAA);
let my_addr = *node.node_addr();
let src_addr = make_node_addr(0x01);
let root_addr = make_node_addr(0xF0);
let src_coords = TreeCoordinate::from_addrs(vec![src_addr, root_addr]).unwrap();
let dest_coords = TreeCoordinate::from_addrs(vec![my_addr, root_addr]).unwrap();
let setup = SessionSetup::new(src_coords, dest_coords);
let dg = SessionDatagram::new(src_addr, my_addr, setup.encode()).with_ttl(0);
let encoded = dg.encode();
let now_ms = std::time::SystemTime::now()
.duration_since(std::time::UNIX_EPOCH)
.unwrap()
.as_millis() as u64;
assert!(node.coord_cache().get(&src_addr, now_ms).is_none());
node.handle_session_datagram(&from, &encoded[1..], false)
.await;
assert_eq!(
node.metrics().forwarding.delivered_packets.get(),
1,
"ttl=0 addressed to this node must be delivered, not dropped"
);
let cached = node.coord_cache().get(&src_addr, now_ms);
assert!(
cached.is_some(),
"warming must not be gated on the TTL: a delivered ttl=0 datagram \
still carries usable coords"
);
assert_eq!(cached.unwrap().root_id(), &root_addr);
}
/// Cache warming is not gated on the TTL, case 2 of 2: a transit datagram
/// that is dropped for hop limit still contributes its plaintext coordinates.
///
/// This is the case the gate suppressed most visibly — the datagram never
/// reaches any other code that could cache coords, so the assertion below is
/// only satisfiable by the unconditional warming call. The `TtlExhausted`
/// charge is asserted alongside it to show the drop did happen, so the test
/// cannot be satisfied by the datagram merely surviving the TTL gate.
#[tokio::test]
async fn test_coord_cache_warming_ttl_zero_transit_drop() {
let mut node = make_node();
let from = make_node_addr(0xAA);
let src_addr = make_node_addr(0x01);
let dest_addr = make_node_addr(0x02);
let root_addr = make_node_addr(0xF0);
let src_coords = TreeCoordinate::from_addrs(vec![src_addr, root_addr]).unwrap();
let dest_coords = TreeCoordinate::from_addrs(vec![dest_addr, root_addr]).unwrap();
let setup = SessionSetup::new(src_coords, dest_coords);
let dg = SessionDatagram::new(src_addr, dest_addr, setup.encode()).with_ttl(0);
let encoded = dg.encode();
let now_ms = std::time::SystemTime::now()
.duration_since(std::time::UNIX_EPOCH)
.unwrap()
.as_millis() as u64;
assert!(node.coord_cache().get(&src_addr, now_ms).is_none());
assert!(node.coord_cache().get(&dest_addr, now_ms).is_none());
node.handle_session_datagram(&from, &encoded[1..], false)
.await;
assert_eq!(
node.metrics().forwarding.ttl_exhausted_packets.get(),
1,
"the transit datagram should have been dropped for hop limit"
);
assert!(
node.coord_cache().get(&src_addr, now_ms).is_some(),
"a transit datagram dropped for hop limit must still warm the cache \
with its source coords"
);
assert!(
node.coord_cache().get(&dest_addr, now_ms).is_some(),
"a transit datagram dropped for hop limit must still warm the cache \
with its destination coords"
);
}
// ============================================================================
// Integration Tests
// ============================================================================
@@ -518,27 +616,68 @@ async fn test_forwarding_hop_limit_prevents_infinite_loops() {
process_available_packets(&mut nodes).await;
}
// No panic, no infinite loop
// The datagram must actually have traversed the chain rather than dying
// somewhere en route: node 1 forwarded it and node 2 delivered it. These
// hold on both sides of the hop-limit change — the per-hop decrement is
// pinned by `test_forwarding_ttl_decrement_is_one_per_hop` below — but
// without them the test asserts nothing and would pass on a chain that
// dropped the datagram at the first hop.
assert_eq!(
nodes[1].node.metrics().forwarding.forwarded_packets.get(),
1,
"node 1 should have forwarded the transit datagram"
);
assert_eq!(
nodes[2].node.metrics().forwarding.delivered_packets.get(),
1,
"node 2 should have delivered the datagram addressed to it"
);
assert_eq!(
nodes[2]
.node
.metrics()
.forwarding
.ttl_exhausted_packets
.get(),
0,
"the destination must not charge a TTL drop for a datagram it delivers"
);
cleanup_nodes(&mut nodes).await;
}
/// Acceptance: a transit datagram arriving with ttl=2 leaves with ttl=1.
/// Acceptance for the hop limit as the *composed* system sees it: the shell's
/// TTL-shaped predicates in `node::dataplane::forwarding` driving the
/// authoritative rule in `Router::route`, across real links.
///
/// Pinned on a live 3-node chain (0 -- 1 -- 2) by where the datagram stops,
/// since the TTL that leaves node 0 is only observable through what the next
/// hop does with it. Both injections are transit at node 0 (external source,
/// destined for node 2), so node 1 is a forwarder in both.
/// The decision is split across those two files on this branch, and the
/// core-level tests exercise only one side of that seam. This is the coverage
/// that runs both together over a hop.
///
/// - ttl=2 in: node 0 must emit ttl=1, which node 1 (transit) drops. If node 0
/// emitted ttl=2 unchanged, node 1 would forward and node 2 would deliver.
/// - ttl=3 in: node 0 emits 2, node 1 emits 1, node 2 delivers (delivery is
/// not TTL-gated). If either hop decremented by more than one, the datagram
/// would have died at node 1 instead.
/// The TTL a node puts on the wire is not directly observable, so it is
/// bracketed from both sides by where the datagram comes to rest on a live
/// 3-node chain (0 -- 1 -- 2). Both injections are handed to node 0 as
/// transit — an external source, addressed to node 2 — so nodes 0 and 1 are
/// both forwarders and only node 2 is the addressed destination.
///
/// Together the two pin the decrement at exactly one per hop and the drop at
/// would-leave-zero.
/// - ttl=2 must reach node 1 as ttl=1 and stop there, because forwarding it
/// again would put it on the wire at zero. Emitting ttl=2 unchanged would
/// instead show up as a delivery at node 2.
/// - ttl=3 must survive both forwarders and be delivered at node 2, which
/// receives it at ttl=1 — delivery is not TTL-gated. Decrementing by more
/// than one per hop would have stopped it at node 1.
#[tokio::test]
async fn test_forwarding_ttl_decrement_is_one_per_hop() {
/// `(forwarded, ttl_exhausted, delivered)` for one node.
fn counts(node: &TestNode) -> (u64, u64, u64) {
let fwd = &node.node.metrics().forwarding;
(
fwd.forwarded_packets.get(),
fwd.ttl_exhausted_packets.get(),
fwd.delivered_packets.get(),
)
}
let edges = vec![(0, 1), (1, 2)];
let mut nodes = run_tree_test(3, &edges, false).await;
verify_tree_convergence(&nodes);
@@ -548,84 +687,57 @@ async fn test_forwarding_ttl_decrement_is_one_per_hop() {
let node2_addr = *nodes[2].node.node_addr();
let external_src = make_node_addr(0xEE);
// --- ttl=2: must die at node 1, one hop short of the destination ---
let dg =
SessionDatagram::new(external_src, node2_addr, vec![0x10, 0x00, 0x00, 0x00]).with_ttl(2);
let encoded = dg.encode();
nodes[0]
.node
.handle_session_datagram(&node0_addr, &encoded[1..], false)
.await;
for _ in 0..3 {
tokio::time::sleep(Duration::from_millis(50)).await;
process_available_packets(&mut nodes).await;
}
assert_eq!(
nodes[0].node.metrics().forwarding.forwarded_packets.get(),
1,
"node 0 should have forwarded the ttl=2 datagram"
);
assert_eq!(
nodes[1]
/// Hand a transit datagram to node 0 and let the chain settle.
async fn inject(nodes: &mut [TestNode], from: &NodeAddr, dg: SessionDatagram) {
let encoded = dg.encode();
nodes[0]
.node
.metrics()
.forwarding
.ttl_exhausted_packets
.get(),
1,
"node 1 should have received ttl=1 and dropped it as TTL-exhausted"
);
assert_eq!(
nodes[1].node.metrics().forwarding.forwarded_packets.get(),
0,
"node 1 must not forward a datagram that would leave with ttl=0"
);
assert_eq!(
nodes[2].node.metrics().forwarding.delivered_packets.get(),
0,
"node 2 must never see the ttl=2 datagram"
);
// --- ttl=3: must survive both transit hops and be delivered at node 2 ---
let dg =
SessionDatagram::new(external_src, node2_addr, vec![0x10, 0x00, 0x00, 0x00]).with_ttl(3);
let encoded = dg.encode();
nodes[0]
.node
.handle_session_datagram(&node0_addr, &encoded[1..], false)
.await;
for _ in 0..3 {
tokio::time::sleep(Duration::from_millis(50)).await;
process_available_packets(&mut nodes).await;
.handle_session_datagram(from, &encoded[1..], false)
.await;
for _ in 0..3 {
tokio::time::sleep(Duration::from_millis(50)).await;
process_available_packets(nodes).await;
}
}
let transit = |ttl: u8| {
SessionDatagram::new(external_src, node2_addr, vec![0x10, 0x00, 0x00, 0x00]).with_ttl(ttl)
};
// ttl=2 in: node 0 emits 1, node 1 has nothing left to emit.
inject(&mut nodes, &node0_addr, transit(2)).await;
assert_eq!(
nodes[0].node.metrics().forwarding.forwarded_packets.get(),
2,
counts(&nodes[0]),
(1, 0, 0),
"node 0 should have forwarded the ttl=2 datagram at ttl=1"
);
assert_eq!(
counts(&nodes[1]),
(0, 1, 0),
"node 1 should have received ttl=1 and dropped it rather than sending at ttl=0"
);
assert_eq!(
counts(&nodes[2]),
(0, 0, 0),
"node 2 must never see a datagram that started two hops away at ttl=2"
);
// ttl=3 in: node 0 emits 2, node 1 emits 1, node 2 delivers at ttl=1.
inject(&mut nodes, &node0_addr, transit(3)).await;
assert_eq!(
counts(&nodes[0]),
(2, 0, 0),
"node 0 should have forwarded the ttl=3 datagram too"
);
assert_eq!(
nodes[1].node.metrics().forwarding.forwarded_packets.get(),
1,
"node 1 should have forwarded the ttl=2 it received"
counts(&nodes[1]),
(1, 1, 0),
"node 1 should have forwarded the ttl=2 it received, and dropped nothing new"
);
assert_eq!(
nodes[1]
.node
.metrics()
.forwarding
.ttl_exhausted_packets
.get(),
1,
"node 1 should not have dropped the second datagram"
);
assert_eq!(
nodes[2].node.metrics().forwarding.delivered_packets.get(),
1,
"node 2 should have delivered the datagram that arrived with ttl=1"
counts(&nodes[2]),
(0, 0, 1),
"node 2 should have delivered the datagram that arrived at ttl=1"
);
cleanup_nodes(&mut nodes).await;
@@ -794,7 +906,7 @@ async fn test_forwarding_with_cache_warming_enables_routing() {
// ============================================================================
use crate::node::TransportDropState;
use crate::node::handlers::session::mark_ipv6_ecn_ce;
use crate::proto::fsp::mark_ipv6_ecn_ce;
use crate::transport::TransportId;
/// Build a minimal IPv6 header (40 bytes) with specified ECN bits.
src/node/tests/handshake.rs
+243 -111
View File
@@ -5,7 +5,7 @@ use super::*;
#[tokio::test]
async fn test_two_node_handshake_udp() {
use crate::config::UdpConfig;
use crate::node::wire::{
use crate::proto::fmp::wire::{
build_encrypted, build_established_header, build_msg1, prepend_inner_header,
};
use crate::transport::udp::UdpTransport;
@@ -53,19 +53,28 @@ async fn test_two_node_handshake_udp() {
let peer_b_node_addr = *peer_b_identity.node_addr();
let link_id_a = node_a.allocate_link_id();
let mut conn_a = PeerConnection::outbound(link_id_a, peer_b_identity, 1000);
// Allocate session index for A's outbound
let our_index_a = node_a.index_allocator.allocate().unwrap();
node_a
.seed_handshake_machine(
HandshakeSeed::outbound(link_id_a, peer_b_identity, 1000)
.with_our_index(our_index_a)
.with_transport_id(transport_id_a)
.with_source_addr(remote_addr_b.clone()),
)
.unwrap();
// Start handshake (generates Noise IK msg1)
let our_keypair_a = node_a.identity().keypair();
let noise_msg1 = conn_a
.start_handshake(our_keypair_a, node_a.startup_epoch(), 1000)
let startup_epoch_a = node_a.startup_epoch();
let noise_msg1 = node_a
.peer_machines
.get_mut(&link_id_a)
.unwrap()
.start_handshake(our_keypair_a, startup_epoch_a, 1000)
.unwrap();
conn_a.set_our_index(our_index_a);
conn_a.set_transport_id(transport_id_a);
conn_a.set_source_addr(remote_addr_b.clone());
// Build wire msg1 and track in node state
let wire_msg1 = build_msg1(our_index_a, &noise_msg1);
@@ -78,7 +87,6 @@ async fn test_two_node_handshake_udp() {
Duration::from_millis(100),
);
node_a.links.insert(link_id_a, link_a);
node_a.connections.insert(link_id_a, conn_a);
node_a
.pending_outbound
.insert((transport_id_a, our_index_a.as_u32()), link_id_a);
@@ -243,7 +251,7 @@ async fn test_two_node_handshake_udp() {
#[tokio::test]
async fn test_run_rx_loop_handshake() {
use crate::config::UdpConfig;
use crate::node::wire::build_msg1;
use crate::proto::fmp::wire::build_msg1;
use crate::transport::udp::UdpTransport;
use tokio::time::Duration;
@@ -285,8 +293,8 @@ async fn test_run_rx_loop_handshake() {
node_b.packet_rx = Some(packet_rx_b);
// Set node state to Running (transports need to be operational)
node_a.state = NodeState::Running;
node_b.state = NodeState::Running;
node_a.supervisor.state = NodeState::Running;
node_b.supervisor.state = NodeState::Running;
// === Phase 1: Node A initiates handshake to Node B ===
@@ -294,16 +302,24 @@ async fn test_run_rx_loop_handshake() {
let peer_b_node_addr = *peer_b_identity.node_addr();
let link_id_a = node_a.allocate_link_id();
let mut conn_a = PeerConnection::outbound(link_id_a, peer_b_identity, 1000);
let our_index_a = node_a.index_allocator.allocate().unwrap();
let our_keypair_a = node_a.identity().keypair();
let noise_msg1 = conn_a
.start_handshake(our_keypair_a, node_a.startup_epoch(), 1000)
node_a
.seed_handshake_machine(
HandshakeSeed::outbound(link_id_a, peer_b_identity, 1000)
.with_our_index(our_index_a)
.with_transport_id(transport_id_a)
.with_source_addr(remote_addr_b.clone()),
)
.unwrap();
let our_keypair_a = node_a.identity().keypair();
let startup_epoch_a = node_a.startup_epoch();
let noise_msg1 = node_a
.peer_machines
.get_mut(&link_id_a)
.unwrap()
.start_handshake(our_keypair_a, startup_epoch_a, 1000)
.unwrap();
conn_a.set_our_index(our_index_a);
conn_a.set_transport_id(transport_id_a);
conn_a.set_source_addr(remote_addr_b.clone());
let wire_msg1 = build_msg1(our_index_a, &noise_msg1);
@@ -315,7 +331,6 @@ async fn test_run_rx_loop_handshake() {
Duration::from_millis(100),
);
node_a.links.insert(link_id_a, link_a);
node_a.connections.insert(link_id_a, conn_a);
node_a
.pending_outbound
.insert((transport_id_a, our_index_a.as_u32()), link_id_a);
@@ -434,7 +449,7 @@ async fn test_run_rx_loop_handshake() {
#[tokio::test]
async fn test_cross_connection_both_initiate() {
use crate::config::UdpConfig;
use crate::node::wire::build_msg1;
use crate::proto::fmp::wire::build_msg1;
use crate::transport::udp::UdpTransport;
use tokio::time::{Duration, timeout};
@@ -483,15 +498,23 @@ async fn test_cross_connection_both_initiate() {
// Node A initiates to Node B
let link_id_a_out = node_a.allocate_link_id();
let mut conn_a = PeerConnection::outbound(link_id_a_out, peer_b_identity, 1000);
let our_index_a = node_a.index_allocator.allocate().unwrap();
let our_keypair_a = node_a.identity().keypair();
let noise_msg1_a = conn_a
.start_handshake(our_keypair_a, node_a.startup_epoch(), 1000)
node_a
.seed_handshake_machine(
HandshakeSeed::outbound(link_id_a_out, peer_b_identity, 1000)
.with_our_index(our_index_a)
.with_transport_id(transport_id_a)
.with_source_addr(remote_addr_b.clone()),
)
.unwrap();
let our_keypair_a = node_a.identity().keypair();
let startup_epoch_a = node_a.startup_epoch();
let noise_msg1_a = node_a
.peer_machines
.get_mut(&link_id_a_out)
.unwrap()
.start_handshake(our_keypair_a, startup_epoch_a, 1000)
.unwrap();
conn_a.set_our_index(our_index_a);
conn_a.set_transport_id(transport_id_a);
conn_a.set_source_addr(remote_addr_b.clone());
let wire_msg1_a = build_msg1(our_index_a, &noise_msg1_a);
@@ -506,22 +529,29 @@ async fn test_cross_connection_both_initiate() {
node_a
.addr_to_link
.insert((transport_id_a, remote_addr_b.clone()), link_id_a_out);
node_a.connections.insert(link_id_a_out, conn_a);
node_a
.pending_outbound
.insert((transport_id_a, our_index_a.as_u32()), link_id_a_out);
// Node B initiates to Node A
let link_id_b_out = node_b.allocate_link_id();
let mut conn_b = PeerConnection::outbound(link_id_b_out, peer_a_identity, 1000);
let our_index_b = node_b.index_allocator.allocate().unwrap();
let our_keypair_b = node_b.identity().keypair();
let noise_msg1_b = conn_b
.start_handshake(our_keypair_b, node_b.startup_epoch(), 1000)
node_b
.seed_handshake_machine(
HandshakeSeed::outbound(link_id_b_out, peer_a_identity, 1000)
.with_our_index(our_index_b)
.with_transport_id(transport_id_b)
.with_source_addr(remote_addr_a.clone()),
)
.unwrap();
let our_keypair_b = node_b.identity().keypair();
let startup_epoch_b = node_b.startup_epoch();
let noise_msg1_b = node_b
.peer_machines
.get_mut(&link_id_b_out)
.unwrap()
.start_handshake(our_keypair_b, startup_epoch_b, 1000)
.unwrap();
conn_b.set_our_index(our_index_b);
conn_b.set_transport_id(transport_id_b);
conn_b.set_source_addr(remote_addr_a.clone());
let wire_msg1_b = build_msg1(our_index_b, &noise_msg1_b);
@@ -536,7 +566,6 @@ async fn test_cross_connection_both_initiate() {
node_b
.addr_to_link
.insert((transport_id_b, remote_addr_a.clone()), link_id_b_out);
node_b.connections.insert(link_id_b_out, conn_b);
node_b
.pending_outbound
.insert((transport_id_b, our_index_b.as_u32()), link_id_b_out);
@@ -662,17 +691,24 @@ async fn test_stale_connection_cleanup() {
// Create outbound connection with a timestamp far in the past
let past_time_ms = 1000; // A very early timestamp
let link_id = node.allocate_link_id();
let mut conn = PeerConnection::outbound(link_id, peer_identity, past_time_ms);
// Allocate session index and set transport info
let our_index = node.index_allocator.allocate().unwrap();
node.seed_handshake_machine(
HandshakeSeed::outbound(link_id, peer_identity, past_time_ms)
.with_our_index(our_index)
.with_transport_id(transport_id)
.with_source_addr(remote_addr.clone()),
)
.unwrap();
let our_keypair = node.identity().keypair();
let _noise_msg1 = conn
.start_handshake(our_keypair, node.startup_epoch(), past_time_ms)
let startup_epoch = node.startup_epoch();
let _noise_msg1 = node
.peer_machines
.get_mut(&link_id)
.unwrap()
.start_handshake(our_keypair, startup_epoch, past_time_ms)
.unwrap();
conn.set_our_index(our_index);
conn.set_transport_id(transport_id);
conn.set_source_addr(remote_addr.clone());
// Set up all the state that initiate_peer_connection would create
let link = Link::connectionless(
@@ -685,7 +721,6 @@ async fn test_stale_connection_cleanup() {
node.links.insert(link_id, link);
node.addr_to_link
.insert((transport_id, remote_addr.clone()), link_id);
node.connections.insert(link_id, conn);
node.pending_outbound
.insert((transport_id, our_index.as_u32()), link_id);
@@ -741,17 +776,23 @@ async fn test_failed_connection_cleanup() {
.map(|d| d.as_millis() as u64)
.unwrap_or(0);
let link_id = node.allocate_link_id();
let mut conn = PeerConnection::outbound(link_id, peer_identity, now_ms);
let our_index = node.index_allocator.allocate().unwrap();
node.seed_handshake_machine(
HandshakeSeed::outbound(link_id, peer_identity, now_ms)
.with_our_index(our_index)
.with_transport_id(transport_id)
.with_source_addr(remote_addr.clone()),
)
.unwrap();
let our_keypair = node.identity().keypair();
let _noise_msg1 = conn
.start_handshake(our_keypair, node.startup_epoch(), now_ms)
let startup_epoch = node.startup_epoch();
let _noise_msg1 = node
.peer_machines
.get_mut(&link_id)
.unwrap()
.start_handshake(our_keypair, startup_epoch, now_ms)
.unwrap();
conn.set_our_index(our_index);
conn.set_transport_id(transport_id);
conn.set_source_addr(remote_addr.clone());
conn.mark_failed(); // Simulate send failure
let link = Link::connectionless(
link_id,
@@ -763,10 +804,27 @@ async fn test_failed_connection_cleanup() {
node.links.insert(link_id, link);
node.addr_to_link
.insert((transport_id, remote_addr.clone()), link_id);
node.connections.insert(link_id, conn);
node.pending_outbound
.insert((transport_id, our_index.as_u32()), link_id);
// Simulate a stored-handshake send failure through the control machine —
// the failure carrier the stale-connection sweep now reads (the leg no
// longer carries a failed phase of its own).
{
let machine = node
.peer_machines
.get_mut(&link_id)
.expect("machine seeded by the handshake seeder");
let alloc = &mut node.index_allocator;
let actions = machine.step(
crate::peer::machine::PeerEvent::HandshakeSendFailed,
now_ms,
alloc,
);
assert!(actions.is_empty());
assert!(machine.is_failed());
}
assert_eq!(node.connection_count(), 1);
// Failed connections should be cleaned up immediately regardless of age
@@ -788,7 +846,7 @@ async fn test_failed_connection_cleanup() {
/// Test that msg1 bytes are stored on connection for resend.
#[tokio::test]
async fn test_msg1_stored_for_resend() {
use crate::node::wire::build_msg1;
use crate::proto::fmp::wire::build_msg1;
let mut node = make_node();
let transport_id = TransportId::new(1);
@@ -801,26 +859,24 @@ async fn test_msg1_stored_for_resend() {
.map(|d| d.as_millis() as u64)
.unwrap_or(0);
let link_id = node.allocate_link_id();
let mut conn = PeerConnection::outbound(link_id, peer_identity, now_ms);
let mut conn = outbound_leg(link_id, peer_identity, now_ms);
let our_index = node.index_allocator.allocate().unwrap();
let our_keypair = node.identity().keypair();
let noise_msg1 = conn
.start_handshake(our_keypair, node.startup_epoch(), now_ms)
.unwrap();
conn.set_our_index(our_index);
conn.set_transport_id(transport_id);
conn.set_source_addr(remote_addr.clone());
conn.set_conn_our_index(our_index);
conn.set_conn_transport_id(transport_id);
conn.set_conn_source_addr(remote_addr.clone());
// Build wire msg1 and store it (as initiate_peer_connection does)
let wire_msg1 = build_msg1(our_index, &noise_msg1);
let resend_interval = node.config().node.rate_limit.handshake_resend_interval_ms;
conn.set_handshake_msg1(wire_msg1.clone(), now_ms + resend_interval);
conn.set_conn_handshake_msg1(wire_msg1.clone(), now_ms + resend_interval);
// Verify stored msg1 matches what was built
assert_eq!(conn.handshake_msg1().unwrap(), &wire_msg1);
assert_eq!(conn.resend_count(), 0);
assert!(conn.next_resend_at_ms() > now_ms);
assert_eq!(conn.conn_handshake_msg1().unwrap(), &wire_msg1);
}
/// Test that resend scheduling respects max_resends and backoff.
@@ -834,20 +890,17 @@ async fn test_resend_scheduling() {
let now_ms = 100_000u64; // Use a fixed time for predictable testing
let link_id = node.allocate_link_id();
let mut conn = PeerConnection::outbound(link_id, peer_identity, now_ms);
let mut conn = outbound_leg(link_id, peer_identity, now_ms);
let our_index = node.index_allocator.allocate().unwrap();
let our_keypair = node.identity().keypair();
let noise_msg1 = conn
.start_handshake(our_keypair, node.startup_epoch(), now_ms)
.unwrap();
conn.set_our_index(our_index);
conn.set_transport_id(transport_id);
conn.set_source_addr(remote_addr.clone());
conn.set_conn_source_addr(remote_addr.clone());
// Store msg1 with first resend at now + 1000ms
let wire_msg1 = crate::node::wire::build_msg1(our_index, &noise_msg1);
conn.set_handshake_msg1(wire_msg1, now_ms + 1000);
let wire_msg1 = crate::proto::fmp::wire::build_msg1(our_index, &noise_msg1);
let link = Link::connectionless(
link_id,
@@ -858,73 +911,152 @@ async fn test_resend_scheduling() {
);
node.links.insert(link_id, link);
node.addr_to_link
.insert((transport_id, remote_addr), link_id);
.insert((transport_id, remote_addr.clone()), link_id);
node.pending_outbound
.insert((transport_id, our_index.as_u32()), link_id);
node.connections.insert(link_id, conn);
// Before resend time: nothing should happen (no transport = can't send,
// but the filter should exclude it because now < next_resend_at)
node.resend_pending_handshakes(now_ms + 500).await;
let conn = node.connections.get(&link_id).unwrap();
assert_eq!(conn.resend_count(), 0, "No resend before scheduled time");
// The msg1-resend counter and its due timer live on the per-peer machine,
// which also carries the pending connection. Dial it to `SentMsg1`
// (connectionless: no connect step) and arm its retransmit timer at
// now + 1000ms, mirroring what a real dial arms.
let mut machine =
crate::peer::machine::PeerMachine::new_outbound(link_id, peer_identity, now_ms);
let _ = machine.step(
crate::peer::machine::PeerEvent::Dial {
transport_id,
remote_addr: remote_addr.clone(),
peer_identity,
connection_oriented: false,
},
now_ms,
&mut node.index_allocator,
);
// The msg1 wire lives on the machine's carrier (the retransmit driver's
// resend source), mirroring `prepare_outbound_msg1`.
machine.set_conn_handshake_msg1(wire_msg1, now_ms + 1000);
machine.set_conn_our_index(our_index);
machine.set_conn_transport_id(transport_id);
machine.set_leg(conn.take_leg().unwrap());
node.peer_machines.insert(link_id, machine);
node.peer_timers.entry(link_id).or_default().insert(
crate::peer::machine::TimerKind::HandshakeRetransmit,
now_ms + 1000,
);
// At resend time: would resend if transport existed. Without transport,
// the send fails silently and resend_count stays at 0.
// This tests the filtering logic — the connection IS a candidate.
node.resend_pending_handshakes(now_ms + 1000).await;
// No transport registered, so send fails — count stays 0.
// That's the expected behavior (transport absence is a transient condition).
let conn = node.connections.get(&link_id).unwrap();
// Before the scheduled time the timer isn't due, so nothing fires.
node.drive_peer_timers(now_ms + 500).await;
assert_eq!(
conn.resend_count(),
node.connection_resend_count(link_id),
0,
"No transport means no resend recorded"
"No resend before scheduled time"
);
// At the scheduled time the timer is due, but no transport is registered so
// the send fails. Record-on-success: the count does NOT advance (and the
// connection is not marked failed) — a failed resend just retries next tick.
node.drive_peer_timers(now_ms + 1000).await;
assert_eq!(
node.connection_resend_count(link_id),
0,
"Failed send records no resend"
);
}
/// Test that msg2 is stored on PeerConnection for responder resend.
#[test]
fn test_msg2_stored_on_connection() {
let mut conn = PeerConnection::inbound(LinkId::new(1), 1000);
/// Test that the timer driver reaps an outbound leg whose machine
/// `HandshakeTimeout` timer has come due (the timeout fold). The reap re-checks
/// the shell `is_timed_out` predicate, then tears the connection down exactly as
/// the old `check_timeouts` Teardown path did.
#[tokio::test]
async fn test_handshake_timeout_drive() {
let mut node = make_node();
let transport_id = TransportId::new(1);
let peer_identity = make_peer_identity();
let remote_addr = TransportAddr::from_string("10.0.0.2:2121");
assert!(conn.handshake_msg2().is_none());
let dial_ms = 1000u64;
let link_id = node.allocate_link_id();
let mut conn = outbound_leg(link_id, peer_identity, dial_ms);
let our_index = node.index_allocator.allocate().unwrap();
let our_keypair = node.identity().keypair();
let _ = conn
.start_handshake(our_keypair, node.startup_epoch(), dial_ms)
.unwrap();
conn.set_conn_source_addr(remote_addr.clone());
let msg2_bytes = vec![0x01, 0x02, 0x03, 0x04];
conn.set_handshake_msg2(msg2_bytes.clone());
let link = Link::connectionless(
link_id,
transport_id,
remote_addr.clone(),
LinkDirection::Outbound,
Duration::from_millis(100),
);
node.links.insert(link_id, link);
node.addr_to_link
.insert((transport_id, remote_addr.clone()), link_id);
node.pending_outbound
.insert((transport_id, our_index.as_u32()), link_id);
assert_eq!(conn.handshake_msg2().unwrap(), &msg2_bytes);
// Machine in SentMsg1, carrying the pending connection, with a
// HandshakeTimeout timer armed at dial + 30s.
let mut machine =
crate::peer::machine::PeerMachine::new_outbound(link_id, peer_identity, dial_ms);
let _ = machine.step(
crate::peer::machine::PeerEvent::Dial {
transport_id,
remote_addr: remote_addr.clone(),
peer_identity,
connection_oriented: false,
},
dial_ms,
&mut node.index_allocator,
);
machine.set_conn_our_index(our_index);
machine.set_conn_transport_id(transport_id);
machine.set_leg(conn.take_leg().unwrap());
node.peer_machines.insert(link_id, machine);
node.peer_timers.entry(link_id).or_default().insert(
crate::peer::machine::TimerKind::HandshakeTimeout,
dial_ms + 30_000,
);
assert_eq!(node.connection_count(), 1);
// Well past dial + 30s: the timer is due and the leg is idle-timed-out.
node.drive_peer_timers(dial_ms + 100_000).await;
assert_eq!(
node.connection_count(),
0,
"Timed-out leg reaped by the timer drive"
);
assert_eq!(node.index_allocator.count(), 0, "Session index freed");
assert!(
!node.peer_machines.contains_key(&link_id),
"Control machine dropped with the reaped connection"
);
assert!(
!node.peer_timers.contains_key(&link_id),
"Timer store dropped with the reaped connection"
);
}
/// Test that resend_count and next_resend_at_ms track correctly.
/// Test that msg2 is stored on the control machine's carrier for responder resend.
#[test]
fn test_resend_count_tracking() {
let peer_identity = make_peer_identity();
let mut conn = PeerConnection::outbound(LinkId::new(1), peer_identity, 1000);
fn test_msg2_stored_on_connection() {
let mut machine = crate::peer::machine::PeerMachine::new_inbound(LinkId::new(1), 1000);
assert_eq!(conn.resend_count(), 0);
assert_eq!(conn.next_resend_at_ms(), 0);
assert!(machine.conn_handshake_msg2().is_none());
// Simulate storing msg1 and scheduling first resend
conn.set_handshake_msg1(vec![0x01], 2000);
assert_eq!(conn.resend_count(), 0);
assert_eq!(conn.next_resend_at_ms(), 2000);
let msg2_bytes = vec![0x01, 0x02, 0x03, 0x04];
machine.set_conn_handshake_msg2(msg2_bytes.clone());
// Record first resend
conn.record_resend(4000); // next at 4000 (2s backoff)
assert_eq!(conn.resend_count(), 1);
assert_eq!(conn.next_resend_at_ms(), 4000);
// Record second resend
conn.record_resend(8000); // next at 8000 (4s backoff)
assert_eq!(conn.resend_count(), 2);
assert_eq!(conn.next_resend_at_ms(), 8000);
assert_eq!(machine.conn_handshake_msg2().unwrap(), &msg2_bytes);
}
/// Test that duplicate msg2 is silently dropped when pending_outbound is already cleared.
#[tokio::test]
async fn test_duplicate_msg2_dropped() {
use crate::node::wire::build_msg2;
use crate::proto::fmp::wire::build_msg2;
use crate::transport::ReceivedPacket;
let mut node = make_node();
@@ -1072,7 +1204,7 @@ async fn test_should_admit_msg1_admits_rekey_when_udp_accept_off() {
///
/// The carve-out predicate must also consult peer state by source
/// address: `current_addr()` is updated from inbound encrypted-frame
/// source addrs (`handlers/encrypted.rs`), so an established peer can
/// source addrs (`dataplane/encrypted.rs`), so an established peer can
/// be matched even when the addr_to_link key is hostname-form and the
/// incoming addr is numeric.
#[tokio::test]
src/node/tests/mmp_chartests.rs
+439
View File
@@ -0,0 +1,439 @@
//! Characterization tests for the three under-tested MMP tick handlers.
//!
//! These lock in the *current* observable behavior of the MMP fan-out and
//! first-RTT paths so a later behavior-neutral sans-IO extraction has an
//! equality oracle. The `check_link_heartbeats` handler already has a good
//! oracle (`heartbeat.rs` + `tcp.rs`) and is not re-covered here; this file
//! targets the three paths with no direct handler tests:
//!
//! * `check_mmp_reports` — link-layer mode/flag fan-out gating
//! * `check_session_mmp_reports` — session mode + PathMtu gating + backoff dedup
//! * `handle_receiver_report` — the first-RTT tree re-evaluation branch
//!
//! Assertions capture what the code does today, surprising or not.
//!
//! Report-generation is probed through the reused `proto/mmp/` primitives
//! (`should_send_report` / `should_send_notification`): after a handler tick,
//! a *consumed* interval reads as "not due" (the report was built) while an
//! *ungated* interval still reads as "due" (the report was suppressed by the
//! mode/flag gate). This survives the later refactor because those primitives
//! stay in `proto/mmp/` unchanged.
//!
//! Two `#[cfg(test)]` production seams are used, both on `ActivePeer`:
//! * `test_init_mmp(mode)` — attach link MMP with a chosen mode to a
//! bare (sessionless) peer, so mode gating is exercisable.
//! * `test_backdate_session_start` — age `session_elapsed_ms()` so a crafted
//! ReceiverReport yields a positive RTT sample (first-RTT trigger).
//!
//! Neither changes any decision logic or threshold.
use super::*;
use crate::config::SessionMmpConfig;
use crate::node::session::{EndToEndState, SessionEntry};
use crate::noise::HandshakeState;
use crate::peer::ActivePeer;
use crate::proto::mmp::{MmpMode, ReceiverReport};
use crate::proto::stp::{ParentDeclaration, TreeCoordinate};
// ===========================================================================
// Helpers
// ===========================================================================
/// Insert a bare (sessionless) peer carrying link-layer MMP state in `mode`.
/// Returns the peer's NodeAddr.
fn insert_link_peer(node: &mut Node, mode: MmpMode) -> NodeAddr {
let identity = make_peer_identity();
let addr = *identity.node_addr();
let mut peer = ActivePeer::new(identity, LinkId::new(1), 0);
peer.test_init_mmp(mode);
node.peers.insert(addr, peer);
addr
}
/// Arm both sender and receiver link-MMP intervals so a report would be built.
fn arm_link_mmp(node: &mut Node, addr: &NodeAddr) {
let mmp = node.get_peer_mut(addr).unwrap().mmp_mut().unwrap();
mmp.sender.record_sent(1, 100, 500);
mmp.receiver
.record_recv(1, 100, 500, false, crate::time::mono_ms());
}
/// Complete an in-memory Noise IK handshake, returning the initiator session.
fn make_noise_session(
our_identity: &crate::Identity,
remote_identity: &crate::Identity,
) -> crate::noise::NoiseSession {
let mut initiator =
HandshakeState::new_initiator(our_identity.keypair(), remote_identity.pubkey_full());
let mut responder = HandshakeState::new_responder(remote_identity.keypair());
let mut init_epoch = [0u8; 8];
rand::Rng::fill_bytes(&mut rand::rng(), &mut init_epoch);
initiator.set_local_epoch(init_epoch);
let mut resp_epoch = [0u8; 8];
rand::Rng::fill_bytes(&mut rand::rng(), &mut resp_epoch);
responder.set_local_epoch(resp_epoch);
let msg1 = initiator.write_message_1().unwrap();
responder.read_message_1(&msg1).unwrap();
let msg2 = responder.write_message_2().unwrap();
initiator.read_message_2(&msg2).unwrap();
initiator.into_session().unwrap()
}
/// Insert an Established session carrying session-layer MMP state in `mode`.
/// Returns the destination NodeAddr.
fn insert_session(node: &mut Node, mode: MmpMode) -> NodeAddr {
let remote = crate::Identity::generate();
let remote_addr = *remote.node_addr();
let session = make_noise_session(node.identity(), &remote);
let mut entry = SessionEntry::new(
remote_addr,
remote.pubkey_full(),
EndToEndState::Established(session),
1000,
true,
);
let cfg = SessionMmpConfig {
mode,
..SessionMmpConfig::default()
};
entry.init_mmp(&cfg);
node.sessions.insert(remote_addr, entry);
remote_addr
}
/// Arm both sender and receiver session-MMP intervals.
fn arm_session_mmp(node: &mut Node, addr: &NodeAddr) {
let mmp = node.sessions.get_mut(addr).unwrap().mmp_mut().unwrap();
mmp.sender.record_sent(1, 100, 500);
mmp.receiver
.record_recv(1, 100, 500, false, crate::time::mono_ms());
}
// ===========================================================================
// check_mmp_reports — link-layer mode fan-out
// ===========================================================================
/// Full mode: both a SenderReport and a ReceiverReport are generated (both
/// intervals consumed).
#[tokio::test]
async fn mmp_full_mode_builds_sender_and_receiver_reports() {
let mut node = make_node();
let addr = insert_link_peer(&mut node, MmpMode::Full);
arm_link_mmp(&mut node, &addr);
node.check_mmp_reports().await;
let mmp = node.get_peer(&addr).unwrap().mmp().unwrap();
let now = crate::time::mono_ms();
assert!(
!mmp.sender.should_send_report(now),
"Full mode consumes the sender interval (SenderReport built)"
);
assert!(
!mmp.receiver.should_send_report(now),
"Full mode consumes the receiver interval (ReceiverReport built)"
);
}
/// Lightweight mode: only a ReceiverReport is generated; the sender interval
/// is left intact (no SenderReport in Lightweight).
#[tokio::test]
async fn mmp_lightweight_mode_builds_receiver_report_only() {
let mut node = make_node();
let addr = insert_link_peer(&mut node, MmpMode::Lightweight);
arm_link_mmp(&mut node, &addr);
node.check_mmp_reports().await;
let mmp = node.get_peer(&addr).unwrap().mmp().unwrap();
let now = crate::time::mono_ms();
assert!(
mmp.sender.should_send_report(now),
"Lightweight mode suppresses the SenderReport (sender interval intact)"
);
assert!(
!mmp.receiver.should_send_report(now),
"Lightweight mode still builds the ReceiverReport (receiver interval consumed)"
);
}
/// Minimal mode: neither report is generated; both intervals stay intact.
#[tokio::test]
async fn mmp_minimal_mode_builds_nothing() {
let mut node = make_node();
let addr = insert_link_peer(&mut node, MmpMode::Minimal);
arm_link_mmp(&mut node, &addr);
node.check_mmp_reports().await;
let mmp = node.get_peer(&addr).unwrap().mmp().unwrap();
let now = crate::time::mono_ms();
assert!(
mmp.sender.should_send_report(now),
"Minimal mode suppresses the SenderReport"
);
assert!(
mmp.receiver.should_send_report(now),
"Minimal mode suppresses the ReceiverReport"
);
}
/// Periodic operator logging fires once per interval: a fresh peer is due for
/// a log, and after one tick the log is marked (not due again within the
/// interval).
#[tokio::test]
async fn mmp_should_log_marks_logged_once_per_interval() {
let mut node = make_node();
let addr = insert_link_peer(&mut node, MmpMode::Full);
assert!(
node.get_peer(&addr)
.unwrap()
.mmp()
.unwrap()
.should_log(crate::time::mono_ms()),
"a freshly created peer is due for its first operator log"
);
node.check_mmp_reports().await;
assert!(
!node
.get_peer(&addr)
.unwrap()
.mmp()
.unwrap()
.should_log(crate::time::mono_ms()),
"after one tick the log is marked and not due again within the interval"
);
}
// ===========================================================================
// check_session_mmp_reports — session mode + PathMtu gating + backoff dedup
// ===========================================================================
/// Full mode session: both SenderReport and ReceiverReport are generated
/// (both intervals consumed) even though the send has no route and fails.
#[tokio::test]
async fn session_full_mode_builds_sender_and_receiver_reports() {
let mut node = make_node();
let addr = insert_session(&mut node, MmpMode::Full);
arm_session_mmp(&mut node, &addr);
node.check_session_mmp_reports().await;
let mmp = node.get_session(&addr).unwrap().mmp().unwrap();
let now = crate::time::mono_ms();
assert!(
!mmp.sender.should_send_report(now),
"Full session consumes the sender interval"
);
assert!(
!mmp.receiver.should_send_report(now),
"Full session consumes the receiver interval"
);
}
/// PathMtu notifications gate on all modes: in Minimal mode neither report is
/// built, yet a PathMtuNotification is still generated when an MTU has been
/// observed.
#[tokio::test]
async fn session_minimal_mode_still_sends_path_mtu() {
let mut node = make_node();
let addr = insert_session(&mut node, MmpMode::Minimal);
arm_session_mmp(&mut node, &addr);
// Observe an MTU so a notification becomes due (all modes).
node.sessions
.get_mut(&addr)
.unwrap()
.mmp_mut()
.unwrap()
.path_mtu
.observe_incoming_mtu(1200);
let now_before = crate::time::mono_ms();
assert!(
node.get_session(&addr)
.unwrap()
.mmp()
.unwrap()
.path_mtu
.should_send_notification(now_before),
"precondition: a PathMtuNotification is due after observing an MTU"
);
node.check_session_mmp_reports().await;
let mmp = node.get_session(&addr).unwrap().mmp().unwrap();
let now = crate::time::mono_ms();
assert!(
mmp.sender.should_send_report(now),
"Minimal mode suppresses the session SenderReport"
);
assert!(
mmp.receiver.should_send_report(now),
"Minimal mode suppresses the session ReceiverReport"
);
assert!(
!mmp.path_mtu.should_send_notification(now),
"PathMtuNotification is generated in Minimal mode (gate is mode-independent)"
);
}
/// Backoff dedup, all-fail side: a Full-mode session generates two reports
/// (SR + RR) to one destination; with no route both sends fail. The
/// per-destination dedup collapses the two failures into exactly ONE
/// `record_send_failure` (consecutive count advances by 1, not 2).
#[tokio::test]
async fn session_backoff_all_reports_fail_records_single_failure() {
let mut node = make_node();
let addr = insert_session(&mut node, MmpMode::Full);
arm_session_mmp(&mut node, &addr);
assert_eq!(
node.get_session(&addr)
.unwrap()
.mmp()
.unwrap()
.sender
.consecutive_send_failures(),
0,
"precondition: no prior send failures"
);
node.check_session_mmp_reports().await;
assert_eq!(
node.get_session(&addr)
.unwrap()
.mmp()
.unwrap()
.sender
.consecutive_send_failures(),
1,
"two failed reports to one dest dedup to a single record_send_failure"
);
}
// ===========================================================================
// handle_receiver_report — first-RTT tree re-evaluation branch
// ===========================================================================
/// Build a peer (NodeAddr strictly smaller than the node's own) that carries
/// link MMP but no RTT yet, and register it in the tree as a self-root with
/// that smaller address. This makes it a mandatory parent-switch target once
/// it becomes eligible. Returns the peer's NodeAddr.
fn setup_smaller_root_peer(node: &mut Node) -> NodeAddr {
let my_addr = *node.node_addr();
let (identity, addr) = loop {
let id = make_peer_identity();
let a = *id.node_addr();
if a < my_addr {
break (id, a);
}
};
let mut peer = ActivePeer::new(identity, LinkId::new(1), 0);
peer.test_init_mmp(MmpMode::Full);
// Age the session so a crafted ReceiverReport yields a positive RTT.
peer.test_backdate_session_start(std::time::Duration::from_secs(10));
node.peers.insert(addr, peer);
// Register the peer as a self-root in the tree at its (smaller) address.
node.tree_state_mut().update_peer(
ParentDeclaration::self_root(addr, 1, 0),
TreeCoordinate::root(addr),
);
addr
}
/// Craft a ReceiverReport whose timestamp echo yields a valid first RTT
/// sample. `highest`/`pkts`/`bytes` advance the cumulative counters so a
/// second report is not dropped as stale/duplicate.
fn craft_rr_payload(highest: u64, pkts: u64, bytes: u64) -> Vec<u8> {
let rr = ReceiverReport {
highest_counter: highest,
cumulative_packets_recv: pkts,
cumulative_bytes_recv: bytes,
timestamp_echo: 1000,
dwell_time: 0,
max_burst_loss: 0,
mean_burst_loss: 0,
jitter: 0,
ecn_ce_count: 0,
owd_trend: 0,
burst_loss_count: 0,
cumulative_reorder_count: 0,
interval_packets_recv: pkts as u32,
interval_bytes_recv: bytes as u32,
};
// handle_receiver_report receives the body with the msg_type byte stripped.
rr.encode()[1..].to_vec()
}
/// A first RTT sample flips the peer eligible for parent selection AND fires
/// the shell-resident tree branch: the node (initially self-root) adopts the
/// smaller-addressed peer as its new root.
#[tokio::test]
async fn first_rtt_flips_peer_eligible_and_triggers_tree_reeval() {
let mut node = make_node();
let addr = setup_smaller_root_peer(&mut node);
assert!(
node.tree_state().is_root(),
"precondition: node starts as its own root"
);
assert!(
!node.get_peer(&addr).unwrap().has_srtt(),
"precondition: peer has no RTT measurement yet"
);
let switches_before = node.metrics().tree.parent_switches.get();
node.handle_receiver_report(&addr, &craft_rr_payload(10, 5, 500))
.await;
assert!(
node.get_peer(&addr).unwrap().has_srtt(),
"first RTT sample makes the peer eligible for parent selection"
);
assert!(
!node.tree_state().is_root(),
"the first-RTT tree branch fired: node adopted a parent"
);
assert_eq!(
node.tree_state().root(),
&addr,
"node switched its root to the smaller-addressed peer"
);
assert!(
node.metrics().tree.parent_switches.get() > switches_before,
"the parent-switch was recorded in the tree metrics"
);
}
/// Regression guard: a *second* ReceiverReport (RTT already initialized, so
/// `first_rtt` is false) does NOT re-enter the tree branch — no further parent
/// switch is recorded.
#[tokio::test]
async fn non_first_receiver_report_does_not_retrigger_tree() {
let mut node = make_node();
let addr = setup_smaller_root_peer(&mut node);
// First report: fires the branch (established by the test above).
node.handle_receiver_report(&addr, &craft_rr_payload(10, 5, 500))
.await;
let switches_after_first = node.metrics().tree.parent_switches.get();
assert!(node.get_peer(&addr).unwrap().has_srtt());
// Second report with advanced counters: first_rtt is now false.
node.handle_receiver_report(&addr, &craft_rr_payload(20, 10, 1000))
.await;
assert_eq!(
node.metrics().tree.parent_switches.get(),
switches_after_first,
"a non-first ReceiverReport does not re-enter the first-RTT tree branch"
);
}

Some files were not shown because too many files have changed in this diff Show More