laantungir/fips
1
0
Fork 0
mirror of https://github.com/jmcorgan/fips.git synced 2026-07-31 03:56:15 +00:00
Code Issues Packages Projects Releases Wiki Activity

Compare commits

base: laantungir/fips:feat/platform-peer-queue
Branches Tags
laantungir/fips:master laantungir/fips:feat/platform-peer-queue laantungir/fips:next laantungir/fips:maint laantungir/fips:ble-v2
laantungir/fips:v0.4.1 laantungir/fips:v0.4.0 laantungir/fips:v0.4.0-rc2 laantungir/fips:v0.4.0-rc1 laantungir/fips:v0.3.0 laantungir/fips:v0.2.1-rel laantungir/fips:v0.2.1 laantungir/fips:v0.2.0-rel laantungir/fips:untagged-837413e700b3aaa62052 laantungir/fips:v0.1.0
...
compare: laantungir/fips:master
Branches Tags
laantungir/fips:feat/platform-peer-queue laantungir/fips:next laantungir/fips:master laantungir/fips:maint laantungir/fips:ble-v2
laantungir/fips:v0.4.1 laantungir/fips:v0.4.0 laantungir/fips:v0.4.0-rc2 laantungir/fips:v0.4.0-rc1 laantungir/fips:v0.3.0 laantungir/fips:v0.2.1-rel laantungir/fips:v0.2.1 laantungir/fips:v0.2.0-rel laantungir/fips:untagged-837413e700b3aaa62052 laantungir/fips:v0.1.0
232 Commits
feat/platform-peer-queue ... master
Author SHA1 Message Date
Johnathan Corgan 83c4e800a5 Merge branch 'maint' 
# Conflicts:
#	src/node/retry.rs
 2026-07-29 02:17:34 +00:00
Johnathan Corgan 13a98ae702 Make a half-scoped reap impossible in ci-cleanup.sh 
The reap runs two selectors, one on the CI label and one on the
compose project, and each flag narrows only its own. So --run-id
alone leaves the project sweep broad and --project-prefix alone
leaves the label sweep broad, and either way the reap still destroys
every concurrent run on the host. Both single-flag forms look scoped
and are not.

The usage text asserted otherwise. It documented --run-id as leaving
other runs alone and --project-prefix as scoping the reap to a single
run, and both claims were false for the same reason. A caller passing
--run-id alone, on the strength of that line, force-removed the
containers of three concurrent runs on 2026-07-29, and its own
comment recorded the belief that it was scoped.

Rather than ask every caller to remember the pair, close the two
half-scoped states here. --run-id now derives the matching project
prefix when none is given, built from the existing base so the two
cannot drift, and an explicit --project-prefix still wins.
--project-prefix without --run-id is refused outright, because there
is nothing to derive a label scope from and the quiet failure is
someone else's run disappearing.

Passing neither flag is untouched: that is the deliberate "reap
everything" form a manual cleanup wants.

Verified against a decoy container labelled as another run: the
scoped reap leaves it up, and reproducing the old broad project sweep
removes it, so the check discriminates rather than passing because
there was nothing to reap. An earlier version of that check ran when
no CI containers existed at all and passed without proving anything.
 2026-07-29 02:09:59 +00:00
Johnathan Corgan 77ed64bb88 Compute every peer's outgoing bloom filter in one sweep 
Each peer must be sent the union of all other peers' inbound filters,
excluding its own contribution. Building that per recipient rebuilt
the whole peer-filter map and re-ORed it once for every peer, so a
tick that announced to R peers did R kilobyte-scale map builds and
R by T merges. At 240 peers this was 20.6 ms per tick, roughly half
the tick body, and it was steady work rather than a tail: the
per-interval maximum had a median of 34.5 ms.

Replace it with a prefix and suffix union sweep that produces every
target's filter in one pass, T merges instead of R by T. The packet
path gets the same treatment, since marking changed peers had the
identical shape once per inbound announce.

The result is exactly equal, not approximately. Merging is a bytewise
OR, so regrouping the unions cannot change the outcome, and a filter
whose size does not match is rejected before any byte is touched, at
every merge site in both the old and new arrangement. Every
accumulator here is default-sized, so an odd-sized peer filter is
skipped in the new code exactly where it was skipped in the old.

Cadence, the debounce, the sequence rule and the fill-ratio cap are
untouched. A sequence number is still drawn after the debounce
re-check and before encoding, so a suppressed peer still consumes
none and an encode failure still burns one.

Known trade-off, measured rather than assumed: the sweep does its
full O(T) work regardless of how many peers are ready, so a tick that
announces to only one or two peers now costs about twice what it did.
Break-even is around three ready peers, and the saving above that
grows without bound. The marking path is a pure win, since it always
targets every peer.
 2026-07-29 01:45:12 +00:00
Johnathan Corgan bf2e7a0892 Derive each peer's npub once instead of once per tick 
The per-tick stats snapshot ran a bech32 encode for every tracked
peer, and for the common mesh peer — one with no hosts-file entry and
no configured alias — it ran a second one, because the display-name
fallback chain bottoms out in the same encode. At 240 peers that was
14.1 ms per tick, a third of the tick body and its second largest
cost, all of it recomputing values that cannot change.

Cache the npub and the shortened npub on the peer at construction. An
npub is a pure function of the peer's public key, and the identity is
never mutated after construction: there is no setter, no identity_mut,
and no assignment to the field anywhere in the tree, so the cache
cannot go stale.

The display name itself is deliberately NOT cached. Two of its inputs
do mutate at runtime — the alias map and the host map, the latter
reloaded on this same tick — so a resolved name stored on the peer
would go stale on an alias change or a hosts reload. Only the
immutable component is memoized.

Tests cover both constructors, that the cached npub matches the
identity, and that the display name still tracks an alias change. The
memoization itself is asserted by pointer stability rather than by
timing, so it is deterministic under load. Three deliberate breaks
were each caught by exactly one test: re-deriving instead of
memoizing, populating one constructor's cache from the wrong source,
and reordering the display-name fallback so it stops honoring aliases.
 2026-07-29 01:32:48 +00:00
Johnathan Corgan 2045b51f5d Watch the Nostr loops that never return, not the one that does 
Every node reported state: degraded permanently. The liveness probe
polled connect_task, which wraps a single Client::connect() call. That
call only spawns a per-relay background connection task and returns,
so the handle finished moments after start on a perfectly healthy
node, the supervisor saw a child exit, and the node latched Degraded
for the rest of its life.

Nothing behaved differently, because every consumer of NodeState
treats Degraded the same as Running. What was lost is the signal: a
genuine degradation was indistinguishable from the permanent false
one.

Watch the three service loops that cannot return by design instead —
the inbound notify loop, the advert publisher, and the refresh ticker.
Each is an unconditional loop, so a finished handle means a panic or
an abort, which is unrecoverable and matches the one-way ChildExited
latch in the supervisor. connect_task and relay_startup_task stay
deliberately unwatched, and the doc comment now says why, since
watching either reproduces this bug exactly.

The tests install task handles directly and check both directions:
that a finished connect_task alongside three live loops reports
healthy, which is the production configuration a few hundred
milliseconds after start, and that each loop dying on its own reports
degraded. Reverting to the old predicate reds four of five; replacing
the predicate with a constant false also reds four of five, so the
fix cannot pass by never reporting degraded at all.
 2026-07-29 01:31:13 +00:00
Johnathan Corgan d6457fa74f Stop awaiting the advert refetch on the retry tick 
process_pending_retries runs inline on the node's 1-second rx-loop
tick. For each due peer it awaited a Nostr relay fetch carrying a
2-second timeout, and discarded the result. With up to sixteen due
peers in one tick body, the timeouts stack: field profiling measured
single 2.00 s stalls as the common case and a worst tick of 12.4 s
against a 1 s period, with every other rx-loop arm delayed behind it
by as much as 4.2 s.

Spawn the refetch instead of awaiting it, matching the pattern the
failure arm of the same loop already uses thirty lines below. The dial
now uses whatever advert is cached at that moment and the refreshed
one lands for the next retry of that peer. Since retries are
backoff-paced, that defers the benefit by one backoff interval rather
than losing it, and the result was already being discarded, so nothing
downstream read it.

The test drives four due peers whose refetches all hang against a
local listener that accepts and never speaks, so the fetch burns its
full timeout with no network egress. Awaited, the call takes 8.0 s;
spawned, it returns in milliseconds. It also asserts every due peer
was still attempted and rescheduled, so a version that skipped the
dial entirely cannot pass it.
 2026-07-29 01:27:49 +00:00
Johnathan Corgan 3c4f003c91 Merge branch 'maint' 
# Conflicts:
#	src/nostr/runtime.rs
#	src/nostr/tests.rs
 2026-07-29 00:36:19 +00:00
Johnathan Corgan 7d0a110f4e Send traversal signals only to relays the client pool actually holds 
A traversal signal is addressed to a merge of the peer's NIP-17 inbox
relays, the relays its advert nominates for signaling, and our own DM
relays. The client pool is built once at startup from our configured
relays and never added to, and send_event_to rejects the entire send
with "relay not found" if any single URL in the list is outside that
pool, before contacting anything. So one relay we are not configured
with, anywhere in that merge, killed the whole attempt -- including the
sends to relays we do share and that would have carried the signal.

In an open-mode window on a public node this made discovery
non-functional: 309 traversal attempts, 290 explicit failures, zero
successes, every failure on "relay not found". Configured peers were
unaffected because they run a matching relay set.

Filter the merged list down to relays the pool holds before sending.
Our own DM relays are always in the merge and always in the pool, so
the result is empty only when no DM relay is configured at all, which
is already a total failure. Comparison is on the normalized RelayUrl
rather than the raw string, because that is how the pool is keyed --
a raw comparison would discard a configured relay spelled with a
trailing slash or a different host case, which is the same defect in
a quieter form.

Merge and filter are one synchronous function so the decision can be
exercised without a relay client. The pool is read via all_relays(),
which is the set send_event_to validates against; relays() is filtered
by service flags and would be narrower.

Two smaller fixes ride along. The responder now resolves its relays
before binding a socket and running STUN, instead of spending a STUN
round trip and holding an offer slot only to discover it has nowhere
to answer. And it gained the empty-list guard the initiator already
had, which gives BootstrapError::MissingRelays a condition it can
reach for the first time -- it was unreachable, since the merge always
appended our own DM relays.
 2026-07-29 00:31:36 +00:00
Johnathan Corgan 2e8fb60970 Let build-deb.sh build with Cargo features, and mark the version when it does 
A measurement run needs the .deb built with a non-default feature, and the
script had no way to express that: its argument loop took only --target,
--version and --no-build, and nothing forwarded a feature list or read one
from the environment. --features <list> now forwards to cargo-deb's native
-F.

The marking is the half that is easy to skip and matters more. The
auto-derived dev Version is built from the crate version, the commit date
and the sha, none of which change when a feature is enabled, so an
instrumented package and a default package of the same commit carried
byte-identical versions. Two consequences, and the second is worse than
the first: the node offers no way to tell which one it is running, and
reverting is an install of a version already present, which no-ops
silently and leaves the instrumented binary in place. That is precisely
the failure the per-commit version was introduced to prevent, reappearing
one level down. The Version now carries a +<features> marker, folded to
dots since underscores and commas are not legal there.

The marker sorts above the unmarked build, checked with dpkg
--compare-versions rather than assumed, so installing a feature build is
an upgrade and reverting is a downgrade: revert with dpkg -i, not apt
install. The ~dev ordering below a tagged release is preserved.

--features is refused together with --no-build, which would stamp the
marker onto whatever binaries happened to be sitting in target/ while
claiming the features had reached them.

Verified end to end rather than by reading: a real --features profiling
build produces a package whose fipsctl accepts the profile subcommand and
whose daemon carries the capture-file header text that only exists under
the feature gate. The release packaging path is untouched, shown by
diffing the cargo invocation the old and new scripts produce across five
argument shapes including the --version/--no-build pair the packaging
workflow uses; all five are identical.

Authored on master rather than maint, which takes bug fixes and CI or
tooling changes but not new capability, so the three copies of this
script now differ by design.
 2026-07-28 17:40:28 +00:00
Johnathan Corgan 8162b7d9fc Cover the capture writer's failure path and split its flush cycle 
The mesh rehearsal could not exercise the write-error stop at all, and
that is not a gap in the rehearsal. Removing the sink file leaves the
writer's descriptor valid, so writes keep succeeding into the unlinked
inode and the capture runs on; mounting a tiny filesystem inside the test
container is refused outright. So the terminal state added for a failing
sink had no coverage from either direction.

Split one flush cycle out of the writer loop as a function returning what
it decided, so the loop owns the waiting and the state transition while
the decision is testable on its own. A sink that fails every write now
drives the error outcome directly, and a working sink is asserted to
continue so the first test cannot pass for a writer that always stops.
The residual gap is recorded at the test: what is covered is that an
error from the sink produces the error outcome rather than the cap
outcome, not that a real full disk reaches that branch.

The rehearsal itself was otherwise clean on a live three-node mesh. The
header carries node, build, platform, tick period and the reading
caveats; every step reports every interval; stopping returns in 53 ms
rather than waiting out the flush interval; an idle status reports no
path and no bytes; arming twice is refused naming the active file; an
unwritable directory fails the command; and the node kept serving
throughout. On an idle node the entry gap sits at one tick period and
arm starvation reads about 1.4 ms, which is small but not zero, so the
measurement is live rather than degenerate.
 2026-07-28 01:30:11 +00:00
Johnathan Corgan 7493153a89 Add a feature-gated profiler for the rx-loop maintenance tick 
The tick arm runs twenty-six housekeeping steps in sequence on the one
runtime thread, and is polled last, so anything slow in it holds up
inbound packets, TUN traffic and control commands behind it. Field
evidence says that happens for over a second at a time, but the
attribution behind that is two months old and predates the
connect-on-send gate, the control read isolation, and the peer lifecycle
rework. This measures it rather than continuing to reason about it.

Per step it records exact count, max and total into fixed static
counters; a dedicated writer thread drains them every ten seconds to a
TSV under /var/log/fips, one file per capture, capped at 32 MB. Nothing
accumulates: the counters are swapped to zero each interval and the
thread holds no history. Arming is `fipsctl profile tick on`, served in
the control accept task so the toggle cannot queue behind the very
behaviour it measures, and it does not survive a restart.

The whole thing is behind a Cargo feature that is off by default,
because the risk worth eliminating is the twenty-six edited call sites
in the hot loop. With the feature off the macro expands to the bare
expression, which makes the default build's neutrality something you
read off the generated code rather than something a benchmark fails to
disprove.

The measurement that matters is how late each tick is against the
deadline it was scheduled for, since the arm is polled last and that
lateness is the delay. Two earlier designs derived it from the interval
between entries and both under-reported: the schedule is fixed, so a
steady delay leaves every gap exactly one period and any gap-derived
figure reads zero under precisely the sustained overload this is meant
to find. The interval hands back its own deadline, so the delay is now a
subtraction with no model behind it, and a test drives three late ticks
at a constant gap to keep it that way.

CI gains a default-features clippy and a feature-on build and test on
both runners, closing the gap left by clippy already running with all
features.
 2026-07-28 00:56:54 +00:00
Johnathan Corgan 52dc21726a Merge branch 'maint' 2026-07-27 17:57:21 +00:00
Johnathan Corgan 2fdc831ce3 Claim the NAT lab's two bridges per run and derive every address from them 
The NAT lab was the last suite pinning fixed IPv4 subnets, so two overlapping
runs collided on the wan and shared-lan bridges.

Each run now claims a free /24 for each bridge, scanning candidates and
advancing on an overlap while still failing fast on any other network-create
error. The claim exports NAT_WAN_PREFIX and NAT_LAN_PREFIX, and every routable
address in the compose file and the suite scripts derives from them, with
defaults that render exactly what the lab used before. The router-side LANs are
deliberately left pinned: they live inside per-container network namespaces,
never become docker networks, and cannot collide.

An external-network overlay lets the suites attach to the networks the run
already claimed instead of creating their own. Two of the three suite scripts
had no overlay hook, so they would have requested the claimed range a second
time; both now have one.

Host veth names are scoped by a short token rather than the full run id, which
overruns the fifteen-character interface-name limit at the default run-id
length, and the cleanup reaper's pattern is widened to match the new shape.

Networks are released inline on every exit path rather than in a trap, since a
trap written inside a shell function replaces the script-level handler and
would disable the whole run's teardown.
 2026-07-27 14:34:46 +00:00
Johnathan Corgan fbff52d85e Give the NAT lab per-run generated configs 
The NAT lab wrote its generated node configs to one shared directory, unlike
the static and firewall labs which already scope theirs by run. The directory
is bind-mounted by compose and read back by the suite scripts after the
containers are up, so two overlapping runs let the second run's generator
overwrite the npubs the first is about to ping.

Scope the directory with FIPS_CI_NAME_SUFFIX at all three places that have to
agree: the generator's output path, the ten compose bind-mounts, and the
CONFIG_DIR the suite scripts read back from. An unset suffix renders the plain
path the lab has always used, so a bare invocation and the GitHub matrix are
unaffected. The run teardown removes the per-run directory alongside the static
and firewall ones, and the gitignore is widened to cover the suffixed form.

Rendering the compose file with every profile and no suffix set reproduces
today's exact ten paths; setting a suffix moves all ten.

This lands on its own, ahead of the network and address work, so that a failure
of the two-overlapping-runs acceptance test can be bisected between the config
fix and the address conversion.
 2026-07-27 14:34:33 +00:00
Johnathan Corgan 76d7e43d60 Merge branch 'maint' 
Carries today's maint batch: the local CI image-scoping work, which gives
each run its own build context and test image tag instead of writing the
shared mutable one, and the retirement of the bloom-storm chaos scenario.

Both apply unchanged here — the compose files, suite lists and matrix legs
they touch are identical on the two lines, so no branch adaptation was
needed. Parity stays symmetric across runners at 21 legs a side on this
branch and 24 on maint, the gap being the three rekey Docker suites maint
keeps by design.
 2026-07-26 17:16:26 +00:00
Johnathan Corgan 93800a503e Retire the bloom-storm chaos scenario from both CI runners 
The scenario guarded a real regression: a mid-chain tree update that
changed neither root nor depth leaking downstream as a sustained bloom
announce storm. But it was never once run against the regressed binary,
and its per-node ceiling was inferred from a post-mortem harness that no
longer exists in the tree. On the only surviving regressed measurement
the tail node's rate scales to roughly 7 sends per 30s, well under the
scenario's ceiling of 40, so it was never established that the assertion
could fire on its own bug class. The ceiling is also uniform per node,
calibrated against the flap target that is legitimately busy rather than
against the tail where the storm actually shows.

This removes coverage rather than relocating it, unlike the two earlier
retirements above it, and the comment in ci-local.sh records that gap
explicitly. The scenario, its README, the link_swap sim primitive and the
mesh-lab dispatch all stay on disk, so it remains runnable by hand.

Parity holds at 24 legs a side. The parity guard was break-checked by
re-adding the GitHub leg alone, which correctly reported the asymmetry.
 2026-07-26 16:51:23 +00:00
Johnathan Corgan cdda660f10 Stop local CI writing the shared mutable test image tag 
The run built per-run images and then retagged them to fips-test:latest as
a compatibility bridge for the consumers that had never been migrated. The
bridge was the defect: while it existed, two concurrent runs shared one
binding, so a suite could start containers from the other run's binaries
and the verdict was recorded against a commit whose code never ran. It
fails silently by construction — the run is green either way, and nothing
compares a running container's binary against the commit under test.

Every consumer now reads the run's image, so the retag is deleted rather
than kept. That is the point of deleting it: a consumer that was missed
fails loudly instead of quietly resolving whichever run wrote the tag last.

Two consequences handled here. The cleanup script ran ip(8) inside
fips-test:latest to reap simulation interfaces, and nothing writes that tag
any more, so it now takes the caller's image, then the run's, then any
surviving test image — the last of which is what keeps an unscoped --reap
working, since that path execs before the run identity is exported. And a
guard checks statically that nothing names the shared tag, because on a
host with a hand-built copy lying around a reintroduction would run green;
it is break-checked against a reintroduced compose consumer and a
reintroduced default. Both runners gate on it, as they do the other guards.
 2026-07-26 15:47:22 +00:00
Johnathan Corgan 49163befd5 Give each local CI run its own docker build context 
Scoping the image tag was never sufficient on its own. Every build read one
unscoped directory in the working tree, into which each run copies the
binaries it just built, so two concurrent runs raced on the contents of the
context as well as on the name of the result — and a run could produce a
correctly-per-run-tagged image built from the other run's binaries.

The run now copies the context's tracked files into its own directory,
installs its binaries there, and builds from it, exporting the path so every
other consumer follows. Deliberately not carried over: a previous run's
binaries, since inheriting them is the failure this prevents. The path is
absolute because compose resolves a relative build context against the
compose file's own directory rather than the working directory, which was
measured rather than assumed.

The chaos entry script gated the whole simulation on a binary in the shared
directory by literal path, so it moves in this same commit: left behind, it
would have failed on a clean checkout and, worse on a host with leftovers,
passed while reading a binary that was not the one under test.

Teardown removes the directory on red runs as well as green, since it holds
only reproducible content and is never the evidence of a failure. The
worker's SIGKILL runs no trap, so the cleanup script also sweeps contexts
left by a preempted run, and the ignore rule keeps a concurrent run from
showing up as untracked working-tree noise.
 2026-07-26 15:42:48 +00:00
Johnathan Corgan af847c68b5 Stop the chaos simulation rebuilding the shared test image per scenario 
The runner rebuilt the shared tag from the shared build context at the
start of every scenario, unconditionally. Local CI runs chaos scenarios in
parallel, so that was several concurrent builds into one name inside a
single run, before any second run is considered, and it silently replaced
whatever image the harness had already built and handed over.

It now uses the caller's image when one is named, and asserts the image is
present rather than building a substitute, because under a harness a miss
means something upstream is broken and building would hide it. A bare run
still builds the shared tag exactly as before, reading the run's own build
context when one is set.
 2026-07-26 15:40:03 +00:00
Johnathan Corgan a718cef8ce Have every suite compose file name the caller's test image 
Only the static family read FIPS_TEST_IMAGE; the firewall, ACL, nat,
sidecar and both Tor compose files named the shared mutable tag directly,
which is why the local CI runner has to retag its per-run image to that
name. They now take the same defaulted form the static file already uses,
so a harness run resolves the image it built and a bare hand run still
resolves the shared tag exactly as before.

Checked by rendering each file through `docker compose config` with the
variables set: every service resolves to the supplied name and none is
left naming the shared tag.
 2026-07-26 15:38:28 +00:00
Johnathan Corgan cbbdf2c13c Stop the firewall and ACL suites rebuilding an image their caller supplied 
Both suites ran `docker compose up -d --build` unconditionally, and their
compose files carry both a build context and an image name, so the flag
rebuilt and retagged the shared test image from the shared build context on
every run. Their --skip-build flag did not cover it: it guarded only the
Rust build above. Under a harness that has already built the image and
handed it over, that rebuild can only replace the binaries under test with
whatever the shared context currently holds.

The nat scripts had the mirror-image problem. Their image guard built a
replacement whenever the image was missing, which is right by hand and
wrong under a harness: when the caller names an image it built, a miss
means something upstream is broken, and manufacturing a substitute hides
that. They now fail loudly in that case and keep building only on the hand
path. The mesh-lab guard reads the same variable and its build hint moves
to the script that still produces the shared tag.
 2026-07-26 15:37:50 +00:00
Johnathan Corgan 24bf4d46d4 Widen the churn-mixed baseline bounds to sit outside the scenario's own spread 
The root-count ceiling of 4 was set from six runs whose observed maximum
was 3. Fourteen runs that recorded a baseline verdict put the real
distribution at 1 to 5 roots, so the ceiling sat at roughly the 93rd
percentile of the scenario's own variance: one sample in fourteen
exceeded it and four sat at or above it. It therefore reddened a share of
runs whatever the daemon did, which is a calibration fault rather than a
convergence one. Every sample ran the file's fixed seed, so the spread is
container timing and not a differing chaos schedule.

The ceiling is now 6, one step beyond the observed maximum, and the
parented floor moves to its complement at 4. That still fails a mesh
where seven or more of ten nodes are islanded, which is the collapse case
the assertion exists to catch; driving evaluate_baseline directly
confirms all fourteen samples pass and 7, 8 and 10 roots still fail.

min_sessions is left at 10 against an observed minimum of 12. It has
never fired, but the comment now records the margin as thin so a future
failure there is read as calibration first.
 2026-07-26 15:22:43 +00:00
Johnathan Corgan cf129cde77 Retire the three ignored Ethernet tests, which covered nothing unique 
All three required root or CAP_NET_RAW, neither runner passes --ignored,
and so none had ever executed in CI. Before deleting them I checked what
each covered and whether anything else covered it:

- The two-node handshake is covered by the ethernet-only chaos scenario,
  whose baseline assertion cannot pass unless handshakes complete over
  AF_PACKET.
- Tree convergence and root election by smallest NodeAddr are covered by
  ethernet-only's single-root assertion and by the in-process
  spanning-tree tests respectively.
- The mixed-transport coexistence property is covered twice over by
  test_tcp_mixed_transport_coexistence and test_ble_mixed_transport, which
  call the same verify_tree_convergence_components helper with the same
  two-component shape and both run today.

So the only thing running them would have added is an AF_PACKET variant of
a property already proven on two other transports.

The counts confirm nothing running was lost: 1392 tests pass before and
after, and the ignored count falls from 7 to 4.

The ethernet-only comment block cited these tests as the reason its
assertion was the only Ethernet coverage in CI. It now records the sharper
fact found while reading them: that coverage is control plane only. Both
Ethernet scenarios disable traffic, so no datagram crosses an Ethernet
link anywhere, which leaves the frame length field that trims NIC minimum
frame padding ahead of AEAD verification unexercised. Deleting these tests
does not widen that gap, because the test named for data exchange did not
exchange any.
 2026-07-26 14:41:47 +00:00
Johnathan Corgan 4315328f2b Stop the log analyzer double-counting outbound peer promotions 
The shared analyzer matched two strings for one event: "Peer promoted to
active", emitted at info, and "Outbound handshake completed", emitted at
debug. Both come from handle_msg2 on the same call path for the same
outbound promotion, so any run at debug level counted every outbound
promotion twice.

Measured rather than argued. On the archived debug-level churn-mixed run
of 2026-07-25, the node logs carry 20 of each string; the old matcher
reported 40 promotions and the new one reports 20. The neighbouring
counters are unmoved at 13 sessions and 33 parent switches.

Info-level runs are unaffected, which was checked rather than assumed: on
the tcp-mesh run of the same date the debug string occurs zero times, and
the old and new matchers both report 7 against 7 raw occurrences. Three
scenarios run at debug, one of them gating, so this corrects the promotion
figure in their artifacts. No assertion reads the counter, so no suite's
verdict changes.

The earlier fix that introduced this added the live string alongside the
dead one where it needed to replace it.
 2026-07-26 14:24:44 +00:00
Johnathan Corgan 44e04eb2e2 Retire the tcp-chain static test topology 
The tcp-chain profile has never been runnable. ping-test.sh dispatches on
chain and mesh only, and the string tcp-chain has never appeared in that
script in its history, so invoking the profile has always fallen through
to the unknown-profile branch: silently asserting nothing before that
branch was made to fail, and exiting 2 since. Neither runner referenced
it either.

What it would have covered is already covered. The chaos tcp-mesh
scenario runs in local CI and gives TCP a discriminating gate: n04's only
edges are TCP, so if the transport were broken n04 could not parent and
the scenario's baseline assertion would fail. It also carries a pure-TCP
two-hop path, n01 to n04 to n05, under netem and link flaps. The only
residual tcp-chain would have added is a mesh with no UDP present
anywhere, which is not worth a fixture that has never run.

Remove the topology, its three compose services, and the documentation
rows. Every profile the static compose file still defines is now
exercised by a suite.

Also correct the chaos README, which has listed tcp-only and tcp-chain in
its transport table since those two scenarios were deleted in 9e63b42.
Neither has existed for months.
 2026-07-26 14:02:12 +00:00
Johnathan Corgan fa49dc1210 Retire the mesh-public static test topology 
The mesh-public profile ran in neither runner: ci-local's static suite
list carries only static-mesh and static-chain, and the GitHub matrix has
only mesh and chain. It also added no coverage over static-mesh.
ping-test.sh and iperf-test.sh branched mesh and mesh-public together and
exercised the same 20 directed pairs among node-a through node-e, and no
script referenced the external node at all. The convergence waits even
used mesh's peer counts rather than mesh-public's, so the extra link to
the public node was never counted, let alone asserted.

Running it would therefore have added a dependency on a live internet
host (test-us01.fips.network) in exchange for zero additional assertions.
Remove the topology, its five compose services, the script branches that
aliased it to mesh, and the documentation rows. An invocation using the
old profile name now fails on ping-test.sh's unknown-profile guard
instead of silently behaving as mesh.

The config generator's external-node support (external_ip,
is_external_node) stays. It has no consumer now, but it is woven into the
config path every remaining topology uses, so removing it would put the
gating suites at risk for no present gain.
 2026-07-26 13:54:14 +00:00
Johnathan Corgan d94300a4f4 Merge branch 'maint' 2026-07-25 19:15:54 +00:00
Johnathan Corgan 0eea7dae13 Make a zero-peer floor unreachable in the shared convergence wait 
wait_for_peers reads the connected-peer count through a pipeline ending in
`|| echo 0`, so a container that never answers contributes 0. That is safe
against a floor of 1 or more, where 0 reads as "not converged yet" and the wait
eventually times out, and it is unsafe against a floor of 0, where the first
read from a dead container satisfies the wait and the caller proceeds as though
convergence had been observed.

No caller passes 0 today; every one passes 1 or more, and the single variable
minimum is advisory. Rather than harden the reader, reject the input: a floor
below 1 is now an error. That makes the shape unreachable instead of repairing
instances of it, and it costs a future caller nothing except a clear message
saying that asserting "exactly zero peers" needs a reader that distinguishes no
answer from zero, which this floor is not.

wait_for_links goes with it. It had no caller anywhere in the tree on any
branch, and it carried the identical fallback, so the only thing it could do was
hand the hazard to whoever called it first. An uncalled helper cannot be wrong
today, which is exactly why it was the risk worth removing rather than the one
worth keeping for symmetry.

The hermetic gate suite gains a case covering both directions, with docker
stubbed so it stays container-free: a floor of 0 is refused with a message
naming why, and the same unreachable container with a floor of 1 still polls its
full budget and times out. Verified by removing the guard and confirming exactly
the three zero-floor assertions red while both floor-of-one controls stay green.
 2026-07-25 19:15:31 +00:00
Johnathan Corgan 1077fd6a7d Stop the firewall peer reader turning a silent container into a count of zero 
wait_for_peers_exact read the connected-peer count through a pipeline ending in
`|| echo 0`, so a container that never answered and a daemon that answered zero
produced the same value. That is only harmless while every caller expects a
non-zero count, which is true here today and is the reason this copy was left
alone when the acl suite's copy was fixed. It leaves the trap armed for whoever
adds the first caller expecting zero: the check would be satisfied on its first
iteration without the property it exists to verify ever being observed.

The read moves into its own function that returns the empty string when the
container does not answer, and the caller treats empty as "no answer" rather
than as a count. A run that never gets an answer now fails saying so, distinctly
from one that answered the wrong count, and the diagnostic peer dump runs only
on the latter, since dumping from a container that cannot answer prints a docker
error rather than evidence.

This is the shape the acl-allowlist suite already uses. The two copies had
diverged on whether a silent container counts as an answer, which is the kind of
disagreement that decides a security assertion in whichever file is read last.

Checked by construction rather than by a green run: driving the old form against
an absent container with an expected count of zero returns success on the first
iteration, and the new form fails; with an expected count of one, the shape both
real callers use, both forms still fail, and a genuine zero from a live daemon
is still reported as zero.
 2026-07-25 19:02:15 +00:00
Johnathan Corgan 0e42c789be Let the firewall suite float its docker subnet 
The compose pinned 172.32.0.0/24 with a per-container ipv4_address, so two
concurrent runs asked docker for the same address space and the second failed
with a pool-overlap error. Request no subnet and let docker assign one from the
daemon pool. Peers address each other by the docker hostname the compose already
sets (host-a, host-b) rather than by literal IP; docker's embedded DNS is
per-network, so the same hostname in two runs resolves inside each run's own
subnet. Nothing this suite asserts on moves as a result: its checks run over the
fips0 overlay, whose addresses are derived from node npubs and are independent of
docker addressing, and the packaged nftables ruleset matches on interface rather
than on any address.

The generated config directory moves under the run suffix for the same reason it
did in the acl suite, and the run teardown gains the matching removal so a run
no longer leaves its directory behind.

Case (b) also wrote curl's output to a fixed path under /tmp. Two concurrent
runs shared that one file, and either run's cleanup landing between the other's
write and read left an empty read, failing the http_code check for a reason
having nothing to do with the firewall. It uses mktemp now.

As in the acl suite, the floating subnet removes one of the two obstacles to
concurrent runs. The compose project name is still fixed; the local CI runner
scopes it externally, a bare hand run does not, and the comment at the site says
so rather than claiming the file is self-sufficient.
 2026-07-25 19:01:56 +00:00
Johnathan Corgan 611f045d33 Let the acl-allowlist suite float its docker subnet 
The compose pinned 172.31.0.0/24 with a per-container ipv4_address, so two
concurrent runs of this suite asked docker for the same address space and the
second failed with a pool-overlap error. Request no subnet instead and let
docker assign one from the daemon pool, which leaves nothing for two runs to
contend for. Peers now address each other by the docker hostname the compose
already sets (host-a through host-f) rather than by literal IP, and docker's
embedded DNS is per-network, so the same hostname in two runs resolves inside
each run's own subnet.

The generated config directory moves under the run suffix in the same change,
because it has to: the generator does rm -rf on it, so a shared directory means
one run deletes the fixtures another is still using, which fails silently rather
than loudly at bring-up. Scoping it requires the generator output path, all five
bind-mount sources per service, and the gitignore entry together; scoping only
the generator leaves the compose reading the unscoped path.

The floating subnet removes one of the two obstacles to running this suite
twice at once, not both: the compose project name is still fixed, and nothing
scopes it for this suite. Recorded at the site so the comment does not claim
more than the change delivers.
 2026-07-25 19:01:16 +00:00
Johnathan Corgan 66d5be2bfb Merge branch 'maint' into master 
Both maint changes carry over unchanged. The hop-limit helpers follow the
module rename from src/protocol/link.rs to src/proto/link.rs, applied by
rename detection and verified at the renamed path rather than assumed; the old
path is not recreated. The rekey config validation and its jitter-constant
import apply as written, the node module exposing the constant on this branch
as well.

The only conflict was the unreleased changelog, where master's own Changed
entries and the incoming hop-limit entry were both correct for their branch.
Resolved as the union, with the incoming entry appended to master's Changed
list.
 2026-07-25 17:14:30 +00:00
Johnathan Corgan 0cb5574077 Reject rekey settings that fire the trigger immediately and forever 
Config validation said nothing about the rekey block, so two settings that
disable rekey in appearance and hammer it in practice were accepted silently.

after_messages of zero makes the message-count arm true on every poll, since
the trigger tests the counter against it with a greater-or-equal. It reads like
a way to switch the arm off and does the opposite.

after_secs at or below the per-session jitter is the same trap on the timer
arm. Each session offsets the interval by a random value drawn from plus or
minus the jitter bound, so a smaller interval saturates to zero on a negative
draw and rekeys on sight, for roughly half of sessions and not the other half.
The rule is expressed against the jitter constant rather than its current
value, so it tracks if the bound ever moves.

Both are checked whether or not rekey is enabled, so turning it on later cannot
surface a configuration error at a surprising moment. Neither has an upper
bound: a very large value is the established way to disable one arm of the
trigger and stays legal.
 2026-07-25 17:07:57 +00:00
Johnathan Corgan 989ae65fc5 Align the datagram hop-limit helpers with the forwarder's semantics 
The forwarding path was corrected to full IP semantics: local delivery is not
hop-limit gated, and forwarding decrements first and drops at zero. Two helpers
on SessionDatagram were left implementing the old rule. decrement_ttl still
checked before decrementing, and can_forward still answered true at a hop limit
of one, where the decrement leaves zero and the datagram is dropped. Neither is
called anywhere today, which is precisely why they are worth correcting rather
than leaving as an invitation to reintroduce the defect.

The TtlExhausted reject doc described behaviour that no longer exists. The
reject is charged for a transit arrival at one as well as at zero, and is never
charged for a datagram addressed to this node, whose delivery is decided ahead
of the test.

Both helpers are pinned by tests at the boundary, which fail against the
previous implementations.
 2026-07-25 17:05:20 +00:00
Johnathan Corgan 7505b86a5d Merge branch 'maint' 
Carries the address-to-link map's corrected doc comment. The only
conflict was the adjacent line naming the discovery config keys, which
this line renamed to node.lookup.* and the maintenance line did not; kept
this branch's names.
 2026-07-25 01:53:15 +00:00
Johnathan Corgan 1eabe08575 Correct what the address-to-link map claims to be for 
The doc said the map "enables dispatching incoming packets to the right
connection before authentication completes." It does not: find_link_by_addr
has no callers outside its own tests on any branch, and encrypted frames
dispatch by session index. Its live readers are the msg1 admission fast
path and the duplicate-inbound-handshake check.

The comment mattered because the map has a trap that invites exactly the
misuse it was advertising. An outbound dial registers the literal
configured address string, which may be a hostname, while an inbound
packet carries the resolved form; TransportAddr compares byte-wise, so
the two never match and a lookup keyed on an inbound address returns "no
such link" for every hostname-configured peer rather than failing. The
entry is also single-valued per key, so an inbound handshake overwrites
an outbound dial's entry for the same address.

Both current readers survive this because each compares a key written in
the same form it reads. That is a property of those two call sites and
not of the map, so the comment now says so, and says not to key a
peer-identity question on it.

Documentation only; no behaviour changes.
 2026-07-25 01:50:40 +00:00
Johnathan Corgan b29908ba8a Pin the rekey counter arm's boundary against the real trigger 
The send-counter arm of the rekey trigger had no test that could fail for
the reason it existed. The integration suites never reached it: they
inject after_messages = 65536 against a ping-driven workload, so every
rekey they observe is time-triggered. The unit test that looked like the
guard reproduced the trigger's OR predicate as a local closure and
asserted against its own copy, so deleting the arm from the real
predicate left it green.

That is the coverage hole that let the data-plane drop across a
message-count-triggered rekey survive: the defect is precisely a rekey of
a young session, which only the counter arm can produce.

Replaces the existing at-threshold check with a boundary test driving
poll_rekey, holding the time arm off so the counter is provably what
decides. The silence-below half is the part that is new, and it earns its
place by measurement: widening the bound to `p.counter >= 1` reds exactly
one test in the library suite, this one. Deletion, narrowing to `>` and
inversion are caught here too, though the time arm's negative tests
already catch a bound removed altogether.

Removes the closure-based test rather than repairing it. Its one
assertion over real code, that a fresh session entry's jitter lies within
the symmetric bound, is already covered over 100 samples elsewhere.

Note this leaves the deployed maintenance line untested on the same arm:
its trigger is inline in an async method that reads the clock directly,
with no pure function to drive, so the same test cannot be written there
without the sans-IO seam this line already has.
 2026-07-25 00:35:16 +00:00
Johnathan Corgan 9213cce6c4 Retire the rekey Docker suites; timing, continuity and variants are in-process 
The three rekey integration suites (rekey, rekey-accept-off,
rekey-outbound-only) are dropped from both runners. Their coverage now
lives in fast, deterministic in-process tests:

- rekey timing and choreography (trigger, K-bit cutover, drain, jitter,
  guards) in the sans-IO poll_rekey tests (proto/fsp, proto/fmp);
- data-plane continuity across a real cutover in
  rekey_cutover_preserves_data_plane (a real IK rekey over loopback);
- the accept-off dual-init regression and the udp.outbound_only rekey
  loop in the should_admit_msg1 and dual-init characterization tests.

Drop them from both runners in lockstep so the parity guard stays green,
and record the retirement in the deliberately-not-run block with a pointer
to the standalone runner. The rekey-test.sh script stays on disk.
 2026-07-24 03:51:13 +00:00
Johnathan Corgan f44de56fd8 Add an in-process test that a rekey cutover preserves the data plane 
Drives a real IK rekey handshake to K-bit cutover between two loopback
nodes and asserts an encrypted datagram sent after the cutover decodes on
the new session, with no spurious peer teardown. The rekey is forced
deterministically: rekey.after_messages = 1 crosses the initiator's
trigger on the first sent datagram, and both sessions are backdated past
the responder's 30s rekey-acceptance gate, so no wall clock is read.

The cutover is proven to actually occur (the live session index changes),
which is what makes the continuity assertion meaningful rather than
vacuous. Adds a make_test_node_with_config loopback helper for setting the
rekey thresholds. The rekey timing and choreography decisions themselves
are already covered exhaustively by the sans-IO poll_rekey tests.
 2026-07-24 03:39:47 +00:00
Johnathan Corgan 768df453b7 Merge branch 'maint' 2026-07-24 02:40:19 +00:00
Johnathan Corgan 9846e85705 Retire the admission-cap Docker suite; the inbound gate is proven in-process 
The inbound max_peers early-gate in handle_msg1 (silent-drop a Msg1 from
a net-new identity at saturation, send no Msg2, admit no peer) is
unit-tested over a real UDP socket by
handle_msg1_silent_drops_at_cap_for_new_peer in src/node/tests/unit.rs:
it saturates a node, sends a Msg1 from a fresh identity, and polls the
sender socket to assert no Msg2 comes back — the same wire-observable
discriminator the Docker suite's tcpdump used — with a sibling test for
the existing-peer bypass. That is a deterministic superset of the Docker
packet-capture assertion.

Drop it from both runners in lockstep (its *_SUITES array, run function,
default-flow loop, --only dispatch arm, and the GitHub matrix leg with
its steps) so the parity guard stays green, and record the retirement in
the deliberately-not-run block with a pointer to the standalone runner.
The admission-cap-test.sh script stays on disk and runs by hand.
 2026-07-24 02:31:43 +00:00
Johnathan Corgan 41ce64ba82 Retire the acl-allowlist Docker suite; the decision is proven in-process 
The ACL admission decision is exhaustively unit-tested per npub over real
loaded allow/deny files (src/node/acl.rs test module: allow-match-wins,
allowlist-miss falls through, deny-only, deny-all, allow_all override,
deny-after-allowlist-miss), and the inbound and outbound handshake-
admission paths are covered in-process over the loopback transport
(src/node/tests/acl.rs: inbound msg1 denial, outbound denial, reload).
The Docker suite's only unique coverage was admission over a real UDP
transport, which every other real-transport suite already exercises, and
its operator-facing log assertion.

Drop it from both runners in lockstep (its *_SUITES array, run function,
default-flow call, --only dispatch arm, and the GitHub matrix leg with
its steps) so the parity guard stays green, and record the retirement in
the deliberately-not-run block with a pointer to the standalone runner.
The testing/acl-allowlist/ suite stays on disk and runs by hand.
 2026-07-24 02:29:38 +00:00
Johnathan Corgan 7cbe1d3d4e Retire the smoke-10 chaos scenario; convergence is covered in-process 
smoke-10 was a no-stressor 10-node tree-convergence sanity check (netem
off, no ping). Its subject, spanning-tree convergence and root election,
is now covered in-process, faster and deterministically, by the loopback
spanning-tree harness (src/node/tests/spanning_tree.rs: ring, star,
chain, 100-node and disconnected-component convergence) plus end-to-end
datagram delivery (src/node/tests/forwarding.rs). Real-UDP convergence
smoke still runs via static-mesh and the other scenarios' baseline
assertions, so no Docker coverage is lost.

Drop it from both runners in lockstep (the CHAOS_SUITES list and the
GitHub chaos matrix) so the parity guard stays green, delete the scenario
YAML, and update the chaos README.
 2026-07-24 01:43:20 +00:00
Johnathan Corgan 08547e8dff Merge maint: retire cost/congestion chaos scenarios 
Bring up the retirement of the six cost-based parent-selection chaos
scenarios (cost-reeval, cost-avoidance, cost-stability, depth-vs-cost,
mixed-technology, bottleneck-parent), which tested a decision the Docker
harness could not exercise reliably.

master already carries the equivalent sans-IO coverage in
src/proto/stp/tests (effective-depth cost selection, hysteresis, cost
degradation) and its transport-drop tests, so this keeps master's src
unchanged and takes only the scenario removals and the CI-list, README and
scenario-comment updates from maint.
 2026-07-23 23:44:31 +00:00
Johnathan Corgan 08a226fb63 Test cost-based parent selection and kernel-drop detection as unit tests 
The cost-selection chaos scenarios (cost-reeval, cost-avoidance,
cost-stability, depth-vs-cost, mixed-technology, bottleneck-parent) tested
TreeState::evaluate_parent's decision logic through a Docker mesh that could
not exercise it reliably: the tree roots at whichever node holds the smallest
NodeAddr, MMP link costs take several measurement windows to settle, and the
parent hold-down plus hysteresis timing all confound the outcome. A
deterministic link-cost flap still produced zero periodic parent switches in a
full run.

Replace those six scenarios with deterministic unit tests in src/tree/tests.rs
that drive evaluate_parent directly: cheaper-link selection at equal depth,
switch-on-cost-change, hysteresis suppressing a marginal change while allowing
a significant one, and the depth-versus-cost effective-depth tradeoff. Each is
constructed so that breaking the cost or hysteresis logic makes it fail.

The congestion kernel-drop signal (SO_RXQ_OVFL) cannot be provoked
deterministically in Docker: a fresh daemon reader keeps up with
container-speed traffic, so the socket receive queue never overflows (an
unshaped run with a 4 KB buffer and heavy traffic recorded zero drops on every
node). Extract the drop-detection edge -- read the cumulative counter, fire an
event only on the transition into a new drop burst -- into
TransportDropState::observe_drops and unit-test it directly. congestion-stress
keeps its ECN and MMP congestion-signal assertions, which do need the real
shaped bottleneck queue.

Remove the retired scenarios from both CI runners and update the chaos README.
 2026-07-23 23:33:01 +00:00
Johnathan Corgan abf2f9ba48 Merge branch 'maint' 2026-07-23 19:51:58 +00:00
Johnathan Corgan be5deee814 Make the gateway integration suite safe for concurrent CI runs 
gateway-lan pinned 172.20.1.0/24 and fd02::/64, so two concurrent local CI
runs collided on "Pool overlaps" at network creation. The IPv4 subnet has no
consumer -- the whole gateway LAN path is IPv6 -- so drop it and let docker
auto-assign. The IPv6 side cannot float, because the LAN clients' resolv.conf
pins the gateway's address as their nameserver and that must be a literal
known before they start, so claim a free /64 per run and thread the prefix
through the compose address pins, a generated resolv.conf, and the test via a
single exported variable.

The claim and the external network ship as a harness-only compose overlay
applied by run_gateway; the base compose keeps a normal gateway-lan network,
so the GitHub matrix and any standalone bring-up are unaffected. With no claim
every address renders exactly as before.
 2026-07-23 19:50:17 +00:00
Johnathan Corgan bb9bca4d83 Accept a degraded systemd as a booted one in the deb-install boot wait 
The boot check piped `systemctl is-system-running --wait` into `grep -qE
'running|degraded'`, but the script runs `set -o pipefail` and is-system-running
exits non-zero for `degraded`. So the pipeline failed on a degraded system even
though grep matched, and the wait looped to its timeout. The older distros reach
`running` (exit 0) and passed; debian:trixie and ubuntu:26.04 reach `degraded`
because a unit that cannot run in a container (systemd-modules-load) fails, so
they timed out at every ceiling -- which is why the earlier timeout increase did
nothing.

Capture the state string and test it directly instead of trusting the pipeline
exit, so a degraded-but-booted system is accepted as intended. Validated: all
five distros pass, including the two that previously failed.
 2026-07-23 19:50:07 +00:00
Johnathan Corgan 2bc0345174 Make the cost-based chaos assertions reliable under parallel CI load 
cost-avoidance and mixed-technology decide a parent by measured link cost
(etx * (1 + srtt_ms/100)). Under the local CI's 4-way-parallel chaos the fast
fiber link's measured srtt spikes from host scheduling and can exceed the
Bluetooth link's, flipping the choice and reddening a run that proves nothing
about the daemon. The old margin was ~0.4 cost units, which a ~24ms one-sided
scheduling blip closes.

Widen the Bluetooth delay to 150-250ms so the margin dwarfs any plausible
one-sided spike. The link still establishes well within the handshake budget
(a ~500ms RTT against a 30s stale-handshake window), so the losing candidate
is still a real, established peer rather than an absent one.

Add a netem mutation exclude_edges option and use it in mixed-technology to
pin the two links n08 chooses between, so a random degradation can't flip the
asserted comparison either. An unknown excluded edge is a hard error, since a
silent no-op would reintroduce the flakiness it exists to remove.

Validated by running both scenarios repeatedly under heavier-than-CI
concurrent load: the parent choice holds every time.
 2026-07-23 18:58:49 +00:00
Johnathan Corgan 9b7a27f219 Claim the chaos network range instead of deriving it, ending the collision 
Two concurrent ci-local runs could not both run chaos. Each child's subnet came
from its position in the suite list, so both runs walked 10.30.0 through
10.30.12 and requested identical ranges; whichever reached the daemon first won
and the other died with a pool overlap. A run index or hashed offset would only
make that unlikely, and it is precisely the failure being removed.

The simulator now claims its range: attempt-create on a candidate, advance on
docker's own overlap error, fail loudly on anything else. Docker's address pool
becomes the arbiter, so an overlap is impossible rather than improbable. The
claim lives in the sim rather than in ci-local because only the process that
creates the network can advance on conflict, and because it fixes the bare
chaos.sh path too, which a ci-local-only fix would have left broken.

The generated compose now declares the network external over the claimed one,
and teardown releases the range from both paths, including the setup-failed
path where a run that fell over after claiming still holds one.

Ordering matters here and is not obvious: node IPs derive from the subnet
during topology generation, and traffic shaping keys its filters on those
addresses, so the claim happens before the topology exists. Claiming later
would give a network on one range and filters on another, which does not fail
at bring-up and instead leaves the shaping matching nothing.

--subnet survives as an explicit pin for when a known range is wanted, and now
fails loudly if that range is taken rather than silently overlapping. ci-local
no longer passes it.

Validated by running two instances of the same scenario deliberately
concurrently: they claimed 10.30.1.0/24 and 10.30.0.0/24, both exited 0 with
their assertions passing, and both released their networks. The negative
control holds too: with a squatter on a pinned range the run aborts with a
message naming the range rather than proceeding.
 2026-07-23 17:09:07 +00:00
Johnathan Corgan f478f51afe Assert mixed-technology's sound criterion, and correct the one that never was 
With the root pinned, n08's test subject is assertable for the first time: its
two candidates sit at equal depth with fiber-grade default netem on one side and
Bluetooth on the other, which is exactly the preference this scenario exists to
test, and it takes the fiber parent in four of four runs. Encoded as written
rather than calibrated from the runs, since "should pick n03" is the spec and
the runs merely confirm it is met.

n06 is deliberately absent, and its documented criterion is corrected rather
than left standing. The header called n03 a fiber parent for n06; the netem
policies in the same file say that link is WiFi and the other candidate is
Bluetooth, so n06 chooses between two impaired links. Its only fiber-grade link
reaches a node at depth 3 that is never a good parent. There was never a reason
to expect it to prefer n03, and the two-of-four split observed is correct
behaviour: in a run where it took the Bluetooth peer, the fiber-labelled
candidate measured a link cost of 3 against the other's 1, so the daemon chose
the cheaper effective depth as it should.

Validated by replaying the assertion against all four archived runs, then
breaking it three ways to confirm it can fail, then one live run for the wiring.
One of the breaks expects n06 to take n03 and reports that it chose n02, which
is the direct demonstration of why n06 is not encoded.

cost-reeval records that its subject now fires in two runs of three, against
zero of five when it was structurally impossible, and why that rate is still not
one an assertion can rest on.
 2026-07-23 17:04:23 +00:00
Johnathan Corgan 9d92cfeab4 Correct three scenario headers that documented conclusions since disproved 
congestion-stress said the transport drop-detection path "is exercised by
nothing". Widening the scan beyond this one scenario finds it firing in six
others, so the path is live and what is dead is this scenario's ability to
provoke it. Records the arithmetic that explains why: at the 1 Mbps cap the
receive queue needs tens of milliseconds of reader stall to overflow, and the
ingress policer discards the same traffic a layer below the socket.

mixed-technology said the implementation does not do what its criteria say.
It does. The criteria were conditioned on a tree rooted at n01 while the mesh
rooted at n09, under which n08's choice was settled by depth before link
technology could matter. The archived parent distributions are relabelled as
describing the old root so nobody calibrates against them.

cost-reeval now records why it had no assertion about its own subject: its
designated subject was the root, so the parent switch it exists to observe
could not occur. The historical logs show the switch on n04 in 13 of 14 runs
when the root still wandered, and only on n01 in the five runs before the fix.

Each header now names what remains to be done and where it is tracked, rather
than leaving a disproved conclusion in place for the next reader to inherit.
 2026-07-23 15:47:28 +00:00
Johnathan Corgan 6319c7f577 Describe the trailing command accurately in the gate's finding message 
For a value-returning helper the trailing echo is the return mechanism, not a
log call. The hazard is the same either way -- it fixes the exit status at zero
-- but calling it a log call misdescribes four of the sites the gate now finds.
 2026-07-23 15:46:04 +00:00
Johnathan Corgan d4a2504f99 Teach the trailing-log gate to see command-substitution call sites 
Found while fixing count_log_pattern: the gate reported that function clean
even though it ends in echo and both callers consume its status. Its call-site
patterns only matched direct forms -- if fn, while fn, fn ||, fn && -- so a
status consumed through command substitution was invisible, because the line
begins with the variable rather than the function name.

That is not an exotic form. It is how a shell function returns a value, and it
is precisely the shape of the swallowed-failure family this gate exists to
catch, so the gate was blind to a large part of its own stated class.

Three patterns added for the assignment, if-guarded and test-expression forms.
The extension finds four real instances, all value-returning helpers whose
trailing echo made their exit status unconditionally zero, and each now carries
an explicit return 0. Also corrects the finding message, which described the
trailing command as a log call; for these it is the return mechanism, and the
hazard is that it fixes the status either way.

Validated by breaking what it guards: a probe reintroducing the defect shape
behind a command substitution is reported and exits 1, where before the
extension it would have passed.
 2026-07-23 15:45:37 +00:00
Johnathan Corgan bf173d8d98 Pin the chaos mesh root to n01 so scenario diagrams describe the real tree 
Every chaos scenario draws n01 at the top of its topology, and until now that
held in three of thirteen. The mesh roots itself at the numerically smallest
NodeAddr, which is a hash of the node's public key and bears no relation to the
node numbering, so which node ended up as root was effectively arbitrary.

The consequences were not cosmetic. cost-reeval rooted at n04, its own
designated test subject, so that node had no parent and the periodic parent
switch the scenario exists to observe could not occur at all. mixed-technology
rooted at n09, which put its two documented parent criteria out of reach and
made correct cost-based selection look like a defect.

Identities are still derived from the mesh name exactly as before and are still
deterministic. What changes is which node id holds which one: they are now
assigned in NodeAddr order, so n01 holds the smallest and is the root. Verified
against a model of the daemon's own derivation that reproduces the previously
observed root for every scenario and n01's address byte for byte; all twelve
pinned scenarios now root at n01.

smoke-10 deliberately opts out via pin_root: false so that root election from an
arbitrary key distribution stays exercised somewhere. Its assertion is a
convergence floor and is root-agnostic, which is why it is the cheapest home for
that. The new key is rejected when non-boolean, and a near-miss spelling is
rejected as unknown; both checked.

Not yet established: the trees themselves change, so the parent-dependent
assertions in bottleneck-parent, cost-avoidance and cost-stability need
re-deriving against live runs. Those are held until the in-flight CI finishes,
because a chaos run rebuilds the shared fips-test image that run is using.
 2026-07-23 15:43:10 +00:00
Johnathan Corgan 34a2561af7 Close the remaining "caught" and "produces red" harness holes 
Four sites where a check could report a verdict it had not established, or
detect a failure and then not turn it red.

assert_no_panic in both NAT suites read `docker logs ... || true`, so a
container that could not be read produced empty output, matched no panic
pattern, and returned success. The assertion's failure mode was
indistinguishable from its success condition. It now reports that absence of
panics is not established.

deb-install's apt capture discarded the exit status, so a failed docker exec
gave an empty capture that matched neither error pattern and reached the pass
branch. The status is now kept and checked before the output is inspected, with
the output still printed on failure.

wait_for_systemd printed a warning and returned success on timeout, so every
check after it read a system that may not have started its units. It now returns
non-zero and the caller abandons that distro leg rather than testing an
unstarted system.

interop's copy of count_log_pattern carried the same defect fixed in rekey: a
node whose logs could not be read contributed zero to eight expect-zero
assertions. Fixed the same way, and its consumer now reports the unreadable case
rather than comparing a sentinel against zero.

Each validated by breaking what it guards and by confirming the healthy path is
unchanged: an absent container now fails each check where it previously passed,
a readable panic-free container still passes, a real panic is still caught, and
a genuinely successful install still passes.
 2026-07-23 15:36:59 +00:00
Johnathan Corgan c8a0ac5fca Stop a node whose logs cannot be read from counting as a clean node 
count_log_pattern summed a per-node `docker logs | grep -c ... || true`, so a
node the harness could not read contributed zero. The six assert_zero_count
callers are negative health assertions -- no panics, no ERROR lines, no AEAD
decrypt failures, no rekey msg2 failures -- and a contributed zero reads as
clean, so one unreadable node silently weakened the assertion and all of them
voided it. This is the family the harness-fallback issue was raised to high
priority for, and it is the one remaining audit residual that can make a green
rekey run lie about protocol code.

The read now fails the count rather than degrading it, printing a sentinel that
names the container. Both callers split the declaration from the assignment,
because `local c=$(fn)` takes local's exit status and discards the function's --
which is how the original defect stayed invisible.

Validated by breaking what it guards rather than by observing green: against
absent containers the old reader returns 0 and assert_zero_count passes
vacuously, while the new one returns rc=1 and reports a failure. Checked the
other direction too, since a fix that reds a legitimately clean run is no use: a
genuine zero across readable nodes still returns 0, and the sum is unchanged at
2+1+0 for a shimmed three-node read.
 2026-07-23 15:30:01 +00:00
Johnathan Corgan c58149b0b5 Merge branch 'maint' 2026-07-23 14:09:43 +00:00
Johnathan Corgan 55331a80b5 Gate on shell functions whose exit status is a log call's 
A bash function returns its last command's status, so one ending in a log
call returns 0 whatever it did, and a caller written as `if func` or
`func || fail` has a gate that cannot fire. This tree found two in one
day: run_chaos ended in record, making every chaos row unconditionally
green since 2026-03-09, and build_fips_for_e2e ended in log, letting five
end-to-end legs test the previous commit's binary and report green. Both
were repaired one at a time. This is the rule that catches the next one.

What it enforces is a contract, not a bug hunt: a function whose status a
caller tests must end in an explicit return rather than leaving its
success value to whatever the last log call produced. That distinction is
deliberate and worth stating, because all three instances found in the
tree were benign — each failure path already returned early, so the
trailing echo reported a real success. Reading the function is the only
way to know that, and the next edit that puts an unguarded command before
the final log converts the benign shape into the defect with nothing to
notice. An explicit return costs a line and makes the class unreachable.
The three are given one here.

Scoped on the call sites rather than the definitions, the same way
check-log-strings.py scopes on what a grep reads rather than what its file
mentions: 315 functions scanned, 42 end in a log call, and only the ones
whose status something consumes are reported. A function that ends by
reporting and is called for its output is idiomatic and is left alone.
Without that scoping the finding list would be 42 long and would stop
being read.

Validated by construction rather than by a green run: injecting a copy of
build_fips_for_e2e's exact shape — unguarded docker cp, then a log as the
last statement, with a `|| { ...; return 1; }` caller — makes the checker
report it and exit 1.

Wired into both runners beside the parity and log-string checks.
 2026-07-23 14:09:04 +00:00
Johnathan Corgan 3181861341 Assert the Link MMP pane exists before asserting its colour 
mmp_focused_pane_indicator checked that the unfocused Link MMP title is
not cyan with assert_ne! over fg_at. fg_at is find(..)? mapped to the
cell's foreground, so it returns None for a title that was never drawn,
and None != Some(Cyan). The assertion therefore passed just as happily
when the pane was missing entirely as when it was present and unstyled.

Assert presence first, so the colour check means "not highlighted"
rather than "not there".

Demonstrated rather than argued. Renaming the pane title in mmp.rs so
"Link MMP" is never rendered leaves the original assertion passing, and
makes the new one fail with the message it was given. Both files restored
after.

This is the only instance of the shape: it is the sole assert_ne! over
fg_at or find in the fipstop tree, and the only assert_ne! in snapshots.rs
at all.

Quartet clean: fmt, build, clippy --all-targets -D warnings, test --lib
at 1376 passed / 0 failed / 7 ignored.
 2026-07-23 14:00:49 +00:00
Johnathan Corgan 0f27bdbd2c Give the UDP microbench a reason and record the retry asymmetry 
The UDP recv microbenchmark carried a bare #[ignore] while its sibling in
the link module carries a self-describing one. Match it, so the reason
appears wherever the test is listed rather than only in the doc comment.

Record why [profile.ci] retries = 2 is not applied locally. The two gates
disagree on purpose: the hosted runner retries a flaky test twice, the
local sweep fails on the first failure. It exists for shared-runner packet
loss, a property of that environment rather than of the code, and applying
it locally would suppress a real local flake — a failure that only
reproduces under load is a robustness bug to fix, not to retry past.
Keeping the local sweep strict is what makes it the sharper gate. An
undocumented asymmetry is indistinguishable from an oversight, which is
why this is written down rather than left to be rediscovered.

The cost is stated rather than hidden: a test that fails once and passes
on retry is reported green with no separate signal, so a genuine
intermittent failure can be absorbed. If that starts mattering the fix is
to surface retried-but-passed tests, not to drop the retries.

Quartet clean: fmt, build --workspace, clippy --all-targets -D warnings,
and test --lib at 1376 passed / 0 failed / 7 ignored.
 2026-07-23 13:48:11 +00:00
Johnathan Corgan d3eaad543d Make a skipped check visible in the verdict rather than in scrollback 
Two suites could report a clean pass for work that did not run.

stun-faults skips Phase 2 when tc netem is unavailable, announcing it
several hundred lines above the result and then printing a bare
"stun-faults-test passed". The verdict line now carries the count and
names each skipped phase, and a run in which every phase was skipped
fails outright, since it tested nothing.

deb-install defines a skip() helper and a SKIP counter, prints "N
skipped" in its summary, and calls skip() from nowhere, so the number can
only ever be 0. Its exit tested FAIL alone, meaning a skip could not have
failed the run even once something did set the counter. Gate on SKIP too.
That changes no current outcome, which is the reason to do it now rather
than later: the machinery and the reported number already existed, so the
first skip path added would have printed "N skipped" beside a zero exit
and read as coverage.

Verified by driving the verdict logic at zero, one and three skips: clean
pass, pass with the skip named, and a non-zero exit when nothing ran.
 2026-07-23 13:44:29 +00:00
Johnathan Corgan 2a6039995d Retire the ECN A/B test framing and record why suites are excluded 
ecn-ab-test.sh was a -test.sh with no path to a failing result: it
asserts nothing, applies no threshold, and nothing invokes it. It also
could not have worked. It read a fixed sim-results/ecn-ab-on/ path while
the runner has written timestamped directories since 2026-03-20, and not
one ecn-ab result directory exists on disk, so it has found neither input
for months and the "+10.2% recv throughput" figure the README carried is
not reproducible from anything available.

Rename it to ecn-ab-compare.sh, fix the path resolution to glob the
timestamped directories, and stop swallowing a failed simulation with
|| true so a broken run cannot feed the comparison.

Deliberately not adding the threshold assertion that would make it
gateable. That needs a calibration corpus, and none exists precisely
because the tool has never produced a kept result; a number invented now
would assert a guess. The prerequisite is a calibration run set and a
decision about what ECN is expected to deliver, which is a protocol
question. Written in the script header rather than left implied.

Add the exclusion block to ci-local.sh naming every suite and scenario
neither runner runs, with the reason for each: interop, boringtun,
iperf-test, this comparison tool, mesh-lab, and the four chaos scenarios
outside CHAOS_SUITES. Until now "not in the suite list" was
indistinguishable from "forgotten".
 2026-07-23 13:40:53 +00:00
Johnathan Corgan 1f149bda32 Run the convergence-gate unit tests in both CI runners 
wait_until_connected decides whether every static suite proceeds or
gives up, so a regression in it turns those suites' verdicts into noise.
Its unit tests existed and were invoked by nothing.

Wire them into ci-local.sh beside the parity and log-string checks,
above the mode branches so --only and --test-only are gated too, and add
the matching step to the ci-parity job in ci.yml. Putting them in one
runner only would create exactly the drift check-ci-parity.sh exists to
catch, and neither placement is visible to that checker since this is not
a matrix suite.

Verified by breaking what it guards rather than by observing green:
setting wait_until_connected's default near-converged slack to 0 fails
case4, records FAIL wait-converge, and exits the sweep 1. Restored after.

They pass, which nothing had established before — 13 assertions, all
green. They also take about 45 seconds rather than the "a few seconds"
the header claimed, and nothing had contradicted that claim because no
runner had ever invoked it. Header corrected with the measurement.

The orphan sweep this called for is done and there are now no
zero-reference *-test.sh files under testing/. It also settles a
disagreement recorded between the audit and the task file: ecn-ab-test.sh
is invoked by nothing, its one reference being a README description
rather than a driver, so the audit was right and the earlier sweep wrong
to call it a documented manual tool. interop-test.sh does have a driver
in interop-stress.sh and iperf-test.sh one in iperf-compare-refs.sh, so
those two were correctly classified.
 2026-07-23 13:38:25 +00:00
Johnathan Corgan 38a60c61da Record that churn-mixed's floors describe the invocation CI overrides 
The baseline floors added for this scenario were calibrated from archived
runs, and those runs are all the 10-node 120-second variant ci-local
invokes as "churn-mixed --nodes 10 --duration 120". The file itself
defaults to 20 nodes and 600 seconds, and --nodes sed-patches num_nodes
in a copy before the scenario loads, so the gating run and a bare
chaos.sh run are different scenarios. Nothing at the site said so.

The floors hold for both, but only because the gating run is the smaller
one. Retuning them against a bare 20-node run, which clears them easily
at 20 answering, 1 root, 19 parented and 69 sessions, would put them past
what the 10-node run reaches and turn CI red while a manual run stayed
green. Say that where someone retuning them will read it.

Confirmed by running the gating invocation directly: 10 answered, 2
roots, 8 parented, 18 sessions, all inside the ranges the floors were
built from. churn-mixed is the only scenario CI overrides this way; the
other twelve run their files as written.
 2026-07-23 07:27:16 +00:00
Johnathan Corgan 3d0a388511 Fail the ping test on an unknown topology profile 
Every section of ping-test.sh is guarded by the profile name, so an
unrecognised profile fell through all of them and the script exited 0
having run no assertion. A typo in the caller's argument produced a green
run that tested nothing. Give it an else branch that names the valid
profiles and exits 2.

No existing leg changes behaviour: the hosted runner gates the ping step
on matrix.type == 'static', which only static-mesh and static-chain
carry, and ci-local passes only the two names in STATIC_SUITES. Worth
noting while confirming that, though: ping-test.sh accepts a mesh-public
profile that neither runner ever passes.

Also record why the convergence waits are `|| true` — they are settling
delays rather than assertions, and the directed-pair pings are what
decides the run — and add admission-cap to the suite list in the
ci-local header, which ran it without listing it.
 2026-07-23 07:13:32 +00:00
Johnathan Corgan 5d3a3d7cad Give the assertion-free chaos scenarios a convergence floor 
Five scenarios named by the audit carried no assertions at all, so a run
in which the mesh never formed exited 0 and reported green. Add a
baseline assertion covering how many nodes answered, how many distinct
roots they agreed on, how many took a parent, and optionally how many
sessions were established, and apply it to those five plus the two cost
scenarios left unasserted by the previous commit.

For ethernet-only, ethernet-mesh, tcp-mesh, smoke-10, mixed-technology
and depth-vs-cost the values are not calibrated: one root and N-1
parented nodes is what a spanning tree is, and all provably-completed
archived runs of each show exactly that. churn-mixed is different and
says so at the site. A scenario that stops and starts nodes on purpose
does not hold a single tree, and its six completed runs end with two or
three roots and seven or eight of ten nodes parented, so its floors sit
one step outside the observed range. That leaves them catching a mesh
that collapsed rather than one that churned, which is the most a sample
of six supports.

This gives Ethernet transport its only assertion anywhere in CI. The
three tests in src/node/tests/ethernet.rs are ignored for requiring
CAP_NET_RAW and neither runner passes --ignored, so until now nothing
exercised that transport with a verdict attached.

The floor is deliberately weak and deliberately not a substitute: it says
the mesh formed, not that it formed the tree the scenario describes.
Where those differ the scenario now says so in its own comments.
 2026-07-23 07:09:36 +00:00
Johnathan Corgan e9ca741e53 Assert parent selection where the cost scenarios actually specify it 
Four scenarios exist to test cost-based parent selection and each named
its expected outcome in a comment that nothing read. Add a tree_parents
assertion mapping a node to the parent it must have in the final tree
snapshot, and encode it for the two scenarios whose stated outcome the
implementation meets: cost-avoidance (n04 takes the fiber n03) and
bottleneck-parent (n06 takes the fiber n03, n09 keeps its only parent
n05). Both hold in all six provably-completed archived runs.

The other two are left unasserted on purpose, and each for a different
reason worth keeping distinct.

mixed-technology says n06 and n08 should both pick the fiber parent n03.
They do not. Across the five completed archived runs n06 picks n10 three
times and n08 picks n04 four times. Encoding the criterion as written
would red the scenario most runs, and encoding the observed behaviour
would bless something nobody specified, so the disagreement is recorded
at the site as an open question. n06 preferring n10 may well be correct
and the comment stale, since n10 reaches it by fiber too.

depth-vs-cost is different: its validation line does not name an outcome
at all, saying only that the choice "reflects the actual cost tradeoff",
which either answer satisfies. The corpus shows both occurring under one
seed, three runs to two. That scenario needs a protocol decision about
which parent is correct before it can have an assertion.

Compare parents by address resolved from the snapshot's own
my_node_addr, and fail rather than skip when a node is absent from the
snapshot or still claims to be its own root. Those two states are the
common ones in the older corpus and both produce the same "no match" a
wrong parent does, so only separating them keeps a harness problem from
reading as a routing verdict.
 2026-07-23 07:05:50 +00:00
Johnathan Corgan 5a11cf091d Give congestion-stress the assertions its criteria described 
The scenario listed four success criteria "verified via post-run
congestion snapshot". Nothing read the snapshot, so the scenario could
not fail on any of them.

Add a congestion_signals assertion taking a floor on the number of nodes
reporting each counter, and encode three of the four. The floors are one
node each because that is what the criteria say; tightening them to the
counts recently observed would assert something nobody wrote down.

The fourth criterion is not encoded, and that is the finding rather than
an omission. No node has reported a non-zero kernel_drop_events in any of
the 182 archived runs, so asserting it would red the scenario
permanently. It is recorded at the site as an unmet criterion and a
coverage gap: the transport drop-detection path the scenario names as its
second signal is exercised by nothing.

Two things about the corpus are worth carrying, both recorded in the
scenario. Only six archived runs carry a status.txt and are therefore
provably completed; all six meet the three encoded criteria with five to
nine nodes reporting each signal. The 176 older runs meet none of them,
and they are not invalid samples: each reached teardown far enough to
write an analysis.txt, and ECN landed before all but one of them. What
changed on 2026-07-22 is not established, since neither the scenario nor
netem.py, traffic.py or control.py has been touched.

Count the nodes reporting a signal rather than the magnitude of any one
counter, since a single node with a large count would satisfy a magnitude
test while proving the signal never propagated. A missing snapshot fails
rather than reading as an absence of congestion.
 2026-07-23 06:59:21 +00:00
Johnathan Corgan aab149e215 Fail a chaos run whose nodes logged errors 
The simulation exit ladder reported panics and failed assertions but said
nothing about ERROR-level log lines, so a run in which every node errored
on every line still exited 0.

Add a max_errors assertion and apply it to every scenario by default
rather than having each one opt in. This is a floor on what a green run
means rather than a property of an individual scenario, and a scenario
that has to ask for the floor is one that can forget to.

The default ceiling of 0 is what the archived corpus supports: across
2416 result directories no node log contains an ERROR-level line, while
the WARN counter extracted by the same code path ranges from 0 to 1135,
so the counter is known to discriminate rather than merely known to read
zero. A scenario that legitimately induces errors raises the ceiling in
its own YAML and says there why.

Reject rust_log: off alongside it, since that is the one level that would
leave the ceiling counting zero whatever the mesh did.
 2026-07-23 06:53:56 +00:00
Johnathan Corgan bc9bef717e Merge branch 'maint' 2026-07-23 05:29:07 +00:00
Johnathan Corgan 73e6917341 Give cost-stability the parent-switch ceiling its criterion described 
The scenario's success criterion existed only as a comment: count
"Parent switched" in n04's log, expect at most 5. Nothing read it, and
the scenario declared no assertions at all, so its exit code could only
report that the mesh came up and nothing crashed.

Add a max_parent_switches assertion and wire that criterion to it.

The assertion takes an optional node scope, which is the part that
matters. The existing sibling counts mesh-wide, and a criterion written
about one node's log is a different quantity from the sum over every
node: across archived runs of this scenario n04 is 1 or 2 while the
mesh-wide total ranges 3 to 6. Asserting the sum against a per-node
threshold would check something other than what was specified, and at
this threshold would also have failed several runs that were fine.

Counting nothing must not read as stability. A node id absent from the
topology fails explicitly rather than matching zero log lines and
sailing under the ceiling, and a scenario that declares a parent-switch
assertion while setting a log level that suppresses the events it counts
is rejected at load rather than passing vacuously after a full run. The
parse rejects a mistyped key, a missing, negative, boolean or
non-integer ceiling, and a node written empty or as a non-string -- that
last one because YAML renders a bare `node:` as null, which would
silently revert to the mesh-wide count.

The scenario comment now records what the threshold is worth against
the 11 completed archived runs rather than a single sample: 5 is well
above anything observed, the daemon's own parent hold-down caps
switches near 6 per run, and nearly every switch lands during initial
tree formation before any mutation fires. So this catches only
near-pathological reparenting. Tightening to 3 would make it a real
detector, but that changes the stated criterion rather than fixing it,
so it is left as a decision.

Verified against a live run: the assertion evaluates n04 at 2 against
mesh-wide 4 and passes, and with the ceiling temporarily set to 0 the
scenario exits 3, the assertion-failure code, which it previously could
not reach.
 2026-07-23 05:19:36 +00:00
Johnathan Corgan 8d7053fa72 Merge branch 'maint' 2026-07-23 04:16:51 +00:00
Johnathan Corgan 300deb0476 Reject an unanswered show_mappings instead of reading it as zero 
The mapping-reclaimed check expects zero mappings, and read the count
with `r.get('data',{}).get('mappings',[])`. An error response carries
no data field, so it parsed to zero and satisfied the check without the
gateway having answered at all. The expected value and the failure
value were the same number, which is the shape that lets an assertion
pass without asking anything.

Require the key to exist and be a list, and exit non-zero otherwise, so
the existing fallback turns an unanswered or malformed response into a
failure. The gateway always emits the key on success, including for an
empty set, so a genuine zero still reads as zero; confirmed against the
running gateway, where the reclamation check passes.

The earlier show_mappings poll shares the parse and is left alone: it
waits for a positive two, which no error response can produce. The
hazard was already documented there, and this is the call site that was
exposed to it.
 2026-07-23 04:16:42 +00:00
Johnathan Corgan 700ac581ee Require a timeout and a drop verdict in the firewall assertions 
Case (a) sends an unallowed inbound request that the ruleset should
drop, and accepted any non-zero curl status as proof. A drop produces
no RST, so curl can only hit its deadline and exit 28; connection
refused, an unroutable address or a missing listener all fail too, with
different codes, and the check counted those as a blocked connection.
Require 28 exactly, so the assertion distinguishes silently dropped
from failed for some other reason. Confirmed against the harness: the
real run returns 28.

The baseline check matched `counter packets`, which any counter rule
satisfies whatever its verdict, including one that accepts. Require the
rendered drop rule instead.

The drop-counter read had the same weakness plus a worse one: it took
field three of the first line mentioning a counter, which is the packet
count only when the line begins with `counter`. On a rule such as
`tcp dport 9 counter packets 42 bytes 3000 drop` it printed the port
number. That shape is reachable, because the shipped ruleset includes
the operator drop-in directory ahead of the trailing default deny and a
drop-in may add its own counted drop. Extract by position within the
matched text, and take the last match rather than the first, since case
(a) falls through to the default deny that the ruleset emits as the
final rule.

Note in place at the baseline check that it still only proves a counted
drop rule exists somewhere in the table, not that it is the trailing
one; asserting rule position is a larger change than this warrants.

The sample output in the README is updated to the new wording.
 2026-07-23 04:16:32 +00:00
Johnathan Corgan 98560594cf Prove ACL rejection per peer by matching a single log line 
The four rejection checks were independent greps over the whole log:
two npubs, a context and a decision. Together they proved only that
node-a mentioned each npub somewhere, rejected somebody on an inbound
handshake, and rejected somebody by denylist. Nothing tied a rejection
to a named peer, and node-a lists both denied peers as auto_connect
peers, so it logs their npubs on the outbound connect path whether or
not any rejection ever happened. The actual message was never matched.

Replace them with a helper that requires every given string on ONE
line, and assert per denied peer: the real message, that peer's npub,
and the denylist decision together. Strings match in any order by
chaining fixed-string greps over the surviving lines, so the assertion
does not depend on how the log formatter orders a message and its
fields. A grep over empty input yields the empty string rather than
anything a caller could mistake for a match, so an unreadable container
times out and fails instead of passing.

Two limits are written at the call sites rather than left implied. The
decision conjunct discriminates nothing, since the two allowing
variants return early and denylist is the only value that can reach
that warning. And node-a authorizes before dialing, so it emits a
fully formed rejection line for each denied peer on the outbound path;
these assertions are satisfiable without the inbound check running at
all, and it is the peer-count assertions that would catch that.

Verified against the running harness: all three find a real matching
line. The defect shape they replace was exercised offline first, with
the three strings spread across three lines, where the old checks
passed and the new ones fail.
 2026-07-23 04:16:16 +00:00
Johnathan Corgan fc8ebd5a06 Merge branch 'maint' 2026-07-23 03:23:18 +00:00
Johnathan Corgan ebf34ae712 Read the wall clock directly for Nostr traversal timestamps 
The traversal clock cached a Unix millisecond value and an Instant at
first use, then served every later call by advancing the cached value
with the monotonic elapsed time. A monotonic clock does not tick while
the host is suspended, so once a machine had slept the returned value
trailed real time by the sleep duration for the rest of the process
lifetime, and never re-synced.

Almost everything that clock feeds is an absolute timestamp. The NIP-40
expiration tags on adverts and traversal signals, and the issuedAt and
expiresAt fields of offers and answers, are all computed as now plus a
TTL; the freshness and cache-pruning paths compare it against a
peer-authored, signed created_at. Once the host had slept longer than
signal_ttl_secs, every offer was published already expired, relays
dropped it, and the initiator timed out waiting for an answer with
traversal broken until a restart.

Read the wall clock on every call instead. This also removes a
mismatch inside the traversal failure-state map, which was written
here from the cached clock but written and read from the node
lifecycle with the real one.

Not platform-specific: monotonic clocks exclude suspended time on
Linux and Windows as well, so this affected any host that suspends.
A laptop is simply where a process lives long enough across a sleep
to notice.

The interval-shaped consumers hold up under a clock step. A forward
step, which is what a resume produces, saturates the punch start delay
to zero, and the attempt's own bounds are monotonic deadlines that are
unaffected. A backward step lengthens that delay and can cost one punch
attempt, which retries. Early eviction from the replay window cannot
admit a replay under the shipped defaults, because the freshness window
a replayed offer must also satisfy is strictly narrower than the replay
window itself.

The added test pins the contract and fires on a host that has genuinely
suspended, but it is not a regression guard for this defect: nothing
reachable from a unit test can simulate a suspend, so on a machine that
has not slept the old implementation passes it too. Its comment says
so rather than leaving a false sense of coverage.

Reported in https://github.com/jmcorgan/fips/issues/128
 2026-07-23 03:23:00 +00:00
Johnathan Corgan a1222d6d75 Merge the open !FIPS access SSID layer from GitHub 2026-07-23 03:02:54 +00:00
ArjenandGitHub f624013b83 feat(openwrt): open !FIPS access SSID layer (#126) 
Access layer for phones and laptops to reach FIPS routers on OpenWrt,
stacked on the 802.11s mesh backhaul from #123. Squashed from five
commits by Arjen (Origami74); their original messages follow.

* feat(openwrt): open !FIPS access SSID — fips-ap-setup helper, default transport binding, how-to

  Client access layer for phones and laptops: every FIPS router
  broadcasts the same open SSID ('!FIPS' — the leading '!' sorts it to
  the top of alphabetically ordered network pickers), forming one
  standard ESS. Clients save it once and roam between all FIPS routers
  natively, with FIPS's Noise IK handshake as the only security layer:

  - fips-ap-setup: opt-in UCI helper that creates the 'fips-ap0' open AP
    (encryption none — security type must be uniform across routers or
    clients treat the ESS as different saved networks), an isolated
    network with a static ULA /64, RA-only odhcpd addressing (stateless
    SLAAC, no DHCP — the minimum that satisfies Android's provisioning
    check; no internet by design, so phones keep cellular as default
    route), and a locked-down fips_ap firewall zone (no path to br-lan
    or the WAN; only ICMPv6, mDNS, and the FIPS transports reachable).
    'remove' subcommand undoes it. Radio setup stays opt-in; a package
    must not commandeer radios on install.
  - fips.yaml: ship 'ap0'/'ap1' Ethernet-transport entries commented out
    (matching the 802.11s mesh backhaul) so a stock install that never
    creates fips-ap* logs no per-boot "interface missing" bind warning;
    fips-ap-setup uncomments the matching block when it creates the
    interface and re-comments it on remove.
  - Regression test: extend shipped_openwrt_config_parses to assert the
    ap0/ap1 entries ship commented out and still parse once uncommented,
    alongside mesh0/mesh1.
  - Packaging: install the helper in ipk/apk/buildroot (three synced
    copies), extend CI structural checks and shellcheck targets.
  - docs/how-to/set-up-open-access-ssid.md: full guide, including the
    one-time 'no internet, stay connected' acceptance (stored per SSID,
    covers every FIPS router) and the security-type-uniformity
    constraint.

* docs(openwrt): correct open-SSID/mesh security framing — peering is open by design

  The fips-ap-setup/fips-mesh-setup comments and both how-tos claimed a
  stranger "cannot pass the FIPS handshake" / "their frames die at the
  handshake" / the handshake surface "drops them". That is wrong: FIPS
  peer admission is open. An inbound handshake from any net-new identity
  is promoted (node::handlers::handshake::promote_connection), gated only
  by the daemon's max-peers cap — there is no allowlist, no PSK, and the
  AuthChallenge path is not wired to admission. The Noise IK handshake
  provides authentication (no impersonation of another identity, no MITM),
  not authorization.

  Restate the model accurately in all five places: a stranger on the open
  SSID (or the open mesh) can associate AND form a FIPS peer link — that
  is the point of open access. Containment is the isolated fips_ap zone
  (no path to br-lan or the WAN) plus the max-peers cap, not the
  handshake. Clarify that AP client isolation is an L2 control only: a
  peered stranger is an overlay peer like any other, so the FIPS overlay,
  not L2, is the trust boundary between clients.

* feat(openwrt): serve DHCPv4 on the access SSID from a fixed roamable subnet

  RA-only addressing satisfied Android's provisioning check but left
  anything expecting IPv4 with a self-assigned address and a "no IP"
  complaint. dnsmasq now leases out of 10.21.<N>.0/24 (N = radio index;
  prefix echoes FIPS port 2121), deliberately identical on every router:
  a roaming client keeps its lease across the ESS, and dnsmasq's
  authoritative mode — the OpenWrt default, pinned by the helper — ACKs
  the renew a foreign router never issued. Lease collisions across
  routers surface as a NAK on renew and the client re-DHCPs.

  The dhcp section's 'dhcpv4 server' is read by both dnsmasq (default
  images) and odhcpd (only with maindhcp), so either arrangement serves.
  A DHCPv4/udp-67 accept rule joins the fips_ap zone; DHCPv6 stays off,
  the ULA RA stays as-is, and nothing depends on an upstream. The zone
  remains isolated — no forwardings, no internet.

* feat(openwrt): enable mDNS rendezvous from fips-ap-setup

  Phone FIPS apps cannot open raw-Ethernet sockets, so DNS-SD is how they
  find the router's daemon — but node.rendezvous.lan defaults to off. Ship
  the lan block commented in fips.yaml (consistent with the apN transport
  entries) and have fips-ap-setup uncomment it when creating the access
  SSID. The awk match is scoped to node.rendezvous because
  transports.ethernet carries a 'lan' entry at the same indent. The switch
  is daemon-wide, so 'remove' deliberately leaves it on rather than guess
  whether other transports rely on it.

* fix(openwrt): bind UDP dual-stack [::]:2121 so access-SSID clients reach it

  The shipped router config bound the UDP transport "0.0.0.0:2121" (IPv4
  wildcard) while the mDNS LAN advert announces every interface address,
  including the router's IPv6 link-local — which phones on the !FIPS
  access SSID rightly prefer (their cellular default route swallows v4,
  and fd00::/8 is captured by the Myco mesh TUN). Result: the client's
  Noise msg1 arrives on an unbound v6 port and is silently lost; the
  handshake resends and times out.

  Symptom chain (observed on-device): mDNS resolve OK, platform push OK,
  "Sent Noise handshake message 1" to [fe80::…%N]:2121, four resends, no
  reply, 30 s stale-timeout.

  OpenWrt is Linux (bindv6only=0), so "[::]" accepts IPv4 via v4-mapped
  addresses too — nothing is lost. packaging/common is deliberately left
  on "0.0.0.0" for now: Windows defaults IPV6_V6ONLY=1, where "[::]"
  would drop v4 instead.
 2026-07-22 20:02:20 -07:00
Johnathan Corgan 16b1bc2c5c Merge branch 'maint' 2026-07-23 02:42:48 +00:00
Johnathan Corgan 291c4312dc Stop the ACL peer-count reader from turning "no answer" into zero 
wait_for_peers_exact fell back to 0 when it could not read a container's peer
count. Two of its six call sites are the ACL denial checks, which expect
exactly 0 connected peers — so a failed docker exec satisfied them on the
first iteration and the isolation property the suite exists to prove was
never observed. Confirmed against a container that does not exist: the old
reader returns "0" and the expect-zero comparison passes.

The reader now returns the empty string when the daemon does not answer,
which no numeric comparison can satisfy, and the timeout message says which
of the two happened — never answered, or answered the wrong number. Same
shape as admission-cap-test.sh's read_peer_count.
 2026-07-23 02:42:36 +00:00
Johnathan Corgan d21f818659 Check that log strings tests match on are still ones the daemon emits 
A test that greps the daemon's log for a message the daemon stopped emitting
does not fail. It stops observing, and an assertion built on it — especially
one expecting a count of zero — then passes because nothing can be seen
rather than because nothing happened. Several findings have come from that
one class, so it is now checked mechanically.

testing/check-log-strings.py extracts the strings test code matches against
daemon log text and requires each to exist in src/. It reads four shapes:
python `"..." in line` tests, the first argument of the shell log helpers,
bash associative-array pattern tables, and greps whose input is daemon log
output. Patterns carry regex syntax, so each is reduced to the longest
literal run of every alternation branch, with escaping resolved in the same
pass — `\.` is a literal dot and `.` is a wildcard, and conflating them would
let a pattern match text that is not there.

Scoping is by what a grep READS, not by what its file mentions. A suite greps
its own analyzer output, fipsctl JSON, Tor's log and ping output, and none of
those have to correspond to a string in src/; scoping on the file produced 34
false positives. Strings that legitimately do not come from src/ are named in
ALLOWED with a reason, so the exceptions are reviewable rather than invisible.

It found six dead strings, three of which were not previously known: a second
copy of "Excessive decrypt failures" in the interop pattern table, a dead
"Handshake error" branch inside an alternation whose other branches still
matched, and "bootstrap failed" in the NAT suite. All six are corrected to
what the daemon emits, or dropped where the surviving branch already covers
the case. Verified by breaking it two ways and confirming it reports.

Also here, since it is the same failure shape one layer down: the shared log
library returned an empty string for a container whose logs it could not
read, so analysing a missing container reported no panics, no errors and no
sessions, and exited 0 — indistinguishable from a clean run. It now reports
which containers could not be read and raises, matching the fix already
landed in the chaos copy.
 2026-07-23 02:42:25 +00:00
Johnathan Corgan fbd0dba870 Merge branch 'maint' 2026-07-23 02:06:06 +00:00
Johnathan Corgan 6c52b0e01e Let the static test network float so concurrent CI runs cannot collide 
Two local CI runs on one host both asked docker for 172.20.0.0/24 and the
second lost its whole static family to "Pool overlaps". Docker honours a
fixed subnet request verbatim, so the only robust fix is to stop making one:
fips-net now requests no subnet and docker assigns from its own pool, which
cannot hand the same range to two runs.

That means node addresses are not known before `up`, so peers address each
other by container hostname instead. The generator emits node-<id>, or the
topology's docker_host where the compose hostname differs — only the gateway
profile, whose services are gw-*. External peers keep the address the
topology gives them, since it is not ours to assign. The resolv.conf mount
stays: dnsmasq is what forwards these names to docker's resolver and .fips
to the daemon, so removing it would take out every .fips assertion.

generated-configs is now per-run as well. A shared directory let two runs
overwrite each other's node configs, which the subnet collision had been
hiding by killing runs before that window opened. The generator, the compose
bind mounts and env_file, the six scripts that read it, and teardown all
follow FIPS_CI_NAME_SUFFIX; unset, every path renders as before. Teardown
keeps the directory after a failed run, where it is the evidence of what the
failing nodes were configured with.

Three things this exposed that were wrong independently:

admission-cap built its tcpdump patterns from the topology file's docker_ip
literals. Floating the subnet makes those match nothing, which would have
left its expect-zero "no Msg2 leaked" assertion passing because it could no
longer see anything at all. It now reads addresses from the running
containers. Restarting the denied peers together also made them swap
addresses, so each peer's counts were really the pair's total; they are
restarted one at a time now, and a check fails the suite outright if two
denied peers ever share an address, because per-peer attribution is
impossible once they do.

Attribute lookups in the generator used a fixed ten-line window and read the
next node's fields when a node omitted an attribute. An external node
followed by an internal one was classified as internal, which under hostname
peering would emit a name that resolves nowhere. Lookups are bounded to the
node's own block; generated output is byte-identical for all eight
topologies.

The rekey outbound-only variant used to rewrite peer addresses to hostnames
to set up its scenario. The generator now does that everywhere, so the
rewrite matched nothing and was silently doing no work. It asserts the
premise instead, and fails if a numeric address ever reappears.

Verified by running three instances of this compose at once — tcp-chain plus
two independent meshes — which drew 10.128.2/3/4.0/24 with no overlap while
both meshes passed ping-test 20/20 over the real .fips path. tcp-chain is
run by neither CI runner, so it was checked by hand: chain peer counts 1/2/1
and multi-hop .fips reachable both directions over TCP.

gateway-lan still pins its own IPv4 and fd02:: ranges and is unchanged here,
so the gateway profile is not yet concurrency-safe.
 2026-07-23 02:05:56 +00:00
Johnathan Corgan c2d6283bd6 Merge branch 'maint' 2026-07-22 22:57:33 +00:00
Johnathan Corgan 428773490f Close the gaps an adversarial review found in the new CI guards 
Four of these are places the parity guard could stop covering something without saying so, which is the failure mode it exists to prevent. Removing the deb-install suite array left it green, because the dispatch cross-check compared against a hardcoded name rather than the array it was meant to read. An arm-shaped line at an unexpected indent, or one written in quotes, was dropped silently by the arm pattern; unexpected indents are now a hard error rather than a quiet skip, and quoted arms parse. A matrix leg with a type of chaos or deb-install but no scenario key raised a Python traceback instead of reporting a problem, and a leg carrying a scenario but no suite key was skipped entirely even though scenario is now the identity for those legs.

The e2e builder no longer folds docker create's stderr into the container id. Docker prints warnings on success as well as failure, so a platform-mismatch warning would have become part of the id and made every later reference to that container fail, turning a working extraction into a spurious red.

Also corrects the comment about registering a new chaos assertion type, which named only one of the two key sets that actually need it.
 2026-07-22 22:41:24 +00:00
Johnathan Corgan 56f00058d4 Run the CI parity guard in both runners 
The guard that keeps "local green" and "GitHub green" meaning the same thing was invoked by nothing. Its only entry point was an exec behind an explicit --check-parity flag, which by exec-ing could not be combined with an actual run, and no workflow step called it at all. A guard nobody runs is a guard that does not exist, which is how the mixed-profile divergence on next survived unnoticed.

It now runs as the first stage of every local run and as its own job on GitHub. Locally it reports through the same result-recording path as every other stage, so a divergence sets the run's exit status instead of scrolling past. It is deliberately placed above the mode branches, so a divergence also fails --only, --test-only and --build-only runs: whichever subset was asked for, the claim that a local result means what a GitHub result means is what has broken. The GitHub side installs pyyaml explicitly rather than assuming the runner image carries it.

--check-parity keeps working exactly as before. Three comment blocks that described the old folded comparison are corrected here, in the workflow, the local runner and the testing README; they were spread across three files and only two mention the guard by name, so the third is best found by searching for what it claims rather than for the script.
 2026-07-22 22:03:08 +00:00
Johnathan Corgan e578a11b7a Compare every CI leg in the parity guard, not a folded token 
The guard collapsed all thirteen chaos legs to the token "chaos" and all five deb-install legs to "deb-install", so sixteen of the thirty-four integration legs were invisible to it. Deleting twelve chaos legs and four deb-install legs from a copy of the workflow left it still reporting "CI parity OK". The fold existed because the guard compared the cosmetic suite name, which really does differ between the runners; both sides have carried a directly comparable scenario field all along, so the fix is to compare that instead, along with the chaos flags.

The local suite set is now discovered by sweeping ci-local.sh for suite arrays rather than reading a hardcoded list of variable names, and the deb-install distro list is read from the suite's own script. A suite dispatched with no backing array is invisible to either approach, so every run_suite arm is now checked to have one and the guard names any that does not. That is not hypothetical: next dispatches its mixed-profile suite without an array, and the guard could only report it as missing from the local runner without being able to say why.

The guard also now refuses to run rather than failing obscurely when python3 or pyyaml is absent, since a parity check that cannot run must not look like one that passed. Verified by breaking each thing it guards: legs deleted from the workflow, drifted chaos flags, a fabricated suite array, and a dispatch arm whose array was removed. That last case initially did not fire because the arm pattern was pinned to the wrong indentation, which the test caught.
 2026-07-22 21:59:28 +00:00
Johnathan Corgan 99cb0a51fd Reject unknown keys when loading a chaos scenario 
The scenario loader read every section and member through raw.get with a default, so a mistyped key was silently ignored and the block it belonged to stayed default-constructed while the YAML still looked correct on the page. Writing "assertion:" for "assertions:" disarmed a scenario's only assertions and left it unable to return a non-zero exit code; "link_flap:" for "link_flaps:" turned off the churn the scenario existed to inject, and a calm mesh then ran under the name of a chaos test.

Keys are now checked at load time, at the top level and inside every section with a fixed schema, including the netem policy bodies, the {min, max} ranges and the assertion bodies themselves. The error names the offending key, the section it appeared in, and the keys that section does understand. Three mappings are deliberately exempt because their keys are names the author chooses rather than a schema, and two sub-trees are passed through whole; all five are listed at the key sets with the reason.

Adding an assertion type now means registering it, so the next person to add one gets a rejection rather than a silent no-op. That coupling is the point and it is written down where they will see it. Verified against the seventeen scenarios in the tree, all of which still load, and against fixtures for each failure mode: a top-level typo, a nested typo, a typo inside an assertion body, and a section left empty, which previously raised an unrelated AttributeError from the next line of the loader.
 2026-07-22 21:56:41 +00:00
Johnathan Corgan e9ca16f3a1 Stop the DNS resolver e2e tests certifying a binary that was never built 
The e2e scenarios build fips and fips-gateway in a builder image and extract them with two docker cp calls whose stderr went to /dev/null and whose exit status was never tested. Because build_fips_for_e2e ended in a log call it returned 0 on every path, so the caller's guard could not fire. A failed extraction left the previous run's binaries in the cache, they satisfied the caller's executable check, and the five e2e legs then exercised the previous commit's code and reported green.

The extraction now deletes the cached binaries before it starts, which is what makes the stale case impossible rather than merely detected, and it checks each docker cp independently, reports the failure with docker's own message, verifies each extracted file is non-empty and executable, and returns an explicit status. This mirrors what the deb-install suite already does.

Confirmed by inducing the failure rather than by observing a green run. With both extractions broken and a stale binary in place, the old code logged "Cached fips (26936 bytes)" and went on to build the runtime image around it; the new code reports both docker errors, deletes the stale binaries and fails the suite. Breaking either extraction on its own is caught separately, and an unmodified run passes 7 of 7.
 2026-07-22 21:54:37 +00:00
Johnathan Corgan 946df54d0f Merge maint into master, bringing the chaos suite's failure reporting with it 
The chaos simulation could not report a failure on this line either: scenarios collided on globally-scoped container names and a shared config directory, the runner swallowed whatever they raised, and the harness discarded their exit codes before anything read them. The five commits scope the names and directories per scenario, give an aborted run its own exit code and let that code reach the summary, stop a failed scenario harvesting whichever containers happen to hold its names, and keep the daemon's reason for a failure instead of throwing it away.

Testing only, and the merge is textual: every chaos file matches maint byte for byte, and the ci-local divergence on this line does not touch the changed regions.
 2026-07-22 09:33:58 +00:00
Johnathan Corgan ab750f60b3 Record why a chaos docker command failed 
Compose commands run with their output captured and with check set, so a failure raised a CalledProcessError and nothing ever looked at what the daemon had said. That exception reports the argv and the exit status and nothing else, so every scenario that died bringing its containers up left a runner log saying only that docker compose returned non-zero exit status 1, and the reason was captured and then discarded. In the run that prompted this work twelve scenarios failed that way and why they failed can no longer be established at all. Log the captured stderr and stdout before the error goes anywhere, keeping the tail of each so a verbose failure cannot fill the log file. A command that times out raises before its exit status is ever read, so that path records whatever it had emitted too, rather than leaving the same hole one shape over. Reproduced against both of the shapes this is meant to catch: an unusable network prefix now records the daemon's parse error, and a name already held by another container now records the conflict and names the container holding it. The success path logs nothing new, and a completed scenario's runner log is unchanged line for line.

The raising behaviour is deliberately untouched, since the abort flag and the exit code that came with it depend on it. The check is applied here rather than by subprocess so that a caller who passed check false gets the same record: both such callers are the compose down in teardown, which today can leave a mesh behind without saying so.

The same omission was costing the veth path its reasons. Failing ip commands had their stderr logged at debug, which the runner does not emit unless asked for verbose output, so a failed pair creation reached the log as nothing more than the names of the two interfaces it could not create. Raise that to a warning. The deletes issued to clear a stale pair are expected to fail and still say nothing.
 2026-07-22 08:26:59 +00:00
Johnathan Corgan 226c6994d3 Harvest chaos results only from a mesh that actually started 
Teardown runs from a finally, so it also runs after a failed setup, and it was guarded only on the topology and the compose file, both of which are set well before any container exists. A scenario that died bringing its containers up therefore ran the whole harvest anyway: final snapshots, docker logs, the analysis, the assertions and the metadata, all addressing containers by names that are global to the host. What it left behind was not an empty directory an investigator would notice but a full and plausible one, in the recorded case ten node logs and an analysis reporting two promotions and two parent switches, every byte of it from a different scenario's mesh. Gate the harvest on whether the containers ever started, and leave the compose down outside that gate so a partly successful start still gets cleaned up.

Record the outcome in a status file in every result directory, naming the run as completed, interrupted, aborted, setup-failed or teardown-failed, alongside the scenario, the seed and the container names it used. With the harvest gated, the presence of an analysis file is now itself proof that the scenario's own mesh existed, and the status file says which of the several ways a run can end applies. A run cut short by a signal keeps the existing exit codes rather than gaining one of its own: what it collected before stopping is real and still worth reporting, the status file records that it was truncated, and every wrapper already reports a Ctrl-C of its own.

Log collection never looked at the status of docker logs. It concatenated stdout and stderr unconditionally, so collecting from containers that were not there wrote the daemon's "No such container" reply into each node log and analysed the result as a mesh with no panics, no errors and no sessions, which reads exactly like a clean run and exits zero. Check the return code, and treat a harvest that cannot read every container, or that reads none, as a failed teardown. A teardown that raises now also reports on the same footing as a run that never started, instead of escaping past the exit codes as a bare traceback and being read as a malformed command line.
 2026-07-22 08:02:38 +00:00
Johnathan Corgan 4f55a281ac Let a failed chaos scenario fail the run 
The simulation caught every exception a scenario raised, logged it, and exited 0, so a scenario that died during setup was recorded as a pass. Local CI discarded the exit code in any case: run_chaos ends in a call whose own last statement is an echo, so it returned 0 whatever the scenario did, and the parallel launcher's wait therefore always succeeded and always recorded a pass. Between them no chaos failure of any kind could turn a local run red, and none could since the script was written.

Carry the abort out of the run and give it its own exit code, ordered ahead of the two content codes so a run that never produced a mesh cannot report that mesh's panic and assertion counts. Return the scenario's status from run_chaos so the launcher sees it. The documented exit codes were stale before this change and are rewritten in full, including that the argument parser rejects a malformed command line with the same code that reports panics.

Expect scenarios that have been reporting green to start reporting red. In the last recorded local run twelve of the thirteen scenarios never got past starting their containers, and all thirteen were counted as passes; those that now execute have not run for months, and their assertions have never been evaluated against a mesh of their own.
 2026-07-22 07:37:48 +00:00
Johnathan Corgan 2ef36c0071 Scope chaos host interface names per scenario and reap orphaned pairs 
Ethernet edges get a veth pair created in the host namespace and then moved into the containers, named from the node ids alone. ethernet-mesh and ethernet-only both connect n01 to n04 and always run together, so one tore down the other's live link. Hash the scenario suffix to four hex characters and insert those after the vh prefix; 15 characters is far too few to carry the suffix itself.

The names are now run-specific and never regenerated, so the delete-before-create no longer reclaims an interface an earlier run abandoned. ci-cleanup.sh takes over, deriving the names from the suffixes a run's teardown hands it. It is the first thing that script removes that is neither a docker object nor labelled, so an unscoped reap can sever a running bare simulation's links; testing/README.md and the --reap help say so.
 2026-07-22 07:35:11 +00:00
Johnathan Corgan 5be7c6d0cb Give each chaos scenario its own container names and config directory 
Chaos scenarios run four at a time under local CI, but every one of them claimed the container names fips-node-nNN and wrote its generated configs and compose file to the same generated-configs/sim directory. Container names are global in Docker and are not scoped by the compose project, so concurrent scenarios collided on both, and a scenario could start containers from a compose file another had overwritten.

Thread the existing FIPS_CI_NAME_SUFFIX into the simulation. run_chaos narrows the run-wide suffix to the scenario, and the sim reads it once when the topology is built, applying it to the container names and to the config directory basename. The compose template renders the name through the topology accessor instead of duplicating the literal, so one expression produces every chaos container name.

The suffix is empty when the variable is unset, so a bare chaos.sh run and the hosted CI jobs render byte-identical names and paths. Verified by rendering every scenario's compose file before and after with the variable unset and diffing.
 2026-07-22 07:35:11 +00:00
Johnathan Corgan 29923cf676 Merge maint into master, reconciling the two hop limit implementations 
Both branches carry the same correction, authored separately because the
sans-IO relocation moved the logic into a core that returns an outcome rather
than acting inline. Git resolved the file move as a rename and offered to merge
the two versions textually, which would have been wrong in both files.

Resolved by keeping master's re-expression wholesale for the handler and its
tests, and taking the changelog entry from maint. Verified rather than assumed:
the resulting tree is byte identical to the reviewed and CI-green master commit
across every code path, and the merge contributes nothing but the fifteen line
changelog entry. Since the code is unchanged from a tree that already passed the
full local suite, that result still stands and no re-run is owed.

Nothing was lost from maint's side. Its tests are written against a file layout
that does not exist here, and master carries equivalent coverage written against
its own shape, including the multi-node chain and the cache warming cases that
the maint version does not have.
 2026-07-22 02:59:44 +00:00
Johnathan Corgan 34431209f5 Follow IP semantics for the session datagram hop limit on the refactored path 
The same correction as on the maintenance line, re-expressed rather than merged,
because the sans-IO relocation moved this logic into a synchronous core that
returns an outcome instead of acting inline.

Delivery to the addressed node is no longer gated on the hop limit, and the
decrement happens before the drop decision, so a datagram that would leave with
zero is not sent. The reachable radius does not change: the old behavior refused
to deliver at zero but was willing to send at zero, and the new behavior is the
mirror of that, so the two cancel.

The shell above the core needed changing too, and that was the real work. It
carried two hop-limit predicates of its own, both justified by comments citing the
ordering that existed before the refactor. The first gated coordinate cache
warming, which this fix requires to be unconditional; left alone it would have
suppressed warming twice over, once for a datagram addressed here and now
delivered, and once for a transit datagram dropped for hop limit whose plaintext
coordinates are still perfectly usable. The second gated next hop resolution,
which carries a cache touch side effect, and now mirrors the core's
would-leave-zero rule instead of the old one. Neither divergence would have
surfaced as a test failure.

That last point is worth stating plainly, because it nearly went the other way.
Restoring the warming gate on top of the corrected core passes the entire suite
without a single failure. The tests could not see the seam between the core rule
and the shell's copy of it. So this adds coverage that can: a three node chain
pinning the decrement across a real hop, which is the only thing exercising the
shell predicates composed with the core, and two warming tests at a zero hop limit
covering both the delivered and the dropped case. Each was checked by putting the
specific defect it targets back and confirming the test fails.

One existing test that claimed to cover hop limit behavior asserted nothing
whatsoever. It asserts now, but those assertions do not discriminate this change,
since the case it exercises behaves identically under both rules. They guard
against the test going vacuous again, not against this defect, and the comment
says so.
 2026-07-22 02:32:34 +00:00
Johnathan Corgan 8ca2362e6c Follow IP semantics for the session datagram hop limit 
The forwarding path tested the hop limit before testing whether the datagram was
addressed to this node, and decremented only on the forwarding branch. Two
consequences followed. A datagram addressed here that arrived with a zero hop
limit was dropped rather than delivered. And a forwarder receiving a transit
datagram with one hop left transmitted it with zero for the next hop to discard,
wasting a transmission on every expiring datagram.

Delivery to the addressed node is no longer gated on the hop limit, and the
decrement now happens before the drop decision, so a datagram that would leave
with zero is not sent. Saturating subtraction folds the two arrivals that cannot
be transmitted, already exhausted and last hop, into a single test.

The reachable radius does not change, which is worth stating because it is easy
to assume otherwise. The old behavior refused to deliver at zero but was willing
to send at zero, and the new behavior is the mirror of that, so the two cancel: a
path of h links still delivers for any source value of h or more. What this
actually buys is the wasted transmission, the exhaustion counter charging at the
node that makes the decision rather than the one after it, conformance with the
specified semantics, and one real gain during a rolling upgrade, where an
unupgraded forwarder feeding an upgraded destination delivers one hop further
than either version does on its own.

Coordinate cache warming moved ahead of both decisions, so a transit datagram
later dropped for hop limit still contributes its plaintext coordinates. The only
arrivals this newly warms from are those with a zero hop limit, and every insert
they can make is already achievable with a value of one, so it grants nothing
that was not already available and adds no memory growth the cap does not bound.

Four tests added: delivery at zero to this node, the transit drop at one, the
transit pass at two, and the one per hop decrement across a three node chain,
which brackets the emitted value from both sides since it is only observable
through what the next hop does. One existing test was renamed because its
destination was this node, so it never exercised the transit path its name
claimed, and it asserted nothing at all.
 2026-07-22 01:53:53 +00:00
Johnathan Corgan 80c956a6fd Merge maint into master after the v0.4.1 rollover 
maint carries only its 0.4.2-dev version bump; master keeps its own
0.5.0-dev development version. Recorded as a linkage merge so later
forward-merges of real fixes land cleanly.
 2026-07-19 19:06:08 +00:00
Johnathan Corgan 75d7077880 Open 0.4.2-dev cycle on maint after v0.4.1 release 
Bump the version to 0.4.2-dev and update the status badge and prose. The
0.4.x patch line continues; v0.4.1 remains the shipped release it points at.

Follows the shape of the v0.4.0 rollover commit. No maint reset was needed
this time: v0.4.1 was cut from maint, so the tag is already on this line and
it simply continues.
 2026-07-19 19:05:13 +00:00
Johnathan Corgan a47ddbd5a5 Merge maint into master after the v0.4.1 release 
Carries the v0.4.1 content up the one-way branch flow. The release itself
belongs to maint, so the parts of this merge that identify a version are
resolved in master's favor and the parts that describe the project's state
are taken from maint.

Kept master's: Cargo.toml and Cargo.lock at 0.5.0-dev, the README status
badge, and the "FIPS is at v0.5.0-dev on the master branch" line. A patch
release consumes no minor version and master's development line is
unaffected by it.

Took from maint: the bloom antipoison FPR default change and the
duplicate-definition fix it rests on, the two docs that describe them, the
new v0.4.1 release notes, the correction to the v0.4.0 release date, and the
[0.4.1] changelog section, which slots below master's own [Unreleased] and
above [0.4.0]. The root RELEASE-NOTES.md mirror moves to v0.4.1 because it
tracks the latest shipped release, which also clears the stale provisional
date it had been carrying.

One line needed splitting rather than choosing: the README said "v0.4.0 has
shipped" inside the paragraph that identifies master as 0.5.0-dev. The
version identity stays master's and the shipped-release pointer moves to
v0.4.1.

Quartet green on the result: 1645 tests passed, clippy clean with -D warnings.
 2026-07-19 18:57:52 +00:00
Johnathan Corgan 15db6471db Set version to 0.4.1 and correct the v0.4.0 release date 
Drop the -dev suffix from the package version and refresh the lockfile.
The version is single-sourced from CARGO_PKG_VERSION, so build.rs and
src/version.rs need no change.

Also fix the archived v0.4.0 release notes, which still read
"2026-06-21 (provisional)" while the changelog records the actual
release date of 2026-06-27. The date was confirmed in the changelog at
the time but not in the two release-notes files that carry it, so the
shipped notes have been showing a wrong date and the word provisional
since that release. The root mirror is regenerated for v0.4.1, so only
the archived copy needed correcting.
 2026-07-19 17:59:37 +00:00
Johnathan Corgan 146d19a8d8 Add v0.4.1 changelog entry and release notes 
Back-fill the changelog for the six user-facing changes on this line
since v0.4.0: the antipoison FPR cap default, the bloom probe and
secp256k1 context performance work, the coordinate-cache and path-MTU
fixes, and the removed parent_switched counter. The Unreleased section
was empty, so all six entries are new.

Write the v0.4.1 release notes and mirror them to the root copy. The
notes lead with the FPR cap change and state plainly that this is the
second raise of that default in two releases, that it buys headroom
rather than fixing the underlying fixed-filter constraint, and that the
structural remedy is the v2 filter work. They also flag the rolling
upgrade window, where an upgraded and a not-yet-upgraded node can
disagree about mesh size, and point consumers of parent_switched at
parent_switches.

Bump the README status badge and the roadmap prose together so the two
agree.
 2026-07-19 17:21:57 +00:00
Johnathan Corgan bda327b5f5 Raise the bloom antipoison FPR cap default to 0.20 
The inbound FilterAnnounce cap at 0.10 rejects aggregates that are
legitimately near their operating ceiling, before the network reaches
the fixed-filter capacity limit. Raise the default to 0.20, which
corresponds to fill 0.7248 at k=5, about 2,114 entries on the 1 KB
filter (Swamidass-Baldi).

Also remove the duplicate default definitions in BloomConfig. Each
default was written twice, once in impl Default and once in the serde
default function, with nothing enforcing that they agree, so a config
file that omits the key took a different path from one that sets it.
impl Default now delegates to the serde default functions, leaving a
single source of truth.
 2026-07-19 17:18:49 +00:00
Johnathan Corgan ea74cd7e58 Merge branch 'maint' 2026-07-19 07:40:07 +00:00
Johnathan Corgan 78377208af Fix two sidecar container references that skipped the per-run suffix 
The isolation loop and the failure-log dump both built container names
without the run suffix, so under a suffixed run they addressed containers
that do not exist.

The consequence was worse than the visible failure. Two of the three
checks in that loop assert a ping FAILS, and a ping into a non-existent
container fails for the wrong reason - so both passed vacuously, and the
security assertion the loop exists for was silently a no-op. Only the
third check, which expects a ping to succeed, noticed anything was wrong.

Add a guard that fails loudly when the container is absent, so a future
naming slip cannot quietly turn these back into no-ops rather than
surfacing as one confusing failure among two false passes.

These sites were missed because the suite passes when run by hand: with no
suffix set the unsuffixed names are correct, and the defect only appears
under the automation that sets one. Verified this time with the suffix
set, which is the condition that matters: 18 of 18, container names
resolving to real containers.
 2026-07-19 07:40:07 +00:00
Johnathan Corgan 37adb13d5b Merge branch 'maint' 2026-07-19 07:17:26 +00:00
Johnathan Corgan 26a579b1c9 Claim a free address range for the sidecar network instead of hardcoding one 
Making the sidecar network name unique per run exposed a latent problem
rather than causing a new one. With a fixed name, compose found any
existing network and reused it, which quietly masked the fact that the
address range was hardcoded. With a unique name there is nothing to
reuse, so the range is requested every time and the run fails outright if
anything already holds it.

Claim the range instead of assuming it. The suite now tries to create its
network on a candidate range and, if the daemon reports an overlap, moves
to the next candidate. The daemon's own address pool becomes the arbiter,
so two concurrent runs cannot land on the same range at all - as opposed
to a range derived from the run identifier, which only makes a collision
unlikely, and a collision is the precise failure being avoided.

The node addresses are derived from whichever range is claimed rather than
written alongside it, including the gateway address the isolation check
probes. Moved to a range that nothing in the tree claims and that sits
outside the daemon's default pool.

Only an overlap is worth advancing on. Any other creation failure is real
and is reported with the daemon's own message, rather than being retried
sixty-four times and buried - a lesson from a failure elsewhere in the
harness this week that was undiagnosable because its error output was
discarded.

All three nodes now attach to the pre-created network as external, so
teardown removes it explicitly.

Verified: three concurrent claims take three distinct ranges; a
non-overlap error fails fast rather than looping; the suite passes 18/18;
and the claimed gateway address is confirmed pingable when not firewalled,
so the isolation assertions still mean something.
 2026-07-19 07:17:19 +00:00
Johnathan Corgan 93a5b71728 Merge branch 'maint' 2026-07-19 06:46:55 +00:00
Johnathan Corgan 3ebb14eda4 Make the admission-cap final peer_count read tolerate a busy daemon 
The suite red-ed with an empty final peer_count while every substantive
assertion passed: both denied peers showed inbound handshake attempts and
zero outbound responses, which is the gate this suite exists to verify.

Two pre-existing defects combined. The final count was sampled once, in
the same second the load driver issues its last round of peer restarts, so
the read can land on a daemon busy with those restarts and return nothing.
And the fallback meant to cover that could never fire, because the shell
applies it to the last stage of the pipeline, which succeeds on empty
input - so an unanswered query was indistinguishable from a genuine zero,
and reported as a cap violation.

Poll the final read for a short window instead, and give the two cases
distinct output. A real regression still fails, just after the retry
window, since the loop exits early only on the expected value. Extract the
read the two phases share rather than leaving them to drift.
 2026-07-19 06:46:50 +00:00
Johnathan Corgan 4ff7de4d81 Merge branch 'maint' 2026-07-19 06:45:18 +00:00
Johnathan Corgan e4a854f6b0 Stop concurrent local CI runs from clobbering each other's containers 
Two runs on one host destroyed each other's containers, producing
mid-test "No such container" failures that look like real defects. The
automated builder runs a full local CI on the same box every few minutes,
so the machine is contended almost always and this has red-ed both a hand
run and an automated gate.

Two independent causes are fixed here. Container names were hardcoded, and
docker names are global rather than scoped by compose project, so two runs
collided on the same name; every name now takes an optional suffix that
the harness sets from the run id. And the cleanup sweep matched a label
shared by every run, so one run's teardown force-removed another's
containers; resources now also carry a per-run label and the sweep can be
narrowed to it.

A third hazard turned up that was not in the original report: the sidecar
suite passes explicit compose project names, which override the run-scoped
project and put it outside the shared prefix entirely. Its project names,
network, and derived container references are now scoped too.

Both are default-off. With the suffix unset, names render exactly as they
do today and a bare compose invocation is unchanged, which is what keeps
the hosted CI and the documentation correct. A cleanup run with no run id
still reaps everything, which is what a manual "clear the box" wants.

The literal-name sweep was not sufficient: six scripts build container
names dynamically from node labels, and two suites create their own
containers outside compose. Those are handled at their construction sites.

Verified: syntax check on all modified scripts; compose validation on
every modified file with the suffix both set and unset; and a synthetic
two-run reproduction that shows the old cleanup destroying a bystander run
and the new one leaving it alone.

Known gap: subnets are still hardcoded, so two concurrent full runs will
still collide on address-pool overlap. That fix reaches into topology
configs, chaos scenarios, diagrams and production source, so it is left
for its own change rather than half-done here.
 2026-07-19 06:41:25 +00:00
Johnathan Corgan 0c3d9a0b73 Correct stale view-trait docs and drop internal shorthand from comments 
The lifecycle view's doc described the shell as implementing it over a
`connections` map that no longer exists. The establish view named that map
too, and was doubly wrong: both of its snapshot builders read only
`peers`. Rewrite both to describe the maps actually read.

Several comments in the node module carried shorthand from private
working notes - bare parenthetical labels, and one reference to an
internal tracker item - none of which mean anything to a reader of this
repo. Rewrite them so each clause states its own reasoning inline. The
paragraph above the tracker reference already carried the rationale, so
that one is simply dropped.

Comment-only; no behavior change.
 2026-07-19 06:11:40 +00:00
Johnathan Corgan 281ed132f1 Merge branch 'master' into refactor-node 2026-07-19 04:25:36 +00:00
Johnathan Corgan c5492f4572 Merge branch 'maint' 2026-07-19 04:25:25 +00:00
Johnathan Corgan 7a97599921 testing: floor the rekey second-cutover wait at the log-analysis threshold 
The rekey suites' Phase 4 polled until the initiator-cutover count
reached its value at phase entry plus two, while Phase 6 asserts an
absolute minimum of four. On an unloaded host the ping phases can
outrun the jittered first rekey cycle, so Phase 4 can begin with a
single cutover on the books, release at three, and leave Phase 6 to
lose a sub-second race against the remaining first-cycle cutovers.
A rekey-outbound-only run failed exactly this way: five counted
cutover lines by t=32.6s of container life, with the Phase-6 count
snapshot at t~32.4s seeing three.

Floor the Phase-4 wait target at four so a released wait guarantees
the Phase-6 assertion — the docker-logs count is monotone over a
container's lifetime. A genuine rekey stall still times out the
bounded wait and fails Phase 6 with the real count.

Also drop the second-cycle framing from the comment, the phase
banner, and the assertion description: with rekey timers resetting at
each cutover a second cycle cannot begin inside this test's window,
so the counted events are first-cycle cutovers spread across the
topology's links, emitted once or twice per completed link rekey
depending on which side's promotion path logs them.
 2026-07-19 04:12:17 +00:00
Johnathan Corgan 7fe1d75637 peer: delete the pending-connection type, leaving the control machine whole 
The per-peer control machine has absorbed every field the pending
connection carried. What remained was a struct holding two Noise
handles beside a duplicate copy of bookkeeping nobody read. Replace it
with a small carrier for the two handles and delete the type.

Presence of that carrier, not the state of the handles inside it, is
what marks a machine as mid-handshake. The distinction is essential
rather than stylistic: a failed handshake drops its initiation handle
and is deliberately retained so the stale sweep can reclaim it, and a
completed one has its session taken before disposal. Deriving presence
from the handles would make both invisible to the sweep, the
connection count, and the peering budget at once, leaking the slot
permanently. A test drives an empty carrier past every presence
predicate and then detaches it, so a future edit cannot quietly couple
the two.

The remote startup epoch now comes from the surviving carrier, which
the handshake operations already wrote at the same two points with the
same value. The paired writes onto the pending connection's own
bookkeeping had no readers left and are gone.

The handshake-phase surface leaves the public API: it was public by
accident rather than design, and the machine behind it is crate
internal. Callers outside the crate that need a view of pending
handshakes go through the operator queries, which are unchanged.

ConnectionState::inbound_with_transport loses its last non-test caller
with the inbound seed and is marked test-only.
 2026-07-19 01:18:05 +00:00
Johnathan Corgan e21e09d7e6 peer: source the handshake identity and index family from the control machine 
The pending connection and the per-peer control machine have carried
duplicate copies of the handshake-phase fields since the machine gained
its own connection state. Read them from the machine and drop the
connection's projections.

The peer identity is the sharp one. The connection learned it from msg1
and the machine's carrier did not, so the two views genuinely disagreed
for inbound connections until the Noise operations moved onto the
machine and began recording each result on both. That is now in place,
so every reader can take the machine's copy unchanged: the establish
snapshot, the promotion sweep for competing connections, the dial and
path in-progress checks, the peering observations and per-peer in-flight
budget, and the stale-connection sweep's retry address.

Also repointed: our_index, their_index, transport_id, started_at, the
stored handshake message bytes, and the idle-timeout check. Two
duplicate index writes on the connection are dropped, both immediately
preceded by the machine-side write of the same value. Reads that
previously came off a connection detached just before its machine was
disposed now capture the machine's value first.

Test seeding follows the establish paths: the seed builders write
our_index to the carrier, which promotion now reads. A new test pins
the inbound identity learn on the carrier and the retry address a
failed inbound connection reports to the sweep, so a silent regression
to a blank identity cannot pass.

ConnectionState::duration loses its last non-test caller with the
connection accessor and is marked test-only.
 2026-07-19 00:25:17 +00:00
Johnathan Corgan e7537929ba node: move link, direction, and peer address onto the control machine 
These three had no counterpart on the control machine, so readers still
reached them through the pending connection. Add machine-side
accessors and repoint every reader, then drop the connection's.

The peer address needed its writes lifted, not just its reads
repointed: the machine's copy was never written. It is now written at
each of the points the connection's copy was — the inbound seed, the
dial, message-2 completion, and the two paths that seed a machine from
a pre-built connection — so promotion and the resend path read a value
with the same provenance at the same time as before.

Link and direction need no lift. Both machine constructors already
seed them from the same arguments the connection is built with, so an
outbound machine carries outbound state and an inbound one inbound,
and the machine's link always equals the connection's. The handshake
operations' direction guards read the machine's copy for the same
reason.

The stale-connection sweep's teardown log and its resend path both now
take the transport and address from the machine, and each keeps its own
check that a pending connection is still attached rather than relying
on the caller to have established it.

Add a test pinning link, direction, and address on the two shapes that
seed a carrier independently — the dial and an accepted message 1. The
cross-connection winner reads both values from a carrier one of those
two already seeded.
 2026-07-18 23:17:21 +00:00
Johnathan Corgan 347cbe60bd peer: move the Noise handshake operations onto the control machine 
The pending connection drove its own Noise handshake while the control
machine held it, so the crypto and the state it produces lived on the
carrier that is going away. Move the six operations onto the machine:
starting and completing an initiation, processing an inbound
initiation, taking the session, testing for one, and dropping the
handle on failure. Bodies are unchanged apart from reaching the
handles through the attached connection.

Each operation now records its results on both carriers. The learned
identity, the remote epoch, and the activity stamp are written to the
machine's own bookkeeping at the same point and with the same value as
they are written to the connection's. The connection's copies still
have readers until those reads are repointed, so both have to be
written; the machine's copy of the learned identity was previously
never populated for an inbound connection, which is why an inbound
pending row showed no expected peer.

Driving the handshake from the machine means the machine has to exist
before the crypto runs. On the inbound path it is built above the
message-1 processing and still kept local, so a rejected message
leaves no registry entry and allocates no index. On the outbound path
it already existed from the dial, so it simply takes the connection
before the index allocation, and both failure arms unwind it as they
did.

Completing message 2 no longer mirrors the activity stamp separately,
since the completion itself now writes both carriers at the point the
mirror was approximating.

Three tests cover what the compiler cannot. A connection whose
handshake failed holds neither Noise handle yet must stay visible to
the stale-connection sweep, or every failed connection would leak and
hold a peering-budget slot forever. A message 1 rejected by the crypto
or by the ACL must leave no machine, no index, and the same rejection
count as before. A dial whose message-1 preparation fails must unwind
the machine registered at dial time; that test drives the
index-allocation failure rather than a crypto failure, which is the
arm this change actually widens, since the allocation now happens with
the connection already attached.
 2026-07-18 22:55:38 +00:00
Johnathan Corgan 11ec16777c node: yield the control machine when iterating pending connections 
`Node::connections()` yielded the pending connection itself, so its
consumers reached the handshake-phase fields through that value. The
pending connection is being folded into the control machine, and the
fields will move off it, so yield the machine (keyed by its link) and
let each consumer reach what it needs from there.

Membership is unchanged: the iterator still selects machines that
carry a pending connection, which is the predicate `connection_count`
and the stale-connection sweep already use. Every consumer takes the
same value from the same place, one hop further out, and no field
changes carrier here.

The method is now internal to the crate. It yields the control
machine, which is not part of the published surface, and nothing
outside the crate iterates pending connections — operator views of
the handshake phase go through the query surface instead.
 2026-07-18 22:06:46 +00:00
Johnathan Corgan 5b09e22956 node: read promotion's carrier fields in one borrow 
`promote_connection` detached the pending connection and then
interleaved reads off that detached value with three separate lookups
of the same control machine. Gather the machine-side fields
(`their_index`, `transport_id`, link stats) in the single `get_mut`
that takes the connection, so the machine is borrowed once.

Behaviour is unchanged. The connection is still taken before anything
is validated, so a rejected promotion leaves the machine with no
pending connection, and the prelude only gathers options: the checks
below it still report the first missing field in the same order
(`our_index`, `their_index`, `transport_id`, `source_addr`).

Link stats move to the prelude, ahead of those checks. The value is
the same either way: they live on the surviving carrier rather than
the detached connection, and nothing between the two points touches
the machine map. The lookup could not fail at the old site either,
since the function had already reached it through a successful lookup
on the same key, so dropping the defaulting arm changes nothing.

Add a test covering the error order and the detach, driving promotion
with connections missing each required field in turn plus every later
one, so an implementation that validated during the prelude would
report the wrong field. Test seeding grows a variant that lets the
caller shape the seed.
 2026-07-18 21:45:09 +00:00
Johnathan Corgan a382b17931 node: seed control machines directly in tests, ahead of removing the leg 
Tests built a free-standing PeerConnection, mutated it, and handed it to
Node::add_connection by value. None of those sites survives the removal of
PeerConnection, so converting them afterwards would mean one enormous commit
that cannot be reviewed honestly. Convert them now, while the struct still
exists and the conversion can be validated against a green tree.

Adds a cfg(test) Node::seed_handshake_machine plus a HandshakeSeed builder,
and rewrites make_completed_connection (now seed_completed_connection) and
the twenty inline builders onto it. add_connection keeps its body and loses
its test callers.

The builder's carrier seeding is a verbatim copy of add_connection's: the two
conditional writes for their_index and transport_id, then set_leg, through the
same entry().or_insert_with() so an existing leg-less machine keeps its
constructor-side fields. Nothing else reaches the carrier -- our_index,
source_addr, post-construction started_at and the stored handshake bytes stay
leg-only. Seeding more than that would let these tests observe a carrier
richer than production's and keep passing even if a later production
write-lift were missed.

The Noise exchange now runs on the already-seeded leg rather than before the
hand-over. That is neutral because the only read of expected_identity is
guarded by is_outbound, and no crypto method allocates a session index.

Also adds a compile-time check that PeerAction is Clone + Eq, which is what
keeps a runtime handle from being smuggled into an action payload.
 2026-07-18 21:24:03 +00:00
Johnathan Corgan a90049d3a1 Merge branch 'master' into refactor-node 
Bring the Android-ready core (target_os gating and app-owned TUN seam)
onto this branch so subsequent work builds on the current mainline tree.
 2026-07-18 20:38:56 +00:00
Johnathan Corgan 791b35c221 node: make the peer machine's connection state the sole session-index carrier 
The session index was carried three ways: on the leg, on the peer machine's own
connection-state copy, and on a separate machine-owned shadow field. Populate the
machine's connection copy on the outbound path (it was written only on the inbound
and cross-connection-swap paths before), then delete the shadow field and route its
readers, the decrypt-registration lifecycle, and the rekey cutover through the
single carrier. The shadow and the connection copy diverged only on paths that are
unreachable or dormant in production (a rekey cutover with no newly allocated index,
the swap winner's later reads, and the never-dispatched timeout/disconnect events),
so the merge is byte-identical for every reachable path. Rewrote the outbound-promote
index tests and the dial-time comment for the single-carrier contract.
 2026-07-18 17:19:44 +00:00
Johnathan Corgan 6a80790742 node: source connection index/transport/stats from the peer machine 
The connection leg's their_index, transport_id and link_stats duplicated the peer
machine's own connection-state copy. Route every production reader through the
machine copy: promotion, the stale-connection reaper, the outbound msg1 resend, and
the in-use / connecting-path checks. Seed the inbound machine's transport_id from the
msg1 packet (the same source the leg used), matching the outbound dial seed, so the
hard-required promote read never sees a missing transport. link_stats is a
never-mutated zero seed, so its leg accessors are removed outright; the leg
their_index/transport_id getters/setters remain only for the test-only connection
builder. Also delete the write-only next_resend_at_ms field from the connection
state (the resend deadline is carried by the machine-armed retransmit timer, not this
field). Byte-identical.
 2026-07-18 16:43:22 +00:00
Johnathan Corgan 60b8acf716 node: source handshake resend buffers from the peer machine 
The connection leg's stored msg1/msg2 handshake-resend buffers duplicated the
peer machine's own connection-state copy. Write the machine copy at the outbound
msg1-prep and the two inbound authorize sites, then source the retransmit
resend-bytes reader, send_stored_msg1, and the pending tier of find_stored_msg2
from the machine. The post-promote active-peer msg2 copy is written from the same
wire bytes, so the pending tier now matching after promotion is value-identical.
Byte-identical resend behavior.
 2026-07-18 16:00:15 +00:00
Johnathan Corgan 4fc295d90a node: source connection peer-identity telemetry from the peer machine 
The expected_peer connection-row field read the peer identity off the leg;
source it from the machine's own connection-state copy instead, matching the
started_at/last_activity treatment. The machine carrier holds the same identity
as the leg for every connection that appears in the view: outbound legs seed
both from the dialed identity, and inbound legs never rest in the connections
view (the machine takes the leg at promotion), so their machine copy is
unobserved. The leg accessor stays for the crypto-extraction path and retry
decisions. Byte-identical.
 2026-07-18 15:32:04 +00:00
Johnathan Corgan b3f2018fce node: source connection start/activity timestamps from the peer machine 
The connection leg's started_at and last_activity duplicated the peer
machine's own connection-state copy. Make the machine copy the sole live
telemetry and timeout carrier: re-stamp it with the leg's provenance when
the leg is attached at handshake start (the msg1-prep clock, not the earlier
dial-time constructor value) and mirror the completion touch, then delete the
leg's last_activity/touch accessors and source the connection-row projection,
show_connections, and the timeout readers from the machine. Byte-identical
for all normal paths.
 2026-07-18 15:11:43 +00:00
Johnathan Corgan 56bbc81a40 node: derive handshake state from the peer machine, delete the leg field 
The connection leg's HandshakeState field duplicated the peer machine's
handshake phase. Delete it and derive the displayed handshake state string
from the machine's PeerState, looked up by link id (mirroring the resend
count). Move the failure signal onto the machine: a send_failed flag that
preserves retransmit eligibility (the machine stays in its handshake phase),
alongside the existing Failed state. The leg's crypto self-gates now guard on
Noise handle presence instead of the deleted phase field, and mark_failed
only drops the handle. Telemetry strings, wire bytes, index allocation, and
the stale-connection reaping are byte-identical for all normal paths.
 2026-07-18 02:26:39 +00:00
Johnathan Corgan b38f8c6ffb node: classify inbound msg1 via a single machine decision site 
handle_msg1 now creates one local machine before classification and routes
on the InboundDecision it returns, instead of calling establish_inbound in
the shell and re-deriving the decision inside each arm. The promote and
restart arms drop their own per-arm machine construction and the redundant
InboundMsg1 step; their phase-1 actions come from the single call. The
effect-bearing arm bodies (rekey-respond, duplicate resend, reject
bookkeeping, and the promote/restart tails) are unchanged.

Behavior-neutral: the same establish_inbound evaluation over the same
snapshot and wire, done once in the machine method rather than once in the
shell, with byte-identical arm bodies and no change to index allocation,
wire sends, machine state, or telemetry.
 2026-07-17 19:17:25 +00:00
ArjenandJohnathan Corgan cf62cff5f4 feat: Android-ready core: target_os gating and app-owned TUN seam 
Make the FIPS core build and run as an embedded Android library. The host
app owns the TUN (e.g. an Android VpnService) and FIPS performs no
system-TUN or CAP_NET_ADMIN operations.

Squashed from the following changes:

- gate desktop transports/TUN by target_os, not features: a plain
  `cargo build` now compiles for every target with no flags. Ethernet (raw
  AF_PACKET / BPF) is gated to linux/macos, so Android (target_os =
  "android", not "linux") self-excludes it as Windows already did; real
  system-TUN ops are gated per linux/macos and Android gets a no-op stub;
  the ipi6_ifindex cast handles it being i32 on Android vs u32 on macOS. No
  Cargo features are introduced; desktop builds are unchanged.

- app-owned TUN seam: Node::enable_app_owned_tun() lets an embedder that
  owns the TUN fd exchange IPv6 packet bytes with FIPS over channels
  instead of FIPS creating a system TUN device. It returns (app_outbound_tx,
  app_inbound_rx): the embedder pushes packets read from its fd into the
  outbound sender (app -> mesh) and pulls packets destined for its fd from
  the inbound receiver (mesh -> app). start() gates system-TUN creation on
  tun_tx being unset, so with the channels pre-installed it skips device
  creation and does no system-TUN ops; both directions reuse the existing
  inbound-shim and run_rx_loop wiring. Packets entering via app_outbound_tx
  bypass handle_tun_packet, so the embedder must push only fd00::/8-destined
  packets and clamp TCP MSS on outbound SYNs; the rustdoc and the
  IPv6-adapter design doc spell this out.

- keep the android target warning-clean so the cross-compile check passes
  clippy -D warnings.

- add an Android cross-compile CI check: cross-compile the library for
  aarch64-linux-android via cargo-ndk and run clippy -D warnings. Android
  ships as an embedded library (the host app owns the TUN), so there is no
  daemon binary to package; this is a check job, not a packaging one.

- docs: list Android as a supported platform.

Tests: app_owned_tun_seam_wires_channels covers the channel round-trip and
the Active state; start_skips_system_tun_when_app_owned runs start() and
asserts no named system device is created.
 2026-07-17 18:25:55 +00:00
Johnathan Corgan 7fe3388f2f peer: return the inbound establish decision from the machine arm 
The inbound msg1 machine arm re-derived the establish decision for
its own action selection while the shell matched on a separately
computed copy. Expose the arm as a method returning the decision
alongside the actions so a driver can route on it directly; the
step dispatch delegates and keeps its action-only shape.

The dead rekey-respond arm is stripped to decision-only and its
helper deleted: it allocated from the real index allocator, wrote
the rekey shadow index, emitted a wrong-framed rekey send, stamped
the dampening clock, and flipped state, none of which happens on
the live path, where the inline respond body owns all effects. The
resend-msg2 arm's send emission is stripped the same way; the
decision already carries the stored bytes.
 2026-07-17 13:47:50 +00:00
Johnathan Corgan 252d16fab9 node: feed msg1 send failure to the peer machine as an event 
send_stored_msg1 marked the embedded leg failed by writing it
directly from the shell. Route the write through a new
HandshakeSendFailed machine event instead: the machine marks its
leg so the stale-connection sweep reclaims it, without leaving the
handshaking state, so retransmit eligibility survives the window
between the failed send and the sweep exactly as before.

send_stored_msg1 gains a now_ms parameter threaded from the action
executor for the step call; the failure arm itself ignores it.
 2026-07-17 07:12:40 +00:00
Johnathan Corgan 94d7b91244 node: route the outbound msg2 establish decision through the machine 
handle_msg2 no longer pre-computes establish_outbound alongside the
machine's own evaluation of the same snapshot. The shell now builds
the snapshot, steps the persistent outbound machine once at the
decision point, and routes on what comes back: the promote action
vector drives promotion through the executor as before, and the
ResolveCrossConnection decision selects the inline swap/keep
resolution bodies, which are unchanged.

The machine step and its defensive transient-rebuild move up from
the promote arm to the decision point; the cross-connection path
keeps its take-leg-then-dispose ordering with the step preceding
both. Comment prose at the touched sites refreshed to describe the
single-decision-site shape.
 2026-07-17 06:59:30 +00:00
Johnathan Corgan 7790eb86bd peer: make the outbound msg2 swap/keep machine arms decision-only 
The machine's on_msg2 cross-connection arms crystallized state and
emitted FreeIndex/RegisterDecryptSession actions that duplicate the
inline shell resolution, but they are unreachable on the live path
(the shell removes the machine before running the swap/keep bodies).
Making them live in that shape would double-free the outbound index
through the executor.

Strip both arms to a single new ResolveCrossConnection { swap }
action: a decision conveyed to the driver, not an effect. The shell
intercepts it and runs the inline resolution, which owns all effects
permanently. The action executor gets a defensive unreachable arm.
The Promote arm and set_their_index are unchanged.
 2026-07-17 06:44:09 +00:00
Johnathan Corgan cf1c957336 node: embed the handshake leg in the peer machine, drop the connections map 
The pending-handshake PeerConnection map and the per-peer control
machine map were parallel LinkId-keyed structures whose keysets must
stay coherent by hand. With every leg now born with a machine, the leg
becomes storage inside its machine (leg: Option<PeerConnection>, pure
storage the machine never reads or drives) and Node loses the
connections field; every access routes through the machine.

The non-mechanical lowerings, each argued at the site: the
rekey-vs-establish gate in handle_msg2 tests leg-absence (an established
peer's machine stays keyed by its link, so machine-presence would
misclassify every rekey msg2 as a fresh establish); the
connecting-predicates, peering observation, and handshake-slot budget
iterate machines-with-legs so connect-window machines (leg not yet
born) are excluded exactly as before and never double-counted against
their pending-connect slot; the cross-connection extract takes the leg
before disposing the machine; the stale reaper takes the leg and leaves
the machine untouched when none is present, matching the old early
return. The map-coherence debug check keeps its machine-has-carrier
direction with the embedded leg as a carrier; the leg-to-machine
direction is now true by construction and its gate const is gone.

connection_count() counts machines with legs; the connections()
iterator, the test seams, and the control-socket connection rows are
re-implemented over the embedded legs with unchanged output.
 2026-07-17 04:27:53 +00:00
Johnathan Corgan 5ccd95cf3f node: add a debug-build peer-map coherence check 
Assert, once per tick in debug builds, that the peer control-machine map
and its carriers stay coherent: every control machine has a live
handshake leg, an active peer, or a pending connect (a machine with none
is a leak the stale reaper can never see), and every handshake leg has a
machine. The second direction sits behind a file-local const so a branch
where handshake-window legs legitimately run machine-less can gate it
off without weakening the leak tripwire.

Teach the add_connection test seam to seed a control machine for the leg
it inserts (derived from the connection's direction and identity), and
remove_connection to dispose it, so the seam-built topologies satisfy
the check. Three unit tests: seam coherence, coherence through a real
promotion, and the orphaned-machine panic.
 2026-07-17 02:09:28 +00:00
Johnathan Corgan 119b85d28e node: correct stale liveness claims in machine, executor, and timer docs 
Several module and field docs still described the per-peer machine as
unwired shadow scaffolding. It has been live for some time: machines are
inserted at dial and inbound msg1, stepped by the handshake handlers and
the rekey-cadence and liveness-reap routers, and the executor's
SwapSendState/CompleteDrain/InvalidateSendState arms are the authoritative
paths (the inline bodies survive only as debug-assert release fallbacks).
Rewrite those docs to the current truth while keeping the still-true
dormancy facts: PeerEvent::Timeout and PeerEvent::Tick are never
dispatched in production, retransmit fires on the machine-armed deadline
while the timeout reaper keys on timer presence with the config
threshold, and the remaining inert executor stubs are SendRekey,
SendLinkMessage, and the connected-UDP arms. Drop the stale
allow(dead_code) on the peer_machines field.
 2026-07-17 01:33:06 +00:00
Johnathan Corgan 74245e80ac peer: remove the dead PeerSlot enum and PeerConnection resend API 
PeerSlot (and its entire impl surface) was referenced only by its own
module tests and the lib.rs re-export; peer storage has always used the
separate connections/peers maps. PeerConnection's resend_count/
next_resend_at_ms/record_resend delegations had no production callers:
the live resend counter is machine-sourced (connection_resend_count reads
the per-peer machine), and the FSP session layer uses SessionEntry's own
methods. ConnectionState::next_resend_at_ms is now test-only (its
remaining callers are the fmp state unit tests) and marked cfg(test).

The PeerSlot unit tests go with the enum; test_resend_count_tracking is
dropped because the delegation target's schedule arithmetic is already
covered by the fmp resend_bookkeeping test.
 2026-07-17 01:32:49 +00:00
Johnathan Corgan e42598a86e Merge branch 'maint' into master 2026-07-16 05:55:20 +00:00
ArjenandJohnathan Corgan 054d17aac5 feat(openwrt): 802.11s open-mesh backhaul support 
Router-to-router radio backhaul over an open 802.11s mesh interface,
with FIPS providing all encryption (Noise IK), authentication, and
routing on top of bare L2 neighbor links. The mesh runs OPEN with
mesh_fwding 0 — SAE would duplicate the Noise layer and force ath10k
raw mode, and FIPS is the routing layer — so the Noise handshake is the
real auth/encryption boundary and FIPS's spanning tree does the routing.

fips-mesh-setup: an opt-in UCI helper that creates a per-radio
mesh-point interface (radio0 -> fips-mesh0, radio1 -> fips-mesh1;
trailing-digit derivation with a free-index fallback and a collision
guard). Radio setup stays opt-in — a package must not commandeer radios
on install. 'remove' takes an optional radio and otherwise removes all
instances. Dual-band routers get one instance per radio; FIPS treats the
two backhaul paths as failover, not multipath: it keeps one active link
per peer (cross-connection resolution picks a single winner), and the
second band stands by, re-establishing the peer after keepalive timeout —
traffic never uses both bands at once.

fips.yaml ships the mesh0/mesh1 Ethernet-transport entries commented
out, so a stock install that never creates fips-mesh* logs no per-boot
"interface missing" bind warning. fips-mesh-setup uncomments the matching
meshN block when it creates the interface and re-comments it on remove,
so the flash-and-drop-in flow needs no manual config edit. The file is
rewritten 0600-first (it may hold an inline nsec) via an atomic replace.

Two field-found silent non-peering causes are surfaced by the helper and
the guide:

- Same channel: mesh points only peer on a shared channel, and 'auto'
  lets each radio pick its own. The helper prints the radio's
  band/channel and warns loudly on 'auto' with the exact uci command to
  pin one; the how-to gains an ordered no-peers triage (channel mismatch,
  on-air scan check, DFS CAC wait, regdomain).
- STA channel capture: a client (sta) interface drags the whole radio to
  its upstream AP's channel, so a mesh pinned elsewhere never joins and
  does not recover until the STA disconnects. The helper warns when the
  target radio carries a STA; the guide documents the incompatibility of a
  roaming uplink with a fixed-channel mesh on the same radio.

Both the create and remove paths run 'wifi reload', which briefly drops
every client AP on all radios; the how-to sets that expectation.

Regression test: the shipped OpenWrt fips.yaml must parse via the real
Config deserializer in both states — as shipped (mesh inactive) and after
the uncomment the helper performs.

Packaging: the helper is installed across the ipk/apk/buildroot paths
(three synced copies), with the CI structural checks and shellcheck
targets extended to cover it. Full guide in
docs/how-to/set-up-80211s-mesh-backhaul.md.
 2026-07-16 05:48:04 +00:00
Johnathan Corgan fbb4fb8879 testing: wait for both second-cycle rekey cutovers before asserting the count 
Phase 4 of the rekey suite waited for only one more initiator cutover
(guaranteeing three) while Phase 6 asserts at least four. The fourth
cutover then had to land in the brief window between the wait returning
and the Phase 6 log snapshot, so host load could push it past the window
and fail the run even though every cutover completed correctly. Wait for
two more cutovers, the full second rekey cycle, matching the Phase 6
threshold, so the asserted count is guaranteed before it is checked.
 2026-07-16 00:42:04 +00:00
Johnathan Corgan 87399795f8 node: observe rekey-msg2 and cross-connection resolution in the per-peer machine 
Feed the per-peer machine two observation events so its control/rekey
shadow state stays coherent with the inline crypto-session installs,
without moving those installs. The crypto effect bodies are unchanged.

On rekey-msg2 completion, feed the machine the responder index on the
success path only (after the pending session is installed); the abandon
path feeds nothing.

On cross-connection resolution, feed the machine the swap outcome: on a
swap it records the new outbound and responder indices to mirror the
session replacement, and on a keep it does nothing. The observation
targets the promoted peer's machine, not the outbound leg whose machine
is removed as the block is entered.

Both observation handlers emit no action; nothing consumes the updated
shadow state yet.
 2026-07-15 23:50:18 +00:00
Johnathan Corgan 765819f52b node: reap the handshake timeout through the per-peer machine timer 
Move the outbound handshake-timeout reap off the unconditional check_timeouts
scan onto the machine-armed HandshakeTimeout timer. A new drive_handshake_timeouts
(run before the retransmit drive, so a timed-out leg is reaped rather than resent
on the same tick) reaps the outbound legs that carry a HandshakeTimeout timer and
have idle-timed-out this tick.

The timer's presence selects the leg (only outbound legs arm one; IK inbound arms
none); the reap threshold is the shell is_timed_out(now, config) predicate, not
the timer's stored deadline. The machine arms the timer from a hardcoded constant
at dial, which is not authoritative for an operator-tuned handshake_timeout_secs,
so reading the threshold from config each tick keeps the reap neutral for any
timeout value and on the last_activity clock exactly as before.

check_timeouts keeps reaping everything else: failed connections (all of them,
promptly) and the idle-timeout of legs without a machine timer (inbound legs, and
any machine-less connection). Total coverage is unchanged. check_timeouts runs
before the timer drive, so a failed outbound leg is reaped there first, its timers
dropped, and the timeout drive never double-fires on it.

Split drive_peer_timers into the timeout drive and the (byte-identical) retransmit
drive. The dormant machine timeout handler is not dispatched, so the session index
is freed once, by cleanup_stale_connection. A regression test drives a timed-out
outbound leg to reap; the existing check_timeouts tests continue to anchor the
residual sweep. Test count 1631 to 1632.
 2026-07-15 20:23:49 +00:00
Johnathan Corgan 5021197f5c node: drive the handshake msg1 resend from the per-peer machine timer 
Move the outbound msg1 resend off the unconditional tick function onto the
machine-armed retransmit timer. The per-peer machine already arms a
HandshakeRetransmit deadline at dial; a new drive_peer_timers fires the due ones
(kind-filtered to the retransmit timer) and homes the resend counter on the
machine, where the operator-visible count now reads from.

The resend decision reads the same operator config as before (interval, backoff,
max), the wire bytes and transport target still come from the shell connection,
and the pure core computes the backoff schedule. As with the deleted
resend_pending_handshakes, the count and reschedule advance only on a successful
send: a failed send neither advances the count nor marks the connection failed,
it just retries on the next tick. The handshake-timeout timer stays on the legacy
check_timeouts path, and the rekey/liveness timers keep their own shell drivers,
so drive_peer_timers deliberately fires only the retransmit kind.

The show_connections resend count is relocated to read from the machine (the
counter's new home) via connection_resend_count; machine-less and inbound
connections report 0, matching what the shell connection reported before. Delete
resend_pending_handshakes and resend_candidates (the latter also from the
LifecycleView trait, its only user). The resend unit test is re-expressed against
the machine + timer path, covering the due check and the record-on-success
semantics.

Known limitation: the first resend interval is armed from the machine's hardcoded
1000ms constant, which equals the config default. Under an operator override of
handshake_resend_interval_ms the first resend diverges from the pre-change
dial+interval; subsequent resends and the cap remain config-driven. Neutralizing
the first-resend override needs the interval threaded into the sync core (the
shell arm would be clobbered by the machine's own timer arming at dial), deferred
to when the connection and machine entities merge. Test count unchanged at 1631.
 2026-07-15 19:38:49 +00:00
Johnathan Corgan 31f5a8c1b7 peer: add a per-peer timer store populated by the machine timer actions 
The control machine already emits SetTimer/CancelTimer actions when it arms the
handshake retransmit and timeout deadlines, but their executor arms were a
single no-op stub and the machine's Timeout event was never dispatched -- the
real work still runs on the legacy tick (check_timeouts,
resend_pending_handshakes).

Add the storage the time-as-input driver will read: a peer_timers map keyed by
LinkId then TimerKind, holding each armed timer's absolute deadline. The SetTimer
arm now inserts (overwrite = reschedule) and CancelTimer removes; the outbound
msg2 promote cancels the two dial-armed handshake timers, since the machine
survives promotion and the entries would otherwise linger in the store.

This store is a shadow: it is written and cleared but no driver reads it yet, so
behavior is unchanged. The legacy tick stays authoritative until the driver that
feeds Timeout and deletes the overlapping tick paths lands.

Route every machine removal through a new remove_peer_machine(link) choke-point
that drops the timer store alongside the machine, replacing the twelve direct
peer_machines.remove sites so no armed timer outlives its machine. TimerKind
gains Hash (to key the store) and Ord (for deterministic driver collection); it
is internal and never serialized. Test count unchanged at 1631.
 2026-07-15 18:49:50 +00:00
Johnathan Corgan 3e7ca90212 peer: drive the connection-oriented outbound dial through the machine 
Route the connection-oriented (TCP/Tor) outbound path through the peer state
machine, matching how the connectionless path already works.
initiate_connection's oriented branch now drives PeerEvent::Dial with
connection_oriented=true; the machine parks in Connecting and emits
OpenTransport, whose executor arm performs the non-blocking transport.connect
and pushes the PendingConnect. When the connect resolves, poll_pending_connects
prepares msg1 in the shell and then drives PeerEvent::TransportConnected, which
sends msg1 via the machine's SendHandshake arm.

The msg1 prepare (index allocation, Noise leaf, wire arming) MUST run in the
shell before the TransportConnected drive: send_stored_msg1 only transmits an
already-armed wire, so a drive-only path would silently send nothing. The
machine's our_index stays unset; the connect-failure path keeps its direct
handshake-timeout handling (TransportFailed stays dormant). The now-unused
Node::start_handshake helper is removed.

Behavior-neutral: same transport.connect, same PendingConnect, same msg1 send
and failure teardown as the removed inline path -- only the driver changes from
inline shell code to the state machine.
 2026-07-15 14:40:58 +00:00
Johnathan Corgan 9588c50063 peer: persist the outbound control machine at dial, not at msg1 prepare 
Move the peer_machines insert for an outbound leg out of the tail of
prepare_outbound_msg1 to a single shared site in initiate_connection, before
the connection-oriented / connectionless fork. This lets the connection-oriented
path find the machine at dial time (so it can be driven through the connect
handshake) without prepare_outbound_msg1 -- which for that path runs after the
connect completes -- clobbering an in-progress machine back to Discovered.

Because the machine now exists before the fallible dial steps, add
peer_machines.remove to every failure path in the widened dial window: the
index-allocation and Noise-leaf failures in prepare_outbound_msg1, the oriented
transport.connect failure, and both poll_pending_connects teardown arms
(handshake-start failure and async connect failure). remove_link does not touch
peer_machines, so these explicit removes are required.

Behavior-neutral: the connectionless path still drives the machine to
Handshaking and sends msg1 identically; the machine's our_index stays unset
(no spurious UnregisterDecryptSession on a later inbound restart); a failed dial
leaves peer_machines empty for that link exactly as before; and no
connection-oriented machine drive is wired here.
 2026-07-15 14:12:49 +00:00
Johnathan Corgan f698da50b6 peer: add unit coverage for the connection-oriented outbound dial path 
The connection-oriented (TCP/Tor) outbound connect->handshake path had no
`cargo test --lib` coverage; it was exercised only by the opt-in Tor
integration suites, and the TCP node tests bypass it via a manual
connectionless handshake helper. Add three unit tests:

- a machine-level test driving Dial{connection_oriented:true} -> Connecting
  (emitting only OpenTransport, no msg1) -> TransportConnected ->
  Handshaking{SentMsg1}, asserting the exact OpenTransport and
  SendHandshake+SetTimer action vectors that start_outbound_handshake emits
  (its oriented reach via on_transport_connected was previously untested; the
  connectionless reach via on_dial was already covered);

- two node-level tests over a real loopback TcpTransport: a successful connect
  that reaches start_handshake (observed via a pending_outbound entry), and a
  connect to a closed port that routes through the failure arm (link torn down,
  no msg1 dispatched).

Tests only; no production change.
 2026-07-15 13:52:34 +00:00
Johnathan Corgan 1f765cfd8f peer: drive the connectionless outbound msg1 send through the machine 
Split the outbound handshake setup out of start_handshake into
prepare_outbound_msg1 (allocate the index, run the Noise leaf, frame and arm
msg1 -- the fallible steps, returning an error the caller propagates) and
send_stored_msg1 (transmit the armed wire). A connectionless dial now runs
prepare in the shell, then drives the control machine, whose SendHandshake
action sends the wire via the executor. Connection-oriented dials keep calling
start_handshake, now prepare followed by the send inline.

The dial event gains a connection_oriented flag so the machine's on_dial sends
msg1 immediately for connectionless transports (no connect step) instead of
opening a transport first. Behavior is unchanged: the same index, Noise leaf,
wire bytes, maps, and send-error handling as before, with index-allocation and
Noise failures still propagated synchronously before the send. A regression
test covers that a promote from the post-dial handshaking state is identical to
the former discovered-state promote.
 2026-07-15 06:15:21 +00:00
Johnathan Corgan 3b99a416ad peer: persist the outbound control machine at dial 
Create and persist the per-peer control machine when an outbound handshake is
dialed, keyed by its link, instead of building a transient at msg2. The msg2
completion path now looks up that persisted machine to drive the promote,
falling back to a transient only if none is present (e.g. a direct-seeded test).

The machine parks in the Discovered state until promotion and is inert to the
liveness reap and rekey cadence while unpromoted, since it is absent from the
peers map. It is removed on every path that ends the outbound leg without
promoting -- the stale-connection reaper, the msg2 authorization-failure arm,
and the cross-connection resolution block -- mirroring the connection's own
lifetime so no dangling machine survives.

Its session index is deliberately left unset on the machine (the shell owns the
index on its connection), so a later inbound restart does not emit a spurious
decrypt-session unregister. A regression test covers that invariant.
 2026-07-15 04:06:44 +00:00
Johnathan Corgan e064c96df3 peer: route peer-loss reports by kind 
Add a LostKind discriminator to the ReportLost action so the executor can
route an un-promoted handshake failure to the connected-guarded reconnect
reflex (note_handshake_timeout) and an established peer's link-death to the
unconditional one (note_link_dead), instead of collapsing every loss to
note_link_dead.

The two loss producers dispatched today, the liveness reap and the
inbound-restart-then-promote arm, both keep the link-dead routing, so
behavior is unchanged. The handshake-timeout and dial-failure producers are
tagged accordingly but stay dormant until their events are dispatched.
 2026-07-15 03:37:31 +00:00
Johnathan Corgan a70c725e48 node: drop stale dead-code allows and a redundant clock read 
Remove the now-stale `#[allow(dead_code)]` on `advance_peer_machine`
and `execute_peer_actions`; both are live (called from the link-dead
reap, rekey cadence routing, and the handshake establish sites).

In `route_rekey_cadence`, reuse the ambient context timestamp for the
machine step instead of taking a second clock sample; the extra
sample's only consumer was the currently-inert cutover drain timer, so
the change is behavior-neutral.
 2026-07-15 00:56:58 +00:00
Johnathan Corgan c8077967cd peer: add a round-trip contract test for the action message type 
Prove the per-peer machine's action type is a runtime-agnostic message
contract: it is Send + Sync + 'static, and every variant round-trips
unchanged through a single-threaded async channel (a current-thread runtime,
sender and receiver on one thread). A wildcard-free match over the variants
makes any future action a compile error here until it is added to the sample,
so a variant that embedded a runtime handle could not slip through unproven.
 2026-07-14 23:36:18 +00:00
Johnathan Corgan 5dfa571908 node: pin promote-failure warnings to the handshake tracing target 
The two promote-failure warn! sites in the peer-action executor emit under
this module's own tracing target rather than the handshake target the rest of
the establish-path logging uses. Pin both (outbound "Failed to promote
connection" and inbound "Failed to promote inbound connection") to
fips::node::handlers::handshake, matching the sibling max-peers reject log and
keeping all establish-path diagnostics grouped under one target regardless of
which module physically emits them.
 2026-07-14 23:23:54 +00:00
Johnathan Corgan 6bebca88ac node: rewrite planning-phase locators out of source comments 
Comments across the per-peer machine, executor, lifecycle supervisor,
and peering reconciler carried internal rollout labels and design-note
section references. Rewrite them to describe the code's behavior
directly. Comment-text only; no code changes.
 2026-07-14 01:38:52 +00:00
Johnathan Corgan 5d5da69a5b node: drive the link-dead peer reap through the per-peer machine 
Route each link-dead peer that the tick sweep's plan_heartbeats decides
to reap through the per-peer machine and executor, replacing the inline
reap body in check_link_heartbeats. The batch decision, the liveness
snapshots (read from the hot-path-written receive clock), and the
heartbeat-send arm stay shell-side and byte-unchanged; the machine only
consumes the decided LinkDeadSuspected, tearing the peer down via
remove_active_peer and reporting the loss to the reconciler exactly as
before, on the same tick with the same wall-clock timestamp. The reap
log stays shell-side.

The machine's link-dead handler no longer emits a decrypt-session
unregister keyed by its shadow index (the full peer teardown already
unregisters the real index; the shadow could have drifted to a reused
index), and its guard now covers the Established state a freshly
promoted peer sits in.

Handshake-timeout, retransmit, and stale-connection cleanup stay inline:
they act on pre-promotion legs that have no machine, and the loss reflex
they use differs from the link-dead one.
 2026-07-13 17:08:35 +00:00
Johnathan Corgan e05b868cf8 node: drive rekey cadence cutover and drain through the peer machine 
Route each Cutover and Drain that the shell-side batch poll_rekey decides
through the per-peer machine and the executor, replacing the inline
effect bodies in check_rekey. The batch decision and its per-peer
snapshots stay shell-side and byte-unchanged: poll_rekey phase-groups all
cutovers, then all drains, then all initiations across the peer set, and
that ordering governs the shared index allocator's free-then-allocate
sequence that appears on the wire, so the machine only consumes the
already-decided actions (a new RekeyConsume event) without re-deciding.
InitiateRekey stays inline (its Noise msg1 build is a shell-side leaf)
with a RekeyInitiated observation feeding the machine so its control
state stays coherent for the next tick's cutover.

The cutover and drain logs, which relocated into the executor in the
prior commit, are pinned back to the fips::node::handlers::rekey tracing
target so they stay visible under the operator's module log filter.

Also clears the machine's shadow draining_index on drain so a later
cross-connection resolution cannot double-free the already-freed index.
 2026-07-13 16:20:45 +00:00
Johnathan Corgan 0ebd1b44c0 node: add the initiator-cutover and drain executor actions (unwired) 
Prepare the establish executor to drive FMP rekey by activating the
SwapSendState action (initiator K-bit cutover via cutover_to_new_session,
with the gated decrypt-worker re-registration) and adding a CompleteDrain
action (erase the drained previous session: free its index, drop its
peers_by_index entry, unregister its decrypt session). Both reproduce the
current inline rekey.rs cutover and drain bodies. The machine's drain
mapping now emits CompleteDrain, using the real drained index rather than
a shadow copy.

Unwired: nothing drives the machine's rekey path yet (live rekey still
runs inline), so these arms are unreachable and the change is
behavior-neutral. The cadence fold that routes cutover and drain through
the machine follows.
 2026-07-13 14:51:11 +00:00
Johnathan Corgan 800cfb23e3 node: relocate decrypt-session registration to the establish executor 
Move register_decrypt_worker_session out of promote_connection into the
executor's PromoteToActive handler, gated on a promoted or
cross-connection-won result. Every live promote now flows through that
one executor path, so registration still fires exactly once at the same
synchronous point; the direct test callers of promote_connection spawn
no worker pool, so the call was already a no-op for them.

Add the cross-connection loser-link teardown (close the losing
transport, remove its link, re-point addr_to_link at the winner) to the
executor as a guarded follow-up. It is unreachable on the current driven
establish paths, which only promote net-new peers, and asserts so, but
keeps the executor complete for when that case is driven.

Remove the now-dead drive_promote_to_active and ConnAction::PromoteToActive.
 2026-07-13 14:20:56 +00:00
Johnathan Corgan e9112cc1bb node: drive net-new outbound establish through the per-peer machine 
Cut the net-new outbound handshake completion (a received msg2 that
promotes a fresh outbound leg to a new peer) over to the per-peer
control machine, mirroring the inbound cutover. handle_msg2 still runs
the msg2 prologue, the ACL check, and the cross-connection swap/keep
arms inline; for the net-new promote it now builds a transient machine,
steps it, and drives promote_connection through the executor. The
session index was already allocated at dial, so there is no two-phase
authorize here.

The wire, index sequence, and peer registry state after promote are
byte-neutral. To keep the promote-failure path neutral, the executor's
cleanup now distinguishes inbound from outbound: an outbound promote
failure records the reject only, matching the prior handler, rather
than the inbound path's link and index teardown.

Cross-connection swap/keep, rekey-msg2, and the dial path stay inline;
the loser-link surgery for the currently unreachable driven
cross-connection case lands with the register relocation next.
 2026-07-13 13:42:08 +00:00
Johnathan Corgan c80a7fdea5 node: drive restart inbound establish through the per-peer machine 
Cut the restart handshake (an inbound msg1 from a peer that reconnected
with a new epoch) over to the per-peer control machine, completing the
inbound establish cutover. The old peer is torn down and the fresh leg
promotes through the same two-step authorize-then-allocate path as the
net-new case: the machine's first step emits the old-peer teardown
(invalidate send-state, report loss), the shell interposes the ACL
check, and the second step allocates the new index and sends msg2. The
old index is freed before the new one is allocated and the msg2 wire
bytes are unchanged, so the sequence stays byte-neutral.

With restart driven through the machine, the shared inline establish
tail that only the restart arm reached is deleted.

Also bounds the peer_machines map (remove_active_peer now drops the
peer's machine entry) and restores the msg2-send, promote, and
index-allocation failure warnings the cutover had dropped.
 2026-07-13 12:46:23 +00:00
Johnathan Corgan 0bf031dd32 node: drive net-new inbound establish through the per-peer machine 
Cut the net-new inbound handshake (a fresh msg1 that promotes to a new
peer, plus the at-capacity reject) over from the inline handle_msg1 logic
to the per-peer control machine. handle_msg1 still classifies via
establish_inbound and still owns the Noise wire step, the late ACL check,
and the promote_connection registry surgery; for the net-new path it now
builds the machine, steps it, and executes the returned actions.

Authorization is interposed between two machine steps so the session
index is allocated only after the ACL check passes: a rejected or
unauthorized msg1 consumes no index, matching the prior order exactly.
The msg2 wire bytes, the index-allocation sequence, and the reject
metrics are all byte-neutral. Restart, resend, rekey-respond, and the
other reject arms stay inline unchanged; they move to the machine once
outbound establish is cut over and every promoted peer has a machine.

Also fills in the executor's send-failure and promote-failure cleanup so
a mid-establish error tears the leg down and frees its index exactly as
before.
 2026-07-13 12:05:17 +00:00
Johnathan Corgan 4a0584a5e9 node/dataplane: add the per-peer machine home and action executor (unwired) 
Add Node.peer_machines (a LinkId-keyed map from the stable link handle to
the per-peer control machine) as the home for the machines, and a new
dataplane/peer_actions.rs holding execute_peer_actions / advance_peer_machine:
the executor that maps each PeerAction the machine emits to its shell call —
frame and send a handshake via build_msg2, drive promote_connection and feed
the PromotionResult back through the machine, tear a peer down via
remove_active_peer, free session indices, report loss via note_link_dead.
Actions for the rekey, connected-UDP, and timer paths are stubbed with notes
for the commits that fold those mechanisms in.

Unwired: nothing drives the machine yet — no live handler path calls the
executor and peer_machines is never populated — so this is behavior-neutral;
the inbound and outbound establish paths still run their existing inline
logic. The executor is cut over path-by-path in the following commits.
 2026-07-13 11:11:20 +00:00
Johnathan Corgan 59155df4e3 peer: split ActivePeer send-state into PeerSendState (two-tier boundary) 
Draw the control/published-send-state boundary inside ActivePeer by
grouping the send-critical fields — the three epoch session slots
{current, previous, pending}, the K-bit flag and session-start, the
transport target, the connected-UDP handles, and the hot counters —
into a new co-located PeerSendState struct. The control-tier fields
(identity, connectivity, declaration/ancestry, filter and tree-announce
groups, remote_epoch, the rekey-negotiation sub-machine, and the rest)
stay on ActivePeer.

Behavior-neutral: a pure field regrouping. Every accessor signature is
unchanged (bodies now read/write self.send.*), so the hot path and the
handlers are byte-untouched; both K-bit cutovers still rotate the three
slots atomically with the same control-tier updates. No Arc/ArcSwap —
the fields are co-located and read by plain borrow; publishing behind a
shared cell is later plumbing for a sharded data plane.
 2026-07-13 10:36:15 +00:00
Johnathan Corgan fcaee74ec0 peer: add the per-peer FMP control state machine (unwired) 
Introduce src/peer/machine.rs: a sans-IO per-peer control FSM that
consolidates the scattered handshake/rekey/timeout driver logic now
spread across node/handlers. The machine is a pure reducer —
step(event, now, index_allocator) -> [action] — that reuses the
existing FMP decision cores (establish_inbound/establish_outbound/
cross_connection_winner/poll_*) rather than reimplementing any
decision, and returns runtime-agnostic actions the driver executes.

Control-tier state only; the published send-state boundary and the
driver wiring land in following commits. The machine is terminal at
Closed — re-dial is the reconciler's, so it holds no cross-attempt
retry state.

Includes eight unit tests: inbound and outbound establish, N:1
identity crystallization, the dual-initiation tie-break,
restart-override, rekey initiator cutover, the data-plane-owned
responder cutover boundary, and liveness -> link-dead -> report-lost.
Unwired — nothing calls it yet.
 2026-07-13 09:58:45 +00:00
Johnathan Corgan 56e3d56c25 peer/connected_udp: own the connected-socket fd with OwnedFd 
Replace the hand-rolled RawFd field and the unsafe Drop on
ConnectedPeerSocket with an OwnedFd, whose own drop glue closes the
fd. from_fd now stores the OwnedFd it already receives instead of
stripping ownership through into_raw_fd, and the manual libc::close is
gone, shrinking the unsafe surface.

Behavior-neutral: the fd still closes exactly once at last-Arc-drop and
as_raw_fd returns the same underlying fd while the socket is alive.
 2026-07-13 09:20:04 +00:00
Johnathan Corgan 7b0590f70e node/lifecycle: wire the runtime child-exit producer 
Wire exit detection for the four directly-observable optional children so
the supervisor FSM's ChildExited edge fires at runtime. A runtime
child-liveness mpsc channel carries a Child on exit: the DNS task and the
two TUN threads self-report when their body returns, and a 2s poll monitor
reports mDNS and Nostr via new is_finished accessors. The rx_loop gains a
select arm that steps the FSM and republishes health (Degraded, since a
running node always has at least one transport up). Transports and worker
pools expose no runtime-exit signal and are left for a follow-up.

Adds LanRendezvous::is_finished and NostrRendezvous::is_finished; the
latter treats a shutdown-taken connect_task as finished so the monitor
terminates after a stop instead of polling forever.
 2026-07-13 08:48:51 +00:00
Johnathan Corgan b93a127623 node/lifecycle: add runtime ChildExited health routing to the supervisor FSM 
Add the sans-IO half of runtime child-liveness monitoring: a ChildExited
event and an on_child_exited handler that routes a runtime task or thread
exit the same way a start failure does. An optional child exiting degrades
the node (Degraded, with the child recorded in the health reasons); the
last transport exiting publishes Failed. There is no restart, and the
handler is inert outside Running (startup, drain, and teardown own their
own child bookkeeping via the pending/up sets). Failed here is a published
health signal only, not a teardown.

The start and runtime paths share the health classification, extracted
from resolve_start_health into classify_health.

The producer that emits the event (the exit-detection wiring) lands in a
following commit; the event variant carries a temporary allow(dead_code)
until then.
 2026-07-13 08:25:45 +00:00
Johnathan Corgan 85a4983dbe node/peering: restore the open-discovery sweep summary log 
The overlay-discovery cutover to the sans-IO reconciler dropped the
operator-facing "open-discovery sweep complete" summary, including its
per-reason skip breakdown, because the sweep logic moved into the
log-free core. Have the core accumulate the enqueue/skip tally as it
reconciles and return it as data; the driver adds the two values only it
holds (the raw cache size and the self-advert filter) and emits the
summary, reproducing the old startup-vs-per-tick summarize gate and the
budget-zero debug path.

The self-advert precedence matches the pre-cutover sweep: a stale or
self-configured own advert is attributed to the age/configured buckets,
not skipped_self.

Behavior-neutral: the tally is pure side-counting; no dial or enqueue
decision changes.
 2026-07-13 08:00:46 +00:00
Johnathan Corgan 5090ab7851 node/peering: drive opportunistic transport-neighbor growth through the reconciler 
Cut the last scattered peering mechanism — opportunistic growth from
transport-neighbor beacons and LAN mDNS — over to the reconciler's
opportunistic layer via a gate-checked reconcile_opportunistic wrapper,
one call per tick slot (transport and LAN stay separate slots so their
per-tick budget and per-peer cap are not shared). The driver keeps the
beacon/mDNS I/O and the path-granular prefilters that read live state
(self, fresh-enough-to-skip, connecting-on-path) and executes the emitted
Connect intents; the core owns the connected/budget/per-peer-cap
decisions. A first-wins per-peer dedup on the LAN path reproduces the old
inline once-per-peer dial now that the snapshot core cannot observe the
intra-tick connecting feedback.

Delete the now-dead discovery_connect_budget helper. With this, all three
scattered peering mechanisms (auto-connect and retry, overlay discovery,
neighbor growth) are unified in one sans-IO reconciler. Behavior-neutral;
unit test count unchanged (1607).
 2026-07-13 06:02:21 +00:00
Johnathan Corgan 03ced618ce node/peering: drive Nostr open-discovery through the reconciler overlay layer 
Cut the overlay (Nostr open-discovery) enqueue over to the sans-IO
reconciler. A gate-checked reconcile_overlay wrapper runs the overlay
layer alone at the discovery tick slot; the monolithic reconcile would
re-fire the always-on retry-dial and double the per-tick dial cap, so the
overlay slot must call only its layer. The driver builds the candidate
pool from the overlay advert cache (self excluded), the cooldown and
configured-npub sets, and the startup/steady max-age, then feeds the
reconciler; the emitted enqueue set drives the per-enqueue identity-cache
and alias pre-seed exactly as before. Enqueued entries are dialed at the
retry slot, preserving the two-phase cadence.

Delete the now-dead enqueue-budget and expiry helpers. Behavior-neutral:
same candidates enqueued in the same order with the same budget, cap, and
expiry. Opportunistic transport-neighbor growth remains imperative and is
cut over next. Unit test count unchanged (1607).
 2026-07-13 05:34:28 +00:00
Johnathan Corgan bf81f422ea node/peering: drive the mandatory-floor and retry mechanism through the reconciler 
Route the auto-connect floor and the connection-retry mechanism through
the sans-IO reconciler instead of the imperative Node methods. A thin
peering driver (note_handshake_timeout / note_link_dead) centralizes the
reflex path with the gate guard and the already-connected check; the
per-tick retry-dial and the startup floor build reconciler inputs and
execute the emitted Connect intents. The three imperative retry methods
are removed and their call sites rerouted.

Wire the drain and startup gates. Entering a bounded drain now clears the
retry schedule and suppresses the peer-loss reconnect reflex (the drain
window runs under the Suspended gate), so the drain no longer fights a
reconnect for the peers it just closed. The startup floor runs under an
explicit Reconciling gate at the existing peer-connect seam, preserving
the current dial position.

Behavior-neutral except the intended drain change. Overlay discovery and
opportunistic transport-neighbor growth remain imperative for now; they
observe the relocated retry schedule and are cut over next. Unit tests
migrated to the driver API with assertions unchanged (1607).
 2026-07-13 05:00:10 +00:00
Johnathan Corgan a0cf593580 node: relocate the peering fields off Node into the Peering owner 
Move retry_pending and pending_connects from Node into the Peering owner
struct (Node.peering), the home introduced with the reconciler core.
Pure mechanical relocation: the two collections now live behind
self.peering, with roughly seventy accessor sites repointed. No decision
logic is touched and the unit-test count is unchanged (1607). The
imperative peering methods still run; the reconciler is wired in the
following commit.
 2026-07-13 04:24:34 +00:00
Johnathan Corgan 5d08d27d3c node/peering: add the sans-IO peering reconciler core (unwired) 
Introduce PeeringReconciler, the synchronous decision core for peer
desired-state, with its input/action vocabulary (Gate, Budget, Observed,
Candidate, DiscoveryPools, Policy, PeeringAction) and the Peering owner
struct. reconcile() computes connect/retry intents across four layers:
the auto-connect mandatory floor, the ceiling-only overlay enqueue, the
opportunistic transport-neighbor growth, and the node.limits ceiling
enforced inline. It mirrors the supervisor sans-IO shape: no I/O, no
clock reads (time enters as a parameter), no runtime handles.

The core owns its own cross-attempt retry schedule so escalating backoff
survives the fresh connection created per re-dial. Gate-guarded reflexes
(on_handshake_timeout / on_link_dead) reproduce the current backoff math
and no-op while draining. Overlay is strictly ceiling-only (no peer-count
set-point). Disconnect is defined but never emitted (no shedding).

Unwired here: the driver still runs the imperative peering methods and
Node holds the live retry map. 11 unit tests drive the pure core with
synthetic inputs. No behavior change.
 2026-07-13 04:10:36 +00:00
Johnathan Corgan b676c9d83a node: rewrap over-width retry_state_iter signature 
Homing the retry schedule under peering/ lengthened the RetryState
module path, pushing the retry_state_iter return-type line past the
100-column max_width. Rewrap to satisfy rustfmt. No behavior change.
 2026-07-13 04:00:41 +00:00
Johnathan Corgan a45eefb58a node: home the peering concept; relocate the connection-retry schedule 
Create src/node/peering/ as the home for the peer desired-state
(homeostatic reconciler) concept and move the cross-attempt connection
retry schedule into it: RetryState plus schedule_retry /
schedule_reconnect / process_pending_retries.

Mechanical relocation only. The methods remain inherent on Node; the
retry_count must persist across re-dials (a fresh connection is created
each attempt), which is why the schedule belongs in the peering home
rather than on a per-connection type. The one-level-deeper module path
requires pub(super) -> pub(in crate::node) to preserve the prior scope,
plus module-path fixups at the reference sites. No behavior change; unit
test count unchanged (1596).
 2026-07-13 03:30:52 +00:00
Johnathan Corgan d61d189572 node: split Running into Full/Degraded, add Failed health state 
Determine node health at start completion instead of unconditionally
reaching Running. Zero transports up is now Failed (fatal): start()
tears down cleanly and returns an error, and the daemon exits. Any
configured optional child that failed to start - a transport beyond the
first, Nostr, mDNS, TUN, DNS, or a worker pool - leaves the node
Degraded but serving, with an operator warning naming what failed. All
configured children up is Full. A child the node was never asked to run
does not count against health.

The published NodeState gains Degraded and Failed variants, both visible
via control queries; Degraded is operational, Failed is not. The
lifecycle FSM gains the health states plus the PublishState action that
drives them - a health fork cannot be a single direct state write, which
is why the earlier commits deferred it to here.

Runtime child-exit health re-evaluation (a running child dying) is a
separate liveness-monitoring mechanism left for a follow-up; this commit
is start-time health only.
 2026-07-13 00:46:31 +00:00
Johnathan Corgan d6ca632251 node: add bounded graceful-shutdown drain phase 
Add an operator-visible Draining phase on daemon shutdown. On the
shutdown signal the node broadcasts Disconnect to all peers, then keeps
serving for a bounded window - up to node.drain_timeout_secs (default 2s),
exiting early once all peers are gone - before tearing down. This lets
in-flight traffic settle and peers observe the disconnect before the
transports close, rather than the previous immediate teardown.

The lifecycle FSM gains a Draining state plus Drain/DrainDeadlineElapsed
events; the run loop observes the shutdown signal and transitions to
draining in place - one continuous loop, so the channel receivers are
never destructively cancelled. The published NodeState gains a Draining
variant, visible via control queries during the window. The immediate
stop() path used by tests and non-daemon callers is unchanged: it still
tears down immediately with no drain wait.

The reconciler-gate actions the drain emits are no-ops until the peering
reconciler lands and consumes them.
 2026-07-12 23:11:28 +00:00
Johnathan Corgan 6c5fd3f4b0 node: extract lifecycle supervisor FSM, migrate substrate fields (behavior-neutral) 
Introduce a sans-IO lifecycle supervisor: a synchronous step(event) ->
[action] state machine (SupervisorFsm) that authors the substrate-child
spawn and teardown order, plus an owner struct (Supervisor) holding the
substrate runtime fields and embedding the FSM. The substrate-lifecycle
fields leave Node's flat list into the owner - state, packet_tx, the TUN
reader/writer handles + shutdown fd + channels, the DNS task + identity
channel, the Nostr/LAN rendezvous drivers, and the encrypt/decrypt worker
pools; the dataplane keeps packet_rx.

start()/stop() become the driver executing the FSM's SpawnChild/StopChild
actions: same children, same order, same warn/debug-and-continue on
optional failures, same logs, same NodeState transitions. Behavior- and
wire-neutral. The only determinism change is that transports now tear
down in ascending-id order, previously nondeterministic HashMap iteration.

The bounded Draining phase and the Running{Full|Degraded} health split
land as separate follow-on commits.
 2026-07-12 21:03:46 +00:00
Johnathan Corgan 434b9726aa node: establish dataplane/ and session/ concept homes (behavior-neutral) 
Reorganize the node module tree by concept rather than by
message-handling verb, as the first step of the node runtime
decomposition. Pure relocation: no wire, config, metric, or log
semantics change; the lib test count is unchanged (1577 passed).

Moves (git mv, 100% rename similarity):
- handlers/{forwarding,rx_loop,connected_udp,dispatch,encrypted}.rs
  -> node/dataplane/ — the whole RX hot path (the select! run loop,
  transit/local forwarding, the link-message router, the RX decrypt
  path with responder K-bit cutover + roam writes, and connected-UDP
  fast-path activation) now lives in one home.
- node/session.rs -> node/session/mod.rs — establishes the session
  concept home for the data/state types. The message-behavior file
  handlers/session.rs stays put for now (folds in with the later FSP
  session step).

The IK/XX-divergent establishment files (handlers/{handshake,rekey,
timeout}.rs) and the deferred-home files (handlers/{mmp,lookup}.rs)
deliberately stay in handlers/, to move once rather than twice.

Every module is reached through impl Node methods, so no call site or
re-export shim was needed. Updated in lockstep with the moves: the
module_path!-derived tracing targets in the two mesh-lab compose-trace
overlays, a structural test's include_str! source path, doc-comments
in proto/routing and the mesh-lab docs, and the stale source-location
citations (node/handlers/{forwarding,rx_loop,encrypted}.rs and
node/session.rs) in doc-comments and the discovery design doc.
 2026-07-12 19:22:08 +00:00
Johnathan Corgan 26d70ebb59 Merge branch 'maint' into master 
Forward-merge three maint fixes, hand-relocated into master's
post-sans-IO / discovery-to-lookup structure:
- drop the redundant TreeMetrics parent_switched counter (keep
  parent_switches), reconciled across the refactored tree/mmp sites and
  the spanning-tree test reads
- keep the tighter path_mtu when applying a LookupResponse, now in
  handlers/lookup.rs after the discovery module rename
- reuse one shared secp256k1 context in the identity module
 2026-07-12 16:51:01 +00:00
Johnathan Corgan 567e6a535e identity: reuse one shared secp256k1 context 
Every sign/verify/key-derive site built a fresh context via
Secp256k1::new(), which allocates a Secp256k1<All> and runs
randomization/blinding table setup on each call. Introduce one
crate-wide LazyLock<Secp256k1<All>> and reuse it across the local, peer,
and auth sites (and their tests). Behavior-neutral: identical secp256k1
API calls, only the context lifetime changes, and the shared All context
still performs the standard construction-time blinding.
 2026-07-12 16:33:53 +00:00
Johnathan Corgan 6011d233c1 discovery: keep tighter path_mtu when applying a LookupResponse 
An originator handling a LookupResponse unconditionally overwrote the
cached path_mtu_lookup entry, so a looser (larger) estimate in a later
response could clobber a tighter value already learned from a reactive
MtuExceeded or PathMtuNotification. Read-and-compare before writing and
keep the minimum, so a looser discovery estimate no longer loosens the
clamp. Add a regression test.
 2026-07-12 16:29:01 +00:00
Johnathan Corgan cb5a32693e tree: remove redundant parent_switched metric counter 
parent_switched was incremented on the line immediately before
parent_switches at every site and never independently, so the two
counters were always identical. Drop parent_switched from TreeMetrics,
its snapshot, TreeStatsSnapshot, the show_tree fixture, and the fipstop
render, keeping parent_switches as the sole counter.
 2026-07-12 15:53:03 +00:00
Johnathan Corgan 9b46b6fa85 fipstop: rename routing-stats "Discovery" labels to "Lookup" 
The routing-stats pane's Discovery Requests/Responses sections show the FMP
overlay coordinate-lookup counters. Rename the section labels and the nested
JSON key they read from "discovery" to "lookup" to match the metric family's
canonical name. The daemon dual-emits both keys, so this reads the current
name and no longer depends on the deprecated "discovery" alias.
 2026-07-12 01:52:54 +00:00
Johnathan Corgan cbc089b820 transport/ethernet: rename config discovery flag to listen 
Rename the Ethernet per-interface config flag from discovery to listen,
so the receive/transmit toggle pair reads as the symmetric announce
(transmit) / listen (receive) neighbor-beacon vocabulary. The old
discovery: key is still accepted via a serde alias, so deployed configs
load unchanged; to_yaml re-emits it under the canonical listen: name.
Marked deprecated for removal at the v2 cutover.

Updates the config field + accessor, the transport listen_enabled local,
the one struct-literal test consumer, the chaos sim config generator,
packaged fips.yaml examples, and the classified operator-facing docs
(ethernet neighbor-beacon subsystem prose; the generic Transport
discovery capability prose is left unchanged). Adds a compat parse test
asserting the legacy alias, the new key, and that deny_unknown_fields
still rejects unknown keys. Behavior-neutral.
 2026-07-11 23:13:28 +00:00
Johnathan Corgan 4c95be0000 transport/ble: rename discovery module to neighbor 
Mirror the ethernet neighbor rename in the BLE transport: the internal
peer-detection buffer becomes NeighborBuffer (from DiscoveryBuffer) and
the module is renamed discovery -> neighbor. The BlueR/bluez API terms
(DiscoveryFilter, set_discovery_filter) and the BLE advertise/scan
mechanism vocabulary are unchanged, as is BleConfig. Behavior-neutral.
 2026-07-11 23:01:48 +00:00
Johnathan Corgan e362ab67a6 transport/ethernet: rename discovery module to neighbor 
Rename the ethernet link-local neighbor-beacon subsystem from the
overloaded "discovery" vocabulary to a neighbor umbrella. The beacon
frame vocabulary is retained (build_beacon/parse_beacon/BEACON_SIZE/
FRAME_TYPE_BEACON/FRAME_TYPE_DATA); only the subsystem and buffer
identifiers change: DiscoveryBuffer becomes NeighborBuffer,
discovery_buffer becomes neighbor_buffer, and DISCOVERY_VERSION becomes
BEACON_VERSION (wire value 0x01 unchanged). Config toggles are left for
a follow-up commit. Behavior-neutral.
 2026-07-11 22:56:05 +00:00
Johnathan Corgan 6c9f55ea80 transport: rename darwin_sockopts to sockopts_macos 
The module is gated target_os="macos" (not the broader Darwin/iOS family),
so the name now tracks the cfg and matches the *_macos.rs file convention
(io_macos.rs). Drops the redundant inner #![cfg(target_os="macos")] (the
decl gate already covers it) and corrects a stale top comment that claimed
the module stays visible on Linux — it is macos-decl-gated, so it never was.
Mechanical rename; no logic change.
 2026-07-11 21:37:00 +00:00
Johnathan Corgan 89a31fd555 transport: move the connected-UDP fast-path handles into the peer module 
ConnectedPeerSocket and PeerRecvDrain are node/peer-side logic: the Transport
trait never touches them, ActivePeer stores them, and node's encrypt worker
drives them. They only happened to live under transport/udp. Relocate the
handle types to a new src/peer/connected_udp/ module (socket.rs + drain.rs),
which the node handler and encrypt worker reach via node -> peer (no new edge;
a node home would have forced a peer -> node cycle).

The udp transport keeps only the kernel-construction seam: open_connected_fd,
now folded into udp/io.rs (the byte-layer home) behind a linux/macos-gated
submodule and re-exported as transport::udp::open_connected_fd. The node
handler builds the fd through it and adopts it via ConnectedPeerSocket::from_fd.

Behavior-neutral: no wire/config/metric/log change; the per-packet hot path
(bare-RawFd send_batch_gso/raw) is untouched. Preserves the Arc multi-owner
contract, the drop-drain-before-socket ordering, and the drain's detach-on-Drop
deadlock avoidance. Reconciles the old cfg(unix)/any(linux,macos) double-gate
onto the single any(linux,macos) predicate.
 2026-07-11 21:30:36 +00:00
Johnathan Corgan ab0a46f2c0 transport: extract udp open_connected_fd syscall body into a standalone fn 
Pulls the connected-UDP socket construction (socket/REUSEADDR/REUSEPORT/
BUFFORCE/bind/connect + darwin tuning) out of ConnectedPeerSocket::open into
a pub(crate) open_connected_fd returning an OwnedFd; open() now delegates and
adopts the fd. Error paths still close the fd via the temporary's Drop; the
success tail transfers ownership through OwnedFd. Behavior identical. Prepares
the fast-path handle types to move to the peer module while the socket
construction stays with the udp transport.
 2026-07-11 21:08:29 +00:00
Johnathan Corgan e839aead7a transport: extract tcp connection-pool types into pool.rs 
Moves the inline TcpConnection/ConnectingEntry structs, the Direction
enum, and the ConnectionPool/ConnectingPool type aliases out of the
1147-line tcp/mod.rs into a dedicated pool.rs, matching the canonical
per-transport layout. Struct fields are pub(crate) so mod.rs can still
construct and read them across the module boundary. Pure relocation; no
logic change.
 2026-07-11 20:26:42 +00:00
Johnathan Corgan 6d6889d0f6 transport: give ethernet a canonical addr.rs home for MAC parsing 
Moves parse_mac_string into ethernet/addr.rs and re-exports it at the
ethernet module root so the ethernet::parse_mac_string path stays
byte-identical for its one external consumer (zero churn there). Its
unit tests stay in mod.rs, calling through the re-export. Pure
relocation; no logic change.
 2026-07-11 20:19:08 +00:00
Johnathan Corgan 196d9492da transport: rename ethernet byte-layer module from socket to io 
Normalizes the ethernet transport onto the canonical byte-layer name
(io.rs), including the per-OS files io_linux.rs/io_macos.rs and their
#[path] wiring. Pure file rename plus module-path and doc-comment
updates; no logic, wire, config, metric, or log change.
 2026-07-11 20:14:37 +00:00
Johnathan Corgan 0f2e91b479 transport: rename udp byte-layer module from socket to io 
Normalizes the udp transport onto the canonical per-transport module
layout (ble is the reference: io/pool/addr/stats). Pure file rename plus
module-path updates at the use sites; no logic, wire, config, metric, or
log change. Behavior identical.
 2026-07-11 20:10:19 +00:00
Johnathan Corgan 32475d859e transport: consolidate tor and nym SOCKS5 dialing into a shared module 
The Nym transport was a near-clone of the Tor transport's outbound
SOCKS5 path. Extract the shared logic into src/transport/socks5/ so both
transports drive one implementation instead of two maintained copies:

- share one SOCKS5 mock server between the tor and nym tests
- extract the common send/receive/connect counters into a shared
  ProxiedStatsBase (snapshot structs and emitted metrics unchanged)
- add a shared Socks5Dialer that collapses all six connect variants;
  the sole dialing difference (tor's per-destination circuit-isolation
  auth vs nym's no-auth) is an enum on the dialer
- route both tor and nym dialing through the shared dialer
- share the proxied connection pool, generic over a per-connection meta
  type that carries tor's inbound/outbound direction counting (nym uses
  the unit type)
- share the proxied receive loop

Behavior-neutral: no wire-format, config-key, emitted-metric, log, or
error-variant change. Tor keeps its control-port, inbound/onion, and
directory surface and its own address validation.
 2026-07-11 18:45:41 +00:00
Johnathan Corgan f2e6b8befb transport: promote FMP stream framing to transport::framing 
The FMP frame-boundary reader lived in transport/tcp/stream.rs, but the
Tor and Nym transports both reached across module boundaries to
'use crate::transport::tcp::stream::read_fmp_packet', a layering smell:
the reader is a shared stream-framing utility, not a TCP-private one.
Move the file to transport/framing.rs and repoint the tcp/tor/nym use
sites, removing the tor->tcp and nym->tcp dependencies. Behavior
unchanged.
 2026-07-11 04:32:38 +00:00
Johnathan Corgan 1aacdfa086 transport: share connection-pool counters via PoolCounters 
TcpStats and TorStats each carried an identical pair of pool_inbound/
pool_outbound atomics plus the same five record_pool_* / pool_inbound_count
methods over them. Move the counter logic into a shared PoolCounters
struct (new transport/stats_common.rs) embedded as a 'pool' field in both.
The public record_pool_* methods stay as thin delegators so all call
sites and the flat pool_inbound/pool_outbound snapshot fields are
unchanged; only the duplicated atomic bookkeeping is now single-sourced.
 2026-07-11 04:30:19 +00:00
Johnathan Corgan a7dfe47663 transport: derive Serialize for EthernetStatsSnapshot 
EthernetStatsSnapshot derived only Clone/Debug/Default, unlike every
other transport's *StatsSnapshot which also derives Serialize. That gap
forced a hand-rolled serde_json::json!{} arm in TransportHandle::
transport_stats() that re-listed all ten fields by hand. Add the
Serialize derive and collapse the arm to the same one-line
serde_json::to_value(...) form the other transports use. The emitted
JSON keys and values are unchanged.
 2026-07-11 04:27:50 +00:00
Johnathan Corgan 8aab71af86 bench: add routing next-hop microbench 
Measures the per-forwarded-packet cost of routing candidate assembly
(routing_candidates over a synthetic RoutingView) against a zero-alloc
reference across 8/32/128/256 peers, with per-call allocation counts via
a counting allocator. Criterion harness; no production code change.
 2026-07-11 01:48:34 +00:00
Johnathan Corgan 2cffc10520 Merge branch 'maint' (bloom single-digest double-hashing) 
Forward-merges the bloom SHA-256-once fix from maint. The bloom filter was
relocated to proto/bloom/ by the sans-IO refactor, so the fix applied cleanly
to proto/bloom/core.rs; the behavior-neutral test was re-homed into
proto/bloom/tests/core.rs (maint carried it in the pre-split bloom/tests.rs).
 2026-07-11 01:32:48 +00:00
Johnathan Corgan 81e4207631 bloom: compute the SHA-256 digest once per double-hash probe 
BloomFilter derived all k hash functions from one SHA-256 digest but
recomputed that digest inside the per-function loop, so every insert and
contains ran SHA-256 hash_count times (5x at the default) over the same
bytes. Hoist the digest out of the loop: base_hashes() computes it once
and returns (h1, h2), and bit_index() derives each of the k indices with
the same (h1 + k*h2) mod m arithmetic. Bit-for-bit identical output; this
is the hottest path in packet forwarding and mesh-size estimation.

Adds a test pinning the bit indices against the double-hashing formula
recomputed independently, proving the refactor is behavior-neutral.
 2026-07-11 01:22:07 +00:00
Johnathan Corgan 1c1ed0d939 Relocate PromotionResult into proto/fmp 
Move the PromotionResult enum (and its impl) out of peer::mod and into
proto/fmp/core.rs, alongside the cross_connection_winner tie-break helper
that was relocated the same way. This is FMP connection-lifecycle result
vocabulary, so it belongs in the FMP subsystem home rather than the peer
module.

Behavior-neutral pure type relocation: consumers import it from
crate::proto::fmp, and the crate-root public path crate::PromotionResult
is preserved via a re-export in lib.rs (mirroring cross_connection_winner).
Full lib suite green at baseline.
 2026-07-10 16:52:18 +00:00
Johnathan Corgan 1c41f73931 Relocate FMP link wire codec into proto/fmp 
Move the FMP mesh-layer wire format (common prefix, encrypted/msg1/msg2
headers, and the build_*/inner-header codec fns) out of node/wire.rs and
into proto/fmp/wire.rs, so the whole FMP wire surface lives with its
subsystem, matching the proto/fsp/wire.rs layout. The wire module becomes
pub(crate) mod wire; callers reach it via crate::proto::fmp::wire.

Behavior-neutral: pure relocation plus import-path rewrites across the
node/peer consumers; no logic change. Full lib suite green at baseline.
 2026-07-10 16:42:17 +00:00
Johnathan Corgan 39ad4d2e67 Extract the nostr rendezvous decision logic into sans-IO state machines 
Carve the nostr rendezvous engine's decision logic out of the async
NostrRendezvous driver into synchronous, clock-injected cores, following
the failure_state.rs pattern: sync state behind std::sync::Mutex, time
passed in as now_ms, no .await and no I/O in the core. The driver performs
all relay/socket I/O and executes the returned plans.

- AdvertMachine (src/nostr/advert.rs): advert publish/cache/fetch/prune
  decision logic; the driver executes a returned PublishPlan.
- TraversalMachine (src/nostr/traversal_machine.rs): the engine-scoped
  cross-session decision state -- in-flight-initiator dedup, the dual-init
  responder suppression election, and the replay/seen-session cache.
- classify_punch_packet (src/nostr/traversal.rs): a pure classifier for
  the punch recv loop's packet branch table.

The driver keeps all I/O, the NAT punch send cadence, the offer-slot
admission semaphore, and the pending-answer oneshot routing. Behavior-
neutral: no wire, config-key, or metric change. Adds unit coverage for
the advert plans, election ordering, replay eviction, initiator dedup,
and punch classification.
 2026-07-10 03:28:11 +00:00
Johnathan Corgan 4d2504f59d Update trace-target references for the relocated rendezvous modules 
Moving the nostr rendezvous engine to src/nostr/ (and mDNS to src/mdns/)
changed the module-path-derived tracing targets from
fips::discovery::nostr::* to fips::nostr::*. Update the RUST_LOG filters
in the NAT and mesh-lab test compose files and the resolve-peers-via-nostr
tutorial's debug recipe to the new targets so trace configs and the
documented journal-watch command keep emitting the intended lines.

Without this, the stun-faults suite's Phase-0 pre-flight (which greps the
daemon journal for the debug-level "STUN observation succeeded" /
"traversal: initiator STUN observed" lines) saw those lines suppressed —
the daemon behaved correctly, but the stale RUST_LOG target hid the
evidence. Test-harness and docs only; no source or behavior change.
 2026-07-09 23:20:28 +00:00
Johnathan Corgan 1208f6a5c2 Consolidate the nostr rendezvous driver into src/nostr 
Pull the rendezvous driver state and the movable driver logic out of the
Node struct into the src/nostr home. A new RendezvousDriver owns the
engine handle and the four bookkeeping fields (traversal start time,
startup-sweep latch, adopted bootstrap-transport set and their npubs)
that previously sat loose on Node; Node holds a single driver field and
reaches the same data through thin accessors, including the rx-loop
hot-path protocol-mismatch hook.

The advert build/refresh, the via_nostr fallback-address resolve, the
overlay-endpoint-to-PeerAddress mapping, and the bootstrap request move
onto the driver, taking their Node inputs explicitly (a transport-
endpoint snapshot for advert building) rather than reading Node fields
directly. Transport/connection-table-bound work (traversal adoption,
bootstrap-transport cleanup, the open-discovery sweep, and the outbound
budget calculators) stays on Node as thin glue that calls into the
driver.

Behavior-neutral relocation: statements moved verbatim, no logic, wire,
config-key, or metric changes. lifecycle.rs shrinks ~230 lines. cargo
fmt/build/clippy clean; lib suite 1547 passing (baseline unchanged).
 2026-07-09 22:13:56 +00:00
Johnathan Corgan 3f80530cc5 Move nostr peer rendezvous and mDNS into dedicated module homes 
Relocate the overlay peer-rendezvous subsystem out of the overloaded
src/discovery/ tree into two focused, independent homes: src/nostr/
(relay-mediated overlay endpoint advertise/resolve/auto-mesh plus NAT
traversal) and src/mdns/ (link-local DNS-SD rendezvous). The two
subsystems are independent, so they get separate homes rather than
sharing one.

Drop the ambiguous "Discovery" stem from their identifiers in favor of
"Rendezvous": NostrDiscovery -> NostrRendezvous, LanDiscovery ->
LanRendezvous, and the matching config, policy, field, and method names.
The former src/discovery.rs handoff types (EstablishedTraversal,
BootstrapHandoffResult, the punch-packet helpers) fold into
src/nostr/handoff and stay reachable via the crate-root re-exports.

Pure relocation and rename: no logic, wire-format, config-key, metric,
or tracing-target changes. The operator-facing node.rendezvous.nostr.*
and node.rendezvous.lan.* config keys and the fips-overlay-v1 advert
namespace are byte-identical. cargo fmt/build/clippy clean; lib test
suite 1547 passing (baseline unchanged).
 2026-07-09 21:51:54 +00:00
Johnathan Corgan 3b401a0cbd Disambiguate the mesh-lookup "discovery" name across the source tree 
The identifier "discovery" named three unrelated subsystems; the FMP
overlay coordinate-lookup subsystem is now consistently "lookup". This
finishes the concept-#1 rename across the shell, config, and metric
layers left after the earlier proto-layer rename:

- Handler module node::handlers::discovery -> node::handlers::lookup, and
  reset_discovery_backoff -> reset_lookup_backoff.
- The Node lookup-engine field Node.discovery -> Node.lookup, renamed by
  resolved binding so the metrics().discovery and node.discovery config
  paths are left untouched.
- The lookup metric types DiscoveryMetrics -> LookupMetrics,
  DiscoveryStatsSnapshot -> LookupStatsSnapshot, and Metrics.discovery ->
  Metrics.lookup.

Two surfaces cross a stability boundary and ship behind a compatibility
window, both marked in-code for removal at the v2 cutover:

- The control-socket metric family is dual-emitted under both "discovery"
  (deprecated alias) and "lookup" so existing dashboards keep working.
- The node.discovery.* config table is split into node.lookup.* (mesh
  lookup scalars) and node.rendezvous.* (nostr/lan peer rendezvous).
  NodeConfig does not deny unknown fields, so a naive rename would make a
  deployed node.discovery: block deserialize into nothing and silently
  revert every setting to default. A deprecated all-Option
  DiscoveryConfigCompat field captures a legacy block and a new post-parse
  Config::normalize_deprecated_keys pass folds it into the new tables with
  a one-time deprecation warning.

Flip the packaged fips.yaml templates to the new keys, add legacy/new/
scalar compat parse tests, and record the split and deprecations in the
CHANGELOG. Behavior-neutral; fmt/clippy clean, lib suite green.
 2026-07-09 15:43:31 +00:00
Johnathan Corgan 0b2212e1e8 proto: bound powi test drift by relative tolerance, not ULP 
The previous portability fix bounded the powi test's drift from std::powi
at an absolute ULP count, but Windows/MSVC drift grows with the exponent
(3 ULP by exp=14), so no fixed ULP bound is portable.

Replace the ULP oracle with a loose relative-tolerance sanity sweep
(1e-11), which any sane libm clears by ~1000x regardless of exponent
while still catching a grossly wrong impl. Add a golden-bit pin for the
exp=14 case that exposed the growth. The powi function is unchanged; the
golden-bit determinism pins already passed on Windows.
 2026-07-09 01:30:18 +00:00
Johnathan Corgan b2ce7cd3c8 proto: make the powi guard test portable across platforms 
The powi guard test asserted our square-and-multiply result was
bit-for-bit equal to f64::powi, which failed the Windows unit-test job by
one ULP. f64::powi is not portably bit-stable: Linux and macOS lower it
to compiler-rt's __powidf2, but Windows/MSVC rounds differently.

Our square-and-multiply is pure IEEE-754 f64 multiplication and is
therefore deterministic across every platform, which is the property that
actually matters for mesh nodes to agree on bloom FPR and backoff timing
regardless of OS. Replace the std bit-equality assertion with golden-bit
pins on our own output (locking that cross-platform determinism) plus a
bounded-drift sanity sweep against std::powi. The powi function itself is
unchanged.
 2026-07-09 01:19:49 +00:00
Johnathan Corgan e3e03f6a5d proto: rename the mesh discovery subsystem module from discovery to lookup 
Rename src/proto/discovery to src/proto/lookup and bring the module's naming
onto the lookup stem, matching its concept: a mesh lookup of a node's
coordinates from its pubkey, sent into the mesh as a bloom-filter-guided
multicast request that returns a unicast response with the coordinates.

- the module directory and its declaration (proto::discovery -> proto::lookup)
- exported types: DiscoveryAction -> LookupAction, DiscoveryBackoff ->
  LookupBackoff, DiscoveryForwardRateLimiter -> LookupForwardRateLimiter,
  MAX_RECENT_DISCOVERY_REQUESTS -> MAX_RECENT_LOOKUP_REQUESTS, and the Discovery
  state struct -> Lookup
- internal terminology: doc comments, the "overlay-lookup" phrasing, the
  empty_discovery/suppressing_discovery test helpers, and the disc
  parameter/variable all take the lookup names
- the already-lookup-named wire types (LookupRequest/LookupResponse) are unchanged

Behavior-neutral: no wire bytes or decision logic change. Two references are
intentionally kept as "discovery": the still-named node::handlers::discovery
shell module and the node.discovery.* config keys, which belong to the broader
disambiguation of the shell, config, and metric surfaces still to come.
 2026-07-08 21:54:24 +00:00
Johnathan Corgan 2b009196b5 proto: no_std hygiene for the fmt/alloc imports and the filter math 
Bring the proto tree closer to a std+alloc-only posture, behavior unchanged on
every decision path:

- Sweep the remaining std::fmt and std::collections imports over to core::fmt
  and alloc::collections across the wire codecs and their tests. Subsystem files
  that shadow the core name with a child core module use the leading-colon
  ::core::fmt form. Imports only.

- Replace the two f64::powi calls (bloom false-positive-rate, FMP backoff timer)
  with a shared core-only square-and-multiply helper in proto/math, bit-identical
  to std::powi (a guard test pins this against every exponent the codecs reach),
  and route the diagnostic estimated-count natural log through libm::log. The FPR
  reject decision and the backoff timer are bit-for-bit unchanged; only the
  debug-only count estimate may differ by at most one ULP.
 2026-07-08 19:00:25 +00:00
Johnathan Corgan 5d13090d8f proto: add a bounds-checked byte reader/writer and adopt it across the wire codecs 
Introduce a shared proto/codec module with a cursor Reader (short reads fail with
MessageTooShort { expected: position + needed, got: total }) and an append Writer,
and adopt them across the seven subsystem wire codecs, replacing the repetitive
manual slicing, try_into, and from_le_bytes extraction. Each existing length check
maps to the reader (an up-front minimum becomes require at position zero, so the
per-field expected values, including the tree-announce expected 99, are reproduced
exactly); the bloom exact-length check stays explicit since it also rejects
over-long payloads. Encoded bytes and decode decisions are unchanged.
 2026-07-08 19:00:25 +00:00
Johnathan Corgan 6538731176 proto: replace the string Malformed error with typed variants and drop thiserror 
Replace Malformed(String) with a no_std-clean set: Malformed(&static str)
for the static decode diagnostics, BadSizeClass { got, max } for the two
bloom size-class checks, and typed sources BadCoord(CoordError) /
BadBloom(BloomError) for the coordinate and bloom construction failures. Add a
dedicated CoordError so proto/coord no longer depends upward on stp TreeError,
resolving the temporary inversion from the coordinate relocation. Drop the
thiserror derive from the four proto error types (Error, TreeError, BloomError,
CoordError) in favor of hand-rolled core::fmt::Display and core::error::Error
impls. thiserror remains in use elsewhere in the crate. Wire decode decisions
are unchanged; only the error type and its diagnostic text change.
 2026-07-08 19:00:25 +00:00
Johnathan Corgan 9697026c81 proto: share a per-address rate limiter and backoff helper across subsystems 
Hoist the two line-for-line-identical per-destination minimum-interval limiters
(discovery forward, routing error) into a shared PerAddrRateLimiter, and fold
the duplicated exponential-backoff math (discovery originator, FMP retry) into a
shared backoff_ms helper, both in a new proto/rate_limit module. The subsystem
limiters become thin delegating newtypes so should_forward/should_send and the
backoff call conventions are preserved. Behavior is byte-identical.
 2026-07-08 19:00:25 +00:00
Johnathan Corgan a2400d823f proto/stp: extract ParentDeclaration and relocate the coordinate type to a shared proto/coord module 
Two behavior-neutral relocations, wire bytes unchanged:

- Move the ParentDeclaration type and its impls out of state.rs into a dedicated
  declaration.rs, re-exported at the same path, and relocate the inline wire.rs
  test module into tests/wire.rs alongside the other stp unit tests (reusing the
  shared test helpers). wire.rs drops to non-test code only.

- Move TreeCoordinate, CoordEntry, and the coordinate wire codec out of
  proto/stp into a shared proto/coord module (a peer of proto/link), re-exported
  from proto::stp so every existing import path keeps resolving unchanged.
  Coordinate is a shared addressing primitive with many non-stp consumers, so it
  no longer belongs under the spanning-tree subsystem. The codec carries a
  documented temporary dependency on stp::TreeError until a dedicated CoordError
  is introduced.
 2026-07-08 19:00:25 +00:00
Johnathan Corgan dc9334e725 proto/stp: hoist the parent flap/hold-down veto to the shell for a clock-free classify core 
evaluate_parent no longer reads the injected clock: it returns a ParentEval of
Mandatory, Discretionary, or None, and the flap/hold-down veto is applied at the
edge via a new TreeState::is_switch_suppressed(now_ms), gating only the
discretionary arm. classify_announce/classify_periodic take the pre-computed
switch_suppressed bool instead of now_ms, so the whole classify ladder is
clock-free; the shell callers (tree announce/periodic re-eval, MMP first-RTT
re-eval, handle_parent_lost) compute the veto verdict. The no-coords parent
case stays discretionary (veto-gated) exactly as before. Behavior unchanged;
the veto tests now assert the moved responsibility.
 2026-07-08 19:00:24 +00:00
Johnathan Corgan 309a91d293 proto/mmp: split state into per-role modules and dissolve the vestigial mmp shell 
Reorganize the MMP subsystem, behavior unchanged throughout:

- Restore the pre-migration role split: move SenderState into sender.rs,
  ReceiverState (with its GapTracker helper) into receiver.rs, MmpMetrics and
  RrLog into metrics.rs, and PathMtuState into path_mtu.rs, leaving the Mmp
  aggregate plus the peer/session state in state.rs (1339 to 196 lines). Home
  the module constants in a new limits.rs and re-export them at the same paths.
  Pure code-motion with module rewiring; visibility unchanged.

- Dissolve the 97-line src/mmp shell, which held only MmpConfig and the
  monotonic mono_ms() clock: move MmpConfig into config/node.rs (re-exported as
  crate::config::MmpConfig), move mono_ms() into a new top-level src/time.rs
  shell time seam, repoint all callers, and delete src/mmp/. Also correct stale
  src/mmp/ doc-comment path labels to proto/mmp/ where they name the protocol
  primitives.
 2026-07-08 19:00:24 +00:00
Johnathan Corgan c1ddbf053c proto/discovery: home the recent-request cap and inject request_id from the shell 
Two behavior-neutral discovery cleanups:

- Move MAX_RECENT_DISCOVERY_REQUESTS out of the node discovery handler into the
  discovery subsystem limits module and re-export it, keeping the two use sites
  unchanged. Same value and semantics; only its home moves.

- Remove the ambient rand draw from LookupRequest by deleting generate() and
  having the shell draw the random request_id and pass it into the existing
  new() constructor. Same per-request u64 draw, now at the shell, leaving the
  discovery codec free of ambient RNG reads.
 2026-07-08 19:00:24 +00:00
Johnathan Corgan b53db662c3 proto/bloom: add BloomFilter::fpr() and use it for the false-positive-rate 
The false-positive-rate primitive (fill ratio raised to the hash count) was
inlined at three sites. Hoist it into a BloomFilter::fpr() method and call it
from all three; the threshold/policy logic stays at the call sites. Behavior
and result bytes are unchanged.
 2026-07-08 19:00:24 +00:00
Johnathan Corgan 4ad5940114 proto/fsp: extract FSP session protocol into sans-IO layout, retire src/protocol 
Migrate the FSP end-to-end session subsystem into src/proto/fsp/ following the
established sans-IO shape, and retire the src/protocol grab-bag now that FSP was
its last occupant.

Relocate the FSP session wire (node/session_wire.rs plus the FSP message types
from protocol/session.rs) into proto/fsp/wire.rs. Hoist the pure decision logic
into proto/fsp/core.rs over plain-data SessionSnapshots returning an ordered
FspAction list the shell drives: session-rekey policy, msg3-resend
classification, post-decrypt epoch reaction, setup/dual-init tie-break,
coords/path-MTU emit-policy, bounded pending-queue, and IPv6 ECN. The
crypto-owning SessionEntry stays shell-side in node/session.rs (matching the FMP
ActivePeer pattern); proto/fsp is wire + core + limits only, with no proto->noise
dependency and no crypto.

Move the coords helpers to proto/stp/ (they serialize TreeCoordinate), and split
SessionMessageType: the encrypted-inner 0x10-0x1F variants stay in proto/fsp/wire.rs
while the 0x20-0x2F routing signals become a new RoutingSignalType in
proto/routing/wire.rs. Migrate the session-MMP shell adapter, which continues to
drive proto/mmp/.

Retire src/protocol: LinkMessageType and SessionDatagram move to a new shared
proto/link.rs, ProtocolError becomes proto::Error (relocated verbatim), the
deprecated MessageType alias and the unimported PROTOCOL_VERSION are dropped, and
src/protocol/ is deleted along with its lib.rs module declaration.

Behavior-neutral: wire bytes unchanged, oracle tests pass unedited except
mod-path relocation; adds rekey/epoch characterization tests and pure
poll/emit-policy core tests.
 2026-07-08 04:57:00 +00:00
Johnathan Corgan 4ed674ea8b proto/bloom: relocate bloom filter into sans-IO layout 
Migrate the v1 bloom filter subsystem into src/proto/bloom/, matching the
discovery/routing/fmp/mmp/stp reference layout and completing the proto/
relocation series for the data/wire subsystems.

- wire.rs: the FilterAnnounce (0x20) codec, moved from protocol/filter.rs
- core.rs: the pure BloomFilter algorithm (hash/insert/contains/merge/
  as_bytes/from_bytes/estimated_count), moved from bloom/filter.rs
- state.rs: BloomState (per-peer inbound store, compute_outgoing_filter,
  the injected-clock send debounce), moved from bloom/state.rs
- limits.rs: the v1 sizing constants
- mod.rs: module wiring + the BloomError enum
- tests/: the unit suite split by target (core/state/wire), no inline tests

no_std+alloc hygiene: core::fmt over std::fmt, the tracing dependency
dropped from the pure filter, and std collections replaced with
BTreeMap/BTreeSet (NodeAddr: Ord) for deterministic iteration. The pure
filter combination stays a BloomState method; the two irreducible shell
gathers (peer_inbound_filters, build_filter_announce) remain in the async
shell. Wire bytes and observable behavior are unchanged; full local CI
green (36/36) including the bloom-storm chaos gate.
 2026-07-07 23:10:12 +00:00
Johnathan Corgan a67801099d proto/stp: sans-IO spanning-tree state machine 
Migrate the full non-async spanning-tree surface into proto/stp/, mirroring the
discovery/routing/fmp/mmp conversions. The classification ladder (parent-switch /
self-root / loop-drop / ancestry-update / periodic-rebroadcast / parent-lost) moves
out of the async node handlers into a pure Stp classify layer returning a
TreeDecision the shell drives, with effect ordering and per-arm invalidation
preserved verbatim. src/tree/ relocates wholesale: TreeState + ParentDeclaration data
+ coordinates into proto/stp/{state,coordinate}, the flap-dampening / hold-down
cluster into a FlapDampener in limits.rs, and the wire codec into wire.rs. The clock
is injected as u64 (wall-clock secs for the escaping declaration timestamp, monotonic
ms for the dampening timers via mmp::mono_ms); declaration crypto is field-partitioned
so sign/verify/hash run in the shell while the in-core modules carry data +
signing_bytes only. Peer maps/sets move to BTree; core/state/coordinate/limits are
core+alloc clean, with wire.rs the one std-tethered file. Behavior-neutral:
characterization tests added for the handler decision arms; convergence suite and
ci-local (36/36) green.
 2026-07-07 17:07:33 +00:00
Johnathan Corgan 50a595a0ed proto/mmp: sans-IO metrics-reporting state machine 2026-07-07 09:24:17 +00:00
Johnathan Corgan 4802792e38 proto/fmp: sans-IO connection-lifecycle state machine 2026-07-07 06:20:02 +00:00
Johnathan Corgan 9ea57b483a proto/routing: sans-IO transit + hop-selection state machine 2026-07-07 04:12:05 +00:00
Johnathan Corgan e03b206f62 proto/discovery: sans-IO state-machine migration + no_std reductions 
Migrate the FMP discovery decision logic out of the async handlers into
synchronous, runtime-agnostic sans-IO state machines owned by the protocol
structs, with I/O pushed to the edges. Pulls the full decision surface into a
pure core (backoff, rate-limit, planners, response routing), consolidates the
tests into a per-module tree with a shared crate testutil, and injects a u64
wall-clock so the core is free of Instant and std time.

Also brings the module toward no_std+alloc: the four discovery maps use
alloc::collections::BTreeMap (HashMap's RandomState is std-only), Arc is spelled
alloc::sync::Arc, the backoff-reset log lives in the shell (the core returns the
cleared count so observability stays out of the pure core), and the crate root
names alloc directly. The one remaining tether is ProtocolError's
std::error::Error coupling in the wire codec.

First subsystem of the broader sans-IO refactor; establishes the extraction
patterns and conventions carried forward to the remaining protocols.
 2026-07-05 21:59:21 +00:00
353 changed files with 48057 additions and 21212 deletions
Expand all files Collapse all files
.config/nextest.toml
+17
View File
@@ -5,6 +5,23 @@ junit = { path = "junit.xml" }
# occasional msg1 under burst load even with the per-edge repair loop.
# Allow a retry rather than failing the whole CI run on a single
# dropped packet.
#
# Deliberately scoped to [profile.ci] and NOT applied locally, which makes
# the two gates disagree: the hosted runner retries a flaky test twice, the
# local sweep fails on the first failure. The asymmetry is intended and this
# is the record of why, since an undocumented one is indistinguishable from
# an oversight.
#
# It is here for shared-runner packet loss, a property of the hosted
# environment and not of the code. Applying it locally would suppress a real
# local flake, and a failure that only reproduces under load is a robustness
# bug to fix rather than to retry past. Keeping the local sweep strict is
# what makes it the sharper of the two gates.
#
# The cost, stated rather than hidden: a test that fails once and passes on
# retry is reported green here with no separate signal, so a genuine
# intermittent failure can be absorbed. If that starts mattering, the fix is
# to surface retried-but-passed tests, not to drop the retries.
retries = 2
[test-groups]
.github/workflows/ci.yml
+88 -196
View File
@@ -38,12 +38,12 @@ env:
# unreliable on GitHub-hosted runners.
# tor-directory — same; live Tor dependency.
#
# Granularity-only differences (same coverage, different matrix shape —
# NOT a divergence):
# deb-install — split here into per-distro legs (debian12/debian13/
# ubuntu22/ubuntu24/ubuntu26) for parallelism; local runs the
# same distro set in one suite.
# dns-resolver — single leg here; runs all scenarios (same as local).
# The two runners express the same work in different matrix shapes, and the
# parity guard compares through that shape rather than around it: chaos legs
# are compared per scenario (and per flag) via their `scenario:` field,
# deb-install legs per distro. The one leg still compared at leg granularity
# is dns-resolver — a single leg here, running all of its scenarios
# internally, exactly as the local suite does.
# ─────────────────────────────────────────────────────────────────────────────
# ─────────────────────────────────────────────────────────────────────────────
@@ -52,6 +52,29 @@ env:
# Builds on Linux x86_64, Linux aarch64, and macOS.
# ─────────────────────────────────────────────────────────────────────────────
jobs:
ci-parity:
name: CI parity
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v6
- name: Install Python deps
run: pip3 install --quiet pyyaml
- name: Check local and GitHub runners cover the same work
run: bash testing/check-ci-parity.sh
- name: Check test log matchers against the strings src/ emits
run: python3 testing/check-log-strings.py
- name: Check no tested function's exit status is a log call's
run: python3 testing/check-trailing-log.py
- name: Check nothing resolves the shared mutable test image
run: bash testing/check-image-scoping.sh
# Hermetic: synthetic ping functions, no containers, ~45s. Lives beside
# the other two so both runners gate on it identically — putting it in
# only one would create exactly the drift check-ci-parity.sh exists to
# catch, and it is invisible to that checker either way since it is not
# a matrix suite.
- name: Run convergence-gate unit tests
run: bash testing/lib/wait-converge-test.sh
fmt:
name: Format check
runs-on: ubuntu-latest
@@ -87,6 +110,59 @@ jobs:
restore-keys: |
${{ runner.os }}-cargo-
- run: cargo clippy --all-targets --all-features -- -D warnings
# An optional feature means two source trees, and --all-features lints
# only one of them. The default build is what ships, so lint it
# explicitly: without this stage, code that compiles only with
# `profiling` enabled would pass CI while breaking every release build.
# Mirrored in testing/ci-local.sh — check-ci-parity.sh compares
# integration suites only and will not catch a stage added to one runner
# and not the other.
- name: Clippy (default features)
run: cargo clippy --all-targets -- -D warnings
- name: Build with the tick-body profiler enabled
run: cargo build --workspace --features profiling
# ───────────────────────────────────────────────────────────────────────────
# Android cross-check
#
# FIPS runs on Android as an embedded library — the host app owns the TUN
# (an Android VpnService), so there are no daemon binaries to package, unlike
# the desktop targets. This job only cross-compiles the library for the
# android target to guard the android-only cfg paths (and the `not(android)`
# exclusions) from silently bit-rotting; nothing else in CI compiles them.
# cargo-ndk wires the NDK toolchain, which is required even for a check
# because `ring` compiles C at build time.
# ───────────────────────────────────────────────────────────────────────────
android-check:
name: Android cross-check (aarch64)
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v6
- name: Install Rust toolchain (+ Android target)
uses: actions-rust-lang/setup-rust-toolchain@v1
with:
target: aarch64-linux-android
components: clippy
cache: false
rustflags: ''
- name: Cache Cargo registry + build
uses: actions/cache@v5
with:
path: |
~/.cargo/registry
~/.cargo/git
target
key: ${{ runner.os }}-cargo-android-${{ hashFiles('**/Cargo.lock') }}
restore-keys: |
${{ runner.os }}-cargo-
- name: Install cargo-ndk
uses: taiki-e/install-action@v2
with:
tool: cargo-ndk
- name: Clippy the library for Android
run: |
export ANDROID_NDK_HOME="${ANDROID_NDK_HOME:-$ANDROID_NDK_LATEST_HOME}"
cargo ndk -t arm64-v8a clippy --lib -- -D warnings
build:
name: Build (${{ matrix.os }})
@@ -228,6 +304,12 @@ jobs:
check_name: Unit Tests Summary
fail_on_failure: false
# The `profiling` feature adds a module, a recorder and a writer thread
# that the default-feature run above never compiles, so its own tests do
# not execute there. Mirrored in testing/ci-local.sh.
- name: Run library tests with the tick-body profiler enabled
run: cargo test --lib --features profiling
# ─────────────────────────────────────────────────────────────────────────────
# Job 2b – Unit tests (macOS)
# ─────────────────────────────────────────────────────────────────────────────
@@ -351,22 +433,6 @@ jobs:
- suite: static-chain
type: static
topology: chain
# ── Rekey integration test ──────────────────────────────────────────
- suite: rekey
type: rekey
topology: rekey
- suite: rekey-accept-off
type: rekey-accept-off
topology: rekey-accept-off
- suite: rekey-outbound-only
type: rekey-outbound-only
topology: rekey-outbound-only
# ── Inbound max_peers admission-cap test ───────────────────────
- suite: admission-cap
type: admission-cap
topology: mesh
- suite: acl-allowlist
type: acl-allowlist
# ── Firewall baseline (fips0 nftables default-deny) ────────────
- suite: firewall
type: firewall
@@ -375,9 +441,6 @@ jobs:
type: gateway
topology: gateway
# ── Chaos / stochastic scenarios ───────────────────────────────────
- suite: chaos-smoke-10
type: chaos
scenario: smoke-10
- suite: churn-mixed-10
type: chaos
scenario: churn-mixed
@@ -391,30 +454,9 @@ jobs:
- suite: tcp-mesh
type: chaos
scenario: tcp-mesh
- suite: bottleneck-parent
type: chaos
scenario: bottleneck-parent
- suite: cost-avoidance
type: chaos
scenario: cost-avoidance
- suite: cost-reeval
type: chaos
scenario: cost-reeval
- suite: cost-stability
type: chaos
scenario: cost-stability
- suite: depth-vs-cost
type: chaos
scenario: depth-vs-cost
- suite: mixed-technology
type: chaos
scenario: mixed-technology
- suite: congestion-stress
type: chaos
scenario: congestion-stress
- suite: bloom-storm
type: chaos
scenario: bloom-storm
# ── Sidecar deployment ──────────────────────────────────────────
- suite: sidecar
type: sidecar
@@ -527,120 +569,6 @@ jobs:
docker compose -f testing/static/docker-compose.yml \
--profile ${{ matrix.topology }} down --volumes --remove-orphans
# ── Rekey integration test ──────────────────────────────────────────────
- name: Generate and inject configs (rekey)
if: matrix.type == 'rekey'
run: |
bash testing/static/scripts/generate-configs.sh rekey
bash testing/static/scripts/rekey-test.sh inject-config
- name: Start containers (rekey)
if: matrix.type == 'rekey'
run: |
docker compose -f testing/static/docker-compose.yml \
--profile rekey up -d
- name: Run rekey test
if: matrix.type == 'rekey'
run: bash testing/static/scripts/rekey-test.sh
- name: Collect logs on failure (rekey)
if: matrix.type == 'rekey' && failure()
run: |
docker compose -f testing/static/docker-compose.yml \
--profile rekey logs --no-color
- name: Stop containers (rekey)
if: matrix.type == 'rekey' && always()
run: |
docker compose -f testing/static/docker-compose.yml \
--profile rekey down --volumes --remove-orphans
# ── Rekey + accept_connections=false variant ──────────────────────────
- name: Generate and inject configs (rekey-accept-off)
if: matrix.type == 'rekey-accept-off'
env:
REKEY_TOPOLOGY: rekey-accept-off
REKEY_ACCEPT_OFF_NODES: b
run: |
bash testing/static/scripts/generate-configs.sh rekey-accept-off
bash testing/static/scripts/rekey-test.sh inject-config
- name: Start containers (rekey-accept-off)
if: matrix.type == 'rekey-accept-off'
run: |
docker compose -f testing/static/docker-compose.yml \
--profile rekey-accept-off up -d
- name: Run rekey test (accept-off variant)
if: matrix.type == 'rekey-accept-off'
env:
REKEY_TOPOLOGY: rekey-accept-off
REKEY_ACCEPT_OFF_NODES: b
run: bash testing/static/scripts/rekey-test.sh
- name: Collect logs on failure (rekey-accept-off)
if: matrix.type == 'rekey-accept-off' && failure()
run: |
docker compose -f testing/static/docker-compose.yml \
--profile rekey-accept-off logs --no-color | tail -300
- name: Stop containers (rekey-accept-off)
if: matrix.type == 'rekey-accept-off' && always()
run: |
docker compose -f testing/static/docker-compose.yml \
--profile rekey-accept-off down --volumes --remove-orphans
# ── Rekey + udp.outbound_only=true variant ─────────────────────────────
- name: Generate and inject configs (rekey-outbound-only)
if: matrix.type == 'rekey-outbound-only'
env:
REKEY_TOPOLOGY: rekey-outbound-only
REKEY_OUTBOUND_ONLY_NODES: b
run: |
bash testing/static/scripts/generate-configs.sh rekey-outbound-only
bash testing/static/scripts/rekey-test.sh inject-config
- name: Start containers (rekey-outbound-only)
if: matrix.type == 'rekey-outbound-only'
run: |
docker compose -f testing/static/docker-compose.yml \
--profile rekey-outbound-only up -d
- name: Run rekey test (outbound-only variant)
if: matrix.type == 'rekey-outbound-only'
env:
REKEY_TOPOLOGY: rekey-outbound-only
REKEY_OUTBOUND_ONLY_NODES: b
run: bash testing/static/scripts/rekey-test.sh
- name: Collect logs on failure (rekey-outbound-only)
if: matrix.type == 'rekey-outbound-only' && failure()
run: |
docker compose -f testing/static/docker-compose.yml \
--profile rekey-outbound-only logs --no-color | tail -300
- name: Stop containers (rekey-outbound-only)
if: matrix.type == 'rekey-outbound-only' && always()
run: |
docker compose -f testing/static/docker-compose.yml \
--profile rekey-outbound-only down --volumes --remove-orphans
# ── ACL allowlist integration test ─────────────────────────────────────
- name: Run ACL allowlist integration test
if: matrix.type == 'acl-allowlist'
run: bash testing/acl-allowlist/test.sh --skip-build --keep-up
- name: Collect logs on failure (acl-allowlist)
if: matrix.type == 'acl-allowlist' && failure()
run: |
docker compose -f testing/acl-allowlist/docker-compose.yml logs --no-color
- name: Stop containers (acl-allowlist)
if: matrix.type == 'acl-allowlist' && always()
run: |
docker compose -f testing/acl-allowlist/docker-compose.yml down --volumes --remove-orphans
# ── Firewall baseline integration test ─────────────────────────────────
- name: Run firewall baseline integration test
if: matrix.type == 'firewall'
@@ -771,42 +699,6 @@ jobs:
docker compose -f testing/static/docker-compose.yml \
--profile gateway down --volumes --remove-orphans
# ── Inbound max_peers admission-cap integration test ────────────────
# Lowers node.max_peers on one mesh node and asserts the inbound cap
# holds under sustained retry pressure: denied peers keep retrying but
# are never promoted to an active session. The admission-cap-test.sh
# assertions are tailored per link-layer handshake variant; the leg
# itself is uniform. Static-style harness on the shared mesh profile.
- name: Generate configs (admission-cap)
if: matrix.type == 'admission-cap'
run: bash testing/static/scripts/generate-configs.sh mesh
- name: Inject admission-cap config (admission-cap)
if: matrix.type == 'admission-cap'
run: bash testing/static/scripts/admission-cap-test.sh inject-config
- name: Start containers (admission-cap)
if: matrix.type == 'admission-cap'
run: |
docker compose -f testing/static/docker-compose.yml \
--profile mesh up -d
- name: Run admission-cap test
if: matrix.type == 'admission-cap'
run: bash testing/static/scripts/admission-cap-test.sh
- name: Collect logs on failure (admission-cap)
if: matrix.type == 'admission-cap' && failure()
run: |
docker compose -f testing/static/docker-compose.yml \
--profile mesh logs --no-color | tail -300
- name: Stop containers (admission-cap)
if: matrix.type == 'admission-cap' && always()
run: |
docker compose -f testing/static/docker-compose.yml \
--profile mesh down --volumes --remove-orphans
# ── Real-deb install integration ────────────────────────────────────
# The deb-install harness builds its own .deb from source in a
# cargo-deb builder image; the pre-built Linux binary from the
.github/workflows/package-openwrt.yml
+5
View File
@@ -285,6 +285,8 @@ jobs:
"$FILES_DIR/etc/fips/firewall.sh"
"$FILES_DIR/etc/hotplug.d/net/99-fips"
"$FILES_DIR/etc/uci-defaults/90-fips-setup"
"$FILES_DIR/usr/bin/fips-mesh-setup"
"$FILES_DIR/usr/bin/fips-ap-setup"
)
fail=0
for f in "${TARGETS[@]}"; do
@@ -404,6 +406,8 @@ jobs:
./usr/bin/fipsctl
./usr/bin/fipstop
./usr/bin/fips-gateway
./usr/bin/fips-mesh-setup
./usr/bin/fips-ap-setup
./etc/init.d/fips
./etc/init.d/fips-gateway
./etc/fips/fips.yaml
@@ -717,6 +721,7 @@ jobs:
for path in \
usr/bin/fips usr/bin/fipsctl usr/bin/fipstop usr/bin/fips-gateway \
usr/bin/fips-mesh-setup usr/bin/fips-ap-setup \
etc/init.d/fips etc/init.d/fips-gateway \
etc/fips/fips.yaml etc/fips/firewall.sh etc/dnsmasq.d/fips.conf \
etc/sysctl.d/fips-gateway.conf etc/sysctl.d/fips-bridge.conf \
.gitignore
+5
View File
@@ -33,6 +33,11 @@ __pycache__/
*.egg-info/
*.egg
# Per-run build contexts created by testing/ci-local.sh. Its teardown normally
# removes them, but the CI worker's SIGKILL runs no trap, so one can survive a
# preempted run; ci-cleanup.sh sweeps the survivors.
/testing/docker-*/
# Runtime artifacts from running fips in-tree during local testing.
# Root-anchored so legitimately-tracked fips.yaml under packaging/ and
# examples/ stays included.
CHANGELOG.md
+143
View File
@@ -9,10 +9,153 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
### Added
- An optional tick-body profiler behind the new `profiling` Cargo feature,
**off by default**. When enabled, `fipsctl profile tick on [--dir PATH]` /
`off` / `status` starts and stops a capture at runtime with no restart. Each
capture writes one tab-separated file (default `/var/log/fips`, capped at
32 MB) carrying, per ten-second interval, the exact count, max and total for
every step of the rx-loop tick arm, the whole-tick span, and gauges for ticks,
peer count, the gap between successive tick-arm entries and the resulting
arm-starvation delay. With the feature off the instrumentation macro is a pure
pass-through, so a default build contains no timing code on the tick path.
`LogsDirectory=fips` was added to the packaged systemd units so the capture
directory is created and cleaned up declaratively.
- `packaging/debian/build-deb.sh --features <list>` builds the `.deb` with a
Cargo feature list, which is how an instrumented package is produced for a
measurement run. The auto-derived dev Version gains a matching `+<features>`
marker, so a feature build and a default build of the same commit are no
longer indistinguishable: without it the two carry byte-identical versions,
an install of one over the other is an apt no-op, and the running node offers
no way to tell which one it has. The marker sorts above the unmarked build, so
installing a feature build is an upgrade and reverting to the default build is
a downgrade — revert with `dpkg -i` rather than `apt install`. `--features` is
refused together with `--no-build`, which would stamp the marker onto binaries
the features never reached.
### Changed
- The Ethernet transport's per-interface `discovery` flag was renamed to
`listen` (`transports.ethernet.*`) to match the symmetric `announce`
(transmit) / `listen` (receive) neighbor-beacon vocabulary. The old
`discovery:` key is still accepted via a serde alias, so deployed configs
continue to load unchanged; `Config::to_yaml()` re-emits it under the
canonical `listen:` name. Update your `fips.yaml` to `listen:`.
- The mesh-lookup control-metrics family is now emitted under the key
`lookup` in `fipsctl stats metrics` and `show routing`. The former key
`discovery` is still emitted as a deprecated alias carrying identical
counters; update dashboards and alerts to read `lookup`.
- The overloaded `node.discovery.*` config table was split into
`node.lookup.*` (mesh-lookup scalars: `ttl`, `attempt_timeouts_secs`,
`recent_expiry_secs`, `backoff_base_secs`, `backoff_max_secs`,
`forward_min_interval_secs`) and `node.rendezvous.*` (peer rendezvous:
`nostr.*`, `lan.*`). A deployed `node.discovery:` block still loads and is
folded into the new tables with a one-time deprecation warning; migrate your
`fips.yaml` to the new keys.
- `SessionDatagram::decrement_ttl` and `SessionDatagram::can_forward` now match
the forwarder's IP hop-limit semantics: `decrement_ttl` decrements first and
reports false when the result is zero, and `can_forward` is true only at a
TTL of 2 or more.
### Deprecated
- The `discovery` metric-family key (control-socket JSON). It is dual-emitted
alongside the new `lookup` key during a migration window and will be removed.
Migrate dashboards/alerts from `discovery.*` to `lookup.*`.
- The `node.discovery.*` config table. Its keys were split into `node.lookup.*`
(mesh-lookup) and `node.rendezvous.*` (peer rendezvous). A legacy
`node.discovery:` block still applies for now with a deprecation warning and
will be removed; migrate to `node.lookup.*` / `node.rendezvous.*`.
- The Ethernet `transports.ethernet.discovery` flag, renamed to
`transports.ethernet.listen`. The old key is still accepted via a serde
alias and will be removed at the v2 cutover; migrate to `listen`.
### Fixed
- Nostr NAT traversal no longer breaks after the host suspends. The traversal
clock cached a Unix timestamp once at startup and advanced it with a
monotonic `Instant`, which does not tick while a machine is asleep, so after
a suspend the daemon's idea of the time trailed real time by the suspend
duration for the rest of the process lifetime. Every NIP-40 expiration it
computed was therefore published already in the past: relays dropped the
offers as expired, the initiator logged a signal timeout waiting for an
answer, and traversal stayed broken until the daemon was restarted. The
clock now reads the wall clock on every call. This is not macOS-specific,
though a laptop that sleeps is where it is easiest to hit; any host that
suspends or hibernates was affected. Reported in
[#128](https://github.com/jmcorgan/fips/issues/128).
- `SessionDatagram` hop-limit handling now follows IP semantics. Delivery to
the addressed node is no longer TTL-gated, and a forwarder decrements before
deciding rather than after, so a datagram that would leave with a TTL of zero
is dropped instead of transmitted. Previously the TTL check ran ahead of the
local-delivery test, so a datagram addressed to this node that arrived with
TTL 0 was dropped, and a forwarder receiving a transit datagram at TTL 1
transmitted it at TTL 0 for the next hop to discard, wasting one transmission
per expiring datagram. The reachable radius is unchanged, because the two
behaviors compensated exactly: a path of `h` links still delivers for any
source TTL of `h` or more. During a rolling upgrade, an unupgraded forwarder
feeding an upgraded destination delivers one hop further than either version
does on its own; no version mix delivers less far. The `TtlExhausted` reject
counter now charges at the node that makes the decision rather than at the
hop after it.
## [0.4.1] - 2026-07-19
### Changed
- `node.bloom.max_inbound_fpr` default raised from `0.10` to `0.20`. The
cap rejects inbound `FilterAnnounce` whose FPR (`fill^k`) exceeds it. On
the fixed 1 KB / k=5 filter, `0.10` corresponds to fill 0.631 (~1,630
reachable entries), and the busiest nodes' aggregates had again begun to
reach it as the mesh grew. `0.20` (fill 0.7248, ~2,114 entries) restores
headroom without materially weakening the antipoison gate: a saturated or
poisoned filter is ~100% FPR and still rejected. This is the second raise
of this cap in two releases; the fixed 1 KB filter is the underlying
constraint, and the structural remedy is the v2 filter work rather than a
further raise. A node running this default accepts announcements that a
v0.4.0 node drops, so during a rolling upgrade the two versions can
disagree about mesh size.
- Bloom filter probing computes its SHA-256 digest once per operation
rather than once per hash function. All k indices were already derived
from a single digest, but the digest was recomputed inside the
per-function loop, so every insert and membership test hashed the same
bytes `hash_count` times (5x at the default). Output is bit-for-bit
identical; this is the hottest path in packet forwarding and mesh-size
estimation.
- Identity operations reuse one shared `secp256k1` context instead of
constructing a fresh one at every sign, verify, and key-derive site.
Each construction allocated a context and ran randomization and blinding
table setup. Behavior is unchanged: the same API calls are made, only the
context lifetime differs, and the shared context still performs the
standard construction-time blinding.
### Fixed
- Spanning tree: the coordinate cache is now invalidated when the parent
link is lost through peer removal. That path reparents or self-roots the
node but omitted the invalidation every other position-change path
performs, so cached entries for downstream destinations kept the node's
now-stale coordinate prefix. Because routing access refreshes an entry's
TTL, an actively routed stale entry never self-expired and was corrected
only by a fresh insert.
- Discovery: applying a `LookupResponse` now keeps the tighter of the
cached and received `path_mtu` rather than overwriting unconditionally.
A looser estimate arriving in a later response could clobber a tighter
value already learned from a reactive `MtuExceeded` or
`PathMtuNotification`, loosening a clamp that had been correctly
tightened.
### Removed
- The `parent_switched` spanning-tree metric counter. It was incremented on
the line immediately before `parent_switches` at every site and never
independently, so the two were always identical. `parent_switches`
remains as the sole counter. Consumers reading `parent_switched` from the
control socket or `fipstop` should use `parent_switches`.
## [0.4.0] - 2026-06-27
### Added
Cargo.lock
Generated
+1
View File
@@ -1087,6 +1087,7 @@ dependencies = [
"hex",
"hkdf",
"libc",
"libm",
"mdns-sd",
"nostr",
"nostr-sdk",
Cargo.toml
+14
View File
@@ -11,12 +11,21 @@ readme = "README.md"
keywords = ["mesh", "p2p", "decentralized", "overlay-network", "nostr"]
categories = ["network-programming", "command-line-utilities", "cryptography"]
[features]
default = []
# Tick-body profiler (`src/instr`). Off by default: enabling it edits 26 call
# sites in the rx loop's hot tick arm, and only a compile-time gate makes the
# default build's neutrality a property of the generated code rather than of a
# runtime check. Build with `--features profiling` for a measurement run.
profiling = []
[dependencies]
ratatui = "0.30"
secp256k1 = { version = "0.30", features = ["rand", "global-context"] }
sha2 = "0.10"
hkdf = "0.12"
ring = "0.17"
libm = "0.2"
rand = "0.10.1"
crossbeam-channel = "0.5"
thiserror = "2.0"
@@ -108,3 +117,8 @@ path = "src/bin/fips-gateway.rs"
[[bin]]
name = "fipstop"
path = "src/bin/fipstop/main.rs"
[[bench]]
name = "routing_next_hop"
path = "benches/routing_next_hop.rs"
harness = false
README.md
+13 -11
View File
@@ -112,17 +112,19 @@ tutorial progression starting at
cargo build --release
```
Requires Rust 1.94.1+ (edition 2024). Linux, macOS, and Windows are
supported; transport availability varies by platform.
Requires Rust 1.94.1+ (edition 2024). Linux, macOS, and Windows run as
standalone daemons; Android is supported as an embedded library (the host
app owns the TUN, e.g. a `VpnService`). Transport availability varies by
platform.
| Transport | Linux | macOS | Windows | OpenWrt |
|-----------|:-----:|:-----:|:-------:|:-------:|
| UDP | ✅ | ✅ | ✅ | ✅ |
| TCP | ✅ | ✅ | ✅ | ✅ |
| Ethernet | ✅ | ✅ | ❌ | ✅ |
| Tor | ✅ | ✅ | ✅ | ✅ |
| Nym | ✅ | ✅ | ✅ | ❌ |
| BLE | ✅ | ❌ | ❌ | ❌ |
| Transport | Linux | macOS | Windows | Android | OpenWrt |
|-----------|:-----:|:-----:|:-------:|:-------:|:-------:|
| UDP | ✅ | ✅ | ✅ | ✅ | ✅ |
| TCP | ✅ | ✅ | ✅ | ✅ | ✅ |
| Ethernet | ✅ | ✅ | ❌ | ❌ | ✅ |
| Tor | ✅ | ✅ | ✅ | ❌ | ✅ |
| Nym | ✅ | ✅ | ✅ | ❌ | ❌ |
| BLE | ✅ | ❌ | ❌ | ❌ | ❌ |
On Linux, a source build requires `libclang` — the LAN gateway's
nftables bindings are generated by `bindgen` at build time, which
@@ -211,7 +213,7 @@ testing/ Docker-based integration test harnesses + chaos simulation
## Status & roadmap
FIPS is at **v0.5.0-dev** on the `master` branch.
[v0.4.0](https://github.com/jmcorgan/fips/releases/tag/v0.4.0) has
[v0.4.1](https://github.com/jmcorgan/fips/releases/tag/v0.4.1) has
shipped; this development line continues the testing-and-polishing
track toward v0.5.0. The core protocol works end-to-end over
UDP, TCP, Ethernet, Tor, Nym, and Bluetooth on a global, public test
RELEASE-NOTES.md
+109 -284
View File
@@ -1,302 +1,134 @@
# FIPS v0.4.0
# FIPS v0.4.1
**Released**: 2026-06-21 (provisional)
**Released**: 2026-07-19
v0.4.0 is the throughput-and-observability release on the v0.3.x wire
format. It adds two new ways for nodes to find and reach each other (the
Nym mixnet transport and opt-in mDNS LAN discovery), overhauls the data
plane for higher single-node throughput and lower per-packet CPU, moves
the entire operator read surface off the data-plane hot path so
observability stays responsive under load, ships a reworked `fipstop`
TUI, and hardens FMP and FSP rekey to be hitless under packet loss in
both directions. It also folds in the accumulated mesh-convergence,
admission-control, and packaging fixes from the maintenance line.
v0.4.1 is a maintenance release on the v0.4.x line. It raises the default
antipoison cap on inbound bloom filter announcements, removes a redundant
spanning-tree metric counter, fixes two convergence and path-MTU bugs, and
cuts per-packet CPU in the bloom and identity paths. There is no wire
format change and no new feature surface.
v0.4.0 is wire-compatible with v0.3.0. Mixed meshes interoperate; there
is no flag-day upgrade. A deployed v0.3.0 node and an upgraded v0.4.0
node peer, rekey, and route normally, so you can roll the upgrade out
across a mesh in any order.
v0.4.1 is wire-compatible with v0.4.0. Nodes can be upgraded one at a time
with no coordinated restart, though one behavior change below is worth
reading before you start a rolling upgrade.
## At a glance
- New outbound Nym mixnet transport with a single-container demo and a
new mixnet-relay example.
- Opt-in mDNS / DNS-SD discovery on the local link.
- Data-plane overhaul: off-task encrypt and decrypt worker pools, GSO,
connected-UDP send path, copy-avoidance on receive, batched macOS
receive.
- The full `show_*` read surface now serves off the receive loop, so
`fipsctl` and `fipstop` stay responsive on loaded nodes; a new
counter-only `show_metrics` query enables a Prometheus scraper at no
hot-path cost.
- Reworked `fipstop` TUI on a machine-verified render-snapshot base.
- Rekey is now hitless under loss and reordering in both directions.
- New packaging targets: an OpenWrt `.apk` for OpenWrt 25+ and a Nix
flake for reproducible from-source builds on Nix/NixOS.
- Six route-class transit counters partition forwarded traffic by its
tree relationship to the next hop, visible via `show_routing` and
`show_status`.
## What's new
### Nym mixnet transport
FIPS can now peer over the [Nym](https://nymtech.net/) mixnet for
metadata-resistant connectivity. The new `transports.nym` transport
makes outbound connections through a `nym-socks5-client` SOCKS5 proxy
that you run alongside the daemon (for example as a service running
alongside the fips daemon, or as a sidecar container). The transport
waits at startup for the nym-socks5-client to become ready before giving
up.
This is a privacy and anonymity deployment mode chosen for its own
properties. It mixes your FIPS traffic into the Nym cover-traffic
network so that link-level observers cannot correlate which mesh peers
are talking. A new `examples/sidecar-nostr-mixnet-relay/` demonstrates a
FIPS-reachable Nostr relay peered across the mixnet end to end, and a
single-container demo ships with the transport.
Enable it by adding a `transports.nym` instance and pointing it at your
running nym-socks5-client. See the transports reference for the field
set.
### mDNS LAN discovery
Nodes on a shared local link can now find each other with zero address
configuration. The opt-in `node.discovery.lan` path runs an mDNS /
DNS-SD responder and browser: each node advertises a FIPS service record
on the link and adopts the peers it discovers. This complements the
existing Nostr-mediated overlay discovery for the common case where the
peers are simply on the same LAN.
Turn it on with `node.discovery.lan.enabled: true`. `service_type` and
`scope` tune the advertised service record and which interfaces
participate. Discovery on the local link needs no relay and no STUN.
### Data-plane throughput overhaul
The receive and send paths were reworked for higher single-node
throughput and lower per-packet CPU, building on the v0.3.0
crypto-backend swap:
- **Off-task encrypt and decrypt.** Per-peer encrypt and decrypt now run
on dedicated worker tasks rather than inline on the receive loop, so a
single busy peer no longer serializes the whole node's crypto.
- **GSO and connected-UDP send.** The Linux send path uses generic
segmentation offload and a connected-UDP socket where available,
cutting syscall overhead on bulk flows.
- **Copy-avoidance on receive.** The receive hot path avoids buffer
copies it previously made per packet.
- **Batched macOS receive.** macOS gains a `recvmsg_x` batched receive,
mirroring the Linux `recvmmsg` batching from v0.3.0.
- **Shared immutable-state context and an atomic metric registry.**
Immutable per-node state moved into a single shared context, and
counters live in an atomic metric registry that the new `show_metrics`
query reads without touching the hot path.
These are all internal to the data plane and require no operator action.
### Observability off the hot path
Every read-only control query now renders from a snapshot published once
per tick into a lock-free `ArcSwap`, served from the control accept task
instead of round-tripping the data-plane receive loop. This covers
`show_status`, `show_stats_*`, `show_peers`, `show_sessions`,
`show_links`, `show_connections`, `show_transports`, `show_mmp`,
`show_tree`, `show_bloom`, `show_cache`, `show_routing`,
`show_identity_cache`, `show_acl`, `show_listening_sockets`, and the new
`show_metrics`. Only the mutating `connect` and `disconnect` commands
still reach the loop.
The practical effect: on a loaded node where the receive loop was busy,
`fipsctl` and `fipstop` queries previously stalled or timed out (the
five-second query pattern operators saw). They now answer promptly
regardless of data-plane load. Per-entity snapshots reuse unchanged rows
by pointer, so the per-tick publish cost stays bounded as peer and
session counts grow.
A new **`show_metrics`** query (surfaced as `fipsctl stats metrics`)
returns a counter-only snapshot of every metric family. It is the
enabler for a Prometheus scraper that pulls node counters at no hot-path
cost.
Six **route-class transit counters** partition transit-forwarded packets
by their tree relationship to the chosen next hop — tree-up, tree-down,
tree-down-cross, cross-link descend, cross-link ascend, and direct-peer
— and the six classes sum to `forwarded_packets`. They surface through
`show_routing` and `show_status`, and the `fipstop` routing tab is
reorganized so its two columns separate own/endpoint traffic from
forwarded/transit traffic with the tree-down-cross line visually flagged.
### Reworked fipstop TUI
`fipstop` gets a rendering, navigation, and read-surface overhaul on a
machine-verified base: a render-snapshot harness asserts the exact text
grid and per-cell style of every view against canned control-socket
output. New daemon-resolved fields surface through the snapshots,
including effective persistence, root and is-root state, a
per-transport-type peer-count map, per-peer effective depth, the root
npub, and the last-sent uptree filter fill ratio with the subtree size
estimate.
A separate fix clears a garbled-screen problem on startup and stray
bytes on quit, most visible over SSH and inside tmux: startup now forces
a full repaint before the first draw, and quit stops and joins the
stdin-poll thread before restoring the terminal, so post-raw-mode
keystrokes no longer echo onto the restored screen.
### Rekey reliability
FMP and FSP session rekey are now hitless under packet loss and
reordering in both directions:
- Inbound frames are authenticated against the pending session before
the K-bit cutover promotes it, so a spoofed or stale frame cannot
derail a rekey in progress.
- Rekey message-1 retransmission is bounded, and the link-dead heartbeat
is rekey-aware so an in-flight rekey is not mistaken for a dead link.
- FSP session rekey holds connectivity across the rekey window under
loss and reordering.
- Dual-initiation races (both peers starting a rekey at once on a
high-latency link) are desynchronized with symmetric jitter so the two
sides converge on one session rather than fighting.
- An exhausted retransmission-budget abort, an expected and self-limiting
outcome on lossy or high-latency links, is logged at debug rather than
warn.
The net operator takeaway: rekey completes cleanly without dropping
traffic, even on lossy or high-latency links, and the log no longer
cries wolf when a rekey gives up and retries.
### New packaging targets
- **OpenWrt `.apk`.** A new `.apk` package targets OpenWrt 25+, where
apk-tools is the mandatory package manager; the existing `.ipk`
continues to cover OpenWrt 24.x and earlier. It is built SDK-free,
reusing the `.ipk` cross-compile and installed-filesystem payload, and
releases publish `.apk` artifacts and checksums alongside `.ipk`. Like
the `.ipk`, the package is unsigned and installed with
`apk add --allow-untrusted`.
- **Nix flake.** A `flake.nix` at the project root builds all four
binaries (`fips`, `fipsctl`, `fips-gateway`, `fipstop`) from source on
Nix/NixOS, pinning the exact toolchain and wiring the native build
dependencies so no host setup is needed beyond Nix with flakes
enabled. It exposes `nix build`, `nix run`, a `nix develop` dev shell,
and `nix flake check`, with `flake.lock` committed for reproducibility.
- `node.bloom.max_inbound_fpr` default moves from `0.10` to `0.20`.
- The `parent_switched` metric counter is gone. Use `parent_switches`.
- Spanning tree no longer serves stale coordinates after a parent link is
lost through peer removal.
- Discovery no longer loosens a path MTU clamp it had correctly tightened.
- Bloom probing and identity operations do measurably less work per call,
with identical results.
## Behavior changes worth flagging
These affect operators on upgrade.
### The inbound filter FPR cap default doubles again
- **Bloom filter antipoison cap raised.** `node.bloom.max_inbound_fpr`
moves from 0.05 to 0.10, accepting filters with a higher derived
false-positive rate before rejecting them. This reduces spurious
filter rejections on larger meshes while keeping the antipoison
protection in place.
- **TCP inbound cap honors `max_connections`.** The TCP inbound accept
ceiling now resolves from explicit per-transport
`max_inbound_connections`, then node-wide
`node.limits.max_connections`, then the built-in default of 256.
Previously the TCP inbound ceiling was hardwired to 256 and ignored
`max_connections`, so raising it had no effect on inbound TCP.
- **Static host aliases hot-reload.** `/etc/fips/hosts` now reloads on
mtime change once per tick rather than only at startup, so display
names in `fipsctl` and `fipstop` reflect edits without a daemon
restart. The peer ACL reloads through the same lock-free snapshot
mechanism.
- **Quieter logs on busy public-mesh nodes.** Routine per-peer
connection-lifecycle and capacity-cap events, no-route session-datagram
drops, and exhausted rekey-budget aborts are demoted to debug, so
genuinely notable info and warn lines are no longer drowned out.
- **More visible drops.** Receive-path silent rejections now flow
through typed reject-reason counters, and discovery counts requests
dropped when the dedup cache is full (`req_dedup_cache_full`, visible
via `show_routing`). Drops that were previously silent are now
countable.
- **Tor connect-refused accounting.** The Tor transport increments its
`connect_refused` statistic (the "Refused" line in `fipstop`) on an
actively-refused SOCKS5 connect, instead of recording every connect
failure as a generic SOCKS5 error.
`node.bloom.max_inbound_fpr` goes from `0.10` to `0.20`. The cap rejects
inbound `FilterAnnounce` frames whose advertised false positive rate
exceeds it. On the fixed 1 KB, k=5 filter, `0.10` corresponds to a fill of
0.631 and roughly 1,630 reachable entries, and the busiest nodes'
aggregates had started reaching that ceiling as the mesh grew. `0.20`
corresponds to a fill of 0.7248 and roughly 2,114 entries.
Be aware that this is the second time in two releases that this default
has doubled, for the same reason both times. That is worth stating plainly
rather than repeating the previous release's framing: raising the cap buys
headroom, it does not fix anything. The real constraint is the fixed 1 KB
filter size, which is a protocol constant. The structural remedy is the v2
filter work, where filter capacity scales with the mesh instead of being
pinned. This release is an interim step to keep legitimate aggregates from
being rejected until that lands. It is not the start of a pattern of
raising the cap once per release, and if you are sizing capacity planning
around this number, plan against the v2 work rather than against a third
raise.
The antipoison property the cap exists for is preserved. A saturated or
deliberately poisoned filter still presents an FPR near 100% and is still
rejected.
**This matters during a rolling upgrade.** A v0.4.1 node accepts a
`FilterAnnounce` with a derived FPR between 0.10 and 0.20; a v0.4.0 node
drops the same frame, and the drop is silent on the wire with no NACK. The
cap also gates the mesh size estimator, which declines to produce a value
when any contributing filter is over the cap. So while a mesh is partly
upgraded, upgraded and not-yet-upgraded nodes can legitimately report
different mesh sizes, or one can report a size while the other reports
unknown. This resolves once every node is on v0.4.1. If you want to avoid
the window entirely, set `node.bloom.max_inbound_fpr: 0.10` explicitly in
your config before upgrading and remove it after the last node is done.
### The `parent_switched` counter is removed
`parent_switched` was incremented on the line immediately before
`parent_switches` at every site and never independently, so the two
counters always held the same value. `parent_switched` is now gone from
the tree metrics, the control socket snapshot, and the `fipstop` tree
view. `parent_switches` remains and is unchanged.
If you scrape the control socket, or have dashboards or alerts referencing
`parent_switched`, point them at `parent_switches`. Anything still asking
for `parent_switched` will find nothing rather than a zero.
## Notable bug fixes
The CHANGELOG has the exhaustive list. This is the operator-relevant
subset of fixes for behavior that shipped in v0.3.0.
### Stale coordinates after losing a parent through peer removal
- **Symmetric peer teardown on manual disconnect.** A manual
`fipsctl disconnect` now sends the peer a scoped Disconnect so both
ends tear down and re-handshake cleanly. Previously a manual
disconnect tore down only the local side, leaving the peer with a
stale session that was never re-adopted as a child and whose bloom
filter was never re-recorded.
- **Gateway holds long-lived and DNS-cached mappings.** `fips-gateway`
no longer drops a virtual-IP mapping while traffic is still flowing.
The mapping TTL clock previously advanced only on DNS re-query, so a
busy long-lived or DNS-cached client could have its mapping reclaimed
mid-flow. The tick now refreshes the mapping whenever conntrack reports
active sessions and recovers a draining mapping to active when traffic
resumes; only genuinely idle mappings drain.
- **Accurate mesh-size estimate under filter overlap.** The mesh-size
estimator now estimates the cardinality of the OR-union of self plus
every connected peer's inbound filter, instead of summing per-filter
cardinalities of tree peers. Summing assumed the filters were disjoint,
so a stale or oversized parent filter or a routing loop inflated the
reported mesh size and a tree rebalance flapped the count. OR-union
deduplicates overlap, equals the old result in the disjoint case, and
removes the estimate's dependence on tree-declaration cache freshness.
- **Single-uplink node reattaches within a round-trip.** A node with one
tree peer, which has periodic parent re-evaluation disabled, was left
self-rooted and unreachable if its one-shot attaching TreeAnnounce was
lost, until the next periodic re-broadcast. Tree-position exchange is
now self-healing on the receive path: a node that hears an announce
advertising a strictly worse root echoes its own declaration back,
provoking the better-rooted peer to re-push its real position
immediately.
- **macOS self-connections work end to end (#117).** Traffic a macOS
node sends to its own `<npub>.fips` address is now delivered locally
for full TCP/UDP, not just `ping6`. The point-to-point `utun` egresses
self-addressed packets into the daemon with an unfinished transport
checksum (macOS offloads it on the `lo0` loopback route), so
re-injecting them verbatim made the local stack drop every segment the
MSS-clamp rewrite did not happen to fix and self-connections
half-opened and hung. The hairpin path now recomputes the TCP/UDP
checksum before re-injection. Linux was unaffected.
When a node's parent link dropped via peer removal, the node correctly
reparented or self-rooted, but skipped the coordinate cache invalidation
that every other position-change path performs. Cached entries for
downstream destinations kept the node's old coordinate prefix. This did
not self-correct the way a stale cache entry normally would: routing
access refreshes an entry's TTL, so an entry that was actively being
routed through never expired, and was only fixed by an unrelated fresh
insert. Both invalidation classes now run on this path, matching the
loop-detection branch.
### Discovery could loosen a tightened path MTU clamp
An originator handling a `LookupResponse` overwrote its cached path MTU
unconditionally. If a reactive `MtuExceeded` or `PathMtuNotification` had
already taught it a tighter value, a later, looser discovery estimate
would clobber that and re-loosen the clamp, risking a return to dropped
oversized packets. The cached and received values are now compared and the
tighter one is kept.
## Upgrade notes
Operator-actionable items moving from v0.3.0 to v0.4.0:
This is a drop-in upgrade from v0.4.0 with no wire format change, no
config migration, and no coordinated restart. Upgrade nodes in whatever
order you like.
- **Wire-compatible, no flag day.** v0.4.0 peers with v0.3.0. Upgrade
nodes in any order. During a rolling upgrade you may see some log lines
on the upgraded side as it interacts with not-yet-upgraded peers;
behavior is correct, log noise only.
- **Bloom antipoison cap default changed.** `node.bloom.max_inbound_fpr`
now defaults to 0.10 (was 0.05). If you set this explicitly, review
whether you still want the old value.
- **New optional config surfaces.** `transports.nym` (outbound Nym
mixnet) and `node.discovery.lan` (mDNS LAN discovery) are both opt-in
and off by default. Adding them is the only way to turn the new paths
on.
- **TCP inbound cap.** If you relied on the old hardwired 256 inbound-TCP
ceiling, note it now honors `max_inbound_connections` then
`node.limits.max_connections` then 256.
- **New observability query.** `fipsctl stats metrics` (the
`show_metrics` control query) returns a counter-only snapshot suitable
for a scraper.
Two things to do rather than assume:
## Getting v0.4.0
1. If you monitor `parent_switched`, move to `parent_switches` before
upgrading, or your dashboards will go blank rather than error.
2. During the rolling window, expect upgraded and not-yet-upgraded nodes
to potentially disagree about mesh size, per the FPR cap section above.
This is expected and self-resolves. Do not chase it as a bug unless it
persists after every node reports `0.4.1`.
If you have pinned `node.bloom.max_inbound_fpr` explicitly in your config,
your setting is honored and nothing changes for you. The change only
affects nodes taking the default.
Downgrading to v0.4.0 is supported and needs no special handling.
## Getting v0.4.1
- **Linux x86_64 / aarch64**: `.deb` and tarball at the
[v0.4.0 release page](https://github.com/jmcorgan/fips/releases/tag/v0.4.0).
[v0.4.1 release page](https://github.com/jmcorgan/fips/releases/tag/v0.4.1).
- **Arch Linux**: `fips` from the AUR.
- **macOS**: `.pkg` at the v0.4.0 release page.
- **Windows**: ZIP at the v0.4.0 release page.
- **macOS**: `.pkg` at the v0.4.1 release page.
- **Windows**: ZIP at the v0.4.1 release page.
- **OpenWrt**: `.ipk` (OpenWrt 24.x and earlier) or `.apk` (OpenWrt 25+)
at the v0.4.0 release page.
- **From source**: `cargo build --release` from a checkout of the v0.4.0
at the v0.4.1 release page.
- **From source**: `cargo build --release` from a checkout of the v0.4.1
tag (Rust 1.94.1 per `rust-toolchain.toml`; `libclang-dev` is a
required Linux build prerequisite).
- **Nix / NixOS**: `nix build .#fips` from a checkout of the v0.4.0 tag
- **Nix / NixOS**: `nix build .#fips` from a checkout of the v0.4.1 tag
builds the binaries from source with the pinned toolchain and no manual
prerequisites (see the Nix section of `packaging/README.md`).
@@ -309,13 +141,6 @@ The full per-commit changelog lives in
Thanks to everyone who contributed code, packaging work, bug reports, or
reviews to this release.
- [@jcorgan](https://github.com/jmcorgan): release shepherd, high-level
design, control read plane, rekey hardening, admission, bug fixes,
testing, packaging, PR coordination, and issue resolution.
- [@mmalmi](https://github.com/mmalmi): opt-in mDNS LAN discovery and
data-plane performance work.
- [@Origami74](https://github.com/Origami74): macOS packaging and
website coordination.
- [@dskvr](https://github.com/dskvr): AUR packaging.
- [@oleksky](https://github.com/oleksky): Nym mixnet transport and the
single-container mixnet demo.
- [@jcorgan](https://github.com/jmcorgan): release shepherd, spanning-tree
and discovery fixes, bloom and identity performance work, antipoison cap
change, and testing.
benches/routing_next_hop.rs
+365
View File
@@ -0,0 +1,365 @@
//! Micro-benchmark quantifying the per-forwarded-packet heap-allocation cost
//! of the routing next-hop candidate-assembly path.
//!
//! `find_next_hop` runs once per forwarded data packet. Its sans-IO core
//! assembles a `Vec<Candidate>` by enumerating every peer through the
//! `RoutingView` seam: `peer_addrs()` materializes a `Vec<NodeAddr>` of all
//! peers, the survivors are snapshotted (each cloning its `TreeCoordinate`),
//! and the result is collected into a second `Vec`. This bench measures that
//! per-call allocation against a fused zero-alloc reference that iterates the
//! peer map directly and borrows coordinates instead of cloning.
//!
//! Visibility caveat: the production `routing_candidates` / `select_best_candidate`
//! / `RoutingView` / `Candidate` are `pub(crate)` (src/proto/routing/core.rs)
//! and are not re-exported at the crate root, so an external bench crate cannot
//! name them. Rather than change production visibility, this file reproduces
//! that path verbatim over the real public `NodeAddr` / `TreeCoordinate` /
//! `CoordEntry` / `BloomFilter` types with the same iterator chain and the same
//! `HashMap`-backed view the shell uses (src/node/mod.rs NodeRoutingView). The
//! allocation behavior is therefore identical to production by construction;
//! only the symbol identity differs.
use std::alloc::{GlobalAlloc, Layout, System};
use std::collections::HashMap;
use std::hint::black_box;
use std::sync::atomic::{AtomicUsize, Ordering};
use criterion::{BenchmarkId, Criterion, criterion_group, criterion_main};
use fips::{BloomFilter, NodeAddr, TreeCoordinate};
// ---------------------------------------------------------------------------
// Counting global allocator: bumps a process-global counter on every heap
// allocation operation (alloc / alloc_zeroed / realloc). Sampled tightly and
// single-threaded in `report_allocs` so no unrelated allocations are captured.
// ---------------------------------------------------------------------------
struct CountingAlloc;
static ALLOCS: AtomicUsize = AtomicUsize::new(0);
unsafe impl GlobalAlloc for CountingAlloc {
unsafe fn alloc(&self, layout: Layout) -> *mut u8 {
ALLOCS.fetch_add(1, Ordering::Relaxed);
unsafe { System.alloc(layout) }
}
unsafe fn dealloc(&self, ptr: *mut u8, layout: Layout) {
unsafe { System.dealloc(ptr, layout) }
}
unsafe fn alloc_zeroed(&self, layout: Layout) -> *mut u8 {
ALLOCS.fetch_add(1, Ordering::Relaxed);
unsafe { System.alloc_zeroed(layout) }
}
unsafe fn realloc(&self, ptr: *mut u8, layout: Layout, new_size: usize) -> *mut u8 {
ALLOCS.fetch_add(1, Ordering::Relaxed);
unsafe { System.realloc(ptr, layout, new_size) }
}
}
#[global_allocator]
static GLOBAL: CountingAlloc = CountingAlloc;
const PEER_COUNTS: [usize; 4] = [8, 32, 128, 256];
/// Fraction of peers whose bloom filter reports the destination reachable.
const REACH_NUMERATOR: usize = 1;
const REACH_DENOMINATOR: usize = 2;
/// Tree depth for synthetic coordinates (self..root), a realistic mesh depth.
const COORD_DEPTH: usize = 8;
// ---------------------------------------------------------------------------
// Reproduction of the pub(crate) routing seam (src/proto/routing/core.rs).
// ---------------------------------------------------------------------------
trait RoutingView {
fn peer_addrs(&self) -> Vec<NodeAddr>;
fn peer_may_reach(&self, peer: &NodeAddr, dest: &NodeAddr) -> bool;
fn peer_can_send(&self, peer: &NodeAddr) -> bool;
fn peer_link_cost(&self, peer: &NodeAddr) -> f64;
fn peer_coords(&self, peer: &NodeAddr) -> Option<TreeCoordinate>;
}
struct Candidate {
addr: NodeAddr,
can_send: bool,
link_cost: f64,
coords: Option<TreeCoordinate>,
}
/// Verbatim from `routing::routing_candidates` (core.rs). Allocates the
/// `peer_addrs` Vec, clones each survivor's coords, and collects into a Vec.
fn routing_candidates(rv: &impl RoutingView, dest: &NodeAddr) -> Vec<Candidate> {
rv.peer_addrs()
.into_iter()
.filter(|peer| rv.peer_may_reach(peer, dest))
.map(|peer| Candidate {
can_send: rv.peer_can_send(&peer),
link_cost: rv.peer_link_cost(&peer),
coords: rv.peer_coords(&peer),
addr: peer,
})
.collect()
}
/// Verbatim from `routing::select_best_candidate` (core.rs). Pure, no alloc.
fn select_best_candidate(
candidates: &[Candidate],
dest_coords: &TreeCoordinate,
my_coords: &TreeCoordinate,
) -> Option<NodeAddr> {
let my_distance = my_coords.distance_to(dest_coords);
let mut best: Option<(&Candidate, f64, usize)> = None;
for candidate in candidates {
if !candidate.can_send {
continue;
}
let cost = candidate.link_cost;
let dist = candidate
.coords
.as_ref()
.map(|pc| pc.distance_to(dest_coords))
.unwrap_or(usize::MAX);
if dist >= my_distance {
continue;
}
let dominated = match &best {
None => true,
Some((_, best_cost, best_dist)) => {
cost < *best_cost
|| (cost == *best_cost && dist < *best_dist)
|| (cost == *best_cost
&& dist == *best_dist
&& candidate.addr < best.as_ref().unwrap().0.addr)
}
};
if dominated {
best = Some((candidate, cost, dist));
}
}
best.map(|(candidate, _, _)| candidate.addr)
}
// ---------------------------------------------------------------------------
// Bench-local view, HashMap-backed exactly like src/node/mod.rs NodeRoutingView.
// ---------------------------------------------------------------------------
struct BenchPeer {
bloom: BloomFilter,
can_send: bool,
link_cost: f64,
}
struct BenchView {
peers: HashMap<NodeAddr, BenchPeer>,
coords: HashMap<NodeAddr, TreeCoordinate>,
}
impl RoutingView for BenchView {
fn peer_addrs(&self) -> Vec<NodeAddr> {
self.peers.keys().copied().collect()
}
fn peer_may_reach(&self, peer: &NodeAddr, dest: &NodeAddr) -> bool {
self.peers.get(peer).is_some_and(|p| p.bloom.contains(dest))
}
fn peer_can_send(&self, peer: &NodeAddr) -> bool {
self.peers.get(peer).is_some_and(|p| p.can_send)
}
fn peer_link_cost(&self, peer: &NodeAddr) -> f64 {
self.peers.get(peer).map_or(f64::INFINITY, |p| p.link_cost)
}
fn peer_coords(&self, peer: &NodeAddr) -> Option<TreeCoordinate> {
self.coords.get(peer).cloned()
}
}
/// Zero-alloc reference: what an iterator/visitor seam would do. Iterates the
/// peer map directly, fuses the may_reach + can_send filters, borrows coords
/// instead of cloning, and tracks the best hop inline. No Vec, no coord clone.
fn resolve_next_hop_zeroalloc(
view: &BenchView,
dest: &NodeAddr,
dest_coords: &TreeCoordinate,
my_coords: &TreeCoordinate,
) -> Option<NodeAddr> {
let my_distance = my_coords.distance_to(dest_coords);
let mut best: Option<(NodeAddr, f64, usize)> = None;
for (addr, peer) in &view.peers {
if !peer.bloom.contains(dest) {
continue;
}
if !peer.can_send {
continue;
}
let cost = peer.link_cost;
let dist = view
.coords
.get(addr)
.map(|pc| pc.distance_to(dest_coords))
.unwrap_or(usize::MAX);
if dist >= my_distance {
continue;
}
let dominated = match &best {
None => true,
Some((best_addr, best_cost, best_dist)) => {
cost < *best_cost
|| (cost == *best_cost && dist < *best_dist)
|| (cost == *best_cost && dist == *best_dist && *addr < *best_addr)
}
};
if dominated {
best = Some((*addr, cost, dist));
}
}
best.map(|(addr, _, _)| addr)
}
// ---------------------------------------------------------------------------
// Scenario construction.
// ---------------------------------------------------------------------------
fn addr(tag: u8, i: u16) -> NodeAddr {
let mut b = [0u8; 16];
b[0] = tag;
b[1..3].copy_from_slice(&i.to_le_bytes());
NodeAddr::from_bytes(b)
}
/// A depth-`COORD_DEPTH` coordinate whose leaf is `leaf`, sharing a fixed
/// interior path and root with `shared_tag`. Peers built with the dest's
/// shared_tag sit close to the destination (distance 2); a distinct shared_tag
/// sits far (near the root), modeling our own position.
fn coord(leaf: NodeAddr, shared_tag: u8) -> TreeCoordinate {
let mut path = Vec::with_capacity(COORD_DEPTH);
path.push(leaf);
for level in 1..(COORD_DEPTH - 1) {
path.push(addr(shared_tag, level as u16));
}
path.push(addr(9, 0)); // common root
TreeCoordinate::from_addrs(path).expect("valid coord path")
}
struct Scenario {
view: BenchView,
dest: NodeAddr,
dest_coords: TreeCoordinate,
my_coords: TreeCoordinate,
}
impl Scenario {
fn new(n: usize) -> Self {
let dest = addr(2, 0);
// Destination path uses interior tag 4; peers reuse tag 4 so survivors
// are close to the destination. Our own coords use tag 5 (far).
let dest_coords = coord(dest, 4);
let my_coords = coord(addr(6, 0), 5);
let mut peers = HashMap::new();
let mut coords = HashMap::new();
for i in 0..n {
let paddr = addr(1, i as u16);
let mut bloom = BloomFilter::new();
// Realistic fill: a handful of unrelated reachable addrs.
for f in 0..4u16 {
bloom.insert(&addr(7, i as u16 * 4 + f));
}
// A controlled fraction advertise the destination as reachable.
if (i % REACH_DENOMINATOR) < REACH_NUMERATOR {
bloom.insert(&dest);
}
peers.insert(
paddr,
BenchPeer {
bloom,
can_send: true,
link_cost: 1.0 + (i as f64) * 0.01,
},
);
// Peers share the destination's interior path (tag 4) → close.
coords.insert(paddr, coord(paddr, 4));
}
Self {
view: BenchView { peers, coords },
dest,
dest_coords,
my_coords,
}
}
fn survivors(&self) -> usize {
self.view
.peers
.values()
.filter(|p| p.bloom.contains(&self.dest))
.count()
}
}
// ---------------------------------------------------------------------------
// Allocation-per-call report (printed once, before criterion timing).
// ---------------------------------------------------------------------------
fn count_allocs<T>(iters: usize, mut f: impl FnMut() -> T) -> f64 {
for _ in 0..8 {
black_box(f());
}
let start = ALLOCS.load(Ordering::Relaxed);
for _ in 0..iters {
black_box(f());
}
let end = ALLOCS.load(Ordering::Relaxed);
(end - start) as f64 / iters as f64
}
fn report_allocs() {
const ITERS: usize = 2000;
println!("\n=== allocations per call (heap alloc ops: alloc+alloc_zeroed+realloc) ===");
println!(
"{:>6} {:>10} {:>16} {:>16}",
"peers", "survivors", "current/call", "zero-alloc/call"
);
for &n in &PEER_COUNTS {
let s = Scenario::new(n);
let survivors = s.survivors();
let current = count_allocs(ITERS, || {
let cands = routing_candidates(&s.view, &s.dest);
select_best_candidate(&cands, &s.dest_coords, &s.my_coords)
});
let zero = count_allocs(ITERS, || {
resolve_next_hop_zeroalloc(&s.view, &s.dest, &s.dest_coords, &s.my_coords)
});
println!("{n:>6} {survivors:>10} {current:>16.2} {zero:>16.2}");
}
println!();
}
fn bench_next_hop(c: &mut Criterion) {
report_allocs();
let mut group = c.benchmark_group("find_next_hop");
for &n in &PEER_COUNTS {
let scenario = Scenario::new(n);
group.bench_with_input(BenchmarkId::new("current_alloc", n), &n, |b, _| {
b.iter(|| {
let cands = routing_candidates(&scenario.view, &scenario.dest);
black_box(select_best_candidate(
&cands,
&scenario.dest_coords,
&scenario.my_coords,
))
});
});
group.bench_with_input(BenchmarkId::new("zero_alloc_ref", n), &n, |b, _| {
b.iter(|| {
black_box(resolve_next_hop_zeroalloc(
&scenario.view,
&scenario.dest,
&scenario.dest_coords,
&scenario.my_coords,
))
});
});
}
group.finish();
}
criterion_group! {
name = benches;
config = Criterion::default().sample_size(50);
targets = bench_next_hop
}
criterion_main!(benches);
docs/design/fips-bloom-filters.md
+2 -2
View File
@@ -360,14 +360,14 @@ control socket and `fipstop` dashboard. (See `compute_mesh_size()` in
The estimator refuses to produce a value when any contributing filter
is above the antipoison FPR cap (`node.bloom.max_inbound_fpr`,
default `0.10`); a partial aggregate would silently underestimate.
default `0.20`); a partial aggregate would silently underestimate.
Consumers handle the resulting `None` by displaying an "unknown"
state rather than a misleading number.
## Antipoison: Inbound FPR Cap
Inbound `FilterAnnounce` payloads are checked against
`node.bloom.max_inbound_fpr` (default `0.10`). Filters whose
`node.bloom.max_inbound_fpr` (default `0.20`). Filters whose
estimated false positive rate exceeds the cap are dropped silently
(no NACK on the wire) — they would otherwise inflate downstream
candidate evaluation cost without contributing useful discrimination.
docs/design/fips-ipv6-adapter.md
+28
View File
@@ -302,6 +302,34 @@ alternative — running under a dedicated unprivileged service
account with the capability granted on the binary — see
[../how-to/run-as-unprivileged-user.md](../how-to/run-as-unprivileged-user.md).
### App-Owned TUN (embedded hosts)
On platforms where FIPS is embedded rather than run as a daemon — notably
Android, where the `VpnService` owns the TUN fd and the app has no
`CAP_NET_ADMIN` — FIPS does not create `fips0` itself. Instead the embedder owns
the fd and exchanges IPv6 packet bytes with FIPS over channels.
`Node::enable_app_owned_tun()` sets this up. It is called after `Node::new` and
before `start()` (and before the node is moved into a background task), mirroring
`control_read_handle()`, and returns two app-side channel ends:
- **app → mesh** — the embedder pushes IPv6 packets read from its fd into
`app_outbound_tx`. These are drained by `run_rx_loop` into `handle_tun_outbound`
and routed exactly as the Reader Thread's output would be.
- **mesh → app** — inbound mesh traffic on port 256 is reconstructed and written
to the node's `tun_tx` (the same sink the Writer Thread reads); the embedder
pulls from `app_inbound_rx` and writes to its fd.
With the channels installed, `start()` skips system-TUN creation (it gates on
`tun_tx` being unset), so FIPS does no `CAP_NET_ADMIN` operations.
Because packets enter via `app_outbound_tx` rather than the Reader Thread, they
**bypass `handle_tun_packet`** — the `fd00::/8` destination filter, the ICMPv6
Destination Unreachable for off-mesh dests (see [Reader Thread](#reader-thread)),
and the [TUN-Side TCP MSS Clamping](#tun-side-tcp-mss-clamping). The embedder is
therefore responsible for routing only `fd00::/8` to its TUN (so only mesh-bound
packets arrive) and for clamping TCP MSS on outbound SYNs.
## Implementation Status
| Feature | Status |
docs/design/fips-nostr-discovery.md
+1 -1
View File
@@ -477,7 +477,7 @@ The TXT record carries three keys (`src/discovery/lan/mod.rs:47-55`):
Once per node tick, the node drains browser events and acts on them in
`poll_lan_discovery()` (`src/node/lifecycle.rs:907`, called from
`src/node/handlers/rx_loop.rs:266`). For each discovered peer it finds
`src/node/dataplane/rx_loop.rs:266`). For each discovered peer it finds
a UDP transport whose family matches the peer address, parses the
`npub` into a `PeerIdentity`, skips peers it is already connected to or
currently connecting to, and otherwise initiates a connection.
docs/design/fips-transport-layer.md
+6 -6
View File
@@ -262,7 +262,7 @@ UDP (1500 vs 1472 MTU).
- **No IP dependency**: Operates below the IP layer. Nodes on the same
Ethernet segment can communicate without IP addresses or routing
infrastructure
- **Broadcast discovery**: Nodes discover each other via periodic beacon
- **Broadcast neighbor detection**: Nodes discover each other via periodic beacon
broadcasts on the shared medium, with no static peer configuration required
- **Higher MTU**: Standard Ethernet frames carry 1500 bytes of payload,
yielding an effective FIPS MTU of 1499 after the frame type prefix
@@ -293,7 +293,7 @@ socket.
| Addressing | 6-byte MAC address |
| Platform | Linux only (`CAP_NET_RAW` required) |
### Beacon Discovery
### Neighbor Beacons
Ethernet nodes discover peers via broadcast beacons sent to
ff:ff:ff:ff:ff:ff. Each beacon is a 34-byte frame containing the sender's
@@ -301,7 +301,7 @@ x-only public key. Receiving nodes extract the MAC source address from the
frame and the public key from the payload, then report the discovered peer
to FMP.
Four configuration flags control discovery behavior — `discovery`
Four configuration flags control neighbor behavior — `listen`
(listen for beacons), `announce` (broadcast beacons), `auto_connect`
(initiate handshakes to discovered peers), and `accept_connections`
(accept inbound handshakes). The flag table and per-flag defaults
@@ -310,13 +310,13 @@ under `transports.ethernet.*`.
A typical discoverable node sets `announce`, `auto_connect`, and
`accept_connections` all true. A passive listener uses just
`discovery: true` to observe the network without announcing itself.
`listen: true` to observe the network without announcing itself.
### WiFi Compatibility
WiFi interfaces in infrastructure (managed) mode work transparently for
unicast — the mac80211 subsystem handles frame translation between 802.11
and 802.3. Broadcast beacon discovery is unreliable in managed mode because
and 802.3. Broadcast neighbor detection is unreliable in managed mode because
access points commonly isolate clients from each other's broadcast traffic.
Startup logging:
@@ -895,7 +895,7 @@ transitions through `Starting` to `Up` (operational). `stop()` moves to
| --------- | ------ | ----- |
| UDP/IP | **Implemented** | Primary transport, AsyncFd/recvmsg, SO_RXQ_OVFL kernel drop detection |
| TCP/IP | **Implemented** | FMP header-based framing, non-blocking connect, per-connection MSS MTU |
| Ethernet | **Implemented** | AF_PACKET SOCK_DGRAM, EtherType 0x2121, beacon discovery, Linux only |
| Ethernet | **Implemented** | AF_PACKET SOCK_DGRAM, EtherType 0x2121, neighbor beacons, Linux only |
| WiFi | **Implemented** (via Ethernet transport, infrastructure mode) | mac80211 translates 802.11↔802.3; broadcast beacons unreliable through APs |
| Tor | **Implemented** | Outbound SOCKS5, inbound via onion service, .onion and clearnet addressing |
| Nym | **Implemented** | Outbound-only SOCKS5 through nym-socks5-client, mixnet anonymity, IP/hostname addressing |
docs/how-to/README.md
+2
View File
@@ -25,4 +25,6 @@ X" to "X is done".
| [persistent-identity.md](persistent-identity.md) | Provision a stable Nostr keypair so the node keeps the same npub across restarts |
| [host-aliases.md](host-aliases.md) | Use shortnames (`test-us01.fips`, `my-laptop.fips`) instead of full npubs by editing `/etc/fips/hosts` or setting peer aliases |
| [set-up-bluetooth-peer.md](set-up-bluetooth-peer.md) | Configure a Bluetooth Low Energy peer link |
| [set-up-80211s-mesh-backhaul.md](set-up-80211s-mesh-backhaul.md) | Link OpenWrt FIPS routers over an open 802.11s radio backhaul (FIPS provides encryption, authentication, and routing) |
| [set-up-open-access-ssid.md](set-up-open-access-ssid.md) | Broadcast the open `!FIPS` access SSID so phones and laptops roam onto the mesh (one ESS: save once, roam every FIPS router) |
| [diagnose-mtu-issues.md](diagnose-mtu-issues.md) | Triage MTU-shaped failures and rule out their imposters (bufferbloat, transport saturation) |
docs/how-to/set-up-80211s-mesh-backhaul.md
+249
View File
@@ -0,0 +1,249 @@
# Set Up an 802.11s Mesh Backhaul (OpenWrt)
Link FIPS routers over radio — no cables, no APs, no shared
infrastructure — by running the Ethernet transport on an open 802.11s
mesh interface. The radio layer provides nothing but L2 frames to
direct neighbors; FIPS provides everything else: encryption and
authentication (Noise IK), peer discovery (Ethernet beacons), and
routing (the spanning tree).
For the transport design, see
[../design/fips-transport-layer.md](../design/fips-transport-layer.md).
For all `transports.ethernet.*` configuration keys, see
[../reference/configuration.md](../reference/configuration.md).
## Why open, why forwarding off
Two deliberate choices distinguish this from a stock 802.11s setup:
- **`encryption none`** — the mesh is open on purpose. Every FIPS peer
link is already authenticated and encrypted by the Noise IK
handshake, so SAE at L2 would duplicate that work, add a shared
credential to provision across routers, and (on ath10k) force the
firmware into its slower raw Tx/Rx mode. A stranger can form an
802.11s peering with your router *and* a FIPS peer link on top of it —
the same open model as mDNS and BLE discovery, where the advert is
only a hint and the handshake authenticates each link (no
impersonation, no MITM) rather than gating who may peer. Admission is
open up to the daemon's max-peers cap. What you concede: any nearby
radio can peer and reach the FIPS overlay surface; L2 metadata (MAC
addresses, frame sizes) is visible in the air; a hostile radio can
burn airtime — all inherent to an open radio link.
- **`mesh_fwding 0`** — disables 802.11s's own HWMP routing so each
mesh link is a plain neighbor link. FIPS is the routing layer; two
routing layers would fight, and broadcast discovery beacons would
flood the whole mesh instead of reaching direct neighbors only.
The interface is **not** bridged into `br-lan` — the FIPS Ethernet
transport binds it directly.
## When to use
- Two or more OpenWrt FIPS routers within radio range of each other,
where running cable is impractical.
- You want the mesh segment to keep working with zero shared
credentials or per-site configuration ("flash and drop in").
It is **not** for connecting phones or laptops — client devices
cannot join an 802.11s mesh. They enter the mesh through a normal AP
on the same router (see constraints below), or over BLE.
## Requirements
- OpenWrt 22.03+ with the FIPS package installed.
- A radio whose driver supports mesh point interfaces. Check with:
```sh
iw list | grep -A 10 "Supported interface modes" | grep "mesh point"
```
The mainstream OpenWrt chips (ath9k, ath10k, mt76) all qualify.
- Ideally a dual- or tri-band router, so one band can be dedicated to
the backhaul (see constraints).
## Step 1 — create the mesh interface(s)
On **each** router, run the helper once per radio you want in the
backhaul:
```sh
fips-mesh-setup radio1
```
This creates an open 802.11s interface with mesh ID `fips-mesh` and
HWMP forwarding off, attaches it to an unmanaged netifd interface (no
IP configuration — none is needed), uncomments the matching `meshN`
transport entry in `/etc/fips/fips.yaml` (see Step 2), and reloads the
radio. Interfaces are named by radio index: `radio0` → `fips-mesh0`,
`radio1` → `fips-mesh1`. Pass a second argument to use a different
mesh ID.
Note: the helper runs `wifi reload`, which re-applies the whole
wireless config and so briefly drops every client AP on all radios for
a few seconds. `fips-mesh-setup remove` reloads the same way. Expect
the blip if clients are connected.
On dual-band routers, meshing **both** bands is worth it: 2.4 GHz
reaches further at lower rates, 5 GHz carries more over shorter
links. Note this is **failover, not multipath**: FIPS keeps one
active link per peer, so traffic uses one band at a time — the other
is a standby that re-establishes the peer if the active link dies
(detection via keepalive timeout, so a cutover takes seconds, not
milliseconds):
```sh
fips-mesh-setup radio0
fips-mesh-setup radio1
```
**Pin the same channel on every backhaul router, per band.** Mesh
points only peer on the same channel, and the mesh inherits whatever
the radio is set to — with `channel 'auto'` (the default on many
devices) each router picks its own and the mesh silently never forms.
The script prints the radio's current band and channel and warns on
`auto`:
```sh
uci set wireless.radio1.channel='36'
uci commit wireless && wifi reload
```
Prefer a non-DFS channel (36–48 on 5 GHz): on DFS channels the radio
must wait ~60 s in CAC before transmitting after every reload.
Equivalent manual UCI (per radio), if you prefer to see what it does:
```sh
uci batch <<'EOF'
set wireless.fips_mesh_radio1=wifi-iface
set wireless.fips_mesh_radio1.device='radio1'
set wireless.fips_mesh_radio1.mode='mesh'
set wireless.fips_mesh_radio1.mesh_id='fips-mesh'
set wireless.fips_mesh_radio1.encryption='none'
set wireless.fips_mesh_radio1.mesh_fwding='0'
set wireless.fips_mesh_radio1.ifname='fips-mesh1'
set wireless.fips_mesh_radio1.network='fips_mesh_radio1'
set network.fips_mesh_radio1=interface
set network.fips_mesh_radio1.proto='none'
EOF
uci commit
wifi reload
```
## Step 2 — check the FIPS transport binding
The `fips.yaml` shipped in the OpenWrt package carries one transport
entry per radio, but **commented out** — so a stock install that never
runs this helper logs no per-boot "interface missing" warning.
`fips-mesh-setup` uncommented the matching `meshN` entry in Step 1, so
there is normally nothing to do here. If you maintain your own config
(or ran the manual UCI above instead of the helper), make sure the
entries are present and uncommented:
```yaml
transports:
ethernet:
mesh0:
interface: "fips-mesh0"
discovery: true
announce: true
auto_connect: true
accept_connections: true
mesh1:
interface: "fips-mesh1"
discovery: true
announce: true
auto_connect: true
accept_connections: true
```
## Step 3 — restart the daemon (order matters)
```sh
/etc/init.d/fips restart
```
Restart fips **after** the mesh interface is up. A transport whose
interface is missing at startup is logged and skipped, not retried —
so if the daemon comes up before the radio, the mesh transport stays
dead until the next restart. (An interface that *vanishes and
returns* after startup is recovered automatically; only the missing-
at-startup case needs this ordering.)
## Verify
L2 first — the 802.11s peering, with a second configured router in
range:
```sh
iw dev fips-mesh0 station dump
```
You should see one station entry per neighbor router, with signal
levels. No entries means a radio problem, not a FIPS problem — triage
in this order:
1. **Channel mismatch** (the most common cause): compare
`iw dev fips-mesh0 info` on both routers — mesh ID *and* channel
must match exactly.
2. **The mesh interface never joined** — `iw dev fips-meshX info`
shows `type mesh point` but **no channel line**, and `station dump`
is empty. Usual cause: a client (`sta`) interface on the same
radio. A STA must follow its upstream AP's channel, the whole
radio follows the STA, and a mesh pinned to a different channel
silently stays down. Check for a STA sharing the radio
(`iw dev`, look for `type managed` on the same phy), compare
`iw dev <sta-iface> info | grep channel`, and re-pin the mesh
channel to match — on every backhaul router.
3. **Is the other router transmitting at all?**
```sh
iw dev fips-mesh0 scan | grep -i -B4 "MESH ID"
```
Its mesh ID visible → transmission works, peering is failing
(mesh ID typo, or one side has encryption set). Nothing visible →
check `wifi status` on the other router, remember the ~60 s DFS
CAC wait, and confirm the country code is set
(`uci get wireless.radio1.country`) — an unset regdomain can
block channels entirely.
4. `logread | grep -iE "mesh|fips-mesh0"` on both sides.
Then the FIPS layer on top:
```sh
logread | grep -i beacon # beacons flowing on the new transport
fipsctl show peers # neighbor authenticated and connected
fipsctl show links # link on the 'ethernet' transport
```
Discovery is automatic: each node beacons its pubkey every few
seconds, and `auto_connect` initiates the Noise handshake on first
sight.
## Constraints
- **Airtime is shared per radio.** All virtual interfaces on one
radio (AP + mesh) share one channel, and multi-hop forwarding on a
single radio roughly halves throughput per hop. On dual/tri-band
hardware, dedicate one band to `fips-mesh0` and serve clients on
the others.
- **AP + mesh coexistence is driver-dependent.** It works on the
mainstream chips (this is the standard Freifunk/Gluon setup), but
check `iw list` under "valid interface combinations" for your
hardware.
- **Clients can't join.** Phones and laptops reach the mesh through
the router's normal AP or via BLE — never through the 802.11s
interface.
- **Radio links are lossy.** A neighbor at the edge of range will
form an 802.11s peering yet deliver a fraction of its frames.
Expect link-quality effects that don't exist on wired Ethernet.
- **A client (STA) uplink on the same radio owns the channel.** The
STA must follow whatever channel its upstream AP uses; every other
interface on that radio follows the STA. A mesh pinned to a
different channel silently never joins, and it does **not** recover
when the STA disconnects — a `wifi reload` (plus a fips restart) is
needed. A *roaming* uplink (travel-router / hotspot-chasing setups)
is fundamentally incompatible with a fixed-channel mesh on the same
radio: dedicate the mesh to the radio the STA never uses, and treat
any mesh sharing a STA radio as best-effort.
docs/how-to/set-up-open-access-ssid.md
+267
View File
@@ -0,0 +1,267 @@
# Set Up the Open FIPS Access SSID (OpenWrt)
Give phones and laptops a way in: every FIPS router broadcasts the
same open SSID — `!FIPS` — from its access radio. Same SSID + unique
BSSIDs is one standard ESS, so a client saves the network once and
roams between all FIPS routers natively, with no per-router setup and
no shared credentials (the Freifunk model). The leading `!` sorts the
network to the top of alphabetically ordered pickers (iOS, desktop
OSes — Android sorts by signal strength) and is part of the name:
SSIDs match byte-for-byte or not at all. The radio layer provides
nothing but open L2 to the nearest router; FIPS provides everything
else: encryption and authentication (Noise IK), discovery
(mDNS/Ethernet beacons), and mobility (the overlay identity survives
roaming, so no 802.11r or L2 tricks are needed).
This is the *access* layer — how clients reach FIPS routers. For the
router-to-router *backhaul*, see
[set-up-80211s-mesh-backhaul.md](set-up-80211s-mesh-backhaul.md).
For all `transports.ethernet.*` configuration keys, see
[../reference/configuration.md](../reference/configuration.md).
## Why open, why this addressing
Three deliberate choices distinguish this from a stock guest network:
- **`encryption none`** — the SSID is open on purpose, and it *must*
be. Clients key a saved network on SSID **plus security type**: if
one router used a PSK and another OWE, the same `FIPS` name would be
three different saved networks and roaming would break. Open is the
only security type that needs zero provisioning, and OWE is left out
for now for exactly this uniformity reason (OWE-transition mode is
inconsistent across client vendors). Every FIPS peer link is already
authenticated and encrypted by the Noise IK handshake. A stranger
can associate *and* form a FIPS peer link — that is the point of open
access; the handshake authenticates each link (no impersonation, no
MITM) but does not gate who may peer, and admission is open up to the
daemon's max-peers cap. What confines a hostile peer is the isolated
`fips_ap` zone (no path to br-lan or the WAN — see below), not the
handshake. What you concede: any nearby device can reach the FIPS
overlay surface (handshake, discovery, lookup, routing) and peer with
the router; L2 metadata is visible in the air; a hostile radio can
burn airtime — all inherent to an open radio link.
- **DHCPv4 from a fixed subnet, plus IPv6 router advertisements.**
dnsmasq leases IPv4 out of `10.21.<N>.0/24` (`N` = the radio index;
the prefix echoes FIPS port 2121). The subnet is deliberately
**identical on every router**: a roaming phone keeps its lease
across routers, and dnsmasq's authoritative mode (the OpenWrt
default, pinned by the helper) ACKs a renew the new router never
issued. odhcpd additionally announces a ULA prefix (`fd..`-range)
for stateless SLAAC; DHCPv6 stays off. FIPS itself only needs
link-local + mDNS, but Android's provisioning check requires an RA
or a DHCP offer and *disconnects* with neither, and plain laptops
expect a real IPv4 address. Works with or without an upstream —
nothing here depends on the WAN. The IPv6 side stays per-router and
disposable; in all cases the FIPS overlay identity, not the IP, is
the mobility anchor.
- **Isolated interface** — its own network and firewall zone, with no
path to `br-lan` and no forwarding to the WAN. Inbound traffic is
rejected except DHCPv4, ICMPv6 (SLAAC itself), mDNS, and the FIPS
transport ports; the raw-Ethernet transport (EtherType 0x2121) is
not IP and never traverses the firewall. AP client isolation is on, so clients
cannot reach each other at L2 — two FIPS phones on one router still
reach each other through the router at the overlay layer.
## The "no internet" behavior (expected, one-time acceptance)
The network intentionally provides **no internet**. On first connect,
a phone's validation probe fails and it asks whether to stay on a
network without internet access — choose **stay connected** and
**don't ask again**. That choice is stored per SSID, so accepting it
once covers every FIPS router anywhere.
After that, the network is marked "connected, no internet"
(unvalidated) and the phone keeps **cellular as its default route**
while staying associated — normal apps never notice the FIPS network
exists. FIPS apps bind their sockets to the Wi-Fi network explicitly,
so mesh traffic flows over Wi-Fi while everything else uses cellular.
## When to use
- Any FIPS router that should serve phones and laptops directly, not
just peer with other routers.
- You want clients to roam between FIPS routers with zero per-router
or per-site configuration.
It is the complement of the 802.11s backhaul: the backhaul links
routers (clients cannot join it), the access SSID admits clients.
Both can share a radio, at an airtime cost (see constraints).
## Requirements
- OpenWrt 22.03+ with the FIPS package installed (fw4; dnsmasq and
odhcpd are part of the default images).
- Any radio — AP mode needs no special driver support.
## Step 1 — create the access point(s)
On **each** router, run the helper once per radio that should serve
clients:
```sh
fips-ap-setup radio0
```
This creates an open AP with SSID `!FIPS` and client isolation, an
isolated network with `10.21.<N>.1/24` and a static ULA `/64`, a
DHCPv4 + RA dhcp config (dnsmasq leases, SLAAC, no DHCPv6), and a
locked-down `fips_ap` firewall zone — then reloads the radio. Interfaces are named by radio index: `radio0` →
`fips-ap0`, `radio1` → `fips-ap1`. Pass a second argument to use a
different SSID — but the SSID, like the security type, must be
identical on **all** routers or clients will treat them as separate
networks and stop roaming.
On dual-band routers, run it for both radios so clients can pick
either band:
```sh
fips-ap-setup radio0
fips-ap-setup radio1
```
**Channels are free per router.** Unlike the mesh backhaul, there is
no same-channel constraint — clients scan when they roam — so leave
each router on whatever channel suits its RF environment.
Equivalent manual UCI (per radio), if you prefer to see what it does
(`fdxx:...` stands for a `/64` out of the router's ULA prefix):
```sh
uci batch <<'EOF'
set wireless.fips_ap_radio0=wifi-iface
set wireless.fips_ap_radio0.device='radio0'
set wireless.fips_ap_radio0.mode='ap'
set wireless.fips_ap_radio0.ssid='!FIPS'
set wireless.fips_ap_radio0.encryption='none'
set wireless.fips_ap_radio0.isolate='1'
set wireless.fips_ap_radio0.ifname='fips-ap0'
set wireless.fips_ap_radio0.network='fips_ap_radio0'
set network.fips_ap_radio0=interface
set network.fips_ap_radio0.proto='static'
set network.fips_ap_radio0.ipaddr='10.21.0.1'
set network.fips_ap_radio0.netmask='255.255.255.0'
set network.fips_ap_radio0.ip6addr='fdxx:xxxx:xxxx:fa00::1/64'
set dhcp.fips_ap_radio0=dhcp
set dhcp.fips_ap_radio0.interface='fips_ap_radio0'
set dhcp.fips_ap_radio0.ra='server'
set dhcp.fips_ap_radio0.ra_default='2'
set dhcp.fips_ap_radio0.dhcpv6='disabled'
set dhcp.fips_ap_radio0.dhcpv4='server'
set dhcp.fips_ap_radio0.start='10'
set dhcp.fips_ap_radio0.limit='200'
EOF
uci commit
wifi reload
```
plus the `fips_ap` firewall zone (input/forward REJECT, no
forwardings, ACCEPT rules for DHCPv4/UDP 67, ICMPv6, UDP 5353/2121,
TCP 8443).
## Step 2 — check the FIPS transport binding
The `fips.yaml` shipped in the OpenWrt package carries one transport
entry per access interface, but **commented out** — so a stock install
that never runs this helper logs no per-boot "interface missing"
warning. `fips-ap-setup` uncommented the matching `apN` entry in Step 1,
and also enabled `node.rendezvous.lan` (the daemon's mDNS/DNS-SD
rendezvous — phone FIPS apps cannot see raw-Ethernet beacons, so mDNS
is how they find the daemon; the switch is daemon-wide and stays on if
you later remove the AP). So there is normally nothing to do here. If
you maintain your own config (or ran the manual UCI above instead of
the helper), make sure both are present and uncommented:
```yaml
node:
rendezvous:
lan:
enabled: true
```
```yaml
transports:
ethernet:
ap0:
interface: "fips-ap0"
discovery: true
announce: true
auto_connect: true
accept_connections: true
ap1:
interface: "fips-ap1"
discovery: true
announce: true
auto_connect: true
accept_connections: true
```
## Step 3 — restart the daemon (order matters)
```sh
/etc/init.d/fips restart
```
Restart fips **after** the AP interface is up. A transport whose
interface is missing at startup is logged and skipped, not retried —
so if the daemon comes up before the radio, the access transport
stays dead until the next restart. (An interface that *vanishes and
returns* after startup is recovered automatically; only the missing-
at-startup case needs this ordering.)
## Verify
L2 and addressing first, with a phone or laptop connected to `!FIPS`:
```sh
iw dev fips-ap0 station dump # one entry per associated client
ip addr show dev fips-ap0 # 10.21.0.1/24 and the fd..::1/64
cat /tmp/dhcp.leases # one lease per connected client
```
No station entries means a radio problem; an association that drops
after ~30 s usually means the client never got an address — check
`logread | grep -e dnsmasq -e odhcpd` and that the
`dhcp.fips_ap_radio0` section survived
(`uci show dhcp | grep fips_ap`).
Then the FIPS layer on top, for a client running FIPS:
```sh
logread | grep -i beacon # beacons flowing on the new transport
fipsctl show peers # client authenticated and connected
```
On the phone itself: the network shows "connected, no internet" and
stays associated — that is the designed steady state, not an error.
## Constraints
- **SSID and security type must be uniform across ALL routers.**
One router with a PSK (or OWE) under the same name splits the ESS
into different saved networks and silently breaks roaming. Never
"harden" a single router.
- **Airtime is shared per radio.** An access AP and a mesh backhaul
on the same radio share one channel. On dual/tri-band hardware,
dedicate a band to the backhaul and serve clients on the others.
- **Strangers can associate and peer — by design.** Open access means
any nearby device can complete the Noise handshake and become a FIPS
peer (up to the max-peers cap); the handshake authenticates each link,
it does not restrict who joins. They reach only the FIPS overlay
surface — the isolated zone gives no path to br-lan or the WAN. Do not
add forwardings to the `fips_ap` zone: that would turn the open SSID
into a hotspot and hand the isolation away.
- **Roaming is client-driven.** Clients decide when to hop BSSIDs
(standard ESS behavior); the IPv4 lease survives the hop (same
subnet everywhere), the SLAAC address renumbers, and FIPS sessions
ride through because the overlay identity is the anchor. Expect a
brief L2 gap during the hop, as on any ESS without 802.11r.
- **The `10.21.<N>.0/24` convention must hold everywhere.** Lease
survival depends on every router serving the same subnet from the
same radio index — the helper guarantees this; don't hand-pick
per-router subnets. Two routers can lease the same address to two
different clients; after a roam the conflict is caught (dnsmasq
NAKs a renew for an address in use) and the client re-DHCPs. If a
laptop is *also* wired to a LAN that really uses `10.21.<N>.0/24`,
its routing table will conflict — a corner case worth knowing, not
designing around: the zone forwards nowhere, so the FIPS side never
reaches beyond the router either way.
docs/reference/cli-fipsctl.md
+52
View File
@@ -115,6 +115,58 @@ Tell the daemon to drop a peer link.
| -------- | ----------- |
| `peer` | npub (bech32) or hostname from `/etc/fips/hosts`. |
### `profile tick <on|off|status>`
> **Reading the output.** Step durations are wall clock measured across `await`
> points, not CPU time: a step that waits on I/O accrues that wait, and other
> tasks may run inside the span. That is the intended measure for head-of-line
> delay, and it means a large step is not necessarily an expensive one.
> `arm_starvation` is measured directly as the entry time minus the deadline
> the interval scheduled that tick for. It is not derived from
> `tick_entry_gap`, which carries no starvation signal on its own: under a
> steady delay every gap is exactly one tick period.
Start, stop and inspect a capture of the rx-loop tick body. **Present
only when both `fipsctl` and the daemon are built with
`--features profiling`**; the feature is off by default, so a stock
package does not carry this subcommand and a stock daemon reports
`profile_tick_*` as an unknown command.
| Subcommand | Control-socket command | Description |
| ---------- | ---------------------- | ----------- |
| `profile tick on` | `profile_tick_on` | Create the capture file and start recording. Fails if a capture is already running (naming the active file) or if the directory cannot be written. |
| `profile tick off` | `profile_tick_off` | Stop the capture. The writer is woken immediately, drains once more and is joined, so the command returns promptly. Succeeds, reporting nothing active, when no capture is running. |
| `profile tick status` | `profile_tick_status` | Report `idle`, `running`, `stopped_by_cap` or `stopped_by_error`, plus the active path, bytes written, flush interval and byte cap. |
`profile tick on` options:
| Flag | Argument | Default | Description |
| ---- | -------- | ------- | ----------- |
| `--dir` | directory path | `/var/log/fips` | Where to write the capture. Created if absent. Use it to profile a non-root `cargo run`, or on a platform whose log root differs. |
One file is written per capture, named `profile-<UTC timestamp>.tsv`.
It opens with a `#`-prefixed header block (node npub, build version,
platform, configured tick period, flush interval, byte cap, start
time), then a tab-separated column header, then one row per measured
step per flush interval:
```text
ts_unix kind domain name count max total unit
```
`kind` is `step` for a timed span and `gauge` for a sampled scalar, so
a gauge value never lands under a duration column; `unit` names the
unit of `max` and `total` for that row. Every step present in the build
gets a row every interval, including zero-count rows. Gauges cover
ticks per interval, peer count, the wall gap between successive
tick-arm entries, and the arm-starvation delay, which is measured
against the deadline the tick was scheduled for rather than derived
from the gap.
A capture stops itself on reaching 32 MB, appending a `#` line saying
so; `profile tick status` then reports `stopped_by_cap` until the next
`on` or `off` clears it.
## Exit Codes
| Code | Meaning |
docs/reference/configuration.md
+8 -8
View File
@@ -277,7 +277,7 @@ Controls tree construction and parent selection.
| Parameter | Type | Default | Description |
|-----------|------|---------|-------------|
| `node.bloom.update_debounce_ms` | u64 | `500` | Debounce interval for filter update propagation |
| `node.bloom.max_inbound_fpr` | f64 | `0.10` | Antipoison cap: reject inbound `FilterAnnounce` frames whose advertised false-positive rate exceeds this value. Valid range `(0.0, 1.0)`. The default `0.10` corresponds to fill 0.631 at k=5 (≈1,630 entries on the 1 KB filter); a saturated/poisoned filter is still ~100% FPR and rejected |
| `node.bloom.max_inbound_fpr` | f64 | `0.20` | Antipoison cap: reject inbound `FilterAnnounce` frames whose advertised false-positive rate exceeds this value. Valid range `(0.0, 1.0)`. The default `0.20` corresponds to fill 0.7248 at k=5 (≈2,114 entries on the 1 KB filter); a saturated/poisoned filter is still ~100% FPR and rejected |
Bloom filter size (1 KB), hash count (5), and size classes are protocol
constants and not configurable.
@@ -436,7 +436,7 @@ Requires `CAP_NET_RAW` or running as root. Linux only.
| `mtu` | u16 | *(auto)* | Override MTU. Default: interface MTU minus 3 (for frame type + length prefix) |
| `recv_buf_size` | usize | `2097152` | Socket receive buffer size in bytes (2 MB) |
| `send_buf_size` | usize | `2097152` | Socket send buffer size in bytes (2 MB) |
| `discovery` | bool | `true` | Listen for discovery beacons from other nodes |
| `listen` | bool | `true` | Listen for neighbor beacons from other nodes |
| `announce` | bool | `false` | Broadcast announcement beacons on the LAN |
| `auto_connect` | bool | `false` | Auto-connect to discovered peers |
| `accept_connections` | bool | `false` | Accept incoming connection attempts from discovered peers |
@@ -450,7 +450,7 @@ transports:
ethernet:
lan:
interface: "eth0"
discovery: true
listen: true
announce: true
backbone:
interface: "eth1"
@@ -458,7 +458,7 @@ transports:
```
Each named instance operates independently with its own socket and
discovery state. The instance name is used in log messages and the
neighbor state. The instance name is used in log messages and the
`name()` method on the Transport trait.
### TCP (`transports.tcp.*`)
@@ -840,7 +840,7 @@ peers:
### Mixed UDP + Ethernet Example
A node bridging internet peers (UDP) and a local Ethernet segment with
beacon discovery:
neighbor beacons:
```yaml
node:
@@ -856,7 +856,7 @@ transports:
mtu: 1472
ethernet:
interface: "eth0"
discovery: true
listen: true
announce: true
auto_connect: true
accept_connections: true
@@ -944,7 +944,7 @@ node:
flap_dampening_secs: 120 # extended hold-down on flap
bloom:
update_debounce_ms: 500
max_inbound_fpr: 0.10 # antipoison cap on inbound FilterAnnounce FPR
max_inbound_fpr: 0.20 # antipoison cap on inbound FilterAnnounce FPR
session:
default_ttl: 64
pending_packets_per_dest: 16
@@ -999,7 +999,7 @@ transports:
# mtu: null # null = interface MTU - 3 (typically 1497)
# recv_buf_size: 2097152 # 2 MB
# send_buf_size: 2097152 # 2 MB
# discovery: true # listen for beacons
# listen: true # listen for beacons
# announce: false # broadcast beacons
# auto_connect: false # connect to discovered peers
# accept_connections: false # accept inbound handshakes
docs/reference/control-socket.md
+15
View File
@@ -159,6 +159,21 @@ not reproduced here to avoid duplicating the source.
Both commands run on the daemon's main task and may block briefly
while the node mutates its state.
#### Profiler toggle (`--features profiling` builds only)
| Command | Params | Behaviour |
| ------- | ------ | --------- |
| `profile_tick_on` | `dir` (optional directory path; default `/var/log/fips`) | Creates the capture file, publishes its path, and starts the writer thread. `data`: `state`, `path`, `interval_secs`, `byte_cap`. Errors if a capture is already running (naming the active file) or the directory is unwritable. |
| `profile_tick_off` | — | Stops the capture, drains once more, joins the writer. `data`: `state`, `stopped`, `stopped_by_cap`, `stopped_by_error`, `path`, `bytes`. |
| `profile_tick_status` | — | `data`: `state` (`idle` / `running` / `stopped_by_cap` / `stopped_by_error`), `path`, `bytes`, `byte_cap`, `interval_secs`. |
Unlike `connect` and `disconnect`, these three are served in the
control accept task rather than on the daemon's main task. All of their
state is process statics and none of them needs `&mut Node`, so
routing them through the main loop would only make the toggle queue
behind the tick body it exists to measure. They are absent from a
default build, where the daemon answers them as unknown commands.
## Gateway Command Catalog
`fips-gateway` exposes a separate control socket with its own command
docs/reference/security.md
+1 -1
View File
@@ -209,7 +209,7 @@ for the metadata-privacy model and the rejection of onion routing.
| --------- | --------------- | ------------ | ------ |
| UDP | None until `bind_addr` set | `0.0.0.0:2121` typical | Operator sets `transports.udp.bind_addr` |
| TCP | None until `bind_addr` set | None — outbound-only without bind | Operator sets `transports.tcp.bind_addr` |
| Ethernet | Listens on configured interface (raw `AF_PACKET`) | EtherType 0x2121 on selected interface | Per-flag `discovery`, `announce`, `auto_connect`, `accept_connections` |
| Ethernet | Listens on configured interface (raw `AF_PACKET`) | EtherType 0x2121 on selected interface | Per-flag `listen`, `announce`, `auto_connect`, `accept_connections` |
| Tor | None until `directory_service` configured | `127.0.0.1:8443` (loopback only) | Operator sets `transports.tor.directory_service` and configures `HiddenServiceDir` in `torrc` |
| BLE | Off by default | n/a | Operator enables `transports.ble.*` |
| Nostr discovery | Off by default | n/a (relay client, not a listener) | Operator sets `node.discovery.nostr.enabled: true` |
docs/releases/release-notes-v0.4.0.md
+1 -1
View File
@@ -1,6 +1,6 @@
# FIPS v0.4.0
**Released**: 2026-06-21 (provisional)
**Released**: 2026-06-27
v0.4.0 is the throughput-and-observability release on the v0.3.x wire
format. It adds two new ways for nodes to find and reach each other (the
docs/releases/release-notes-v0.4.1.md
+146
View File
@@ -0,0 +1,146 @@
# FIPS v0.4.1
**Released**: 2026-07-19
v0.4.1 is a maintenance release on the v0.4.x line. It raises the default
antipoison cap on inbound bloom filter announcements, removes a redundant
spanning-tree metric counter, fixes two convergence and path-MTU bugs, and
cuts per-packet CPU in the bloom and identity paths. There is no wire
format change and no new feature surface.
v0.4.1 is wire-compatible with v0.4.0. Nodes can be upgraded one at a time
with no coordinated restart, though one behavior change below is worth
reading before you start a rolling upgrade.
## At a glance
- `node.bloom.max_inbound_fpr` default moves from `0.10` to `0.20`.
- The `parent_switched` metric counter is gone. Use `parent_switches`.
- Spanning tree no longer serves stale coordinates after a parent link is
lost through peer removal.
- Discovery no longer loosens a path MTU clamp it had correctly tightened.
- Bloom probing and identity operations do measurably less work per call,
with identical results.
## Behavior changes worth flagging
### The inbound filter FPR cap default doubles again
`node.bloom.max_inbound_fpr` goes from `0.10` to `0.20`. The cap rejects
inbound `FilterAnnounce` frames whose advertised false positive rate
exceeds it. On the fixed 1 KB, k=5 filter, `0.10` corresponds to a fill of
0.631 and roughly 1,630 reachable entries, and the busiest nodes'
aggregates had started reaching that ceiling as the mesh grew. `0.20`
corresponds to a fill of 0.7248 and roughly 2,114 entries.
Be aware that this is the second time in two releases that this default
has doubled, for the same reason both times. That is worth stating plainly
rather than repeating the previous release's framing: raising the cap buys
headroom, it does not fix anything. The real constraint is the fixed 1 KB
filter size, which is a protocol constant. The structural remedy is the v2
filter work, where filter capacity scales with the mesh instead of being
pinned. This release is an interim step to keep legitimate aggregates from
being rejected until that lands. It is not the start of a pattern of
raising the cap once per release, and if you are sizing capacity planning
around this number, plan against the v2 work rather than against a third
raise.
The antipoison property the cap exists for is preserved. A saturated or
deliberately poisoned filter still presents an FPR near 100% and is still
rejected.
**This matters during a rolling upgrade.** A v0.4.1 node accepts a
`FilterAnnounce` with a derived FPR between 0.10 and 0.20; a v0.4.0 node
drops the same frame, and the drop is silent on the wire with no NACK. The
cap also gates the mesh size estimator, which declines to produce a value
when any contributing filter is over the cap. So while a mesh is partly
upgraded, upgraded and not-yet-upgraded nodes can legitimately report
different mesh sizes, or one can report a size while the other reports
unknown. This resolves once every node is on v0.4.1. If you want to avoid
the window entirely, set `node.bloom.max_inbound_fpr: 0.10` explicitly in
your config before upgrading and remove it after the last node is done.
### The `parent_switched` counter is removed
`parent_switched` was incremented on the line immediately before
`parent_switches` at every site and never independently, so the two
counters always held the same value. `parent_switched` is now gone from
the tree metrics, the control socket snapshot, and the `fipstop` tree
view. `parent_switches` remains and is unchanged.
If you scrape the control socket, or have dashboards or alerts referencing
`parent_switched`, point them at `parent_switches`. Anything still asking
for `parent_switched` will find nothing rather than a zero.
## Notable bug fixes
### Stale coordinates after losing a parent through peer removal
When a node's parent link dropped via peer removal, the node correctly
reparented or self-rooted, but skipped the coordinate cache invalidation
that every other position-change path performs. Cached entries for
downstream destinations kept the node's old coordinate prefix. This did
not self-correct the way a stale cache entry normally would: routing
access refreshes an entry's TTL, so an entry that was actively being
routed through never expired, and was only fixed by an unrelated fresh
insert. Both invalidation classes now run on this path, matching the
loop-detection branch.
### Discovery could loosen a tightened path MTU clamp
An originator handling a `LookupResponse` overwrote its cached path MTU
unconditionally. If a reactive `MtuExceeded` or `PathMtuNotification` had
already taught it a tighter value, a later, looser discovery estimate
would clobber that and re-loosen the clamp, risking a return to dropped
oversized packets. The cached and received values are now compared and the
tighter one is kept.
## Upgrade notes
This is a drop-in upgrade from v0.4.0 with no wire format change, no
config migration, and no coordinated restart. Upgrade nodes in whatever
order you like.
Two things to do rather than assume:
1. If you monitor `parent_switched`, move to `parent_switches` before
upgrading, or your dashboards will go blank rather than error.
2. During the rolling window, expect upgraded and not-yet-upgraded nodes
to potentially disagree about mesh size, per the FPR cap section above.
This is expected and self-resolves. Do not chase it as a bug unless it
persists after every node reports `0.4.1`.
If you have pinned `node.bloom.max_inbound_fpr` explicitly in your config,
your setting is honored and nothing changes for you. The change only
affects nodes taking the default.
Downgrading to v0.4.0 is supported and needs no special handling.
## Getting v0.4.1
- **Linux x86_64 / aarch64**: `.deb` and tarball at the
[v0.4.1 release page](https://github.com/jmcorgan/fips/releases/tag/v0.4.1).
- **Arch Linux**: `fips` from the AUR.
- **macOS**: `.pkg` at the v0.4.1 release page.
- **Windows**: ZIP at the v0.4.1 release page.
- **OpenWrt**: `.ipk` (OpenWrt 24.x and earlier) or `.apk` (OpenWrt 25+)
at the v0.4.1 release page.
- **From source**: `cargo build --release` from a checkout of the v0.4.1
tag (Rust 1.94.1 per `rust-toolchain.toml`; `libclang-dev` is a
required Linux build prerequisite).
- **Nix / NixOS**: `nix build .#fips` from a checkout of the v0.4.1 tag
builds the binaries from source with the pinned toolchain and no manual
prerequisites (see the Nix section of `packaging/README.md`).
The full per-commit changelog lives in
[`CHANGELOG.md`](../../CHANGELOG.md). Issues and discussion at
[github.com/jmcorgan/fips](https://github.com/jmcorgan/fips).
## Contributors
Thanks to everyone who contributed code, packaging work, bug reports, or
reviews to this release.
- [@jcorgan](https://github.com/jmcorgan): release shepherd, spanning-tree
and discovery fixes, bloom and identity performance work, antipoison cap
change, and testing.
docs/tutorials/ground-up-mesh.md
+9 -9
View File
@@ -71,7 +71,7 @@ supplies the rest:
- **Addressing**: the `fips0` adapter takes an `fd97:...` ULA
derived from the npub. No DHCP. No SLAAC. The address is
cryptographically tied to the identity.
- **Discovery**: each daemon broadcasts a small beacon on the
- **Neighbor detection**: each daemon broadcasts a small beacon on the
link advertising its npub; the other daemon's listener picks
it up and dials in over the same link.
- **Routing**: the FIPS mesh layer builds its own spanning tree
@@ -177,7 +177,7 @@ different interface names — that is normal.
Edit `/etc/fips/fips.yaml` on **both** nodes. Under
`transports:`, add an `ethernet:` block. The key settings are
the four discovery flags — both nodes must opt in to all four,
the four neighbor flags — both nodes must opt in to all four,
and they default to off:
```yaml
@@ -185,7 +185,7 @@ transports:
ethernet:
interface: "<eth>" # the name from Step 1
announce: true # broadcast our beacon on the link
discovery: true # listen for beacons (default; shown for clarity)
listen: true # listen for beacons (default; shown for clarity)
auto_connect: true # dial peers we discover
accept_connections: true # accept dial-ins from peers we discover
```
@@ -194,7 +194,7 @@ Each flag does one thing:
- `announce: true` — emit a small beacon every
`beacon_interval_secs` (default 30s) carrying our npub.
- `discovery: true` — listen for incoming beacons; populate a
- `listen: true` — listen for incoming beacons; populate a
candidate-peer list keyed by source MAC and observed npub.
- `auto_connect: true` — when we see a beacon from an npub
we have not yet peered with, initiate the outbound Noise
@@ -218,7 +218,7 @@ is "all four flags on both ends."
> lan:
> interface: "eth0"
> announce: true
> discovery: true
> listen: true
> auto_connect: true
> accept_connections: true
> dongle:
@@ -227,7 +227,7 @@ is "all four flags on both ends."
> # ...
> ```
>
> Each named instance runs its own socket and discovery state.
> Each named instance runs its own socket and neighbor state.
> A single ground-up link only needs the flat form shown
> first; named instances become useful when the same node
> bridges multiple physical segments.
@@ -390,7 +390,7 @@ What you do need on the AP side:
networks and "secure" enterprise APs ship with it on.
When client isolation is on, the AP refuses to forward
station-to-station frames — the broadcast beacons never
arrive at the other node, and discovery fails silently.
arrive at the other node, and neighbor detection fails silently.
If beacons aren't crossing, this is the first thing to
check.
@@ -401,7 +401,7 @@ adapter name.
### Bluetooth LE (experimental but works)
BLE is a separate transport (`transports.ble.*`) with its own
discovery model — L2CAP advertisements rather than raw L2
neighbor-detection model — L2CAP advertisements rather than raw L2
broadcasts. The shape of the tutorial is the same (advertise +
scan + auto-connect + accept), but the prerequisites are
different: BlueZ, `bluetoothd`, an HCI adapter, and the
@@ -424,7 +424,7 @@ Windows builds skip it.
a radio link), `CAP_NET_RAW`, and a few config flags on each
end are sufficient. The mesh supplies its own identity,
addressing, discovery, and routing.
- **Discovery is a four-flag opt-in.** `announce`, `discovery`,
- **Neighbor detection is a four-flag opt-in.** `announce`, `listen`,
`auto_connect`, and `accept_connections` each control one
thing; both ends must agree before a link will form.
- **The two modes coexist.** Overlay peers and ground-up peers
docs/tutorials/resolve-peers-via-nostr.md
+1 -1
View File
@@ -178,7 +178,7 @@ The resolution itself happens at debug-log level, so you will
not see it in the default-level journal. The user-facing way to
confirm everything worked is `fipsctl show peers` in the next
step. (To watch the resolution in the journal, run the daemon
manually with `RUST_LOG=fips::discovery::nostr=debug`; not
manually with `RUST_LOG=fips::nostr=debug`; not
necessary for this tutorial.)
## Step 5: Verify the resolved endpoint
packaging/common/fips.yaml
+18 -9
View File
@@ -10,12 +10,21 @@ node:
#
# Or set an explicit key (overrides persistent):
# nsec: "nsec1..."
discovery:
# Optional Nostr-mediated overlay endpoint discovery.
# Mesh-lookup protocol (node.lookup.*): the overlay coordinate-lookup engine
# (mesh address -> coordinates). Defaults shown; uncomment to override.
# lookup:
# ttl: 64
# attempt_timeouts_secs: [1, 2, 4, 8]
# recent_expiry_secs: 10
# backoff_base_secs: 0
# backoff_max_secs: 0
# forward_min_interval_secs: 2
rendezvous:
# Optional Nostr-mediated overlay endpoint rendezvous.
# nostr:
# enabled: true
# policy: configured_only # disabled | configured_only | open
# open_discovery_max_pending: 64 # caps queued open-discovery retries
# open_discovery_max_pending: 64 # caps queued open-rendezvous retries
# app: "fips-overlay-v1"
# advertise: true
# advert_relays:
@@ -34,17 +43,17 @@ node:
# - "stun:stun.cloudflare.com:3478"
# - "stun:global.stun.twilio.com:3478"
#
# Optional mDNS-based LAN discovery for sub-second same-LAN pairing.
# Optional mDNS-based LAN rendezvous for sub-second same-LAN pairing.
# Opt-in (default false): default-off avoids a per-LAN identity
# broadcast on nodes that have deliberately disabled other discovery
# broadcast on nodes that have deliberately disabled other rendezvous
# channels, and avoids any multicast surprise on upgrade. Requires an
# operational UDP transport (the advertised port is the one peers dial).
# lan:
# enabled: false
# # Optional application/network scope carried in the LAN-only TXT
# # record. Browsers that set a scope ignore adverts for other scopes.
# # Kept separate from the Nostr discovery `app` tag so relay-visible
# # adverts can stay generic while LAN discovery stays per-private-network.
# # Kept separate from the Nostr rendezvous `app` tag so relay-visible
# # adverts can stay generic while LAN rendezvous stays per-private-network.
# # scope: "lab-floor-3"
# # Advanced: overrides the mDNS service type. Leave unset in normal
# # use — only needed to run multiple isolated services on one
@@ -89,7 +98,7 @@ transports:
# Ethernet transport — uncomment and set your interface name.
# ethernet:
# interface: "eth0"
# discovery: true
# listen: true
# announce: true
# auto_connect: true
# accept_connections: true
@@ -147,5 +156,5 @@ peers: []
# - transport: udp
# addr: "test-us01.fips.network:2121" # IP or hostname (e.g., "peer.example.com:2121")
# - transport: udp
# addr: "nat" # Use node.discovery.nostr for Nostr/STUN hole punching
# addr: "nat" # Use node.rendezvous.nostr for Nostr/STUN hole punching
# connect_policy: auto_connect
packaging/debian/build-deb.sh
+44 -2
View File
@@ -2,6 +2,7 @@
# Build a .deb package for FIPS using cargo-deb.
#
# Usage: ./build-deb.sh [--target <triple>] [--version <version>] [--no-build]
# [--features <list>]
#
# Prerequisites: cargo-deb (install with: cargo install cargo-deb)
# Output: deploy/fips_<version>_<arch>.deb
@@ -19,6 +20,9 @@ Options:
--target <triple> Rust target triple to build/package
--version <version> Override Debian package version
--no-build Package existing binaries without running cargo build
--features <list> Cargo features to build with (comma-separated). Marks the
auto-derived Version so the package is distinguishable
from a default build of the same commit.
-h, --help Show this help
EOF
}
@@ -26,6 +30,7 @@ EOF
TARGET_TRIPLE=""
VERSION_OVERRIDE=""
NO_BUILD=0
FEATURES=""
while [[ $# -gt 0 ]]; do
case "$1" in
@@ -41,6 +46,10 @@ while [[ $# -gt 0 ]]; do
NO_BUILD=1
shift
;;
--features)
FEATURES="${2:?missing value for --features}"
shift 2
;;
-h|--help)
usage
exit 0
@@ -53,6 +62,16 @@ while [[ $# -gt 0 ]]; do
esac
done
# A feature build that skips the build step would stamp a feature-marked Version
# onto whatever binaries already sit in target/, which is the one outcome the
# marking exists to prevent. Refuse rather than emit a package that misdescribes
# itself.
if [[ -n "${FEATURES}" && "${NO_BUILD}" -eq 1 ]]; then
echo "--features cannot be combined with --no-build: the features would not" >&2
echo "reach the binaries, but the Version would claim they had." >&2
exit 1
fi
cd "${PROJECT_ROOT}"
# Ensure cargo-deb is available
@@ -81,13 +100,33 @@ if [[ -z "${VERSION_OVERRIDE}" ]]; then
if [[ -n "$(git status --porcelain 2>/dev/null)" ]]; then
DIRTY_SUFFIX=".dirty"
fi
# Debian Version: <upstream>~dev+git<YYYYMMDD>.<sha>[.dirty]-1
# A feature build of a given commit is a different package from the
# default build of that same commit, but nothing else in this version
# says so: the crate version, the date and the sha are all identical.
# Without a marker the two are byte-identical versions, so installing
# one over the other is an apt no-op (the very failure the per-commit
# version above exists to prevent) and the node offers no way to tell
# which one it is running. Underscores and commas are not legal in a
# Debian version, so the feature list is folded to dots.
FEATURE_SUFFIX=""
if [[ -n "${FEATURES}" ]]; then
FEATURE_SUFFIX="+$(printf '%s' "${FEATURES}" | tr -c 'a-zA-Z0-9.' '.')"
fi
# Debian Version: <upstream>~dev+git<YYYYMMDD>.<sha>[.dirty][+<features>]-1
# The "~" makes every dev build sort BEFORE the eventual tagged
# release; the date+sha makes consecutive dev builds compare as
# different versions; the trailing "-1" is the Debian revision.
VERSION_OVERRIDE="${BASE_VERSION}~dev+git${GIT_DATE}.${GIT_SHA}${DIRTY_SUFFIX}-1"
# The feature suffix sorts ABOVE the unsuffixed build, so installing a
# feature build is an upgrade and reverting to the default build is a
# downgrade — which apt refuses without being told to, and `dpkg -i`
# performs. Revert with `dpkg -i`, not `apt install`.
VERSION_OVERRIDE="${BASE_VERSION}~dev+git${GIT_DATE}.${GIT_SHA}${DIRTY_SUFFIX}${FEATURE_SUFFIX}-1"
echo "Auto-derived dev Version: ${VERSION_OVERRIDE}"
fi
elif [[ -n "${FEATURES}" ]]; then
echo "Warning: --version was given with --features, so the Version carries no" >&2
echo "feature marker and this package is indistinguishable from a default" >&2
echo "build of the same commit. Mark it yourself if that matters." >&2
fi
# Build the .deb package
@@ -105,6 +144,9 @@ fi
if [[ "${NO_BUILD}" -eq 1 ]]; then
cargo_args+=(--no-build)
fi
if [[ -n "${FEATURES}" ]]; then
cargo_args+=(--features "${FEATURES}")
fi
cargo "${cargo_args[@]}"
# Move output to deploy/
packaging/debian/fips.service
+5
View File
@@ -19,6 +19,11 @@ RestartSec=5
RuntimeDirectory=fips
RuntimeDirectoryMode=0750
# Log directory (/var/log/fips/), where the built-in tick-body profiler writes
# its capture files. Declared so systemd creates it on start and removes it on
# purge; the daemon runs as root and already has access without it.
LogsDirectory=fips
# Security hardening (daemon runs as root for TUN and raw sockets)
ProtectHome=yes
PrivateTmp=yes
packaging/openwrt-apk/build-apk.sh
+2
View File
@@ -182,6 +182,8 @@ install -m 0755 "$RELEASE_DIR/fips" "$STAGE_DIR/usr/bin/fips"
install -m 0755 "$RELEASE_DIR/fipsctl" "$STAGE_DIR/usr/bin/fipsctl"
install -m 0755 "$RELEASE_DIR/fipstop" "$STAGE_DIR/usr/bin/fipstop"
install -m 0755 "$RELEASE_DIR/fips-gateway" "$STAGE_DIR/usr/bin/fips-gateway"
install -m 0755 "$FILES_DIR/usr/bin/fips-mesh-setup" "$STAGE_DIR/usr/bin/fips-mesh-setup"
install -m 0755 "$FILES_DIR/usr/bin/fips-ap-setup" "$STAGE_DIR/usr/bin/fips-ap-setup"
install -d "$STAGE_DIR/etc/init.d"
install -m 0755 "$FILES_DIR/etc/init.d/fips" "$STAGE_DIR/etc/init.d/fips"
packaging/openwrt-ipk/Makefile
+6
View File
@@ -96,6 +96,12 @@ define Package/fips/install
$(INSTALL_BIN) $(RUST_RELEASE_DIR)/fipstop $(1)/usr/bin/fipstop
$(INSTALL_BIN) $(RUST_RELEASE_DIR)/fips-gateway $(1)/usr/bin/fips-gateway
# 802.11s mesh backhaul setup helper
$(INSTALL_BIN) $(CURDIR)/files/usr/bin/fips-mesh-setup $(1)/usr/bin/fips-mesh-setup
# Open "FIPS" access SSID setup helper
$(INSTALL_BIN) $(CURDIR)/files/usr/bin/fips-ap-setup $(1)/usr/bin/fips-ap-setup
# procd init script
$(INSTALL_DIR) $(1)/etc/init.d
$(INSTALL_BIN) $(CURDIR)/files/etc/init.d/fips $(1)/etc/init.d/fips
packaging/openwrt-ipk/README.md
+1
View File
@@ -14,6 +14,7 @@ For ad-hoc deployment without the build system, see
| `/usr/bin/fipsctl` | CLI control tool (`fipsctl show peers`, `fipsctl show links`, …) |
| `/usr/bin/fipstop` | Live TUI dashboard |
| `/usr/bin/fips-gateway` | Outbound LAN gateway service (not started by default) |
| `/usr/bin/fips-mesh-setup` | Opt-in helper — creates an open 802.11s mesh interface for router↔router backhaul |
| `/etc/init.d/fips` | procd service for the daemon (auto-start, crash respawn) |
| `/etc/init.d/fips-gateway` | procd service for the gateway (disabled by default) |
| `/etc/fips/fips.yaml` | Node configuration (edit before first start) |
packaging/openwrt-ipk/build-ipk.sh
+2
View File
@@ -161,6 +161,8 @@ install -m 0755 "$RELEASE_DIR/fips" "$DATA_DIR/usr/bin/fips"
install -m 0755 "$RELEASE_DIR/fipsctl" "$DATA_DIR/usr/bin/fipsctl"
install -m 0755 "$RELEASE_DIR/fipstop" "$DATA_DIR/usr/bin/fipstop"
install -m 0755 "$RELEASE_DIR/fips-gateway" "$DATA_DIR/usr/bin/fips-gateway"
install -m 0755 "$FILES_DIR/usr/bin/fips-mesh-setup" "$DATA_DIR/usr/bin/fips-mesh-setup"
install -m 0755 "$FILES_DIR/usr/bin/fips-ap-setup" "$DATA_DIR/usr/bin/fips-ap-setup"
install -d "$DATA_DIR/etc/init.d"
install -m 0755 "$FILES_DIR/etc/init.d/fips" "$DATA_DIR/etc/init.d/fips"
packaging/openwrt-ipk/files/etc/fips/fips.yaml
+79 -8
View File
@@ -10,12 +10,21 @@ node:
#
# Or set an explicit key (overrides persistent):
# nsec: "nsec1..."
discovery:
# Optional Nostr-mediated overlay endpoint discovery.
# Mesh-lookup protocol (node.lookup.*): the overlay coordinate-lookup engine
# (mesh address -> coordinates). Defaults shown; uncomment to override.
# lookup:
# ttl: 64
# attempt_timeouts_secs: [1, 2, 4, 8]
# recent_expiry_secs: 10
# backoff_base_secs: 0
# backoff_max_secs: 0
# forward_min_interval_secs: 2
rendezvous:
# Optional Nostr-mediated overlay endpoint rendezvous.
# nostr:
# enabled: true
# policy: configured_only # disabled | configured_only | open
# open_discovery_max_pending: 64 # caps queued open-discovery retries
# open_discovery_max_pending: 64 # caps queued open-rendezvous retries
# app: "fips-overlay-v1"
# advertise: true
# advert_relays:
@@ -34,6 +43,14 @@ node:
# - "stun:stun.cloudflare.com:3478"
# - "stun:global.stun.twilio.com:3478"
# mDNS/DNS-SD peer rendezvous on the local link. Ships commented (the
# daemon default is off); 'fips-ap-setup' uncomments it when creating
# the access SSID — phone FIPS apps cannot see raw-Ethernet beacons,
# so mDNS is how they find this router's daemon. Daemon-wide switch,
# left enabled on 'fips-ap-setup remove'.
# lan:
# enabled: true
tun:
enabled: true
name: fips0
@@ -55,7 +72,11 @@ dns:
transports:
udp:
bind_addr: "0.0.0.0:2121"
# Dual-stack wildcard, not "0.0.0.0": access-SSID clients (phones) learn
# this node's addresses from the mDNS advert and prefer the IPv6
# link-local — a v4-only bind silently drops their Noise msg1.
# OpenWrt is Linux (bindv6only=0), so "[::]" accepts v4 too.
bind_addr: "[::]:2121"
# advertise_on_nostr: true
# public: false # false => advertise udp:nat; true => advertise bound host:port
# accept_connections: true # default; refuse inbound msg1 when false
@@ -74,23 +95,73 @@ transports:
ethernet:
wan:
interface: "eth0"
discovery: true
listen: true
announce: true
auto_connect: true
accept_connections: true
wwan:
interface: "phy0-sta0"
discovery: true
listen: true
announce: true
auto_connect: true
accept_connections: true
lan:
interface: "br-lan"
discovery: true
listen: true
announce: true
auto_connect: true
accept_connections: true
# 802.11s mesh backhaul between FIPS routers. These entries ship
# commented out so a stock install that never creates fips-mesh*
# logs no per-boot "interface missing" bind warning. Running
# 'fips-mesh-setup <radio>' creates the interface AND uncomments the
# matching block here (once per radio; radio0 -> fips-mesh0, radio1 ->
# fips-mesh1); 'fips-mesh-setup remove' re-comments it. Restart fips
# after — a transport whose interface is missing at startup is skipped,
# not retried. Dual-band routers can mesh on both bands at once —
# failover, not multipath: FIPS keeps one active link per peer, the
# other band stands by. The mesh runs OPEN (no SAE) with 802.11s
# forwarding off: FIPS's Noise handshake is the encryption and
# authentication, and FIPS is the routing layer. See
# docs/how-to/set-up-80211s-mesh-backhaul.md.
# mesh0:
# interface: "fips-mesh0"
# discovery: true
# announce: true
# auto_connect: true
# accept_connections: true
# mesh1:
# interface: "fips-mesh1"
# discovery: true
# announce: true
# auto_connect: true
# accept_connections: true
# Open "!FIPS" access SSID for phones and laptops running FIPS. These
# entries ship commented out so a stock install that never creates
# fips-ap* logs no per-boot "interface missing" bind warning. Running
# 'fips-ap-setup <radio>' creates the interface AND uncomments the
# matching block here (once per radio; radio0 -> fips-ap0, radio1 ->
# fips-ap1); 'fips-ap-setup remove' re-comments it. Restart fips after
# — a transport whose interface is missing at startup is skipped, not
# retried. The SSID is OPEN and isolated on purpose: FIPS's Noise
# handshake is the only security layer, and associated clients reach
# nothing but the FIPS handshake surface. See
# docs/how-to/set-up-open-access-ssid.md.
# ap0:
# interface: "fips-ap0"
# discovery: true
# announce: true
# auto_connect: true
# accept_connections: true
# ap1:
# interface: "fips-ap1"
# discovery: true
# announce: true
# auto_connect: true
# accept_connections: true
# Bluetooth Low Energy transport — requires BlueZ and the 'ble' feature.
# ble:
# adapter: "hci0"
@@ -121,5 +192,5 @@ peers: []
# - transport: udp
# addr: "test-us01.fips.network:2121" # IP or hostname (e.g., "peer.example.com:2121")
# - transport: udp
# addr: "nat" # Use node.discovery.nostr for Nostr/STUN hole punching
# addr: "nat" # Use node.rendezvous.nostr for Nostr/STUN hole punching
# connect_policy: auto_connect
packaging/openwrt-ipk/files/usr/bin/fips-ap-setup
Executable
+412
View File
@@ -0,0 +1,412 @@
#!/bin/sh
# fips-ap-setup — configure the open "FIPS" access SSID for phones/laptops.
#
# Usage:
# fips-ap-setup <radio> [ssid] e.g. fips-ap-setup radio0
# fips-ap-setup remove [radio] no radio: remove all instances
#
# Creates an open AP on the given radio so client devices running FIPS can
# reach the router. Every FIPS router broadcasts the SAME SSID ("!FIPS" by
# default — the leading '!' sorts it to the top of alphabetically ordered
# network pickers): same SSID + unique BSSIDs is one standard ESS, so a
# phone saves the network once and roams between all FIPS routers natively.
#
# - encryption 'none' — the AP is OPEN on purpose. FIPS's Noise IK
# handshake authenticates and encrypts everything above the radio, and
# the security type must be uniform across ALL routers anyway: clients
# key a saved network on SSID + security type, so one router with a PSK
# splits the ESS into a different saved network. A stranger can
# associate AND form a FIPS peer link — that is the point of open
# access. The Noise handshake authenticates each link (no
# impersonation of another identity, no MITM); it does NOT gate who
# may peer. Admission is open up to the daemon's max-peers cap; the
# firewall zone below is what confines every client to the FIPS
# overlay (no path to br-lan or the WAN).
# - DHCPv4 + RA IPv6 — dnsmasq serves DHCPv4 from a FIXED subnet,
# 10.21.<N>.0/24 (echoes FIPS port 2121), identical on every router:
# a roaming phone keeps its lease across routers, and dnsmasq's
# authoritative mode (the OpenWrt default) ACKs the renew a foreign
# router never issued. odhcpd additionally announces a ULA prefix in
# router advertisements (stateless SLAAC); DHCPv6 stays off. FIPS
# itself only needs link-local + mDNS, but client provisioning checks
# (Android disconnects without an RA or a DHCP offer) and plain
# laptops both want a real address. The network provides no internet,
# so phones mark it unvalidated and keep cellular as the default
# route while staying associated.
# - ISOLATED — own network and firewall zone: no path to
# br-lan, no forwarding to the WAN, and AP client isolation on.
# Associated clients reach only the FIPS handshake surface.
#
# Interfaces are named per radio index (radio0 -> fips-ap0, radio1 ->
# fips-ap1). Unlike the 802.11s backhaul there is NO same-channel
# constraint — clients scan when they roam, so every router picks its
# access channels freely.
#
# The shipped /etc/fips/fips.yaml carries 'ap0' and 'ap1' entries under
# 'transports.ethernet' bound to these names, but commented out — a stock
# install that never creates fips-ap* then logs no bind warning. This
# helper uncomments the matching entry when it creates an interface and
# re-comments it on remove, so the daemon binds the transport without a
# manual config edit. It also uncomments the node.rendezvous.lan block
# (mDNS/DNS-SD — how phone FIPS apps discover the daemon); that switch is
# daemon-wide and stays on at remove. After an interface is up, restart
# fips.
# See docs/how-to/set-up-open-access-ssid.md for the full guide.
DEFAULT_SSID="!FIPS"
CONFIG="/etc/fips/fips.yaml"
# Replace $CONFIG with the rewritten $CONFIG.tmp. Force mode 0600 first: the
# package installs fips.yaml 0600 (it may hold an inline 'nsec' private key),
# and a fresh tmp file would otherwise land world-readable after the move.
ap_config_write() {
chmod 600 "$CONFIG.tmp" && mv "$CONFIG.tmp" "$CONFIG"
}
# Uncomment the 'ap<idx>' transports.ethernet block in $CONFIG (created by
# 'fips-ap-setup'). Reversible with ap_config_disable. Returns:
# 0 enabled (or already active) 1 no config file 2 no such block
ap_config_enable() {
idx="$1"
[ -f "$CONFIG" ] || return 1
grep -q "^ ap$idx:" "$CONFIG" && return 0
grep -q "^ # ap$idx:" "$CONFIG" || return 2
awk -v idx="$idx" '
$0 ~ ("^ # ap" idx ":[ \t]*$") { blk = 1; sub(/^ # /, " "); print; next }
blk && /^ # / { sub(/^ # /, " "); print; next }
{ blk = 0; print }
' "$CONFIG" > "$CONFIG.tmp" && ap_config_write
}
# Uncomment the 'lan' block under node.rendezvous in $CONFIG — the daemon's
# mDNS/DNS-SD responder+browser. Phone FIPS apps cannot open raw-Ethernet
# sockets, so mDNS is how they find this router's daemon. The match is
# scoped to node.rendezvous: transports.ethernet also has a 'lan' entry at
# the same indent. Daemon-wide switch — enabled here, deliberately NOT
# re-commented on remove (other transports use it once on). Returns:
# 0 enabled (or already active) 1 no config file 2 no such block
lan_rendezvous_enable() {
[ -f "$CONFIG" ] || return 1
state="$(awk '
/^[A-Za-z_]/ { top = $1 }
top == "node:" && /^ [A-Za-z_]/ { sec = $1 }
top == "node:" && sec == "rendezvous:" && /^ lan:[ \t]*$/ { print "active"; exit }
top == "node:" && sec == "rendezvous:" && /^ # lan:[ \t]*$/ { print "commented"; exit }
' "$CONFIG")"
case "$state" in
active) return 0 ;;
commented) ;;
*) return 2 ;;
esac
awk '
/^[A-Za-z_]/ { top = $1 }
top == "node:" && /^ [A-Za-z_]/ { sec = $1 }
top == "node:" && sec == "rendezvous:" && $0 ~ /^ # lan:[ \t]*$/ { blk = 1; sub(/^ # /, " "); print; next }
blk && /^ # / { sub(/^ # /, " "); print; next }
{ blk = 0; print }
' "$CONFIG" > "$CONFIG.tmp" && ap_config_write
}
# Re-comment the 'ap<idx>' block so the daemon stops binding it (and stops
# warning about the now-missing interface). Inverse of ap_config_enable.
ap_config_disable() {
idx="$1"
[ -f "$CONFIG" ] || return 1
grep -q "^ ap$idx:" "$CONFIG" || return 0
awk -v idx="$idx" '
$0 ~ ("^ ap" idx ":[ \t]*$") { blk = 1; sub(/^ /, " # "); print; next }
blk && /^ / { sub(/^ /, " # "); print; next }
{ blk = 0; print }
' "$CONFIG" > "$CONFIG.tmp" && ap_config_write
}
usage() {
echo "Usage: fips-ap-setup <radio> [ssid]" >&2
echo " fips-ap-setup remove [radio]" >&2
echo "Radios on this device:" >&2
uci show wireless 2>/dev/null | sed -n "s/^wireless\.\([^.]*\)=wifi-device$/ \1/p" >&2
exit 1
}
# List the UCI section names of fips-managed access-point wifi-ifaces.
ap_sections() {
uci show wireless 2>/dev/null | sed -n "s/^wireless\.\(fips_ap[^.=]*\)=wifi-iface$/\1/p"
}
# ---------------------------------------------------------------------------
# remove [radio] — delete the wireless, network, dhcp, and firewall sections
# created below
# ---------------------------------------------------------------------------
if [ "$1" = "remove" ]; then
if [ -n "$2" ]; then
SECTIONS="fips_ap_$(printf '%s' "$2" | tr -c 'a-zA-Z0-9_' '_')"
else
SECTIONS="$(ap_sections)"
fi
[ -n "$SECTIONS" ] || {
echo "No fips access-point instances configured."
exit 0
}
for section in $SECTIONS; do
ifname="$(uci -q get "wireless.$section.ifname")"
uci -q delete "wireless.$section"
uci -q delete "network.$section"
uci -q delete "dhcp.$section"
uci -q del_list "firewall.fips_ap.network=$section"
# Re-comment the matching ap<N> transport in fips.yaml so the
# daemon stops warning about the interface we just removed.
idx="$(printf '%s' "$ifname" | sed -n 's/.*[^0-9]\([0-9]\{1,\}\)$/\1/p')"
[ -n "$idx" ] && ap_config_disable "$idx"
echo "Removed ${ifname:-$section}."
done
# Drop the shared zone and its rules once the last instance is gone.
if [ -z "$(uci -q get firewall.fips_ap.network)" ]; then
uci -q delete firewall.fips_ap
uci -q delete firewall.fips_ap_icmpv6
uci -q delete firewall.fips_ap_dhcpv4
uci -q delete firewall.fips_ap_mdns
uci -q delete firewall.fips_ap_fips_udp
uci -q delete firewall.fips_ap_fips_tcp
fi
uci commit wireless
uci commit network
uci commit dhcp
uci commit firewall
wifi reload
/etc/init.d/dnsmasq reload
/etc/init.d/odhcpd reload
/etc/init.d/firewall reload
echo "Restart fips: /etc/init.d/fips restart"
exit 0
fi
RADIO="$1"
SSID="${2:-$DEFAULT_SSID}"
[ -n "$RADIO" ] || usage
if [ "$(uci -q get "wireless.$RADIO")" != "wifi-device" ]; then
echo "Error: '$RADIO' is not a wifi-device in /etc/config/wireless." >&2
usage
fi
# One instance per radio: section fips_ap_<radio>, netdev fips-ap<N>
# where N is the radio's trailing index (radio0 -> fips-ap0). For radios
# named without a trailing number, fall back to the first free index.
SECTION="fips_ap_$(printf '%s' "$RADIO" | tr -c 'a-zA-Z0-9_' '_')"
IDX="$(printf '%s' "$RADIO" | sed -n 's/.*[^0-9]\([0-9]\{1,\}\)$/\1/p')"
[ -n "$IDX" ] || IDX="$(printf '%s' "$RADIO" | sed -n 's/^\([0-9]\{1,\}\)$/\1/p')"
if [ -z "$IDX" ]; then
IDX=0
while uci show wireless 2>/dev/null | grep -q "\.ifname='fips-ap$IDX'"; do
IDX=$((IDX + 1))
done
fi
AP_IFNAME="fips-ap$IDX"
# Refuse a name collision from another radio's instance (e.g. two radios
# whose names end in the same digit) rather than silently hijacking it.
OWNER="$(uci show wireless 2>/dev/null \
| sed -n "s/^wireless\.\(fips_ap[^.=]*\)\.ifname='$AP_IFNAME'$/\1/p")"
if [ -n "$OWNER" ] && [ "$OWNER" != "$SECTION" ]; then
echo "Error: $AP_IFNAME is already used by section '$OWNER'." >&2
echo "Remove it first: fips-ap-setup remove" >&2
exit 1
fi
# ---------------------------------------------------------------------------
# Wireless: open AP with client isolation. Clients of the same AP cannot
# exchange L2 frames directly — two FIPS phones on one router still reach
# each other through the router at the overlay layer. Isolation is an L2
# control only: a stranger who peers is an overlay peer like any other, so
# the FIPS overlay (not L2) is the trust boundary between clients.
# ---------------------------------------------------------------------------
uci -q delete "wireless.$SECTION"
uci set "wireless.$SECTION=wifi-iface"
uci set "wireless.$SECTION.device=$RADIO"
uci set "wireless.$SECTION.mode=ap"
uci set "wireless.$SECTION.ssid=$SSID"
uci set "wireless.$SECTION.encryption=none"
uci set "wireless.$SECTION.isolate=1"
uci set "wireless.$SECTION.ifname=$AP_IFNAME"
uci set "wireless.$SECTION.network=$SECTION"
# Radios ship disabled on fresh OpenWrt installs; a disabled radio would
# leave the AP down with no error anywhere visible.
if [ "$(uci -q get "wireless.$RADIO.disabled")" = "1" ]; then
echo "Note: enabling $RADIO (was disabled)."
uci -q delete "wireless.$RADIO.disabled"
fi
CHANNEL="$(uci -q get "wireless.$RADIO.channel")"
BAND="$(uci -q get "wireless.$RADIO.band")"
# ---------------------------------------------------------------------------
# Network: IPv4 from the fixed convention 10.21.<IDX>.1/24 — deterministic,
# so every router serving the same radio index lands on the same subnet and
# a roaming client's lease stays valid. IPv6 is a static ULA /64 so odhcpd
# has a prefix to announce; that space is per-router and disposable — a
# roaming phone SLAACs a fresh address on each router, and the FIPS overlay
# identity (not the IP) is the mobility anchor. The ULA is derived from the
# router's global ULA prefix; a re-run keeps the address already configured.
# ---------------------------------------------------------------------------
AP_ADDR="$(uci -q get "network.$SECTION.ip6addr")"
case "$AP_ADDR" in
fd*) ;; # keep the existing address on re-run
*)
ULA_BASE=""
ULA_PREFIX="$(uci -q get network.globals.ula_prefix)"
case "$ULA_PREFIX" in
fd*::/48) ULA_BASE="${ULA_PREFIX%::/48}" ;;
esac
if [ -z "$ULA_BASE" ]; then
HEX="$(head -c 5 /dev/urandom | hexdump -e '5/1 "%02x"')"
ULA_BASE="fd$(printf '%s' "$HEX" | cut -c1-2):$(printf '%s' "$HEX" | cut -c3-6):$(printf '%s' "$HEX" | cut -c7-10)"
echo "Note: no usable ULA prefix in network.globals — generated $ULA_BASE::/48 for this AP."
fi
# 64000 = 0xfa00 — high subnet IDs keep clear of br-lan's low
# ip6assign allocations from the same ULA prefix.
AP_ADDR="$ULA_BASE:$(printf '%04x' $((64000 + IDX)))::1/64"
;;
esac
AP_ADDR4="10.21.$IDX.1"
uci -q delete "network.$SECTION"
uci set "network.$SECTION=interface"
uci set "network.$SECTION.proto=static"
uci set "network.$SECTION.ipaddr=$AP_ADDR4"
uci set "network.$SECTION.netmask=255.255.255.0"
uci set "network.$SECTION.ip6addr=$AP_ADDR"
# ---------------------------------------------------------------------------
# DHCP/RA: dnsmasq DHCPv4 leases out of 10.21.<IDX>.0/24, plus router
# advertisements for the ULA (stateless SLAAC, no DHCPv6). ra_default '2'
# announces a default router even without an upstream default route:
# Android's provisioning wants address + route + DNS, and its validation
# probe then fails by design (no internet), so the phone keeps cellular as
# the default route. 'dhcpv4 server' is read by BOTH dnsmasq (the default
# DHCPv4 server) and odhcpd (serves v4 only when odhcpd.maindhcp is set),
# so either arrangement hands out leases.
# ---------------------------------------------------------------------------
uci -q delete "dhcp.$SECTION"
uci set "dhcp.$SECTION=dhcp"
uci set "dhcp.$SECTION.interface=$SECTION"
uci set "dhcp.$SECTION.ra=server"
uci set "dhcp.$SECTION.ra_default=2"
uci set "dhcp.$SECTION.dhcpv6=disabled"
uci set "dhcp.$SECTION.dhcpv4=server"
uci set "dhcp.$SECTION.start=10"
uci set "dhcp.$SECTION.limit=200"
# Authoritative is the OpenWrt default, but roaming correctness depends on
# it (a foreign router must ACK a lease it never issued), so pin it.
[ -n "$(uci -q get dhcp.@dnsmasq[0])" ] && uci set dhcp.@dnsmasq[0].authoritative=1
# ---------------------------------------------------------------------------
# Firewall: one shared 'fips_ap' zone for all instances. Everything is
# rejected except what a FIPS client needs — DHCPv4 (addressing), ICMPv6
# (SLAAC itself), mDNS (discovery), and the FIPS UDP/TCP transports (the
# handshake surface).
# The raw-Ethernet transport (EtherType 0x2121) is not IP and never
# traverses the firewall. No forwardings exist, so there is no path to
# br-lan or the WAN.
# ---------------------------------------------------------------------------
if [ "$(uci -q get firewall.fips_ap)" != "zone" ]; then
uci set firewall.fips_ap=zone
fi
uci set firewall.fips_ap.name=fips_ap
uci set firewall.fips_ap.input=REJECT
uci set firewall.fips_ap.output=ACCEPT
uci set firewall.fips_ap.forward=REJECT
uci -q del_list "firewall.fips_ap.network=$SECTION"
uci add_list "firewall.fips_ap.network=$SECTION"
# ap_rule <section-suffix> <name> <proto> [dest_port]
ap_rule() {
rule="firewall.fips_ap_$1"
uci -q delete "$rule"
uci set "$rule=rule"
uci set "$rule.name=$2"
uci set "$rule.src=fips_ap"
uci set "$rule.proto=$3"
uci set "$rule.target=ACCEPT"
[ -z "${4:-}" ] || uci set "$rule.dest_port=$4"
}
ap_rule icmpv6 "FIPS-AP-ICMPv6" icmp
uci set firewall.fips_ap_icmpv6.family=ipv6
ap_rule dhcpv4 "FIPS-AP-DHCPv4" udp 67
uci set firewall.fips_ap_dhcpv4.family=ipv4
ap_rule mdns "FIPS-AP-mDNS" udp 5353
ap_rule fips_udp "FIPS-AP-FIPS-UDP" udp 2121
ap_rule fips_tcp "FIPS-AP-FIPS-TCP" tcp 8443
uci commit wireless
uci commit network
uci commit dhcp
uci commit firewall
wifi reload
/etc/init.d/dnsmasq reload
/etc/init.d/odhcpd reload
/etc/init.d/firewall reload
# Enable the matching ap<N> transport in the shipped fips.yaml (it ships
# commented out). Tailor the restart hint to what we could do.
ap_config_enable "$IDX"
case $? in
0) TRANSPORT_NOTE="The ap$IDX transport in $CONFIG that binds '$AP_IFNAME' is
now uncommented and enabled." ;;
1) TRANSPORT_NOTE="No $CONFIG found — add a transports.ethernet entry binding
interface '$AP_IFNAME' by hand." ;;
*) TRANSPORT_NOTE="No 'ap$IDX' entry in $CONFIG — add a transports.ethernet
entry binding interface '$AP_IFNAME' by hand (copy the ap0 block)." ;;
esac
# Phones discover the daemon via mDNS, not raw-Ethernet beacons — make sure
# the daemon-wide mDNS rendezvous is on.
if lan_rendezvous_enable; then
MDNS_NOTE="node.rendezvous.lan (mDNS) is enabled — phone FIPS apps
discover this router via DNS-SD."
else
MDNS_NOTE="Could not enable mDNS in $CONFIG — set
'node.rendezvous.lan.enabled: true' by hand; phone FIPS apps rely
on it to discover this router."
fi
cat <<EOF
Created open access SSID '$SSID' as $AP_IFNAME on $RADIO \
(band ${BAND:-?}, channel ${CHANNEL:-auto}).
DHCPv4 on $AP_ADDR4/24 and RA IPv6 on $AP_ADDR — no internet,
isolated from br-lan and the WAN.
ALL FIPS routers must broadcast this SSID with the same security type
(open) — phones then save it once and roam between routers as one
network. The 10.21.$IDX.0/24 subnet is the same on every router on
purpose: leases survive roaming. Unlike the mesh backhaul, channels
are free per router. On a dual-band router, run fips-ap-setup for the
other radio too so clients can pick either band.
On first connect a phone warns that the network has no internet —
choose "stay connected" and "don't ask again". That choice is stored
per SSID, so it covers every FIPS router.
Next steps:
1. $TRANSPORT_NOTE
2. $MDNS_NOTE
Restart the daemon AFTER the interface is up — a transport whose
interface is missing at startup is skipped, not retried:
/etc/init.d/fips restart
3. Associate a phone or laptop running FIPS and verify:
iw dev $AP_IFNAME station dump
and the FIPS link on top of it:
fipsctl show peers
Run 'fips-ap-setup remove' to undo all instances, or
'fips-ap-setup remove $RADIO' for just this one.
EOF
packaging/openwrt-ipk/files/usr/bin/fips-mesh-setup
Executable
+261
View File
@@ -0,0 +1,261 @@
#!/bin/sh
# fips-mesh-setup — configure open 802.11s mesh interfaces for FIPS backhaul.
#
# Usage:
# fips-mesh-setup <radio> [mesh-id] e.g. fips-mesh-setup radio1
# fips-mesh-setup remove [radio] no radio: remove all instances
#
# Creates a mesh-point interface on the given radio and leaves everything
# above L2 to FIPS. Run once per radio: dual-band routers can mesh on both
# bands at once (2.4 GHz reaches further, 5 GHz carries more). Note this is
# failover, not multipath — FIPS keeps one active link per peer; the other
# band stands by and reconnects the peer if the active link dies.
#
# - encryption 'none' — the mesh is OPEN on purpose. FIPS's Noise IK
# handshake authenticates and encrypts every peer link, so SAE would
# only duplicate that (and on ath10k it forces the slower raw Tx/Rx
# firmware mode). A stranger can form an 802.11s peering AND a FIPS
# peer link — the Noise handshake authenticates each link (no
# impersonation of another identity, no MITM), it does not gate who
# may peer. Admission is open up to the daemon's max-peers cap.
# - mesh_fwding '0' — disables 802.11s HWMP forwarding so each mesh
# link is a plain L2 neighbor link. FIPS is the routing layer; two
# routing layers would fight.
#
# Interfaces are named per radio index (radio0 -> fips-mesh0, radio1 ->
# fips-mesh1) and are intentionally NOT bridged into br-lan: the FIPS
# Ethernet transport binds each directly and runs discovery beacons over it.
#
# The shipped /etc/fips/fips.yaml carries 'mesh0' and 'mesh1' entries under
# 'transports.ethernet' bound to these names, but commented out — a stock
# install that never creates fips-mesh* then logs no bind warning. This
# helper uncomments the matching entry when it creates an interface and
# re-comments it on remove, so the daemon binds the transport without a
# manual config edit. After an interface is up, restart fips.
# See docs/how-to/set-up-80211s-mesh-backhaul.md for the full guide.
DEFAULT_MESH_ID="fips-mesh"
CONFIG="/etc/fips/fips.yaml"
# Replace $CONFIG with the rewritten $CONFIG.tmp. Force mode 0600 first: the
# package installs fips.yaml 0600 (it may hold an inline 'nsec' private key),
# and a fresh tmp file would otherwise land world-readable after the move.
mesh_config_write() {
chmod 600 "$CONFIG.tmp" && mv "$CONFIG.tmp" "$CONFIG"
}
# Uncomment the 'mesh<idx>' transports.ethernet block in $CONFIG (created by
# 'fips-mesh-setup'). Reversible with mesh_config_disable. Returns:
# 0 enabled (or already active) 1 no config file 2 no such block
mesh_config_enable() {
idx="$1"
[ -f "$CONFIG" ] || return 1
grep -q "^ mesh$idx:" "$CONFIG" && return 0
grep -q "^ # mesh$idx:" "$CONFIG" || return 2
awk -v idx="$idx" '
$0 ~ ("^ # mesh" idx ":[ \t]*$") { blk = 1; sub(/^ # /, " "); print; next }
blk && /^ # / { sub(/^ # /, " "); print; next }
{ blk = 0; print }
' "$CONFIG" > "$CONFIG.tmp" && mesh_config_write
}
# Re-comment the 'mesh<idx>' block so the daemon stops binding it (and stops
# warning about the now-missing interface). Inverse of mesh_config_enable.
mesh_config_disable() {
idx="$1"
[ -f "$CONFIG" ] || return 1
grep -q "^ mesh$idx:" "$CONFIG" || return 0
awk -v idx="$idx" '
$0 ~ ("^ mesh" idx ":[ \t]*$") { blk = 1; sub(/^ /, " # "); print; next }
blk && /^ / { sub(/^ /, " # "); print; next }
{ blk = 0; print }
' "$CONFIG" > "$CONFIG.tmp" && mesh_config_write
}
usage() {
echo "Usage: fips-mesh-setup <radio> [mesh-id]" >&2
echo " fips-mesh-setup remove [radio]" >&2
echo "Radios on this device:" >&2
uci show wireless 2>/dev/null | sed -n "s/^wireless\.\([^.]*\)=wifi-device$/ \1/p" >&2
exit 1
}
# List the UCI section names of fips-managed mesh wifi-ifaces.
mesh_sections() {
uci show wireless 2>/dev/null | sed -n "s/^wireless\.\(fips_mesh[^.=]*\)=wifi-iface$/\1/p"
}
# ---------------------------------------------------------------------------
# remove [radio] — delete the wireless and network sections created below
# ---------------------------------------------------------------------------
if [ "$1" = "remove" ]; then
if [ -n "$2" ]; then
SECTIONS="fips_mesh_$(printf '%s' "$2" | tr -c 'a-zA-Z0-9_' '_')"
else
SECTIONS="$(mesh_sections)"
fi
[ -n "$SECTIONS" ] || {
echo "No fips mesh instances configured."
exit 0
}
for section in $SECTIONS; do
ifname="$(uci -q get "wireless.$section.ifname")"
uci -q delete "wireless.$section"
uci -q delete "network.$section"
# Re-comment the matching mesh<N> transport in fips.yaml so the
# daemon stops warning about the interface we just removed.
idx="$(printf '%s' "$ifname" | sed -n 's/.*[^0-9]\([0-9]\{1,\}\)$/\1/p')"
[ -n "$idx" ] && mesh_config_disable "$idx"
echo "Removed ${ifname:-$section}."
done
uci commit wireless
uci commit network
# 'wifi reload' re-applies the whole wireless config, so it briefly drops
# every client AP on all radios (a few seconds) — expected on remove.
wifi reload
echo "Restart fips: /etc/init.d/fips restart"
exit 0
fi
RADIO="$1"
MESH_ID="${2:-$DEFAULT_MESH_ID}"
[ -n "$RADIO" ] || usage
if [ "$(uci -q get "wireless.$RADIO")" != "wifi-device" ]; then
echo "Error: '$RADIO' is not a wifi-device in /etc/config/wireless." >&2
usage
fi
# One instance per radio: section fips_mesh_<radio>, netdev fips-mesh<N>
# where N is the radio's trailing index (radio0 -> fips-mesh0). For radios
# named without a trailing number, fall back to the first free index.
SECTION="fips_mesh_$(printf '%s' "$RADIO" | tr -c 'a-zA-Z0-9_' '_')"
IDX="$(printf '%s' "$RADIO" | sed -n 's/.*[^0-9]\([0-9]\{1,\}\)$/\1/p')"
[ -n "$IDX" ] || IDX="$(printf '%s' "$RADIO" | sed -n 's/^\([0-9]\{1,\}\)$/\1/p')"
if [ -z "$IDX" ]; then
IDX=0
while uci show wireless 2>/dev/null | grep -q "\.ifname='fips-mesh$IDX'"; do
IDX=$((IDX + 1))
done
fi
MESH_IFNAME="fips-mesh$IDX"
# Refuse a name collision from another radio's instance (e.g. two radios
# whose names end in the same digit) rather than silently hijacking it.
OWNER="$(uci show wireless 2>/dev/null \
| sed -n "s/^wireless\.\(fips_mesh[^.=]*\)\.ifname='$MESH_IFNAME'$/\1/p")"
if [ -n "$OWNER" ] && [ "$OWNER" != "$SECTION" ]; then
echo "Error: $MESH_IFNAME is already used by section '$OWNER'." >&2
echo "Remove it first: fips-mesh-setup remove" >&2
exit 1
fi
# ---------------------------------------------------------------------------
# Driver capability check (advisory — config below is harmless either way)
# ---------------------------------------------------------------------------
if command -v iw >/dev/null 2>&1; then
if ! iw list 2>/dev/null | grep -q "\* mesh point"; then
echo "Warning: no radio on this device advertises 'mesh point' support" >&2
echo "(iw list | grep 'mesh point'). The interface may fail to come up." >&2
fi
fi
# ---------------------------------------------------------------------------
# Wireless: open 802.11s mesh point, HWMP forwarding off
# ---------------------------------------------------------------------------
uci -q delete "wireless.$SECTION"
uci set "wireless.$SECTION=wifi-iface"
uci set "wireless.$SECTION.device=$RADIO"
uci set "wireless.$SECTION.mode=mesh"
uci set "wireless.$SECTION.mesh_id=$MESH_ID"
uci set "wireless.$SECTION.encryption=none"
uci set "wireless.$SECTION.mesh_fwding=0"
uci set "wireless.$SECTION.ifname=$MESH_IFNAME"
uci set "wireless.$SECTION.network=$SECTION"
# Radios ship disabled on fresh OpenWrt installs; a disabled radio would
# leave the mesh interface down with no error anywhere visible.
if [ "$(uci -q get "wireless.$RADIO.disabled")" = "1" ]; then
echo "Note: enabling $RADIO (was disabled)."
uci -q delete "wireless.$RADIO.disabled"
fi
# The mesh inherits the radio's channel, and mesh points only peer on the
# same channel. 'auto' lets each router pick its own — the classic silent
# non-peering cause — so surface the setting loudly.
CHANNEL="$(uci -q get "wireless.$RADIO.channel")"
BAND="$(uci -q get "wireless.$RADIO.band")"
if [ -z "$CHANNEL" ] || [ "$CHANNEL" = "auto" ]; then
echo "Warning: $RADIO channel is '${CHANNEL:-unset}' — each router may" >&2
echo "auto-select a different channel and mesh points only peer on the" >&2
echo "same one. Pin the same channel on every backhaul router, e.g.:" >&2
echo " uci set wireless.$RADIO.channel='36' && uci commit wireless && wifi reload" >&2
fi
# A client (sta) interface on the same radio follows its upstream AP's
# channel and drags every other interface with it — a mesh pinned to a
# different channel silently never joins, and does not recover when the
# STA disconnects.
for s in $(uci show wireless 2>/dev/null | sed -n "s/^wireless\.\([^.]*\)\.mode='sta'$/\1/p"); do
if [ "$(uci -q get "wireless.$s.device")" = "$RADIO" ]; then
echo "Warning: $RADIO also carries client interface '$s' (mode 'sta')." >&2
echo "The whole radio follows that STA's upstream channel — a mesh" >&2
echo "pinned to a different channel stays down silently. Align the" >&2
echo "mesh channel with the upstream AP, or put the mesh on a radio" >&2
echo "without a STA (a roaming uplink is incompatible with a" >&2
echo "fixed-channel mesh on the same radio)." >&2
fi
done
# ---------------------------------------------------------------------------
# Network: unmanaged interface so netifd brings the netdev up. No IP config —
# the FIPS Ethernet transport speaks raw frames on it.
# ---------------------------------------------------------------------------
uci -q delete "network.$SECTION"
uci set "network.$SECTION=interface"
uci set "network.$SECTION.proto=none"
uci commit wireless
uci commit network
# 'wifi reload' re-applies the whole wireless config, so it briefly drops
# every client AP on all radios (a few seconds) — expected when adding a mesh.
wifi reload
# Enable the matching mesh<N> transport in the shipped fips.yaml (it ships
# commented out). Tailor the restart hint to what we could do.
mesh_config_enable "$IDX"
case $? in
0) TRANSPORT_NOTE="The mesh$IDX transport in $CONFIG that binds '$MESH_IFNAME' is
now uncommented and enabled." ;;
1) TRANSPORT_NOTE="No $CONFIG found — add a transports.ethernet entry binding
interface '$MESH_IFNAME' by hand." ;;
*) TRANSPORT_NOTE="No 'mesh$IDX' entry in $CONFIG — add a transports.ethernet
entry binding interface '$MESH_IFNAME' by hand (copy the mesh0 block)." ;;
esac
cat <<EOF
Created open 802.11s mesh '$MESH_ID' as $MESH_IFNAME on $RADIO \
(band ${BAND:-?}, channel ${CHANNEL:-auto}).
ALL routers in this backhaul must share this mesh ID AND channel
(per band). On a dual-band router, run fips-mesh-setup for the other
radio too — second band is a standby path (failover, not multipath).
Next steps:
1. $TRANSPORT_NOTE
Restart the daemon AFTER the interface is up — a transport whose
interface is missing at startup is skipped, not retried:
/etc/init.d/fips restart
2. Verify L2 peering with a second FIPS router in range:
iw dev $MESH_IFNAME station dump
and the FIPS link on top of it:
fipsctl show peers
Run 'fips-mesh-setup remove' to undo all instances, or
'fips-mesh-setup remove $RADIO' for just this one.
EOF
packaging/systemd/README.install.md
+1 -1
View File
@@ -68,7 +68,7 @@ and set the interface name:
transports:
ethernet:
interface: "eth0"
discovery: true
listen: true
announce: true
auto_connect: true
accept_connections: true
packaging/systemd/fips.service
+5
View File
@@ -14,6 +14,11 @@ RestartSec=5
RuntimeDirectory=fips
RuntimeDirectoryMode=0750
# Log directory (/var/log/fips/), where the built-in tick-body profiler writes
# its capture files. Declared so systemd creates it on start and removes it on
# purge; the daemon runs as root and already has access without it.
LogsDirectory=fips
# Security hardening (daemon runs as root for TUN and raw sockets)
ProtectHome=yes
PrivateTmp=yes
src/bin/fips.rs
+14 -17
View File
@@ -8,7 +8,7 @@ use fips::config::{IdentitySource, resolve_identity};
use fips::version;
use fips::{Config, Node};
use std::path::PathBuf;
use tracing::{debug, error, info, warn};
use tracing::{debug, error, info};
use tracing_subscriber::{EnvFilter, fmt};
/// FIPS mesh network daemon
@@ -157,26 +157,23 @@ async fn run_daemon(
info!("FIPS running");
// Run the RX event loop until shutdown signal.
// stop() drops the packet channel, causing run_rx_loop to exit.
tokio::select! {
result = node.run_rx_loop() => {
match result {
Ok(()) => info!("RX loop exited"),
Err(e) => error!("RX loop error: {}", e),
}
}
_ = shutdown_signal => {
info!("Shutdown signal received");
}
// Serve until the shutdown signal, then drain in place before returning.
// The rx loop observes the signal directly, so its channels are never
// destructively cancelled — they live in the loop's locals across serve and
// drain, and are dropped only on clean exit (after which teardown does not
// need them). On the signal the loop broadcasts a shutdown Disconnect and
// waits (bounded by node.drain_timeout_secs) for peers to clear.
match node.run_rx_loop_with_shutdown(shutdown_signal).await {
Ok(()) => info!("RX loop exited"),
Err(e) => error!("RX loop error: {}", e),
}
info!("FIPS shutting down");
// Stop the node (shuts down transports, TUN, I/O threads)
if let Err(e) = node.stop().await {
warn!("Error during shutdown: {}", e);
}
// Close the drain window (if the loop drained) and tear down. A drained
// loop tears down without re-broadcasting; a loop that exited some other
// way falls back to the immediate stop().
node.finish_shutdown().await;
info!("FIPS shutdown complete");
}
src/bin/fipsctl.rs
+45
View File
@@ -76,6 +76,37 @@ enum Commands {
#[command(subcommand)]
what: StatsCommands,
},
/// Control the built-in profiler (requires a `--features profiling` build)
#[cfg(feature = "profiling")]
Profile {
#[command(subcommand)]
what: ProfileCommands,
},
}
#[cfg(feature = "profiling")]
#[derive(Subcommand, Debug)]
enum ProfileCommands {
/// Profile the rx-loop tick body
Tick {
#[command(subcommand)]
action: ProfileTickAction,
},
}
#[cfg(feature = "profiling")]
#[derive(Subcommand, Debug)]
enum ProfileTickAction {
/// Start a capture
On {
/// Directory for the capture file (default /var/log/fips)
#[arg(long)]
dir: Option<PathBuf>,
},
/// Stop the running capture
Off,
/// Report capture state
Status,
}
#[derive(Subcommand, Debug)]
@@ -471,6 +502,20 @@ fn main() {
build_command("show_stats_history", params)
}
},
#[cfg(feature = "profiling")]
Commands::Profile { what } => match what {
ProfileCommands::Tick { action } => match action {
ProfileTickAction::On { dir } => match dir {
Some(dir) => build_command(
"profile_tick_on",
serde_json::json!({"dir": dir.display().to_string()}),
),
None => build_query("profile_tick_on"),
},
ProfileTickAction::Off => build_query("profile_tick_off"),
ProfileTickAction::Status => build_query("profile_tick_status"),
},
},
Commands::Keygen { .. } => unreachable!(),
};
src/bin/fipstop/ui/routing.rs
+22 -22
View File
@@ -134,8 +134,8 @@ fn draw_routing_stats(
let cols =
Layout::horizontal([Constraint::Percentage(50), Constraint::Percentage(50)]).split(inner);
// Shorthand for a nested counter value (e.g. discovery.req_received).
let disc = |key: &str| helpers::nested_u64(data, "discovery", key);
// Shorthand for a nested counter value (e.g. lookup.req_received).
let lookup = |key: &str| helpers::nested_u64(data, "lookup", key);
let err = |key: &str| helpers::nested_u64(data, "error_signals", key);
let cong = |key: &str| helpers::nested_u64(data, "congestion", key);
@@ -174,32 +174,32 @@ fn draw_routing_stats(
));
left.push(Line::from(""));
left.extend(section(
"Discovery Requests",
"Lookup Requests",
&[
("Received", disc("req_received")),
("Forwarded", disc("req_forwarded")),
("Initiated", disc("req_initiated")),
("Deduplicated", disc("req_deduplicated")),
("Target Is Us", disc("req_target_is_us")),
("Duplicate", disc("req_duplicate")),
("Bloom Miss", disc("req_bloom_miss")),
("Backoff Suppressed", disc("req_backoff_suppressed")),
("Fwd Rate Limited", disc("req_forward_rate_limited")),
("TTL Exhausted", disc("req_ttl_exhausted")),
("Decode Error", disc("req_decode_error")),
("Received", lookup("req_received")),
("Forwarded", lookup("req_forwarded")),
("Initiated", lookup("req_initiated")),
("Deduplicated", lookup("req_deduplicated")),
("Target Is Us", lookup("req_target_is_us")),
("Duplicate", lookup("req_duplicate")),
("Bloom Miss", lookup("req_bloom_miss")),
("Backoff Suppressed", lookup("req_backoff_suppressed")),
("Fwd Rate Limited", lookup("req_forward_rate_limited")),
("TTL Exhausted", lookup("req_ttl_exhausted")),
("Decode Error", lookup("req_decode_error")),
],
));
left.push(Line::from(""));
left.extend(section(
"Discovery Responses",
"Lookup Responses",
&[
("Received", disc("resp_received")),
("Accepted", disc("resp_accepted")),
("Forwarded", disc("resp_forwarded")),
("Timed Out", disc("resp_timed_out")),
("Identity Miss", disc("resp_identity_miss")),
("Proof Failed", disc("resp_proof_failed")),
("Decode Error", disc("resp_decode_error")),
("Received", lookup("resp_received")),
("Accepted", lookup("resp_accepted")),
("Forwarded", lookup("resp_forwarded")),
("Timed Out", lookup("resp_timed_out")),
("Identity Miss", lookup("resp_identity_miss")),
("Proof Failed", lookup("resp_proof_failed")),
("Decode Error", lookup("resp_decode_error")),
],
));
src/bin/fipstop/ui/snapshots.rs
+9
View File
@@ -1189,6 +1189,15 @@ fn mmp_focused_pane_indicator() {
});
// The focused Session MMP title is cyan; the unfocused Link MMP title is not.
assert_eq!(testkit::fg_at(&buf, "Session MMP"), Some(Color::Cyan));
// Presence before colour. `fg_at` is `find(..)?` mapped to the cell's fg, so
// it returns None for a title that was never drawn, and None != Some(Cyan) --
// meaning the assertion below passed when the Link MMP pane was missing
// entirely. Asserting it is on screen first is what makes the next line read
// "not highlighted" rather than "not there".
assert!(
testkit::find(&buf, "Link MMP").is_some(),
"the unfocused Link MMP title should still be rendered"
);
assert_ne!(testkit::fg_at(&buf, "Link MMP"), Some(Color::Cyan));
}
src/bin/fipstop/ui/tree.rs
-4
View File
@@ -174,10 +174,6 @@ fn draw_stats(frame: &mut Frame, data: &serde_json::Value, scroll: u16, focused:
&helpers::nested_u64(data, "stats", "sig_failed"),
),
helpers::kv_line("Stale", &helpers::nested_u64(data, "stats", "stale")),
helpers::kv_line(
"Parent Switched",
&helpers::nested_u64(data, "stats", "parent_switched"),
),
helpers::kv_line(
"Loop Detected",
&helpers::nested_u64(data, "stats", "loop_detected"),
src/bloom/mod.rs
-60
View File
@@ -1,60 +0,0 @@
//! Bloom Filter Implementation
//!
//! 1KB Bloom filters for reachability in FIPS routing. Each node
//! maintains filters that summarize which destinations are reachable
//! through each peer, enabling efficient routing decisions without
//! global network knowledge.
//!
//! ## v1 Parameters
//!
//! - Size: 1 KB (8,192 bits) - sized for actual ~400-800 entry occupancy
//! - Hash functions: k=5 - optimal at ~1,200 entries, good for 800-1,600
//! - Bandwidth: 1 KB/announce (75% reduction from original 4KB design)
//!
//! These parameters are right-sized for typical network occupancy of
//! ~250-800 entries per node.
mod filter;
mod state;
use thiserror::Error;
pub use filter::BloomFilter;
pub use state::BloomState;
/// Default filter size in bits (1KB = 8,192 bits).
///
/// Sized for ~800-1,600 entries. FPR ~0.05% at 400 entries, ~0.9% at 800.
/// This is v1 protocol default (size_class=1).
pub const DEFAULT_FILTER_SIZE_BITS: usize = 8192;
/// Default filter size in bytes (1KB).
pub const DEFAULT_FILTER_SIZE_BYTES: usize = DEFAULT_FILTER_SIZE_BITS / 8;
/// Default number of hash functions.
///
/// k=5 is optimal at ~1,200 entries and a good compromise for 800-1,600.
/// At 400 entries: FPR ~0.05%. At 800 entries: FPR ~0.9%.
pub const DEFAULT_HASH_COUNT: u8 = 5;
/// Size class for v1 protocol (1 KB filters).
pub const V1_SIZE_CLASS: u8 = 1;
/// Filter sizes by size_class: bytes = 512 << size_class
pub const SIZE_CLASS_BYTES: [usize; 4] = [512, 1024, 2048, 4096];
/// Errors related to Bloom filter operations.
#[derive(Debug, Error)]
pub enum BloomError {
#[error("invalid filter size: expected {expected} bits, got {got}")]
InvalidSize { expected: usize, got: usize },
#[error("filter size must be a multiple of 8, got {0}")]
SizeNotByteAligned(usize),
#[error("hash count must be positive")]
ZeroHashCount,
}
#[cfg(test)]
mod tests;
src/cache/coord_cache.rs
Vendored
+1 -1
View File
@@ -9,7 +9,7 @@ use std::collections::HashMap;
use super::CacheStats;
use super::entry::CacheEntry;
use crate::NodeAddr;
use crate::tree::TreeCoordinate;
use crate::proto::stp::TreeCoordinate;
/// Default maximum entries in coordinate cache.
pub const DEFAULT_COORD_CACHE_SIZE: usize = 50_000;
src/cache/entry.rs
Vendored
+1 -1
View File
@@ -1,6 +1,6 @@
//! Cache entry with TTL and LRU tracking.
use crate::tree::TreeCoordinate;
use crate::proto::stp::TreeCoordinate;
/// A cached coordinate entry.
#[derive(Clone, Debug)]
src/config/mod.rs
+317 -31
View File
@@ -24,6 +24,7 @@ mod node;
mod peer;
mod transport;
use crate::node::REKEY_JITTER_SECS;
use crate::upper::config::{DnsConfig, TunConfig};
use crate::{Identity, IdentityError};
use serde::{Deserialize, Serialize};
@@ -33,9 +34,9 @@ use thiserror::Error;
#[cfg(target_os = "linux")]
pub use gateway::{ConntrackConfig, GatewayConfig, GatewayDnsConfig, PortForward, Proto};
pub use node::{
BloomConfig, BuffersConfig, CacheConfig, ControlConfig, DiscoveryConfig, LimitsConfig,
NodeConfig, NostrDiscoveryConfig, NostrDiscoveryPolicy, RateLimitConfig, RekeyConfig,
RetryConfig, SessionConfig, SessionMmpConfig, TreeConfig,
BloomConfig, BuffersConfig, CacheConfig, ControlConfig, LimitsConfig, LookupConfig, MmpConfig,
NodeConfig, NostrRendezvousConfig, NostrRendezvousPolicy, RateLimitConfig, RekeyConfig,
RendezvousConfig, RetryConfig, SessionConfig, SessionMmpConfig, TreeConfig,
};
pub use peer::{ConnectPolicy, PeerAddress, PeerConfig};
pub use transport::{
@@ -489,10 +490,59 @@ impl Config {
source: e,
})?;
serde_yaml::from_str(&contents).map_err(|e| ConfigError::ParseYaml {
path: path.to_path_buf(),
source: e,
})
let mut config: Config =
serde_yaml::from_str(&contents).map_err(|e| ConfigError::ParseYaml {
path: path.to_path_buf(),
source: e,
})?;
config.normalize_deprecated_keys();
Ok(config)
}
/// COMPAT (drop at the v2 cutover): fold a deprecated `node.discovery:`
/// block into the `node.lookup.*` (mesh-lookup scalars) and
/// `node.rendezvous.*` (nostr/LAN peer rendezvous) tables that replaced it.
///
/// Runs at every deserialize boundary (see `load_file`). A present legacy
/// field fills the corresponding new-table field, so a config that predates
/// the split keeps behaving identically. When a legacy block is seen, a
/// one-time deprecation warning names the old→new key moves. Exposed to the
/// crate so config tests that deserialize directly can invoke it.
pub(crate) fn normalize_deprecated_keys(&mut self) {
let Some(compat) = self.node.discovery.take() else {
return;
};
tracing::warn!(
target: "fips::config",
"`node.discovery.*` is deprecated and will be removed: mesh-lookup \
scalars moved to `node.lookup.*`, and peer-rendezvous keys moved to \
`node.rendezvous.nostr.*` / `node.rendezvous.lan.*`. Please migrate; \
a legacy `node.discovery` block still applies for now."
);
if let Some(v) = compat.ttl {
self.node.lookup.ttl = v;
}
if let Some(v) = compat.attempt_timeouts_secs {
self.node.lookup.attempt_timeouts_secs = v;
}
if let Some(v) = compat.recent_expiry_secs {
self.node.lookup.recent_expiry_secs = v;
}
if let Some(v) = compat.backoff_base_secs {
self.node.lookup.backoff_base_secs = v;
}
if let Some(v) = compat.backoff_max_secs {
self.node.lookup.backoff_max_secs = v;
}
if let Some(v) = compat.forward_min_interval_secs {
self.node.lookup.forward_min_interval_secs = v;
}
if let Some(v) = compat.nostr {
self.node.rendezvous.nostr = v;
}
if let Some(v) = compat.lan {
self.node.rendezvous.lan = v;
}
}
/// Get the standard search paths in priority order (lowest to highest).
@@ -600,7 +650,7 @@ impl Config {
/// Validate cross-field configuration invariants.
pub fn validate(&self) -> Result<(), ConfigError> {
let nostr = &self.node.discovery.nostr;
let nostr = &self.node.rendezvous.nostr;
let any_transport_advertises_on_nostr = self
.transports
@@ -620,13 +670,13 @@ impl Config {
if any_transport_advertises_on_nostr && !nostr.enabled {
return Err(ConfigError::Validation(
"at least one transport has `advertise_on_nostr = true`, but `node.discovery.nostr.enabled` is false".to_string(),
"at least one transport has `advertise_on_nostr = true`, but `node.rendezvous.nostr.enabled` is false".to_string(),
));
}
if self.peers.iter().any(|peer| peer.via_nostr) && !nostr.enabled {
return Err(ConfigError::Validation(
"at least one peer has `via_nostr = true`, but `node.discovery.nostr.enabled` is false".to_string(),
"at least one peer has `via_nostr = true`, but `node.rendezvous.nostr.enabled` is false".to_string(),
));
}
@@ -648,12 +698,12 @@ impl Config {
if nostr.enabled && has_nat_udp_advert {
if nostr.dm_relays.is_empty() {
return Err(ConfigError::Validation(
"NAT UDP advert publishing requires `node.discovery.nostr.dm_relays` to be non-empty".to_string(),
"NAT UDP advert publishing requires `node.rendezvous.nostr.dm_relays` to be non-empty".to_string(),
));
}
if nostr.stun_servers.is_empty() {
return Err(ConfigError::Validation(
"NAT UDP advert publishing requires `node.discovery.nostr.stun_servers` to be non-empty".to_string(),
"NAT UDP advert publishing requires `node.rendezvous.nostr.stun_servers` to be non-empty".to_string(),
));
}
}
@@ -686,6 +736,32 @@ impl Config {
}
}
// Reject rekey triggers that fire immediately and forever. Both
// arms are checked regardless of `node.rekey.enabled` so that
// turning rekey on later cannot surface a config error at a
// surprising moment. There is deliberately no upper bound:
// u64::MAX is the idiom for disabling one arm of the trigger.
let rekey = &self.node.rekey;
if rekey.after_messages == 0 {
return Err(ConfigError::Validation(
"`node.rekey.after_messages` must be at least 1; 0 fires the message-count trigger on every poll instead of disabling it. \
Use a very large value to effectively disable the message-count trigger."
.to_string(),
));
}
let jitter_secs = REKEY_JITTER_SECS.unsigned_abs();
if rekey.after_secs <= jitter_secs {
return Err(ConfigError::Validation(format!(
"`node.rekey.after_secs` is {}, but must be greater than the per-session rekey jitter of {jitter_secs}s; \
each session offsets the interval by a random value in [-{jitter_secs}, +{jitter_secs}] seconds, so a smaller interval saturates to zero \
and rekeys on sight for roughly half of sessions. \
Use a very large value to effectively disable the timer trigger.",
rekey.after_secs
)));
}
Ok(())
}
@@ -721,6 +797,84 @@ node:
assert!(config.has_identity());
}
/// The fips.yaml shipped in the OpenWrt package must keep parsing as the
/// config schema evolves. Both the 802.11s mesh backhaul entries
/// (docs/how-to/set-up-80211s-mesh-backhaul.md) and the open-access SSID
/// entries (docs/how-to/set-up-open-access-ssid.md) ship commented out —
/// one per radio, so dual-band routers can run either on both bands — so
/// a stock install that never creates fips-mesh*/fips-ap* logs no
/// per-boot bind warning; `fips-mesh-setup`/`fips-ap-setup` uncomment the
/// matching block when they create the interface. Verify both states
/// parse: as shipped (both inactive), and after the uncomment the helpers
/// perform.
#[test]
fn shipped_openwrt_config_parses() {
let yaml = include_str!("../../packaging/openwrt-ipk/files/etc/fips/fips.yaml");
// As shipped: parses, and the mesh/ap entries are commented out (a
// running daemon binds no fips-mesh*/fips-ap* transport, no warning).
let config: Config = serde_yaml::from_str(yaml).expect("shipped OpenWrt fips.yaml");
for name in ["mesh0", "mesh1", "ap0", "ap1"] {
assert!(
!config
.transports
.ethernet
.iter()
.any(|(n, _)| n == Some(name)),
"{name} must ship commented out, not active, in fips.yaml"
);
}
// What `fips-mesh-setup`/`fips-ap-setup` produce: uncomment each
// block, which must still parse into a transport bound to the right
// netdev.
let uncommented =
uncomment_transport_blocks(&uncomment_transport_blocks(yaml, "mesh"), "ap");
let config: Config = serde_yaml::from_str(&uncommented)
.expect("fips.yaml with mesh and ap transports uncommented");
for (name, interface) in [
("mesh0", "fips-mesh0"),
("mesh1", "fips-mesh1"),
("ap0", "fips-ap0"),
("ap1", "fips-ap1"),
] {
assert!(
config
.transports
.ethernet
.iter()
.any(|(n, eth)| n == Some(name) && eth.interface == interface),
"{name} entry missing after uncommenting shipped fips.yaml"
);
}
}
/// Mirror the setup helpers' block uncomment: strip the ` # ` prefix
/// from each `# <prefix><N>:` header and its ` # ` continuation
/// lines, leaving every other comment untouched.
fn uncomment_transport_blocks(yaml: &str, prefix: &str) -> String {
let header = format!(" # {prefix}");
let mut out = String::new();
let mut in_block = false;
for line in yaml.lines() {
let is_header = line
.strip_prefix(&header)
.and_then(|r| r.strip_suffix(':'))
.is_some_and(|n| !n.is_empty() && n.bytes().all(|b| b.is_ascii_digit()));
if is_header {
in_block = true;
out.push_str(&line.replacen(" # ", " ", 1));
} else if in_block && line.starts_with(" # ") {
out.push_str(&line.replacen(" # ", " ", 1));
} else {
in_block = false;
out.push_str(line);
}
out.push('\n');
}
out
}
#[test]
fn test_parse_yaml_with_hex() {
let yaml = r#"
@@ -1261,7 +1415,9 @@ peers:
}
#[test]
fn test_parse_nostr_discovery_config() {
fn test_parse_legacy_discovery_nostr_config_compat() {
// COMPAT (drop at the v2 cutover): a deprecated `node.discovery.nostr`
// block must fold into `node.rendezvous.nostr` via normalize.
let yaml = r#"
node:
discovery:
@@ -1285,26 +1441,27 @@ peers:
- transport: udp
addr: "nat"
"#;
let config: Config = serde_yaml::from_str(yaml).unwrap();
assert!(config.node.discovery.nostr.enabled);
assert!(!config.node.discovery.nostr.advertise);
assert_eq!(config.node.discovery.nostr.app, "fips.nat.test.v1");
assert_eq!(config.node.discovery.nostr.signal_ttl_secs, 45);
let mut config: Config = serde_yaml::from_str(yaml).unwrap();
config.normalize_deprecated_keys();
assert!(config.node.rendezvous.nostr.enabled);
assert!(!config.node.rendezvous.nostr.advertise);
assert_eq!(config.node.rendezvous.nostr.app, "fips.nat.test.v1");
assert_eq!(config.node.rendezvous.nostr.signal_ttl_secs, 45);
assert_eq!(
config.node.discovery.nostr.policy,
NostrDiscoveryPolicy::ConfiguredOnly
config.node.rendezvous.nostr.policy,
NostrRendezvousPolicy::ConfiguredOnly
);
assert_eq!(config.node.discovery.nostr.open_discovery_max_pending, 12);
assert_eq!(config.node.rendezvous.nostr.open_discovery_max_pending, 12);
assert_eq!(
config.node.discovery.nostr.advert_relays,
config.node.rendezvous.nostr.advert_relays,
vec!["wss://relay-a.example".to_string()]
);
assert_eq!(
config.node.discovery.nostr.dm_relays,
config.node.rendezvous.nostr.dm_relays,
vec!["wss://relay-b.example".to_string()]
);
assert_eq!(
config.node.discovery.nostr.stun_servers,
config.node.rendezvous.nostr.stun_servers,
vec!["stun:stun.example.org:3478".to_string()]
);
assert_eq!(
@@ -1314,6 +1471,55 @@ peers:
assert!(config.peers[0].via_nostr);
}
#[test]
fn test_parse_lookup_and_rendezvous_new_keys() {
// The post-split keys parse directly, with no deprecated block and no
// normalize warning.
let yaml = r#"
node:
lookup:
ttl: 7
attempt_timeouts_secs: [3, 6]
forward_min_interval_secs: 9
rendezvous:
nostr:
enabled: true
app: "fips.new.keys.v1"
"#;
let mut config: Config = serde_yaml::from_str(yaml).unwrap();
config.normalize_deprecated_keys();
assert_eq!(config.node.lookup.ttl, 7);
assert_eq!(config.node.lookup.attempt_timeouts_secs, vec![3, 6]);
assert_eq!(config.node.lookup.forward_min_interval_secs, 9);
// Unset scalar keeps its default.
assert_eq!(config.node.lookup.recent_expiry_secs, 10);
assert!(config.node.rendezvous.nostr.enabled);
assert_eq!(config.node.rendezvous.nostr.app, "fips.new.keys.v1");
assert!(config.node.discovery.is_none());
}
#[test]
fn test_legacy_discovery_lookup_scalars_compat() {
// COMPAT (drop at the v2 cutover): legacy `node.discovery` mesh-lookup
// scalars must fold into `node.lookup`; unset keys keep their defaults.
let yaml = r#"
node:
discovery:
ttl: 5
backoff_base_secs: 4
backoff_max_secs: 30
"#;
let mut config: Config = serde_yaml::from_str(yaml).unwrap();
config.normalize_deprecated_keys();
assert_eq!(config.node.lookup.ttl, 5);
assert_eq!(config.node.lookup.backoff_base_secs, 4);
assert_eq!(config.node.lookup.backoff_max_secs, 30);
// Unset legacy scalar leaves the new-table default intact.
assert_eq!(config.node.lookup.attempt_timeouts_secs, vec![1, 2, 4, 8]);
// The compat block is consumed by normalize.
assert!(config.node.discovery.is_none());
}
#[test]
fn test_validate_transport_advert_requires_nostr_enabled() {
let mut config = Config::default();
@@ -1321,7 +1527,7 @@ peers:
advertise_on_nostr: Some(true),
..Default::default()
});
config.node.discovery.nostr.enabled = false;
config.node.rendezvous.nostr.enabled = false;
let err = config.validate().expect_err("validation should fail");
assert!(err.to_string().contains("advertise_on_nostr"));
@@ -1337,7 +1543,7 @@ peers:
}],
..Default::default()
};
config.node.discovery.nostr.enabled = false;
config.node.rendezvous.nostr.enabled = false;
let err = config.validate().expect_err("validation should fail");
assert!(err.to_string().contains("via_nostr"));
@@ -1358,7 +1564,7 @@ peers:
// Empty addresses + via_nostr=true + nostr.enabled=true → ok.
config.peers[0].via_nostr = true;
config.node.discovery.nostr.enabled = true;
config.node.rendezvous.nostr.enabled = true;
config
.validate()
.expect("via_nostr should allow empty addresses");
@@ -1367,8 +1573,8 @@ peers:
#[test]
fn test_validate_nat_udp_advert_requires_relays_and_stun() {
let mut config = Config::default();
config.node.discovery.nostr.enabled = true;
config.node.discovery.nostr.dm_relays.clear();
config.node.rendezvous.nostr.enabled = true;
config.node.rendezvous.nostr.dm_relays.clear();
config.transports.udp = TransportInstances::Single(UdpConfig {
advertise_on_nostr: Some(true),
public: Some(false),
@@ -1378,8 +1584,8 @@ peers:
let err = config.validate().expect_err("validation should fail");
assert!(err.to_string().contains("dm_relays"));
config.node.discovery.nostr.dm_relays = vec!["wss://relay.example".to_string()];
config.node.discovery.nostr.stun_servers.clear();
config.node.rendezvous.nostr.dm_relays = vec!["wss://relay.example".to_string()];
config.node.rendezvous.nostr.stun_servers.clear();
let err = config.validate().expect_err("validation should fail");
assert!(err.to_string().contains("stun_servers"));
}
@@ -1459,6 +1665,86 @@ peers:
.expect("outbound_only should be exempt from the loopback check");
}
#[test]
fn test_validate_default_rekey_settings_ok() {
Config::default()
.validate()
.expect("shipped default rekey settings must validate");
}
#[test]
fn test_validate_rekey_after_messages_zero_rejected() {
let mut config = Config::default();
config.node.rekey.after_messages = 0;
let err = config.validate().expect_err("validation should fail");
let msg = err.to_string();
assert!(msg.contains("after_messages"), "got: {msg}");
}
#[test]
fn test_validate_rekey_after_messages_one_accepted() {
let mut config = Config::default();
config.node.rekey.after_messages = 1;
config
.validate()
.expect("after_messages = 1 rekeys every message, which is wasteful but well defined");
}
#[test]
fn test_validate_rekey_after_secs_at_or_below_jitter_rejected() {
let jitter = REKEY_JITTER_SECS.unsigned_abs();
for after_secs in [0, 1, jitter - 1, jitter] {
let mut config = Config::default();
config.node.rekey.after_secs = after_secs;
match config.validate() {
Err(e) => assert!(e.to_string().contains("after_secs"), "got: {e}"),
Ok(()) => panic!("after_secs = {after_secs} should be rejected"),
}
}
}
#[test]
fn test_validate_rekey_after_secs_just_above_jitter_accepted() {
let mut config = Config::default();
config.node.rekey.after_secs = REKEY_JITTER_SECS.unsigned_abs() + 1;
config
.validate()
.expect("one second above the jitter bound leaves a non-zero effective interval");
}
#[test]
fn test_validate_rekey_unbounded_values_accepted() {
let mut config = Config::default();
config.node.rekey.after_secs = u64::MAX;
config.node.rekey.after_messages = u64::MAX;
config
.validate()
.expect("u64::MAX disables an arm of the trigger and must stay legal");
}
#[test]
fn test_validate_rekey_checked_even_when_disabled() {
let mut config = Config::default();
config.node.rekey.enabled = false;
config.node.rekey.after_messages = 0;
let err = config.validate().expect_err("validation should fail");
assert!(err.to_string().contains("after_messages"));
let mut config = Config::default();
config.node.rekey.enabled = false;
config.node.rekey.after_secs = REKEY_JITTER_SECS.unsigned_abs();
let err = config.validate().expect_err("validation should fail");
assert!(err.to_string().contains("after_secs"));
}
#[test]
fn test_outbound_only_forces_ephemeral_bind() {
let cfg = UdpConfig {
src/config/node.rs
+226 -80
View File
@@ -7,7 +7,7 @@
use serde::{Deserialize, Serialize};
use super::IdentityConfig;
use crate::mmp::{DEFAULT_LOG_INTERVAL_SECS, DEFAULT_OWD_WINDOW_SIZE, MmpConfig, MmpMode};
use crate::proto::mmp::{DEFAULT_LOG_INTERVAL_SECS, DEFAULT_OWD_WINDOW_SIZE, MmpMode};
// ============================================================================
// Node Configuration Subsections
@@ -186,48 +186,42 @@ impl CacheConfig {
}
}
/// Discovery protocol (`node.discovery.*`).
/// Mesh-lookup protocol (`node.lookup.*`): the overlay coordinate-lookup
/// engine (address → coordinates). The peer-rendezvous keys that used to
/// share this table (`nostr`/`lan`) now live under [`RendezvousConfig`]
/// (`node.rendezvous.*`).
#[derive(Debug, Clone, Serialize, Deserialize)]
pub struct DiscoveryConfig {
/// Hop limit for LookupRequest flood (`node.discovery.ttl`).
#[serde(default = "DiscoveryConfig::default_ttl")]
pub struct LookupConfig {
/// Hop limit for LookupRequest flood (`node.lookup.ttl`).
#[serde(default = "LookupConfig::default_ttl")]
pub ttl: u8,
/// Per-attempt timeouts in seconds (`node.discovery.attempt_timeouts_secs`).
/// Per-attempt timeouts in seconds (`node.lookup.attempt_timeouts_secs`).
/// Each entry is the time to wait for a response before sending the next
/// LookupRequest (with a fresh request_id). Sequence length determines the
/// total number of attempts before declaring the destination unreachable.
/// Default `[1, 2, 4, 8]` gives 4 attempts and a 15s total budget.
#[serde(default = "DiscoveryConfig::default_attempt_timeouts_secs")]
#[serde(default = "LookupConfig::default_attempt_timeouts_secs")]
pub attempt_timeouts_secs: Vec<u64>,
/// Dedup cache expiry in seconds (`node.discovery.recent_expiry_secs`).
#[serde(default = "DiscoveryConfig::default_recent_expiry_secs")]
/// Dedup cache expiry in seconds (`node.lookup.recent_expiry_secs`).
#[serde(default = "LookupConfig::default_recent_expiry_secs")]
pub recent_expiry_secs: u64,
/// Base backoff after lookup failure in seconds (`node.discovery.backoff_base_secs`).
/// Base backoff after lookup failure in seconds (`node.lookup.backoff_base_secs`).
/// Doubles per consecutive failure up to `backoff_max_secs`. Defaults to 0
/// (no post-failure suppression); the per-attempt sequence in
/// `attempt_timeouts_secs` provides the only retry pacing.
#[serde(default = "DiscoveryConfig::default_backoff_base_secs")]
#[serde(default = "LookupConfig::default_backoff_base_secs")]
pub backoff_base_secs: u64,
/// Maximum backoff cap in seconds (`node.discovery.backoff_max_secs`).
#[serde(default = "DiscoveryConfig::default_backoff_max_secs")]
/// Maximum backoff cap in seconds (`node.lookup.backoff_max_secs`).
#[serde(default = "LookupConfig::default_backoff_max_secs")]
pub backoff_max_secs: u64,
/// Minimum interval between forwarded lookups for the same target in seconds
/// (`node.discovery.forward_min_interval_secs`).
/// (`node.lookup.forward_min_interval_secs`).
/// Defense-in-depth against misbehaving nodes.
#[serde(default = "DiscoveryConfig::default_forward_min_interval_secs")]
#[serde(default = "LookupConfig::default_forward_min_interval_secs")]
pub forward_min_interval_secs: u64,
/// Nostr-mediated overlay endpoint discovery.
#[serde(default = "DiscoveryConfig::default_nostr")]
pub nostr: NostrDiscoveryConfig,
/// mDNS / DNS-SD peer discovery on the local link. Identity surface
/// is a strict subset of what `nostr.advertise` already publishes
/// publicly, so there's no marginal privacy cost; the latency win
/// for same-LAN peers is large (sub-second pairing, no relay).
#[serde(default = "DiscoveryConfig::default_lan")]
pub lan: crate::discovery::lan::LanDiscoveryConfig,
}
impl Default for DiscoveryConfig {
impl Default for LookupConfig {
fn default() -> Self {
Self {
ttl: 64,
@@ -236,13 +230,11 @@ impl Default for DiscoveryConfig {
backoff_base_secs: 0,
backoff_max_secs: 0,
forward_min_interval_secs: 2,
nostr: NostrDiscoveryConfig::default(),
lan: crate::discovery::lan::LanDiscoveryConfig::default(),
}
}
}
impl DiscoveryConfig {
impl LookupConfig {
fn default_ttl() -> u8 {
64
}
@@ -261,12 +253,45 @@ impl DiscoveryConfig {
fn default_forward_min_interval_secs() -> u64 {
2
}
fn default_nostr() -> NostrDiscoveryConfig {
NostrDiscoveryConfig::default()
}
fn default_lan() -> crate::discovery::lan::LanDiscoveryConfig {
crate::discovery::lan::LanDiscoveryConfig::default()
}
}
/// Peer rendezvous (`node.rendezvous.*`): how the node finds peers to connect
/// to at all — Nostr-mediated overlay endpoints and mDNS/DNS-SD on the local
/// link. Distinct from mesh lookup ([`LookupConfig`]), which finds coordinates
/// for an already-known mesh address.
#[derive(Debug, Clone, Default, Serialize, Deserialize)]
pub struct RendezvousConfig {
/// Nostr-mediated overlay endpoint rendezvous (`node.rendezvous.nostr.*`).
#[serde(default)]
pub nostr: NostrRendezvousConfig,
/// mDNS / DNS-SD peer rendezvous on the local link (`node.rendezvous.lan.*`).
/// Identity surface is a strict subset of what `nostr.advertise` already
/// publishes publicly, so there's no marginal privacy cost; the latency
/// win for same-LAN peers is large (sub-second pairing, no relay).
#[serde(default)]
pub lan: crate::mdns::LanRendezvousConfig,
}
/// COMPAT (drop at the v2 cutover): a deprecated legacy `node.discovery:` block.
///
/// The `node.discovery.*` table was split into `node.lookup.*` (mesh-lookup
/// scalars) and `node.rendezvous.*` (nostr/LAN peer rendezvous). Because
/// `NodeConfig` does not deny unknown fields, a still-deployed `node.discovery:`
/// block would otherwise deserialize into nothing and silently revert every
/// lookup/rendezvous setting to its default. This all-`Option` mirror captures
/// it so [`Config::normalize_deprecated_keys`] can fold it into the new tables
/// with a one-time deprecation warning; unset legacy keys stay `None` and leave
/// the new-table defaults intact.
#[derive(Debug, Clone, Deserialize)]
pub(crate) struct DiscoveryConfigCompat {
pub ttl: Option<u8>,
pub attempt_timeouts_secs: Option<Vec<u64>>,
pub recent_expiry_secs: Option<u64>,
pub backoff_base_secs: Option<u64>,
pub backoff_max_secs: Option<u64>,
pub forward_min_interval_secs: Option<u64>,
pub nostr: Option<NostrRendezvousConfig>,
pub lan: Option<crate::mdns::LanRendezvousConfig>,
}
/// Nostr advert discovery policy.
@@ -278,33 +303,33 @@ impl DiscoveryConfig {
/// - `open`: also consider adverts for non-configured peers
#[derive(Debug, Clone, Copy, Default, PartialEq, Eq, Serialize, Deserialize)]
#[serde(rename_all = "snake_case")]
pub enum NostrDiscoveryPolicy {
pub enum NostrRendezvousPolicy {
Disabled,
#[default]
ConfiguredOnly,
Open,
}
/// Nostr-mediated overlay endpoint discovery (`node.discovery.nostr.*`).
/// Nostr-mediated overlay endpoint discovery (`node.rendezvous.nostr.*`).
#[derive(Debug, Clone, Serialize, Deserialize)]
#[serde(deny_unknown_fields)]
pub struct NostrDiscoveryConfig {
pub struct NostrRendezvousConfig {
/// Enable Nostr-signaled traversal bootstrap.
#[serde(default)]
pub enabled: bool,
/// Publish service advertisements so remote peers can bootstrap inbound.
#[serde(default = "NostrDiscoveryConfig::default_advertise")]
#[serde(default = "NostrRendezvousConfig::default_advertise")]
pub advertise: bool,
/// Relay URLs used for service advertisements.
#[serde(default = "NostrDiscoveryConfig::default_advert_relays")]
#[serde(default = "NostrRendezvousConfig::default_advert_relays")]
pub advert_relays: Vec<String>,
/// Relay URLs used for encrypted signaling events.
#[serde(default = "NostrDiscoveryConfig::default_dm_relays")]
#[serde(default = "NostrRendezvousConfig::default_dm_relays")]
pub dm_relays: Vec<String>,
/// STUN servers used for local reflexive address discovery.
/// Outbound observation uses only this local list; peer-advertised STUN
/// values are informational and are not treated as egress targets.
#[serde(default = "NostrDiscoveryConfig::default_stun_servers")]
#[serde(default = "NostrRendezvousConfig::default_stun_servers")]
pub stun_servers: Vec<String>,
/// Whether to advertise local (RFC 1918 / ULA) interface addresses as
/// host candidates in the traversal offer.
@@ -318,85 +343,85 @@ pub struct NostrDiscoveryConfig {
#[serde(default)]
pub share_local_candidates: bool,
/// Traversal application namespace and advert identifier suffix.
#[serde(default = "NostrDiscoveryConfig::default_app")]
#[serde(default = "NostrRendezvousConfig::default_app")]
pub app: String,
/// Signaling TTL in seconds.
#[serde(default = "NostrDiscoveryConfig::default_signal_ttl_secs")]
#[serde(default = "NostrRendezvousConfig::default_signal_ttl_secs")]
pub signal_ttl_secs: u64,
/// Policy for advert-derived endpoint discovery.
#[serde(default)]
pub policy: NostrDiscoveryPolicy,
pub policy: NostrRendezvousPolicy,
/// Max number of open-discovery peers queued for outbound retry/connection
/// at once. Prevents unbounded queue growth from ambient advert traffic.
#[serde(default = "NostrDiscoveryConfig::default_open_discovery_max_pending")]
#[serde(default = "NostrRendezvousConfig::default_open_discovery_max_pending")]
pub open_discovery_max_pending: usize,
/// Max concurrent inbound traversal offers processed at once.
/// Acts as a rate limit against offer spam from relays.
#[serde(default = "NostrDiscoveryConfig::default_max_concurrent_incoming_offers")]
#[serde(default = "NostrRendezvousConfig::default_max_concurrent_incoming_offers")]
pub max_concurrent_incoming_offers: usize,
/// Max cached overlay adverts retained from relay traffic.
/// Bounds memory under ambient advert volume.
#[serde(default = "NostrDiscoveryConfig::default_advert_cache_max_entries")]
#[serde(default = "NostrRendezvousConfig::default_advert_cache_max_entries")]
pub advert_cache_max_entries: usize,
/// Max seen-session IDs retained for replay detection.
/// Oldest entries are evicted when the cap is exceeded.
#[serde(default = "NostrDiscoveryConfig::default_seen_sessions_max_entries")]
#[serde(default = "NostrRendezvousConfig::default_seen_sessions_max_entries")]
pub seen_sessions_max_entries: usize,
/// Overall punch attempt timeout in seconds.
#[serde(default = "NostrDiscoveryConfig::default_attempt_timeout_secs")]
#[serde(default = "NostrRendezvousConfig::default_attempt_timeout_secs")]
pub attempt_timeout_secs: u64,
/// Replay tracking retention window in seconds.
#[serde(default = "NostrDiscoveryConfig::default_replay_window_secs")]
#[serde(default = "NostrRendezvousConfig::default_replay_window_secs")]
pub replay_window_secs: u64,
/// Delay before punch traffic starts.
#[serde(default = "NostrDiscoveryConfig::default_punch_start_delay_ms")]
#[serde(default = "NostrRendezvousConfig::default_punch_start_delay_ms")]
pub punch_start_delay_ms: u64,
/// Interval between punch packets.
#[serde(default = "NostrDiscoveryConfig::default_punch_interval_ms")]
#[serde(default = "NostrRendezvousConfig::default_punch_interval_ms")]
pub punch_interval_ms: u64,
/// How long to keep punching before failure.
#[serde(default = "NostrDiscoveryConfig::default_punch_duration_ms")]
#[serde(default = "NostrRendezvousConfig::default_punch_duration_ms")]
pub punch_duration_ms: u64,
/// Advert TTL in seconds.
#[serde(default = "NostrDiscoveryConfig::default_advert_ttl_secs")]
#[serde(default = "NostrRendezvousConfig::default_advert_ttl_secs")]
pub advert_ttl_secs: u64,
/// How often adverts are refreshed in seconds.
#[serde(default = "NostrDiscoveryConfig::default_advert_refresh_secs")]
#[serde(default = "NostrRendezvousConfig::default_advert_refresh_secs")]
pub advert_refresh_secs: u64,
/// Settle delay in seconds after Nostr discovery starts before the
/// one-shot startup sweep of cached adverts runs. Allows the relay
/// subscription backlog to populate the in-memory advert cache.
/// Only used under `policy: open`. Default: 5.
#[serde(default = "NostrDiscoveryConfig::default_startup_sweep_delay_secs")]
#[serde(default = "NostrRendezvousConfig::default_startup_sweep_delay_secs")]
pub startup_sweep_delay_secs: u64,
/// Maximum age in seconds for cached adverts considered by the
/// one-shot startup sweep. Adverts whose `created_at` is older than
/// `now - startup_sweep_max_age_secs` are skipped. Only used under
/// `policy: open`. Default: 3600 (1 hour).
#[serde(default = "NostrDiscoveryConfig::default_startup_sweep_max_age_secs")]
#[serde(default = "NostrRendezvousConfig::default_startup_sweep_max_age_secs")]
pub startup_sweep_max_age_secs: u64,
/// Number of consecutive NAT-traversal failures against a peer before
/// an extended cooldown is applied to throttle further offer publishes.
/// At this threshold the daemon also actively re-fetches the peer's
/// advert from `advert_relays` to evict cache entries for peers that
/// have gone away. Default: 5.
#[serde(default = "NostrDiscoveryConfig::default_failure_streak_threshold")]
#[serde(default = "NostrRendezvousConfig::default_failure_streak_threshold")]
pub failure_streak_threshold: u32,
/// Cooldown applied to a peer once `failure_streak_threshold` is hit.
/// Suppresses both open-discovery sweep enqueues and per-attempt
/// retry firings until elapsed. Default: 1800 (30 minutes).
#[serde(default = "NostrDiscoveryConfig::default_extended_cooldown_secs")]
#[serde(default = "NostrRendezvousConfig::default_extended_cooldown_secs")]
pub extended_cooldown_secs: u64,
/// Minimum interval between `NAT traversal failed` WARN log lines for
/// the same peer. Subsequent failures inside the window log at DEBUG.
/// Reduces log spam on public-test nodes with many cache-learned
/// peers. Default: 300 (5 minutes).
#[serde(default = "NostrDiscoveryConfig::default_warn_log_interval_secs")]
#[serde(default = "NostrRendezvousConfig::default_warn_log_interval_secs")]
pub warn_log_interval_secs: u64,
/// Maximum entries retained in the per-npub failure-state map.
/// Bounds memory under high cache turnover. Oldest entries (by last
/// failure time) evicted when the cap is exceeded. Default: 4096.
#[serde(default = "NostrDiscoveryConfig::default_failure_state_max_entries")]
#[serde(default = "NostrRendezvousConfig::default_failure_state_max_entries")]
pub failure_state_max_entries: usize,
/// Cooldown applied after observing a fatal protocol mismatch on a
/// Nostr-adopted bootstrap transport (e.g. `Unknown FMP version`
@@ -404,11 +429,11 @@ pub struct NostrDiscoveryConfig {
/// of `extended_cooldown_secs` and much longer because the mismatch
/// is structural — re-traversing the peer is wasted effort until one
/// side upgrades. Default: 86400 (24 hours).
#[serde(default = "NostrDiscoveryConfig::default_protocol_mismatch_cooldown_secs")]
#[serde(default = "NostrRendezvousConfig::default_protocol_mismatch_cooldown_secs")]
pub protocol_mismatch_cooldown_secs: u64,
}
impl Default for NostrDiscoveryConfig {
impl Default for NostrRendezvousConfig {
fn default() -> Self {
Self {
enabled: false,
@@ -419,7 +444,7 @@ impl Default for NostrDiscoveryConfig {
share_local_candidates: false,
app: Self::default_app(),
signal_ttl_secs: Self::default_signal_ttl_secs(),
policy: NostrDiscoveryPolicy::default(),
policy: NostrRendezvousPolicy::default(),
open_discovery_max_pending: Self::default_open_discovery_max_pending(),
max_concurrent_incoming_offers: Self::default_max_concurrent_incoming_offers(),
advert_cache_max_entries: Self::default_advert_cache_max_entries(),
@@ -442,7 +467,7 @@ impl Default for NostrDiscoveryConfig {
}
}
impl NostrDiscoveryConfig {
impl NostrRendezvousConfig {
fn default_advertise() -> bool {
true
}
@@ -635,8 +660,8 @@ pub struct BloomConfig {
pub update_debounce_ms: u64,
/// Antipoison cap: reject inbound FilterAnnounce whose FPR exceeds
/// this value (`node.bloom.max_inbound_fpr`). Valid range `(0.0, 1.0)`.
/// Default `0.10` ≈ fill 0.631 at k=5 ≈ ~1,630 entries on the 1 KB
/// filter (Swamidass–Baldi). Raised from 0.05 so aggregates that are
/// Default `0.20` ≈ fill 0.7248 at k=5 ≈ ~2,114 entries on the 1 KB
/// filter (Swamidass–Baldi). Raised from 0.10 so aggregates that are
/// legitimately near their operating ceiling are not rejected before
/// the network reaches the fixed-filter capacity limit; conceptually
/// distinct from future autoscaling hysteresis setpoints — same unit,
@@ -648,8 +673,8 @@ pub struct BloomConfig {
impl Default for BloomConfig {
fn default() -> Self {
Self {
update_debounce_ms: 500,
max_inbound_fpr: 0.10,
update_debounce_ms: Self::default_update_debounce_ms(),
max_inbound_fpr: Self::default_max_inbound_fpr(),
}
}
}
@@ -659,7 +684,7 @@ impl BloomConfig {
500
}
fn default_max_inbound_fpr() -> f64 {
0.10
0.20
}
}
@@ -726,6 +751,41 @@ impl SessionConfig {
}
}
/// MMP configuration (`node.mmp.*`).
#[derive(Debug, Clone, Serialize, Deserialize)]
pub struct MmpConfig {
/// Operating mode (`node.mmp.mode`).
#[serde(default)]
pub mode: MmpMode,
/// Periodic operator log interval in seconds (`node.mmp.log_interval_secs`).
#[serde(default = "MmpConfig::default_log_interval_secs")]
pub log_interval_secs: u64,
/// OWD trend ring buffer size (`node.mmp.owd_window_size`).
#[serde(default = "MmpConfig::default_owd_window_size")]
pub owd_window_size: usize,
}
impl Default for MmpConfig {
fn default() -> Self {
Self {
mode: MmpMode::default(),
log_interval_secs: DEFAULT_LOG_INTERVAL_SECS,
owd_window_size: DEFAULT_OWD_WINDOW_SIZE,
}
}
}
impl MmpConfig {
fn default_log_interval_secs() -> u64 {
DEFAULT_LOG_INTERVAL_SECS
}
fn default_owd_window_size() -> usize {
DEFAULT_OWD_WINDOW_SIZE
}
}
/// Session-layer Metrics Measurement Protocol (`node.session_mmp.*`).
///
/// Separate from link-layer `node.mmp.*` to allow independent mode/interval
@@ -970,6 +1030,19 @@ pub struct NodeConfig {
#[serde(default = "NodeConfig::default_link_dead_timeout_secs")]
pub link_dead_timeout_secs: u64,
/// Graceful-shutdown drain deadline in seconds (`node.drain_timeout_secs`).
/// The bounded `Draining` phase broadcasts a shutdown `Disconnect` and then
/// waits up to this long for peers to clear before tearing down, early-
/// exiting as soon as all peers are gone. `None` selects the 2-second
/// default (see [`NodeConfig::drain_timeout`]).
///
/// Kept `Option` deliberately: `NodeConfig` has no `deny_unknown_fields`, so
/// a naive non-`Option` add with a `default` fn would silently rewrite the
/// value into deployed configs on the next serialize. The `Option` +
/// `skip_serializing_if` keeps absent configs absent.
#[serde(default, skip_serializing_if = "Option::is_none")]
pub drain_timeout_secs: Option<u64>,
/// Resource limits (`node.limits.*`).
#[serde(default)]
pub limits: LimitsConfig,
@@ -986,9 +1059,19 @@ pub struct NodeConfig {
#[serde(default)]
pub cache: CacheConfig,
/// Discovery protocol (`node.discovery.*`).
/// Mesh-lookup protocol (`node.lookup.*`).
#[serde(default)]
pub discovery: DiscoveryConfig,
pub lookup: LookupConfig,
/// Peer rendezvous (`node.rendezvous.*`).
#[serde(default)]
pub rendezvous: RendezvousConfig,
/// COMPAT (drop at the v2 cutover): a deprecated legacy `node.discovery:`
/// block, folded into `lookup`/`rendezvous` by
/// [`Config::normalize_deprecated_keys`]. Never re-serialized.
#[serde(default, skip_serializing)]
pub(crate) discovery: Option<DiscoveryConfigCompat>,
/// Spanning tree (`node.tree.*`).
#[serde(default)]
@@ -1041,11 +1124,14 @@ impl Default for NodeConfig {
base_rtt_ms: 100,
heartbeat_interval_secs: 10,
link_dead_timeout_secs: 30,
drain_timeout_secs: None,
limits: LimitsConfig::default(),
rate_limit: RateLimitConfig::default(),
retry: RetryConfig::default(),
cache: CacheConfig::default(),
discovery: DiscoveryConfig::default(),
lookup: LookupConfig::default(),
rendezvous: RendezvousConfig::default(),
discovery: None,
tree: TreeConfig::default(),
bloom: BloomConfig::default(),
session: SessionConfig::default(),
@@ -1089,12 +1175,72 @@ impl NodeConfig {
fn default_link_dead_timeout_secs() -> u64 {
30
}
/// Graceful-shutdown drain deadline as a `Duration`.
///
/// Returns the configured `drain_timeout_secs`, or the 2-second default
/// when unset. Used by the daemon's bounded `Draining` phase.
pub fn drain_timeout(&self) -> std::time::Duration {
std::time::Duration::from_secs(self.drain_timeout_secs.unwrap_or(2))
}
}
#[cfg(test)]
mod tests {
use super::*;
#[test]
fn test_config_default() {
let config = MmpConfig::default();
assert_eq!(config.mode, MmpMode::Full);
assert_eq!(config.log_interval_secs, 30);
assert_eq!(config.owd_window_size, 32);
}
#[test]
fn test_config_yaml_parse() {
let yaml = r#"
mode: lightweight
log_interval_secs: 60
owd_window_size: 48
"#;
let config: MmpConfig = serde_yaml::from_str(yaml).unwrap();
assert_eq!(config.mode, MmpMode::Lightweight);
assert_eq!(config.log_interval_secs, 60);
assert_eq!(config.owd_window_size, 48);
}
#[test]
fn test_config_yaml_partial() {
let yaml = "mode: minimal";
let config: MmpConfig = serde_yaml::from_str(yaml).unwrap();
assert_eq!(config.mode, MmpMode::Minimal);
assert_eq!(config.log_interval_secs, DEFAULT_LOG_INTERVAL_SECS);
assert_eq!(config.owd_window_size, DEFAULT_OWD_WINDOW_SIZE);
}
#[test]
fn test_drain_timeout_default_and_override() {
// Unset → the 2-second default.
let c = NodeConfig::default();
assert_eq!(c.drain_timeout_secs, None);
assert_eq!(c.drain_timeout(), std::time::Duration::from_secs(2));
// Explicit override is honored.
let c2 = NodeConfig {
drain_timeout_secs: Some(10),
..NodeConfig::default()
};
assert_eq!(c2.drain_timeout(), std::time::Duration::from_secs(10));
// A zero override is a valid (immediate) drain, not the default.
let c3 = NodeConfig {
drain_timeout_secs: Some(0),
..NodeConfig::default()
};
assert_eq!(c3.drain_timeout(), std::time::Duration::from_secs(0));
}
#[test]
fn test_ecn_config_defaults() {
let c = EcnConfig::default();
@@ -1123,27 +1269,27 @@ mod tests {
}
#[test]
fn test_nostr_discovery_startup_sweep_defaults() {
let c = NostrDiscoveryConfig::default();
fn test_nostr_rendezvous_startup_sweep_defaults() {
let c = NostrRendezvousConfig::default();
assert_eq!(c.startup_sweep_delay_secs, 5);
assert_eq!(c.startup_sweep_max_age_secs, 3_600);
}
#[test]
fn test_nostr_discovery_startup_sweep_yaml_override() {
fn test_nostr_rendezvous_startup_sweep_yaml_override() {
let yaml = "enabled: true\npolicy: open\nstartup_sweep_delay_secs: 10\nstartup_sweep_max_age_secs: 1800\n";
let c: NostrDiscoveryConfig = serde_yaml::from_str(yaml).unwrap();
let c: NostrRendezvousConfig = serde_yaml::from_str(yaml).unwrap();
assert!(c.enabled);
assert_eq!(c.policy, NostrDiscoveryPolicy::Open);
assert_eq!(c.policy, NostrRendezvousPolicy::Open);
assert_eq!(c.startup_sweep_delay_secs, 10);
assert_eq!(c.startup_sweep_max_age_secs, 1_800);
}
#[test]
fn test_nostr_discovery_startup_sweep_partial_yaml_uses_defaults() {
fn test_nostr_rendezvous_startup_sweep_partial_yaml_uses_defaults() {
// Only override delay; max_age should fall back to default.
let yaml = "enabled: true\nstartup_sweep_delay_secs: 30\n";
let c: NostrDiscoveryConfig = serde_yaml::from_str(yaml).unwrap();
let c: NostrRendezvousConfig = serde_yaml::from_str(yaml).unwrap();
assert_eq!(c.startup_sweep_delay_secs, 30);
assert_eq!(c.startup_sweep_max_age_secs, 3_600);
}
src/config/transport.rs
+25 -6
View File
@@ -282,9 +282,10 @@ pub struct EthernetConfig {
#[serde(default, skip_serializing_if = "Option::is_none")]
pub send_buf_size: Option<usize>,
/// Listen for discovery beacons from other nodes. Default: true.
#[serde(default, skip_serializing_if = "Option::is_none")]
pub discovery: Option<bool>,
/// Listen for neighbor beacons from other nodes. Default: true.
/// (Renamed from `discovery`; the old key is still accepted.)
#[serde(default, alias = "discovery", skip_serializing_if = "Option::is_none")]
pub listen: Option<bool>,
/// Broadcast announcement beacons on the LAN. Default: false.
#[serde(default, skip_serializing_if = "Option::is_none")]
@@ -319,9 +320,9 @@ impl EthernetConfig {
self.send_buf_size.unwrap_or(DEFAULT_ETHERNET_SEND_BUF)
}
/// Whether to listen for discovery beacons. Default: true.
pub fn discovery(&self) -> bool {
self.discovery.unwrap_or(true)
/// Whether to listen for neighbor beacons. Default: true.
pub fn listen(&self) -> bool {
self.listen.unwrap_or(true)
}
/// Whether to broadcast announcement beacons. Default: false.
@@ -1046,4 +1047,22 @@ mod tests {
assert_eq!(parse_bind_port("[::]:443"), Some(443));
assert_eq!(parse_bind_port("not-a-socket-addr"), None);
}
#[test]
fn ethernet_listen_accepts_legacy_discovery_alias_and_rejects_unknown() {
// (a) The legacy `discovery:` key is still accepted via serde alias.
let legacy: EthernetConfig =
serde_yaml::from_str("interface: eth0\ndiscovery: true\n").unwrap();
assert_eq!(legacy.listen, Some(true));
// (b) The new canonical `listen:` key parses into the renamed field.
let renamed: EthernetConfig =
serde_yaml::from_str("interface: eth0\nlisten: true\n").unwrap();
assert_eq!(renamed.listen, Some(true));
// (c) `deny_unknown_fields` still rejects an unknown ethernet key.
let bogus: Result<EthernetConfig, _> =
serde_yaml::from_str("interface: eth0\nbogus: true\n");
assert!(bogus.is_err());
}
}
src/control/queries.rs
+23 -14
View File
@@ -236,7 +236,7 @@ pub fn show_peers(node: &Node) -> Value {
// Per-npub Nostr-traversal failure-state snapshot, indexed by npub
// for O(1) per-peer lookup. Empty if Nostr discovery is disabled.
let nostr_state: std::collections::HashMap<String, _> = node
.nostr_discovery_handle()
.nostr_rendezvous_handle()
.map(|d| {
d.failure_state_snapshot()
.into_iter()
@@ -1303,17 +1303,18 @@ pub fn show_connections(node: &Node) -> Value {
let now = now_ms();
let connections: Vec<Value> = node
.connections()
.map(|conn| {
.map(|(_, machine)| {
let link_id = machine.link_id();
let mut conn_json = json!({
"link_id": conn.link_id().as_u64(),
"direction": format!("{}", conn.direction()),
"handshake_state": format!("{}", conn.handshake_state()),
"started_at_ms": conn.started_at(),
"idle_ms": now.saturating_sub(conn.last_activity()),
"resend_count": conn.resend_count(),
"link_id": link_id.as_u64(),
"direction": format!("{}", machine.conn_direction()),
"handshake_state": node.connection_handshake_state(link_id),
"started_at_ms": node.connection_started_at(link_id),
"idle_ms": now.saturating_sub(node.connection_last_activity(link_id)),
"resend_count": node.connection_resend_count(link_id),
});
if let Some(identity) = conn.expected_identity() {
if let Some(identity) = node.connection_expected_identity(link_id) {
conn_json["expected_peer"] = json!(identity.npub());
}
@@ -1487,7 +1488,9 @@ pub fn show_routing(node: &Node) -> Value {
"recent_requests": node.recent_request_count(),
"retries": retries,
"forwarding": serde_json::to_value(metrics.forwarding.snapshot()).unwrap_or_default(),
"discovery": serde_json::to_value(metrics.discovery.snapshot()).unwrap_or_default(),
// COMPAT: `discovery` is the deprecated alias for `lookup`; drop at the v2 cutover.
"discovery": serde_json::to_value(metrics.lookup.snapshot()).unwrap_or_default(),
"lookup": serde_json::to_value(metrics.lookup.snapshot()).unwrap_or_default(),
"error_signals": serde_json::to_value(metrics.errors.snapshot()).unwrap_or_default(),
"congestion": serde_json::to_value(metrics.congestion.snapshot()).unwrap_or_default(),
})
@@ -1543,7 +1546,9 @@ pub(crate) fn show_routing_from_handle(handle: &super::read_handle::ControlReadH
"recent_requests": view.recent_requests,
"retries": retries,
"forwarding": serde_json::to_value(metrics.forwarding.snapshot()).unwrap_or_default(),
"discovery": serde_json::to_value(metrics.discovery.snapshot()).unwrap_or_default(),
// COMPAT: `discovery` is the deprecated alias for `lookup`; drop at the v2 cutover.
"discovery": serde_json::to_value(metrics.lookup.snapshot()).unwrap_or_default(),
"lookup": serde_json::to_value(metrics.lookup.snapshot()).unwrap_or_default(),
"error_signals": serde_json::to_value(metrics.errors.snapshot()).unwrap_or_default(),
"congestion": serde_json::to_value(metrics.congestion.snapshot()).unwrap_or_default(),
})
@@ -2328,7 +2333,9 @@ pub(crate) fn show_metrics_from_handle(handle: &super::read_handle::ControlReadH
let m = handle.metrics();
json!({
"forwarding": m.forwarding.snapshot(),
"discovery": m.discovery.snapshot(),
// COMPAT: `discovery` is the deprecated alias for `lookup`; drop at the v2 cutover.
"discovery": m.lookup.snapshot(),
"lookup": m.lookup.snapshot(),
"tree": m.tree.snapshot(),
"bloom": m.bloom.snapshot(),
"congestion": m.congestion.snapshot(),
@@ -2761,12 +2768,12 @@ mod tests {
/// Structural confirmation that the rx_loop no longer dispatches `show_*`:
/// the rx_loop source carries no `queries::dispatch` call and no
/// `starts_with("show_")` routing branch. Reads the committed source of
/// `src/node/handlers/rx_loop.rs` and asserts both markers are absent. This
/// `src/node/dataplane/rx_loop.rs` and asserts both markers are absent. This
/// is the milestone's "remove `show_*` from the data-plane dispatch path"
/// invariant, guarded against regression.
#[test]
fn rx_loop_has_no_show_dispatch() {
let src = include_str!("../node/handlers/rx_loop.rs");
let src = include_str!("../node/dataplane/rx_loop.rs");
assert!(
!src.contains("queries::dispatch"),
"rx_loop must not call queries::dispatch (show_* served off-loop)"
@@ -2794,7 +2801,9 @@ mod tests {
let expected_families = [
("forwarding", "received_packets"),
// `discovery` is the deprecated dual-emit alias for `lookup`; drop at the v2 cutover.
("discovery", "req_received"),
("lookup", "req_received"),
("tree", "accepted"),
("bloom", "accepted"),
("congestion", "ce_forwarded"),
src/control/read_handle.rs
+31
View File
@@ -118,10 +118,41 @@ impl ControlReadHandle {
/// Cutover queries (R1) read only `NodeContext` / `MetricsRegistry` (the state
/// the read handle already bundles) plus host-OS facts (`/proc`, nftables), so
/// they render entirely in the control task without touching `Node`.
///
/// **It now also carries mutating commands**, namely the `profile_tick_*`
/// family under the `profiling` feature. They are served here rather than on
/// the rx_loop deliberately: all of their state is process statics, they need
/// no `&mut Node`, and routing them through the loop would make the toggle
/// queue behind the very behavior it exists to measure.
pub(crate) fn snapshot_dispatch(request: &Request, handle: &ControlReadHandle) -> Option<Response> {
use crate::control::queries;
match request.command.as_str() {
// Tick-body profiler toggle. Present only in a `--features profiling`
// build; otherwise these fall through to the rx_loop dispatch, which
// reports them as unknown commands.
#[cfg(feature = "profiling")]
"profile_tick_on" => {
let dir = request
.params
.as_ref()
.and_then(|p| p.get("dir"))
.and_then(|v| v.as_str());
let context = handle.context();
let npub = context.identity.npub();
let period = context.config.node.tick_interval_secs;
Some(match crate::instr::capture::start(dir, &npub, period) {
Ok(value) => Response::ok(value),
Err(e) => Response::error(e),
})
}
#[cfg(feature = "profiling")]
"profile_tick_off" => Some(match crate::instr::capture::stop() {
Ok(value) => Response::ok(value),
Err(e) => Response::error(e),
}),
#[cfg(feature = "profiling")]
"profile_tick_status" => Some(Response::ok(crate::instr::capture::status())),
"show_listening_sockets" => Some(Response::ok(
queries::show_listening_sockets_from_handle(handle),
)),
src/control/snapshots/show_routing.json
+24
View File
@@ -63,6 +63,30 @@
"ttl_exhausted_packets": 0
},
"identity_cache_entries": 0,
"lookup": {
"req_backoff_suppressed": 0,
"req_bloom_miss": 0,
"req_decode_error": 0,
"req_dedup_cache_full": 0,
"req_deduplicated": 0,
"req_duplicate": 0,
"req_fallback_forwarded": 0,
"req_forward_rate_limited": 0,
"req_forwarded": 0,
"req_initiated": 0,
"req_no_tree_peer": 0,
"req_received": 0,
"req_target_is_us": 0,
"req_ttl_exhausted": 0,
"resp_accepted": 0,
"resp_decode_error": 0,
"resp_forwarded": 0,
"resp_identity_miss": 0,
"resp_no_route": 0,
"resp_proof_failed": 0,
"resp_received": 0,
"resp_timed_out": 0
},
"pending_lookups": [],
"pending_tun_destinations": 0,
"pending_tun_packets": 0,
src/control/snapshots/show_tree.json
-1
View File
@@ -24,7 +24,6 @@
"loop_detected": 0,
"outbound_sign_failed": 0,
"parent_losses": 0,
"parent_switched": 0,
"parent_switches": 0,
"rate_limited": 0,
"received": 0,
src/identity/auth.rs
+3 -3
View File
@@ -1,7 +1,7 @@
//! Authentication challenge-response protocol.
use rand::Rng;
use secp256k1::{Secp256k1, XOnlyPublicKey};
use secp256k1::XOnlyPublicKey;
use sha2::{Digest, Sha256};
use super::{IdentityError, NodeAddr};
@@ -34,9 +34,9 @@ impl AuthChallenge {
/// Verify a response to this challenge.
pub fn verify(&self, response: &AuthResponse) -> Result<NodeAddr, IdentityError> {
let digest = auth_challenge_digest(&self.0, response.timestamp);
let secp = Secp256k1::new();
secp.verify_schnorr(&response.signature, &digest, &response.pubkey)
super::SECP
.verify_schnorr(&response.signature, &digest, &response.pubkey)
.map_err(|_| IdentityError::SignatureVerificationFailed)?;
Ok(NodeAddr::from_pubkey(&response.pubkey))
src/identity/local.rs
+4 -7
View File
@@ -1,6 +1,6 @@
//! Local node identity with signing capability.
use secp256k1::{Keypair, PublicKey, Secp256k1, SecretKey, XOnlyPublicKey};
use secp256k1::{Keypair, PublicKey, SecretKey, XOnlyPublicKey};
use std::fmt;
use super::auth::{AuthResponse, auth_challenge_digest};
@@ -42,8 +42,7 @@ impl Identity {
/// Create an identity from a secret key.
pub fn from_secret_key(secret_key: SecretKey) -> Self {
let secp = Secp256k1::new();
let keypair = Keypair::from_secret_key(&secp, &secret_key);
let keypair = Keypair::from_secret_key(&super::SECP, &secret_key);
Self::from_keypair(keypair)
}
@@ -93,9 +92,8 @@ impl Identity {
/// Sign arbitrary data with this identity's secret key.
pub fn sign(&self, data: &[u8]) -> secp256k1::schnorr::Signature {
let secp = Secp256k1::new();
let digest = sha256(data);
secp.sign_schnorr(&digest, &self.keypair)
super::SECP.sign_schnorr(&digest, &self.keypair)
}
/// Create an authentication response for a challenge.
@@ -103,8 +101,7 @@ impl Identity {
/// The response signs: SHA256("fips-auth-v1" || challenge || timestamp)
pub fn sign_challenge(&self, challenge: &[u8; 32], timestamp: u64) -> AuthResponse {
let digest = auth_challenge_digest(challenge, timestamp);
let secp = Secp256k1::new();
let signature = secp.sign_schnorr(&digest, &self.keypair);
let signature = super::SECP.sign_schnorr(&digest, &self.keypair);
AuthResponse {
pubkey: self.pubkey(),
timestamp,
src/identity/mod.rs
+12
View File
@@ -11,6 +11,9 @@ mod local;
mod node_addr;
mod peer;
use std::sync::LazyLock;
use secp256k1::{All, Secp256k1};
use sha2::{Digest, Sha256};
use thiserror::Error;
@@ -21,6 +24,15 @@ pub use local::Identity;
pub use node_addr::NodeAddr;
pub use peer::PeerIdentity;
/// Shared secp256k1 context reused across all identity operations.
///
/// `Secp256k1::new()` allocates a `Secp256k1<All>` and runs randomization /
/// blinding table setup; it is designed to be created once and reused rather
/// than rebuilt per sign / verify / key-derive call. This single `All` context
/// serves both signing and verification across the identity module and still
/// performs the standard construction-time blinding.
pub(crate) static SECP: LazyLock<Secp256k1<All>> = LazyLock::new(Secp256k1::new);
/// FIPS address prefix (IPv6 ULA range).
pub const FIPS_ADDRESS_PREFIX: u8 = 0xfd;
src/identity/peer.rs
+3 -3
View File
@@ -1,6 +1,6 @@
//! Remote peer identity (public key only, no signing capability).
use secp256k1::{Parity, PublicKey, Secp256k1, XOnlyPublicKey};
use secp256k1::{Parity, PublicKey, XOnlyPublicKey};
use std::fmt;
use super::encoding::{decode_npub, encode_npub};
@@ -107,9 +107,9 @@ impl PeerIdentity {
/// Verify a signature from this peer.
pub fn verify(&self, data: &[u8], signature: &secp256k1::schnorr::Signature) -> bool {
let secp = Secp256k1::new();
let digest = sha256(data);
secp.verify_schnorr(signature, &digest, &self.pubkey)
super::SECP
.verify_schnorr(signature, &digest, &self.pubkey)
.is_ok()
}
}
src/identity/tests.rs
+4 -5
View File
@@ -1,7 +1,7 @@
use std::collections::HashSet;
use std::net::Ipv6Addr;
use secp256k1::{Keypair, Secp256k1, SecretKey};
use secp256k1::{Keypair, SecretKey};
use super::*;
@@ -161,10 +161,10 @@ fn test_identity_sign() {
let sig = identity.sign(data);
// Verify the signature manually
let secp = secp256k1::Secp256k1::new();
let digest = super::sha256(data);
assert!(
secp.verify_schnorr(&sig, &digest, &identity.pubkey())
super::SECP
.verify_schnorr(&sig, &digest, &identity.pubkey())
.is_ok()
);
}
@@ -580,13 +580,12 @@ fn test_peer_identity_pubkey_full_even_parity_fallback() {
#[test]
fn test_peer_identity_pubkey_full_preserved_parity() {
// Create two identities and find one with odd parity to make this test meaningful
let secp = Secp256k1::new();
let secret_bytes: [u8; 32] = [
0x01, 0x02, 0x03, 0x04, 0x05, 0x06, 0x07, 0x08, 0x09, 0x0a, 0x0b, 0x0c, 0x0d, 0x0e, 0x0f,
0x10, 0x11, 0x12, 0x13, 0x14, 0x15, 0x16, 0x17, 0x18, 0x19, 0x1a, 0x1b, 0x1c, 0x1d, 0x1e,
0x1f, 0x20,
];
let keypair = Keypair::from_seckey_slice(&secp, &secret_bytes).unwrap();
let keypair = Keypair::from_seckey_slice(&super::SECP, &secret_bytes).unwrap();
let full_pubkey = keypair.public_key();
let peer = PeerIdentity::from_pubkey_full(full_pubkey);
src/instr/capture.rs
+418
View File
@@ -0,0 +1,418 @@
//! Capture lifecycle: the arm/disarm state machine, the sink file, and the
//! `fipsctl`-facing operations.
//!
//! The toggle — not the writer — creates and opens the sink and publishes its
//! path, so an unwritable directory fails the `on` command loudly instead of
//! being discovered later by a background thread with nobody to report to.
//!
//! Capture state is a single atomic state machine (`Idle`, `Running`,
//! `StoppedByCap`) transitioned by `compare_exchange`. Every accepted control
//! connection is served by its own spawned task, so two simultaneous `on`
//! requests are genuinely concurrent and must not both create a writer.
use std::fs::File;
use std::io::Write;
use std::path::PathBuf;
use std::sync::Mutex;
use std::sync::atomic::{AtomicBool, AtomicU8, AtomicU64, Ordering};
use std::time::{Duration, SystemTime, UNIX_EPOCH};
use super::recorder;
use super::writer;
/// Default sink directory. Overridable per capture with `--dir`.
pub(crate) const DEFAULT_DIR: &str = "/var/log/fips";
/// Writer flush interval.
pub(crate) const INTERVAL: Duration = Duration::from_secs(10);
/// Size at which a capture stops itself. Reaching it stops the capture rather
/// than rotating: the point of a capture is a bounded, self-describing window.
pub(crate) const BYTE_CAP: u64 = 32 * 1024 * 1024;
pub(crate) const IDLE: u8 = 0;
pub(crate) const RUNNING: u8 = 1;
pub(crate) const STOPPED_BY_CAP: u8 = 2;
/// The writer could not write and stopped itself. Distinct from a cap stop:
/// a capture that died on a full disk produced a truncated window, and calling
/// that "stopped_by_cap" tells the operator it ran to its limit when it did
/// not. The trailer line explaining it goes to the same failing file, so the
/// state is the only signal that survives.
pub(crate) const STOPPED_BY_ERROR: u8 = 3;
static STATE: AtomicU8 = AtomicU8::new(IDLE);
static GATE: AtomicBool = AtomicBool::new(false);
static BYTES: AtomicU64 = AtomicU64::new(0);
static ACTIVE_PATH: Mutex<Option<PathBuf>> = Mutex::new(None);
static WRITER: Mutex<Option<writer::Handle>> = Mutex::new(None);
/// The per-tick gate. One relaxed load per tick when the feature is compiled in
/// and no capture is running.
#[inline]
pub(crate) fn gate() -> bool {
GATE.load(Ordering::Relaxed)
}
pub(crate) fn bytes_written() -> u64 {
BYTES.load(Ordering::Relaxed)
}
pub(crate) fn add_bytes(n: u64) -> u64 {
BYTES.fetch_add(n, Ordering::Relaxed) + n
}
fn active_path() -> Option<PathBuf> {
ACTIVE_PATH
.lock()
.unwrap_or_else(|e| e.into_inner())
.clone()
}
fn path_display() -> String {
active_path()
.map(|p| p.display().to_string())
.unwrap_or_else(|| "<none>".to_string())
}
fn state_name(state: u8) -> &'static str {
match state {
RUNNING => "running",
STOPPED_BY_CAP => "stopped_by_cap",
STOPPED_BY_ERROR => "stopped_by_error",
_ => "idle",
}
}
/// Called by the writer when it stops itself. `terminal` is `STOPPED_BY_CAP`
/// or `STOPPED_BY_ERROR`. Returns true if this call is the one that stopped it.
pub(crate) fn mark_stopped(terminal: u8) -> bool {
debug_assert!(terminal == STOPPED_BY_CAP || terminal == STOPPED_BY_ERROR);
GATE.store(false, Ordering::Relaxed);
STATE
.compare_exchange(RUNNING, terminal, Ordering::AcqRel, Ordering::Acquire)
.is_ok()
}
/// Join the writer thread, if one exists. Never called while holding another
/// lock the writer might want.
fn reap() {
let handle = WRITER.lock().unwrap_or_else(|e| e.into_inner()).take();
if let Some(handle) = handle {
handle.stop_and_join();
}
}
/// Arm a capture.
///
/// Opens the sink first and only then starts the writer, so a bad `--dir` is
/// reported to the caller rather than logged into the void.
pub(crate) fn start(
dir: Option<&str>,
node_npub: &str,
tick_period_secs: u64,
) -> Result<serde_json::Value, String> {
claim()?;
match open_sink(dir, node_npub, tick_period_secs) {
Ok((file, path, header_len)) => {
recorder::reset();
BYTES.store(header_len, Ordering::Relaxed);
match writer::spawn(file) {
Ok(handle) => {
*WRITER.lock().unwrap_or_else(|e| e.into_inner()) = Some(handle);
*ACTIVE_PATH.lock().unwrap_or_else(|e| e.into_inner()) = Some(path.clone());
GATE.store(true, Ordering::Release);
Ok(serde_json::json!({
"state": "running",
"path": path.display().to_string(),
"interval_secs": INTERVAL.as_secs(),
"byte_cap": BYTE_CAP,
}))
}
Err(e) => {
let _ = std::fs::remove_file(&path);
BYTES.store(0, Ordering::Relaxed);
STATE.store(IDLE, Ordering::Release);
Err(format!("cannot start profile writer thread: {e}"))
}
}
}
Err(e) => {
BYTES.store(0, Ordering::Relaxed);
STATE.store(IDLE, Ordering::Release);
Err(e)
}
}
}
/// Take the capture slot, reaping a cap-stopped predecessor if that is what is
/// in the way.
fn claim() -> Result<(), String> {
match STATE.compare_exchange(IDLE, RUNNING, Ordering::AcqRel, Ordering::Acquire) {
Ok(_) => Ok(()),
Err(RUNNING) => Err(format!("capture already running: {}", path_display())),
Err(stopped @ (STOPPED_BY_CAP | STOPPED_BY_ERROR)) => {
reap();
*ACTIVE_PATH.lock().unwrap_or_else(|e| e.into_inner()) = None;
STATE
.compare_exchange(stopped, RUNNING, Ordering::AcqRel, Ordering::Acquire)
.map(|_| ())
.map_err(|_| "capture state changed concurrently; retry".to_string())
}
Err(_) => Err("capture in an unexpected state".to_string()),
}
}
/// Disarm the capture. Succeeds when nothing is running, reporting so.
pub(crate) fn stop() -> Result<serde_json::Value, String> {
let previous = STATE.load(Ordering::Acquire);
if previous == IDLE {
return Ok(serde_json::json!({"state": "idle", "stopped": false}));
}
GATE.store(false, Ordering::Release);
// The writer wakes on the stop message rather than after the interval, so
// this join returns promptly instead of parking the caller for up to one
// flush interval.
reap();
let path = path_display();
*ACTIVE_PATH.lock().unwrap_or_else(|e| e.into_inner()) = None;
let bytes = bytes_written();
// Clear the counter with the slot: a later `status` while idle must not
// report the previous capture's byte total as though a capture were live.
BYTES.store(0, Ordering::Relaxed);
STATE.store(IDLE, Ordering::Release);
Ok(serde_json::json!({
"state": "idle",
"stopped": true,
"stopped_by_cap": previous == STOPPED_BY_CAP,
"stopped_by_error": previous == STOPPED_BY_ERROR,
"path": path,
"bytes": bytes,
}))
}
/// Report capture state. Distinguishes all four states.
pub(crate) fn status() -> serde_json::Value {
let state = STATE.load(Ordering::Acquire);
serde_json::json!({
"state": state_name(state),
"path": active_path().map(|p| p.display().to_string()),
"bytes": bytes_written(),
"byte_cap": BYTE_CAP,
"interval_secs": INTERVAL.as_secs(),
})
}
/// Stop and reap at daemon teardown. Idempotent.
pub(crate) fn shutdown() {
if STATE.load(Ordering::Acquire) != IDLE {
let _ = stop();
}
}
/// Create the sink file and write its header block. Returns the open file, its
/// path, and the number of header bytes written.
fn open_sink(
dir: Option<&str>,
node_npub: &str,
tick_period_secs: u64,
) -> Result<(File, PathBuf, u64), String> {
let dir = PathBuf::from(dir.unwrap_or(DEFAULT_DIR));
std::fs::create_dir_all(&dir)
.map_err(|e| format!("cannot use profile directory {}: {e}", dir.display()))?;
let start_unix = SystemTime::now()
.duration_since(UNIX_EPOCH)
.map(|d| d.as_secs())
.unwrap_or(0);
let path = dir.join(format!("profile-{}.tsv", compact_utc(start_unix)));
let mut file = File::create(&path)
.map_err(|e| format!("cannot create profile file {}: {e}", path.display()))?;
let header = format!(
"# fips tick profile\n\
# node\t{node}\n\
# build\t{build}\n\
# platform\t{platform}\n\
# tick_period_secs\t{period}\n\
# interval_secs\t{interval}\n\
# byte_cap\t{cap}\n\
# start_utc\t{start_utc}\n\
# start_unix\t{start_unix}\n\
# NOTE\tstep durations are WALL CLOCK across await points, not CPU time:\n\
# NOTE\ta step that awaits I/O accrues the wait, and other tasks may run\n\
# NOTE\tinside that span. That is the intended measure for head-of-line\n\
# NOTE\tdelay; do not read a large step as CPU cost.\n\
# NOTE\tarm_starvation is measured directly as (entry time - the deadline\n\
# NOTE\tthe interval scheduled the tick for). It is NOT derived from\n\
# NOTE\ttick_entry_gap, which carries no starvation signal by itself:\n\
# NOTE\tunder a steady delay every gap is exactly one tick period.\n\
ts_unix\tkind\tdomain\tname\tcount\tmax\ttotal\tunit\n",
node = node_npub,
build = crate::version::short_version(),
platform = std::env::consts::OS,
period = tick_period_secs,
interval = INTERVAL.as_secs(),
cap = BYTE_CAP,
start_utc = iso_utc(start_unix),
start_unix = start_unix,
);
file.write_all(header.as_bytes())
.map_err(|e| format!("cannot write profile header to {}: {e}", path.display()))?;
Ok((file, path, header.len() as u64))
}
/// Break a Unix timestamp into UTC `(year, month, day, hour, minute, second)`.
///
/// Hinnant's `civil_from_days`, era-based. No date crate is in the dependency
/// set and one filename stamp does not justify adding one.
fn utc_parts(unix: u64) -> (i64, u32, u32, u32, u32, u32) {
let days = (unix / 86_400) as i64;
let secs = unix % 86_400;
let z = days + 719_468;
let era = z.div_euclid(146_097);
let doe = z.rem_euclid(146_097);
let yoe = (doe - doe / 1_460 + doe / 36_524 - doe / 146_096) / 365;
let y = yoe + era * 400;
let doy = doe - (365 * yoe + yoe / 4 - yoe / 100);
let mp = (5 * doy + 2) / 153;
let d = (doy - (153 * mp + 2) / 5 + 1) as u32;
let m = (if mp < 10 { mp + 3 } else { mp - 9 }) as u32;
let y = if m <= 2 { y + 1 } else { y };
(
y,
m,
d,
(secs / 3_600) as u32,
((secs % 3_600) / 60) as u32,
(secs % 60) as u32,
)
}
/// `20260727T191500Z` — filename-safe.
fn compact_utc(unix: u64) -> String {
let (y, mo, d, h, mi, s) = utc_parts(unix);
format!("{y:04}{mo:02}{d:02}T{h:02}{mi:02}{s:02}Z")
}
/// `2026-07-27T19:15:00Z` — for the header block.
fn iso_utc(unix: u64) -> String {
let (y, mo, d, h, mi, s) = utc_parts(unix);
format!("{y:04}-{mo:02}-{d:02}T{h:02}:{mi:02}:{s:02}Z")
}
#[cfg(test)]
mod tests {
use super::*;
#[test]
fn utc_parts_matches_known_instants() {
assert_eq!(utc_parts(0), (1970, 1, 1, 0, 0, 0));
assert_eq!(utc_parts(946_684_800), (2000, 1, 1, 0, 0, 0));
// 2026-07-27T19:15:00Z
assert_eq!(utc_parts(1_785_179_700), (2026, 7, 27, 19, 15, 0));
// Leap day.
assert_eq!(utc_parts(1_709_164_800), (2024, 2, 29, 0, 0, 0));
}
#[test]
fn stamps_render_expected_shapes() {
assert_eq!(compact_utc(1_785_179_700), "20260727T191500Z");
assert_eq!(iso_utc(1_785_179_700), "2026-07-27T19:15:00Z");
}
// The lock these tests take is shared with the recorder tests, which
// mutate the same statics. See `crate::instr::test_serial`.
#[test]
fn capture_round_trip_writes_header_and_rows() {
let _guard = crate::instr::test_serial();
let dir = tempfile::tempdir().expect("tempdir");
let dir_str = dir.path().to_str().unwrap().to_string();
let started = start(Some(&dir_str), "npub1test", 1).expect("start");
assert_eq!(started["state"], "running");
assert!(gate(), "gate must be armed while running");
let path = PathBuf::from(started["path"].as_str().unwrap());
// A second `on` is refused while one is running, and names the file.
let refused = start(Some(&dir_str), "npub1test", 1).unwrap_err();
assert!(refused.contains(&path.display().to_string()), "{refused}");
// Feed one observation so the drained rows are not all zero.
recorder::record(
recorder::Domain::Tick,
recorder::Step::WholeTick,
Duration::from_millis(7),
);
// Stopping wakes the writer immediately; it drains once more and joins.
let stopped = stop().expect("stop");
assert_eq!(stopped["stopped"], true);
assert_eq!(stopped["stopped_by_cap"], false);
assert!(!gate(), "gate must be clear after stop");
let text = std::fs::read_to_string(&path).expect("read capture");
assert!(text.starts_with("# fips tick profile\n"), "{text}");
assert!(text.contains("# node\tnpub1test\n"), "{text}");
assert!(
text.contains("ts_unix\tkind\tdomain\tname\tcount\tmax\ttotal\tunit\n"),
"{text}"
);
// The final drain emitted one row per emitted step, plus the gauges.
let rows: Vec<&str> = text
.lines()
.filter(|l| l.starts_with(|c: char| c.is_ascii_digit()))
.collect();
let expected_steps = recorder::STEPS.iter().filter(|s| s.emitted()).count();
assert_eq!(rows.len(), expected_steps + recorder::N_GAUGES);
// The 7 ms observation above is in the whole-tick row, converted to
// microseconds. Bounds rather than equality: the gate is process-wide,
// so a node under test elsewhere in this binary may have ticked into
// the same capture window.
let whole_tick = rows
.iter()
.find(|r| r.contains("\tstep\ttick\twhole_tick\t"))
.expect("whole_tick row");
let fields: Vec<&str> = whole_tick.split('\t').collect();
assert_eq!(fields.last(), Some(&"us"), "{whole_tick}");
assert!(
fields[4].parse::<u64>().unwrap() >= 1,
"count: {whole_tick}"
);
assert!(
fields[5].parse::<u64>().unwrap() >= 7_000,
"max: {whole_tick}"
);
assert!(
rows.iter()
.any(|r| r.contains("\tgauge\ttick\tarm_starvation\t")),
"{text}"
);
// A stop with nothing running is not an error.
let again = stop().expect("second stop");
assert_eq!(again["stopped"], false);
}
#[test]
fn start_fails_loudly_on_an_unwritable_directory() {
let _guard = crate::instr::test_serial();
let err = start(Some("/proc/fips-profile-should-not-exist"), "npub1test", 1)
.expect_err("must fail");
assert!(err.contains("profile directory"), "{err}");
// The failed attempt must leave the slot free for the next try.
assert_eq!(STATE.load(Ordering::Acquire), IDLE);
assert!(!gate());
}
#[test]
fn status_reports_the_bounds_it_is_enforcing() {
let _guard = crate::instr::test_serial();
let value = status();
assert_eq!(value["byte_cap"], BYTE_CAP);
assert_eq!(value["interval_secs"], INTERVAL.as_secs());
}
}
src/instr/mod.rs
+171
View File
@@ -0,0 +1,171 @@
//! Tick-body instrumentation.
//!
//! A purpose-built, feature-gated profiler for the rx-loop tick arm. It exists
//! to answer one question with field data: which subsystem step dominates the
//! tick body, and how long does the tick arm wait behind the other `select!`
//! arms before it runs at all.
//!
//! # Shape
//!
//! - Everything that costs anything at runtime is behind the `profiling` Cargo
//! feature, which is **off by default**. The default build's neutrality is a
//! property of the generated code, not of a runtime check.
//! - The instrumentation macro is defined twice, once per feature state. The
//! feature-off definition is a pure pass-through: it expands to the measured
//! expression and nothing else, so no timing code exists in a default build.
//! - The always-present surface — [`gate`], [`tick_entry`], [`tick_gauges`],
//! [`shutdown`] — exists in both feature states because the call sites in
//! `rx_loop.rs` and the lifecycle teardown must compile either way. Their
//! feature-off forms are empty (and [`gate`] is a `const fn` returning
//! `false`), so they cost nothing.
//! - The module is named `instr` rather than `profiling` so that it sorts
//! before `node` in `lib.rs`'s alphabetical module list: a `#[macro_use]`
//! module must be declared before the modules that use its macros.
//!
//! # Data model
//!
//! Domain above step: [`Domain`] carries exactly one variant today
//! (`Domain::Tick`). The primitive, the recorder, the writer and the `fipsctl`
//! surface all take a domain, so adding a data-path domain later is additive.
//! No second domain is declared until something records into it.
//!
//! Per (domain, step) the recorder keeps an exact count, max and total in fixed
//! static `AtomicU64` arrays — no histogram, no accumulation, a fixed footprint
//! regardless of run length. Gauges (ticks per interval, peer count, and the
//! arm-starvation figures) live in a parallel array and are emitted with an
//! explicit row kind so a gauge value never lands under a duration column.
#[cfg(feature = "profiling")]
pub(crate) mod capture;
#[cfg(feature = "profiling")]
mod recorder;
#[cfg(feature = "profiling")]
mod writer;
#[cfg(feature = "profiling")]
pub(crate) use recorder::{Domain, Step, now, record};
// ---------------------------------------------------------------------------
// The macro pair.
//
// Every path in the body is `$crate::`-qualified. `macro_rules!` bodies are not
// path-hygienic: an unqualified `Instant::now()` or `record(..)` would resolve
// at the *call site* (`rx_loop.rs`), where neither name is in scope. Importing
// them there is worse still, because the imports would be unused in the
// feature-off build and red it under `-D warnings`.
//
// `$e` is evaluated exactly once in both forms, which is what makes nesting the
// whole-tick span around the per-step spans safe.
// ---------------------------------------------------------------------------
/// Time `$e` as one step of `$domain`, when `$on` is true.
///
/// `$on` is the per-tick gate hoist: the enable flag is read once at the top of
/// the tick arm into a local, and that local is passed explicitly to every
/// invocation, because macro hygiene makes a call-site local invisible inside
/// the macro body.
#[cfg(feature = "profiling")]
macro_rules! instr_step {
($on:expr, $domain:expr, $step:expr, $e:expr) => {{
let t0 = if $on {
Some($crate::instr::now())
} else {
None
};
let r = $e;
if let Some(t) = t0 {
$crate::instr::record($domain, $step, t.elapsed());
}
r
}};
}
/// Feature-off form: a pure pass-through. The expansion contains no clock read,
/// no counter update and no reference to the recorder — only the measured
/// expression, plus a discard of the gate local so it is not unused.
#[cfg(not(feature = "profiling"))]
macro_rules! instr_step {
($on:expr, $domain:expr, $step:expr, $e:expr) => {{
let _ = &$on;
$e
}};
}
// ---------------------------------------------------------------------------
// Always-present surface.
// ---------------------------------------------------------------------------
/// Whether a capture is armed. Read **once per tick** into a local that is then
/// passed to each `instr_step!` invocation, so the feature-on-but-idle cost of
/// the whole tick arm is a single relaxed load.
#[cfg(feature = "profiling")]
#[inline]
pub(crate) fn gate() -> bool {
capture::gate()
}
/// Feature-off gate: a `const fn` returning `false`, so the whole tick arm
/// folds to the uninstrumented sequence at compile time.
#[cfg(not(feature = "profiling"))]
#[inline]
pub(crate) const fn gate() -> bool {
false
}
/// Record how late this tick-arm entry is against its scheduled deadline.
///
/// The arm is polled **last** under `biased;`, so its lateness is the time it
/// spent waiting behind the packet, TUN and control arms. `tokio::time::
/// interval::tick` returns the deadline it was scheduled for, so this is a
/// direct subtraction rather than a model. Two earlier designs derived it from
/// the inter-entry gap instead and both under-reported: one by the previous
/// body, the other by reporting only the first difference of the delay, so a
/// sustained stall read as zero. The inter-entry gap is still recorded as its
/// own gauge, but it carries no starvation signal on its own.
#[cfg(feature = "profiling")]
#[inline]
pub(crate) fn tick_entry(on: bool, deadline: std::time::Instant, now: std::time::Instant) {
recorder::tick_entry(on, deadline, now);
}
#[cfg(not(feature = "profiling"))]
#[inline]
pub(crate) fn tick_entry(_on: bool, _deadline: std::time::Instant, _now: std::time::Instant) {}
/// Sample the per-tick gauges taken from node state.
#[cfg(feature = "profiling")]
#[inline]
pub(crate) fn tick_gauges(on: bool, peers: u64) {
recorder::tick_gauges(on, peers);
}
#[cfg(not(feature = "profiling"))]
#[inline]
pub(crate) fn tick_gauges(_on: bool, _peers: u64) {}
/// Stop and reap any running capture at daemon teardown. Idempotent.
#[cfg(feature = "profiling")]
pub(crate) fn shutdown() {
capture::shutdown();
}
#[cfg(not(feature = "profiling"))]
pub(crate) fn shutdown() {}
/// One serialization lock for every test in this module tree.
///
/// The recorder counters and the capture state machine are the *same* process
/// statics: `capture::start` calls `recorder::reset`, and `capture::stop`
/// drains every slot. Two suites with their own locks therefore do not
/// serialize against each other, and the feature-on stage runs tests as
/// threads in one process, so a capture round-trip can zero the counters a
/// recorder test is mid-way through asserting on. One lock for both.
#[cfg(all(test, feature = "profiling"))]
pub(crate) static TEST_SERIAL: std::sync::Mutex<()> = std::sync::Mutex::new(());
/// Take the shared test lock, recovering from a poisoned mutex so one failing
/// test does not cascade into every other one.
#[cfg(all(test, feature = "profiling"))]
pub(crate) fn test_serial() -> std::sync::MutexGuard<'static, ()> {
TEST_SERIAL.lock().unwrap_or_else(|e| e.into_inner())
}
src/instr/recorder.rs
+478
View File
@@ -0,0 +1,478 @@
//! Fixed-footprint recorder: exact count / max / total per (domain, step).
//!
//! All state is process statics, not `Node` state, because the `fipsctl`
//! handler that arms and disarms a capture runs in the control accept task and
//! has no `&Node` — that is the whole point of serving it off-loop, so it
//! cannot queue behind the behavior it is measuring.
//!
//! The writer thread is the only reader. It takes each interval's figures with
//! `swap(0)`, so there are no "previous value" arrays to carry and the counters
//! are per-interval by construction.
use std::sync::LazyLock;
use std::sync::atomic::{AtomicU64, Ordering::Relaxed};
use std::time::{Duration, Instant};
/// Measurement domain. Structural only: one variant today.
///
/// A data-path domain is deliberately **not** declared until something records
/// into it. What generalizes here is the enum, the counter table and the
/// writer; the per-tick gate hoist does not, so a data-path domain will need
/// its own gate strategy.
#[derive(Copy, Clone, Debug, PartialEq, Eq)]
#[repr(usize)]
pub(crate) enum Domain {
Tick = 0,
}
pub(crate) const N_DOMAINS: usize = 1;
pub(crate) const DOMAINS: [Domain; N_DOMAINS] = [Domain::Tick];
impl Domain {
pub(crate) const fn name(self) -> &'static str {
match self {
Domain::Tick => "tick",
}
}
}
/// One measured step of the rx-loop tick arm, in call order, plus the
/// whole-body span.
///
/// `as usize` indexes the counter arrays, so the discriminants are dense and
/// `WholeTick` is last (it defines `N_STEPS`). Variants are declared
/// unconditionally — see [`Step::emitted`] for how the two platform- and
/// profile-conditional steps are kept out of the emitted table.
#[derive(Copy, Clone, Debug, PartialEq, Eq)]
#[repr(usize)]
pub(crate) enum Step {
CheckTimeouts = 0,
ReloadPeerAcl,
ReloadHostMap,
PollPendingConnects,
PollNostrRendezvous,
PollLanRendezvous,
DrivePeerTimers,
ResendPendingRekeys,
ResendPendingSessionHandshakes,
ResendPendingSessionMsg3,
PurgeIdleSessions,
ProcessPendingRetries,
CheckTreeState,
CheckBloomState,
ComputeMeshSize,
RecordStatsHistory,
CheckMmpReports,
CheckSessionMmpReports,
CheckLinkHeartbeats,
CheckRekey,
CheckSessionRekey,
CheckPendingLookups,
PollTransportDiscovery,
SampleTransportCongestion,
ActivateConnectedUdpSessions,
DebugAssertPeerMapsCoherent,
/// The whole tick-arm body, from before `check_timeouts` to after the last
/// step. Composes safely with the per-step spans because the macro
/// evaluates its measured expression exactly once.
WholeTick,
}
pub(crate) const N_STEPS: usize = Step::WholeTick as usize + 1;
/// Every step, in emission order. Index `i` of this table is `STEPS[i] as
/// usize`; `steps_table_is_dense` asserts it.
pub(crate) const STEPS: [Step; N_STEPS] = [
Step::CheckTimeouts,
Step::ReloadPeerAcl,
Step::ReloadHostMap,
Step::PollPendingConnects,
Step::PollNostrRendezvous,
Step::PollLanRendezvous,
Step::DrivePeerTimers,
Step::ResendPendingRekeys,
Step::ResendPendingSessionHandshakes,
Step::ResendPendingSessionMsg3,
Step::PurgeIdleSessions,
Step::ProcessPendingRetries,
Step::CheckTreeState,
Step::CheckBloomState,
Step::ComputeMeshSize,
Step::RecordStatsHistory,
Step::CheckMmpReports,
Step::CheckSessionMmpReports,
Step::CheckLinkHeartbeats,
Step::CheckRekey,
Step::CheckSessionRekey,
Step::CheckPendingLookups,
Step::PollTransportDiscovery,
Step::SampleTransportCongestion,
Step::ActivateConnectedUdpSessions,
Step::DebugAssertPeerMapsCoherent,
Step::WholeTick,
];
impl Step {
pub(crate) const fn name(self) -> &'static str {
match self {
Step::CheckTimeouts => "check_timeouts",
Step::ReloadPeerAcl => "reload_peer_acl",
Step::ReloadHostMap => "reload_host_map",
Step::PollPendingConnects => "poll_pending_connects",
Step::PollNostrRendezvous => "poll_nostr_rendezvous",
Step::PollLanRendezvous => "poll_lan_rendezvous",
Step::DrivePeerTimers => "drive_peer_timers",
Step::ResendPendingRekeys => "resend_pending_rekeys",
Step::ResendPendingSessionHandshakes => "resend_pending_session_handshakes",
Step::ResendPendingSessionMsg3 => "resend_pending_session_msg3",
Step::PurgeIdleSessions => "purge_idle_sessions",
Step::ProcessPendingRetries => "process_pending_retries",
Step::CheckTreeState => "check_tree_state",
Step::CheckBloomState => "check_bloom_state",
Step::ComputeMeshSize => "compute_mesh_size",
Step::RecordStatsHistory => "record_stats_history",
Step::CheckMmpReports => "check_mmp_reports",
Step::CheckSessionMmpReports => "check_session_mmp_reports",
Step::CheckLinkHeartbeats => "check_link_heartbeats",
Step::CheckRekey => "check_rekey",
Step::CheckSessionRekey => "check_session_rekey",
Step::CheckPendingLookups => "check_pending_lookups",
Step::PollTransportDiscovery => "poll_transport_discovery",
Step::SampleTransportCongestion => "sample_transport_congestion",
Step::ActivateConnectedUdpSessions => "activate_connected_udp_sessions",
Step::DebugAssertPeerMapsCoherent => "debug_assert_peer_maps_coherent",
Step::WholeTick => "whole_tick",
}
}
/// Whether this step gets a row in this build.
///
/// Two steps are conditionally compiled at their call sites. Emitting a row
/// for them in a build where the call site does not exist would publish a
/// count that is structurally zero forever, which reads as "this step never
/// runs" rather than "this step is not in this build". The predicates below
/// are the same `cfg` expressions that gate the call sites in
/// `node::dataplane::rx_loop`; keep them in step.
pub(crate) const fn emitted(self) -> bool {
match self {
Step::ActivateConnectedUdpSessions => {
cfg!(any(target_os = "linux", target_os = "macos"))
}
Step::DebugAssertPeerMapsCoherent => cfg!(debug_assertions),
_ => true,
}
}
}
/// A scalar sampled once per tick, as opposed to a duration.
///
/// Gauges carry their own row kind and their own unit in the output so a gauge
/// value can never be read as a duration.
#[derive(Copy, Clone, Debug, PartialEq, Eq)]
#[repr(usize)]
pub(crate) enum Gauge {
Ticks = 0,
Peers,
TickGap,
ArmStarvation,
}
pub(crate) const N_GAUGES: usize = Gauge::ArmStarvation as usize + 1;
pub(crate) const GAUGES: [Gauge; N_GAUGES] = [
Gauge::Ticks,
Gauge::Peers,
Gauge::TickGap,
Gauge::ArmStarvation,
];
impl Gauge {
pub(crate) const fn name(self) -> &'static str {
match self {
Gauge::Ticks => "ticks",
Gauge::Peers => "peers",
Gauge::TickGap => "tick_entry_gap",
Gauge::ArmStarvation => "arm_starvation",
}
}
/// Unit of the `max` and `total` columns for this gauge.
pub(crate) const fn unit(self) -> &'static str {
match self {
Gauge::Ticks => "ticks",
Gauge::Peers => "peers",
Gauge::TickGap | Gauge::ArmStarvation => "us",
}
}
/// Whether the gauge's stored values are nanosecond durations that the
/// writer converts to microseconds.
pub(crate) const fn is_duration(self) -> bool {
matches!(self, Gauge::TickGap | Gauge::ArmStarvation)
}
}
const N_SLOTS: usize = N_DOMAINS * N_STEPS;
static COUNT: [AtomicU64; N_SLOTS] = [const { AtomicU64::new(0) }; N_SLOTS];
static MAX_NS: [AtomicU64; N_SLOTS] = [const { AtomicU64::new(0) }; N_SLOTS];
static TOTAL_NS: [AtomicU64; N_SLOTS] = [const { AtomicU64::new(0) }; N_SLOTS];
static G_COUNT: [AtomicU64; N_GAUGES] = [const { AtomicU64::new(0) }; N_GAUGES];
static G_MAX: [AtomicU64; N_GAUGES] = [const { AtomicU64::new(0) }; N_GAUGES];
static G_TOTAL: [AtomicU64; N_GAUGES] = [const { AtomicU64::new(0) }; N_GAUGES];
/// Monotonic baseline so tick-arm entry times fit in an atomic. Offset by one
/// on store so that zero can mean "no previous entry".
static BASE: LazyLock<Instant> = LazyLock::new(Instant::now);
static PREV_ENTRY_NS: AtomicU64 = AtomicU64::new(0);
#[inline]
const fn slot(domain: Domain, step: Step) -> usize {
(domain as usize * N_STEPS) + step as usize
}
/// Read the clock for a step span.
#[inline]
pub(crate) fn now() -> Instant {
Instant::now()
}
/// Record one observation of `step`.
#[inline]
pub(crate) fn record(domain: Domain, step: Step, elapsed: Duration) {
let ns = elapsed.as_nanos() as u64;
let idx = slot(domain, step);
COUNT[idx].fetch_add(1, Relaxed);
TOTAL_NS[idx].fetch_add(ns, Relaxed);
MAX_NS[idx].fetch_max(ns, Relaxed);
}
#[inline]
fn record_gauge(gauge: Gauge, value: u64) {
let idx = gauge as usize;
G_COUNT[idx].fetch_add(1, Relaxed);
G_TOTAL[idx].fetch_add(value, Relaxed);
G_MAX[idx].fetch_max(value, Relaxed);
}
/// Sample the inter-entry gap and the measured arm-starvation delay.
pub(crate) fn tick_entry(on: bool, deadline: Instant, now: Instant) {
if !on {
return;
}
// Offset by one so that a stored zero unambiguously means "no previous
// entry", even for an entry that lands on the baseline instant.
let stamp = BASE.elapsed().as_nanos() as u64 + 1;
let late = now.saturating_duration_since(deadline).as_nanos() as u64;
tick_entry_at(stamp, late);
}
/// The clock-free half of [`tick_entry`]: both times are inputs, so the
/// arithmetic can be driven with synthetic stamps in a test.
///
/// `late_ns` is how far past its scheduled deadline this entry was, measured
/// directly rather than derived. Two earlier designs derived it from the
/// inter-entry gap and were both wrong: subtracting the previous body
/// understated it by exactly the body, and subtracting `max(period, body)`
/// reported the *first difference* of the delay, so a sustained stall — the
/// overload regime this measurement exists to characterize — read as zero
/// forever. `tokio::time::interval::tick` hands back the deadline it was
/// scheduled for, so the delay is a subtraction with no model behind it.
fn tick_entry_at(stamp: u64, late_ns: u64) {
let prev = PREV_ENTRY_NS.swap(stamp, Relaxed);
record_gauge(Gauge::Ticks, 1);
record_gauge(Gauge::ArmStarvation, late_ns);
if prev == 0 {
// First entry of this capture: there is no previous entry to measure a
// gap against, and the idle interval before arming is not a gap. The
// lateness above does not depend on a previous entry, so it still counts.
return;
}
record_gauge(Gauge::TickGap, stamp.saturating_sub(prev));
}
/// Sample the gauges that come from node state.
pub(crate) fn tick_gauges(on: bool, peers: u64) {
if !on {
return;
}
record_gauge(Gauge::Peers, peers);
}
/// Take (and clear) this interval's figures for one step: count, max ns, total
/// ns. Called only by the writer thread.
pub(crate) fn take_step(domain: Domain, step: Step) -> (u64, u64, u64) {
let idx = slot(domain, step);
(
COUNT[idx].swap(0, Relaxed),
MAX_NS[idx].swap(0, Relaxed),
TOTAL_NS[idx].swap(0, Relaxed),
)
}
/// Take (and clear) this interval's figures for one gauge.
pub(crate) fn take_gauge(gauge: Gauge) -> (u64, u64, u64) {
let idx = gauge as usize;
(
G_COUNT[idx].swap(0, Relaxed),
G_MAX[idx].swap(0, Relaxed),
G_TOTAL[idx].swap(0, Relaxed),
)
}
/// Zero every counter so a capture starts from a clean slate.
pub(crate) fn reset() {
for i in 0..N_SLOTS {
COUNT[i].store(0, Relaxed);
MAX_NS[i].store(0, Relaxed);
TOTAL_NS[i].store(0, Relaxed);
}
for i in 0..N_GAUGES {
G_COUNT[i].store(0, Relaxed);
G_MAX[i].store(0, Relaxed);
G_TOTAL[i].store(0, Relaxed);
}
PREV_ENTRY_NS.store(0, Relaxed);
}
#[cfg(test)]
mod tests {
use super::*;
use std::sync::MutexGuard;
/// Shared with the capture tests: they mutate the same statics. See
/// `crate::instr::test_serial`.
fn serial() -> MutexGuard<'static, ()> {
crate::instr::test_serial()
}
#[test]
fn steps_table_is_dense() {
for (i, step) in STEPS.iter().enumerate() {
assert_eq!(*step as usize, i, "step {} is out of order", step.name());
}
assert_eq!(STEPS.len(), N_STEPS);
}
#[test]
fn gauges_table_is_dense() {
for (i, gauge) in GAUGES.iter().enumerate() {
assert_eq!(*gauge as usize, i, "gauge {} is out of order", gauge.name());
}
assert_eq!(GAUGES.len(), N_GAUGES);
}
#[test]
fn step_names_are_unique() {
let mut names: Vec<&str> = STEPS.iter().map(|s| s.name()).collect();
names.sort_unstable();
let before = names.len();
names.dedup();
assert_eq!(before, names.len(), "duplicate step name");
}
#[test]
fn emitted_row_count_matches_build() {
let emitted = STEPS.iter().filter(|s| s.emitted()).count();
// 24 unconditional subsystem steps + the whole-tick span, plus the two
// conditionally-compiled steps where this build has them.
let mut expected = 25;
if cfg!(any(target_os = "linux", target_os = "macos")) {
expected += 1;
}
if cfg!(debug_assertions) {
expected += 1;
}
assert_eq!(emitted, expected);
}
#[test]
fn record_accumulates_count_max_and_total() {
let _guard = serial();
reset();
record(Domain::Tick, Step::CheckRekey, Duration::from_nanos(10));
record(Domain::Tick, Step::CheckRekey, Duration::from_nanos(30));
let (count, max, total) = take_step(Domain::Tick, Step::CheckRekey);
assert_eq!((count, max, total), (2, 30, 40));
// Taking clears the slot.
assert_eq!(take_step(Domain::Tick, Step::CheckRekey), (0, 0, 0));
}
#[test]
fn starvation_is_the_measured_lateness_of_the_entry() {
let _guard = serial();
reset();
// The interval hands back the deadline it was scheduled for, so the
// delay is `now - deadline` and nothing is derived from the period, the
// previous entry, or the previous body.
PREV_ENTRY_NS.store(1_000_000_000, Relaxed);
tick_entry_at(1_100_000_000, 50_000_000);
assert_eq!(take_gauge(Gauge::TickGap), (1, 100_000_000, 100_000_000));
assert_eq!(
take_gauge(Gauge::ArmStarvation),
(1, 50_000_000, 50_000_000)
);
assert_eq!(take_gauge(Gauge::Ticks).0, 1);
reset();
}
#[test]
fn sustained_lateness_is_reported_on_every_tick() {
let _guard = serial();
reset();
// The regime the two earlier designs both hid. Three consecutive entries
// each 50 ms past their deadline, one period apart, i.e. the arm waiting
// a constant amount behind the other select arms every round. The gaps
// are all exactly one period, so any formula derived from the
// inter-entry gap reports zero here; measured lateness reports 50 ms
// three times, which is the truth.
PREV_ENTRY_NS.store(1_000_000_000, Relaxed);
tick_entry_at(1_050_000_000, 50_000_000);
tick_entry_at(1_100_000_000, 50_000_000);
tick_entry_at(1_150_000_000, 50_000_000);
let (count, max, total) = take_gauge(Gauge::ArmStarvation);
assert_eq!(count, 3);
assert_eq!(max, 50_000_000);
assert_eq!(total, 150_000_000);
// ...and the gap alone carries no signal about it: every gap is one
// period, exactly as it would be on a perfectly healthy node.
assert_eq!(take_gauge(Gauge::TickGap), (3, 50_000_000, 150_000_000));
reset();
}
#[test]
fn first_entry_of_a_capture_records_no_gap_but_still_records_lateness() {
let _guard = serial();
reset();
tick_entry_at(500, 7_000_000);
assert_eq!(take_gauge(Gauge::Ticks).0, 1);
assert_eq!(take_gauge(Gauge::TickGap), (0, 0, 0));
// Lateness does not depend on a previous entry, so the first tick of a
// capture still contributes one.
assert_eq!(take_gauge(Gauge::ArmStarvation), (1, 7_000_000, 7_000_000));
reset();
}
#[test]
fn an_on_schedule_entry_reports_no_starvation() {
let _guard = serial();
reset();
PREV_ENTRY_NS.store(1_000_000_000, Relaxed);
tick_entry_at(1_050_000_000, 0);
assert_eq!(take_gauge(Gauge::ArmStarvation), (1, 0, 0));
assert_eq!(take_gauge(Gauge::TickGap), (1, 50_000_000, 50_000_000));
reset();
}
#[test]
fn gate_off_records_nothing() {
let _guard = serial();
reset();
let t = Instant::now();
tick_entry(false, t, t);
tick_gauges(false, 42);
assert_eq!(take_gauge(Gauge::Ticks), (0, 0, 0));
assert_eq!(take_gauge(Gauge::Peers), (0, 0, 0));
}
}
src/instr/writer.rs
+218
View File
@@ -0,0 +1,218 @@
//! The capture writer: a dedicated, named OS thread with an explicit
//! lifecycle.
//!
//! There is no worker-thread lifecycle in this codebase to copy — the crypto
//! worker pools are never torn down and drop their join handles at spawn — so
//! this one is designed here.
//!
//! Two properties matter:
//!
//! - **It is an OS thread, not a spawned task.** The runtime is
//! `current_thread`, so file I/O on a task would run on the rx loop's own
//! thread and stall every `select!` arm, including the one being measured.
//! - **It waits on a channel with a timeout, not on a sleep.** `recv_timeout`
//! returns immediately when the toggle sends stop, so `off` performs a final
//! drain and joins promptly instead of parking the caller for up to a full
//! flush interval.
//!
//! The thread is created lazily when a capture starts, so a node that never
//! arms one never has the thread.
use std::fs::File;
use std::io::Write;
use std::sync::mpsc::{self, Receiver, RecvTimeoutError, Sender};
use std::thread::{self, JoinHandle};
use std::time::{SystemTime, UNIX_EPOCH};
use super::capture::{self, BYTE_CAP, INTERVAL};
use super::recorder::{self, DOMAINS, GAUGES, STEPS};
/// Owner-side handle to the writer thread.
pub(crate) struct Handle {
stop_tx: Sender<()>,
join: JoinHandle<()>,
}
impl Handle {
/// Wake the writer, let it drain once more, and join it.
pub(crate) fn stop_and_join(self) {
// A send error means the thread already exited (cap stop); joining is
// still correct and returns at once.
let _ = self.stop_tx.send(());
let _ = self.join.join();
}
}
/// Start the writer thread on an already-open sink.
pub(crate) fn spawn(file: File) -> std::io::Result<Handle> {
let (stop_tx, stop_rx) = mpsc::channel();
let join = thread::Builder::new()
.name("fips-profile".to_string())
.spawn(move || run(file, stop_rx))?;
Ok(Handle { stop_tx, join })
}
/// What one flush cycle decided. Separated from [`run`] so the terminal paths
/// can be driven in a test with a failing sink: the loop below owns the waiting
/// and the state transition, this owns the decision.
#[derive(Debug, PartialEq, Eq)]
enum Cycle {
Continue,
CapReached,
WriteFailed,
}
/// Drain one interval into the sink and decide whether the capture goes on.
fn flush_cycle<W: Write>(file: &mut W) -> Cycle {
if flush(file).is_err() {
// The sink is gone or full; stop rather than spinning on a broken file
// for the rest of the run.
let _ = note(file, "capture stopped: write error");
return Cycle::WriteFailed;
}
if capture::bytes_written() >= BYTE_CAP {
let _ = note(
file,
&format!("capture stopped: byte cap {BYTE_CAP} reached"),
);
let _ = file.flush();
return Cycle::CapReached;
}
Cycle::Continue
}
fn run<W: Write>(mut file: W, stop_rx: Receiver<()>) {
loop {
match stop_rx.recv_timeout(INTERVAL) {
// Stop requested, or the owner went away: final drain, then exit.
Ok(()) | Err(RecvTimeoutError::Disconnected) => {
let _ = flush(&mut file);
let _ = file.flush();
return;
}
Err(RecvTimeoutError::Timeout) => match flush_cycle(&mut file) {
Cycle::Continue => {}
Cycle::WriteFailed => {
// The trailer went to the same failing file, so it is not a
// signal that survives. `stop` and a subsequent `on` both
// clear the state without surfacing it, so an operator would
// otherwise never learn the window was truncated.
tracing::warn!(
target: "fips::instr",
"profile capture stopped: write error on the sink"
);
capture::mark_stopped(capture::STOPPED_BY_ERROR);
return;
}
Cycle::CapReached => {
capture::mark_stopped(capture::STOPPED_BY_CAP);
return;
}
},
}
}
}
/// Append a `#`-prefixed trailer line.
fn note<W: Write>(file: &mut W, text: &str) -> std::io::Result<()> {
let line = format!("# {text}\n");
file.write_all(line.as_bytes())?;
capture::add_bytes(line.len() as u64);
Ok(())
}
/// Emit one interval: every step of every domain, then the gauges.
///
/// Every emitted step gets a row every interval, including zero-count rows, so
/// "this step did not run" is visible rather than absent. The two steps whose
/// call sites are conditionally compiled are excluded in builds that do not
/// have them, so no row is structurally zero forever.
fn flush<W: Write>(file: &mut W) -> std::io::Result<()> {
let ts = SystemTime::now()
.duration_since(UNIX_EPOCH)
.map(|d| d.as_secs())
.unwrap_or(0);
let mut out = String::with_capacity(4096);
for domain in DOMAINS {
for step in STEPS {
if !step.emitted() {
continue;
}
let (count, max_ns, total_ns) = recorder::take_step(domain, step);
out.push_str(&format!(
"{ts}\tstep\t{domain}\t{name}\t{count}\t{max}\t{total}\tus\n",
domain = domain.name(),
name = step.name(),
max = max_ns / 1_000,
total = total_ns / 1_000,
));
}
}
// Gauges carry the tick domain today; the row kind and the unit column keep
// them distinguishable from the duration rows above.
for gauge in GAUGES {
let (count, mut max, mut total) = recorder::take_gauge(gauge);
if gauge.is_duration() {
max /= 1_000;
total /= 1_000;
}
out.push_str(&format!(
"{ts}\tgauge\t{domain}\t{name}\t{count}\t{max}\t{total}\t{unit}\n",
domain = recorder::Domain::Tick.name(),
name = gauge.name(),
unit = gauge.unit(),
));
}
file.write_all(out.as_bytes())?;
capture::add_bytes(out.len() as u64);
Ok(())
}
#[cfg(test)]
mod tests {
use super::*;
/// A sink that fails every write, so the writer's error path is driven by a
/// real `Err` rather than asserted about.
struct AlwaysFails;
impl Write for AlwaysFails {
fn write(&mut self, _buf: &[u8]) -> std::io::Result<usize> {
Err(std::io::Error::new(
std::io::ErrorKind::StorageFull,
"no space left on device",
))
}
fn flush(&mut self) -> std::io::Result<()> {
Ok(())
}
}
/// A failing sink ends the capture as a write error, which `run` turns into
/// `STOPPED_BY_ERROR` — not into a byte-cap stop. The distinction is what
/// tells an operator that a window is truncated rather than complete.
///
/// **Coverage note.** This drives the decision, not the filesystem
/// condition. The mesh rehearsal cannot produce one: removing the file
/// leaves the writer's descriptor valid and writes keep succeeding into the
/// unlinked inode, and mounting a tiny filesystem inside the test container
/// is refused. So "a real ENOSPC reaches this branch" stays unexercised;
/// what is covered is that an `Err` from the sink produces the error
/// outcome and not the cap outcome.
#[test]
fn a_failing_sink_ends_the_cycle_as_an_error_not_a_cap() {
let _guard = crate::instr::test_serial();
assert_eq!(flush_cycle(&mut AlwaysFails), Cycle::WriteFailed);
}
/// The healthy path must not be reported as either terminal state, or the
/// test above would pass for a writer that always stops.
#[test]
fn a_working_sink_continues() {
let _guard = crate::instr::test_serial();
let mut sink: Vec<u8> = Vec::new();
assert_eq!(flush_cycle(&mut sink), Cycle::Continue);
}
}
src/lib.rs
+53 -20
View File
@@ -3,22 +3,34 @@
//! A distributed, decentralized network routing protocol for mesh nodes
//! connecting over arbitrary transports.
pub mod bloom;
// Name the `alloc` crate directly so the sans-IO protocol cores can spell their
// heap-type imports in `no_std`-forward form (`alloc::sync::Arc`,
// `alloc::collections::BTreeMap`). The crate remains `std`; this only reduces the
// distance to extracting the pure cores into a `no_std` crate later.
extern crate alloc;
pub mod cache;
pub mod config;
pub mod control;
pub mod discovery;
#[cfg(target_os = "linux")]
pub mod gateway;
pub mod identity;
pub mod mmp;
// Declared before `node` (and named to sort there) because it carries
// `#[macro_use]`: the tick instrumentation macro must be in scope for the
// modules that follow.
#[macro_use]
pub(crate) mod instr;
pub mod mdns;
pub mod node;
pub mod noise;
pub mod nostr;
pub mod peer;
pub mod perf_profile;
pub mod protocol;
pub(crate) mod proto;
#[cfg(test)]
pub(crate) mod testutil;
mod time;
pub mod transport;
pub mod tree;
pub mod upper;
pub mod utils;
pub mod version;
@@ -33,14 +45,16 @@ pub use identity::{
pub use config::{Config, ConfigError, IdentityConfig, NymConfig, TorConfig, UdpConfig};
pub use upper::config::{DnsConfig, TunConfig};
// Re-export discovery types
pub use discovery::{BootstrapHandoffResult, EstablishedTraversal};
// Re-export nostr rendezvous handoff types
pub use nostr::{BootstrapHandoffResult, EstablishedTraversal, is_punch_packet};
// Re-export tree types
pub use tree::{CoordEntry, ParentDeclaration, TreeCoordinate, TreeError, TreeState};
// Re-export tree types (relocated from tree:: to proto::stp)
pub use proto::stp::{
CoordEntry, CoordError, ParentDeclaration, TreeCoordinate, TreeError, TreeState,
};
// Re-export bloom filter types
pub use bloom::{BloomError, BloomFilter, BloomState};
// Re-export bloom filter types (relocated from bloom:: to proto::bloom)
pub use proto::bloom::{BloomError, BloomFilter, BloomState};
// Re-export transport types
pub use transport::udp::UdpTransport;
@@ -50,21 +64,40 @@ pub use transport::{
TransportState, TransportType, packet_channel,
};
// Re-export protocol types
pub use protocol::{
CoordsRequired, FilterAnnounce, HandshakeMessageType, LinkMessageType, LookupRequest,
LookupResponse, PathBroken, ProtocolError, SessionAck, SessionDatagram, SessionFlags,
SessionMessageType, SessionSetup, TreeAnnounce,
// Re-export link-layer types (relocated from protocol:: to proto::link)
pub use proto::link::{LinkMessageType, SessionDatagram};
// Re-export the shared protocol error (relocated from protocol:: to proto::Error)
pub use proto::Error;
// Re-export FSP session wire types (relocated from protocol:: to proto::fsp)
pub use proto::fsp::{SessionAck, SessionFlags, SessionMessageType, SessionSetup};
// Re-export STP wire types (relocated from protocol:: to proto::stp)
pub use proto::stp::TreeAnnounce;
// Re-export bloom wire types (relocated from protocol:: to proto::bloom)
pub use proto::bloom::FilterAnnounce;
// Re-export discovery wire types (relocated from protocol:: to proto::lookup)
pub use proto::lookup::{LookupRequest, LookupResponse};
// Re-export routing wire types (relocated from protocol:: to proto::routing)
pub use proto::routing::{
COORDS_REQUIRED_SIZE, CoordsRequired, MTU_EXCEEDED_SIZE, MtuExceeded, PathBroken,
};
// Re-export FMP link-framing wire type (relocated from protocol:: to proto::fmp)
pub use proto::fmp::HandshakeMessageType;
// Re-export cache types
pub use cache::{CacheEntry, CacheError, CacheStats, CoordCache};
// Re-export FMP tie-break helper and promotion result (relocated from peer:: to proto::fmp)
pub use proto::fmp::{PromotionResult, cross_connection_winner};
// Re-export peer types
pub use peer::{
ActivePeer, ConnectivityState, HandshakeState, PeerConnection, PeerError, PeerSlot,
PromotionResult, cross_connection_winner,
};
pub use peer::{ActivePeer, ConnectivityState, PeerError};
// Re-export node types
pub use node::{Node, NodeError, NodeState, UpdatePeersOutcome};
src/discovery/lan/mod.rs → src/mdns/mod.rs
+28 -19
View File
@@ -54,8 +54,12 @@ pub const TXT_KEY_SCOPE: &str = "scope";
/// `PROTOCOL_VERSION`).
pub const TXT_KEY_VERSION: &str = "v";
/// FIPS protocol version advertised in the mDNS TXT `v` key. Kept in sync
/// with the Nostr rendezvous `PROTOCOL_VERSION` (same value, `"1"`).
const TXT_PROTOCOL_VERSION: &str = "1";
#[derive(Debug, Error)]
pub enum LanDiscoveryError {
pub enum LanRendezvousError {
#[error("mDNS daemon init failed: {0}")]
Daemon(String),
#[error("mDNS register failed: {0}")]
@@ -79,7 +83,7 @@ pub struct LanDiscoveredPeer {
pub observed_at: Instant,
}
/// Browser-side events surfaced by `LanDiscovery::drain_events`.
/// Browser-side events surfaced by `LanRendezvous::drain_events`.
#[derive(Debug, Clone)]
pub enum LanEvent {
Discovered(LanDiscoveredPeer),
@@ -87,17 +91,17 @@ pub enum LanEvent {
/// Runtime configuration for the mDNS responder + browser.
#[derive(Debug, Clone, serde::Serialize, serde::Deserialize)]
pub struct LanDiscoveryConfig {
pub struct LanRendezvousConfig {
/// Master switch. Default: `false` — LAN discovery is opt-in. Operators
/// who want sub-second same-LAN pairing enable it via
/// `node.discovery.lan.enabled: true`. Default-off avoids reintroducing
/// `node.rendezvous.lan.enabled: true`. Default-off avoids reintroducing
/// a per-LAN identity broadcast on nodes that have deliberately disabled
/// other discovery channels, and avoids any multicast surprise on upgrade.
#[serde(default = "LanDiscoveryConfig::default_enabled")]
#[serde(default = "LanRendezvousConfig::default_enabled")]
pub enabled: bool,
/// Overridable service type, primarily so integration tests can run
/// multiple isolated services on the same loopback interface.
#[serde(default = "LanDiscoveryConfig::default_service_type")]
#[serde(default = "LanRendezvousConfig::default_service_type")]
pub service_type: String,
/// Optional application/network scope carried in the LAN-only TXT
/// record. Browsers that set a scope ignore adverts for other scopes.
@@ -109,7 +113,7 @@ pub struct LanDiscoveryConfig {
pub scope: Option<String>,
}
impl Default for LanDiscoveryConfig {
impl Default for LanRendezvousConfig {
fn default() -> Self {
Self {
enabled: Self::default_enabled(),
@@ -119,7 +123,7 @@ impl Default for LanDiscoveryConfig {
}
}
impl LanDiscoveryConfig {
impl LanRendezvousConfig {
fn default_enabled() -> bool {
false
}
@@ -129,7 +133,7 @@ impl LanDiscoveryConfig {
}
/// Running mDNS responder + browser bound to the node's UDP advert port.
pub struct LanDiscovery {
pub struct LanRendezvous {
daemon: ServiceDaemon,
own_npub: String,
instance_fullname: String,
@@ -137,7 +141,12 @@ pub struct LanDiscovery {
event_pump: tokio::task::JoinHandle<()>,
}
impl LanDiscovery {
impl LanRendezvous {
/// Whether the mDNS event-pump task has exited (runtime liveness).
pub fn is_finished(&self) -> bool {
self.event_pump.is_finished()
}
/// Start the mDNS responder and browser.
///
/// `advertised_port` is the UDP port the operational UDP transport
@@ -148,16 +157,16 @@ impl LanDiscovery {
identity: &Identity,
scope: Option<String>,
advertised_port: u16,
config: LanDiscoveryConfig,
) -> Result<Arc<Self>, LanDiscoveryError> {
config: LanRendezvousConfig,
) -> Result<Arc<Self>, LanRendezvousError> {
if !config.enabled {
return Err(LanDiscoveryError::Disabled);
return Err(LanRendezvousError::Disabled);
}
if advertised_port == 0 {
return Err(LanDiscoveryError::NoAdvertisedPort);
return Err(LanRendezvousError::NoAdvertisedPort);
}
let daemon = ServiceDaemon::new().map_err(|e| LanDiscoveryError::Daemon(e.to_string()))?;
let daemon = ServiceDaemon::new().map_err(|e| LanRendezvousError::Daemon(e.to_string()))?;
let npub = identity.npub();
// mDNS DNS labels are capped at 63 bytes. 16 bech32 chars of npub
@@ -176,7 +185,7 @@ impl LanDiscovery {
}
props.insert(
TXT_KEY_VERSION.to_string(),
super::nostr::PROTOCOL_VERSION.to_string(),
TXT_PROTOCOL_VERSION.to_string(),
);
// host_ipv4 is set to "127.0.0.1" *and* enable_addr_auto() is
@@ -193,18 +202,18 @@ impl LanDiscovery {
advertised_port,
Some(props),
)
.map_err(|e| LanDiscoveryError::Register(e.to_string()))?
.map_err(|e| LanRendezvousError::Register(e.to_string()))?
.enable_addr_auto();
let instance_fullname = service_info.get_fullname().to_string();
daemon
.register(service_info)
.map_err(|e| LanDiscoveryError::Register(e.to_string()))?;
.map_err(|e| LanRendezvousError::Register(e.to_string()))?;
let browse_rx = daemon
.browse(&config.service_type)
.map_err(|e| LanDiscoveryError::Browse(e.to_string()))?;
.map_err(|e| LanRendezvousError::Browse(e.to_string()))?;
let (events_tx, events_rx) = tokio::sync::mpsc::unbounded_channel();
let own_npub = npub.clone();
src/discovery/lan/tests.rs → src/mdns/tests.rs
+12 -12
View File
@@ -4,7 +4,7 @@ use std::time::Duration;
use crate::Identity;
use mdns_sd::ScopedIp;
use super::{LanDiscovery, LanDiscoveryConfig, LanEvent};
use super::{LanEvent, LanRendezvous, LanRendezvousConfig};
/// Distinct service type per test run so concurrent cargo-test workers
/// on the same machine don't cross-feed each other's adverts via the
@@ -15,8 +15,8 @@ fn isolated_service_type(tag: &str) -> String {
format!("_fipstest-{tag}-{rand:08x}._udp.local.")
}
fn config_for(service_type: String) -> LanDiscoveryConfig {
LanDiscoveryConfig {
fn config_for(service_type: String) -> LanRendezvousConfig {
LanRendezvousConfig {
enabled: true,
service_type,
scope: None,
@@ -47,7 +47,7 @@ fn non_link_local_ipv6_advert_is_preserved() {
}
async fn wait_for_peer(
discovery: &LanDiscovery,
discovery: &LanRendezvous,
expected_npub: &str,
timeout: Duration,
) -> Option<super::LanDiscoveredPeer> {
@@ -64,7 +64,7 @@ async fn wait_for_peer(
None
}
/// Two LanDiscovery instances on isolated service types — `a` browses
/// Two LanRendezvous instances on isolated service types — `a` browses
/// only its own type and never sees `b`, and vice versa. Sanity check
/// that the scope-isolation defense works (we'd lose isolation if mdns-
/// sd ever leaked across service types).
@@ -76,7 +76,7 @@ async fn isolated_service_types_do_not_cross_feed() {
let service_a = isolated_service_type("isolated-a");
let service_b = isolated_service_type("isolated-b");
let lan_a = LanDiscovery::start(
let lan_a = LanRendezvous::start(
&identity_a,
Some("scope-x".to_string()),
61001,
@@ -84,7 +84,7 @@ async fn isolated_service_types_do_not_cross_feed() {
)
.await
.expect("start a");
let lan_b = LanDiscovery::start(
let lan_b = LanRendezvous::start(
&identity_b,
Some("scope-x".to_string()),
61002,
@@ -118,7 +118,7 @@ async fn isolated_service_types_do_not_cross_feed() {
assert!(!saw_a_from_b, "isolated service types must not cross-feed");
}
/// Two LanDiscovery instances on the same service type and the same
/// Two LanRendezvous instances on the same service type and the same
/// scope: each should observe the other's advert within a few seconds.
/// Exercises the responder + browser + TXT plumbing end-to-end.
///
@@ -136,7 +136,7 @@ async fn matched_scope_peers_observe_each_other() {
let service = isolated_service_type("matched");
let lan_a = LanDiscovery::start(
let lan_a = LanRendezvous::start(
&identity_a,
Some("scope-shared".to_string()),
61101,
@@ -144,7 +144,7 @@ async fn matched_scope_peers_observe_each_other() {
)
.await
.expect("start a");
let lan_b = LanDiscovery::start(
let lan_b = LanRendezvous::start(
&identity_b,
Some("scope-shared".to_string()),
61102,
@@ -181,7 +181,7 @@ async fn cross_scope_advert_is_filtered() {
let service = isolated_service_type("cross-scope");
let lan_a = LanDiscovery::start(
let lan_a = LanRendezvous::start(
&identity_a,
Some("scope-a".to_string()),
61201,
@@ -189,7 +189,7 @@ async fn cross_scope_advert_is_filtered() {
)
.await
.expect("start a");
let lan_b = LanDiscovery::start(
let lan_b = LanRendezvous::start(
&identity_b,
Some("scope-b".to_string()),
61202,
src/mmp/metrics.rs
-556
View File
@@ -1,556 +0,0 @@
//! MMP derived metrics.
//!
//! `MmpMetrics` processes incoming ReceiverReports (from our peer) and
//! maintains derived metrics: SRTT, loss rate, goodput, ETX, and dual
//! EWMA trend indicators. Updated by the sender side when it receives
//! a ReceiverReport about its own traffic.
use crate::mmp::algorithms::{DualEwma, SrttEstimator, compute_etx};
use crate::mmp::report::ReceiverReport;
use std::time::Instant;
use tracing::trace;
/// Derived MMP metrics, updated from incoming ReceiverReports.
///
/// This lives on the sender side: when we receive a ReceiverReport from
/// our peer describing what they observed about our traffic, we process
/// it here to compute RTT, loss, goodput, and trend indicators.
pub struct MmpMetrics {
/// Smoothed RTT from timestamp echo.
pub srtt: SrttEstimator,
/// Dual EWMA trend detectors.
pub rtt_trend: DualEwma,
pub loss_trend: DualEwma,
pub goodput_trend: DualEwma,
pub jitter_trend: DualEwma,
pub etx_trend: DualEwma,
/// Forward delivery ratio (what fraction of our frames the peer received).
pub delivery_ratio_forward: f64,
/// Reverse delivery ratio (set when we compute from our own receiver state).
pub delivery_ratio_reverse: f64,
/// ETX computed from bidirectional delivery ratios.
pub etx: f64,
/// Smoothed goodput in bytes/sec (forward direction: what the peer received from us).
pub goodput_bps: f64,
// --- State for delta computation ---
/// Previous ReceiverReport's cumulative counters (for computing interval deltas).
prev_rr_cum_packets: u64,
prev_rr_cum_bytes: u64,
prev_rr_highest_counter: u64,
prev_rr_ecn_ce: u32,
prev_rr_reorder: u32,
/// Time of previous ReceiverReport (for goodput rate computation).
prev_rr_time: Option<Instant>,
/// Whether we have a previous ReceiverReport for delta computation.
has_prev_rr: bool,
// --- State for reverse delivery ratio delta computation ---
/// Previous reverse-side cumulative packets received (our receiver state).
prev_reverse_packets: u64,
/// Previous reverse-side highest counter (our receiver state).
prev_reverse_highest: u64,
/// Whether we have a previous reverse-side snapshot for delta computation.
has_prev_reverse: bool,
}
impl MmpMetrics {
/// Reset state derived from ReceiverReport counters for rekey cutover.
///
/// The new session starts with counter 0, so the prev_rr deltas must
/// be reset to avoid computing bogus loss/goodput from the counter
/// discontinuity. RTT (SRTT) is preserved since it remains valid.
pub fn reset_for_rekey(&mut self) {
self.prev_rr_cum_packets = 0;
self.prev_rr_cum_bytes = 0;
self.prev_rr_highest_counter = 0;
self.prev_rr_ecn_ce = 0;
self.prev_rr_reorder = 0;
self.prev_rr_time = None;
self.has_prev_rr = false;
self.delivery_ratio_forward = 1.0;
self.prev_reverse_packets = 0;
self.prev_reverse_highest = 0;
self.has_prev_reverse = false;
// Keep srtt, etx, trends, goodput_bps — they'll refresh from data
}
pub fn new() -> Self {
Self {
srtt: SrttEstimator::new(),
rtt_trend: DualEwma::new(),
loss_trend: DualEwma::new(),
goodput_trend: DualEwma::new(),
jitter_trend: DualEwma::new(),
etx_trend: DualEwma::new(),
delivery_ratio_forward: 1.0,
delivery_ratio_reverse: 1.0,
etx: 1.0,
goodput_bps: 0.0,
prev_rr_cum_packets: 0,
prev_rr_cum_bytes: 0,
prev_rr_highest_counter: 0,
prev_rr_ecn_ce: 0,
prev_rr_reorder: 0,
prev_rr_time: None,
has_prev_rr: false,
prev_reverse_packets: 0,
prev_reverse_highest: 0,
has_prev_reverse: false,
}
}
/// Process an incoming ReceiverReport (from the peer about our traffic).
///
/// `our_timestamp_ms` is the current session-relative time in ms (for RTT).
/// `now` is the current monotonic time (for goodput rate computation).
///
/// Returns `true` if this report produced the first SRTT measurement
/// (transition from uninitialized to initialized).
pub fn process_receiver_report(
&mut self,
rr: &ReceiverReport,
our_timestamp_ms: u32,
now: Instant,
) -> bool {
let had_srtt = self.srtt.initialized();
if self.has_prev_rr {
let counters_regressed = rr.highest_counter < self.prev_rr_highest_counter
|| rr.cumulative_packets_recv < self.prev_rr_cum_packets
|| rr.cumulative_bytes_recv < self.prev_rr_cum_bytes
|| rr.ecn_ce_count < self.prev_rr_ecn_ce
|| rr.cumulative_reorder_count < self.prev_rr_reorder;
let duplicate_counters = rr.highest_counter == self.prev_rr_highest_counter
&& rr.cumulative_packets_recv == self.prev_rr_cum_packets
&& rr.cumulative_bytes_recv == self.prev_rr_cum_bytes
&& rr.ecn_ce_count == self.prev_rr_ecn_ce
&& rr.cumulative_reorder_count == self.prev_rr_reorder;
// Safe to drop: reports are only built after interval data, so
// a fresh report always advances at least one cumulative counter.
if counters_regressed || duplicate_counters {
trace!(
highest_counter = rr.highest_counter,
prev_highest_counter = self.prev_rr_highest_counter,
cumulative_packets_recv = rr.cumulative_packets_recv,
prev_cumulative_packets_recv = self.prev_rr_cum_packets,
cumulative_bytes_recv = rr.cumulative_bytes_recv,
prev_cumulative_bytes_recv = self.prev_rr_cum_bytes,
"Ignoring stale MMP ReceiverReport"
);
return false;
}
}
// --- RTT from timestamp echo ---
// RTT = now - echoed_timestamp - dwell_time
if rr.timestamp_echo > 0 {
let echo_ms = rr.timestamp_echo;
let dwell_ms = u32::from(rr.dwell_time);
let rtt_sample_ms = echo_ms
.checked_add(dwell_ms)
.and_then(|send_done_ms| our_timestamp_ms.checked_sub(send_done_ms));
match rtt_sample_ms {
Some(rtt_ms) if rtt_ms > 0 => {
let rtt_us = (rtt_ms as i64) * 1000;
trace!(
our_ts = our_timestamp_ms,
echo = echo_ms,
dwell = dwell_ms,
rtt_ms = rtt_ms,
srtt_ms = self.srtt.srtt_us() as f64 / 1000.0,
"RTT sample from timestamp echo"
);
self.srtt.update(rtt_us);
self.rtt_trend.update(rtt_us as f64);
}
_ => {
trace!(
our_ts = our_timestamp_ms,
echo = echo_ms,
dwell = dwell_ms,
"Ignoring invalid MMP RTT sample"
);
}
}
}
// --- Loss rate from cumulative counters ---
// Delta: frames the peer should have received vs. actually received
if self.has_prev_rr {
let counter_span = rr
.highest_counter
.saturating_sub(self.prev_rr_highest_counter);
let packets_delta = rr
.cumulative_packets_recv
.saturating_sub(self.prev_rr_cum_packets);
if counter_span > 0 {
let delivery = (packets_delta as f64) / (counter_span as f64);
self.delivery_ratio_forward = delivery.clamp(0.0, 1.0);
let loss_rate = 1.0 - self.delivery_ratio_forward;
self.loss_trend.update(loss_rate);
self.etx = compute_etx(self.delivery_ratio_forward, self.delivery_ratio_reverse);
self.etx_trend.update(self.etx);
}
}
// --- Goodput from cumulative bytes + time delta ---
if self.has_prev_rr {
let bytes_delta = rr
.cumulative_bytes_recv
.saturating_sub(self.prev_rr_cum_bytes);
self.goodput_trend.update(bytes_delta as f64);
// Compute bytes/sec if we have a time reference
if let Some(prev_time) = self.prev_rr_time {
let elapsed = now.duration_since(prev_time);
let secs = elapsed.as_secs_f64();
if secs > 0.0 {
let bps = bytes_delta as f64 / secs;
// EWMA smoothing: α = 1/4
if self.goodput_bps == 0.0 {
self.goodput_bps = bps;
} else {
self.goodput_bps += (bps - self.goodput_bps) * 0.25;
}
}
}
}
// --- Jitter trend ---
self.jitter_trend.update(rr.jitter as f64);
// --- Save for next delta ---
self.prev_rr_cum_packets = rr.cumulative_packets_recv;
self.prev_rr_cum_bytes = rr.cumulative_bytes_recv;
self.prev_rr_highest_counter = rr.highest_counter;
self.prev_rr_ecn_ce = rr.ecn_ce_count;
self.prev_rr_reorder = rr.cumulative_reorder_count;
self.prev_rr_time = Some(now);
self.has_prev_rr = true;
!had_srtt && self.srtt.initialized()
}
/// Update the reverse delivery ratio from our own receiver state.
///
/// Computes a per-interval delta (same as forward ratio) rather than
/// a lifetime cumulative ratio, so ETX responds to recent conditions.
pub fn update_reverse_delivery(&mut self, our_recv_packets: u64, peer_highest: u64) {
if self.has_prev_reverse {
let counter_span = peer_highest.saturating_sub(self.prev_reverse_highest);
let packets_delta = our_recv_packets.saturating_sub(self.prev_reverse_packets);
if counter_span > 0 {
let delivery = (packets_delta as f64) / (counter_span as f64);
self.delivery_ratio_reverse = delivery.clamp(0.0, 1.0);
self.etx = compute_etx(self.delivery_ratio_forward, self.delivery_ratio_reverse);
self.etx_trend.update(self.etx);
}
}
self.prev_reverse_packets = our_recv_packets;
self.prev_reverse_highest = peer_highest;
self.has_prev_reverse = true;
}
/// Current smoothed RTT in milliseconds, or `None` if not yet measured.
pub fn srtt_ms(&self) -> Option<f64> {
if self.srtt.initialized() {
Some(self.srtt.srtt_us() as f64 / 1000.0)
} else {
None
}
}
/// Current loss rate (0.0 = no loss, 1.0 = total loss).
pub fn loss_rate(&self) -> f64 {
1.0 - self.delivery_ratio_forward
}
/// Smoothed loss rate (long-term EWMA), or `None` if not yet initialized.
pub fn smoothed_loss(&self) -> Option<f64> {
if self.loss_trend.initialized() {
Some(self.loss_trend.long())
} else {
None
}
}
/// Smoothed ETX (long-term EWMA), or `None` if not yet initialized.
pub fn smoothed_etx(&self) -> Option<f64> {
if self.etx_trend.initialized() {
Some(self.etx_trend.long())
} else {
None
}
}
/// Current smoothed goodput in bytes/sec, or 0 if not yet measured.
pub fn goodput_bps(&self) -> f64 {
self.goodput_bps
}
/// Cumulative ECN CE count from the most recent ReceiverReport.
pub fn last_ecn_ce_count(&self) -> u32 {
self.prev_rr_ecn_ce
}
}
impl Default for MmpMetrics {
fn default() -> Self {
Self::new()
}
}
// ============================================================================
// Tests
// ============================================================================
#[cfg(test)]
mod tests {
use super::*;
use std::time::Duration;
fn make_rr(
highest_counter: u64,
cum_packets: u64,
cum_bytes: u64,
timestamp_echo: u32,
dwell: u16,
jitter: u32,
) -> ReceiverReport {
ReceiverReport {
highest_counter,
cumulative_packets_recv: cum_packets,
cumulative_bytes_recv: cum_bytes,
timestamp_echo,
dwell_time: dwell,
max_burst_loss: 0,
mean_burst_loss: 0,
jitter,
ecn_ce_count: 0,
owd_trend: 0,
burst_loss_count: 0,
cumulative_reorder_count: 0,
interval_packets_recv: 0,
interval_bytes_recv: 0,
}
}
#[test]
fn test_rtt_from_echo() {
let mut m = MmpMetrics::new();
let now = Instant::now();
// Peer echoes timestamp 1000ms, dwell=5ms, our current time=1050ms
let rr = make_rr(10, 10, 5000, 1000, 5, 0);
m.process_receiver_report(&rr, 1050, now);
assert!(m.srtt.initialized());
// RTT = 1050 - 1000 - 5 = 45ms
let srtt_ms = m.srtt_ms().unwrap();
assert!((srtt_ms - 45.0).abs() < 1.0, "srtt={srtt_ms}, expected ~45");
}
#[test]
fn test_ignores_duplicate_receiver_report_after_valid_sample() {
let mut m = MmpMetrics::new();
let t0 = Instant::now();
let rr1 = make_rr(10, 10, 5_000, 1_000, 5, 0);
m.process_receiver_report(&rr1, 1_050, t0);
let rr2 = make_rr(20, 18, 14_000, 1_100, 5, 0);
m.process_receiver_report(&rr2, 1_150, t0 + Duration::from_secs(1));
let baseline_srtt_ms = m.srtt_ms().unwrap();
let baseline_loss = m.loss_rate();
let baseline_goodput = m.goodput_bps();
assert!(baseline_loss > 0.0);
assert!(baseline_goodput > 0.0);
// A duplicate of the same counters arriving later would be a 4.895s
// RTT sample if accepted. It is stale and must not move metrics.
m.process_receiver_report(&rr2, 6_000, t0 + Duration::from_secs(5));
assert_eq!(m.srtt_ms().unwrap(), baseline_srtt_ms);
assert_eq!(m.loss_rate(), baseline_loss);
assert_eq!(m.goodput_bps(), baseline_goodput);
}
#[test]
fn test_ignores_out_of_order_receiver_report_after_valid_sample() {
let mut m = MmpMetrics::new();
let now = Instant::now();
let valid_rr = make_rr(20, 20, 10000, 1000, 5, 0);
m.process_receiver_report(&valid_rr, 1050, now);
let baseline_srtt_ms = m.srtt_ms().unwrap();
let old_rr = make_rr(10, 10, 5000, 1000, 0, 0);
m.process_receiver_report(&old_rr, 6000, now + Duration::from_secs(5));
let srtt_ms = m.srtt_ms().unwrap();
assert_eq!(srtt_ms, baseline_srtt_ms);
}
#[test]
fn test_ignores_wrapped_rtt_sample() {
let mut m = MmpMetrics::new();
let now = Instant::now();
let wrapped_rr = make_rr(10, 10, 5000, u32::MAX - 10, 20, 0);
m.process_receiver_report(&wrapped_rr, 15, now);
assert!(m.srtt_ms().is_none());
}
#[test]
fn test_ignores_future_rtt_sample() {
let mut m = MmpMetrics::new();
let now = Instant::now();
let future_rr = make_rr(10, 10, 5_000, 2_000, 5, 0);
m.process_receiver_report(&future_rr, 1_000, now);
assert!(m.srtt_ms().is_none());
}
#[test]
fn test_loss_rate_computation() {
let mut m = MmpMetrics::new();
let t0 = Instant::now();
// First report: baseline
let rr1 = make_rr(100, 100, 50000, 0, 0, 0);
m.process_receiver_report(&rr1, 0, t0);
// Second report: 200 counters sent, 190 received (5% loss)
let rr2 = make_rr(300, 290, 145000, 0, 0, 0);
m.process_receiver_report(&rr2, 0, t0 + Duration::from_secs(1));
let loss = m.loss_rate();
assert!((loss - 0.05).abs() < 0.01, "loss={loss}, expected ~0.05");
}
#[test]
fn test_etx_updates() {
let mut m = MmpMetrics::new();
assert_eq!(m.etx, 1.0); // initial: perfect
// Simulate some loss via forward ratio
m.delivery_ratio_forward = 0.9;
// First call establishes the baseline (no ETX update yet)
m.update_reverse_delivery(100, 100);
assert_eq!(m.etx, 1.0); // still perfect — baseline only
// Second call: 190 of 200 frames received (5% loss)
m.update_reverse_delivery(290, 300);
assert!(m.etx > 1.0);
assert!(m.etx < 2.0);
}
#[test]
fn test_no_rtt_without_echo() {
let mut m = MmpMetrics::new();
let now = Instant::now();
let rr = make_rr(10, 10, 5000, 0, 0, 0);
m.process_receiver_report(&rr, 1000, now);
assert!(m.srtt_ms().is_none());
}
#[test]
fn test_jitter_trend() {
let mut m = MmpMetrics::new();
let t0 = Instant::now();
let rr1 = make_rr(10, 10, 5000, 0, 0, 100);
m.process_receiver_report(&rr1, 0, t0);
let rr2 = make_rr(20, 20, 10000, 0, 0, 500);
m.process_receiver_report(&rr2, 0, t0 + Duration::from_secs(1));
assert!(m.jitter_trend.initialized());
// Short-term should be closer to 500 than long-term
assert!(m.jitter_trend.short() > m.jitter_trend.long());
}
#[test]
fn test_goodput_bps() {
let mut m = MmpMetrics::new();
let t0 = Instant::now();
// First report: baseline (50KB received)
let rr1 = make_rr(100, 100, 50_000, 0, 0, 0);
m.process_receiver_report(&rr1, 0, t0);
assert_eq!(m.goodput_bps(), 0.0); // no rate yet (first report)
// Second report 1s later: 150KB total (100KB delta in 1s = 100KB/s)
let rr2 = make_rr(300, 290, 150_000, 0, 0, 0);
m.process_receiver_report(&rr2, 0, t0 + Duration::from_secs(1));
assert!(
m.goodput_bps() > 90_000.0,
"goodput={}, expected ~100000",
m.goodput_bps()
);
assert!(
m.goodput_bps() < 110_000.0,
"goodput={}, expected ~100000",
m.goodput_bps()
);
}
#[test]
fn test_reverse_delivery_delta() {
let mut m = MmpMetrics::new();
// First call: baseline only, no ratio update
m.update_reverse_delivery(100, 100);
assert_eq!(m.delivery_ratio_reverse, 1.0); // unchanged from default
// Second call: perfect delivery (200 new frames, all received)
m.update_reverse_delivery(300, 300);
assert!((m.delivery_ratio_reverse - 1.0).abs() < 0.001);
// Third call: 50% loss (100 frames sent, 50 received)
m.update_reverse_delivery(350, 400);
assert!(
(m.delivery_ratio_reverse - 0.5).abs() < 0.001,
"reverse={}, expected 0.5",
m.delivery_ratio_reverse
);
}
#[test]
fn test_reverse_delivery_rekey_reset() {
let mut m = MmpMetrics::new();
// Establish baseline and one measurement
m.update_reverse_delivery(100, 100);
m.update_reverse_delivery(300, 300);
assert!((m.delivery_ratio_reverse - 1.0).abs() < 0.001);
// Rekey resets reverse state
m.reset_for_rekey();
// First call after rekey: baseline only
m.update_reverse_delivery(50, 50);
// delivery_ratio_reverse was reset to 1.0 by reset_for_rekey's
// clearing of delivery_ratio_forward; reverse is not explicitly
// reset — but the delta state is, so next call computes fresh.
assert_eq!(m.delivery_ratio_reverse, 1.0);
// Second call after rekey: 80% delivery
m.update_reverse_delivery(90, 100);
assert!(
(m.delivery_ratio_reverse - 0.8).abs() < 0.001,
"reverse={}, expected 0.8",
m.delivery_ratio_reverse
);
}
}
src/mmp/mod.rs
-555
View File
@@ -1,555 +0,0 @@
//! Metrics Measurement Protocol (MMP) — link-layer instantiation.
//!
//! Measures link quality between adjacent peers: RTT, loss, jitter,
//! throughput, one-way delay trend, and ETX. Operates on the per-frame
//! hooks (counter, timestamp, flags) introduced by the FMP wire format
//! revision.
//!
//! Three operating modes trade measurement fidelity for overhead:
//! - **Full**: sender + receiver reports at RTT-adaptive intervals
//! - **Lightweight**: receiver reports only (infer loss from counters)
//! - **Minimal**: spin bit + CE echo only, no reports
use serde::{Deserialize, Serialize};
use std::fmt::{self, Debug};
use std::time::{Duration, Instant};
// Sub-modules
pub mod algorithms;
pub mod metrics;
pub mod receiver;
pub mod report;
pub mod sender;
// Re-exports
pub use algorithms::{
DualEwma, JitterEstimator, OwdTrendDetector, SpinBitState, SrttEstimator, compute_etx,
};
pub use metrics::MmpMetrics;
pub use receiver::ReceiverState;
pub use report::{ReceiverReport, SenderReport};
pub use sender::SenderState;
// Session-layer re-exports
// MmpSessionState and PathMtuState are defined in this file
// ============================================================================
// Constants
// ============================================================================
/// SenderReport body size (after msg_type byte): 3 reserved + 44 payload = 47.
pub const SENDER_REPORT_BODY_SIZE: usize = 47;
/// ReceiverReport body size (after msg_type byte): 3 reserved + 64 payload = 67.
pub const RECEIVER_REPORT_BODY_SIZE: usize = 67;
/// SenderReport total wire size including inner header: 5 + 47 = 52.
pub const SENDER_REPORT_WIRE_SIZE: usize = 52;
/// ReceiverReport total wire size including inner header: 5 + 67 = 72.
pub const RECEIVER_REPORT_WIRE_SIZE: usize = 72;
// --- EWMA parameters (as shift amounts for integer arithmetic) ---
/// Jitter EWMA: α = 1/16 (RFC 3550 §6.4.1).
pub const JITTER_ALPHA_SHIFT: u32 = 4;
/// SRTT: α = 1/8 (Jacobson, RFC 6298).
pub const SRTT_ALPHA_SHIFT: u32 = 3;
/// RTTVAR: β = 1/4 (Jacobson, RFC 6298).
pub const RTTVAR_BETA_SHIFT: u32 = 2;
/// Dual EWMA short-term: α = 1/4.
pub const EWMA_SHORT_ALPHA: f64 = 0.25;
/// Dual EWMA long-term: α = 1/32.
pub const EWMA_LONG_ALPHA: f64 = 1.0 / 32.0;
// --- Timing defaults (milliseconds) ---
/// Default report interval before SRTT is available (cold start).
pub const DEFAULT_COLD_START_INTERVAL_MS: u64 = 200;
/// Minimum report interval (SRTT clamp floor).
///
/// Raised from 100ms to 1000ms: parent re-evaluation runs every 60s,
/// so 60 samples/cycle is more than sufficient for EWMA convergence (~10).
/// The cold-start phase uses `DEFAULT_COLD_START_INTERVAL_MS` (200ms) for
/// fast initial SRTT convergence before transitioning to this floor.
pub const MIN_REPORT_INTERVAL_MS: u64 = 1_000;
/// Maximum report interval (SRTT clamp ceiling).
pub const MAX_REPORT_INTERVAL_MS: u64 = 5_000;
/// Number of SRTT samples before transitioning from cold-start to normal floor.
///
/// During cold-start, report intervals use `DEFAULT_COLD_START_INTERVAL_MS` as
/// the floor to gather SRTT samples quickly. After this many updates, the floor
/// switches to `MIN_REPORT_INTERVAL_MS`.
pub const COLD_START_SAMPLES: u32 = 5;
/// Default OWD ring buffer capacity.
pub const DEFAULT_OWD_WINDOW_SIZE: usize = 32;
/// Default operator log interval in seconds.
pub const DEFAULT_LOG_INTERVAL_SECS: u64 = 30;
// --- Session-layer timing defaults ---
// Session reports are routed end-to-end (bandwidth cost on every transit link),
// so intervals are higher than link-layer.
/// Session-layer minimum report interval.
pub const MIN_SESSION_REPORT_INTERVAL_MS: u64 = 500;
/// Session-layer maximum report interval.
pub const MAX_SESSION_REPORT_INTERVAL_MS: u64 = 10_000;
/// Session-layer cold-start report interval (before SRTT is available).
pub const SESSION_COLD_START_INTERVAL_MS: u64 = 1_000;
// ============================================================================
// Operating Mode
// ============================================================================
/// MMP operating mode.
#[derive(Debug, Default, Clone, Copy, PartialEq, Eq, Serialize, Deserialize)]
#[serde(rename_all = "lowercase")]
pub enum MmpMode {
/// Sender + receiver reports at RTT-adaptive intervals. Maximum fidelity.
#[default]
Full,
/// Receiver reports only. Loss inferred from counter gaps.
Lightweight,
/// Spin bit + CE echo only. No reports exchanged.
Minimal,
}
impl fmt::Display for MmpMode {
fn fmt(&self, f: &mut fmt::Formatter<'_>) -> fmt::Result {
match self {
MmpMode::Full => write!(f, "full"),
MmpMode::Lightweight => write!(f, "lightweight"),
MmpMode::Minimal => write!(f, "minimal"),
}
}
}
// ============================================================================
// Configuration
// ============================================================================
/// MMP configuration (`node.mmp.*`).
#[derive(Debug, Clone, Serialize, Deserialize)]
pub struct MmpConfig {
/// Operating mode (`node.mmp.mode`).
#[serde(default)]
pub mode: MmpMode,
/// Periodic operator log interval in seconds (`node.mmp.log_interval_secs`).
#[serde(default = "MmpConfig::default_log_interval_secs")]
pub log_interval_secs: u64,
/// OWD trend ring buffer size (`node.mmp.owd_window_size`).
#[serde(default = "MmpConfig::default_owd_window_size")]
pub owd_window_size: usize,
}
impl Default for MmpConfig {
fn default() -> Self {
Self {
mode: MmpMode::default(),
log_interval_secs: DEFAULT_LOG_INTERVAL_SECS,
owd_window_size: DEFAULT_OWD_WINDOW_SIZE,
}
}
}
impl MmpConfig {
fn default_log_interval_secs() -> u64 {
DEFAULT_LOG_INTERVAL_SECS
}
fn default_owd_window_size() -> usize {
DEFAULT_OWD_WINDOW_SIZE
}
}
// ============================================================================
// Per-Peer MMP State
// ============================================================================
/// Combined MMP state for a single peer link.
///
/// Wraps sender, receiver, metrics, and spin bit state. One instance
/// per `ActivePeer`.
pub struct MmpPeerState {
pub sender: SenderState,
pub receiver: ReceiverState,
pub metrics: MmpMetrics,
pub spin_bit: SpinBitState,
mode: MmpMode,
log_interval: Duration,
last_log_time: Option<Instant>,
}
impl MmpPeerState {
/// Create MMP state for a new peer link.
///
/// `is_initiator`: true if this node initiated the Noise handshake
/// (determines spin bit role).
pub fn new(config: &MmpConfig, is_initiator: bool) -> Self {
Self {
sender: SenderState::new(),
receiver: ReceiverState::new(config.owd_window_size),
metrics: MmpMetrics::new(),
spin_bit: SpinBitState::new(is_initiator),
mode: config.mode,
log_interval: Duration::from_secs(config.log_interval_secs),
last_log_time: None,
}
}
/// Reset counter-dependent state for rekey cutover.
pub fn reset_for_rekey(&mut self, now: Instant) {
self.receiver.reset_for_rekey(now);
self.metrics.reset_for_rekey();
}
/// Current operating mode.
pub fn mode(&self) -> MmpMode {
self.mode
}
/// Check if it's time to emit a periodic metrics log.
pub fn should_log(&self, now: Instant) -> bool {
match self.last_log_time {
None => true,
Some(last) => now.duration_since(last) >= self.log_interval,
}
}
/// Mark that a periodic log was emitted.
pub fn mark_logged(&mut self, now: Instant) {
self.last_log_time = Some(now);
}
}
// ============================================================================
// Per-Session MMP State (session-layer instantiation)
// ============================================================================
/// Combined MMP state for a single end-to-end session.
///
/// Wraps sender, receiver, metrics, spin bit, and path MTU state.
/// One instance per established `SessionEntry`.
pub struct MmpSessionState {
pub sender: SenderState,
pub receiver: ReceiverState,
pub metrics: MmpMetrics,
pub spin_bit: SpinBitState,
mode: MmpMode,
log_interval: Duration,
last_log_time: Option<Instant>,
pub path_mtu: PathMtuState,
}
impl MmpSessionState {
/// Create MMP state for a new session.
///
/// `is_initiator`: true if this node initiated the Noise handshake
/// (determines spin bit role).
pub fn new(config: &crate::config::SessionMmpConfig, is_initiator: bool) -> Self {
Self {
sender: SenderState::new_with_cold_start(SESSION_COLD_START_INTERVAL_MS),
receiver: ReceiverState::new_with_cold_start(
config.owd_window_size,
SESSION_COLD_START_INTERVAL_MS,
),
metrics: MmpMetrics::new(),
spin_bit: SpinBitState::new(is_initiator),
mode: config.mode,
log_interval: Duration::from_secs(config.log_interval_secs),
last_log_time: None,
path_mtu: PathMtuState::new(),
}
}
/// Reset counter-dependent state for rekey cutover.
pub fn reset_for_rekey(&mut self, now: Instant) {
self.receiver.reset_for_rekey(now);
self.metrics.reset_for_rekey();
}
/// Current operating mode.
pub fn mode(&self) -> MmpMode {
self.mode
}
/// Check if it's time to emit a periodic metrics log.
pub fn should_log(&self, now: Instant) -> bool {
match self.last_log_time {
None => true,
Some(last) => now.duration_since(last) >= self.log_interval,
}
}
/// Mark that a periodic log was emitted.
pub fn mark_logged(&mut self, now: Instant) {
self.last_log_time = Some(now);
}
}
impl Debug for MmpSessionState {
fn fmt(&self, f: &mut fmt::Formatter<'_>) -> fmt::Result {
f.debug_struct("MmpSessionState")
.field("mode", &self.mode)
.field("path_mtu", &self.path_mtu.current_mtu())
.finish_non_exhaustive()
}
}
// ============================================================================
// Path MTU State (session-layer only)
// ============================================================================
/// Path MTU tracking for a single session.
///
/// Destination side: observes `path_mtu` from incoming SessionDatagram envelopes
/// and generates PathMtuNotification messages back to the source.
///
/// Source side: applies received PathMtuNotification to limit outbound datagram
/// size. Decrease is immediate; increase requires 3 consecutive notifications.
pub struct PathMtuState {
/// Current effective path MTU (what we use for sending).
current_mtu: u16,
/// Last observed path MTU from incoming datagrams (destination-side).
last_observed_mtu: u16,
/// Whether the observed MTU has changed since the last notification.
observed_changed: bool,
/// Last time a PathMtuNotification was sent.
last_notification_time: Option<Instant>,
/// Notification interval: max(10s, 5 * SRTT). Default 10s.
notification_interval: Duration,
/// For source-side increase tracking: consecutive higher-value notifications.
consecutive_increase_count: u8,
/// Time of the first notification in the current increase sequence.
first_increase_time: Option<Instant>,
/// The MTU value being proposed for increase.
pending_increase_mtu: u16,
}
impl PathMtuState {
/// Create path MTU state with no initial measurement.
pub fn new() -> Self {
Self {
current_mtu: u16::MAX,
last_observed_mtu: u16::MAX,
observed_changed: false,
last_notification_time: None,
notification_interval: Duration::from_secs(10),
consecutive_increase_count: 0,
first_increase_time: None,
pending_increase_mtu: 0,
}
}
/// Current effective path MTU (source-side, for sending).
pub fn current_mtu(&self) -> u16 {
self.current_mtu
}
/// Last observed incoming path MTU (destination-side).
pub fn last_observed_mtu(&self) -> u16 {
self.last_observed_mtu
}
/// Update notification interval from SRTT: max(10s, 5 * SRTT).
pub fn update_interval_from_srtt(&mut self, srtt_ms: f64) {
let five_srtt = Duration::from_millis((srtt_ms * 5.0) as u64);
self.notification_interval = five_srtt.max(Duration::from_secs(10));
}
/// Seed source-side current_mtu from outbound transport MTU.
///
/// Called on each send. Only decreases (never increases) the current_mtu
/// so the destination's PathMtuNotification can still raise it later.
/// Ensures current_mtu doesn't stay at u16::MAX before any notification
/// arrives from the destination.
pub fn seed_source_mtu(&mut self, outbound_mtu: u16) {
if outbound_mtu < self.current_mtu {
self.current_mtu = outbound_mtu;
}
}
// --- Destination side ---
/// Observe the path_mtu from an incoming SessionDatagram envelope.
///
/// Called on the destination (receiver) side for every session message.
pub fn observe_incoming_mtu(&mut self, path_mtu: u16) {
if path_mtu != self.last_observed_mtu {
self.observed_changed = true;
self.last_observed_mtu = path_mtu;
}
}
/// Check if a PathMtuNotification should be sent.
///
/// Send on first measurement, on decrease (immediate), or periodic
/// confirmation at the notification interval.
pub fn should_send_notification(&self, now: Instant) -> bool {
if self.last_observed_mtu == u16::MAX {
return false; // No measurement yet
}
match self.last_notification_time {
None => true, // First measurement
Some(last) => {
// Immediate on decrease
if self.observed_changed && self.last_observed_mtu < self.current_mtu {
return true;
}
// Periodic confirmation
now.duration_since(last) >= self.notification_interval
}
}
}
/// Build a PathMtuNotification from current state.
///
/// Returns the path_mtu value to send. Caller handles encoding.
pub fn build_notification(&mut self, now: Instant) -> Option<u16> {
if self.last_observed_mtu == u16::MAX {
return None;
}
self.last_notification_time = Some(now);
self.observed_changed = false;
Some(self.last_observed_mtu)
}
// --- Source side ---
/// Apply a received PathMtuNotification.
///
/// - Decrease: immediate (take the lower value).
/// - Increase: require 3 consecutive notifications with the same higher
/// value, spanning at least 2 * notification_interval.
///
/// Returns `true` if the effective MTU changed.
pub fn apply_notification(&mut self, reported_mtu: u16, now: Instant) -> bool {
if reported_mtu < self.current_mtu {
// Decrease: immediate
self.current_mtu = reported_mtu;
self.consecutive_increase_count = 0;
self.first_increase_time = None;
return true;
}
if reported_mtu > self.current_mtu {
// Increase: track consecutive notifications
if reported_mtu == self.pending_increase_mtu {
self.consecutive_increase_count += 1;
} else {
// Different value: reset sequence
self.pending_increase_mtu = reported_mtu;
self.consecutive_increase_count = 1;
self.first_increase_time = Some(now);
}
// Accept increase after 3 consecutive spanning 2 * interval
if self.consecutive_increase_count >= 3
&& let Some(first_time) = self.first_increase_time
{
let required = self.notification_interval * 2;
if now.duration_since(first_time) >= required {
self.current_mtu = reported_mtu;
self.consecutive_increase_count = 0;
self.first_increase_time = None;
return true;
}
}
}
// No change (equal or increase not yet confirmed)
false
}
}
impl Default for PathMtuState {
fn default() -> Self {
Self::new()
}
}
impl Debug for MmpPeerState {
fn fmt(&self, f: &mut fmt::Formatter<'_>) -> fmt::Result {
f.debug_struct("MmpPeerState")
.field("mode", &self.mode)
.finish_non_exhaustive()
}
}
// ============================================================================
// Tests
// ============================================================================
#[cfg(test)]
mod tests {
use super::*;
#[test]
fn test_mode_default() {
assert_eq!(MmpMode::default(), MmpMode::Full);
}
#[test]
fn test_mode_display() {
assert_eq!(MmpMode::Full.to_string(), "full");
assert_eq!(MmpMode::Lightweight.to_string(), "lightweight");
assert_eq!(MmpMode::Minimal.to_string(), "minimal");
}
#[test]
fn test_mode_serde_roundtrip() {
let yaml = "full";
let mode: MmpMode = serde_yaml::from_str(yaml).unwrap();
assert_eq!(mode, MmpMode::Full);
let yaml = "lightweight";
let mode: MmpMode = serde_yaml::from_str(yaml).unwrap();
assert_eq!(mode, MmpMode::Lightweight);
let yaml = "minimal";
let mode: MmpMode = serde_yaml::from_str(yaml).unwrap();
assert_eq!(mode, MmpMode::Minimal);
}
#[test]
fn test_config_default() {
let config = MmpConfig::default();
assert_eq!(config.mode, MmpMode::Full);
assert_eq!(config.log_interval_secs, 30);
assert_eq!(config.owd_window_size, 32);
}
#[test]
fn test_config_yaml_parse() {
let yaml = r#"
mode: lightweight
log_interval_secs: 60
owd_window_size: 48
"#;
let config: MmpConfig = serde_yaml::from_str(yaml).unwrap();
assert_eq!(config.mode, MmpMode::Lightweight);
assert_eq!(config.log_interval_secs, 60);
assert_eq!(config.owd_window_size, 48);
}
#[test]
fn test_config_yaml_partial() {
let yaml = "mode: minimal";
let config: MmpConfig = serde_yaml::from_str(yaml).unwrap();
assert_eq!(config.mode, MmpMode::Minimal);
assert_eq!(config.log_interval_secs, DEFAULT_LOG_INTERVAL_SECS);
assert_eq!(config.owd_window_size, DEFAULT_OWD_WINDOW_SIZE);
}
}
src/mmp/report.rs
-385
View File
@@ -1,385 +0,0 @@
//! MMP report wire format: SenderReport and ReceiverReport.
//!
//! Serialization and deserialization for the two report types exchanged
//! between link-layer peers. Wire format follows the MMP design doc.
use crate::protocol::ProtocolError;
// ============================================================================
// SenderReport (msg_type 0x01, 48-byte body including type byte)
// ============================================================================
/// Link-layer sender report.
///
/// Wire layout (48 bytes total, sent as link message):
/// ```text
/// [0] msg_type = 0x01
/// [1-3] reserved (zero)
/// [4-11] interval_start_counter: u64 LE
/// [12-19] interval_end_counter: u64 LE
/// [20-23] interval_start_timestamp: u32 LE
/// [24-27] interval_end_timestamp: u32 LE
/// [28-31] interval_bytes_sent: u32 LE
/// [32-39] cumulative_packets_sent: u64 LE
/// [40-47] cumulative_bytes_sent: u64 LE
/// ```
#[derive(Debug, Clone, PartialEq, Eq)]
pub struct SenderReport {
pub interval_start_counter: u64,
pub interval_end_counter: u64,
pub interval_start_timestamp: u32,
pub interval_end_timestamp: u32,
pub interval_bytes_sent: u32,
pub cumulative_packets_sent: u64,
pub cumulative_bytes_sent: u64,
}
/// ReceiverReport (msg_type 0x02, 68-byte body including type byte)
///
/// Wire layout (68 bytes total, sent as link message):
/// ```text
/// [0] msg_type = 0x02
/// [1-3] reserved (zero)
/// [4-11] highest_counter: u64 LE
/// [12-19] cumulative_packets_recv: u64 LE
/// [20-27] cumulative_bytes_recv: u64 LE
/// [28-31] timestamp_echo: u32 LE
/// [32-33] dwell_time: u16 LE
/// [34-35] max_burst_loss: u16 LE
/// [36-37] mean_burst_loss: u16 LE (u8.8 fixed-point)
/// [38-39] reserved: u16 LE
/// [40-43] jitter: u32 LE (microseconds)
/// [44-47] ecn_ce_count: u32 LE
/// [48-51] owd_trend: i32 LE (µs/s)
/// [52-55] burst_loss_count: u32 LE
/// [56-59] cumulative_reorder_count: u32 LE
/// [60-63] interval_packets_recv: u32 LE
/// [64-67] interval_bytes_recv: u32 LE
/// ```
#[derive(Debug, Clone, PartialEq, Eq)]
pub struct ReceiverReport {
pub highest_counter: u64,
pub cumulative_packets_recv: u64,
pub cumulative_bytes_recv: u64,
pub timestamp_echo: u32,
pub dwell_time: u16,
pub max_burst_loss: u16,
pub mean_burst_loss: u16,
pub jitter: u32,
pub ecn_ce_count: u32,
pub owd_trend: i32,
pub burst_loss_count: u32,
pub cumulative_reorder_count: u32,
pub interval_packets_recv: u32,
pub interval_bytes_recv: u32,
}
// Encode/decode will be implemented in Step 2.
impl SenderReport {
/// Encode to wire format (48 bytes: msg_type + 3 reserved + 44 payload).
pub fn encode(&self) -> Vec<u8> {
let mut buf = Vec::with_capacity(48);
buf.push(0x01); // msg_type
buf.extend_from_slice(&[0u8; 3]); // reserved
buf.extend_from_slice(&self.interval_start_counter.to_le_bytes());
buf.extend_from_slice(&self.interval_end_counter.to_le_bytes());
buf.extend_from_slice(&self.interval_start_timestamp.to_le_bytes());
buf.extend_from_slice(&self.interval_end_timestamp.to_le_bytes());
buf.extend_from_slice(&self.interval_bytes_sent.to_le_bytes());
buf.extend_from_slice(&self.cumulative_packets_sent.to_le_bytes());
buf.extend_from_slice(&self.cumulative_bytes_sent.to_le_bytes());
buf
}
/// Decode from payload after msg_type byte has been consumed.
///
/// `payload` starts at the reserved bytes (offset 1 in the wire format).
pub fn decode(payload: &[u8]) -> Result<Self, ProtocolError> {
if payload.len() < 47 {
return Err(ProtocolError::MessageTooShort {
expected: 47,
got: payload.len(),
});
}
// Skip 3 reserved bytes
let p = &payload[3..];
Ok(Self {
interval_start_counter: u64::from_le_bytes(p[0..8].try_into().unwrap()),
interval_end_counter: u64::from_le_bytes(p[8..16].try_into().unwrap()),
interval_start_timestamp: u32::from_le_bytes(p[16..20].try_into().unwrap()),
interval_end_timestamp: u32::from_le_bytes(p[20..24].try_into().unwrap()),
interval_bytes_sent: u32::from_le_bytes(p[24..28].try_into().unwrap()),
cumulative_packets_sent: u64::from_le_bytes(p[28..36].try_into().unwrap()),
cumulative_bytes_sent: u64::from_le_bytes(p[36..44].try_into().unwrap()),
})
}
}
impl ReceiverReport {
/// Encode to wire format (68 bytes: msg_type + 3 reserved + 64 payload).
pub fn encode(&self) -> Vec<u8> {
let mut buf = Vec::with_capacity(68);
buf.push(0x02); // msg_type
buf.extend_from_slice(&[0u8; 3]); // reserved
buf.extend_from_slice(&self.highest_counter.to_le_bytes());
buf.extend_from_slice(&self.cumulative_packets_recv.to_le_bytes());
buf.extend_from_slice(&self.cumulative_bytes_recv.to_le_bytes());
buf.extend_from_slice(&self.timestamp_echo.to_le_bytes());
buf.extend_from_slice(&self.dwell_time.to_le_bytes());
buf.extend_from_slice(&self.max_burst_loss.to_le_bytes());
buf.extend_from_slice(&self.mean_burst_loss.to_le_bytes());
buf.extend_from_slice(&[0u8; 2]); // reserved
buf.extend_from_slice(&self.jitter.to_le_bytes());
buf.extend_from_slice(&self.ecn_ce_count.to_le_bytes());
buf.extend_from_slice(&self.owd_trend.to_le_bytes());
buf.extend_from_slice(&self.burst_loss_count.to_le_bytes());
buf.extend_from_slice(&self.cumulative_reorder_count.to_le_bytes());
buf.extend_from_slice(&self.interval_packets_recv.to_le_bytes());
buf.extend_from_slice(&self.interval_bytes_recv.to_le_bytes());
buf
}
/// Decode from payload after msg_type byte has been consumed.
///
/// `payload` starts at the reserved bytes (offset 1 in the wire format).
pub fn decode(payload: &[u8]) -> Result<Self, ProtocolError> {
if payload.len() < 67 {
return Err(ProtocolError::MessageTooShort {
expected: 67,
got: payload.len(),
});
}
// Skip 3 reserved bytes
let p = &payload[3..];
Ok(Self {
highest_counter: u64::from_le_bytes(p[0..8].try_into().unwrap()),
cumulative_packets_recv: u64::from_le_bytes(p[8..16].try_into().unwrap()),
cumulative_bytes_recv: u64::from_le_bytes(p[16..24].try_into().unwrap()),
timestamp_echo: u32::from_le_bytes(p[24..28].try_into().unwrap()),
dwell_time: u16::from_le_bytes(p[28..30].try_into().unwrap()),
max_burst_loss: u16::from_le_bytes(p[30..32].try_into().unwrap()),
mean_burst_loss: u16::from_le_bytes(p[32..34].try_into().unwrap()),
// skip 2 reserved bytes at p[34..36]
jitter: u32::from_le_bytes(p[36..40].try_into().unwrap()),
ecn_ce_count: u32::from_le_bytes(p[40..44].try_into().unwrap()),
owd_trend: i32::from_le_bytes(p[44..48].try_into().unwrap()),
burst_loss_count: u32::from_le_bytes(p[48..52].try_into().unwrap()),
cumulative_reorder_count: u32::from_le_bytes(p[52..56].try_into().unwrap()),
interval_packets_recv: u32::from_le_bytes(p[56..60].try_into().unwrap()),
interval_bytes_recv: u32::from_le_bytes(p[60..64].try_into().unwrap()),
})
}
}
// ============================================================================
// Conversions between link-layer and session-layer report types
// ============================================================================
use crate::protocol::{SessionReceiverReport, SessionSenderReport};
impl From<&SenderReport> for SessionSenderReport {
fn from(r: &SenderReport) -> Self {
Self {
interval_start_counter: r.interval_start_counter,
interval_end_counter: r.interval_end_counter,
interval_start_timestamp: r.interval_start_timestamp,
interval_end_timestamp: r.interval_end_timestamp,
interval_bytes_sent: r.interval_bytes_sent,
cumulative_packets_sent: r.cumulative_packets_sent,
cumulative_bytes_sent: r.cumulative_bytes_sent,
}
}
}
impl From<&SessionSenderReport> for SenderReport {
fn from(r: &SessionSenderReport) -> Self {
Self {
interval_start_counter: r.interval_start_counter,
interval_end_counter: r.interval_end_counter,
interval_start_timestamp: r.interval_start_timestamp,
interval_end_timestamp: r.interval_end_timestamp,
interval_bytes_sent: r.interval_bytes_sent,
cumulative_packets_sent: r.cumulative_packets_sent,
cumulative_bytes_sent: r.cumulative_bytes_sent,
}
}
}
impl From<&ReceiverReport> for SessionReceiverReport {
fn from(r: &ReceiverReport) -> Self {
Self {
highest_counter: r.highest_counter,
cumulative_packets_recv: r.cumulative_packets_recv,
cumulative_bytes_recv: r.cumulative_bytes_recv,
timestamp_echo: r.timestamp_echo,
dwell_time: r.dwell_time,
max_burst_loss: r.max_burst_loss,
mean_burst_loss: r.mean_burst_loss,
jitter: r.jitter,
ecn_ce_count: r.ecn_ce_count,
owd_trend: r.owd_trend,
burst_loss_count: r.burst_loss_count,
cumulative_reorder_count: r.cumulative_reorder_count,
interval_packets_recv: r.interval_packets_recv,
interval_bytes_recv: r.interval_bytes_recv,
}
}
}
impl From<&SessionReceiverReport> for ReceiverReport {
fn from(r: &SessionReceiverReport) -> Self {
Self {
highest_counter: r.highest_counter,
cumulative_packets_recv: r.cumulative_packets_recv,
cumulative_bytes_recv: r.cumulative_bytes_recv,
timestamp_echo: r.timestamp_echo,
dwell_time: r.dwell_time,
max_burst_loss: r.max_burst_loss,
mean_burst_loss: r.mean_burst_loss,
jitter: r.jitter,
ecn_ce_count: r.ecn_ce_count,
owd_trend: r.owd_trend,
burst_loss_count: r.burst_loss_count,
cumulative_reorder_count: r.cumulative_reorder_count,
interval_packets_recv: r.interval_packets_recv,
interval_bytes_recv: r.interval_bytes_recv,
}
}
}
// ============================================================================
// Tests
// ============================================================================
#[cfg(test)]
mod tests {
use super::*;
fn sample_sender_report() -> SenderReport {
SenderReport {
interval_start_counter: 100,
interval_end_counter: 200,
interval_start_timestamp: 5000,
interval_end_timestamp: 6000,
interval_bytes_sent: 50_000,
cumulative_packets_sent: 10_000,
cumulative_bytes_sent: 5_000_000,
}
}
fn sample_receiver_report() -> ReceiverReport {
ReceiverReport {
highest_counter: 195,
cumulative_packets_recv: 9_500,
cumulative_bytes_recv: 4_750_000,
timestamp_echo: 5900,
dwell_time: 5,
max_burst_loss: 3,
mean_burst_loss: 384, // 1.5 in u8.8
jitter: 1200,
ecn_ce_count: 0,
owd_trend: -50,
burst_loss_count: 2,
cumulative_reorder_count: 10,
interval_packets_recv: 95,
interval_bytes_recv: 47_500,
}
}
#[test]
fn test_sender_report_encode_size() {
let sr = sample_sender_report();
let encoded = sr.encode();
assert_eq!(encoded.len(), 48);
assert_eq!(encoded[0], 0x01); // msg_type
}
#[test]
fn test_sender_report_roundtrip() {
let sr = sample_sender_report();
let encoded = sr.encode();
// decode expects payload after msg_type
let decoded = SenderReport::decode(&encoded[1..]).unwrap();
assert_eq!(sr, decoded);
}
#[test]
fn test_sender_report_too_short() {
let result = SenderReport::decode(&[0u8; 10]);
assert!(result.is_err());
}
#[test]
fn test_receiver_report_encode_size() {
let rr = sample_receiver_report();
let encoded = rr.encode();
assert_eq!(encoded.len(), 68);
assert_eq!(encoded[0], 0x02); // msg_type
}
#[test]
fn test_receiver_report_roundtrip() {
let rr = sample_receiver_report();
let encoded = rr.encode();
// decode expects payload after msg_type
let decoded = ReceiverReport::decode(&encoded[1..]).unwrap();
assert_eq!(rr, decoded);
}
#[test]
fn test_receiver_report_too_short() {
let result = ReceiverReport::decode(&[0u8; 10]);
assert!(result.is_err());
}
#[test]
fn test_sender_report_zero_values() {
let sr = SenderReport {
interval_start_counter: 0,
interval_end_counter: 0,
interval_start_timestamp: 0,
interval_end_timestamp: 0,
interval_bytes_sent: 0,
cumulative_packets_sent: 0,
cumulative_bytes_sent: 0,
};
let encoded = sr.encode();
let decoded = SenderReport::decode(&encoded[1..]).unwrap();
assert_eq!(sr, decoded);
}
#[test]
fn test_receiver_report_max_values() {
let rr = ReceiverReport {
highest_counter: u64::MAX,
cumulative_packets_recv: u64::MAX,
cumulative_bytes_recv: u64::MAX,
timestamp_echo: u32::MAX,
dwell_time: u16::MAX,
max_burst_loss: u16::MAX,
mean_burst_loss: u16::MAX,
jitter: u32::MAX,
ecn_ce_count: u32::MAX,
owd_trend: i32::MAX,
burst_loss_count: u32::MAX,
cumulative_reorder_count: u32::MAX,
interval_packets_recv: u32::MAX,
interval_bytes_recv: u32::MAX,
};
let encoded = rr.encode();
let decoded = ReceiverReport::decode(&encoded[1..]).unwrap();
assert_eq!(rr, decoded);
}
#[test]
fn test_receiver_report_negative_owd_trend() {
let rr = ReceiverReport {
owd_trend: -12345,
..sample_receiver_report()
};
let encoded = rr.encode();
let decoded = ReceiverReport::decode(&encoded[1..]).unwrap();
assert_eq!(decoded.owd_trend, -12345);
}
}
src/mmp/sender.rs
-418
View File
@@ -1,418 +0,0 @@
//! MMP sender state machine.
//!
//! Tracks what this node has sent to a specific peer and produces
//! SenderReport messages on demand. One `SenderState` per active peer.
use std::time::{Duration, Instant};
use crate::mmp::report::SenderReport;
use crate::mmp::{
COLD_START_SAMPLES, DEFAULT_COLD_START_INTERVAL_MS, MAX_REPORT_INTERVAL_MS,
MIN_REPORT_INTERVAL_MS,
};
/// Per-peer sender-side MMP state.
///
/// Records cumulative and interval counters for every frame transmitted
/// to this peer. Produces `SenderReport` snapshots on demand.
pub struct SenderState {
// --- Cumulative (lifetime) ---
cumulative_packets_sent: u64,
cumulative_bytes_sent: u64,
// --- Current interval ---
interval_start_counter: u64,
interval_start_timestamp: u32,
interval_bytes_sent: u32,
/// Counter of the most recently sent frame.
last_counter: u64,
/// Timestamp of the most recently sent frame.
last_timestamp: u32,
/// Whether any frames have been sent in the current interval.
interval_has_data: bool,
// --- Report timing ---
last_report_time: Option<Instant>,
report_interval: Duration,
// --- Send failure backoff ---
/// Consecutive send failure count for backoff calculation.
consecutive_send_failures: u32,
// --- Cold-start tracking ---
/// Number of SRTT-based interval updates received.
srtt_sample_count: u32,
}
impl SenderState {
pub fn new() -> Self {
Self::new_with_cold_start(DEFAULT_COLD_START_INTERVAL_MS)
}
/// Create with a custom cold-start interval (ms).
///
/// Used by session-layer MMP which needs a longer initial interval
/// since reports consume bandwidth on every transit link.
pub fn new_with_cold_start(cold_start_ms: u64) -> Self {
Self {
cumulative_packets_sent: 0,
cumulative_bytes_sent: 0,
interval_start_counter: 0,
interval_start_timestamp: 0,
interval_bytes_sent: 0,
last_counter: 0,
last_timestamp: 0,
interval_has_data: false,
last_report_time: None,
report_interval: Duration::from_millis(cold_start_ms),
consecutive_send_failures: 0,
srtt_sample_count: 0,
}
}
/// Record a frame sent to this peer.
///
/// Called on the TX path for every encrypted link message.
/// `counter` is the AEAD nonce/counter, `timestamp` is the inner header
/// session-relative timestamp (ms), `bytes` is the wire payload size.
pub fn record_sent(&mut self, counter: u64, timestamp: u32, bytes: usize) {
if !self.interval_has_data {
self.interval_start_counter = counter;
self.interval_start_timestamp = timestamp;
self.interval_has_data = true;
}
self.last_counter = counter;
self.last_timestamp = timestamp;
self.interval_bytes_sent = self.interval_bytes_sent.saturating_add(bytes as u32);
self.cumulative_packets_sent += 1;
self.cumulative_bytes_sent += bytes as u64;
}
/// Build a SenderReport from current state and reset the interval.
///
/// Returns `None` if no frames have been sent since the last report.
pub fn build_report(&mut self, now: Instant) -> Option<SenderReport> {
if !self.interval_has_data {
return None;
}
let report = SenderReport {
interval_start_counter: self.interval_start_counter,
interval_end_counter: self.last_counter,
interval_start_timestamp: self.interval_start_timestamp,
interval_end_timestamp: self.last_timestamp,
interval_bytes_sent: self.interval_bytes_sent,
cumulative_packets_sent: self.cumulative_packets_sent,
cumulative_bytes_sent: self.cumulative_bytes_sent,
};
// Reset interval
self.interval_has_data = false;
self.interval_bytes_sent = 0;
self.last_report_time = Some(now);
Some(report)
}
/// Check if it's time to send a report.
///
/// When consecutive send failures have occurred, the effective interval
/// is multiplied by an exponential backoff factor (2^failures, capped at 32×).
pub fn should_send_report(&self, now: Instant) -> bool {
if !self.interval_has_data {
return false;
}
match self.last_report_time {
None => true, // Never sent a report — send immediately
Some(last) => {
let effective = self
.report_interval
.mul_f64(self.send_failure_backoff_multiplier());
now.duration_since(last) >= effective
}
}
}
/// Record a send failure. Returns the new consecutive failure count.
pub fn record_send_failure(&mut self) -> u32 {
self.consecutive_send_failures += 1;
self.consecutive_send_failures
}
/// Record a successful send. Returns the previous failure count (for summary logging).
pub fn record_send_success(&mut self) -> u32 {
let prev = self.consecutive_send_failures;
self.consecutive_send_failures = 0;
prev
}
/// Get the backoff multiplier based on consecutive failures.
///
/// Returns 1.0 for no failures, 2.0 for 1 failure, 4.0 for 2, ...
/// capped at 32.0 (5 failures).
pub fn send_failure_backoff_multiplier(&self) -> f64 {
if self.consecutive_send_failures == 0 {
1.0
} else {
2.0_f64.powi(self.consecutive_send_failures.min(5) as i32)
}
}
/// Update the report interval based on SRTT (link-layer defaults).
///
/// Sender reports at 2× SRTT clamped to [floor, MAX]. During cold-start
/// (first `COLD_START_SAMPLES` updates), the floor is the cold-start
/// interval (200ms) for fast SRTT convergence. After that, it rises to
/// `MIN_REPORT_INTERVAL_MS` (1000ms) for steady-state efficiency.
pub fn update_report_interval_from_srtt(&mut self, srtt_us: i64) {
self.srtt_sample_count = self.srtt_sample_count.saturating_add(1);
let floor = if self.srtt_sample_count <= COLD_START_SAMPLES {
DEFAULT_COLD_START_INTERVAL_MS
} else {
MIN_REPORT_INTERVAL_MS
};
self.update_report_interval_with_bounds(srtt_us, floor, MAX_REPORT_INTERVAL_MS);
}
/// Update the report interval based on SRTT with custom bounds.
///
/// Used by session-layer MMP which needs higher clamp values since
/// each report consumes bandwidth on every transit link.
pub fn update_report_interval_with_bounds(&mut self, srtt_us: i64, min_ms: u64, max_ms: u64) {
if srtt_us <= 0 {
return;
}
let interval_us = (srtt_us * 2) as u64;
let interval_ms = (interval_us / 1000).clamp(min_ms, max_ms);
self.report_interval = Duration::from_millis(interval_ms);
}
// --- Accessors ---
pub fn cumulative_packets_sent(&self) -> u64 {
self.cumulative_packets_sent
}
pub fn cumulative_bytes_sent(&self) -> u64 {
self.cumulative_bytes_sent
}
pub fn report_interval(&self) -> Duration {
self.report_interval
}
pub fn consecutive_send_failures(&self) -> u32 {
self.consecutive_send_failures
}
}
impl Default for SenderState {
fn default() -> Self {
Self::new()
}
}
// ============================================================================
// Tests
// ============================================================================
#[cfg(test)]
mod tests {
use super::*;
#[test]
fn test_new_sender_state() {
let s = SenderState::new();
assert_eq!(s.cumulative_packets_sent(), 0);
assert_eq!(s.cumulative_bytes_sent(), 0);
}
#[test]
fn test_record_sent() {
let mut s = SenderState::new();
s.record_sent(1, 100, 500);
s.record_sent(2, 200, 600);
assert_eq!(s.cumulative_packets_sent(), 2);
assert_eq!(s.cumulative_bytes_sent(), 1100);
}
#[test]
fn test_build_report_empty() {
let mut s = SenderState::new();
assert!(s.build_report(Instant::now()).is_none());
}
#[test]
fn test_build_report() {
let mut s = SenderState::new();
s.record_sent(10, 1000, 500);
s.record_sent(11, 1100, 600);
s.record_sent(12, 1200, 400);
let report = s.build_report(Instant::now()).unwrap();
assert_eq!(report.interval_start_counter, 10);
assert_eq!(report.interval_end_counter, 12);
assert_eq!(report.interval_start_timestamp, 1000);
assert_eq!(report.interval_end_timestamp, 1200);
assert_eq!(report.interval_bytes_sent, 1500);
assert_eq!(report.cumulative_packets_sent, 3);
assert_eq!(report.cumulative_bytes_sent, 1500);
}
#[test]
fn test_build_report_resets_interval() {
let mut s = SenderState::new();
s.record_sent(1, 100, 500);
let _ = s.build_report(Instant::now());
// Second report with no new data returns None
assert!(s.build_report(Instant::now()).is_none());
// New data starts a fresh interval
s.record_sent(2, 200, 300);
let report = s.build_report(Instant::now()).unwrap();
assert_eq!(report.interval_start_counter, 2);
assert_eq!(report.interval_bytes_sent, 300);
// Cumulative continues
assert_eq!(report.cumulative_packets_sent, 2);
assert_eq!(report.cumulative_bytes_sent, 800);
}
#[test]
fn test_should_send_report_no_data() {
let s = SenderState::new();
assert!(!s.should_send_report(Instant::now()));
}
#[test]
fn test_should_send_report_first_time() {
let mut s = SenderState::new();
s.record_sent(1, 100, 500);
assert!(s.should_send_report(Instant::now()));
}
#[test]
fn test_should_send_report_respects_interval() {
let mut s = SenderState::new();
let t0 = Instant::now();
s.record_sent(1, 100, 500);
let _ = s.build_report(t0);
s.record_sent(2, 200, 500);
// Immediately after report — should not send
assert!(!s.should_send_report(t0));
// After interval elapses
let t1 = t0 + s.report_interval() + Duration::from_millis(1);
assert!(s.should_send_report(t1));
}
#[test]
fn test_update_report_interval_cold_start() {
let mut s = SenderState::new();
// During cold-start, floor is 200ms (DEFAULT_COLD_START_INTERVAL_MS)
// 50ms RTT → 100ms sender interval (2× SRTT), clamped to cold-start floor 200ms
s.update_report_interval_from_srtt(50_000);
assert_eq!(s.report_interval(), Duration::from_millis(200));
// 500ms RTT → 1000ms sender interval (above cold-start floor)
s.update_report_interval_from_srtt(500_000);
assert_eq!(s.report_interval(), Duration::from_millis(1000));
}
#[test]
fn test_update_report_interval_after_cold_start() {
let mut s = SenderState::new();
// Burn through cold-start samples (COLD_START_SAMPLES = 5)
for _ in 0..COLD_START_SAMPLES {
s.update_report_interval_from_srtt(500_000);
}
// 6th sample: now in steady state, floor is MIN_REPORT_INTERVAL_MS (1000ms)
// 50ms RTT → 100ms sender interval (2× SRTT), clamped to 1000ms
s.update_report_interval_from_srtt(50_000);
assert_eq!(
s.report_interval(),
Duration::from_millis(MIN_REPORT_INTERVAL_MS)
);
// 3s RTT → 6s, clamped to max 5s
s.update_report_interval_from_srtt(3_000_000);
assert_eq!(
s.report_interval(),
Duration::from_millis(MAX_REPORT_INTERVAL_MS)
);
}
#[test]
fn test_backoff_multiplier_progression() {
let mut s = SenderState::new();
// No failures → multiplier 1.0
assert_eq!(s.send_failure_backoff_multiplier(), 1.0);
assert_eq!(s.consecutive_send_failures(), 0);
// Progressive failures: 2^1, 2^2, 2^3, 2^4, 2^5
let expected = [2.0, 4.0, 8.0, 16.0, 32.0];
for (i, &exp) in expected.iter().enumerate() {
let count = s.record_send_failure();
assert_eq!(count, (i + 1) as u32);
assert_eq!(s.send_failure_backoff_multiplier(), exp);
}
// Beyond 5 failures: stays capped at 32.0
s.record_send_failure(); // 6th
assert_eq!(s.send_failure_backoff_multiplier(), 32.0);
s.record_send_failure(); // 7th
assert_eq!(s.send_failure_backoff_multiplier(), 32.0);
}
#[test]
fn test_backoff_reset_on_success() {
let mut s = SenderState::new();
// Accumulate failures
s.record_send_failure();
s.record_send_failure();
s.record_send_failure();
assert_eq!(s.consecutive_send_failures(), 3);
assert_eq!(s.send_failure_backoff_multiplier(), 8.0);
// Success resets and returns previous count
let prev = s.record_send_success();
assert_eq!(prev, 3);
assert_eq!(s.consecutive_send_failures(), 0);
assert_eq!(s.send_failure_backoff_multiplier(), 1.0);
}
#[test]
fn test_backoff_success_with_no_prior_failures() {
let mut s = SenderState::new();
// Success with no failures returns 0
let prev = s.record_send_success();
assert_eq!(prev, 0);
assert_eq!(s.consecutive_send_failures(), 0);
}
#[test]
fn test_should_send_report_respects_backoff() {
let mut s = SenderState::new();
let t0 = Instant::now();
s.record_sent(1, 100, 500);
let _ = s.build_report(t0);
// Record a failure: multiplier becomes 2.0
s.record_send_failure();
s.record_sent(2, 200, 500);
// At 1× interval: should NOT send (backoff requires 2×)
let t1 = t0 + s.report_interval() + Duration::from_millis(1);
assert!(!s.should_send_report(t1));
// At 2× interval: should send
let t2 = t0 + s.report_interval() * 2 + Duration::from_millis(1);
assert!(s.should_send_report(t2));
}
}
src/node/bloom.rs
+31 -23
View File
@@ -4,12 +4,12 @@
//! including debounced propagation to peers.
use crate::NodeAddr;
use crate::bloom::BloomFilter;
use crate::protocol::FilterAnnounce;
use crate::proto::bloom::BloomFilter;
use crate::proto::bloom::FilterAnnounce;
use super::reject::BloomReject;
use super::{Node, NodeError};
use std::collections::HashMap;
use std::collections::BTreeMap;
use tracing::{debug, warn};
impl Node {
@@ -17,8 +17,8 @@ impl Node {
///
/// Returns a map of (peer_node_addr -> filter) for peers that
/// have sent us a FilterAnnounce.
pub(super) fn peer_inbound_filters(&self) -> HashMap<NodeAddr, BloomFilter> {
let mut filters = HashMap::new();
pub(super) fn peer_inbound_filters(&self) -> BTreeMap<NodeAddr, BloomFilter> {
let mut filters = BTreeMap::new();
for (addr, peer) in &self.peers {
if self.is_tree_peer(addr)
&& let Some(filter) = peer.inbound_filter()
@@ -29,27 +29,19 @@ impl Node {
filters
}
/// Build a FilterAnnounce for a specific peer.
///
/// The outgoing filter excludes the destination peer's own filter
/// to prevent routing loops (don't tell a peer about destinations
/// reachable only through them).
fn build_filter_announce(&mut self, exclude_peer: &NodeAddr) -> FilterAnnounce {
let peer_filters = self.peer_inbound_filters();
let filter = self
.bloom_state
.compute_outgoing_filter(exclude_peer, &peer_filters);
let sequence = self.bloom_state.next_sequence();
FilterAnnounce::new(filter, sequence)
}
/// Send a FilterAnnounce to a specific peer, respecting debounce.
///
/// `filter` is the outgoing filter for this peer, already computed
/// with the destination peer's own contribution excluded to prevent
/// routing loops (don't tell a peer about destinations reachable
/// only through them).
///
/// If the peer is rate-limited, the update stays pending for
/// delivery on the next tick cycle.
pub(super) async fn send_filter_announce_to_peer(
&mut self,
peer_addr: &NodeAddr,
filter: BloomFilter,
) -> Result<(), NodeError> {
let now_ms = std::time::SystemTime::now()
.duration_since(std::time::UNIX_EPOCH)
@@ -64,7 +56,7 @@ impl Node {
}
// Build and encode
let announce = self.build_filter_announce(peer_addr);
let announce = FilterAnnounce::new(filter, self.bloom_state.next_sequence());
let sent_filter = announce.filter.clone();
let encoded = announce.encode().map_err(|e| NodeError::SendFailed {
node_addr: *peer_addr,
@@ -87,7 +79,7 @@ impl Node {
// operator to see one clear message, not spam.
let max_fpr = self.config().node.bloom.max_inbound_fpr;
let out_fill = sent_filter.fill_ratio();
let out_fpr = out_fill.powi(sent_filter.hash_count() as i32);
let out_fpr = sent_filter.fpr();
if out_fpr > max_fpr {
let now = std::time::Instant::now();
let should_warn = self
@@ -142,8 +134,24 @@ impl Node {
.copied()
.collect();
if ready.is_empty() {
return;
}
// One snapshot and one union pass for the whole ready set. The
// send path never mutates peer inbound filters or the tree state,
// and the rx loop holds `&mut self` across the awaits, so the
// snapshot cannot go stale mid-loop.
let peer_filters = self.peer_inbound_filters();
let mut outgoing = self
.bloom_state
.compute_outgoing_filters(&ready, &peer_filters);
for peer_addr in ready {
if let Err(e) = self.send_filter_announce_to_peer(&peer_addr).await {
let Some(filter) = outgoing.remove(&peer_addr) else {
continue;
};
if let Err(e) = self.send_filter_announce_to_peer(&peer_addr, filter).await {
debug!(
peer = %self.peer_display_name(&peer_addr),
error = %e,
@@ -213,7 +221,7 @@ impl Node {
// to wipe a victim's contribution to aggregation.
let max_fpr = self.config().node.bloom.max_inbound_fpr;
let fill = announce.filter.fill_ratio();
let fpr = fill.powi(announce.filter.hash_count() as i32);
let fpr = announce.filter.fpr();
if fpr > max_fpr {
self.metrics()
.bloom
src/node/handlers/connected_udp.rs → src/node/dataplane/connected_udp.rs
+18 -16
View File
@@ -45,12 +45,10 @@ impl Node {
/// (e.g. only non-UDP transports). Enabled on Linux and macOS:
/// both kernels route a matching peer 5-tuple to the connected
/// socket when it shares the wildcard listen port via SO_REUSEPORT.
/// Only compiled on Linux/macOS — the sole caller (the rx_loop tick) is
/// gated the same way, so on other targets (android) there is nothing to do.
#[cfg(any(target_os = "linux", target_os = "macos"))]
pub(in crate::node) async fn activate_connected_udp_sessions(&mut self) {
#[cfg(not(any(target_os = "linux", target_os = "macos")))]
{
// No-op on platforms without the connected-UDP fast path.
}
#[cfg(any(target_os = "linux", target_os = "macos"))]
{
if !connected_udp_enabled() {
return;
@@ -142,20 +140,24 @@ impl Node {
(peer_sa, local, recv_buf, send_buf, tx)
};
// Open the connected socket on the kernel side.
let socket = std::sync::Arc::new(
crate::transport::udp::connected_peer::ConnectedPeerSocket::open(
local_addr,
peer_socket_addr,
recv_buf,
send_buf,
)
.map_err(|e| format!("ConnectedPeerSocket::open: {e}"))?,
);
// Open the connected socket on the kernel side, then adopt the
// fd into the owning handle.
let owned = crate::transport::udp::open_connected_fd(
local_addr,
peer_socket_addr,
recv_buf,
send_buf,
)
.map_err(|e| format!("open_connected_fd: {e}"))?;
let socket = std::sync::Arc::new(crate::peer::connected_udp::ConnectedPeerSocket::from_fd(
owned,
peer_socket_addr,
local_addr,
));
// Spawn the drain thread. It feeds `packet_tx` exactly like
// the wildcard listen socket — rx_loop dispatches identically.
let drain = crate::transport::udp::peer_drain::PeerRecvDrain::spawn(
let drain = crate::peer::connected_udp::PeerRecvDrain::spawn(
socket.clone(),
transport_id,
peer_socket_addr,
src/node/handlers/dispatch.rs → src/node/dataplane/dispatch.rs
+11 -2
View File
@@ -73,7 +73,7 @@ impl Node {
/// entries — other removal paths (link-dead, decrypt failure, peer
/// restart) all schedule reconnect.
pub(in crate::node) fn handle_disconnect(&mut self, from: &NodeAddr, payload: &[u8]) {
let disconnect = match crate::protocol::Disconnect::decode(payload) {
let disconnect = match crate::proto::fmp::Disconnect::decode(payload) {
Ok(msg) => msg,
Err(e) => {
debug!(from = %self.peer_display_name(from), error = %e, "Malformed disconnect message");
@@ -93,7 +93,7 @@ impl Node {
.duration_since(std::time::UNIX_EPOCH)
.map(|d| d.as_millis() as u64)
.unwrap_or(0);
self.schedule_reconnect(addr, now_ms);
self.note_link_dead(addr, now_ms);
}
/// Remove an active peer and clean up all associated state.
@@ -187,6 +187,15 @@ impl Node {
// Remove link and address mapping
self.remove_link(&link_id);
// Bound `peer_machines`: drop this peer's machine
// entry, keyed by the `link_id` derived above BEFORE the `peers` removal.
// This cleans up the OLD peer's machine on an inbound restart and prevents
// unbounded growth on the establish success path. NEUTRAL: nothing on the
// live path reads `peer_machines` except the establish executor, which only
// ever touches the in-flight establish's (distinct) `link_id`; no reader
// depends on a stale entry, so removal changes no behavior — it only bounds
// the map.
self.remove_peer_machine(link_id);
if let Some(transport_id) = transport_id {
self.cleanup_bootstrap_transport_if_unused(transport_id);
}
src/node/handlers/encrypted.rs → src/node/dataplane/encrypted.rs
+13 -12
View File
@@ -1,10 +1,11 @@
//! Encrypted frame handling (hot path).
use crate::node::Node;
use crate::node::wire::{EncryptedHeader, FLAG_CE, FLAG_KEY_EPOCH, FLAG_SP, strip_inner_header};
use crate::noise::NoiseError;
use crate::proto::fmp::wire::{
EncryptedHeader, FLAG_CE, FLAG_KEY_EPOCH, FLAG_SP, strip_inner_header,
};
use crate::transport::ReceivedPacket;
use std::time::Instant;
use tracing::{debug, trace, warn};
/// Force-remove a peer after this many consecutive decryption failures.
@@ -171,7 +172,7 @@ impl Node {
#[cfg(unix)]
{
let cache_key = (packet.transport_id, header.receiver_idx.as_u32());
if let Some(workers) = self.decrypt_workers.as_ref().cloned()
if let Some(workers) = self.supervisor.decrypt_workers.as_ref().cloned()
&& self.decrypt_registered_sessions.contains(&cache_key)
{
let job = crate::node::decrypt_worker::DecryptJob {
@@ -259,7 +260,7 @@ impl Node {
};
// MMP per-frame processing and statistics
let now = Instant::now();
let now_ms = crate::time::mono_ms();
let ce_flag = header.flags & FLAG_CE != 0;
let sp_flag = header.flags & FLAG_SP != 0;
@@ -270,9 +271,9 @@ impl Node {
timestamp,
packet.data.len(),
ce_flag,
now,
now_ms,
);
let _spin_rtt = mmp.spin_bit.rx_observe(sp_flag, header.counter, now);
let _spin_rtt = mmp.spin_bit.rx_observe(sp_flag, header.counter, now_ms);
}
peer.set_current_addr(packet.transport_id, packet.remote_addr.clone());
peer.link_stats_mut()
@@ -355,7 +356,7 @@ impl Node {
} else {
return;
};
let now = Instant::now();
let now_ms = crate::time::mono_ms();
let mut address_changed = false;
if let Some(peer) = self.peers.get_mut(node_addr) {
peer.reset_decrypt_failures();
@@ -365,8 +366,8 @@ impl Node {
peer.touch(packet_timestamp_ms);
if let Some(mmp) = peer.mmp_mut() {
mmp.receiver
.record_recv(fmp_counter, inner_ts, packet_len, ce_flag, now);
let _spin_rtt = mmp.spin_bit.rx_observe(sp_flag, fmp_counter, now);
.record_recv(fmp_counter, inner_ts, packet_len, ce_flag, now_ms);
let _spin_rtt = mmp.spin_bit.rx_observe(sp_flag, fmp_counter, now_ms);
}
}
// Address rotation invalidates the per-peer connect()-ed UDP
@@ -450,7 +451,7 @@ impl Node {
/// black-hole the session.
#[cfg(unix)]
pub(in crate::node) fn register_decrypt_worker_session(&mut self, node_addr: &crate::NodeAddr) {
let Some(workers) = self.decrypt_workers.as_ref().cloned() else {
let Some(workers) = self.supervisor.decrypt_workers.as_ref().cloned() else {
return;
};
let (cache_key, state) = {
@@ -491,7 +492,7 @@ impl Node {
&mut self,
cache_key: (crate::transport::TransportId, u32),
) {
if let Some(workers) = self.decrypt_workers.as_ref() {
if let Some(workers) = self.supervisor.decrypt_workers.as_ref() {
workers.unregister_session(cache_key);
}
self.decrypt_registered_sessions.remove(&cache_key);
@@ -533,7 +534,7 @@ impl Node {
.duration_since(std::time::UNIX_EPOCH)
.map(|d| d.as_millis() as u64)
.unwrap_or(0);
self.schedule_reconnect(addr, now_ms);
self.note_link_dead(addr, now_ms);
}
}
}
src/node/handlers/forwarding.rs → src/node/dataplane/forwarding.rs
+215 -150
View File
@@ -1,21 +1,22 @@
//! SessionDatagram forwarding handler.
//!
//! Handles incoming SessionDatagram (0x00) link messages: decodes the
//! envelope, enforces hop limits, performs coordinate cache warming from
//! plaintext session-layer headers, routes to the next hop or delivers
//! locally, and generates error signals on routing failure.
//! envelope, performs coordinate cache warming from plaintext session-layer
//! headers, pre-resolves the next hop for forwardable transit datagrams, and
//! drives the routing core's outcome — local delivery when the datagram is
//! addressed to this node, the transit hop-limit drop, the forward, or the
//! error signal generated on routing failure.
use crate::NodeAddr;
use crate::node::reject::ForwardingReject;
use crate::node::session_wire::{
use crate::node::{Node, NodeError, NodeRoutingView};
use crate::proto::fsp::wire::{
FSP_COMMON_PREFIX_SIZE, FSP_HEADER_SIZE, FSP_PHASE_ESTABLISHED, FSP_PHASE_MSG1, FSP_PHASE_MSG2,
FspCommonPrefix, parse_encrypted_coords,
};
use crate::node::{Node, NodeError};
use crate::protocol::{
CoordsRequired, MtuExceeded, PathBroken, SessionAck, SessionDatagram, SessionDatagramRef,
SessionSetup,
};
use crate::proto::fsp::{SessionAck, SessionSetup};
use crate::proto::link::{SessionDatagram, SessionDatagramRef};
use crate::proto::routing::{DropReason, NextHop, RouteAction, RouteOutcome};
use std::time::{Duration, Instant};
use tracing::{debug, warn};
@@ -43,123 +44,169 @@ impl Node {
}
};
// TTL enforcement: decrement for forwarding and drop only if the
// received datagram was already exhausted.
if datagram_ref.ttl == 0 {
self.metrics()
.forwarding
.record_reject_bytes(ForwardingReject::TtlExhausted, payload.len());
debug!(
src = %datagram_ref.src_addr,
dest = %datagram_ref.dest_addr,
"SessionDatagram TTL exhausted, dropping"
);
return;
}
let forwarded_ttl = datagram_ref.ttl - 1;
let my_addr = *self.node_addr();
// Coordinate cache warming from plaintext session-layer headers
// Coordinate cache warming from plaintext session-layer headers. Runs
// ahead of both the delivery and the TTL decisions the core makes: the
// coords a peer put on the wire are equally valid whichever way those
// go, and the only arrivals this newly warms from are those with an
// exhausted TTL, whose every insert is already achievable at TTL 1.
self.try_warm_coord_cache_ref(&datagram_ref);
// Local delivery: dispatch to session layer handlers without
// materializing an owned SessionDatagram payload Vec.
if datagram_ref.dest_addr == *self.node_addr() {
self.metrics().forwarding.record_delivered(payload.len());
self.handle_session_payload(
&datagram_ref.src_addr,
datagram_ref.payload,
datagram_ref.path_mtu,
incoming_ce,
)
.await;
return;
}
// Pre-resolve the next hop only for datagrams the core can actually
// forward: not locally destined, and carrying a TTL that survives the
// decrement (`ttl > 1` — the shell-side mirror of the core's
// would-leave-zero drop). This keeps `find_next_hop`'s coord-cache
// LRU-touch side effect scoped to genuine forwards, as it was when the
// TTL test ran inline ahead of it. Warming above has already run, so
// the resolution observes freshly cached coords.
let next_hop = if datagram_ref.dest_addr != my_addr && datagram_ref.ttl > 1 {
self.resolve_next_hop(&datagram_ref.dest_addr)
} else {
None
};
let mut datagram = datagram_ref.into_owned();
datagram.ttl = forwarded_ttl;
// Read local congestion once and reuse it for both the CE decision
// (via the view) and the congestion metric/log below, keeping
// `detect_congestion` the single source of truth.
let congested = next_hop
.as_ref()
.map(|nh| self.detect_congestion(&nh.addr))
.unwrap_or(false);
// Find next hop toward destination
let next_hop_addr = match self.find_next_hop(&datagram.dest_addr) {
Some(peer) => *peer.node_addr(),
None => {
// Borrow the routing tables disjointly from `&mut self.routing` for
// the pure decision, then release both before driving the outcome.
let outcome = {
let view = NodeRoutingView {
coord_cache: &self.coord_cache,
peers: &self.peers,
tree_state: &self.tree_state,
congested,
};
self.routing
.route(&datagram_ref, &my_addr, incoming_ce, next_hop, &view)
};
match outcome {
RouteOutcome::Drop {
reason: DropReason::TtlExhausted,
} => {
self.metrics()
.forwarding
.record_reject_bytes(ForwardingReject::TtlExhausted, payload.len());
debug!(
src = %datagram_ref.src_addr,
dest = %datagram_ref.dest_addr,
ttl = datagram_ref.ttl,
"SessionDatagram TTL exhausted, dropping"
);
}
RouteOutcome::DeliverLocal => {
// Local delivery: dispatch to session layer handlers without
// materializing an owned SessionDatagram payload Vec.
self.metrics().forwarding.record_delivered(payload.len());
self.handle_session_payload(
&datagram_ref.src_addr,
datagram_ref.payload,
datagram_ref.path_mtu,
incoming_ce,
)
.await;
}
RouteOutcome::NoRoute => {
self.metrics()
.forwarding
.record_reject_bytes(ForwardingReject::NoRoute, payload.len());
let original = datagram_ref.into_owned();
debug!(
src = %self.peer_display_name(&datagram.src_addr),
dest = %self.peer_display_name(&datagram.dest_addr),
src = %self.peer_display_name(&original.src_addr),
dest = %self.peer_display_name(&original.dest_addr),
bytes = payload.len(),
"Dropping transit SessionDatagram: no route to destination"
);
self.send_routing_error(&datagram).await;
return;
self.send_routing_error(&original).await;
}
};
RouteOutcome::Forward {
next_hop,
bytes,
outgoing_ce,
} => {
let dest = datagram_ref.dest_addr;
// Apply path_mtu min() from the outgoing link's transport MTU
if let Some(peer) = self.peers.get(&next_hop_addr)
// ECN CE relay: congestion was detected locally above; emit the
// metric and rate-limited log at the transit chokepoint.
if congested {
self.metrics().congestion.congestion_detected.inc();
let now = Instant::now();
let should_log = self
.last_congestion_log
.map(|t| now.duration_since(t) >= Duration::from_secs(5))
.unwrap_or(true);
if should_log {
self.last_congestion_log = Some(now);
debug!(next_hop = %next_hop, "Congestion detected, CE flag set on forwarded packet");
}
}
match self
.send_encrypted_link_message_with_ce(&next_hop, &bytes, outgoing_ce)
.await
{
Err(NodeError::MtuExceeded { mtu, .. }) => {
self.metrics()
.forwarding
.record_reject_bytes(ForwardingReject::MtuExceeded, payload.len());
self.send_mtu_exceeded_error(dest, datagram_ref.src_addr, mtu)
.await;
}
Err(e) => {
self.metrics()
.forwarding
.record_reject_bytes(ForwardingReject::SendError, payload.len());
debug!(
next_hop = %next_hop,
dest = %dest,
error = %e,
"Failed to forward SessionDatagram"
);
}
Ok(()) => {
self.metrics().forwarding.record_forwarded(bytes.len());
// Classify this transit forward by route class (partition
// of forwarded_packets). Done here, at the data-plane
// chokepoint, so the error-signal routing callers of
// find_next_hop are excluded.
let class = self.classify_forward(&dest, &next_hop);
self.metrics().forwarding.record_route_class(class);
if outgoing_ce {
self.metrics().congestion.ce_forwarded.inc();
}
}
}
}
}
}
/// Resolve the next hop toward `dest` into its address plus the outgoing
/// link's transport MTU. Returns `None` when there is no route.
///
/// The MTU defaults to `u16::MAX` (a no-op min-fold) when the peer's
/// transport is not resolvable, matching the pre-refactor inline behavior
/// where the MTU `if let` chain simply did not fire.
fn resolve_next_hop(&mut self, dest: &NodeAddr) -> Option<NextHop> {
let addr = *self.find_next_hop(dest)?.node_addr();
let link_mtu = if let Some(peer) = self.peers.get(&addr)
&& let Some(tid) = peer.transport_id()
&& let Some(transport) = self.transports.get(&tid)
{
if let Some(addr) = peer.current_addr() {
datagram.path_mtu = datagram.path_mtu.min(transport.link_mtu(addr));
} else {
datagram.path_mtu = datagram.path_mtu.min(transport.mtu());
}
}
// ECN CE relay: propagate incoming CE and detect local congestion
let local_congestion = self.detect_congestion(&next_hop_addr);
let outgoing_ce = incoming_ce || local_congestion;
if local_congestion {
self.metrics().congestion.congestion_detected.inc();
let now = Instant::now();
let should_log = self
.last_congestion_log
.map(|t| now.duration_since(t) >= Duration::from_secs(5))
.unwrap_or(true);
if should_log {
self.last_congestion_log = Some(now);
debug!(next_hop = %next_hop_addr, "Congestion detected, CE flag set on forwarded packet");
}
}
// Forward: re-encode (includes 0x00 type byte) and send
let encoded = datagram.encode();
if let Err(e) = self
.send_encrypted_link_message_with_ce(&next_hop_addr, &encoded, outgoing_ce)
.await
{
match e {
NodeError::MtuExceeded { mtu, .. } => {
self.metrics()
.forwarding
.record_reject_bytes(ForwardingReject::MtuExceeded, payload.len());
self.send_mtu_exceeded_error(&datagram, mtu).await;
}
_ => {
self.metrics()
.forwarding
.record_reject_bytes(ForwardingReject::SendError, payload.len());
debug!(
next_hop = %next_hop_addr,
dest = %datagram.dest_addr,
error = %e,
"Failed to forward SessionDatagram"
);
}
match peer.current_addr() {
Some(link_addr) => transport.link_mtu(link_addr),
None => transport.mtu(),
}
} else {
self.metrics().forwarding.record_forwarded(encoded.len());
// Classify this transit forward by route class (partition of
// forwarded_packets). Done here, at the data-plane chokepoint, so
// the error-signal routing callers of find_next_hop are excluded.
let class = self.classify_forward(&datagram.dest_addr, &next_hop_addr);
self.metrics().forwarding.record_route_class(class);
if outgoing_ce {
self.metrics().congestion.ce_forwarded.inc();
}
}
u16::MAX
};
Some(NextHop { addr, link_mtu })
}
/// Attempt to warm the coordinate cache from session-layer payload headers.
@@ -260,35 +307,41 @@ impl Node {
/// If we can't route the error back to the source either, drop silently.
/// No cascading errors.
async fn send_routing_error(&mut self, original: &SessionDatagram) {
// Rate limit: one error signal per destination per 100ms
if !self
.routing_error_rate_limiter
.should_send(&original.dest_addr)
{
return;
}
let my_addr = *self.node_addr();
let now_ms = std::time::SystemTime::now()
.duration_since(std::time::UNIX_EPOCH)
.map(|d| d.as_millis() as u64)
.unwrap_or(0);
let default_ttl = self.config().node.session.default_ttl;
let error_payload =
if let Some(coords) = self.coord_cache().get(&original.dest_addr, now_ms) {
let coords = coords.clone();
PathBroken::new(original.dest_addr, my_addr)
.with_last_coords(coords)
.encode()
} else {
CoordsRequired::new(original.dest_addr, my_addr).encode()
// Pure decision: rate-limit gate + PathBroken/CoordsRequired choice +
// error-PDU encode. Borrow the routing tables disjointly from
// `&mut self.routing`, then release them before the reverse-hop lookup.
let action = {
let view = NodeRoutingView {
coord_cache: &self.coord_cache,
peers: &self.peers,
tree_state: &self.tree_state,
congested: false,
};
self.routing.synth_routing_error(
&original.dest_addr,
&original.src_addr,
&my_addr,
&view,
now_ms,
default_ttl,
)
};
let RouteAction::SendError { toward, bytes } = match action {
Some(action) => action,
// Rate limited: drop silently. No cascading errors.
None => return,
};
let error_dg = SessionDatagram::new(my_addr, original.src_addr, error_payload)
.with_ttl(self.config().node.session.default_ttl);
let next_hop_addr = match self.find_next_hop(&original.src_addr) {
// Resolve the reverse link hop only now, after the gate passed, so
// `find_next_hop`'s coord-cache touch keeps its pre-refactor scope.
let next_hop_addr = match self.find_next_hop(&toward) {
Some(peer) => *peer.node_addr(),
None => {
debug!(
@@ -300,9 +353,8 @@ impl Node {
}
};
let encoded = error_dg.encode();
if let Err(e) = self
.send_encrypted_link_message(&next_hop_addr, &encoded)
.send_encrypted_link_message(&next_hop_addr, &bytes)
.await
{
debug!(
@@ -324,37 +376,50 @@ impl Node {
/// Called when `send_encrypted_link_message()` fails with
/// `NodeError::MtuExceeded` during forwarding. The signal tells the
/// source the bottleneck MTU so it can immediately reduce its path MTU.
async fn send_mtu_exceeded_error(&mut self, original: &SessionDatagram, bottleneck_mtu: u16) {
// Rate limit: reuse routing_error_rate_limiter keyed on dest_addr
if !self
.routing_error_rate_limiter
.should_send(&original.dest_addr)
{
return;
}
///
/// `dest` is the failed datagram's destination (rate-limit key); `toward`
/// is its source, where the signal is routed back.
async fn send_mtu_exceeded_error(
&mut self,
dest: NodeAddr,
toward: NodeAddr,
bottleneck_mtu: u16,
) {
let my_addr = *self.node_addr();
let now_ms = Self::now_ms();
let default_ttl = self.config().node.session.default_ttl;
let error_payload = MtuExceeded::new(original.dest_addr, my_addr, bottleneck_mtu).encode();
// Pure decision: rate-limit gate + MtuExceeded PDU + encode.
let action = self.routing.synth_mtu_exceeded(
&dest,
&toward,
&my_addr,
bottleneck_mtu,
now_ms,
default_ttl,
);
let RouteAction::SendError { toward, bytes } = match action {
Some(action) => action,
// Rate limited: drop silently. No cascading errors.
None => return,
};
let error_dg = SessionDatagram::new(my_addr, original.src_addr, error_payload)
.with_ttl(self.config().node.session.default_ttl);
let next_hop_addr = match self.find_next_hop(&original.src_addr) {
// Resolve the reverse link hop only now, after the gate passed, so
// `find_next_hop`'s coord-cache touch keeps its pre-refactor scope.
let next_hop_addr = match self.find_next_hop(&toward) {
Some(peer) => *peer.node_addr(),
None => {
debug!(
src = %original.src_addr,
dest = %original.dest_addr,
src = %toward,
dest = %dest,
"Cannot route MtuExceeded signal back to source, dropping"
);
return;
}
};
let encoded = error_dg.encode();
if let Err(e) = self
.send_encrypted_link_message(&next_hop_addr, &encoded)
.send_encrypted_link_message(&next_hop_addr, &bytes)
.await
{
debug!(
@@ -364,8 +429,8 @@ impl Node {
);
} else {
debug!(
original_dest = %original.dest_addr,
error_dest = %original.src_addr,
original_dest = %dest,
error_dest = %toward,
bottleneck_mtu,
"Sent MtuExceeded error signal"
);
src/node/dataplane/mod.rs
+18
View File
@@ -0,0 +1,18 @@
//! Data plane: the RX `select!` loop and the per-packet forwarding path.
//!
//! Holds the whole hot path in one home: the `select!` run loop
//! (`rx_loop`), transit/local datagram forwarding (`forwarding`), the
//! link-message router (`dispatch`), the RX decrypt path including responder
//! K-bit cutover and address-roam writes (`encrypted`), and the per-peer
//! connected-UDP fast-path socket activation (`connected_udp`). Each module
//! contributes `impl Node` methods driven by the run loop.
#[cfg(unix)]
pub(crate) mod connected_udp;
mod dispatch;
mod encrypted;
mod forwarding;
mod peer_actions;
mod rx_loop;
pub(in crate::node) use peer_actions::PeerActionCtx;
src/node/dataplane/peer_actions.rs
+557
View File
@@ -0,0 +1,557 @@
//! Executor for the per-peer control machine's [`PeerAction`]s.
//!
//! The per-peer FSM in [`crate::peer::machine`] is a sans-IO reducer: it decides
//! *what* must happen and returns a `Vec<PeerAction>`; this module is the *doing*
//! half — the thin driver that maps each action onto the exact shell call it
//! stands for (`build_msg2` + `transport.send`, `promote_connection`,
//! `remove_active_peer`, `index_allocator.free`, `note_link_dead`, …).
//!
//! ## Progressive cutover
//!
//! The executor is wired incrementally. Live today: the inbound establish
//! (`handle_msg1` → `step(InboundMsg1)`), the outbound msg2 promote
//! (`handle_msg2` looks up the dial-persisted machine), the connectionless
//! outbound msg1 send (`SendHandshake` with `their_index == None` →
//! `send_stored_msg1`, driven from `initiate_connection`), the
//! connection-oriented dial (`OpenTransport` performs the non-blocking
//! `transport.connect`; `TransportConnected` drives the connect-resolution msg1
//! send from `poll_pending_connects`), the rekey cadence (`check_rekey` →
//! `route_rekey_cadence` → `RekeyConsume`, driving the `SwapSendState` and
//! `CompleteDrain` arms), and the liveness reap (`route_link_dead` →
//! `LinkDeadSuspected`, driving `InvalidateSendState` → `remove_active_peer`).
//!
//! The genuine inert stubs remaining are `SendRekey`, `SendLinkMessage`, and
//! the connected-UDP arms. `RegisterDecryptSession` is a deliberate no-op —
//! see its arm for the note.
//!
//! The timer arms (`SetTimer`/`CancelTimer`) populate/clear the per-peer timer
//! store (`peer_timers`). The `HandshakeRetransmit` and `HandshakeTimeout`
//! deadlines are read and fired by `drive_peer_timers` (the handshake resend +
//! reap home). The rekey/liveness kinds are still SHADOW — driven by their own
//! shell drivers — so populating them stays behavior-neutral.
use crate::PeerIdentity;
use crate::node::Node;
use crate::node::reject::{HandshakeReject, RejectReason};
use crate::peer::machine::{LostKind, PeerAction, PeerEvent};
use crate::proto::fmp::PromotionResult;
use crate::proto::fmp::wire::build_msg2;
use crate::transport::{LinkId, TransportAddr, TransportId};
use crate::utils::index::SessionIndex;
use std::collections::VecDeque;
use tracing::{debug, trace, warn};
/// Ambient shell facts a [`PeerAction`] executor needs that the machine's
/// runtime-agnostic action payloads deliberately omit (verified identity,
/// transport target, the msg2 framing indices, the promotion timestamp).
///
/// Unlike a machine event/action payload this is **executor-side**, so it may
/// hold real values resolved from the wire context (cf. `handle_msg1`'s
/// `wire`/`packet` locals and `promote_connection`'s ambient args). It is
/// built fresh per driven step by the caller at cutover time.
#[allow(dead_code)]
pub(in crate::node) struct PeerActionCtx {
/// The authenticated peer identity: `PromoteToActive` /
/// `InvalidateSendState` resolve their `NodeAddr` from this.
pub(in crate::node) verified_identity: PeerIdentity,
/// The transport the exchange is happening over (msg2 send target, decrypt
/// cache-key transport half).
pub(in crate::node) transport_id: TransportId,
/// The peer's wire address (msg2 send target).
pub(in crate::node) remote_addr: TransportAddr,
/// Our session index for this exchange (msg2 framing sender_idx).
pub(in crate::node) our_index: Option<SessionIndex>,
/// The peer's session index for this exchange (msg2 framing
/// receiver_idx).
pub(in crate::node) their_index: Option<SessionIndex>,
/// The wire timestamp driving this step (promotion ts / loss-report clock).
pub(in crate::node) now_ms: u64,
/// Establish direction for this exchange. Discriminates the
/// `PromoteToActive` failure cleanup: the pre-refactor inbound
/// (`handle_msg1`) and outbound (`handle_msg2`) promote-Err arms were NOT
/// byte-identical, so the executor must reproduce each. `false` = inbound
/// (drop link + reverse map + free index), `true` = outbound (record the
/// reject only; leave the dead link/`addr_to_link` for the stale-connection
/// reaper, matching old `handle_msg2`).
pub(in crate::node) is_outbound: bool,
}
impl Node {
/// Advance the machine for `link` by one event and execute the resulting
/// actions.
///
/// The borrow structure the whole seam turns on: the machine
/// needs `&mut IndexAllocator` as a synchronous capability *while it is
/// itself borrowed mutably out of `peer_machines`*. `peer_machines` and
/// `index_allocator` are **distinct `Node` fields**, so the collect below is
/// a disjoint two-field borrow the checker accepts; once the actions are
/// collected both borrows drop and the executor runs against `&mut self`.
pub(in crate::node) async fn advance_peer_machine(
&mut self,
link: LinkId,
event: PeerEvent,
now: u64,
ambient: &PeerActionCtx,
) {
let actions = match self.peer_machines.get_mut(&link) {
// Disjoint field borrow: `self.peer_machines` (the map entry) and
// `self.index_allocator` (the capability) are separate fields.
Some(machine) => machine.step(event, now, &mut self.index_allocator),
None => return,
};
self.execute_peer_actions(link, ambient, actions).await;
}
/// Map each [`PeerAction`] onto its shell call.
///
/// `PromoteToActive` feeds its [`PromotionResult`](crate::proto::fmp::PromotionResult)
/// back into the machine and appends the follow-up actions to the same
/// worklist — a queue rather than self-recursion so the async executor stays a
/// single flat future (no boxing) and the emitted order is preserved (the
/// establish sequences always end in `PromoteToActive`, so its follow-ups run
/// after any siblings).
pub(in crate::node) async fn execute_peer_actions(
&mut self,
link: LinkId,
ambient: &PeerActionCtx,
actions: Vec<PeerAction>,
) {
let mut queue: VecDeque<PeerAction> = actions.into();
while let Some(action) = queue.pop_front() {
match action {
PeerAction::OpenTransport {
transport_id,
remote_addr,
} => {
// Outbound connection-oriented dial. `initiate_connection`'s
// oriented branch drove the machine to `Connecting`, which
// emitted this action. Perform the non-blocking
// `transport.connect` and, on success, push the
// `PendingConnect` for `poll_pending_connects` to resolve. On
// connect error, tear down the dial-window state (link,
// reverse map, control machine) and abort the queue — the
// executor-local mirror of the old inline
// `initiate_connection` connect+push.
if let Some(transport) = self.transports.get(&transport_id) {
match transport.connect(&remote_addr).await {
Ok(()) => {
debug!(
transport_id = %transport_id,
remote_addr = %remote_addr,
link_id = %link,
"Transport connect initiated (non-blocking)"
);
self.peering
.pending_connects
.push(crate::node::PendingConnect {
link_id: link,
transport_id,
remote_addr,
peer_identity: ambient.verified_identity,
});
}
Err(_e) => {
self.links.remove(&link);
self.addr_to_link.remove(&(transport_id, remote_addr));
self.remove_peer_machine(link);
return;
}
}
}
}
PeerAction::SendHandshake { bytes } => {
// Two outbound directions share this action, discriminated by
// `their_index`:
// msg2 (`their_index == Some`): the machine payload is the
// UNFRAMED Noise msg2; frame it with our/their index
// (`build_msg2`) and send.
// msg1 (`their_index == None`): a fresh outbound handshake;
// the machine's empty payload is ignored — the shell already
// allocated the index, ran the Noise leaf, and armed the
// wire at dial (`prepare_outbound_msg1`); this just sends the
// stored wire (see `send_stored_msg1`).
if let (Some(sender_idx), Some(receiver_idx)) =
(ambient.our_index, ambient.their_index)
{
let frame = build_msg2(sender_idx, receiver_idx, &bytes);
// Surface the send Result. A missing transport skips
// the send and continues (mirrors `handle_msg1`'s
// `if let Some(transport)` guard); a send *error* runs the
// pre-refactor msg2-send-failure cleanup (`handle_msg1`
// L494-503) and ABORTS the remaining queue so the queued
// `PromoteToActive` never runs.
let send_err = match self.transports.get(&ambient.transport_id) {
Some(transport) => {
transport.send(&ambient.remote_addr, &frame).await.err()
}
None => None,
};
if let Some(e) = send_err {
// Restored pre-refactor msg2-send-failure warn!
// (`handle_msg1` L665): the send error text is surfaced
// at the executor point where the failure is now handled.
warn!(link_id = %link, error = %e, "Failed to send msg2");
self.links.remove(&link);
self.addr_to_link
.remove(&(ambient.transport_id, ambient.remote_addr.clone()));
if let Some(idx) = ambient.our_index {
let _ = self.index_allocator.free(idx);
}
self.remove_peer_machine(link);
self.stats_mut()
.record_reject(RejectReason::Handshake(HandshakeReject::BadState));
return;
}
} else {
// msg1: the shell already allocated the index, ran the
// Noise leaf, and armed the wire on the connection at dial
// (`prepare_outbound_msg1`); send the stored wire. The
// machine's empty payload is ignored.
let _ = bytes;
self.send_stored_msg1(
link,
ambient.transport_id,
&ambient.remote_addr,
ambient.now_ms,
)
.await;
}
}
PeerAction::SendRekey { .. } => {
// Rekey msg2 framing (`build_msg2(our_new_index, …)`,
// `handshake.rs:365`) + send. Rekey fold is not yet wired.
}
PeerAction::SendLinkMessage { .. } => {
// Encrypt + send a link-control frame (heartbeat / filter
// / tree / disconnect). Data-plane-owned; not yet wired.
}
PeerAction::PromoteToActive { link: promote_link } => {
// Ambient supplies the verified identity + promotion ts
// that `promote_connection` needs (resolved from the wire ctx).
match self.promote_connection(
promote_link,
ambient.verified_identity,
ambient.now_ms,
) {
Ok(result) => {
// The decrypt-worker registration relocated
// OUT of `promote_connection` into THIS single executor
// arm — the one live caller of `promote_connection` (both
// the inbound `handle_msg1` and outbound `handle_msg2`
// net-new establish paths reach it here). Register iff the
// promotion actually created or replaced a peer
// (`Promoted | CrossConnectionWon`), NEVER on
// `CrossConnectionLost`. Run synchronously right after
// `promote_connection` returns, before feeding
// `PromotionResolved` and before any await — the exact
// synchronous point (and Promoted/Won gating) of the
// pre-refactor in-`promote_connection` call. No-op when
// the worker pool isn't spawned (`register_...` early-
// returns), so the direct `promote_connection` test
// callers (which bypass this executor) are unaffected.
#[cfg(unix)]
match result {
PromotionResult::Promoted(node_addr)
| PromotionResult::CrossConnectionWon { node_addr, .. } => {
self.register_decrypt_worker_session(&node_addr);
}
PromotionResult::CrossConnectionLost { .. } => {}
}
// Feed the outcome back into the machine and fold the
// follow-up actions (RegisterDecryptSession — now a
// redundant no-op, see its arm — and the cross-conn index
// frees) into the worklist. Disjoint field borrow again.
let follow = match self.peer_machines.get_mut(&promote_link) {
Some(machine) => machine.step(
PeerEvent::PromotionResolved { result },
ambient.now_ms,
&mut self.index_allocator,
),
None => Vec::new(),
};
queue.extend(follow);
// Defensive cross-connection loser-link surgery.
// LINK-ONLY: close the losing transport connection, drop
// its link, and re-point `addr_to_link`, reproducing the
// pre-refactor inline `handle_msg2`/`handle_msg1` per-arm
// order EXACTLY. The index-plane frees/unregisters are
// owned by the machine's `PromotionResolved{Won/Lost}`
// follow-up (queued just above), so NOTHING here touches
// an index — no double-free.
//
// UNREACHABLE on every current driven path: the inbound
// and outbound net-new establish arms only route to the
// machine when no promoted peer exists for the node_addr
// (and `RestartThenPromote` removes the old peer first),
// so `promote_connection` always returns `Promoted`. The
// `debug_assert!(false, ..)` catches any future path that
// drives a cross-connection through the executor without
// the matching send-state handling.
match result {
PromotionResult::CrossConnectionWon { loser_link_id, .. } => {
debug_assert!(
false,
"executor CrossConnectionWon is unreachable on \
driven net-new establish paths"
);
// Close the losing transport connection (no-op for
// connectionless) via the LOSER link's own
// transport/addr, then drop the losing link.
if let Some(loser_link) = self.links.get(&loser_link_id) {
let loser_tid = loser_link.transport_id();
let loser_addr = loser_link.remote_addr().clone();
if let Some(transport) = self.transports.get(&loser_tid) {
transport.close_connection(&loser_addr).await;
}
}
self.remove_link(&loser_link_id);
// Point `addr_to_link` at the winning (current)
// link.
self.addr_to_link.insert(
(ambient.transport_id, ambient.remote_addr.clone()),
promote_link,
);
}
PromotionResult::CrossConnectionLost { winner_link_id } => {
debug_assert!(
false,
"executor CrossConnectionLost is unreachable on \
driven net-new establish paths"
);
// Close this (losing) connection, drop its link,
// and restore `addr_to_link` to the winner.
if let Some(transport) =
self.transports.get(&ambient.transport_id)
{
transport.close_connection(&ambient.remote_addr).await;
}
self.remove_link(&promote_link);
self.addr_to_link.insert(
(ambient.transport_id, ambient.remote_addr.clone()),
winner_link_id,
);
}
PromotionResult::Promoted(_) => {}
}
}
Err(e) => {
// Promotion failed. `promote_connection` already
// removed `connections[link]` and (on error) handled its
// own index internally. The pre-refactor inbound and
// outbound promote-Err arms were NOT byte-identical, so
// discriminate on `ambient.is_outbound`. The queue is
// drained (PromoteToActive is the last establish action),
// so no explicit abort.
if ambient.is_outbound {
// OLD outbound (`handle_msg2` promote-Err): warn +
// record_reject ONLY. NO `remove_link`, NO
// `index_allocator.free`, NO `addr_to_link` removal —
// the dead link/addr_to_link/pending_outbound were
// left for the 30s stale-connection reaper
// (`promote_connection` already handled
// `connections[link]`/its index on error). Restored
// pre-refactor outbound warn! ("Failed to promote
// connection").
//
// The outbound machine was persisted at dial; it is
// additive state that did not exist pre-refactor, so
// removing it on promote failure is neutral vs old and
// prevents a leak.
warn!(
target: "fips::node::handlers::handshake",
link_id = %promote_link,
error = %e,
"Failed to promote connection"
);
self.stats_mut().record_reject(RejectReason::Handshake(
HandshakeReject::BadState,
));
self.remove_peer_machine(promote_link);
} else {
// OLD inbound (`handle_msg1` L587-591): drop the link
// + reverse map, free our index, discard the machine,
// and record the reject. Restored pre-refactor inbound
// promote-failure warn! (`handle_msg1` L757).
warn!(
target: "fips::node::handlers::handshake",
link_id = %promote_link,
error = %e,
"Failed to promote inbound connection"
);
self.remove_link(&promote_link);
if let Some(idx) = ambient.our_index {
let _ = self.index_allocator.free(idx);
}
self.remove_peer_machine(promote_link);
self.stats_mut().record_reject(RejectReason::Handshake(
HandshakeReject::BadState,
));
}
}
}
}
PeerAction::ResolveCrossConnection { .. } => {
// A decision token, not an effect: the outbound msg2
// handler intercepts it and runs the inline swap/keep
// resolution itself, so it must never reach the executor.
debug_assert!(
false,
"ResolveCrossConnection is intercepted by the msg2 \
handler and must never reach the executor"
);
}
PeerAction::SwapSendState { .. } => {
// Initiator cutover: the live authoritative rekey-cadence
// path, routed here from `check_rekey` via
// `route_rekey_cadence` → `PeerEvent::RekeyConsume`; the
// inline body survives only as `cutover_peer_inline`, a
// debug-assert release fallback. `addr` is resolved
// from the ambient verified identity (as `InvalidateSendState`
// does). The decrypt re-register folds HERE, gated on
// `did_cutover` — the generic `RegisterDecryptSession` arm stays a
// no-op so a promote never double-registers.
let node_addr = *ambient.verified_identity.node_addr();
let did_cutover = if let Some(peer) = self.peers.get_mut(&node_addr) {
if let Some(_old_our_index) = peer.cutover_to_new_session() {
// New index was pre-registered in peers_by_index
// during msg2 handling (handshake.rs).
debug_assert!(
peer.transport_id().is_some()
&& peer.our_index().is_some()
&& self.peers_by_index.contains_key(&(
peer.transport_id().unwrap(),
peer.our_index().unwrap().as_u32()
)),
"peers_by_index should contain pre-registered new index after cutover"
);
debug!(
// Pin the target to the pre-refactor module: this
// cutover log relocated from handlers/rekey.rs into
// the executor, but operators (and the test harness)
// filter it under fips::node::handlers::rekey. Keeping
// the target preserves the observable log contract.
target: "fips::node::handlers::rekey",
peer = %self.peer_display_name(&node_addr),
"Rekey cutover complete (initiator), K-bit flipped"
);
true
} else {
false
}
} else {
false
};
// Re-register the new session with the decrypt worker — the
// cache_key (transport_id, our_index) just changed, so the
// old worker entry is stale and every packet on the new
// session would miss the worker's HashMap lookup.
#[cfg(unix)]
if did_cutover {
self.register_decrypt_worker_session(&node_addr);
}
#[cfg(not(unix))]
let _ = did_cutover;
}
PeerAction::CompleteDrain { peer: node_addr } => {
// Initiator drain completion: the live authoritative
// rekey-cadence path, routed here from `check_rekey` via
// `route_rekey_cadence` → `PeerEvent::RekeyConsume`; the
// inline body survives only as `drain_peer_inline`, a
// debug-assert release fallback. Extract the real previous
// index + transport_id under the peer borrow, drop the
// borrow, then run the cache_key cleanup (which takes
// &mut self for unregister_decrypt_worker_session).
let drained = self.peers.get_mut(&node_addr).and_then(|peer| {
peer.complete_drain().map(|idx| (idx, peer.transport_id()))
});
if let Some((old_our_index, transport_id)) = drained {
if let Some(tid) = transport_id {
let cache_key = (tid, old_our_index.as_u32());
self.peers_by_index.remove(&cache_key);
#[cfg(unix)]
self.unregister_decrypt_worker_session(cache_key);
}
let _ = self.index_allocator.free(old_our_index);
trace!(
// Pin to the pre-refactor module (see the cutover log
// above) so the relocated drain log stays visible under
// the operator's fips::node::handlers::rekey filter.
target: "fips::node::handlers::rekey",
peer = %self.peer_display_name(&node_addr),
old_index = %old_our_index,
"Drain complete, previous session erased"
);
}
}
PeerAction::InvalidateSendState => {
// The FULL teardown. `remove_active_peer`
// (`dispatch.rs:107`) frees the four index slots
// (current/rekey/pending/previous), drops `peers_by_index`,
// unregisters the decrypt worker, removes the FSP `sessions`
// entry and `pending_tun_packets`. The machine emits NO
// `FreeIndex` for those slots, so there is no double-free.
self.remove_active_peer(ambient.verified_identity.node_addr());
}
PeerAction::RegisterDecryptSession { index } => {
let _ = index;
// No-op by design. The decrypt-worker
// registration relocated into the `PromoteToActive` Ok arm above, gated on
// the returned `PromotionResult`, so it runs once per live
// promote (Promoted/Won) at the pre-refactor synchronous point.
// This machine-emitted action is now redundant with that arm;
// kept as an inert no-op (rather than removing the emission) so
// the machine's action sequence and its unit tests stay
// unchanged. The keyed-by-NodeAddr register does not need the
// machine's `index` payload.
}
PeerAction::UnregisterDecryptSession { index } => {
// Executor supplies `transport_id` from ambient; keyed by
// (tid, index) like `remove_active_peer` / the rekey drain path.
#[cfg(unix)]
self.unregister_decrypt_worker_session((ambient.transport_id, index.as_u32()));
#[cfg(not(unix))]
let _ = index;
}
PeerAction::FreeIndex { index } => {
let _ = self.index_allocator.free(index);
}
PeerAction::ActivateConnectedUdp | PeerAction::TeardownConnectedUdp => {
// Connected-UDP plane ownership (`connected_udp.rs`).
}
PeerAction::SetTimer { kind, at_ms } => {
// Populate the per-peer timer store (overwrite = reschedule).
// The `HandshakeRetransmit` and `HandshakeTimeout` deadlines
// are read + fired by `drive_peer_timers`. Rekey/liveness kinds
// are still SHADOW here — they keep their own shell drivers —
// so populating them stays behavior-neutral.
self.peer_timers
.entry(link)
.or_default()
.insert(kind, at_ms);
}
PeerAction::CancelTimer { kind } => {
if let Some(timers) = self.peer_timers.get_mut(&link) {
timers.remove(&kind);
}
}
PeerAction::ReportLost { peer, kind } => {
// The single loss token, routed to the reconciler reflex the
// `kind` names: an un-promoted handshake attempt takes the
// connected-guarded `note_handshake_timeout` (`driver.rs:28`),
// an established peer's link-death takes the unconditional
// `note_link_dead` (`driver.rs:48`).
match kind {
LostKind::HandshakeTimeout => {
self.note_handshake_timeout(peer, ambient.now_ms);
}
LostKind::LinkDead => {
self.note_link_dead(peer, ambient.now_ms);
}
}
}
}
}
}
}
src/node/handlers/rx_loop.rs → src/node/dataplane/rx_loop.rs
+192 -45
View File
@@ -1,10 +1,10 @@
//! RX event loop and packet dispatch.
use crate::control::{ControlSocket, commands};
use crate::node::wire::{
use crate::node::{Node, NodeError};
use crate::proto::fmp::wire::{
COMMON_PREFIX_SIZE, CommonPrefix, FMP_VERSION, PHASE_ESTABLISHED, PHASE_MSG1, PHASE_MSG2,
};
use crate::node::{Node, NodeError};
use crate::transport::ReceivedPacket;
use std::time::Duration;
use tracing::{debug, info, warn};
@@ -42,12 +42,42 @@ impl Node {
/// This method takes ownership of the packet_rx channel and runs
/// until the channel is closed (typically when stop() is called).
pub async fn run_rx_loop(&mut self) -> Result<(), NodeError> {
// No shutdown observer → today's infinite loop, byte-identical. All
// existing callers/tests use this; `pending()` never fires, so the
// shutdown/deadline arms below stay permanently disabled.
self.run_rx_loop_with_shutdown(std::future::pending()).await
}
/// The rx event loop, which serves until `shutdown` fires and then drains
/// **in place** before returning.
///
/// The channel receivers are moved into this frame's locals and live across
/// both serve and drain, so — unlike a `select!`-cancelled loop — they are
/// never destructively dropped mid-flight; they are released only on clean
/// exit, after which teardown does not need them.
///
/// - While serving (`drain_deadline == None`) the loop is behaviorally
/// identical to before: the shutdown arm, the deadline arm, and the
/// peers-empty early-exit are all guarded off, so the hot per-packet path
/// and the `biased` order of the real arms are unchanged.
/// - When `shutdown` fires, the loop calls [`Node::enter_drain`] once
/// (broadcast Disconnect, gate the reconciler off) and arms the bounded
/// deadline, then keeps servicing inbound/tick/peer-removal until all
/// peers clear or the deadline elapses, then returns. The caller
/// ([`Node::finish_shutdown`]) closes the window and tears down.
pub async fn run_rx_loop_with_shutdown(
&mut self,
shutdown: impl std::future::Future<Output = ()>,
) -> Result<(), NodeError> {
tokio::pin!(shutdown);
// `None` = serving; `Some(deadline)` = draining (bounded window).
let mut drain_deadline: Option<tokio::time::Instant> = None;
let mut packet_rx = self.packet_rx.take().ok_or(NodeError::NotStarted)?;
// Take the TUN outbound receiver, or create a dummy channel that never
// produces messages (when TUN is disabled). Holding the sender prevents
// the channel from closing.
let (mut tun_outbound_rx, _tun_guard) = match self.tun_outbound_rx.take() {
let (mut tun_outbound_rx, _tun_guard) = match self.supervisor.tun_outbound_rx.take() {
Some(rx) => (rx, None),
None => {
let (tx, rx) = tokio::sync::mpsc::channel(1);
@@ -57,7 +87,7 @@ impl Node {
// Take the DNS identity receiver, or create a dummy channel (when DNS
// is disabled). Same pattern as TUN outbound.
let (mut dns_identity_rx, _dns_guard) = match self.dns_identity_rx.take() {
let (mut dns_identity_rx, _dns_guard) = match self.supervisor.dns_identity_rx.take() {
Some(rx) => (rx, None),
None => {
let (tx, rx) = tokio::sync::mpsc::channel(1);
@@ -65,8 +95,20 @@ impl Node {
}
};
let mut tick =
tokio::time::interval(Duration::from_secs(self.config().node.tick_interval_secs));
// Take the runtime child-liveness receiver, or a dummy channel (when the
// node was seeded straight into Running without a start()). Holding the
// dummy sender in the guard keeps the channel open. Same pattern as TUN
// outbound / DNS identity.
let (mut child_exit_rx, _child_exit_guard) = match self.child_exit_rx.take() {
Some(rx) => (rx, None),
None => {
let (tx, rx) = tokio::sync::mpsc::channel(1);
(rx, Some(tx))
}
};
let tick_period = Duration::from_secs(self.config().node.tick_interval_secs);
let mut tick = tokio::time::interval(tick_period);
// Set up control socket channel
let (control_tx, mut control_rx) =
@@ -122,6 +164,13 @@ impl Node {
crate::perf_profile::maybe_spawn_reporter();
loop {
// Bounded drain mode: break as soon as all peers have cleared. In
// normal mode (`None`) this short-circuits before touching
// `self.peers`, so the loop is byte-identical.
if drain_deadline.is_some() && self.peers.is_empty() {
info!("Drain complete: all peers cleared, ending drain loop");
break;
}
tokio::select! {
biased;
// Decrypt-worker fallback drains FIRST. Under sustained
@@ -214,6 +263,27 @@ impl Node {
}
}
}
// Runtime child-liveness. Placed AFTER `packet_rx` so the hot
// inbound path keeps its `biased` priority. A directly-observable
// child (TUN threads, DNS/mDNS/Nostr) exited on its own; feed the
// FSM, which republishes health (Degraded here — a Running node
// always has ≥1 transport up). `on_child_exited` only ever emits
// `PublishState`; other variants are ignored defensively.
maybe_child = child_exit_rx.recv() => {
if let Some(child) = maybe_child {
let actions = self
.supervisor
.fsm
.step(crate::node::lifecycle::supervisor::Event::ChildExited { child });
for action in actions {
if let crate::node::lifecycle::supervisor::Action::PublishState(ns) =
action
{
self.supervisor.state = ns;
}
}
}
}
Some(ipv6_packet) = tun_outbound_rx.recv() => {
self.handle_tun_outbound(ipv6_packet).await;
let mut drained = 0;
@@ -249,41 +319,114 @@ impl Node {
).await;
let _ = response_tx.send(response);
}
_ = tick.tick() => {
self.check_timeouts();
let now_ms = Self::now_ms();
self.reload_peer_acl().await;
// The host map hot-reloads on the same tick as the ACL. It
// is polled separately from `reload_peer_acl` because the
// ACL's embedded alias reloader and this snapshot are
// distinct resources; the `path_mtu_lookup` cache and the
// `nostr_discovery` subsystem are deliberately excluded
// from `Reloadable` since neither reloads from a backing
// file (see `node::reloadable`).
self.reload_host_map().await;
self.poll_pending_connects().await;
self.poll_nostr_discovery().await;
self.poll_lan_discovery().await;
self.resend_pending_handshakes(now_ms).await;
self.resend_pending_rekeys(now_ms).await;
self.resend_pending_session_handshakes(now_ms).await;
self.resend_pending_session_msg3(now_ms).await;
self.purge_idle_sessions(now_ms);
self.process_pending_retries(now_ms).await;
self.check_tree_state().await;
self.check_bloom_state().await;
self.compute_mesh_size();
self.record_stats_history();
self.check_mmp_reports().await;
self.check_session_mmp_reports().await;
self.check_link_heartbeats().await;
self.check_rekey().await;
self.check_session_rekey().await;
self.check_pending_lookups(now_ms).await;
self.poll_transport_discovery().await;
self.sample_transport_congestion();
#[cfg(any(target_os = "linux", target_os = "macos"))]
self.activate_connected_udp_sessions().await;
deadline = tick.tick() => {
// Tick-body instrumentation. The gate is read ONCE per tick
// into `instr_on`, which is then passed explicitly to every
// `instr_step!` invocation — macro hygiene makes a call-site
// local invisible inside the macro body. With the
// `profiling` feature off, `gate()` is a `const fn`
// returning false and the macro is a pure pass-through, so
// the whole arm compiles to the uninstrumented sequence.
//
// `tick_entry` records how late this entry is against the
// deadline the interval scheduled it for. That is the
// measurement this instrumentation exists for: the arm is
// polled LAST under `biased;`, so the
// lateness IS the time it spent waiting behind the packet,
// TUN and control arms. `tick()` hands back its scheduled
// deadline, so this is a subtraction rather than a model.
// The whole-tick span below measures the body alone.
let instr_on = crate::instr::gate();
crate::instr::tick_entry(instr_on, deadline.into_std(), std::time::Instant::now());
instr_step!(instr_on, crate::instr::Domain::Tick, crate::instr::Step::WholeTick, {
instr_step!(instr_on, crate::instr::Domain::Tick, crate::instr::Step::CheckTimeouts,
self.check_timeouts());
let now_ms = Self::now_ms();
instr_step!(instr_on, crate::instr::Domain::Tick, crate::instr::Step::ReloadPeerAcl,
self.reload_peer_acl().await);
// The host map hot-reloads on the same tick as the ACL. It
// is polled separately from `reload_peer_acl` because the
// ACL's embedded alias reloader and this snapshot are
// distinct resources; the `path_mtu_lookup` cache and the
// `nostr_rendezvous` subsystem are deliberately excluded
// from `Reloadable` since neither reloads from a backing
// file (see `node::reloadable`).
instr_step!(instr_on, crate::instr::Domain::Tick, crate::instr::Step::ReloadHostMap,
self.reload_host_map().await);
instr_step!(instr_on, crate::instr::Domain::Tick, crate::instr::Step::PollPendingConnects,
self.poll_pending_connects().await);
instr_step!(instr_on, crate::instr::Domain::Tick, crate::instr::Step::PollNostrRendezvous,
self.poll_nostr_rendezvous().await);
instr_step!(instr_on, crate::instr::Domain::Tick, crate::instr::Step::PollLanRendezvous,
self.poll_lan_rendezvous().await);
instr_step!(instr_on, crate::instr::Domain::Tick, crate::instr::Step::DrivePeerTimers,
self.drive_peer_timers(now_ms).await);
instr_step!(instr_on, crate::instr::Domain::Tick, crate::instr::Step::ResendPendingRekeys,
self.resend_pending_rekeys(now_ms).await);
instr_step!(instr_on, crate::instr::Domain::Tick, crate::instr::Step::ResendPendingSessionHandshakes,
self.resend_pending_session_handshakes(now_ms).await);
instr_step!(instr_on, crate::instr::Domain::Tick, crate::instr::Step::ResendPendingSessionMsg3,
self.resend_pending_session_msg3(now_ms).await);
instr_step!(instr_on, crate::instr::Domain::Tick, crate::instr::Step::PurgeIdleSessions,
self.purge_idle_sessions(now_ms));
instr_step!(instr_on, crate::instr::Domain::Tick, crate::instr::Step::ProcessPendingRetries,
self.process_pending_retries(now_ms).await);
instr_step!(instr_on, crate::instr::Domain::Tick, crate::instr::Step::CheckTreeState,
self.check_tree_state().await);
instr_step!(instr_on, crate::instr::Domain::Tick, crate::instr::Step::CheckBloomState,
self.check_bloom_state().await);
instr_step!(instr_on, crate::instr::Domain::Tick, crate::instr::Step::ComputeMeshSize,
self.compute_mesh_size());
instr_step!(instr_on, crate::instr::Domain::Tick, crate::instr::Step::RecordStatsHistory,
self.record_stats_history());
instr_step!(instr_on, crate::instr::Domain::Tick, crate::instr::Step::CheckMmpReports,
self.check_mmp_reports().await);
instr_step!(instr_on, crate::instr::Domain::Tick, crate::instr::Step::CheckSessionMmpReports,
self.check_session_mmp_reports().await);
instr_step!(instr_on, crate::instr::Domain::Tick, crate::instr::Step::CheckLinkHeartbeats,
self.check_link_heartbeats().await);
instr_step!(instr_on, crate::instr::Domain::Tick, crate::instr::Step::CheckRekey,
self.check_rekey().await);
instr_step!(instr_on, crate::instr::Domain::Tick, crate::instr::Step::CheckSessionRekey,
self.check_session_rekey().await);
instr_step!(instr_on, crate::instr::Domain::Tick, crate::instr::Step::CheckPendingLookups,
self.check_pending_lookups(now_ms).await);
instr_step!(instr_on, crate::instr::Domain::Tick, crate::instr::Step::PollTransportDiscovery,
self.poll_transport_discovery().await);
instr_step!(instr_on, crate::instr::Domain::Tick, crate::instr::Step::SampleTransportCongestion,
self.sample_transport_congestion());
#[cfg(any(target_os = "linux", target_os = "macos"))]
instr_step!(instr_on, crate::instr::Domain::Tick, crate::instr::Step::ActivateConnectedUdpSessions,
self.activate_connected_udp_sessions().await);
// Debug-build sweep of the peer-lifecycle map invariant
// (leaked machines / machine-less legs); two map scans,
// compiled out of release builds.
#[cfg(debug_assertions)]
instr_step!(instr_on, crate::instr::Domain::Tick, crate::instr::Step::DebugAssertPeerMapsCoherent,
self.debug_assert_peer_maps_coherent());
});
crate::instr::tick_gauges(instr_on, self.peers.len() as u64);
}
// Shutdown signal → enter the bounded drain in place, ONCE.
// Gated on `is_none()` so it only fires while serving; after
// entering drain the arm is disabled (the completed signal is
// never polled again) and the deadline arm below bounds the
// window. Placed after the real arms so their `biased` priority
// is unchanged, and inert while serving with `pending()`.
_ = &mut shutdown, if drain_deadline.is_none() => {
self.enter_drain().await;
drain_deadline =
Some(tokio::time::Instant::now() + self.config().node.drain_timeout());
}
// Bounded drain deadline (drain mode only). Placed LAST so the
// `biased` priority of the normal arms is unchanged, and gated
// on `is_some()` so in normal mode the branch is disabled — the
// future is created but never polled and never fires.
_ = tokio::time::sleep_until(
drain_deadline.unwrap_or_else(tokio::time::Instant::now)
), if drain_deadline.is_some() => {
info!("Drain deadline elapsed, ending drain loop");
break;
}
}
}
@@ -319,12 +462,16 @@ impl Node {
// though no msg1/msg2 exchange can ever succeed. Bump the
// discovery-layer cooldown to the long protocol-mismatch
// window and emit a single WARN per fresh observation.
if self.bootstrap_transports.contains(&packet.transport_id)
if self
.supervisor
.nostr_rendezvous
.is_bootstrap_transport(&packet.transport_id)
&& let Some(npub) = self
.bootstrap_transport_npubs
.get(&packet.transport_id)
.supervisor
.nostr_rendezvous
.bootstrap_transport_npub(&packet.transport_id)
.cloned()
&& let Some(handle) = self.nostr_discovery_handle()
&& let Some(handle) = self.nostr_rendezvous_handle()
{
let now_ms = Self::now_ms();
let cooldown_secs = handle.protocol_mismatch_cooldown_secs();
src/node/decrypt_worker.rs
+5 -5
View File
@@ -561,7 +561,7 @@ mod tests {
let open_cipher = LessSafeKey::new(unbound2);
let counter: u64 = 7;
const HDR: usize = crate::node::wire::ESTABLISHED_HEADER_SIZE;
const HDR: usize = crate::proto::fmp::wire::ESTABLISHED_HEADER_SIZE;
// Build a wire packet `[16-byte header][4-byte inner ts][1 byte link msg]`
// with capacity for the trailing AEAD tag. Header bytes
// double as AAD and as the on-wire prefix.
@@ -569,7 +569,7 @@ mod tests {
// Header: fill the flags byte (the second byte) with both
// FLAG_CE and FLAG_SP set; the rest is uninterpreted by the
// worker (it just AADs the whole 16 bytes).
let flags_byte = crate::node::wire::FLAG_CE | crate::node::wire::FLAG_SP;
let flags_byte = crate::proto::fmp::wire::FLAG_CE | crate::proto::fmp::wire::FLAG_SP;
let mut header = [0u8; HDR];
header[1] = flags_byte;
wire.extend_from_slice(&header);
@@ -626,11 +626,11 @@ mod tests {
"fmp_flags must round-trip from DecryptJob to DecryptFallback"
);
assert!(
fallback.fmp_flags & crate::node::wire::FLAG_CE != 0,
fallback.fmp_flags & crate::proto::fmp::wire::FLAG_CE != 0,
"FLAG_CE bit lost on worker path"
);
assert!(
fallback.fmp_flags & crate::node::wire::FLAG_SP != 0,
fallback.fmp_flags & crate::proto::fmp::wire::FLAG_SP != 0,
"FLAG_SP bit lost on worker path"
);
}
@@ -724,7 +724,7 @@ mod tests {
let open_cipher = LessSafeKey::new(unbound);
let counter: u64 = 11;
const HDR: usize = crate::node::wire::ESTABLISHED_HEADER_SIZE;
const HDR: usize = crate::proto::fmp::wire::ESTABLISHED_HEADER_SIZE;
let header = [0u8; HDR];
let mut wire = Vec::with_capacity(HDR + 4 + 1 + 16);
wire.extend_from_slice(&header);
src/node/discovery_rate_limit.rs
-376
View File
@@ -1,376 +0,0 @@
//! Discovery protocol rate limiting and backoff.
//!
//! Two complementary mechanisms:
//!
//! - **`DiscoveryBackoff`** (originator-side, optional): Exponential
//! suppression of fresh lookups after the per-attempt sequence in
//! `node.discovery.attempt_timeouts_secs` has been exhausted.
//! **Disabled by default** (base/cap = 0); the per-attempt sequence
//! is the only retry pacing in the standard configuration. Reset on
//! topology changes (parent change, new peer, first RTT, reconnection).
//!
//! - **`DiscoveryForwardRateLimiter`** (transit-side): Per-target minimum
//! interval for forwarded requests. Defense-in-depth against misbehaving
//! nodes generating fresh request_ids at high rate.
use crate::NodeAddr;
use std::collections::HashMap;
use std::time::{Duration, Instant};
// ============================================================================
// Originator-side: Discovery Backoff
// ============================================================================
/// Default base backoff after first lookup failure. `0` = disabled.
const DEFAULT_BACKOFF_BASE_SECS: u64 = 0;
/// Default maximum backoff cap. `0` = disabled.
const DEFAULT_BACKOFF_MAX_SECS: u64 = 0;
/// Backoff multiplier per consecutive failure.
const BACKOFF_MULTIPLIER: u64 = 2;
/// Exponential backoff for failed discovery lookups.
///
/// Tracks targets whose lookups have timed out and suppresses
/// re-initiation with increasing delays. Cleared on topology changes.
pub struct DiscoveryBackoff {
/// Maps target → (suppress_until, consecutive_failures).
entries: HashMap<NodeAddr, BackoffEntry>,
/// Base backoff duration (first failure).
base: Duration,
/// Maximum backoff cap.
max: Duration,
}
struct BackoffEntry {
/// Don't re-initiate until this instant.
suppress_until: Instant,
/// Consecutive failures (drives exponential backoff).
failures: u32,
}
impl DiscoveryBackoff {
/// Create with default parameters (disabled — base/cap = 0).
pub fn new() -> Self {
Self::with_params(DEFAULT_BACKOFF_BASE_SECS, DEFAULT_BACKOFF_MAX_SECS)
}
/// Create with custom base and max backoff in seconds.
pub fn with_params(base_secs: u64, max_secs: u64) -> Self {
Self {
entries: HashMap::new(),
base: Duration::from_secs(base_secs),
max: Duration::from_secs(max_secs),
}
}
/// Check if a lookup for this target is suppressed.
///
/// Returns true if the target is in backoff and should not be
/// looked up yet.
pub fn is_suppressed(&self, target: &NodeAddr) -> bool {
if let Some(entry) = self.entries.get(target) {
Instant::now() < entry.suppress_until
} else {
false
}
}
/// Record a lookup failure (timeout) for a target.
///
/// Increments the failure count and sets the next suppression
/// window using exponential backoff.
pub fn record_failure(&mut self, target: &NodeAddr) {
let now = Instant::now();
let failures = self.entries.get(target).map_or(0, |e| e.failures) + 1;
let backoff_secs = self
.base
.as_secs()
.saturating_mul(BACKOFF_MULTIPLIER.saturating_pow(failures.saturating_sub(1)));
let backoff = Duration::from_secs(backoff_secs.min(self.max.as_secs()));
self.entries.insert(
*target,
BackoffEntry {
suppress_until: now + backoff,
failures,
},
);
}
/// Record a successful lookup — remove backoff for this target.
pub fn record_success(&mut self, target: &NodeAddr) {
self.entries.remove(target);
}
/// Clear all backoff entries.
///
/// Called on topology changes that might make previously-unreachable
/// targets reachable (parent change, new peer, first RTT, reconnection).
pub fn reset_all(&mut self) {
self.entries.clear();
}
/// Whether any entries exist.
pub fn is_empty(&self) -> bool {
self.entries.is_empty()
}
/// Current number of entries.
pub fn entry_count(&self) -> usize {
self.entries.len()
}
/// Get the failure count for a target (for logging).
pub fn failure_count(&self, target: &NodeAddr) -> u32 {
self.entries.get(target).map_or(0, |e| e.failures)
}
#[cfg(test)]
pub fn len(&self) -> usize {
self.entries.len()
}
}
impl Default for DiscoveryBackoff {
fn default() -> Self {
Self::new()
}
}
// ============================================================================
// Transit-side: Discovery Forward Rate Limiter
// ============================================================================
/// Default minimum interval between forwarded lookups for the same target.
const DEFAULT_FORWARD_MIN_INTERVAL: Duration = Duration::from_secs(2);
/// Maximum age of entries before cleanup.
const FORWARD_MAX_AGE: Duration = Duration::from_secs(60);
/// Rate limiter for forwarded discovery requests.
///
/// Tracks the last time a LookupRequest was forwarded for each target
/// and enforces a minimum interval to prevent floods from misbehaving
/// nodes generating fresh request_ids.
pub struct DiscoveryForwardRateLimiter {
last_forwarded: HashMap<NodeAddr, Instant>,
min_interval: Duration,
max_age: Duration,
}
impl DiscoveryForwardRateLimiter {
/// Create with default parameters (2s interval).
pub fn new() -> Self {
Self {
last_forwarded: HashMap::new(),
min_interval: DEFAULT_FORWARD_MIN_INTERVAL,
max_age: FORWARD_MAX_AGE,
}
}
/// Create with a custom minimum interval.
pub fn with_interval(min_interval: Duration) -> Self {
Self {
last_forwarded: HashMap::new(),
min_interval,
max_age: FORWARD_MAX_AGE,
}
}
/// Check if we should forward a lookup for this target.
///
/// Returns true if enough time has passed since the last forward
/// for this target. Updates internal state when returning true.
pub fn should_forward(&mut self, target: &NodeAddr) -> bool {
let now = Instant::now();
if let Some(&last) = self.last_forwarded.get(target)
&& now.duration_since(last) < self.min_interval
{
return false;
}
self.last_forwarded.insert(*target, now);
self.cleanup(now);
true
}
/// Replace the minimum interval (e.g., set to zero to disable).
#[cfg(test)]
pub fn set_interval(&mut self, interval: Duration) {
self.min_interval = interval;
}
/// Remove entries older than max_age.
fn cleanup(&mut self, now: Instant) {
self.last_forwarded
.retain(|_, &mut last| now.duration_since(last) < self.max_age);
}
#[cfg(test)]
pub fn len(&self) -> usize {
self.last_forwarded.len()
}
}
impl Default for DiscoveryForwardRateLimiter {
fn default() -> Self {
Self::new()
}
}
// ============================================================================
// Tests
// ============================================================================
#[cfg(test)]
mod tests {
use super::*;
use std::thread;
fn addr(val: u8) -> NodeAddr {
let mut bytes = [0u8; 16];
bytes[0] = val;
NodeAddr::from_bytes(bytes)
}
// --- DiscoveryBackoff tests ---
#[test]
fn test_backoff_not_suppressed_initially() {
let backoff = DiscoveryBackoff::new();
assert!(!backoff.is_suppressed(&addr(1)));
}
#[test]
fn test_backoff_suppressed_after_failure() {
// Backoff is opt-in; exercise the suppression path with explicit params.
let mut backoff = DiscoveryBackoff::with_params(30, 300);
backoff.record_failure(&addr(1));
assert!(backoff.is_suppressed(&addr(1)));
// Different target not affected
assert!(!backoff.is_suppressed(&addr(2)));
}
#[test]
fn test_backoff_cleared_on_success() {
let mut backoff = DiscoveryBackoff::with_params(30, 300);
backoff.record_failure(&addr(1));
assert!(backoff.is_suppressed(&addr(1)));
backoff.record_success(&addr(1));
assert!(!backoff.is_suppressed(&addr(1)));
}
#[test]
fn test_backoff_reset_all() {
let mut backoff = DiscoveryBackoff::new();
backoff.record_failure(&addr(1));
backoff.record_failure(&addr(2));
assert_eq!(backoff.len(), 2);
backoff.reset_all();
assert_eq!(backoff.len(), 0);
assert!(!backoff.is_suppressed(&addr(1)));
}
#[test]
fn test_backoff_exponential() {
let mut backoff = DiscoveryBackoff::with_params(1, 300);
// First failure: 1s backoff
backoff.record_failure(&addr(1));
assert_eq!(backoff.failure_count(&addr(1)), 1);
// Second failure: 2s backoff
backoff.record_failure(&addr(1));
assert_eq!(backoff.failure_count(&addr(1)), 2);
// Third failure: 4s backoff
backoff.record_failure(&addr(1));
assert_eq!(backoff.failure_count(&addr(1)), 3);
}
#[test]
fn test_backoff_expires() {
let mut backoff = DiscoveryBackoff::with_params(0, 0);
backoff.record_failure(&addr(1));
// With 0s backoff, should not be suppressed
assert!(!backoff.is_suppressed(&addr(1)));
}
#[test]
fn test_backoff_capped() {
let mut backoff = DiscoveryBackoff::with_params(1, 10);
// Record many failures
for _ in 0..20 {
backoff.record_failure(&addr(1));
}
// Backoff should be capped at max (10s), not overflow
let entry = backoff.entries.get(&addr(1)).unwrap();
let remaining = entry.suppress_until.duration_since(Instant::now());
assert!(remaining <= Duration::from_secs(11));
}
// --- DiscoveryForwardRateLimiter tests ---
#[test]
fn test_forward_first_allowed() {
let mut limiter = DiscoveryForwardRateLimiter::new();
assert!(limiter.should_forward(&addr(1)));
}
#[test]
fn test_forward_rapid_rate_limited() {
let mut limiter = DiscoveryForwardRateLimiter::new();
assert!(limiter.should_forward(&addr(1)));
assert!(!limiter.should_forward(&addr(1)));
assert!(!limiter.should_forward(&addr(1)));
}
#[test]
fn test_forward_different_targets_independent() {
let mut limiter = DiscoveryForwardRateLimiter::new();
assert!(limiter.should_forward(&addr(1)));
assert!(limiter.should_forward(&addr(2)));
assert!(!limiter.should_forward(&addr(1)));
assert!(!limiter.should_forward(&addr(2)));
}
#[test]
fn test_forward_allowed_after_interval() {
let mut limiter = DiscoveryForwardRateLimiter::with_interval(Duration::from_millis(100));
assert!(limiter.should_forward(&addr(1)));
thread::sleep(Duration::from_millis(110));
assert!(limiter.should_forward(&addr(1)));
}
#[test]
fn test_forward_cleanup_removes_old() {
let mut limiter = DiscoveryForwardRateLimiter::new();
assert!(limiter.should_forward(&addr(1)));
assert!(limiter.should_forward(&addr(2)));
assert_eq!(limiter.len(), 2);
let future = Instant::now() + Duration::from_secs(61);
limiter.cleanup(future);
assert_eq!(limiter.len(), 0);
}
#[test]
fn test_forward_cleanup_preserves_recent() {
let mut limiter = DiscoveryForwardRateLimiter::new();
assert!(limiter.should_forward(&addr(1)));
assert_eq!(limiter.len(), 1);
limiter.cleanup(Instant::now());
assert_eq!(limiter.len(), 1);
}
}
src/node/encrypt_worker.rs
+16 -19
View File
@@ -50,9 +50,9 @@
// warnings rather than gate every function individually.
#![cfg_attr(not(unix), allow(dead_code))]
use crate::node::session_wire::FSP_HEADER_SIZE;
use crate::node::wire::ESTABLISHED_HEADER_SIZE;
use crate::transport::udp::socket::AsyncUdpSocket;
use crate::proto::fmp::wire::ESTABLISHED_HEADER_SIZE;
use crate::proto::fsp::wire::FSP_HEADER_SIZE;
use crate::transport::udp::io::AsyncUdpSocket;
#[cfg(not(target_os = "macos"))]
use crossbeam_channel::{Receiver, SendError, Sender, TrySendError, bounded};
use ring::aead::{Aad, LessSafeKey, Nonce};
@@ -132,8 +132,7 @@ pub(crate) struct FmpSendJob {
/// the job completes and the worker drops it, only the peer's
/// strong ref remains.
#[cfg(any(target_os = "linux", target_os = "macos"))]
pub connected_socket:
Option<std::sync::Arc<crate::transport::udp::connected_peer::ConnectedPeerSocket>>,
pub connected_socket: Option<std::sync::Arc<crate::peer::connected_udp::ConnectedPeerSocket>>,
/// Bulk endpoint data may be dropped when the kernel reports UDP
/// send-queue exhaustion. Control/rekey frames keep retrying so
/// congestion cannot strand the session.
@@ -716,8 +715,7 @@ fn mac_now_ms() -> u64 {
struct MacSequencedSendFlow {
key: MacSendFlowKey,
socket: AsyncUdpSocket,
connected_socket:
Option<std::sync::Arc<crate::transport::udp::connected_peer::ConnectedPeerSocket>>,
connected_socket: Option<std::sync::Arc<crate::peer::connected_udp::ConnectedPeerSocket>>,
dest_addr: SocketAddr,
next_seq: std::sync::atomic::AtomicU64,
last_used_ms: std::sync::atomic::AtomicU64,
@@ -754,9 +752,7 @@ impl MacSequencedSendFlow {
fn spawn(
key: MacSendFlowKey,
socket: AsyncUdpSocket,
connected_socket: Option<
std::sync::Arc<crate::transport::udp::connected_peer::ConnectedPeerSocket>,
>,
connected_socket: Option<std::sync::Arc<crate::peer::connected_udp::ConnectedPeerSocket>>,
dest_addr: SocketAddr,
now_ms: u64,
) -> Arc<Self> {
@@ -1024,8 +1020,7 @@ fn flush_batch_sync(
struct EncryptedGroup {
socket: AsyncUdpSocket,
#[cfg(any(target_os = "linux", target_os = "macos"))]
connected_socket:
Option<std::sync::Arc<crate::transport::udp::connected_peer::ConnectedPeerSocket>>,
connected_socket: Option<std::sync::Arc<crate::peer::connected_udp::ConnectedPeerSocket>>,
dest_addr: SocketAddr,
wire_packets: Vec<Vec<u8>>,
drop_on_backpressure: bool,
@@ -1710,7 +1705,7 @@ fn send_batch_gso(
}
/// Direct `sendmmsg(2)` wrapper for the sync worker. The
/// `transport::udp::socket` module's existing `send_batch` is
/// `transport::udp::io` module's existing `send_batch` is
/// pub(crate) on `UdpRawSocket`, but we don't have a handle to the
/// raw socket from here — we just have the FD. Re-implementing
/// inline is ~15 lines and avoids tunnelling the inner socket
@@ -1780,7 +1775,7 @@ fn send_batch_raw(
#[cfg(all(test, unix))]
mod unix_tests {
use super::*;
use crate::transport::udp::socket::UdpRawSocket;
use crate::transport::udp::io::UdpRawSocket;
use ring::aead::{LessSafeKey, UnboundKey};
use std::net::UdpSocket;
@@ -1904,10 +1899,12 @@ mod unix_tests {
#[test]
fn pipelined_send_wire_layout_roundtrips_canonical_decoders() {
use crate::NodeAddr;
use crate::node::session_wire::build_fsp_header;
use crate::node::wire::{EncryptedHeader, FLAG_KEY_EPOCH, build_established_header};
use crate::noise::TAG_SIZE;
use crate::protocol::{LinkMessageType, SESSION_DATAGRAM_HEADER_SIZE, SessionDatagramRef};
use crate::proto::fmp::wire::{EncryptedHeader, FLAG_KEY_EPOCH, build_established_header};
use crate::proto::fsp::wire::build_fsp_header;
use crate::proto::link::{
LinkMessageType, SESSION_DATAGRAM_HEADER_SIZE, SessionDatagramRef,
};
use crate::utils::index::SessionIndex;
let rt = tokio::runtime::Builder::new_current_thread()
@@ -2218,7 +2215,7 @@ mod tests {
/// AsRawFd impl.
#[test]
fn flush_batch_routes_each_target_separately() {
use crate::transport::udp::socket::UdpRawSocket;
use crate::transport::udp::io::UdpRawSocket;
use ring::aead::{LessSafeKey, UnboundKey};
use std::net::UdpSocket;
@@ -2264,7 +2261,7 @@ mod tests {
const B_WIRE: usize = 16 + B_PLAINTEXT + 16; // 96
fn make_job(
socket: crate::transport::udp::socket::AsyncUdpSocket,
socket: crate::transport::udp::io::AsyncUdpSocket,
cipher: &LessSafeKey,
counter: u64,
dest: SocketAddr,
src/node/handlers/discovery.rs
-738
View File
@@ -1,738 +0,0 @@
//! LookupRequest/LookupResponse discovery protocol handlers.
//!
//! Handles coordinate discovery via bloom-filter-guided tree routing.
//! Requests are forwarded only to tree peers (parent + children) whose
//! bloom filter contains the target. TTL and request_id dedup provide
//! safety bounds.
use crate::node::reject::DiscoveryReject;
use crate::node::{Node, RecentRequest};
use crate::protocol::{LookupRequest, LookupResponse};
use crate::transport::{TransportAddr, TransportId};
use crate::{NodeAddr, PeerIdentity};
use tracing::{debug, info, trace, warn};
const MAX_RECENT_DISCOVERY_REQUESTS: usize = 4096;
impl Node {
/// Handle an incoming LookupRequest from a peer.
///
/// Processing steps:
/// 1. Decode and validate
/// 2. Check request_id for duplicates (dedup / reverse-path routing)
/// 3. Record request for reverse-path forwarding
/// 4. Lazy purge expired entries
/// 5. If we're the target, generate and send response
/// 6. If TTL > 0, forward to tree peers whose bloom filter matches
pub(in crate::node) async fn handle_lookup_request(&mut self, from: &NodeAddr, payload: &[u8]) {
self.metrics().discovery.req_received.inc();
let request = match LookupRequest::decode(payload) {
Ok(req) => req,
Err(e) => {
self.metrics()
.discovery
.record_reject(DiscoveryReject::ReqDecodeError);
debug!(from = %self.peer_display_name(from), error = %e, "Malformed LookupRequest");
return;
}
};
let now_ms = Self::now_ms();
self.purge_expired_requests(now_ms);
// Dedup: drop if we've already seen this request_id.
// Also serves as loop protection — tree routing is loop-free,
// but request_id dedup catches edge cases during tree restructuring.
if self.recent_requests.contains_key(&request.request_id) {
self.metrics()
.discovery
.record_reject(DiscoveryReject::ReqDuplicate);
debug!(
request_id = request.request_id,
from = %self.peer_display_name(from),
"Duplicate LookupRequest, dropping"
);
return;
}
if self.recent_requests.len() >= MAX_RECENT_DISCOVERY_REQUESTS {
self.metrics()
.discovery
.record_reject(DiscoveryReject::ReqDedupCacheFull);
debug!(
request_id = request.request_id,
from = %self.peer_display_name(from),
recent_requests = self.recent_requests.len(),
max_recent_requests = MAX_RECENT_DISCOVERY_REQUESTS,
"Discovery request dedup cache full, dropping LookupRequest"
);
return;
}
// Record for reverse-path forwarding and dedup
self.recent_requests
.insert(request.request_id, RecentRequest::new(*from, now_ms));
// Are we the target?
if request.target == *self.node_addr() {
self.metrics().discovery.req_target_is_us.inc();
debug!(
request_id = request.request_id,
origin = %self.peer_display_name(&request.origin),
"We are the lookup target, generating response"
);
self.send_lookup_response(&request).await;
return;
}
// Forward if TTL permits
if request.can_forward() {
// Transit-side rate limit: collapse rapid-fire lookups for the
// same target from misbehaving nodes generating fresh request_ids.
if !self
.discovery_forward_limiter
.should_forward(&request.target)
{
self.metrics().discovery.req_forward_rate_limited.inc();
debug!(
request_id = request.request_id,
target = %self.peer_display_name(&request.target),
"Forward rate limited, suppressing LookupRequest"
);
return;
}
self.metrics().discovery.req_forwarded.inc();
self.forward_lookup_request(request).await;
} else {
self.metrics()
.discovery
.record_reject(DiscoveryReject::ReqTtlExhausted);
debug!(
request_id = request.request_id,
target = %self.peer_display_name(&request.target),
"LookupRequest TTL exhausted"
);
}
}
/// Handle an incoming LookupResponse from a peer.
///
/// Processing steps:
/// 1. Decode and validate
/// 2. Check recent_requests to determine if we originated or are forwarding
/// 3. If originator: verify proof signature, then cache target_coords and path_mtu in coord_cache
/// 4. If transit: apply path_mtu min(outgoing_link_mtu), reverse-path forward to from_peer
pub(in crate::node) async fn handle_lookup_response(
&mut self,
from: &NodeAddr,
payload: &[u8],
) {
self.metrics().discovery.resp_received.inc();
let mut response = match LookupResponse::decode(payload) {
Ok(resp) => resp,
Err(e) => {
self.metrics()
.discovery
.record_reject(DiscoveryReject::RespDecodeError);
debug!(from = %self.peer_display_name(from), error = %e, "Malformed LookupResponse");
return;
}
};
let now_ms = Self::now_ms();
// Check if we forwarded this request (transit node) or originated it
if let Some(recent) = self.recent_requests.get_mut(&response.request_id) {
// Already forwarded a response for this request — drop to
// prevent response routing loops.
if recent.response_forwarded {
debug!(
request_id = response.request_id,
target = %self.peer_display_name(&response.target),
"Response already forwarded for this request, dropping"
);
return;
}
recent.response_forwarded = true;
// Transit node: reverse-path forward
let from_peer = recent.from_peer;
self.metrics().discovery.resp_forwarded.inc();
// Apply path_mtu min() from the outgoing link's transport MTU
self.apply_outgoing_link_mtu_to_response(&mut response, &from_peer);
debug!(
request_id = response.request_id,
target = %self.peer_display_name(&response.target),
next_hop = %self.peer_display_name(&from_peer),
path_mtu = response.path_mtu,
"Reverse-path forwarding LookupResponse"
);
let encoded = response.encode();
if let Err(e) = self.send_encrypted_link_message(&from_peer, &encoded).await {
debug!(
next_hop = %self.peer_display_name(&from_peer),
error = %e,
"Failed to forward LookupResponse"
);
}
} else {
// We originated this request — verify proof before caching
let target = response.target;
let path_mtu = response.path_mtu;
// Look up the target's public key from identity_cache
let mut prefix = [0u8; 15];
prefix.copy_from_slice(&target.as_bytes()[0..15]);
let target_pubkey = match self.lookup_by_fips_prefix(&prefix) {
Some((_addr, pubkey)) => pubkey,
None => {
self.metrics()
.discovery
.record_reject(DiscoveryReject::RespIdentityMiss);
warn!(
request_id = response.request_id,
target = %self.peer_display_name(&target),
"identity_cache miss for lookup target, cannot verify proof"
);
return;
}
};
// Verify the proof signature
let (xonly, _parity) = target_pubkey.x_only_public_key();
let peer_id = PeerIdentity::from_pubkey(xonly);
let proof_data =
LookupResponse::proof_bytes(response.request_id, &target, &response.target_coords);
if !peer_id.verify(&proof_data, &response.proof) {
self.metrics()
.discovery
.record_reject(DiscoveryReject::RespProofFailed);
warn!(
request_id = response.request_id,
target = %self.peer_display_name(&target),
"LookupResponse proof verification failed, discarding"
);
return;
}
self.metrics().discovery.resp_accepted.inc();
// Clear backoff on success — target is reachable
self.discovery_backoff.record_success(&target);
info!(
request_id = response.request_id,
target = %self.peer_display_name(&target),
depth = response.target_coords.depth(),
path_mtu = path_mtu,
"Discovery succeeded, proof verified, route cached"
);
self.coord_cache
.insert_with_path_mtu(target, response.target_coords, now_ms, path_mtu);
// Mirror path_mtu into the FipsAddress-keyed read-only lookup
// map used by the TUN reader/writer at TCP MSS clamp time.
let fips_addr = crate::FipsAddress::from_node_addr(&target);
match self.path_mtu_lookup.write() {
Ok(mut map) => {
let prior = map.insert(fips_addr, path_mtu);
debug!(
target = %self.peer_display_name(&target),
fips_addr = %fips_addr,
path_mtu = path_mtu,
prior = ?prior,
map_len = map.len(),
"Wrote path_mtu_lookup from discovery LookupResponse"
);
}
Err(e) => {
warn!(
target = %self.peer_display_name(&target),
fips_addr = %fips_addr,
path_mtu = path_mtu,
error = %e,
"path_mtu_lookup write lock poisoned; clamp will not see this update"
);
}
}
// Clean up pending lookup tracking
self.pending_lookups.remove(&target);
// If an established session exists, reset the warmup counter.
let n = self.config().node.session.coords_warmup_packets;
if let Some(entry) = self.sessions.get_mut(&target)
&& entry.is_established()
{
entry.set_coords_warmup_remaining(n);
debug!(
dest = %self.peer_display_name(&target),
warmup_packets = n,
"Reset coords warmup after discovery for existing session"
);
}
// If we have pending TUN packets for this target, retry session
// initiation. The coord_cache now has coords, so find_next_hop()
// should succeed.
if let Some(packets) = self.pending_tun_packets.get(&target) {
debug!(
dest = %self.peer_display_name(&target),
queued_packets = packets.len(),
"Retrying queued packets after discovery"
);
self.retry_session_after_discovery(target).await;
}
}
}
/// Generate and send a LookupResponse when we are the target.
async fn send_lookup_response(&mut self, request: &LookupRequest) {
let our_coords = self.tree_state().my_coords().clone();
// Sign proof: Identity::sign hashes with SHA-256 internally
let proof_data =
LookupResponse::proof_bytes(request.request_id, &request.target, &our_coords);
let proof = self.identity().sign(&proof_data);
let mut response =
LookupResponse::new(request.request_id, request.target, our_coords, proof);
// Route toward origin via reverse path.
let next_hop_addr = if let Some(recent) = self.recent_requests.get(&request.request_id) {
recent.from_peer
} else {
// Fallback: try greedy tree routing toward origin
match self.find_next_hop(&request.origin) {
Some(peer) => *peer.node_addr(),
None => {
debug!(
origin = %self.peer_display_name(&request.origin),
"Cannot route LookupResponse: no reverse path or tree route to origin"
);
self.metrics()
.discovery
.record_reject(DiscoveryReject::RespNoRoute);
return;
}
}
};
// Fold our outgoing-link MTU into path_mtu so the target-edge link
// appears in the bottleneck calculation. Without this, the response
// leaves the target with path_mtu = u16::MAX and only intermediate
// transits min-fold; the target's first reverse-path hop is missed.
self.apply_outgoing_link_mtu_to_response(&mut response, &next_hop_addr);
debug!(
request_id = request.request_id,
origin = %self.peer_display_name(&request.origin),
next_hop = %self.peer_display_name(&next_hop_addr),
path_mtu = response.path_mtu,
"Sending LookupResponse"
);
let encoded = response.encode();
if let Err(e) = self
.send_encrypted_link_message(&next_hop_addr, &encoded)
.await
{
debug!(
next_hop = %self.peer_display_name(&next_hop_addr),
error = %e,
"Failed to send LookupResponse"
);
}
}
/// Forward a LookupRequest to eligible peers.
///
/// Primary path: tree peers (parent + children) whose bloom filter
/// contains the target. Restricting to tree peers follows the spanning
/// tree partition, producing a single directed path.
///
/// Fallback: if no tree peer's bloom matches, try non-tree peers whose
/// bloom contains the target. This recovers from dead ends caused by
/// stale bloom filters, tree restructuring, or transit node failures.
async fn forward_lookup_request(&mut self, mut request: LookupRequest) {
if !request.forward() {
return;
}
// Collect tree peers whose bloom filter contains the target
let forward_to: Vec<NodeAddr> = self
.peers
.iter()
.filter(|(addr, peer)| self.is_tree_peer(addr) && peer.may_reach(&request.target))
.map(|(addr, _)| *addr)
.collect();
// Fallback: if no tree peer matches, try non-tree bloom-matching peers
let (forward_to, used_fallback) = if forward_to.is_empty() {
let fallback: Vec<NodeAddr> = self
.peers
.iter()
.filter(|(addr, peer)| !self.is_tree_peer(addr) && peer.may_reach(&request.target))
.map(|(addr, _)| *addr)
.collect();
if fallback.is_empty() {
self.metrics().discovery.req_no_tree_peer.inc();
trace!(
request_id = request.request_id,
"No eligible peers to forward LookupRequest"
);
return;
}
(fallback, true)
} else {
(forward_to, false)
};
if used_fallback {
self.metrics().discovery.req_fallback_forwarded.inc();
debug!(
request_id = request.request_id,
target = %self.peer_display_name(&request.target),
ttl = request.ttl,
peer_count = forward_to.len(),
"Forwarding LookupRequest via non-tree fallback"
);
} else {
debug!(
request_id = request.request_id,
target = %self.peer_display_name(&request.target),
ttl = request.ttl,
peer_count = forward_to.len(),
"Forwarding LookupRequest"
);
}
let encoded = request.encode();
for peer_addr in forward_to {
if let Err(e) = self.send_encrypted_link_message(&peer_addr, &encoded).await {
debug!(
peer = %self.peer_display_name(&peer_addr),
error = %e,
"Failed to forward LookupRequest to peer"
);
}
}
}
/// Initiate a discovery lookup for a target node.
///
/// Creates a LookupRequest and sends it to tree peers whose bloom
/// filters contain the target. Returns the number of peers sent to.
/// The originator does NOT record the request_id in recent_requests,
/// so when the response arrives, it's recognized as "our request".
pub(in crate::node) async fn initiate_lookup(&mut self, target: &NodeAddr, ttl: u8) -> usize {
self.metrics().discovery.req_initiated.inc();
let origin = *self.node_addr();
let origin_coords = self.tree_state().my_coords().clone();
let request = LookupRequest::generate(*target, origin, origin_coords, ttl, 0);
// Send only to tree peers whose bloom filter contains the target
let peer_addrs: Vec<NodeAddr> = self
.peers
.iter()
.filter(|(addr, peer)| self.is_tree_peer(addr) && peer.may_reach(target))
.map(|(addr, _)| *addr)
.collect();
let peer_count = peer_addrs.len();
debug!(
request_id = request.request_id,
target = %self.peer_display_name(target),
ttl = ttl,
peer_count = peer_count,
total_peers = self.peers.len(),
"Discovery lookup initiated"
);
if peer_count == 0 {
return 0;
}
let encoded = request.encode();
for peer_addr in peer_addrs {
if let Err(e) = self.send_encrypted_link_message(&peer_addr, &encoded).await {
debug!(
peer = %self.peer_display_name(&peer_addr),
error = %e,
"Failed to send LookupRequest to peer"
);
}
}
peer_count
}
/// Initiate a discovery lookup if one is not already pending for this target.
///
/// Checks: pending dedup, post-failure backoff (off by default), bloom
/// filter pre-check. If all pass, sends the first attempt's LookupRequest.
/// Subsequent attempts (with fresh request_ids) are scheduled by
/// [`Self::check_pending_lookups`] when each attempt's per-attempt timeout
/// expires, using the sequence in `node.discovery.attempt_timeouts_secs`.
pub(in crate::node) async fn maybe_initiate_lookup(&mut self, dest: &NodeAddr) {
let now_ms = Self::now_ms();
// Dedup: any pending lookup means we are already trying.
if self.pending_lookups.contains_key(dest) {
self.metrics().discovery.req_deduplicated.inc();
debug!(
target_node = %self.peer_display_name(dest),
"Discovery lookup deduplicated, already pending"
);
return;
}
// Optional post-failure suppression. Defaults are 0/0 (inert);
// operators can opt in by setting `node.discovery.backoff_*_secs`.
if self.discovery_backoff.is_suppressed(dest) {
self.metrics().discovery.req_backoff_suppressed.inc();
debug!(
target_node = %self.peer_display_name(dest),
failures = self.discovery_backoff.failure_count(dest),
"Discovery lookup suppressed by backoff"
);
return;
}
// Bloom filter pre-check: if no peer's filter contains the target,
// it's not in the mesh — skip the lookup and record as failure.
let reachable = self.peers.values().any(|peer| peer.may_reach(dest));
if !reachable {
self.metrics().discovery.req_bloom_miss.inc();
self.discovery_backoff.record_failure(dest);
debug!(
target_node = %self.peer_display_name(dest),
"Discovery skipped, target not in any peer bloom filter"
);
return;
}
self.pending_lookups
.insert(*dest, PendingLookup::new(now_ms));
let ttl = self.config().node.discovery.ttl;
let sent = self.initiate_lookup(dest, ttl).await;
// If no tree peers had the target, fail immediately
if sent == 0 {
self.pending_lookups.remove(dest);
self.discovery_backoff.record_failure(dest);
debug!(
target_node = %self.peer_display_name(dest),
"Discovery failed, no tree peers with bloom match"
);
}
}
/// Check pending lookups for next-attempt or final timeout.
///
/// Called periodically from the tick handler. The lookup state machine
/// runs through `node.discovery.attempt_timeouts_secs` (default
/// `[1, 2, 4, 8]`): each entry is the deadline for one attempt. When the
/// current attempt's deadline elapses:
/// - If more entries remain: send the next attempt with a fresh
/// `request_id`.
/// - Otherwise: declare the destination unreachable, drop queued packets,
/// and emit ICMPv6 destination-unreachable for each.
pub(in crate::node) async fn check_pending_lookups(&mut self, now_ms: u64) {
let timeouts = self.config().node.discovery.attempt_timeouts_secs.clone();
let max_attempts = timeouts.len() as u8;
// Collect targets needing action
let mut to_retry: Vec<NodeAddr> = Vec::new();
let mut to_timeout: Vec<NodeAddr> = Vec::new();
for (&target, entry) in &self.pending_lookups {
let attempt_idx = (entry.attempt as usize).saturating_sub(1);
let attempt_timeout_ms = timeouts.get(attempt_idx).copied().unwrap_or(0) * 1000;
if now_ms.saturating_sub(entry.last_sent_ms) >= attempt_timeout_ms {
if entry.attempt >= max_attempts {
to_timeout.push(target);
} else {
to_retry.push(target);
}
}
}
// Process retries
for target in to_retry {
if let Some(entry) = self.pending_lookups.get_mut(&target) {
entry.attempt += 1;
entry.last_sent_ms = now_ms;
let attempt = entry.attempt;
let ttl = self.config().node.discovery.ttl;
let sent = self.initiate_lookup(&target, ttl).await;
if sent > 0 {
debug!(
target_node = %self.peer_display_name(&target),
attempt = attempt,
"Discovery retry sent"
);
}
}
}
// Process timeouts
for addr in to_timeout {
self.metrics().discovery.resp_timed_out.inc();
self.pending_lookups.remove(&addr);
// Record failure for optional backoff
self.discovery_backoff.record_failure(&addr);
let failures = self.discovery_backoff.failure_count(&addr);
let queued = self.pending_tun_packets.remove(&addr);
let pkt_count = queued.as_ref().map_or(0, |p| p.len());
info!(
target_node = %self.peer_display_name(&addr),
queued_packets = pkt_count,
failures = failures,
"Discovery lookup timed out, destination unreachable"
);
if let Some(packets) = queued {
for pkt in &packets {
self.send_icmpv6_dest_unreachable(pkt);
}
}
}
}
/// Reset discovery backoff on topology changes.
pub(in crate::node) fn reset_discovery_backoff(&mut self) {
if !self.discovery_backoff.is_empty() {
debug!(
entries = self.discovery_backoff.entry_count(),
"Resetting discovery backoff on topology change"
);
self.discovery_backoff.reset_all();
}
}
/// Remove expired entries from the recent_requests cache.
fn purge_expired_requests(&mut self, current_time_ms: u64) {
let expiry_ms = self.config().node.discovery.recent_expiry_secs * 1000;
self.recent_requests
.retain(|_, entry| !entry.is_expired(current_time_ms, expiry_ms));
}
/// Min-fold our outgoing-link MTU into a LookupResponse's `path_mtu`.
///
/// Used at both transit-side reverse-path forward and at the target's
/// own send_lookup_response. The link MTU we apply is the MTU of the
/// transport+addr we'll use to deliver the response toward `next_hop`.
/// No-op when `next_hop` is not a directly-connected peer or its
/// transport is not registered.
pub(in crate::node) fn apply_outgoing_link_mtu_to_response(
&self,
response: &mut LookupResponse,
next_hop: &NodeAddr,
) {
if let Some(peer) = self.peers.get(next_hop)
&& let Some(tid) = peer.transport_id()
&& let Some(transport) = self.transports.get(&tid)
{
let link_mtu = if let Some(addr) = peer.current_addr() {
transport.link_mtu(addr)
} else {
transport.mtu()
};
response.path_mtu = response.path_mtu.min(link_mtu);
}
}
/// Seed `path_mtu_lookup` for a directly-connected peer.
///
/// Called when an FMP link-layer peer is promoted to active. The seed
/// value is the local outgoing-link MTU on the peer's transport, which
/// is the actual link constraint for direct-link traffic. Stored only
/// when no tighter value exists: discovery's reverse-path bottleneck
/// or MMP `MtuExceeded` reactive learning take precedence when smaller.
///
/// Without this seed, configured/auto-connect peers (which establish
/// sessions without going through the discovery Lookup flow) leave
/// `path_mtu_lookup` empty for their FipsAddress, causing
/// `per_flow_max_mss` to fall back to the global ceiling and the
/// SYN-time TCP MSS clamp to over-estimate the effective path.
pub(in crate::node) fn seed_path_mtu_for_link_peer(
&self,
peer_addr: &NodeAddr,
transport_id: TransportId,
addr: &TransportAddr,
) {
let Some(transport) = self.transports.get(&transport_id) else {
debug!(
peer = %self.peer_display_name(peer_addr),
transport_id = %transport_id,
"seed_path_mtu_for_link_peer: transport not registered, skipping seed"
);
return;
};
let link_mtu = transport.link_mtu(addr);
let fips_addr = crate::FipsAddress::from_node_addr(peer_addr);
let Ok(mut map) = self.path_mtu_lookup.write() else {
warn!(
peer = %self.peer_display_name(peer_addr),
"seed_path_mtu_for_link_peer: path_mtu_lookup write lock poisoned"
);
return;
};
match map.get(&fips_addr).copied() {
Some(existing) if existing <= link_mtu => {
// Keep the tighter learned value; never loosen the clamp.
debug!(
peer = %self.peer_display_name(peer_addr),
fips_addr = %fips_addr,
link_mtu = link_mtu,
existing = existing,
"seed_path_mtu_for_link_peer: keeping tighter existing value"
);
}
other => {
map.insert(fips_addr, link_mtu);
debug!(
peer = %self.peer_display_name(peer_addr),
fips_addr = %fips_addr,
link_mtu = link_mtu,
prior = ?other,
map_len = map.len(),
"seed_path_mtu_for_link_peer: wrote link MTU"
);
}
}
}
}
/// Tracks a pending discovery lookup with retry state.
pub struct PendingLookup {
/// When the lookup was first initiated.
pub initiated_ms: u64,
/// When the last attempt was sent.
pub last_sent_ms: u64,
/// Current attempt number (1 = initial, 2 = first retry, ...).
pub attempt: u8,
}
impl PendingLookup {
pub fn new(now_ms: u64) -> Self {
Self {
initiated_ms: now_ms,
last_sent_ms: now_ms,
attempt: 1,
}
}
}
src/node/handlers/handshake.rs
+887 -562
View File
File diff suppressed because it is too large Load Diff
src/node/handlers/lookup.rs
+739
View File
@@ -0,0 +1,739 @@
//! LookupRequest/LookupResponse mesh lookup protocol handlers.
//!
//! Handles coordinate lookup via bloom-filter-guided tree routing.
//! Requests are forwarded only to tree peers (parent + children) whose
//! bloom filter contains the target. TTL and request_id dedup provide
//! safety bounds.
use crate::node::Node;
use crate::node::reject::DiscoveryReject;
use crate::proto::lookup::{
LookupAction, LookupRequest, LookupResponse, MAX_RECENT_LOOKUP_REQUESTS,
};
use crate::transport::{TransportAddr, TransportId};
use crate::{NodeAddr, PeerIdentity};
use tracing::{debug, info, trace, warn};
/// Shell adapter exposing the live routing tables to the sans-IO discovery
/// core's `RoutingView` read seam. Lives in `node` so it can read `Node`'s
/// private `peers` map and call the crate-private tree/bloom predicates.
///
/// Holding `&Node` whole is fine for the forward path because it does not
/// also need `&mut self.lookup` concurrently. A later commit whose core
/// step needs `&mut discovery` while reading routing state should narrow this
/// to borrow only `peers` + `tree_state` instead of the whole node.
struct NodeRoutingView<'a> {
node: &'a Node,
}
impl crate::proto::lookup::RoutingView for NodeRoutingView<'_> {
fn is_tree_peer(&self, addr: &NodeAddr) -> bool {
self.node.is_tree_peer(addr)
}
fn peers_reaching(&self, target: &NodeAddr) -> Vec<NodeAddr> {
self.node
.peers
.iter()
.filter(|(_, peer)| peer.may_reach(target))
.map(|(addr, _)| *addr)
.collect()
}
}
impl Node {
/// Handle an incoming LookupRequest from a peer.
///
/// Processing steps:
/// 1. Decode and validate
/// 2. Check request_id for duplicates (dedup / reverse-path routing)
/// 3. Record request for reverse-path forwarding
/// 4. Lazy purge expired entries
/// 5. If we're the target, generate and send response
/// 6. If TTL > 0, forward to tree peers whose bloom filter matches
pub(in crate::node) async fn handle_lookup_request(&mut self, from: &NodeAddr, payload: &[u8]) {
self.metrics().lookup.req_received.inc();
let request = match LookupRequest::decode(payload) {
Ok(req) => req,
Err(e) => {
self.metrics()
.lookup
.record_reject(DiscoveryReject::ReqDecodeError);
debug!(from = %self.peer_display_name(from), error = %e, "Malformed LookupRequest");
return;
}
};
let now_ms = Self::now_ms();
let recent_expiry_ms = self.config().node.lookup.recent_expiry_secs * 1000;
let my_addr = *self.node_addr();
use crate::proto::lookup::RequestOutcome;
match crate::proto::lookup::classify_request(
&mut self.lookup,
&request,
from,
&my_addr,
now_ms,
recent_expiry_ms,
MAX_RECENT_LOOKUP_REQUESTS,
) {
RequestOutcome::Duplicate => {
self.metrics()
.lookup
.record_reject(DiscoveryReject::ReqDuplicate);
debug!(
request_id = request.request_id,
from = %self.peer_display_name(from),
"Duplicate LookupRequest, dropping"
);
}
RequestOutcome::DedupCacheFull { len } => {
self.metrics()
.lookup
.record_reject(DiscoveryReject::ReqDedupCacheFull);
debug!(
request_id = request.request_id,
from = %self.peer_display_name(from),
recent_requests = len,
max_recent_requests = MAX_RECENT_LOOKUP_REQUESTS,
"Discovery request dedup cache full, dropping LookupRequest"
);
}
RequestOutcome::RespondAsTarget => {
self.metrics().lookup.req_target_is_us.inc();
debug!(
request_id = request.request_id,
origin = %self.peer_display_name(&request.origin),
"We are the lookup target, generating response"
);
self.send_lookup_response(&request).await;
}
RequestOutcome::Forward => {
self.metrics().lookup.req_forwarded.inc();
self.forward_lookup_request(request).await;
}
RequestOutcome::ForwardRateLimited => {
self.metrics().lookup.req_forward_rate_limited.inc();
debug!(
request_id = request.request_id,
target = %self.peer_display_name(&request.target),
"Forward rate limited, suppressing LookupRequest"
);
}
RequestOutcome::TtlExhausted => {
self.metrics()
.lookup
.record_reject(DiscoveryReject::ReqTtlExhausted);
debug!(
request_id = request.request_id,
target = %self.peer_display_name(&request.target),
"LookupRequest TTL exhausted"
);
}
}
}
/// Handle an incoming LookupResponse from a peer.
///
/// Processing steps:
/// 1. Decode and validate
/// 2. Check recent_requests to determine if we originated or are forwarding
/// 3. If originator: verify proof signature, then cache target_coords and path_mtu in coord_cache
/// 4. If transit: apply path_mtu min(outgoing_link_mtu), reverse-path forward to from_peer
pub(in crate::node) async fn handle_lookup_response(
&mut self,
from: &NodeAddr,
payload: &[u8],
) {
self.metrics().lookup.resp_received.inc();
let mut response = match LookupResponse::decode(payload) {
Ok(resp) => resp,
Err(e) => {
self.metrics()
.lookup
.record_reject(DiscoveryReject::RespDecodeError);
debug!(from = %self.peer_display_name(from), error = %e, "Malformed LookupResponse");
return;
}
};
let now_ms = Self::now_ms();
// Check if we forwarded this request (transit node) or originated it
match crate::proto::lookup::classify_response(&mut self.lookup, response.request_id) {
crate::proto::lookup::ResponseRoute::AlreadyForwarded => {
// Already forwarded a response for this request — drop to
// prevent response routing loops.
debug!(
request_id = response.request_id,
target = %self.peer_display_name(&response.target),
"Response already forwarded for this request, dropping"
);
}
crate::proto::lookup::ResponseRoute::Transit { from_peer } => {
// Transit node: reverse-path forward
self.metrics().lookup.resp_forwarded.inc();
// Apply path_mtu min() from the outgoing link's transport MTU
self.apply_outgoing_link_mtu_to_response(&mut response, &from_peer);
debug!(
request_id = response.request_id,
target = %self.peer_display_name(&response.target),
next_hop = %self.peer_display_name(&from_peer),
path_mtu = response.path_mtu,
"Reverse-path forwarding LookupResponse"
);
let encoded = response.encode();
if let Err(e) = self.send_encrypted_link_message(&from_peer, &encoded).await {
debug!(
next_hop = %self.peer_display_name(&from_peer),
error = %e,
"Failed to forward LookupResponse"
);
}
}
crate::proto::lookup::ResponseRoute::Originator => {
// We originated this request — verify proof before caching
let target = response.target;
let path_mtu = response.path_mtu;
// Look up the target's public key from identity_cache
let mut prefix = [0u8; 15];
prefix.copy_from_slice(&target.as_bytes()[0..15]);
let target_pubkey = match self.lookup_by_fips_prefix(&prefix) {
Some((_addr, pubkey)) => pubkey,
None => {
self.metrics()
.lookup
.record_reject(DiscoveryReject::RespIdentityMiss);
warn!(
request_id = response.request_id,
target = %self.peer_display_name(&target),
"identity_cache miss for lookup target, cannot verify proof"
);
return;
}
};
// Verify the proof signature
let (xonly, _parity) = target_pubkey.x_only_public_key();
let peer_id = PeerIdentity::from_pubkey(xonly);
let proof_data = LookupResponse::proof_bytes(
response.request_id,
&target,
&response.target_coords,
);
if !peer_id.verify(&proof_data, &response.proof) {
self.metrics()
.lookup
.record_reject(DiscoveryReject::RespProofFailed);
warn!(
request_id = response.request_id,
target = %self.peer_display_name(&target),
"LookupResponse proof verification failed, discarding"
);
return;
}
self.metrics().lookup.resp_accepted.inc();
info!(
request_id = response.request_id,
target = %self.peer_display_name(&target),
depth = response.target_coords.depth(),
path_mtu = path_mtu,
"Discovery succeeded, proof verified, route cached"
);
// Apply the accept-side effects: the core clears the success
// state (backoff + pending lookup) and returns the
// cross-subsystem effects for us to drive.
let actions = crate::proto::lookup::on_response_accepted(
&mut self.lookup,
&target,
response.target_coords,
now_ms,
path_mtu,
);
self.drive_response_actions(actions).await;
}
}
}
/// Drive the cross-subsystem effects returned by the discovery core's
/// accept-side planning. Each arm reproduces the original inline effect
/// exactly (same metrics/logs/writes, same order).
async fn drive_response_actions(&mut self, actions: Vec<LookupAction>) {
for action in actions {
match action {
LookupAction::CacheCoords {
target,
coords,
now_ms,
path_mtu,
} => {
self.coord_cache
.insert_with_path_mtu(target, coords, now_ms, path_mtu);
}
LookupAction::WritePathMtu { target, path_mtu } => {
// Mirror path_mtu into the FipsAddress-keyed read-only lookup
// map used by the TUN reader/writer at TCP MSS clamp time.
let fips_addr = crate::FipsAddress::from_node_addr(&target);
match self.path_mtu_lookup.write() {
Ok(mut map) => match map.get(&fips_addr).copied() {
Some(existing) if existing <= path_mtu => {
// Keep the tighter learned value; never loosen
// the clamp. A reactive MtuExceeded or
// PathMtuNotification tighten takes precedence
// over a looser discovery estimate
// (cross-carrier keep-tighter).
debug!(
target = %self.peer_display_name(&target),
fips_addr = %fips_addr,
path_mtu = path_mtu,
existing = existing,
"LookupResponse: keeping tighter existing path_mtu_lookup value"
);
}
other => {
map.insert(fips_addr, path_mtu);
debug!(
target = %self.peer_display_name(&target),
fips_addr = %fips_addr,
path_mtu = path_mtu,
prior = ?other,
map_len = map.len(),
"Wrote path_mtu_lookup from discovery LookupResponse"
);
}
},
Err(e) => {
warn!(
target = %self.peer_display_name(&target),
fips_addr = %fips_addr,
path_mtu = path_mtu,
error = %e,
"path_mtu_lookup write lock poisoned; clamp will not see this update"
);
}
}
}
LookupAction::ResetWarmupIfEstablished { target } => {
// If an established session exists, reset the warmup counter.
let n = self.config().node.session.coords_warmup_packets;
if let Some(entry) = self.sessions.get_mut(&target)
&& entry.is_established()
{
entry.set_coords_warmup_remaining(n);
debug!(
dest = %self.peer_display_name(&target),
warmup_packets = n,
"Reset coords warmup after discovery for existing session"
);
}
}
LookupAction::RetryQueuedPackets { target } => {
// If we have pending TUN packets for this target, retry session
// initiation. The coord_cache now has coords, so find_next_hop()
// should succeed.
if let Some(packets) = self.pending_tun_packets.get(&target) {
debug!(
dest = %self.peer_display_name(&target),
queued_packets = packets.len(),
"Retrying queued packets after discovery"
);
self.retry_session_after_discovery(target).await;
}
}
LookupAction::SendLink { peer, bytes } => {
if let Err(e) = self.send_encrypted_link_message(&peer, &bytes).await {
debug!(
peer = %self.peer_display_name(&peer),
error = %e,
"Failed to send discovery link message"
);
}
}
}
}
}
/// Generate and send a LookupResponse when we are the target.
async fn send_lookup_response(&mut self, request: &LookupRequest) {
let our_coords = self.tree_state().my_coords().clone();
// Sign proof: Identity::sign hashes with SHA-256 internally
let proof_data =
LookupResponse::proof_bytes(request.request_id, &request.target, &our_coords);
let proof = self.identity().sign(&proof_data);
let mut response =
LookupResponse::new(request.request_id, request.target, our_coords, proof);
// Route toward origin. The reverse-path decision (the peer the request
// arrived from, recorded in recent_requests) is the sans-IO core's; the
// greedy tree-route fallback is a &mut coord-cache op kept in the shell.
use crate::proto::lookup::ResponseRouteDecision;
let next_hop_addr = match crate::proto::lookup::plan_response_route(
&self.lookup,
request.request_id,
) {
ResponseRouteDecision::ReversePath(peer) => peer,
ResponseRouteDecision::NeedsTreeRoute => match self.find_next_hop(&request.origin) {
Some(peer) => *peer.node_addr(),
None => {
debug!(
origin = %self.peer_display_name(&request.origin),
"Cannot route LookupResponse: no reverse path or tree route to origin"
);
self.metrics()
.lookup
.record_reject(DiscoveryReject::RespNoRoute);
return;
}
},
};
// Fold our outgoing-link MTU into path_mtu so the target-edge link
// appears in the bottleneck calculation. Without this, the response
// leaves the target with path_mtu = u16::MAX and only intermediate
// transits min-fold; the target's first reverse-path hop is missed.
self.apply_outgoing_link_mtu_to_response(&mut response, &next_hop_addr);
debug!(
request_id = request.request_id,
origin = %self.peer_display_name(&request.origin),
next_hop = %self.peer_display_name(&next_hop_addr),
path_mtu = response.path_mtu,
"Sending LookupResponse"
);
let encoded = response.encode();
if let Err(e) = self
.send_encrypted_link_message(&next_hop_addr, &encoded)
.await
{
debug!(
next_hop = %self.peer_display_name(&next_hop_addr),
error = %e,
"Failed to send LookupResponse"
);
}
}
/// Forward a LookupRequest to eligible peers.
///
/// Primary path: tree peers (parent + children) whose bloom filter
/// contains the target. Restricting to tree peers follows the spanning
/// tree partition, producing a single directed path.
///
/// Fallback: if no tree peer's bloom matches, try non-tree peers whose
/// bloom contains the target. This recovers from dead ends caused by
/// stale bloom filters, tree restructuring, or transit node failures.
async fn forward_lookup_request(&mut self, mut request: LookupRequest) {
// Plan the forward with the sans-IO decision core. The core owns the
// TTL decrement, tree/fallback peer selection, and single-encode
// fan-out; the shell keeps all metrics/logging and drives the sends.
let outcome = {
let rv = NodeRoutingView { node: self };
crate::proto::lookup::plan_forward(&mut request, &rv)
};
match outcome {
crate::proto::lookup::ForwardOutcome::TtlExhausted => {}
crate::proto::lookup::ForwardOutcome::NoPeers => {
self.metrics().lookup.req_no_tree_peer.inc();
trace!(
request_id = request.request_id,
"No eligible peers to forward LookupRequest"
);
}
crate::proto::lookup::ForwardOutcome::Forward {
actions,
used_fallback,
} => {
let peer_count = actions.len();
if used_fallback {
self.metrics().lookup.req_fallback_forwarded.inc();
debug!(
request_id = request.request_id,
target = %self.peer_display_name(&request.target),
ttl = request.ttl,
peer_count,
"Forwarding LookupRequest via non-tree fallback"
);
} else {
debug!(
request_id = request.request_id,
target = %self.peer_display_name(&request.target),
ttl = request.ttl,
peer_count,
"Forwarding LookupRequest"
);
}
for action in actions {
if let LookupAction::SendLink { peer, bytes } = action
&& let Err(e) = self.send_encrypted_link_message(&peer, &bytes).await
{
debug!(
peer = %self.peer_display_name(&peer),
error = %e,
"Failed to forward LookupRequest to peer"
);
}
}
}
}
}
/// Initiate a discovery lookup for a target node.
///
/// Creates a LookupRequest and sends it to tree peers whose bloom
/// filters contain the target. Returns the number of peers sent to.
/// The originator does NOT record the request_id in recent_requests,
/// so when the response arrives, it's recognized as "our request".
pub(in crate::node) async fn initiate_lookup(&mut self, target: &NodeAddr, ttl: u8) -> usize {
self.metrics().lookup.req_initiated.inc();
let origin = *self.node_addr();
let origin_coords = self.tree_state().my_coords().clone();
let request_id = {
use rand::RngExt;
rand::rng().random()
};
let request = LookupRequest::new(request_id, *target, origin, origin_coords, ttl, 0);
// Tree-peer bloom-match selection + single encode live in the sans-IO
// core. The core keeps the tree-only (no non-tree fallback) behavior;
// the shell drives the sends and keeps all metrics/logging.
let actions = {
let rv = NodeRoutingView { node: self };
crate::proto::lookup::plan_initiate(&request, &rv)
};
let peer_count = actions.len();
debug!(
request_id = request.request_id,
target = %self.peer_display_name(target),
ttl = ttl,
peer_count = peer_count,
total_peers = self.peers.len(),
"Discovery lookup initiated"
);
for action in actions {
if let LookupAction::SendLink { peer, bytes } = action
&& let Err(e) = self.send_encrypted_link_message(&peer, &bytes).await
{
debug!(
peer = %self.peer_display_name(&peer),
error = %e,
"Failed to send LookupRequest to peer"
);
}
}
peer_count
}
/// Initiate a discovery lookup if one is not already pending for this target.
///
/// Checks: pending dedup, post-failure backoff (off by default), bloom
/// filter pre-check. If all pass, sends the first attempt's LookupRequest.
/// Subsequent attempts (with fresh request_ids) are scheduled by
/// [`Self::check_pending_lookups`] when each attempt's per-attempt timeout
/// expires, using the sequence in `node.lookup.attempt_timeouts_secs`.
pub(in crate::node) async fn maybe_initiate_lookup(&mut self, dest: &NodeAddr) {
let now_ms = Self::now_ms();
// Bloom filter pre-check (view read) BEFORE the core call: if no peer's
// filter contains the target, it's not in the mesh. Reading `self.peers`
// here keeps the `&mut self.lookup` borrow in `initiate_gate` from
// overlapping the immutable peer-table read.
let reachable = self.peers.values().any(|peer| peer.may_reach(dest));
use crate::proto::lookup::InitiateDecision;
match crate::proto::lookup::initiate_gate(&mut self.lookup, dest, now_ms, reachable) {
InitiateDecision::Deduplicated => {
self.metrics().lookup.req_deduplicated.inc();
debug!(
target_node = %self.peer_display_name(dest),
"Discovery lookup deduplicated, already pending"
);
}
InitiateDecision::Suppressed { failures } => {
self.metrics().lookup.req_backoff_suppressed.inc();
debug!(
target_node = %self.peer_display_name(dest),
failures = failures,
"Discovery lookup suppressed by backoff"
);
}
InitiateDecision::BloomMiss => {
self.metrics().lookup.req_bloom_miss.inc();
debug!(
target_node = %self.peer_display_name(dest),
"Discovery skipped, target not in any peer bloom filter"
);
}
InitiateDecision::Proceed => {
let ttl = self.config().node.lookup.ttl;
let sent = self.initiate_lookup(dest, ttl).await;
// If no tree peers had the target, fail immediately
if sent == 0 {
crate::proto::lookup::initiate_failed(&mut self.lookup, dest, now_ms);
debug!(
target_node = %self.peer_display_name(dest),
"Discovery failed, no tree peers with bloom match"
);
}
}
}
}
/// Check pending lookups for next-attempt or final timeout.
///
/// Called periodically from the tick handler. The lookup state machine
/// runs through `node.lookup.attempt_timeouts_secs` (default
/// `[1, 2, 4, 8]`): each entry is the deadline for one attempt. When the
/// current attempt's deadline elapses:
/// - If more entries remain: send the next attempt with a fresh
/// `request_id`.
/// - Otherwise: declare the destination unreachable, drop queued packets,
/// and emit ICMPv6 destination-unreachable for each.
pub(in crate::node) async fn check_pending_lookups(&mut self, now_ms: u64) {
let attempt_timeouts = self.config().node.lookup.attempt_timeouts_secs.clone();
let outcome =
crate::proto::lookup::poll_pending(&mut self.lookup, now_ms, &attempt_timeouts);
for (target, attempt) in outcome.retries {
let ttl = self.config().node.lookup.ttl;
let sent = self.initiate_lookup(&target, ttl).await;
if sent > 0 {
debug!(
target_node = %self.peer_display_name(&target),
attempt = attempt,
"Discovery retry sent"
);
}
}
for (addr, failures) in outcome.timeouts {
self.metrics().lookup.resp_timed_out.inc();
let queued = self.pending_tun_packets.remove(&addr);
let pkt_count = queued.as_ref().map_or(0, |p| p.len());
info!(
target_node = %self.peer_display_name(&addr),
queued_packets = pkt_count,
failures = failures,
"Discovery lookup timed out, destination unreachable"
);
if let Some(packets) = queued {
for pkt in &packets {
self.send_icmpv6_dest_unreachable(pkt);
}
}
}
}
/// Reset discovery backoff on topology changes.
pub(in crate::node) fn reset_lookup_backoff(&mut self) {
let cleared = self.lookup.reset_backoff();
if cleared > 0 {
debug!(
entries = cleared,
"Resetting discovery backoff on topology change"
);
}
}
/// Min-fold our outgoing-link MTU into a LookupResponse's `path_mtu`.
///
/// Used at both transit-side reverse-path forward and at the target's
/// own send_lookup_response. The link MTU we apply is the MTU of the
/// transport+addr we'll use to deliver the response toward `next_hop`.
/// No-op when `next_hop` is not a directly-connected peer or its
/// transport is not registered.
pub(in crate::node) fn apply_outgoing_link_mtu_to_response(
&self,
response: &mut LookupResponse,
next_hop: &NodeAddr,
) {
if let Some(peer) = self.peers.get(next_hop)
&& let Some(tid) = peer.transport_id()
&& let Some(transport) = self.transports.get(&tid)
{
let link_mtu = if let Some(addr) = peer.current_addr() {
transport.link_mtu(addr)
} else {
transport.mtu()
};
response.path_mtu = response.path_mtu.min(link_mtu);
}
}
/// Seed `path_mtu_lookup` for a directly-connected peer.
///
/// Called when an FMP link-layer peer is promoted to active. The seed
/// value is the local outgoing-link MTU on the peer's transport, which
/// is the actual link constraint for direct-link traffic. Stored only
/// when no tighter value exists: discovery's reverse-path bottleneck
/// or MMP `MtuExceeded` reactive learning take precedence when smaller.
///
/// Without this seed, configured/auto-connect peers (which establish
/// sessions without going through the discovery Lookup flow) leave
/// `path_mtu_lookup` empty for their FipsAddress, causing
/// `per_flow_max_mss` to fall back to the global ceiling and the
/// SYN-time TCP MSS clamp to over-estimate the effective path.
pub(in crate::node) fn seed_path_mtu_for_link_peer(
&self,
peer_addr: &NodeAddr,
transport_id: TransportId,
addr: &TransportAddr,
) {
let Some(transport) = self.transports.get(&transport_id) else {
debug!(
peer = %self.peer_display_name(peer_addr),
transport_id = %transport_id,
"seed_path_mtu_for_link_peer: transport not registered, skipping seed"
);
return;
};
let link_mtu = transport.link_mtu(addr);
let fips_addr = crate::FipsAddress::from_node_addr(peer_addr);
let Ok(mut map) = self.path_mtu_lookup.write() else {
warn!(
peer = %self.peer_display_name(peer_addr),
"seed_path_mtu_for_link_peer: path_mtu_lookup write lock poisoned"
);
return;
};
match map.get(&fips_addr).copied() {
Some(existing) if existing <= link_mtu => {
// Keep the tighter learned value; never loosen the clamp.
debug!(
peer = %self.peer_display_name(peer_addr),
fips_addr = %fips_addr,
link_mtu = link_mtu,
existing = existing,
"seed_path_mtu_for_link_peer: keeping tighter existing value"
);
}
other => {
map.insert(fips_addr, link_mtu);
debug!(
peer = %self.peer_display_name(peer_addr),
fips_addr = %fips_addr,
link_mtu = link_mtu,
prior = ?other,
map_len = map.len(),
"seed_path_mtu_for_link_peer: wrote link MTU"
);
}
}
}
}
src/node/handlers/mmp.rs
+306 -337
View File
@@ -5,20 +5,63 @@
//! and teardown metric logs.
use crate::NodeAddr;
use crate::mmp::MmpMode;
use crate::mmp::MmpSessionState;
use crate::mmp::report::{ReceiverReport, SenderReport};
use crate::node::Node;
use crate::node::dataplane::PeerActionCtx;
use crate::node::reject::{MmpReject, RejectReason, TreeReject};
use crate::protocol::{
LinkMessageType, PathMtuNotification, SessionMessageType, SessionReceiverReport,
SessionSenderReport,
use crate::node::tree::sign_declaration;
use crate::peer::machine::PeerEvent;
use crate::proto::link::LinkMessageType;
use crate::proto::mmp::{
LinkReportKind, LinkReportSnapshot, MmpAction, PeerLivenessSnapshot, ReceiverReport, RrLog,
SenderReport,
};
use crate::proto::stp::ParentEval;
use crate::transport::{TransportAddr, TransportId};
use std::time::{Duration, Instant};
use tracing::{debug, info, trace, warn};
/// Emit the operator `trace!` point for a processed ReceiverReport outcome.
///
/// These log points used to live inside `MmpMetrics::process_receiver_report`;
/// the sans-IO migration returns the outcome as an [`RrLog`] and re-emits it
/// here, shell-side, preserving the original field set, content, and (relative
/// to the surrounding handler logs) ordering. The original traces carried no
/// peer identifier, so none is added here.
pub(super) fn log_rr_outcome(rr: &ReceiverReport, our_timestamp_ms: u32, log: RrLog) {
match log {
RrLog::Stale {
prev_highest,
prev_packets,
prev_bytes,
} => trace!(
highest_counter = rr.highest_counter,
prev_highest_counter = prev_highest,
cumulative_packets_recv = rr.cumulative_packets_recv,
prev_cumulative_packets_recv = prev_packets,
cumulative_bytes_recv = rr.cumulative_bytes_recv,
prev_cumulative_bytes_recv = prev_bytes,
"Ignoring stale MMP ReceiverReport"
),
RrLog::RttSample { rtt_ms, srtt_ms } => trace!(
our_ts = our_timestamp_ms,
echo = rr.timestamp_echo,
dwell = u32::from(rr.dwell_time),
rtt_ms = rtt_ms,
srtt_ms = srtt_ms,
"RTT sample from timestamp echo"
),
RrLog::InvalidRtt => trace!(
our_ts = our_timestamp_ms,
echo = rr.timestamp_echo,
dwell = u32::from(rr.dwell_time),
"Ignoring invalid MMP RTT sample"
),
RrLog::None => {}
}
}
/// Format bytes/sec as human-readable throughput.
fn format_throughput(bps: f64) -> String {
pub(in crate::node) fn format_throughput(bps: f64) -> String {
if bps == 0.0 {
"n/a".to_string()
} else if bps >= 1_000_000.0 {
@@ -113,10 +156,12 @@ impl Node {
// Process the report: computes RTT from timestamp echo, updates
// loss rate, goodput rate, jitter trend, and ETX.
let now = Instant::now();
let first_rtt = mmp
.metrics
.process_receiver_report(&rr, our_timestamp_ms, now);
let now_ms = crate::time::mono_ms();
let (first_rtt, rr_log) =
mmp.metrics
.process_receiver_report(&rr, our_timestamp_ms, now_ms);
// Re-emit the operator trace the core used to log mid-decision.
log_rr_outcome(&rr, our_timestamp_ms, rr_log);
// Feed SRTT back to sender/receiver report interval tuning
if let Some(srtt_ms) = mmp.metrics.srtt_ms() {
@@ -144,25 +189,43 @@ impl Node {
// Trigger re-evaluation so the node doesn't wait for the next
// periodic tick or TreeAnnounce.
if first_rtt {
let peer_costs: std::collections::HashMap<crate::NodeAddr, f64> = self
let peer_costs: std::collections::BTreeMap<crate::NodeAddr, f64> = self
.peers
.iter()
.filter(|(_, p)| p.has_srtt())
.map(|(a, p)| (*a, p.link_cost()))
.collect();
if let Some(new_parent) = self.tree_state.evaluate_parent(&peer_costs) {
// Wall-clock seconds for the escaping declaration timestamp;
// monotonic ms for the flap-dampening / hold-down timers.
let now_secs = std::time::SystemTime::now()
.duration_since(std::time::UNIX_EPOCH)
.map(|d| d.as_secs())
.unwrap_or(0);
let mono_now_ms = crate::time::mono_ms();
// Compute the flap-dampening / hold-down veto at the edge; a mandatory
// switch bypasses it, a discretionary one is taken only if not suppressed.
let switch_suppressed = self.tree_state.is_switch_suppressed(mono_now_ms);
let new_parent = match self
.tree_state
.evaluate_parent(&peer_costs, &std::collections::BTreeSet::new())
{
ParentEval::Mandatory(p) => Some(p),
ParentEval::Discretionary(p) if !switch_suppressed => Some(p),
ParentEval::Discretionary(_) | ParentEval::None => None,
};
if let Some(new_parent) = new_parent {
let new_seq = self.tree_state.my_declaration().sequence() + 1;
let timestamp = std::time::SystemTime::now()
.duration_since(std::time::UNIX_EPOCH)
.map(|d| d.as_secs())
.unwrap_or(0);
let flap_dampened = self.tree_state.set_parent(new_parent, new_seq, timestamp);
let flap_dampened =
self.tree_state
.set_parent(new_parent, new_seq, now_secs, mono_now_ms);
self.tree_state.recompute_coords();
// Clone identity once: sign_declaration borrows &mut tree_state while
// the identity() accessor borrows all of &self, so an owned copy avoids
// the split-borrow conflict on this infrequent parent-switch path.
let our_identity = self.identity().clone();
if let Err(e) = self.tree_state.sign_declaration(&our_identity) {
if let Err(e) =
sign_declaration(self.tree_state.my_declaration_mut(), &our_identity)
{
warn!(error = %e, "Failed to sign declaration after first-RTT parent eval");
self.metrics()
.tree
@@ -172,8 +235,7 @@ impl Node {
// Surgical invalidation — see CoordCache::invalidate_via_node doc.
self.coord_cache
.invalidate_via_node(our_identity.node_addr());
self.reset_discovery_backoff();
self.metrics().tree.parent_switched.inc();
self.reset_lookup_backoff();
self.metrics().tree.parent_switches.inc();
info!(
new_parent = %self.peer_display_name(&new_parent),
@@ -191,10 +253,12 @@ impl Node {
let all_peers: Vec<crate::NodeAddr> = self.peers.keys().copied().collect();
self.bloom_state.mark_all_updates_needed(all_peers);
} else if !self.tree_state.is_root() && self.tree_state.should_be_root() {
self.tree_state.become_root();
self.tree_state.become_root(now_secs);
// Clone identity once (see the parent-switch branch above for why).
let our_identity = self.identity().clone();
if let Err(e) = self.tree_state.sign_declaration(&our_identity) {
if let Err(e) =
sign_declaration(self.tree_state.my_declaration_mut(), &our_identity)
{
warn!(error = %e, "Failed to sign self-root declaration after first-RTT");
self.metrics()
.tree
@@ -204,8 +268,7 @@ impl Node {
// Surgical invalidation — see CoordCache::invalidate_other_roots doc.
self.coord_cache
.invalidate_other_roots(our_identity.node_addr());
self.reset_discovery_backoff();
self.metrics().tree.parent_switched.inc();
self.reset_lookup_backoff();
self.metrics().tree.parent_switches.inc();
info!(
new_root = %self.tree_state.root(),
@@ -223,65 +286,91 @@ impl Node {
///
/// Called from the tick handler. Also emits periodic operator logs.
pub(in crate::node) async fn check_mmp_reports(&mut self) {
let now = Instant::now();
let now_ms = crate::time::mono_ms();
// Collect peers that need reports (can't borrow self mutably while iterating)
let mut sender_reports: Vec<(NodeAddr, Vec<u8>)> = Vec::new();
let mut receiver_reports: Vec<(NodeAddr, Vec<u8>)> = Vec::new();
// Build one report-gating snapshot per peer, resolving every timing read
// shell-side into a `bool`. `send_sr`/`send_rr` are `true` on the master
// (IK) line — there is no profile negotiation here; the forward-merge to
// `-next` wires them to `peer.send_sr()`/`peer.send_rr()` (plan spot c).
// The snapshots own only `NodeAddr`/`MmpMode`/`bool`, so the
// peer-iteration borrow is released before the pure decision runs and the
// driving loop mutates the reporting state.
let snapshots: Vec<LinkReportSnapshot> = self
.peers
.iter()
.filter_map(|(node_addr, peer)| {
let mmp = peer.mmp()?;
Some(LinkReportSnapshot {
peer: *node_addr,
mode: mmp.mode(),
send_sr: true,
send_rr: true,
sr_due: mmp.sender.should_send_report(now_ms),
rr_due: mmp.receiver.should_send_report(now_ms),
log_due: mmp.should_log(now_ms),
})
})
.collect();
for (node_addr, peer) in self.peers.iter_mut() {
// Compute display name before taking mutable MMP borrow
let peer_name = self
.peer_aliases
.get(node_addr)
.cloned()
.unwrap_or_else(|| peer.identity().short_npub());
let actions = self.mmp.plan_link_reports(&snapshots);
let Some(mmp) = peer.mmp_mut() else {
continue;
};
let mode = mmp.mode();
// Sender reports: Full mode only
if mode == MmpMode::Full
&& mmp.sender.should_send_report(now)
&& let Some(sr) = mmp.sender.build_report(now)
{
sender_reports.push((*node_addr, sr.encode()));
}
// Receiver reports: Full and Lightweight modes
if mode != MmpMode::Minimal
&& mmp.receiver.should_send_report(now)
&& let Some(rr) = mmp.receiver.build_report(now)
{
receiver_reports.push((*node_addr, rr.encode()));
}
// Periodic operator logging
if mmp.should_log(now) {
Self::log_mmp_metrics(&peer_name, mmp);
mmp.mark_logged(now);
}
}
// Send collected reports
for (node_addr, encoded) in sender_reports {
if let Err(e) = self.send_encrypted_link_message(&node_addr, &encoded).await {
debug!(peer = %self.peer_display_name(&node_addr), error = %e, "Failed to send SenderReport");
}
}
for (node_addr, encoded) in receiver_reports {
if let Err(e) = self.send_encrypted_link_message(&node_addr, &encoded).await {
debug!(peer = %self.peer_display_name(&node_addr), error = %e, "Failed to send ReceiverReport");
// Drive the planned actions in their phase-grouped order (all logs, then
// all SenderReports, then all ReceiverReports). Logs run first because the
// operator log reads cumulative_packets_sent, which each report send
// advances (send_encrypted_link_message -> sender.record_sent); the
// pre-refactor handler logged during its collect pass, before any send.
// `build_report` (which advances the interval state) is called only on a
// SendLinkReport action, exactly as the pre-refactor gate did.
for action in actions {
match action {
MmpAction::SendLinkReport { peer, kind } => {
let encoded = self
.peers
.get_mut(&peer)
.and_then(|p| p.mmp_mut())
.and_then(|mmp| match kind {
LinkReportKind::Sender => {
mmp.sender.build_report(now_ms).map(|sr| sr.encode())
}
LinkReportKind::Receiver => {
mmp.receiver.build_report(now_ms).map(|rr| rr.encode())
}
});
if let Some(encoded) = encoded
&& let Err(e) = self.send_encrypted_link_message(&peer, &encoded).await
{
let label = match kind {
LinkReportKind::Sender => "Failed to send SenderReport",
LinkReportKind::Receiver => "Failed to send ReceiverReport",
};
debug!(peer = %self.peer_display_name(&peer), error = %e, "{}", label);
}
}
MmpAction::LogLink { peer } => {
// Resolve the display name exactly as the pre-refactor loop
// did (alias, else short_npub) — not `peer_display_name`,
// which also consults the host map.
let peer_name = self.peer_aliases.get(&peer).cloned().unwrap_or_else(|| {
self.peers
.get(&peer)
.map(|p| p.identity().short_npub())
.unwrap_or_default()
});
if let Some(mmp) = self.peers.get_mut(&peer).and_then(|p| p.mmp_mut()) {
Self::log_mmp_metrics(&peer_name, mmp);
mmp.mark_logged(now_ms);
}
}
MmpAction::ReapPeer { .. }
| MmpAction::Heartbeat { .. }
| MmpAction::SendSessionReport { .. }
| MmpAction::LogSession { .. } => {}
}
}
}
/// Emit periodic MMP metrics for a peer.
fn log_mmp_metrics(peer_name: &str, mmp: &crate::mmp::MmpPeerState) {
fn log_mmp_metrics(peer_name: &str, mmp: &crate::proto::mmp::MmpPeerState) {
let m = &mmp.metrics;
let rtt_str = if m.rtt_trend.initialized() {
@@ -309,7 +398,10 @@ impl Node {
}
/// Emit a teardown log summarizing lifetime MMP metrics for a removed peer.
pub(in crate::node) fn log_mmp_teardown(peer_name: &str, mmp: &crate::mmp::MmpPeerState) {
pub(in crate::node) fn log_mmp_teardown(
peer_name: &str,
mmp: &crate::proto::mmp::MmpPeerState,
) {
let m = &mmp.metrics;
let jitter_ms = mmp.receiver.jitter_us() as f64 / 1000.0;
@@ -334,205 +426,6 @@ impl Node {
);
}
// === Session-layer MMP ===
/// Check all sessions for pending MMP reports and send them.
///
/// Called from the tick handler. Also emits periodic session MMP logs.
/// Uses the collect-then-send pattern to avoid borrowing conflicts.
pub(in crate::node) async fn check_session_mmp_reports(&mut self) {
let now = Instant::now();
// Collect reports to send: (dest_addr, msg_type, encoded_body)
let mut reports: Vec<(NodeAddr, u8, Vec<u8>)> = Vec::new();
for (dest_addr, entry) in self.sessions.iter_mut() {
// Compute display name before taking mutable MMP borrow
let session_name = self
.peer_aliases
.get(dest_addr)
.cloned()
.unwrap_or_else(|| {
let (xonly, _) = entry.remote_pubkey().x_only_public_key();
crate::PeerIdentity::from_pubkey(xonly).short_npub()
});
let Some(mmp) = entry.mmp_mut() else {
continue;
};
let mode = mmp.mode();
// Sender reports: Full mode only
if mode == MmpMode::Full
&& mmp.sender.should_send_report(now)
&& let Some(sr) = mmp.sender.build_report(now)
{
let session_sr: SessionSenderReport = SessionSenderReport::from(&sr);
reports.push((
*dest_addr,
SessionMessageType::SenderReport.to_byte(),
session_sr.encode(),
));
}
// Receiver reports: Full and Lightweight modes
if mode != MmpMode::Minimal
&& mmp.receiver.should_send_report(now)
&& let Some(rr) = mmp.receiver.build_report(now)
{
let session_rr: SessionReceiverReport = SessionReceiverReport::from(&rr);
reports.push((
*dest_addr,
SessionMessageType::ReceiverReport.to_byte(),
session_rr.encode(),
));
}
// PathMtu notifications (all modes)
if mmp.path_mtu.should_send_notification(now)
&& let Some(mtu_value) = mmp.path_mtu.build_notification(now)
{
let notif = PathMtuNotification::new(mtu_value);
reports.push((
*dest_addr,
SessionMessageType::PathMtuNotification.to_byte(),
notif.encode(),
));
}
// Periodic operator logging
if mmp.should_log(now) {
Self::log_session_mmp_metrics(&session_name, mmp);
mmp.mark_logged(now);
}
}
// Send collected reports via session-layer encryption.
// Track per-destination success/failure for backoff and log suppression.
let mut send_results: Vec<(NodeAddr, bool)> = Vec::new();
for (dest_addr, msg_type, body) in reports {
match self.send_session_msg(&dest_addr, msg_type, &body).await {
Ok(()) => {
send_results.push((dest_addr, true));
}
Err(e) => {
// Peek at current failure count for log suppression
let failures = self
.sessions
.get(&dest_addr)
.and_then(|entry| entry.mmp())
.map(|mmp| mmp.sender.consecutive_send_failures())
.unwrap_or(0);
if failures < 3 {
debug!(
dest = %self.peer_display_name(&dest_addr),
msg_type,
error = %e,
"Failed to send session MMP report"
);
} else if failures == 3 {
debug!(
dest = %self.peer_display_name(&dest_addr),
"Suppressing further session MMP send failure logs"
);
}
// failures > 3: silently suppressed
send_results.push((dest_addr, false));
}
}
}
// Update backoff state from send results.
// Deduplicate: a destination counts as success if ANY report succeeded,
// failure only if ALL reports for that destination failed.
let mut dest_success: std::collections::HashMap<NodeAddr, bool> =
std::collections::HashMap::new();
for (dest, ok) in &send_results {
let entry = dest_success.entry(*dest).or_insert(false);
if *ok {
*entry = true;
}
}
for (dest_addr, success) in dest_success {
if let Some(entry) = self.sessions.get_mut(&dest_addr)
&& let Some(mmp) = entry.mmp_mut()
{
if success {
let prev = mmp.sender.record_send_success();
if prev > 3 {
debug!(
dest = %self.peer_display_name(&dest_addr),
consecutive_failures = prev,
"Resumed session MMP reporting"
);
}
} else {
mmp.sender.record_send_failure();
}
}
}
}
/// Emit periodic session MMP metrics.
fn log_session_mmp_metrics(session_name: &str, mmp: &MmpSessionState) {
let m = &mmp.metrics;
let rtt_str = if m.rtt_trend.initialized() {
format!("{:.1}ms", m.rtt_trend.long() / 1000.0)
} else {
"n/a".to_string()
};
let loss_str = if m.loss_trend.initialized() {
format!("{:.1}%", m.loss_trend.long() * 100.0)
} else {
"n/a".to_string()
};
let jitter_ms = mmp.receiver.jitter_us() as f64 / 1000.0;
debug!(
session = %session_name,
rtt = %rtt_str,
loss = %loss_str,
jitter = format_args!("{:.1}ms", jitter_ms),
goodput = %format_throughput(m.goodput_bps()),
mtu = mmp.path_mtu.last_observed_mtu(),
tx_pkts = mmp.sender.cumulative_packets_sent(),
rx_pkts = mmp.receiver.cumulative_packets_recv(),
"MMP session metrics"
);
}
/// Emit a teardown log summarizing lifetime session MMP metrics.
pub(in crate::node) fn log_session_mmp_teardown(session_name: &str, mmp: &MmpSessionState) {
let m = &mmp.metrics;
let jitter_ms = mmp.receiver.jitter_us() as f64 / 1000.0;
let rtt_str = match m.srtt_ms() {
Some(rtt) => format!("{:.1}ms", rtt),
None => "n/a".to_string(),
};
let loss_str = format!("{:.1}%", m.loss_rate() * 100.0);
debug!(
session = %session_name,
rtt = %rtt_str,
loss = %loss_str,
jitter = format_args!("{:.1}ms", jitter_ms),
etx = format_args!("{:.2}", m.etx),
goodput = %format_throughput(m.goodput_bps()),
send_mtu = mmp.path_mtu.current_mtu(),
observed_mtu = mmp.path_mtu.last_observed_mtu(),
tx_pkts = mmp.sender.cumulative_packets_sent(),
tx_bytes = mmp.sender.cumulative_bytes_sent(),
rx_pkts = mmp.receiver.cumulative_packets_recv(),
rx_bytes = mmp.receiver.cumulative_bytes_recv(),
"MMP session teardown"
);
}
/// Send heartbeats and remove dead peers.
///
/// Called from the tick handler. Sends a 1-byte heartbeat to each peer
@@ -540,83 +433,159 @@ impl Node {
/// hasn't sent us a frame within the link dead timeout.
pub(in crate::node) async fn check_link_heartbeats(&mut self) {
let now = Instant::now();
// Monotonic ms for the MMP receiver's injected-`u64` liveness clock; the
// Instant `now` is still used for the shell-owned heartbeat timing and
// the session-start fallback (both `ActivePeer` Instants).
let now_ms = crate::time::mono_ms();
let heartbeat_interval = Duration::from_secs(self.config().node.heartbeat_interval_secs);
let dead_timeout = Duration::from_secs(self.config().node.link_dead_timeout_secs);
let dead_timeout_ms = dead_timeout.as_millis() as u64;
let max_resends = self.config().node.rate_limit.handshake_max_resends;
let heartbeat_msg = [LinkMessageType::Heartbeat.to_byte()];
// Collect heartbeats to send and dead peers to remove
let mut heartbeats: Vec<NodeAddr> = Vec::new();
let mut dead_peers: Vec<NodeAddr> = Vec::new();
// Build one liveness snapshot per peer, resolving every clock read and
// the rekey-suppression predicate shell-side. The snapshots own only
// `NodeAddr`/`bool`, so the peer-iteration borrow is released before the
// pure decision runs and the driving loop mutates the registry.
let snapshots: Vec<PeerLivenessSnapshot> = self
.peers
.iter()
.map(|(node_addr, peer)| {
// Check liveness via the MMP receiver's last-received monotonic
// ms. Fall back to session_start (an `ActivePeer` Instant) for
// peers that never sent data, keeping that branch in Instant
// space so no monotonic-ms epoch conversion is needed.
let time_dead = if let Some(mmp) = peer.mmp() {
match mmp.receiver.last_recv_ms() {
Some(last_ms) => now_ms.saturating_sub(last_ms) >= dead_timeout_ms,
None => now.duration_since(peer.session_start()) >= dead_timeout,
}
} else {
false
};
for (node_addr, peer) in self.peers.iter() {
// Check liveness via MMP receiver last_recv_time.
// Fall back to session_start for peers that never sent data.
let time_dead = if let Some(mmp) = peer.mmp() {
let reference_time = mmp
.receiver
.last_recv_time()
.unwrap_or(peer.session_start());
now.duration_since(reference_time) >= dead_timeout
} else {
false
};
// Suppress teardown while an FMP rekey is genuinely in flight
// with budget left: a rekey-handshake link is not silent. The
// msg1 resend cap guarantees this terminates (abandon on
// exhaustion or cutover on completion clears
// `rekey_in_progress`), so a truly dead link is reaped on the
// next cycle.
let rekey_active = peer.rekey_in_progress()
&& peer.rekey_msg1_resend_count() < max_resends
&& peer.rekey_msg1().is_some();
// Suppress teardown while an FMP rekey is genuinely in flight with
// budget left: a rekey-handshake link is not silent. The msg1
// resend cap guarantees this terminates (abandon on exhaustion or
// cutover on completion clears `rekey_in_progress`), so a truly
// dead link is reaped on the next cycle.
let rekey_active = peer.rekey_in_progress()
&& peer.rekey_msg1_resend_count() < max_resends
&& peer.rekey_msg1().is_some();
// Check if heartbeat is due.
let heartbeat_due = match peer.last_heartbeat_sent() {
None => true,
Some(last) => now.duration_since(last) >= heartbeat_interval,
};
let is_dead = time_dead && !rekey_active;
if is_dead {
dead_peers.push(*node_addr);
continue;
}
PeerLivenessSnapshot {
peer: *node_addr,
time_dead,
rekey_active,
heartbeat_due,
}
})
.collect();
// Check if heartbeat is due
let needs_heartbeat = match peer.last_heartbeat_sent() {
None => true,
Some(last) => now.duration_since(last) >= heartbeat_interval,
};
if needs_heartbeat {
heartbeats.push(*node_addr);
}
}
let actions = self.mmp.plan_heartbeats(&snapshots);
// Remove dead peers and schedule auto-reconnect
// Wall-clock basis for reconnect scheduling, sourced once (as before).
let now_ms = std::time::SystemTime::now()
.duration_since(std::time::UNIX_EPOCH)
.map(|d| d.as_millis() as u64)
.unwrap_or(0);
for addr in &dead_peers {
debug!(
peer = %self.peer_display_name(addr),
timeout_secs = self.config().node.link_dead_timeout_secs,
"Removing peer: link dead timeout"
);
self.remove_active_peer(addr);
self.schedule_reconnect(*addr, now_ms);
}
// Send heartbeats (skip peers we just removed)
for addr in heartbeats {
if dead_peers.contains(&addr) {
continue;
}
if let Some(peer) = self.peers.get_mut(&addr) {
peer.mark_heartbeat_sent(now);
}
if let Err(e) = self
.send_encrypted_link_message(&addr, &heartbeat_msg)
.await
{
trace!(peer = %self.peer_display_name(&addr), error = %e, "Failed to send heartbeat");
// Drive the planned actions: all reaps first (each removed +
// reconnect-scheduled), then all heartbeats (a just-reaped peer is never
// heartbeated — the core never emits both for the same peer).
for action in actions {
match action {
MmpAction::ReapPeer { peer } => {
// Log SHELL-SIDE before routing so the reap keeps the
// `fips::node::handlers::mmp` tracing target (no relocation into
// the executor, no target pin needed).
debug!(
peer = %self.peer_display_name(&peer),
timeout_secs = self.config().node.link_dead_timeout_secs,
"Removing peer: link dead timeout"
);
self.route_link_dead(peer, now_ms).await;
}
MmpAction::Heartbeat { peer } => {
if let Some(p) = self.peers.get_mut(&peer) {
p.mark_heartbeat_sent(now);
}
if let Err(e) = self
.send_encrypted_link_message(&peer, &heartbeat_msg)
.await
{
trace!(peer = %self.peer_display_name(&peer), error = %e, "Failed to send heartbeat");
}
}
MmpAction::SendLinkReport { .. }
| MmpAction::LogLink { .. }
| MmpAction::SendSessionReport { .. }
| MmpAction::LogSession { .. } => {}
}
}
}
/// Route a link-dead liveness reap through the peer machine + executor.
/// Mirrors [`route_rekey_cadence`](Node::route_rekey_cadence): the
/// shell already decided (the tick sweep's `plan_heartbeats` batch emitted
/// this `ReapPeer` in phase order), so the machine only CONSUMES the decision
/// via [`PeerEvent::LinkDeadSuspected`]. The resulting executor arms
/// (`InvalidateSendState` → `remove_active_peer`, `ReportLost` →
/// `note_link_dead`) reproduce the pre-refactor inline reap body exactly.
///
/// An established peer always has a `peer_machine`. If the peer vanished
/// between snapshot and effect, the old inline body was already a
/// no-op, so we return; if the machine is absent (which should be impossible)
/// we fall back to the byte-identical inline body under a `debug_assert`.
///
/// `now_ms` is the sweep's hoisted wall-clock ms (the same value the old reap
/// fed `note_link_dead`); it flows to the executor `ReportLost` arm via
/// `ambient.now_ms`.
async fn route_link_dead(&mut self, node_addr: NodeAddr, now_ms: u64) {
let link = match self.peers.get(&node_addr) {
Some(peer) => peer.link_id(),
None => return,
};
if !self.peer_machines.contains_key(&link) {
debug_assert!(false, "peer machine present for every established peer");
self.remove_active_peer(&node_addr);
self.note_link_dead(node_addr, now_ms);
return;
}
let ambient = self.link_dead_ctx(&node_addr, now_ms);
self.advance_peer_machine(link, PeerEvent::LinkDeadSuspected, Self::now_ms(), &ambient)
.await;
}
/// Ambient shell facts for the routed liveness reap. Mirrors
/// [`rekey_cadence_ctx`](Node::rekey_cadence_ctx). The executor reads only
/// `verified_identity` (`InvalidateSendState` → `remove_active_peer` resolves
/// its `NodeAddr` from it, so it must equal `node_addr`) and `now_ms`
/// (`ReportLost` → `note_link_dead`, the wall-clock reconnect basis). The
/// transport/index/direction fields are unused by these two arms and are
/// populated best-effort for coherence. `now_ms` is threaded in (rather than
/// re-read) so the value fed to `note_link_dead` is byte-identical to the old
/// reap's hoisted wall-clock for every peer in the sweep.
fn link_dead_ctx(&self, node_addr: &NodeAddr, now_ms: u64) -> PeerActionCtx {
let peer = &self.peers[node_addr];
PeerActionCtx {
verified_identity: *peer.identity(),
transport_id: peer.transport_id().unwrap_or_else(|| TransportId::new(0)),
remote_addr: peer
.current_addr()
.cloned()
.unwrap_or_else(|| TransportAddr::new(Vec::new())),
our_index: peer.our_index(),
their_index: peer.their_index(),
now_ms,
is_outbound: false,
}
}
}
src/node/handlers/mod.rs
+2 -8
View File
@@ -1,14 +1,8 @@
//! RX event loop and message handlers.
//! Message handlers: per-message-type behavior on `impl Node`.
#[cfg(unix)]
pub(crate) mod connected_udp;
pub(crate) mod discovery;
mod dispatch;
mod encrypted;
mod forwarding;
mod handshake;
pub(crate) mod lookup;
mod mmp;
mod rekey;
mod rx_loop;
pub(in crate::node) mod session;
mod timeout;
src/node/handlers/rekey.rs
+407 -307
View File
@@ -7,28 +7,28 @@
use crate::NodeAddr;
use crate::node::Node;
use crate::node::wire::build_msg1;
use crate::node::dataplane::PeerActionCtx;
use crate::noise::HandshakeState;
use crate::protocol::{SessionDatagram, SessionSetup};
use crate::peer::machine::PeerEvent;
use crate::proto::fmp::wire::build_msg1;
use crate::proto::fmp::{ConnAction, LifecycleView, PeerSnapshot, RekeyCfg, RekeyResendSnapshot};
use crate::proto::fsp::{
FspAction, RekeyMsg3ResendSnapshot, SessionSetup, SessionSnapshot, cutover_timer_elapsed,
};
use crate::proto::link::SessionDatagram;
use crate::transport::{TransportAddr, TransportId};
use tracing::{debug, trace, warn};
/// Keep previous session alive for this long after cutover.
///
/// FMP-scoped copy for `check_rekey`; the FSP session-rekey timing bounds live
/// in `crate::proto::fsp::limits`.
const DRAIN_WINDOW_SECS: u64 = 10;
/// Suppress local rekey initiation for this long after receiving
/// a peer's rekey msg1.
/// a peer's rekey msg1. FMP-scoped copy for `check_rekey`.
const REKEY_DAMPENING_SECS: u64 = 30;
/// Liveness bound on how long the FSP rekey initiator holds the
/// `current` + `pending` state before cutting over to the new epoch.
///
/// This is NOT safety-critical: overlapping-epoch trial-decrypt covers
/// any skew between the two endpoints' cutovers. The timer only bounds
/// how long the initiator advertises the old K-bit. An opportunistic
/// early cutover also fires if the initiator authenticates a peer frame
/// against its own `pending` session (the responder cut over first).
const FSP_CUTOVER_DELAY_MS: u64 = 2000;
impl Node {
/// Periodic rekey check. Called from the tick loop.
///
@@ -41,121 +41,225 @@ impl Node {
return;
}
let rekey_after_secs = self.config().node.rekey.after_secs;
let rekey_after_messages = self.config().node.rekey.after_messages;
let cfg = RekeyCfg {
after_secs: self.config().node.rekey.after_secs,
after_messages: self.config().node.rekey.after_messages,
};
// Collect peers that need action (to avoid borrow conflicts)
let mut peers_to_cutover: Vec<NodeAddr> = Vec::new();
let mut peers_to_drain: Vec<NodeAddr> = Vec::new();
let mut peers_to_rekey: Vec<NodeAddr> = Vec::new();
for (node_addr, peer) in &self.peers {
if !peer.has_session() || !peer.is_healthy() {
continue;
}
// 1. Initiator-side cutover: we completed a rekey and have
// a pending session ready. Cut over on the next tick.
if peer.pending_new_session().is_some() && !peer.rekey_in_progress() {
peers_to_cutover.push(*node_addr);
continue;
}
// 2. Drain window expiry
if peer.is_draining() && peer.drain_expired(DRAIN_WINDOW_SECS) {
peers_to_drain.push(*node_addr);
}
// 3. Rekey trigger
if peer.rekey_in_progress() {
continue;
}
if peer.is_rekey_dampened(REKEY_DAMPENING_SECS) {
continue;
}
let elapsed = peer.session_established_at().elapsed().as_secs();
let counter = peer
.noise_session()
.map(|s| s.current_send_counter())
.unwrap_or(0);
// Apply per-session symmetric jitter to desynchronize
// dual-initiation in symmetric-start meshes.
let effective_after_secs =
rekey_after_secs.saturating_add_signed(peer.rekey_jitter_secs());
if elapsed >= effective_after_secs || counter >= rekey_after_messages {
peers_to_rekey.push(*node_addr);
// The shell snapshots each healthy peer's rekey ages/flags (every clock
// read resolved here); the core decides cutover/drain/trigger with no
// clock, phase-grouped to preserve the pre-refactor execution order.
// The batch `poll_rekey` + snapshots STAY SHELL-SIDE and BYTE-UNCHANGED:
// the cross-peer phase-grouping (all Cutover → all Drain →
// all InitiateRekey) governs the shared `index_allocator` free-then-alloc
// SEQUENCE that appears on the wire. The machine must NOT re-poll; it
// CONSUMES each decided `ConnAction` in the same order the batch returned.
let snapshots = self.rekey_peers();
for action in self.fmp.poll_rekey(snapshots, &cfg) {
match action {
// Initiator cutover: route the decided action through the peer
// machine + executor. The executor's `SwapSendState` arm
// reproduces the pre-refactor cutover body EXACTLY.
ConnAction::Cutover { peer: node_addr } => {
self.route_rekey_cadence(node_addr, ConnAction::Cutover { peer: node_addr })
.await;
}
// Drain completion: route through the machine + executor. The
// executor's `CompleteDrain` arm reads the REAL previous index
// from `complete_drain()` and frees it at the same point the old
// inline body did (index-order preserving).
ConnAction::Drain { peer: node_addr } => {
self.route_rekey_cadence(node_addr, ConnAction::Drain { peer: node_addr })
.await;
}
// Initiate a new rekey: STAYS INLINE (the Noise msg1 build +
// index allocation are a shell-side leaf, byte-unchanged). Feed
// the machine a `RekeyInitiated` observation afterward so its
// control state stays coherent for the next tick's Cutover/Drain.
ConnAction::InitiateRekey { peer: node_addr } => {
self.initiate_rekey(&node_addr).await;
self.observe_rekey_initiated(&node_addr);
}
#[allow(unreachable_patterns)]
_ => {}
}
}
}
// Execute cutover for initiator side
for node_addr in peers_to_cutover {
let did_cutover = if let Some(peer) = self.peers.get_mut(&node_addr) {
if let Some(_old_our_index) = peer.cutover_to_new_session() {
// New index was pre-registered in peers_by_index
// during msg2 handling (handshake.rs).
debug_assert!(
peer.transport_id().is_some()
&& peer.our_index().is_some()
&& self.peers_by_index.contains_key(&(
peer.transport_id().unwrap(),
peer.our_index().unwrap().as_u32()
)),
"peers_by_index should contain pre-registered new index after cutover"
);
debug!(
peer = %self.peer_display_name(&node_addr),
"Rekey cutover complete (initiator), K-bit flipped"
);
true
} else {
false
}
/// Route a cadence-decided `Cutover`/`Drain` `ConnAction` through the peer
/// machine + executor. The shell already decided (batch `poll_rekey`);
/// the machine consumes via [`PeerEvent::RekeyConsume`] WITHOUT re-polling,
/// preserving the phase order. The `SwapSendState`/`CompleteDrain` executor
/// arms reproduce the pre-refactor inline effect bodies exactly.
///
/// An established peer always has a `peer_machine`. If the peer vanished
/// between snapshot and effect, the old inline body was a no-op, so we do
/// nothing; if the machine is absent (which should be impossible) we fall
/// back to the byte-identical inline body under a `debug_assert`.
async fn route_rekey_cadence(&mut self, node_addr: NodeAddr, action: ConnAction) {
let link = match self.peers.get(&node_addr) {
Some(peer) => peer.link_id(),
None => return,
};
if !self.peer_machines.contains_key(&link) {
debug_assert!(
false,
"peer machine present for every established rekey peer"
);
match action {
ConnAction::Cutover { peer } => self.cutover_peer_inline(&peer),
ConnAction::Drain { peer } => self.drain_peer_inline(&peer),
_ => {}
}
return;
}
let ambient = self.rekey_cadence_ctx(&node_addr);
self.advance_peer_machine(
link,
PeerEvent::RekeyConsume { action },
ambient.now_ms,
&ambient,
)
.await;
}
/// Feed the machine the `RekeyInitiated` observation after the inline
/// `initiate_rekey`. The obs emits no action, so there is no executor
/// pass — a bare `step` keeps the machine's control state coherent.
fn observe_rekey_initiated(&mut self, node_addr: &NodeAddr) {
let link = match self.peers.get(node_addr) {
Some(peer) => peer.link_id(),
None => return,
};
if let Some(machine) = self.peer_machines.get_mut(&link) {
let acts = machine.step(
PeerEvent::RekeyInitiated,
Self::now_ms(),
&mut self.index_allocator,
);
debug_assert!(acts.is_empty(), "RekeyInitiated is a pure observation");
} else {
debug_assert!(
false,
"peer machine present for every established rekey peer"
);
}
}
/// Ambient shell facts for the routed cadence Cutover/Drain step. Only
/// `verified_identity` is read by the `SwapSendState`/`CompleteDrain`
/// executor arms — `SwapSendState` resolves its `NodeAddr` from it (so it must
/// equal `node_addr`), and `CompleteDrain` carries its peer in the action
/// payload. The transport/index/direction fields are unused by these two arms
/// (they matter only to `PromoteToActive`, never emitted on this path) and are
/// populated best-effort for coherence.
fn rekey_cadence_ctx(&self, node_addr: &NodeAddr) -> PeerActionCtx {
let peer = &self.peers[node_addr];
PeerActionCtx {
verified_identity: *peer.identity(),
transport_id: peer.transport_id().unwrap_or_else(|| TransportId::new(0)),
remote_addr: peer
.current_addr()
.cloned()
.unwrap_or_else(|| TransportAddr::new(Vec::new())),
our_index: peer.our_index(),
their_index: peer.their_index(),
now_ms: Self::now_ms(),
is_outbound: false,
}
}
/// Pre-refactor initiator cutover body, retained as the release fallback for
/// the (should-be-impossible) missing-machine case. Byte-identical to the old
/// inline `ConnAction::Cutover` arm and to the executor's `SwapSendState` arm.
fn cutover_peer_inline(&mut self, node_addr: &NodeAddr) {
let did_cutover = if let Some(peer) = self.peers.get_mut(node_addr) {
if let Some(_old_our_index) = peer.cutover_to_new_session() {
// New index was pre-registered in peers_by_index during msg2
// handling (handshake.rs).
debug_assert!(
peer.transport_id().is_some()
&& peer.our_index().is_some()
&& self.peers_by_index.contains_key(&(
peer.transport_id().unwrap(),
peer.our_index().unwrap().as_u32()
)),
"peers_by_index should contain pre-registered new index after cutover"
);
debug!(
peer = %self.peer_display_name(node_addr),
"Rekey cutover complete (initiator), K-bit flipped"
);
true
} else {
false
};
// Re-register the new session with the decrypt worker — the
// cache_key (transport_id, our_index) just changed, so the
// old worker entry is stale and every packet on the new
// session would miss the worker's HashMap lookup.
#[cfg(unix)]
if did_cutover {
self.register_decrypt_worker_session(&node_addr);
}
#[cfg(not(unix))]
let _ = did_cutover;
} else {
false
};
// Re-register the new session with the decrypt worker — the cache_key
// (transport_id, our_index) just changed, so the old worker entry is
// stale and every packet on the new session would miss the lookup.
#[cfg(unix)]
if did_cutover {
self.register_decrypt_worker_session(node_addr);
}
#[cfg(not(unix))]
let _ = did_cutover;
}
// Execute drain completion
for node_addr in peers_to_drain {
// Extract the old index and transport_id under the peer
// borrow, then drop the borrow so the cache_key cleanup
// below can take &mut self for unregister_decrypt_worker_session.
let drained = self
.peers
.get_mut(&node_addr)
.and_then(|peer| peer.complete_drain().map(|idx| (idx, peer.transport_id())));
if let Some((old_our_index, transport_id)) = drained {
if let Some(tid) = transport_id {
let cache_key = (tid, old_our_index.as_u32());
self.peers_by_index.remove(&cache_key);
#[cfg(unix)]
self.unregister_decrypt_worker_session(cache_key);
}
let _ = self.index_allocator.free(old_our_index);
trace!(
peer = %self.peer_display_name(&node_addr),
old_index = %old_our_index,
"Drain complete, previous session erased"
);
/// Pre-refactor drain-completion body, retained as the release fallback for
/// the (should-be-impossible) missing-machine case. Byte-identical to the old
/// inline `ConnAction::Drain` arm and to the executor's `CompleteDrain` arm.
fn drain_peer_inline(&mut self, node_addr: &NodeAddr) {
// Extract the old index and transport_id under the peer borrow, then drop
// the borrow so the cache_key cleanup below can take &mut self for
// unregister_decrypt_worker_session.
let drained = self
.peers
.get_mut(node_addr)
.and_then(|peer| peer.complete_drain().map(|idx| (idx, peer.transport_id())));
if let Some((old_our_index, transport_id)) = drained {
if let Some(tid) = transport_id {
let cache_key = (tid, old_our_index.as_u32());
self.peers_by_index.remove(&cache_key);
#[cfg(unix)]
self.unregister_decrypt_worker_session(cache_key);
}
let _ = self.index_allocator.free(old_our_index);
trace!(
peer = %self.peer_display_name(node_addr),
old_index = %old_our_index,
"Drain complete, previous session erased"
);
}
}
// Initiate new rekeys
for node_addr in peers_to_rekey {
self.initiate_rekey(&node_addr).await;
}
/// Snapshot every healthy peer with a session for the rekey decision,
/// pre-computing its monotonic ages and timer predicates so the pure core
/// applies the thresholds without reading a clock (see [`PeerSnapshot`]).
///
/// Lives here, beside the drain/dampening constants and the FSP analog, so
/// the forward-merge onto `next` reconciles rekey timing in one place.
pub(in crate::node) fn rekey_peer_snapshots(&self) -> Vec<PeerSnapshot> {
self.peers
.iter()
.filter(|(_, peer)| peer.has_session() && peer.is_healthy())
.map(|(node_addr, peer)| PeerSnapshot {
addr: *node_addr,
has_pending: peer.pending_new_session().is_some(),
rekey_in_progress: peer.rekey_in_progress(),
is_draining: peer.is_draining(),
drain_expired: peer.drain_expired(DRAIN_WINDOW_SECS),
is_dampened: peer.is_rekey_dampened(REKEY_DAMPENING_SECS),
elapsed_secs: peer.session_established_at().elapsed().as_secs(),
counter: peer
.noise_session()
.map(|s| s.current_send_counter())
.unwrap_or(0),
jitter_secs: peer.rekey_jitter_secs(),
})
.collect()
}
/// Initiate an outbound rekey to a peer.
@@ -260,60 +364,74 @@ impl Node {
let backoff = self.config().node.rate_limit.handshake_resend_backoff;
let max_resends = self.config().node.rate_limit.handshake_max_resends;
// Collect peers needing action
let mut to_resend: Vec<(NodeAddr, Vec<u8>)> = Vec::new();
let mut to_abandon: Vec<NodeAddr> = Vec::new();
// The shell snapshots each in-flight rekey (resend-due predicate
// resolved here); the core classifies abandon-vs-resend and computes
// the backoff, abandons first.
let candidates = self.rekey_resend_candidates(now_ms);
for action in
self.fmp
.poll_rekey_resends(candidates, now_ms, interval_ms, backoff, max_resends)
{
match action {
// Abandon rekey cycles that exhausted their retransmission budget.
ConnAction::AbandonRekey { peer: node_addr } => {
if let Some(peer) = self.peers.get_mut(&node_addr) {
peer.abandon_rekey();
}
debug!(
peer = %self.peer_display_name(&node_addr),
"FMP rekey aborted: msg1 unconfirmed after max retransmissions, abandoning cycle"
);
}
ConnAction::ResendRekeyMsg1 {
peer: node_addr,
bytes,
next_resend_at_ms,
} => {
let (transport_id, remote_addr) = match self.peers.get(&node_addr) {
Some(p) => match (p.transport_id(), p.current_addr()) {
(Some(tid), Some(addr)) => (tid, addr.clone()),
_ => continue,
},
None => continue,
};
for (node_addr, peer) in &self.peers {
if !peer.rekey_in_progress() || peer.rekey_msg1().is_none() {
continue;
}
if peer.rekey_msg1_resend_count() >= max_resends {
to_abandon.push(*node_addr);
continue;
}
if peer.needs_msg1_resend(now_ms) {
to_resend.push((*node_addr, peer.rekey_msg1().unwrap().to_vec()));
let sent = if let Some(transport) = self.transports.get(&transport_id) {
transport.send(&remote_addr, &bytes).await.is_ok()
} else {
false
};
if sent && let Some(peer) = self.peers.get_mut(&node_addr) {
peer.record_rekey_msg1_resend(next_resend_at_ms);
let count = peer.rekey_msg1_resend_count();
trace!(
peer = %self.peer_display_name(&node_addr),
resend = count,
"Resent rekey msg1"
);
}
}
#[allow(unreachable_patterns)]
_ => {}
}
}
}
// Abandon rekey cycles that exhausted their retransmission budget.
for node_addr in to_abandon {
if let Some(peer) = self.peers.get_mut(&node_addr) {
peer.abandon_rekey();
}
debug!(
peer = %self.peer_display_name(&node_addr),
"FMP rekey aborted: msg1 unconfirmed after max retransmissions, abandoning cycle"
);
}
for (node_addr, msg1_bytes) in to_resend {
let (transport_id, remote_addr) = match self.peers.get(&node_addr) {
Some(p) => match (p.transport_id(), p.current_addr()) {
(Some(tid), Some(addr)) => (tid, addr.clone()),
_ => continue,
},
None => continue,
};
let sent = if let Some(transport) = self.transports.get(&transport_id) {
transport.send(&remote_addr, &msg1_bytes).await.is_ok()
} else {
false
};
if sent && let Some(peer) = self.peers.get_mut(&node_addr) {
let count = peer.rekey_msg1_resend_count() + 1;
let next = now_ms + (interval_ms as f64 * backoff.powi(count as i32)) as u64;
peer.record_rekey_msg1_resend(next);
trace!(
peer = %self.peer_display_name(&node_addr),
resend = count,
"Resent rekey msg1"
);
}
}
/// Snapshot every peer with a rekey handshake in flight (and a stored
/// msg1) for the retransmission decision, pre-evaluating the resend-due
/// predicate against `now_ms` so the core reads no clock.
pub(in crate::node) fn rekey_resend_snapshots(&self, now_ms: u64) -> Vec<RekeyResendSnapshot> {
self.peers
.iter()
.filter(|(_, peer)| peer.rekey_in_progress() && peer.rekey_msg1().is_some())
.map(|(node_addr, peer)| RekeyResendSnapshot {
peer: *node_addr,
resend_count: peer.rekey_msg1_resend_count(),
needs_resend: peer.needs_msg1_resend(now_ms),
msg1: peer.rekey_msg1().unwrap().to_vec(),
})
.collect()
}
/// Retransmit FSP rekey msg3 until the responder is confirmed on the
@@ -352,66 +470,77 @@ impl Node {
let ttl = self.config().node.session.default_ttl;
let my_addr = *self.node_addr();
// Collect rekey initiators whose msg3 retransmission is due.
let mut to_resend: Vec<(NodeAddr, Vec<u8>)> = Vec::new();
let mut to_abandon: Vec<NodeAddr> = Vec::new();
for (node_addr, entry) in &self.sessions {
// Only the rekey initiator retains a msg3 payload.
let payload = match entry.rekey_msg3_payload() {
Some(p) => p,
None => continue,
};
if entry.rekey_msg3_next_resend_ms() == 0 || now_ms < entry.rekey_msg3_next_resend_ms()
{
continue;
}
if entry.rekey_msg3_resend_count() >= max_resends {
to_abandon.push(*node_addr);
continue;
}
to_resend.push((*node_addr, payload.to_vec()));
}
// Abandon rekey cycles that exhausted their retransmission budget.
for node_addr in to_abandon {
if let Some(entry) = self.sessions.get_mut(&node_addr) {
entry.abandon_rekey();
}
debug!(
peer = %self.peer_display_name(&node_addr),
"FSP rekey aborted: msg3 unconfirmed after max retransmissions, abandoning cycle"
);
}
// Retransmit msg3 for cycles still within budget.
for (node_addr, payload) in to_resend {
let mut datagram = SessionDatagram::new(my_addr, node_addr, payload).with_ttl(ttl);
let sent = match self.send_session_datagram(&mut datagram).await {
Ok(_) => true,
Err(e) => {
// The shell snapshots each session retaining a msg3 payload (resend-due
// predicate resolved here); the core classifies abandon-vs-resend,
// abandons first.
let candidates = self.rekey_msg3_resend_snapshots(now_ms);
for action in self.fsp.poll_rekey_msg3_resends(candidates, max_resends) {
match action {
FspAction::AbandonRekey { addr } => {
if let Some(entry) = self.sessions.get_mut(&addr) {
entry.abandon_rekey();
}
debug!(
peer = %self.peer_display_name(&node_addr),
error = %e,
"FSP rekey msg3 retransmission failed"
peer = %self.peer_display_name(&addr),
"FSP rekey aborted: msg3 unconfirmed after max retransmissions, abandoning cycle"
);
false
}
};
FspAction::ResendSessionMsg3 { addr } => {
let payload = match self
.sessions
.get(&addr)
.and_then(|e| e.rekey_msg3_payload())
{
Some(p) => p.to_vec(),
None => continue,
};
let mut datagram = SessionDatagram::new(my_addr, addr, payload).with_ttl(ttl);
let sent = match self.send_session_datagram(&mut datagram).await {
Ok(_) => true,
Err(e) => {
debug!(
peer = %self.peer_display_name(&addr),
error = %e,
"FSP rekey msg3 retransmission failed"
);
false
}
};
if sent && let Some(entry) = self.sessions.get_mut(&node_addr) {
let count = entry.rekey_msg3_resend_count() + 1;
let next = now_ms + (interval_ms as f64 * backoff.powi(count as i32)) as u64;
entry.record_rekey_msg3_resend(next);
trace!(
peer = %self.peer_display_name(&node_addr),
resend = count,
"Resent FSP rekey msg3"
);
if sent && let Some(entry) = self.sessions.get_mut(&addr) {
let count = entry.rekey_msg3_resend_count() + 1;
let next =
now_ms + (interval_ms as f64 * backoff.powi(count as i32)) as u64;
entry.record_rekey_msg3_resend(next);
trace!(
peer = %self.peer_display_name(&addr),
resend = count,
"Resent FSP rekey msg3"
);
}
}
#[allow(unreachable_patterns)]
_ => {}
}
}
}
/// Snapshot every session retaining a rekey-msg3 payload for the
/// retransmission decision, pre-evaluating the resend-due predicate against
/// `now_ms` so the core reads no clock.
fn rekey_msg3_resend_snapshots(&self, now_ms: u64) -> Vec<RekeyMsg3ResendSnapshot> {
self.sessions
.iter()
.filter(|(_, entry)| entry.rekey_msg3_payload().is_some())
.map(|(node_addr, entry)| RekeyMsg3ResendSnapshot {
addr: *node_addr,
resend_count: entry.rekey_msg3_resend_count(),
resend_due: entry.rekey_msg3_next_resend_ms() != 0
&& now_ms >= entry.rekey_msg3_next_resend_ms(),
})
.collect()
}
/// Periodic session (FSP) rekey check. Called from the tick loop.
///
/// For each established session:
@@ -429,100 +558,71 @@ impl Node {
return;
}
let rekey_after_secs = self.config().node.rekey.after_secs;
let rekey_after_messages = self.config().node.rekey.after_messages;
let cfg = crate::proto::fsp::RekeyCfg {
after_secs: self.config().node.rekey.after_secs,
after_messages: self.config().node.rekey.after_messages,
};
let now_ms = Self::now_ms();
let drain_ms = DRAIN_WINDOW_SECS * 1000;
let dampening_ms = REKEY_DAMPENING_SECS * 1000;
let mut sessions_to_cutover: Vec<NodeAddr> = Vec::new();
let mut sessions_to_drain: Vec<NodeAddr> = Vec::new();
let mut sessions_to_rekey: Vec<NodeAddr> = Vec::new();
for (node_addr, entry) in &self.sessions {
if !entry.is_established() {
continue;
}
// 1. Initiator-side cutover (option A): completed rekey,
// pending session ready, liveness timer elapsed. This is
// an unconditional timer, NOT gated on responder progress —
// overlapping-epoch trial-decrypt covers the cutover skew,
// so flipping the K-bit here is always safe. An
// opportunistic early cutover also happens in
// `handle_encrypted_session_msg` if the initiator
// authenticates a peer frame against its own `pending`.
if entry.pending_new_session().is_some()
&& !entry.has_rekey_in_progress()
&& entry.is_rekey_initiator()
&& now_ms.saturating_sub(entry.rekey_completed_ms()) >= FSP_CUTOVER_DELAY_MS
{
sessions_to_cutover.push(*node_addr);
continue;
}
// 2. Drain window expiry
if entry.is_draining() && entry.drain_expired(now_ms, drain_ms) {
sessions_to_drain.push(*node_addr);
}
// 3. Rekey trigger
if entry.has_rekey_in_progress() {
continue;
}
if entry.pending_new_session().is_some() {
continue; // Pending session present, awaiting cutover
}
if entry.rekey_msg3_payload().is_some() {
// Initiator already cut over on its liveness timer but is
// still retransmitting msg3 to a responder not yet
// confirmed on the new epoch. Don't start another rekey
// until the current cycle's msg3 is delivered or abandoned.
continue;
}
if entry.is_rekey_dampened(now_ms, dampening_ms) {
continue;
}
let elapsed_secs = now_ms.saturating_sub(entry.session_start_ms()) / 1000;
let counter = entry.send_counter();
// Apply per-session symmetric jitter to desynchronize
// dual-initiation in symmetric-start meshes.
let effective_after_secs =
rekey_after_secs.saturating_add_signed(entry.rekey_jitter_secs());
if elapsed_secs >= effective_after_secs || counter >= rekey_after_messages {
sessions_to_rekey.push(*node_addr);
// The shell snapshots each established session's rekey ages/flags
// (every clock read resolved here); the core decides
// cutover/drain/trigger with no clock, phase-grouped to preserve the
// pre-refactor execution order.
let snapshots = self.session_rekey_snapshots(now_ms);
for action in self.fsp.poll_rekey(snapshots, &cfg) {
match action {
FspAction::CutOver { addr } => {
if let Some(entry) = self.sessions.get_mut(&addr)
&& entry.cutover_to_new_session(now_ms)
{
debug!(
peer = %self.peer_display_name(&addr),
"FSP rekey cutover complete (initiator), K-bit flipped"
);
}
}
FspAction::CompleteDrain { addr } => {
if let Some(entry) = self.sessions.get_mut(&addr) {
entry.complete_drain();
trace!(
peer = %self.peer_display_name(&addr),
"FSP drain complete, previous session erased"
);
}
}
FspAction::InitiateRekey { addr } => {
self.initiate_session_rekey(&addr).await;
}
#[allow(unreachable_patterns)]
_ => {}
}
}
}
// Execute cutover for initiator side
for node_addr in sessions_to_cutover {
if let Some(entry) = self.sessions.get_mut(&node_addr)
&& entry.cutover_to_new_session(now_ms)
{
debug!(
peer = %self.peer_display_name(&node_addr),
"FSP rekey cutover complete (initiator), K-bit flipped"
);
}
}
// Execute drain completion
for node_addr in sessions_to_drain {
if let Some(entry) = self.sessions.get_mut(&node_addr) {
entry.complete_drain();
trace!(
peer = %self.peer_display_name(&node_addr),
"FSP drain complete, previous session erased"
);
}
}
// Initiate new rekeys
for node_addr in sessions_to_rekey {
self.initiate_session_rekey(&node_addr).await;
}
/// Snapshot every established session for the FSP rekey decision,
/// pre-computing its monotonic age and timer predicates so the pure core
/// applies the thresholds without reading a clock (see [`SessionSnapshot`]).
fn session_rekey_snapshots(&self, now_ms: u64) -> Vec<SessionSnapshot> {
let drain_ms = crate::proto::fsp::limits::DRAIN_WINDOW_SECS * 1000;
let dampening_ms = crate::proto::fsp::limits::REKEY_DAMPENING_SECS * 1000;
self.sessions
.iter()
.filter(|(_, entry)| entry.is_established())
.map(|(node_addr, entry)| SessionSnapshot {
addr: *node_addr,
has_pending: entry.pending_new_session().is_some(),
rekey_in_progress: entry.has_rekey_in_progress(),
is_rekey_initiator: entry.is_rekey_initiator(),
cutover_timer_elapsed: cutover_timer_elapsed(now_ms, entry.rekey_completed_ms()),
is_draining: entry.is_draining(),
drain_expired: entry.drain_expired(now_ms, drain_ms),
has_rekey_msg3_payload: entry.rekey_msg3_payload().is_some(),
is_dampened: entry.is_rekey_dampened(now_ms, dampening_ms),
elapsed_secs: now_ms.saturating_sub(entry.session_start_ms()) / 1000,
counter: entry.send_counter(),
jitter_secs: entry.rekey_jitter_secs(),
})
.collect()
}
/// Initiate an FSP session rekey.
src/node/handlers/session.rs
+397 -154
View File
@@ -6,34 +6,39 @@
//! encrypted data, and error signals (CoordsRequired, PathBroken).
use crate::NodeAddr;
use crate::mmp::report::ReceiverReport;
use crate::mmp::{MAX_SESSION_REPORT_INTERVAL_MS, MIN_SESSION_REPORT_INTERVAL_MS};
use crate::node::handlers::mmp::format_throughput;
use crate::node::reject::{RejectReason, SessionReject};
use crate::node::session::{EndToEndState, EpochSlot, SessionEntry};
use crate::node::session_wire::{
FSP_COMMON_PREFIX_SIZE, FSP_FLAG_CP, FSP_FLAG_K, FSP_HEADER_SIZE, FSP_PHASE_ESTABLISHED,
FSP_PHASE_MSG1, FSP_PHASE_MSG2, FSP_PHASE_MSG3, FSP_PORT_HEADER_SIZE, FSP_PORT_IPV6_SHIM,
FspCommonPrefix, FspEncryptedHeader, build_fsp_header, fsp_prepend_inner_header,
fsp_strip_inner_header, parse_encrypted_coords,
};
#[cfg(unix)]
use crate::node::wire::{
ESTABLISHED_HEADER_SIZE, FLAG_KEY_EPOCH, FLAG_SP, build_established_header,
};
use crate::node::{Node, NodeError};
use crate::noise::{
HandshakeState, XK_HANDSHAKE_MSG1_SIZE, XK_HANDSHAKE_MSG2_SIZE, XK_HANDSHAKE_MSG3_SIZE,
};
#[cfg(unix)]
use crate::protocol::LinkMessageType;
#[cfg(unix)]
use crate::protocol::SESSION_DATAGRAM_HEADER_SIZE;
use crate::protocol::{
CoordsRequired, FspInnerFlags, MtuExceeded, PathBroken, PathMtuNotification, SessionAck,
SessionDatagram, SessionMessageType, SessionMsg3, SessionReceiverReport, SessionSenderReport,
SessionSetup,
use crate::proto::fmp::wire::{
ESTABLISHED_HEADER_SIZE, FLAG_KEY_EPOCH, FLAG_SP, build_established_header,
};
use crate::protocol::{coords_wire_size, encode_coords};
use crate::proto::fsp::wire::{
FSP_COMMON_PREFIX_SIZE, FSP_FLAG_CP, FSP_FLAG_K, FSP_HEADER_SIZE, FSP_PHASE_ESTABLISHED,
FSP_PHASE_MSG1, FSP_PHASE_MSG2, FSP_PHASE_MSG3, FSP_PORT_HEADER_SIZE, FSP_PORT_IPV6_SHIM,
FspCommonPrefix, FspEncryptedHeader, build_fsp_header, fsp_prepend_inner_header,
fsp_strip_inner_header, parse_encrypted_coords,
};
use crate::proto::fsp::{
DecryptSlot, EpochReaction, FspAction, FspInnerFlags, SessionAck, SessionMessageType,
SessionMsg3, SessionSetup, mark_ipv6_ecn_ce,
};
#[cfg(unix)]
use crate::proto::link::LinkMessageType;
#[cfg(unix)]
use crate::proto::link::SESSION_DATAGRAM_HEADER_SIZE;
use crate::proto::link::SessionDatagram;
use crate::proto::mmp::{
BackoffUpdate, MmpAction, MmpSessionState, PathMtuNotification, ReceiverReport, SendResult,
SessionReceiverReport, SessionReportKind, SessionReportSnapshot, SessionSenderReport,
};
use crate::proto::mmp::{MAX_SESSION_REPORT_INTERVAL_MS, MIN_SESSION_REPORT_INTERVAL_MS};
use crate::proto::routing::{CoordsRequired, MtuExceeded, PathBroken, RoutingSignalType};
use crate::proto::stp::{coords_wire_size, encode_coords};
#[cfg(unix)]
use crate::transport::TransportHandle;
use crate::upper::icmp::FIPS_OVERHEAD;
@@ -51,8 +56,8 @@ struct PipelinedSend<'a> {
timestamp: u32,
fsp_flags: u8,
inner_plaintext: &'a [u8],
my_coords: Option<&'a crate::tree::TreeCoordinate>,
dest_coords: Option<&'a crate::tree::TreeCoordinate>,
my_coords: Option<&'a crate::proto::stp::TreeCoordinate>,
dest_coords: Option<&'a crate::proto::stp::TreeCoordinate>,
}
impl Node {
@@ -104,14 +109,14 @@ impl Node {
}
let error_type = inner[0];
let error_body = &inner[1..];
match SessionMessageType::from_byte(error_type) {
Some(SessionMessageType::CoordsRequired) => {
match RoutingSignalType::from_byte(error_type) {
Some(RoutingSignalType::CoordsRequired) => {
self.handle_coords_required(error_body).await;
}
Some(SessionMessageType::PathBroken) => {
Some(RoutingSignalType::PathBroken) => {
self.handle_path_broken(error_body).await;
}
Some(SessionMessageType::MtuExceeded) => {
Some(RoutingSignalType::MtuExceeded) => {
self.handle_mtu_exceeded(error_body).await;
}
_ => {
@@ -166,11 +171,14 @@ impl Node {
match parse_encrypted_coords(coord_data) {
Ok((src_coords, dest_coords, bytes_consumed)) => {
let now_ms = Self::now_ms();
if let Some(coords) = src_coords {
self.coord_cache.insert(*src_addr, coords, now_ms);
}
if let Some(coords) = dest_coords {
self.coord_cache.insert(*self.node_addr(), coords, now_ms);
let my_addr = *self.node_addr();
for action in
self.fsp
.plan_cache_coords(*src_addr, my_addr, src_coords, dest_coords)
{
if let FspAction::CacheCoords { addr, coords } = action {
self.coord_cache.insert(addr, coords, now_ms);
}
}
ciphertext_offset += bytes_consumed;
}
@@ -248,44 +256,55 @@ impl Node {
}
};
// React to the epoch the frame decrypted against.
match slot {
EpochSlot::Pending => {
// A frame that authenticates against `pending` is itself
// the cutover signal — proof the peer derived the new
// session and moved to it. Promote now: current →
// previous, pending → current, flip the K-bit. The
// header K-bit is no longer the gating event; the
// authenticated decrypt is.
// React to the epoch the frame decrypted against. The shell opened
// the frame; the core classifies the post-decrypt reaction over the
// plain-data slot + session flags, and the shell applies the
// `SessionEntry` mutation.
let decrypt_slot = match slot {
EpochSlot::Current => DecryptSlot::Current,
EpochSlot::Pending => DecryptSlot::Pending,
EpochSlot::Previous => DecryptSlot::Previous,
};
match self.fsp.classify_epoch(
decrypt_slot,
entry.rekey_msg3_payload().is_some(),
entry.pending_new_session().is_some(),
) {
EpochReaction::PromoteConfirming => {
// A frame that authenticates against `pending` is itself the
// cutover signal — proof the peer derived the new session and
// moved to it. The peer received msg3, so confirm it on the new
// epoch (stop retransmitting) before `handle_peer_kbit_flip`
// consumes the pending session, then promote.
info!(
peer = %self.peer_display_name(src_addr),
"Peer FSP new-epoch frame authenticated, FSP rekey cutover complete, promoting new session"
);
// The peer derived the new session, so it received msg3:
// confirm it on the new epoch and stop retransmitting.
// `handle_peer_kbit_flip` consumes the pending session,
// so confirm first.
if entry.rekey_msg3_payload().is_some() {
entry.confirm_peer_new_epoch();
}
entry.confirm_peer_new_epoch();
entry.handle_peer_kbit_flip(now_ms);
}
EpochSlot::Current => {
// If we still retain a msg3 retransmission payload but no
// longer hold a `pending` session, we are the rekey
// initiator that already cut over on its own timer:
// `current` is now the new epoch, so a frame decrypting
// against it confirms the responder reached the new
// epoch. Stop retransmitting msg3.
if entry.rekey_msg3_payload().is_some() && entry.pending_new_session().is_none() {
entry.confirm_peer_new_epoch();
}
EpochReaction::Promote => {
// Promote now: current → previous, pending → current, flip the
// K-bit. The header K-bit is only a hint; the authenticated
// decrypt is the gating event.
info!(
peer = %self.peer_display_name(src_addr),
"Peer FSP new-epoch frame authenticated, FSP rekey cutover complete, promoting new session"
);
entry.handle_peer_kbit_flip(now_ms);
}
EpochSlot::Previous => {
// The peer is still on the old epoch. `fsp_trial_decrypt`
// already refreshed the drain deadline so the `previous`
// slot is not retired while the peer keeps using it —
// no further state change here, just deliver.
EpochReaction::ConfirmResponder => {
// We are the rekey initiator that already cut over on its own
// timer: `current` is now the new epoch, so a frame decrypting
// against it confirms the responder reached it. Stop
// retransmitting msg3.
entry.confirm_peer_new_epoch();
}
EpochReaction::None => {
// Steady-state `current`, or an old-epoch `previous` straggler:
// `fsp_trial_decrypt` already refreshed the drain deadline so
// the `previous` slot is not retired while the peer keeps using
// it — no further state change, just deliver.
}
}
@@ -305,16 +324,16 @@ impl Node {
if let Some(entry) = self.sessions.get_mut(src_addr)
&& let Some(mmp) = entry.mmp_mut()
{
let now = std::time::Instant::now();
let now_ms = crate::time::mono_ms();
mmp.receiver
.record_recv(header.counter, timestamp, plaintext.len(), ce_flag, now);
.record_recv(header.counter, timestamp, plaintext.len(), ce_flag, now_ms);
// Spin bit: advance state machine for correct TX reflection.
// RTT samples not fed into SRTT — timestamp-echo provides
// accurate RTT; spin bit includes variable inter-frame delays.
let inner_flags = FspInnerFlags::from_byte(inner_flags_byte);
let _spin_rtt = mmp
.spin_bit
.rx_observe(inner_flags.spin_bit, header.counter, now);
.rx_observe(inner_flags.spin_bit, header.counter, now_ms);
}
// Feed path_mtu from datagram envelope to MMP path MTU tracking.
@@ -355,7 +374,7 @@ impl Node {
mark_ipv6_ecn_ce(&mut packet);
self.metrics().congestion.ce_received.inc();
}
if let Some(tun_tx) = &self.tun_tx {
if let Some(tun_tx) = &self.supervisor.tun_tx {
if let Err(e) = tun_tx.send(packet) {
debug!(error = %e, "Failed to deliver decompressed IPv6 packet to TUN");
}
@@ -444,7 +463,7 @@ impl Node {
if let Some(existing) = self.sessions.get(src_addr) {
if existing.is_initiating() {
// Simultaneous initiation: smaller NodeAddr wins as initiator
if self.identity().node_addr() < src_addr {
if crate::proto::fsp::initiation_winner(self.identity().node_addr(), src_addr) {
// We win — drop their setup, they'll process ours
debug!(
src = %self.peer_display_name(src_addr),
@@ -482,7 +501,10 @@ impl Node {
// simultaneously. Apply tie-breaker — smaller NodeAddr
// wins as initiator (same as initial session setup).
if rekey_in_progress {
if self.identity().node_addr() < src_addr {
if crate::proto::fsp::initiation_winner(
self.identity().node_addr(),
src_addr,
) {
// We win as initiator — drop their msg1.
debug!(
src = %self.peer_display_name(src_addr),
@@ -919,6 +941,221 @@ impl Node {
// === Session-layer MMP report handlers ===
/// Check all sessions for pending MMP reports and send them.
///
/// Called from the tick handler. Also emits periodic session MMP logs.
/// Uses the collect-then-send pattern to avoid borrowing conflicts.
pub(in crate::node) async fn check_session_mmp_reports(&mut self) {
let now_ms = crate::time::mono_ms();
// Build one report-gating snapshot per session, resolving every timing
// read shell-side into a `bool`. The snapshots own only
// `NodeAddr`/`MmpMode`/`bool`, so the session-iteration borrow is released
// before the pure decision runs and the driving loop mutates the
// reporting state / performs the sends.
let snapshots: Vec<SessionReportSnapshot> = self
.sessions
.iter()
.filter_map(|(dest_addr, entry)| {
let mmp = entry.mmp()?;
Some(SessionReportSnapshot {
dest: *dest_addr,
mode: mmp.mode(),
sr_due: mmp.sender.should_send_report(now_ms),
rr_due: mmp.receiver.should_send_report(now_ms),
mtu_due: mmp.path_mtu.should_send_notification(now_ms),
log_due: mmp.should_log(now_ms),
})
})
.collect();
let actions = self.mmp.plan_session_reports(&snapshots);
// Drive the planned actions in phase-grouped order (all logs, then the
// sends in per-session SR/RR/MTU order). Logs run first because the
// session operator log reads cumulative_packets_sent, which each send
// advances (send_session_msg -> sender.record_sent); the pre-refactor
// handler logged during its collect pass, before any send. Each build
// (`build_report`/`build_notification`, which advance interval/
// notification state) runs only on its SendSessionReport action, exactly
// as the pre-refactor collect pass did. Per-destination success/failure
// is collected for the backoff dedup + failure-log suppression.
let mut send_results: Vec<SendResult> = Vec::new();
for action in actions {
match action {
MmpAction::LogSession { dest } => {
// Resolve the display name exactly as the pre-refactor loop
// did (alias, else short_npub from the session's remote key).
let session_name = self.peer_aliases.get(&dest).cloned().unwrap_or_else(|| {
self.sessions
.get(&dest)
.map(|entry| {
let (xonly, _) = entry.remote_pubkey().x_only_public_key();
crate::PeerIdentity::from_pubkey(xonly).short_npub()
})
.unwrap_or_default()
});
if let Some(mmp) = self.sessions.get_mut(&dest).and_then(|e| e.mmp_mut()) {
Self::log_session_mmp_metrics(&session_name, mmp);
mmp.mark_logged(now_ms);
}
}
MmpAction::SendSessionReport { dest, kind } => {
let built = self
.sessions
.get_mut(&dest)
.and_then(|entry| entry.mmp_mut())
.and_then(|mmp| match kind {
SessionReportKind::Sender => {
mmp.sender.build_report(now_ms).map(|sr| {
(
SessionMessageType::SenderReport.to_byte(),
SessionSenderReport::from(&sr).encode(),
)
})
}
SessionReportKind::Receiver => {
mmp.receiver.build_report(now_ms).map(|rr| {
(
SessionMessageType::ReceiverReport.to_byte(),
SessionReceiverReport::from(&rr).encode(),
)
})
}
SessionReportKind::PathMtu => {
mmp.path_mtu.build_notification(now_ms).map(|mtu_value| {
(
SessionMessageType::PathMtuNotification.to_byte(),
PathMtuNotification::new(mtu_value).encode(),
)
})
}
});
let Some((msg_type, body)) = built else {
continue;
};
match self.send_session_msg(&dest, msg_type, &body).await {
Ok(()) => send_results.push(SendResult { dest, ok: true }),
Err(e) => {
// Peek at current failure count for log suppression
// (unchanged by the backoff apply, which runs later).
let failures = self
.sessions
.get(&dest)
.and_then(|entry| entry.mmp())
.map(|mmp| mmp.sender.consecutive_send_failures())
.unwrap_or(0);
if failures < 3 {
debug!(
dest = %self.peer_display_name(&dest),
msg_type,
error = %e,
"Failed to send session MMP report"
);
} else if failures == 3 {
debug!(
dest = %self.peer_display_name(&dest),
"Suppressing further session MMP send failure logs"
);
}
// failures > 3: silently suppressed
send_results.push(SendResult { dest, ok: false });
}
}
}
MmpAction::ReapPeer { .. }
| MmpAction::Heartbeat { .. }
| MmpAction::SendLinkReport { .. }
| MmpAction::LogLink { .. } => {}
}
}
// Deduplicate send results per destination (any-ok -> success, all-fail
// -> failure) and apply the backoff state transition for each dest.
for update in self.mmp.plan_backoff(&send_results) {
match update {
BackoffUpdate::Success { dest } => {
if let Some(mmp) = self.sessions.get_mut(&dest).and_then(|e| e.mmp_mut()) {
let prev = mmp.sender.record_send_success();
if prev > 3 {
debug!(
dest = %self.peer_display_name(&dest),
consecutive_failures = prev,
"Resumed session MMP reporting"
);
}
}
}
BackoffUpdate::Failure { dest } => {
if let Some(mmp) = self.sessions.get_mut(&dest).and_then(|e| e.mmp_mut()) {
mmp.sender.record_send_failure();
}
}
}
}
}
/// Emit periodic session MMP metrics.
fn log_session_mmp_metrics(session_name: &str, mmp: &MmpSessionState) {
let m = &mmp.metrics;
let rtt_str = if m.rtt_trend.initialized() {
format!("{:.1}ms", m.rtt_trend.long() / 1000.0)
} else {
"n/a".to_string()
};
let loss_str = if m.loss_trend.initialized() {
format!("{:.1}%", m.loss_trend.long() * 100.0)
} else {
"n/a".to_string()
};
let jitter_ms = mmp.receiver.jitter_us() as f64 / 1000.0;
debug!(
session = %session_name,
rtt = %rtt_str,
loss = %loss_str,
jitter = format_args!("{:.1}ms", jitter_ms),
goodput = %format_throughput(m.goodput_bps()),
mtu = mmp.path_mtu.last_observed_mtu(),
tx_pkts = mmp.sender.cumulative_packets_sent(),
rx_pkts = mmp.receiver.cumulative_packets_recv(),
"MMP session metrics"
);
}
/// Emit a teardown log summarizing lifetime session MMP metrics.
pub(in crate::node) fn log_session_mmp_teardown(session_name: &str, mmp: &MmpSessionState) {
let m = &mmp.metrics;
let jitter_ms = mmp.receiver.jitter_us() as f64 / 1000.0;
let rtt_str = match m.srtt_ms() {
Some(rtt) => format!("{:.1}ms", rtt),
None => "n/a".to_string(),
};
let loss_str = format!("{:.1}%", m.loss_rate() * 100.0);
debug!(
session = %session_name,
rtt = %rtt_str,
loss = %loss_str,
jitter = format_args!("{:.1}ms", jitter_ms),
etx = format_args!("{:.2}", m.etx),
goodput = %format_throughput(m.goodput_bps()),
send_mtu = mmp.path_mtu.current_mtu(),
observed_mtu = mmp.path_mtu.last_observed_mtu(),
tx_pkts = mmp.sender.cumulative_packets_sent(),
tx_bytes = mmp.sender.cumulative_bytes_sent(),
rx_pkts = mmp.receiver.cumulative_packets_recv(),
rx_bytes = mmp.receiver.cumulative_bytes_recv(),
"MMP session teardown"
);
}
/// Handle an incoming session-layer SenderReport (msg_type 0x11).
///
/// Informational only — the peer is telling us about what they sent.
@@ -974,9 +1211,11 @@ impl Node {
return;
};
let now = std::time::Instant::now();
mmp.metrics
.process_receiver_report(&rr, our_timestamp_ms, now);
let (_first_rtt, rr_log) =
mmp.metrics
.process_receiver_report(&rr, our_timestamp_ms, crate::time::mono_ms());
// Re-emit the operator trace the core used to log mid-decision.
super::mmp::log_rr_outcome(&rr, our_timestamp_ms, rr_log);
// Feed SRTT back to sender/receiver report interval tuning (session-layer bounds)
if let Some(srtt_ms) = mmp.metrics.srtt_ms() {
@@ -1042,8 +1281,9 @@ impl Node {
};
let old_mtu = mmp.path_mtu.current_mtu();
let now = std::time::Instant::now();
let changed = mmp.path_mtu.apply_notification(notif.path_mtu, now);
let changed = mmp
.path_mtu
.apply_notification(notif.path_mtu, crate::time::mono_ms());
let new_mtu = mmp.path_mtu.current_mtu();
if !changed {
@@ -1065,28 +1305,34 @@ impl Node {
// tighter of existing-or-new — never loosen the clamp.
let fips_addr = crate::FipsAddress::from_node_addr(src_addr);
match self.path_mtu_lookup.write() {
Ok(mut map) => match map.get(&fips_addr).copied() {
Some(existing) if existing <= new_mtu => {
Ok(mut map) => {
// Read existing, decide, and apply the write under one guard so
// the keep-tighter update stays atomic.
let prior = map.get(&fips_addr).copied();
let actions = self.fsp.plan_path_mtu_tighten(fips_addr, prior, new_mtu);
if actions.is_empty() {
debug!(
dest = %peer_name,
fips_addr = %fips_addr,
new_mtu,
existing,
existing = prior.unwrap_or(new_mtu),
"PathMtuNotification: keeping tighter existing path_mtu_lookup value"
);
}
other => {
map.insert(fips_addr, new_mtu);
debug!(
dest = %peer_name,
fips_addr = %fips_addr,
new_mtu,
prior = ?other,
map_len = map.len(),
"PathMtuNotification: tightened path_mtu_lookup"
);
for action in actions {
if let FspAction::TightenPathMtuLookup { fips_addr, mtu } = action {
map.insert(fips_addr, mtu);
debug!(
dest = %peer_name,
fips_addr = %fips_addr,
new_mtu,
prior = ?prior,
map_len = map.len(),
"PathMtuNotification: tightened path_mtu_lookup"
);
}
}
},
}
Err(e) => {
warn!(
dest = %peer_name,
@@ -1125,7 +1371,7 @@ impl Node {
// Send standalone CoordsWarmup immediately (rate-limited)
if self
.coords_response_rate_limiter
.should_send(&msg.dest_addr)
.should_send(&msg.dest_addr, Self::now_ms())
{
if let Some(entry) = self.sessions.get(&msg.dest_addr)
&& entry.is_established()
@@ -1141,12 +1387,19 @@ impl Node {
// Only trigger discovery if we have the target's identity cached —
// otherwise we can't verify the LookupResponse proof.
if self.has_cached_identity(&msg.dest_addr) {
self.maybe_initiate_lookup(&msg.dest_addr).await;
} else {
let has_cached_identity = self.has_cached_identity(&msg.dest_addr);
let actions = self
.fsp
.plan_coords_required_lookup(msg.dest_addr, has_cached_identity);
if actions.is_empty() {
debug!(dest = %msg.dest_addr,
"Skipping discovery after CoordsRequired: no cached identity for target");
}
for action in actions {
if let FspAction::InitiateLookup { dest } = action {
self.maybe_initiate_lookup(&dest).await;
}
}
// Reset coords warmup counter so the next N packets also include
// COORDS_PRESENT, re-warming transit caches along the path.
@@ -1186,7 +1439,7 @@ impl Node {
// Send standalone CoordsWarmup immediately (rate-limited)
if self
.coords_response_rate_limiter
.should_send(&msg.dest_addr)
.should_send(&msg.dest_addr, Self::now_ms())
{
if let Some(entry) = self.sessions.get(&msg.dest_addr)
&& entry.is_established()
@@ -1200,16 +1453,26 @@ impl Node {
"PathBroken response rate-limited, skipping standalone CoordsWarmup");
}
// Invalidate stale cached coordinates
self.coord_cache.remove(&msg.dest_addr);
// Trigger re-discovery to get fresh coordinates, but only if we have
// the target's identity cached — otherwise we can't verify the
// LookupResponse proof. This avoids a race when the XK responder
// receives PathBroken before msg3 completes (identity unknown).
if self.has_cached_identity(&msg.dest_addr) {
self.maybe_initiate_lookup(&msg.dest_addr).await;
} else {
// Invalidate stale cached coordinates, then (only if the target's
// identity is cached — else the LookupResponse proof cannot be verified,
// e.g. when the XK responder receives PathBroken before msg3 completes)
// trigger re-discovery. The core emits invalidate-then-lookup in order.
let has_cached_identity = self.has_cached_identity(&msg.dest_addr);
let actions = self
.fsp
.plan_path_broken(msg.dest_addr, has_cached_identity);
for action in actions {
match action {
FspAction::InvalidateCoords { addr } => {
self.coord_cache.remove(&addr);
}
FspAction::InitiateLookup { dest } => {
self.maybe_initiate_lookup(&dest).await;
}
_ => {}
}
}
if !has_cached_identity {
debug!(dest = %msg.dest_addr,
"Skipping discovery after PathBroken: no cached identity for target");
}
@@ -1256,8 +1519,10 @@ impl Node {
&& let Some(mmp) = entry.mmp_mut()
{
let old_mtu = mmp.path_mtu.current_mtu();
let now = std::time::Instant::now();
if mmp.path_mtu.apply_notification(msg.mtu, now) {
if mmp
.path_mtu
.apply_notification(msg.mtu, crate::time::mono_ms())
{
let new_mtu = mmp.path_mtu.current_mtu();
info!(
dest = %peer_name,
@@ -1277,28 +1542,34 @@ impl Node {
// tighter of existing-or-new — never loosen the clamp.
let fips_addr = crate::FipsAddress::from_node_addr(&msg.dest_addr);
match self.path_mtu_lookup.write() {
Ok(mut map) => match map.get(&fips_addr).copied() {
Some(existing) if existing <= msg.mtu => {
Ok(mut map) => {
// Read existing, decide, and apply the write under one guard so
// the keep-tighter update stays atomic.
let prior = map.get(&fips_addr).copied();
let actions = self.fsp.plan_path_mtu_tighten(fips_addr, prior, msg.mtu);
if actions.is_empty() {
debug!(
dest = %peer_name,
fips_addr = %fips_addr,
bottleneck_mtu = msg.mtu,
existing,
existing = prior.unwrap_or(msg.mtu),
"Reactive MtuExceeded: keeping tighter existing path_mtu_lookup value"
);
}
other => {
map.insert(fips_addr, msg.mtu);
debug!(
dest = %peer_name,
fips_addr = %fips_addr,
bottleneck_mtu = msg.mtu,
prior = ?other,
map_len = map.len(),
"Reactive MtuExceeded: tightened path_mtu_lookup"
);
for action in actions {
if let FspAction::TightenPathMtuLookup { fips_addr, mtu } = action {
map.insert(fips_addr, mtu);
debug!(
dest = %peer_name,
fips_addr = %fips_addr,
bottleneck_mtu = msg.mtu,
prior = ?prior,
map_len = map.len(),
"Reactive MtuExceeded: tightened path_mtu_lookup"
);
}
}
},
}
Err(e) => {
warn!(
dest = %peer_name,
@@ -1561,7 +1832,7 @@ impl Node {
send: PipelinedSend<'_>,
) -> Result<bool, NodeError> {
let dest_addr = send.dest_addr;
let Some(workers) = self.encrypt_workers.as_ref().cloned() else {
let Some(workers) = self.supervisor.encrypt_workers.as_ref().cloned() else {
return Ok(false);
};
@@ -2065,7 +2336,10 @@ impl Node {
/// Returns our own coordinates as a fallback (the SessionSetup will
/// carry src_coords for return path routing; empty dest_coords
/// would fail wire encoding since TreeCoordinate requires ≥1 entry).
pub(in crate::node) fn get_dest_coords(&self, dest: &NodeAddr) -> crate::tree::TreeCoordinate {
pub(in crate::node) fn get_dest_coords(
&self,
dest: &NodeAddr,
) -> crate::proto::stp::TreeCoordinate {
let now_ms = Self::now_ms();
if let Some(coords) = self.coord_cache.get(dest, now_ms) {
return coords.clone();
@@ -2174,7 +2448,7 @@ impl Node {
let our_ipv6 = FipsAddress::from_node_addr(self.node_addr()).to_ipv6();
if let Some(response) =
build_dest_unreachable(original_packet, DestUnreachableCode::NoRoute, our_ipv6)
&& let Some(tun_tx) = &self.tun_tx
&& let Some(tun_tx) = &self.supervisor.tun_tx
{
let _ = tun_tx.send(response);
}
@@ -2209,7 +2483,7 @@ impl Node {
// causes a PMTUD blackhole when both src and ICMP-src are local.
let dest_addr = Ipv6Addr::from(<[u8; 16]>::try_from(&original_packet[24..40]).unwrap());
if let Some(response) = build_packet_too_big(original_packet, mtu, dest_addr)
&& let Some(tun_tx) = &self.tun_tx
&& let Some(tun_tx) = &self.supervisor.tun_tx
{
debug!(
original_src = %src_addr,
@@ -2234,10 +2508,7 @@ impl Node {
let per_dest = self.config().node.session.pending_packets_per_dest;
let queue = self.pending_tun_packets.entry(dest_addr).or_default();
if queue.len() >= per_dest {
queue.pop_front(); // Drop oldest
}
queue.push_back(packet);
crate::proto::fsp::push_bounded_pending(queue, packet, per_dest);
}
/// Flush pending packets for a destination whose session just reached Established.
@@ -2288,31 +2559,3 @@ impl Node {
}
}
}
/// Mark ECN-CE in an IPv6 packet's Traffic Class field.
///
/// IPv6 Traffic Class occupies bits across bytes 0 and 1:
/// byte[0] bits[3:0] = TC[7:4]
/// byte[1] bits[7:4] = TC[3:0]
/// ECN is TC[1:0]. Only marks CE (0b11) if the packet is ECN-capable
/// (ECT(0) or ECT(1)). Packets with ECN=0b00 (Not-ECT) are never marked
/// per RFC 3168.
///
/// No checksum update needed: IPv6 has no header checksum, and the Traffic
/// Class field is not part of the TCP/UDP pseudo-header.
pub(in crate::node) fn mark_ipv6_ecn_ce(packet: &mut [u8]) {
if packet.len() < 2 {
return;
}
// Extract 8-bit Traffic Class from IPv6 header bytes 0-1
let tc = ((packet[0] & 0x0F) << 4) | (packet[1] >> 4);
let ecn = tc & 0x03;
// Only mark CE on ECN-capable packets (ECT(0)=0b10 or ECT(1)=0b01)
if ecn == 0 {
return;
}
// Set both ECN bits to 1 (CE = 0b11)
let new_tc = tc | 0x03;
packet[0] = (packet[0] & 0xF0) | (new_tc >> 4);
packet[1] = (new_tc << 4) | (packet[1] & 0x0F);
}
src/node/handlers/timeout.rs
+298 -77
View File
@@ -2,58 +2,108 @@
//! and handshake message resend scheduling.
use crate::node::Node;
use crate::peer::HandshakeState;
use crate::peer::machine::TimerKind;
use crate::proto::fmp::{
ConnAction, ConnSnapshot, LifecycleView, PeerSnapshot, RekeyResendSnapshot,
};
use crate::transport::LinkId;
use tracing::{debug, info};
impl LifecycleView for Node {
fn stale_connections(&self, now_ms: u64, timeout_ms: u64) -> Vec<ConnSnapshot> {
// `is_failed()` legs are always reaped here (~1s), as before. The
// idle-timeout is reaped here ONLY for legs whose timeout is not already
// driven by a machine `HandshakeTimeout` timer — i.e. inbound legs (IK
// inbound arms none). Outbound legs with an armed timer are reaped by
// `drive_handshake_timeouts`, so excluding them here avoids a double
// reap.
self.peer_machines
.iter()
.filter(|(_, machine)| machine.leg().is_some())
.filter(|(link_id, machine)| {
machine.is_failed()
|| (machine.conn_is_timed_out(now_ms, timeout_ms)
&& !self.peer_timers.get(*link_id).is_some_and(|timers| {
timers.contains_key(&TimerKind::HandshakeTimeout)
}))
})
.map(|(link_id, machine)| ConnSnapshot {
link: *link_id,
is_outbound: machine.conn_is_outbound(),
retry_addr: machine.conn_expected_identity().map(|id| *id.node_addr()),
resend_count: 0,
msg1: Vec::new(),
})
.collect()
}
fn rekey_peers(&self) -> Vec<PeerSnapshot> {
// The snapshot builder lives in `rekey` beside its drain/dampening
// constants; the read-seam unifies here.
self.rekey_peer_snapshots()
}
fn rekey_resend_candidates(&self, now_ms: u64) -> Vec<RekeyResendSnapshot> {
self.rekey_resend_snapshots(now_ms)
}
}
impl Node {
/// Check for timed-out handshake connections and clean them up.
///
/// Called periodically by the RX event loop. Removes connections that have
/// been idle longer than the configured handshake timeout or are in Failed state.
///
/// The stale/failed predicate and every registry mutation stay shell-side;
/// the retry-then-teardown choreography is the pure
/// [`Fmp::poll_timeouts`](crate::proto::fmp::Fmp::poll_timeouts) decision.
pub(in crate::node) fn check_timeouts(&mut self) {
if self.connections.is_empty() {
if self.connection_count() == 0 {
return;
}
let now_ms = Self::now_ms();
let timeout_ms = self.config().node.rate_limit.handshake_timeout_secs * 1000;
let stale: Vec<LinkId> = self
.connections
.iter()
.filter(|(_, conn)| conn.is_timed_out(now_ms, timeout_ms) || conn.is_failed())
.map(|(link_id, _)| *link_id)
.collect();
for link_id in stale {
// Log and schedule retry before cleanup (need connection state)
if let Some(conn) = self.connections.get(&link_id) {
let direction = conn.direction();
let idle_ms = conn.idle_time(now_ms);
if conn.is_failed() {
debug!(
link_id = %link_id,
direction = %direction,
"Failed handshake connection cleaned up"
);
} else {
debug!(
link_id = %link_id,
direction = %direction,
idle_secs = idle_ms / 1000,
"Stale handshake connection timed out"
);
}
// Schedule retry for failed outbound auto-connect peers
if conn.is_outbound()
&& let Some(identity) = conn.expected_identity()
{
self.schedule_retry(*identity.node_addr(), now_ms);
let stale = self.stale_connections(now_ms, timeout_ms);
for action in self.fmp.poll_timeouts(stale) {
match action {
ConnAction::ScheduleRetry { peer } => self.note_handshake_timeout(peer, now_ms),
ConnAction::Teardown { link } => {
// Log before cleanup (needs live connection state). The
// failure signal is now read from the control machine; the
// leg still carries direction/idle for the log fields.
let is_failed = self
.peer_machines
.get(&link)
.is_some_and(|machine| machine.is_failed());
if let Some(machine) = self
.peer_machines
.get(&link)
.filter(|machine| machine.leg().is_some())
{
let direction = machine.conn_direction();
if is_failed {
debug!(
link_id = %link,
direction = %direction,
"Failed handshake connection cleaned up"
);
} else {
debug!(
link_id = %link,
direction = %direction,
idle_secs =
now_ms.saturating_sub(self.connection_last_activity(link)) / 1000,
"Stale handshake connection timed out"
);
}
}
self.cleanup_stale_connection(link, now_ms);
}
#[allow(unreachable_patterns)]
_ => {}
}
self.cleanup_stale_connection(link_id, now_ms);
}
}
@@ -63,15 +113,32 @@ impl Node {
/// the link and address mapping. Does not log — callers provide context-appropriate
/// log messages.
pub(in crate::node) fn cleanup_stale_connection(&mut self, link_id: LinkId, _now_ms: u64) {
let conn = match self.connections.remove(&link_id) {
// Take the connection off its machine BEFORE disposing the machine
// (the machine owns it), keeping it readable for the index/link
// cleanup below. The machine shares the connection's `link_id` and
// lifetime; dropping it here means a reaped handshake leg leaves no
// dangling machine. A no-op for promoted peers — `promote_connection`
// already consumed their connection, so this reaper never runs for
// them.
let _detached_leg = match self
.peer_machines
.get_mut(&link_id)
.and_then(|machine| machine.take_leg())
{
Some(c) => c,
None => return,
};
let transport_id = conn.transport_id();
// Read the transport ID and session index off the surviving carrier
// before disposing the machine (the leg no longer projects them).
let (transport_id, our_index) = match self.peer_machines.get(&link_id) {
Some(machine) => (machine.conn_transport_id(), machine.our_index()),
None => (None, None),
};
self.remove_peer_machine(link_id);
// Free session index and pending_outbound if allocated
if let Some(idx) = conn.our_index() {
if let Some(tid) = conn.transport_id() {
if let Some(idx) = our_index {
if let Some(tid) = transport_id {
self.pending_outbound.remove(&(tid, idx.as_u32()));
}
let _ = self.index_allocator.free(idx);
@@ -84,53 +151,196 @@ impl Node {
}
}
/// Resend handshake messages for pending connections.
/// Act on the per-peer machine timers this tick.
///
/// For outbound connections in SentMsg1 state, resends the stored msg1
/// with exponential backoff. Called periodically from the RX event loop.
pub(in crate::node) async fn resend_pending_handshakes(&mut self, now_ms: u64) {
if self.connections.is_empty() {
/// The sans-IO machine arms `SetTimer`/`CancelTimer` actions into
/// [`peer_timers`](Node::peer_timers); this is the shell driver that acts on
/// them: timeout reaps idle-timed-out outbound legs, retransmit resends the
/// due msg1s. Handshake-TIMEOUT is driven before handshake-RETRANSMIT so a
/// timed-out leg is reaped rather than resent on the same tick. The
/// rekey/liveness kinds keep their own shell drivers, so only the two
/// handshake kinds are driven here.
pub(in crate::node) async fn drive_peer_timers(&mut self, now_ms: u64) {
if self.peer_timers.is_empty() {
return;
}
self.drive_handshake_timeouts(now_ms);
self.drive_handshake_retransmits(now_ms).await;
}
/// Reap the outbound legs whose machine `HandshakeTimeout` timer marks them
/// as machine-timeout-owned and which have idle-timed-out this tick.
///
/// The timer's PRESENCE selects the leg (only OUTBOUND legs arm one — IK
/// inbound arms none); the reap THRESHOLD is the shell `is_timed_out(now,
/// config)` predicate, NOT the timer's stored deadline. This matters because
/// the machine arms the timer from a hardcoded constant at dial, which is not
/// authoritative for an operator-tuned `handshake_timeout_secs` — reading the
/// threshold from config each tick keeps the reap neutral for any config, and
/// off the `last_activity` clock exactly as the old `check_timeouts` did. A
/// timed-out leg is reaped by the old Teardown path: the outbound retry
/// reflex, then `cleanup_stale_connection` (which drops the machine + timers).
///
/// `check_timeouts` keeps reaping everything else — `is_failed()` legs and the
/// idle-timeout of legs without a machine timer (inbound legs).
fn drive_handshake_timeouts(&mut self, now_ms: u64) {
let timeout_ms = self.config().node.rate_limit.handshake_timeout_secs * 1000;
let timer_links: Vec<LinkId> = self
.peer_timers
.iter()
.filter(|(_, timers)| timers.contains_key(&TimerKind::HandshakeTimeout))
.map(|(link, _)| *link)
.collect();
for link in timer_links {
// The idle-timeout threshold reads the survivor carrier's
// last-activity; presence of a pending handshake is what decides
// between reaping and dropping an orphan timer.
let timed_out = self
.peer_machines
.get(&link)
.is_some_and(|machine| machine.conn_is_timed_out(now_ms, timeout_ms));
let (reap, retry_peer) = match self.has_pending_leg(&link) {
true if timed_out => {
let retry_peer = if self
.peer_machines
.get(&link)
.is_some_and(|machine| machine.conn_is_outbound())
{
self.peer_machines
.get(&link)
.and_then(|machine| machine.conn_expected_identity())
.map(|id| *id.node_addr())
} else {
None
};
(true, retry_peer)
}
// Not yet idle-timed-out: leave the timer for a later tick.
true => (false, None),
false => {
// Orphan timer (connection already reaped elsewhere) — drop it.
if let Some(timers) = self.peer_timers.get_mut(&link) {
timers.remove(&TimerKind::HandshakeTimeout);
}
(false, None)
}
};
if reap {
if let Some(peer) = retry_peer {
self.note_handshake_timeout(peer, now_ms);
}
debug!(link_id = %link, "Handshake connection timed out");
self.cleanup_stale_connection(link, now_ms);
}
}
}
/// Fire due handshake-retransmit timers: resend the stored msg1.
///
/// The pre-fold `resend_pending_handshakes` logic, re-homed: the *due* signal
/// is the machine-armed timer (not the connection's `next_resend_at_ms`), and
/// the resend counter lives on the machine (the operator-visible count reads
/// from there). The wire bytes and transport target still come from the shell
/// connection, the pure core computes the backoff schedule, and — matching the
/// old shell exactly — the count and reschedule advance only on a successful
/// send; a failed send neither advances the count nor marks the connection
/// failed, it just retries next tick.
async fn drive_handshake_retransmits(&mut self, now_ms: u64) {
let max_resends = self.config().node.rate_limit.handshake_max_resends;
let interval_ms = self.config().node.rate_limit.handshake_resend_interval_ms;
let backoff = self.config().node.rate_limit.handshake_resend_backoff;
// Collect resend candidates: outbound, in SentMsg1, with stored msg1,
// under max resends, and past the scheduled time.
let candidates: Vec<(LinkId, Vec<u8>)> = self
.connections
// Collect due retransmit timers (kind-filtered).
let due: Vec<LinkId> = self
.peer_timers
.iter()
.filter(|(_, conn)| {
conn.is_outbound()
&& conn.handshake_state() == HandshakeState::SentMsg1
&& conn.resend_count() < max_resends
&& conn.next_resend_at_ms() > 0
&& now_ms >= conn.next_resend_at_ms()
})
.filter_map(|(link_id, conn)| {
conn.handshake_msg1().map(|msg1| (*link_id, msg1.to_vec()))
.filter(|(_, timers)| {
timers
.get(&TimerKind::HandshakeRetransmit)
.is_some_and(|&at_ms| now_ms >= at_ms)
})
.map(|(link, _)| *link)
.collect();
if due.is_empty() {
return;
}
for (link_id, msg1_bytes) in candidates {
// Get transport and address info from the connection
let (transport_id, remote_addr) = match self.connections.get(&link_id) {
Some(conn) => match (conn.transport_id(), conn.source_addr()) {
(Some(tid), Some(addr)) => (tid, addr.clone()),
_ => continue,
},
// Classify each due link against the machine + connection. A timer whose
// machine has left `SentMsg1` (promoted/gone) or has hit the resend cap
// is dropped — no more resends, exactly as the old shell stopped
// selecting a capped/settled connection; the handshake-timeout reaper
// takes it from there.
let mut candidates: Vec<ConnSnapshot> = Vec::new();
let mut drop_timers: Vec<LinkId> = Vec::new();
for link in due {
let armed = match self.peer_machines.get(&link) {
Some(machine)
if machine.is_handshaking_sent_msg1()
&& machine.resend_count() < max_resends =>
{
machine.resend_count()
}
Some(_) => {
drop_timers.push(link);
continue;
}
None => {
drop_timers.push(link);
continue;
}
};
match self
.peer_machines
.get(&link)
.and_then(|machine| machine.conn_handshake_msg1())
{
// Armed but the stored wire isn't there yet — leave the timer and
// retry next tick (matches the old candidate filter skipping it).
None => continue,
Some(msg1) => candidates.push(ConnSnapshot {
link,
is_outbound: true,
retry_addr: None,
resend_count: armed,
msg1: msg1.to_vec(),
}),
}
}
for link in drop_timers {
if let Some(timers) = self.peer_timers.get_mut(&link) {
timers.remove(&TimerKind::HandshakeRetransmit);
}
}
for action in self
.fmp
.poll_resends(candidates, now_ms, interval_ms, backoff)
{
let ConnAction::ResendMsg1 {
link,
bytes,
next_resend_at_ms,
} = action
else {
continue;
};
let (transport_id, remote_addr) = match self.peer_machines.get(&link) {
Some(machine) if machine.leg().is_some() => {
match (machine.conn_transport_id(), machine.conn_source_addr()) {
(Some(tid), Some(addr)) => (tid, addr.clone()),
_ => continue,
}
}
_ => continue,
};
// Send the stored msg1
let sent = if let Some(transport) = self.transports.get(&transport_id) {
match transport.send(&remote_addr, &msg1_bytes).await {
match transport.send(&remote_addr, &bytes).await {
Ok(_) => true,
Err(e) => {
debug!(
link_id = %link_id,
link_id = %link,
error = %e,
"Handshake msg1 resend failed"
);
@@ -141,15 +351,26 @@ impl Node {
false
};
if sent && let Some(conn) = self.connections.get_mut(&link_id) {
let count = conn.resend_count() + 1;
let next = now_ms + (interval_ms as f64 * backoff.powi(count as i32)) as u64;
conn.record_resend(next);
debug!(
link_id = %link_id,
resend = count,
"Resent handshake msg1"
);
if sent {
if let Some(machine) = self.peer_machines.get_mut(&link) {
machine.record_resend(next_resend_at_ms);
debug!(
link_id = %link,
resend = machine.resend_count(),
"Resent handshake msg1"
);
}
self.peer_timers
.entry(link)
.or_default()
.insert(TimerKind::HandshakeRetransmit, next_resend_at_ms);
} else {
// Failed send: keep retrying at the tick cadence (the old shell
// left next_resend_at_ms unchanged so the connection stayed due).
self.peer_timers
.entry(link)
.or_default()
.insert(TimerKind::HandshakeRetransmit, now_ms);
}
}
}
@@ -204,7 +425,7 @@ impl Node {
.collect();
for (dest_addr, payload) in candidates {
use crate::protocol::SessionDatagram;
use crate::proto::link::SessionDatagram;
let mut datagram = SessionDatagram::new(my_addr, dest_addr, payload).with_ttl(ttl);
let sent = match self.send_session_datagram(&mut datagram).await {
src/node/lifecycle.rs
-2761
View File
File diff suppressed because it is too large Load Diff
src/node/lifecycle/mod.rs
+3321
View File
File diff suppressed because it is too large Load Diff
src/node/lifecycle/supervisor.rs
+1364
View File
File diff suppressed because it is too large Load Diff
src/node/metrics.rs
+15 -50
View File
@@ -15,8 +15,8 @@ use std::sync::atomic::{AtomicU64, Ordering};
use crate::node::reject::{BloomReject, DiscoveryReject, ForwardingReject, TreeReject};
use crate::node::stats::{
BloomStatsSnapshot, CongestionStatsSnapshot, DiscoveryStatsSnapshot, ErrorSignalStatsSnapshot,
ForwardingStatsSnapshot, TreeStatsSnapshot,
BloomStatsSnapshot, CongestionStatsSnapshot, ErrorSignalStatsSnapshot, ForwardingStatsSnapshot,
LookupStatsSnapshot, TreeStatsSnapshot,
};
/// An atomic counter.
@@ -90,43 +90,10 @@ pub struct ForwardingMetrics {
}
/// Route class of a transit-forwarded packet, classified from tree
/// coordinates at the forwarding decision point. The six variants
/// partition `forwarded_packets` exactly.
///
/// Two variants are up-and-over forwards (destination not in the chosen
/// peer's subtree); they differ in whether they depend on a child
/// advertising cross-link reach *upward* to its parent:
/// - `TreeDownCross`: the chosen peer is our tree descendant, but the
/// destination is *not* in that child's subtree. The forward only fired
/// because the child advertised cross-link reach upward to us, beyond its
/// own subtree. If children advertised only their subtree upward, this
/// forward would route up instead, so its count measures how much
/// forwarding depends on the upward cross-link advertisement — the
/// dive-to-tree-child cut-through.
/// - `CrosslinkAscend`: the chosen peer is lateral (neither ancestor nor
/// descendant) and the destination is not in its subtree. This is a node
/// using its *own* cross-link, learned via the peer's split-horizon
/// advertisement to its neighbors, so it does not depend on any upward
/// advertisement. Tracked alongside `TreeDownCross` as the lateral
/// up-and-over contrast.
#[derive(Debug, Clone, Copy, PartialEq, Eq)]
pub enum RouteClass {
/// Chosen peer is our ancestor (tree-up).
TreeUp,
/// Chosen peer is our descendant and dest is in its subtree (canonical
/// tree-down).
TreeDown,
/// Chosen peer is our descendant but dest is *not* in its subtree: the
/// dive-to-tree-child cut-through enabled by upward cross-link
/// advertisement.
TreeDownCross,
/// Chosen peer is lateral and dest is in its subtree (subtree entry).
CrosslinkDescend,
/// Chosen peer is lateral and dest is not in its subtree (up-and-over).
CrosslinkAscend,
/// Chosen peer is the destination itself (degenerate direct hop).
DirectPeer,
}
/// coordinates at the forwarding decision point. Defined by the sans-IO
/// routing core and re-exported here for the forwarding-metrics counters
/// ([`ForwardingMetrics::record_route_class`]).
pub(crate) use crate::proto::routing::RouteClass;
impl ForwardingMetrics {
/// Record a received packet of `bytes` payload (packets and bytes).
@@ -235,7 +202,7 @@ impl ForwardingMetrics {
/// Discovery metric counters.
#[derive(Default)]
pub struct DiscoveryMetrics {
pub struct LookupMetrics {
pub req_received: Padded<Counter>,
pub req_decode_error: Counter,
pub req_duplicate: Counter,
@@ -260,7 +227,7 @@ pub struct DiscoveryMetrics {
pub resp_timed_out: Counter,
}
impl DiscoveryMetrics {
impl LookupMetrics {
/// Mirror of `DiscoveryStats::record_reject`: route a typed discovery
/// rejection to its counter.
#[inline]
@@ -278,8 +245,8 @@ impl DiscoveryMetrics {
}
/// Sample every counter into a serializable snapshot.
pub fn snapshot(&self) -> DiscoveryStatsSnapshot {
DiscoveryStatsSnapshot {
pub fn snapshot(&self) -> LookupStatsSnapshot {
LookupStatsSnapshot {
req_received: self.req_received.get(),
req_decode_error: self.req_decode_error.get(),
req_duplicate: self.req_duplicate.get(),
@@ -317,7 +284,6 @@ pub struct TreeMetrics {
pub stale: Counter,
pub ancestry_invalid: Counter,
pub accepted: Counter,
pub parent_switched: Counter,
pub loop_detected: Counter,
pub ancestry_changed: Counter,
pub sent: Counter,
@@ -351,7 +317,6 @@ impl TreeMetrics {
stale: self.stale.get(),
ancestry_invalid: self.ancestry_invalid.get(),
accepted: self.accepted.get(),
parent_switched: self.parent_switched.get(),
loop_detected: self.loop_detected.get(),
ancestry_changed: self.ancestry_changed.get(),
sent: self.sent.get(),
@@ -462,7 +427,7 @@ impl ErrorMetrics {
#[derive(Default)]
pub struct MetricsRegistry {
pub forwarding: ForwardingMetrics,
pub discovery: DiscoveryMetrics,
pub lookup: LookupMetrics,
pub tree: TreeMetrics,
pub bloom: BloomMetrics,
pub congestion: CongestionMetrics,
@@ -490,7 +455,7 @@ mod tests {
#[test]
fn discovery_record_reject_routes_to_field() {
let m = DiscoveryMetrics::default();
let m = LookupMetrics::default();
m.record_reject(DiscoveryReject::ReqDuplicate);
m.record_reject(DiscoveryReject::ReqDuplicate);
m.record_reject(DiscoveryReject::RespNoRoute);
@@ -501,7 +466,7 @@ mod tests {
#[test]
fn discovery_direct_counters_increment() {
let m = DiscoveryMetrics::default();
let m = LookupMetrics::default();
m.req_received.inc();
m.req_forwarded.inc();
m.req_forwarded.inc();
@@ -534,9 +499,9 @@ mod tests {
fn registry_subcounters_are_independent() {
let r = MetricsRegistry::new();
r.forwarding.record_received(10);
r.discovery.req_received.inc();
r.lookup.req_received.inc();
assert_eq!(r.forwarding.received_packets.get(), 1);
assert_eq!(r.forwarding.received_bytes.get(), 10);
assert_eq!(r.discovery.req_received.get(), 1);
assert_eq!(r.lookup.req_received.get(), 1);
}
}
src/node/mod.rs
+607 -448
View File
File diff suppressed because it is too large Load Diff
src/node/peering/driver.rs
+156
View File
@@ -0,0 +1,156 @@
//! Thin async driver for the peering reconciler.
//!
//! These `impl Node` methods are the I/O edge of the sans-IO
//! [`super::reconcile::PeeringReconciler`]: they snapshot the live dataplane
//! maps into the reconciler's plain-data inputs, invoke the pure core, and
//! perform the dial / advert-refetch I/O each [`PeeringAction`] names. They also
//! host the two gate-guarded reflex wrappers every peer-loss call site routes
//! through, so drain suppression and the connected-guard live in one place.
//!
//! The `Policy` / `Observed` / `Budget` builders these methods consume live in
//! [`crate::node::lifecycle`] next to the surviving budget helpers and limit
//! constants they wrap.
use crate::identity::NodeAddr;
use crate::node::{Node, NodeError};
use tracing::warn;
use super::reconcile::{DiscoveryPools, Gate, PeeringAction};
impl Node {
/// Reflex: an outbound handshake timed out (replaces the old
/// `Node::schedule_retry` call sites).
///
/// Replicates `schedule_retry`'s connected-guard — the pure core cannot
/// observe the peers map, so the driver drops the event when the peer is
/// already connected — then feeds the gate-guarded reconciler reflex with
/// the gate derived from the live published state.
pub(in crate::node) fn note_handshake_timeout(&mut self, node_addr: NodeAddr, now_ms: u64) {
if self.peers.contains_key(&node_addr) {
return;
}
let policy =
self.build_peering_policy(self.config().auto_connect_peers().cloned().collect());
let gate = Gate::from_state(self.supervisor.state);
let _ = self
.peering
.reconciler
.on_handshake_timeout(node_addr, now_ms, &policy, gate);
}
/// Reflex: a link went dead / a peer was lost (replaces the old
/// `Node::schedule_reconnect` call sites).
///
/// No connected-guard — the peer is already gone by the time a link-dead /
/// disconnect event fires (`schedule_reconnect` had none). The gate is
/// derived from the live published state so a drain self-suppresses the
/// reconnect.
pub(in crate::node) fn note_link_dead(&mut self, node_addr: NodeAddr, now_ms: u64) {
let policy =
self.build_peering_policy(self.config().auto_connect_peers().cloned().collect());
let gate = Gate::from_state(self.supervisor.state);
let _ = self
.peering
.reconciler
.on_link_dead(node_addr, now_ms, &policy, gate);
}
/// Process pending retries whose time has arrived (replaces the old
/// `Node::process_pending_retries` body).
///
/// The pure retry-dial phase owns the decision — drop expired entries, refuse
/// to grow when admission binds, dial the first `retry_per_tick` due entries
/// (bumping their `retry_after_ms` past the handshake window). This driver
/// performs the advert-refetch + dial I/O each emitted `Connect` names, and
/// on an immediate dial error feeds the `on_handshake_timeout` reflex so the
/// optimistic re-fire suppression is overwritten by proper backoff. During a
/// drain the gate is `Suspended`, so the reconcile clears the schedule and
/// emits nothing.
pub(in crate::node) async fn process_pending_retries(&mut self, now_ms: u64) {
if self.peering.reconciler.retry_pending.is_empty() {
return;
}
// Retry-dial cadence slot: empty config floor and empty discovery
// pools, so only the retry-dial phase acts.
let policy = self.build_peering_policy(Vec::new());
let observed = self.observe_peering();
let budget = self.build_peering_budget();
let gate = Gate::from_state(self.supervisor.state);
let actions = self.peering.reconciler.reconcile(
&policy,
&observed,
&budget,
&DiscoveryPools::default(),
now_ms,
gate,
);
for action in actions {
let PeeringAction::Connect(candidate) = action else {
continue;
};
let Some(identity) = candidate.identity else {
continue;
};
let node_addr = *identity.node_addr();
let Some(peer_config) = self
.peering
.reconciler
.retry_pending
.get(&node_addr)
.map(|state| state.peer_config.clone())
else {
continue;
};
// Kick off a refresh of the peer's overlay advert. The cache is
// read-only on hit, so a retry without a refetch dials the same
// cached endpoint — and the most common reason a peer landed in the
// retry schedule is that endpoint just stopped working (NAT rebind,
// port change, peer restart).
//
// Fire-and-forget, NOT awaited: this runs inline on the 1s rx-loop
// tick, and the fetch carries a 2s relay timeout that would stall
// the tick — and every other rx-loop arm with it — by up to 2s per
// due peer. So the dial below uses whatever advert is cached now
// and the refreshed one lands for the *next* retry of this peer.
// Retries are backoff-paced, so that defers the benefit by one
// backoff interval rather than losing it.
if let Some(bootstrap) = self.supervisor.nostr_rendezvous.engine_arc() {
let npub = peer_config.npub.clone();
tokio::spawn(async move {
let _ = bootstrap.refetch_advert_for_stale_check(&npub).await;
});
}
match self.initiate_peer_connection(&peer_config).await {
// The core already pushed `retry_after_ms` past the handshake
// window; a successful promotion clears the entry, a later
// timeout re-fires the reflex with proper backoff.
Ok(()) => {}
Err(e) => {
warn!(
peer = %self.peer_display_name(&node_addr),
error = %e,
"Retry connection initiation failed"
);
// No-transport failures usually mean the cached overlay
// advert is stale; force a re-fetch so the next tick picks up
// fresh endpoints.
if matches!(e, NodeError::NoTransportForType(_))
&& let Some(bootstrap) = self.supervisor.nostr_rendezvous.engine_arc()
{
let npub = peer_config.npub.clone();
tokio::spawn(async move {
let _ = bootstrap.refetch_advert_for_stale_check(&npub).await;
});
}
// Immediate failure counts as an attempt: overwrite the
// optimistic re-fire suppression with backoff.
self.note_handshake_timeout(node_addr, now_ms);
}
}
}
}
}
src/node/peering/mod.rs
+14
View File
@@ -0,0 +1,14 @@
//! Peering homeostasis — the desired-state controller for the node's peer set.
//!
//! This module is the home for the peering-reconciler concept: config defines a
//! desired peer set; the reconciler converges the observed set toward it
//! (auto-connect floor, overlay pool, transport-neighbor growth) under the
//! `node.limits` ceiling. Startup and steady-state are the same loop.
//!
//! The cross-attempt retry schedule (`retry.rs`) lives here because a fresh
//! connection is created per re-dial, so the escalating backoff count must
//! persist in the reconciler, not per-connection.
pub(in crate::node) mod driver;
pub(in crate::node) mod reconcile;
pub(in crate::node) mod retry;
src/node/peering/reconcile.rs
+1447
View File
File diff suppressed because it is too large Load Diff
src/node/peering/retry.rs
+49
View File
@@ -0,0 +1,49 @@
//! Cross-attempt retry state for auto-connect peers.
//!
//! [`RetryState`] is the durable per-peer schedule entry the peering reconciler
//! owns (it lives in [`crate::node::peering::reconcile::PeeringReconciler`], not
//! on a per-connection object, because a fresh connection is created per re-dial
//! so the escalating backoff count must persist across attempts). The decision
//! logic that reads and mutates it — the retry-dial phase and the
//! `on_handshake_timeout` / `on_link_dead` reflexes — lives in the sans-IO
//! reconciler core; the driver wrappers that feed it (retry-dial I/O, the
//! gate-guarded reflex call sites) live in [`super::driver`].
use crate::config::PeerConfig;
/// Per-tick cap on retry-dial connection attempts (ceiling).
pub(in crate::node) const MAX_RETRY_CONNECTIONS_PER_TICK: usize = 16;
/// Tracks retry state for a peer across connection attempts.
pub struct RetryState {
/// The peer config to use for initiating retries.
pub peer_config: PeerConfig,
/// Number of retries attempted so far.
pub retry_count: u32,
/// Timestamp (Unix ms) when the next retry should be attempted.
pub retry_after_ms: u64,
/// Whether this is an auto-reconnect (unlimited retries, ignores max_retries).
pub reconnect: bool,
/// Optional absolute expiry for this retry entry (Unix ms).
///
/// When set, retries are dropped after this point even if reconnect logic
/// would otherwise continue.
pub expires_at_ms: Option<u64>,
}
impl RetryState {
/// Create a new retry state for a peer.
pub fn new(peer_config: PeerConfig) -> Self {
Self {
peer_config,
retry_count: 0,
retry_after_ms: 0,
reconnect: false,
expires_at_ms: None,
}
}
}
src/node/reject.rs
+6 -2
View File
@@ -222,7 +222,7 @@ pub enum MmpReject {
/// Forwarding-path rejection reasons.
///
/// Each variant corresponds to a silent-rejection path in
/// `src/node/handlers/forwarding.rs::handle_session_datagram`. Matching
/// `src/node/dataplane/forwarding.rs::handle_session_datagram`. Matching
/// `ForwardingStats` counters already track packets and bytes for each
/// outcome; `record_reject` mirrors the packet-count side of the bump
/// for parity with the other rejection clusters.
@@ -232,7 +232,11 @@ pub enum ForwardingReject {
/// `SessionDatagramRef::decode` returned an error. Tracked via
/// [`ForwardingStats::decode_error_packets`](crate::node::stats::ForwardingStats).
DecodeError,
/// Datagram arrived with TTL=0 — already exhausted, no forward.
/// Transit datagram whose TTL would reach zero on this hop, so it is
/// dropped rather than forwarded. Charged for an arrival at TTL 1 as
/// well as an already-exhausted arrival at TTL 0. Never charged for a
/// datagram addressed to this node, whose delivery is not TTL-gated and
/// is decided ahead of this test.
/// Tracked via
/// [`ForwardingStats::ttl_exhausted_packets`](crate::node::stats::ForwardingStats).
TtlExhausted,
src/node/reloadable.rs
+1 -1
View File
@@ -41,7 +41,7 @@
//! file. There is nothing to poll. (Its read side could adopt the same
//! lock-free `ArcSwap` shape in the future, but that is an optimization, not
//! a reload.)
//! - `nostr_discovery` is an async spawned subsystem, not a snapshot of disk
//! - `nostr_rendezvous` is an async spawned subsystem, not a snapshot of disk
//! state.
//!
//! Both [`HostMapReloadable`] and the peer ACL reloader currently stat
src/node/retry.rs
-417
View File
@@ -1,417 +0,0 @@
//! Connection retry logic for auto-connect peers.
//!
//! When an outbound handshake fails (timeout or send error), the node can
//! automatically retry with exponential backoff. Retry state lives on Node
//! (not PeerConnection) because each retry creates a fresh connection.
use super::{Node, NodeError};
use crate::PeerIdentity;
use crate::config::PeerConfig;
use crate::identity::NodeAddr;
use tracing::{debug, info, warn};
// MAX_BACKOFF_MS is now derived from config: node.retry.max_backoff_secs * 1000
const MAX_RETRY_CONNECTIONS_PER_TICK: usize = 16;
/// Tracks retry state for a peer across connection attempts.
pub struct RetryState {
/// The peer config to use for initiating retries.
pub peer_config: PeerConfig,
/// Number of retries attempted so far.
pub retry_count: u32,
/// Timestamp (Unix ms) when the next retry should be attempted.
pub retry_after_ms: u64,
/// Whether this is an auto-reconnect (unlimited retries, ignores max_retries).
pub reconnect: bool,
/// Optional absolute expiry for this retry entry (Unix ms).
///
/// When set, retries are dropped after this point even if reconnect logic
/// would otherwise continue.
pub expires_at_ms: Option<u64>,
}
impl RetryState {
/// Create a new retry state for a peer.
pub fn new(peer_config: PeerConfig) -> Self {
Self {
peer_config,
retry_count: 0,
retry_after_ms: 0,
reconnect: false,
expires_at_ms: None,
}
}
/// Calculate the backoff delay in milliseconds for the current retry count.
///
/// Uses exponential backoff: `base_interval_ms * 2^retry_count`,
/// capped at `MAX_BACKOFF_MS`.
pub fn backoff_ms(&self, base_interval_ms: u64, max_backoff_ms: u64) -> u64 {
let multiplier = 1u64.checked_shl(self.retry_count).unwrap_or(u64::MAX);
base_interval_ms
.saturating_mul(multiplier)
.min(max_backoff_ms)
}
}
impl Node {
/// Schedule a retry for a failed outbound connection, if applicable.
///
/// Only schedules if the peer is an auto-connect peer and max retries
/// have not been exhausted (unless `reconnect` is true, which retries
/// indefinitely). Does nothing if the peer is already connected or has
/// a connection in progress.
pub(super) fn schedule_retry(&mut self, node_addr: NodeAddr, now_ms: u64) {
let retry_cfg = &self.config().node.retry;
let max_retries = retry_cfg.max_retries;
if max_retries == 0 {
return;
}
// Don't retry if peer is already connected
if self.peers.contains_key(&node_addr) {
return;
}
let base_interval_ms = retry_cfg.base_interval_secs * 1000;
let max_backoff_ms = retry_cfg.max_backoff_secs * 1000;
let peer_name = self.peer_display_name(&node_addr);
if let Some(state) = self.retry_pending.get_mut(&node_addr) {
// Already tracking — increment
state.retry_count += 1;
if !state.reconnect && state.retry_count > max_retries {
info!(
peer = %peer_name,
attempts = state.retry_count,
"Max retries exhausted, giving up on peer"
);
self.retry_pending.remove(&node_addr);
return;
}
let delay = state.backoff_ms(base_interval_ms, max_backoff_ms);
state.retry_after_ms = now_ms + delay;
debug!(
peer = %peer_name,
retry = state.retry_count,
reconnect = state.reconnect,
delay_secs = delay / 1000,
"Scheduling connection retry"
);
} else {
// First failure — find the matching PeerConfig
let peer_config = self
.config()
.auto_connect_peers()
.find(|pc| {
PeerIdentity::from_npub(&pc.npub)
.map(|id| *id.node_addr() == node_addr)
.unwrap_or(false)
})
.cloned();
if let Some(pc) = peer_config {
let mut state = RetryState::new(pc);
state.retry_count = 1;
state.reconnect = true;
let delay = state.backoff_ms(base_interval_ms, max_backoff_ms);
state.retry_after_ms = now_ms + delay;
debug!(
peer = %self.peer_display_name(&node_addr),
delay_secs = delay / 1000,
"First connection attempt failed, scheduling retry"
);
self.retry_pending.insert(node_addr, state);
}
// If not found in auto_connect_peers, no retry (one-shot connection)
}
}
/// Schedule auto-reconnect for a peer removed by MMP dead timeout.
///
/// Looks up the peer in auto-connect config and checks `auto_reconnect`.
/// If enabled, feeds the peer into the retry system with unlimited retries.
///
/// If a retry entry already exists (e.g. from a previous failed handshake
/// attempt during an earlier reconnect cycle), the existing retry count is
/// preserved and incremented rather than reset to zero. This ensures
/// exponential backoff accumulates across repeated link-dead events instead
/// of resetting to the base interval on every peer removal.
pub(super) fn schedule_reconnect(&mut self, node_addr: NodeAddr, now_ms: u64) {
// Find peer in auto-connect config
let peer_config = self
.config()
.auto_connect_peers()
.find(|pc| {
PeerIdentity::from_npub(&pc.npub)
.map(|id| *id.node_addr() == node_addr)
.unwrap_or(false)
})
.cloned();
let Some(pc) = peer_config else {
return; // Not an auto-connect peer, no reconnect
};
if !pc.auto_reconnect {
debug!(
peer = %self.peer_display_name(&node_addr),
"Auto-reconnect disabled for peer, skipping"
);
return;
}
let base_interval_ms = self.config().node.retry.base_interval_secs * 1000;
let max_backoff_ms = self.config().node.retry.max_backoff_secs * 1000;
let peer_name = self.peer_display_name(&node_addr);
// If we already have accumulated backoff from previous failed attempts,
// preserve and bump it rather than resetting to zero. This prevents the
// exponential backoff from being discarded on each link-dead cycle.
if let Some(state) = self.retry_pending.get_mut(&node_addr) {
state.reconnect = true;
state.retry_count += 1;
let delay = state.backoff_ms(base_interval_ms, max_backoff_ms);
state.retry_after_ms = now_ms + delay;
debug!(
peer = %peer_name,
retry = state.retry_count,
delay_secs = delay / 1000,
"Scheduling auto-reconnect after link-dead removal (backoff preserved)"
);
return;
}
let mut state = RetryState::new(pc);
state.reconnect = true;
let delay = state.backoff_ms(base_interval_ms, max_backoff_ms);
state.retry_after_ms = now_ms + delay;
debug!(
peer = %peer_name,
delay_secs = delay / 1000,
"Scheduling auto-reconnect after link-dead removal"
);
self.retry_pending.insert(node_addr, state);
}
/// Process pending retries whose time has arrived.
///
/// For each due retry, initiates a fresh connection attempt. The retry
/// entry stays in `retry_pending` until the connection succeeds (cleared
/// in `promote_connection`) or max retries are exhausted (cleared in
/// `schedule_retry`).
pub(super) async fn process_pending_retries(&mut self, now_ms: u64) {
if self.retry_pending.is_empty() {
return;
}
let expired: Vec<NodeAddr> = self
.retry_pending
.iter()
.filter_map(|(addr, state)| {
state
.expires_at_ms
.filter(|expires_at_ms| now_ms >= *expires_at_ms)
.map(|_| *addr)
})
.collect();
for node_addr in expired {
self.retry_pending.remove(&node_addr);
info!(
peer = %self.peer_display_name(&node_addr),
"Retry window expired, dropping pending retry state"
);
}
if self.retry_pending.is_empty() {
return;
}
if !self.outbound_admission_check() {
debug!(
peers = self.peers.len(),
max_peers = self.max_peers(),
retry_pending = self.retry_pending.len(),
"Suppressing auto-reconnect retries: at capacity"
);
return;
}
// Collect retries that are due
let due: Vec<NodeAddr> = self
.retry_pending
.iter()
.filter(|(_, state)| now_ms >= state.retry_after_ms)
.map(|(addr, _)| *addr)
.collect();
let deferred = due.len().saturating_sub(MAX_RETRY_CONNECTIONS_PER_TICK);
if deferred > 0 {
debug!(
due = due.len(),
processing = MAX_RETRY_CONNECTIONS_PER_TICK,
deferred,
"Retry processing budget exhausted; deferring remaining peers"
);
}
for node_addr in due.into_iter().take(MAX_RETRY_CONNECTIONS_PER_TICK) {
// Peer may have connected inbound while we waited
if self.peers.contains_key(&node_addr) {
self.retry_pending.remove(&node_addr);
continue;
}
let state = match self.retry_pending.get(&node_addr) {
Some(s) => s,
None => continue,
};
debug!(
peer = %self.peer_display_name(&node_addr),
retry = state.retry_count,
"Attempting connection retry"
);
let peer_config = state.peer_config.clone();
// Refresh the peer's overlay advert before retrying. The cache is
// read-only on hit (see fetch_advert), so every retry without a
// refetch dials the same cached endpoint — and the most common
// reason a peer ended up in retry_pending is that the cached
// endpoint just stopped working (NAT rebind, port change, peer
// restart on a different port). Without this refresh the retry
// loop dials the same dead address forever.
//
// refetch_advert_for_stale_check uses the relay's advert as
// ground truth: replaces the cache if there's a newer one,
// evicts if the relay has nothing, otherwise leaves it. Cheap
// (one Filter fetch with 2s timeout) and bounded by the retry
// backoff cadence.
if let Some(bootstrap) = self.nostr_discovery.clone() {
let _ = bootstrap
.refetch_advert_for_stale_check(&peer_config.npub)
.await;
}
match self.initiate_peer_connection(&peer_config).await {
Ok(()) => {
// Push retry_after_ms past the handshake timeout window so
// we don't re-fire on the next tick. If the handshake
// succeeds, promote_connection() clears retry_pending. If
// it times out, check_timeouts() calls schedule_retry()
// which bumps the counter and applies proper backoff.
let hs_timeout_ms = self.config().node.rate_limit.handshake_timeout_secs * 1000;
if let Some(state) = self.retry_pending.get_mut(&node_addr) {
state.retry_after_ms = now_ms + hs_timeout_ms;
}
debug!(
peer = %self.peer_display_name(&node_addr),
"Retry connection initiated, suppressing re-fire for {}s",
self.config().node.rate_limit.handshake_timeout_secs,
);
}
Err(e) => {
warn!(
peer = %self.peer_display_name(&node_addr),
error = %e,
"Retry connection initiation failed"
);
// No-transport failures usually mean the cached overlay
// advert is stale (peer rebound NAT, switched relay, etc.).
// The advert cache is read-only inside fetch_advert, so
// every retry returns the same dead address until the
// entry expires. Force a re-fetch so the next retry tick
// picks up fresh endpoints.
if matches!(e, NodeError::NoTransportForType(_))
&& let Some(bootstrap) = self.nostr_discovery.clone()
{
let npub = peer_config.npub.clone();
tokio::spawn(async move {
let _ = bootstrap.refetch_advert_for_stale_check(&npub).await;
});
}
// Immediate failure counts as an attempt — schedule next retry
// (reconnect flag is preserved on existing retry_pending entry)
self.schedule_retry(node_addr, now_ms);
}
}
}
}
}
#[cfg(test)]
mod tests {
use super::*;
use crate::config::PeerConfig;
const TEST_MAX_BACKOFF_MS: u64 = 300_000;
#[test]
fn test_backoff_exponential() {
let state = RetryState {
peer_config: PeerConfig::default(),
retry_count: 0,
retry_after_ms: 0,
reconnect: false,
expires_at_ms: None,
};
// base = 5000ms
assert_eq!(state.backoff_ms(5000, TEST_MAX_BACKOFF_MS), 5000); // 5s * 2^0
let state = RetryState {
retry_count: 1,
..state
};
assert_eq!(state.backoff_ms(5000, TEST_MAX_BACKOFF_MS), 10_000); // 5s * 2^1
let state = RetryState {
retry_count: 2,
..state
};
assert_eq!(state.backoff_ms(5000, TEST_MAX_BACKOFF_MS), 20_000); // 5s * 2^2
let state = RetryState {
retry_count: 3,
..state
};
assert_eq!(state.backoff_ms(5000, TEST_MAX_BACKOFF_MS), 40_000); // 5s * 2^3
let state = RetryState {
retry_count: 4,
..state
};
assert_eq!(state.backoff_ms(5000, TEST_MAX_BACKOFF_MS), 80_000); // 5s * 2^4
}
#[test]
fn test_backoff_cap() {
let state = RetryState {
peer_config: PeerConfig::default(),
retry_count: 20, // 2^20 * 5000 would be huge
retry_after_ms: 0,
reconnect: false,
expires_at_ms: None,
};
assert_eq!(
state.backoff_ms(5000, TEST_MAX_BACKOFF_MS),
TEST_MAX_BACKOFF_MS
);
}
#[test]
fn test_backoff_zero_base() {
let state = RetryState {
peer_config: PeerConfig::default(),
retry_count: 3,
retry_after_ms: 0,
reconnect: false,
expires_at_ms: None,
};
assert_eq!(state.backoff_ms(0, TEST_MAX_BACKOFF_MS), 0);
}
}
src/node/routing_error_rate_limit.rs
-161
View File
@@ -1,161 +0,0 @@
//! Routing error signal rate limiting.
//!
//! Prevents routing error floods (CoordsRequired / PathBroken) by
//! rate-limiting error signals per destination address at transit nodes.
use crate::NodeAddr;
use std::collections::HashMap;
use std::time::{Duration, Instant};
/// Rate limiter for routing error signals (CoordsRequired / PathBroken).
///
/// Tracks the last time a routing error was sent for each destination
/// address and enforces a minimum interval to prevent floods.
pub struct RoutingErrorRateLimiter {
/// Maps destination NodeAddr to the last time we sent an error about it.
last_sent: HashMap<NodeAddr, Instant>,
/// Minimum interval between error signals for the same destination.
min_interval: Duration,
/// Maximum age of entries before cleanup.
max_age: Duration,
}
impl RoutingErrorRateLimiter {
/// Create a new rate limiter.
///
/// Default: max 10 errors/sec per destination (100ms interval).
pub fn new() -> Self {
Self {
last_sent: HashMap::new(),
min_interval: Duration::from_millis(100),
max_age: Duration::from_secs(10),
}
}
/// Create a rate limiter with a custom minimum interval.
pub fn with_interval(min_interval: Duration) -> Self {
Self {
last_sent: HashMap::new(),
min_interval,
max_age: Duration::from_secs(10),
}
}
/// Check if we should send a routing error for this destination.
///
/// Returns true if enough time has passed since the last error for
/// this destination, or if this is the first error. Updates internal
/// state when returning true.
pub fn should_send(&mut self, dest_addr: &NodeAddr) -> bool {
let now = Instant::now();
if let Some(&last) = self.last_sent.get(dest_addr)
&& now.duration_since(last) < self.min_interval
{
return false;
}
self.last_sent.insert(*dest_addr, now);
self.cleanup(now);
true
}
/// Remove entries older than max_age.
fn cleanup(&mut self, now: Instant) {
self.last_sent
.retain(|_, &mut last| now.duration_since(last) < self.max_age);
}
#[cfg(test)]
pub fn len(&self) -> usize {
self.last_sent.len()
}
}
impl Default for RoutingErrorRateLimiter {
fn default() -> Self {
Self::new()
}
}
#[cfg(test)]
mod tests {
use super::*;
use std::thread;
fn addr(val: u8) -> NodeAddr {
let mut bytes = [0u8; 16];
bytes[0] = val;
NodeAddr::from_bytes(bytes)
}
#[test]
fn test_first_send_allowed() {
let mut limiter = RoutingErrorRateLimiter::new();
assert!(limiter.should_send(&addr(1)));
}
#[test]
fn test_rapid_sends_rate_limited() {
let mut limiter = RoutingErrorRateLimiter::new();
assert!(limiter.should_send(&addr(1)));
assert!(!limiter.should_send(&addr(1)));
assert!(!limiter.should_send(&addr(1)));
}
#[test]
fn test_different_destinations_independent() {
let mut limiter = RoutingErrorRateLimiter::new();
assert!(limiter.should_send(&addr(1)));
assert!(limiter.should_send(&addr(2)));
assert!(!limiter.should_send(&addr(1)));
assert!(!limiter.should_send(&addr(2)));
}
#[test]
fn test_send_allowed_after_interval() {
let mut limiter = RoutingErrorRateLimiter::new();
assert!(limiter.should_send(&addr(1)));
thread::sleep(Duration::from_millis(110));
assert!(limiter.should_send(&addr(1)));
}
#[test]
fn test_cleanup_removes_old_entries() {
let mut limiter = RoutingErrorRateLimiter::new();
assert!(limiter.should_send(&addr(1)));
assert!(limiter.should_send(&addr(2)));
assert_eq!(limiter.len(), 2);
let future = Instant::now() + Duration::from_secs(11);
limiter.cleanup(future);
assert_eq!(limiter.len(), 0);
}
#[test]
fn test_cleanup_preserves_recent_entries() {
let mut limiter = RoutingErrorRateLimiter::new();
assert!(limiter.should_send(&addr(1)));
assert_eq!(limiter.len(), 1);
limiter.cleanup(Instant::now());
assert_eq!(limiter.len(), 1);
}
#[test]
fn test_with_interval_custom_rate() {
let mut limiter = RoutingErrorRateLimiter::with_interval(Duration::from_millis(500));
assert!(limiter.should_send(&addr(1)));
assert!(!limiter.should_send(&addr(1)));
// Still rate-limited after 200ms (would pass with default 100ms)
thread::sleep(Duration::from_millis(200));
assert!(!limiter.should_send(&addr(1)));
// Allowed after 500ms total
thread::sleep(Duration::from_millis(350));
assert!(limiter.should_send(&addr(1)));
}
}

Some files were not shown because too many files have changed in this diff Show More