The reap runs two selectors, one on the CI label and one on the
compose project, and each flag narrows only its own. So --run-id
alone leaves the project sweep broad and --project-prefix alone
leaves the label sweep broad, and either way the reap still destroys
every concurrent run on the host. Both single-flag forms look scoped
and are not.
The usage text asserted otherwise. It documented --run-id as leaving
other runs alone and --project-prefix as scoping the reap to a single
run, and both claims were false for the same reason. A caller passing
--run-id alone, on the strength of that line, force-removed the
containers of three concurrent runs on 2026-07-29, and its own
comment recorded the belief that it was scoped.
Rather than ask every caller to remember the pair, close the two
half-scoped states here. --run-id now derives the matching project
prefix when none is given, built from the existing base so the two
cannot drift, and an explicit --project-prefix still wins.
--project-prefix without --run-id is refused outright, because there
is nothing to derive a label scope from and the quiet failure is
someone else's run disappearing.
Passing neither flag is untouched: that is the deliberate "reap
everything" form a manual cleanup wants.
Verified against a decoy container labelled as another run: the
scoped reap leaves it up, and reproducing the old broad project sweep
removes it, so the check discriminates rather than passing because
there was nothing to reap. An earlier version of that check ran when
no CI containers existed at all and passed without proving anything.
Each peer must be sent the union of all other peers' inbound filters,
excluding its own contribution. Building that per recipient rebuilt
the whole peer-filter map and re-ORed it once for every peer, so a
tick that announced to R peers did R kilobyte-scale map builds and
R by T merges. At 240 peers this was 20.6 ms per tick, roughly half
the tick body, and it was steady work rather than a tail: the
per-interval maximum had a median of 34.5 ms.
Replace it with a prefix and suffix union sweep that produces every
target's filter in one pass, T merges instead of R by T. The packet
path gets the same treatment, since marking changed peers had the
identical shape once per inbound announce.
The result is exactly equal, not approximately. Merging is a bytewise
OR, so regrouping the unions cannot change the outcome, and a filter
whose size does not match is rejected before any byte is touched, at
every merge site in both the old and new arrangement. Every
accumulator here is default-sized, so an odd-sized peer filter is
skipped in the new code exactly where it was skipped in the old.
Cadence, the debounce, the sequence rule and the fill-ratio cap are
untouched. A sequence number is still drawn after the debounce
re-check and before encoding, so a suppressed peer still consumes
none and an encode failure still burns one.
Known trade-off, measured rather than assumed: the sweep does its
full O(T) work regardless of how many peers are ready, so a tick that
announces to only one or two peers now costs about twice what it did.
Break-even is around three ready peers, and the saving above that
grows without bound. The marking path is a pure win, since it always
targets every peer.
The per-tick stats snapshot ran a bech32 encode for every tracked
peer, and for the common mesh peer — one with no hosts-file entry and
no configured alias — it ran a second one, because the display-name
fallback chain bottoms out in the same encode. At 240 peers that was
14.1 ms per tick, a third of the tick body and its second largest
cost, all of it recomputing values that cannot change.
Cache the npub and the shortened npub on the peer at construction. An
npub is a pure function of the peer's public key, and the identity is
never mutated after construction: there is no setter, no identity_mut,
and no assignment to the field anywhere in the tree, so the cache
cannot go stale.
The display name itself is deliberately NOT cached. Two of its inputs
do mutate at runtime — the alias map and the host map, the latter
reloaded on this same tick — so a resolved name stored on the peer
would go stale on an alias change or a hosts reload. Only the
immutable component is memoized.
Tests cover both constructors, that the cached npub matches the
identity, and that the display name still tracks an alias change. The
memoization itself is asserted by pointer stability rather than by
timing, so it is deterministic under load. Three deliberate breaks
were each caught by exactly one test: re-deriving instead of
memoizing, populating one constructor's cache from the wrong source,
and reordering the display-name fallback so it stops honoring aliases.
Every node reported state: degraded permanently. The liveness probe
polled connect_task, which wraps a single Client::connect() call. That
call only spawns a per-relay background connection task and returns,
so the handle finished moments after start on a perfectly healthy
node, the supervisor saw a child exit, and the node latched Degraded
for the rest of its life.
Nothing behaved differently, because every consumer of NodeState
treats Degraded the same as Running. What was lost is the signal: a
genuine degradation was indistinguishable from the permanent false
one.
Watch the three service loops that cannot return by design instead —
the inbound notify loop, the advert publisher, and the refresh ticker.
Each is an unconditional loop, so a finished handle means a panic or
an abort, which is unrecoverable and matches the one-way ChildExited
latch in the supervisor. connect_task and relay_startup_task stay
deliberately unwatched, and the doc comment now says why, since
watching either reproduces this bug exactly.
The tests install task handles directly and check both directions:
that a finished connect_task alongside three live loops reports
healthy, which is the production configuration a few hundred
milliseconds after start, and that each loop dying on its own reports
degraded. Reverting to the old predicate reds four of five; replacing
the predicate with a constant false also reds four of five, so the
fix cannot pass by never reporting degraded at all.
process_pending_retries runs inline on the node's 1-second rx-loop
tick. For each due peer it awaited a Nostr relay fetch carrying a
2-second timeout, and discarded the result. With up to sixteen due
peers in one tick body, the timeouts stack: field profiling measured
single 2.00 s stalls as the common case and a worst tick of 12.4 s
against a 1 s period, with every other rx-loop arm delayed behind it
by as much as 4.2 s.
Spawn the refetch instead of awaiting it, matching the pattern the
failure arm of the same loop already uses thirty lines below. The dial
now uses whatever advert is cached at that moment and the refreshed
one lands for the next retry of that peer. Since retries are
backoff-paced, that defers the benefit by one backoff interval rather
than losing it, and the result was already being discarded, so nothing
downstream read it.
The test drives four due peers whose refetches all hang against a
local listener that accepts and never speaks, so the fetch burns its
full timeout with no network egress. Awaited, the call takes 8.0 s;
spawned, it returns in milliseconds. It also asserts every due peer
was still attempted and rescheduled, so a version that skipped the
dial entirely cannot pass it.
A traversal signal is addressed to a merge of the peer's NIP-17 inbox
relays, the relays its advert nominates for signaling, and our own DM
relays. The client pool is built once at startup from our configured
relays and never added to, and send_event_to rejects the entire send
with "relay not found" if any single URL in the list is outside that
pool, before contacting anything. So one relay we are not configured
with, anywhere in that merge, killed the whole attempt -- including the
sends to relays we do share and that would have carried the signal.
In an open-mode window on a public node this made discovery
non-functional: 309 traversal attempts, 290 explicit failures, zero
successes, every failure on "relay not found". Configured peers were
unaffected because they run a matching relay set.
Filter the merged list down to relays the pool holds before sending.
Our own DM relays are always in the merge and always in the pool, so
the result is empty only when no DM relay is configured at all, which
is already a total failure. Comparison is on the normalized RelayUrl
rather than the raw string, because that is how the pool is keyed --
a raw comparison would discard a configured relay spelled with a
trailing slash or a different host case, which is the same defect in
a quieter form.
Merge and filter are one synchronous function so the decision can be
exercised without a relay client. The pool is read via all_relays(),
which is the set send_event_to validates against; relays() is filtered
by service flags and would be narrower.
Two smaller fixes ride along. The responder now resolves its relays
before binding a socket and running STUN, instead of spending a STUN
round trip and holding an offer slot only to discover it has nowhere
to answer. And it gained the empty-list guard the initiator already
had, which gives BootstrapError::MissingRelays a condition it can
reach for the first time -- it was unreachable, since the merge always
appended our own DM relays.
A measurement run needs the .deb built with a non-default feature, and the
script had no way to express that: its argument loop took only --target,
--version and --no-build, and nothing forwarded a feature list or read one
from the environment. --features <list> now forwards to cargo-deb's native
-F.
The marking is the half that is easy to skip and matters more. The
auto-derived dev Version is built from the crate version, the commit date
and the sha, none of which change when a feature is enabled, so an
instrumented package and a default package of the same commit carried
byte-identical versions. Two consequences, and the second is worse than
the first: the node offers no way to tell which one it is running, and
reverting is an install of a version already present, which no-ops
silently and leaves the instrumented binary in place. That is precisely
the failure the per-commit version was introduced to prevent, reappearing
one level down. The Version now carries a +<features> marker, folded to
dots since underscores and commas are not legal there.
The marker sorts above the unmarked build, checked with dpkg
--compare-versions rather than assumed, so installing a feature build is
an upgrade and reverting is a downgrade: revert with dpkg -i, not apt
install. The ~dev ordering below a tagged release is preserved.
--features is refused together with --no-build, which would stamp the
marker onto whatever binaries happened to be sitting in target/ while
claiming the features had reached them.
Verified end to end rather than by reading: a real --features profiling
build produces a package whose fipsctl accepts the profile subcommand and
whose daemon carries the capture-file header text that only exists under
the feature gate. The release packaging path is untouched, shown by
diffing the cargo invocation the old and new scripts produce across five
argument shapes including the --version/--no-build pair the packaging
workflow uses; all five are identical.
Authored on master rather than maint, which takes bug fixes and CI or
tooling changes but not new capability, so the three copies of this
script now differ by design.
The mesh rehearsal could not exercise the write-error stop at all, and
that is not a gap in the rehearsal. Removing the sink file leaves the
writer's descriptor valid, so writes keep succeeding into the unlinked
inode and the capture runs on; mounting a tiny filesystem inside the test
container is refused outright. So the terminal state added for a failing
sink had no coverage from either direction.
Split one flush cycle out of the writer loop as a function returning what
it decided, so the loop owns the waiting and the state transition while
the decision is testable on its own. A sink that fails every write now
drives the error outcome directly, and a working sink is asserted to
continue so the first test cannot pass for a writer that always stops.
The residual gap is recorded at the test: what is covered is that an
error from the sink produces the error outcome rather than the cap
outcome, not that a real full disk reaches that branch.
The rehearsal itself was otherwise clean on a live three-node mesh. The
header carries node, build, platform, tick period and the reading
caveats; every step reports every interval; stopping returns in 53 ms
rather than waiting out the flush interval; an idle status reports no
path and no bytes; arming twice is refused naming the active file; an
unwritable directory fails the command; and the node kept serving
throughout. On an idle node the entry gap sits at one tick period and
arm starvation reads about 1.4 ms, which is small but not zero, so the
measurement is live rather than degenerate.
The tick arm runs twenty-six housekeeping steps in sequence on the one
runtime thread, and is polled last, so anything slow in it holds up
inbound packets, TUN traffic and control commands behind it. Field
evidence says that happens for over a second at a time, but the
attribution behind that is two months old and predates the
connect-on-send gate, the control read isolation, and the peer lifecycle
rework. This measures it rather than continuing to reason about it.
Per step it records exact count, max and total into fixed static
counters; a dedicated writer thread drains them every ten seconds to a
TSV under /var/log/fips, one file per capture, capped at 32 MB. Nothing
accumulates: the counters are swapped to zero each interval and the
thread holds no history. Arming is `fipsctl profile tick on`, served in
the control accept task so the toggle cannot queue behind the very
behaviour it measures, and it does not survive a restart.
The whole thing is behind a Cargo feature that is off by default,
because the risk worth eliminating is the twenty-six edited call sites
in the hot loop. With the feature off the macro expands to the bare
expression, which makes the default build's neutrality something you
read off the generated code rather than something a benchmark fails to
disprove.
The measurement that matters is how late each tick is against the
deadline it was scheduled for, since the arm is polled last and that
lateness is the delay. Two earlier designs derived it from the interval
between entries and both under-reported: the schedule is fixed, so a
steady delay leaves every gap exactly one period and any gap-derived
figure reads zero under precisely the sustained overload this is meant
to find. The interval hands back its own deadline, so the delay is now a
subtraction with no model behind it, and a test drives three late ticks
at a constant gap to keep it that way.
CI gains a default-features clippy and a feature-on build and test on
both runners, closing the gap left by clippy already running with all
features.
The NAT lab was the last suite pinning fixed IPv4 subnets, so two overlapping
runs collided on the wan and shared-lan bridges.
Each run now claims a free /24 for each bridge, scanning candidates and
advancing on an overlap while still failing fast on any other network-create
error. The claim exports NAT_WAN_PREFIX and NAT_LAN_PREFIX, and every routable
address in the compose file and the suite scripts derives from them, with
defaults that render exactly what the lab used before. The router-side LANs are
deliberately left pinned: they live inside per-container network namespaces,
never become docker networks, and cannot collide.
An external-network overlay lets the suites attach to the networks the run
already claimed instead of creating their own. Two of the three suite scripts
had no overlay hook, so they would have requested the claimed range a second
time; both now have one.
Host veth names are scoped by a short token rather than the full run id, which
overruns the fifteen-character interface-name limit at the default run-id
length, and the cleanup reaper's pattern is widened to match the new shape.
Networks are released inline on every exit path rather than in a trap, since a
trap written inside a shell function replaces the script-level handler and
would disable the whole run's teardown.
The NAT lab wrote its generated node configs to one shared directory, unlike
the static and firewall labs which already scope theirs by run. The directory
is bind-mounted by compose and read back by the suite scripts after the
containers are up, so two overlapping runs let the second run's generator
overwrite the npubs the first is about to ping.
Scope the directory with FIPS_CI_NAME_SUFFIX at all three places that have to
agree: the generator's output path, the ten compose bind-mounts, and the
CONFIG_DIR the suite scripts read back from. An unset suffix renders the plain
path the lab has always used, so a bare invocation and the GitHub matrix are
unaffected. The run teardown removes the per-run directory alongside the static
and firewall ones, and the gitignore is widened to cover the suffixed form.
Rendering the compose file with every profile and no suffix set reproduces
today's exact ten paths; setting a suffix moves all ten.
This lands on its own, ahead of the network and address work, so that a failure
of the two-overlapping-runs acceptance test can be bisected between the config
fix and the address conversion.
Carries today's maint batch: the local CI image-scoping work, which gives
each run its own build context and test image tag instead of writing the
shared mutable one, and the retirement of the bloom-storm chaos scenario.
Both apply unchanged here — the compose files, suite lists and matrix legs
they touch are identical on the two lines, so no branch adaptation was
needed. Parity stays symmetric across runners at 21 legs a side on this
branch and 24 on maint, the gap being the three rekey Docker suites maint
keeps by design.
The scenario guarded a real regression: a mid-chain tree update that
changed neither root nor depth leaking downstream as a sustained bloom
announce storm. But it was never once run against the regressed binary,
and its per-node ceiling was inferred from a post-mortem harness that no
longer exists in the tree. On the only surviving regressed measurement
the tail node's rate scales to roughly 7 sends per 30s, well under the
scenario's ceiling of 40, so it was never established that the assertion
could fire on its own bug class. The ceiling is also uniform per node,
calibrated against the flap target that is legitimately busy rather than
against the tail where the storm actually shows.
This removes coverage rather than relocating it, unlike the two earlier
retirements above it, and the comment in ci-local.sh records that gap
explicitly. The scenario, its README, the link_swap sim primitive and the
mesh-lab dispatch all stay on disk, so it remains runnable by hand.
Parity holds at 24 legs a side. The parity guard was break-checked by
re-adding the GitHub leg alone, which correctly reported the asymmetry.
The run built per-run images and then retagged them to fips-test:latest as
a compatibility bridge for the consumers that had never been migrated. The
bridge was the defect: while it existed, two concurrent runs shared one
binding, so a suite could start containers from the other run's binaries
and the verdict was recorded against a commit whose code never ran. It
fails silently by construction — the run is green either way, and nothing
compares a running container's binary against the commit under test.
Every consumer now reads the run's image, so the retag is deleted rather
than kept. That is the point of deleting it: a consumer that was missed
fails loudly instead of quietly resolving whichever run wrote the tag last.
Two consequences handled here. The cleanup script ran ip(8) inside
fips-test:latest to reap simulation interfaces, and nothing writes that tag
any more, so it now takes the caller's image, then the run's, then any
surviving test image — the last of which is what keeps an unscoped --reap
working, since that path execs before the run identity is exported. And a
guard checks statically that nothing names the shared tag, because on a
host with a hand-built copy lying around a reintroduction would run green;
it is break-checked against a reintroduced compose consumer and a
reintroduced default. Both runners gate on it, as they do the other guards.
Scoping the image tag was never sufficient on its own. Every build read one
unscoped directory in the working tree, into which each run copies the
binaries it just built, so two concurrent runs raced on the contents of the
context as well as on the name of the result — and a run could produce a
correctly-per-run-tagged image built from the other run's binaries.
The run now copies the context's tracked files into its own directory,
installs its binaries there, and builds from it, exporting the path so every
other consumer follows. Deliberately not carried over: a previous run's
binaries, since inheriting them is the failure this prevents. The path is
absolute because compose resolves a relative build context against the
compose file's own directory rather than the working directory, which was
measured rather than assumed.
The chaos entry script gated the whole simulation on a binary in the shared
directory by literal path, so it moves in this same commit: left behind, it
would have failed on a clean checkout and, worse on a host with leftovers,
passed while reading a binary that was not the one under test.
Teardown removes the directory on red runs as well as green, since it holds
only reproducible content and is never the evidence of a failure. The
worker's SIGKILL runs no trap, so the cleanup script also sweeps contexts
left by a preempted run, and the ignore rule keeps a concurrent run from
showing up as untracked working-tree noise.
The runner rebuilt the shared tag from the shared build context at the
start of every scenario, unconditionally. Local CI runs chaos scenarios in
parallel, so that was several concurrent builds into one name inside a
single run, before any second run is considered, and it silently replaced
whatever image the harness had already built and handed over.
It now uses the caller's image when one is named, and asserts the image is
present rather than building a substitute, because under a harness a miss
means something upstream is broken and building would hide it. A bare run
still builds the shared tag exactly as before, reading the run's own build
context when one is set.
Only the static family read FIPS_TEST_IMAGE; the firewall, ACL, nat,
sidecar and both Tor compose files named the shared mutable tag directly,
which is why the local CI runner has to retag its per-run image to that
name. They now take the same defaulted form the static file already uses,
so a harness run resolves the image it built and a bare hand run still
resolves the shared tag exactly as before.
Checked by rendering each file through `docker compose config` with the
variables set: every service resolves to the supplied name and none is
left naming the shared tag.
Both suites ran `docker compose up -d --build` unconditionally, and their
compose files carry both a build context and an image name, so the flag
rebuilt and retagged the shared test image from the shared build context on
every run. Their --skip-build flag did not cover it: it guarded only the
Rust build above. Under a harness that has already built the image and
handed it over, that rebuild can only replace the binaries under test with
whatever the shared context currently holds.
The nat scripts had the mirror-image problem. Their image guard built a
replacement whenever the image was missing, which is right by hand and
wrong under a harness: when the caller names an image it built, a miss
means something upstream is broken, and manufacturing a substitute hides
that. They now fail loudly in that case and keep building only on the hand
path. The mesh-lab guard reads the same variable and its build hint moves
to the script that still produces the shared tag.
The root-count ceiling of 4 was set from six runs whose observed maximum
was 3. Fourteen runs that recorded a baseline verdict put the real
distribution at 1 to 5 roots, so the ceiling sat at roughly the 93rd
percentile of the scenario's own variance: one sample in fourteen
exceeded it and four sat at or above it. It therefore reddened a share of
runs whatever the daemon did, which is a calibration fault rather than a
convergence one. Every sample ran the file's fixed seed, so the spread is
container timing and not a differing chaos schedule.
The ceiling is now 6, one step beyond the observed maximum, and the
parented floor moves to its complement at 4. That still fails a mesh
where seven or more of ten nodes are islanded, which is the collapse case
the assertion exists to catch; driving evaluate_baseline directly
confirms all fourteen samples pass and 7, 8 and 10 roots still fail.
min_sessions is left at 10 against an observed minimum of 12. It has
never fired, but the comment now records the margin as thin so a future
failure there is read as calibration first.
All three required root or CAP_NET_RAW, neither runner passes --ignored,
and so none had ever executed in CI. Before deleting them I checked what
each covered and whether anything else covered it:
- The two-node handshake is covered by the ethernet-only chaos scenario,
whose baseline assertion cannot pass unless handshakes complete over
AF_PACKET.
- Tree convergence and root election by smallest NodeAddr are covered by
ethernet-only's single-root assertion and by the in-process
spanning-tree tests respectively.
- The mixed-transport coexistence property is covered twice over by
test_tcp_mixed_transport_coexistence and test_ble_mixed_transport, which
call the same verify_tree_convergence_components helper with the same
two-component shape and both run today.
So the only thing running them would have added is an AF_PACKET variant of
a property already proven on two other transports.
The counts confirm nothing running was lost: 1392 tests pass before and
after, and the ignored count falls from 7 to 4.
The ethernet-only comment block cited these tests as the reason its
assertion was the only Ethernet coverage in CI. It now records the sharper
fact found while reading them: that coverage is control plane only. Both
Ethernet scenarios disable traffic, so no datagram crosses an Ethernet
link anywhere, which leaves the frame length field that trims NIC minimum
frame padding ahead of AEAD verification unexercised. Deleting these tests
does not widen that gap, because the test named for data exchange did not
exchange any.
The shared analyzer matched two strings for one event: "Peer promoted to
active", emitted at info, and "Outbound handshake completed", emitted at
debug. Both come from handle_msg2 on the same call path for the same
outbound promotion, so any run at debug level counted every outbound
promotion twice.
Measured rather than argued. On the archived debug-level churn-mixed run
of 2026-07-25, the node logs carry 20 of each string; the old matcher
reported 40 promotions and the new one reports 20. The neighbouring
counters are unmoved at 13 sessions and 33 parent switches.
Info-level runs are unaffected, which was checked rather than assumed: on
the tcp-mesh run of the same date the debug string occurs zero times, and
the old and new matchers both report 7 against 7 raw occurrences. Three
scenarios run at debug, one of them gating, so this corrects the promotion
figure in their artifacts. No assertion reads the counter, so no suite's
verdict changes.
The earlier fix that introduced this added the live string alongside the
dead one where it needed to replace it.
The tcp-chain profile has never been runnable. ping-test.sh dispatches on
chain and mesh only, and the string tcp-chain has never appeared in that
script in its history, so invoking the profile has always fallen through
to the unknown-profile branch: silently asserting nothing before that
branch was made to fail, and exiting 2 since. Neither runner referenced
it either.
What it would have covered is already covered. The chaos tcp-mesh
scenario runs in local CI and gives TCP a discriminating gate: n04's only
edges are TCP, so if the transport were broken n04 could not parent and
the scenario's baseline assertion would fail. It also carries a pure-TCP
two-hop path, n01 to n04 to n05, under netem and link flaps. The only
residual tcp-chain would have added is a mesh with no UDP present
anywhere, which is not worth a fixture that has never run.
Remove the topology, its three compose services, and the documentation
rows. Every profile the static compose file still defines is now
exercised by a suite.
Also correct the chaos README, which has listed tcp-only and tcp-chain in
its transport table since those two scenarios were deleted in 9e63b42.
Neither has existed for months.
The mesh-public profile ran in neither runner: ci-local's static suite
list carries only static-mesh and static-chain, and the GitHub matrix has
only mesh and chain. It also added no coverage over static-mesh.
ping-test.sh and iperf-test.sh branched mesh and mesh-public together and
exercised the same 20 directed pairs among node-a through node-e, and no
script referenced the external node at all. The convergence waits even
used mesh's peer counts rather than mesh-public's, so the extra link to
the public node was never counted, let alone asserted.
Running it would therefore have added a dependency on a live internet
host (test-us01.fips.network) in exchange for zero additional assertions.
Remove the topology, its five compose services, the script branches that
aliased it to mesh, and the documentation rows. An invocation using the
old profile name now fails on ping-test.sh's unknown-profile guard
instead of silently behaving as mesh.
The config generator's external-node support (external_ip,
is_external_node) stays. It has no consumer now, but it is woven into the
config path every remaining topology uses, so removing it would put the
gating suites at risk for no present gain.
wait_for_peers reads the connected-peer count through a pipeline ending in
`|| echo 0`, so a container that never answers contributes 0. That is safe
against a floor of 1 or more, where 0 reads as "not converged yet" and the wait
eventually times out, and it is unsafe against a floor of 0, where the first
read from a dead container satisfies the wait and the caller proceeds as though
convergence had been observed.
No caller passes 0 today; every one passes 1 or more, and the single variable
minimum is advisory. Rather than harden the reader, reject the input: a floor
below 1 is now an error. That makes the shape unreachable instead of repairing
instances of it, and it costs a future caller nothing except a clear message
saying that asserting "exactly zero peers" needs a reader that distinguishes no
answer from zero, which this floor is not.
wait_for_links goes with it. It had no caller anywhere in the tree on any
branch, and it carried the identical fallback, so the only thing it could do was
hand the hazard to whoever called it first. An uncalled helper cannot be wrong
today, which is exactly why it was the risk worth removing rather than the one
worth keeping for symmetry.
The hermetic gate suite gains a case covering both directions, with docker
stubbed so it stays container-free: a floor of 0 is refused with a message
naming why, and the same unreachable container with a floor of 1 still polls its
full budget and times out. Verified by removing the guard and confirming exactly
the three zero-floor assertions red while both floor-of-one controls stay green.
wait_for_peers_exact read the connected-peer count through a pipeline ending in
`|| echo 0`, so a container that never answered and a daemon that answered zero
produced the same value. That is only harmless while every caller expects a
non-zero count, which is true here today and is the reason this copy was left
alone when the acl suite's copy was fixed. It leaves the trap armed for whoever
adds the first caller expecting zero: the check would be satisfied on its first
iteration without the property it exists to verify ever being observed.
The read moves into its own function that returns the empty string when the
container does not answer, and the caller treats empty as "no answer" rather
than as a count. A run that never gets an answer now fails saying so, distinctly
from one that answered the wrong count, and the diagnostic peer dump runs only
on the latter, since dumping from a container that cannot answer prints a docker
error rather than evidence.
This is the shape the acl-allowlist suite already uses. The two copies had
diverged on whether a silent container counts as an answer, which is the kind of
disagreement that decides a security assertion in whichever file is read last.
Checked by construction rather than by a green run: driving the old form against
an absent container with an expected count of zero returns success on the first
iteration, and the new form fails; with an expected count of one, the shape both
real callers use, both forms still fail, and a genuine zero from a live daemon
is still reported as zero.
The compose pinned 172.32.0.0/24 with a per-container ipv4_address, so two
concurrent runs asked docker for the same address space and the second failed
with a pool-overlap error. Request no subnet and let docker assign one from the
daemon pool. Peers address each other by the docker hostname the compose already
sets (host-a, host-b) rather than by literal IP; docker's embedded DNS is
per-network, so the same hostname in two runs resolves inside each run's own
subnet. Nothing this suite asserts on moves as a result: its checks run over the
fips0 overlay, whose addresses are derived from node npubs and are independent of
docker addressing, and the packaged nftables ruleset matches on interface rather
than on any address.
The generated config directory moves under the run suffix for the same reason it
did in the acl suite, and the run teardown gains the matching removal so a run
no longer leaves its directory behind.
Case (b) also wrote curl's output to a fixed path under /tmp. Two concurrent
runs shared that one file, and either run's cleanup landing between the other's
write and read left an empty read, failing the http_code check for a reason
having nothing to do with the firewall. It uses mktemp now.
As in the acl suite, the floating subnet removes one of the two obstacles to
concurrent runs. The compose project name is still fixed; the local CI runner
scopes it externally, a bare hand run does not, and the comment at the site says
so rather than claiming the file is self-sufficient.
The compose pinned 172.31.0.0/24 with a per-container ipv4_address, so two
concurrent runs of this suite asked docker for the same address space and the
second failed with a pool-overlap error. Request no subnet instead and let
docker assign one from the daemon pool, which leaves nothing for two runs to
contend for. Peers now address each other by the docker hostname the compose
already sets (host-a through host-f) rather than by literal IP, and docker's
embedded DNS is per-network, so the same hostname in two runs resolves inside
each run's own subnet.
The generated config directory moves under the run suffix in the same change,
because it has to: the generator does rm -rf on it, so a shared directory means
one run deletes the fixtures another is still using, which fails silently rather
than loudly at bring-up. Scoping it requires the generator output path, all five
bind-mount sources per service, and the gitignore entry together; scoping only
the generator leaves the compose reading the unscoped path.
The floating subnet removes one of the two obstacles to running this suite
twice at once, not both: the compose project name is still fixed, and nothing
scopes it for this suite. Recorded at the site so the comment does not claim
more than the change delivers.
Both maint changes carry over unchanged. The hop-limit helpers follow the
module rename from src/protocol/link.rs to src/proto/link.rs, applied by
rename detection and verified at the renamed path rather than assumed; the old
path is not recreated. The rekey config validation and its jitter-constant
import apply as written, the node module exposing the constant on this branch
as well.
The only conflict was the unreleased changelog, where master's own Changed
entries and the incoming hop-limit entry were both correct for their branch.
Resolved as the union, with the incoming entry appended to master's Changed
list.
Config validation said nothing about the rekey block, so two settings that
disable rekey in appearance and hammer it in practice were accepted silently.
after_messages of zero makes the message-count arm true on every poll, since
the trigger tests the counter against it with a greater-or-equal. It reads like
a way to switch the arm off and does the opposite.
after_secs at or below the per-session jitter is the same trap on the timer
arm. Each session offsets the interval by a random value drawn from plus or
minus the jitter bound, so a smaller interval saturates to zero on a negative
draw and rekeys on sight, for roughly half of sessions and not the other half.
The rule is expressed against the jitter constant rather than its current
value, so it tracks if the bound ever moves.
Both are checked whether or not rekey is enabled, so turning it on later cannot
surface a configuration error at a surprising moment. Neither has an upper
bound: a very large value is the established way to disable one arm of the
trigger and stays legal.
The forwarding path was corrected to full IP semantics: local delivery is not
hop-limit gated, and forwarding decrements first and drops at zero. Two helpers
on SessionDatagram were left implementing the old rule. decrement_ttl still
checked before decrementing, and can_forward still answered true at a hop limit
of one, where the decrement leaves zero and the datagram is dropped. Neither is
called anywhere today, which is precisely why they are worth correcting rather
than leaving as an invitation to reintroduce the defect.
The TtlExhausted reject doc described behaviour that no longer exists. The
reject is charged for a transit arrival at one as well as at zero, and is never
charged for a datagram addressed to this node, whose delivery is decided ahead
of the test.
Both helpers are pinned by tests at the boundary, which fail against the
previous implementations.
Carries the address-to-link map's corrected doc comment. The only
conflict was the adjacent line naming the discovery config keys, which
this line renamed to node.lookup.* and the maintenance line did not; kept
this branch's names.
The doc said the map "enables dispatching incoming packets to the right
connection before authentication completes." It does not: find_link_by_addr
has no callers outside its own tests on any branch, and encrypted frames
dispatch by session index. Its live readers are the msg1 admission fast
path and the duplicate-inbound-handshake check.
The comment mattered because the map has a trap that invites exactly the
misuse it was advertising. An outbound dial registers the literal
configured address string, which may be a hostname, while an inbound
packet carries the resolved form; TransportAddr compares byte-wise, so
the two never match and a lookup keyed on an inbound address returns "no
such link" for every hostname-configured peer rather than failing. The
entry is also single-valued per key, so an inbound handshake overwrites
an outbound dial's entry for the same address.
Both current readers survive this because each compares a key written in
the same form it reads. That is a property of those two call sites and
not of the map, so the comment now says so, and says not to key a
peer-identity question on it.
Documentation only; no behaviour changes.
The send-counter arm of the rekey trigger had no test that could fail for
the reason it existed. The integration suites never reached it: they
inject after_messages = 65536 against a ping-driven workload, so every
rekey they observe is time-triggered. The unit test that looked like the
guard reproduced the trigger's OR predicate as a local closure and
asserted against its own copy, so deleting the arm from the real
predicate left it green.
That is the coverage hole that let the data-plane drop across a
message-count-triggered rekey survive: the defect is precisely a rekey of
a young session, which only the counter arm can produce.
Replaces the existing at-threshold check with a boundary test driving
poll_rekey, holding the time arm off so the counter is provably what
decides. The silence-below half is the part that is new, and it earns its
place by measurement: widening the bound to `p.counter >= 1` reds exactly
one test in the library suite, this one. Deletion, narrowing to `>` and
inversion are caught here too, though the time arm's negative tests
already catch a bound removed altogether.
Removes the closure-based test rather than repairing it. Its one
assertion over real code, that a fresh session entry's jitter lies within
the symmetric bound, is already covered over 100 samples elsewhere.
Note this leaves the deployed maintenance line untested on the same arm:
its trigger is inline in an async method that reads the clock directly,
with no pure function to drive, so the same test cannot be written there
without the sans-IO seam this line already has.
The three rekey integration suites (rekey, rekey-accept-off,
rekey-outbound-only) are dropped from both runners. Their coverage now
lives in fast, deterministic in-process tests:
- rekey timing and choreography (trigger, K-bit cutover, drain, jitter,
guards) in the sans-IO poll_rekey tests (proto/fsp, proto/fmp);
- data-plane continuity across a real cutover in
rekey_cutover_preserves_data_plane (a real IK rekey over loopback);
- the accept-off dual-init regression and the udp.outbound_only rekey
loop in the should_admit_msg1 and dual-init characterization tests.
Drop them from both runners in lockstep so the parity guard stays green,
and record the retirement in the deliberately-not-run block with a pointer
to the standalone runner. The rekey-test.sh script stays on disk.
Drives a real IK rekey handshake to K-bit cutover between two loopback
nodes and asserts an encrypted datagram sent after the cutover decodes on
the new session, with no spurious peer teardown. The rekey is forced
deterministically: rekey.after_messages = 1 crosses the initiator's
trigger on the first sent datagram, and both sessions are backdated past
the responder's 30s rekey-acceptance gate, so no wall clock is read.
The cutover is proven to actually occur (the live session index changes),
which is what makes the continuity assertion meaningful rather than
vacuous. Adds a make_test_node_with_config loopback helper for setting the
rekey thresholds. The rekey timing and choreography decisions themselves
are already covered exhaustively by the sans-IO poll_rekey tests.
The inbound max_peers early-gate in handle_msg1 (silent-drop a Msg1 from
a net-new identity at saturation, send no Msg2, admit no peer) is
unit-tested over a real UDP socket by
handle_msg1_silent_drops_at_cap_for_new_peer in src/node/tests/unit.rs:
it saturates a node, sends a Msg1 from a fresh identity, and polls the
sender socket to assert no Msg2 comes back — the same wire-observable
discriminator the Docker suite's tcpdump used — with a sibling test for
the existing-peer bypass. That is a deterministic superset of the Docker
packet-capture assertion.
Drop it from both runners in lockstep (its *_SUITES array, run function,
default-flow loop, --only dispatch arm, and the GitHub matrix leg with
its steps) so the parity guard stays green, and record the retirement in
the deliberately-not-run block with a pointer to the standalone runner.
The admission-cap-test.sh script stays on disk and runs by hand.
The ACL admission decision is exhaustively unit-tested per npub over real
loaded allow/deny files (src/node/acl.rs test module: allow-match-wins,
allowlist-miss falls through, deny-only, deny-all, allow_all override,
deny-after-allowlist-miss), and the inbound and outbound handshake-
admission paths are covered in-process over the loopback transport
(src/node/tests/acl.rs: inbound msg1 denial, outbound denial, reload).
The Docker suite's only unique coverage was admission over a real UDP
transport, which every other real-transport suite already exercises, and
its operator-facing log assertion.
Drop it from both runners in lockstep (its *_SUITES array, run function,
default-flow call, --only dispatch arm, and the GitHub matrix leg with
its steps) so the parity guard stays green, and record the retirement in
the deliberately-not-run block with a pointer to the standalone runner.
The testing/acl-allowlist/ suite stays on disk and runs by hand.
smoke-10 was a no-stressor 10-node tree-convergence sanity check (netem
off, no ping). Its subject, spanning-tree convergence and root election,
is now covered in-process, faster and deterministically, by the loopback
spanning-tree harness (src/node/tests/spanning_tree.rs: ring, star,
chain, 100-node and disconnected-component convergence) plus end-to-end
datagram delivery (src/node/tests/forwarding.rs). Real-UDP convergence
smoke still runs via static-mesh and the other scenarios' baseline
assertions, so no Docker coverage is lost.
Drop it from both runners in lockstep (the CHAOS_SUITES list and the
GitHub chaos matrix) so the parity guard stays green, delete the scenario
YAML, and update the chaos README.
Bring up the retirement of the six cost-based parent-selection chaos
scenarios (cost-reeval, cost-avoidance, cost-stability, depth-vs-cost,
mixed-technology, bottleneck-parent), which tested a decision the Docker
harness could not exercise reliably.
master already carries the equivalent sans-IO coverage in
src/proto/stp/tests (effective-depth cost selection, hysteresis, cost
degradation) and its transport-drop tests, so this keeps master's src
unchanged and takes only the scenario removals and the CI-list, README and
scenario-comment updates from maint.
The cost-selection chaos scenarios (cost-reeval, cost-avoidance,
cost-stability, depth-vs-cost, mixed-technology, bottleneck-parent) tested
TreeState::evaluate_parent's decision logic through a Docker mesh that could
not exercise it reliably: the tree roots at whichever node holds the smallest
NodeAddr, MMP link costs take several measurement windows to settle, and the
parent hold-down plus hysteresis timing all confound the outcome. A
deterministic link-cost flap still produced zero periodic parent switches in a
full run.
Replace those six scenarios with deterministic unit tests in src/tree/tests.rs
that drive evaluate_parent directly: cheaper-link selection at equal depth,
switch-on-cost-change, hysteresis suppressing a marginal change while allowing
a significant one, and the depth-versus-cost effective-depth tradeoff. Each is
constructed so that breaking the cost or hysteresis logic makes it fail.
The congestion kernel-drop signal (SO_RXQ_OVFL) cannot be provoked
deterministically in Docker: a fresh daemon reader keeps up with
container-speed traffic, so the socket receive queue never overflows (an
unshaped run with a 4 KB buffer and heavy traffic recorded zero drops on every
node). Extract the drop-detection edge -- read the cumulative counter, fire an
event only on the transition into a new drop burst -- into
TransportDropState::observe_drops and unit-test it directly. congestion-stress
keeps its ECN and MMP congestion-signal assertions, which do need the real
shaped bottleneck queue.
Remove the retired scenarios from both CI runners and update the chaos README.
gateway-lan pinned 172.20.1.0/24 and fd02::/64, so two concurrent local CI
runs collided on "Pool overlaps" at network creation. The IPv4 subnet has no
consumer -- the whole gateway LAN path is IPv6 -- so drop it and let docker
auto-assign. The IPv6 side cannot float, because the LAN clients' resolv.conf
pins the gateway's address as their nameserver and that must be a literal
known before they start, so claim a free /64 per run and thread the prefix
through the compose address pins, a generated resolv.conf, and the test via a
single exported variable.
The claim and the external network ship as a harness-only compose overlay
applied by run_gateway; the base compose keeps a normal gateway-lan network,
so the GitHub matrix and any standalone bring-up are unaffected. With no claim
every address renders exactly as before.
The boot check piped `systemctl is-system-running --wait` into `grep -qE
'running|degraded'`, but the script runs `set -o pipefail` and is-system-running
exits non-zero for `degraded`. So the pipeline failed on a degraded system even
though grep matched, and the wait looped to its timeout. The older distros reach
`running` (exit 0) and passed; debian:trixie and ubuntu:26.04 reach `degraded`
because a unit that cannot run in a container (systemd-modules-load) fails, so
they timed out at every ceiling -- which is why the earlier timeout increase did
nothing.
Capture the state string and test it directly instead of trusting the pipeline
exit, so a degraded-but-booted system is accepted as intended. Validated: all
five distros pass, including the two that previously failed.
cost-avoidance and mixed-technology decide a parent by measured link cost
(etx * (1 + srtt_ms/100)). Under the local CI's 4-way-parallel chaos the fast
fiber link's measured srtt spikes from host scheduling and can exceed the
Bluetooth link's, flipping the choice and reddening a run that proves nothing
about the daemon. The old margin was ~0.4 cost units, which a ~24ms one-sided
scheduling blip closes.
Widen the Bluetooth delay to 150-250ms so the margin dwarfs any plausible
one-sided spike. The link still establishes well within the handshake budget
(a ~500ms RTT against a 30s stale-handshake window), so the losing candidate
is still a real, established peer rather than an absent one.
Add a netem mutation exclude_edges option and use it in mixed-technology to
pin the two links n08 chooses between, so a random degradation can't flip the
asserted comparison either. An unknown excluded edge is a hard error, since a
silent no-op would reintroduce the flakiness it exists to remove.
Validated by running both scenarios repeatedly under heavier-than-CI
concurrent load: the parent choice holds every time.
Two concurrent ci-local runs could not both run chaos. Each child's subnet came
from its position in the suite list, so both runs walked 10.30.0 through
10.30.12 and requested identical ranges; whichever reached the daemon first won
and the other died with a pool overlap. A run index or hashed offset would only
make that unlikely, and it is precisely the failure being removed.
The simulator now claims its range: attempt-create on a candidate, advance on
docker's own overlap error, fail loudly on anything else. Docker's address pool
becomes the arbiter, so an overlap is impossible rather than improbable. The
claim lives in the sim rather than in ci-local because only the process that
creates the network can advance on conflict, and because it fixes the bare
chaos.sh path too, which a ci-local-only fix would have left broken.
The generated compose now declares the network external over the claimed one,
and teardown releases the range from both paths, including the setup-failed
path where a run that fell over after claiming still holds one.
Ordering matters here and is not obvious: node IPs derive from the subnet
during topology generation, and traffic shaping keys its filters on those
addresses, so the claim happens before the topology exists. Claiming later
would give a network on one range and filters on another, which does not fail
at bring-up and instead leaves the shaping matching nothing.
--subnet survives as an explicit pin for when a known range is wanted, and now
fails loudly if that range is taken rather than silently overlapping. ci-local
no longer passes it.
Validated by running two instances of the same scenario deliberately
concurrently: they claimed 10.30.1.0/24 and 10.30.0.0/24, both exited 0 with
their assertions passing, and both released their networks. The negative
control holds too: with a squatter on a pinned range the run aborts with a
message naming the range rather than proceeding.
With the root pinned, n08's test subject is assertable for the first time: its
two candidates sit at equal depth with fiber-grade default netem on one side and
Bluetooth on the other, which is exactly the preference this scenario exists to
test, and it takes the fiber parent in four of four runs. Encoded as written
rather than calibrated from the runs, since "should pick n03" is the spec and
the runs merely confirm it is met.
n06 is deliberately absent, and its documented criterion is corrected rather
than left standing. The header called n03 a fiber parent for n06; the netem
policies in the same file say that link is WiFi and the other candidate is
Bluetooth, so n06 chooses between two impaired links. Its only fiber-grade link
reaches a node at depth 3 that is never a good parent. There was never a reason
to expect it to prefer n03, and the two-of-four split observed is correct
behaviour: in a run where it took the Bluetooth peer, the fiber-labelled
candidate measured a link cost of 3 against the other's 1, so the daemon chose
the cheaper effective depth as it should.
Validated by replaying the assertion against all four archived runs, then
breaking it three ways to confirm it can fail, then one live run for the wiring.
One of the breaks expects n06 to take n03 and reports that it chose n02, which
is the direct demonstration of why n06 is not encoded.
cost-reeval records that its subject now fires in two runs of three, against
zero of five when it was structurally impossible, and why that rate is still not
one an assertion can rest on.
congestion-stress said the transport drop-detection path "is exercised by
nothing". Widening the scan beyond this one scenario finds it firing in six
others, so the path is live and what is dead is this scenario's ability to
provoke it. Records the arithmetic that explains why: at the 1 Mbps cap the
receive queue needs tens of milliseconds of reader stall to overflow, and the
ingress policer discards the same traffic a layer below the socket.
mixed-technology said the implementation does not do what its criteria say.
It does. The criteria were conditioned on a tree rooted at n01 while the mesh
rooted at n09, under which n08's choice was settled by depth before link
technology could matter. The archived parent distributions are relabelled as
describing the old root so nobody calibrates against them.
cost-reeval now records why it had no assertion about its own subject: its
designated subject was the root, so the parent switch it exists to observe
could not occur. The historical logs show the switch on n04 in 13 of 14 runs
when the root still wandered, and only on n01 in the five runs before the fix.
Each header now names what remains to be done and where it is tracked, rather
than leaving a disproved conclusion in place for the next reader to inherit.
For a value-returning helper the trailing echo is the return mechanism, not a
log call. The hazard is the same either way -- it fixes the exit status at zero
-- but calling it a log call misdescribes four of the sites the gate now finds.
Found while fixing count_log_pattern: the gate reported that function clean
even though it ends in echo and both callers consume its status. Its call-site
patterns only matched direct forms -- if fn, while fn, fn ||, fn && -- so a
status consumed through command substitution was invisible, because the line
begins with the variable rather than the function name.
That is not an exotic form. It is how a shell function returns a value, and it
is precisely the shape of the swallowed-failure family this gate exists to
catch, so the gate was blind to a large part of its own stated class.
Three patterns added for the assignment, if-guarded and test-expression forms.
The extension finds four real instances, all value-returning helpers whose
trailing echo made their exit status unconditionally zero, and each now carries
an explicit return 0. Also corrects the finding message, which described the
trailing command as a log call; for these it is the return mechanism, and the
hazard is that it fixes the status either way.
Validated by breaking what it guards: a probe reintroducing the defect shape
behind a command substitution is reported and exits 1, where before the
extension it would have passed.
Every chaos scenario draws n01 at the top of its topology, and until now that
held in three of thirteen. The mesh roots itself at the numerically smallest
NodeAddr, which is a hash of the node's public key and bears no relation to the
node numbering, so which node ended up as root was effectively arbitrary.
The consequences were not cosmetic. cost-reeval rooted at n04, its own
designated test subject, so that node had no parent and the periodic parent
switch the scenario exists to observe could not occur at all. mixed-technology
rooted at n09, which put its two documented parent criteria out of reach and
made correct cost-based selection look like a defect.
Identities are still derived from the mesh name exactly as before and are still
deterministic. What changes is which node id holds which one: they are now
assigned in NodeAddr order, so n01 holds the smallest and is the root. Verified
against a model of the daemon's own derivation that reproduces the previously
observed root for every scenario and n01's address byte for byte; all twelve
pinned scenarios now root at n01.
smoke-10 deliberately opts out via pin_root: false so that root election from an
arbitrary key distribution stays exercised somewhere. Its assertion is a
convergence floor and is root-agnostic, which is why it is the cheapest home for
that. The new key is rejected when non-boolean, and a near-miss spelling is
rejected as unknown; both checked.
Not yet established: the trees themselves change, so the parent-dependent
assertions in bottleneck-parent, cost-avoidance and cost-stability need
re-deriving against live runs. Those are held until the in-flight CI finishes,
because a chaos run rebuilds the shared fips-test image that run is using.
Four sites where a check could report a verdict it had not established, or
detect a failure and then not turn it red.
assert_no_panic in both NAT suites read `docker logs ... || true`, so a
container that could not be read produced empty output, matched no panic
pattern, and returned success. The assertion's failure mode was
indistinguishable from its success condition. It now reports that absence of
panics is not established.
deb-install's apt capture discarded the exit status, so a failed docker exec
gave an empty capture that matched neither error pattern and reached the pass
branch. The status is now kept and checked before the output is inspected, with
the output still printed on failure.
wait_for_systemd printed a warning and returned success on timeout, so every
check after it read a system that may not have started its units. It now returns
non-zero and the caller abandons that distro leg rather than testing an
unstarted system.
interop's copy of count_log_pattern carried the same defect fixed in rekey: a
node whose logs could not be read contributed zero to eight expect-zero
assertions. Fixed the same way, and its consumer now reports the unreadable case
rather than comparing a sentinel against zero.
Each validated by breaking what it guards and by confirming the healthy path is
unchanged: an absent container now fails each check where it previously passed,
a readable panic-free container still passes, a real panic is still caught, and
a genuinely successful install still passes.
count_log_pattern summed a per-node `docker logs | grep -c ... || true`, so a
node the harness could not read contributed zero. The six assert_zero_count
callers are negative health assertions -- no panics, no ERROR lines, no AEAD
decrypt failures, no rekey msg2 failures -- and a contributed zero reads as
clean, so one unreadable node silently weakened the assertion and all of them
voided it. This is the family the harness-fallback issue was raised to high
priority for, and it is the one remaining audit residual that can make a green
rekey run lie about protocol code.
The read now fails the count rather than degrading it, printing a sentinel that
names the container. Both callers split the declaration from the assignment,
because `local c=$(fn)` takes local's exit status and discards the function's --
which is how the original defect stayed invisible.
Validated by breaking what it guards rather than by observing green: against
absent containers the old reader returns 0 and assert_zero_count passes
vacuously, while the new one returns rc=1 and reports a failure. Checked the
other direction too, since a fix that reds a legitimately clean run is no use: a
genuine zero across readable nodes still returns 0, and the sum is unchanged at
2+1+0 for a shimmed three-node read.
A bash function returns its last command's status, so one ending in a log
call returns 0 whatever it did, and a caller written as `if func` or
`func || fail` has a gate that cannot fire. This tree found two in one
day: run_chaos ended in record, making every chaos row unconditionally
green since 2026-03-09, and build_fips_for_e2e ended in log, letting five
end-to-end legs test the previous commit's binary and report green. Both
were repaired one at a time. This is the rule that catches the next one.
What it enforces is a contract, not a bug hunt: a function whose status a
caller tests must end in an explicit return rather than leaving its
success value to whatever the last log call produced. That distinction is
deliberate and worth stating, because all three instances found in the
tree were benign — each failure path already returned early, so the
trailing echo reported a real success. Reading the function is the only
way to know that, and the next edit that puts an unguarded command before
the final log converts the benign shape into the defect with nothing to
notice. An explicit return costs a line and makes the class unreachable.
The three are given one here.
Scoped on the call sites rather than the definitions, the same way
check-log-strings.py scopes on what a grep reads rather than what its file
mentions: 315 functions scanned, 42 end in a log call, and only the ones
whose status something consumes are reported. A function that ends by
reporting and is called for its output is idiomatic and is left alone.
Without that scoping the finding list would be 42 long and would stop
being read.
Validated by construction rather than by a green run: injecting a copy of
build_fips_for_e2e's exact shape — unguarded docker cp, then a log as the
last statement, with a `|| { ...; return 1; }` caller — makes the checker
report it and exit 1.
Wired into both runners beside the parity and log-string checks.
mmp_focused_pane_indicator checked that the unfocused Link MMP title is
not cyan with assert_ne! over fg_at. fg_at is find(..)? mapped to the
cell's foreground, so it returns None for a title that was never drawn,
and None != Some(Cyan). The assertion therefore passed just as happily
when the pane was missing entirely as when it was present and unstyled.
Assert presence first, so the colour check means "not highlighted"
rather than "not there".
Demonstrated rather than argued. Renaming the pane title in mmp.rs so
"Link MMP" is never rendered leaves the original assertion passing, and
makes the new one fail with the message it was given. Both files restored
after.
This is the only instance of the shape: it is the sole assert_ne! over
fg_at or find in the fipstop tree, and the only assert_ne! in snapshots.rs
at all.
Quartet clean: fmt, build, clippy --all-targets -D warnings, test --lib
at 1376 passed / 0 failed / 7 ignored.
The UDP recv microbenchmark carried a bare #[ignore] while its sibling in
the link module carries a self-describing one. Match it, so the reason
appears wherever the test is listed rather than only in the doc comment.
Record why [profile.ci] retries = 2 is not applied locally. The two gates
disagree on purpose: the hosted runner retries a flaky test twice, the
local sweep fails on the first failure. It exists for shared-runner packet
loss, a property of that environment rather than of the code, and applying
it locally would suppress a real local flake — a failure that only
reproduces under load is a robustness bug to fix, not to retry past.
Keeping the local sweep strict is what makes it the sharper gate. An
undocumented asymmetry is indistinguishable from an oversight, which is
why this is written down rather than left to be rediscovered.
The cost is stated rather than hidden: a test that fails once and passes
on retry is reported green with no separate signal, so a genuine
intermittent failure can be absorbed. If that starts mattering the fix is
to surface retried-but-passed tests, not to drop the retries.
Quartet clean: fmt, build --workspace, clippy --all-targets -D warnings,
and test --lib at 1376 passed / 0 failed / 7 ignored.
Two suites could report a clean pass for work that did not run.
stun-faults skips Phase 2 when tc netem is unavailable, announcing it
several hundred lines above the result and then printing a bare
"stun-faults-test passed". The verdict line now carries the count and
names each skipped phase, and a run in which every phase was skipped
fails outright, since it tested nothing.
deb-install defines a skip() helper and a SKIP counter, prints "N
skipped" in its summary, and calls skip() from nowhere, so the number can
only ever be 0. Its exit tested FAIL alone, meaning a skip could not have
failed the run even once something did set the counter. Gate on SKIP too.
That changes no current outcome, which is the reason to do it now rather
than later: the machinery and the reported number already existed, so the
first skip path added would have printed "N skipped" beside a zero exit
and read as coverage.
Verified by driving the verdict logic at zero, one and three skips: clean
pass, pass with the skip named, and a non-zero exit when nothing ran.
ecn-ab-test.sh was a -test.sh with no path to a failing result: it
asserts nothing, applies no threshold, and nothing invokes it. It also
could not have worked. It read a fixed sim-results/ecn-ab-on/ path while
the runner has written timestamped directories since 2026-03-20, and not
one ecn-ab result directory exists on disk, so it has found neither input
for months and the "+10.2% recv throughput" figure the README carried is
not reproducible from anything available.
Rename it to ecn-ab-compare.sh, fix the path resolution to glob the
timestamped directories, and stop swallowing a failed simulation with
|| true so a broken run cannot feed the comparison.
Deliberately not adding the threshold assertion that would make it
gateable. That needs a calibration corpus, and none exists precisely
because the tool has never produced a kept result; a number invented now
would assert a guess. The prerequisite is a calibration run set and a
decision about what ECN is expected to deliver, which is a protocol
question. Written in the script header rather than left implied.
Add the exclusion block to ci-local.sh naming every suite and scenario
neither runner runs, with the reason for each: interop, boringtun,
iperf-test, this comparison tool, mesh-lab, and the four chaos scenarios
outside CHAOS_SUITES. Until now "not in the suite list" was
indistinguishable from "forgotten".
wait_until_connected decides whether every static suite proceeds or
gives up, so a regression in it turns those suites' verdicts into noise.
Its unit tests existed and were invoked by nothing.
Wire them into ci-local.sh beside the parity and log-string checks,
above the mode branches so --only and --test-only are gated too, and add
the matching step to the ci-parity job in ci.yml. Putting them in one
runner only would create exactly the drift check-ci-parity.sh exists to
catch, and neither placement is visible to that checker since this is not
a matrix suite.
Verified by breaking what it guards rather than by observing green:
setting wait_until_connected's default near-converged slack to 0 fails
case4, records FAIL wait-converge, and exits the sweep 1. Restored after.
They pass, which nothing had established before — 13 assertions, all
green. They also take about 45 seconds rather than the "a few seconds"
the header claimed, and nothing had contradicted that claim because no
runner had ever invoked it. Header corrected with the measurement.
The orphan sweep this called for is done and there are now no
zero-reference *-test.sh files under testing/. It also settles a
disagreement recorded between the audit and the task file: ecn-ab-test.sh
is invoked by nothing, its one reference being a README description
rather than a driver, so the audit was right and the earlier sweep wrong
to call it a documented manual tool. interop-test.sh does have a driver
in interop-stress.sh and iperf-test.sh one in iperf-compare-refs.sh, so
those two were correctly classified.
The baseline floors added for this scenario were calibrated from archived
runs, and those runs are all the 10-node 120-second variant ci-local
invokes as "churn-mixed --nodes 10 --duration 120". The file itself
defaults to 20 nodes and 600 seconds, and --nodes sed-patches num_nodes
in a copy before the scenario loads, so the gating run and a bare
chaos.sh run are different scenarios. Nothing at the site said so.
The floors hold for both, but only because the gating run is the smaller
one. Retuning them against a bare 20-node run, which clears them easily
at 20 answering, 1 root, 19 parented and 69 sessions, would put them past
what the 10-node run reaches and turn CI red while a manual run stayed
green. Say that where someone retuning them will read it.
Confirmed by running the gating invocation directly: 10 answered, 2
roots, 8 parented, 18 sessions, all inside the ranges the floors were
built from. churn-mixed is the only scenario CI overrides this way; the
other twelve run their files as written.
Every section of ping-test.sh is guarded by the profile name, so an
unrecognised profile fell through all of them and the script exited 0
having run no assertion. A typo in the caller's argument produced a green
run that tested nothing. Give it an else branch that names the valid
profiles and exits 2.
No existing leg changes behaviour: the hosted runner gates the ping step
on matrix.type == 'static', which only static-mesh and static-chain
carry, and ci-local passes only the two names in STATIC_SUITES. Worth
noting while confirming that, though: ping-test.sh accepts a mesh-public
profile that neither runner ever passes.
Also record why the convergence waits are `|| true` — they are settling
delays rather than assertions, and the directed-pair pings are what
decides the run — and add admission-cap to the suite list in the
ci-local header, which ran it without listing it.
Five scenarios named by the audit carried no assertions at all, so a run
in which the mesh never formed exited 0 and reported green. Add a
baseline assertion covering how many nodes answered, how many distinct
roots they agreed on, how many took a parent, and optionally how many
sessions were established, and apply it to those five plus the two cost
scenarios left unasserted by the previous commit.
For ethernet-only, ethernet-mesh, tcp-mesh, smoke-10, mixed-technology
and depth-vs-cost the values are not calibrated: one root and N-1
parented nodes is what a spanning tree is, and all provably-completed
archived runs of each show exactly that. churn-mixed is different and
says so at the site. A scenario that stops and starts nodes on purpose
does not hold a single tree, and its six completed runs end with two or
three roots and seven or eight of ten nodes parented, so its floors sit
one step outside the observed range. That leaves them catching a mesh
that collapsed rather than one that churned, which is the most a sample
of six supports.
This gives Ethernet transport its only assertion anywhere in CI. The
three tests in src/node/tests/ethernet.rs are ignored for requiring
CAP_NET_RAW and neither runner passes --ignored, so until now nothing
exercised that transport with a verdict attached.
The floor is deliberately weak and deliberately not a substitute: it says
the mesh formed, not that it formed the tree the scenario describes.
Where those differ the scenario now says so in its own comments.
Four scenarios exist to test cost-based parent selection and each named
its expected outcome in a comment that nothing read. Add a tree_parents
assertion mapping a node to the parent it must have in the final tree
snapshot, and encode it for the two scenarios whose stated outcome the
implementation meets: cost-avoidance (n04 takes the fiber n03) and
bottleneck-parent (n06 takes the fiber n03, n09 keeps its only parent
n05). Both hold in all six provably-completed archived runs.
The other two are left unasserted on purpose, and each for a different
reason worth keeping distinct.
mixed-technology says n06 and n08 should both pick the fiber parent n03.
They do not. Across the five completed archived runs n06 picks n10 three
times and n08 picks n04 four times. Encoding the criterion as written
would red the scenario most runs, and encoding the observed behaviour
would bless something nobody specified, so the disagreement is recorded
at the site as an open question. n06 preferring n10 may well be correct
and the comment stale, since n10 reaches it by fiber too.
depth-vs-cost is different: its validation line does not name an outcome
at all, saying only that the choice "reflects the actual cost tradeoff",
which either answer satisfies. The corpus shows both occurring under one
seed, three runs to two. That scenario needs a protocol decision about
which parent is correct before it can have an assertion.
Compare parents by address resolved from the snapshot's own
my_node_addr, and fail rather than skip when a node is absent from the
snapshot or still claims to be its own root. Those two states are the
common ones in the older corpus and both produce the same "no match" a
wrong parent does, so only separating them keeps a harness problem from
reading as a routing verdict.
The scenario listed four success criteria "verified via post-run
congestion snapshot". Nothing read the snapshot, so the scenario could
not fail on any of them.
Add a congestion_signals assertion taking a floor on the number of nodes
reporting each counter, and encode three of the four. The floors are one
node each because that is what the criteria say; tightening them to the
counts recently observed would assert something nobody wrote down.
The fourth criterion is not encoded, and that is the finding rather than
an omission. No node has reported a non-zero kernel_drop_events in any of
the 182 archived runs, so asserting it would red the scenario
permanently. It is recorded at the site as an unmet criterion and a
coverage gap: the transport drop-detection path the scenario names as its
second signal is exercised by nothing.
Two things about the corpus are worth carrying, both recorded in the
scenario. Only six archived runs carry a status.txt and are therefore
provably completed; all six meet the three encoded criteria with five to
nine nodes reporting each signal. The 176 older runs meet none of them,
and they are not invalid samples: each reached teardown far enough to
write an analysis.txt, and ECN landed before all but one of them. What
changed on 2026-07-22 is not established, since neither the scenario nor
netem.py, traffic.py or control.py has been touched.
Count the nodes reporting a signal rather than the magnitude of any one
counter, since a single node with a large count would satisfy a magnitude
test while proving the signal never propagated. A missing snapshot fails
rather than reading as an absence of congestion.
The simulation exit ladder reported panics and failed assertions but said
nothing about ERROR-level log lines, so a run in which every node errored
on every line still exited 0.
Add a max_errors assertion and apply it to every scenario by default
rather than having each one opt in. This is a floor on what a green run
means rather than a property of an individual scenario, and a scenario
that has to ask for the floor is one that can forget to.
The default ceiling of 0 is what the archived corpus supports: across
2416 result directories no node log contains an ERROR-level line, while
the WARN counter extracted by the same code path ranges from 0 to 1135,
so the counter is known to discriminate rather than merely known to read
zero. A scenario that legitimately induces errors raises the ceiling in
its own YAML and says there why.
Reject rust_log: off alongside it, since that is the one level that would
leave the ceiling counting zero whatever the mesh did.
The scenario's success criterion existed only as a comment: count
"Parent switched" in n04's log, expect at most 5. Nothing read it, and
the scenario declared no assertions at all, so its exit code could only
report that the mesh came up and nothing crashed.
Add a max_parent_switches assertion and wire that criterion to it.
The assertion takes an optional node scope, which is the part that
matters. The existing sibling counts mesh-wide, and a criterion written
about one node's log is a different quantity from the sum over every
node: across archived runs of this scenario n04 is 1 or 2 while the
mesh-wide total ranges 3 to 6. Asserting the sum against a per-node
threshold would check something other than what was specified, and at
this threshold would also have failed several runs that were fine.
Counting nothing must not read as stability. A node id absent from the
topology fails explicitly rather than matching zero log lines and
sailing under the ceiling, and a scenario that declares a parent-switch
assertion while setting a log level that suppresses the events it counts
is rejected at load rather than passing vacuously after a full run. The
parse rejects a mistyped key, a missing, negative, boolean or
non-integer ceiling, and a node written empty or as a non-string -- that
last one because YAML renders a bare `node:` as null, which would
silently revert to the mesh-wide count.
The scenario comment now records what the threshold is worth against
the 11 completed archived runs rather than a single sample: 5 is well
above anything observed, the daemon's own parent hold-down caps
switches near 6 per run, and nearly every switch lands during initial
tree formation before any mutation fires. So this catches only
near-pathological reparenting. Tightening to 3 would make it a real
detector, but that changes the stated criterion rather than fixing it,
so it is left as a decision.
Verified against a live run: the assertion evaluates n04 at 2 against
mesh-wide 4 and passes, and with the ceiling temporarily set to 0 the
scenario exits 3, the assertion-failure code, which it previously could
not reach.
The mapping-reclaimed check expects zero mappings, and read the count
with `r.get('data',{}).get('mappings',[])`. An error response carries
no data field, so it parsed to zero and satisfied the check without the
gateway having answered at all. The expected value and the failure
value were the same number, which is the shape that lets an assertion
pass without asking anything.
Require the key to exist and be a list, and exit non-zero otherwise, so
the existing fallback turns an unanswered or malformed response into a
failure. The gateway always emits the key on success, including for an
empty set, so a genuine zero still reads as zero; confirmed against the
running gateway, where the reclamation check passes.
The earlier show_mappings poll shares the parse and is left alone: it
waits for a positive two, which no error response can produce. The
hazard was already documented there, and this is the call site that was
exposed to it.
Case (a) sends an unallowed inbound request that the ruleset should
drop, and accepted any non-zero curl status as proof. A drop produces
no RST, so curl can only hit its deadline and exit 28; connection
refused, an unroutable address or a missing listener all fail too, with
different codes, and the check counted those as a blocked connection.
Require 28 exactly, so the assertion distinguishes silently dropped
from failed for some other reason. Confirmed against the harness: the
real run returns 28.
The baseline check matched `counter packets`, which any counter rule
satisfies whatever its verdict, including one that accepts. Require the
rendered drop rule instead.
The drop-counter read had the same weakness plus a worse one: it took
field three of the first line mentioning a counter, which is the packet
count only when the line begins with `counter`. On a rule such as
`tcp dport 9 counter packets 42 bytes 3000 drop` it printed the port
number. That shape is reachable, because the shipped ruleset includes
the operator drop-in directory ahead of the trailing default deny and a
drop-in may add its own counted drop. Extract by position within the
matched text, and take the last match rather than the first, since case
(a) falls through to the default deny that the ruleset emits as the
final rule.
Note in place at the baseline check that it still only proves a counted
drop rule exists somewhere in the table, not that it is the trailing
one; asserting rule position is a larger change than this warrants.
The sample output in the README is updated to the new wording.
The four rejection checks were independent greps over the whole log:
two npubs, a context and a decision. Together they proved only that
node-a mentioned each npub somewhere, rejected somebody on an inbound
handshake, and rejected somebody by denylist. Nothing tied a rejection
to a named peer, and node-a lists both denied peers as auto_connect
peers, so it logs their npubs on the outbound connect path whether or
not any rejection ever happened. The actual message was never matched.
Replace them with a helper that requires every given string on ONE
line, and assert per denied peer: the real message, that peer's npub,
and the denylist decision together. Strings match in any order by
chaining fixed-string greps over the surviving lines, so the assertion
does not depend on how the log formatter orders a message and its
fields. A grep over empty input yields the empty string rather than
anything a caller could mistake for a match, so an unreadable container
times out and fails instead of passing.
Two limits are written at the call sites rather than left implied. The
decision conjunct discriminates nothing, since the two allowing
variants return early and denylist is the only value that can reach
that warning. And node-a authorizes before dialing, so it emits a
fully formed rejection line for each denied peer on the outbound path;
these assertions are satisfiable without the inbound check running at
all, and it is the peer-count assertions that would catch that.
Verified against the running harness: all three find a real matching
line. The defect shape they replace was exercised offline first, with
the three strings spread across three lines, where the old checks
passed and the new ones fail.
The traversal clock cached a Unix millisecond value and an Instant at
first use, then served every later call by advancing the cached value
with the monotonic elapsed time. A monotonic clock does not tick while
the host is suspended, so once a machine had slept the returned value
trailed real time by the sleep duration for the rest of the process
lifetime, and never re-synced.
Almost everything that clock feeds is an absolute timestamp. The NIP-40
expiration tags on adverts and traversal signals, and the issuedAt and
expiresAt fields of offers and answers, are all computed as now plus a
TTL; the freshness and cache-pruning paths compare it against a
peer-authored, signed created_at. Once the host had slept longer than
signal_ttl_secs, every offer was published already expired, relays
dropped it, and the initiator timed out waiting for an answer with
traversal broken until a restart.
Read the wall clock on every call instead. This also removes a
mismatch inside the traversal failure-state map, which was written
here from the cached clock but written and read from the node
lifecycle with the real one.
Not platform-specific: monotonic clocks exclude suspended time on
Linux and Windows as well, so this affected any host that suspends.
A laptop is simply where a process lives long enough across a sleep
to notice.
The interval-shaped consumers hold up under a clock step. A forward
step, which is what a resume produces, saturates the punch start delay
to zero, and the attempt's own bounds are monotonic deadlines that are
unaffected. A backward step lengthens that delay and can cost one punch
attempt, which retries. Early eviction from the replay window cannot
admit a replay under the shipped defaults, because the freshness window
a replayed offer must also satisfy is strictly narrower than the replay
window itself.
The added test pins the contract and fires on a host that has genuinely
suspended, but it is not a regression guard for this defect: nothing
reachable from a unit test can simulate a suspend, so on a machine that
has not slept the old implementation passes it too. Its comment says
so rather than leaving a false sense of coverage.
Reported in https://github.com/jmcorgan/fips/issues/128
Access layer for phones and laptops to reach FIPS routers on OpenWrt,
stacked on the 802.11s mesh backhaul from #123. Squashed from five
commits by Arjen (Origami74); their original messages follow.
* feat(openwrt): open !FIPS access SSID — fips-ap-setup helper, default transport binding, how-to
Client access layer for phones and laptops: every FIPS router
broadcasts the same open SSID ('!FIPS' — the leading '!' sorts it to
the top of alphabetically ordered network pickers), forming one
standard ESS. Clients save it once and roam between all FIPS routers
natively, with FIPS's Noise IK handshake as the only security layer:
- fips-ap-setup: opt-in UCI helper that creates the 'fips-ap0' open AP
(encryption none — security type must be uniform across routers or
clients treat the ESS as different saved networks), an isolated
network with a static ULA /64, RA-only odhcpd addressing (stateless
SLAAC, no DHCP — the minimum that satisfies Android's provisioning
check; no internet by design, so phones keep cellular as default
route), and a locked-down fips_ap firewall zone (no path to br-lan
or the WAN; only ICMPv6, mDNS, and the FIPS transports reachable).
'remove' subcommand undoes it. Radio setup stays opt-in; a package
must not commandeer radios on install.
- fips.yaml: ship 'ap0'/'ap1' Ethernet-transport entries commented out
(matching the 802.11s mesh backhaul) so a stock install that never
creates fips-ap* logs no per-boot "interface missing" bind warning;
fips-ap-setup uncomments the matching block when it creates the
interface and re-comments it on remove.
- Regression test: extend shipped_openwrt_config_parses to assert the
ap0/ap1 entries ship commented out and still parse once uncommented,
alongside mesh0/mesh1.
- Packaging: install the helper in ipk/apk/buildroot (three synced
copies), extend CI structural checks and shellcheck targets.
- docs/how-to/set-up-open-access-ssid.md: full guide, including the
one-time 'no internet, stay connected' acceptance (stored per SSID,
covers every FIPS router) and the security-type-uniformity
constraint.
* docs(openwrt): correct open-SSID/mesh security framing — peering is open by design
The fips-ap-setup/fips-mesh-setup comments and both how-tos claimed a
stranger "cannot pass the FIPS handshake" / "their frames die at the
handshake" / the handshake surface "drops them". That is wrong: FIPS
peer admission is open. An inbound handshake from any net-new identity
is promoted (node::handlers::handshake::promote_connection), gated only
by the daemon's max-peers cap — there is no allowlist, no PSK, and the
AuthChallenge path is not wired to admission. The Noise IK handshake
provides authentication (no impersonation of another identity, no MITM),
not authorization.
Restate the model accurately in all five places: a stranger on the open
SSID (or the open mesh) can associate AND form a FIPS peer link — that
is the point of open access. Containment is the isolated fips_ap zone
(no path to br-lan or the WAN) plus the max-peers cap, not the
handshake. Clarify that AP client isolation is an L2 control only: a
peered stranger is an overlay peer like any other, so the FIPS overlay,
not L2, is the trust boundary between clients.
* feat(openwrt): serve DHCPv4 on the access SSID from a fixed roamable subnet
RA-only addressing satisfied Android's provisioning check but left
anything expecting IPv4 with a self-assigned address and a "no IP"
complaint. dnsmasq now leases out of 10.21.<N>.0/24 (N = radio index;
prefix echoes FIPS port 2121), deliberately identical on every router:
a roaming client keeps its lease across the ESS, and dnsmasq's
authoritative mode — the OpenWrt default, pinned by the helper — ACKs
the renew a foreign router never issued. Lease collisions across
routers surface as a NAK on renew and the client re-DHCPs.
The dhcp section's 'dhcpv4 server' is read by both dnsmasq (default
images) and odhcpd (only with maindhcp), so either arrangement serves.
A DHCPv4/udp-67 accept rule joins the fips_ap zone; DHCPv6 stays off,
the ULA RA stays as-is, and nothing depends on an upstream. The zone
remains isolated — no forwardings, no internet.
* feat(openwrt): enable mDNS rendezvous from fips-ap-setup
Phone FIPS apps cannot open raw-Ethernet sockets, so DNS-SD is how they
find the router's daemon — but node.rendezvous.lan defaults to off. Ship
the lan block commented in fips.yaml (consistent with the apN transport
entries) and have fips-ap-setup uncomment it when creating the access
SSID. The awk match is scoped to node.rendezvous because
transports.ethernet carries a 'lan' entry at the same indent. The switch
is daemon-wide, so 'remove' deliberately leaves it on rather than guess
whether other transports rely on it.
* fix(openwrt): bind UDP dual-stack [::]:2121 so access-SSID clients reach it
The shipped router config bound the UDP transport "0.0.0.0:2121" (IPv4
wildcard) while the mDNS LAN advert announces every interface address,
including the router's IPv6 link-local — which phones on the !FIPS
access SSID rightly prefer (their cellular default route swallows v4,
and fd00::/8 is captured by the Myco mesh TUN). Result: the client's
Noise msg1 arrives on an unbound v6 port and is silently lost; the
handshake resends and times out.
Symptom chain (observed on-device): mDNS resolve OK, platform push OK,
"Sent Noise handshake message 1" to [fe80::…%N]:2121, four resends, no
reply, 30 s stale-timeout.
OpenWrt is Linux (bindv6only=0), so "[::]" accepts IPv4 via v4-mapped
addresses too — nothing is lost. packaging/common is deliberately left
on "0.0.0.0" for now: Windows defaults IPV6_V6ONLY=1, where "[::]"
would drop v4 instead.
wait_for_peers_exact fell back to 0 when it could not read a container's peer
count. Two of its six call sites are the ACL denial checks, which expect
exactly 0 connected peers — so a failed docker exec satisfied them on the
first iteration and the isolation property the suite exists to prove was
never observed. Confirmed against a container that does not exist: the old
reader returns "0" and the expect-zero comparison passes.
The reader now returns the empty string when the daemon does not answer,
which no numeric comparison can satisfy, and the timeout message says which
of the two happened — never answered, or answered the wrong number. Same
shape as admission-cap-test.sh's read_peer_count.
A test that greps the daemon's log for a message the daemon stopped emitting
does not fail. It stops observing, and an assertion built on it — especially
one expecting a count of zero — then passes because nothing can be seen
rather than because nothing happened. Several findings have come from that
one class, so it is now checked mechanically.
testing/check-log-strings.py extracts the strings test code matches against
daemon log text and requires each to exist in src/. It reads four shapes:
python `"..." in line` tests, the first argument of the shell log helpers,
bash associative-array pattern tables, and greps whose input is daemon log
output. Patterns carry regex syntax, so each is reduced to the longest
literal run of every alternation branch, with escaping resolved in the same
pass — `\.` is a literal dot and `.` is a wildcard, and conflating them would
let a pattern match text that is not there.
Scoping is by what a grep READS, not by what its file mentions. A suite greps
its own analyzer output, fipsctl JSON, Tor's log and ping output, and none of
those have to correspond to a string in src/; scoping on the file produced 34
false positives. Strings that legitimately do not come from src/ are named in
ALLOWED with a reason, so the exceptions are reviewable rather than invisible.
It found six dead strings, three of which were not previously known: a second
copy of "Excessive decrypt failures" in the interop pattern table, a dead
"Handshake error" branch inside an alternation whose other branches still
matched, and "bootstrap failed" in the NAT suite. All six are corrected to
what the daemon emits, or dropped where the surviving branch already covers
the case. Verified by breaking it two ways and confirming it reports.
Also here, since it is the same failure shape one layer down: the shared log
library returned an empty string for a container whose logs it could not
read, so analysing a missing container reported no panics, no errors and no
sessions, and exited 0 — indistinguishable from a clean run. It now reports
which containers could not be read and raises, matching the fix already
landed in the chaos copy.
Two local CI runs on one host both asked docker for 172.20.0.0/24 and the
second lost its whole static family to "Pool overlaps". Docker honours a
fixed subnet request verbatim, so the only robust fix is to stop making one:
fips-net now requests no subnet and docker assigns from its own pool, which
cannot hand the same range to two runs.
That means node addresses are not known before `up`, so peers address each
other by container hostname instead. The generator emits node-<id>, or the
topology's docker_host where the compose hostname differs — only the gateway
profile, whose services are gw-*. External peers keep the address the
topology gives them, since it is not ours to assign. The resolv.conf mount
stays: dnsmasq is what forwards these names to docker's resolver and .fips
to the daemon, so removing it would take out every .fips assertion.
generated-configs is now per-run as well. A shared directory let two runs
overwrite each other's node configs, which the subnet collision had been
hiding by killing runs before that window opened. The generator, the compose
bind mounts and env_file, the six scripts that read it, and teardown all
follow FIPS_CI_NAME_SUFFIX; unset, every path renders as before. Teardown
keeps the directory after a failed run, where it is the evidence of what the
failing nodes were configured with.
Three things this exposed that were wrong independently:
admission-cap built its tcpdump patterns from the topology file's docker_ip
literals. Floating the subnet makes those match nothing, which would have
left its expect-zero "no Msg2 leaked" assertion passing because it could no
longer see anything at all. It now reads addresses from the running
containers. Restarting the denied peers together also made them swap
addresses, so each peer's counts were really the pair's total; they are
restarted one at a time now, and a check fails the suite outright if two
denied peers ever share an address, because per-peer attribution is
impossible once they do.
Attribute lookups in the generator used a fixed ten-line window and read the
next node's fields when a node omitted an attribute. An external node
followed by an internal one was classified as internal, which under hostname
peering would emit a name that resolves nowhere. Lookups are bounded to the
node's own block; generated output is byte-identical for all eight
topologies.
The rekey outbound-only variant used to rewrite peer addresses to hostnames
to set up its scenario. The generator now does that everywhere, so the
rewrite matched nothing and was silently doing no work. It asserts the
premise instead, and fails if a numeric address ever reappears.
Verified by running three instances of this compose at once — tcp-chain plus
two independent meshes — which drew 10.128.2/3/4.0/24 with no overlap while
both meshes passed ping-test 20/20 over the real .fips path. tcp-chain is
run by neither CI runner, so it was checked by hand: chain peer counts 1/2/1
and multi-hop .fips reachable both directions over TCP.
gateway-lan still pins its own IPv4 and fd02:: ranges and is unchanged here,
so the gateway profile is not yet concurrency-safe.
Four of these are places the parity guard could stop covering something without saying so, which is the failure mode it exists to prevent. Removing the deb-install suite array left it green, because the dispatch cross-check compared against a hardcoded name rather than the array it was meant to read. An arm-shaped line at an unexpected indent, or one written in quotes, was dropped silently by the arm pattern; unexpected indents are now a hard error rather than a quiet skip, and quoted arms parse. A matrix leg with a type of chaos or deb-install but no scenario key raised a Python traceback instead of reporting a problem, and a leg carrying a scenario but no suite key was skipped entirely even though scenario is now the identity for those legs.
The e2e builder no longer folds docker create's stderr into the container id. Docker prints warnings on success as well as failure, so a platform-mismatch warning would have become part of the id and made every later reference to that container fail, turning a working extraction into a spurious red.
Also corrects the comment about registering a new chaos assertion type, which named only one of the two key sets that actually need it.
The guard that keeps "local green" and "GitHub green" meaning the same thing was invoked by nothing. Its only entry point was an exec behind an explicit --check-parity flag, which by exec-ing could not be combined with an actual run, and no workflow step called it at all. A guard nobody runs is a guard that does not exist, which is how the mixed-profile divergence on next survived unnoticed.
It now runs as the first stage of every local run and as its own job on GitHub. Locally it reports through the same result-recording path as every other stage, so a divergence sets the run's exit status instead of scrolling past. It is deliberately placed above the mode branches, so a divergence also fails --only, --test-only and --build-only runs: whichever subset was asked for, the claim that a local result means what a GitHub result means is what has broken. The GitHub side installs pyyaml explicitly rather than assuming the runner image carries it.
--check-parity keeps working exactly as before. Three comment blocks that described the old folded comparison are corrected here, in the workflow, the local runner and the testing README; they were spread across three files and only two mention the guard by name, so the third is best found by searching for what it claims rather than for the script.
The guard collapsed all thirteen chaos legs to the token "chaos" and all five deb-install legs to "deb-install", so sixteen of the thirty-four integration legs were invisible to it. Deleting twelve chaos legs and four deb-install legs from a copy of the workflow left it still reporting "CI parity OK". The fold existed because the guard compared the cosmetic suite name, which really does differ between the runners; both sides have carried a directly comparable scenario field all along, so the fix is to compare that instead, along with the chaos flags.
The local suite set is now discovered by sweeping ci-local.sh for suite arrays rather than reading a hardcoded list of variable names, and the deb-install distro list is read from the suite's own script. A suite dispatched with no backing array is invisible to either approach, so every run_suite arm is now checked to have one and the guard names any that does not. That is not hypothetical: next dispatches its mixed-profile suite without an array, and the guard could only report it as missing from the local runner without being able to say why.
The guard also now refuses to run rather than failing obscurely when python3 or pyyaml is absent, since a parity check that cannot run must not look like one that passed. Verified by breaking each thing it guards: legs deleted from the workflow, drifted chaos flags, a fabricated suite array, and a dispatch arm whose array was removed. That last case initially did not fire because the arm pattern was pinned to the wrong indentation, which the test caught.
The scenario loader read every section and member through raw.get with a default, so a mistyped key was silently ignored and the block it belonged to stayed default-constructed while the YAML still looked correct on the page. Writing "assertion:" for "assertions:" disarmed a scenario's only assertions and left it unable to return a non-zero exit code; "link_flap:" for "link_flaps:" turned off the churn the scenario existed to inject, and a calm mesh then ran under the name of a chaos test.
Keys are now checked at load time, at the top level and inside every section with a fixed schema, including the netem policy bodies, the {min, max} ranges and the assertion bodies themselves. The error names the offending key, the section it appeared in, and the keys that section does understand. Three mappings are deliberately exempt because their keys are names the author chooses rather than a schema, and two sub-trees are passed through whole; all five are listed at the key sets with the reason.
Adding an assertion type now means registering it, so the next person to add one gets a rejection rather than a silent no-op. That coupling is the point and it is written down where they will see it. Verified against the seventeen scenarios in the tree, all of which still load, and against fixtures for each failure mode: a top-level typo, a nested typo, a typo inside an assertion body, and a section left empty, which previously raised an unrelated AttributeError from the next line of the loader.
The e2e scenarios build fips and fips-gateway in a builder image and extract them with two docker cp calls whose stderr went to /dev/null and whose exit status was never tested. Because build_fips_for_e2e ended in a log call it returned 0 on every path, so the caller's guard could not fire. A failed extraction left the previous run's binaries in the cache, they satisfied the caller's executable check, and the five e2e legs then exercised the previous commit's code and reported green.
The extraction now deletes the cached binaries before it starts, which is what makes the stale case impossible rather than merely detected, and it checks each docker cp independently, reports the failure with docker's own message, verifies each extracted file is non-empty and executable, and returns an explicit status. This mirrors what the deb-install suite already does.
Confirmed by inducing the failure rather than by observing a green run. With both extractions broken and a stale binary in place, the old code logged "Cached fips (26936 bytes)" and went on to build the runtime image around it; the new code reports both docker errors, deletes the stale binaries and fails the suite. Breaking either extraction on its own is caught separately, and an unmodified run passes 7 of 7.
The chaos simulation could not report a failure on this line either: scenarios collided on globally-scoped container names and a shared config directory, the runner swallowed whatever they raised, and the harness discarded their exit codes before anything read them. The five commits scope the names and directories per scenario, give an aborted run its own exit code and let that code reach the summary, stop a failed scenario harvesting whichever containers happen to hold its names, and keep the daemon's reason for a failure instead of throwing it away.
Testing only, and the merge is textual: every chaos file matches maint byte for byte, and the ci-local divergence on this line does not touch the changed regions.
Compose commands run with their output captured and with check set, so a failure raised a CalledProcessError and nothing ever looked at what the daemon had said. That exception reports the argv and the exit status and nothing else, so every scenario that died bringing its containers up left a runner log saying only that docker compose returned non-zero exit status 1, and the reason was captured and then discarded. In the run that prompted this work twelve scenarios failed that way and why they failed can no longer be established at all. Log the captured stderr and stdout before the error goes anywhere, keeping the tail of each so a verbose failure cannot fill the log file. A command that times out raises before its exit status is ever read, so that path records whatever it had emitted too, rather than leaving the same hole one shape over. Reproduced against both of the shapes this is meant to catch: an unusable network prefix now records the daemon's parse error, and a name already held by another container now records the conflict and names the container holding it. The success path logs nothing new, and a completed scenario's runner log is unchanged line for line.
The raising behaviour is deliberately untouched, since the abort flag and the exit code that came with it depend on it. The check is applied here rather than by subprocess so that a caller who passed check false gets the same record: both such callers are the compose down in teardown, which today can leave a mesh behind without saying so.
The same omission was costing the veth path its reasons. Failing ip commands had their stderr logged at debug, which the runner does not emit unless asked for verbose output, so a failed pair creation reached the log as nothing more than the names of the two interfaces it could not create. Raise that to a warning. The deletes issued to clear a stale pair are expected to fail and still say nothing.
Teardown runs from a finally, so it also runs after a failed setup, and it was guarded only on the topology and the compose file, both of which are set well before any container exists. A scenario that died bringing its containers up therefore ran the whole harvest anyway: final snapshots, docker logs, the analysis, the assertions and the metadata, all addressing containers by names that are global to the host. What it left behind was not an empty directory an investigator would notice but a full and plausible one, in the recorded case ten node logs and an analysis reporting two promotions and two parent switches, every byte of it from a different scenario's mesh. Gate the harvest on whether the containers ever started, and leave the compose down outside that gate so a partly successful start still gets cleaned up.
Record the outcome in a status file in every result directory, naming the run as completed, interrupted, aborted, setup-failed or teardown-failed, alongside the scenario, the seed and the container names it used. With the harvest gated, the presence of an analysis file is now itself proof that the scenario's own mesh existed, and the status file says which of the several ways a run can end applies. A run cut short by a signal keeps the existing exit codes rather than gaining one of its own: what it collected before stopping is real and still worth reporting, the status file records that it was truncated, and every wrapper already reports a Ctrl-C of its own.
Log collection never looked at the status of docker logs. It concatenated stdout and stderr unconditionally, so collecting from containers that were not there wrote the daemon's "No such container" reply into each node log and analysed the result as a mesh with no panics, no errors and no sessions, which reads exactly like a clean run and exits zero. Check the return code, and treat a harvest that cannot read every container, or that reads none, as a failed teardown. A teardown that raises now also reports on the same footing as a run that never started, instead of escaping past the exit codes as a bare traceback and being read as a malformed command line.
The simulation caught every exception a scenario raised, logged it, and exited 0, so a scenario that died during setup was recorded as a pass. Local CI discarded the exit code in any case: run_chaos ends in a call whose own last statement is an echo, so it returned 0 whatever the scenario did, and the parallel launcher's wait therefore always succeeded and always recorded a pass. Between them no chaos failure of any kind could turn a local run red, and none could since the script was written.
Carry the abort out of the run and give it its own exit code, ordered ahead of the two content codes so a run that never produced a mesh cannot report that mesh's panic and assertion counts. Return the scenario's status from run_chaos so the launcher sees it. The documented exit codes were stale before this change and are rewritten in full, including that the argument parser rejects a malformed command line with the same code that reports panics.
Expect scenarios that have been reporting green to start reporting red. In the last recorded local run twelve of the thirteen scenarios never got past starting their containers, and all thirteen were counted as passes; those that now execute have not run for months, and their assertions have never been evaluated against a mesh of their own.
Ethernet edges get a veth pair created in the host namespace and then moved into the containers, named from the node ids alone. ethernet-mesh and ethernet-only both connect n01 to n04 and always run together, so one tore down the other's live link. Hash the scenario suffix to four hex characters and insert those after the vh prefix; 15 characters is far too few to carry the suffix itself.
The names are now run-specific and never regenerated, so the delete-before-create no longer reclaims an interface an earlier run abandoned. ci-cleanup.sh takes over, deriving the names from the suffixes a run's teardown hands it. It is the first thing that script removes that is neither a docker object nor labelled, so an unscoped reap can sever a running bare simulation's links; testing/README.md and the --reap help say so.
Chaos scenarios run four at a time under local CI, but every one of them claimed the container names fips-node-nNN and wrote its generated configs and compose file to the same generated-configs/sim directory. Container names are global in Docker and are not scoped by the compose project, so concurrent scenarios collided on both, and a scenario could start containers from a compose file another had overwritten.
Thread the existing FIPS_CI_NAME_SUFFIX into the simulation. run_chaos narrows the run-wide suffix to the scenario, and the sim reads it once when the topology is built, applying it to the container names and to the config directory basename. The compose template renders the name through the topology accessor instead of duplicating the literal, so one expression produces every chaos container name.
The suffix is empty when the variable is unset, so a bare chaos.sh run and the hosted CI jobs render byte-identical names and paths. Verified by rendering every scenario's compose file before and after with the variable unset and diffing.
Both branches carry the same correction, authored separately because the
sans-IO relocation moved the logic into a core that returns an outcome rather
than acting inline. Git resolved the file move as a rename and offered to merge
the two versions textually, which would have been wrong in both files.
Resolved by keeping master's re-expression wholesale for the handler and its
tests, and taking the changelog entry from maint. Verified rather than assumed:
the resulting tree is byte identical to the reviewed and CI-green master commit
across every code path, and the merge contributes nothing but the fifteen line
changelog entry. Since the code is unchanged from a tree that already passed the
full local suite, that result still stands and no re-run is owed.
Nothing was lost from maint's side. Its tests are written against a file layout
that does not exist here, and master carries equivalent coverage written against
its own shape, including the multi-node chain and the cache warming cases that
the maint version does not have.
The same correction as on the maintenance line, re-expressed rather than merged,
because the sans-IO relocation moved this logic into a synchronous core that
returns an outcome instead of acting inline.
Delivery to the addressed node is no longer gated on the hop limit, and the
decrement happens before the drop decision, so a datagram that would leave with
zero is not sent. The reachable radius does not change: the old behavior refused
to deliver at zero but was willing to send at zero, and the new behavior is the
mirror of that, so the two cancel.
The shell above the core needed changing too, and that was the real work. It
carried two hop-limit predicates of its own, both justified by comments citing the
ordering that existed before the refactor. The first gated coordinate cache
warming, which this fix requires to be unconditional; left alone it would have
suppressed warming twice over, once for a datagram addressed here and now
delivered, and once for a transit datagram dropped for hop limit whose plaintext
coordinates are still perfectly usable. The second gated next hop resolution,
which carries a cache touch side effect, and now mirrors the core's
would-leave-zero rule instead of the old one. Neither divergence would have
surfaced as a test failure.
That last point is worth stating plainly, because it nearly went the other way.
Restoring the warming gate on top of the corrected core passes the entire suite
without a single failure. The tests could not see the seam between the core rule
and the shell's copy of it. So this adds coverage that can: a three node chain
pinning the decrement across a real hop, which is the only thing exercising the
shell predicates composed with the core, and two warming tests at a zero hop limit
covering both the delivered and the dropped case. Each was checked by putting the
specific defect it targets back and confirming the test fails.
One existing test that claimed to cover hop limit behavior asserted nothing
whatsoever. It asserts now, but those assertions do not discriminate this change,
since the case it exercises behaves identically under both rules. They guard
against the test going vacuous again, not against this defect, and the comment
says so.
The forwarding path tested the hop limit before testing whether the datagram was
addressed to this node, and decremented only on the forwarding branch. Two
consequences followed. A datagram addressed here that arrived with a zero hop
limit was dropped rather than delivered. And a forwarder receiving a transit
datagram with one hop left transmitted it with zero for the next hop to discard,
wasting a transmission on every expiring datagram.
Delivery to the addressed node is no longer gated on the hop limit, and the
decrement now happens before the drop decision, so a datagram that would leave
with zero is not sent. Saturating subtraction folds the two arrivals that cannot
be transmitted, already exhausted and last hop, into a single test.
The reachable radius does not change, which is worth stating because it is easy
to assume otherwise. The old behavior refused to deliver at zero but was willing
to send at zero, and the new behavior is the mirror of that, so the two cancel: a
path of h links still delivers for any source value of h or more. What this
actually buys is the wasted transmission, the exhaustion counter charging at the
node that makes the decision rather than the one after it, conformance with the
specified semantics, and one real gain during a rolling upgrade, where an
unupgraded forwarder feeding an upgraded destination delivers one hop further
than either version does on its own.
Coordinate cache warming moved ahead of both decisions, so a transit datagram
later dropped for hop limit still contributes its plaintext coordinates. The only
arrivals this newly warms from are those with a zero hop limit, and every insert
they can make is already achievable with a value of one, so it grants nothing
that was not already available and adds no memory growth the cap does not bound.
Four tests added: delivery at zero to this node, the transit drop at one, the
transit pass at two, and the one per hop decrement across a three node chain,
which brackets the emitted value from both sides since it is only observable
through what the next hop does. One existing test was renamed because its
destination was this node, so it never exercised the transit path its name
claimed, and it asserted nothing at all.
maint carries only its 0.4.2-dev version bump; master keeps its own
0.5.0-dev development version. Recorded as a linkage merge so later
forward-merges of real fixes land cleanly.
Bump the version to 0.4.2-dev and update the status badge and prose. The
0.4.x patch line continues; v0.4.1 remains the shipped release it points at.
Follows the shape of the v0.4.0 rollover commit. No maint reset was needed
this time: v0.4.1 was cut from maint, so the tag is already on this line and
it simply continues.
Carries the v0.4.1 content up the one-way branch flow. The release itself
belongs to maint, so the parts of this merge that identify a version are
resolved in master's favor and the parts that describe the project's state
are taken from maint.
Kept master's: Cargo.toml and Cargo.lock at 0.5.0-dev, the README status
badge, and the "FIPS is at v0.5.0-dev on the master branch" line. A patch
release consumes no minor version and master's development line is
unaffected by it.
Took from maint: the bloom antipoison FPR default change and the
duplicate-definition fix it rests on, the two docs that describe them, the
new v0.4.1 release notes, the correction to the v0.4.0 release date, and the
[0.4.1] changelog section, which slots below master's own [Unreleased] and
above [0.4.0]. The root RELEASE-NOTES.md mirror moves to v0.4.1 because it
tracks the latest shipped release, which also clears the stale provisional
date it had been carrying.
One line needed splitting rather than choosing: the README said "v0.4.0 has
shipped" inside the paragraph that identifies master as 0.5.0-dev. The
version identity stays master's and the shipped-release pointer moves to
v0.4.1.
Quartet green on the result: 1645 tests passed, clippy clean with -D warnings.
Drop the -dev suffix from the package version and refresh the lockfile.
The version is single-sourced from CARGO_PKG_VERSION, so build.rs and
src/version.rs need no change.
Also fix the archived v0.4.0 release notes, which still read
"2026-06-21 (provisional)" while the changelog records the actual
release date of 2026-06-27. The date was confirmed in the changelog at
the time but not in the two release-notes files that carry it, so the
shipped notes have been showing a wrong date and the word provisional
since that release. The root mirror is regenerated for v0.4.1, so only
the archived copy needed correcting.
Back-fill the changelog for the six user-facing changes on this line
since v0.4.0: the antipoison FPR cap default, the bloom probe and
secp256k1 context performance work, the coordinate-cache and path-MTU
fixes, and the removed parent_switched counter. The Unreleased section
was empty, so all six entries are new.
Write the v0.4.1 release notes and mirror them to the root copy. The
notes lead with the FPR cap change and state plainly that this is the
second raise of that default in two releases, that it buys headroom
rather than fixing the underlying fixed-filter constraint, and that the
structural remedy is the v2 filter work. They also flag the rolling
upgrade window, where an upgraded and a not-yet-upgraded node can
disagree about mesh size, and point consumers of parent_switched at
parent_switches.
Bump the README status badge and the roadmap prose together so the two
agree.
The inbound FilterAnnounce cap at 0.10 rejects aggregates that are
legitimately near their operating ceiling, before the network reaches
the fixed-filter capacity limit. Raise the default to 0.20, which
corresponds to fill 0.7248 at k=5, about 2,114 entries on the 1 KB
filter (Swamidass-Baldi).
Also remove the duplicate default definitions in BloomConfig. Each
default was written twice, once in impl Default and once in the serde
default function, with nothing enforcing that they agree, so a config
file that omits the key took a different path from one that sets it.
impl Default now delegates to the serde default functions, leaving a
single source of truth.
The isolation loop and the failure-log dump both built container names
without the run suffix, so under a suffixed run they addressed containers
that do not exist.
The consequence was worse than the visible failure. Two of the three
checks in that loop assert a ping FAILS, and a ping into a non-existent
container fails for the wrong reason - so both passed vacuously, and the
security assertion the loop exists for was silently a no-op. Only the
third check, which expects a ping to succeed, noticed anything was wrong.
Add a guard that fails loudly when the container is absent, so a future
naming slip cannot quietly turn these back into no-ops rather than
surfacing as one confusing failure among two false passes.
These sites were missed because the suite passes when run by hand: with no
suffix set the unsuffixed names are correct, and the defect only appears
under the automation that sets one. Verified this time with the suffix
set, which is the condition that matters: 18 of 18, container names
resolving to real containers.
Making the sidecar network name unique per run exposed a latent problem
rather than causing a new one. With a fixed name, compose found any
existing network and reused it, which quietly masked the fact that the
address range was hardcoded. With a unique name there is nothing to
reuse, so the range is requested every time and the run fails outright if
anything already holds it.
Claim the range instead of assuming it. The suite now tries to create its
network on a candidate range and, if the daemon reports an overlap, moves
to the next candidate. The daemon's own address pool becomes the arbiter,
so two concurrent runs cannot land on the same range at all - as opposed
to a range derived from the run identifier, which only makes a collision
unlikely, and a collision is the precise failure being avoided.
The node addresses are derived from whichever range is claimed rather than
written alongside it, including the gateway address the isolation check
probes. Moved to a range that nothing in the tree claims and that sits
outside the daemon's default pool.
Only an overlap is worth advancing on. Any other creation failure is real
and is reported with the daemon's own message, rather than being retried
sixty-four times and buried - a lesson from a failure elsewhere in the
harness this week that was undiagnosable because its error output was
discarded.
All three nodes now attach to the pre-created network as external, so
teardown removes it explicitly.
Verified: three concurrent claims take three distinct ranges; a
non-overlap error fails fast rather than looping; the suite passes 18/18;
and the claimed gateway address is confirmed pingable when not firewalled,
so the isolation assertions still mean something.
The suite red-ed with an empty final peer_count while every substantive
assertion passed: both denied peers showed inbound handshake attempts and
zero outbound responses, which is the gate this suite exists to verify.
Two pre-existing defects combined. The final count was sampled once, in
the same second the load driver issues its last round of peer restarts, so
the read can land on a daemon busy with those restarts and return nothing.
And the fallback meant to cover that could never fire, because the shell
applies it to the last stage of the pipeline, which succeeds on empty
input - so an unanswered query was indistinguishable from a genuine zero,
and reported as a cap violation.
Poll the final read for a short window instead, and give the two cases
distinct output. A real regression still fails, just after the retry
window, since the loop exits early only on the expected value. Extract the
read the two phases share rather than leaving them to drift.
Two runs on one host destroyed each other's containers, producing
mid-test "No such container" failures that look like real defects. The
automated builder runs a full local CI on the same box every few minutes,
so the machine is contended almost always and this has red-ed both a hand
run and an automated gate.
Two independent causes are fixed here. Container names were hardcoded, and
docker names are global rather than scoped by compose project, so two runs
collided on the same name; every name now takes an optional suffix that
the harness sets from the run id. And the cleanup sweep matched a label
shared by every run, so one run's teardown force-removed another's
containers; resources now also carry a per-run label and the sweep can be
narrowed to it.
A third hazard turned up that was not in the original report: the sidecar
suite passes explicit compose project names, which override the run-scoped
project and put it outside the shared prefix entirely. Its project names,
network, and derived container references are now scoped too.
Both are default-off. With the suffix unset, names render exactly as they
do today and a bare compose invocation is unchanged, which is what keeps
the hosted CI and the documentation correct. A cleanup run with no run id
still reaps everything, which is what a manual "clear the box" wants.
The literal-name sweep was not sufficient: six scripts build container
names dynamically from node labels, and two suites create their own
containers outside compose. Those are handled at their construction sites.
Verified: syntax check on all modified scripts; compose validation on
every modified file with the suffix both set and unset; and a synthetic
two-run reproduction that shows the old cleanup destroying a bystander run
and the new one leaving it alone.
Known gap: subnets are still hardcoded, so two concurrent full runs will
still collide on address-pool overlap. That fix reaches into topology
configs, chaos scenarios, diagrams and production source, so it is left
for its own change rather than half-done here.
The lifecycle view's doc described the shell as implementing it over a
`connections` map that no longer exists. The establish view named that map
too, and was doubly wrong: both of its snapshot builders read only
`peers`. Rewrite both to describe the maps actually read.
Several comments in the node module carried shorthand from private
working notes - bare parenthetical labels, and one reference to an
internal tracker item - none of which mean anything to a reader of this
repo. Rewrite them so each clause states its own reasoning inline. The
paragraph above the tracker reference already carried the rationale, so
that one is simply dropped.
Comment-only; no behavior change.
The rekey suites' Phase 4 polled until the initiator-cutover count
reached its value at phase entry plus two, while Phase 6 asserts an
absolute minimum of four. On an unloaded host the ping phases can
outrun the jittered first rekey cycle, so Phase 4 can begin with a
single cutover on the books, release at three, and leave Phase 6 to
lose a sub-second race against the remaining first-cycle cutovers.
A rekey-outbound-only run failed exactly this way: five counted
cutover lines by t=32.6s of container life, with the Phase-6 count
snapshot at t~32.4s seeing three.
Floor the Phase-4 wait target at four so a released wait guarantees
the Phase-6 assertion — the docker-logs count is monotone over a
container's lifetime. A genuine rekey stall still times out the
bounded wait and fails Phase 6 with the real count.
Also drop the second-cycle framing from the comment, the phase
banner, and the assertion description: with rekey timers resetting at
each cutover a second cycle cannot begin inside this test's window,
so the counted events are first-cycle cutovers spread across the
topology's links, emitted once or twice per completed link rekey
depending on which side's promotion path logs them.
The per-peer control machine has absorbed every field the pending
connection carried. What remained was a struct holding two Noise
handles beside a duplicate copy of bookkeeping nobody read. Replace it
with a small carrier for the two handles and delete the type.
Presence of that carrier, not the state of the handles inside it, is
what marks a machine as mid-handshake. The distinction is essential
rather than stylistic: a failed handshake drops its initiation handle
and is deliberately retained so the stale sweep can reclaim it, and a
completed one has its session taken before disposal. Deriving presence
from the handles would make both invisible to the sweep, the
connection count, and the peering budget at once, leaking the slot
permanently. A test drives an empty carrier past every presence
predicate and then detaches it, so a future edit cannot quietly couple
the two.
The remote startup epoch now comes from the surviving carrier, which
the handshake operations already wrote at the same two points with the
same value. The paired writes onto the pending connection's own
bookkeeping had no readers left and are gone.
The handshake-phase surface leaves the public API: it was public by
accident rather than design, and the machine behind it is crate
internal. Callers outside the crate that need a view of pending
handshakes go through the operator queries, which are unchanged.
ConnectionState::inbound_with_transport loses its last non-test caller
with the inbound seed and is marked test-only.
The pending connection and the per-peer control machine have carried
duplicate copies of the handshake-phase fields since the machine gained
its own connection state. Read them from the machine and drop the
connection's projections.
The peer identity is the sharp one. The connection learned it from msg1
and the machine's carrier did not, so the two views genuinely disagreed
for inbound connections until the Noise operations moved onto the
machine and began recording each result on both. That is now in place,
so every reader can take the machine's copy unchanged: the establish
snapshot, the promotion sweep for competing connections, the dial and
path in-progress checks, the peering observations and per-peer in-flight
budget, and the stale-connection sweep's retry address.
Also repointed: our_index, their_index, transport_id, started_at, the
stored handshake message bytes, and the idle-timeout check. Two
duplicate index writes on the connection are dropped, both immediately
preceded by the machine-side write of the same value. Reads that
previously came off a connection detached just before its machine was
disposed now capture the machine's value first.
Test seeding follows the establish paths: the seed builders write
our_index to the carrier, which promotion now reads. A new test pins
the inbound identity learn on the carrier and the retry address a
failed inbound connection reports to the sweep, so a silent regression
to a blank identity cannot pass.
ConnectionState::duration loses its last non-test caller with the
connection accessor and is marked test-only.
These three had no counterpart on the control machine, so readers still
reached them through the pending connection. Add machine-side
accessors and repoint every reader, then drop the connection's.
The peer address needed its writes lifted, not just its reads
repointed: the machine's copy was never written. It is now written at
each of the points the connection's copy was — the inbound seed, the
dial, message-2 completion, and the two paths that seed a machine from
a pre-built connection — so promotion and the resend path read a value
with the same provenance at the same time as before.
Link and direction need no lift. Both machine constructors already
seed them from the same arguments the connection is built with, so an
outbound machine carries outbound state and an inbound one inbound,
and the machine's link always equals the connection's. The handshake
operations' direction guards read the machine's copy for the same
reason.
The stale-connection sweep's teardown log and its resend path both now
take the transport and address from the machine, and each keeps its own
check that a pending connection is still attached rather than relying
on the caller to have established it.
Add a test pinning link, direction, and address on the two shapes that
seed a carrier independently — the dial and an accepted message 1. The
cross-connection winner reads both values from a carrier one of those
two already seeded.
The pending connection drove its own Noise handshake while the control
machine held it, so the crypto and the state it produces lived on the
carrier that is going away. Move the six operations onto the machine:
starting and completing an initiation, processing an inbound
initiation, taking the session, testing for one, and dropping the
handle on failure. Bodies are unchanged apart from reaching the
handles through the attached connection.
Each operation now records its results on both carriers. The learned
identity, the remote epoch, and the activity stamp are written to the
machine's own bookkeeping at the same point and with the same value as
they are written to the connection's. The connection's copies still
have readers until those reads are repointed, so both have to be
written; the machine's copy of the learned identity was previously
never populated for an inbound connection, which is why an inbound
pending row showed no expected peer.
Driving the handshake from the machine means the machine has to exist
before the crypto runs. On the inbound path it is built above the
message-1 processing and still kept local, so a rejected message
leaves no registry entry and allocates no index. On the outbound path
it already existed from the dial, so it simply takes the connection
before the index allocation, and both failure arms unwind it as they
did.
Completing message 2 no longer mirrors the activity stamp separately,
since the completion itself now writes both carriers at the point the
mirror was approximating.
Three tests cover what the compiler cannot. A connection whose
handshake failed holds neither Noise handle yet must stay visible to
the stale-connection sweep, or every failed connection would leak and
hold a peering-budget slot forever. A message 1 rejected by the crypto
or by the ACL must leave no machine, no index, and the same rejection
count as before. A dial whose message-1 preparation fails must unwind
the machine registered at dial time; that test drives the
index-allocation failure rather than a crypto failure, which is the
arm this change actually widens, since the allocation now happens with
the connection already attached.
`Node::connections()` yielded the pending connection itself, so its
consumers reached the handshake-phase fields through that value. The
pending connection is being folded into the control machine, and the
fields will move off it, so yield the machine (keyed by its link) and
let each consumer reach what it needs from there.
Membership is unchanged: the iterator still selects machines that
carry a pending connection, which is the predicate `connection_count`
and the stale-connection sweep already use. Every consumer takes the
same value from the same place, one hop further out, and no field
changes carrier here.
The method is now internal to the crate. It yields the control
machine, which is not part of the published surface, and nothing
outside the crate iterates pending connections — operator views of
the handshake phase go through the query surface instead.
`promote_connection` detached the pending connection and then
interleaved reads off that detached value with three separate lookups
of the same control machine. Gather the machine-side fields
(`their_index`, `transport_id`, link stats) in the single `get_mut`
that takes the connection, so the machine is borrowed once.
Behaviour is unchanged. The connection is still taken before anything
is validated, so a rejected promotion leaves the machine with no
pending connection, and the prelude only gathers options: the checks
below it still report the first missing field in the same order
(`our_index`, `their_index`, `transport_id`, `source_addr`).
Link stats move to the prelude, ahead of those checks. The value is
the same either way: they live on the surviving carrier rather than
the detached connection, and nothing between the two points touches
the machine map. The lookup could not fail at the old site either,
since the function had already reached it through a successful lookup
on the same key, so dropping the defaulting arm changes nothing.
Add a test covering the error order and the detach, driving promotion
with connections missing each required field in turn plus every later
one, so an implementation that validated during the prelude would
report the wrong field. Test seeding grows a variant that lets the
caller shape the seed.
Tests built a free-standing PeerConnection, mutated it, and handed it to
Node::add_connection by value. None of those sites survives the removal of
PeerConnection, so converting them afterwards would mean one enormous commit
that cannot be reviewed honestly. Convert them now, while the struct still
exists and the conversion can be validated against a green tree.
Adds a cfg(test) Node::seed_handshake_machine plus a HandshakeSeed builder,
and rewrites make_completed_connection (now seed_completed_connection) and
the twenty inline builders onto it. add_connection keeps its body and loses
its test callers.
The builder's carrier seeding is a verbatim copy of add_connection's: the two
conditional writes for their_index and transport_id, then set_leg, through the
same entry().or_insert_with() so an existing leg-less machine keeps its
constructor-side fields. Nothing else reaches the carrier -- our_index,
source_addr, post-construction started_at and the stored handshake bytes stay
leg-only. Seeding more than that would let these tests observe a carrier
richer than production's and keep passing even if a later production
write-lift were missed.
The Noise exchange now runs on the already-seeded leg rather than before the
hand-over. That is neutral because the only read of expected_identity is
guarded by is_outbound, and no crypto method allocates a session index.
Also adds a compile-time check that PeerAction is Clone + Eq, which is what
keeps a runtime handle from being smuggled into an action payload.
The session index was carried three ways: on the leg, on the peer machine's own
connection-state copy, and on a separate machine-owned shadow field. Populate the
machine's connection copy on the outbound path (it was written only on the inbound
and cross-connection-swap paths before), then delete the shadow field and route its
readers, the decrypt-registration lifecycle, and the rekey cutover through the
single carrier. The shadow and the connection copy diverged only on paths that are
unreachable or dormant in production (a rekey cutover with no newly allocated index,
the swap winner's later reads, and the never-dispatched timeout/disconnect events),
so the merge is byte-identical for every reachable path. Rewrote the outbound-promote
index tests and the dial-time comment for the single-carrier contract.
The connection leg's their_index, transport_id and link_stats duplicated the peer
machine's own connection-state copy. Route every production reader through the
machine copy: promotion, the stale-connection reaper, the outbound msg1 resend, and
the in-use / connecting-path checks. Seed the inbound machine's transport_id from the
msg1 packet (the same source the leg used), matching the outbound dial seed, so the
hard-required promote read never sees a missing transport. link_stats is a
never-mutated zero seed, so its leg accessors are removed outright; the leg
their_index/transport_id getters/setters remain only for the test-only connection
builder. Also delete the write-only next_resend_at_ms field from the connection
state (the resend deadline is carried by the machine-armed retransmit timer, not this
field). Byte-identical.
The connection leg's stored msg1/msg2 handshake-resend buffers duplicated the
peer machine's own connection-state copy. Write the machine copy at the outbound
msg1-prep and the two inbound authorize sites, then source the retransmit
resend-bytes reader, send_stored_msg1, and the pending tier of find_stored_msg2
from the machine. The post-promote active-peer msg2 copy is written from the same
wire bytes, so the pending tier now matching after promotion is value-identical.
Byte-identical resend behavior.
The expected_peer connection-row field read the peer identity off the leg;
source it from the machine's own connection-state copy instead, matching the
started_at/last_activity treatment. The machine carrier holds the same identity
as the leg for every connection that appears in the view: outbound legs seed
both from the dialed identity, and inbound legs never rest in the connections
view (the machine takes the leg at promotion), so their machine copy is
unobserved. The leg accessor stays for the crypto-extraction path and retry
decisions. Byte-identical.
The connection leg's started_at and last_activity duplicated the peer
machine's own connection-state copy. Make the machine copy the sole live
telemetry and timeout carrier: re-stamp it with the leg's provenance when
the leg is attached at handshake start (the msg1-prep clock, not the earlier
dial-time constructor value) and mirror the completion touch, then delete the
leg's last_activity/touch accessors and source the connection-row projection,
show_connections, and the timeout readers from the machine. Byte-identical
for all normal paths.
The connection leg's HandshakeState field duplicated the peer machine's
handshake phase. Delete it and derive the displayed handshake state string
from the machine's PeerState, looked up by link id (mirroring the resend
count). Move the failure signal onto the machine: a send_failed flag that
preserves retransmit eligibility (the machine stays in its handshake phase),
alongside the existing Failed state. The leg's crypto self-gates now guard on
Noise handle presence instead of the deleted phase field, and mark_failed
only drops the handle. Telemetry strings, wire bytes, index allocation, and
the stale-connection reaping are byte-identical for all normal paths.
handle_msg1 now creates one local machine before classification and routes
on the InboundDecision it returns, instead of calling establish_inbound in
the shell and re-deriving the decision inside each arm. The promote and
restart arms drop their own per-arm machine construction and the redundant
InboundMsg1 step; their phase-1 actions come from the single call. The
effect-bearing arm bodies (rekey-respond, duplicate resend, reject
bookkeeping, and the promote/restart tails) are unchanged.
Behavior-neutral: the same establish_inbound evaluation over the same
snapshot and wire, done once in the machine method rather than once in the
shell, with byte-identical arm bodies and no change to index allocation,
wire sends, machine state, or telemetry.
Make the FIPS core build and run as an embedded Android library. The host
app owns the TUN (e.g. an Android VpnService) and FIPS performs no
system-TUN or CAP_NET_ADMIN operations.
Squashed from the following changes:
- gate desktop transports/TUN by target_os, not features: a plain
`cargo build` now compiles for every target with no flags. Ethernet (raw
AF_PACKET / BPF) is gated to linux/macos, so Android (target_os =
"android", not "linux") self-excludes it as Windows already did; real
system-TUN ops are gated per linux/macos and Android gets a no-op stub;
the ipi6_ifindex cast handles it being i32 on Android vs u32 on macOS. No
Cargo features are introduced; desktop builds are unchanged.
- app-owned TUN seam: Node::enable_app_owned_tun() lets an embedder that
owns the TUN fd exchange IPv6 packet bytes with FIPS over channels
instead of FIPS creating a system TUN device. It returns (app_outbound_tx,
app_inbound_rx): the embedder pushes packets read from its fd into the
outbound sender (app -> mesh) and pulls packets destined for its fd from
the inbound receiver (mesh -> app). start() gates system-TUN creation on
tun_tx being unset, so with the channels pre-installed it skips device
creation and does no system-TUN ops; both directions reuse the existing
inbound-shim and run_rx_loop wiring. Packets entering via app_outbound_tx
bypass handle_tun_packet, so the embedder must push only fd00::/8-destined
packets and clamp TCP MSS on outbound SYNs; the rustdoc and the
IPv6-adapter design doc spell this out.
- keep the android target warning-clean so the cross-compile check passes
clippy -D warnings.
- add an Android cross-compile CI check: cross-compile the library for
aarch64-linux-android via cargo-ndk and run clippy -D warnings. Android
ships as an embedded library (the host app owns the TUN), so there is no
daemon binary to package; this is a check job, not a packaging one.
- docs: list Android as a supported platform.
Tests: app_owned_tun_seam_wires_channels covers the channel round-trip and
the Active state; start_skips_system_tun_when_app_owned runs start() and
asserts no named system device is created.
The inbound msg1 machine arm re-derived the establish decision for
its own action selection while the shell matched on a separately
computed copy. Expose the arm as a method returning the decision
alongside the actions so a driver can route on it directly; the
step dispatch delegates and keeps its action-only shape.
The dead rekey-respond arm is stripped to decision-only and its
helper deleted: it allocated from the real index allocator, wrote
the rekey shadow index, emitted a wrong-framed rekey send, stamped
the dampening clock, and flipped state, none of which happens on
the live path, where the inline respond body owns all effects. The
resend-msg2 arm's send emission is stripped the same way; the
decision already carries the stored bytes.
send_stored_msg1 marked the embedded leg failed by writing it
directly from the shell. Route the write through a new
HandshakeSendFailed machine event instead: the machine marks its
leg so the stale-connection sweep reclaims it, without leaving the
handshaking state, so retransmit eligibility survives the window
between the failed send and the sweep exactly as before.
send_stored_msg1 gains a now_ms parameter threaded from the action
executor for the step call; the failure arm itself ignores it.
handle_msg2 no longer pre-computes establish_outbound alongside the
machine's own evaluation of the same snapshot. The shell now builds
the snapshot, steps the persistent outbound machine once at the
decision point, and routes on what comes back: the promote action
vector drives promotion through the executor as before, and the
ResolveCrossConnection decision selects the inline swap/keep
resolution bodies, which are unchanged.
The machine step and its defensive transient-rebuild move up from
the promote arm to the decision point; the cross-connection path
keeps its take-leg-then-dispose ordering with the step preceding
both. Comment prose at the touched sites refreshed to describe the
single-decision-site shape.
The machine's on_msg2 cross-connection arms crystallized state and
emitted FreeIndex/RegisterDecryptSession actions that duplicate the
inline shell resolution, but they are unreachable on the live path
(the shell removes the machine before running the swap/keep bodies).
Making them live in that shape would double-free the outbound index
through the executor.
Strip both arms to a single new ResolveCrossConnection { swap }
action: a decision conveyed to the driver, not an effect. The shell
intercepts it and runs the inline resolution, which owns all effects
permanently. The action executor gets a defensive unreachable arm.
The Promote arm and set_their_index are unchanged.
The pending-handshake PeerConnection map and the per-peer control
machine map were parallel LinkId-keyed structures whose keysets must
stay coherent by hand. With every leg now born with a machine, the leg
becomes storage inside its machine (leg: Option<PeerConnection>, pure
storage the machine never reads or drives) and Node loses the
connections field; every access routes through the machine.
The non-mechanical lowerings, each argued at the site: the
rekey-vs-establish gate in handle_msg2 tests leg-absence (an established
peer's machine stays keyed by its link, so machine-presence would
misclassify every rekey msg2 as a fresh establish); the
connecting-predicates, peering observation, and handshake-slot budget
iterate machines-with-legs so connect-window machines (leg not yet
born) are excluded exactly as before and never double-counted against
their pending-connect slot; the cross-connection extract takes the leg
before disposing the machine; the stale reaper takes the leg and leaves
the machine untouched when none is present, matching the old early
return. The map-coherence debug check keeps its machine-has-carrier
direction with the embedded leg as a carrier; the leg-to-machine
direction is now true by construction and its gate const is gone.
connection_count() counts machines with legs; the connections()
iterator, the test seams, and the control-socket connection rows are
re-implemented over the embedded legs with unchanged output.
Assert, once per tick in debug builds, that the peer control-machine map
and its carriers stay coherent: every control machine has a live
handshake leg, an active peer, or a pending connect (a machine with none
is a leak the stale reaper can never see), and every handshake leg has a
machine. The second direction sits behind a file-local const so a branch
where handshake-window legs legitimately run machine-less can gate it
off without weakening the leak tripwire.
Teach the add_connection test seam to seed a control machine for the leg
it inserts (derived from the connection's direction and identity), and
remove_connection to dispose it, so the seam-built topologies satisfy
the check. Three unit tests: seam coherence, coherence through a real
promotion, and the orphaned-machine panic.
Several module and field docs still described the per-peer machine as
unwired shadow scaffolding. It has been live for some time: machines are
inserted at dial and inbound msg1, stepped by the handshake handlers and
the rekey-cadence and liveness-reap routers, and the executor's
SwapSendState/CompleteDrain/InvalidateSendState arms are the authoritative
paths (the inline bodies survive only as debug-assert release fallbacks).
Rewrite those docs to the current truth while keeping the still-true
dormancy facts: PeerEvent::Timeout and PeerEvent::Tick are never
dispatched in production, retransmit fires on the machine-armed deadline
while the timeout reaper keys on timer presence with the config
threshold, and the remaining inert executor stubs are SendRekey,
SendLinkMessage, and the connected-UDP arms. Drop the stale
allow(dead_code) on the peer_machines field.
PeerSlot (and its entire impl surface) was referenced only by its own
module tests and the lib.rs re-export; peer storage has always used the
separate connections/peers maps. PeerConnection's resend_count/
next_resend_at_ms/record_resend delegations had no production callers:
the live resend counter is machine-sourced (connection_resend_count reads
the per-peer machine), and the FSP session layer uses SessionEntry's own
methods. ConnectionState::next_resend_at_ms is now test-only (its
remaining callers are the fmp state unit tests) and marked cfg(test).
The PeerSlot unit tests go with the enum; test_resend_count_tracking is
dropped because the delegation target's schedule arithmetic is already
covered by the fmp resend_bookkeeping test.
Router-to-router radio backhaul over an open 802.11s mesh interface,
with FIPS providing all encryption (Noise IK), authentication, and
routing on top of bare L2 neighbor links. The mesh runs OPEN with
mesh_fwding 0 — SAE would duplicate the Noise layer and force ath10k
raw mode, and FIPS is the routing layer — so the Noise handshake is the
real auth/encryption boundary and FIPS's spanning tree does the routing.
fips-mesh-setup: an opt-in UCI helper that creates a per-radio
mesh-point interface (radio0 -> fips-mesh0, radio1 -> fips-mesh1;
trailing-digit derivation with a free-index fallback and a collision
guard). Radio setup stays opt-in — a package must not commandeer radios
on install. 'remove' takes an optional radio and otherwise removes all
instances. Dual-band routers get one instance per radio; FIPS treats the
two backhaul paths as failover, not multipath: it keeps one active link
per peer (cross-connection resolution picks a single winner), and the
second band stands by, re-establishing the peer after keepalive timeout —
traffic never uses both bands at once.
fips.yaml ships the mesh0/mesh1 Ethernet-transport entries commented
out, so a stock install that never creates fips-mesh* logs no per-boot
"interface missing" bind warning. fips-mesh-setup uncomments the matching
meshN block when it creates the interface and re-comments it on remove,
so the flash-and-drop-in flow needs no manual config edit. The file is
rewritten 0600-first (it may hold an inline nsec) via an atomic replace.
Two field-found silent non-peering causes are surfaced by the helper and
the guide:
- Same channel: mesh points only peer on a shared channel, and 'auto'
lets each radio pick its own. The helper prints the radio's
band/channel and warns loudly on 'auto' with the exact uci command to
pin one; the how-to gains an ordered no-peers triage (channel mismatch,
on-air scan check, DFS CAC wait, regdomain).
- STA channel capture: a client (sta) interface drags the whole radio to
its upstream AP's channel, so a mesh pinned elsewhere never joins and
does not recover until the STA disconnects. The helper warns when the
target radio carries a STA; the guide documents the incompatibility of a
roaming uplink with a fixed-channel mesh on the same radio.
Both the create and remove paths run 'wifi reload', which briefly drops
every client AP on all radios; the how-to sets that expectation.
Regression test: the shipped OpenWrt fips.yaml must parse via the real
Config deserializer in both states — as shipped (mesh inactive) and after
the uncomment the helper performs.
Packaging: the helper is installed across the ipk/apk/buildroot paths
(three synced copies), with the CI structural checks and shellcheck
targets extended to cover it. Full guide in
docs/how-to/set-up-80211s-mesh-backhaul.md.
Phase 4 of the rekey suite waited for only one more initiator cutover
(guaranteeing three) while Phase 6 asserts at least four. The fourth
cutover then had to land in the brief window between the wait returning
and the Phase 6 log snapshot, so host load could push it past the window
and fail the run even though every cutover completed correctly. Wait for
two more cutovers, the full second rekey cycle, matching the Phase 6
threshold, so the asserted count is guaranteed before it is checked.
Feed the per-peer machine two observation events so its control/rekey
shadow state stays coherent with the inline crypto-session installs,
without moving those installs. The crypto effect bodies are unchanged.
On rekey-msg2 completion, feed the machine the responder index on the
success path only (after the pending session is installed); the abandon
path feeds nothing.
On cross-connection resolution, feed the machine the swap outcome: on a
swap it records the new outbound and responder indices to mirror the
session replacement, and on a keep it does nothing. The observation
targets the promoted peer's machine, not the outbound leg whose machine
is removed as the block is entered.
Both observation handlers emit no action; nothing consumes the updated
shadow state yet.
Move the outbound handshake-timeout reap off the unconditional check_timeouts
scan onto the machine-armed HandshakeTimeout timer. A new drive_handshake_timeouts
(run before the retransmit drive, so a timed-out leg is reaped rather than resent
on the same tick) reaps the outbound legs that carry a HandshakeTimeout timer and
have idle-timed-out this tick.
The timer's presence selects the leg (only outbound legs arm one; IK inbound arms
none); the reap threshold is the shell is_timed_out(now, config) predicate, not
the timer's stored deadline. The machine arms the timer from a hardcoded constant
at dial, which is not authoritative for an operator-tuned handshake_timeout_secs,
so reading the threshold from config each tick keeps the reap neutral for any
timeout value and on the last_activity clock exactly as before.
check_timeouts keeps reaping everything else: failed connections (all of them,
promptly) and the idle-timeout of legs without a machine timer (inbound legs, and
any machine-less connection). Total coverage is unchanged. check_timeouts runs
before the timer drive, so a failed outbound leg is reaped there first, its timers
dropped, and the timeout drive never double-fires on it.
Split drive_peer_timers into the timeout drive and the (byte-identical) retransmit
drive. The dormant machine timeout handler is not dispatched, so the session index
is freed once, by cleanup_stale_connection. A regression test drives a timed-out
outbound leg to reap; the existing check_timeouts tests continue to anchor the
residual sweep. Test count 1631 to 1632.
Move the outbound msg1 resend off the unconditional tick function onto the
machine-armed retransmit timer. The per-peer machine already arms a
HandshakeRetransmit deadline at dial; a new drive_peer_timers fires the due ones
(kind-filtered to the retransmit timer) and homes the resend counter on the
machine, where the operator-visible count now reads from.
The resend decision reads the same operator config as before (interval, backoff,
max), the wire bytes and transport target still come from the shell connection,
and the pure core computes the backoff schedule. As with the deleted
resend_pending_handshakes, the count and reschedule advance only on a successful
send: a failed send neither advances the count nor marks the connection failed,
it just retries on the next tick. The handshake-timeout timer stays on the legacy
check_timeouts path, and the rekey/liveness timers keep their own shell drivers,
so drive_peer_timers deliberately fires only the retransmit kind.
The show_connections resend count is relocated to read from the machine (the
counter's new home) via connection_resend_count; machine-less and inbound
connections report 0, matching what the shell connection reported before. Delete
resend_pending_handshakes and resend_candidates (the latter also from the
LifecycleView trait, its only user). The resend unit test is re-expressed against
the machine + timer path, covering the due check and the record-on-success
semantics.
Known limitation: the first resend interval is armed from the machine's hardcoded
1000ms constant, which equals the config default. Under an operator override of
handshake_resend_interval_ms the first resend diverges from the pre-change
dial+interval; subsequent resends and the cap remain config-driven. Neutralizing
the first-resend override needs the interval threaded into the sync core (the
shell arm would be clobbered by the machine's own timer arming at dial), deferred
to when the connection and machine entities merge. Test count unchanged at 1631.
The control machine already emits SetTimer/CancelTimer actions when it arms the
handshake retransmit and timeout deadlines, but their executor arms were a
single no-op stub and the machine's Timeout event was never dispatched -- the
real work still runs on the legacy tick (check_timeouts,
resend_pending_handshakes).
Add the storage the time-as-input driver will read: a peer_timers map keyed by
LinkId then TimerKind, holding each armed timer's absolute deadline. The SetTimer
arm now inserts (overwrite = reschedule) and CancelTimer removes; the outbound
msg2 promote cancels the two dial-armed handshake timers, since the machine
survives promotion and the entries would otherwise linger in the store.
This store is a shadow: it is written and cleared but no driver reads it yet, so
behavior is unchanged. The legacy tick stays authoritative until the driver that
feeds Timeout and deletes the overlapping tick paths lands.
Route every machine removal through a new remove_peer_machine(link) choke-point
that drops the timer store alongside the machine, replacing the twelve direct
peer_machines.remove sites so no armed timer outlives its machine. TimerKind
gains Hash (to key the store) and Ord (for deterministic driver collection); it
is internal and never serialized. Test count unchanged at 1631.
Route the connection-oriented (TCP/Tor) outbound path through the peer state
machine, matching how the connectionless path already works.
initiate_connection's oriented branch now drives PeerEvent::Dial with
connection_oriented=true; the machine parks in Connecting and emits
OpenTransport, whose executor arm performs the non-blocking transport.connect
and pushes the PendingConnect. When the connect resolves, poll_pending_connects
prepares msg1 in the shell and then drives PeerEvent::TransportConnected, which
sends msg1 via the machine's SendHandshake arm.
The msg1 prepare (index allocation, Noise leaf, wire arming) MUST run in the
shell before the TransportConnected drive: send_stored_msg1 only transmits an
already-armed wire, so a drive-only path would silently send nothing. The
machine's our_index stays unset; the connect-failure path keeps its direct
handshake-timeout handling (TransportFailed stays dormant). The now-unused
Node::start_handshake helper is removed.
Behavior-neutral: same transport.connect, same PendingConnect, same msg1 send
and failure teardown as the removed inline path -- only the driver changes from
inline shell code to the state machine.
Move the peer_machines insert for an outbound leg out of the tail of
prepare_outbound_msg1 to a single shared site in initiate_connection, before
the connection-oriented / connectionless fork. This lets the connection-oriented
path find the machine at dial time (so it can be driven through the connect
handshake) without prepare_outbound_msg1 -- which for that path runs after the
connect completes -- clobbering an in-progress machine back to Discovered.
Because the machine now exists before the fallible dial steps, add
peer_machines.remove to every failure path in the widened dial window: the
index-allocation and Noise-leaf failures in prepare_outbound_msg1, the oriented
transport.connect failure, and both poll_pending_connects teardown arms
(handshake-start failure and async connect failure). remove_link does not touch
peer_machines, so these explicit removes are required.
Behavior-neutral: the connectionless path still drives the machine to
Handshaking and sends msg1 identically; the machine's our_index stays unset
(no spurious UnregisterDecryptSession on a later inbound restart); a failed dial
leaves peer_machines empty for that link exactly as before; and no
connection-oriented machine drive is wired here.
The connection-oriented (TCP/Tor) outbound connect->handshake path had no
`cargo test --lib` coverage; it was exercised only by the opt-in Tor
integration suites, and the TCP node tests bypass it via a manual
connectionless handshake helper. Add three unit tests:
- a machine-level test driving Dial{connection_oriented:true} -> Connecting
(emitting only OpenTransport, no msg1) -> TransportConnected ->
Handshaking{SentMsg1}, asserting the exact OpenTransport and
SendHandshake+SetTimer action vectors that start_outbound_handshake emits
(its oriented reach via on_transport_connected was previously untested; the
connectionless reach via on_dial was already covered);
- two node-level tests over a real loopback TcpTransport: a successful connect
that reaches start_handshake (observed via a pending_outbound entry), and a
connect to a closed port that routes through the failure arm (link torn down,
no msg1 dispatched).
Tests only; no production change.
Split the outbound handshake setup out of start_handshake into
prepare_outbound_msg1 (allocate the index, run the Noise leaf, frame and arm
msg1 -- the fallible steps, returning an error the caller propagates) and
send_stored_msg1 (transmit the armed wire). A connectionless dial now runs
prepare in the shell, then drives the control machine, whose SendHandshake
action sends the wire via the executor. Connection-oriented dials keep calling
start_handshake, now prepare followed by the send inline.
The dial event gains a connection_oriented flag so the machine's on_dial sends
msg1 immediately for connectionless transports (no connect step) instead of
opening a transport first. Behavior is unchanged: the same index, Noise leaf,
wire bytes, maps, and send-error handling as before, with index-allocation and
Noise failures still propagated synchronously before the send. A regression
test covers that a promote from the post-dial handshaking state is identical to
the former discovered-state promote.
Create and persist the per-peer control machine when an outbound handshake is
dialed, keyed by its link, instead of building a transient at msg2. The msg2
completion path now looks up that persisted machine to drive the promote,
falling back to a transient only if none is present (e.g. a direct-seeded test).
The machine parks in the Discovered state until promotion and is inert to the
liveness reap and rekey cadence while unpromoted, since it is absent from the
peers map. It is removed on every path that ends the outbound leg without
promoting -- the stale-connection reaper, the msg2 authorization-failure arm,
and the cross-connection resolution block -- mirroring the connection's own
lifetime so no dangling machine survives.
Its session index is deliberately left unset on the machine (the shell owns the
index on its connection), so a later inbound restart does not emit a spurious
decrypt-session unregister. A regression test covers that invariant.
Add a LostKind discriminator to the ReportLost action so the executor can
route an un-promoted handshake failure to the connected-guarded reconnect
reflex (note_handshake_timeout) and an established peer's link-death to the
unconditional one (note_link_dead), instead of collapsing every loss to
note_link_dead.
The two loss producers dispatched today, the liveness reap and the
inbound-restart-then-promote arm, both keep the link-dead routing, so
behavior is unchanged. The handshake-timeout and dial-failure producers are
tagged accordingly but stay dormant until their events are dispatched.
Remove the now-stale `#[allow(dead_code)]` on `advance_peer_machine`
and `execute_peer_actions`; both are live (called from the link-dead
reap, rekey cadence routing, and the handshake establish sites).
In `route_rekey_cadence`, reuse the ambient context timestamp for the
machine step instead of taking a second clock sample; the extra
sample's only consumer was the currently-inert cutover drain timer, so
the change is behavior-neutral.
Prove the per-peer machine's action type is a runtime-agnostic message
contract: it is Send + Sync + 'static, and every variant round-trips
unchanged through a single-threaded async channel (a current-thread runtime,
sender and receiver on one thread). A wildcard-free match over the variants
makes any future action a compile error here until it is added to the sample,
so a variant that embedded a runtime handle could not slip through unproven.
The two promote-failure warn! sites in the peer-action executor emit under
this module's own tracing target rather than the handshake target the rest of
the establish-path logging uses. Pin both (outbound "Failed to promote
connection" and inbound "Failed to promote inbound connection") to
fips::node::handlers::handshake, matching the sibling max-peers reject log and
keeping all establish-path diagnostics grouped under one target regardless of
which module physically emits them.
Comments across the per-peer machine, executor, lifecycle supervisor,
and peering reconciler carried internal rollout labels and design-note
section references. Rewrite them to describe the code's behavior
directly. Comment-text only; no code changes.
Route each link-dead peer that the tick sweep's plan_heartbeats decides
to reap through the per-peer machine and executor, replacing the inline
reap body in check_link_heartbeats. The batch decision, the liveness
snapshots (read from the hot-path-written receive clock), and the
heartbeat-send arm stay shell-side and byte-unchanged; the machine only
consumes the decided LinkDeadSuspected, tearing the peer down via
remove_active_peer and reporting the loss to the reconciler exactly as
before, on the same tick with the same wall-clock timestamp. The reap
log stays shell-side.
The machine's link-dead handler no longer emits a decrypt-session
unregister keyed by its shadow index (the full peer teardown already
unregisters the real index; the shadow could have drifted to a reused
index), and its guard now covers the Established state a freshly
promoted peer sits in.
Handshake-timeout, retransmit, and stale-connection cleanup stay inline:
they act on pre-promotion legs that have no machine, and the loss reflex
they use differs from the link-dead one.
Route each Cutover and Drain that the shell-side batch poll_rekey decides
through the per-peer machine and the executor, replacing the inline
effect bodies in check_rekey. The batch decision and its per-peer
snapshots stay shell-side and byte-unchanged: poll_rekey phase-groups all
cutovers, then all drains, then all initiations across the peer set, and
that ordering governs the shared index allocator's free-then-allocate
sequence that appears on the wire, so the machine only consumes the
already-decided actions (a new RekeyConsume event) without re-deciding.
InitiateRekey stays inline (its Noise msg1 build is a shell-side leaf)
with a RekeyInitiated observation feeding the machine so its control
state stays coherent for the next tick's cutover.
The cutover and drain logs, which relocated into the executor in the
prior commit, are pinned back to the fips::node::handlers::rekey tracing
target so they stay visible under the operator's module log filter.
Also clears the machine's shadow draining_index on drain so a later
cross-connection resolution cannot double-free the already-freed index.
Prepare the establish executor to drive FMP rekey by activating the
SwapSendState action (initiator K-bit cutover via cutover_to_new_session,
with the gated decrypt-worker re-registration) and adding a CompleteDrain
action (erase the drained previous session: free its index, drop its
peers_by_index entry, unregister its decrypt session). Both reproduce the
current inline rekey.rs cutover and drain bodies. The machine's drain
mapping now emits CompleteDrain, using the real drained index rather than
a shadow copy.
Unwired: nothing drives the machine's rekey path yet (live rekey still
runs inline), so these arms are unreachable and the change is
behavior-neutral. The cadence fold that routes cutover and drain through
the machine follows.
Move register_decrypt_worker_session out of promote_connection into the
executor's PromoteToActive handler, gated on a promoted or
cross-connection-won result. Every live promote now flows through that
one executor path, so registration still fires exactly once at the same
synchronous point; the direct test callers of promote_connection spawn
no worker pool, so the call was already a no-op for them.
Add the cross-connection loser-link teardown (close the losing
transport, remove its link, re-point addr_to_link at the winner) to the
executor as a guarded follow-up. It is unreachable on the current driven
establish paths, which only promote net-new peers, and asserts so, but
keeps the executor complete for when that case is driven.
Remove the now-dead drive_promote_to_active and ConnAction::PromoteToActive.
Cut the net-new outbound handshake completion (a received msg2 that
promotes a fresh outbound leg to a new peer) over to the per-peer
control machine, mirroring the inbound cutover. handle_msg2 still runs
the msg2 prologue, the ACL check, and the cross-connection swap/keep
arms inline; for the net-new promote it now builds a transient machine,
steps it, and drives promote_connection through the executor. The
session index was already allocated at dial, so there is no two-phase
authorize here.
The wire, index sequence, and peer registry state after promote are
byte-neutral. To keep the promote-failure path neutral, the executor's
cleanup now distinguishes inbound from outbound: an outbound promote
failure records the reject only, matching the prior handler, rather
than the inbound path's link and index teardown.
Cross-connection swap/keep, rekey-msg2, and the dial path stay inline;
the loser-link surgery for the currently unreachable driven
cross-connection case lands with the register relocation next.
Cut the restart handshake (an inbound msg1 from a peer that reconnected
with a new epoch) over to the per-peer control machine, completing the
inbound establish cutover. The old peer is torn down and the fresh leg
promotes through the same two-step authorize-then-allocate path as the
net-new case: the machine's first step emits the old-peer teardown
(invalidate send-state, report loss), the shell interposes the ACL
check, and the second step allocates the new index and sends msg2. The
old index is freed before the new one is allocated and the msg2 wire
bytes are unchanged, so the sequence stays byte-neutral.
With restart driven through the machine, the shared inline establish
tail that only the restart arm reached is deleted.
Also bounds the peer_machines map (remove_active_peer now drops the
peer's machine entry) and restores the msg2-send, promote, and
index-allocation failure warnings the cutover had dropped.
Cut the net-new inbound handshake (a fresh msg1 that promotes to a new
peer, plus the at-capacity reject) over from the inline handle_msg1 logic
to the per-peer control machine. handle_msg1 still classifies via
establish_inbound and still owns the Noise wire step, the late ACL check,
and the promote_connection registry surgery; for the net-new path it now
builds the machine, steps it, and executes the returned actions.
Authorization is interposed between two machine steps so the session
index is allocated only after the ACL check passes: a rejected or
unauthorized msg1 consumes no index, matching the prior order exactly.
The msg2 wire bytes, the index-allocation sequence, and the reject
metrics are all byte-neutral. Restart, resend, rekey-respond, and the
other reject arms stay inline unchanged; they move to the machine once
outbound establish is cut over and every promoted peer has a machine.
Also fills in the executor's send-failure and promote-failure cleanup so
a mid-establish error tears the leg down and frees its index exactly as
before.
Add Node.peer_machines (a LinkId-keyed map from the stable link handle to
the per-peer control machine) as the home for the machines, and a new
dataplane/peer_actions.rs holding execute_peer_actions / advance_peer_machine:
the executor that maps each PeerAction the machine emits to its shell call —
frame and send a handshake via build_msg2, drive promote_connection and feed
the PromotionResult back through the machine, tear a peer down via
remove_active_peer, free session indices, report loss via note_link_dead.
Actions for the rekey, connected-UDP, and timer paths are stubbed with notes
for the commits that fold those mechanisms in.
Unwired: nothing drives the machine yet — no live handler path calls the
executor and peer_machines is never populated — so this is behavior-neutral;
the inbound and outbound establish paths still run their existing inline
logic. The executor is cut over path-by-path in the following commits.
Draw the control/published-send-state boundary inside ActivePeer by
grouping the send-critical fields — the three epoch session slots
{current, previous, pending}, the K-bit flag and session-start, the
transport target, the connected-UDP handles, and the hot counters —
into a new co-located PeerSendState struct. The control-tier fields
(identity, connectivity, declaration/ancestry, filter and tree-announce
groups, remote_epoch, the rekey-negotiation sub-machine, and the rest)
stay on ActivePeer.
Behavior-neutral: a pure field regrouping. Every accessor signature is
unchanged (bodies now read/write self.send.*), so the hot path and the
handlers are byte-untouched; both K-bit cutovers still rotate the three
slots atomically with the same control-tier updates. No Arc/ArcSwap —
the fields are co-located and read by plain borrow; publishing behind a
shared cell is later plumbing for a sharded data plane.
Introduce src/peer/machine.rs: a sans-IO per-peer control FSM that
consolidates the scattered handshake/rekey/timeout driver logic now
spread across node/handlers. The machine is a pure reducer —
step(event, now, index_allocator) -> [action] — that reuses the
existing FMP decision cores (establish_inbound/establish_outbound/
cross_connection_winner/poll_*) rather than reimplementing any
decision, and returns runtime-agnostic actions the driver executes.
Control-tier state only; the published send-state boundary and the
driver wiring land in following commits. The machine is terminal at
Closed — re-dial is the reconciler's, so it holds no cross-attempt
retry state.
Includes eight unit tests: inbound and outbound establish, N:1
identity crystallization, the dual-initiation tie-break,
restart-override, rekey initiator cutover, the data-plane-owned
responder cutover boundary, and liveness -> link-dead -> report-lost.
Unwired — nothing calls it yet.
Replace the hand-rolled RawFd field and the unsafe Drop on
ConnectedPeerSocket with an OwnedFd, whose own drop glue closes the
fd. from_fd now stores the OwnedFd it already receives instead of
stripping ownership through into_raw_fd, and the manual libc::close is
gone, shrinking the unsafe surface.
Behavior-neutral: the fd still closes exactly once at last-Arc-drop and
as_raw_fd returns the same underlying fd while the socket is alive.
Wire exit detection for the four directly-observable optional children so
the supervisor FSM's ChildExited edge fires at runtime. A runtime
child-liveness mpsc channel carries a Child on exit: the DNS task and the
two TUN threads self-report when their body returns, and a 2s poll monitor
reports mDNS and Nostr via new is_finished accessors. The rx_loop gains a
select arm that steps the FSM and republishes health (Degraded, since a
running node always has at least one transport up). Transports and worker
pools expose no runtime-exit signal and are left for a follow-up.
Adds LanRendezvous::is_finished and NostrRendezvous::is_finished; the
latter treats a shutdown-taken connect_task as finished so the monitor
terminates after a stop instead of polling forever.
Add the sans-IO half of runtime child-liveness monitoring: a ChildExited
event and an on_child_exited handler that routes a runtime task or thread
exit the same way a start failure does. An optional child exiting degrades
the node (Degraded, with the child recorded in the health reasons); the
last transport exiting publishes Failed. There is no restart, and the
handler is inert outside Running (startup, drain, and teardown own their
own child bookkeeping via the pending/up sets). Failed here is a published
health signal only, not a teardown.
The start and runtime paths share the health classification, extracted
from resolve_start_health into classify_health.
The producer that emits the event (the exit-detection wiring) lands in a
following commit; the event variant carries a temporary allow(dead_code)
until then.
The overlay-discovery cutover to the sans-IO reconciler dropped the
operator-facing "open-discovery sweep complete" summary, including its
per-reason skip breakdown, because the sweep logic moved into the
log-free core. Have the core accumulate the enqueue/skip tally as it
reconciles and return it as data; the driver adds the two values only it
holds (the raw cache size and the self-advert filter) and emits the
summary, reproducing the old startup-vs-per-tick summarize gate and the
budget-zero debug path.
The self-advert precedence matches the pre-cutover sweep: a stale or
self-configured own advert is attributed to the age/configured buckets,
not skipped_self.
Behavior-neutral: the tally is pure side-counting; no dial or enqueue
decision changes.
Cut the last scattered peering mechanism — opportunistic growth from
transport-neighbor beacons and LAN mDNS — over to the reconciler's
opportunistic layer via a gate-checked reconcile_opportunistic wrapper,
one call per tick slot (transport and LAN stay separate slots so their
per-tick budget and per-peer cap are not shared). The driver keeps the
beacon/mDNS I/O and the path-granular prefilters that read live state
(self, fresh-enough-to-skip, connecting-on-path) and executes the emitted
Connect intents; the core owns the connected/budget/per-peer-cap
decisions. A first-wins per-peer dedup on the LAN path reproduces the old
inline once-per-peer dial now that the snapshot core cannot observe the
intra-tick connecting feedback.
Delete the now-dead discovery_connect_budget helper. With this, all three
scattered peering mechanisms (auto-connect and retry, overlay discovery,
neighbor growth) are unified in one sans-IO reconciler. Behavior-neutral;
unit test count unchanged (1607).
Cut the overlay (Nostr open-discovery) enqueue over to the sans-IO
reconciler. A gate-checked reconcile_overlay wrapper runs the overlay
layer alone at the discovery tick slot; the monolithic reconcile would
re-fire the always-on retry-dial and double the per-tick dial cap, so the
overlay slot must call only its layer. The driver builds the candidate
pool from the overlay advert cache (self excluded), the cooldown and
configured-npub sets, and the startup/steady max-age, then feeds the
reconciler; the emitted enqueue set drives the per-enqueue identity-cache
and alias pre-seed exactly as before. Enqueued entries are dialed at the
retry slot, preserving the two-phase cadence.
Delete the now-dead enqueue-budget and expiry helpers. Behavior-neutral:
same candidates enqueued in the same order with the same budget, cap, and
expiry. Opportunistic transport-neighbor growth remains imperative and is
cut over next. Unit test count unchanged (1607).
Route the auto-connect floor and the connection-retry mechanism through
the sans-IO reconciler instead of the imperative Node methods. A thin
peering driver (note_handshake_timeout / note_link_dead) centralizes the
reflex path with the gate guard and the already-connected check; the
per-tick retry-dial and the startup floor build reconciler inputs and
execute the emitted Connect intents. The three imperative retry methods
are removed and their call sites rerouted.
Wire the drain and startup gates. Entering a bounded drain now clears the
retry schedule and suppresses the peer-loss reconnect reflex (the drain
window runs under the Suspended gate), so the drain no longer fights a
reconnect for the peers it just closed. The startup floor runs under an
explicit Reconciling gate at the existing peer-connect seam, preserving
the current dial position.
Behavior-neutral except the intended drain change. Overlay discovery and
opportunistic transport-neighbor growth remain imperative for now; they
observe the relocated retry schedule and are cut over next. Unit tests
migrated to the driver API with assertions unchanged (1607).
Move retry_pending and pending_connects from Node into the Peering owner
struct (Node.peering), the home introduced with the reconciler core.
Pure mechanical relocation: the two collections now live behind
self.peering, with roughly seventy accessor sites repointed. No decision
logic is touched and the unit-test count is unchanged (1607). The
imperative peering methods still run; the reconciler is wired in the
following commit.
Introduce PeeringReconciler, the synchronous decision core for peer
desired-state, with its input/action vocabulary (Gate, Budget, Observed,
Candidate, DiscoveryPools, Policy, PeeringAction) and the Peering owner
struct. reconcile() computes connect/retry intents across four layers:
the auto-connect mandatory floor, the ceiling-only overlay enqueue, the
opportunistic transport-neighbor growth, and the node.limits ceiling
enforced inline. It mirrors the supervisor sans-IO shape: no I/O, no
clock reads (time enters as a parameter), no runtime handles.
The core owns its own cross-attempt retry schedule so escalating backoff
survives the fresh connection created per re-dial. Gate-guarded reflexes
(on_handshake_timeout / on_link_dead) reproduce the current backoff math
and no-op while draining. Overlay is strictly ceiling-only (no peer-count
set-point). Disconnect is defined but never emitted (no shedding).
Unwired here: the driver still runs the imperative peering methods and
Node holds the live retry map. 11 unit tests drive the pure core with
synthetic inputs. No behavior change.
Homing the retry schedule under peering/ lengthened the RetryState
module path, pushing the retry_state_iter return-type line past the
100-column max_width. Rewrap to satisfy rustfmt. No behavior change.
Create src/node/peering/ as the home for the peer desired-state
(homeostatic reconciler) concept and move the cross-attempt connection
retry schedule into it: RetryState plus schedule_retry /
schedule_reconnect / process_pending_retries.
Mechanical relocation only. The methods remain inherent on Node; the
retry_count must persist across re-dials (a fresh connection is created
each attempt), which is why the schedule belongs in the peering home
rather than on a per-connection type. The one-level-deeper module path
requires pub(super) -> pub(in crate::node) to preserve the prior scope,
plus module-path fixups at the reference sites. No behavior change; unit
test count unchanged (1596).
Determine node health at start completion instead of unconditionally
reaching Running. Zero transports up is now Failed (fatal): start()
tears down cleanly and returns an error, and the daemon exits. Any
configured optional child that failed to start - a transport beyond the
first, Nostr, mDNS, TUN, DNS, or a worker pool - leaves the node
Degraded but serving, with an operator warning naming what failed. All
configured children up is Full. A child the node was never asked to run
does not count against health.
The published NodeState gains Degraded and Failed variants, both visible
via control queries; Degraded is operational, Failed is not. The
lifecycle FSM gains the health states plus the PublishState action that
drives them - a health fork cannot be a single direct state write, which
is why the earlier commits deferred it to here.
Runtime child-exit health re-evaluation (a running child dying) is a
separate liveness-monitoring mechanism left for a follow-up; this commit
is start-time health only.
Add an operator-visible Draining phase on daemon shutdown. On the
shutdown signal the node broadcasts Disconnect to all peers, then keeps
serving for a bounded window - up to node.drain_timeout_secs (default 2s),
exiting early once all peers are gone - before tearing down. This lets
in-flight traffic settle and peers observe the disconnect before the
transports close, rather than the previous immediate teardown.
The lifecycle FSM gains a Draining state plus Drain/DrainDeadlineElapsed
events; the run loop observes the shutdown signal and transitions to
draining in place - one continuous loop, so the channel receivers are
never destructively cancelled. The published NodeState gains a Draining
variant, visible via control queries during the window. The immediate
stop() path used by tests and non-daemon callers is unchanged: it still
tears down immediately with no drain wait.
The reconciler-gate actions the drain emits are no-ops until the peering
reconciler lands and consumes them.
Introduce a sans-IO lifecycle supervisor: a synchronous step(event) ->
[action] state machine (SupervisorFsm) that authors the substrate-child
spawn and teardown order, plus an owner struct (Supervisor) holding the
substrate runtime fields and embedding the FSM. The substrate-lifecycle
fields leave Node's flat list into the owner - state, packet_tx, the TUN
reader/writer handles + shutdown fd + channels, the DNS task + identity
channel, the Nostr/LAN rendezvous drivers, and the encrypt/decrypt worker
pools; the dataplane keeps packet_rx.
start()/stop() become the driver executing the FSM's SpawnChild/StopChild
actions: same children, same order, same warn/debug-and-continue on
optional failures, same logs, same NodeState transitions. Behavior- and
wire-neutral. The only determinism change is that transports now tear
down in ascending-id order, previously nondeterministic HashMap iteration.
The bounded Draining phase and the Running{Full|Degraded} health split
land as separate follow-on commits.
Reorganize the node module tree by concept rather than by
message-handling verb, as the first step of the node runtime
decomposition. Pure relocation: no wire, config, metric, or log
semantics change; the lib test count is unchanged (1577 passed).
Moves (git mv, 100% rename similarity):
- handlers/{forwarding,rx_loop,connected_udp,dispatch,encrypted}.rs
-> node/dataplane/ — the whole RX hot path (the select! run loop,
transit/local forwarding, the link-message router, the RX decrypt
path with responder K-bit cutover + roam writes, and connected-UDP
fast-path activation) now lives in one home.
- node/session.rs -> node/session/mod.rs — establishes the session
concept home for the data/state types. The message-behavior file
handlers/session.rs stays put for now (folds in with the later FSP
session step).
The IK/XX-divergent establishment files (handlers/{handshake,rekey,
timeout}.rs) and the deferred-home files (handlers/{mmp,lookup}.rs)
deliberately stay in handlers/, to move once rather than twice.
Every module is reached through impl Node methods, so no call site or
re-export shim was needed. Updated in lockstep with the moves: the
module_path!-derived tracing targets in the two mesh-lab compose-trace
overlays, a structural test's include_str! source path, doc-comments
in proto/routing and the mesh-lab docs, and the stale source-location
citations (node/handlers/{forwarding,rx_loop,encrypted}.rs and
node/session.rs) in doc-comments and the discovery design doc.
Forward-merge three maint fixes, hand-relocated into master's
post-sans-IO / discovery-to-lookup structure:
- drop the redundant TreeMetrics parent_switched counter (keep
parent_switches), reconciled across the refactored tree/mmp sites and
the spanning-tree test reads
- keep the tighter path_mtu when applying a LookupResponse, now in
handlers/lookup.rs after the discovery module rename
- reuse one shared secp256k1 context in the identity module
Every sign/verify/key-derive site built a fresh context via
Secp256k1::new(), which allocates a Secp256k1<All> and runs
randomization/blinding table setup on each call. Introduce one
crate-wide LazyLock<Secp256k1<All>> and reuse it across the local, peer,
and auth sites (and their tests). Behavior-neutral: identical secp256k1
API calls, only the context lifetime changes, and the shared All context
still performs the standard construction-time blinding.
An originator handling a LookupResponse unconditionally overwrote the
cached path_mtu_lookup entry, so a looser (larger) estimate in a later
response could clobber a tighter value already learned from a reactive
MtuExceeded or PathMtuNotification. Read-and-compare before writing and
keep the minimum, so a looser discovery estimate no longer loosens the
clamp. Add a regression test.
parent_switched was incremented on the line immediately before
parent_switches at every site and never independently, so the two
counters were always identical. Drop parent_switched from TreeMetrics,
its snapshot, TreeStatsSnapshot, the show_tree fixture, and the fipstop
render, keeping parent_switches as the sole counter.
The routing-stats pane's Discovery Requests/Responses sections show the FMP
overlay coordinate-lookup counters. Rename the section labels and the nested
JSON key they read from "discovery" to "lookup" to match the metric family's
canonical name. The daemon dual-emits both keys, so this reads the current
name and no longer depends on the deprecated "discovery" alias.
Rename the Ethernet per-interface config flag from discovery to listen,
so the receive/transmit toggle pair reads as the symmetric announce
(transmit) / listen (receive) neighbor-beacon vocabulary. The old
discovery: key is still accepted via a serde alias, so deployed configs
load unchanged; to_yaml re-emits it under the canonical listen: name.
Marked deprecated for removal at the v2 cutover.
Updates the config field + accessor, the transport listen_enabled local,
the one struct-literal test consumer, the chaos sim config generator,
packaged fips.yaml examples, and the classified operator-facing docs
(ethernet neighbor-beacon subsystem prose; the generic Transport
discovery capability prose is left unchanged). Adds a compat parse test
asserting the legacy alias, the new key, and that deny_unknown_fields
still rejects unknown keys. Behavior-neutral.
Mirror the ethernet neighbor rename in the BLE transport: the internal
peer-detection buffer becomes NeighborBuffer (from DiscoveryBuffer) and
the module is renamed discovery -> neighbor. The BlueR/bluez API terms
(DiscoveryFilter, set_discovery_filter) and the BLE advertise/scan
mechanism vocabulary are unchanged, as is BleConfig. Behavior-neutral.
Rename the ethernet link-local neighbor-beacon subsystem from the
overloaded "discovery" vocabulary to a neighbor umbrella. The beacon
frame vocabulary is retained (build_beacon/parse_beacon/BEACON_SIZE/
FRAME_TYPE_BEACON/FRAME_TYPE_DATA); only the subsystem and buffer
identifiers change: DiscoveryBuffer becomes NeighborBuffer,
discovery_buffer becomes neighbor_buffer, and DISCOVERY_VERSION becomes
BEACON_VERSION (wire value 0x01 unchanged). Config toggles are left for
a follow-up commit. Behavior-neutral.
The module is gated target_os="macos" (not the broader Darwin/iOS family),
so the name now tracks the cfg and matches the *_macos.rs file convention
(io_macos.rs). Drops the redundant inner #![cfg(target_os="macos")] (the
decl gate already covers it) and corrects a stale top comment that claimed
the module stays visible on Linux — it is macos-decl-gated, so it never was.
Mechanical rename; no logic change.
ConnectedPeerSocket and PeerRecvDrain are node/peer-side logic: the Transport
trait never touches them, ActivePeer stores them, and node's encrypt worker
drives them. They only happened to live under transport/udp. Relocate the
handle types to a new src/peer/connected_udp/ module (socket.rs + drain.rs),
which the node handler and encrypt worker reach via node -> peer (no new edge;
a node home would have forced a peer -> node cycle).
The udp transport keeps only the kernel-construction seam: open_connected_fd,
now folded into udp/io.rs (the byte-layer home) behind a linux/macos-gated
submodule and re-exported as transport::udp::open_connected_fd. The node
handler builds the fd through it and adopts it via ConnectedPeerSocket::from_fd.
Behavior-neutral: no wire/config/metric/log change; the per-packet hot path
(bare-RawFd send_batch_gso/raw) is untouched. Preserves the Arc multi-owner
contract, the drop-drain-before-socket ordering, and the drain's detach-on-Drop
deadlock avoidance. Reconciles the old cfg(unix)/any(linux,macos) double-gate
onto the single any(linux,macos) predicate.
Pulls the connected-UDP socket construction (socket/REUSEADDR/REUSEPORT/
BUFFORCE/bind/connect + darwin tuning) out of ConnectedPeerSocket::open into
a pub(crate) open_connected_fd returning an OwnedFd; open() now delegates and
adopts the fd. Error paths still close the fd via the temporary's Drop; the
success tail transfers ownership through OwnedFd. Behavior identical. Prepares
the fast-path handle types to move to the peer module while the socket
construction stays with the udp transport.
Moves the inline TcpConnection/ConnectingEntry structs, the Direction
enum, and the ConnectionPool/ConnectingPool type aliases out of the
1147-line tcp/mod.rs into a dedicated pool.rs, matching the canonical
per-transport layout. Struct fields are pub(crate) so mod.rs can still
construct and read them across the module boundary. Pure relocation; no
logic change.
Moves parse_mac_string into ethernet/addr.rs and re-exports it at the
ethernet module root so the ethernet::parse_mac_string path stays
byte-identical for its one external consumer (zero churn there). Its
unit tests stay in mod.rs, calling through the re-export. Pure
relocation; no logic change.
Normalizes the ethernet transport onto the canonical byte-layer name
(io.rs), including the per-OS files io_linux.rs/io_macos.rs and their
#[path] wiring. Pure file rename plus module-path and doc-comment
updates; no logic, wire, config, metric, or log change.
Normalizes the udp transport onto the canonical per-transport module
layout (ble is the reference: io/pool/addr/stats). Pure file rename plus
module-path updates at the use sites; no logic, wire, config, metric, or
log change. Behavior identical.
The Nym transport was a near-clone of the Tor transport's outbound
SOCKS5 path. Extract the shared logic into src/transport/socks5/ so both
transports drive one implementation instead of two maintained copies:
- share one SOCKS5 mock server between the tor and nym tests
- extract the common send/receive/connect counters into a shared
ProxiedStatsBase (snapshot structs and emitted metrics unchanged)
- add a shared Socks5Dialer that collapses all six connect variants;
the sole dialing difference (tor's per-destination circuit-isolation
auth vs nym's no-auth) is an enum on the dialer
- route both tor and nym dialing through the shared dialer
- share the proxied connection pool, generic over a per-connection meta
type that carries tor's inbound/outbound direction counting (nym uses
the unit type)
- share the proxied receive loop
Behavior-neutral: no wire-format, config-key, emitted-metric, log, or
error-variant change. Tor keeps its control-port, inbound/onion, and
directory surface and its own address validation.
The FMP frame-boundary reader lived in transport/tcp/stream.rs, but the
Tor and Nym transports both reached across module boundaries to
'use crate::transport::tcp::stream::read_fmp_packet', a layering smell:
the reader is a shared stream-framing utility, not a TCP-private one.
Move the file to transport/framing.rs and repoint the tcp/tor/nym use
sites, removing the tor->tcp and nym->tcp dependencies. Behavior
unchanged.
TcpStats and TorStats each carried an identical pair of pool_inbound/
pool_outbound atomics plus the same five record_pool_* / pool_inbound_count
methods over them. Move the counter logic into a shared PoolCounters
struct (new transport/stats_common.rs) embedded as a 'pool' field in both.
The public record_pool_* methods stay as thin delegators so all call
sites and the flat pool_inbound/pool_outbound snapshot fields are
unchanged; only the duplicated atomic bookkeeping is now single-sourced.
EthernetStatsSnapshot derived only Clone/Debug/Default, unlike every
other transport's *StatsSnapshot which also derives Serialize. That gap
forced a hand-rolled serde_json::json!{} arm in TransportHandle::
transport_stats() that re-listed all ten fields by hand. Add the
Serialize derive and collapse the arm to the same one-line
serde_json::to_value(...) form the other transports use. The emitted
JSON keys and values are unchanged.
Measures the per-forwarded-packet cost of routing candidate assembly
(routing_candidates over a synthetic RoutingView) against a zero-alloc
reference across 8/32/128/256 peers, with per-call allocation counts via
a counting allocator. Criterion harness; no production code change.
Forward-merges the bloom SHA-256-once fix from maint. The bloom filter was
relocated to proto/bloom/ by the sans-IO refactor, so the fix applied cleanly
to proto/bloom/core.rs; the behavior-neutral test was re-homed into
proto/bloom/tests/core.rs (maint carried it in the pre-split bloom/tests.rs).
BloomFilter derived all k hash functions from one SHA-256 digest but
recomputed that digest inside the per-function loop, so every insert and
contains ran SHA-256 hash_count times (5x at the default) over the same
bytes. Hoist the digest out of the loop: base_hashes() computes it once
and returns (h1, h2), and bit_index() derives each of the k indices with
the same (h1 + k*h2) mod m arithmetic. Bit-for-bit identical output; this
is the hottest path in packet forwarding and mesh-size estimation.
Adds a test pinning the bit indices against the double-hashing formula
recomputed independently, proving the refactor is behavior-neutral.
Move the PromotionResult enum (and its impl) out of peer::mod and into
proto/fmp/core.rs, alongside the cross_connection_winner tie-break helper
that was relocated the same way. This is FMP connection-lifecycle result
vocabulary, so it belongs in the FMP subsystem home rather than the peer
module.
Behavior-neutral pure type relocation: consumers import it from
crate::proto::fmp, and the crate-root public path crate::PromotionResult
is preserved via a re-export in lib.rs (mirroring cross_connection_winner).
Full lib suite green at baseline.
Move the FMP mesh-layer wire format (common prefix, encrypted/msg1/msg2
headers, and the build_*/inner-header codec fns) out of node/wire.rs and
into proto/fmp/wire.rs, so the whole FMP wire surface lives with its
subsystem, matching the proto/fsp/wire.rs layout. The wire module becomes
pub(crate) mod wire; callers reach it via crate::proto::fmp::wire.
Behavior-neutral: pure relocation plus import-path rewrites across the
node/peer consumers; no logic change. Full lib suite green at baseline.
Carve the nostr rendezvous engine's decision logic out of the async
NostrRendezvous driver into synchronous, clock-injected cores, following
the failure_state.rs pattern: sync state behind std::sync::Mutex, time
passed in as now_ms, no .await and no I/O in the core. The driver performs
all relay/socket I/O and executes the returned plans.
- AdvertMachine (src/nostr/advert.rs): advert publish/cache/fetch/prune
decision logic; the driver executes a returned PublishPlan.
- TraversalMachine (src/nostr/traversal_machine.rs): the engine-scoped
cross-session decision state -- in-flight-initiator dedup, the dual-init
responder suppression election, and the replay/seen-session cache.
- classify_punch_packet (src/nostr/traversal.rs): a pure classifier for
the punch recv loop's packet branch table.
The driver keeps all I/O, the NAT punch send cadence, the offer-slot
admission semaphore, and the pending-answer oneshot routing. Behavior-
neutral: no wire, config-key, or metric change. Adds unit coverage for
the advert plans, election ordering, replay eviction, initiator dedup,
and punch classification.
Moving the nostr rendezvous engine to src/nostr/ (and mDNS to src/mdns/)
changed the module-path-derived tracing targets from
fips::discovery::nostr::* to fips::nostr::*. Update the RUST_LOG filters
in the NAT and mesh-lab test compose files and the resolve-peers-via-nostr
tutorial's debug recipe to the new targets so trace configs and the
documented journal-watch command keep emitting the intended lines.
Without this, the stun-faults suite's Phase-0 pre-flight (which greps the
daemon journal for the debug-level "STUN observation succeeded" /
"traversal: initiator STUN observed" lines) saw those lines suppressed —
the daemon behaved correctly, but the stale RUST_LOG target hid the
evidence. Test-harness and docs only; no source or behavior change.
Pull the rendezvous driver state and the movable driver logic out of the
Node struct into the src/nostr home. A new RendezvousDriver owns the
engine handle and the four bookkeeping fields (traversal start time,
startup-sweep latch, adopted bootstrap-transport set and their npubs)
that previously sat loose on Node; Node holds a single driver field and
reaches the same data through thin accessors, including the rx-loop
hot-path protocol-mismatch hook.
The advert build/refresh, the via_nostr fallback-address resolve, the
overlay-endpoint-to-PeerAddress mapping, and the bootstrap request move
onto the driver, taking their Node inputs explicitly (a transport-
endpoint snapshot for advert building) rather than reading Node fields
directly. Transport/connection-table-bound work (traversal adoption,
bootstrap-transport cleanup, the open-discovery sweep, and the outbound
budget calculators) stays on Node as thin glue that calls into the
driver.
Behavior-neutral relocation: statements moved verbatim, no logic, wire,
config-key, or metric changes. lifecycle.rs shrinks ~230 lines. cargo
fmt/build/clippy clean; lib suite 1547 passing (baseline unchanged).
Relocate the overlay peer-rendezvous subsystem out of the overloaded
src/discovery/ tree into two focused, independent homes: src/nostr/
(relay-mediated overlay endpoint advertise/resolve/auto-mesh plus NAT
traversal) and src/mdns/ (link-local DNS-SD rendezvous). The two
subsystems are independent, so they get separate homes rather than
sharing one.
Drop the ambiguous "Discovery" stem from their identifiers in favor of
"Rendezvous": NostrDiscovery -> NostrRendezvous, LanDiscovery ->
LanRendezvous, and the matching config, policy, field, and method names.
The former src/discovery.rs handoff types (EstablishedTraversal,
BootstrapHandoffResult, the punch-packet helpers) fold into
src/nostr/handoff and stay reachable via the crate-root re-exports.
Pure relocation and rename: no logic, wire-format, config-key, metric,
or tracing-target changes. The operator-facing node.rendezvous.nostr.*
and node.rendezvous.lan.* config keys and the fips-overlay-v1 advert
namespace are byte-identical. cargo fmt/build/clippy clean; lib test
suite 1547 passing (baseline unchanged).
The identifier "discovery" named three unrelated subsystems; the FMP
overlay coordinate-lookup subsystem is now consistently "lookup". This
finishes the concept-#1 rename across the shell, config, and metric
layers left after the earlier proto-layer rename:
- Handler module node::handlers::discovery -> node::handlers::lookup, and
reset_discovery_backoff -> reset_lookup_backoff.
- The Node lookup-engine field Node.discovery -> Node.lookup, renamed by
resolved binding so the metrics().discovery and node.discovery config
paths are left untouched.
- The lookup metric types DiscoveryMetrics -> LookupMetrics,
DiscoveryStatsSnapshot -> LookupStatsSnapshot, and Metrics.discovery ->
Metrics.lookup.
Two surfaces cross a stability boundary and ship behind a compatibility
window, both marked in-code for removal at the v2 cutover:
- The control-socket metric family is dual-emitted under both "discovery"
(deprecated alias) and "lookup" so existing dashboards keep working.
- The node.discovery.* config table is split into node.lookup.* (mesh
lookup scalars) and node.rendezvous.* (nostr/lan peer rendezvous).
NodeConfig does not deny unknown fields, so a naive rename would make a
deployed node.discovery: block deserialize into nothing and silently
revert every setting to default. A deprecated all-Option
DiscoveryConfigCompat field captures a legacy block and a new post-parse
Config::normalize_deprecated_keys pass folds it into the new tables with
a one-time deprecation warning.
Flip the packaged fips.yaml templates to the new keys, add legacy/new/
scalar compat parse tests, and record the split and deprecations in the
CHANGELOG. Behavior-neutral; fmt/clippy clean, lib suite green.
The previous portability fix bounded the powi test's drift from std::powi
at an absolute ULP count, but Windows/MSVC drift grows with the exponent
(3 ULP by exp=14), so no fixed ULP bound is portable.
Replace the ULP oracle with a loose relative-tolerance sanity sweep
(1e-11), which any sane libm clears by ~1000x regardless of exponent
while still catching a grossly wrong impl. Add a golden-bit pin for the
exp=14 case that exposed the growth. The powi function is unchanged; the
golden-bit determinism pins already passed on Windows.
The powi guard test asserted our square-and-multiply result was
bit-for-bit equal to f64::powi, which failed the Windows unit-test job by
one ULP. f64::powi is not portably bit-stable: Linux and macOS lower it
to compiler-rt's __powidf2, but Windows/MSVC rounds differently.
Our square-and-multiply is pure IEEE-754 f64 multiplication and is
therefore deterministic across every platform, which is the property that
actually matters for mesh nodes to agree on bloom FPR and backoff timing
regardless of OS. Replace the std bit-equality assertion with golden-bit
pins on our own output (locking that cross-platform determinism) plus a
bounded-drift sanity sweep against std::powi. The powi function itself is
unchanged.
Rename src/proto/discovery to src/proto/lookup and bring the module's naming
onto the lookup stem, matching its concept: a mesh lookup of a node's
coordinates from its pubkey, sent into the mesh as a bloom-filter-guided
multicast request that returns a unicast response with the coordinates.
- the module directory and its declaration (proto::discovery -> proto::lookup)
- exported types: DiscoveryAction -> LookupAction, DiscoveryBackoff ->
LookupBackoff, DiscoveryForwardRateLimiter -> LookupForwardRateLimiter,
MAX_RECENT_DISCOVERY_REQUESTS -> MAX_RECENT_LOOKUP_REQUESTS, and the Discovery
state struct -> Lookup
- internal terminology: doc comments, the "overlay-lookup" phrasing, the
empty_discovery/suppressing_discovery test helpers, and the disc
parameter/variable all take the lookup names
- the already-lookup-named wire types (LookupRequest/LookupResponse) are unchanged
Behavior-neutral: no wire bytes or decision logic change. Two references are
intentionally kept as "discovery": the still-named node::handlers::discovery
shell module and the node.discovery.* config keys, which belong to the broader
disambiguation of the shell, config, and metric surfaces still to come.
Bring the proto tree closer to a std+alloc-only posture, behavior unchanged on
every decision path:
- Sweep the remaining std::fmt and std::collections imports over to core::fmt
and alloc::collections across the wire codecs and their tests. Subsystem files
that shadow the core name with a child core module use the leading-colon
::core::fmt form. Imports only.
- Replace the two f64::powi calls (bloom false-positive-rate, FMP backoff timer)
with a shared core-only square-and-multiply helper in proto/math, bit-identical
to std::powi (a guard test pins this against every exponent the codecs reach),
and route the diagnostic estimated-count natural log through libm::log. The FPR
reject decision and the backoff timer are bit-for-bit unchanged; only the
debug-only count estimate may differ by at most one ULP.
Introduce a shared proto/codec module with a cursor Reader (short reads fail with
MessageTooShort { expected: position + needed, got: total }) and an append Writer,
and adopt them across the seven subsystem wire codecs, replacing the repetitive
manual slicing, try_into, and from_le_bytes extraction. Each existing length check
maps to the reader (an up-front minimum becomes require at position zero, so the
per-field expected values, including the tree-announce expected 99, are reproduced
exactly); the bloom exact-length check stays explicit since it also rejects
over-long payloads. Encoded bytes and decode decisions are unchanged.
Replace Malformed(String) with a no_std-clean set: Malformed(&static str)
for the static decode diagnostics, BadSizeClass { got, max } for the two
bloom size-class checks, and typed sources BadCoord(CoordError) /
BadBloom(BloomError) for the coordinate and bloom construction failures. Add a
dedicated CoordError so proto/coord no longer depends upward on stp TreeError,
resolving the temporary inversion from the coordinate relocation. Drop the
thiserror derive from the four proto error types (Error, TreeError, BloomError,
CoordError) in favor of hand-rolled core::fmt::Display and core::error::Error
impls. thiserror remains in use elsewhere in the crate. Wire decode decisions
are unchanged; only the error type and its diagnostic text change.
Hoist the two line-for-line-identical per-destination minimum-interval limiters
(discovery forward, routing error) into a shared PerAddrRateLimiter, and fold
the duplicated exponential-backoff math (discovery originator, FMP retry) into a
shared backoff_ms helper, both in a new proto/rate_limit module. The subsystem
limiters become thin delegating newtypes so should_forward/should_send and the
backoff call conventions are preserved. Behavior is byte-identical.
Two behavior-neutral relocations, wire bytes unchanged:
- Move the ParentDeclaration type and its impls out of state.rs into a dedicated
declaration.rs, re-exported at the same path, and relocate the inline wire.rs
test module into tests/wire.rs alongside the other stp unit tests (reusing the
shared test helpers). wire.rs drops to non-test code only.
- Move TreeCoordinate, CoordEntry, and the coordinate wire codec out of
proto/stp into a shared proto/coord module (a peer of proto/link), re-exported
from proto::stp so every existing import path keeps resolving unchanged.
Coordinate is a shared addressing primitive with many non-stp consumers, so it
no longer belongs under the spanning-tree subsystem. The codec carries a
documented temporary dependency on stp::TreeError until a dedicated CoordError
is introduced.
evaluate_parent no longer reads the injected clock: it returns a ParentEval of
Mandatory, Discretionary, or None, and the flap/hold-down veto is applied at the
edge via a new TreeState::is_switch_suppressed(now_ms), gating only the
discretionary arm. classify_announce/classify_periodic take the pre-computed
switch_suppressed bool instead of now_ms, so the whole classify ladder is
clock-free; the shell callers (tree announce/periodic re-eval, MMP first-RTT
re-eval, handle_parent_lost) compute the veto verdict. The no-coords parent
case stays discretionary (veto-gated) exactly as before. Behavior unchanged;
the veto tests now assert the moved responsibility.
Reorganize the MMP subsystem, behavior unchanged throughout:
- Restore the pre-migration role split: move SenderState into sender.rs,
ReceiverState (with its GapTracker helper) into receiver.rs, MmpMetrics and
RrLog into metrics.rs, and PathMtuState into path_mtu.rs, leaving the Mmp
aggregate plus the peer/session state in state.rs (1339 to 196 lines). Home
the module constants in a new limits.rs and re-export them at the same paths.
Pure code-motion with module rewiring; visibility unchanged.
- Dissolve the 97-line src/mmp shell, which held only MmpConfig and the
monotonic mono_ms() clock: move MmpConfig into config/node.rs (re-exported as
crate::config::MmpConfig), move mono_ms() into a new top-level src/time.rs
shell time seam, repoint all callers, and delete src/mmp/. Also correct stale
src/mmp/ doc-comment path labels to proto/mmp/ where they name the protocol
primitives.
Two behavior-neutral discovery cleanups:
- Move MAX_RECENT_DISCOVERY_REQUESTS out of the node discovery handler into the
discovery subsystem limits module and re-export it, keeping the two use sites
unchanged. Same value and semantics; only its home moves.
- Remove the ambient rand draw from LookupRequest by deleting generate() and
having the shell draw the random request_id and pass it into the existing
new() constructor. Same per-request u64 draw, now at the shell, leaving the
discovery codec free of ambient RNG reads.
The false-positive-rate primitive (fill ratio raised to the hash count) was
inlined at three sites. Hoist it into a BloomFilter::fpr() method and call it
from all three; the threshold/policy logic stays at the call sites. Behavior
and result bytes are unchanged.
Migrate the FSP end-to-end session subsystem into src/proto/fsp/ following the
established sans-IO shape, and retire the src/protocol grab-bag now that FSP was
its last occupant.
Relocate the FSP session wire (node/session_wire.rs plus the FSP message types
from protocol/session.rs) into proto/fsp/wire.rs. Hoist the pure decision logic
into proto/fsp/core.rs over plain-data SessionSnapshots returning an ordered
FspAction list the shell drives: session-rekey policy, msg3-resend
classification, post-decrypt epoch reaction, setup/dual-init tie-break,
coords/path-MTU emit-policy, bounded pending-queue, and IPv6 ECN. The
crypto-owning SessionEntry stays shell-side in node/session.rs (matching the FMP
ActivePeer pattern); proto/fsp is wire + core + limits only, with no proto->noise
dependency and no crypto.
Move the coords helpers to proto/stp/ (they serialize TreeCoordinate), and split
SessionMessageType: the encrypted-inner 0x10-0x1F variants stay in proto/fsp/wire.rs
while the 0x20-0x2F routing signals become a new RoutingSignalType in
proto/routing/wire.rs. Migrate the session-MMP shell adapter, which continues to
drive proto/mmp/.
Retire src/protocol: LinkMessageType and SessionDatagram move to a new shared
proto/link.rs, ProtocolError becomes proto::Error (relocated verbatim), the
deprecated MessageType alias and the unimported PROTOCOL_VERSION are dropped, and
src/protocol/ is deleted along with its lib.rs module declaration.
Behavior-neutral: wire bytes unchanged, oracle tests pass unedited except
mod-path relocation; adds rekey/epoch characterization tests and pure
poll/emit-policy core tests.
Migrate the v1 bloom filter subsystem into src/proto/bloom/, matching the
discovery/routing/fmp/mmp/stp reference layout and completing the proto/
relocation series for the data/wire subsystems.
- wire.rs: the FilterAnnounce (0x20) codec, moved from protocol/filter.rs
- core.rs: the pure BloomFilter algorithm (hash/insert/contains/merge/
as_bytes/from_bytes/estimated_count), moved from bloom/filter.rs
- state.rs: BloomState (per-peer inbound store, compute_outgoing_filter,
the injected-clock send debounce), moved from bloom/state.rs
- limits.rs: the v1 sizing constants
- mod.rs: module wiring + the BloomError enum
- tests/: the unit suite split by target (core/state/wire), no inline tests
no_std+alloc hygiene: core::fmt over std::fmt, the tracing dependency
dropped from the pure filter, and std collections replaced with
BTreeMap/BTreeSet (NodeAddr: Ord) for deterministic iteration. The pure
filter combination stays a BloomState method; the two irreducible shell
gathers (peer_inbound_filters, build_filter_announce) remain in the async
shell. Wire bytes and observable behavior are unchanged; full local CI
green (36/36) including the bloom-storm chaos gate.
Migrate the full non-async spanning-tree surface into proto/stp/, mirroring the
discovery/routing/fmp/mmp conversions. The classification ladder (parent-switch /
self-root / loop-drop / ancestry-update / periodic-rebroadcast / parent-lost) moves
out of the async node handlers into a pure Stp classify layer returning a
TreeDecision the shell drives, with effect ordering and per-arm invalidation
preserved verbatim. src/tree/ relocates wholesale: TreeState + ParentDeclaration data
+ coordinates into proto/stp/{state,coordinate}, the flap-dampening / hold-down
cluster into a FlapDampener in limits.rs, and the wire codec into wire.rs. The clock
is injected as u64 (wall-clock secs for the escaping declaration timestamp, monotonic
ms for the dampening timers via mmp::mono_ms); declaration crypto is field-partitioned
so sign/verify/hash run in the shell while the in-core modules carry data +
signing_bytes only. Peer maps/sets move to BTree; core/state/coordinate/limits are
core+alloc clean, with wire.rs the one std-tethered file. Behavior-neutral:
characterization tests added for the handler decision arms; convergence suite and
ci-local (36/36) green.
Migrate the FMP discovery decision logic out of the async handlers into
synchronous, runtime-agnostic sans-IO state machines owned by the protocol
structs, with I/O pushed to the edges. Pulls the full decision surface into a
pure core (backoff, rate-limit, planners, response routing), consolidates the
tests into a per-module tree with a shared crate testutil, and injects a u64
wall-clock so the core is free of Instant and std time.
Also brings the module toward no_std+alloc: the four discovery maps use
alloc::collections::BTreeMap (HashMap's RandomState is std-only), Arc is spelled
alloc::sync::Arc, the backoff-reset log lives in the shell (the core returns the
cleared count so observability stays out of the pure core), and the crate root
names alloc directly. The one remaining tether is ProtocolError's
std::error::Error coupling in the wire codec.
First subsystem of the broader sans-IO refactor; establishes the extraction
patterns and conventions carried forward to the remaining protocols.
handle_peer_removal_tree_cleanup reparents or self-roots the node when
its parent link drops, but omitted the coordinate-cache invalidation that
every other position-change path performs. Cached entries for downstream
destinations kept the node's now-stale coordinate prefix, and because
find_next_hop refreshes the TTL on every routing access, an actively
routed stale entry never self-expired — corrected only by a fresh insert.
Mirror the loop-detection branch: inside the parent-loss changed block,
invalidate both classes — invalidate_via_node (reparent) and
invalidate_other_roots (self-root). Add regression tests for a parent-link
removal that reparents (via-node entry dropped, same-root sibling
preserved) and one that self-roots (both via-node and stale old-root
entries dropped).
Add a "Running CI locally" section to testing/README.md covering the
ci-local.sh orchestrator: the per-run FIPS_CI_RUN_ID override and how it
scopes compose projects, image tags, and per-chaos-child subnets so
simultaneous runs on one host never collide; the cancellation exit codes
(130/143 vs 0/1) and how a preempting worker interprets them; and the
label-based cleanup via --reap / ci-cleanup.sh.
A CI worker may preempt an in-flight ci-local.sh run (SIGTERM, then SIGKILL
after a grace period) to restart on a newer commit. For that kill to be safe,
the script must clean up after itself and never let a dying run collide with
its restart. It previously had no signal handling, shared the default compose
project name across runs, and tore down each suite only at the suite end.
- Derive a per-run id (honoring FIPS_CI_RUN_ID, else short-sha+random) and
namespace every docker resource to it: a fipsci_<run>_<suite> compose project
per suite and per parallel chaos child, and per-run image tags
(fips-test:<run>, fips-test-app:<run>) retagged to :latest only after both
builds succeed so :latest never points at a half-built image.
- Install a bounded, idempotent teardown trap on SIGTERM/SIGINT (+ EXIT): reap
parallel chaos children, then force-remove this run's docker resources via
the new ci-cleanup.sh, wrapped in timeout so a stuck down cannot wedge it.
- Exit 143 (SIGTERM) / 130 (SIGINT), distinct from 0 (pass) / 1 (failed), so a
preempting worker tells a cancelled run from a real failure.
- Add ci-cleanup.sh (also ci-local.sh --reap): force-removes leftover CI
resources by the com.corganlabs.fips-ci=1 label and the fipsci_ project
prefix, robust to however a prior run died.
- Label every per-suite docker resource so the label sweep reaps it after a
SIGKILL regardless of network name: direct docker run/network resources, the
sidecar compose services, and every per-suite compose network (acl-allowlist,
boringtun, firewall, nat, static, both tor suites, and the chaos generator
template). Parametrize the static/sidecar compose image refs so the per-run
tags are honored.
- Give each parallel chaos child a unique /24 from 10.30.x (a new --subnet
override on the sim CLI, assigned per-child in ci-local.sh) so parallel
children never collide on a shared docker subnet, and a chaos net can never
span a fixed-subnet suite (sidecar/static in 172.20.x). 10.30.x sits outside
docker's default-address-pool range, so an auto-assigned net cannot land on
it either; node IPs derive from the subnet, so no scenario config changes.
Replace the fixed post-first-rekey and post-second-rekey settle sleeps in
the rekey integration test, and the fixed-deadline baseline wait in the
interop test, with the deterministic wait_until_connected progress-aware
polling helper already used for the initial baseline convergence.
Each converted site fails fast with structured diagnostics when the mesh
is stuck and extends its deadline only while pairwise reachability is
still climbing, removing the wall-clock settle windows that produced
intermittent connectivity failures after a rekey.
maint carries only its 0.4.1-dev version bump; master keeps its own
0.5.0-dev development version. Recorded as a linkage merge so later
forward-merges of real fixes land cleanly.
The shared fips.yaml ships ethernet.wan.interface: "eth0", the default
WAN port on OpenWrt 24 and earlier. OpenWrt 25 (DSA) boards, which the
.apk package targets, name the WAN port "wan" instead. Rewrite the
staged copy at build time so the as-installed config binds the Ethernet
transport to the right port out of the box, without maintaining a second
copy of the config file. The .ipk package keeps "eth0".
Update the openwrt README to document eth0 (24) vs wan (25/DSA) and add
a CHANGELOG entry.
A transient timeout uploading the built .ipk to the Blossom CDN
(blossom.primal.net) failed the entire OpenWrt Package run, and because
the GitHub release job depends on the build jobs, a CDN blip would block
every OpenWrt artifact from the release. Blossom/nostr distribution is
supplementary; the package already ships as a GitHub release artifact.
Wrap the upload in a 3-attempt retry with backoff to absorb transient
timeouts, mark the step continue-on-error so a persistent outage no
longer fails the build, and guard the NIP-94 publish on the upload
succeeding so no event is created with an empty URL. Applied to both the
.ipk and .apk build jobs.
Claude-Session: https://claude.ai/code/session_01A2pYfSypNmmG4HyHwZuLex
Prepare the source tree for the v0.4.0 release cut, leaving the version
at 0.4.0-dev (the version bump rides a separate release-candidate
commit).
- CHANGELOG: backfill the missing entry for the route-class transit
counters and fipstop routing-tab reorg, then reorganize the Unreleased
block from a flat per-commit list into topic-grouped subsections
mirroring the 0.3.0 entry (coalescing interim fixes into net-effect
descriptions without dropping technical detail), and stamp it as
[0.4.0] - 2026-06-21 with a fresh empty Unreleased block. The date is
provisional and reconfirmed at the final tag.
- Release notes: refresh both RELEASE-NOTES.md and the versioned archive
(kept byte-identical) to cover the OpenWrt .apk packaging, the Nix
flake, the macOS self-traffic checksum fix (#117), and the route-class
transit counters.
- README: bump the status badge to v0.4.0 so it matches the prose.
Claude-Session: https://claude.ai/code/session_01A2pYfSypNmmG4HyHwZuLex
Add six forwarding counters that partition transit-forwarded packets by their
tree relationship to the chosen next hop: tree-up (peer is our ancestor),
tree-down (peer is our descendant and the destination is within its subtree),
tree-down-cross (peer is our descendant but the destination is outside its
subtree), cross-link descend (lateral peer, destination within its subtree),
cross-link ascend (lateral peer, destination outside its subtree), and
direct-peer. The six classes sum to forwarded_packets, asserted by a unit
test. Classification is computed from tree coordinates at the transit
chokepoint, so the error-signal routing callers are excluded.
The two "outside the chosen peer's subtree" classes are both up-and-over
forwards but differ in what they depend on. Tree-down-cross is the
dive-to-tree-child cut-through: we forward down to our own child for a
destination not beneath it, which is only possible because the child
advertised cross-link reach upward to us, beyond its own subtree. Its count
measures how much forwarding depends on that upward advertisement, i.e. what
would change if cross-link advertisements were narrowed to subtree-entry only.
Cross-link ascend, by contrast, uses the node's own lateral cross-link learned
from a peer's split-horizon advertisement, so it does not depend on any upward
advertisement.
Surface the counters through the forwarding stats snapshot (control socket,
show_routing and show_status) and reorganize the fipstop routing tab so its
two columns separate own/endpoint traffic (received, delivered, originated)
from forwarded/transit traffic (the route-class breakdown and drop reasons),
with the tree-down-cross line visually flagged.
Add OpenWrt .apk packaging for OpenWrt 25+, where apk-tools is the
mandatory package manager. The existing .ipk continues to cover OpenWrt
24.x and earlier. Built SDK-free like the .ipk: it reuses the
cargo-zigbuild cross-compile and the shared installed-filesystem payload
under openwrt-ipk/files, and assembles the ADB container with the
official `apk mkpkg` applet from apk-tools 3.0.5 built from source, so no
OpenWrt SDK image is needed.
The package-openwrt workflow is refactored so a single compile-binaries
job cross-compiles and strips each arch once; both the .ipk and .apk
packagers consume the binaries via a new --bin-dir flag instead of each
recompiling. A build-apk job (aarch64, x86_64) builds apk-tools,
packages, and structurally verifies the .apk with `apk adbdump`. Releases
now publish .apk artifacts and checksums alongside .ipk; the release
download is scoped to fips_* so the shared raw-binary artifacts are not
swept into the published release. apk-version.sh maps a release tag or
commit height to an apk-tools-valid version, covered by a case-table test.
Packages are unsigned, installed with `apk add --allow-untrusted`,
matching the .ipk posture.
Also fix the OpenWrt control socket: the init script now pre-creates
/run/fips before starting the daemon, the procd equivalent of the systemd
unit's RuntimeDirectory=fips. Without it, on a fresh boot the daemon
resolves its control socket to /tmp (since /run/fips does not yet exist),
while fips-gateway later creates /run/fips for its own gateway.sock,
leaving fipsctl/fipstop resolving a /run/fips/control.sock the daemon
never bound.
Self-addressed TCP/UDP connections to a node's own <npub>.fips address
half-opened and hung on macOS. macOS routes self-traffic as loopback (a
LOCAL route via lo0), which defers the transport TX checksum, but the
point-to-point utun then egresses the packet into the daemon with only
the pseudo-header partial checksum present. The hairpin path added in
9a9e90a re-injected these verbatim, so the local stack dropped every
segment whose checksum MSS clamping didn't happen to rewrite: the
SYN/SYN-ACK got through (clamping recomputes them) but the bare ACK,
data, and FIN were dropped for a bad checksum, leaving the listener
stuck in SYN_RCVD.
Recompute the TCP/UDP checksum for self-addressed packets on the hairpin
path before re-injection, completing the self-delivery 9a9e90a started
(which only covered ICMP and the TCP handshake). Linux is unaffected: it
loops self-traffic via lo before the TUN, so the hairpin branch never
fires and checksums are already valid.
Confirmed on macOS: a self-connect that previously timed out now
completes in ~9ms with payload delivered.
The linux/macos/windows package workflows each publish a
checksums-<platform>.txt sha256 file with their release artifacts, but
the OpenWrt workflow attached only the .ipk files with no checksum
coverage. Generate checksums-openwrt.txt over the .ipk outputs using the
same sha256sum idiom as the other platforms and add it to the release
file set.
Add a flake that builds all four binaries (fips, fipsctl, fips-gateway,
fipstop) on Linux and macOS, pinning the exact toolchain from
rust-toolchain.toml (1.94.1 + rustfmt/clippy) via fenix so Nix builds
match CI and the AUR/Debian packaging.
Wire up the build-time native deps the source tree needs: pkg-config and
bindgenHook (libclang) for the rustables/libdbus-sys bindgen step, and
dbus for bluer's BLE support. autoPatchelfHook rewrites the binary RPATHs
so the daemon resolves libdbus-1.so.3 and libgcc_s.so.1 from the Nix store
at runtime — without it `fips` fails to load on NixOS, which has no global
/usr/lib.
Outputs: packages.{default,fips}, apps for each binary, checks.fips, and a
devShell with the pinned toolchain plus cargo-edit. Tests are skipped in
the package build since they exercise TUN devices, raw sockets, and mDNS
that aren't available in the sandbox, mirroring the AUR/Debian packaging.
Verified end-to-end in a pure Nix store (nixos/nix container): nix build,
nix flake check, and running all four binaries succeed against the
committed flake.lock.
Documented across the install and developer docs: a Nix / NixOS section in
packaging/README.md, the from-source guide in docs/getting-started.md
(noting the flake produces binaries only, with NixOS system integration
through the system configuration rather than the installer), the CHANGELOG,
README, CONTRIBUTING, and the v0.4.0 release notes.
Co-authored-by: Johnathan Corgan <johnathan@corganlabs.com>
A packet destined for our own mesh address reached the TUN reader on
macOS (utun egresses self-traffic into the daemon despite the lo0 host
route) and was pushed onto the mesh outbound path, where it was dropped
for lack of a session/route to self. Hairpin self-addressed packets back
to the TUN writer instead, so ping6 and connections to our own
<npub>.fips address are delivered locally.
On Linux the kernel already loops self-traffic via `lo` before it reaches
the TUN, so the branch never fires there; the check is kept unconditional
as a daemon-level delivery invariant and to keep it covered by Linux CI.
The fips build runs bindgen (via the rustables crate that powers the LAN
gateway's nftables bindings), which needs libclang.so at build time.
makepkg -s installs only declared makedepends, so without clang the AUR
build panics with "Unable to find libclang". Add clang to the makedepends
of both PKGBUILD and PKGBUILD-git so makepkg installs it and AUR users
get the dependency. Surfaced by the always-on aur-build CI job.
Bring the source tree to its finished v0.4.0 content state ahead of the
release candidate. Documentation, changelog, release notes, and
packaging metadata only; no code or version-string changes.
CHANGELOG.md: backfill the operator-visible changes that landed since
v0.3.0 (the show_metrics scraper query and fipsctl stats metrics, the
discovery dedup-cache-full counter, the off-rx_loop control read
surface and its new daemon-resolved fields, the fipstop TUI overhaul,
the TCP inbound cap now honoring max_connections, host-map hot-reload,
log-noise demotions, and the net bug fixes), topic-grouped rather than
replayed per commit; intra-cycle fixes folded into their feature
entries; the duplicate Unreleased Fixed section merged into one.
Release notes: author docs/releases/release-notes-v0.4.0.md to the
operator-upgrade bar and mirror it byte-for-byte into the root
RELEASE-NOTES.md. Attribute each feature to its author in Contributors
(Nym transport and the mixnet demo to @oleksky, opt-in mDNS LAN
discovery to @mmalmi).
README.md: add the Nym transport to the support matrix (Linux, macOS,
Windows; OpenWrt pending verification), the multi-transport bullet, and
the "What works today" list; fold mDNS LAN discovery into the existing
Nostr-discovery bullets; refresh the status narrative to v0.4.0 over a
global, public test mesh.
packaging/common/fips.yaml: add commented Nym transport and mDNS LAN
discovery example stanzas with verified field names and defaults.
Cargo.toml: add homepage, keywords, and categories crate metadata.
docs/reference: update the cli-fips version example to the released
form; document the new control-socket output fields and the typed
RejectReason families in control-socket.md; sync cli-fipstop.md with the
overhauled TUI keybindings.
poll_lan_discovery's comment said Noise XX, but LAN-discovered peers dial over
UDP through initiate_connection, which uses Noise IK (IK at FMP). compute_mesh_size's
header comment still described the obsolete sum-of-disjoint-subtrees estimate;
the function OR-unions every connected peer's inbound filter plus self and
estimates cardinality once (matching the body comment). Comment-only, no
behavior change.
Pre-cut documentation pass for the 0.4.0 release, verified against current source.
Corrections:
- fipsctl: stale 'show identities'/'show node' -> 'show status'
(host-a-service, run-as-unprivileged-user)
- mesh address derivation: first 16 bytes of SHA-256(pubkey) with the leading
byte set to 0xfd, not a fixed fd97: prefix (reach-mesh-services,
ipv6-adapter-walkthrough)
- gateway control socket mode 0660 -> 0770 (troubleshoot-gateway)
- Tor example: add advertised_port: 8443 so the published port matches the
prose (enable-nostr-discovery)
- bloom mesh-size estimate rewritten to the OR-union-of-peer-filters algorithm;
plus mtu deep-link, gateway pool wording, and a NAT failure-mode line
- examples: delete orphaned nostr-rs-relay config, accept inbound to the local
8443 TCP listener, fix fd::/8 -> fd00::/8 typos, dotless wireguard alias
Additions:
- new Nym mixnet transport section (fips-transport-layer) and the architecture
transport list
- new LAN/mDNS discovery section (fips-nostr-discovery)
- reference docs: Nym transport, LAN discovery, and new control/stats surfaces;
drop ble from the connect transport list
Run the AUR package through makepkg + namcap on the same triggers as the
other package workflows (pushes to master/maint/next, pull requests,
tags, and manual dispatch), so a broken PKGBUILD is caught continuously
rather than only at release time. The build runs in an Arch container
(makepkg/namcap are not on ubuntu-latest) and packages the checked-out
tree from a local git-archive tarball, so it works for branch/PR builds
and unreleased rc tags that have no published GitHub source archive yet.
makepkg runs with --nocheck since the test suite is already covered by
ci.yml; this job validates packaging.
Publishing to the AUR is unchanged in intent but now gated to a real
(non-prerelease) release tag push, plus the existing manual-dispatch
republish path for packaging-only pkgrel bumps; it depends on the build
job so a package that fails to build or lint is never published. Branch
pushes and pull requests build and lint but never publish.
The PKGBUILD-patching logic is factored into a shared
packaging/aur/patch-pkgbuild.sh used by both jobs; the build path
sanitizes the version for makepkg (which forbids '-' in pkgver).
The GitHub deb-install matrix ran debian12/ubuntu24/ubuntu26, but the
local harness (testing/deb-install/test.sh) runs five distros. Add the
missing debian13 (trixie) and ubuntu22 legs so the cloud gate covers the
same distro set as local CI. Each new leg invokes the existing test.sh
scenario, so per-distro behavior is identical to the local run.
Update the granularity-only parity notes in ci.yml, ci-local.sh, and
check-ci-parity.sh to list the full distro set.
Pull the available point releases that sit within the current version
constraints: tun 0.8.11, tokio-socks 0.5.3, socket2 0.6.4, nostr 0.44.3,
nostr-relay-pool 0.44.1, ratatui 0.30.1, serde_json 1.0.150, simple-dns
0.11.3, mdns-sd 0.19.2, plus the transitive maintenance crowd. Lockfile
only; no Cargo.toml constraint changes.
Defer the out-of-semver crypto and identity majors (secp256k1 0.31, sha2
0.11 with hkdf 0.13, bech32 0.12) and the mdns-sd 0.20 bump to a separate
coordinated refresh with its own handshake, identity, and LAN validation.
The connect_refused stat counter (the Refused line in fipstop) was
defined but never incremented: every SOCKS5 connect failure recorded
socks5_errors instead, so the counter sat at zero and the operator-facing
gauge was permanently misleading. Both the synchronous connect path and
the background connect_async task now count a genuine SOCKS5 REP=0x05
refusal as connect_refused and every other failure as a socks5_error,
distinguishing them precisely via tokio_socks::Error::ConnectionRefused.
Extends the mock SOCKS5 server with a configurable reply code and adds
two tests covering the refused and general-failure paths.
Add an outbound-only Nym mixnet transport that tunnels FMP peer links
through a local nym-socks5-client SOCKS5 proxy into the Nym mixnet. It
structurally mirrors the Tor SOCKS5 transport (connection pool,
connect-on-send background promotion, FMP-v0 framing reused from TCP)
with the onion, inbound-listener, and control-port machinery removed.
Wires the transport through the full TransportHandle dispatch, NymConfig
(standard transport-instance pattern), and node instantiation, and
surfaces its counters in fipstop. Includes a mock SOCKS5 harness and unit
coverage for the address-parsing paths.
Also adds an isolated single-container example
(examples/sidecar-nostr-mixnet-relay/) demonstrating FIPS peering across
the mixnet end to end. No new crate dependencies: tokio_socks, socket2,
and futures are already pulled in by the Tor transport.
Two independent rendering glitches, both most visible over SSH and inside
tmux:
Startup: ratatui::try_init() enters the alternate screen but never clears
it, and the first terminal.draw() only emits cells that differ from an
assumed-blank internal buffer. On terminals that don't hand back a cleared
alternate buffer (notably tmux, and amplified by SSH latency) the prior
contents show through. Force a full repaint with terminal.clear() before
the first draw.
Quit: the input EventHandler spawned a detached thread that polled stdin in
a loop outliving the main loop, so at quit it kept reading after raw mode
was disabled and stray bytes (a keystroke or a terminal query response)
echoed onto the restored screen. Give the thread a stop flag and join it
before restoring the terminal; poll on a short fixed interval (decoupled
from the refresh tick) so quit stays responsive.
git log --oneline -1
When an FMP msg1 or FSP msg3 rekey retransmission budget is exhausted, the
cycle is abandoned and retried on the next timer. On lossy or high-latency
links this is an expected, self-limiting outcome: the existing session stays
valid and keeps carrying traffic, so the abort is not a failure that warrants
operator-level visibility. Demote both abandon-cycle messages from warn to
debug to cut steady-state log noise on nodes with many flapping peers.
A manual disconnect tore down only the local side and sent the peer nothing, so
the peer kept its session and never re-emitted its tree and filter
announcements; on reconnect it was never re-adopted as a child and its bloom
filter was never recorded. Send the disconnected peer a scoped Disconnect, the
same message graceful shutdown sends to all peers, so both sides tear down and
re-handshake cleanly on the next connection.
Reworks the fipstop TUI across its rendering, the control read surface it
draws from, and its interaction model, on a machine-verified base.
Test infrastructure:
- Add a ratatui TestBackend snapshot harness (testkit + snapshots
modules) that renders any ui::draw_* into an in-memory Buffer from
canned show_* JSON and asserts the text grid plus per-cell style.
Layout, columns, alignment, labels, grouping, and colour are now
checkable under cargo test; every render below ships a snapshot.
Control read surface (each new field emitted byte-identically on the
live and off-loop builders, published once from the tick, with schema
fixtures regenerated and the parity asserts holding):
- show_status: effective persistence (persistent || nsec.is_some());
root and is_root; and a per-configured-transport-type peer-count map
in which idle-but-configured types stay visible at zero.
- show_peers: per-peer effective_depth (depth + link_cost, the value
evaluate_parent ranks on), null when unmeasured or coordless so
fipstop never recomputes it.
- show_tree: root_npub, resolved once daemon-side (self when root, an
attested peer npub, or an identity-cache hit).
- show_bloom: the last-actually-sent uptree filter fill ratio and
subtree estimate, null for a root or before the first announce.
- show_mmp: session-layer srtt, loss, and etx trend labels.
Rendering:
- Display a 6-byte non-UTF-8 TransportAddr as a colon-separated MAC at
the type layer, so daemon logs, fipsctl, and JSON consumers all
benefit; non-6-byte payloads stay bare hex.
- Right-justify the Bloom Peer Filters numerics into aligned fixed-width
columns, render the Routing panes through a kv_lines helper that shares
one value column across a key-value group, and right-justify the Graphs
by-peer summary columns.
- Truncate an over-long peer name (the npub shown when no friendly name
exists) in the Tree, Bloom, and MMP peer lists so it no longer runs
into the next column.
- Group the Peers table by role (parent, then STP children, then other)
and render it as a full grouped view with styled group labels and
blank separators; the selection stays a peer index and the cursor only
ever lands on a peer row. Apply the same role grouping to the Tree and
Bloom peer lists, joining each peer's role from the peers view by node
address.
- Show min in the Graphs plot titles, rest a steady non-zero metric on
the baseline as a row of dots, render a genuine zero as an empty plot,
and keep a distinct no-data placeholder.
- Replace the metric-by-peer grid, which squeezed plots to nothing once
peers overflowed, with a master/detail Graphs view: a scrollable
per-peer summary list that expands (Enter) to a full-pane btop plot,
with up/down to flip peer, n/N to switch statistic, m to cycle mode,
and Esc to return.
- Put inline colored trend arrows on the Link and Session MMP values
(drawn only on a rising or falling trend, with a fixed blank slot when
stable so the value columns stay aligned), via a shared helper.
- Cycle column sorting on the Link MMP, Session MMP, and Graphs by-peer
tables (one key cycles the active column, another toggles direction),
with the active column marked in each table's header.
- Render the new daemon-surfaced fields: the dashboard root line (a
self-is-root marker, otherwise a truncated root hex), a
transports-by-type line, and an "approx. mesh estimate" line; an
effective_depth column and lines on the Peers, peer-detail, and Tree
sites from the single daemon derivation, showing a dash placeholder
when unmeasured rather than a misleading zero; the full Tree root hex
plus an Npub line; and the Bloom uptree fill and subtree-estimate lines.
Interaction model:
- Add a declarative keybinding registry keyed by (Tab, UiMode) that both
the context footer and the ? help overlay render from, so the two
cannot drift; a test asserts every registry key has a dispatch handler.
- Add a modal ? help overlay, and a context-aware footer that shows the
current state's actions first, drops global hints when the terminal is
narrow, and always keeps a Help affordance as the overflow path.
- Generalize per-pane focus and scroll state on App, wired across the
Tree, Filters, Routing, and MMP tabs (f cycles pane focus and the
focused pane scrolls instead of clipping its overflow); on the MMP tab
the column sort acts on the focused pane. Esc deselects the active row
when no detail is open (detail-close still takes priority).
- Add a Del-disconnect confirmation modal naming the peer, the only
state-mutating action, issuing the control-socket disconnect on confirm
and noting that the peer stays disconnected until manually reconnected.
Complete the control-plane read-isolation work: every pure-read show_*
query now renders in the control accept task from published read
snapshots, so none round-trips the data-plane receive loop. Only the
mutating connect/disconnect commands still reach that loop.
Three subsystem snapshots are published via ArcSwap and served through the
read handle's snapshot_dispatch:
- A routing read view (spanning tree, bloom filters, coordinate cache,
identity cache, and the discovery F-queue summary scalars), published
from the tick, serving show_tree/show_bloom/show_cache/show_routing/
show_identity_cache.
- A per-entity read view (peers, sessions, links, connections, transports,
and the MMP link/session views) as Vec<Arc<Row>> tables reconciled
against the prior snapshot so a republish reuses unchanged rows by
pointer and re-allocates only changed or new rows, keeping the per-tick
publish cost bounded as the peer/session count grows. Serves
show_peers/show_sessions/show_links/show_connections/show_transports/
show_mmp.
- The stats snapshot is extended with the peer-ACL status and a per-peer
metadata map (is_active, npub, display name), resolved at publish time,
serving show_acl and the two per-peer stats queries.
Display names and other cross-subsystem fields are resolved at publish
time; time-relative fields are derived at render time from captured
absolute timestamps, so rendered output is byte-identical to the prior
on-loop handlers, which are retained as the equality oracle.
With every read query served off-loop, the show_* branch is removed from
the rx_loop control handler and the now-dead on-loop dispatcher deleted.
The snapshot projections are forward-compatible with the later structural
extraction of the derived-state and session tables: they become thin
views over the extracted types without changing the read-handle interface.
Introduce a read-snapshot plane so pure-snapshot control queries render in
the control-socket task instead of round-tripping the rx_loop, removing the
head-of-line coupling that let a busy or slow rx_loop time out fipsctl and
fipstop observability.
- ControlReadHandle: a cloneable bundle the control accept loop holds, over
the node's already-shared NodeContext and MetricsRegistry plus an
ArcSwap-published StatsSnapshot. A snapshot_dispatch seam serves cut-over
commands off-loop and falls through to the rx_loop for the rest, keeping
the rx_loop's ownership of Node intact.
- StatsSnapshot is published from the tick (the natural and sole mutator of
stats_history), carrying the history rings plus the scalar gauges and
counts show_status reports. Readers serve the latest snapshot
unconditionally, with staleness bounded by the tick interval and no
IO_TIMEOUT-coupled fallback.
- Off-loop now: show_status, show_stats_history, show_stats_all_history,
show_listening_sockets, show_stats_list, and a new counter-only
show_metrics (exposed as fipsctl "stats metrics", the enabler for a
Prometheus scraper at no hot-path cost). Queries that need live per-entity
state (peers, links, sessions, routing, and the per-peer stats variants)
stay on the rx_loop path pending later phases.
Quartet green; forward-merge to next verified clean.
The pool's TTL clock (VirtualIpMapping.last_referenced) advanced only on
DNS re-query, never on traffic, and the mapping-TTL is wired equal to the
DNS TTL, so an in-use mapping was forced to drain at TTL and reclaimed at
the first zero-conntrack tick (a stale drain_start gave no grace effective
protection), breaking long-lived, bursty, or DNS-cached clients.
In tick(), refresh last_referenced whenever conntrack reports sessions > 0
so an actively used mapping never ages out, and recover a Draining mapping
to Active (clearing drain_start) when traffic resumes, so a later drain
gets a fresh grace window instead of a stale one. The Active arm now only
drains an idle mapping. The DNS-TTL / idle-reclaim-TTL wiring is unchanged.
Adds regression tests for continuous-traffic-survives-past-TTL, bursty
drain-then-recover, and fresh-grace-on-redrain.
The per-transport TCP inbound cap was hardwired to 256 and never read
node.limits.max_connections, so raising max_connections was a silent
no-op for inbound TCP. Resolve the effective cap with precedence:
explicit per-transport max_inbound_connections, then node-wide
max_connections, then the built-in default of 256. Established peers
remain bounded node-wide by add_connection, so deriving the per-transport
raw-accept ceiling from max_connections does not admit more real peers
across multiple transports.
Add effective_max_inbound on the TCP transport with a node_max_connections
setter wired from create_transports, plus a precedence unit test.
Estimate the OR-union cardinality over self plus every connected peer's
inbound filter, dropping the parent/child tree gating in
compute_mesh_size. Filter propagation is split-horizon, so cross-links
advertise near-complete mesh views; unioning all peers yields the same
set as the tree-only union in steady state (OR dedups overlap and no
filter can over-count) while damping the node-count flap on parent
switches, since dropping the parent no longer collapses the upward leg.
This also removes the estimate's dependence on tree-declaration cache
freshness.
Rename the debug-log child_count to contributor_count, adapt the two
membership-invariant tests to all-peers semantics, and add a test that
the estimate stays stable across a parent drop when a healthy cross-link
is present.
The inbound FilterAnnounce FPR cap rejects filters whose false-positive
rate (fill^k) exceeds the configured maximum. On the fixed 1 KB / k=5
filter, 0.05 corresponds to fill 0.549 (~1,300 reachable entries), and
the busiest nodes' aggregates were beginning to hit that ceiling as the
mesh grew. Raise the default to 0.10 (fill 0.631, ~1,630 entries) to
restore headroom toward the fixed-filter capacity limit. A saturated or
poisoned filter is ~100% FPR and remains rejected, so the antipoison
gate is not materially weakened.
Updates the config default, the config-reference and bloom-filter
design docs, and the changelog.
The sidecar chain test intends node-a to be a standalone root ("node-a: no
outbound peers"), but started node-a without clearing FIPS_PEER_*, so it
inherited the external peer default from testing/sidecar/.env (a real public
mesh node, test-us01.fips.network) and auto-connected to the live mesh. The
whole a-b-c chain then attached under an external root, inflating tree depth and
breaking test isolation; this also produced spurious multi-hop failures.
Set FIPS_PEER_NPUB/FIPS_PEER_ADDR empty for node-a, matching how node-b and
node-c already get explicit inline peers, so the suite is hermetic regardless of
the .env default. Validated against the unmodified .env: node-a comes up with a
single link to node-b, no external attachment, multi-hop passes 18/18.
Before bf77ece (Fix DNS responder silent-drop on systemd-resolved
deployments, 2026-04-29) the daemon defaulted dns.bind_addr to "::"
(wildcard, accepted v4 traffic too), so the macOS pkg's resolver shim
of `nameserver 127.0.0.1` reached the daemon fine over v4 loopback.
That commit tightened the default to "::1" — IPv6 loopback only,
which on Linux/macOS does not accept v4-mapped traffic — to defuse a
mesh-interface filter / IPV6_PKTINFO bug that was silently dropping
.fips queries on systemd-resolved hosts. The Linux side was updated
in the same commit: fips-dns-setup now writes [::1]:5354 in every
backend, and the gateway's DEFAULT_DNS_UPSTREAM moved to [::1]:5354
with an inline comment about the v4/v6 mismatch.
The macOS resolver shim in packaging/macos/build-pkg.sh was missed in
that sweep. Since 2026-04-29, every macOS install has shipped
/etc/resolver/fips with `nameserver 127.0.0.1` while the daemon
listened on `::1`, so .fips hostnames don't resolve via getaddrinfo
(ping6, curl, etc.) even though `dig @::1 -p 5354 …` works.
The mismatch is easy to miss: mDNSResponder swallows the timeout,
VPN clients that hijack DNS (NetworkExtension match-domain : *) mask
it entirely, and the symptom looks like "discovery hasn't found the
peer yet". Switch the shim to nameserver ::1 to match the daemon.
A node with a single tree peer has its periodic parent re-evaluation
disabled (it needs at least two peers for a meaningful comparison), so
it depends entirely on its peer pushing a TreeAnnounce for it to attach.
That push happens once at promotion time plus on the parent's slow
periodic no-change re-broadcast. If the one-shot attaching announce is
lost, the single-uplink node falls back to self-root and cannot recover
until the next periodic re-broadcast (reeval_interval_secs later),
stranding it out of the tree and unreachable end-to-end in the interim.
Make tree-position exchange self-healing on the receive path: when an
accepted TreeAnnounce advertises a root strictly worse (higher NodeAddr,
since election is smallest-wins) than our own, echo our current
declaration back to that peer. A stranded self-root node's announce now
provokes its better-rooted peer to re-push its real position immediately,
so the node re-attaches within a round-trip instead of waiting for the
periodic cadence.
Echo only in that one direction. If the peer's root is lower (better)
than ours, we are the stale side: the peer would ignore our worse root
anyway and we converge via the parent re-evaluation that follows, so
echoing back is pure waste and would double announce traffic in the
learning direction during a root change or partition merge. Equal roots
are already converged. The echo is bounded by the existing per-peer
500 ms tree-announce rate limiter and is a no-op once the peer adopts our
root, so it adds no traffic in a converged mesh.
Add a spanning-tree unit test that drives a converged child back to
self-root with its peer-ancestry view of the root cleared (modelling the
lost attaching announce), and asserts the root re-pushes on the
resulting root disagreement and the child re-attaches.
wait_until_connected fail-fasted on stall even when the mesh was all but
converged, abandoning the run with budget still unspent because one hard
pair straggled to come up. On the rekey-outbound-only topology this turned a
rare deep-node timing straggle (stacked discovery backoff + late bloom
propagation, which clears well inside the budget) into a false baseline RED.
Add a near_converged_slack threshold (default 2): when the number of
still-failing pairs is at or below the slack and the stall window elapses,
keep polling toward max_secs instead of bailing. A mesh genuinely far from
convergence still fast-bails on stall, and a never-converging pair still
hits the hard cap, so a real regression is never masked.
Add a self-contained behavioral test (wait-converge-test.sh) covering the
four contract cases: near-converged hold (with a slack=0 contrast that
proves the slack is what saves the run), far-from-converged fast-bail,
never-converging hard cap, and four-argument backward compatibility.
Ignore duplicate or counter-regressed ReceiverReports before updating
RTT, loss, goodput, or ETX, so a delayed or reordered report can no
longer poison link metrics. Compute the RTT-from-echo sample with
checked timestamp arithmetic and reject zero, negative, or out-of-range
results instead of risking wrap or underflow on untrusted wire values.
On the sender side, when receiver dwell time overflows the u16 wire
field, suppress the timestamp echo (send 0) and saturate dwell to
u16::MAX rather than truncating, so a bogus small RTT cannot be formed.
Adds duplicate, out-of-order, wrapped-add, and future-dated (checked_sub)
sample tests, asserts loss and goodput stay unchanged on a dropped
duplicate, and covers the dwell-overflow echo suppression. Documents the
behavior in the MMP design note and CHANGELOG.
Co-authored-by: Johnathan Corgan <johnathan@corganlabs.com>
The interop harness only tested a full mesh, where every pair is a direct
one-hop FMP link. That left multi-hop forwarding, routing, coordinate
handling, and mesh-size estimation untested across versions.
Add an opt-in multi-hop topology and three new checks:
- generate-configs.sh: FIPS_INTEROP_EDGES selects an explicit undirected
edge list instead of the full mesh, with symmetric peering, connectivity
validation (no isolated node, graph connected), and per-node degree plus
edge metadata in nodes.env. Unset means full mesh, unchanged.
- interop-test.sh: a --topology flag with a built-in multihop-3v-cycle
(six nodes, two of each version, one cycle, two leaves). The per-node
peer check now expects each node's adjacency degree rather than N-1, and
the all-pairs ping now also exercises cross-version forwarding over
non-adjacent pairs. Adds a control-differential data-plane continuity
stream across the rekey window (Phase 5b) and a strict plus/minus 25
percent mesh-size convergence check (Phase 7).
- interop-stress.sh: forward --topology through to the per-rep driver
(mutually exclusive with a positional node-spec).
Phase 1 makes the strict retrying ping authoritative. The convergence
detector (one fully-clean, no-retry sweep of every directed pair) is now an
advisory settle-wait whose timeout is non-fatal; the strict retrying ping
decides pass/fail, matching how the post-rekey phases already work. Under
packet loss a clean no-retry sweep of a large mesh is statistically unlikely
even when the mesh is healthy (30 pairs at 2 percent loss leaves only about
55 percent of sweeps clean), so the detector otherwise falsely failed runs
whose strict ping reported full reachability. CONVERGENCE_TIMEOUT is now
env-overridable and defaults to 90s for larger meshes under loss.
Full-mesh runs are unaffected. Validated end-to-end with the
multihop-3v-cycle topology across three versions: all 30 directed pairs
reachable including multi-hop routed pairs, zero data-plane loss through
both rekey cutovers, and mesh-size converging to the true node count on
every node.
Brings the maint-line OR-union mesh-size estimator fix and the
per-peer / capacity-cap log-level demotions forward to master.
Conflict resolution: compute_mesh_size resolved to master's config()
accessor form carrying maint's OR-union rewrite (drop the stale summing
`total`); the log-level demotions auto-merged into master's handler
versions. The equivalent master-only log "connected UDP socket installed"
(connected_udp.rs, a file that does not exist on maint) was demoted
info -> debug as part of this merge so the connected-UDP path matches the
rest of the per-peer lifecycle logging.
On a saturated public-mesh node the connection-lifecycle and capacity-cap
events fire continuously and drown out the genuinely notable INFO/WARN
lines. Demote them to debug and drop a redundant duplicate:
- FMP K-bit cutover promotion (encrypted): info -> debug
- "Connection promoted to active peer" (handshake): info -> debug, and
remove the duplicate "Inbound peer promoted to active" line that
shadowed it on the inbound path
- "Peer restart detected" (handshake): info -> debug
- "Peer removed and state cleaned up" (dispatch): info -> debug
- "Rejecting inbound TCP connection (max_inbound_connections reached)"
(tcp): warn -> debug
- "Congestion detected, CE flag set on forwarded packet" (forwarding):
warn -> debug
- "Removing peer: link dead timeout" (mmp): warn -> debug
These are expected, high-frequency conditions on a busy public node (new
and reconnecting peers, ECN CE marking, the inbound connection cap, and
link-dead churn), not operator-actionable signals.
The mesh-size estimator summed the per-filter cardinality of the parent
filter and each child filter, which assumes those filters are perfectly
disjoint. When they overlap -- a stale or oversized parent filter, or a
routing loop -- the sum over-counts and inflates the reported mesh size
to as much as several times the true size.
Estimate the cardinality of the OR-union of the contributing filters
(self + parent + children) once instead. OR is idempotent, so any
overlap is deduplicated: the result equals the old sum in the disjoint
case and stays correct under overlap. The union is seeded from a clone
of a contributing filter so it keeps that filter's size class, and a
filter whose size class does not match is skipped rather than panicking.
The refuse-to-estimate behavior on a saturated or above-cap filter is
preserved.
Add a regression test with overlapping parent and child filters where
the naive sum over-counts and the union estimate tracks the distinct
member count.
The transport layer used Mutex::lock().unwrap() at ten sites across the
UDP, BLE, and Ethernet code. A std mutex poisons if a thread panics
while holding it, after which every lock().unwrap() on that same mutex
also panics, turning one fault into a cascade. These critical sections
only perform short HashMap/Vec operations on locally constructed values
and are not reachable from peer input, but the idiom is fragile against
any future in-section panic. Replace each with
lock().unwrap_or_else(|e| e.into_inner()), which recovers the guarded
data and removes the cascade with no new dependency and no call-graph
change.
Also replace four self.local_addr.unwrap() calls in the UDP start and
adopt paths with a sentinel fallback. The value is provably set just
above each log line today, but the unwrap is brittle against a future
reordering; logging an unbound sentinel is harmless and cannot panic.
Linux source builds pull in rustables, whose build script runs bindgen
to generate nftables bindings for the LAN gateway. bindgen needs
libclang.so on the build host, so a clean source build fails with
'Unable to find libclang' unless libclang-dev (or llvm) is installed.
The prerequisite text in README.md and CONTRIBUTING.md previously
listed only the optional BLE dependencies, and packaging/README.md had
no source-build prerequisite list at all. Document libclang-dev as a
mandatory Linux build dependency, distinct from the optional BLE deps,
and note that it is build-time only so pre-built .deb installs are
unaffected.
Record the changes that landed since the last changelog update: the
dual-auto_connect traversal-session election, the FMP link-layer rekey
reliability fixes (bounded msg1 retransmission with rekey-aware
heartbeat, and authenticate-before-cutover), the in-process loopback
test transport and progress-aware convergence wait that remove CI flake
classes, the local/GitHub CI suite-parity and single-source toolchain
selection, and the packaging change shipping the config as an example
that postinst seeds when absent.
Installing /etc/fips/fips.yaml as a live dpkg conf-file collides with a
configuration-management-rendered or operator-edited config on upgrade:
dpkg either prompts interactively (keep/replace), stalling unattended
upgrades, or clobbers the local file. Ship the default config as
/usr/share/doc/fips/fips.yaml.example (mode 644) and drop it from
conf-files. postinst now seeds /etc/fips/fips.yaml from the example only
when it does not already exist (mode 600), yielding to any existing
config without a prompt or clobber. Add ConditionPathExists for the
config to the service unit so a missing config skips the unit cleanly
rather than crash-looping.
A busy node opens roughly three file descriptors per established UDP peer
(a connect()-ed socket plus a 2-FD drain self-pipe), so the default 1024
soft RLIMIT_NOFILE is exhausted near 320 peers and further peer admission,
handshakes, and discovery fail with EMFILE. Document the FD budget, the
symptom, and the systemd (LimitNOFILE drop-in) and OpenWrt (procd nofile)
procedures to raise it, plus how to verify the per-peer ratio is bounded.
Link the new guide from the how-to index.
Bring the local runner (testing/ci-local.sh) and GitHub CI into agreement
on two axes.
Suite coverage: the admission-cap integration suite ran only locally, so a
regression in it could never turn the GitHub gate red. Add an admission-cap
leg to the integration matrix, add testing/check-ci-parity.sh (wired as
'ci-local.sh --check-parity') to diff the two suite sets and fail on
unexpected drift, and document the deliberate local-only (live-Tor) and
granularity-only differences in a comment block atop both runners.
Toolchain selection: every CI and packaging job installed its toolchain
with dtolnay/rust-toolchain@stable, but the rust-toolchain.toml channel pin
overrode it for all compilation, wasting an install and printing a
misleading rustc version. Switch the stable call sites to
actions-rust-lang/setup-rust-toolchain, which reads rust-toolchain.toml as
the single source of truth. Keep the explicit cache steps (cache: false on
the new action), set rustflags empty so the action does not impose a global
-D warnings the previous setup never applied, fold the macOS cross-compile
target into the action input, and leave the OpenWrt nightly Tier-3 leg on
dtolnay/rust-toolchain@nightly.
When two peers each auto_connect to the other, each runs both an
initiator and a responder NAT-traversal session and binds a separate
UDP socket per session. Each side adopts only the first Established
event and drops the loser session's socket; when the two sides adopt
mismatched sessions, each sends its Noise msg1 to a peer port the peer
has already stopped draining, and both handshakes stall.
Deterministically keep the session initiated by the smaller NodeAddr,
decided on the responder path: decline an incoming offer only when we
also have an in-flight outbound initiator for the same peer and our
NodeAddr is smaller. The peer's redundant initiator then times out,
leaving a single matching socket pair on both ends. Asymmetric
(one-sided) auto_connect has no co-active initiator and is never
suppressed, so connectivity is preserved; an undecidable NodeAddr falls
through to answering.
Reuses the NodeAddr tie-breaker convention already used by the
cross-connection and rekey dual-init paths. Adds a unit test for the
election helper.
Add wait_until_connected to the shared convergence helpers: it polls a
suite's own pairwise pings (the signal it actually asserts on), returns
as soon as every pair is reachable, extends its deadline while the
reachable-pair count is still climbing, and gives up only when progress
stalls.
Use it in the rekey, static-mesh, and sidecar suites in place of the
fixed wall-clock baseline timeout and the blind sleep, which timed out
under concurrent CI load while the mesh was still converging.
Add a Loopback variant to TransportHandle backed by an unbounded
in-process channel and a shared address-to-receiver registry, so
node-level multi-node tests deliver packets directly between nodes
instead of over real localhost UDP sockets. This removes the kernel
UDP receive-buffer overflow that dropped handshake packets when many
tests ran in parallel under CPU contention, and lets the large-network
convergence tests run reliably in the default suite again (their
parallel-load ignore markers are removed).
The new transport and its enum variant are cfg(test)-gated, so the
daemon build is unaffected.
The FMP rekey msg1 resend driver retransmitted indefinitely with no cap
and no abandon, so a rekey that never completed kept resending msg1
forever. Give it a retransmission budget: cap resends at
handshake_max_resends with exponential backoff and abandon the rekey
cycle cleanly once the budget is exhausted, mirroring the FSP session
rekey msg3 driver.
With the cap in place the link-dead heartbeat can safely become
rekey-aware: check_link_heartbeats now suppresses teardown while a rekey
is in progress with msg1 budget remaining, instead of reaping a link
that is still actively carrying rekey-handshake traffic. The suppression
terminates deterministically (the budget abandons on exhaustion, cutover
clears the in-progress flag), so a genuinely dead link is still reaped on
the next cycle.
Adds a rekey_msg1_resend_count counter on ActivePeer reset at every
rekey-clear and cutover site, msg1 resend-budget unit tests, and two-node
heartbeat suppression/resume/regression integration tests.
Remove the duplicated immutable fields (config, identity, startup_epoch,
started_at, is_leaf_only, max_connections/peers/links) from the Node
struct so the Arc<NodeContext> bundle is the single source of truth.
Previously Node owned these fields and a parallel context copy, kept in
lockstep by rebuild_context() at every mutation site — pure overhead that
existed only because of the duplication.
- Replace rebuild_context() with replace_context(): a clone-edit-swap of
the whole Arc. The per-instance context stays immutable; mutation swaps
the Arc. This is the sole runtime mutation path (constructors, leaf_only,
update_peers).
- Add Copy-returning accessors startup_epoch() and max_connections()/
max_peers()/max_links(); migrate the remaining direct field readers onto
the accessors. node_addr()/npub()/Debug now read identity/is_leaf_only
from the context.
- update_peers reads the pre-update peer set from the live context Arc
before building a fresh Config + context and swapping — preserving the
read-before-write ordering its mutation-window test depends on.
- Remove the test-only set_max_* setters; tests set the limits on Config at
construction instead (new make_node_with_max_peers/links helpers).
- Add a ci-local guard that fails if the Node struct re-declares a bundled
field, so the single-store invariant can't silently regress.
cargo test --lib 1291/0; clippy -D warnings and release build clean.
Store node counters in an atomic metric registry read through &self, and
introduce a shared NodeContext bundle holding the effectively-immutable
fields (config, identity, startup epoch, capability limits). Source the
immutable config and identity reads across the receive hot path, the
handshake/session/mmp/encrypted state machines, and the discovery, tree,
bloom, retry, and lifecycle modules through the context accessors rather
than direct field reads. The Node fields and the context are rebuilt in
lockstep at every mutation site.
PeerRecvDrain::drop previously called std::thread::join on the worker
thread synchronously. The worker uses packet_tx.blocking_send on a
tokio mpsc Sender, which internally parks the worker via
tokio::block_on on the same current_thread runtime that drives
rx_loop. Calling join from inside remove_active_peer (which runs on
the runtime thread, the runtime's sole driver) created a circular
wait:
- rx_loop blocks in libc futex via Thread::join
- the worker being joined cannot observe the stop flag because the
runtime that polls it is the very thread now blocked joining it
- all other PeerRecvDrain workers park on the same runtime via
block_on, so a single peer's removal wedges every worker on the
daemon
The /proc snapshot from a production wedge showed exactly this
shape: 107 of 108 threads in futex_do_wait, 101 of them named
fips-peer-drain. fipsctl became unresponsive (EAGAIN on control
socket), SIGTERM was ignored, and Docker SIGKILLed the container
after the 10 s grace period. Two confirmed wedges on the public
test deployment (52 min and 23 min uptime), plus a third on the
admission-gate-Msg2-silent-drop build at 2 min 21 sec — all ending
with the identical "Peer removed and state cleaned up
tree_changed=false" final log line preceding total silence.
Fix: detach the std::thread instead of joining. The stop flag plus
self-pipe write already signal the worker to exit; the worker's
kernel-level libc::poll inside the drain loop sees the wake, checks
the flag, exits, and the OS reclaims the thread state independently
of the JoinHandle being dropped.
The trigger was statistically amplified by aggressive multi-npub-
from-one-NAT peer reconnect patterns at the moment of the 30 s
link-dead-timeout peer-removal, but not bounded to them. Any
peer whose disconnect happens with the per-peer drain worker
parked in block_on can fire the bug. The admission-gate work
that landed earlier in this branch line compressed more handshake
work per rx_loop tick, increasing the rate at which workers are
parked in block_on and so reducing time-to-wedge — but the
underlying bug pre-dated the admission gate and pre-dated this
fix branch.
The deployed wedged daemon is mitigated operationally by blocking
the trigger IP at the host firewall; this commit removes the bug
class entirely.
Bring the refactor-hotpath integration branch into master: explicit
RejectReason counters for previously-silent receive-path drop sites
across the tree, discovery, and handshake handlers, plus the Reloadable
trait and ArcSwap-backed hot-reload consolidation for the host map and
peer ACL. Includes the new discovery dedup-cache-full reject counter.
The discovery request dedup cache (recent_requests) silently dropped
LookupRequests once it reached MAX_RECENT_DISCOVERY_REQUESTS, with no
counter to surface the condition. Add a DiscoveryReject::ReqDedupCacheFull
reject reason backed by a req_dedup_cache_full counter on DiscoveryStats,
mirroring the existing duplicate-request counter, and record it at the
drop site so the rejection is visible in show_routing.
Bring the runtime peer-list refresh and opt-in mDNS LAN discovery work
on master into the receive-path RejectReason / reloadable-config
integration branch. Code files auto-merge clean; the only conflict is
the CHANGELOG Unreleased section, resolved as the union of both sets of
entries.
Add scoped mDNS / DNS-SD discovery for peers on the same local link,
giving sub-second pairing without a relay or NAT-traversal roundtrip.
A node advertises its npub, protocol version, and an optional network
scope over link-local multicast, and browses for matching adverts to
initiate Noise handshakes against same-LAN peers.
LAN discovery is disabled by default; operators enable it with
node.discovery.lan.enabled: true. Default-off avoids reintroducing a
per-LAN identity broadcast on nodes that have deliberately disabled
other discovery channels, and avoids any multicast surprise on upgrade.
The startup advertised-port picker now excludes bootstrap transports
and selects a non-bootstrap operational UDP transport with a stable
lowest-id selector, so the advertised port is deterministic across
restarts rather than dependent on HashMap iteration order. This
matches the per-dial transport selection used for discovered peers.
Co-authored-by: Johnathan Corgan <johnathan@corganlabs.com>
Add Node::update_peers for runtime peer-list refresh. It re-derives the
active peer connections from a new peer configuration, adding newly
configured peers and removing those no longer present, while keeping
links to peers that remain in the set rather than tearing every
connection down. The call returns an UpdatePeersOutcome summarizing the
added, removed, and retained peers.
PeerAddress gains a seen_at_ms recency field (with_seen_at_ms). Active
path selection now sorts address candidates by recency so the most
recently observed address wins when concurrent path probes race.
complete_rekey_msg2 now returns the remote peer's startup epoch
alongside the new Noise session, letting the rekey path detect a peer
restart and clear stale session state. A stale FSP session is cleared
when a peer restart is detected during FMP rekey or cross-connection
promotion, so the session-layer map no longer lingers out of sync with
the freshly promoted peer.
Per-tick work budgets bound the connection churn in a single node tick
(MAX_DISCOVERY_CONNECTS_PER_TICK, MAX_RETRY_CONNECTIONS_PER_TICK,
MAX_PARALLEL_PATH_CANDIDATES_PER_PEER); work beyond a tick's budget is
deferred to the next tick rather than discarded.
Co-authored-by: Johnathan Corgan <johnathan@corganlabs.com>
Move PeerAclReloader onto the Reloadable trait: its ACL snapshot is now
published through an arc_swap::ArcSwap so the authorization hot path reads
it without locking, and the former check_reload becomes the trait's
reload(). The node tick calls self.peer_acl.reload().await.
Wire the host map into the tick as well. The host map snapshot was
previously taken once at construction and never polled; it now hot-reloads
on /etc/fips/hosts mtime changes once per tick, alongside the ACL, so
hostname display reflects edits without a restart.
The path_mtu_lookup cache (event-driven, populated from observed traffic)
and the nostr_discovery subsystem (an async spawned task) are deliberately
left off the trait: neither reloads from a backing file, so a no-op reload()
would be misleading. The rationale is documented on the trait module.
The host map and the ACL's embedded alias reloader still stat /etc/fips/hosts
independently each tick. A single small-file stat per tick is cheap, so the
duplicate is left in place; sharing one mtime observation between the two is
a possible future cleanup.
Tests: a node-level test exercises the host-map tick reload end to end
through peer_display_name; the ACL reloader tests are updated to drive the
async reload().
Add a `Reloadable` trait that normalizes the node's reloadable
configuration/resource pattern onto a single contract built around an
`arc_swap::ArcSwap` snapshot: a lock-free `load()` for the hot read path
and an async `reload()` that re-reads the backing source and atomically
swaps in a fresh snapshot. The trait carries the canonical Arc-wrapper
template documentation (single-writer node tick, many-reader hot path,
whole-snapshot swap so readers never observe a partial update).
Migrate the host map to this trait via a new `HostMapReloadable` that
reuses the existing load/merge/mtime helpers in upper::hosts. The Node
`host_map` field changes from `Arc<HostMap>` to `HostMapReloadable`, and
`peer_display_name` reads through a lock-free guard. The initial snapshot
is byte-identical to the previous construction, so behavior is unchanged.
The host map is still snapshotted once at construction and not polled;
`reload()` is exercised only by unit tests for now. Wiring the periodic
poll into the node tick, and deduplicating the hosts-file stat against
the ACL reloader's embedded copy, is left as a follow-up.
Add `arc-swap` as a dependency. Unit tests cover initial load (base +
file, base only), change/no-change/deletion/creation detection,
base-preserved-on-reload, and equivalence of the initial snapshot to the
pre-migration construction.
Introduce a typed RejectReason enum and a NodeStats::record_reject
dispatch so every receive-path rejection-and-return site bumps a
machine-readable per-subsystem counter while keeping its operator-facing
log line. The top-level variants mirror the existing NodeStats subsystem
split (Tree, Bloom, Discovery, Forwarding) and add Handshake, Session,
Mmp, and Transport categories; HandshakeStats, SessionStats, and MmpStats
are new sub-stats.
Wired clusters: tree and MMP outbound sign-failure; the FSP session
unknown-session and state-machine cluster; the Noise IK handshake
state-machine cluster (msg1/msg2); and the decode / crypto / cap /
semantic tail across bloom, discovery, forwarding, mmp, and tree. The
TreeStats::ancestry_invalid counter, present since the scaffold but never
incremented, is now bumped from the validate_semantics ancestry rejection.
Several handshake, MMP, tree, and discovery paths that previously had no
counter at all are now counted, including the send_lookup_response
no-route drop (DiscoveryStats::resp_no_route).
Existing direct counters at the bloom / discovery / forwarding sites are
retained alongside the new dispatch while the rollout is in progress (the
bloom_poison tests expect the transitional +2 delta); a later change
collapses the duplicate increment.
Raise the in-process backpressure headroom in make_test_node_with_mtu
(request an 8 MiB recv_buf_size on UdpConfig and grow packet_channel from
256 to 8192) to reduce localhost-UDP receive overflow under parallel-CPU
scheduler contention, and mark the large-network convergence tests
#[ignore] so cargo test --lib stays green by default. The ignored tests
remain runnable on demand with --ignored or --test-threads=1.
Add per-direction pool_inbound/pool_outbound counters to TcpStats and
TorStats, updated at every pool-insert, receive-loop-exit, transport-stop,
and send-failure removal site. Compare the max_inbound_connections cap
against pool_inbound rather than the combined pool length, so outbound
connect-on-send connections no longer consume the operator-facing inbound
budget. The configuration field name and operator semantics are preserved;
only the cap-check comparison and accounting change.
New integration scenario verifying the early-gate silent-drop behavior
of the inbound max_peers admission check at sustained scale, using the
existing 5-node mesh topology with one node's node.limits.max_peers
lowered to 1. This forces 2 of the cap'd node's 3 configured peers
into a sustained denied state, and asserts via tcpdump that no Msg2
responses go back to those denied peers across a 60s capture window.
A background load-driver restarts the denied peer containers every 15s
to reset their auto-reconnect exponential backoff (5s base / 300s cap),
producing fresh Msg1 bursts each cycle. Without this loop the gate
fires ~3-4 times per denied peer in a 60s window; with restarts the
observed rate is 15 per denied peer (~30 total firings), high enough
that any Msg2 leakage would be caught with strong statistical
confidence.
Local run on this branch: cap'd node-c converged to peer_count=1 with
node-b admitted; nodes d and e sustained-retried as denied; tcpdump
captured 30 inbound Msg1 (len 84) packets from the denied pair and 0
outbound Msg2 (len 104) packets, with final peer_count unchanged.
Files:
testing/static/scripts/admission-cap-test.sh — new test script with
inject-config subcommand (sets node.limits.max_peers) and a
3-phase test driver (converge, capture-with-load, per-peer assert)
testing/ci-local.sh — register admission-cap as a new suite category
(ADMISSION_SUITES), wire run_admission_cap function, add to
run_suite dispatch, list_suites, and the default integration sweep
Together with the existing unit-level coverage in src/node/tests/unit.rs
(handle_msg1_silent_drops_at_cap_for_new_peer with mock-transport Msg2
discriminator, and handle_msg1_admits_existing_peer_at_cap as the
bypass regression guard), the gate's silent-drop behavior is now
verified both at single-firing wire-observable resolution and at
sustained multi-firing cross-process scale.
Move the max_peers cap check in handle_msg1 forward, from the late
check inside promote_connection (which fires after Msg2 has already
been built and put on the wire) to an early position after identity
verification but before index allocation and the Msg2 send. When the
gate fires for a net-new identity, the Msg1 is silent-dropped — no
response goes back to the peer, no AEAD compute or wire bytes are
spent.
Bypass preserved for known peers (reconnect / cross-connection): if
the sender's NodeAddr is already in self.peers, or if a pending
outbound connection is in flight to the same identity, the gate is
skipped so legitimate maintenance traffic continues to work. The
late check inside promote_connection is intentionally retained as
defense-in-depth against future call sites or a disconnect racing
between the early-gate decision and promotion.
Wire-cost rationale: a 45 s tcpdump at saturation observed ~3.6
cap-denials/s steady-state, each previously paying the full Noise IK
responder crypto + Msg2 (~104 B) on the wire before being rejected.
The bigger value is cleaner peer-side semantics — the peer no longer
sees a fake-completed handshake whose data frames subsequently fail
decryption locally.
Two new unit tests cover the cases:
- handle_msg1_silent_drops_at_cap_for_new_peer drives a wire-pumped
Msg1 from a fresh identity into a saturated node and asserts no
Msg2 reaches the sender socket. Stash-verifies as FAIL on the
pre-fix tree (Msg2 hits the wire) and PASS post-fix.
- handle_msg1_admits_existing_peer_at_cap drives a Msg1 from an
identity already in self.peers and asserts the gate does not evict
it. This is a regression check (the no-gate tree behaves the same
way here, but the test guards against an accidental future gate
that breaks known-peer admit).
Brings two structural fixes landed on maint:
- compute_mesh_size: explicit parent skip in the children loop, so the
disjoint-subtree invariant no longer depends on peer_declaration cache
freshness.
- max_peers: outbound connection-initiation gated on the cap (auto-reconnect
retries, Nostr-mediated discovery established adoption, and both sides
of the NAT-traversal punch sequence). Inbound msg1 admission gate
unchanged.
node.limits.max_peers was honored only on inbound msg1 admission
(handshake.rs handle_msg1 returns PeerLimitExceeded when peers.len
is at the cap). Four outbound initiation paths proceeded unconditionally
at capacity: auto-reconnect retries (process_pending_retries),
Nostr-mediated discovery's BootstrapEvent::Established adoption
(poll_nostr_discovery), NAT-traversal punch initiation (the outgoing
side of the offer/answer/punch sequence in the Nostr discovery
runtime), and NAT-traversal punch response (the incoming side of the
same sequence). A saturated node burned CPU, UDP probes, STUN
observations, and Nostr relay traffic on connections that the inbound
gate would reject the moment they reached msg1.
Introduce Node::outbound_admission_check (peers.len < max_peers, or
true when max_peers == 0 as the no-cap sentinel) and gate the four
paths. The discovery runtime lives in a separate task and does not
hold a Node reference; bridge via an Arc<AtomicBool> the runtime
reads and Node refreshes once per tick from outbound_admission_check.
The atomic granularity is intentionally loose: one-tick lag is
acceptable because the inbound msg1 gate continues to be the
authoritative cap, and in-flight handshakes started below the cap
are allowed to complete.
Inbound gate at handshake.rs is unchanged.
The mesh-size estimator's children loop relied on the cached
peer_declaration(parent_id).parent_id() != my_addr check to exclude
the parent. That cached view briefly disagrees with our own latest
my_declaration().parent_id() during the window between a local
parent-switch and the new parent's next inbound TreeAnnounce: the
peer-declaration cache still names us as the parent's parent, so the
parent is iterated as if it were a child and its (typically dominant)
bloom cardinality is added a second time. Symptom: estimated mesh size
displayed in fipsctl show status and fipstop nearly-but-not-exactly
doubles during tree rebalancing.
Make the invariant structural with an explicit peer_addr == parent_id
skip at the head of the children loop. Per-peer 500 ms rate-limiter
and overall recompute cadence are unchanged.
Adds a regression test that constructs the stale-peer-declaration
scenario directly and asserts the parent is not double-counted.
Adds a tree/mmp-targeted compose overlay (compose-trace-tree.yml) and
a new FIPS_MESH_LAB_TRACE_TREE env-var gate in run_rekey_family,
layered independently of the existing FIPS_MESH_LAB_TRACE rekey-class
overlay. Trace targets: fips::node::tree, fips::tree,
fips::node::handlers::mmp, fips::node::handlers::handshake.
Used for tree-partition race investigations during multi-peer startup
where evaluate_parent inputs, send_tree_announce_to_all recipient
enumeration, and process_receiver_report first-RTT triggers all need
to be visible together to bracket the loss window.
Extend parse_rekey to emit phase1_status / phase1_baseline_passed /
phase1_baseline_total fields in signature.json by scraping rekey-test.sh's
"Best observed baseline before timeout: N/M passed" line (the timeout
path) and "Pre-rekey baseline (all 20 pairs): N/M passed" (the success
path). Phase-5-shape parsing is unchanged.
Extend mechanism_match_rekey to also fire on the Phase 1 characteristic
12/20 split — the multi-hop-routing-failure shape where direct-peer pairs
pass and the four multi-hop pairs (x 2 directions) fail. The Phase 5
predicate remains intact for the pre-existing flake class.
Add FIPS_MESH_LAB_NO_RESOURCE_LIMITS=1 to run_rekey_family for
unconstrained characterization runs where the goal is to surface a race
or scheduling artefact rather than reproduce GHA pressure. Default
behaviour (variable unset) keeps the compose-resource-limits.yml overlay
engaged.
Documented in README.md and the in-script env-var header block.
Closes the eventually-consistent gap in spanning-tree state
distribution. Every existing send_tree_announce_to_all call site
gates on a local state-change event (parent switch, self-root
promotion, ancestry change, peer promotion, parent loss). Once a
partition latches — for example a parent-switch announce stranded
in the brief cross-init handshake swap window, where the announce
arrives on a session-index whose decrypt-worker entry has been
unregistered — neither side's state changes again, so neither
side ever re-broadcasts. The existing 60 s check_periodic_parent_reeval
was a re-evaluation, not a re-broadcast: it short-circuited
silently on no-change. Production-side healing depended on
incidental link churn; lab harnesses with stable docker-bridge
links had no equivalent path.
Add a final else branch that fires send_tree_announce_to_all
unconditionally on the no-change path, alongside the existing
switch and self-promote arms. Receivers coalesce by sequence
comparison (ParentDeclaration::is_fresher_than) and short-circuit
at the `if !updated` gate in handle_tree_announce; same-sequence
repeats drop silently with no cascade. The per-peer 500 ms
rate-limiter is well below this 60 s cadence and does not suppress
the heartbeat broadcast.
The fix is a general protocol-robustness improvement: it addresses
any in-flight TreeAnnounce loss class, not only the specific
cross-init swap-window drop site.
testing/static/scripts/rekey-test.sh BASELINE_CONVERGENCE_TIMEOUT
60 -> 65 so a partition healed by the periodic broadcast at T+60
lands inside the convergence window. wait_for_full_baseline
early-exits on PASS, so successful reps see no extra wall-clock.
The cross-connection-won path in handle_msg1 removes the old peer and frees
its allocated index, but does not unregister the old (transport_id, our_index)
cache_key from the decrypt worker pool. The orphan entry sits in the
per-shard HashMap until the index allocator recycles old_idx to a different
peer and that peer's register_decrypt_worker_session call overwrites it.
In the interim, any decrypt job that lands at the recycled cache_key
resolves to the wrong session and AEAD silently fails — observed as
multi-hop routing failure in 5-node static-mesh on next-branch where
bidirectional auto_connect drives cross-connections at every peer pair
on startup.
The tick body's per-peer check_* loops (heartbeats, bloom
announces, MMP reports, tree announces) called transport.send
for every active peer, which on TCP/Tor fell through to a 5 s
connect-on-send wait for any peer whose pool entry was not yet
established. That wedged the entire tick body for the full
connect_timeout_ms per unreachable peer; under post-restart
convergence on a high-peer mesh, this cascaded into multi-
second tick stalls. On master, the same mechanism also starved
the per-tick control-snapshot republish and pushed fipsctl
queries onto an mpsc fallback that was itself queued behind
the wedged rx_loop, producing the 5-second fipsctl head-of-line
pattern operators observed on loaded nodes.
Gate send_encrypted_link_message_with_ce on
transport.connection_state before the send: proceed only when
Connected; on None, kick off a non-blocking background connect
(idempotent — TransportHandle::connect dedupes against the
connecting pool and spawns the timeout-bounded TcpStream::connect
inside its own tokio task) and fail this send fast with a
clear "transport connection not ready" error. A subsequent
tick retries once the pool has an entry. The reconnect
lifecycle (check_link_heartbeats, process_pending_retries,
poll_pending_connects) is unchanged. The connect-on-send
branch in transport.send_async itself remains in place for
code paths that legitimately need synchronous connect (e.g.,
explicit operator-driven fipsctl connect).
When both peers' Nostr-mediated UDP punches complete within the
same scheduling window, each side's `BootstrapEvent::Established`
event arrives with `is_connecting_to_peer` already true: each side
received an inbound msg1 from the peer's pre-punch outbound
attempt, which created a connecting-state record. The deduplication
skip then fires on both sides, neither installs the fresh
traversal socket as canonical, and the peer-adoption budget
(45 s) expires. Cross-node wall-clock alignment of the skip log
line in observed failures was within ~1 ms — simultaneous dual-
fire under contention, the dual-initiation pattern.
Apply the deterministic NodeAddr tie-breaker already used at
`handlers/handshake.rs:269` for rekey dual-initiation and in
`peer::cross_connection_winner` for cross-connection resolution.
Smaller NodeAddr wins as adopter: enumerate the in-flight
connections whose `expected_identity` points at this peer, tear
them down via the canonical `cleanup_stale_connection` helper, and
fall through to `adopt_established_traversal`. Larger NodeAddr
loses and keeps the existing `continue` semantics; the loser's
in-flight outbound is reconciled by `handle_msg1`'s cross-
connection logic when the winner's fresh msg1 arrives over the
adopted socket.
`cleanup_stale_connection` visibility bumped from module-private
to `pub(in crate::node)` so it is callable from `lifecycle.rs`.
The defensive re-check inside `adopt_established_traversal`
itself is left as-is — after the outer cleanup the winner reaches
it with `is_connecting_to_peer == false`, so the inner skip
won't trip. The `BootstrapEvent::Failed` arm is unchanged: there
is no winning outcome on dual failure, and the existing skip +
retry-schedule semantics are correct.
Three deltas to the mesh-lab nat-lan suite for stall characterization:
- FIPS_NAT_LAN_CPUSET env-var-driven CPU-pinning sidecar in
run_nat_lan, mirroring the bloom-storm pattern. Pinning is needed
because the mesh-lab compose-resource-limits.yml override is
rekey-family service-name specific (rekey-* / rekey-accept-off-* /
rekey-outbound-only-*), so it does not constrain the nat-lan
containers. Default cpuset 0,1 mimics a GHA 2-core runner; empty
disables the sidecar.
- New compose-trace-nat.yml overlay that bumps RUST_LOG to trace on
discovery::nostr, transport::udp, node::lifecycle,
handlers::handshake, handlers::forwarding — the modules covering
the cross-init / adoption / handshake path. Picked up by the
nat-test.sh COMPOSE array via a new FIPS_NAT_EXTRA_COMPOSE
colon-separated env-var hook. run_nat_lan sets this hook when
FIPS_MESH_LAB_TRACE is non-empty; the README env-var section
updated to reflect that FIPS_MESH_LAB_TRACE now applies to nat-lan
in addition to the rekey-family.
- parse_nat_lan extended with a per-node stall_signature emitting
last-occurrence timestamps for eight event categories (startup,
discovery, adoption, handshake_init, msg2_sent, cross_init_ignore_*,
handshake_failed) plus derived last_meaningful_event_ts,
last_event_category, silent_gap_s. Top-level stall_class binned as
no_timeout / silent / localized / distributed / incomplete from
the per-node categories. Aggregation phase consumes the per-rep
signatures across a characterization run to classify stall
mechanism.
Wired support in nat-test.sh: FIPS_NAT_EXTRA_COMPOSE colon-separated
list of repo-relative or absolute compose files layered onto the
base via the COMPOSE array; FIPS_NAT_SKIP_FINAL_CLEANUP gates the
success-path teardown so the mesh-lab harness can capture docker
logs before tearing down (failure paths already returned without
cleanup, leaving stall-state containers intact for capture).
Smoke-tested on idle profile with TRACE on: 1 rep PASS, 32/36 TRACE
lines per node, signature.json events all populated with the
expected category timestamps.
The bloom-storm scenario's bloom_send_rate ceiling has been bumped
from 30 to 40 sends per node over the trailing 30 s window. A 59-rep
characterization run under `github-runner-equivalent` pressure with
per-container CPU pinning to `cpuset=0,1` (mimicking a 2-core
`ubuntu-latest` runner) measured n04 (the structural max-spike node)
at mean 24.4, P99 29, max 30. The original ceiling of 30 sat at the
lab's structural max, leaving no headroom for the asymmetric transient
spikes observed on GitHub Actions (n04=34 on master CI run 25933972365,
re-fired on run 26008950865). GHA fires do not reproduce on this lab
host even with the cpuset sidecar applied.
Rationale: lab max + ~2σ ≈ 39.4 → round to 40, giving 33 % margin over
the lab maximum while staying well below the deployment-scale storm
rate (~480× steady state). The companion `min_parent_switches` guard
is unchanged.
See README.md alongside this file for the updated threshold derivation.
Wires the bloom-storm chaos scenario into the mesh-lab harness as
a first-class suite, with optional per-container CPU pinning to
mimic GitHub Actions' 2-core ubuntu-latest budget.
Dispatch path — three new run-loop.sh functions plus the
dispatch_suite and dispatch_mechanism_match case-arm additions:
- `run_bloom_storm` invokes `bash testing/chaos/scripts/chaos.sh
bloom-storm` and captures stdout+stderr into the rep's
test-output.log. Chaos uses its own python sim runner
(`python3 -m sim`), not docker-compose, so this suite gets no
per-container compose override, no separate `docker logs`
capture, and no in-container netem injection — the chaos
scenario yaml owns its own netem and link-swap config.
- `parse_bloom_storm` extracts the bloom_send_rate result
(pass/fail/unknown), ceiling, max-observed per-node delta,
offenders list, full per-node delta distribution, the companion
min_parent_switches result, and panic + error counts. Lands in
the rep's signature.json. Two parser details: assertion greps
are anchored on `^(PASS|FAIL)` so they only match the bare
end-of-run summary line, not python-logger-prefixed lines that
contain the same substring; and `grep -c` panic/error counts
use `; true` + a defensive empty-string check instead of the
common `|| echo 0` fallback (`grep -c` exits 1 on zero matches
while also printing "0", so the fallback would corrupt the
count to "0\\n0").
- `mechanism_match_bloom_storm` returns true when a rep both
fails the bloom_send_rate assertion and the FAIL line carries
a named offender (filtering the harness-side "failed to sample
window endpoints" sub-failure out of the mechanism count).
CPU-pinning sidecar — bloom-storm's chaos sim spawns containers
directly via the docker SDK, so the mesh-lab compose-resource-
limits override does not apply. A poll-and-pin loop around the
chaos.sh invocation lists \`fips-*\` containers every 0.5 s and
applies \`docker update --cpuset-cpus <set>\` to each. Pinning is
idempotent (re-applying the same cpuset is a no-op). Default
cpuset \`0,1\` mimics the GHA 2-core budget; override via
\`FIPS_BLOOM_STORM_CPUSET=<set>\` (any comma-separated CPU list),
or set to the empty string to disable. Only applies to the
bloom-storm suite; other suites' dispatch paths are unchanged.
README's "Suites supported" entry covers the assertion class, and
the \`FIPS_BLOOM_STORM_CPUSET\` knob is documented alongside the
other mesh-lab env-var knobs.
The cross-connection-won branch of `promote_connection` builds a
fresh ActivePeer with a new Noise session and our_index, inserts
it into peers, and registers identity, but did not hand the new
session to the decrypt shard worker pool. The normal-promotion
tail in the same function does make that call. A session
established via the cross-connection race path therefore missed
the worker fast-path for its lifetime, falling back to inline
decryption on the rx loop. Correctness was unaffected, but the
throughput/latency benefit of the worker pool was lost for peerspromoted through that path.
Mirror the normal-promotion tail and call
`register_decrypt_worker_session` after the fresh ActivePeer is
inserted into `self.peers` in the `this_wins` arm.
Logs source, destination, and payload size at the existing no-route
drop site so investigations can attribute transit drops without
enabling trace-level instrumentation. Diagnostic-only; no behavior
change on the success path.
An FSP session rekey could leave the two endpoints holding different
key sets for a brief window: if a handshake message was lost in
transit, one side rotated to the new keys while the other did not.
Traffic sealed in one key epoch then reached a peer still on the
other epoch and failed to decrypt, producing bursts of AEAD
decryption failures and dropped connectivity until a later rekey
cycle reconverged the pair. Choreographing the cutover order cannot
close this window: any fixed ordering still leaves a skew that
packet reordering widens.
Make rekey correctness independent of cutover timing by overlapping
the key epochs on the receive path. During a rekey transition the
receiver trial-decrypts each frame against every live session it
holds: current, the not-yet-promoted pending session, and the
draining previous session. The K-bit becomes a hint that orders the
trial-decrypt cascade rather than a hard gate, and a frame that
authenticates against the pending session is itself the cutover
signal. No rotation ordering and no packet reordering can then cause
a decryption failure.
The pre-rekey Noise session is held in the `previous` slot until the
peer has demonstrably moved off it. Its drain deadline is anchored
on the most recent frame the peer authenticated against that slot,
refreshed each time the trial-decrypt cascade lands there, rather
than on a fixed wall-clock timer started unilaterally at the local
cutover. A peer that never received the new keys keeps authenticating
against `previous` and the slot stays live; without this, a fixed
timer would erase the only key set that could decrypt the peer's
frames, producing a permanent silent decrypt failure on a live data
path. A peer that never catches up is handled by the existing FSP
session liveness path rather than by silent decrypt failure.
The lost-handshake liveness gap is closed separately by retransmitting
the third rekey handshake message until the peer is confirmed on the
new keys, with a bounded retry budget after which the rekey cycle is
cleanly abandoned and retried on the next timer.
Adds unit tests covering the trial-decrypt cascade (epoch selection,
promotion on pending decrypt, reordered old-epoch stragglers after
cutover, per-slot replay-window integrity), the msg3 retransmission
lifecycle, and the peer-progress-aware drain retirement.
Commit 57a089f6 (the GitHub #102 fix) landed without a CHANGELOG
entry. Add the `[Unreleased]` / `### Fixed` line so the macOS
package-integrity fix is on record before the v0.3.1 cut.
The AUR `fips` and `fips-git` packages did not install the
`fips-dns-setup` and `fips-dns-teardown` helper scripts that
`fips-dns.service` runs. The Debian package ships them to
`/usr/lib/fips/` through the `[package.metadata.deb]` assets, but the
AUR `package()` functions never replicated those install steps, so
`fips-dns.service` failed to start on Arch with "Unable to locate
executable /usr/lib/fips/fips-dns-setup".
Add the two `install -Dm0755` lines to both PKGBUILDs so the AUR
packages match the Debian layout.
Also harden the transition between the `fips` and `fips-git`
packages: each PKGBUILD now declares the other variant's `-debug`
split package as a conflict and opts out of the debug split, so a
stale debug build cannot retain ownership of installed files when
switching between the release and VCS packages. The `aur-publish`
workflow gains a validated `pkgrel` dispatch input so corrected
packaging can be republished against an existing release tag without
retagging.
Fixes#98
(cherry picked from commit 4cf550e23d)
The published v0.3.0 macOS installer is a structurally corrupt xar
archive: pkgutil and xar reject it even though its SHA-256 matches the
published checksum.
build-pkg.sh derived the architecture suffix in the .pkg filename from
`uname -m`. On the Apple-silicon macOS runner that always reports
arm64, so the cross-compiled x86_64 build also named its output
fips-<version>-macos-arm64.pkg. The release job downloads both build
artifacts with merge-multiple into one directory, where the two
identically named files collide and tear into a malformed result. The
x86_64 package never reaches the release at all.
Derive the package architecture from the Rust target triple, which is
authoritative for cross-compiles, instead of from the build host. Each
matrix leg now produces a distinctly named, arch-correct package, so
the two artifacts no longer collide.
Add a SHA-256 integrity chain so a corrupt or mismatched asset cannot
be published again:
- Capture the .pkg SHA-256 on the macOS runner, after the on-runner
structural verification, into a sidecar file carried in the artifact.
- Add a verify-handoff job that runs on every trigger and asserts each
downloaded .pkg still matches its macOS-runner SHA-256.
- Gate the release job on verify-handoff and repeat the check on the
exact bytes about to be published.
The build step now asserts it produced the expected arch-named package
so a regression in the naming fails loudly rather than as a silent
collision.
Relates to #102. The published v0.3.0 macOS assets still need to be
rebuilt and reuploaded separately.
Moves both AEAD layers (ChaCha20-Poly1305, one round per layer per
packet) plus the sendmsg syscall off the rx_loop task onto a per-shard
worker pool, adds per-peer connect(2)-ed UDP with SO_REUSEPORT, and
uses Linux UDP GSO (sendmsg+UDP_SEGMENT — kernel splits one super-skb
into N on-the-wire datagrams in a single TX-stack walk) when packets
in a batch are uniform-size. Same kernel primitive WireGuard's
in-kernel module and BoringTun use to hit 2.5–3.2 Gbps single-stream.
Single TCP stream on a 5-node docker-bridge mesh, 5 x 15 s x P=1:
A→D: 1379 → 2708 Mbps (1.96x, RTT +0.12 ms)
A→E: 1394 → 2663 Mbps (1.91x, RTT +0.11 ms)
E→A: 1406 → 2624 Mbps (1.87x, RTT +0.19 ms)
Static-peer pairs only — every CoV under 3%, 0 outliers, 0% ICMP
loss. The ~+100 µs RTT is the worker queue handoff cost; AEAD +
sendmmsg now run on a separate core in exchange.
What lands:
- src/node/encrypt_worker.rs: std::thread + crossbeam_channel
workers; hash-by-destination dispatch pins a TCP flow to one
worker so wire ordering is preserved; per-worker sendmmsg(2)
batching up to 32; Linux uses sendmsg(2)+UDP_SEGMENT when
packets in a group are uniform-size.
- src/node/decrypt_worker.rs: receive-side mirror. Each shard owns
its session's recv cipher + replay window in a thread-local
HashMap (no shared RwLock/Mutex). Sessions are handed off at
promote_connection and re-registered on K-bit flip / rekey
cutover.
- src/node/handlers/session.rs try_send_session_data_pipelined:
FSP+FMP both seal in-place in the worker on one wire-buffer
alloc; no intermediate inner_plaintext / fsp_payload Vecs.
- src/transport/udp/connected_peer.rs + peer_drain.rs: per-peer
connect(2)-ed UDP socket with SO_REUSEPORT (set on the listen
socket too — without that, EADDRINUSE on activation and every
packet falls back to the wildcard path); the worker sends with
msg_name=NULL and the kernel uses its cached 5-tuple. Tick-
driven activation in handlers/connected_udp.rs, idempotent.
- src/transport/udp/mod.rs: mem::replace the recvmmsg backing buffer
instead of buf.to_vec() per packet — single pointer swap, no
MTU-sized memcpy.
- src/protocol/link.rs SessionDatagramRef: zero-copy borrowed view
used by handle_session_datagram for the bulk local-delivery
path; handle_session_payload takes the borrowed payload
directly (no payload[35..].to_vec()).
- src/transport/mod.rs TransportAddr::from_socket_addr: collapses
the two-alloc from_string(addr.to_string()) pattern to one.
- src/node/handlers/rx_loop.rs: decrypt-fallback drain promoted
ahead of packet_rx in the select! (TCP ACK starvation fix);
interleaved fallback drain every 32 packets inside the rx burst
loop.
- noise::Session: send_cipher_clone / recv_cipher_clone /
recv_replay_snapshot_owned / take_send_counter / accept_replay
so off-task workers can hold a cloned cipher + reserved counter
while the dispatcher keeps replay/counter sequencing serial.
CipherState::cipher_clone returns a refcount-bumped LessSafeKey.
AsyncUdpSocket: AsRawFd so workers issue raw sendmmsg / sendmsg
without going through the tokio reactor.
- Worker pool sizing: both default to num_cpus, overridable via
FIPS_ENCRYPT_WORKERS=N / FIPS_DECRYPT_WORKERS=N. Per-peer
connected UDP can be disabled via FIPS_CONNECTED_UDP=0.
- src/perf_profile.rs: optional per-stage timing reporter under
FIPS_PERF=1 (or FIPS_PIPELINE_TRACE=1). Off by default; zero
overhead when disabled.
- All cfg(unix)-gated. Windows continues on the existing tokio-
based send/recv.
Decrypt worker session lifecycle:
- Node::unregister_decrypt_worker_session mirrors the existing
register helper. Wired at the two natural sites that already
iterate peers_by_index: the rekey drain-completion block in
handlers/rekey.rs (drops the worker entry for the old our_index
once the drain window has expired and the cache_key is
unreachable to any in-flight OLD-K packet), and remove_active_peer
in handlers/dispatch.rs (drops the worker entry for each of the
four index slots: current, rekey, pending, previous). Only
our_index is normally registered; unregister_session is fire-
and-forget for missing entries, so calling unconditionally on
all four slots is correct and bounds the cleanup without per-
slot accounting. Without these callers the per-worker sessions
HashMap and the Node's decrypt_registered_sessions set would
grow monotonically per rekey on long-lived peers.
Testing:
- testing/static/scripts/bench-multirun.sh: multi-run iperf3 +
ping bench. N reruns (default 5), median / min / max / CoV % /
per-run outlier flag, avg ping RTT, ICMP loss %, TCP retransmit
total. Plain client→dest labels + topology header. Pre-bench
peer-convergence check (FIPS_BENCH_CONVERGE_SECS, default 15);
per-path route verification via stats.bytes_sent deltas — fails
fast if traffic exits via a non-static-peer link.
- testing/static/docker-compose.yml: passes FIPS_ENCRYPT_WORKERS /
FIPS_DECRYPT_WORKERS / FIPS_PERF through to containers for A/B
benchmarking without rebuilds.
- testing/static/scripts/iperf-test.sh: same plain client→dest
labels + topology header (was multihop/direct/N hop, which
conflated topology distance with on-wire path).
- .config/nextest.toml: synthetic UDP node tests serialized
through a max-threads=1 test group. Localhost handshakes drop
on shared CI runners under parallel load; one-at-a-time keeps
assertions reliable.
- src/node/tests/spanning_tree.rs: repair_missing_edge_handshakes
— retries up to 5 times for synthetic edges whose msg1 was
dropped, with a drain after each edge retry instead of after
each attempt's full burst.
- src/node/decrypt_worker.rs::tests: two unit tests asserting
WorkerMsg::UnregisterSession removes the worker-thread session
HashMap entry (handle_msg_unregister_session_removes_entry) and
is a no-op for never-seen cache_keys
(handle_msg_unregister_session_idempotent_on_unknown_key), which
is the safety invariant the unconditional unregister calls at
the four index slots in remove_active_peer rely on.
- src/node/encrypt_worker.rs::unix_tests
pipelined_send_wire_layout_roundtrips_canonical_decoders: mirrors
the encoder geometry of try_send_session_data_pipelined (no
coords, the common established-session path), runs the worker's
real seal + send via flush_direct_batch_sync, and decodes the
resulting wire packet using only canonical receive-side decoders
(EncryptedHeader::parse, SessionDatagramRef::decode, FSP header
parse, noise::open). Any divergence between the hand-rolled
encoder offsets (fsp_aad_offset, fsp_plaintext_offset) and the
decoders fails at one of the parse / open / decode steps before
the inner-plaintext assertion fires. Complements the existing
fsp_preseal_runs_before_outer_fmp_seal test which covers the
seal-ordering invariant with synthetic headers but does not
exercise the wire-layout invariant.
CHANGELOG.md [Unreleased] # Changed entry added describing the
worker-pool threading model, hash-by-destination dispatch,
sendmmsg/UDP_GSO, per-peer connected UDP, the operator-facing env
vars, and the bench numbers above.
Cherry-picks from mmalmi/master (paths translated from
crates/fips-core/src/ to src/): 9b7c723, 0deb5cb, 13f7339, e036c0e,
3740a68, 3792f83, 8510193, 4910b07, e53f545, e4e2896, 5fe4af5,
1d01ada, 8c37008, e12469e, 6eb2860.
Co-authored-by: Johnathan Corgan <johnathan@corganlabs.com>
Publish the 13-criteria PR review checklist the maintainer runs on
every incoming PR so contributors (and their coding agents) can run
the same pass before opening, surfacing problems before the review
round trip. CONTRIBUTING.md gets a new 'Self-review against the
project review checklist' subsection under 'Submitting pull requests'
and a Further Reading entry. CHANGELOG [Unreleased] gets an Added
entry.
Two [Unreleased] / Changed entries that should have landed alongside
the originating commits but didn't:
- macOS recvmsg_x batched receive (originally 59225ccf): completes
the Linux-equivalent inbound batching shape on Apple builds. Now
sequenced before the rx zero-copy entry so the section reads as a
coherent receive-path progression.
- Platform-specific test-build warning cleanup (originally 6bd40640,
PR #93): non-behavioral; documents the gating decisions that keep
cross-platform builds warning-clean.
Two Fixed entries appended to [Unreleased]:
- The coord cache surgical invalidation (49bd2104): replaces the
global CoordCache::clear() at parent-switch / become-root /
loop-detection / root-change sites with two targeted methods
(invalidate_via_node, invalidate_other_roots). Preserves cache
entries that remain correct after the topology change.
- The rekey-test strict-ping retry (306e4555): Phase 1 / 3 / 5
per-pair pings now retry up to 4 attempts. Brings the ICMP-noise
miss-floor from ~33% per phase to ~3.2e-6 at 1% loss without
changing the failure-shape signal the asserts target. Test
scaffold only, no daemon code changes.
The non-Linux test build was emitting warnings from code that is
intentionally platform-specific: the nftables firewall parser is
Linux-only, the utun address-family helper is only used in macOS
TUN paths, and one macOS Ethernet test module trips a clippy
layout lint. These warnings made focused test runs noisy and
encouraged bundling unrelated warning fixes into behavioral PRs.
- Gate the firewall parser dead-code allowance to non-Linux
targets, where the parser is compiled but not used.
- Mark the macOS utun helper and long TUN reader entry point with
narrow allowances.
- Rewrite the small MAC-copy loop to satisfy clippy and mark the
macOS Ethernet test module layout explicitly.
No runtime behavior change.
The Phase 1, Phase 3, and Phase 5 strict asserts each fire a
single ping per directed pair. Under low-level packet loss
(e.g. 1% i.i.d. per-direction loss from a CI runner under
pressure), a single-shot round-trip fails at ~2% per pair, so a
20-pair strict assert misses with probability
1 - (0.98)^20 = ~33% per phase from ICMP noise alone, well above
the routing-state signal the asserts are meant to catch.
ping_one gains a max_attempts parameter (default 1, preserving
existing call sites). On failure it retries up to
MAX_PING_ATTEMPTS-1 additional times with PING_RETRY_DELAY
seconds between attempts. Per-pair worst case under the defaults
(4 attempts, 1 s spacing, 5 s ping6 -W timeout) is 4*5 + 3 = 23 s;
per-rep worst case scales with the failing-pair count.
Successful retries log "OK (RTT, attempt N)"; exhausted retries
log "FAIL (after N attempts)".
The retry budget is wired into all three strict asserts:
- Phase 1 final ping_all (after wait_for_full_baseline converges)
- Phase 3 ping_all (post-first-rekey)
- Phase 5 ping_all (post-second-rekey)
The wait_for_full_baseline convergence loop itself stays
single-shot. Its job is to detect when the mesh first sees a
fully clean 20-pair batch, and retries inside the loop would
conflate transient ping loss with still-converging routing
state.
No daemon code changes.
Replaces the unconditional `CoordCache::clear()` calls at parent-switch,
become-root, and loop-detection sites with two targeted invalidation
methods scoped to what actually makes an entry stale:
- `invalidate_via_node(node_addr)`: drop entries whose cached
destination ancestry contains `node_addr`. Used at parent-position-
change sites — our prefix changed, so destinations downstream of
us have stale-prefix coords.
- `invalidate_other_roots(current_root)`: drop entries rooted under
a different root than the current one. Used at root-change sites.
Under the previous global flush, parent switches blanked the cache
across the board, leaving `find_next_hop` returning `None` for every
non-direct-peer destination until the cache passively re-warmed via
incoming TreeAnnounces / SessionSetup. Surgical invalidation
preserves entries that remain correct after the topology change.
The cached coord describes a destination's tree position; that
position only goes stale relative to our own routing decisions when
our own prefix changes (entries we are downstream of) or the root
changes (entries in a different tree). Peer removal does not
invalidate cached coords: `Node::find_next_hop` recomputes the
next-hop decision on every call against the current peer set, bloom
filters, and tree state, and Discovery already triggers on
`no route to destination` errors when a destination becomes
unroutable through us. The peer-removal site retains the original
"no cache invalidation" behavior.
Each method returns the count of entries removed for observability.
Unit tests cover each method against the cases enumerated in the
acceptance criterion.
- Borrowed SessionDatagramRef decoder is used in the forwarding
handler so local delivery and coordinate-cache warming no longer
allocate or copy the session payload. The owned SessionDatagram is
materialized only when re-encoding for the next hop.
- Owned SessionDatagram::decode is reimplemented as Ref::decode +
into_owned, so the two decoders cannot drift.
- recvmmsg / recvmsg_x (Linux + macOS) receive loop moves each filled
slot buffer into ReceivedPacket via mem::replace instead of cloning
it; a fresh empty buffer is installed for the next syscall.
- TransportAddr is formatted directly from the SocketAddr without
going through an intermediate String.
Focused decode bench: ref 1.6 ns/op vs owned 34.7 ns/op (21.4x).
End-to-end iperf is neutral as expected for a ~30 ns saving per
packet.
Unit tests added:
- test_session_datagram_ref_decode_borrows_payload (verifies the
payload slice pointer equals the input slice's offset 35, a real
zero-copy invariant guard against accidental future to_vec)
- bench_session_datagram_decode_owned_vs_ref (ignored, run with
--ignored --nocapture)
- test_transport_addr_from_socket_addr
The Linux recv path drains up to 32 datagrams per kernel wakeup via
recvmmsg(2), amortising the per-syscall + per-task-wakeup cost across
the burst. macOS still fell through to single-packet recv_from, so
the same overhead capped inbound rate on Apple builds.
Add an equivalent batch path for Darwin using recvmsg_x(2). It is a
xnu-private syscall (not in the public SDK) but is the canonical
amortisation primitive on macOS — same shape used by quinn-udp for
the same reason. ABI is the public msghdr layout plus a trailing
msg_datalen (per-datagram bytes-received output), declared via
`unsafe extern "C"` against a local repr(C) `msghdr_x`.
Same `(count, kernel_drops)` contract as the Linux `recv_batch`. macOS
has no SO_RXQ_OVFL equivalent, so `kernel_drops` is always 0 — the
1Hz `sample_transport_congestion()` detector simply sees no kernel
drop signal on Apple hosts (it already tolerates that, since the
field has been 0 there pre-batching too).
cmsg buffer is intentionally null: we never consume ancillary data on
this path, and quinn-udp documents that `recvmsg_x` does not overwrite
`msg_controllen` on macOS 10.15+ (zeroed init is the only safe state).
The udp_receive_loop dispatch widens from cfg(linux) to
cfg(any(linux, macos)); the per-packet recv_from path is now used
only on the remaining unix targets (BSDs etc.) and Windows.
Add test_burst_recv_batch exercising 10 in-flight datagrams to
verify per-datagram boundaries and arrival order across the batch.
Add an ignored bench_udp_recv_amortization measuring recv-side
syscall amortization across 1/2/4/8 sender threads on dedicated
blocking std threads (kernel rx queue stays saturated regardless of
tokio scheduling). Sample numbers on aarch64-apple-darwin (100B
payloads, 3s windows):
senders=1: recv_from 398k pps recv_batch 432k pps 1.09x
senders=2: recv_from 353k pps recv_batch 608k pps 1.72x
senders=4: recv_from 322k pps recv_batch 503k pps 1.56x
senders=8: recv_from 353k pps recv_batch 515k pps 1.46x
Gate the Linux-only IpAddr import in control::listening behind a
cfg(target_os = "linux") so the macOS test build is warning-clean
now that test code paths there compile.
New testing/boringtun/ harness runs two Cloudflare BoringTun userspace
WireGuard containers with iperf3 between them, giving a single-hop
userspace tunnel baseline for comparison against FIPS throughput
numbers. Local WG key generation runs through the harness image so the
host needs no wireguard-tools.
New testing/static/scripts/iperf-compare-refs.sh builds two git refs
into separate fips-test:* images via git worktree and runs the same
static iperf topology against both, with RUNS-based repetition and
aggregate avg/min/max reporting.
testing/static/scripts/iperf-test.sh gains DURATION, PARALLEL,
SETTLE_SECONDS, IPERF_TIMEOUT env knobs and a per-path iperf timeout.
testing/static/docker-compose.yml selects the image under test via
FIPS_TEST_IMAGE; testing/scripts/build.sh respects CARGO_TARGET_DIR.
Author benchmark on aarch64 Docker Desktop:
boringtun bob -> alice : 1000.13 Mbits/sec
Skip BootstrapEvent::Established and BootstrapEvent::Failed dispatch
in poll_nostr_discovery for peers that are already connected or
actively handshaking. Without these guards, stale traversal events
arriving after a peer connected through a different path would
either attempt to adopt a redundant socket against the live
connection (Established) or poison the per-peer failure-state
cooldown and trigger redundant retraversal via schedule_retry /
try_peer_addresses (Failed).
The four guard sites use a new is_connecting_to_peer helper extracted
from the existing closure inside initiate_peer_connection; the helper
checks for an in-flight outbound handshake state. adopt_established_traversal
gains a defense-in-depth check returning PeerAlreadyExists when called
against an already-promoted peer, so the invariant holds if a future
caller bypasses the outer dispatch guard.
Side benefit: narrows a cooldown-poisoning vector previously available
to an attacker injecting stale failure events for an active peer.
Test coverage for the new behavior:
- test_try_peer_addresses_skips_connected_peer
- test_try_peer_addresses_skips_connecting_peer
- test_nostr_traversal_failure_skips_connected_peer (Failed-arm event
injection)
- test_nostr_traversal_established_skips_connected_peer (Established-arm
event injection, mirror of the Failed test)
- test_adopted_traversal_skips_already_connected_peer
(adopt_established_traversal defense-in-depth)
CHANGELOG entry under [Unreleased] / Fixed.
Closes#87
Bring [Unreleased] into sync with all maint commits since v0.3.0:
- Add a Fixed entry for the acl-allowlist test-script poll-assertion
conversion (commit e9dd316) that was previously missing.
- Add the AUR-publish workflow rewrite and new fips-git VCS workflow
(commit 9bf9701) which had no entry, and merge them with the
ci.yml cancel-in-progress block under a single "CI and
release-publish workflows hardened" entry so the three workflow
changes read as one operational theme.
Pure changelog content reshuffle. No code touched.
Add a top-level concurrency block to ci.yml keyed on
(workflow, ref) with cancel-in-progress: true. Pushes (including
force-pushes) to the same ref now retire any in-flight run for
that ref rather than letting the superseded and current-tip runs
both burn runner minutes.
Motivated by a 2026-05-14 force-push experience where two CI runs
ran concurrently against a feature branch — the original push at
c76ec99 continued for ~17 minutes alongside the amended push at
927ef47, despite only the latter being the live tip.
Scope deliberately limited to ci.yml. Tag-triggered release-build
workflows (package-*.yml, aur-publish-*.yml) are untouched — they
operate on per-tag refs that already form distinct concurrency
groups and release artifact builds generally should not be
cancellable by unrelated activity.
Phase 5's per-pair connectivity check ran immediately after the second
rekey cycle, with no settle for routing reconvergence. Under
GitHub-runner CPU contention, post-rekey parent-switches and
coord-cache flushes can take longer than the per-ping 5s timeout for
a small fraction of pairs (1-3 of 20 typically), even though the
rekey mechanism itself completes cleanly. Phase 6 log analysis stays
all-green on these failed runs; the failure is purely connectivity
timing.
Mirror Phase 3's existing 12-second settle pattern: reuse REKEY_SETTLE
and emit the same banner shape. Two-line change. Cost on the success
path is a fixed 12s per suite run.
Convert assert_log_contains from a one-shot grep snapshot into a
bounded poll that retries until the pattern appears or the timeout
elapses (default 15s). Same wait-with-timeout shape as
wait_for_peers_exact above it in the file.
The pre-existing flake on next-branch CI is structural: under XX
handshake, the cross-connection tie-breaker selects which side
reaches its ACL-check point first. When container-a wins the
tie-breaker on the first attempt against c and d, only the
outbound-handshake-context rejection fires immediately, and the
inbound-handshake-context rejection only emits on a later retry
when c or d's msg1 lands while a has no pending outbound. On one
2026-05-14 run the inbound rejection appeared 63ms after the test
had given up. The race window is small but real.
Polling the log instead of one-shot reading absorbs the
millisecond-to-second variance without slowing the success path
(the helper returns as soon as the pattern appears).
Add a per-session signed jitter offset (uniform [-15, +15] seconds)
to the rekey timer triggers in check_rekey (FMP) and check_session_rekey
(FSP). The configured `node.rekey.after_secs` becomes the nominal
interval rather than a floor; mean is preserved. Desynchronizes
both endpoints in symmetric-start meshes so the dual-initiation
race stops occurring rather than being resolved after the fact by
the smaller-NodeAddr tie-breaker.
Per-session storage means each rekey cutover reconstructs the
session and redraws the jitter naturally — successive cycles get
independent offsets, preventing drift back into sync.
Three entries under [Unreleased] for the three commits since the
v0.3.0 release tag:
- Sidecar example: FIPS_UDP_MTU env override (commit 32a3b58)
- CONTRIBUTING.md overhaul + new docs/branching.md (commit 538ce07)
- Rekey-test Phase 1 baseline-convergence headroom 36s -> 60s
(commit 6533276)
The previous CONTRIBUTING.md read like a stock Rust contributing
template that mentioned FIPS. Replace it with an entry-point doc
that gives new contributors the FIPS-specific mental model they
need to make a useful first PR: the FMP/FSP layering and why
mesh-level changes need multi-node testing, the three-branch
release model and how to choose a target branch, structured bug
reporting expectations, and PR submission requirements (scope
discipline, the local-CI ladder, separate requirements for feature
PRs vs bug-fix PRs, squash-merge mechanics).
Add a contributor-facing AI coding assistant policy: use is
welcome, but the contributor must do a thorough manual review and
editorial pass before submission. The agent is a tool; the
contributor is accountable for the submission. Review effort
scales with submission effort -- unreviewed agent output will
receive an agent reply in turn, without human review.
Add docs/branching.md as the long-form companion covering the
release workflow, version conventions, and merge-direction
rationale. The new CONTRIBUTING.md is the day-to-day entry point;
docs/branching.md is the reference.
All cross-references resolve against current HEAD. Previous
stale links to fips-intro.md, fips-wire-formats.md, and
fips-configuration.md are gone; the doc points at the current
docs/design/ layout, docs/getting-started.md,
docs/tutorials/join-the-test-mesh.md, and testing/README.md.
No code changes.
The Phase 1 pre-rekey baseline in `wait_for_full_baseline`
occasionally times out on GitHub-hosted runners with one ping pair
failing to converge inside the BASELINE_CONVERGENCE_TIMEOUT window.
Phases 2–6 always pass cleanly when this happens — the rekey itself
is fine, the mesh just hasn't finished spanning-tree + bloom-filter
convergence by the time Phase 1 starts pinging.
The wait loop returns as soon as all 20 pairs converge, so the cost
on the success path is unchanged (typical local CI returns well
under the old 36s). The bump only adds headroom on the failure
path. Operators previously worked around this with
`gh run rerun --failed`; this aims to retire that workaround for
the IK/XK lines.
Distinct from the next-branch XX rekey dual-init race, which is a
real protocol bug tracked separately.
The sidecar entrypoint hardcoded `udp.mtu: 1472`, the Docker-bridge
IPv4 maximum (1500 MTU - 8 UDP - 20 IPv4 header). Promote it to
`FIPS_UDP_MTU` (defaulting to 1472, preserving behavior) so non-Docker
reuses of the example can override without editing the script. Plumb
the env var through `docker-compose.yml` so a host-level setting
reaches the container, and add a row to the README's env-var table.
Also annotate the static-CI node template with a comment explaining
the same 1472 rationale and the daemon's 1280 default. The template
keeps 1472 as the literal value since the CI suite runs on a Docker
bridge where that's correct.
No behavior change unless the host explicitly sets FIPS_UDP_MTU.
The v0.3.0 stable AUR push silently failed: with updpkgsums: true,
makepkg downloaded fips-<ver>.tar.gz into the AUR working tree, where
it was then staged by the deploy action and rejected by AUR's 488 KiB
max-blob hook.
Fetch the upstream source tarball and compute its b2sum in CI, patch
pkgver and the b2sums SKIP placeholder in PKGBUILD in-place, then
publish with updpkgsums: false so the AUR clone stays metadata-only.
Recompute and patch the fips.sysusers / fips.tmpfiles asset b2sums in
the same step so they stay in sync with the local files; this safety
net was previously provided by updpkgsums.
Add aur-publish-git.yml for the VCS fips-git PKGBUILD, triggered on
master pushes that touch PKGBUILD-git or its companion files plus
workflow_dispatch. pkgver is computed at build time by the PKGBUILD's
pkgver() function, so this workflow is not tied to release tags.
Add a workflow_dispatch tag input on the stable workflow so historical
release tags can be re-published manually, and drop continue-on-error:
true so future regressions surface in CI.
Marker merge to record the maint dev-line as known on master
without taking maint's v0.3.1-dev opening commit (master is on
v0.4.0-dev). Future bug-fix forward-merges from maint to master
land cleanly on top of this base.
Reset maint to v0.3.0 to retire the v0.2.x maintenance window and
open a new tracking branch for v0.3.1-dev bug-fix work.
- Cargo: 0.3.0 → 0.3.1-dev
- CHANGELOG: fresh [Unreleased] block above [0.3.0]
- README: badge v0.3.0 → v0.3.1--dev
2026-05-12 13:36:22 +00:00
460 changed files with 82322 additions and 22032 deletions
maint ────────●──●──●──●──●────────────── (bug fixes for the latest release)
```
### maint
- Reset to each minor release tag at release time
- Accepts only bug fixes for functionality that shipped in the
latest release
- No new features, no API changes, no wire-format changes
- Patch releases tag from here (e.g., `v0.3.1`, `v0.3.2`)
- Periodically merged forward into `master` so fixes propagate
### master
- Compatible development for the next feature release
- Multiple feature releases may ship from master (`v0.4.0`, `v0.5.0`)
before `next` promotes
- No wire-format breaking changes; no API breaks
- Receives merges from `maint` so released-line fixes flow forward
- Periodically merged forward into `next`
### next
- Accumulates work that breaks wire format, API, or compatibility
- Receives merges from `master` so it stays current with bug fixes
and compatible feature work
- Cargo version on `next` is the expected release version with a
`-dev` suffix, updated if `master` ships additional minor
releases first
- Becomes the new `master` at the next breaking release; at the same
point the old `master` becomes the new `maint`
## Versioning
While the project is in the `0.x` era, semver treats minor bumps as
potentially breaking. Both `master` and `next` bump the minor version;
the distinction between compatible and breaking is captured in the
changelog and in which branch the work landed on.
The `-dev` suffix in `Cargo.toml` indicates an unreleased development
state on the branch.
## Merge Direction
Fixes and features flow in **one direction only**: `maint → master → next`.
Never merge backward (`next` into `master`, or `master` into `maint`).
```text
maint ──→ master ──→ next
```
This guarantees:
- Bug fixes shipped in a release reach all subsequent branches
- Compatible features reach `next`
- Wire-format-breaking work stays isolated on `next` until release
If you submit a PR on `next` that should also be on master or maint
(rare, since the criteria for needing it on multiple branches are
usually mutually exclusive), the PR stays on its target; the
maintainer either backports as a separate commit on the upstream
branch or asks you to.
## Choosing a Branch for Your PR
Pick the branch that matches the scope of your change:
| Your change | Target branch | Why |
| --- | --- | --- |
| Bug fix in a feature that shipped in the latest release | `maint` | Fix forward-merges to `master` and `next` |
| Bug fix in code added on `master` since the last release (not in any released version) | `master` | The released v0.x.y line is unaffected, so `maint` does not need the change |
| Bug fix in code added on `next` (wire-format-breaking work) | `next` | The bug only exists where the breaking work exists |
| New feature that does not break wire format or API | `master` | Becomes part of the next compatible release |
| Wire-format breaking change, API break, or fundamental protocol shape change | `next` | Stays isolated until the next forklift release |
| Documentation, CI, or contributor-facing changes | `maint` if they apply to released material, else `master` | Forward-merges propagate naturally |
If you are not sure, ask in the related issue. The safest defaults
are `master` for new features and `maint` for bug fixes; the
maintainer will retarget the PR if needed.
## Release Workflow
### Bug fix release (from `maint`)
1. Fix on `maint`
2. Bump patch version, tag (e.g., `v0.3.1`)
3. Merge `maint` into `master`
4. Merge `master` into `next`
### Compatible feature release (from `master`)
1. Finalize features on `master`
2. Merge `maint` into `master` to pick up any pending fixes
3. Set version, tag (e.g., `v0.4.0`)
4. Reset `maint` to the new tag
5. Bump `master` to the next `-dev` version
6. Merge `master` into `next`
### Breaking release (from `next`)
1. Finalize features on `next`
2. Merge `master` into `next` to pick up pending fixes and features
3. Assign version as the next minor after `master`'s last release, tag
4.`master` becomes the new `maint`
5.`next` becomes the new `master`
6. Create a new `next` branch from `master`
## Practical Guidelines
- **Commit to the appropriate branch for the scope of the change.**
Do not commit bug fixes to `master` when they apply to the latest
release — put them on `maint` and let the forward-merge propagate.
- **Feature branches base off the long-lived branch they target.**
Create with `git checkout -b my-feature maint` (or `master` or
`next`), not `git checkout -b my-feature origin/maint`. The
`origin/`-prefixed form auto-sets the new branch's upstream to
the source ref, which can cause `git push` to land on the wrong
ref under some configurations.
- **When in doubt about whether a change is compatible**, target
`next`. The maintainer can advise on retargeting.
- **Resolve merge conflicts on the receiving branch**, preserving
both the inherited fix and the new development.
- **PRs are merged via squash-merge.** One logical change per PR
becomes one commit on the destination branch, making bisect
| `node.discovery.lan.enabled` | bool | `false` | Master switch. LAN discovery is opt-in; default-off avoids an unexpected per-link identity multicast on upgrade. |
| `node.discovery.lan.service_type` | string | `_fips._udp.local.` | DNS-SD service type. Overridable mainly so integration tests can isolate multiple services on one loopback interface. |
| `node.discovery.lan.scope` | string (optional) | unset | Application/network scope carried in the LAN-only `scope` TXT record. Kept deliberately separate from the public Nostr `app` tag. When unset, the scope falls back to the derived Nostr `app` value. |
The identity surface published over mDNS (`npub`, version, optional
scope) is a strict subset of what `nostr.advertise` already publishes
publicly, so enabling LAN discovery adds no marginal privacy cost
beyond making the node's presence observable on its own local link.
### Relationship to Nostr discovery
The two mechanisms are complementary and independent:
| | Nostr-mediated | LAN/mDNS |
| --- | --- | --- |
| Reach | Internet-wide, via relays | Same broadcast domain only |
| Signaling channel | Public Nostr relays | mDNS multicast on the local link |
| NAT traversal | STUN + UDP hole-punch for `udp:nat` peers | None — endpoint is link-routable by construction |
| Identity carrier | signed kind 37195 advert (authenticated at publish) | unauthenticated mDNS TXT (routing hint only) |
| Identity proof | FMP Noise IK on the connection | FMP Noise IK on the connection |
| [enable-nostr-discovery.md](enable-nostr-discovery.md) | Turn on Nostr-mediated discovery (3 capabilities — resolve, advertise, open — across 5 scenarios) |
| [deploy-tor-onion.md](deploy-tor-onion.md) | Run a Tor onion service for inbound FIPS connections |
| [tune-udp-buffers.md](tune-udp-buffers.md) | Set host sysctls so FIPS UDP sockets don't get clamped |
| [tune-file-descriptors.md](tune-file-descriptors.md) | Raise `RLIMIT_NOFILE` so a busy node doesn't exhaust file descriptors (`EMFILE`) as peer count grows |
| [run-as-unprivileged-user.md](run-as-unprivileged-user.md) | Run the daemon under a dedicated unprivileged service account (drops the default-root posture) |
| [deploy-gateway.md](deploy-gateway.md) | Manually deploy `fips-gateway` on a non-OpenWrt Linux host (LAN-to-mesh outbound + mesh-to-LAN inbound port-forwards). For the OpenWrt path, see the gateway tutorial. |
| [troubleshoot-gateway.md](troubleshoot-gateway.md) | Diagnostic recipes for the gateway, organised by half (outbound, inbound, common) |
| [persistent-identity.md](persistent-identity.md) | Provision a stable Nostr keypair so the node keeps the same npub across restarts |
| [host-aliases.md](host-aliases.md) | Use shortnames (`test-us01.fips`, `my-laptop.fips`) instead of full npubs by editing `/etc/fips/hosts` or setting peer aliases |
| [set-up-bluetooth-peer.md](set-up-bluetooth-peer.md) | Configure a Bluetooth Low Energy peer link |
| [set-up-80211s-mesh-backhaul.md](set-up-80211s-mesh-backhaul.md) | Link OpenWrt FIPS routers over an open 802.11s radio backhaul (FIPS provides encryption, authentication, and routing) |
| [set-up-open-access-ssid.md](set-up-open-access-ssid.md) | Broadcast the open `!FIPS` access SSID so phones and laptops roam onto the mesh (one ESS: save once, roam every FIPS router) |
| [diagnose-mtu-issues.md](diagnose-mtu-issues.md) | Triage MTU-shaped failures and rule out their imposters (bufferbloat, transport saturation) |
| `stats list` | `show_stats_list` | Enumerate available metrics, their units, and the per-ring retention windows. |
| `stats metrics` | `show_metrics` | Dump current counter values for every protocol metric family (`forwarding`, `discovery`, `tree`, `bloom`, `congestion`, `errors`). |
| `stats peers` | `show_stats_peers` | List peers tracked in stats history (active or recently active). |
| `stats history <metric> [options]` | `show_stats_history` | Fetch a time-series window for one metric. |
@@ -104,7 +105,7 @@ Tell the daemon to dial a peer over a specific transport.
| -------- | ----------- |
| `peer` | npub (bech32) or hostname from `/etc/fips/hosts`. |
| `address` | Transport endpoint, e.g. `192.168.1.10:2121`, `[2001:db8::1]:2121`, or a Tor onion. FIPS-mesh ULAs (`fd00::/8`) are rejected for the IP-based transports (udp, tcp, ethernet). |
| `transport` | One of `udp`, `tcp`, `tor`, `ethernet`. |
| `transport` | One of `udp`, `tcp`, `tor`,`nym`,`ethernet`. The named transport must be configured and running. |
### `disconnect <peer>`
@@ -114,6 +115,58 @@ Tell the daemon to drop a peer link.
| -------- | ----------- |
| `peer` | npub (bech32) or hostname from `/etc/fips/hosts`. |
### `profile tick <on|off|status>`
> **Reading the output.** Step durations are wall clock measured across `await`
> points, not CPU time: a step that waits on I/O accrues that wait, and other
> tasks may run inside the span. That is the intended measure for head-of-line
> delay, and it means a large step is not necessarily an expensive one.
> `arm_starvation` is measured directly as the entry time minus the deadline
> the interval scheduled that tick for. It is not derived from
> `tick_entry_gap`, which carries no starvation signal on its own: under a
> steady delay every gap is exactly one tick period.
Start, stop and inspect a capture of the rx-loop tick body. **Present
only when both `fipsctl` and the daemon are built with
`--features profiling`**; the feature is off by default, so a stock
package does not carry this subcommand and a stock daemon reports
| `profile tick on` | `profile_tick_on` | Create the capture file and start recording. Fails if a capture is already running (naming the active file) or if the directory cannot be written. |
| `profile tick off` | `profile_tick_off` | Stop the capture. The writer is woken immediately, drains once more and is joined, so the command returns promptly. Succeeds, reporting nothing active, when no capture is running. |
| `profile tick status` | `profile_tick_status` | Report `idle`, `running`, `stopped_by_cap` or `stopped_by_error`, plus the active path, bytes written, flush interval and byte cap. |
`profile tick on` options:
| Flag | Argument | Default | Description |
| ---- | -------- | ------- | ----------- |
| `--dir` | directory path | `/var/log/fips` | Where to write the capture. Created if absent. Use it to profile a non-root `cargo run`, or on a platform whose log root differs. |
One file is written per capture, named `profile-<UTC timestamp>.tsv`.
It opens with a `#`-prefixed header block (node npub, build version,
@@ -235,6 +235,29 @@ addresses for the punch socket port.
During punching, compatible private-subnet candidates and reflexive candidates
are attempted in parallel; the first successful path wins.
#### LAN Discovery (`node.discovery.lan.*`)
Peer discovery on the local link via mDNS / DNS-SD (RFC 6762 / RFC
6763). When enabled, the node publishes a `_fips._udp.local.` service
advert carrying its `npub` (and optional scope) and concurrently
browses for the same service type to learn same-broadcast-domain peers.
The result is sub-second peer pairing with no Nostr-relay roundtrip,
STUN observation, or NAT traversal: the observed endpoint is by
construction routable from the consumer's LAN.
mDNS adverts are unauthenticated, so a LAN advert is treated only as a
routing hint. Identity is still proven end-to-end by the Noise XX
handshake the node initiates against the observed endpoint; a spoofed
advert carrying another peer's npub fails the handshake and is dropped.
LAN discovery requires an active UDP transport (peers dial the
advertised UDP port to begin the handshake).
| Parameter | Type | Default | Description |
|-----------|------|---------|-------------|
| `node.discovery.lan.enabled` | bool | `false` | Master switch. Opt-in: enable for sub-second same-LAN pairing. Default-off avoids reintroducing a per-LAN identity broadcast on nodes that have deliberately disabled other discovery channels |
| `node.discovery.lan.service_type` | string | `"_fips._udp.local."` | DNS-SD service type. Primarily an override for integration tests running multiple isolated services on one loopback interface; leave at the default in production |
| `node.discovery.lan.scope` | string | *(none)* | Optional application/network scope carried in a `scope=<name>` TXT entry. Browsers with a scope set only surface peers advertising the same scope, so nodes on the same physical LAN configured for different mesh networks do not cross-feed. Intentionally separate from `node.discovery.nostr.app` so relay-visible adverts can stay generic while LAN discovery is isolated per private network |
### Spanning Tree (`node.tree.*`)
Controls tree construction and parent selection.
@@ -254,7 +277,7 @@ Controls tree construction and parent selection.
| `node.bloom.max_inbound_fpr` | f64 | `0.05` | Antipoison cap: reject inbound `FilterAnnounce` frames whose advertised false-positive rate exceeds this value. Valid range `(0.0, 1.0)`. The default `0.05` corresponds to fill 0.549 at k=5 (≈3,200 entries on the 1 KB filter) |
| `node.bloom.max_inbound_fpr` | f64 | `0.20` | Antipoison cap: reject inbound `FilterAnnounce` frames whose advertised false-positive rate exceeds this value. Valid range `(0.0, 1.0)`. The default `0.20` corresponds to fill 0.7248 at k=5 (≈2,114 entries on the 1 KB filter); a saturated/poisoned filter is still ~100% FPR and rejected |
Bloom filter size (1 KB), hash count (5), and size classes are protocol
constants and not configurable.
@@ -413,7 +436,7 @@ Requires `CAP_NET_RAW` or running as root. Linux only.
| `mtu` | u16 | *(auto)* | Override MTU. Default: interface MTU minus 3 (for frame type + length prefix) |
| `transports.nym.connect_timeout_ms` | u64 | `300000` | Outbound connect timeout in milliseconds. Mixnet SOCKS5 connections traverse 3 mix nodes with timing obfuscation and can take several minutes, so this is generous (300s). |
| `transports.nym.mtu` | u16 | `1400` | Default MTU |
| `transports.nym.startup_timeout_secs` | u64 | `120` | Seconds to wait for `nym-socks5-client` to become ready at startup before giving up |
**Named instances.** Like other transports, multiple Nym instances can
be configured with named sub-keys for different SOCKS5 proxy endpoints.
### BLE (`transports.ble.*`)
Bluetooth Low Energy transport using L2CAP Connection-Oriented Channels.
@@ -798,7 +840,7 @@ peers:
### Mixed UDP + Ethernet Example
A node bridging internet peers (UDP) and a local Ethernet segment with
| `show_status` | — | `version`, `npub`, `node_addr`, `ipv6_addr`, `state`, `is_leaf_only`, `is_root` (bool — this node is the spanning-tree root), `root` (hex node-addr of the current tree root), `persistent` (bool — identity is persisted, i.e. `persistent` set or an `nsec` configured), `peer_count`, `session_count`, `link_count`, `transport_count`, `connection_count`, `transport_peer_counts` (object mapping transport-type name to its connected-peer count; configured transports appear with `0`), `tun_state`, `tun_name`, `effective_ipv6_mtu`, `control_socket`, `pid`, `exe_path`, `uptime_secs`, `estimated_mesh_size`, `forwarding`, `sparklines`. |
| `show_peers` | — | `peers[]` — per-peer object: `node_addr`, `npub`, `display_name`, `ipv6_addr`, `connectivity`, `link_id`, `direction`, `transport_addr`, `transport_type`, `is_parent`, `is_child`, `tree_depth`, `effective_depth` (`tree_depth + link_cost` — the metric `evaluate_parent` ranks on; `null` when the peer has no coords, or is unmeasured while another peer has an SRTT sample, per the cold-start gate), `stats`, `noise`, `current_k_bit`, `mmp`, plus optional `nostr_traversal`, `rekey_in_progress`, `rekey_draining`. |
| `show_bloom` | — | `own_node_addr`, `is_leaf_only`, `sequence`, `leaf_dependent_count`, `leaf_dependents[]`, `peer_filters[]`, `uptree_fill_ratio` (fill ratio of the last filter actually sent to the tree parent), `uptree_estimated_count` (cardinality estimate of that uptree filter — this node's whole subtree under split-horizon, not the mesh; both are `null` for a root node or before the first announce), `stats`. |
| `show_mmp` | — | `peers[]` (link-layer per peer), `sessions[]` (session-layer per session). Each entry includes loss/RTT/ETX/goodput, smoothed values, trends. |
| `show_metrics` | — | Flat snapshot of every counter family in the metrics registry: `forwarding`, `discovery`, `tree`, `bloom`, `congestion`, `errors`. Each value is that family's counter snapshot object. Counter-only — gauges/histograms that need the live node are excluded. Served off the main loop. Silent-rejection sites classify their reason as a typed `RejectReason` and increment the matching per-family counter exposed here — see [Rejection reasons](#rejection-reasons). |
Each rejection increments the corresponding counter in its family's
stats, surfaced through `show_metrics` (the `tree`, `bloom`,
`discovery`, and `forwarding` families carry their own counters; the
`errors` family and the remaining subsystem counters carry the rest).
The full per-family variant list lives in `src/node/reject.rs`; it is
not reproduced here to avoid duplicating the source.
### Mutating commands
| Command | Required params | Behaviour |
| ------- | --------------- | --------- |
| `connect` | `npub` (bech32), `address` (transport endpoint), `transport` (`udp`, `tcp`, `tor`, `ethernet`) | Asks the node to dial the peer over the named transport. Returns the API result on success or an error string on failure. |
| `connect` | `npub` (bech32), `address` (transport endpoint), `transport` (`udp`, `tcp`, `tor`,`nym`,`ethernet`) | Asks the node to dial the peer over the named transport. The named transport must be configured and running. Returns the API result on success or an error string on failure. |
| `disconnect` | `npub` (bech32) | Asks the node to drop the link to the named peer. |
Both commands run on the daemon's main task and may block briefly
| `profile_tick_on` | `dir` (optional directory path; default `/var/log/fips`) | Creates the capture file, publishes its path, and starts the writer thread. `data`: `state`, `path`, `interval_secs`, `byte_cap`. Errors if a capture is already running (naming the active file) or the directory is unwritable. |
| `profile_tick_off` | — | Stops the capture, drains once more, joins the writer. `data`: `state`, `stopped`, `stopped_by_cap`, `stopped_by_error`, `path`, `bytes`. |
# fakeroot makes the packaged files root-owned even though CI runs unprivileged.
DESCRIPTION="FIPS Mesh Network Daemon. Distributed, decentralized mesh networking over UDP, TCP, and raw Ethernet, with a TUN interface (fips0), ULA IPv6 addressing, and a .fips DNS responder."
/// Render a sequence of values as Unicode block characters.
///
/// Returns an empty string for empty input. Constant series render as a
Some files were not shown because too many files have changed in this diff
Show More
Reference in New Issue
Block a user
Blocking a user prevents them from interacting with repositories, such as opening or commenting on pull requests or issues. Learn more about blocking a user.
Johnathan Corgan
ArjenandGitHub
sandwichandJohnathan Corgan
olekskyandJohnathan Corgan
Martti MalmiandJohnathan Corgan