laantungir/fips
1
0
Fork 0
mirror of https://github.com/jmcorgan/fips.git synced 2026-08-01 20:36:15 +00:00
Code Issues Packages Projects Releases Wiki Activity

Compare commits

base: laantungir/fips:v0.4.0-rc1
Branches Tags
laantungir/fips:master laantungir/fips:feat/platform-peer-queue laantungir/fips:next laantungir/fips:maint laantungir/fips:ble-v2
laantungir/fips:v0.4.1 laantungir/fips:v0.4.0 laantungir/fips:v0.4.0-rc2 laantungir/fips:v0.4.0-rc1 laantungir/fips:v0.3.0 laantungir/fips:v0.2.1-rel laantungir/fips:v0.2.1 laantungir/fips:v0.2.0-rel laantungir/fips:untagged-837413e700b3aaa62052 laantungir/fips:v0.1.0
..
compare: laantungir/fips:maint
Branches Tags
laantungir/fips:feat/platform-peer-queue laantungir/fips:next laantungir/fips:master laantungir/fips:maint laantungir/fips:ble-v2
laantungir/fips:v0.4.1 laantungir/fips:v0.4.0 laantungir/fips:v0.4.0-rc2 laantungir/fips:v0.4.0-rc1 laantungir/fips:v0.3.0 laantungir/fips:v0.2.1-rel laantungir/fips:v0.2.1 laantungir/fips:v0.2.0-rel laantungir/fips:untagged-837413e700b3aaa62052 laantungir/fips:v0.1.0
102 Commits
v0.4.0-rc1 .. maint
Author SHA1 Message Date
Johnathan Corgan 13a98ae702 Make a half-scoped reap impossible in ci-cleanup.sh 
The reap runs two selectors, one on the CI label and one on the
compose project, and each flag narrows only its own. So --run-id
alone leaves the project sweep broad and --project-prefix alone
leaves the label sweep broad, and either way the reap still destroys
every concurrent run on the host. Both single-flag forms look scoped
and are not.

The usage text asserted otherwise. It documented --run-id as leaving
other runs alone and --project-prefix as scoping the reap to a single
run, and both claims were false for the same reason. A caller passing
--run-id alone, on the strength of that line, force-removed the
containers of three concurrent runs on 2026-07-29, and its own
comment recorded the belief that it was scoped.

Rather than ask every caller to remember the pair, close the two
half-scoped states here. --run-id now derives the matching project
prefix when none is given, built from the existing base so the two
cannot drift, and an explicit --project-prefix still wins.
--project-prefix without --run-id is refused outright, because there
is nothing to derive a label scope from and the quiet failure is
someone else's run disappearing.

Passing neither flag is untouched: that is the deliberate "reap
everything" form a manual cleanup wants.

Verified against a decoy container labelled as another run: the
scoped reap leaves it up, and reproducing the old broad project sweep
removes it, so the check discriminates rather than passing because
there was nothing to reap. An earlier version of that check ran when
no CI containers existed at all and passed without proving anything.
 2026-07-29 02:09:59 +00:00
Johnathan Corgan 77ed64bb88 Compute every peer's outgoing bloom filter in one sweep 
Each peer must be sent the union of all other peers' inbound filters,
excluding its own contribution. Building that per recipient rebuilt
the whole peer-filter map and re-ORed it once for every peer, so a
tick that announced to R peers did R kilobyte-scale map builds and
R by T merges. At 240 peers this was 20.6 ms per tick, roughly half
the tick body, and it was steady work rather than a tail: the
per-interval maximum had a median of 34.5 ms.

Replace it with a prefix and suffix union sweep that produces every
target's filter in one pass, T merges instead of R by T. The packet
path gets the same treatment, since marking changed peers had the
identical shape once per inbound announce.

The result is exactly equal, not approximately. Merging is a bytewise
OR, so regrouping the unions cannot change the outcome, and a filter
whose size does not match is rejected before any byte is touched, at
every merge site in both the old and new arrangement. Every
accumulator here is default-sized, so an odd-sized peer filter is
skipped in the new code exactly where it was skipped in the old.

Cadence, the debounce, the sequence rule and the fill-ratio cap are
untouched. A sequence number is still drawn after the debounce
re-check and before encoding, so a suppressed peer still consumes
none and an encode failure still burns one.

Known trade-off, measured rather than assumed: the sweep does its
full O(T) work regardless of how many peers are ready, so a tick that
announces to only one or two peers now costs about twice what it did.
Break-even is around three ready peers, and the saving above that
grows without bound. The marking path is a pure win, since it always
targets every peer.
 2026-07-29 01:45:12 +00:00
Johnathan Corgan bf2e7a0892 Derive each peer's npub once instead of once per tick 
The per-tick stats snapshot ran a bech32 encode for every tracked
peer, and for the common mesh peer — one with no hosts-file entry and
no configured alias — it ran a second one, because the display-name
fallback chain bottoms out in the same encode. At 240 peers that was
14.1 ms per tick, a third of the tick body and its second largest
cost, all of it recomputing values that cannot change.

Cache the npub and the shortened npub on the peer at construction. An
npub is a pure function of the peer's public key, and the identity is
never mutated after construction: there is no setter, no identity_mut,
and no assignment to the field anywhere in the tree, so the cache
cannot go stale.

The display name itself is deliberately NOT cached. Two of its inputs
do mutate at runtime — the alias map and the host map, the latter
reloaded on this same tick — so a resolved name stored on the peer
would go stale on an alias change or a hosts reload. Only the
immutable component is memoized.

Tests cover both constructors, that the cached npub matches the
identity, and that the display name still tracks an alias change. The
memoization itself is asserted by pointer stability rather than by
timing, so it is deterministic under load. Three deliberate breaks
were each caught by exactly one test: re-deriving instead of
memoizing, populating one constructor's cache from the wrong source,
and reordering the display-name fallback so it stops honoring aliases.
 2026-07-29 01:32:48 +00:00
Johnathan Corgan d6457fa74f Stop awaiting the advert refetch on the retry tick 
process_pending_retries runs inline on the node's 1-second rx-loop
tick. For each due peer it awaited a Nostr relay fetch carrying a
2-second timeout, and discarded the result. With up to sixteen due
peers in one tick body, the timeouts stack: field profiling measured
single 2.00 s stalls as the common case and a worst tick of 12.4 s
against a 1 s period, with every other rx-loop arm delayed behind it
by as much as 4.2 s.

Spawn the refetch instead of awaiting it, matching the pattern the
failure arm of the same loop already uses thirty lines below. The dial
now uses whatever advert is cached at that moment and the refreshed
one lands for the next retry of that peer. Since retries are
backoff-paced, that defers the benefit by one backoff interval rather
than losing it, and the result was already being discarded, so nothing
downstream read it.

The test drives four due peers whose refetches all hang against a
local listener that accepts and never speaks, so the fetch burns its
full timeout with no network egress. Awaited, the call takes 8.0 s;
spawned, it returns in milliseconds. It also asserts every due peer
was still attempted and rescheduled, so a version that skipped the
dial entirely cannot pass it.
 2026-07-29 01:27:49 +00:00
Johnathan Corgan 7d0a110f4e Send traversal signals only to relays the client pool actually holds 
A traversal signal is addressed to a merge of the peer's NIP-17 inbox
relays, the relays its advert nominates for signaling, and our own DM
relays. The client pool is built once at startup from our configured
relays and never added to, and send_event_to rejects the entire send
with "relay not found" if any single URL in the list is outside that
pool, before contacting anything. So one relay we are not configured
with, anywhere in that merge, killed the whole attempt -- including the
sends to relays we do share and that would have carried the signal.

In an open-mode window on a public node this made discovery
non-functional: 309 traversal attempts, 290 explicit failures, zero
successes, every failure on "relay not found". Configured peers were
unaffected because they run a matching relay set.

Filter the merged list down to relays the pool holds before sending.
Our own DM relays are always in the merge and always in the pool, so
the result is empty only when no DM relay is configured at all, which
is already a total failure. Comparison is on the normalized RelayUrl
rather than the raw string, because that is how the pool is keyed --
a raw comparison would discard a configured relay spelled with a
trailing slash or a different host case, which is the same defect in
a quieter form.

Merge and filter are one synchronous function so the decision can be
exercised without a relay client. The pool is read via all_relays(),
which is the set send_event_to validates against; relays() is filtered
by service flags and would be narrower.

Two smaller fixes ride along. The responder now resolves its relays
before binding a socket and running STUN, instead of spending a STUN
round trip and holding an offer slot only to discover it has nowhere
to answer. And it gained the empty-list guard the initiator already
had, which gives BootstrapError::MissingRelays a condition it can
reach for the first time -- it was unreachable, since the merge always
appended our own DM relays.
 2026-07-29 00:31:36 +00:00
Johnathan Corgan 2fdc831ce3 Claim the NAT lab's two bridges per run and derive every address from them 
The NAT lab was the last suite pinning fixed IPv4 subnets, so two overlapping
runs collided on the wan and shared-lan bridges.

Each run now claims a free /24 for each bridge, scanning candidates and
advancing on an overlap while still failing fast on any other network-create
error. The claim exports NAT_WAN_PREFIX and NAT_LAN_PREFIX, and every routable
address in the compose file and the suite scripts derives from them, with
defaults that render exactly what the lab used before. The router-side LANs are
deliberately left pinned: they live inside per-container network namespaces,
never become docker networks, and cannot collide.

An external-network overlay lets the suites attach to the networks the run
already claimed instead of creating their own. Two of the three suite scripts
had no overlay hook, so they would have requested the claimed range a second
time; both now have one.

Host veth names are scoped by a short token rather than the full run id, which
overruns the fifteen-character interface-name limit at the default run-id
length, and the cleanup reaper's pattern is widened to match the new shape.

Networks are released inline on every exit path rather than in a trap, since a
trap written inside a shell function replaces the script-level handler and
would disable the whole run's teardown.
 2026-07-27 14:34:46 +00:00
Johnathan Corgan fbff52d85e Give the NAT lab per-run generated configs 
The NAT lab wrote its generated node configs to one shared directory, unlike
the static and firewall labs which already scope theirs by run. The directory
is bind-mounted by compose and read back by the suite scripts after the
containers are up, so two overlapping runs let the second run's generator
overwrite the npubs the first is about to ping.

Scope the directory with FIPS_CI_NAME_SUFFIX at all three places that have to
agree: the generator's output path, the ten compose bind-mounts, and the
CONFIG_DIR the suite scripts read back from. An unset suffix renders the plain
path the lab has always used, so a bare invocation and the GitHub matrix are
unaffected. The run teardown removes the per-run directory alongside the static
and firewall ones, and the gitignore is widened to cover the suffixed form.

Rendering the compose file with every profile and no suffix set reproduces
today's exact ten paths; setting a suffix moves all ten.

This lands on its own, ahead of the network and address work, so that a failure
of the two-overlapping-runs acceptance test can be bisected between the config
fix and the address conversion.
 2026-07-27 14:34:33 +00:00
Johnathan Corgan 93800a503e Retire the bloom-storm chaos scenario from both CI runners 
The scenario guarded a real regression: a mid-chain tree update that
changed neither root nor depth leaking downstream as a sustained bloom
announce storm. But it was never once run against the regressed binary,
and its per-node ceiling was inferred from a post-mortem harness that no
longer exists in the tree. On the only surviving regressed measurement
the tail node's rate scales to roughly 7 sends per 30s, well under the
scenario's ceiling of 40, so it was never established that the assertion
could fire on its own bug class. The ceiling is also uniform per node,
calibrated against the flap target that is legitimately busy rather than
against the tail where the storm actually shows.

This removes coverage rather than relocating it, unlike the two earlier
retirements above it, and the comment in ci-local.sh records that gap
explicitly. The scenario, its README, the link_swap sim primitive and the
mesh-lab dispatch all stay on disk, so it remains runnable by hand.

Parity holds at 24 legs a side. The parity guard was break-checked by
re-adding the GitHub leg alone, which correctly reported the asymmetry.
 2026-07-26 16:51:23 +00:00
Johnathan Corgan cdda660f10 Stop local CI writing the shared mutable test image tag 
The run built per-run images and then retagged them to fips-test:latest as
a compatibility bridge for the consumers that had never been migrated. The
bridge was the defect: while it existed, two concurrent runs shared one
binding, so a suite could start containers from the other run's binaries
and the verdict was recorded against a commit whose code never ran. It
fails silently by construction — the run is green either way, and nothing
compares a running container's binary against the commit under test.

Every consumer now reads the run's image, so the retag is deleted rather
than kept. That is the point of deleting it: a consumer that was missed
fails loudly instead of quietly resolving whichever run wrote the tag last.

Two consequences handled here. The cleanup script ran ip(8) inside
fips-test:latest to reap simulation interfaces, and nothing writes that tag
any more, so it now takes the caller's image, then the run's, then any
surviving test image — the last of which is what keeps an unscoped --reap
working, since that path execs before the run identity is exported. And a
guard checks statically that nothing names the shared tag, because on a
host with a hand-built copy lying around a reintroduction would run green;
it is break-checked against a reintroduced compose consumer and a
reintroduced default. Both runners gate on it, as they do the other guards.
 2026-07-26 15:47:22 +00:00
Johnathan Corgan 49163befd5 Give each local CI run its own docker build context 
Scoping the image tag was never sufficient on its own. Every build read one
unscoped directory in the working tree, into which each run copies the
binaries it just built, so two concurrent runs raced on the contents of the
context as well as on the name of the result — and a run could produce a
correctly-per-run-tagged image built from the other run's binaries.

The run now copies the context's tracked files into its own directory,
installs its binaries there, and builds from it, exporting the path so every
other consumer follows. Deliberately not carried over: a previous run's
binaries, since inheriting them is the failure this prevents. The path is
absolute because compose resolves a relative build context against the
compose file's own directory rather than the working directory, which was
measured rather than assumed.

The chaos entry script gated the whole simulation on a binary in the shared
directory by literal path, so it moves in this same commit: left behind, it
would have failed on a clean checkout and, worse on a host with leftovers,
passed while reading a binary that was not the one under test.

Teardown removes the directory on red runs as well as green, since it holds
only reproducible content and is never the evidence of a failure. The
worker's SIGKILL runs no trap, so the cleanup script also sweeps contexts
left by a preempted run, and the ignore rule keeps a concurrent run from
showing up as untracked working-tree noise.
 2026-07-26 15:42:48 +00:00
Johnathan Corgan af847c68b5 Stop the chaos simulation rebuilding the shared test image per scenario 
The runner rebuilt the shared tag from the shared build context at the
start of every scenario, unconditionally. Local CI runs chaos scenarios in
parallel, so that was several concurrent builds into one name inside a
single run, before any second run is considered, and it silently replaced
whatever image the harness had already built and handed over.

It now uses the caller's image when one is named, and asserts the image is
present rather than building a substitute, because under a harness a miss
means something upstream is broken and building would hide it. A bare run
still builds the shared tag exactly as before, reading the run's own build
context when one is set.
 2026-07-26 15:40:03 +00:00
Johnathan Corgan a718cef8ce Have every suite compose file name the caller's test image 
Only the static family read FIPS_TEST_IMAGE; the firewall, ACL, nat,
sidecar and both Tor compose files named the shared mutable tag directly,
which is why the local CI runner has to retag its per-run image to that
name. They now take the same defaulted form the static file already uses,
so a harness run resolves the image it built and a bare hand run still
resolves the shared tag exactly as before.

Checked by rendering each file through `docker compose config` with the
variables set: every service resolves to the supplied name and none is
left naming the shared tag.
 2026-07-26 15:38:28 +00:00
Johnathan Corgan cbbdf2c13c Stop the firewall and ACL suites rebuilding an image their caller supplied 
Both suites ran `docker compose up -d --build` unconditionally, and their
compose files carry both a build context and an image name, so the flag
rebuilt and retagged the shared test image from the shared build context on
every run. Their --skip-build flag did not cover it: it guarded only the
Rust build above. Under a harness that has already built the image and
handed it over, that rebuild can only replace the binaries under test with
whatever the shared context currently holds.

The nat scripts had the mirror-image problem. Their image guard built a
replacement whenever the image was missing, which is right by hand and
wrong under a harness: when the caller names an image it built, a miss
means something upstream is broken, and manufacturing a substitute hides
that. They now fail loudly in that case and keep building only on the hand
path. The mesh-lab guard reads the same variable and its build hint moves
to the script that still produces the shared tag.
 2026-07-26 15:37:50 +00:00
Johnathan Corgan 24bf4d46d4 Widen the churn-mixed baseline bounds to sit outside the scenario's own spread 
The root-count ceiling of 4 was set from six runs whose observed maximum
was 3. Fourteen runs that recorded a baseline verdict put the real
distribution at 1 to 5 roots, so the ceiling sat at roughly the 93rd
percentile of the scenario's own variance: one sample in fourteen
exceeded it and four sat at or above it. It therefore reddened a share of
runs whatever the daemon did, which is a calibration fault rather than a
convergence one. Every sample ran the file's fixed seed, so the spread is
container timing and not a differing chaos schedule.

The ceiling is now 6, one step beyond the observed maximum, and the
parented floor moves to its complement at 4. That still fails a mesh
where seven or more of ten nodes are islanded, which is the collapse case
the assertion exists to catch; driving evaluate_baseline directly
confirms all fourteen samples pass and 7, 8 and 10 roots still fail.

min_sessions is left at 10 against an observed minimum of 12. It has
never fired, but the comment now records the margin as thin so a future
failure there is read as calibration first.
 2026-07-26 15:22:43 +00:00
Johnathan Corgan cf129cde77 Retire the three ignored Ethernet tests, which covered nothing unique 
All three required root or CAP_NET_RAW, neither runner passes --ignored,
and so none had ever executed in CI. Before deleting them I checked what
each covered and whether anything else covered it:

- The two-node handshake is covered by the ethernet-only chaos scenario,
  whose baseline assertion cannot pass unless handshakes complete over
  AF_PACKET.
- Tree convergence and root election by smallest NodeAddr are covered by
  ethernet-only's single-root assertion and by the in-process
  spanning-tree tests respectively.
- The mixed-transport coexistence property is covered twice over by
  test_tcp_mixed_transport_coexistence and test_ble_mixed_transport, which
  call the same verify_tree_convergence_components helper with the same
  two-component shape and both run today.

So the only thing running them would have added is an AF_PACKET variant of
a property already proven on two other transports.

The counts confirm nothing running was lost: 1392 tests pass before and
after, and the ignored count falls from 7 to 4.

The ethernet-only comment block cited these tests as the reason its
assertion was the only Ethernet coverage in CI. It now records the sharper
fact found while reading them: that coverage is control plane only. Both
Ethernet scenarios disable traffic, so no datagram crosses an Ethernet
link anywhere, which leaves the frame length field that trims NIC minimum
frame padding ahead of AEAD verification unexercised. Deleting these tests
does not widen that gap, because the test named for data exchange did not
exchange any.
 2026-07-26 14:41:47 +00:00
Johnathan Corgan 4315328f2b Stop the log analyzer double-counting outbound peer promotions 
The shared analyzer matched two strings for one event: "Peer promoted to
active", emitted at info, and "Outbound handshake completed", emitted at
debug. Both come from handle_msg2 on the same call path for the same
outbound promotion, so any run at debug level counted every outbound
promotion twice.

Measured rather than argued. On the archived debug-level churn-mixed run
of 2026-07-25, the node logs carry 20 of each string; the old matcher
reported 40 promotions and the new one reports 20. The neighbouring
counters are unmoved at 13 sessions and 33 parent switches.

Info-level runs are unaffected, which was checked rather than assumed: on
the tcp-mesh run of the same date the debug string occurs zero times, and
the old and new matchers both report 7 against 7 raw occurrences. Three
scenarios run at debug, one of them gating, so this corrects the promotion
figure in their artifacts. No assertion reads the counter, so no suite's
verdict changes.

The earlier fix that introduced this added the live string alongside the
dead one where it needed to replace it.
 2026-07-26 14:24:44 +00:00
Johnathan Corgan 44e04eb2e2 Retire the tcp-chain static test topology 
The tcp-chain profile has never been runnable. ping-test.sh dispatches on
chain and mesh only, and the string tcp-chain has never appeared in that
script in its history, so invoking the profile has always fallen through
to the unknown-profile branch: silently asserting nothing before that
branch was made to fail, and exiting 2 since. Neither runner referenced
it either.

What it would have covered is already covered. The chaos tcp-mesh
scenario runs in local CI and gives TCP a discriminating gate: n04's only
edges are TCP, so if the transport were broken n04 could not parent and
the scenario's baseline assertion would fail. It also carries a pure-TCP
two-hop path, n01 to n04 to n05, under netem and link flaps. The only
residual tcp-chain would have added is a mesh with no UDP present
anywhere, which is not worth a fixture that has never run.

Remove the topology, its three compose services, and the documentation
rows. Every profile the static compose file still defines is now
exercised by a suite.

Also correct the chaos README, which has listed tcp-only and tcp-chain in
its transport table since those two scenarios were deleted in 9e63b42.
Neither has existed for months.
 2026-07-26 14:02:12 +00:00
Johnathan Corgan fa49dc1210 Retire the mesh-public static test topology 
The mesh-public profile ran in neither runner: ci-local's static suite
list carries only static-mesh and static-chain, and the GitHub matrix has
only mesh and chain. It also added no coverage over static-mesh.
ping-test.sh and iperf-test.sh branched mesh and mesh-public together and
exercised the same 20 directed pairs among node-a through node-e, and no
script referenced the external node at all. The convergence waits even
used mesh's peer counts rather than mesh-public's, so the extra link to
the public node was never counted, let alone asserted.

Running it would therefore have added a dependency on a live internet
host (test-us01.fips.network) in exchange for zero additional assertions.
Remove the topology, its five compose services, the script branches that
aliased it to mesh, and the documentation rows. An invocation using the
old profile name now fails on ping-test.sh's unknown-profile guard
instead of silently behaving as mesh.

The config generator's external-node support (external_ip,
is_external_node) stays. It has no consumer now, but it is woven into the
config path every remaining topology uses, so removing it would put the
gating suites at risk for no present gain.
 2026-07-26 13:54:14 +00:00
Johnathan Corgan 0eea7dae13 Make a zero-peer floor unreachable in the shared convergence wait 
wait_for_peers reads the connected-peer count through a pipeline ending in
`|| echo 0`, so a container that never answers contributes 0. That is safe
against a floor of 1 or more, where 0 reads as "not converged yet" and the wait
eventually times out, and it is unsafe against a floor of 0, where the first
read from a dead container satisfies the wait and the caller proceeds as though
convergence had been observed.

No caller passes 0 today; every one passes 1 or more, and the single variable
minimum is advisory. Rather than harden the reader, reject the input: a floor
below 1 is now an error. That makes the shape unreachable instead of repairing
instances of it, and it costs a future caller nothing except a clear message
saying that asserting "exactly zero peers" needs a reader that distinguishes no
answer from zero, which this floor is not.

wait_for_links goes with it. It had no caller anywhere in the tree on any
branch, and it carried the identical fallback, so the only thing it could do was
hand the hazard to whoever called it first. An uncalled helper cannot be wrong
today, which is exactly why it was the risk worth removing rather than the one
worth keeping for symmetry.

The hermetic gate suite gains a case covering both directions, with docker
stubbed so it stays container-free: a floor of 0 is refused with a message
naming why, and the same unreachable container with a floor of 1 still polls its
full budget and times out. Verified by removing the guard and confirming exactly
the three zero-floor assertions red while both floor-of-one controls stay green.
 2026-07-25 19:15:31 +00:00
Johnathan Corgan 1077fd6a7d Stop the firewall peer reader turning a silent container into a count of zero 
wait_for_peers_exact read the connected-peer count through a pipeline ending in
`|| echo 0`, so a container that never answered and a daemon that answered zero
produced the same value. That is only harmless while every caller expects a
non-zero count, which is true here today and is the reason this copy was left
alone when the acl suite's copy was fixed. It leaves the trap armed for whoever
adds the first caller expecting zero: the check would be satisfied on its first
iteration without the property it exists to verify ever being observed.

The read moves into its own function that returns the empty string when the
container does not answer, and the caller treats empty as "no answer" rather
than as a count. A run that never gets an answer now fails saying so, distinctly
from one that answered the wrong count, and the diagnostic peer dump runs only
on the latter, since dumping from a container that cannot answer prints a docker
error rather than evidence.

This is the shape the acl-allowlist suite already uses. The two copies had
diverged on whether a silent container counts as an answer, which is the kind of
disagreement that decides a security assertion in whichever file is read last.

Checked by construction rather than by a green run: driving the old form against
an absent container with an expected count of zero returns success on the first
iteration, and the new form fails; with an expected count of one, the shape both
real callers use, both forms still fail, and a genuine zero from a live daemon
is still reported as zero.
 2026-07-25 19:02:15 +00:00
Johnathan Corgan 0e42c789be Let the firewall suite float its docker subnet 
The compose pinned 172.32.0.0/24 with a per-container ipv4_address, so two
concurrent runs asked docker for the same address space and the second failed
with a pool-overlap error. Request no subnet and let docker assign one from the
daemon pool. Peers address each other by the docker hostname the compose already
sets (host-a, host-b) rather than by literal IP; docker's embedded DNS is
per-network, so the same hostname in two runs resolves inside each run's own
subnet. Nothing this suite asserts on moves as a result: its checks run over the
fips0 overlay, whose addresses are derived from node npubs and are independent of
docker addressing, and the packaged nftables ruleset matches on interface rather
than on any address.

The generated config directory moves under the run suffix for the same reason it
did in the acl suite, and the run teardown gains the matching removal so a run
no longer leaves its directory behind.

Case (b) also wrote curl's output to a fixed path under /tmp. Two concurrent
runs shared that one file, and either run's cleanup landing between the other's
write and read left an empty read, failing the http_code check for a reason
having nothing to do with the firewall. It uses mktemp now.

As in the acl suite, the floating subnet removes one of the two obstacles to
concurrent runs. The compose project name is still fixed; the local CI runner
scopes it externally, a bare hand run does not, and the comment at the site says
so rather than claiming the file is self-sufficient.
 2026-07-25 19:01:56 +00:00
Johnathan Corgan 611f045d33 Let the acl-allowlist suite float its docker subnet 
The compose pinned 172.31.0.0/24 with a per-container ipv4_address, so two
concurrent runs of this suite asked docker for the same address space and the
second failed with a pool-overlap error. Request no subnet instead and let
docker assign one from the daemon pool, which leaves nothing for two runs to
contend for. Peers now address each other by the docker hostname the compose
already sets (host-a through host-f) rather than by literal IP, and docker's
embedded DNS is per-network, so the same hostname in two runs resolves inside
each run's own subnet.

The generated config directory moves under the run suffix in the same change,
because it has to: the generator does rm -rf on it, so a shared directory means
one run deletes the fixtures another is still using, which fails silently rather
than loudly at bring-up. Scoping it requires the generator output path, all five
bind-mount sources per service, and the gitignore entry together; scoping only
the generator leaves the compose reading the unscoped path.

The floating subnet removes one of the two obstacles to running this suite
twice at once, not both: the compose project name is still fixed, and nothing
scopes it for this suite. Recorded at the site so the comment does not claim
more than the change delivers.
 2026-07-25 19:01:16 +00:00
Johnathan Corgan 0cb5574077 Reject rekey settings that fire the trigger immediately and forever 
Config validation said nothing about the rekey block, so two settings that
disable rekey in appearance and hammer it in practice were accepted silently.

after_messages of zero makes the message-count arm true on every poll, since
the trigger tests the counter against it with a greater-or-equal. It reads like
a way to switch the arm off and does the opposite.

after_secs at or below the per-session jitter is the same trap on the timer
arm. Each session offsets the interval by a random value drawn from plus or
minus the jitter bound, so a smaller interval saturates to zero on a negative
draw and rekeys on sight, for roughly half of sessions and not the other half.
The rule is expressed against the jitter constant rather than its current
value, so it tracks if the bound ever moves.

Both are checked whether or not rekey is enabled, so turning it on later cannot
surface a configuration error at a surprising moment. Neither has an upper
bound: a very large value is the established way to disable one arm of the
trigger and stays legal.
 2026-07-25 17:07:57 +00:00
Johnathan Corgan 989ae65fc5 Align the datagram hop-limit helpers with the forwarder's semantics 
The forwarding path was corrected to full IP semantics: local delivery is not
hop-limit gated, and forwarding decrements first and drops at zero. Two helpers
on SessionDatagram were left implementing the old rule. decrement_ttl still
checked before decrementing, and can_forward still answered true at a hop limit
of one, where the decrement leaves zero and the datagram is dropped. Neither is
called anywhere today, which is precisely why they are worth correcting rather
than leaving as an invitation to reintroduce the defect.

The TtlExhausted reject doc described behaviour that no longer exists. The
reject is charged for a transit arrival at one as well as at zero, and is never
charged for a datagram addressed to this node, whose delivery is decided ahead
of the test.

Both helpers are pinned by tests at the boundary, which fail against the
previous implementations.
 2026-07-25 17:05:20 +00:00
Johnathan Corgan 1eabe08575 Correct what the address-to-link map claims to be for 
The doc said the map "enables dispatching incoming packets to the right
connection before authentication completes." It does not: find_link_by_addr
has no callers outside its own tests on any branch, and encrypted frames
dispatch by session index. Its live readers are the msg1 admission fast
path and the duplicate-inbound-handshake check.

The comment mattered because the map has a trap that invites exactly the
misuse it was advertising. An outbound dial registers the literal
configured address string, which may be a hostname, while an inbound
packet carries the resolved form; TransportAddr compares byte-wise, so
the two never match and a lookup keyed on an inbound address returns "no
such link" for every hostname-configured peer rather than failing. The
entry is also single-valued per key, so an inbound handshake overwrites
an outbound dial's entry for the same address.

Both current readers survive this because each compares a key written in
the same form it reads. That is a property of those two call sites and
not of the map, so the comment now says so, and says not to key a
peer-identity question on it.

Documentation only; no behaviour changes.
 2026-07-25 01:50:40 +00:00
Johnathan Corgan 9846e85705 Retire the admission-cap Docker suite; the inbound gate is proven in-process 
The inbound max_peers early-gate in handle_msg1 (silent-drop a Msg1 from
a net-new identity at saturation, send no Msg2, admit no peer) is
unit-tested over a real UDP socket by
handle_msg1_silent_drops_at_cap_for_new_peer in src/node/tests/unit.rs:
it saturates a node, sends a Msg1 from a fresh identity, and polls the
sender socket to assert no Msg2 comes back — the same wire-observable
discriminator the Docker suite's tcpdump used — with a sibling test for
the existing-peer bypass. That is a deterministic superset of the Docker
packet-capture assertion.

Drop it from both runners in lockstep (its *_SUITES array, run function,
default-flow loop, --only dispatch arm, and the GitHub matrix leg with
its steps) so the parity guard stays green, and record the retirement in
the deliberately-not-run block with a pointer to the standalone runner.
The admission-cap-test.sh script stays on disk and runs by hand.
 2026-07-24 02:31:43 +00:00
Johnathan Corgan 41ce64ba82 Retire the acl-allowlist Docker suite; the decision is proven in-process 
The ACL admission decision is exhaustively unit-tested per npub over real
loaded allow/deny files (src/node/acl.rs test module: allow-match-wins,
allowlist-miss falls through, deny-only, deny-all, allow_all override,
deny-after-allowlist-miss), and the inbound and outbound handshake-
admission paths are covered in-process over the loopback transport
(src/node/tests/acl.rs: inbound msg1 denial, outbound denial, reload).
The Docker suite's only unique coverage was admission over a real UDP
transport, which every other real-transport suite already exercises, and
its operator-facing log assertion.

Drop it from both runners in lockstep (its *_SUITES array, run function,
default-flow call, --only dispatch arm, and the GitHub matrix leg with
its steps) so the parity guard stays green, and record the retirement in
the deliberately-not-run block with a pointer to the standalone runner.
The testing/acl-allowlist/ suite stays on disk and runs by hand.
 2026-07-24 02:29:38 +00:00
Johnathan Corgan 7cbe1d3d4e Retire the smoke-10 chaos scenario; convergence is covered in-process 
smoke-10 was a no-stressor 10-node tree-convergence sanity check (netem
off, no ping). Its subject, spanning-tree convergence and root election,
is now covered in-process, faster and deterministically, by the loopback
spanning-tree harness (src/node/tests/spanning_tree.rs: ring, star,
chain, 100-node and disconnected-component convergence) plus end-to-end
datagram delivery (src/node/tests/forwarding.rs). Real-UDP convergence
smoke still runs via static-mesh and the other scenarios' baseline
assertions, so no Docker coverage is lost.

Drop it from both runners in lockstep (the CHAOS_SUITES list and the
GitHub chaos matrix) so the parity guard stays green, delete the scenario
YAML, and update the chaos README.
 2026-07-24 01:43:20 +00:00
Johnathan Corgan 08a226fb63 Test cost-based parent selection and kernel-drop detection as unit tests 
The cost-selection chaos scenarios (cost-reeval, cost-avoidance,
cost-stability, depth-vs-cost, mixed-technology, bottleneck-parent) tested
TreeState::evaluate_parent's decision logic through a Docker mesh that could
not exercise it reliably: the tree roots at whichever node holds the smallest
NodeAddr, MMP link costs take several measurement windows to settle, and the
parent hold-down plus hysteresis timing all confound the outcome. A
deterministic link-cost flap still produced zero periodic parent switches in a
full run.

Replace those six scenarios with deterministic unit tests in src/tree/tests.rs
that drive evaluate_parent directly: cheaper-link selection at equal depth,
switch-on-cost-change, hysteresis suppressing a marginal change while allowing
a significant one, and the depth-versus-cost effective-depth tradeoff. Each is
constructed so that breaking the cost or hysteresis logic makes it fail.

The congestion kernel-drop signal (SO_RXQ_OVFL) cannot be provoked
deterministically in Docker: a fresh daemon reader keeps up with
container-speed traffic, so the socket receive queue never overflows (an
unshaped run with a 4 KB buffer and heavy traffic recorded zero drops on every
node). Extract the drop-detection edge -- read the cumulative counter, fire an
event only on the transition into a new drop burst -- into
TransportDropState::observe_drops and unit-test it directly. congestion-stress
keeps its ECN and MMP congestion-signal assertions, which do need the real
shaped bottleneck queue.

Remove the retired scenarios from both CI runners and update the chaos README.
 2026-07-23 23:33:01 +00:00
Johnathan Corgan be5deee814 Make the gateway integration suite safe for concurrent CI runs 
gateway-lan pinned 172.20.1.0/24 and fd02::/64, so two concurrent local CI
runs collided on "Pool overlaps" at network creation. The IPv4 subnet has no
consumer -- the whole gateway LAN path is IPv6 -- so drop it and let docker
auto-assign. The IPv6 side cannot float, because the LAN clients' resolv.conf
pins the gateway's address as their nameserver and that must be a literal
known before they start, so claim a free /64 per run and thread the prefix
through the compose address pins, a generated resolv.conf, and the test via a
single exported variable.

The claim and the external network ship as a harness-only compose overlay
applied by run_gateway; the base compose keeps a normal gateway-lan network,
so the GitHub matrix and any standalone bring-up are unaffected. With no claim
every address renders exactly as before.
 2026-07-23 19:50:17 +00:00
Johnathan Corgan bb9bca4d83 Accept a degraded systemd as a booted one in the deb-install boot wait 
The boot check piped `systemctl is-system-running --wait` into `grep -qE
'running|degraded'`, but the script runs `set -o pipefail` and is-system-running
exits non-zero for `degraded`. So the pipeline failed on a degraded system even
though grep matched, and the wait looped to its timeout. The older distros reach
`running` (exit 0) and passed; debian:trixie and ubuntu:26.04 reach `degraded`
because a unit that cannot run in a container (systemd-modules-load) fails, so
they timed out at every ceiling -- which is why the earlier timeout increase did
nothing.

Capture the state string and test it directly instead of trusting the pipeline
exit, so a degraded-but-booted system is accepted as intended. Validated: all
five distros pass, including the two that previously failed.
 2026-07-23 19:50:07 +00:00
Johnathan Corgan 2bc0345174 Make the cost-based chaos assertions reliable under parallel CI load 
cost-avoidance and mixed-technology decide a parent by measured link cost
(etx * (1 + srtt_ms/100)). Under the local CI's 4-way-parallel chaos the fast
fiber link's measured srtt spikes from host scheduling and can exceed the
Bluetooth link's, flipping the choice and reddening a run that proves nothing
about the daemon. The old margin was ~0.4 cost units, which a ~24ms one-sided
scheduling blip closes.

Widen the Bluetooth delay to 150-250ms so the margin dwarfs any plausible
one-sided spike. The link still establishes well within the handshake budget
(a ~500ms RTT against a 30s stale-handshake window), so the losing candidate
is still a real, established peer rather than an absent one.

Add a netem mutation exclude_edges option and use it in mixed-technology to
pin the two links n08 chooses between, so a random degradation can't flip the
asserted comparison either. An unknown excluded edge is a hard error, since a
silent no-op would reintroduce the flakiness it exists to remove.

Validated by running both scenarios repeatedly under heavier-than-CI
concurrent load: the parent choice holds every time.
 2026-07-23 18:58:49 +00:00
Johnathan Corgan 9b7a27f219 Claim the chaos network range instead of deriving it, ending the collision 
Two concurrent ci-local runs could not both run chaos. Each child's subnet came
from its position in the suite list, so both runs walked 10.30.0 through
10.30.12 and requested identical ranges; whichever reached the daemon first won
and the other died with a pool overlap. A run index or hashed offset would only
make that unlikely, and it is precisely the failure being removed.

The simulator now claims its range: attempt-create on a candidate, advance on
docker's own overlap error, fail loudly on anything else. Docker's address pool
becomes the arbiter, so an overlap is impossible rather than improbable. The
claim lives in the sim rather than in ci-local because only the process that
creates the network can advance on conflict, and because it fixes the bare
chaos.sh path too, which a ci-local-only fix would have left broken.

The generated compose now declares the network external over the claimed one,
and teardown releases the range from both paths, including the setup-failed
path where a run that fell over after claiming still holds one.

Ordering matters here and is not obvious: node IPs derive from the subnet
during topology generation, and traffic shaping keys its filters on those
addresses, so the claim happens before the topology exists. Claiming later
would give a network on one range and filters on another, which does not fail
at bring-up and instead leaves the shaping matching nothing.

--subnet survives as an explicit pin for when a known range is wanted, and now
fails loudly if that range is taken rather than silently overlapping. ci-local
no longer passes it.

Validated by running two instances of the same scenario deliberately
concurrently: they claimed 10.30.1.0/24 and 10.30.0.0/24, both exited 0 with
their assertions passing, and both released their networks. The negative
control holds too: with a squatter on a pinned range the run aborts with a
message naming the range rather than proceeding.
 2026-07-23 17:09:07 +00:00
Johnathan Corgan f478f51afe Assert mixed-technology's sound criterion, and correct the one that never was 
With the root pinned, n08's test subject is assertable for the first time: its
two candidates sit at equal depth with fiber-grade default netem on one side and
Bluetooth on the other, which is exactly the preference this scenario exists to
test, and it takes the fiber parent in four of four runs. Encoded as written
rather than calibrated from the runs, since "should pick n03" is the spec and
the runs merely confirm it is met.

n06 is deliberately absent, and its documented criterion is corrected rather
than left standing. The header called n03 a fiber parent for n06; the netem
policies in the same file say that link is WiFi and the other candidate is
Bluetooth, so n06 chooses between two impaired links. Its only fiber-grade link
reaches a node at depth 3 that is never a good parent. There was never a reason
to expect it to prefer n03, and the two-of-four split observed is correct
behaviour: in a run where it took the Bluetooth peer, the fiber-labelled
candidate measured a link cost of 3 against the other's 1, so the daemon chose
the cheaper effective depth as it should.

Validated by replaying the assertion against all four archived runs, then
breaking it three ways to confirm it can fail, then one live run for the wiring.
One of the breaks expects n06 to take n03 and reports that it chose n02, which
is the direct demonstration of why n06 is not encoded.

cost-reeval records that its subject now fires in two runs of three, against
zero of five when it was structurally impossible, and why that rate is still not
one an assertion can rest on.
 2026-07-23 17:04:23 +00:00
Johnathan Corgan 9d92cfeab4 Correct three scenario headers that documented conclusions since disproved 
congestion-stress said the transport drop-detection path "is exercised by
nothing". Widening the scan beyond this one scenario finds it firing in six
others, so the path is live and what is dead is this scenario's ability to
provoke it. Records the arithmetic that explains why: at the 1 Mbps cap the
receive queue needs tens of milliseconds of reader stall to overflow, and the
ingress policer discards the same traffic a layer below the socket.

mixed-technology said the implementation does not do what its criteria say.
It does. The criteria were conditioned on a tree rooted at n01 while the mesh
rooted at n09, under which n08's choice was settled by depth before link
technology could matter. The archived parent distributions are relabelled as
describing the old root so nobody calibrates against them.

cost-reeval now records why it had no assertion about its own subject: its
designated subject was the root, so the parent switch it exists to observe
could not occur. The historical logs show the switch on n04 in 13 of 14 runs
when the root still wandered, and only on n01 in the five runs before the fix.

Each header now names what remains to be done and where it is tracked, rather
than leaving a disproved conclusion in place for the next reader to inherit.
 2026-07-23 15:47:28 +00:00
Johnathan Corgan 6319c7f577 Describe the trailing command accurately in the gate's finding message 
For a value-returning helper the trailing echo is the return mechanism, not a
log call. The hazard is the same either way -- it fixes the exit status at zero
-- but calling it a log call misdescribes four of the sites the gate now finds.
 2026-07-23 15:46:04 +00:00
Johnathan Corgan d4a2504f99 Teach the trailing-log gate to see command-substitution call sites 
Found while fixing count_log_pattern: the gate reported that function clean
even though it ends in echo and both callers consume its status. Its call-site
patterns only matched direct forms -- if fn, while fn, fn ||, fn && -- so a
status consumed through command substitution was invisible, because the line
begins with the variable rather than the function name.

That is not an exotic form. It is how a shell function returns a value, and it
is precisely the shape of the swallowed-failure family this gate exists to
catch, so the gate was blind to a large part of its own stated class.

Three patterns added for the assignment, if-guarded and test-expression forms.
The extension finds four real instances, all value-returning helpers whose
trailing echo made their exit status unconditionally zero, and each now carries
an explicit return 0. Also corrects the finding message, which described the
trailing command as a log call; for these it is the return mechanism, and the
hazard is that it fixes the status either way.

Validated by breaking what it guards: a probe reintroducing the defect shape
behind a command substitution is reported and exits 1, where before the
extension it would have passed.
 2026-07-23 15:45:37 +00:00
Johnathan Corgan bf173d8d98 Pin the chaos mesh root to n01 so scenario diagrams describe the real tree 
Every chaos scenario draws n01 at the top of its topology, and until now that
held in three of thirteen. The mesh roots itself at the numerically smallest
NodeAddr, which is a hash of the node's public key and bears no relation to the
node numbering, so which node ended up as root was effectively arbitrary.

The consequences were not cosmetic. cost-reeval rooted at n04, its own
designated test subject, so that node had no parent and the periodic parent
switch the scenario exists to observe could not occur at all. mixed-technology
rooted at n09, which put its two documented parent criteria out of reach and
made correct cost-based selection look like a defect.

Identities are still derived from the mesh name exactly as before and are still
deterministic. What changes is which node id holds which one: they are now
assigned in NodeAddr order, so n01 holds the smallest and is the root. Verified
against a model of the daemon's own derivation that reproduces the previously
observed root for every scenario and n01's address byte for byte; all twelve
pinned scenarios now root at n01.

smoke-10 deliberately opts out via pin_root: false so that root election from an
arbitrary key distribution stays exercised somewhere. Its assertion is a
convergence floor and is root-agnostic, which is why it is the cheapest home for
that. The new key is rejected when non-boolean, and a near-miss spelling is
rejected as unknown; both checked.

Not yet established: the trees themselves change, so the parent-dependent
assertions in bottleneck-parent, cost-avoidance and cost-stability need
re-deriving against live runs. Those are held until the in-flight CI finishes,
because a chaos run rebuilds the shared fips-test image that run is using.
 2026-07-23 15:43:10 +00:00
Johnathan Corgan 34a2561af7 Close the remaining "caught" and "produces red" harness holes 
Four sites where a check could report a verdict it had not established, or
detect a failure and then not turn it red.

assert_no_panic in both NAT suites read `docker logs ... || true`, so a
container that could not be read produced empty output, matched no panic
pattern, and returned success. The assertion's failure mode was
indistinguishable from its success condition. It now reports that absence of
panics is not established.

deb-install's apt capture discarded the exit status, so a failed docker exec
gave an empty capture that matched neither error pattern and reached the pass
branch. The status is now kept and checked before the output is inspected, with
the output still printed on failure.

wait_for_systemd printed a warning and returned success on timeout, so every
check after it read a system that may not have started its units. It now returns
non-zero and the caller abandons that distro leg rather than testing an
unstarted system.

interop's copy of count_log_pattern carried the same defect fixed in rekey: a
node whose logs could not be read contributed zero to eight expect-zero
assertions. Fixed the same way, and its consumer now reports the unreadable case
rather than comparing a sentinel against zero.

Each validated by breaking what it guards and by confirming the healthy path is
unchanged: an absent container now fails each check where it previously passed,
a readable panic-free container still passes, a real panic is still caught, and
a genuinely successful install still passes.
 2026-07-23 15:36:59 +00:00
Johnathan Corgan c8a0ac5fca Stop a node whose logs cannot be read from counting as a clean node 
count_log_pattern summed a per-node `docker logs | grep -c ... || true`, so a
node the harness could not read contributed zero. The six assert_zero_count
callers are negative health assertions -- no panics, no ERROR lines, no AEAD
decrypt failures, no rekey msg2 failures -- and a contributed zero reads as
clean, so one unreadable node silently weakened the assertion and all of them
voided it. This is the family the harness-fallback issue was raised to high
priority for, and it is the one remaining audit residual that can make a green
rekey run lie about protocol code.

The read now fails the count rather than degrading it, printing a sentinel that
names the container. Both callers split the declaration from the assignment,
because `local c=$(fn)` takes local's exit status and discards the function's --
which is how the original defect stayed invisible.

Validated by breaking what it guards rather than by observing green: against
absent containers the old reader returns 0 and assert_zero_count passes
vacuously, while the new one returns rc=1 and reports a failure. Checked the
other direction too, since a fix that reds a legitimately clean run is no use: a
genuine zero across readable nodes still returns 0, and the sum is unchanged at
2+1+0 for a shimmed three-node read.
 2026-07-23 15:30:01 +00:00
Johnathan Corgan 55331a80b5 Gate on shell functions whose exit status is a log call's 
A bash function returns its last command's status, so one ending in a log
call returns 0 whatever it did, and a caller written as `if func` or
`func || fail` has a gate that cannot fire. This tree found two in one
day: run_chaos ended in record, making every chaos row unconditionally
green since 2026-03-09, and build_fips_for_e2e ended in log, letting five
end-to-end legs test the previous commit's binary and report green. Both
were repaired one at a time. This is the rule that catches the next one.

What it enforces is a contract, not a bug hunt: a function whose status a
caller tests must end in an explicit return rather than leaving its
success value to whatever the last log call produced. That distinction is
deliberate and worth stating, because all three instances found in the
tree were benign — each failure path already returned early, so the
trailing echo reported a real success. Reading the function is the only
way to know that, and the next edit that puts an unguarded command before
the final log converts the benign shape into the defect with nothing to
notice. An explicit return costs a line and makes the class unreachable.
The three are given one here.

Scoped on the call sites rather than the definitions, the same way
check-log-strings.py scopes on what a grep reads rather than what its file
mentions: 315 functions scanned, 42 end in a log call, and only the ones
whose status something consumes are reported. A function that ends by
reporting and is called for its output is idiomatic and is left alone.
Without that scoping the finding list would be 42 long and would stop
being read.

Validated by construction rather than by a green run: injecting a copy of
build_fips_for_e2e's exact shape — unguarded docker cp, then a log as the
last statement, with a `|| { ...; return 1; }` caller — makes the checker
report it and exit 1.

Wired into both runners beside the parity and log-string checks.
 2026-07-23 14:09:04 +00:00
Johnathan Corgan 3181861341 Assert the Link MMP pane exists before asserting its colour 
mmp_focused_pane_indicator checked that the unfocused Link MMP title is
not cyan with assert_ne! over fg_at. fg_at is find(..)? mapped to the
cell's foreground, so it returns None for a title that was never drawn,
and None != Some(Cyan). The assertion therefore passed just as happily
when the pane was missing entirely as when it was present and unstyled.

Assert presence first, so the colour check means "not highlighted"
rather than "not there".

Demonstrated rather than argued. Renaming the pane title in mmp.rs so
"Link MMP" is never rendered leaves the original assertion passing, and
makes the new one fail with the message it was given. Both files restored
after.

This is the only instance of the shape: it is the sole assert_ne! over
fg_at or find in the fipstop tree, and the only assert_ne! in snapshots.rs
at all.

Quartet clean: fmt, build, clippy --all-targets -D warnings, test --lib
at 1376 passed / 0 failed / 7 ignored.
 2026-07-23 14:00:49 +00:00
Johnathan Corgan 0f27bdbd2c Give the UDP microbench a reason and record the retry asymmetry 
The UDP recv microbenchmark carried a bare #[ignore] while its sibling in
the link module carries a self-describing one. Match it, so the reason
appears wherever the test is listed rather than only in the doc comment.

Record why [profile.ci] retries = 2 is not applied locally. The two gates
disagree on purpose: the hosted runner retries a flaky test twice, the
local sweep fails on the first failure. It exists for shared-runner packet
loss, a property of that environment rather than of the code, and applying
it locally would suppress a real local flake — a failure that only
reproduces under load is a robustness bug to fix, not to retry past.
Keeping the local sweep strict is what makes it the sharper gate. An
undocumented asymmetry is indistinguishable from an oversight, which is
why this is written down rather than left to be rediscovered.

The cost is stated rather than hidden: a test that fails once and passes
on retry is reported green with no separate signal, so a genuine
intermittent failure can be absorbed. If that starts mattering the fix is
to surface retried-but-passed tests, not to drop the retries.

Quartet clean: fmt, build --workspace, clippy --all-targets -D warnings,
and test --lib at 1376 passed / 0 failed / 7 ignored.
 2026-07-23 13:48:11 +00:00
Johnathan Corgan d3eaad543d Make a skipped check visible in the verdict rather than in scrollback 
Two suites could report a clean pass for work that did not run.

stun-faults skips Phase 2 when tc netem is unavailable, announcing it
several hundred lines above the result and then printing a bare
"stun-faults-test passed". The verdict line now carries the count and
names each skipped phase, and a run in which every phase was skipped
fails outright, since it tested nothing.

deb-install defines a skip() helper and a SKIP counter, prints "N
skipped" in its summary, and calls skip() from nowhere, so the number can
only ever be 0. Its exit tested FAIL alone, meaning a skip could not have
failed the run even once something did set the counter. Gate on SKIP too.
That changes no current outcome, which is the reason to do it now rather
than later: the machinery and the reported number already existed, so the
first skip path added would have printed "N skipped" beside a zero exit
and read as coverage.

Verified by driving the verdict logic at zero, one and three skips: clean
pass, pass with the skip named, and a non-zero exit when nothing ran.
 2026-07-23 13:44:29 +00:00
Johnathan Corgan 2a6039995d Retire the ECN A/B test framing and record why suites are excluded 
ecn-ab-test.sh was a -test.sh with no path to a failing result: it
asserts nothing, applies no threshold, and nothing invokes it. It also
could not have worked. It read a fixed sim-results/ecn-ab-on/ path while
the runner has written timestamped directories since 2026-03-20, and not
one ecn-ab result directory exists on disk, so it has found neither input
for months and the "+10.2% recv throughput" figure the README carried is
not reproducible from anything available.

Rename it to ecn-ab-compare.sh, fix the path resolution to glob the
timestamped directories, and stop swallowing a failed simulation with
|| true so a broken run cannot feed the comparison.

Deliberately not adding the threshold assertion that would make it
gateable. That needs a calibration corpus, and none exists precisely
because the tool has never produced a kept result; a number invented now
would assert a guess. The prerequisite is a calibration run set and a
decision about what ECN is expected to deliver, which is a protocol
question. Written in the script header rather than left implied.

Add the exclusion block to ci-local.sh naming every suite and scenario
neither runner runs, with the reason for each: interop, boringtun,
iperf-test, this comparison tool, mesh-lab, and the four chaos scenarios
outside CHAOS_SUITES. Until now "not in the suite list" was
indistinguishable from "forgotten".
 2026-07-23 13:40:53 +00:00
Johnathan Corgan 1f149bda32 Run the convergence-gate unit tests in both CI runners 
wait_until_connected decides whether every static suite proceeds or
gives up, so a regression in it turns those suites' verdicts into noise.
Its unit tests existed and were invoked by nothing.

Wire them into ci-local.sh beside the parity and log-string checks,
above the mode branches so --only and --test-only are gated too, and add
the matching step to the ci-parity job in ci.yml. Putting them in one
runner only would create exactly the drift check-ci-parity.sh exists to
catch, and neither placement is visible to that checker since this is not
a matrix suite.

Verified by breaking what it guards rather than by observing green:
setting wait_until_connected's default near-converged slack to 0 fails
case4, records FAIL wait-converge, and exits the sweep 1. Restored after.

They pass, which nothing had established before — 13 assertions, all
green. They also take about 45 seconds rather than the "a few seconds"
the header claimed, and nothing had contradicted that claim because no
runner had ever invoked it. Header corrected with the measurement.

The orphan sweep this called for is done and there are now no
zero-reference *-test.sh files under testing/. It also settles a
disagreement recorded between the audit and the task file: ecn-ab-test.sh
is invoked by nothing, its one reference being a README description
rather than a driver, so the audit was right and the earlier sweep wrong
to call it a documented manual tool. interop-test.sh does have a driver
in interop-stress.sh and iperf-test.sh one in iperf-compare-refs.sh, so
those two were correctly classified.
 2026-07-23 13:38:25 +00:00
Johnathan Corgan 38a60c61da Record that churn-mixed's floors describe the invocation CI overrides 
The baseline floors added for this scenario were calibrated from archived
runs, and those runs are all the 10-node 120-second variant ci-local
invokes as "churn-mixed --nodes 10 --duration 120". The file itself
defaults to 20 nodes and 600 seconds, and --nodes sed-patches num_nodes
in a copy before the scenario loads, so the gating run and a bare
chaos.sh run are different scenarios. Nothing at the site said so.

The floors hold for both, but only because the gating run is the smaller
one. Retuning them against a bare 20-node run, which clears them easily
at 20 answering, 1 root, 19 parented and 69 sessions, would put them past
what the 10-node run reaches and turn CI red while a manual run stayed
green. Say that where someone retuning them will read it.

Confirmed by running the gating invocation directly: 10 answered, 2
roots, 8 parented, 18 sessions, all inside the ranges the floors were
built from. churn-mixed is the only scenario CI overrides this way; the
other twelve run their files as written.
 2026-07-23 07:27:16 +00:00
Johnathan Corgan 3d0a388511 Fail the ping test on an unknown topology profile 
Every section of ping-test.sh is guarded by the profile name, so an
unrecognised profile fell through all of them and the script exited 0
having run no assertion. A typo in the caller's argument produced a green
run that tested nothing. Give it an else branch that names the valid
profiles and exits 2.

No existing leg changes behaviour: the hosted runner gates the ping step
on matrix.type == 'static', which only static-mesh and static-chain
carry, and ci-local passes only the two names in STATIC_SUITES. Worth
noting while confirming that, though: ping-test.sh accepts a mesh-public
profile that neither runner ever passes.

Also record why the convergence waits are `|| true` — they are settling
delays rather than assertions, and the directed-pair pings are what
decides the run — and add admission-cap to the suite list in the
ci-local header, which ran it without listing it.
 2026-07-23 07:13:32 +00:00
Johnathan Corgan 5d3a3d7cad Give the assertion-free chaos scenarios a convergence floor 
Five scenarios named by the audit carried no assertions at all, so a run
in which the mesh never formed exited 0 and reported green. Add a
baseline assertion covering how many nodes answered, how many distinct
roots they agreed on, how many took a parent, and optionally how many
sessions were established, and apply it to those five plus the two cost
scenarios left unasserted by the previous commit.

For ethernet-only, ethernet-mesh, tcp-mesh, smoke-10, mixed-technology
and depth-vs-cost the values are not calibrated: one root and N-1
parented nodes is what a spanning tree is, and all provably-completed
archived runs of each show exactly that. churn-mixed is different and
says so at the site. A scenario that stops and starts nodes on purpose
does not hold a single tree, and its six completed runs end with two or
three roots and seven or eight of ten nodes parented, so its floors sit
one step outside the observed range. That leaves them catching a mesh
that collapsed rather than one that churned, which is the most a sample
of six supports.

This gives Ethernet transport its only assertion anywhere in CI. The
three tests in src/node/tests/ethernet.rs are ignored for requiring
CAP_NET_RAW and neither runner passes --ignored, so until now nothing
exercised that transport with a verdict attached.

The floor is deliberately weak and deliberately not a substitute: it says
the mesh formed, not that it formed the tree the scenario describes.
Where those differ the scenario now says so in its own comments.
 2026-07-23 07:09:36 +00:00
Johnathan Corgan e9ca741e53 Assert parent selection where the cost scenarios actually specify it 
Four scenarios exist to test cost-based parent selection and each named
its expected outcome in a comment that nothing read. Add a tree_parents
assertion mapping a node to the parent it must have in the final tree
snapshot, and encode it for the two scenarios whose stated outcome the
implementation meets: cost-avoidance (n04 takes the fiber n03) and
bottleneck-parent (n06 takes the fiber n03, n09 keeps its only parent
n05). Both hold in all six provably-completed archived runs.

The other two are left unasserted on purpose, and each for a different
reason worth keeping distinct.

mixed-technology says n06 and n08 should both pick the fiber parent n03.
They do not. Across the five completed archived runs n06 picks n10 three
times and n08 picks n04 four times. Encoding the criterion as written
would red the scenario most runs, and encoding the observed behaviour
would bless something nobody specified, so the disagreement is recorded
at the site as an open question. n06 preferring n10 may well be correct
and the comment stale, since n10 reaches it by fiber too.

depth-vs-cost is different: its validation line does not name an outcome
at all, saying only that the choice "reflects the actual cost tradeoff",
which either answer satisfies. The corpus shows both occurring under one
seed, three runs to two. That scenario needs a protocol decision about
which parent is correct before it can have an assertion.

Compare parents by address resolved from the snapshot's own
my_node_addr, and fail rather than skip when a node is absent from the
snapshot or still claims to be its own root. Those two states are the
common ones in the older corpus and both produce the same "no match" a
wrong parent does, so only separating them keeps a harness problem from
reading as a routing verdict.
 2026-07-23 07:05:50 +00:00
Johnathan Corgan 5a11cf091d Give congestion-stress the assertions its criteria described 
The scenario listed four success criteria "verified via post-run
congestion snapshot". Nothing read the snapshot, so the scenario could
not fail on any of them.

Add a congestion_signals assertion taking a floor on the number of nodes
reporting each counter, and encode three of the four. The floors are one
node each because that is what the criteria say; tightening them to the
counts recently observed would assert something nobody wrote down.

The fourth criterion is not encoded, and that is the finding rather than
an omission. No node has reported a non-zero kernel_drop_events in any of
the 182 archived runs, so asserting it would red the scenario
permanently. It is recorded at the site as an unmet criterion and a
coverage gap: the transport drop-detection path the scenario names as its
second signal is exercised by nothing.

Two things about the corpus are worth carrying, both recorded in the
scenario. Only six archived runs carry a status.txt and are therefore
provably completed; all six meet the three encoded criteria with five to
nine nodes reporting each signal. The 176 older runs meet none of them,
and they are not invalid samples: each reached teardown far enough to
write an analysis.txt, and ECN landed before all but one of them. What
changed on 2026-07-22 is not established, since neither the scenario nor
netem.py, traffic.py or control.py has been touched.

Count the nodes reporting a signal rather than the magnitude of any one
counter, since a single node with a large count would satisfy a magnitude
test while proving the signal never propagated. A missing snapshot fails
rather than reading as an absence of congestion.
 2026-07-23 06:59:21 +00:00
Johnathan Corgan aab149e215 Fail a chaos run whose nodes logged errors 
The simulation exit ladder reported panics and failed assertions but said
nothing about ERROR-level log lines, so a run in which every node errored
on every line still exited 0.

Add a max_errors assertion and apply it to every scenario by default
rather than having each one opt in. This is a floor on what a green run
means rather than a property of an individual scenario, and a scenario
that has to ask for the floor is one that can forget to.

The default ceiling of 0 is what the archived corpus supports: across
2416 result directories no node log contains an ERROR-level line, while
the WARN counter extracted by the same code path ranges from 0 to 1135,
so the counter is known to discriminate rather than merely known to read
zero. A scenario that legitimately induces errors raises the ceiling in
its own YAML and says there why.

Reject rust_log: off alongside it, since that is the one level that would
leave the ceiling counting zero whatever the mesh did.
 2026-07-23 06:53:56 +00:00
Johnathan Corgan 73e6917341 Give cost-stability the parent-switch ceiling its criterion described 
The scenario's success criterion existed only as a comment: count
"Parent switched" in n04's log, expect at most 5. Nothing read it, and
the scenario declared no assertions at all, so its exit code could only
report that the mesh came up and nothing crashed.

Add a max_parent_switches assertion and wire that criterion to it.

The assertion takes an optional node scope, which is the part that
matters. The existing sibling counts mesh-wide, and a criterion written
about one node's log is a different quantity from the sum over every
node: across archived runs of this scenario n04 is 1 or 2 while the
mesh-wide total ranges 3 to 6. Asserting the sum against a per-node
threshold would check something other than what was specified, and at
this threshold would also have failed several runs that were fine.

Counting nothing must not read as stability. A node id absent from the
topology fails explicitly rather than matching zero log lines and
sailing under the ceiling, and a scenario that declares a parent-switch
assertion while setting a log level that suppresses the events it counts
is rejected at load rather than passing vacuously after a full run. The
parse rejects a mistyped key, a missing, negative, boolean or
non-integer ceiling, and a node written empty or as a non-string -- that
last one because YAML renders a bare `node:` as null, which would
silently revert to the mesh-wide count.

The scenario comment now records what the threshold is worth against
the 11 completed archived runs rather than a single sample: 5 is well
above anything observed, the daemon's own parent hold-down caps
switches near 6 per run, and nearly every switch lands during initial
tree formation before any mutation fires. So this catches only
near-pathological reparenting. Tightening to 3 would make it a real
detector, but that changes the stated criterion rather than fixing it,
so it is left as a decision.

Verified against a live run: the assertion evaluates n04 at 2 against
mesh-wide 4 and passes, and with the ceiling temporarily set to 0 the
scenario exits 3, the assertion-failure code, which it previously could
not reach.
 2026-07-23 05:19:36 +00:00
Johnathan Corgan 300deb0476 Reject an unanswered show_mappings instead of reading it as zero 
The mapping-reclaimed check expects zero mappings, and read the count
with `r.get('data',{}).get('mappings',[])`. An error response carries
no data field, so it parsed to zero and satisfied the check without the
gateway having answered at all. The expected value and the failure
value were the same number, which is the shape that lets an assertion
pass without asking anything.

Require the key to exist and be a list, and exit non-zero otherwise, so
the existing fallback turns an unanswered or malformed response into a
failure. The gateway always emits the key on success, including for an
empty set, so a genuine zero still reads as zero; confirmed against the
running gateway, where the reclamation check passes.

The earlier show_mappings poll shares the parse and is left alone: it
waits for a positive two, which no error response can produce. The
hazard was already documented there, and this is the call site that was
exposed to it.
 2026-07-23 04:16:42 +00:00
Johnathan Corgan 700ac581ee Require a timeout and a drop verdict in the firewall assertions 
Case (a) sends an unallowed inbound request that the ruleset should
drop, and accepted any non-zero curl status as proof. A drop produces
no RST, so curl can only hit its deadline and exit 28; connection
refused, an unroutable address or a missing listener all fail too, with
different codes, and the check counted those as a blocked connection.
Require 28 exactly, so the assertion distinguishes silently dropped
from failed for some other reason. Confirmed against the harness: the
real run returns 28.

The baseline check matched `counter packets`, which any counter rule
satisfies whatever its verdict, including one that accepts. Require the
rendered drop rule instead.

The drop-counter read had the same weakness plus a worse one: it took
field three of the first line mentioning a counter, which is the packet
count only when the line begins with `counter`. On a rule such as
`tcp dport 9 counter packets 42 bytes 3000 drop` it printed the port
number. That shape is reachable, because the shipped ruleset includes
the operator drop-in directory ahead of the trailing default deny and a
drop-in may add its own counted drop. Extract by position within the
matched text, and take the last match rather than the first, since case
(a) falls through to the default deny that the ruleset emits as the
final rule.

Note in place at the baseline check that it still only proves a counted
drop rule exists somewhere in the table, not that it is the trailing
one; asserting rule position is a larger change than this warrants.

The sample output in the README is updated to the new wording.
 2026-07-23 04:16:32 +00:00
Johnathan Corgan 98560594cf Prove ACL rejection per peer by matching a single log line 
The four rejection checks were independent greps over the whole log:
two npubs, a context and a decision. Together they proved only that
node-a mentioned each npub somewhere, rejected somebody on an inbound
handshake, and rejected somebody by denylist. Nothing tied a rejection
to a named peer, and node-a lists both denied peers as auto_connect
peers, so it logs their npubs on the outbound connect path whether or
not any rejection ever happened. The actual message was never matched.

Replace them with a helper that requires every given string on ONE
line, and assert per denied peer: the real message, that peer's npub,
and the denylist decision together. Strings match in any order by
chaining fixed-string greps over the surviving lines, so the assertion
does not depend on how the log formatter orders a message and its
fields. A grep over empty input yields the empty string rather than
anything a caller could mistake for a match, so an unreadable container
times out and fails instead of passing.

Two limits are written at the call sites rather than left implied. The
decision conjunct discriminates nothing, since the two allowing
variants return early and denylist is the only value that can reach
that warning. And node-a authorizes before dialing, so it emits a
fully formed rejection line for each denied peer on the outbound path;
these assertions are satisfiable without the inbound check running at
all, and it is the peer-count assertions that would catch that.

Verified against the running harness: all three find a real matching
line. The defect shape they replace was exercised offline first, with
the three strings spread across three lines, where the old checks
passed and the new ones fail.
 2026-07-23 04:16:16 +00:00
Johnathan Corgan ebf34ae712 Read the wall clock directly for Nostr traversal timestamps 
The traversal clock cached a Unix millisecond value and an Instant at
first use, then served every later call by advancing the cached value
with the monotonic elapsed time. A monotonic clock does not tick while
the host is suspended, so once a machine had slept the returned value
trailed real time by the sleep duration for the rest of the process
lifetime, and never re-synced.

Almost everything that clock feeds is an absolute timestamp. The NIP-40
expiration tags on adverts and traversal signals, and the issuedAt and
expiresAt fields of offers and answers, are all computed as now plus a
TTL; the freshness and cache-pruning paths compare it against a
peer-authored, signed created_at. Once the host had slept longer than
signal_ttl_secs, every offer was published already expired, relays
dropped it, and the initiator timed out waiting for an answer with
traversal broken until a restart.

Read the wall clock on every call instead. This also removes a
mismatch inside the traversal failure-state map, which was written
here from the cached clock but written and read from the node
lifecycle with the real one.

Not platform-specific: monotonic clocks exclude suspended time on
Linux and Windows as well, so this affected any host that suspends.
A laptop is simply where a process lives long enough across a sleep
to notice.

The interval-shaped consumers hold up under a clock step. A forward
step, which is what a resume produces, saturates the punch start delay
to zero, and the attempt's own bounds are monotonic deadlines that are
unaffected. A backward step lengthens that delay and can cost one punch
attempt, which retries. Early eviction from the replay window cannot
admit a replay under the shipped defaults, because the freshness window
a replayed offer must also satisfy is strictly narrower than the replay
window itself.

The added test pins the contract and fires on a host that has genuinely
suspended, but it is not a regression guard for this defect: nothing
reachable from a unit test can simulate a suspend, so on a machine that
has not slept the old implementation passes it too. Its comment says
so rather than leaving a false sense of coverage.

Reported in https://github.com/jmcorgan/fips/issues/128
 2026-07-23 03:23:00 +00:00
Johnathan Corgan 291c4312dc Stop the ACL peer-count reader from turning "no answer" into zero 
wait_for_peers_exact fell back to 0 when it could not read a container's peer
count. Two of its six call sites are the ACL denial checks, which expect
exactly 0 connected peers — so a failed docker exec satisfied them on the
first iteration and the isolation property the suite exists to prove was
never observed. Confirmed against a container that does not exist: the old
reader returns "0" and the expect-zero comparison passes.

The reader now returns the empty string when the daemon does not answer,
which no numeric comparison can satisfy, and the timeout message says which
of the two happened — never answered, or answered the wrong number. Same
shape as admission-cap-test.sh's read_peer_count.
 2026-07-23 02:42:36 +00:00
Johnathan Corgan d21f818659 Check that log strings tests match on are still ones the daemon emits 
A test that greps the daemon's log for a message the daemon stopped emitting
does not fail. It stops observing, and an assertion built on it — especially
one expecting a count of zero — then passes because nothing can be seen
rather than because nothing happened. Several findings have come from that
one class, so it is now checked mechanically.

testing/check-log-strings.py extracts the strings test code matches against
daemon log text and requires each to exist in src/. It reads four shapes:
python `"..." in line` tests, the first argument of the shell log helpers,
bash associative-array pattern tables, and greps whose input is daemon log
output. Patterns carry regex syntax, so each is reduced to the longest
literal run of every alternation branch, with escaping resolved in the same
pass — `\.` is a literal dot and `.` is a wildcard, and conflating them would
let a pattern match text that is not there.

Scoping is by what a grep READS, not by what its file mentions. A suite greps
its own analyzer output, fipsctl JSON, Tor's log and ping output, and none of
those have to correspond to a string in src/; scoping on the file produced 34
false positives. Strings that legitimately do not come from src/ are named in
ALLOWED with a reason, so the exceptions are reviewable rather than invisible.

It found six dead strings, three of which were not previously known: a second
copy of "Excessive decrypt failures" in the interop pattern table, a dead
"Handshake error" branch inside an alternation whose other branches still
matched, and "bootstrap failed" in the NAT suite. All six are corrected to
what the daemon emits, or dropped where the surviving branch already covers
the case. Verified by breaking it two ways and confirming it reports.

Also here, since it is the same failure shape one layer down: the shared log
library returned an empty string for a container whose logs it could not
read, so analysing a missing container reported no panics, no errors and no
sessions, and exited 0 — indistinguishable from a clean run. It now reports
which containers could not be read and raises, matching the fix already
landed in the chaos copy.
 2026-07-23 02:42:25 +00:00
Johnathan Corgan 6c52b0e01e Let the static test network float so concurrent CI runs cannot collide 
Two local CI runs on one host both asked docker for 172.20.0.0/24 and the
second lost its whole static family to "Pool overlaps". Docker honours a
fixed subnet request verbatim, so the only robust fix is to stop making one:
fips-net now requests no subnet and docker assigns from its own pool, which
cannot hand the same range to two runs.

That means node addresses are not known before `up`, so peers address each
other by container hostname instead. The generator emits node-<id>, or the
topology's docker_host where the compose hostname differs — only the gateway
profile, whose services are gw-*. External peers keep the address the
topology gives them, since it is not ours to assign. The resolv.conf mount
stays: dnsmasq is what forwards these names to docker's resolver and .fips
to the daemon, so removing it would take out every .fips assertion.

generated-configs is now per-run as well. A shared directory let two runs
overwrite each other's node configs, which the subnet collision had been
hiding by killing runs before that window opened. The generator, the compose
bind mounts and env_file, the six scripts that read it, and teardown all
follow FIPS_CI_NAME_SUFFIX; unset, every path renders as before. Teardown
keeps the directory after a failed run, where it is the evidence of what the
failing nodes were configured with.

Three things this exposed that were wrong independently:

admission-cap built its tcpdump patterns from the topology file's docker_ip
literals. Floating the subnet makes those match nothing, which would have
left its expect-zero "no Msg2 leaked" assertion passing because it could no
longer see anything at all. It now reads addresses from the running
containers. Restarting the denied peers together also made them swap
addresses, so each peer's counts were really the pair's total; they are
restarted one at a time now, and a check fails the suite outright if two
denied peers ever share an address, because per-peer attribution is
impossible once they do.

Attribute lookups in the generator used a fixed ten-line window and read the
next node's fields when a node omitted an attribute. An external node
followed by an internal one was classified as internal, which under hostname
peering would emit a name that resolves nowhere. Lookups are bounded to the
node's own block; generated output is byte-identical for all eight
topologies.

The rekey outbound-only variant used to rewrite peer addresses to hostnames
to set up its scenario. The generator now does that everywhere, so the
rewrite matched nothing and was silently doing no work. It asserts the
premise instead, and fails if a numeric address ever reappears.

Verified by running three instances of this compose at once — tcp-chain plus
two independent meshes — which drew 10.128.2/3/4.0/24 with no overlap while
both meshes passed ping-test 20/20 over the real .fips path. tcp-chain is
run by neither CI runner, so it was checked by hand: chain peer counts 1/2/1
and multi-hop .fips reachable both directions over TCP.

gateway-lan still pins its own IPv4 and fd02:: ranges and is unchanged here,
so the gateway profile is not yet concurrency-safe.
 2026-07-23 02:05:56 +00:00
Johnathan Corgan 428773490f Close the gaps an adversarial review found in the new CI guards 
Four of these are places the parity guard could stop covering something without saying so, which is the failure mode it exists to prevent. Removing the deb-install suite array left it green, because the dispatch cross-check compared against a hardcoded name rather than the array it was meant to read. An arm-shaped line at an unexpected indent, or one written in quotes, was dropped silently by the arm pattern; unexpected indents are now a hard error rather than a quiet skip, and quoted arms parse. A matrix leg with a type of chaos or deb-install but no scenario key raised a Python traceback instead of reporting a problem, and a leg carrying a scenario but no suite key was skipped entirely even though scenario is now the identity for those legs.

The e2e builder no longer folds docker create's stderr into the container id. Docker prints warnings on success as well as failure, so a platform-mismatch warning would have become part of the id and made every later reference to that container fail, turning a working extraction into a spurious red.

Also corrects the comment about registering a new chaos assertion type, which named only one of the two key sets that actually need it.
 2026-07-22 22:41:24 +00:00
Johnathan Corgan 56f00058d4 Run the CI parity guard in both runners 
The guard that keeps "local green" and "GitHub green" meaning the same thing was invoked by nothing. Its only entry point was an exec behind an explicit --check-parity flag, which by exec-ing could not be combined with an actual run, and no workflow step called it at all. A guard nobody runs is a guard that does not exist, which is how the mixed-profile divergence on next survived unnoticed.

It now runs as the first stage of every local run and as its own job on GitHub. Locally it reports through the same result-recording path as every other stage, so a divergence sets the run's exit status instead of scrolling past. It is deliberately placed above the mode branches, so a divergence also fails --only, --test-only and --build-only runs: whichever subset was asked for, the claim that a local result means what a GitHub result means is what has broken. The GitHub side installs pyyaml explicitly rather than assuming the runner image carries it.

--check-parity keeps working exactly as before. Three comment blocks that described the old folded comparison are corrected here, in the workflow, the local runner and the testing README; they were spread across three files and only two mention the guard by name, so the third is best found by searching for what it claims rather than for the script.
 2026-07-22 22:03:08 +00:00
Johnathan Corgan e578a11b7a Compare every CI leg in the parity guard, not a folded token 
The guard collapsed all thirteen chaos legs to the token "chaos" and all five deb-install legs to "deb-install", so sixteen of the thirty-four integration legs were invisible to it. Deleting twelve chaos legs and four deb-install legs from a copy of the workflow left it still reporting "CI parity OK". The fold existed because the guard compared the cosmetic suite name, which really does differ between the runners; both sides have carried a directly comparable scenario field all along, so the fix is to compare that instead, along with the chaos flags.

The local suite set is now discovered by sweeping ci-local.sh for suite arrays rather than reading a hardcoded list of variable names, and the deb-install distro list is read from the suite's own script. A suite dispatched with no backing array is invisible to either approach, so every run_suite arm is now checked to have one and the guard names any that does not. That is not hypothetical: next dispatches its mixed-profile suite without an array, and the guard could only report it as missing from the local runner without being able to say why.

The guard also now refuses to run rather than failing obscurely when python3 or pyyaml is absent, since a parity check that cannot run must not look like one that passed. Verified by breaking each thing it guards: legs deleted from the workflow, drifted chaos flags, a fabricated suite array, and a dispatch arm whose array was removed. That last case initially did not fire because the arm pattern was pinned to the wrong indentation, which the test caught.
 2026-07-22 21:59:28 +00:00
Johnathan Corgan 99cb0a51fd Reject unknown keys when loading a chaos scenario 
The scenario loader read every section and member through raw.get with a default, so a mistyped key was silently ignored and the block it belonged to stayed default-constructed while the YAML still looked correct on the page. Writing "assertion:" for "assertions:" disarmed a scenario's only assertions and left it unable to return a non-zero exit code; "link_flap:" for "link_flaps:" turned off the churn the scenario existed to inject, and a calm mesh then ran under the name of a chaos test.

Keys are now checked at load time, at the top level and inside every section with a fixed schema, including the netem policy bodies, the {min, max} ranges and the assertion bodies themselves. The error names the offending key, the section it appeared in, and the keys that section does understand. Three mappings are deliberately exempt because their keys are names the author chooses rather than a schema, and two sub-trees are passed through whole; all five are listed at the key sets with the reason.

Adding an assertion type now means registering it, so the next person to add one gets a rejection rather than a silent no-op. That coupling is the point and it is written down where they will see it. Verified against the seventeen scenarios in the tree, all of which still load, and against fixtures for each failure mode: a top-level typo, a nested typo, a typo inside an assertion body, and a section left empty, which previously raised an unrelated AttributeError from the next line of the loader.
 2026-07-22 21:56:41 +00:00
Johnathan Corgan e9ca16f3a1 Stop the DNS resolver e2e tests certifying a binary that was never built 
The e2e scenarios build fips and fips-gateway in a builder image and extract them with two docker cp calls whose stderr went to /dev/null and whose exit status was never tested. Because build_fips_for_e2e ended in a log call it returned 0 on every path, so the caller's guard could not fire. A failed extraction left the previous run's binaries in the cache, they satisfied the caller's executable check, and the five e2e legs then exercised the previous commit's code and reported green.

The extraction now deletes the cached binaries before it starts, which is what makes the stale case impossible rather than merely detected, and it checks each docker cp independently, reports the failure with docker's own message, verifies each extracted file is non-empty and executable, and returns an explicit status. This mirrors what the deb-install suite already does.

Confirmed by inducing the failure rather than by observing a green run. With both extractions broken and a stale binary in place, the old code logged "Cached fips (26936 bytes)" and went on to build the runtime image around it; the new code reports both docker errors, deletes the stale binaries and fails the suite. Breaking either extraction on its own is caught separately, and an unmodified run passes 7 of 7.
 2026-07-22 21:54:37 +00:00
Johnathan Corgan ab750f60b3 Record why a chaos docker command failed 
Compose commands run with their output captured and with check set, so a failure raised a CalledProcessError and nothing ever looked at what the daemon had said. That exception reports the argv and the exit status and nothing else, so every scenario that died bringing its containers up left a runner log saying only that docker compose returned non-zero exit status 1, and the reason was captured and then discarded. In the run that prompted this work twelve scenarios failed that way and why they failed can no longer be established at all. Log the captured stderr and stdout before the error goes anywhere, keeping the tail of each so a verbose failure cannot fill the log file. A command that times out raises before its exit status is ever read, so that path records whatever it had emitted too, rather than leaving the same hole one shape over. Reproduced against both of the shapes this is meant to catch: an unusable network prefix now records the daemon's parse error, and a name already held by another container now records the conflict and names the container holding it. The success path logs nothing new, and a completed scenario's runner log is unchanged line for line.

The raising behaviour is deliberately untouched, since the abort flag and the exit code that came with it depend on it. The check is applied here rather than by subprocess so that a caller who passed check false gets the same record: both such callers are the compose down in teardown, which today can leave a mesh behind without saying so.

The same omission was costing the veth path its reasons. Failing ip commands had their stderr logged at debug, which the runner does not emit unless asked for verbose output, so a failed pair creation reached the log as nothing more than the names of the two interfaces it could not create. Raise that to a warning. The deletes issued to clear a stale pair are expected to fail and still say nothing.
 2026-07-22 08:26:59 +00:00
Johnathan Corgan 226c6994d3 Harvest chaos results only from a mesh that actually started 
Teardown runs from a finally, so it also runs after a failed setup, and it was guarded only on the topology and the compose file, both of which are set well before any container exists. A scenario that died bringing its containers up therefore ran the whole harvest anyway: final snapshots, docker logs, the analysis, the assertions and the metadata, all addressing containers by names that are global to the host. What it left behind was not an empty directory an investigator would notice but a full and plausible one, in the recorded case ten node logs and an analysis reporting two promotions and two parent switches, every byte of it from a different scenario's mesh. Gate the harvest on whether the containers ever started, and leave the compose down outside that gate so a partly successful start still gets cleaned up.

Record the outcome in a status file in every result directory, naming the run as completed, interrupted, aborted, setup-failed or teardown-failed, alongside the scenario, the seed and the container names it used. With the harvest gated, the presence of an analysis file is now itself proof that the scenario's own mesh existed, and the status file says which of the several ways a run can end applies. A run cut short by a signal keeps the existing exit codes rather than gaining one of its own: what it collected before stopping is real and still worth reporting, the status file records that it was truncated, and every wrapper already reports a Ctrl-C of its own.

Log collection never looked at the status of docker logs. It concatenated stdout and stderr unconditionally, so collecting from containers that were not there wrote the daemon's "No such container" reply into each node log and analysed the result as a mesh with no panics, no errors and no sessions, which reads exactly like a clean run and exits zero. Check the return code, and treat a harvest that cannot read every container, or that reads none, as a failed teardown. A teardown that raises now also reports on the same footing as a run that never started, instead of escaping past the exit codes as a bare traceback and being read as a malformed command line.
 2026-07-22 08:02:38 +00:00
Johnathan Corgan 4f55a281ac Let a failed chaos scenario fail the run 
The simulation caught every exception a scenario raised, logged it, and exited 0, so a scenario that died during setup was recorded as a pass. Local CI discarded the exit code in any case: run_chaos ends in a call whose own last statement is an echo, so it returned 0 whatever the scenario did, and the parallel launcher's wait therefore always succeeded and always recorded a pass. Between them no chaos failure of any kind could turn a local run red, and none could since the script was written.

Carry the abort out of the run and give it its own exit code, ordered ahead of the two content codes so a run that never produced a mesh cannot report that mesh's panic and assertion counts. Return the scenario's status from run_chaos so the launcher sees it. The documented exit codes were stale before this change and are rewritten in full, including that the argument parser rejects a malformed command line with the same code that reports panics.

Expect scenarios that have been reporting green to start reporting red. In the last recorded local run twelve of the thirteen scenarios never got past starting their containers, and all thirteen were counted as passes; those that now execute have not run for months, and their assertions have never been evaluated against a mesh of their own.
 2026-07-22 07:37:48 +00:00
Johnathan Corgan 2ef36c0071 Scope chaos host interface names per scenario and reap orphaned pairs 
Ethernet edges get a veth pair created in the host namespace and then moved into the containers, named from the node ids alone. ethernet-mesh and ethernet-only both connect n01 to n04 and always run together, so one tore down the other's live link. Hash the scenario suffix to four hex characters and insert those after the vh prefix; 15 characters is far too few to carry the suffix itself.

The names are now run-specific and never regenerated, so the delete-before-create no longer reclaims an interface an earlier run abandoned. ci-cleanup.sh takes over, deriving the names from the suffixes a run's teardown hands it. It is the first thing that script removes that is neither a docker object nor labelled, so an unscoped reap can sever a running bare simulation's links; testing/README.md and the --reap help say so.
 2026-07-22 07:35:11 +00:00
Johnathan Corgan 5be7c6d0cb Give each chaos scenario its own container names and config directory 
Chaos scenarios run four at a time under local CI, but every one of them claimed the container names fips-node-nNN and wrote its generated configs and compose file to the same generated-configs/sim directory. Container names are global in Docker and are not scoped by the compose project, so concurrent scenarios collided on both, and a scenario could start containers from a compose file another had overwritten.

Thread the existing FIPS_CI_NAME_SUFFIX into the simulation. run_chaos narrows the run-wide suffix to the scenario, and the sim reads it once when the topology is built, applying it to the container names and to the config directory basename. The compose template renders the name through the topology accessor instead of duplicating the literal, so one expression produces every chaos container name.

The suffix is empty when the variable is unset, so a bare chaos.sh run and the hosted CI jobs render byte-identical names and paths. Verified by rendering every scenario's compose file before and after with the variable unset and diffing.
 2026-07-22 07:35:11 +00:00
Johnathan Corgan 8ca2362e6c Follow IP semantics for the session datagram hop limit 
The forwarding path tested the hop limit before testing whether the datagram was
addressed to this node, and decremented only on the forwarding branch. Two
consequences followed. A datagram addressed here that arrived with a zero hop
limit was dropped rather than delivered. And a forwarder receiving a transit
datagram with one hop left transmitted it with zero for the next hop to discard,
wasting a transmission on every expiring datagram.

Delivery to the addressed node is no longer gated on the hop limit, and the
decrement now happens before the drop decision, so a datagram that would leave
with zero is not sent. Saturating subtraction folds the two arrivals that cannot
be transmitted, already exhausted and last hop, into a single test.

The reachable radius does not change, which is worth stating because it is easy
to assume otherwise. The old behavior refused to deliver at zero but was willing
to send at zero, and the new behavior is the mirror of that, so the two cancel: a
path of h links still delivers for any source value of h or more. What this
actually buys is the wasted transmission, the exhaustion counter charging at the
node that makes the decision rather than the one after it, conformance with the
specified semantics, and one real gain during a rolling upgrade, where an
unupgraded forwarder feeding an upgraded destination delivers one hop further
than either version does on its own.

Coordinate cache warming moved ahead of both decisions, so a transit datagram
later dropped for hop limit still contributes its plaintext coordinates. The only
arrivals this newly warms from are those with a zero hop limit, and every insert
they can make is already achievable with a value of one, so it grants nothing
that was not already available and adds no memory growth the cap does not bound.

Four tests added: delivery at zero to this node, the transit drop at one, the
transit pass at two, and the one per hop decrement across a three node chain,
which brackets the emitted value from both sides since it is only observable
through what the next hop does. One existing test was renamed because its
destination was this node, so it never exercised the transit path its name
claimed, and it asserted nothing at all.
 2026-07-22 01:53:53 +00:00
Johnathan Corgan 75d7077880 Open 0.4.2-dev cycle on maint after v0.4.1 release 
Bump the version to 0.4.2-dev and update the status badge and prose. The
0.4.x patch line continues; v0.4.1 remains the shipped release it points at.

Follows the shape of the v0.4.0 rollover commit. No maint reset was needed
this time: v0.4.1 was cut from maint, so the tag is already on this line and
it simply continues.
 2026-07-19 19:05:13 +00:00
Johnathan Corgan 15db6471db Set version to 0.4.1 and correct the v0.4.0 release date 
Drop the -dev suffix from the package version and refresh the lockfile.
The version is single-sourced from CARGO_PKG_VERSION, so build.rs and
src/version.rs need no change.

Also fix the archived v0.4.0 release notes, which still read
"2026-06-21 (provisional)" while the changelog records the actual
release date of 2026-06-27. The date was confirmed in the changelog at
the time but not in the two release-notes files that carry it, so the
shipped notes have been showing a wrong date and the word provisional
since that release. The root mirror is regenerated for v0.4.1, so only
the archived copy needed correcting.
 2026-07-19 17:59:37 +00:00
Johnathan Corgan 146d19a8d8 Add v0.4.1 changelog entry and release notes 
Back-fill the changelog for the six user-facing changes on this line
since v0.4.0: the antipoison FPR cap default, the bloom probe and
secp256k1 context performance work, the coordinate-cache and path-MTU
fixes, and the removed parent_switched counter. The Unreleased section
was empty, so all six entries are new.

Write the v0.4.1 release notes and mirror them to the root copy. The
notes lead with the FPR cap change and state plainly that this is the
second raise of that default in two releases, that it buys headroom
rather than fixing the underlying fixed-filter constraint, and that the
structural remedy is the v2 filter work. They also flag the rolling
upgrade window, where an upgraded and a not-yet-upgraded node can
disagree about mesh size, and point consumers of parent_switched at
parent_switches.

Bump the README status badge and the roadmap prose together so the two
agree.
 2026-07-19 17:21:57 +00:00
Johnathan Corgan bda327b5f5 Raise the bloom antipoison FPR cap default to 0.20 
The inbound FilterAnnounce cap at 0.10 rejects aggregates that are
legitimately near their operating ceiling, before the network reaches
the fixed-filter capacity limit. Raise the default to 0.20, which
corresponds to fill 0.7248 at k=5, about 2,114 entries on the 1 KB
filter (Swamidass-Baldi).

Also remove the duplicate default definitions in BloomConfig. Each
default was written twice, once in impl Default and once in the serde
default function, with nothing enforcing that they agree, so a config
file that omits the key took a different path from one that sets it.
impl Default now delegates to the serde default functions, leaving a
single source of truth.
 2026-07-19 17:18:49 +00:00
Johnathan Corgan 78377208af Fix two sidecar container references that skipped the per-run suffix 
The isolation loop and the failure-log dump both built container names
without the run suffix, so under a suffixed run they addressed containers
that do not exist.

The consequence was worse than the visible failure. Two of the three
checks in that loop assert a ping FAILS, and a ping into a non-existent
container fails for the wrong reason - so both passed vacuously, and the
security assertion the loop exists for was silently a no-op. Only the
third check, which expects a ping to succeed, noticed anything was wrong.

Add a guard that fails loudly when the container is absent, so a future
naming slip cannot quietly turn these back into no-ops rather than
surfacing as one confusing failure among two false passes.

These sites were missed because the suite passes when run by hand: with no
suffix set the unsuffixed names are correct, and the defect only appears
under the automation that sets one. Verified this time with the suffix
set, which is the condition that matters: 18 of 18, container names
resolving to real containers.
 2026-07-19 07:40:07 +00:00
Johnathan Corgan 26a579b1c9 Claim a free address range for the sidecar network instead of hardcoding one 
Making the sidecar network name unique per run exposed a latent problem
rather than causing a new one. With a fixed name, compose found any
existing network and reused it, which quietly masked the fact that the
address range was hardcoded. With a unique name there is nothing to
reuse, so the range is requested every time and the run fails outright if
anything already holds it.

Claim the range instead of assuming it. The suite now tries to create its
network on a candidate range and, if the daemon reports an overlap, moves
to the next candidate. The daemon's own address pool becomes the arbiter,
so two concurrent runs cannot land on the same range at all - as opposed
to a range derived from the run identifier, which only makes a collision
unlikely, and a collision is the precise failure being avoided.

The node addresses are derived from whichever range is claimed rather than
written alongside it, including the gateway address the isolation check
probes. Moved to a range that nothing in the tree claims and that sits
outside the daemon's default pool.

Only an overlap is worth advancing on. Any other creation failure is real
and is reported with the daemon's own message, rather than being retried
sixty-four times and buried - a lesson from a failure elsewhere in the
harness this week that was undiagnosable because its error output was
discarded.

All three nodes now attach to the pre-created network as external, so
teardown removes it explicitly.

Verified: three concurrent claims take three distinct ranges; a
non-overlap error fails fast rather than looping; the suite passes 18/18;
and the claimed gateway address is confirmed pingable when not firewalled,
so the isolation assertions still mean something.
 2026-07-19 07:17:19 +00:00
Johnathan Corgan 3ebb14eda4 Make the admission-cap final peer_count read tolerate a busy daemon 
The suite red-ed with an empty final peer_count while every substantive
assertion passed: both denied peers showed inbound handshake attempts and
zero outbound responses, which is the gate this suite exists to verify.

Two pre-existing defects combined. The final count was sampled once, in
the same second the load driver issues its last round of peer restarts, so
the read can land on a daemon busy with those restarts and return nothing.
And the fallback meant to cover that could never fire, because the shell
applies it to the last stage of the pipeline, which succeeds on empty
input - so an unanswered query was indistinguishable from a genuine zero,
and reported as a cap violation.

Poll the final read for a short window instead, and give the two cases
distinct output. A real regression still fails, just after the retry
window, since the loop exits early only on the expected value. Extract the
read the two phases share rather than leaving them to drift.
 2026-07-19 06:46:50 +00:00
Johnathan Corgan e4a854f6b0 Stop concurrent local CI runs from clobbering each other's containers 
Two runs on one host destroyed each other's containers, producing
mid-test "No such container" failures that look like real defects. The
automated builder runs a full local CI on the same box every few minutes,
so the machine is contended almost always and this has red-ed both a hand
run and an automated gate.

Two independent causes are fixed here. Container names were hardcoded, and
docker names are global rather than scoped by compose project, so two runs
collided on the same name; every name now takes an optional suffix that
the harness sets from the run id. And the cleanup sweep matched a label
shared by every run, so one run's teardown force-removed another's
containers; resources now also carry a per-run label and the sweep can be
narrowed to it.

A third hazard turned up that was not in the original report: the sidecar
suite passes explicit compose project names, which override the run-scoped
project and put it outside the shared prefix entirely. Its project names,
network, and derived container references are now scoped too.

Both are default-off. With the suffix unset, names render exactly as they
do today and a bare compose invocation is unchanged, which is what keeps
the hosted CI and the documentation correct. A cleanup run with no run id
still reaps everything, which is what a manual "clear the box" wants.

The literal-name sweep was not sufficient: six scripts build container
names dynamically from node labels, and two suites create their own
containers outside compose. Those are handled at their construction sites.

Verified: syntax check on all modified scripts; compose validation on
every modified file with the suffix both set and unset; and a synthetic
two-run reproduction that shows the old cleanup destroying a bystander run
and the new one leaving it alone.

Known gap: subnets are still hardcoded, so two concurrent full runs will
still collide on address-pool overlap. That fix reaches into topology
configs, chaos scenarios, diagrams and production source, so it is left
for its own change rather than half-done here.
 2026-07-19 06:41:25 +00:00
Johnathan Corgan 7a97599921 testing: floor the rekey second-cutover wait at the log-analysis threshold 
The rekey suites' Phase 4 polled until the initiator-cutover count
reached its value at phase entry plus two, while Phase 6 asserts an
absolute minimum of four. On an unloaded host the ping phases can
outrun the jittered first rekey cycle, so Phase 4 can begin with a
single cutover on the books, release at three, and leave Phase 6 to
lose a sub-second race against the remaining first-cycle cutovers.
A rekey-outbound-only run failed exactly this way: five counted
cutover lines by t=32.6s of container life, with the Phase-6 count
snapshot at t~32.4s seeing three.

Floor the Phase-4 wait target at four so a released wait guarantees
the Phase-6 assertion — the docker-logs count is monotone over a
container's lifetime. A genuine rekey stall still times out the
bounded wait and fails Phase 6 with the real count.

Also drop the second-cycle framing from the comment, the phase
banner, and the assertion description: with rekey timers resetting at
each cutover a second cycle cannot begin inside this test's window,
so the counted events are first-cycle cutovers spread across the
topology's links, emitted once or twice per completed link rekey
depending on which side's promotion path logs them.
 2026-07-19 04:12:17 +00:00
Johnathan Corgan fbb4fb8879 testing: wait for both second-cycle rekey cutovers before asserting the count 
Phase 4 of the rekey suite waited for only one more initiator cutover
(guaranteeing three) while Phase 6 asserts at least four. The fourth
cutover then had to land in the brief window between the wait returning
and the Phase 6 log snapshot, so host load could push it past the window
and fail the run even though every cutover completed correctly. Wait for
two more cutovers, the full second rekey cycle, matching the Phase 6
threshold, so the asserted count is guaranteed before it is checked.
 2026-07-16 00:42:04 +00:00
Johnathan Corgan 567e6a535e identity: reuse one shared secp256k1 context 
Every sign/verify/key-derive site built a fresh context via
Secp256k1::new(), which allocates a Secp256k1<All> and runs
randomization/blinding table setup on each call. Introduce one
crate-wide LazyLock<Secp256k1<All>> and reuse it across the local, peer,
and auth sites (and their tests). Behavior-neutral: identical secp256k1
API calls, only the context lifetime changes, and the shared All context
still performs the standard construction-time blinding.
 2026-07-12 16:33:53 +00:00
Johnathan Corgan 6011d233c1 discovery: keep tighter path_mtu when applying a LookupResponse 
An originator handling a LookupResponse unconditionally overwrote the
cached path_mtu_lookup entry, so a looser (larger) estimate in a later
response could clobber a tighter value already learned from a reactive
MtuExceeded or PathMtuNotification. Read-and-compare before writing and
keep the minimum, so a looser discovery estimate no longer loosens the
clamp. Add a regression test.
 2026-07-12 16:29:01 +00:00
Johnathan Corgan cb5a32693e tree: remove redundant parent_switched metric counter 
parent_switched was incremented on the line immediately before
parent_switches at every site and never independently, so the two
counters were always identical. Drop parent_switched from TreeMetrics,
its snapshot, TreeStatsSnapshot, the show_tree fixture, and the fipstop
render, keeping parent_switches as the sole counter.
 2026-07-12 15:53:03 +00:00
Johnathan Corgan 81e4207631 bloom: compute the SHA-256 digest once per double-hash probe 
BloomFilter derived all k hash functions from one SHA-256 digest but
recomputed that digest inside the per-function loop, so every insert and
contains ran SHA-256 hash_count times (5x at the default) over the same
bytes. Hoist the digest out of the loop: base_hashes() computes it once
and returns (h1, h2), and bit_index() derives each of the k indices with
the same (h1 + k*h2) mod m arithmetic. Bit-for-bit identical output; this
is the hottest path in packet forwarding and mesh-size estimation.

Adds a test pinning the bit indices against the double-hashing formula
recomputed independently, proving the refactor is behavior-neutral.
 2026-07-11 01:22:07 +00:00
Johnathan Corgan 1d277e67c7 fix(tree): invalidate coord cache on parent-lost via peer removal 
handle_peer_removal_tree_cleanup reparents or self-roots the node when
its parent link drops, but omitted the coordinate-cache invalidation that
every other position-change path performs. Cached entries for downstream
destinations kept the node's now-stale coordinate prefix, and because
find_next_hop refreshes the TTL on every routing access, an actively
routed stale entry never self-expired — corrected only by a fresh insert.

Mirror the loop-detection branch: inside the parent-loss changed block,
invalidate both classes — invalidate_via_node (reparent) and
invalidate_other_roots (self-root). Add regression tests for a parent-link
removal that reparents (via-node entry dropped, same-root sibling
preserved) and one that self-roots (both via-node and stale old-root
entries dropped).
 2026-07-02 22:03:36 +00:00
Johnathan Corgan 2491091868 docs(testing): document ci-local.sh run isolation, preemption, and cleanup 
Add a "Running CI locally" section to testing/README.md covering the
ci-local.sh orchestrator: the per-run FIPS_CI_RUN_ID override and how it
scopes compose projects, image tags, and per-chaos-child subnets so
simultaneous runs on one host never collide; the cancellation exit codes
(130/143 vs 0/1) and how a preempting worker interprets them; and the
label-based cleanup via --reap / ci-cleanup.sh.
 2026-06-29 16:47:37 +00:00
Johnathan Corgan 965de26239 testing: make ci-local.sh preemption-safe with per-run docker isolation 
A CI worker may preempt an in-flight ci-local.sh run (SIGTERM, then SIGKILL
after a grace period) to restart on a newer commit. For that kill to be safe,
the script must clean up after itself and never let a dying run collide with
its restart. It previously had no signal handling, shared the default compose
project name across runs, and tore down each suite only at the suite end.

- Derive a per-run id (honoring FIPS_CI_RUN_ID, else short-sha+random) and
  namespace every docker resource to it: a fipsci_<run>_<suite> compose project
  per suite and per parallel chaos child, and per-run image tags
  (fips-test:<run>, fips-test-app:<run>) retagged to :latest only after both
  builds succeed so :latest never points at a half-built image.
- Install a bounded, idempotent teardown trap on SIGTERM/SIGINT (+ EXIT): reap
  parallel chaos children, then force-remove this run's docker resources via
  the new ci-cleanup.sh, wrapped in timeout so a stuck down cannot wedge it.
- Exit 143 (SIGTERM) / 130 (SIGINT), distinct from 0 (pass) / 1 (failed), so a
  preempting worker tells a cancelled run from a real failure.
- Add ci-cleanup.sh (also ci-local.sh --reap): force-removes leftover CI
  resources by the com.corganlabs.fips-ci=1 label and the fipsci_ project
  prefix, robust to however a prior run died.
- Label every per-suite docker resource so the label sweep reaps it after a
  SIGKILL regardless of network name: direct docker run/network resources, the
  sidecar compose services, and every per-suite compose network (acl-allowlist,
  boringtun, firewall, nat, static, both tor suites, and the chaos generator
  template). Parametrize the static/sidecar compose image refs so the per-run
  tags are honored.
- Give each parallel chaos child a unique /24 from 10.30.x (a new --subnet
  override on the sim CLI, assigned per-child in ci-local.sh) so parallel
  children never collide on a shared docker subnet, and a chaos net can never
  span a fixed-subnet suite (sidecar/static in 172.20.x). 10.30.x sits outside
  docker's default-address-pool range, so an auto-assigned net cannot land on
  it either; node IPs derive from the subnet, so no scenario config changes.
 2026-06-29 14:23:46 +00:00
Johnathan Corgan ab915d0479 testing: convert remaining fixed-sleep settles to progress-aware polling 
Replace the fixed post-first-rekey and post-second-rekey settle sleeps in
the rekey integration test, and the fixed-deadline baseline wait in the
interop test, with the deterministic wait_until_connected progress-aware
polling helper already used for the initial baseline convergence.

Each converted site fails fast with structured diagnostics when the mesh
is stuck and extends its deadline only while pairwise reachability is
still climbing, removing the wall-clock settle windows that produced
intermittent connectivity failures after a rekey.
 2026-06-29 00:40:06 +00:00
Johnathan Corgan 8f30924fc7 Open 0.4.1-dev cycle on maint after v0.4.0 release 
Fast-forward maint to the v0.4.0 release and open the 0.4.x patch
series: bump the version to 0.4.1-dev and update the status badge.
 2026-06-27 18:29:34 +00:00
Johnathan Corgan d5ee526f0e Release v0.4.0: bump version and finalize CHANGELOG date 
Bump the crate version from 0.4.0-dev to 0.4.0 and stamp the 0.4.0
CHANGELOG heading with the release date.
 2026-06-27 15:57:58 +00:00
ArjenandJohnathan Corgan 22a5b3e5c6 packaging(openwrt): default .apk WAN port to DSA name 'wan' 
The shared fips.yaml ships ethernet.wan.interface: "eth0", the default
WAN port on OpenWrt 24 and earlier. OpenWrt 25 (DSA) boards, which the
.apk package targets, name the WAN port "wan" instead. Rewrite the
staged copy at build time so the as-installed config binds the Ethernet
transport to the right port out of the box, without maintaining a second
copy of the config file. The .ipk package keeps "eth0".

Update the openwrt README to document eth0 (24) vs wan (25/DSA) and add
a CHANGELOG entry.
 2026-06-26 03:31:17 +00:00
Johnathan Corgan 3ea7ca1fd1 ci(openwrt): retry Blossom upload and skip nostr event on failure 
A transient timeout uploading the built .ipk to the Blossom CDN
(blossom.primal.net) failed the entire OpenWrt Package run, and because
the GitHub release job depends on the build jobs, a CDN blip would block
every OpenWrt artifact from the release. Blossom/nostr distribution is
supplementary; the package already ships as a GitHub release artifact.

Wrap the upload in a 3-attempt retry with backoff to absorb transient
timeouts, mark the step continue-on-error so a persistent outage no
longer fails the build, and guard the NIP-94 publish on the upload
succeeding so no event is created with an empty URL. Applied to both the
.ipk and .apk build jobs.

Claude-Session: https://claude.ai/code/session_01A2pYfSypNmmG4HyHwZuLex
 2026-06-21 14:40:50 +00:00
Johnathan Corgan 262d98a8eb Finalize v0.4.0 release content: CHANGELOG, release notes, README 
Prepare the source tree for the v0.4.0 release cut, leaving the version
at 0.4.0-dev (the version bump rides a separate release-candidate
commit).

- CHANGELOG: backfill the missing entry for the route-class transit
  counters and fipstop routing-tab reorg, then reorganize the Unreleased
  block from a flat per-commit list into topic-grouped subsections
  mirroring the 0.3.0 entry (coalescing interim fixes into net-effect
  descriptions without dropping technical detail), and stamp it as
  [0.4.0] - 2026-06-21 with a fresh empty Unreleased block. The date is
  provisional and reconfirmed at the final tag.
- Release notes: refresh both RELEASE-NOTES.md and the versioned archive
  (kept byte-identical) to cover the OpenWrt .apk packaging, the Nix
  flake, the macOS self-traffic checksum fix (#117), and the route-class
  transit counters.
- README: bump the status badge to v0.4.0 so it matches the prose.

Claude-Session: https://claude.ai/code/session_01A2pYfSypNmmG4HyHwZuLex
 2026-06-21 13:45:19 +00:00
Johnathan Corgan 274b09d4ff node: classify transit forwards by route class; regroup fipstop routing tab 
Add six forwarding counters that partition transit-forwarded packets by their
tree relationship to the chosen next hop: tree-up (peer is our ancestor),
tree-down (peer is our descendant and the destination is within its subtree),
tree-down-cross (peer is our descendant but the destination is outside its
subtree), cross-link descend (lateral peer, destination within its subtree),
cross-link ascend (lateral peer, destination outside its subtree), and
direct-peer. The six classes sum to forwarded_packets, asserted by a unit
test. Classification is computed from tree coordinates at the transit
chokepoint, so the error-signal routing callers are excluded.

The two "outside the chosen peer's subtree" classes are both up-and-over
forwards but differ in what they depend on. Tree-down-cross is the
dive-to-tree-child cut-through: we forward down to our own child for a
destination not beneath it, which is only possible because the child
advertised cross-link reach upward to us, beyond its own subtree. Its count
measures how much forwarding depends on that upward advertisement, i.e. what
would change if cross-link advertisements were narrowed to subtree-entry only.
Cross-link ascend, by contrast, uses the node's own lateral cross-link learned
from a peer's split-horizon advertisement, so it does not depend on any upward
advertisement.

Surface the counters through the forwarding stats snapshot (control socket,
show_routing and show_status) and reorganize the fipstop routing tab so its
two columns separate own/endpoint traffic (received, delivered, originated)
from forwarded/transit traffic (the route-class breakdown and drop reasons),
with the tree-down-cross line visually flagged.
 2026-06-20 14:56:54 +00:00
ArjenandJohnathan Corgan 0f1fd18c25 packaging(openwrt): SDK-free .apk build for OpenWrt 25+ and control-socket fix 
Add OpenWrt .apk packaging for OpenWrt 25+, where apk-tools is the
mandatory package manager. The existing .ipk continues to cover OpenWrt
24.x and earlier. Built SDK-free like the .ipk: it reuses the
cargo-zigbuild cross-compile and the shared installed-filesystem payload
under openwrt-ipk/files, and assembles the ADB container with the
official `apk mkpkg` applet from apk-tools 3.0.5 built from source, so no
OpenWrt SDK image is needed.

The package-openwrt workflow is refactored so a single compile-binaries
job cross-compiles and strips each arch once; both the .ipk and .apk
packagers consume the binaries via a new --bin-dir flag instead of each
recompiling. A build-apk job (aarch64, x86_64) builds apk-tools,
packages, and structurally verifies the .apk with `apk adbdump`. Releases
now publish .apk artifacts and checksums alongside .ipk; the release
download is scoped to fips_* so the shared raw-binary artifacts are not
swept into the published release. apk-version.sh maps a release tag or
commit height to an apk-tools-valid version, covered by a case-table test.
Packages are unsigned, installed with `apk add --allow-untrusted`,
matching the .ipk posture.

Also fix the OpenWrt control socket: the init script now pre-creates
/run/fips before starting the daemon, the procd equivalent of the systemd
unit's RuntimeDirectory=fips. Without it, on a fresh boot the daemon
resolves its control socket to /tmp (since /run/fips does not yet exist),
while fips-gateway later creates /run/fips for its own gateway.sock,
leaving fipsctl/fipstop resolving a /run/fips/control.sock the daemon
never bound.
 2026-06-20 14:44:05 +00:00
ArjenandGitHub 225fab29ab fix(tun): complete L4 checksum on hairpinned self-traffic (macOS) (#117) 
Self-addressed TCP/UDP connections to a node's own <npub>.fips address
half-opened and hung on macOS. macOS routes self-traffic as loopback (a
LOCAL route via lo0), which defers the transport TX checksum, but the
point-to-point utun then egresses the packet into the daemon with only
the pseudo-header partial checksum present. The hairpin path added in
9a9e90a re-injected these verbatim, so the local stack dropped every
segment whose checksum MSS clamping didn't happen to rewrite: the
SYN/SYN-ACK got through (clamping recomputes them) but the bare ACK,
data, and FIN were dropped for a bad checksum, leaving the listener
stuck in SYN_RCVD.

Recompute the TCP/UDP checksum for self-addressed packets on the hairpin
path before re-injection, completing the self-delivery 9a9e90a started
(which only covered ICMP and the TCP handshake). Linux is unaffected: it
loops self-traffic via lo before the TUN, so the hairpin branch never
fires and checksums are already valid.

Confirmed on macOS: a self-connect that previously timed out now
completes in ~9ms with payload delivered.
 2026-06-18 08:36:57 -07:00
ArjenandJohnathan Corgan effd69bd53 ci(openwrt): bump Node 20 actions to Node 24 majors 
checkout v4->v6, cache v4->v5, upload-artifact v4->v7,
download-artifact v4->v8 to clear the Node 20 runner deprecation.
 2026-06-17 17:58:37 +00:00
Johnathan Corgan 3749853716 packaging(openwrt): publish checksums-openwrt.txt alongside .ipk artifacts 
The linux/macos/windows package workflows each publish a
checksums-<platform>.txt sha256 file with their release artifacts, but
the OpenWrt workflow attached only the .ipk files with no checksum
coverage. Generate checksums-openwrt.txt over the .ipk outputs using the
same sha256sum idiom as the other platforms and add it to the release
file set.
 2026-06-17 17:58:37 +00:00
ArjenandJohnathan Corgan 289e5f8571 ci: bump Node 20 actions to Node 24 majors 
Clear the GitHub Node 20 runner deprecation across CI, AUR, and the
Linux/macOS/Windows packaging workflows:

  actions/checkout          v4 -> v6
  actions/cache             v4 -> v5
  actions/upload-artifact   v4 -> v7
  actions/download-artifact v4 -> v8

package-openwrt.yml is handled separately on fix/openwrt-checksums.
 2026-06-17 17:58:37 +00:00
sandwichandJohnathan Corgan 3733349d33 packaging(nix): add a Nix flake for reproducible from-source builds 
Add a flake that builds all four binaries (fips, fipsctl, fips-gateway,
fipstop) on Linux and macOS, pinning the exact toolchain from
rust-toolchain.toml (1.94.1 + rustfmt/clippy) via fenix so Nix builds
match CI and the AUR/Debian packaging.

Wire up the build-time native deps the source tree needs: pkg-config and
bindgenHook (libclang) for the rustables/libdbus-sys bindgen step, and
dbus for bluer's BLE support. autoPatchelfHook rewrites the binary RPATHs
so the daemon resolves libdbus-1.so.3 and libgcc_s.so.1 from the Nix store
at runtime — without it `fips` fails to load on NixOS, which has no global
/usr/lib.

Outputs: packages.{default,fips}, apps for each binary, checks.fips, and a
devShell with the pinned toolchain plus cargo-edit. Tests are skipped in
the package build since they exercise TUN devices, raw sockets, and mDNS
that aren't available in the sandbox, mirroring the AUR/Debian packaging.

Verified end-to-end in a pure Nix store (nixos/nix container): nix build,
nix flake check, and running all four binaries succeed against the
committed flake.lock.

Documented across the install and developer docs: a Nix / NixOS section in
packaging/README.md, the from-source guide in docs/getting-started.md
(noting the flake produces binaries only, with NixOS system integration
through the system configuration rather than the installer), the CHANGELOG,
README, CONTRIBUTING, and the v0.4.0 release notes.

Co-authored-by: Johnathan Corgan <johnathan@corganlabs.com>
 2026-06-17 16:36:38 +00:00
ArjenandJohnathan Corgan 9a9e90a32c fix(tun): loop back self-addressed packets for local delivery 
A packet destined for our own mesh address reached the TUN reader on
macOS (utun egresses self-traffic into the daemon despite the lo0 host
route) and was pushed onto the mesh outbound path, where it was dropped
for lack of a session/route to self. Hairpin self-addressed packets back
to the TUN writer instead, so ping6 and connections to our own
<npub>.fips address are delivered locally.

On Linux the kernel already loops self-traffic via `lo` before it reaches
the TUN, so the branch never fires there; the check is kept unconditional
as a daemon-level delivery invariant and to keep it covered by Linux CI.
 2026-06-17 14:32:17 +00:00
159 changed files with 10014 additions and 2935 deletions
Expand all files Collapse all files
.config/nextest.toml
+17
View File
@@ -5,6 +5,23 @@ junit = { path = "junit.xml" }
# occasional msg1 under burst load even with the per-edge repair loop.
# Allow a retry rather than failing the whole CI run on a single
# dropped packet.
#
# Deliberately scoped to [profile.ci] and NOT applied locally, which makes
# the two gates disagree: the hosted runner retries a flaky test twice, the
# local sweep fails on the first failure. The asymmetry is intended and this
# is the record of why, since an undocumented one is indistinguishable from
# an oversight.
#
# It is here for shared-runner packet loss, a property of the hosted
# environment and not of the code. Applying it locally would suppress a real
# local flake, and a failure that only reproduces under load is a robustness
# bug to fix rather than to retry past. Keeping the local sweep strict is
# what makes it the sharper of the two gates.
#
# The cost, stated rather than hidden: a test that fails once and passes on
# retry is reported green here with no separate signal, so a genuine
# intermittent failure can be absorbed. If that starts mattering, the fix is
# to surface retried-but-passed tests, not to drop the retries.
retries = 2
[test-groups]
.github/workflows/aur-publish-git.yml
+1 -1
View File
@@ -17,7 +17,7 @@ jobs:
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v4
- uses: actions/checkout@v6
- name: Patch PKGBUILD-git b2sums for local assets
run: |
.github/workflows/aur-publish.yml
+2 -2
View File
@@ -41,7 +41,7 @@ jobs:
set -euo pipefail
pacman -Sy --noconfirm --needed base-devel namcap git curl
- uses: actions/checkout@v4
- uses: actions/checkout@v6
- name: Resolve package version
id: ver
@@ -176,7 +176,7 @@ jobs:
echo "version=${TAG#v}" >> "$GITHUB_OUTPUT"
echo "pkgrel=${PKGREL}" >> "$GITHUB_OUTPUT"
- uses: actions/checkout@v4
- uses: actions/checkout@v6
with:
ref: ${{ steps.tag.outputs.tag }}
.github/workflows/ci.yml
+45 -103
View File
@@ -38,12 +38,12 @@ env:
# unreliable on GitHub-hosted runners.
# tor-directory — same; live Tor dependency.
#
# Granularity-only differences (same coverage, different matrix shape —
# NOT a divergence):
# deb-install — split here into per-distro legs (debian12/debian13/
# ubuntu22/ubuntu24/ubuntu26) for parallelism; local runs the
# same distro set in one suite.
# dns-resolver — single leg here; runs all scenarios (same as local).
# The two runners express the same work in different matrix shapes, and the
# parity guard compares through that shape rather than around it: chaos legs
# are compared per scenario (and per flag) via their `scenario:` field,
# deb-install legs per distro. The one leg still compared at leg granularity
# is dns-resolver — a single leg here, running all of its scenarios
# internally, exactly as the local suite does.
# ─────────────────────────────────────────────────────────────────────────────
# ─────────────────────────────────────────────────────────────────────────────
@@ -52,11 +52,34 @@ env:
# Builds on Linux x86_64, Linux aarch64, and macOS.
# ─────────────────────────────────────────────────────────────────────────────
jobs:
ci-parity:
name: CI parity
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v6
- name: Install Python deps
run: pip3 install --quiet pyyaml
- name: Check local and GitHub runners cover the same work
run: bash testing/check-ci-parity.sh
- name: Check test log matchers against the strings src/ emits
run: python3 testing/check-log-strings.py
- name: Check no tested function's exit status is a log call's
run: python3 testing/check-trailing-log.py
- name: Check nothing resolves the shared mutable test image
run: bash testing/check-image-scoping.sh
# Hermetic: synthetic ping functions, no containers, ~45s. Lives beside
# the other two so both runners gate on it identically — putting it in
# only one would create exactly the drift check-ci-parity.sh exists to
# catch, and it is invisible to that checker either way since it is not
# a matrix suite.
- name: Run convergence-gate unit tests
run: bash testing/lib/wait-converge-test.sh
fmt:
name: Format check
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v4
- uses: actions/checkout@v6
- uses: actions-rust-lang/setup-rust-toolchain@v1
with:
components: rustfmt
@@ -68,7 +91,7 @@ jobs:
name: Clippy
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v4
- uses: actions/checkout@v6
- name: Install system dependencies
run: sudo apt-get update && sudo apt-get install -y libdbus-1-dev
- uses: actions-rust-lang/setup-rust-toolchain@v1
@@ -77,7 +100,7 @@ jobs:
cache: false
rustflags: ''
- name: Cache Cargo registry + build
uses: actions/cache@v4
uses: actions/cache@v5
with:
path: |
~/.cargo/registry
@@ -102,7 +125,7 @@ jobs:
- os: windows-latest
steps:
- uses: actions/checkout@v4
- uses: actions/checkout@v6
- name: Set SOURCE_DATE_EPOCH from git (Unix)
if: runner.os != 'Windows'
@@ -130,7 +153,7 @@ jobs:
rustflags: ''
- name: Cache Cargo registry + build
uses: actions/cache@v4
uses: actions/cache@v5
with:
path: |
~/.cargo/registry
@@ -159,7 +182,7 @@ jobs:
# Upload the Linux binary so integration jobs can use it without rebuilding
- name: Upload Linux binary
if: matrix.os == 'ubuntu-latest'
uses: actions/upload-artifact@v4
uses: actions/upload-artifact@v7
with:
name: fips-linux
path: |
@@ -180,7 +203,7 @@ jobs:
runs-on: ubuntu-latest
needs: [build]
steps:
- uses: actions/checkout@v4
- uses: actions/checkout@v6
- name: Set SOURCE_DATE_EPOCH from git
run: echo "SOURCE_DATE_EPOCH=$(git log -1 --format=%ct)" >> "$GITHUB_ENV"
@@ -195,7 +218,7 @@ jobs:
rustflags: ''
- name: Cache Cargo registry + build
uses: actions/cache@v4
uses: actions/cache@v5
with:
path: |
~/.cargo/registry
@@ -236,7 +259,7 @@ jobs:
runs-on: macos-latest
needs: [build]
steps:
- uses: actions/checkout@v4
- uses: actions/checkout@v6
- name: Set SOURCE_DATE_EPOCH from git
run: echo "SOURCE_DATE_EPOCH=$(git log -1 --format=%ct)" >> "$GITHUB_ENV"
@@ -248,7 +271,7 @@ jobs:
rustflags: ''
- name: Cache Cargo registry + build
uses: actions/cache@v4
uses: actions/cache@v5
with:
path: |
~/.cargo/registry
@@ -271,7 +294,7 @@ jobs:
name: Unit tests (Windows)
runs-on: windows-latest
steps:
- uses: actions/checkout@v4
- uses: actions/checkout@v6
- name: Install Rust toolchain
uses: actions-rust-lang/setup-rust-toolchain@v1
@@ -280,7 +303,7 @@ jobs:
rustflags: ''
- name: Cache Cargo registry + build
uses: actions/cache@v4
uses: actions/cache@v5
with:
path: |
~/.cargo/registry
@@ -309,7 +332,7 @@ jobs:
name: PowerShell lint (Windows packaging)
runs-on: windows-latest
steps:
- uses: actions/checkout@v4
- uses: actions/checkout@v6
- name: Run PSScriptAnalyzer
shell: pwsh
@@ -361,12 +384,6 @@ jobs:
- suite: rekey-outbound-only
type: rekey-outbound-only
topology: rekey-outbound-only
# ── Inbound max_peers admission-cap test ───────────────────────
- suite: admission-cap
type: admission-cap
topology: mesh
- suite: acl-allowlist
type: acl-allowlist
# ── Firewall baseline (fips0 nftables default-deny) ────────────
- suite: firewall
type: firewall
@@ -375,9 +392,6 @@ jobs:
type: gateway
topology: gateway
# ── Chaos / stochastic scenarios ───────────────────────────────────
- suite: chaos-smoke-10
type: chaos
scenario: smoke-10
- suite: churn-mixed-10
type: chaos
scenario: churn-mixed
@@ -391,30 +405,9 @@ jobs:
- suite: tcp-mesh
type: chaos
scenario: tcp-mesh
- suite: bottleneck-parent
type: chaos
scenario: bottleneck-parent
- suite: cost-avoidance
type: chaos
scenario: cost-avoidance
- suite: cost-reeval
type: chaos
scenario: cost-reeval
- suite: cost-stability
type: chaos
scenario: cost-stability
- suite: depth-vs-cost
type: chaos
scenario: depth-vs-cost
- suite: mixed-technology
type: chaos
scenario: mixed-technology
- suite: congestion-stress
type: chaos
scenario: congestion-stress
- suite: bloom-storm
type: chaos
scenario: bloom-storm
# ── Sidecar deployment ──────────────────────────────────────────
- suite: sidecar
type: sidecar
@@ -478,11 +471,11 @@ jobs:
type: dns-resolver
steps:
- uses: actions/checkout@v4
- uses: actions/checkout@v6
# Fetch the pre-built Linux binary from job 1
- name: Download Linux binary
uses: actions/download-artifact@v4
uses: actions/download-artifact@v8
with:
name: fips-linux
path: _bin
@@ -626,21 +619,6 @@ jobs:
docker compose -f testing/static/docker-compose.yml \
--profile rekey-outbound-only down --volumes --remove-orphans
# ── ACL allowlist integration test ─────────────────────────────────────
- name: Run ACL allowlist integration test
if: matrix.type == 'acl-allowlist'
run: bash testing/acl-allowlist/test.sh --skip-build --keep-up
- name: Collect logs on failure (acl-allowlist)
if: matrix.type == 'acl-allowlist' && failure()
run: |
docker compose -f testing/acl-allowlist/docker-compose.yml logs --no-color
- name: Stop containers (acl-allowlist)
if: matrix.type == 'acl-allowlist' && always()
run: |
docker compose -f testing/acl-allowlist/docker-compose.yml down --volumes --remove-orphans
# ── Firewall baseline integration test ─────────────────────────────────
- name: Run firewall baseline integration test
if: matrix.type == 'firewall'
@@ -668,7 +646,7 @@ jobs:
- name: Upload sim results on failure (chaos)
if: matrix.type == 'chaos' && failure()
uses: actions/upload-artifact@v4
uses: actions/upload-artifact@v7
with:
name: sim-results-${{ matrix.scenario }}
path: testing/chaos/sim-results/
@@ -771,42 +749,6 @@ jobs:
docker compose -f testing/static/docker-compose.yml \
--profile gateway down --volumes --remove-orphans
# ── Inbound max_peers admission-cap integration test ────────────────
# Lowers node.max_peers on one mesh node and asserts the inbound cap
# holds under sustained retry pressure: denied peers keep retrying but
# are never promoted to an active session. The admission-cap-test.sh
# assertions are tailored per link-layer handshake variant; the leg
# itself is uniform. Static-style harness on the shared mesh profile.
- name: Generate configs (admission-cap)
if: matrix.type == 'admission-cap'
run: bash testing/static/scripts/generate-configs.sh mesh
- name: Inject admission-cap config (admission-cap)
if: matrix.type == 'admission-cap'
run: bash testing/static/scripts/admission-cap-test.sh inject-config
- name: Start containers (admission-cap)
if: matrix.type == 'admission-cap'
run: |
docker compose -f testing/static/docker-compose.yml \
--profile mesh up -d
- name: Run admission-cap test
if: matrix.type == 'admission-cap'
run: bash testing/static/scripts/admission-cap-test.sh
- name: Collect logs on failure (admission-cap)
if: matrix.type == 'admission-cap' && failure()
run: |
docker compose -f testing/static/docker-compose.yml \
--profile mesh logs --no-color | tail -300
- name: Stop containers (admission-cap)
if: matrix.type == 'admission-cap' && always()
run: |
docker compose -f testing/static/docker-compose.yml \
--profile mesh down --volumes --remove-orphans
# ── Real-deb install integration ────────────────────────────────────
# The deb-install harness builds its own .deb from source in a
# cargo-deb builder image; the pre-built Linux binary from the
.github/workflows/package-linux.yml
+5 -5
View File
@@ -19,7 +19,7 @@ jobs:
outputs:
linux_package_version: ${{ steps.linux_version.outputs.linux_package_version }}
steps:
- uses: actions/checkout@v4
- uses: actions/checkout@v6
with:
fetch-depth: 0
@@ -61,7 +61,7 @@ jobs:
deb_arch: arm64
steps:
- uses: actions/checkout@v4
- uses: actions/checkout@v6
with:
fetch-depth: 0
@@ -79,7 +79,7 @@ jobs:
- name: Cache Cargo registry + build
if: ${{ env.ACT != 'true' }}
uses: actions/cache@v4
uses: actions/cache@v5
with:
path: |
~/.cargo/registry
@@ -140,7 +140,7 @@ jobs:
- name: Upload artifact (GitHub only)
if: ${{ env.ACT != 'true' }}
uses: actions/upload-artifact@v4
uses: actions/upload-artifact@v7
with:
name: fips_${{ needs.determine-versioning.outputs.linux_package_version }}_${{ matrix.artifact_arch }}_linux
path: |
@@ -164,7 +164,7 @@ jobs:
steps:
- name: Download Linux artifacts
uses: actions/download-artifact@v4
uses: actions/download-artifact@v8
with:
path: dist
merge-multiple: true
.github/workflows/package-macos.yml
+6 -6
View File
@@ -19,7 +19,7 @@ jobs:
outputs:
macos_package_version: ${{ steps.macos_version.outputs.macos_package_version }}
steps:
- uses: actions/checkout@v4
- uses: actions/checkout@v6
with:
fetch-depth: 0
@@ -61,7 +61,7 @@ jobs:
target: x86_64-apple-darwin
steps:
- uses: actions/checkout@v4
- uses: actions/checkout@v6
with:
fetch-depth: 0
@@ -76,7 +76,7 @@ jobs:
rustflags: ''
- name: Cache Cargo registry + build
uses: actions/cache@v4
uses: actions/cache@v5
with:
path: |
~/.cargo/registry
@@ -209,7 +209,7 @@ jobs:
( cd "$(dirname "$PKG")" && shasum -a 256 "$(basename "$PKG")" | tee "$(basename "$PKG").sha256" )
- name: Upload artifact
uses: actions/upload-artifact@v4
uses: actions/upload-artifact@v7
with:
name: fips_${{ needs.determine-versioning.outputs.macos_package_version }}_${{ matrix.arch }}_macos
path: |
@@ -229,7 +229,7 @@ jobs:
steps:
- name: Download macOS artifacts
uses: actions/download-artifact@v4
uses: actions/download-artifact@v8
with:
path: dist
merge-multiple: true
@@ -283,7 +283,7 @@ jobs:
steps:
- name: Download macOS artifacts
uses: actions/download-artifact@v4
uses: actions/download-artifact@v8
with:
path: dist
merge-multiple: true
.github/workflows/package-openwrt.yml
+431 -30
View File
@@ -24,9 +24,10 @@ jobs:
runs-on: ubuntu-latest
outputs:
package_version: ${{ steps.version.outputs.package_version }}
apk_version: ${{ steps.version.outputs.apk_version }}
release_channel: ${{ steps.channel.outputs.release_channel }}
steps:
- uses: actions/checkout@v4
- uses: actions/checkout@v6
with:
fetch-depth: 0
@@ -35,13 +36,20 @@ jobs:
shell: bash
run: |
: ${GITHUB_OUTPUT:=/tmp/github_output}
# package_version is the human-readable label used in artifact
# filenames; apk_version is the apk-tools-compatible string embedded
# in the .apk metadata. apk_version is built directly from the same
# structured inputs (tag, or commit height) — no reparse of the
# flattened package_version. See packaging/openwrt-apk/apk-version.sh.
if [[ "$GITHUB_REF" == refs/tags/* ]]; then
echo "package_version=${GITHUB_REF_NAME}" >> "$GITHUB_OUTPUT"
echo "apk_version=$(sh packaging/openwrt-apk/apk-version.sh tag "${GITHUB_REF_NAME}")" >> "$GITHUB_OUTPUT"
else
BRANCH=$(echo "$GITHUB_REF_NAME" | sed 's|/|-|g')
HEIGHT=$(git rev-list --count HEAD)
HASH=$(git rev-parse --short HEAD)
echo "package_version=${BRANCH}.${HEIGHT}.${HASH}" >> "$GITHUB_OUTPUT"
echo "apk_version=$(sh packaging/openwrt-apk/apk-version.sh dev "${HEIGHT}")" >> "$GITHUB_OUTPUT"
fi
- name: Determine release channel
@@ -64,8 +72,8 @@ jobs:
echo "release_channel=dev" >> "$GITHUB_OUTPUT"
fi
build:
name: Build .ipk (${{ matrix.openwrt_arch }})
compile-binaries:
name: Cross-compile (${{ matrix.openwrt_arch }})
runs-on: ubuntu-latest
needs: determine-versioning
@@ -96,18 +104,10 @@ jobs:
# x86 routers / VMs
steps:
- uses: actions/checkout@v4
- uses: actions/checkout@v6
with:
fetch-depth: 0
- name: Set SOURCE_DATE_EPOCH from git
run: echo "SOURCE_DATE_EPOCH=$(git log -1 --format=%ct)" >> "$GITHUB_ENV"
- name: Initialize
run: |
PACKAGE_FILENAME=${{ env.PACKAGE_NAME }}_${{ needs.determine-versioning.outputs.package_version }}_${{ matrix.openwrt_arch }}.ipk
echo "PACKAGE_FILENAME=$PACKAGE_FILENAME" >> $GITHUB_ENV
- name: Install Rust toolchain (stable)
if: matrix.rust_channel == 'stable'
uses: actions-rust-lang/setup-rust-toolchain@v1
@@ -124,7 +124,7 @@ jobs:
- name: Cache Cargo registry + build
if: ${{ env.ACT != 'true' }}
uses: actions/cache@v4
uses: actions/cache@v5
with:
path: |
~/.cargo/registry
@@ -153,6 +153,64 @@ jobs:
- name: Install llvm-strip
run: sudo apt-get update && sudo apt-get install -y --no-install-recommends llvm
# Cross-compile + strip once; both the .ipk and .apk packagers consume
# these artifacts via --bin-dir, so the Rust build runs a single time
# per architecture instead of once per package format.
- name: Cross-compile and strip binaries
run: |
set -euo pipefail
cargo zigbuild --release --target ${{ matrix.rust_target }} \
--bin fips --bin fipsctl --bin fipstop --bin fips-gateway
RELEASE_DIR="target/${{ matrix.rust_target }}/release"
mkdir -p out
for b in fips fipsctl fipstop fips-gateway; do
llvm-strip "$RELEASE_DIR/$b" 2>/dev/null || true
cp "$RELEASE_DIR/$b" "out/$b"
done
ls -lh out/
- name: Upload binaries artifact
uses: actions/upload-artifact@v7
with:
name: fips-bins-${{ matrix.openwrt_arch }}
path: out/
retention-days: 1
build:
name: Build .ipk (${{ matrix.openwrt_arch }})
runs-on: ubuntu-latest
needs: [determine-versioning, compile-binaries]
strategy:
fail-fast: false
matrix:
# Must be a subset of compile-binaries' arches (this job consumes those
# binary artifacts). Currently both ship aarch64 + x86_64.
include:
- build_arch: aarch64
openwrt_arch: aarch64_cortex-a53
- build_arch: x86_64
openwrt_arch: x86_64
steps:
- uses: actions/checkout@v6
with:
fetch-depth: 0
- name: Set SOURCE_DATE_EPOCH from git
run: echo "SOURCE_DATE_EPOCH=$(git log -1 --format=%ct)" >> "$GITHUB_ENV"
- name: Initialize
run: |
PACKAGE_FILENAME=${{ env.PACKAGE_NAME }}_${{ needs.determine-versioning.outputs.package_version }}_${{ matrix.openwrt_arch }}.ipk
echo "PACKAGE_FILENAME=$PACKAGE_FILENAME" >> $GITHUB_ENV
- name: Download prebuilt binaries
uses: actions/download-artifact@v8
with:
name: fips-bins-${{ matrix.openwrt_arch }}
path: bins
- name: Install nak
shell: bash
run: |
@@ -201,8 +259,7 @@ jobs:
- name: Build .ipk
env:
PKG_VERSION: ${{ needs.determine-versioning.outputs.package_version }}
LLVM_STRIP: llvm-strip
run: ./packaging/openwrt-ipk/build-ipk.sh --arch ${{ matrix.build_arch }}
run: ./packaging/openwrt-ipk/build-ipk.sh --arch ${{ matrix.build_arch }} --bin-dir "$GITHUB_WORKSPACE/bins"
- name: Install shellcheck (if missing)
shell: bash
@@ -389,13 +446,13 @@ jobs:
- name: SHA-256 hashes
run: |
echo "==> Binaries:"
sha256sum target/${{ matrix.rust_target }}/release/fips target/${{ matrix.rust_target }}/release/fipsctl target/${{ matrix.rust_target }}/release/fipstop
sha256sum bins/fips bins/fipsctl bins/fipstop
echo "==> Package:"
sha256sum dist/${{ env.PACKAGE_FILENAME }}
- name: Upload artifact (GitHub only)
if: ${{ env.ACT != 'true' }}
uses: actions/upload-artifact@v4
uses: actions/upload-artifact@v7
with:
name: ${{ env.PACKAGE_FILENAME }}
path: dist/${{ env.PACKAGE_FILENAME }}
@@ -403,6 +460,7 @@ jobs:
- name: Upload to Blossom
id: blossom_upload
continue-on-error: true
shell: bash
env:
BLOSSOM_SERVER: "https://blossom.primal.net"
@@ -410,17 +468,30 @@ jobs:
run: |
: ${GITHUB_OUTPUT:=/tmp/github_output}
UPLOAD_RESPONSE=$(nak blossom upload \
--server "$BLOSSOM_SERVER" \
--sec "$NSEC" \
"dist/${{ env.PACKAGE_FILENAME }}" < /dev/null)
FILE_HASH=""
for attempt in 1 2 3; do
if UPLOAD_RESPONSE=$(nak blossom upload \
--server "$BLOSSOM_SERVER" \
--sec "$NSEC" \
"dist/${{ env.PACKAGE_FILENAME }}" < /dev/null); then
echo "Upload response (attempt $attempt):"
echo "$UPLOAD_RESPONSE"
FILE_HASH=$(echo "$UPLOAD_RESPONSE" | jq -r '.sha256')
if [ -n "$FILE_HASH" ] && [ "$FILE_HASH" != "null" ]; then
break
fi
echo "Upload response had no sha256 (attempt $attempt)"
else
echo "Blossom upload timed out or failed (attempt $attempt)"
fi
FILE_HASH=""
[ "$attempt" -lt 3 ] && sleep $((attempt * 10))
done
echo "Upload response:"
echo "$UPLOAD_RESPONSE"
FILE_HASH=$(echo "$UPLOAD_RESPONSE" | jq -r '.sha256')
if [ -z "$FILE_HASH" ] || [ "$FILE_HASH" = "null" ]; then
echo "Failed to extract hash from upload response"
echo "Blossom upload did not succeed after 3 attempts; non-fatal."
echo "The package still ships as a GitHub release artifact; only the"
echo "supplementary Blossom/nostr distribution is skipped this run."
exit 1
fi
@@ -431,6 +502,7 @@ jobs:
- name: Publish NIP-94 release event
id: publish
if: steps.blossom_upload.outcome == 'success'
shell: bash
env:
RELAYS: "wss://relay.damus.io wss://nos.lol wss://nostr.mom wss://offchain.pub"
@@ -452,6 +524,7 @@ jobs:
--tag A="${{ matrix.openwrt_arch }}" \
--tag v="$VERSION" \
--tag n="${{ env.PACKAGE_NAME }}" \
--tag format="ipk" \
--tag compression="none" \
> event.json 2> event.err
@@ -509,23 +582,351 @@ jobs:
echo " Release EventId: ${{ steps.publish.outputs.eventId }}"
echo " Blossom URL: ${{ steps.blossom_upload.outputs.url }}"
build-apk:
name: Build .apk (${{ matrix.openwrt_arch }})
runs-on: ubuntu-latest
needs: [determine-versioning, compile-binaries]
strategy:
fail-fast: false
matrix:
# Must be a subset of compile-binaries' arches.
include:
- build_arch: aarch64
openwrt_arch: aarch64_cortex-a53
# MT3000, MT6000, Flint 2, RPi 3/4/5 on OpenWrt 25+
- build_arch: x86_64
openwrt_arch: x86_64
# x86 routers / VMs on OpenWrt 25+
env:
# apk-tools commit OpenWrt pins for the .apk (ADB) format. Keep in sync
# with package/system/apk/Makefile in the targeted OpenWrt release so the
# packages we produce are readable by the apk on the device.
APK_TOOLS_VERSION: "3.0.5"
APK_TOOLS_COMMIT: "b5a31c0d865342ad80be10d68f1bb3d3ad9b0866"
steps:
- uses: actions/checkout@v6
with:
fetch-depth: 0
- name: Set SOURCE_DATE_EPOCH from git
run: echo "SOURCE_DATE_EPOCH=$(git log -1 --format=%ct)" >> "$GITHUB_ENV"
- name: Initialize
run: |
PACKAGE_FILENAME=${{ env.PACKAGE_NAME }}_${{ needs.determine-versioning.outputs.package_version }}_${{ matrix.openwrt_arch }}.apk
echo "PACKAGE_FILENAME=$PACKAGE_FILENAME" >> $GITHUB_ENV
- name: Download prebuilt binaries
uses: actions/download-artifact@v8
with:
name: fips-bins-${{ matrix.openwrt_arch }}
path: bins
- name: Install fakeroot
run: sudo apt-get update && sudo apt-get install -y --no-install-recommends fakeroot
# apk mkpkg lives in apk-tools v3, which is not packaged for Ubuntu, so we
# build the pinned release from source. This is the SDK-free equivalent of
# how the .ipk path uses plain tar — one small C tool, no OpenWrt SDK.
- name: Build apk-tools (${{ env.APK_TOOLS_VERSION }}) from source
run: |
set -euo pipefail
sudo apt-get install -y --no-install-recommends \
git ca-certificates build-essential meson ninja-build pkg-config \
zlib1g-dev libssl-dev libzstd-dev liblzma-dev lua5.4-dev scdoc
git clone --quiet https://gitlab.alpinelinux.org/alpine/apk-tools.git /tmp/apk-tools
cd /tmp/apk-tools
git checkout --quiet "${APK_TOOLS_COMMIT}"
meson setup build
ninja -C build src/apk
APK_BIN=/tmp/apk-tools/build/src/apk
"$APK_BIN" --version 2>/dev/null || "$APK_BIN" version 2>/dev/null || true
echo "APK_BIN=$APK_BIN" >> "$GITHUB_ENV"
- name: Build .apk
env:
PKG_VERSION: ${{ needs.determine-versioning.outputs.package_version }}
APK_VERSION: ${{ needs.determine-versioning.outputs.apk_version }}
run: ./packaging/openwrt-apk/build-apk.sh --arch ${{ matrix.build_arch }} --bin-dir "$GITHUB_WORKSPACE/bins"
- name: Verify apk structural integrity
shell: bash
run: |
set -euo pipefail
APK="dist/${{ env.PACKAGE_FILENAME }}"
if [ ! -s "$APK" ]; then
echo "FAIL: produced apk not found or empty at $APK"
exit 1
fi
echo "==> file type:"; file "$APK"
# apk v3 packages are ADB containers; dump the whole manifest with the
# apk-tools we just built. Print it in full so the exact schema is
# always visible in the log if an assertion needs adjusting.
DUMP=$(mktemp)
"$APK_BIN" adbdump "$APK" > "$DUMP" 2>/dev/null || {
echo "FAIL: 'apk adbdump' could not read $APK"; exit 1; }
echo "==> full adbdump:"; cat "$DUMP"
fail=0
# Package metadata (flat keys under info:).
for needle in "name: fips" "version: ${{ needs.determine-versioning.outputs.apk_version }}" "arch: ${{ matrix.openwrt_arch }}"; do
if grep -qF "$needle" "$DUMP"; then
echo " PASS meta: $needle"
else
echo " FAIL meta: missing '$needle'"; fail=1
fi
done
# installed-size reflects the bundled binaries (4 stripped Rust
# binaries, several MB). A payload regression that drops them shows up
# here regardless of how the path tree is formatted.
SIZE=$(awk '/^[[:space:]]*installed-size:/ {print $2; exit}' "$DUMP")
echo " installed-size: ${SIZE:-unknown}"
if [ -z "${SIZE:-}" ] || [ "$SIZE" -lt 1000000 ]; then
echo " FAIL: installed-size implausibly small (binaries missing?)"; fail=1
else
echo " PASS: installed-size >= 1MB"
fi
# The adbdump paths: block is hierarchical. Each directory is a
# top-level list item "- name: <full relative dir>"; its files are
# "- name: <basename>" nested one indent level deeper under "files:".
# (There are no "path:" keys.) Reconstruct full file paths by keying
# off the indentation of the directory-level list items.
RECON=$(awk '
/^paths:/ {p=1; diri=-1; next}
p && /^[^ #-]/ {p=0} # a new top-level key ends paths:
!p {next}
match($0, /^ *- /) {
ind=RLENGTH; rest=substr($0, RLENGTH+1)
if (diri==-1) diri=ind # first list item = directory indent
if (ind==diri) { # directory entry (or the root acl: entry)
if (rest ~ /^name: /) { dir=rest; sub(/^name: /,"",dir) } else dir=""
next
}
if (rest ~ /^name: /) { # deeper item = a file under files:
f=rest; sub(/^name: /,"",f); print (dir==""?f:dir"/"f)
}
}
' "$DUMP")
echo "==> reconstructed paths:"; printf '%s\n' "$RECON"
for path in \
usr/bin/fips usr/bin/fipsctl usr/bin/fipstop usr/bin/fips-gateway \
etc/init.d/fips etc/init.d/fips-gateway \
etc/fips/fips.yaml etc/fips/firewall.sh etc/dnsmasq.d/fips.conf \
etc/sysctl.d/fips-gateway.conf etc/sysctl.d/fips-bridge.conf \
etc/hotplug.d/net/99-fips etc/uci-defaults/90-fips-setup \
lib/upgrade/keep.d/fips; do
if printf '%s\n' "$RECON" | grep -qxF "$path"; then
echo " PASS path: $path"
else
echo " FAIL path: missing $path"; fail=1
fi
done
if [ "$fail" -ne 0 ]; then
echo "apk structural verification FAILED"
exit 1
fi
echo "apk structural verification PASS"
- name: SHA-256 hashes
run: |
echo "==> Binaries:"
sha256sum bins/fips bins/fipsctl bins/fipstop
echo "==> Package:"
sha256sum dist/${{ env.PACKAGE_FILENAME }}
- name: Upload artifact (GitHub only)
if: ${{ env.ACT != 'true' }}
uses: actions/upload-artifact@v7
with:
name: ${{ env.PACKAGE_FILENAME }}
path: dist/${{ env.PACKAGE_FILENAME }}
retention-days: 30
- name: Install nak
shell: bash
run: |
NAK_VERSION="0.16.2"
ARCH=$(uname -m)
case "$ARCH" in
x86_64|amd64) NAK_ARCH="amd64" ;;
aarch64|arm64) NAK_ARCH="arm64" ;;
*) echo "Unsupported architecture: $ARCH"; exit 1 ;;
esac
curl -fsSL "https://github.com/fiatjaf/nak/releases/download/v${NAK_VERSION}/nak-v${NAK_VERSION}-linux-${NAK_ARCH}" \
-o /usr/local/bin/nak
chmod +x /usr/local/bin/nak
nak --version
- name: Install jq
run: |
if ! command -v jq &>/dev/null; then
sudo apt-get update && sudo apt-get install -y jq
fi
# Priority: HIVE_CI_NSEC from env (loom job) > repo secret > generate ephemeral
- name: Resolve signing key
id: keys
shell: bash
env:
SECRET_NSEC: ${{ secrets.HIVE_CI_NSEC }}
run: |
: ${GITHUB_OUTPUT:=/tmp/github_output}
if [ -n "${HIVE_CI_NSEC:-}" ]; then
echo "Using HIVE_CI_NSEC from loom job environment"
NSEC="$HIVE_CI_NSEC"
elif [ -n "$SECRET_NSEC" ]; then
echo "Using HIVE_CI_NSEC from repository secrets"
NSEC="$SECRET_NSEC"
else
echo "No nsec provided -- generating ephemeral keypair"
NSEC=$(nak key generate)
fi
PUBKEY=$(echo "$NSEC" | nak key public)
echo "::add-mask::$NSEC"
echo "nsec=$NSEC" >> "$GITHUB_OUTPUT"
echo "pubkey=$PUBKEY" >> "$GITHUB_OUTPUT"
echo "Publisher pubkey (hex): $PUBKEY"
- name: Upload to Blossom
id: blossom_upload
continue-on-error: true
shell: bash
env:
BLOSSOM_SERVER: "https://blossom.primal.net"
NSEC: ${{ steps.keys.outputs.nsec }}
run: |
: ${GITHUB_OUTPUT:=/tmp/github_output}
FILE_HASH=""
for attempt in 1 2 3; do
if UPLOAD_RESPONSE=$(nak blossom upload \
--server "$BLOSSOM_SERVER" \
--sec "$NSEC" \
"dist/${{ env.PACKAGE_FILENAME }}" < /dev/null); then
echo "Upload response (attempt $attempt):"
echo "$UPLOAD_RESPONSE"
FILE_HASH=$(echo "$UPLOAD_RESPONSE" | jq -r '.sha256')
if [ -n "$FILE_HASH" ] && [ "$FILE_HASH" != "null" ]; then
break
fi
echo "Upload response had no sha256 (attempt $attempt)"
else
echo "Blossom upload timed out or failed (attempt $attempt)"
fi
FILE_HASH=""
[ "$attempt" -lt 3 ] && sleep $((attempt * 10))
done
if [ -z "$FILE_HASH" ] || [ "$FILE_HASH" = "null" ]; then
echo "Blossom upload did not succeed after 3 attempts; non-fatal."
echo "The package still ships as a GitHub release artifact; only the"
echo "supplementary Blossom/nostr distribution is skipped this run."
exit 1
fi
BLOSSOM_URL="${BLOSSOM_SERVER}/${FILE_HASH}"
echo "url=$BLOSSOM_URL" >> "$GITHUB_OUTPUT"
echo "hash=$FILE_HASH" >> "$GITHUB_OUTPUT"
echo "Uploaded to Blossom: $BLOSSOM_URL"
- name: Publish NIP-94 release event
id: publish
if: steps.blossom_upload.outcome == 'success'
shell: bash
env:
RELAYS: "wss://relay.damus.io wss://nos.lol wss://nostr.mom wss://offchain.pub"
NSEC: ${{ steps.keys.outputs.nsec }}
run: |
: ${GITHUB_OUTPUT:=/tmp/github_output}
set -e
VERSION="${{ needs.determine-versioning.outputs.package_version }}"
CHANNEL="${{ needs.determine-versioning.outputs.release_channel }}"
nak event --sec "$NSEC" -k 1063 \
-c "FIPS Package: ${{ env.PACKAGE_NAME }} for ${{ matrix.openwrt_arch }} (apk)" \
--tag url="${{ steps.blossom_upload.outputs.url }}" \
--tag m="application/octet-stream" \
--tag x="${{ steps.blossom_upload.outputs.hash }}" \
--tag ox="${{ steps.blossom_upload.outputs.hash }}" \
--tag filename="${{ env.PACKAGE_FILENAME }}" \
--tag A="${{ matrix.openwrt_arch }}" \
--tag v="$VERSION" \
--tag n="${{ env.PACKAGE_NAME }}" \
--tag format="apk" \
--tag compression="none" \
> event.json 2> event.err
if [ ! -s event.json ]; then
echo "Failed to create event"
cat event.err 2>/dev/null || true
exit 1
fi
echo "=== Event JSON ==="
cat event.json
echo "=================="
EVENT_ID=$(jq -r '.id' event.json)
if [ -z "$EVENT_ID" ] || [ "$EVENT_ID" = "null" ]; then
echo "Failed to extract event ID"
exit 1
fi
# Publish to relays
cat event.json | nak event $RELAYS 2>&1
echo "eventId=$EVENT_ID" >> "$GITHUB_OUTPUT"
echo "Published NIP-94 event: $EVENT_ID"
- name: Build Summary
run: |
echo "Build Summary for ${{ matrix.openwrt_arch }} (apk):"
echo " Package: ${{ env.PACKAGE_FILENAME }}"
echo " apk version: ${{ needs.determine-versioning.outputs.apk_version }}"
echo " Release EventId: ${{ steps.publish.outputs.eventId }}"
echo " Blossom URL: ${{ steps.blossom_upload.outputs.url }}"
release:
name: Publish GitHub Release (github only)
runs-on: ubuntu-latest
needs: build
needs: [build, build-apk]
if: startsWith(github.ref, 'refs/tags/')
permissions:
contents: write
steps:
- name: Download all .ipk artifacts
uses: actions/download-artifact@v4
- name: Download package artifacts
uses: actions/download-artifact@v8
with:
# Only the .ipk/.apk packages (named fips_<ver>_<arch>.*), not the
# fips-bins-* raw-binary artifacts shared between the build jobs.
pattern: fips_*
path: dist
merge-multiple: true
- name: Generate OpenWrt release checksums
run: |
cd dist
find . -maxdepth 1 -type f \( -name '*.ipk' -o -name '*.apk' \) -printf '%P\n' \
| LC_ALL=C sort \
| xargs sha256sum \
> checksums-openwrt.txt
- name: Create release
uses: softprops/action-gh-release@v2
with:
files: dist/*.ipk
files: |
dist/*.ipk
dist/*.apk
dist/checksums-openwrt.txt
generate_release_notes: true
.github/workflows/package-windows.yml
+5 -5
View File
@@ -19,7 +19,7 @@ jobs:
outputs:
package_version: ${{ steps.version.outputs.package_version }}
steps:
- uses: actions/checkout@v4
- uses: actions/checkout@v6
with:
fetch-depth: 0
@@ -52,7 +52,7 @@ jobs:
needs: determine-versioning
steps:
- uses: actions/checkout@v4
- uses: actions/checkout@v6
with:
fetch-depth: 0
@@ -69,7 +69,7 @@ jobs:
rustflags: ''
- name: Cache Cargo registry + build
uses: actions/cache@v4
uses: actions/cache@v5
with:
path: |
~/.cargo/registry
@@ -146,7 +146,7 @@ jobs:
}
- name: Upload artifact
uses: actions/upload-artifact@v4
uses: actions/upload-artifact@v7
with:
name: fips_${{ needs.determine-versioning.outputs.package_version }}_x86_64_windows
path: deploy/fips-*-windows-*.zip
@@ -170,7 +170,7 @@ jobs:
steps:
- name: Download Windows artifacts
uses: actions/download-artifact@v4
uses: actions/download-artifact@v8
with:
path: dist
merge-multiple: true
.gitignore
+5
View File
@@ -33,6 +33,11 @@ __pycache__/
*.egg-info/
*.egg
# Per-run build contexts created by testing/ci-local.sh. Its teardown normally
# removes them, but the CI worker's SIGKILL runs no trap, so one can survive a
# preempted run; ci-cleanup.sh sweeps the survivors.
/testing/docker-*/
# Runtime artifacts from running fips in-tree during local testing.
# Root-anchored so legitimately-tracked fips.yaml under packaging/ and
# examples/ stays included.
CHANGELOG.md
+492 -295
View File
@@ -9,6 +9,103 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
### Added
### Changed
- `SessionDatagram::decrement_ttl` and `SessionDatagram::can_forward` now match
the forwarder's IP hop-limit semantics: `decrement_ttl` decrements first and
reports false when the result is zero, and `can_forward` is true only at a
TTL of 2 or more.
### Fixed
- Nostr NAT traversal no longer breaks after the host suspends. The traversal
clock cached a Unix timestamp once at startup and advanced it with a
monotonic `Instant`, which does not tick while a machine is asleep, so after
a suspend the daemon's idea of the time trailed real time by the suspend
duration for the rest of the process lifetime. Every NIP-40 expiration it
computed was therefore published already in the past: relays dropped the
offers as expired, the initiator logged a signal timeout waiting for an
answer, and traversal stayed broken until the daemon was restarted. The
clock now reads the wall clock on every call. This is not macOS-specific,
though a laptop that sleeps is where it is easiest to hit; any host that
suspends or hibernates was affected. Reported in
[#128](https://github.com/jmcorgan/fips/issues/128).
- `SessionDatagram` hop-limit handling now follows IP semantics. Delivery to
the addressed node is no longer TTL-gated, and a forwarder decrements before
deciding rather than after, so a datagram that would leave with a TTL of zero
is dropped instead of transmitted. Previously the TTL check ran ahead of the
local-delivery test, so a datagram addressed to this node that arrived with
TTL 0 was dropped, and a forwarder receiving a transit datagram at TTL 1
transmitted it at TTL 0 for the next hop to discard, wasting one transmission
per expiring datagram. The reachable radius is unchanged, because the two
behaviors compensated exactly: a path of `h` links still delivers for any
source TTL of `h` or more. During a rolling upgrade, an unupgraded forwarder
feeding an upgraded destination delivers one hop further than either version
does on its own; no version mix delivers less far. The `TtlExhausted` reject
counter now charges at the node that makes the decision rather than at the
hop after it.
## [0.4.1] - 2026-07-19
### Changed
- `node.bloom.max_inbound_fpr` default raised from `0.10` to `0.20`. The
cap rejects inbound `FilterAnnounce` whose FPR (`fill^k`) exceeds it. On
the fixed 1 KB / k=5 filter, `0.10` corresponds to fill 0.631 (~1,630
reachable entries), and the busiest nodes' aggregates had again begun to
reach it as the mesh grew. `0.20` (fill 0.7248, ~2,114 entries) restores
headroom without materially weakening the antipoison gate: a saturated or
poisoned filter is ~100% FPR and still rejected. This is the second raise
of this cap in two releases; the fixed 1 KB filter is the underlying
constraint, and the structural remedy is the v2 filter work rather than a
further raise. A node running this default accepts announcements that a
v0.4.0 node drops, so during a rolling upgrade the two versions can
disagree about mesh size.
- Bloom filter probing computes its SHA-256 digest once per operation
rather than once per hash function. All k indices were already derived
from a single digest, but the digest was recomputed inside the
per-function loop, so every insert and membership test hashed the same
bytes `hash_count` times (5x at the default). Output is bit-for-bit
identical; this is the hottest path in packet forwarding and mesh-size
estimation.
- Identity operations reuse one shared `secp256k1` context instead of
constructing a fresh one at every sign, verify, and key-derive site.
Each construction allocated a context and ran randomization and blinding
table setup. Behavior is unchanged: the same API calls are made, only the
context lifetime differs, and the shared context still performs the
standard construction-time blinding.
### Fixed
- Spanning tree: the coordinate cache is now invalidated when the parent
link is lost through peer removal. That path reparents or self-roots the
node but omitted the invalidation every other position-change path
performs, so cached entries for downstream destinations kept the node's
now-stale coordinate prefix. Because routing access refreshes an entry's
TTL, an actively routed stale entry never self-expired and was corrected
only by a fresh insert.
- Discovery: applying a `LookupResponse` now keeps the tighter of the
cached and received `path_mtu` rather than overwriting unconditionally.
A looser estimate arriving in a later response could clobber a tighter
value already learned from a reactive `MtuExceeded` or
`PathMtuNotification`, loosening a clamp that had been correctly
tightened.
### Removed
- The `parent_switched` spanning-tree metric counter. It was incremented on
the line immediately before `parent_switches` at every site and never
independently, so the two were always identical. `parent_switches`
remains as the sole counter. Consumers reading `parent_switched` from the
control socket or `fipstop` should use `parent_switches`.
## [0.4.0] - 2026-06-27
### Added
#### Transports (Nym, mDNS LAN discovery)
- Nym mixnet transport (`transports.nym`) for outbound peer links
tunneled through a local `nym-socks5-client` SOCKS5 proxy into the
Nym mixnet, as a privacy transport alongside Tor. Outbound-only and
@@ -16,6 +113,26 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
crate dependencies. A single-container example
(`examples/sidecar-nostr-mixnet-relay/`) demonstrates FIPS peering
across the mixnet end to end.
- Opt-in mDNS / DNS-SD LAN discovery for sub-second pairing of peers on
the same local link, without a relay or NAT-traversal roundtrip.
Disabled by default; operators enable it with
`node.discovery.lan.enabled: true`. Configurable service type and an
optional `node.discovery.lan.scope` that isolates discovery to peers
sharing the same private-network scope. The advertised UDP port is
chosen from a non-bootstrap operational UDP transport using a stable
selector, so it is deterministic across restarts.
#### Admission / peer-list management
- `Node::update_peers` for runtime peer-list refresh, returning an
`UpdatePeersOutcome` summarizing added, removed, and retained peers.
Re-derives active peer connections from a new peer configuration
without dropping links to peers that remain in the set.
`PeerAddress` gains a `seen_at_ms` recency field (with
`with_seen_at_ms`) used to prefer more recently observed addresses.
#### Data-plane / metrics / observability
- Typed `RejectReason` classification for receive-path silent-rejection
sites across the node. Each rejection-and-return path now passes a
typed reason to `NodeStats::record_reject`, which routes it to a
@@ -28,47 +145,73 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
site that was previously silent. Several handshake, MMP, tree, and
discovery rejection paths that had no counter at all are now counted,
including the `send_lookup_response` no-route drop
(`DiscoveryStats::resp_no_route`). Existing
direct counters at the bloom / discovery / forwarding sites are
retained alongside the new dispatch while the rollout is in progress;
a later change collapses the duplicate increment.
(`DiscoveryStats::resp_no_route`).
- Internal atomic metric registry (`Arc<MetricsRegistry>`) that shadows
the plain-`u64` `NodeStats` counters, written alongside them and
validated by a whole-struct debug-build parity check. Covers the
forwarding receive counters, the full discovery counter family, and the
tree, bloom, congestion, and error-signal counter families so far, with
tree, bloom, congestion, and error-signal counter families, with
the hottest counters cache-line padded. Behavior-neutral:
`NodeStats` remains the serving path. Groundwork for sampling metrics
without contending the receive loop.
- `Node::update_peers` for runtime peer-list refresh, returning an
`UpdatePeersOutcome` summarizing added, removed, and retained peers.
Re-derives active peer connections from a new peer configuration
without dropping links to peers that remain in the set.
`PeerAddress` gains a `seen_at_ms` recency field (with
`with_seen_at_ms`) used to prefer more recently observed addresses.
- Opt-in mDNS / DNS-SD LAN discovery for sub-second pairing of peers on
the same local link, without a relay or NAT-traversal roundtrip.
Disabled by default; operators enable it with
`node.discovery.lan.enabled: true`. Configurable service type and an
optional `node.discovery.lan.scope` that isolates discovery to peers
sharing the same private-network scope. The advertised UDP port is
chosen from a non-bootstrap operational UDP transport using a stable
selector, so it is deterministic across restarts.
- `fipsctl stats metrics`, backed by a new counter-only `show_metrics`
control query that dumps the atomic metric registry as flat counter
name/value pairs. Serves a Prometheus-style scraper that samples node
counters without contending the receive loop.
- `pool_inbound` and `pool_outbound` counters on the TCP and Tor
transport stats (`TcpStats`, `TorStats`). Per-direction accounting
is updated at every pool-insert and receive-loop-exit site, plus on
transport stop and on send-failure-driven removal. Surfaces through
`TcpStatsSnapshot` and `TorStatsSnapshot` for `show_transports`.
- `fipsctl stats metrics`, backed by a new counter-only `show_metrics`
control query that dumps the atomic metric registry as flat counter
name/value pairs. Serves a Prometheus-style scraper that samples node
counters without contending the receive loop.
#### Spanning-tree / mesh-size / routing
- Six route-class transit counters that partition transit-forwarded
packets by their tree relationship to the chosen next hop: tree-up
(peer is our ancestor), tree-down (peer is our descendant and the
destination is within its subtree), tree-down-cross (peer is our
descendant but the destination is outside its subtree), cross-link
descend (lateral peer, destination within its subtree), cross-link
ascend (lateral peer, destination outside its subtree), and
direct-peer. The six classes sum to `forwarded_packets` (asserted by a
unit test) and are computed from tree coordinates at the transit
chokepoint, so error-signal routing callers are excluded. They surface
through the forwarding stats snapshot via `show_routing` and
`show_status`.
- Discovery now counts `LookupRequest`s dropped when the dedup cache is
full. A saturated `recent_requests` cache
(`MAX_RECENT_DISCOVERY_REQUESTS`) previously dropped requests
silently; a new `DiscoveryStats::req_dedup_cache_full` counter (typed
reject reason `DiscoveryReject::ReqDedupCacheFull`) makes the drop
visible through `show_routing`.
#### Packaging & deployment
- OpenWrt `.apk` packaging (`packaging/openwrt-apk/`, `make apk`) for
OpenWrt 25+, where apk-tools is the mandatory package manager (the
existing `.ipk` continues to cover OpenWrt 24.x and earlier). Built
SDK-free: it reuses the `.ipk` cross-compile (`cargo-zigbuild`) and the
shared installed-filesystem payload, and assembles the package with
`apk mkpkg` from apk-tools 3.0.5 built from source — no OpenWrt SDK
image. A `build-apk` CI job (aarch64, x86_64) builds and structurally
verifies the package; releases now publish `.apk` artifacts and
checksums alongside `.ipk`. Packages are unsigned, installed with
`apk add --allow-untrusted`, matching the `.ipk` posture.
- Nix flake (`flake.nix` at the project root) for reproducible
from-source builds on Nix/NixOS. Builds all four binaries (`fips`,
`fipsctl`, `fips-gateway`, `fipstop`), pins the exact toolchain from
`rust-toolchain.toml` via fenix, and wires the build-time native
dependencies (`libclang` for `bindgen`, plus `dbus` and `pkg-config`),
so it needs no host setup beyond Nix with flakes enabled. Flake inputs
are lock-pinned (`flake.lock` committed) for reproducibility, and the
flake exposes `nix build`, `nix run`, a `nix develop` dev shell with the
pinned toolchain, and `nix flake check`. The flake produces binaries
(and a NixOS `packages.<system>.fips` output); the systemd/service
integration that the `.deb`/tarball installers provide is handled
through the NixOS configuration instead.
#### Docs & contributor tooling
- [`PR-REVIEW.md`](PR-REVIEW.md) — the 13-criteria PR review checklist
the maintainer runs against every incoming PR, published at the
repo root so contributors can run the same pass on their own change
@@ -90,17 +233,14 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
### Changed
#### FMP/FSP rekey reliability
- `complete_rekey_msg2` now returns the remote peer's startup epoch
alongside the new Noise session, so the rekey path can detect a peer
restart and clear stale session state.
- Active-peer path selection now sorts address candidates by recency
(`seen_at_ms`), preferring the most recently observed address when
racing concurrent path probes.
- Per-tick work budgets bound the connection churn done in a single
node tick: `MAX_DISCOVERY_CONNECTS_PER_TICK`,
`MAX_RETRY_CONNECTIONS_PER_TICK`, and
`MAX_PARALLEL_PATH_CANDIDATES_PER_PEER`. Work beyond a tick's budget
is deferred to the next tick rather than discarded.
#### NAT traversal / Nostr discovery
- Nostr discovery startup is now non-blocking. `Node::start` no
longer waits for relay connect, subscribe, or initial advert
publish before returning. A slow or unreachable relay no longer
@@ -109,6 +249,20 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
Subscribe retries with exponential backoff (2 s base, 60 s cap),
publish attempts time out at 10 s, and the new tasks are aborted
cleanly on `Node::stop`.
#### Spanning-tree / mesh-size / routing
- Active-peer path selection now sorts address candidates by recency
(`seen_at_ms`), preferring the most recently observed address when
racing concurrent path probes.
- Per-tick work budgets bound the connection churn done in a single
node tick: `MAX_DISCOVERY_CONNECTS_PER_TICK`,
`MAX_RETRY_CONNECTIONS_PER_TICK`, and
`MAX_PARALLEL_PATH_CANDIDATES_PER_PEER`. Work beyond a tick's budget
is deferred to the next tick rather than discarded.
#### Admission / peer caps
- `node.bloom.max_inbound_fpr` default raised from `0.05` to `0.10`. The
cap rejects inbound `FilterAnnounce` whose FPR (`fill^k`) exceeds it. On
the fixed 1 KB / k=5 filter, `0.05` corresponds to fill 0.549 (~1,300
@@ -124,6 +278,9 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
per-transport `max_inbound_connections`, then node-wide
`max_connections`, then the built-in default of 256. Established peers
remain bounded node-wide by `add_connection`.
#### Data-plane / worker-pool / metrics / observability
- The control-socket read surface is now served off the `rx_loop`.
Every pure-read `show_*` query — `show_status`, the `show_stats_*`
family, `show_listening_sockets`, the new `show_metrics`,
@@ -148,11 +305,6 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
snapshot fields above. Built on a ratatui `TestBackend`
render-snapshot harness that asserts the text grid and per-cell
style of every `ui::draw_*` against canned `show_*` JSON.
- Static host aliases in `/etc/fips/hosts` now hot-reload on mtime
change instead of only at daemon startup, so `fipsctl`/`fipstop`
display names reflect edits without a restart. The peer ACL and host
map both reload once per node tick through a new lock-free
`Reloadable` snapshot.
- Steady-state log noise reduced on saturated public-mesh nodes.
Routine per-peer connection-lifecycle and capacity-cap events are
demoted from info/warn to debug — FMP K-bit cutover promotion,
@@ -163,43 +315,6 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
are no longer drowned out. An exhausted FMP-msg1 / FSP-msg3 rekey
retransmission-budget abort (an expected, self-limiting outcome on
lossy or high-latency links) is likewise demoted from warn to debug.
- Sidecar example (`examples/sidecar-nostr-relay`): `udp.mtu` is now
overridable via the `FIPS_UDP_MTU` environment variable, defaulting to
1472 (preserving prior behavior). Plumbed through `docker-compose.yml`
and documented in the README env-var table. Annotated the static-CI
node template `mtu: 1472` literal with the same Docker-bridge
rationale and a pointer at the daemon's 1280 default.
- Overhauled `CONTRIBUTING.md`: replaced generic Rust-template framing
with a FIPS-specific entry point covering the four-layer
architecture, branch model and PR-target selection, structured bug
reporting, scope discipline and local-CI requirements, an AI coding
assistant policy, and project communication channels. Added
`docs/branching.md` as the long-form companion covering the release
workflow, version conventions, and merge-direction rationale.
- CI and release-publish workflows hardened:
- `ci.yml` declares a top-level `concurrency` block keyed on
`(workflow, ref)` with `cancel-in-progress: true`. Force-pushes
and rapid successive pushes to the same ref now retire any
in-flight run rather than letting superseded and current-tip runs
both burn runner minutes.
- `aur-publish.yml` rewritten to fetch the upstream source tarball
and compute its `b2sum` in CI, then patch `pkgver` and the
`b2sums` SKIP placeholder in `PKGBUILD` in-place. Previously
`updpkgsums: true` downloaded the tarball into the AUR working
tree, where it was rejected by AUR's 488 KiB max-blob hook —
silently no-op'ing the v0.3.0 stable AUR push. `fips.sysusers` /
`fips.tmpfiles` asset b2sums are recomputed in the same step to
stay in sync with the local files. `workflow_dispatch` gains a
tag input so historical release tags can be re-published
manually, and `continue-on-error: true` is dropped so future
regressions surface in CI.
- New `aur-publish-git.yml` workflow for the `fips-git` VCS
PKGBUILD, triggered on master pushes touching `PKGBUILD-git` or
companion files plus `workflow_dispatch`. `pkgver` is computed at
build time by the PKGBUILD's `pkgver()` function, so this workflow
is not tied to release tags.
- Tag-triggered `package-*` release-build workflows remain
untouched.
- macOS UDP receive path now batches up to 32 datagrams per kernel
wakeup via `recvmsg_x(2)`, matching the Linux `recvmmsg(2)`
amortization shape introduced in v0.3.0. Previously macOS fell
@@ -272,6 +387,22 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
and the connected-UDP drain no longer busy-spins on a poll error
(#106).
#### Transports & config
- Static host aliases in `/etc/fips/hosts` now hot-reload on mtime
change instead of only at daemon startup, so `fipsctl`/`fipstop`
display names reflect edits without a restart. The peer ACL and host
map both reload once per node tick through a new lock-free
`Reloadable` snapshot.
- Sidecar example (`examples/sidecar-nostr-relay`): `udp.mtu` is now
overridable via the `FIPS_UDP_MTU` environment variable, defaulting to
1472 (preserving prior behavior). Plumbed through `docker-compose.yml`
and documented in the README env-var table. Annotated the static-CI
node template `mtu: 1472` literal with the same Docker-bridge
rationale and a pointer at the daemon's 1280 default.
#### Packaging & deployment
- The Debian package no longer ships `/etc/fips/fips.yaml` as a dpkg
conf-file. The default configuration is installed as an example at
`/usr/share/fips/fips.yaml.example`, and `postinst` seeds
@@ -283,6 +414,36 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
example is placed under `/usr/share/fips`, deliberately outside
`/usr/share/doc`, which minimal and container installs path-exclude
(so the install-time seed source is never dropped).
- openwrt: the `.apk` package now defaults `ethernet.wan` to the
OpenWrt 25 DSA port name `wan`; the `.ipk` package keeps `eth0` for
OpenWrt 24 and earlier.
#### CI & test-harness reliability
- CI and release-publish workflows hardened:
- `ci.yml` declares a top-level `concurrency` block keyed on
`(workflow, ref)` with `cancel-in-progress: true`. Force-pushes
and rapid successive pushes to the same ref now retire any
in-flight run rather than letting superseded and current-tip runs
both burn runner minutes.
- `aur-publish.yml` rewritten to fetch the upstream source tarball
and compute its `b2sum` in CI, then patch `pkgver` and the
`b2sums` SKIP placeholder in `PKGBUILD` in-place. Previously
`updpkgsums: true` downloaded the tarball into the AUR working
tree, where it was rejected by AUR's 488 KiB max-blob hook —
silently no-op'ing the v0.3.0 stable AUR push. `fips.sysusers` /
`fips.tmpfiles` asset b2sums are recomputed in the same step to
stay in sync with the local files. `workflow_dispatch` gains a
tag input so historical release tags can be re-published
manually, and `continue-on-error: true` is dropped so future
regressions surface in CI.
- New `aur-publish-git.yml` workflow for the `fips-git` VCS
PKGBUILD, triggered on master pushes touching `PKGBUILD-git` or
companion files plus `workflow_dispatch`. `pkgver` is computed at
build time by the PKGBUILD's `pkgver()` function, so this workflow
is not tied to release tags.
- Tag-triggered `package-*` release-build workflows remain
untouched.
- Local and GitHub CI integration coverage brought into parity, and
the Rust toolchain selection given a single source of truth:
- The `admission-cap` integration suite, previously run only by
@@ -305,39 +466,70 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
`-D warnings` is newly imposed. The OpenWrt nightly Tier-3 leg
keeps `@nightly`.
#### Docs & contributor tooling
- Overhauled `CONTRIBUTING.md`: replaced generic Rust-template framing
with a FIPS-specific entry point covering the four-layer
architecture, branch model and PR-target selection, structured bug
reporting, scope discipline and local-CI requirements, an AI coding
assistant policy, and project communication channels. Added
`docs/branching.md` as the long-form companion covering the release
workflow, version conventions, and merge-direction rationale.
### Fixed
- The Tor transport now increments its `connect_refused` statistic (the
"Refused" line in fipstop) when a SOCKS5 connection is actively
refused, instead of recording every connect failure as a generic
SOCKS5 error. The counter previously stayed at zero.
- MMP sender metrics now ignore duplicate or regressed receiver reports
before updating RTT, loss, goodput, or ETX. Receiver reports also
suppress timestamp echo when dwell time overflows, so stale reports
cannot inflate SRTT.
- Six discovery counters (`req_decode_error`, `req_duplicate`,
`req_ttl_exhausted`, `resp_decode_error`, `resp_identity_miss`,
`resp_proof_failed`) no longer double-count. Each was incremented both
by a direct bump and again through the typed reject dispatch; the
redundant direct increment is removed, so each counts once per event.
- Six bloom counters (`decode_error`, `invalid`, `non_v1`,
`unknown_peer`, `stale`, `fill_exceeded`) no longer double-count. Each
was incremented both by a direct bump and again through the typed
reject dispatch; the redundant direct increment is removed, so each
counts once per event.
- Five forwarding reject packet counters (`decode_error_packets`,
`ttl_exhausted_packets`, `drop_no_route_packets`,
`drop_mtu_exceeded_packets`, `drop_send_error_packets`) no longer
double-count. Each was incremented both by the byte-aware outcome
recorder and again through the typed reject dispatch; the two calls are
collapsed into a single byte-aware reject entry point, so packets and
bytes each count once per event.
#### FMP/FSP rekey reliability
- FMP link-layer rekey is now reliable under packet loss, bringing it up
to the FSP session layer's rekey discipline. The rekey msg1
retransmission driver was previously uncapped and never abandoned, so a
rekey that never completed resent msg1 forever; it now uses a bounded
retransmission budget (`handshake_max_resends` with exponential
backoff) and abandons the rekey cycle cleanly once the budget is
exhausted, mirroring the FSP rekey msg3 driver. With the cap in place
the link-dead heartbeat is rekey-aware: `check_link_heartbeats` no
longer reaps a link that is still actively carrying rekey-handshake
traffic, while a genuinely dead link is still reaped once the budget
abandons. At the K-bit cutover the receiver now authenticates an
inbound frame against the pending session before promoting it, instead
of promoting on the bare header K-bit; under jitter a node could
otherwise promote a stale pending session, leaving the two endpoints on
different keys and silently dropping traffic until the link died — the
same failure class already closed on FSP, now closed on FMP.
- FSP session rekey is now hitless under packet loss and reordering.
Previously, a rekey could leave the two endpoints holding different
key sets for a brief window — if a handshake message was lost in
transit one side rotated keys while the other did not, and traffic
sealed in one key epoch reached a peer still on the other epoch and
failed to decrypt, producing bursts of AEAD decryption failures and
dropped connectivity until a later rekey reconverged the pair. The
receive path now trial-decrypts each frame against every live key
epoch (current, pending, and the draining previous session) for the
duration of the rekey transition, so no rotation ordering and no
packet reordering can cause a decryption failure. The previous-epoch
slot is retained as long as the peer keeps using it, with its drain
deadline anchored on the last frame the peer authenticates against
it rather than a fixed wall-clock timer, so a peer that did not
receive the new keys is not stranded by a silent permanent decrypt
failure. The lost-handshake case is closed by retransmitting the
third rekey handshake message until the peer is confirmed on the
new keys, with a bounded retry budget after which the rekey cycle
is cleanly abandoned and retried. There are no FSP decryption
failures across a rekey under lossy, jittery links.
- ±15s symmetric jitter is applied per session to the FMP and FSP rekey
timer trigger, eliminating the steady-state dual-initiation race in
symmetric-start meshes (previously the smaller-NodeAddr tie-breaker
resolved correctness only after every cycle's collision).
`node.rekey.after_secs` becomes the nominal interval rather than a
floor; the mean is preserved.
- A stale FSP (session-layer) session is now cleared when a peer
restart is detected during FMP rekey or cross-connection promotion.
Previously the old session could linger after the peer came back
with a new startup epoch, leaving the session-layer map out of sync
with the freshly promoted peer.
#### NAT traversal / Nostr discovery
- Two nodes that each `auto_connect` to the other no longer stall their
Nostr-mediated NAT-traversal handshake. Each side ran both an
initiator and a responder traversal session, binding a separate UDP
@@ -350,43 +542,55 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
its own NodeAddr is smaller — so one matching socket pair survives on
both ends and the peer's redundant initiator times out harmlessly.
One-sided (asymmetric) `auto_connect` has no co-active initiator and is
never suppressed, so connectivity is preserved. (Distinct from the
cross-init adoption tie-breaker below, which dedups two simultaneous
punches but does not stop each node from running redundant
initiator + responder sessions.)
- FMP link-layer rekey is now reliable under packet loss, bringing it up
to the FSP session layer's rekey discipline:
- The rekey msg1 retransmission driver was uncapped and never
abandoned, so a rekey that never completed resent msg1 forever. It
now uses a bounded retransmission budget (`handshake_max_resends`
with exponential backoff) and abandons the rekey cycle cleanly once
the budget is exhausted, mirroring the FSP rekey msg3 driver. With
the cap in place the link-dead heartbeat is rekey-aware:
`check_link_heartbeats` no longer reaps a link that is still
actively carrying rekey-handshake traffic, while a genuinely dead
link is still reaped once the budget abandons.
- At the K-bit cutover the receiver now authenticates an inbound frame
against the pending session before promoting it, instead of
promoting on the bare header K-bit. Under jitter a node could
otherwise promote a stale pending session, leaving the two endpoints
on different keys and silently dropping traffic until the link died
— the same failure class already closed on FSP, now closed on FMP.
- Node-level multi-node tests no longer flake under parallel CPU load.
They previously delivered handshake packets over real localhost UDP,
whose kernel receive buffer could overflow and drop a packet when many
tests ran concurrently, panicking the large-network convergence tests.
A `cfg(test)`-only loopback `TransportHandle` variant now delivers
packets directly between nodes over an unbounded in-process channel, so
there is no socket buffer to overflow, and the previously-quarantined
large-network tests run in the default suite again. The shipping daemon
build is unaffected (the variant is test-gated).
- Integration suites that wait for the mesh to converge no longer
false-fail under concurrent CI load. The rekey, static-mesh, and
sidecar suites replace a fixed wall-clock baseline timeout (and a blind
sleep) with a progress-aware wait that polls the suite's own pairwise
pings, returns as soon as every pair is reachable, extends its deadline
while the reachable-pair count is still climbing, and gives up only
when progress stalls.
never suppressed, so connectivity is preserved.
- NAT-traversal cross-init adoption is now deterministic under
simultaneous dual-initiation. Previously, when two peers'
Nostr-mediated UDP punches completed within the same scheduling
window, each side's bootstrap-completion event arrived with an
in-flight handshake already recorded against the other peer (each
side had received an inbound msg1 from the other's pre-punch
outbound attempt). The deduplication skip then fired on both
sides, neither installed the fresh traversal socket as canonical,
and the 45-second peer-adoption budget expired with both nodes
stuck waiting for an adoption that never happened. The handler now
applies the same deterministic NodeAddr tie-breaker the codebase
already uses for rekey dual-initiation and cross-connection
resolution: the smaller NodeAddr wins as adopter, tears down its
in-flight handshake state, and proceeds with adoption; the larger
NodeAddr keeps the skip semantics, and its in-flight outbound is
reconciled by the cross-connection logic when the winner's fresh
msg1 arrives over the adopted socket. The dual cross-init stall is
eliminated; cross-init NAT-traversal completes in well under a
second even under host CPU contention.
- Nostr-discovered NAT-traversal events (`BootstrapEvent::Established`
and `BootstrapEvent::Failed`) for peers that are already connected
or actively handshaking are now short-circuited at the
`poll_nostr_discovery` dispatch sites before any cooldown
bookkeeping or fallback retry scheduling runs. Stale `Failed` events
previously poisoned the per-peer failure-state cooldown of healthy
peers and could trigger redundant retraversal attempts via
`schedule_retry` / `try_peer_addresses`; stale `Established`
handoffs could attempt to adopt a second socket against a live
connection. A defense-in-depth guard was added to
`adopt_established_traversal` so the same invariant holds if a
future caller bypasses the outer dispatch check. As a side benefit,
narrows a cooldown-poisoning vector previously available to an
attacker injecting stale failure events for an active peer.
- Nostr discovery now filters unroutable direct UDP/TCP advert
endpoints. Publisher and validator retain only endpoints that parse as
concrete socket addresses with routable IPs and nonzero ports;
`udp:nat` rendezvous endpoints and Tor endpoints pass through
unchanged. Adverts that collapse to zero usable endpoints after
filtering are rejected with a clear "missing publicly routable
endpoints" error. Before this change, misconfigured nodes could
publish RFC1918, loopback, link-local, CGNAT 100.64/10, IPv6 ULA,
or IPv6 link-local endpoints into Nostr discovery, and consumers
would cache and dial them; in mixed LAN/VPN/NAT environments, that
could prefer a misleading one-way private path over the intended
`udp:nat` bootstrap.
#### Admission / peer caps
- TCP and Tor `max_inbound_connections` admission cap is now compared
against the per-direction inbound count (`pool_inbound`) rather than
the combined pool size. Outbound connect-on-send connections share
@@ -415,10 +619,13 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
reconnect / cross-connection bypass for known peers still fires) and
before index allocation + Msg2 wire send. The late cap check inside
`promote_connection` is intentionally retained as
defense-in-depth. Wire savings observed in a 45 s ops tcpdump at
defense-in-depth. Wire savings observed in a 45 s tcpdump at
saturation: ~3.6 cap-denials/s × Msg2 (~104 B + AEAD compute) each.
Bigger win is cleaner peer-side semantics — no fake-completed
handshake whose subsequent data frames fail decryption on this side.
#### Spanning-tree / mesh-size / routing
- The mesh-size estimator (`compute_mesh_size`) no longer over-counts
under filter overlap. It previously summed the per-filter cardinality
of the parent and each child filter, which assumes the filters are
@@ -429,12 +636,11 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
nearly-but-not-exactly doubling during rebalancing). The estimator now
computes the cardinality of the OR-union over self plus every
connected peer's inbound filter, dropping the parent/child tree gating
entirely.
OR is idempotent, so any overlap is deduplicated — the result equals
the old sum in the disjoint case, stays correct under overlap, damps
the parent-switch flap, and removes the estimate's dependence on
tree-declaration cache freshness. The per-peer 500 ms rate-limiter and
overall recompute cadence are unchanged.
entirely. OR is idempotent, so any overlap is deduplicated — the
result equals the old sum in the disjoint case, stays correct under
overlap, damps the parent-switch flap, and removes the estimate's
dependence on tree-declaration cache freshness. The per-peer 500 ms
rate-limiter and overall recompute cadence are unchanged.
- Spanning-tree state distribution is now eventually-consistent.
Previously every `send_tree_announce_to_all` call site fired only
on a local state-change event (parent switch, self-root promotion,
@@ -472,7 +678,20 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
that peer, provoking the better-rooted peer to re-push its real
position immediately. The echo fires only in that one direction and is
bounded by the existing per-peer rate limiter.
- Coord cache invalidation made surgical at parent-position-change
and root-change sites. Replaces the previous unconditional
`CoordCache::clear()` calls with two targeted methods:
`invalidate_via_node(node_addr)` (drops entries whose cached
ancestry contains the changed node, used at parent-switch /
become-root / loop-detection sites) and `invalidate_other_roots`
(drops entries from a different tree, used at root-change sites).
The previous global flush left `find_next_hop` returning `None`
for every non-direct-peer destination after every parent switch
until the cache passively re-warmed; surgical invalidation
preserves entries that remain correct across the topology change.
Peer-removal retains the original "no invalidation" behavior
(`find_next_hop` already recomputes against the current peer set
every call, and Discovery handles "no route" on demand).
- `rx_loop` tick-arm stall under convergence-phase mesh pressure
is eliminated. Previously, the tick body's per-peer `check_*`
loops (heartbeats, bloom announces, MMP reports, tree announces)
@@ -504,45 +723,133 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
synchronous connect (e.g., explicit operator-driven
`fipsctl connect`); the tick path just no longer trips it.
- NAT-traversal cross-init adoption is now deterministic under
simultaneous dual-initiation. Previously, when two peers'
Nostr-mediated UDP punches completed within the same scheduling
window, each side's bootstrap-completion event arrived with an
in-flight handshake already recorded against the other peer (each
side had received an inbound msg1 from the other's pre-punch
outbound attempt). The deduplication skip then fired on both
sides, neither installed the fresh traversal socket as canonical,
and the 45-second peer-adoption budget expired with both nodes
stuck waiting for an adoption that never happened. The handler now
applies the same deterministic NodeAddr tie-breaker the codebase
already uses for rekey dual-initiation and cross-connection
resolution: the smaller NodeAddr wins as adopter, tears down its
in-flight handshake state, and proceeds with adoption; the larger
NodeAddr keeps the skip semantics, and its in-flight outbound is
reconciled by the cross-connection logic when the winner's fresh
msg1 arrives over the adopted socket. The dual cross-init stall is
eliminated; cross-init NAT-traversal completes in well under a
second even under host CPU contention.
- FSP session rekey is now hitless under packet loss and reordering.
Previously, a rekey could leave the two endpoints holding different
key sets for a brief window — if a handshake message was lost in
transit one side rotated keys while the other did not, and traffic
sealed in one key epoch reached a peer still on the other epoch and
failed to decrypt, producing bursts of AEAD decryption failures and
dropped connectivity until a later rekey reconverged the pair. The
receive path now trial-decrypts each frame against every live key
epoch (current, pending, and the draining previous session) for the
duration of the rekey transition, so no rotation ordering and no
packet reordering can cause a decryption failure. The previous-epoch
slot is retained as long as the peer keeps using it, with its drain
deadline anchored on the last frame the peer authenticates against
it rather than a fixed wall-clock timer, so a peer that did not
receive the new keys is not stranded by a silent permanent decrypt
failure. The lost-handshake case is closed by retransmitting the
third rekey handshake message until the peer is confirmed on the
new keys, with a bounded retry budget after which the rekey cycle
is cleanly abandoned and retried. There are no FSP decryption
failures across a rekey under lossy, jittery links.
#### Data-plane / metrics / observability
- The Tor transport now increments its `connect_refused` statistic (the
"Refused" line in fipstop) when a SOCKS5 connection is actively
refused, instead of recording every connect failure as a generic
SOCKS5 error. The counter previously stayed at zero.
- MMP sender metrics now ignore duplicate or regressed receiver reports
before updating RTT, loss, goodput, or ETX. Receiver reports also
suppress timestamp echo when dwell time overflows, so stale reports
cannot inflate SRTT.
- Reject-reason counters no longer double-count now that the rollout's
interim direct increments are removed. Six discovery counters
(`req_decode_error`, `req_duplicate`, `req_ttl_exhausted`,
`resp_decode_error`, `resp_identity_miss`, `resp_proof_failed`), six
bloom counters (`decode_error`, `invalid`, `non_v1`, `unknown_peer`,
`stale`, `fill_exceeded`), and five forwarding reject packet counters
(`decode_error_packets`, `ttl_exhausted_packets`,
`drop_no_route_packets`, `drop_mtu_exceeded_packets`,
`drop_send_error_packets`) were each incremented both by a direct bump
and again through the typed reject dispatch. The redundant direct
increments are removed — for the forwarding family the two calls are
collapsed into a single byte-aware reject entry point — so each counter
(and, for forwarding, its byte tally) counts once per event.
- Transport-layer mutex poisoning no longer cascades. Ten
`Mutex::lock().unwrap()` sites across the UDP, BLE, and Ethernet
transports would turn a single panic (poisoning the mutex) into a
cascade of panics on every subsequent lock. Each is replaced with
`lock().unwrap_or_else(|e| e.into_inner())`, recovering the guarded
data with no new dependency and no call-graph change; four
`local_addr.unwrap()` calls on the UDP start/adopt paths get a
provably-safe sentinel fallback. The critical sections are short,
locally-scoped, and not reachable from peer input, so this is
robustness hardening, not a remotely-triggerable fix.
#### Peer lifecycle / gateway
- A manual `fipsctl disconnect` now notifies the peer so teardown is
symmetric. Previously a manual disconnect tore down only the local
side and sent the peer nothing, so the peer kept its session and never
re-emitted its tree and filter announcements; on reconnect it was
never re-adopted as a child and its bloom filter was never recorded.
The local side now sends the disconnected peer a scoped `Disconnect`
(the same message graceful shutdown sends), so both ends tear down and
re-handshake cleanly on the next connection.
- `fips-gateway` no longer drops long-lived or DNS-cached client
mappings while traffic is still flowing. The virtual-IP pool's TTL
clock advanced only on DNS re-query, never on traffic, and the mapping
TTL is wired equal to the DNS TTL, so an in-use mapping was forced to
drain at TTL and reclaimed at the first zero-conntrack tick — breaking
long-lived, bursty, or DNS-cached clients. The tick now refreshes the
mapping's last-referenced time whenever conntrack reports active
sessions, and recovers a draining mapping to active (with a fresh
grace window) when traffic resumes; only genuinely idle mappings
drain.
#### macOS self-traffic / resolver
- Self-addressed mesh traffic is now delivered locally on macOS instead
of being dropped, for both `ping6` and full TCP/UDP. The point-to-point
`utun` interface egresses self-addressed traffic into the daemon, which
previously pushed it onto the mesh outbound path where it was dropped
for lack of a route to self; such packets are now hairpinned back to
the TUN for inbound delivery. macOS first routes self-addressed packets
as loopback (a `LOCAL` route via `lo0`), which leaves their transport
TX checksum offloaded and unfinished, so re-injecting them verbatim
made the local stack drop every segment whose checksum MSS clamping did
not happen to rewrite (the SYN and SYN-ACK got through, but the bare
ACK, data, and FIN were dropped, so connections to a node's own
`<npub>.fips` service half-opened and hung). The hairpin path now
recomputes the TCP/UDP checksum before re-injection, so full
self-connections — not just `ping6` — to a node's own `<npub>.fips`
address work. Linux was unaffected (the kernel already loops
self-traffic via `lo`). (#117)
- macOS `.fips` name resolution now works on a fresh install: the
shipped resolver shim points at `::1`, matching the daemon's default
IPv6 DNS listener, instead of `127.0.0.1`. The mismatched shim
(`nameserver 127.0.0.1` while the daemon listens on `::1`) broke
`getaddrinfo` for `.fips` on every macOS install since the resolver
was introduced.
#### CI & test-harness reliability
- Node-level multi-node tests no longer flake under parallel CPU load.
They previously delivered handshake packets over real localhost UDP,
whose kernel receive buffer could overflow and drop a packet when many
tests ran concurrently, panicking the large-network convergence tests.
A `cfg(test)`-only loopback `TransportHandle` variant now delivers
packets directly between nodes over an unbounded in-process channel, so
there is no socket buffer to overflow, and the previously-quarantined
large-network tests run in the default suite again. The shipping daemon
build is unaffected (the variant is test-gated).
- Integration suites that wait for the mesh to converge no longer
false-fail under concurrent CI load. The rekey, static-mesh, and
sidecar suites replace a fixed wall-clock baseline timeout (and a blind
sleep) with a progress-aware wait that polls the suite's own pairwise
pings, returns as soon as every pair is reachable, extends its deadline
while the reachable-pair count is still climbing, and gives up only
when progress stalls.
- Rekey integration test (`testing/static/scripts/rekey-test.sh`) no
longer false-fails on GitHub runners under packet loss and CPU
contention. Phase 1, Phase 3, and Phase 5 strict per-pair pings retry
up to 4 attempts (configurable via `MAX_PING_ATTEMPTS` /
`PING_RETRY_DELAY`) — under 1% per-direction loss, single-shot 20-pair
ping_all misses ~33% per phase from ICMP noise alone, and the
4-attempt retry brings that floor to ~3.2e-6 per phase; the
`wait_for_full_baseline` convergence loop stays single-shot so retries
there cannot conflate transient ping loss with still-converging routing
state. Phase 1 baseline-convergence headroom is bumped from 36s to 60s
to eliminate the intermittent Phase 1 timeout that previously required
a `gh run rerun --failed`, and a post-second-rekey settle window is
added in Phase 5 (mirroring Phase 3's 12-second pattern) to close the
post-rekey per-pair-ping flake from convergence exceeding the per-ping
5-second timeout. Test scaffold only; no daemon code changes, and the
success path is unchanged because the wait loops return as soon as all
20 pairs converge.
- ACL-allowlist integration test (`testing/acl-allowlist/test.sh`):
converted `assert_log_contains` from a one-shot `docker logs | grep`
snapshot into a bounded poll with the same wait-with-timeout shape
as `wait_for_peers_exact`. Absorbs the millisecond-to-second
variance in the XX-handshake cross-connection tie-breaker: the
inbound-handshake-context rejection can land tens of milliseconds
after the test's previous one-shot grep gave up, producing a
pre-existing flake on CI. Success-path cost is unchanged — the helper
returns as soon as the pattern appears.
#### Packaging & deployment
- AUR packaging: the `fips` and `fips-git` PKGBUILDs now install the
`fips-dns-setup` and `fips-dns-teardown` helpers into
`/usr/lib/fips/`, matching the Debian package. The AUR `package()`
@@ -564,113 +871,9 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
the arch-named file is present and carries a SHA-256 integrity
chain from the build runner through to `gh release upload`, so a
recurrence fails CI instead of publishing.
- Nostr discovery: filter unroutable direct UDP/TCP advert endpoints.
Publisher and validator now retain only endpoints that parse as
concrete socket addresses with routable IPs and nonzero ports.
`udp:nat` rendezvous endpoints and Tor endpoints pass through
unchanged. Adverts that collapse to zero usable endpoints after
filtering are rejected with a clear "missing publicly routable
endpoints" error. Before this change, misconfigured nodes could
publish RFC1918, loopback, link-local, CGNAT 100.64/10, IPv6 ULA,
or IPv6 link-local endpoints into Nostr discovery, and consumers
would cache and dial them; in mixed LAN/VPN/NAT environments, that
could prefer a misleading one-way private path over the intended
`udp:nat` bootstrap.
- Coord cache invalidation made surgical at parent-position-change
and root-change sites. Replaces the previous unconditional
`CoordCache::clear()` calls with two targeted methods:
`invalidate_via_node(node_addr)` (drops entries whose cached
ancestry contains the changed node, used at parent-switch /
become-root / loop-detection sites) and `invalidate_other_roots`
(drops entries from a different tree, used at root-change sites).
The previous global flush left `find_next_hop` returning `None`
for every non-direct-peer destination after every parent switch
until the cache passively re-warmed; surgical invalidation
preserves entries that remain correct across the topology change.
Peer-removal retains the original "no invalidation" behavior
(`find_next_hop` already recomputes against the current peer set
every call, and Discovery handles "no route" on demand).
- Rekey integration test (`testing/static/scripts/rekey-test.sh`):
Phase 1, Phase 3, and Phase 5 strict per-pair pings now retry up
to 4 attempts (configurable via `MAX_PING_ATTEMPTS` /
`PING_RETRY_DELAY`). Under low-level packet loss (1% per
direction), single-shot 20-pair ping_all misses with probability
~33% per phase from ICMP noise alone, masking the routing-state
signal the asserts target. The 4-attempt retry brings that floor
to ~3.2e-6 per phase. The `wait_for_full_baseline` convergence
loop itself stays single-shot — retries there would conflate
transient ping loss with still-converging routing state. Test
scaffold only; no daemon code changes.
- Apply ±15s symmetric jitter per session to the FMP and FSP rekey
timer trigger. Eliminates the steady-state dual-initiation race
in symmetric-start meshes; previously the smaller-NodeAddr
tie-breaker resolved correctness after every cycle's collision.
`node.rekey.after_secs` becomes the nominal interval rather than
a floor; mean is preserved.
- Rekey integration test (`testing/static/scripts/rekey-test.sh`):
bumped Phase 1 baseline-convergence headroom from 36s to 60s.
Eliminates the intermittent GitHub-runner Phase 1 timeout that
previously required `gh run rerun --failed`. Cost on the success
path is unchanged because the wait loop returns as soon as all 20
pairs converge.
- Rekey integration test (`testing/static/scripts/rekey-test.sh`):
added a post-second-rekey settle window in Phase 5, mirroring
Phase 3's existing 12-second pattern. Closes the intermittent
GitHub-runner Phase 5 per-pair-ping flake caused by post-rekey
routing convergence exceeding the per-ping 5-second timeout under
runner CPU contention. Cost on the success path is a fixed 12s per
suite run.
- ACL-allowlist integration test (`testing/acl-allowlist/test.sh`):
converted `assert_log_contains` from a one-shot `docker logs | grep`
snapshot into a bounded poll with the same wait-with-timeout shape
as `wait_for_peers_exact`. Absorbs the millisecond-to-second
variance in the XX-handshake cross-connection tie-breaker: the
inbound-handshake-context rejection can land tens of milliseconds
after the test's previous one-shot grep gave up, producing a
pre-existing flake on next-branch CI. Success-path cost is
unchanged — the helper returns as soon as the pattern appears.
- Nostr-discovered NAT-traversal events (`BootstrapEvent::Established`
and `BootstrapEvent::Failed`) for peers that are already connected
or actively handshaking are now short-circuited at the
`poll_nostr_discovery` dispatch sites before any cooldown
bookkeeping or fallback retry scheduling runs. Stale `Failed` events
previously poisoned the per-peer failure-state cooldown of healthy
peers and could trigger redundant retraversal attempts via
`schedule_retry` / `try_peer_addresses`; stale `Established`
handoffs could attempt to adopt a second socket against a live
connection. A defense-in-depth guard was added to
`adopt_established_traversal` so the same invariant holds if a
future caller bypasses the outer dispatch check. As a side benefit,
narrows a cooldown-poisoning vector previously available to an
attacker injecting stale failure events for an active peer.
- A manual `fipsctl disconnect` now notifies the peer so teardown is
symmetric. Previously a manual disconnect tore down only the local
side and sent the peer nothing, so the peer kept its session and never
re-emitted its tree and filter announcements; on reconnect it was
never re-adopted as a child and its bloom filter was never recorded.
The local side now sends the disconnected peer a scoped `Disconnect`
(the same message graceful shutdown sends), so both ends tear down and
re-handshake cleanly on the next connection.
- `fips-gateway` no longer drops long-lived or DNS-cached client
mappings while traffic is still flowing. The virtual-IP pool's TTL
clock advanced only on DNS re-query, never on traffic, and the mapping
TTL is wired equal to the DNS TTL, so an in-use mapping was forced to
drain at TTL and reclaimed at the first zero-conntrack tick — breaking
long-lived, bursty, or DNS-cached clients. The tick now refreshes the
mapping's last-referenced time whenever conntrack reports active
sessions, and recovers a draining mapping to active (with a fresh
grace window) when traffic resumes; only genuinely idle mappings
drain.
- Transport-layer mutex poisoning no longer cascades. Ten
`Mutex::lock().unwrap()` sites across the UDP, BLE, and Ethernet
transports would turn a single panic (poisoning the mutex) into a
cascade of panics on every subsequent lock. Each is replaced with
`lock().unwrap_or_else(|e| e.into_inner())`, recovering the guarded
data with no new dependency and no call-graph change; four
`local_addr.unwrap()` calls on the UDP start/adopt paths get a
provably-safe sentinel fallback. The critical sections are short,
locally-scoped, and not reachable from peer input, so this is
robustness hardening, not a remotely-triggerable fix.
#### fipstop
- `fipstop` no longer renders a garbled screen on startup or leaves
stray bytes on quit, most visible over SSH and inside tmux. Startup
forces a full repaint (`terminal.clear()`) before the first draw so
@@ -678,12 +881,6 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
stdin-poll thread a stop flag and joins it before restoring the
terminal, so post-raw-mode keystrokes or terminal query responses no
longer echo onto the restored screen.
- macOS `.fips` name resolution now works on a fresh install: the
shipped resolver shim points at `::1`, matching the daemon's default
IPv6 DNS listener, instead of `127.0.0.1`. The mismatched shim
(`nameserver 127.0.0.1` while the daemon listens on `::1`) broke
`getaddrinfo` for `.fips` on every macOS install since the resolver
was introduced.
## [0.3.0] - 2026-05-11
CONTRIBUTING.md
+4
View File
@@ -42,6 +42,10 @@ and fail without it. BLE-capable builds additionally need `bluez`,
`libdbus-1-dev`, and `pkg-config` installed; the default build picks
up BLE if those are present and skips it cleanly if not.
On Nix, `nix develop` provides the pinned toolchain and all of these
build prerequisites without any manual install; see the Nix / NixOS
section of [packaging/README.md](packaging/README.md).
For multi-node integration runs, Docker is required. The harness
under [testing/](testing/) starts containerized topologies and
exercises real mesh behavior; see [testing/README.md](testing/README.md)
Cargo.lock
Generated
+1 -1
View File
@@ -1074,7 +1074,7 @@ checksum = "9844ddc3a6e533d62bba727eb6c28b5d360921d5175e9ff0f1e621a5c590a4d5"
[[package]]
name = "fips"
version = "0.4.0-rc.1"
version = "0.4.2-dev"
dependencies = [
"arc-swap",
"bech32",
Cargo.toml
+1 -1
View File
@@ -1,6 +1,6 @@
[package]
name = "fips"
version = "0.4.0-rc.1"
version = "0.4.2-dev"
edition = "2024"
description = "A distributed, decentralized network routing protocol for mesh nodes connecting over arbitrary transports"
license = "MIT"
README.md
+17 -9
View File
@@ -3,7 +3,7 @@
![banner](docs/logos/fips_banner.png)
[![License: MIT](https://img.shields.io/badge/license-MIT-blue.svg)](LICENSE)
[![Rust](https://img.shields.io/badge/rust-1.85%2B-orange.svg)](https://www.rust-lang.org/)
[![Status](https://img.shields.io/badge/status-v0.4.0--dev-green.svg)](#status--roadmap)
[![Status](https://img.shields.io/badge/status-v0.4.2--dev-green.svg)](#status--roadmap)
A self-organizing encrypted mesh network built on Nostr identities,
capable of operating over arbitrary transports without central
@@ -98,9 +98,9 @@ This installs the daemon, CLI tools (`fipsctl`, `fipstop`), the
optional `fips-gateway` service, systemd units, and a default
`/etc/fips/fips.yaml` you can edit before starting.
For macOS, Windows, OpenWrt, the systemd tarball, or a from-source
build, see [docs/getting-started.md](docs/getting-started.md) for
the full multi-platform installation guide.
For macOS, Windows, OpenWrt, the systemd tarball, a Nix flake, or a
from-source build, see [docs/getting-started.md](docs/getting-started.md)
for the full multi-platform installation guide.
To join a live mesh and reach your first peer, follow the new-user
tutorial progression starting at
@@ -143,6 +143,12 @@ Nym (mixnet) transport builds on all desktop platforms. The OpenWrt
availability on the target; it will flip to ✅ only if confirmed
buildable there.
Alternatively, the repo ships a [Nix flake](flake.nix): `nix develop`
drops you into a shell with the pinned toolchain and every build
prerequisite (libclang, dbus, pkg-config) already provided, and
`nix build .#fips` builds all four binaries with no host setup. See the
Nix / NixOS section of [packaging/README.md](packaging/README.md).
## Documentation
`docs/` is organised by reader purpose:
@@ -204,12 +210,14 @@ testing/ Docker-based integration test harnesses + chaos simulation
## Status & roadmap
FIPS is at **v0.4.0**. The core protocol works end-to-end over
FIPS is at **v0.4.2-dev** on the `maint` branch.
[v0.4.1](https://github.com/jmcorgan/fips/releases/tag/v0.4.1) has
shipped; this line carries patch-level fixes for the 0.4.x series. The
core protocol works end-to-end over
UDP, TCP, Ethernet, Tor, Nym, and Bluetooth on a global, public test
mesh of thousands of nodes. v0.4.0 builds on the v0.3.0 testing-and-polishing
track, adding the Nym mixnet transport and mDNS LAN discovery
alongside the existing Nostr-mediated peer discovery, UDP NAT
traversal, peer ACL, and packaging hardening. New wire-format work
mesh of thousands of nodes. v0.4.0 added the Nym mixnet transport and
mDNS LAN discovery alongside the existing Nostr-mediated peer discovery,
UDP NAT traversal, peer ACL, and packaging hardening. New wire-format work
continues to be staged on the `next` branch for the subsequent
release line.
RELEASE-NOTES.md
+112 -245
View File
@@ -1,262 +1,136 @@
# FIPS v0.4.0
# FIPS v0.4.1
**Released**: 2026-06-DD (provisional)
**Released**: 2026-07-19
v0.4.0 is the throughput-and-observability release on the v0.3.x wire
format. It adds two new ways for nodes to find and reach each other (the
Nym mixnet transport and opt-in mDNS LAN discovery), overhauls the data
plane for higher single-node throughput and lower per-packet CPU, moves
the entire operator read surface off the data-plane hot path so
observability stays responsive under load, ships a reworked `fipstop`
TUI, and hardens FMP and FSP rekey to be hitless under packet loss in
both directions. It also folds in the accumulated mesh-convergence,
admission-control, and packaging fixes from the maintenance line.
v0.4.1 is a maintenance release on the v0.4.x line. It raises the default
antipoison cap on inbound bloom filter announcements, removes a redundant
spanning-tree metric counter, fixes two convergence and path-MTU bugs, and
cuts per-packet CPU in the bloom and identity paths. There is no wire
format change and no new feature surface.
v0.4.0 is wire-compatible with v0.3.0. Mixed meshes interoperate; there
is no flag-day upgrade. A deployed v0.3.0 node and an upgraded v0.4.0
node peer, rekey, and route normally, so you can roll the upgrade out
across a mesh in any order.
v0.4.1 is wire-compatible with v0.4.0. Nodes can be upgraded one at a time
with no coordinated restart, though one behavior change below is worth
reading before you start a rolling upgrade.
## At a glance
- New outbound Nym mixnet transport with a single-container demo and a
new mixnet-relay example.
- Opt-in mDNS / DNS-SD discovery on the local link.
- Data-plane overhaul: off-task encrypt and decrypt worker pools, GSO,
connected-UDP send path, copy-avoidance on receive, batched macOS
receive.
- The full `show_*` read surface now serves off the receive loop, so
`fipsctl` and `fipstop` stay responsive on loaded nodes; a new
counter-only `show_metrics` query enables a Prometheus scraper at no
hot-path cost.
- Reworked `fipstop` TUI on a machine-verified render-snapshot base.
- Rekey is now hitless under loss and reordering in both directions.
## What's new
### Nym mixnet transport
FIPS can now peer over the [Nym](https://nymtech.net/) mixnet for
metadata-resistant connectivity. The new `transports.nym` transport
makes outbound connections through a `nym-socks5-client` SOCKS5 proxy
that you run alongside the daemon (for example as a service running
alongside the fips daemon, or as a sidecar container). The transport
waits at startup for the nym-socks5-client to become ready before giving
up.
This is a privacy and anonymity deployment mode chosen for its own
properties. It mixes your FIPS traffic into the Nym cover-traffic
network so that link-level observers cannot correlate which mesh peers
are talking. A new `examples/sidecar-nostr-mixnet-relay/` demonstrates a
FIPS-reachable Nostr relay peered across the mixnet end to end, and a
single-container demo ships with the transport.
Enable it by adding a `transports.nym` instance and pointing it at your
running nym-socks5-client. See the transports reference for the field
set.
### mDNS LAN discovery
Nodes on a shared local link can now find each other with zero address
configuration. The opt-in `node.discovery.lan` path runs an mDNS /
DNS-SD responder and browser: each node advertises a FIPS service record
on the link and adopts the peers it discovers. This complements the
existing Nostr-mediated overlay discovery for the common case where the
peers are simply on the same LAN.
Turn it on with `node.discovery.lan.enabled: true`. `service_type` and
`scope` tune the advertised service record and which interfaces
participate. Discovery on the local link needs no relay and no STUN.
### Data-plane throughput overhaul
The receive and send paths were reworked for higher single-node
throughput and lower per-packet CPU, building on the v0.3.0
crypto-backend swap:
- **Off-task encrypt and decrypt.** Per-peer encrypt and decrypt now run
on dedicated worker tasks rather than inline on the receive loop, so a
single busy peer no longer serializes the whole node's crypto.
- **GSO and connected-UDP send.** The Linux send path uses generic
segmentation offload and a connected-UDP socket where available,
cutting syscall overhead on bulk flows.
- **Copy-avoidance on receive.** The receive hot path avoids buffer
copies it previously made per packet.
- **Batched macOS receive.** macOS gains a `recvmsg_x` batched receive,
mirroring the Linux `recvmmsg` batching from v0.3.0.
- **Shared immutable-state context and an atomic metric registry.**
Immutable per-node state moved into a single shared context, and
counters live in an atomic metric registry that the new `show_metrics`
query reads without touching the hot path.
These are all internal to the data plane and require no operator action.
### Observability off the hot path
Every read-only control query now renders from a snapshot published once
per tick into a lock-free `ArcSwap`, served from the control accept task
instead of round-tripping the data-plane receive loop. This covers
`show_status`, `show_stats_*`, `show_peers`, `show_sessions`,
`show_links`, `show_connections`, `show_transports`, `show_mmp`,
`show_tree`, `show_bloom`, `show_cache`, `show_routing`,
`show_identity_cache`, `show_acl`, `show_listening_sockets`, and the new
`show_metrics`. Only the mutating `connect` and `disconnect` commands
still reach the loop.
The practical effect: on a loaded node where the receive loop was busy,
`fipsctl` and `fipstop` queries previously stalled or timed out (the
five-second query pattern operators saw). They now answer promptly
regardless of data-plane load. Per-entity snapshots reuse unchanged rows
by pointer, so the per-tick publish cost stays bounded as peer and
session counts grow.
A new **`show_metrics`** query (surfaced as `fipsctl stats metrics`)
returns a counter-only snapshot of every metric family. It is the
enabler for a Prometheus scraper that pulls node counters at no hot-path
cost.
### Reworked fipstop TUI
`fipstop` gets a rendering, navigation, and read-surface overhaul on a
machine-verified base: a render-snapshot harness asserts the exact text
grid and per-cell style of every view against canned control-socket
output. New daemon-resolved fields surface through the snapshots,
including effective persistence, root and is-root state, a
per-transport-type peer-count map, per-peer effective depth, the root
npub, and the last-sent uptree filter fill ratio with the subtree size
estimate.
A separate fix clears a garbled-screen problem on startup and stray
bytes on quit, most visible over SSH and inside tmux: startup now forces
a full repaint before the first draw, and quit stops and joins the
stdin-poll thread before restoring the terminal, so post-raw-mode
keystrokes no longer echo onto the restored screen.
### Rekey reliability
FMP and FSP session rekey are now hitless under packet loss and
reordering in both directions:
- Inbound frames are authenticated against the pending session before
the K-bit cutover promotes it, so a spoofed or stale frame cannot
derail a rekey in progress.
- Rekey message-1 retransmission is bounded, and the link-dead heartbeat
is rekey-aware so an in-flight rekey is not mistaken for a dead link.
- FSP session rekey holds connectivity across the rekey window under
loss and reordering.
- Dual-initiation races (both peers starting a rekey at once on a
high-latency link) are desynchronized with symmetric jitter so the two
sides converge on one session rather than fighting.
- An exhausted retransmission-budget abort, an expected and self-limiting
outcome on lossy or high-latency links, is logged at debug rather than
warn.
The net operator takeaway: rekey completes cleanly without dropping
traffic, even on lossy or high-latency links, and the log no longer
cries wolf when a rekey gives up and retries.
- `node.bloom.max_inbound_fpr` default moves from `0.10` to `0.20`.
- The `parent_switched` metric counter is gone. Use `parent_switches`.
- Spanning tree no longer serves stale coordinates after a parent link is
lost through peer removal.
- Discovery no longer loosens a path MTU clamp it had correctly tightened.
- Bloom probing and identity operations do measurably less work per call,
with identical results.
## Behavior changes worth flagging
These affect operators on upgrade.
### The inbound filter FPR cap default doubles again
- **Bloom filter antipoison cap raised.** `node.bloom.max_inbound_fpr`
moves from 0.05 to 0.10, accepting filters with a higher derived
false-positive rate before rejecting them. This reduces spurious
filter rejections on larger meshes while keeping the antipoison
protection in place.
- **TCP inbound cap honors `max_connections`.** The TCP inbound accept
ceiling now resolves from explicit per-transport
`max_inbound_connections`, then node-wide
`node.limits.max_connections`, then the built-in default of 256.
Previously the TCP inbound ceiling was hardwired to 256 and ignored
`max_connections`, so raising it had no effect on inbound TCP.
- **Static host aliases hot-reload.** `/etc/fips/hosts` now reloads on
mtime change once per tick rather than only at startup, so display
names in `fipsctl` and `fipstop` reflect edits without a daemon
restart. The peer ACL reloads through the same lock-free snapshot
mechanism.
- **Quieter logs on busy public-mesh nodes.** Routine per-peer
connection-lifecycle and capacity-cap events, no-route session-datagram
drops, and exhausted rekey-budget aborts are demoted to debug, so
genuinely notable info and warn lines are no longer drowned out.
- **More visible drops.** Receive-path silent rejections now flow
through typed reject-reason counters, and discovery counts requests
dropped when the dedup cache is full (`req_dedup_cache_full`, visible
via `show_routing`). Drops that were previously silent are now
countable.
- **Tor connect-refused accounting.** The Tor transport increments its
`connect_refused` statistic (the "Refused" line in `fipstop`) on an
actively-refused SOCKS5 connect, instead of recording every connect
failure as a generic SOCKS5 error.
`node.bloom.max_inbound_fpr` goes from `0.10` to `0.20`. The cap rejects
inbound `FilterAnnounce` frames whose advertised false positive rate
exceeds it. On the fixed 1 KB, k=5 filter, `0.10` corresponds to a fill of
0.631 and roughly 1,630 reachable entries, and the busiest nodes'
aggregates had started reaching that ceiling as the mesh grew. `0.20`
corresponds to a fill of 0.7248 and roughly 2,114 entries.
Be aware that this is the second time in two releases that this default
has doubled, for the same reason both times. That is worth stating plainly
rather than repeating the previous release's framing: raising the cap buys
headroom, it does not fix anything. The real constraint is the fixed 1 KB
filter size, which is a protocol constant. The structural remedy is the v2
filter work, where filter capacity scales with the mesh instead of being
pinned. This release is an interim step to keep legitimate aggregates from
being rejected until that lands. It is not the start of a pattern of
raising the cap once per release, and if you are sizing capacity planning
around this number, plan against the v2 work rather than against a third
raise.
The antipoison property the cap exists for is preserved. A saturated or
deliberately poisoned filter still presents an FPR near 100% and is still
rejected.
**This matters during a rolling upgrade.** A v0.4.1 node accepts a
`FilterAnnounce` with a derived FPR between 0.10 and 0.20; a v0.4.0 node
drops the same frame, and the drop is silent on the wire with no NACK. The
cap also gates the mesh size estimator, which declines to produce a value
when any contributing filter is over the cap. So while a mesh is partly
upgraded, upgraded and not-yet-upgraded nodes can legitimately report
different mesh sizes, or one can report a size while the other reports
unknown. This resolves once every node is on v0.4.1. If you want to avoid
the window entirely, set `node.bloom.max_inbound_fpr: 0.10` explicitly in
your config before upgrading and remove it after the last node is done.
### The `parent_switched` counter is removed
`parent_switched` was incremented on the line immediately before
`parent_switches` at every site and never independently, so the two
counters always held the same value. `parent_switched` is now gone from
the tree metrics, the control socket snapshot, and the `fipstop` tree
view. `parent_switches` remains and is unchanged.
If you scrape the control socket, or have dashboards or alerts referencing
`parent_switched`, point them at `parent_switches`. Anything still asking
for `parent_switched` will find nothing rather than a zero.
## Notable bug fixes
The CHANGELOG has the exhaustive list. This is the operator-relevant
subset of fixes for behavior that shipped in v0.3.0.
### Stale coordinates after losing a parent through peer removal
- **Symmetric peer teardown on manual disconnect.** A manual
`fipsctl disconnect` now sends the peer a scoped Disconnect so both
ends tear down and re-handshake cleanly. Previously a manual
disconnect tore down only the local side, leaving the peer with a
stale session that was never re-adopted as a child and whose bloom
filter was never re-recorded.
- **Gateway holds long-lived and DNS-cached mappings.** `fips-gateway`
no longer drops a virtual-IP mapping while traffic is still flowing.
The mapping TTL clock previously advanced only on DNS re-query, so a
busy long-lived or DNS-cached client could have its mapping reclaimed
mid-flow. The tick now refreshes the mapping whenever conntrack reports
active sessions and recovers a draining mapping to active when traffic
resumes; only genuinely idle mappings drain.
- **Accurate mesh-size estimate under filter overlap.** The mesh-size
estimator now estimates the cardinality of the OR-union of self plus
every connected peer's inbound filter, instead of summing per-filter
cardinalities of tree peers. Summing assumed the filters were disjoint,
so a stale or oversized parent filter or a routing loop inflated the
reported mesh size and a tree rebalance flapped the count. OR-union
deduplicates overlap, equals the old result in the disjoint case, and
removes the estimate's dependence on tree-declaration cache freshness.
- **Single-uplink node reattaches within a round-trip.** A node with one
tree peer, which has periodic parent re-evaluation disabled, was left
self-rooted and unreachable if its one-shot attaching TreeAnnounce was
lost, until the next periodic re-broadcast. Tree-position exchange is
now self-healing on the receive path: a node that hears an announce
advertising a strictly worse root echoes its own declaration back,
provoking the better-rooted peer to re-push its real position
immediately.
When a node's parent link dropped via peer removal, the node correctly
reparented or self-rooted, but skipped the coordinate cache invalidation
that every other position-change path performs. Cached entries for
downstream destinations kept the node's old coordinate prefix. This did
not self-correct the way a stale cache entry normally would: routing
access refreshes an entry's TTL, so an entry that was actively being
routed through never expired, and was only fixed by an unrelated fresh
insert. Both invalidation classes now run on this path, matching the
loop-detection branch.
### Discovery could loosen a tightened path MTU clamp
An originator handling a `LookupResponse` overwrote its cached path MTU
unconditionally. If a reactive `MtuExceeded` or `PathMtuNotification` had
already taught it a tighter value, a later, looser discovery estimate
would clobber that and re-loosen the clamp, risking a return to dropped
oversized packets. The cached and received values are now compared and the
tighter one is kept.
## Upgrade notes
Operator-actionable items moving from v0.3.0 to v0.4.0:
This is a drop-in upgrade from v0.4.0 with no wire format change, no
config migration, and no coordinated restart. Upgrade nodes in whatever
order you like.
- **Wire-compatible, no flag day.** v0.4.0 peers with v0.3.0. Upgrade
nodes in any order. During a rolling upgrade you may see some log lines
on the upgraded side as it interacts with not-yet-upgraded peers;
behavior is correct, log noise only.
- **Bloom antipoison cap default changed.** `node.bloom.max_inbound_fpr`
now defaults to 0.10 (was 0.05). If you set this explicitly, review
whether you still want the old value.
- **New optional config surfaces.** `transports.nym` (outbound Nym
mixnet) and `node.discovery.lan` (mDNS LAN discovery) are both opt-in
and off by default. Adding them is the only way to turn the new paths
on.
- **TCP inbound cap.** If you relied on the old hardwired 256 inbound-TCP
ceiling, note it now honors `max_inbound_connections` then
`node.limits.max_connections` then 256.
- **New observability query.** `fipsctl stats metrics` (the
`show_metrics` control query) returns a counter-only snapshot suitable
for a scraper.
Two things to do rather than assume:
## Getting v0.4.0
1. If you monitor `parent_switched`, move to `parent_switches` before
upgrading, or your dashboards will go blank rather than error.
2. During the rolling window, expect upgraded and not-yet-upgraded nodes
to potentially disagree about mesh size, per the FPR cap section above.
This is expected and self-resolves. Do not chase it as a bug unless it
persists after every node reports `0.4.1`.
If you have pinned `node.bloom.max_inbound_fpr` explicitly in your config,
your setting is honored and nothing changes for you. The change only
affects nodes taking the default.
Downgrading to v0.4.0 is supported and needs no special handling.
## Getting v0.4.1
- **Linux x86_64 / aarch64**: `.deb` and tarball at the
[v0.4.0 release page](https://github.com/jmcorgan/fips/releases/tag/v0.4.0).
[v0.4.1 release page](https://github.com/jmcorgan/fips/releases/tag/v0.4.1).
- **Arch Linux**: `fips` from the AUR.
- **macOS**: `.pkg` at the v0.4.0 release page.
- **Windows**: ZIP at the v0.4.0 release page.
- **OpenWrt**: `.ipk` at the v0.4.0 release page.
- **From source**: `cargo build --release` from a checkout of the v0.4.0
- **macOS**: `.pkg` at the v0.4.1 release page.
- **Windows**: ZIP at the v0.4.1 release page.
- **OpenWrt**: `.ipk` (OpenWrt 24.x and earlier) or `.apk` (OpenWrt 25+)
at the v0.4.1 release page.
- **From source**: `cargo build --release` from a checkout of the v0.4.1
tag (Rust 1.94.1 per `rust-toolchain.toml`; `libclang-dev` is a
required Linux build prerequisite).
- **Nix / NixOS**: `nix build .#fips` from a checkout of the v0.4.1 tag
builds the binaries from source with the pinned toolchain and no manual
prerequisites (see the Nix section of `packaging/README.md`).
The full per-commit changelog lives in
[`CHANGELOG.md`](../../CHANGELOG.md). Issues and discussion at
@@ -267,13 +141,6 @@ The full per-commit changelog lives in
Thanks to everyone who contributed code, packaging work, bug reports, or
reviews to this release.
- [@jcorgan](https://github.com/jmcorgan): release shepherd, high-level
design, control read plane, rekey hardening, admission, bug fixes,
testing, packaging, PR coordination, and issue resolution.
- [@mmalmi](https://github.com/mmalmi): opt-in mDNS LAN discovery and
data-plane performance work.
- [@Origami74](https://github.com/Origami74): macOS packaging and
website coordination.
- [@dskvr](https://github.com/dskvr): AUR packaging.
- [@oleksky](https://github.com/oleksky): Nym mixnet transport and the
single-container mixnet demo.
- [@jcorgan](https://github.com/jmcorgan): release shepherd, spanning-tree
and discovery fixes, bloom and identity performance work, antipoison cap
change, and testing.
docs/design/fips-bloom-filters.md
+2 -2
View File
@@ -360,14 +360,14 @@ control socket and `fipstop` dashboard. (See `compute_mesh_size()` in
The estimator refuses to produce a value when any contributing filter
is above the antipoison FPR cap (`node.bloom.max_inbound_fpr`,
default `0.10`); a partial aggregate would silently underestimate.
default `0.20`); a partial aggregate would silently underestimate.
Consumers handle the resulting `None` by displaying an "unknown"
state rather than a misleading number.
## Antipoison: Inbound FPR Cap
Inbound `FilterAnnounce` payloads are checked against
`node.bloom.max_inbound_fpr` (default `0.10`). Filters whose
`node.bloom.max_inbound_fpr` (default `0.20`). Filters whose
estimated false positive rate exceeds the cap are dropped silently
(no NACK on the wire) — they would otherwise inflate downstream
candidate evaluation cost without contributing useful discrimination.
docs/getting-started.md
+17
View File
@@ -88,6 +88,23 @@ See [packaging/README.md](../packaging/README.md) for per-format
build details, cross-target options, and the full `make` target
list.
### With Nix (flake)
On Nix/NixOS, a [flake](../flake.nix) at the project root builds the
binaries from source with the pinned toolchain and no manual
prerequisite install:
```sh
nix build .#fips # all four binaries, into ./result/bin
nix develop # dev shell with the toolchain + build deps
```
This path produces binaries only — it does not run the installer, so
there are no systemd units, no `fips` group, and no default `fips.yaml`.
On NixOS, wire the daemon in through your system configuration using the
flake's `packages.<system>.fips` output instead. See the Nix / NixOS
section of [packaging/README.md](../packaging/README.md).
## What's installed and running
Here's what the installer leaves on your machine, what's
docs/reference/configuration.md
+2 -2
View File
@@ -277,7 +277,7 @@ Controls tree construction and parent selection.
| Parameter | Type | Default | Description |
|-----------|------|---------|-------------|
| `node.bloom.update_debounce_ms` | u64 | `500` | Debounce interval for filter update propagation |
| `node.bloom.max_inbound_fpr` | f64 | `0.10` | Antipoison cap: reject inbound `FilterAnnounce` frames whose advertised false-positive rate exceeds this value. Valid range `(0.0, 1.0)`. The default `0.10` corresponds to fill 0.631 at k=5 (≈1,630 entries on the 1 KB filter); a saturated/poisoned filter is still ~100% FPR and rejected |
| `node.bloom.max_inbound_fpr` | f64 | `0.20` | Antipoison cap: reject inbound `FilterAnnounce` frames whose advertised false-positive rate exceeds this value. Valid range `(0.0, 1.0)`. The default `0.20` corresponds to fill 0.7248 at k=5 (≈2,114 entries on the 1 KB filter); a saturated/poisoned filter is still ~100% FPR and rejected |
Bloom filter size (1 KB), hash count (5), and size classes are protocol
constants and not configurable.
@@ -944,7 +944,7 @@ node:
flap_dampening_secs: 120 # extended hold-down on flap
bloom:
update_debounce_ms: 500
max_inbound_fpr: 0.10 # antipoison cap on inbound FilterAnnounce FPR
max_inbound_fpr: 0.20 # antipoison cap on inbound FilterAnnounce FPR
session:
default_ttl: 64
pending_packets_per_dest: 16
docs/releases/release-notes-v0.4.0.md
+44 -2
View File
@@ -1,6 +1,6 @@
# FIPS v0.4.0
**Released**: 2026-06-DD (provisional)
**Released**: 2026-06-27
v0.4.0 is the throughput-and-observability release on the v0.3.x wire
format. It adds two new ways for nodes to find and reach each other (the
@@ -31,6 +31,11 @@ across a mesh in any order.
hot-path cost.
- Reworked `fipstop` TUI on a machine-verified render-snapshot base.
- Rekey is now hitless under loss and reordering in both directions.
- New packaging targets: an OpenWrt `.apk` for OpenWrt 25+ and a Nix
flake for reproducible from-source builds on Nix/NixOS.
- Six route-class transit counters partition forwarded traffic by its
tree relationship to the next hop, visible via `show_routing` and
`show_status`.
## What's new
@@ -115,6 +120,14 @@ returns a counter-only snapshot of every metric family. It is the
enabler for a Prometheus scraper that pulls node counters at no hot-path
cost.
Six **route-class transit counters** partition transit-forwarded packets
by their tree relationship to the chosen next hop — tree-up, tree-down,
tree-down-cross, cross-link descend, cross-link ascend, and direct-peer
— and the six classes sum to `forwarded_packets`. They surface through
`show_routing` and `show_status`, and the `fipstop` routing tab is
reorganized so its two columns separate own/endpoint traffic from
forwarded/transit traffic with the tree-down-cross line visually flagged.
### Reworked fipstop TUI
`fipstop` gets a rendering, navigation, and read-surface overhaul on a
@@ -155,6 +168,22 @@ The net operator takeaway: rekey completes cleanly without dropping
traffic, even on lossy or high-latency links, and the log no longer
cries wolf when a rekey gives up and retries.
### New packaging targets
- **OpenWrt `.apk`.** A new `.apk` package targets OpenWrt 25+, where
apk-tools is the mandatory package manager; the existing `.ipk`
continues to cover OpenWrt 24.x and earlier. It is built SDK-free,
reusing the `.ipk` cross-compile and installed-filesystem payload, and
releases publish `.apk` artifacts and checksums alongside `.ipk`. Like
the `.ipk`, the package is unsigned and installed with
`apk add --allow-untrusted`.
- **Nix flake.** A `flake.nix` at the project root builds all four
binaries (`fips`, `fipsctl`, `fips-gateway`, `fipstop`) from source on
Nix/NixOS, pinning the exact toolchain and wiring the native build
dependencies so no host setup is needed beyond Nix with flakes
enabled. It exposes `nix build`, `nix run`, a `nix develop` dev shell,
and `nix flake check`, with `flake.lock` committed for reproducibility.
## Behavior changes worth flagging
These affect operators on upgrade.
@@ -223,6 +252,15 @@ subset of fixes for behavior that shipped in v0.3.0.
advertising a strictly worse root echoes its own declaration back,
provoking the better-rooted peer to re-push its real position
immediately.
- **macOS self-connections work end to end (#117).** Traffic a macOS
node sends to its own `<npub>.fips` address is now delivered locally
for full TCP/UDP, not just `ping6`. The point-to-point `utun` egresses
self-addressed packets into the daemon with an unfinished transport
checksum (macOS offloads it on the `lo0` loopback route), so
re-injecting them verbatim made the local stack drop every segment the
MSS-clamp rewrite did not happen to fix and self-connections
half-opened and hung. The hairpin path now recomputes the TCP/UDP
checksum before re-injection. Linux was unaffected.
## Upgrade notes
@@ -253,10 +291,14 @@ Operator-actionable items moving from v0.3.0 to v0.4.0:
- **Arch Linux**: `fips` from the AUR.
- **macOS**: `.pkg` at the v0.4.0 release page.
- **Windows**: ZIP at the v0.4.0 release page.
- **OpenWrt**: `.ipk` at the v0.4.0 release page.
- **OpenWrt**: `.ipk` (OpenWrt 24.x and earlier) or `.apk` (OpenWrt 25+)
at the v0.4.0 release page.
- **From source**: `cargo build --release` from a checkout of the v0.4.0
tag (Rust 1.94.1 per `rust-toolchain.toml`; `libclang-dev` is a
required Linux build prerequisite).
- **Nix / NixOS**: `nix build .#fips` from a checkout of the v0.4.0 tag
builds the binaries from source with the pinned toolchain and no manual
prerequisites (see the Nix section of `packaging/README.md`).
The full per-commit changelog lives in
[`CHANGELOG.md`](../../CHANGELOG.md). Issues and discussion at
docs/releases/release-notes-v0.4.1.md
+146
View File
@@ -0,0 +1,146 @@
# FIPS v0.4.1
**Released**: 2026-07-19
v0.4.1 is a maintenance release on the v0.4.x line. It raises the default
antipoison cap on inbound bloom filter announcements, removes a redundant
spanning-tree metric counter, fixes two convergence and path-MTU bugs, and
cuts per-packet CPU in the bloom and identity paths. There is no wire
format change and no new feature surface.
v0.4.1 is wire-compatible with v0.4.0. Nodes can be upgraded one at a time
with no coordinated restart, though one behavior change below is worth
reading before you start a rolling upgrade.
## At a glance
- `node.bloom.max_inbound_fpr` default moves from `0.10` to `0.20`.
- The `parent_switched` metric counter is gone. Use `parent_switches`.
- Spanning tree no longer serves stale coordinates after a parent link is
lost through peer removal.
- Discovery no longer loosens a path MTU clamp it had correctly tightened.
- Bloom probing and identity operations do measurably less work per call,
with identical results.
## Behavior changes worth flagging
### The inbound filter FPR cap default doubles again
`node.bloom.max_inbound_fpr` goes from `0.10` to `0.20`. The cap rejects
inbound `FilterAnnounce` frames whose advertised false positive rate
exceeds it. On the fixed 1 KB, k=5 filter, `0.10` corresponds to a fill of
0.631 and roughly 1,630 reachable entries, and the busiest nodes'
aggregates had started reaching that ceiling as the mesh grew. `0.20`
corresponds to a fill of 0.7248 and roughly 2,114 entries.
Be aware that this is the second time in two releases that this default
has doubled, for the same reason both times. That is worth stating plainly
rather than repeating the previous release's framing: raising the cap buys
headroom, it does not fix anything. The real constraint is the fixed 1 KB
filter size, which is a protocol constant. The structural remedy is the v2
filter work, where filter capacity scales with the mesh instead of being
pinned. This release is an interim step to keep legitimate aggregates from
being rejected until that lands. It is not the start of a pattern of
raising the cap once per release, and if you are sizing capacity planning
around this number, plan against the v2 work rather than against a third
raise.
The antipoison property the cap exists for is preserved. A saturated or
deliberately poisoned filter still presents an FPR near 100% and is still
rejected.
**This matters during a rolling upgrade.** A v0.4.1 node accepts a
`FilterAnnounce` with a derived FPR between 0.10 and 0.20; a v0.4.0 node
drops the same frame, and the drop is silent on the wire with no NACK. The
cap also gates the mesh size estimator, which declines to produce a value
when any contributing filter is over the cap. So while a mesh is partly
upgraded, upgraded and not-yet-upgraded nodes can legitimately report
different mesh sizes, or one can report a size while the other reports
unknown. This resolves once every node is on v0.4.1. If you want to avoid
the window entirely, set `node.bloom.max_inbound_fpr: 0.10` explicitly in
your config before upgrading and remove it after the last node is done.
### The `parent_switched` counter is removed
`parent_switched` was incremented on the line immediately before
`parent_switches` at every site and never independently, so the two
counters always held the same value. `parent_switched` is now gone from
the tree metrics, the control socket snapshot, and the `fipstop` tree
view. `parent_switches` remains and is unchanged.
If you scrape the control socket, or have dashboards or alerts referencing
`parent_switched`, point them at `parent_switches`. Anything still asking
for `parent_switched` will find nothing rather than a zero.
## Notable bug fixes
### Stale coordinates after losing a parent through peer removal
When a node's parent link dropped via peer removal, the node correctly
reparented or self-rooted, but skipped the coordinate cache invalidation
that every other position-change path performs. Cached entries for
downstream destinations kept the node's old coordinate prefix. This did
not self-correct the way a stale cache entry normally would: routing
access refreshes an entry's TTL, so an entry that was actively being
routed through never expired, and was only fixed by an unrelated fresh
insert. Both invalidation classes now run on this path, matching the
loop-detection branch.
### Discovery could loosen a tightened path MTU clamp
An originator handling a `LookupResponse` overwrote its cached path MTU
unconditionally. If a reactive `MtuExceeded` or `PathMtuNotification` had
already taught it a tighter value, a later, looser discovery estimate
would clobber that and re-loosen the clamp, risking a return to dropped
oversized packets. The cached and received values are now compared and the
tighter one is kept.
## Upgrade notes
This is a drop-in upgrade from v0.4.0 with no wire format change, no
config migration, and no coordinated restart. Upgrade nodes in whatever
order you like.
Two things to do rather than assume:
1. If you monitor `parent_switched`, move to `parent_switches` before
upgrading, or your dashboards will go blank rather than error.
2. During the rolling window, expect upgraded and not-yet-upgraded nodes
to potentially disagree about mesh size, per the FPR cap section above.
This is expected and self-resolves. Do not chase it as a bug unless it
persists after every node reports `0.4.1`.
If you have pinned `node.bloom.max_inbound_fpr` explicitly in your config,
your setting is honored and nothing changes for you. The change only
affects nodes taking the default.
Downgrading to v0.4.0 is supported and needs no special handling.
## Getting v0.4.1
- **Linux x86_64 / aarch64**: `.deb` and tarball at the
[v0.4.1 release page](https://github.com/jmcorgan/fips/releases/tag/v0.4.1).
- **Arch Linux**: `fips` from the AUR.
- **macOS**: `.pkg` at the v0.4.1 release page.
- **Windows**: ZIP at the v0.4.1 release page.
- **OpenWrt**: `.ipk` (OpenWrt 24.x and earlier) or `.apk` (OpenWrt 25+)
at the v0.4.1 release page.
- **From source**: `cargo build --release` from a checkout of the v0.4.1
tag (Rust 1.94.1 per `rust-toolchain.toml`; `libclang-dev` is a
required Linux build prerequisite).
- **Nix / NixOS**: `nix build .#fips` from a checkout of the v0.4.1 tag
builds the binaries from source with the pinned toolchain and no manual
prerequisites (see the Nix section of `packaging/README.md`).
The full per-commit changelog lives in
[`CHANGELOG.md`](../../CHANGELOG.md). Issues and discussion at
[github.com/jmcorgan/fips](https://github.com/jmcorgan/fips).
## Contributors
Thanks to everyone who contributed code, packaging work, bug reports, or
reviews to this release.
- [@jcorgan](https://github.com/jmcorgan): release shepherd, spanning-tree
and discovery fixes, bloom and identity performance work, antipoison cap
change, and testing.
flake.lock
Generated
+100
View File
@@ -0,0 +1,100 @@
{
"nodes": {
"fenix": {
"inputs": {
"nixpkgs": [
"nixpkgs"
],
"rust-analyzer-src": "rust-analyzer-src"
},
"locked": {
"lastModified": 1781527054,
"narHash": "sha256-1fX9ev2Fh5QoKQ41G9dYutjo5j/jywu6tZse5Eb1Ck4=",
"owner": "nix-community",
"repo": "fenix",
"rev": "8c2e51dffefc040a21975da7abf6f252c8c9b783",
"type": "github"
},
"original": {
"owner": "nix-community",
"repo": "fenix",
"type": "github"
}
},
"flake-utils": {
"inputs": {
"systems": "systems"
},
"locked": {
"lastModified": 1731533236,
"narHash": "sha256-l0KFg5HjrsfsO/JpG+r7fRrqm12kzFHyUHqHCVpMMbI=",
"owner": "numtide",
"repo": "flake-utils",
"rev": "11707dc2f618dd54ca8739b309ec4fc024de578b",
"type": "github"
},
"original": {
"owner": "numtide",
"repo": "flake-utils",
"type": "github"
}
},
"nixpkgs": {
"locked": {
"lastModified": 1781074563,
"narHash": "sha256-md8WlXOlfnIeHeOScMTTHFyf2d6iaTwPl2apR5EQ3P4=",
"owner": "NixOS",
"repo": "nixpkgs",
"rev": "9ae611a455b90cf061d8f332b977e387bda8e1ca",
"type": "github"
},
"original": {
"owner": "NixOS",
"ref": "nixos-unstable",
"repo": "nixpkgs",
"type": "github"
}
},
"root": {
"inputs": {
"fenix": "fenix",
"flake-utils": "flake-utils",
"nixpkgs": "nixpkgs"
}
},
"rust-analyzer-src": {
"flake": false,
"locked": {
"lastModified": 1781453968,
"narHash": "sha256-+V3nK4pCngbmgyVGXY6Kkrlevp4ocPkJJLf2aqwkDNA=",
"owner": "rust-lang",
"repo": "rust-analyzer",
"rev": "cc272809a173c2c11d0e479d639c811c1eacf049",
"type": "github"
},
"original": {
"owner": "rust-lang",
"ref": "nightly",
"repo": "rust-analyzer",
"type": "github"
}
},
"systems": {
"locked": {
"lastModified": 1681028828,
"narHash": "sha256-Vy1rq5AaRuLzOxct8nz4T6wlgyUR7zLU309k9mBC768=",
"owner": "nix-systems",
"repo": "default",
"rev": "da67096a3b9bf56a91d16901293e51ba5b49a27e",
"type": "github"
},
"original": {
"owner": "nix-systems",
"repo": "default",
"type": "github"
}
}
},
"root": "root",
"version": 7
}
flake.nix
+128
View File
@@ -0,0 +1,128 @@
{
description = "FIPS — a distributed, decentralized network routing protocol for mesh nodes connecting over arbitrary transports";
inputs = {
nixpkgs.url = "github:NixOS/nixpkgs/nixos-unstable";
flake-utils.url = "github:numtide/flake-utils";
fenix = {
url = "github:nix-community/fenix";
inputs.nixpkgs.follows = "nixpkgs";
};
};
outputs =
{
self,
nixpkgs,
flake-utils,
fenix,
}:
flake-utils.lib.eachDefaultSystem (
system:
let
pkgs = import nixpkgs { inherit system; };
# Honor the toolchain the repo pins in rust-toolchain.toml
# (channel 1.94.1 + rustfmt, clippy) so Nix builds match CI and the
# AUR/Debian packaging exactly, including the edition-2024 frontend.
rustToolchain = fenix.packages.${system}.fromToolchainFile {
file = ./rust-toolchain.toml;
sha256 = "sha256-zC8E38iDVJ1oPIzCqTk/Ujo9+9kx9dXq7wAwPMpkpg0=";
};
rustPlatform = pkgs.makeRustPlatform {
cargo = rustToolchain;
rustc = rustToolchain;
};
cargoToml = pkgs.lib.importTOML ./Cargo.toml;
# libdbus-sys (pulled in transitively by `bluer`, Linux/glibc only)
# runs `bindgen` against the system D-Bus headers at build time.
nativeBuildInputs = [
pkgs.pkg-config
rustPlatform.bindgenHook # sets LIBCLANG_PATH + clang for bindgen
];
buildInputs = pkgs.lib.optionals pkgs.stdenv.isLinux [
pkgs.dbus # libdbus-1.so.3, linked via bluer→libdbus-sys
pkgs.stdenv.cc.cc.lib # libgcc_s.so.1, needed by every Rust binary
];
fips = rustPlatform.buildRustPackage {
pname = "fips";
version = cargoToml.package.version;
src = pkgs.lib.cleanSourceWith {
src = ./.;
# Drop the build dir and the usual editor/VCS noise so the source
# hash is stable and unrelated edits don't trigger rebuilds.
filter =
path: type:
(pkgs.lib.cleanSourceFilter path type) && (baseNameOf path != "target");
};
cargoLock.lockFile = ./Cargo.lock;
inherit buildInputs;
# autoPatchelfHook rewrites the RPATH of the built binaries so the
# daemon finds libdbus-1.so.3 (linked via bluer→libdbus-sys) in the
# Nix store at runtime — without it the `fips` binary fails to load
# on NixOS where there is no global /usr/lib.
nativeBuildInputs =
nativeBuildInputs ++ pkgs.lib.optionals pkgs.stdenv.isLinux [ pkgs.autoPatchelfHook ];
# The test suite exercises TUN devices, raw sockets and mDNS, none of
# which exist in the build sandbox. The AUR/Debian packaging likewise
# ships the release binaries without running the integration tests
# here, so keep the package build hermetic and skip them.
doCheck = false;
meta = {
description = cargoToml.package.description;
homepage = cargoToml.package.homepage;
license = pkgs.lib.licenses.mit;
mainProgram = "fips";
platforms = pkgs.lib.platforms.linux ++ pkgs.lib.platforms.darwin;
};
};
mkApp = name: {
type = "app";
program = "${fips}/bin/${name}";
meta.description = "Run the ${name} binary from the FIPS package";
};
in
{
packages = {
default = fips;
fips = fips;
};
apps = {
default = mkApp "fips";
fips = mkApp "fips";
fipsctl = mkApp "fipsctl";
fips-gateway = mkApp "fips-gateway";
fipstop = mkApp "fipstop";
};
# `nix flake check` builds the package (and thus validates the flake on
# the current system).
checks.fips = fips;
devShells.default = pkgs.mkShell {
inherit buildInputs;
nativeBuildInputs = nativeBuildInputs ++ [
rustToolchain
pkgs.cargo-edit
];
# Point rust-analyzer at the matching std sources.
RUST_SRC_PATH = "${rustToolchain}/lib/rustlib/src/rust/library";
};
formatter = pkgs.nixfmt;
}
);
}
packaging/Makefile
+6 -2
View File
@@ -6,7 +6,8 @@
# Usage:
# make deb Build a Debian/Ubuntu .deb package
# make tarball Build a systemd install tarball
# make ipk Build an OpenWrt .ipk package
# make ipk Build an OpenWrt .ipk package (opkg, OpenWrt 24.x and earlier)
# make apk Build an OpenWrt .apk package (apk-tools, mandatory on OpenWrt 25+)
# make aur Build fips-git AUR package and validate with namcap
# make pkg Build a macOS .pkg installer
# make zip Build a Windows .zip package
@@ -17,7 +18,7 @@ SHELL := /bin/bash
PACKAGING_DIR := $(dir $(abspath $(lastword $(MAKEFILE_LIST))))
PROJECT_ROOT := $(abspath $(PACKAGING_DIR)/..)
.PHONY: all deb tarball ipk aur pkg zip clean
.PHONY: all deb tarball ipk apk aur pkg zip clean
all: deb tarball
@@ -30,6 +31,9 @@ tarball:
ipk:
@bash $(PACKAGING_DIR)/openwrt-ipk/build-ipk.sh
apk:
@bash $(PACKAGING_DIR)/openwrt-apk/build-apk.sh
aur:
@bash $(PACKAGING_DIR)/aur/build-aur.sh
packaging/README.md
+49 -5
View File
@@ -8,7 +8,8 @@ All build outputs go to `deploy/` at the project root.
```sh
make deb # Debian/Ubuntu .deb
make tarball # systemd install tarball
make ipk # OpenWrt .ipk
make ipk # OpenWrt .ipk (opkg, OpenWrt 24.x and earlier)
make apk # OpenWrt .apk (apk-tools, mandatory on OpenWrt 25+)
make aur # Arch Linux AUR package (fips-git, local build + namcap)
make pkg # macOS .pkg installer
make zip # Windows .zip package
@@ -46,7 +47,8 @@ packaging/
debian/ Debian/Ubuntu .deb packaging via cargo-deb
macos/ macOS .pkg installer via pkgbuild
systemd/ Generic Linux systemd tarball packaging
openwrt/ OpenWrt .ipk packaging via cargo-zigbuild
openwrt-ipk/ OpenWrt .ipk packaging via cargo-zigbuild (opkg)
openwrt-apk/ OpenWrt .apk packaging via cargo-zigbuild + apk mkpkg
windows/ Windows .zip package with service scripts
```
@@ -100,7 +102,7 @@ sudo ./fips-<version>-linux-<arch>/install.sh
See [systemd/README.install.md](systemd/README.install.md) for full
installation and configuration instructions.
### OpenWrt (`.ipk`)
### OpenWrt (`.ipk`, opkg — OpenWrt 24.x and earlier)
Cross-compiled with cargo-zigbuild and assembled as a standard `.ipk`
archive. Supports aarch64, mipsel, mips, arm, and x86\_64 targets.
@@ -110,12 +112,33 @@ archive. Supports aarch64, mipsel, mips, arm, and x86\_64 targets.
make ipk
# Build for a specific architecture
bash packaging/openwrt/build-ipk.sh --arch mipsel
bash packaging/openwrt-ipk/build-ipk.sh --arch mipsel
```
See [openwrt/README.md](openwrt/README.md) for router-specific
See [openwrt-ipk/README.md](openwrt-ipk/README.md) for router-specific
installation instructions.
### OpenWrt (`.apk`, apk-tools — mandatory on OpenWrt 25+)
OpenWrt 25 makes apk-tools the mandatory package manager (it is opt-in on
24.10). Same SDK-free approach
(cargo-zigbuild), but the `.apk` container is assembled by `apk mkpkg`
rather than hand-rolled, so the build additionally needs an apk-tools v3
`apk` binary built from source. The installed-filesystem payload is shared
with the `.ipk` package.
```sh
# Build (default: aarch64; also x86_64)
make apk
# Build for a specific architecture
bash packaging/openwrt-apk/build-apk.sh --arch x86_64
```
Packages are unsigned; install with `apk add --allow-untrusted`. See
[openwrt-apk/README.md](openwrt-apk/README.md) for building apk-tools and
router-specific installation.
### macOS (`.pkg`)
Built with `pkgbuild` (included with Xcode command-line tools). Installs
@@ -177,6 +200,27 @@ yay -S fips # release build from latest tag
See [aur/README.md](aur/README.md) for AUR publication instructions
and maintainer guide.
### Nix / NixOS (flake)
A [flake](../flake.nix) at the project root builds all four binaries
(`fips`, `fipsctl`, `fips-gateway`, `fipstop`) from source. It pins the
exact toolchain from `rust-toolchain.toml` via
[fenix](https://github.com/nix-community/fenix) and wires up the
build-time native dependencies (`libclang` for `bindgen`, plus `dbus`
and `pkg-config` for BLE), so it needs no system setup beyond Nix with
flakes enabled.
```sh
nix build .#fips # build the package (all four binaries)
nix run .#fips -- --help # run a binary directly
nix run .#fipsctl -- status
nix develop # dev shell with the pinned toolchain + cargo-edit
nix flake check # build + validate the flake
```
Add to a NixOS configuration via the flake's `packages.<system>.fips`
output, e.g. `environment.systemPackages = [ fips.packages.${system}.default ];`.
## Shared Assets
`common/` contains assets used across packaging formats:
packaging/openwrt-apk/README.md
+98
View File
@@ -0,0 +1,98 @@
# FIPS OpenWrt Package (apk)
Builds a FIPS `.apk` for **OpenWrt 25+**, where apk-tools is the mandatory
package manager. apk is also available opt-in on **24.10** (where opkg remains
the default). For OpenWrt 24.x and earlier, the `.ipk` package in
[`../openwrt-ipk/`](../openwrt-ipk/) still works.
Like the `.ipk` build, this is **SDK-free**: it cross-compiles with
`cargo-zigbuild` and assembles the package directly — no OpenWrt SDK image. The
`.ipk` format is a plain tar.gz we can hand-roll, but the `.apk` (apk-tools v3
ADB) container is not, so we drive the official `apk mkpkg` applet — the same
tool OpenWrt's own [`include/package-pack.mk`](https://github.com/openwrt/openwrt/blob/main/include/package-pack.mk)
calls. The only extra requirement over the `.ipk` build is the `apk` binary.
## Layout
| File | Purpose |
|---|---|
| `build-apk.sh` | Cross-compile + assemble the `.apk` via `apk mkpkg` |
| `apk-version.sh` | Map a release tag / commit height to an apk-tools-valid version |
| `apk-version.test.sh` | Case-table test for `apk-version.sh` (`sh apk-version.test.sh`) |
The installed-filesystem payload (init scripts, `fips.yaml`, sysctl drop-ins,
hotplug, uci-defaults, …) is **shared** with the `.ipk` package — there is one
canonical copy in [`../openwrt-ipk/files/`](../openwrt-ipk/files/). `build-apk.sh`
stages from there, so the two packages always ship the same files. Keep the
staging block in `build-apk.sh` in sync with `../openwrt-ipk/build-ipk.sh`.
## Versioning
apk-tools enforces a strict version grammar
(`<digit>(.<digit>)*(_<suffix><digit>*)*(-r<N>)`). `apk-version.sh` builds a
valid version from structured inputs rather than rewriting an already-flattened
string:
| Input | apk version |
|---|---|
| `tag v1.2.3` | `1.2.3-r0` |
| `tag v1.2.3-rc1` | `1.2.3_rc1-r0` |
| `dev 1234` (commit height) | `0.0.0_git1234-r0` |
The human-readable version (`v1.2.3`, `master.123.abcdef0`) is still used for the
artifact filename; only the metadata embedded in the package is normalized.
## Building
### Prerequisites
| Requirement | Notes |
|---|---|
| `cargo install cargo-zigbuild` + `zig` | Rust musl cross-compilation (as for `.ipk`) |
| apk-tools v3 `apk` binary | Provides `apk mkpkg`; not packaged for most distros — build from source |
| `fakeroot` | Optional; makes packaged files root-owned on an unprivileged build host |
apk-tools is not in Debian/Ubuntu repos, so build the pinned release from source.
Pin the same commit the targeted OpenWrt release ships (see
`package/system/apk/Makefile` upstream) so the `.apk` is readable by the device's
`apk`. CI builds **3.0.5** (`b5a31c0d…`):
```bash
sudo apt-get install -y build-essential meson ninja-build pkg-config \
zlib1g-dev libssl-dev libzstd-dev liblzma-dev lua5.4-dev scdoc
git clone https://gitlab.alpinelinux.org/alpine/apk-tools.git
cd apk-tools && git checkout b5a31c0d865342ad80be10d68f1bb3d3ad9b0866
meson setup build && ninja -C build src/apk
export APK_BIN="$PWD/build/src/apk"
```
### Build the package
```bash
# from the repo root
./packaging/openwrt-apk/build-apk.sh --arch aarch64 # or x86_64, mipsel, mips, arm
```
Output: `dist/fips_<version>_<openwrt-arch>.apk`. Override the version with
`PKG_VERSION` (filename) and `APK_VERSION` (embedded metadata); otherwise both are
derived from git.
## Installing on the router
Packages are **unsigned** (the same posture as our `.ipk`), so install with
`--allow-untrusted`:
```bash
scp -O dist/fips_<version>_<arch>.apk root@192.168.1.1:/tmp/
ssh root@192.168.1.1 apk add --allow-untrusted /tmp/fips_<version>_<arch>.apk
```
On OpenWrt 25.x, installing from a *signed repository* requires the publisher's
key; a single `--allow-untrusted` package install does not. If we ever publish an
apk feed, add ECDSA (prime256v1) signing via `apk mkpkg --sign` and distribute the
public key to `/etc/apk/keys/`.
`/etc/fips/fips.yaml` is marked as a config file (via
`/lib/apk/packages/fips.conffiles`), so apk preserves local edits across upgrades,
and `/lib/upgrade/keep.d/fips` preserves `/etc/fips/` across `sysupgrade` — the
same guarantees as the `.ipk` package.
packaging/openwrt-apk/apk-version.sh
Executable
+74
View File
@@ -0,0 +1,74 @@
#!/bin/sh
# Emit an apk-tools-compatible version string for FIPS.
#
# apk-tools enforces a strict version grammar:
# <digit>(.<digit>)*(_<suffix><digit>*)*(-r<N>)
# where <suffix> is a recognised pre-release/post-release token
# (alpha, beta, pre, rc, cvs, svn, git, hg, p).
#
# Unlike a regex rewrite of an already-flattened version string, this
# helper builds the apk version directly from the *structured* inputs the
# caller already has (a release tag, or a commit height). There is no
# parsing-back-out of a "branch.height.hash" blob, so there is no fragile
# reparse step to get wrong.
#
# Usage:
# apk-version.sh tag <git-tag> # e.g. v1.2.3, v1.2.3-rc1
# apk-version.sh dev <height> # e.g. 1234 (git rev-list --count HEAD)
# apk-version.sh auto # derive from the current git checkout
#
# Examples:
# apk-version.sh tag v1.2.3 -> 1.2.3-r0
# apk-version.sh tag v1.2.3-rc1 -> 1.2.3_rc1-r0
# apk-version.sh dev 1234 -> 0.0.0_git1234-r0
set -eu
mode="${1:-auto}"
case "$mode" in
tag) raw_tag="${2:?tag mode requires a tag argument}"; height="" ;;
dev) raw_tag=""; height="${2:?dev mode requires a height argument}" ;;
auto)
if raw_tag="$(git describe --exact-match --tags 2>/dev/null)"; then
height=""
else
raw_tag=""
height="$(git rev-list --count HEAD 2>/dev/null || echo 0)"
fi
;;
*)
echo "usage: $0 [auto | tag <git-tag> | dev <height>]" >&2
exit 2
;;
esac
if [ -n "$raw_tag" ]; then
# Release tag: vX.Y.Z or vX.Y.Z-<pre>. Strip the leading 'v', split the
# core (X.Y.Z) from the pre-release token, and map our hyphen separator
# to apk's '_' pre-release marker.
body="${raw_tag#v}"
core="${body%%-*}"
case "$body" in
*-*) pre="${body#*-}" ;;
*) pre="" ;;
esac
case "$pre" in
"") suffix="" ;;
alpha*|beta*|pre*|rc*) suffix="_${pre}" ;;
*)
# Unknown pre-release token: apk would reject or misorder it, so
# drop it rather than emit an invalid version. The human-readable
# PACKAGE_VERSION (the raw tag) is still used for the filename.
suffix=""
;;
esac
printf '%s%s-r0\n' "$core" "$suffix"
else
# Untagged build: no meaningful semver, so anchor at 0.0.0 and encode the
# monotonic commit height as a _git pre-release component. This keeps apk's
# ordering sane across dev builds without smuggling the hash/branch into a
# field that cannot represent them.
printf '0.0.0_git%s-r0\n' "${height:-0}"
fi
packaging/openwrt-apk/apk-version.test.sh
Executable
+45
View File
@@ -0,0 +1,45 @@
#!/bin/sh
# Case-table test for apk-version.sh. Run: sh apk-version.test.sh
set -eu
HERE="$(cd "$(dirname "$0")" && pwd)"
SUT="$HERE/apk-version.sh"
fail=0
check() {
# check <expected> <args...>
expected="$1"; shift
actual="$(sh "$SUT" "$@")"
if [ "$actual" = "$expected" ]; then
printf ' PASS %-22s -> %s\n' "$*" "$actual"
else
printf ' FAIL %-22s -> %s (expected %s)\n' "$*" "$actual" "$expected"
fail=1
fi
}
echo "== apk-version.sh =="
# Plain release tags.
check "1.2.3-r0" tag v1.2.3
check "0.4.0-r0" tag v0.4.0
check "10.20.30-r0" tag v10.20.30
# Pre-release tags: hyphen separator becomes apk's '_' marker.
check "1.2.3_rc1-r0" tag v1.2.3-rc1
check "1.2.3_alpha1-r0" tag v1.2.3-alpha1
check "1.2.3_beta2-r0" tag v1.2.3-beta2
check "1.2.3_pre1-r0" tag v1.2.3-pre1
# Unknown pre-release token is dropped (apk cannot represent it).
check "1.2.3-r0" tag v1.2.3-weird9
# Dev builds: monotonic commit height as a _git component.
check "0.0.0_git1234-r0" dev 1234
check "0.0.0_git0-r0" dev 0
if [ "$fail" -ne 0 ]; then
echo "FAILED"
exit 1
fi
echo "OK"
packaging/openwrt-apk/build-apk.sh
Executable
+296
View File
@@ -0,0 +1,296 @@
#!/bin/bash
# Build a FIPS .apk package for OpenWrt without the OpenWrt SDK.
#
# apk-tools (.apk) is the mandatory package manager from OpenWrt 25 onward; it
# is also available opt-in on 24.10, where opkg (.ipk) remains the default. The
# .ipk package in ../openwrt-ipk/ still covers OpenWrt 24.x and earlier; this
# .apk package is what you need on 25+. Unlike the .ipk format (a plain tar.gz
# of tarballs that we
# assemble by hand in ../openwrt-ipk/build-ipk.sh), the .apk container is the
# apk-tools v3 ADB format, which is impractical to hand-roll. Instead we drive
# the official `apk mkpkg` applet — the same tool OpenWrt's build system calls
# in include/package-pack.mk — so no SDK is required, only the `apk` binary.
#
# Usage:
# ./packaging/openwrt-apk/build-apk.sh [--arch <name>]
#
# Architectures (--arch): aarch64 [default], x86_64, mipsel, mips, arm
# (the apk CI matrix ships aarch64 + x86_64; the rest are buildable locally).
#
# Output: dist/fips_<version>_<openwrt-arch>.apk
#
# Prerequisites:
# cargo install cargo-zigbuild (Rust musl cross-compilation)
# apk-tools v3 `apk` binary on PATH, or pointed at via APK_BIN=/path/to/apk
# (build from source — see README.md; CI builds apk-tools 3.0.5).
# fakeroot (optional but recommended; makes packaged files root-owned).
#
# Install on a router (packages are unsigned, like our .ipk):
# scp -O dist/fips_<version>_<arch>.apk root@192.168.1.1:/tmp/
# ssh root@192.168.1.1 apk add --allow-untrusted /tmp/fips_<version>_<arch>.apk
set -euo pipefail
# ---------------------------------------------------------------------------
# Arguments
# ---------------------------------------------------------------------------
ARCH="aarch64"
BIN_DIR="" # if set, use prebuilt binaries from here instead of compiling
while [[ $# -gt 0 ]]; do
case "$1" in
--arch) ARCH="$2"; shift 2 ;;
--arch=*) ARCH="${1#*=}"; shift ;;
--bin-dir) BIN_DIR="$2"; shift 2 ;;
--bin-dir=*) BIN_DIR="${1#*=}"; shift ;;
*) echo "Unknown argument: $1" >&2; exit 1 ;;
esac
done
# ---------------------------------------------------------------------------
# Architecture mapping
#
# RUST_TARGET — passed to cargo --target
# OPENWRT_ARCH — apk "arch:" field and the package filename
#
# Kept in sync with ../openwrt-ipk/build-ipk.sh (same target table).
# ---------------------------------------------------------------------------
case "$ARCH" in
aarch64)
RUST_TARGET="aarch64-unknown-linux-musl"
OPENWRT_ARCH="aarch64_cortex-a53"
;;
mipsel)
RUST_TARGET="mipsel-unknown-linux-musl"
OPENWRT_ARCH="mipsel_24kc"
;;
mips)
RUST_TARGET="mips-unknown-linux-musl"
OPENWRT_ARCH="mips_24kc"
;;
arm)
RUST_TARGET="arm-unknown-linux-musleabihf"
OPENWRT_ARCH="arm_cortex-a7"
;;
x86_64)
RUST_TARGET="x86_64-unknown-linux-musl"
OPENWRT_ARCH="x86_64"
;;
*)
echo "Unknown arch: $ARCH" >&2
echo "Valid: aarch64, mipsel, mips, arm, x86_64" >&2
exit 1
;;
esac
# ---------------------------------------------------------------------------
# Paths
# ---------------------------------------------------------------------------
SCRIPT_DIR="$(cd "$(dirname "$0")" && pwd)"
PROJECT_ROOT="$(cd "$SCRIPT_DIR/../.." && pwd)"
# The installed-filesystem payload (init scripts, config, sysctl, etc.) is
# shared with the .ipk package; there is one canonical copy in openwrt-ipk/.
FILES_DIR="$PROJECT_ROOT/packaging/openwrt-ipk/files"
DIST_DIR="$PROJECT_ROOT/dist"
PKG_NAME="fips"
# Human-readable version for the filename (e.g. v0.4.0 or master.123.abcdef0),
# mirroring the .ipk artifacts and the CI/NIP-94 plumbing.
PKG_VERSION="${PKG_VERSION:-$(cd "$PROJECT_ROOT" && git describe --tags --always --dirty 2>/dev/null || echo "0.1.0")}"
# apk-tools-compatible version embedded inside the package metadata.
APK_VERSION="${APK_VERSION:-$(cd "$PROJECT_ROOT" && sh "$SCRIPT_DIR/apk-version.sh" auto)}"
APK_BIN="${APK_BIN:-apk}"
if ! command -v "$APK_BIN" >/dev/null 2>&1; then
echo "Error: apk-tools binary not found (looked for '$APK_BIN')." >&2
echo " Build apk-tools v3 from source or set APK_BIN=/path/to/apk." >&2
echo " See packaging/openwrt-apk/README.md." >&2
exit 1
fi
echo "==> Building $PKG_NAME $PKG_VERSION (apk version $APK_VERSION) for $OPENWRT_ARCH ($RUST_TARGET)"
# ---------------------------------------------------------------------------
# 1. Obtain binaries
#
# Either use a directory of prebuilt binaries (--bin-dir; CI cross-compiles
# once in a shared job and hands them to both the .ipk and .apk packagers), or
# compile from source here for a self-contained local build.
# ---------------------------------------------------------------------------
if [ -n "$BIN_DIR" ]; then
RELEASE_DIR="$BIN_DIR"
echo "==> Using prebuilt binaries from $RELEASE_DIR"
for bin in fips fipsctl fipstop fips-gateway; do
[ -f "$RELEASE_DIR/$bin" ] || {
echo "Error: prebuilt binary not found: $RELEASE_DIR/$bin" >&2
exit 1
}
done
else
if ! command -v cargo-zigbuild &>/dev/null; then
echo "Error: cargo-zigbuild not found." >&2
echo " Install: cargo install cargo-zigbuild" >&2
exit 1
fi
if ! rustup target list --installed | grep -q "^$RUST_TARGET$"; then
echo "==> Adding Rust target $RUST_TARGET..."
rustup target add "$RUST_TARGET"
fi
echo "==> Compiling..."
cd "$PROJECT_ROOT"
cargo zigbuild \
--release \
--target "$RUST_TARGET" \
--bin fips \
--bin fipsctl \
--bin fipstop \
--bin fips-gateway
RELEASE_DIR="$PROJECT_ROOT/target/$RUST_TARGET/release"
echo "==> Stripping binaries..."
STRIP="${LLVM_STRIP:-strip}"
for bin in fips fipsctl fipstop fips-gateway; do
"$STRIP" "$RELEASE_DIR/$bin" 2>/dev/null || true
done
fi
SIZE=$(du -sh "$RELEASE_DIR/fips" | cut -f1)
echo " fips: $SIZE"
# ---------------------------------------------------------------------------
# 2. Stage the installed filesystem tree (--files root for apk mkpkg)
# ---------------------------------------------------------------------------
# This block is the same payload as ../openwrt-ipk/build-ipk.sh; keep the two
# in sync. The CI apk structural check asserts every path below is present.
WORK_DIR="$(mktemp -d)"
trap 'rm -rf "$WORK_DIR"' EXIT
STAGE_DIR="$WORK_DIR/root" # becomes the package's filesystem
SCRIPTS_DIR="$WORK_DIR/scripts" # maintainer scripts (metadata, not payload)
mkdir -p "$STAGE_DIR" "$SCRIPTS_DIR"
install -d "$STAGE_DIR/usr/bin"
install -m 0755 "$RELEASE_DIR/fips" "$STAGE_DIR/usr/bin/fips"
install -m 0755 "$RELEASE_DIR/fipsctl" "$STAGE_DIR/usr/bin/fipsctl"
install -m 0755 "$RELEASE_DIR/fipstop" "$STAGE_DIR/usr/bin/fipstop"
install -m 0755 "$RELEASE_DIR/fips-gateway" "$STAGE_DIR/usr/bin/fips-gateway"
install -d "$STAGE_DIR/etc/init.d"
install -m 0755 "$FILES_DIR/etc/init.d/fips" "$STAGE_DIR/etc/init.d/fips"
install -m 0755 "$FILES_DIR/etc/init.d/fips-gateway" "$STAGE_DIR/etc/init.d/fips-gateway"
install -d "$STAGE_DIR/etc/fips"
install -m 0600 "$FILES_DIR/etc/fips/fips.yaml" "$STAGE_DIR/etc/fips/fips.yaml"
install -m 0755 "$FILES_DIR/etc/fips/firewall.sh" "$STAGE_DIR/etc/fips/firewall.sh"
# The shared fips.yaml ships ethernet.wan.interface: "eth0", the OpenWrt 24
# default. This .apk package targets OpenWrt 25+ (DSA), where the WAN port is
# named "wan", so ship "wan" as the default. Patching the staged copy keeps the
# as-installed config correct for the platform without maintaining a second copy
# of the file; operators can still edit /etc/fips/fips.yaml for non-standard boards.
sed -i 's|interface: "eth0"|interface: "wan"|' "$STAGE_DIR/etc/fips/fips.yaml"
install -d "$STAGE_DIR/etc/dnsmasq.d"
install -m 0644 "$FILES_DIR/etc/dnsmasq.d/fips.conf" "$STAGE_DIR/etc/dnsmasq.d/fips.conf"
install -d "$STAGE_DIR/etc/sysctl.d"
install -m 0644 "$FILES_DIR/etc/sysctl.d/fips-bridge.conf" "$STAGE_DIR/etc/sysctl.d/fips-bridge.conf"
install -m 0644 "$FILES_DIR/etc/sysctl.d/fips-gateway.conf" "$STAGE_DIR/etc/sysctl.d/fips-gateway.conf"
install -d "$STAGE_DIR/etc/hotplug.d/net"
install -m 0755 "$FILES_DIR/etc/hotplug.d/net/99-fips" "$STAGE_DIR/etc/hotplug.d/net/99-fips"
install -d "$STAGE_DIR/etc/uci-defaults"
install -m 0755 "$FILES_DIR/etc/uci-defaults/90-fips-setup" "$STAGE_DIR/etc/uci-defaults/90-fips-setup"
install -d "$STAGE_DIR/lib/upgrade/keep.d"
install -m 0644 "$FILES_DIR/lib/upgrade/keep.d/fips" "$STAGE_DIR/lib/upgrade/keep.d/fips"
# ---- conffiles ----
# apk mkpkg discovers config files from /lib/apk/packages/<name>.conffiles
# inside the --files tree (same mechanism OpenWrt's package-pack.mk uses).
# Listing fips.yaml here makes apk preserve user edits across upgrades, the
# apk equivalent of opkg's conffiles handling.
install -d "$STAGE_DIR/lib/apk/packages"
cat > "$STAGE_DIR/lib/apk/packages/${PKG_NAME}.conffiles" <<'EOF'
/etc/fips/fips.yaml
EOF
# ---- maintainer scripts ----
# Map our opkg maintainer scripts onto apk's lifecycle phases:
# opkg postinst -> apk post-install (enable + start services)
# opkg prerm -> apk pre-deinstall (stop + disable services)
cat > "$SCRIPTS_DIR/post-install" <<'EOF'
#!/bin/sh
# Run first-boot UCI setup (the script deletes itself when done).
if [ -x /etc/uci-defaults/90-fips-setup ]; then
/etc/uci-defaults/90-fips-setup && rm -f /etc/uci-defaults/90-fips-setup
fi
/etc/init.d/fips enable
/etc/init.d/fips start
/etc/init.d/fips-gateway enable
/etc/init.d/fips-gateway start
exit 0
EOF
cat > "$SCRIPTS_DIR/pre-deinstall" <<'EOF'
#!/bin/sh
/etc/init.d/fips-gateway stop 2>/dev/null || true
/etc/init.d/fips-gateway disable 2>/dev/null || true
/etc/init.d/fips stop 2>/dev/null || true
/etc/init.d/fips disable 2>/dev/null || true
exit 0
EOF
chmod 0755 "$SCRIPTS_DIR/post-install" "$SCRIPTS_DIR/pre-deinstall"
# ---------------------------------------------------------------------------
# 3. Assemble the .apk via apk mkpkg
# ---------------------------------------------------------------------------
# fakeroot makes the packaged files root-owned even though CI runs unprivileged.
DESCRIPTION="FIPS Mesh Network Daemon. Distributed, decentralized mesh networking over UDP, TCP, and raw Ethernet, with a TUN interface (fips0), ULA IPv6 addressing, and a .fips DNS responder."
DEPENDS="kmod-tun kmod-br-netfilter kmod-nft-nat kmod-nf-conntrack ip-full"
PKG_FILENAME="${PKG_NAME}_${PKG_VERSION}_${OPENWRT_ARCH}.apk"
mkdir -p "$DIST_DIR"
FAKEROOT=""
if command -v fakeroot >/dev/null 2>&1; then
FAKEROOT="fakeroot"
else
echo "Warning: fakeroot not found — packaged files will be owned by the build user." >&2
fi
$FAKEROOT "$APK_BIN" mkpkg \
--info "name:$PKG_NAME" \
--info "version:$APK_VERSION" \
--info "description:$DESCRIPTION" \
--info "arch:$OPENWRT_ARCH" \
--info "license:MIT" \
--info "origin:$PKG_NAME" \
--info "url:https://github.com/jmcorgan/fips" \
--info "maintainer:FIPS Network" \
--info "depends:$DEPENDS" \
--script "post-install:$SCRIPTS_DIR/post-install" \
--script "pre-deinstall:$SCRIPTS_DIR/pre-deinstall" \
--files "$STAGE_DIR" \
--output "$DIST_DIR/$PKG_FILENAME"
echo ""
echo "==> Done: dist/$PKG_FILENAME"
echo " $(du -sh "$DIST_DIR/$PKG_FILENAME" | cut -f1)"
echo ""
echo "Install on router (OpenWrt 25+, or 24.10 with apk enabled):"
echo " scp -O dist/$PKG_FILENAME root@192.168.1.1:/tmp/"
echo " ssh root@192.168.1.1 apk add --allow-untrusted /tmp/$PKG_FILENAME"
packaging/openwrt-ipk/README.md
+4 -1
View File
@@ -132,7 +132,10 @@ The default config enables:
For Ethernet transport, uncomment the `ethernet:` section and set the correct
physical interface names for your router. **Always use physical port names
(`eth0`, `eth1`), never bridge names (`br-lan`).** See
(`eth0`, `eth1`, or DSA port names like `wan`/`lan1`), never bridge names
(`br-lan`).** The shipped default WAN port is `eth0` (OpenWrt 24); on OpenWrt
25 (DSA) boards the WAN port is named `wan` — the `.apk` package ships that
default. Run `ip link show` to confirm the names on your board. See
[`deploy/native/README.md`](../../deploy/native/README.md) for details.
## Service management
packaging/openwrt-ipk/build-ipk.sh
+47 -33
View File
@@ -27,11 +27,14 @@ set -euo pipefail
# ---------------------------------------------------------------------------
ARCH="aarch64"
BIN_DIR="" # if set, use prebuilt binaries from here instead of compiling
while [[ $# -gt 0 ]]; do
case "$1" in
--arch) ARCH="$2"; shift 2 ;;
--arch=*) ARCH="${1#*=}"; shift ;;
--bin-dir) BIN_DIR="$2"; shift 2 ;;
--bin-dir=*) BIN_DIR="${1#*=}"; shift ;;
*) echo "Unknown argument: $1" >&2; exit 1 ;;
esac
done
@@ -86,44 +89,55 @@ PKG_VERSION="${PKG_VERSION:-$(cd "$PROJECT_ROOT" && git describe --tags --always
echo "==> Building $PKG_NAME $PKG_VERSION for $OPENWRT_ARCH ($RUST_TARGET)"
# ---------------------------------------------------------------------------
# Prerequisites
# 1. Obtain binaries
#
# Either use a directory of prebuilt binaries (--bin-dir; CI cross-compiles
# once in a shared job and hands them to both the .ipk and .apk packagers), or
# compile from source here for a self-contained local build.
# ---------------------------------------------------------------------------
if ! command -v cargo-zigbuild &>/dev/null; then
echo "Error: cargo-zigbuild not found." >&2
echo " Install: cargo install cargo-zigbuild" >&2
exit 1
if [ -n "$BIN_DIR" ]; then
RELEASE_DIR="$BIN_DIR"
echo "==> Using prebuilt binaries from $RELEASE_DIR"
for bin in fips fipsctl fipstop fips-gateway; do
[ -f "$RELEASE_DIR/$bin" ] || {
echo "Error: prebuilt binary not found: $RELEASE_DIR/$bin" >&2
exit 1
}
done
else
if ! command -v cargo-zigbuild &>/dev/null; then
echo "Error: cargo-zigbuild not found." >&2
echo " Install: cargo install cargo-zigbuild" >&2
exit 1
fi
if ! rustup target list --installed | grep -q "^$RUST_TARGET$"; then
echo "==> Adding Rust target $RUST_TARGET..."
rustup target add "$RUST_TARGET"
fi
echo "==> Compiling..."
cd "$PROJECT_ROOT"
cargo zigbuild \
--release \
--target "$RUST_TARGET" \
--bin fips \
--bin fipsctl \
--bin fipstop \
--bin fips-gateway
RELEASE_DIR="$PROJECT_ROOT/target/$RUST_TARGET/release"
echo "==> Stripping binaries..."
STRIP="${LLVM_STRIP:-strip}"
for bin in fips fipsctl fipstop fips-gateway; do
"$STRIP" "$RELEASE_DIR/$bin" 2>/dev/null || true
done
fi
if ! rustup target list --installed | grep -q "^$RUST_TARGET$"; then
echo "==> Adding Rust target $RUST_TARGET..."
rustup target add "$RUST_TARGET"
fi
# ---------------------------------------------------------------------------
# 1. Build
# ---------------------------------------------------------------------------
echo "==> Compiling..."
cd "$PROJECT_ROOT"
cargo zigbuild \
--release \
--target "$RUST_TARGET" \
--bin fips \
--bin fipsctl \
--bin fipstop \
--bin fips-gateway
RELEASE_DIR="$PROJECT_ROOT/target/$RUST_TARGET/release"
echo "==> Stripping binaries..."
STRIP="${LLVM_STRIP:-strip}"
for bin in fips fipsctl fipstop fips-gateway; do
"$STRIP" "$RELEASE_DIR/$bin" 2>/dev/null || true
done
SIZE=$(du -sh "$RELEASE_DIR/fips" | cut -f1)
echo " fips: $SIZE after strip"
echo " fips: $SIZE"
# ---------------------------------------------------------------------------
# 2. Assemble .ipk
packaging/openwrt-ipk/files/etc/init.d/fips
+11
View File
@@ -12,6 +12,17 @@ start_service() {
# Ensure TUN module is loaded before starting the daemon.
modprobe tun 2>/dev/null || true
# Pre-create the control-socket runtime directory so the daemon binds the
# canonical /run/fips/control.sock instead of falling back to /tmp. This is
# the procd equivalent of the systemd unit's RuntimeDirectory=fips (and the
# fips.tmpfiles "d /run/fips 0750 root fips" entry); OpenWrt was the only
# platform missing it. Without it, fips-gateway — which creates /run/fips
# for its own gateway.sock — makes fipsctl/fipstop resolve a control socket
# under /run/fips that the daemon actually bound under /tmp.
mkdir -p /run/fips
chmod 0750 /run/fips
chgrp fips /run/fips 2>/dev/null || true
procd_open_instance
procd_set_param command "$PROG" --config "$CONFIG"
# Respawn: restart after 5 s, give up after 5 consecutive failures within
src/bin/fipstop/ui/routing.rs
+131 -38
View File
@@ -83,6 +83,35 @@ fn fwd_value(data: &serde_json::Value, pkt_key: &str, byte_key: &str) -> String
format!("{} pkts ({})", pkts, helpers::format_bytes(bytes))
}
/// Read a raw forwarding counter as a u64 (0 if missing), for arithmetic
/// (percentages, derived totals) that the string-returning helpers can't do.
fn fwd_count(data: &serde_json::Value, key: &str) -> u64 {
data.get("forwarding")
.and_then(|f| f.get(key))
.and_then(|v| v.as_u64())
.unwrap_or(0)
}
/// Total mesh egress = locally-originated + transit-forwarded, formatted as
/// "N pkts (B)". There is no single daemon counter for everything this node
/// transmits to peers, so it is derived from its two contributors.
fn mesh_tx_value(data: &serde_json::Value) -> String {
let pkts = fwd_count(data, "originated_packets") + fwd_count(data, "forwarded_packets");
let bytes = fwd_count(data, "originated_bytes") + fwd_count(data, "forwarded_bytes");
format!("{} pkts ({})", pkts, helpers::format_bytes(bytes))
}
/// Format a route-class count as "N (xx.x%)" where the percentage is the class's
/// share of total forwarded (transit) packets. Zero forwarded yields "0.0%".
fn route_class_value(count: u64, total_forwarded: u64) -> String {
let pct = if total_forwarded > 0 {
count as f64 / total_forwarded as f64 * 100.0
} else {
0.0
};
format!("{count} ({pct:.1}%)")
}
/// Build a section: a styled header line followed by the kv pairs rendered
/// through the group helper so the section's values share a left edge.
fn section(title: &str, pairs: &[(&str, String)]) -> Vec<Line<'static>> {
@@ -110,49 +139,39 @@ fn draw_routing_stats(
let err = |key: &str| helpers::nested_u64(data, "error_signals", key);
let cong = |key: &str| helpers::nested_u64(data, "congestion", key);
// Left column: Forwarding + Discovery. Each section's values share a left
// edge via the kv_lines group helper.
// The node is an interface adapter between the local host stack and the
// mesh; the left column reads each side as a Transmitted/Received pair.
//
// Local Stack — traffic crossing the TUN / local-origination boundary:
// Transmitted is what the host injects into the mesh (originated), Received
// is what the mesh hands up to the host (delivered).
let mut left = section(
"Forwarding",
"Local Stack",
&[
(
"Transmitted",
fwd_value(data, "originated_packets", "originated_bytes"),
),
(
"Received",
fwd_value(data, "delivered_packets", "delivered_bytes"),
),
],
);
left.push(Line::from(""));
// Mesh — traffic crossing the peer-link boundary: Transmitted is everything
// this node puts on the wire (originated + forwarded, derived), Received is
// the ingress aggregate from peers (own-delivered + transit + drops).
left.extend(section(
"Mesh",
&[
("Transmitted", mesh_tx_value(data)),
(
"Received",
fwd_value(data, "received_packets", "received_bytes"),
),
(
"Delivered",
fwd_value(data, "delivered_packets", "delivered_bytes"),
),
(
"Forwarded",
fwd_value(data, "forwarded_packets", "forwarded_bytes"),
),
(
"Originated",
fwd_value(data, "originated_packets", "originated_bytes"),
),
(
"Decode Error",
fwd_value(data, "decode_error_packets", "decode_error_bytes"),
),
(
"TTL Exhausted",
fwd_value(data, "ttl_exhausted_packets", "ttl_exhausted_bytes"),
),
(
"No Route",
fwd_value(data, "drop_no_route_packets", "drop_no_route_bytes"),
),
(
"MTU Exceeded",
fwd_value(data, "drop_mtu_exceeded_packets", "drop_mtu_exceeded_bytes"),
),
(
"Send Error",
fwd_value(data, "drop_send_error_packets", "drop_send_error_bytes"),
),
],
);
));
left.push(Line::from(""));
left.extend(section(
"Discovery Requests",
@@ -184,15 +203,89 @@ fn draw_routing_stats(
],
));
// Right column: Error Signals + Congestion
// Right column — "Forwarded" (transit / routed through this node).
// Forwarded total, then the route-class breakdown (a percentage partition
// of the total), then the transit-path drop reasons.
let fwd_total = fwd_count(data, "forwarded_packets");
let mut right = section(
"Forwarded",
&[(
"Forwarded",
fwd_value(data, "forwarded_packets", "forwarded_bytes"),
)],
);
// Blank separator after the Forwarded total, matching the spacing between
// every other section pair; the total and its route-class breakdown read
// as two distinct groups.
right.push(Line::from(""));
// Route-class breakdown: a partition of Forwarded, each line annotated with
// its share of the total. Tree-down cross — the dive-to-tree-child
// cut-through — is the last class; Tree-down + Tree-down cross sum to the
// pre-split tree-down total.
right.extend(section(
"Route Class",
&[
(
"Direct Peer",
route_class_value(fwd_count(data, "route_direct_peer"), fwd_total),
),
(
"Tree-down",
route_class_value(fwd_count(data, "route_tree_down"), fwd_total),
),
(
"Tree-up",
route_class_value(fwd_count(data, "route_tree_up"), fwd_total),
),
(
"Cross-link descend",
route_class_value(fwd_count(data, "route_crosslink_descend"), fwd_total),
),
(
"Cross-link ascend",
route_class_value(fwd_count(data, "route_crosslink_ascend"), fwd_total),
),
(
"Tree-down cross",
route_class_value(fwd_count(data, "route_tree_down_cross"), fwd_total),
),
],
));
right.push(Line::from(""));
right.extend(section(
"Dropped",
&[
(
"No Route",
fwd_value(data, "drop_no_route_packets", "drop_no_route_bytes"),
),
(
"TTL Exhausted",
fwd_value(data, "ttl_exhausted_packets", "ttl_exhausted_bytes"),
),
(
"Decode Error",
fwd_value(data, "decode_error_packets", "decode_error_bytes"),
),
(
"MTU Exceeded",
fwd_value(data, "drop_mtu_exceeded_packets", "drop_mtu_exceeded_bytes"),
),
(
"Send Error",
fwd_value(data, "drop_send_error_packets", "drop_send_error_bytes"),
),
],
));
right.push(Line::from(""));
right.extend(section(
"Error Signals",
&[
("Coords Required", err("coords_required")),
("Path Broken", err("path_broken")),
("MTU Exceeded", err("mtu_exceeded")),
],
);
));
right.push(Line::from(""));
right.extend(section(
"Congestion",
src/bin/fipstop/ui/snapshots.rs
+15 -1
View File
@@ -1134,7 +1134,12 @@ fn routing_focused_pane_scrolls() {
let mut app1 = app_with(Tab::Routing, data);
app1.data.insert(Tab::Cache, json!({}));
app1.focused_pane.insert(Tab::Routing, 2);
app1.scroll_offsets.insert((Tab::Routing, 2), 6);
// Congestion is the last section of the right ("Forwarded") column, below
// the route-class breakdown, Dropped, and Error Signals groups. The left
// column is the taller of the two, so scrolling fully to the bottom would
// over-scroll the right column past Congestion; this offset lands the
// Congestion region inside the short window instead.
app1.scroll_offsets.insert((Tab::Routing, 2), 24);
let buf1 = testkit::render(100, 20, |frame, area| {
super::routing::draw(frame, &app1, area);
});
@@ -1184,6 +1189,15 @@ fn mmp_focused_pane_indicator() {
});
// The focused Session MMP title is cyan; the unfocused Link MMP title is not.
assert_eq!(testkit::fg_at(&buf, "Session MMP"), Some(Color::Cyan));
// Presence before colour. `fg_at` is `find(..)?` mapped to the cell's fg, so
// it returns None for a title that was never drawn, and None != Some(Cyan) --
// meaning the assertion below passed when the Link MMP pane was missing
// entirely. Asserting it is on screen first is what makes the next line read
// "not highlighted" rather than "not there".
assert!(
testkit::find(&buf, "Link MMP").is_some(),
"the unfocused Link MMP title should still be rendered"
);
assert_ne!(testkit::fg_at(&buf, "Link MMP"), Some(Color::Cyan));
}
src/bin/fipstop/ui/tree.rs
-4
View File
@@ -174,10 +174,6 @@ fn draw_stats(frame: &mut Frame, data: &serde_json::Value, scroll: u16, focused:
&helpers::nested_u64(data, "stats", "sig_failed"),
),
helpers::kv_line("Stale", &helpers::nested_u64(data, "stats", "stale")),
helpers::kv_line(
"Parent Switched",
&helpers::nested_u64(data, "stats", "parent_switched"),
),
helpers::kv_line(
"Loop Detected",
&helpers::nested_u64(data, "stats", "loop_detected"),
src/bloom/filter.rs
+19 -9
View File
@@ -69,16 +69,18 @@ impl BloomFilter {
/// Insert a NodeAddr into the filter.
pub fn insert(&mut self, node_addr: &NodeAddr) {
let (h1, h2) = Self::base_hashes(node_addr.as_bytes());
for i in 0..self.hash_count {
let bit_index = self.hash(node_addr.as_bytes(), i);
let bit_index = self.bit_index(h1, h2, i);
self.set_bit(bit_index);
}
}
/// Insert raw bytes into the filter.
pub fn insert_bytes(&mut self, data: &[u8]) {
let (h1, h2) = Self::base_hashes(data);
for i in 0..self.hash_count {
let bit_index = self.hash(data, i);
let bit_index = self.bit_index(h1, h2, i);
self.set_bit(bit_index);
}
}
@@ -93,8 +95,9 @@ impl BloomFilter {
/// Check if the filter might contain raw bytes.
pub fn contains_bytes(&self, data: &[u8]) -> bool {
let (h1, h2) = Self::base_hashes(data);
for i in 0..self.hash_count {
let bit_index = self.hash(data, i);
let bit_index = self.bit_index(h1, h2, i);
if !self.get_bit(bit_index) {
return false;
}
@@ -196,21 +199,28 @@ impl BloomFilter {
self.hash_count
}
/// Compute a hash index for the given data and hash function number.
/// Compute the two base hashes for `data` with a single SHA-256 digest.
///
/// Uses double hashing: h(x,i) = (h1(x) + i*h2(x)) mod m
fn hash(&self, data: &[u8], k: u8) -> usize {
// Use first 16 bytes of SHA-256 for h1 and h2
/// Double hashing derives the k hash functions from two base hashes:
/// h(x,i) = (h1(x) + i*h2(x)) mod m. Computing the digest once here and
/// reusing `(h1, h2)` across all k functions avoids re-hashing per k.
fn base_hashes(data: &[u8]) -> (u64, u64) {
// Use first 16 bytes of SHA-256 for h1 and h2.
use sha2::{Digest, Sha256};
let mut hasher = Sha256::new();
hasher.update(data);
let hash = hasher.finalize();
// h1 from first 8 bytes
// h1 from first 8 bytes, h2 from next 8 bytes (little-endian).
let h1 = u64::from_le_bytes(hash[0..8].try_into().unwrap());
// h2 from next 8 bytes
let h2 = u64::from_le_bytes(hash[8..16].try_into().unwrap());
(h1, h2)
}
/// Derive the bit index for hash function `k` from the base hashes.
///
/// Uses double hashing: h(x,k) = (h1(x) + k*h2(x)) mod m.
fn bit_index(&self, h1: u64, h2: u64, k: u8) -> usize {
let combined = h1.wrapping_add((k as u64).wrapping_mul(h2));
(combined as usize) % self.num_bits
}
src/bloom/state.rs
+65 -7
View File
@@ -172,21 +172,79 @@ impl BloomState {
peer_addrs: &[NodeAddr],
peer_filters: &HashMap<NodeAddr, BloomFilter>,
) {
for peer_addr in peer_addrs {
if peer_addr == exclude_from {
continue;
}
let new_filter = self.compute_outgoing_filter(peer_addr, peer_filters);
let changed = match self.last_sent_filters.get(peer_addr) {
let targets: Vec<NodeAddr> = peer_addrs
.iter()
.filter(|addr| *addr != exclude_from)
.copied()
.collect();
for (peer_addr, new_filter) in self.compute_outgoing_filters(&targets, peer_filters) {
let changed = match self.last_sent_filters.get(&peer_addr) {
Some(last) => *last != new_filter,
None => true, // never sent → must send
};
if changed {
self.pending_updates.insert(*peer_addr);
self.pending_updates.insert(peer_addr);
}
}
}
/// Compute the outgoing filter for many peers in one pass.
///
/// Equivalent to calling [`compute_outgoing_filter`](Self::compute_outgoing_filter)
/// once per target, but linear in the number of contributing peer
/// filters instead of quadratic. The per-peer call rebuilds the whole
/// union from scratch, so computing it for every peer costs
/// O(targets × filters) 1 KB merges; announce fan-out on a
/// large node does exactly that, once per tick and again on every
/// inbound announce.
///
/// The split-horizon exclusion is the only thing that differs between
/// targets, so the union of "everything except peer i" is assembled
/// from a running prefix union and a precomputed suffix union. Merging
/// is a bytewise OR, which is commutative and associative, so the
/// result is bit-identical to the per-peer computation.
pub fn compute_outgoing_filters(
&self,
targets: &[NodeAddr],
peer_filters: &HashMap<NodeAddr, BloomFilter>,
) -> HashMap<NodeAddr, BloomFilter> {
let base = self.base_filter();
let keys: Vec<NodeAddr> = peer_filters.keys().copied().collect();
let n = keys.len();
// suffix[i] = union of peer_filters[keys[i..]]; suffix[n] is empty.
let mut suffix = vec![BloomFilter::new(); n + 1];
for i in (0..n).rev() {
let mut acc = suffix[i + 1].clone();
// Size mismatches are skipped, exactly as in the per-peer path.
let _ = acc.merge(&peer_filters[&keys[i]]);
suffix[i] = acc;
}
// Filter for a target that contributes nothing: everything merged.
let mut all = base.clone();
let _ = all.merge(&suffix[0]);
let mut per_key: HashMap<NodeAddr, BloomFilter> = HashMap::with_capacity(n);
let mut prefix = BloomFilter::new();
for i in 0..n {
let mut outgoing = base.clone();
let _ = outgoing.merge(&prefix);
let _ = outgoing.merge(&suffix[i + 1]);
per_key.insert(keys[i], outgoing);
let _ = prefix.merge(&peer_filters[&keys[i]]);
}
targets
.iter()
.map(|target| {
let filter = per_key.get(target).cloned().unwrap_or_else(|| all.clone());
(*target, filter)
})
.collect()
}
/// Compute the outgoing filter for a specific peer.
///
/// The filter includes:
src/bloom/tests.rs
+145
View File
@@ -228,6 +228,84 @@ fn test_bloom_filter_insert_bytes_contains_bytes() {
assert!(filter.contains_bytes(data2));
}
#[test]
fn test_bloom_filter_bit_indices_match_double_hashing_formula() {
use sha2::{Digest, Sha256};
// Independently recompute the documented double-hashing bit indices:
// one SHA-256 digest of the input, h1 = bytes[0..8] LE, h2 = bytes[8..16]
// LE, then for k in 0..hash_count: (h1 + k*h2) mod num_bits. This pins
// bit-identical behavior regardless of the internal implementation.
fn expected_indices(data: &[u8], num_bits: usize, hash_count: u8) -> Vec<usize> {
let digest = Sha256::digest(data);
let h1 = u64::from_le_bytes(digest[0..8].try_into().unwrap());
let h2 = u64::from_le_bytes(digest[8..16].try_into().unwrap());
(0..hash_count)
.map(|k| {
let combined = h1.wrapping_add((k as u64).wrapping_mul(h2));
(combined as usize) % num_bits
})
.collect()
}
fn bit_is_set(filter: &BloomFilter, index: usize) -> bool {
let byte = filter.as_bytes()[index / 8];
(byte >> (index % 8)) & 1 == 1
}
let configs = [(1024usize, 5u8), (8192usize, 7u8)];
let inputs: [&[u8]; 4] = [b"", b"alpha", b"the quick brown fox", &[0u8, 1, 2, 3, 255]];
for (num_bits, hash_count) in configs {
for data in inputs {
let mut filter = BloomFilter::with_params(num_bits, hash_count).unwrap();
let expected = expected_indices(data, num_bits, hash_count);
filter.insert_bytes(data);
// Every expected bit is set.
for &idx in &expected {
assert!(
bit_is_set(&filter, idx),
"expected bit {} set for input {:?} (num_bits={}, k={})",
idx,
data,
num_bits,
hash_count
);
}
// No unexpected bits are set: the set-bit count never exceeds the
// number of distinct expected indices.
use std::collections::HashSet;
let distinct: HashSet<usize> = expected.iter().copied().collect();
assert_eq!(
filter.count_ones(),
distinct.len(),
"unexpected bits set for input {:?}",
data
);
// contains reports the inserted item as present.
assert!(filter.contains_bytes(data));
}
}
// NodeAddr path uses the same formula over its byte view.
let node = make_node_addr(7);
let mut filter = BloomFilter::with_params(1024, 5).unwrap();
let expected = expected_indices(node.as_bytes(), 1024, 5);
filter.insert(&node);
for &idx in &expected {
assert!(bit_is_set(&filter, idx));
}
assert!(filter.contains(&node));
// Spot-check a definitely-absent item is reported absent.
let absent = make_node_addr(200);
assert!(!filter.contains(&absent));
}
#[test]
fn test_bloom_filter_estimated_count_saturated() {
// Create a small filter with all bits set
@@ -606,3 +684,70 @@ fn test_bloom_state_mark_changed_peers_excludes_source() {
assert!(!state.needs_update(&peer1));
}
#[test]
fn test_compute_outgoing_filters_matches_per_peer() {
let node = make_node_addr(0);
let mut state = BloomState::new(node);
state.add_leaf_dependent(make_node_addr(200));
state.add_leaf_dependent(make_node_addr(201));
// Six contributing peers with overlapping content, plus one whose
// filter is a different size and must be skipped by both paths.
let mut peer_filters = HashMap::new();
for i in 1u8..=6 {
let mut filter = BloomFilter::new();
for j in 0..5u8 {
filter.insert(&make_node_addr(i.wrapping_mul(7).wrapping_add(j)));
}
peer_filters.insert(make_node_addr(i), filter);
}
let odd_peer = make_node_addr(7);
let mut odd = BloomFilter::with_params(4096, DEFAULT_HASH_COUNT).unwrap();
odd.insert(&make_node_addr(99));
peer_filters.insert(odd_peer, odd);
// Targets: every contributing peer, the odd-sized one, and two peers
// that contribute nothing (non-tree peers get announces too).
let mut targets: Vec<NodeAddr> = (1u8..=7).map(make_node_addr).collect();
targets.push(make_node_addr(120));
targets.push(make_node_addr(121));
let batch = state.compute_outgoing_filters(&targets, &peer_filters);
assert_eq!(batch.len(), targets.len());
for target in &targets {
let expected = state.compute_outgoing_filter(target, &peer_filters);
assert_eq!(
batch.get(target),
Some(&expected),
"batch filter for {:?} differs from per-peer computation",
target
);
}
// Split horizon is real, not vacuous: a contributing peer's own
// entries must be absent from its own outgoing filter, and present
// in another peer's.
let peer3 = make_node_addr(3);
let own_entry = make_node_addr(3u8.wrapping_mul(7));
assert!(!batch[&peer3].contains(&own_entry));
assert!(batch[&make_node_addr(1)].contains(&own_entry));
}
#[test]
fn test_compute_outgoing_filters_empty_inputs() {
let node = make_node_addr(0);
let state = BloomState::new(node);
let peer_filters = HashMap::new();
assert!(
state
.compute_outgoing_filters(&[], &peer_filters)
.is_empty()
);
let target = make_node_addr(1);
let batch = state.compute_outgoing_filters(&[target], &peer_filters);
assert_eq!(batch[&target], state.base_filter());
}
src/config/mod.rs
+107
View File
@@ -24,6 +24,7 @@ mod node;
mod peer;
mod transport;
use crate::node::REKEY_JITTER_SECS;
use crate::upper::config::{DnsConfig, TunConfig};
use crate::{Identity, IdentityError};
use serde::{Deserialize, Serialize};
@@ -686,6 +687,32 @@ impl Config {
}
}
// Reject rekey triggers that fire immediately and forever. Both
// arms are checked regardless of `node.rekey.enabled` so that
// turning rekey on later cannot surface a config error at a
// surprising moment. There is deliberately no upper bound:
// u64::MAX is the idiom for disabling one arm of the trigger.
let rekey = &self.node.rekey;
if rekey.after_messages == 0 {
return Err(ConfigError::Validation(
"`node.rekey.after_messages` must be at least 1; 0 fires the message-count trigger on every poll instead of disabling it. \
Use a very large value to effectively disable the message-count trigger."
.to_string(),
));
}
let jitter_secs = REKEY_JITTER_SECS.unsigned_abs();
if rekey.after_secs <= jitter_secs {
return Err(ConfigError::Validation(format!(
"`node.rekey.after_secs` is {}, but must be greater than the per-session rekey jitter of {jitter_secs}s; \
each session offsets the interval by a random value in [-{jitter_secs}, +{jitter_secs}] seconds, so a smaller interval saturates to zero \
and rekeys on sight for roughly half of sessions. \
Use a very large value to effectively disable the timer trigger.",
rekey.after_secs
)));
}
Ok(())
}
@@ -1459,6 +1486,86 @@ peers:
.expect("outbound_only should be exempt from the loopback check");
}
#[test]
fn test_validate_default_rekey_settings_ok() {
Config::default()
.validate()
.expect("shipped default rekey settings must validate");
}
#[test]
fn test_validate_rekey_after_messages_zero_rejected() {
let mut config = Config::default();
config.node.rekey.after_messages = 0;
let err = config.validate().expect_err("validation should fail");
let msg = err.to_string();
assert!(msg.contains("after_messages"), "got: {msg}");
}
#[test]
fn test_validate_rekey_after_messages_one_accepted() {
let mut config = Config::default();
config.node.rekey.after_messages = 1;
config
.validate()
.expect("after_messages = 1 rekeys every message, which is wasteful but well defined");
}
#[test]
fn test_validate_rekey_after_secs_at_or_below_jitter_rejected() {
let jitter = REKEY_JITTER_SECS.unsigned_abs();
for after_secs in [0, 1, jitter - 1, jitter] {
let mut config = Config::default();
config.node.rekey.after_secs = after_secs;
match config.validate() {
Err(e) => assert!(e.to_string().contains("after_secs"), "got: {e}"),
Ok(()) => panic!("after_secs = {after_secs} should be rejected"),
}
}
}
#[test]
fn test_validate_rekey_after_secs_just_above_jitter_accepted() {
let mut config = Config::default();
config.node.rekey.after_secs = REKEY_JITTER_SECS.unsigned_abs() + 1;
config
.validate()
.expect("one second above the jitter bound leaves a non-zero effective interval");
}
#[test]
fn test_validate_rekey_unbounded_values_accepted() {
let mut config = Config::default();
config.node.rekey.after_secs = u64::MAX;
config.node.rekey.after_messages = u64::MAX;
config
.validate()
.expect("u64::MAX disables an arm of the trigger and must stay legal");
}
#[test]
fn test_validate_rekey_checked_even_when_disabled() {
let mut config = Config::default();
config.node.rekey.enabled = false;
config.node.rekey.after_messages = 0;
let err = config.validate().expect_err("validation should fail");
assert!(err.to_string().contains("after_messages"));
let mut config = Config::default();
config.node.rekey.enabled = false;
config.node.rekey.after_secs = REKEY_JITTER_SECS.unsigned_abs();
let err = config.validate().expect_err("validation should fail");
assert!(err.to_string().contains("after_secs"));
}
#[test]
fn test_outbound_only_forces_ephemeral_bind() {
let cfg = UdpConfig {
src/config/node.rs
+5 -5
View File
@@ -635,8 +635,8 @@ pub struct BloomConfig {
pub update_debounce_ms: u64,
/// Antipoison cap: reject inbound FilterAnnounce whose FPR exceeds
/// this value (`node.bloom.max_inbound_fpr`). Valid range `(0.0, 1.0)`.
/// Default `0.10` ≈ fill 0.631 at k=5 ≈ ~1,630 entries on the 1 KB
/// filter (Swamidass–Baldi). Raised from 0.05 so aggregates that are
/// Default `0.20` ≈ fill 0.7248 at k=5 ≈ ~2,114 entries on the 1 KB
/// filter (Swamidass–Baldi). Raised from 0.10 so aggregates that are
/// legitimately near their operating ceiling are not rejected before
/// the network reaches the fixed-filter capacity limit; conceptually
/// distinct from future autoscaling hysteresis setpoints — same unit,
@@ -648,8 +648,8 @@ pub struct BloomConfig {
impl Default for BloomConfig {
fn default() -> Self {
Self {
update_debounce_ms: 500,
max_inbound_fpr: 0.10,
update_debounce_ms: Self::default_update_debounce_ms(),
max_inbound_fpr: Self::default_max_inbound_fpr(),
}
}
}
@@ -659,7 +659,7 @@ impl BloomConfig {
500
}
fn default_max_inbound_fpr() -> f64 {
0.10
0.20
}
}
src/control/snapshots/show_routing.json
+6
View File
@@ -53,6 +53,12 @@
"originated_packets": 0,
"received_bytes": 0,
"received_packets": 0,
"route_crosslink_ascend": 0,
"route_crosslink_descend": 0,
"route_direct_peer": 0,
"route_tree_down": 0,
"route_tree_down_cross": 0,
"route_tree_up": 0,
"ttl_exhausted_bytes": 0,
"ttl_exhausted_packets": 0
},
src/control/snapshots/show_status.json
+6
View File
@@ -22,6 +22,12 @@
"originated_packets": 0,
"received_bytes": 0,
"received_packets": 0,
"route_crosslink_ascend": 0,
"route_crosslink_descend": 0,
"route_direct_peer": 0,
"route_tree_down": 0,
"route_tree_down_cross": 0,
"route_tree_up": 0,
"ttl_exhausted_bytes": 0,
"ttl_exhausted_packets": 0
},
src/control/snapshots/show_tree.json
-1
View File
@@ -24,7 +24,6 @@
"loop_detected": 0,
"outbound_sign_failed": 0,
"parent_losses": 0,
"parent_switched": 0,
"parent_switches": 0,
"rate_limited": 0,
"received": 0,
src/discovery/nostr/runtime.rs
+87 -17
View File
@@ -1302,6 +1302,16 @@ impl NostrDiscovery {
self.mark_session_seen(&offer.session_id).await?;
// Resolve the answer's relays before binding a socket and running STUN.
// Nothing in the relay choice depends on what STUN observes, and an offer
// from a peer we share no relay with cannot be answered at all — doing it
// in this order spends a STUN round trip, and holds an offer slot for its
// duration, only to discard the result.
let relays = self.preferred_signal_relays(sender, None).await?;
if relays.is_empty() {
return Err(BootstrapError::MissingRelays(offer.sender_npub.clone()));
}
let base_socket = std::net::UdpSocket::bind(("0.0.0.0", 0))?;
base_socket.set_nonblocking(true)?;
let (reflexive_address, local_addresses, stun_server) = observe_traversal_addresses(
@@ -1336,7 +1346,6 @@ impl NostrDiscovery {
(!accepted).then_some("no-usable-addresses".to_string()),
Some(offer_received_at),
);
let relays = self.preferred_signal_relays(sender, None).await?;
let answer_event = self.send_signal(&relays, sender, &answer).await?;
debug!(
peer = %peer_short,
@@ -1466,22 +1475,21 @@ impl NostrDiscovery {
target_pubkey: PublicKey,
advert: Option<&OverlayAdvert>,
) -> Result<Vec<String>, BootstrapError> {
let mut merged = self.find_recipient_inbox_relays(target_pubkey).await?;
if let Some(advert) = advert
&& let Some(relays) = advert.signal_relays.as_ref()
{
for relay in relays {
if !merged.contains(relay) {
merged.push(relay.clone());
}
}
}
for relay in &self.config.dm_relays {
if !merged.contains(relay) {
merged.push(relay.clone());
}
}
Ok(merged)
let inbox = self.find_recipient_inbox_relays(target_pubkey).await?;
let pool: HashSet<RelayUrl> = self.client.pool().all_relays().await.into_keys().collect();
let usable = signal_relays(
&inbox,
advert.and_then(|advert| advert.signal_relays.as_deref()),
&self.config.dm_relays,
&pool,
);
debug!(
peer = %target_pubkey.to_bech32().map(|npub| short_npub(&npub)).unwrap_or_default(),
inbox = inbox.len(),
usable = usable.len(),
"traversal: signal relays resolved against the client pool"
);
Ok(usable)
}
async fn find_recipient_inbox_relays(
@@ -1732,6 +1740,56 @@ impl NostrDiscovery {
}
}
/// Retain only the candidates the client pool actually holds.
///
/// `send_event_to` rejects the whole send with `RelayNotFound` if any single URL
/// is outside the pool, so a signal addressed to a peer's advertised relays fails
/// entirely on one relay we are not configured with. Filtering first turns that
/// into a send to the relays we share.
///
/// Comparison is on the normalized `RelayUrl` rather than the raw string, because
/// the pool is keyed that way: a candidate spelled `wss://relay.example/` matches
/// a configured `wss://relay.example`. Order is preserved, candidates that fail
/// to parse are dropped, and duplicates that normalize alike are collapsed.
fn retain_pooled_relays(candidates: &[String], pool: &HashSet<RelayUrl>) -> Vec<String> {
let mut seen: HashSet<RelayUrl> = HashSet::new();
let mut usable = Vec::with_capacity(candidates.len());
for candidate in candidates {
let Ok(url) = RelayUrl::parse(candidate) else {
continue;
};
if pool.contains(&url) && seen.insert(url.clone()) {
usable.push(url.to_string());
}
}
usable
}
/// Choose the relays a traversal signal for one peer should be sent to.
///
/// The candidates are the peer's NIP-17 inbox relays, then the relays its advert
/// nominates for signaling, then our own DM relays — remote-supplied first, ours
/// last, so a peer's preference is honored where we can act on it. The result is
/// whatever survives [`retain_pooled_relays`].
///
/// This is the whole decision, kept synchronous so it can be exercised without a
/// relay client: the caller's only job is to supply the fetched inbox list and
/// the pool.
pub(super) fn signal_relays(
inbox: &[String],
advert_signal: Option<&[String]>,
dm_relays: &[String],
pool: &HashSet<RelayUrl>,
) -> Vec<String> {
let mut merged: Vec<String> = inbox.to_vec();
for relay in advert_signal.unwrap_or_default().iter().chain(dm_relays) {
if !merged.contains(relay) {
merged.push(relay.clone());
}
}
retain_pooled_relays(&merged, pool)
}
#[cfg(test)]
impl NostrDiscovery {
/// Build a minimal `NostrDiscovery` for unit tests. No relay client is
@@ -1803,6 +1861,18 @@ impl NostrDiscovery {
}
}
/// Point the test instance's advert relays at explicit URLs. Unit tests
/// that exercise `refetch_advert_for_stale_check` use this to replace the
/// default public relay list with a local blackhole, so the refetch runs
/// its full 2s timeout without touching the network.
pub(crate) async fn set_advert_relays_for_test(&mut self, relays: Vec<String>) {
for url in &relays {
let _ = self.client.add_relay(url.as_str()).await;
}
self.client.connect().await;
self.config.advert_relays = relays;
}
/// Insert a cached advert directly into the in-memory cache. Used by
/// unit tests to set up consumer-side state without needing live relays.
pub(crate) async fn insert_advert_for_test(&self, npub: String, advert: CachedOverlayAdvert) {
src/discovery/nostr/tests.rs
+189 -3
View File
@@ -1,13 +1,15 @@
use nostr::prelude::{EventBuilder, Kind, Tag, Timestamp};
use std::collections::HashSet;
use super::runtime::{NostrDiscovery, suppress_responder_for_own_initiator};
use nostr::prelude::{EventBuilder, Kind, RelayUrl, Tag, Timestamp};
use super::runtime::{NostrDiscovery, signal_relays, suppress_responder_for_own_initiator};
use super::signal::{
FreshnessOutcome, build_signal_event, create_traversal_answer, create_traversal_offer,
estimate_clock_skew, validate_offer_freshness, validate_traversal_answer_for_offer,
};
use super::stun::{parse_stun_binding_success, parse_stun_url};
use super::traversal::{
PunchStrategy, build_punch_packet, parse_punch_packet, plan_punch_targets,
PunchStrategy, build_punch_packet, now_ms, parse_punch_packet, plan_punch_targets,
planned_remote_endpoints, session_hash,
};
use super::{
@@ -670,3 +672,187 @@ fn responder_suppression_election() {
&smaller, &smaller, true
));
}
#[test]
fn now_ms_tracks_the_wall_clock() {
use std::time::{SystemTime, UNIX_EPOCH};
fn wall_ms() -> u64 {
SystemTime::now()
.duration_since(UNIX_EPOCH)
.expect("system clock is after the Unix epoch")
.as_millis() as u64
}
// Bracket a sample between two independent wall-clock reads taken either
// side of it. This is the property the traversal clock has to hold for the
// NIP-40 expiration tags it computes to be in the future when published.
//
// Read this for what it is: it pins the contract (Unix epoch, milliseconds,
// tracking real time) and it fires on a host that has actually suspended,
// where the sample falls below `before` by the suspend duration. It is NOT a
// regression guard for the anchored-clock defect. Nothing reachable from a
// unit test can simulate a suspend, so on a machine that has not slept, an
// anchored implementation passes this -- deterministically when this is the
// first caller of `now_ms()` in the binary, and otherwise with a probability
// set by the fractional millisecond the anchor happened to capture.
let before = wall_ms();
let sampled = now_ms();
let after = wall_ms();
assert!(
sampled >= before,
"now_ms() is behind the wall clock: {sampled} < {before}"
);
assert!(
sampled <= after,
"now_ms() is ahead of the wall clock: {sampled} > {after}"
);
}
fn pool(urls: &[&str]) -> HashSet<RelayUrl> {
urls.iter()
.map(|url| RelayUrl::parse(url).expect("test pool url parses"))
.collect()
}
fn candidates(urls: &[&str]) -> Vec<String> {
urls.iter().map(|url| url.to_string()).collect()
}
#[test]
fn out_of_pool_relay_does_not_suppress_the_shared_ones() {
let usable = signal_relays(
&candidates(&[
"wss://relay.damus.io",
"wss://temp.iris.to",
"wss://nos.lol",
]),
None,
&[],
&pool(&[
"wss://relay.damus.io",
"wss://nos.lol",
"wss://offchain.pub",
]),
);
assert_eq!(
usable,
vec![
"wss://relay.damus.io".to_string(),
"wss://nos.lol".to_string()
],
"the unknown relay must be dropped without taking the shared ones with it"
);
}
#[test]
fn trailing_slash_and_host_case_variants_are_retained() {
let usable = signal_relays(
&candidates(&["wss://Relay.Damus.io/", "wss://nos.lol"]),
None,
&[],
&pool(&["wss://relay.damus.io", "wss://nos.lol"]),
);
assert_eq!(
usable.len(),
2,
"normalized spellings of a configured relay are the same relay: {usable:?}"
);
}
#[test]
fn duplicates_that_normalize_alike_are_collapsed() {
let usable = signal_relays(
&candidates(&["wss://nos.lol", "wss://nos.lol/", "wss://NOS.LOL"]),
None,
&[],
&pool(&["wss://nos.lol"]),
);
assert_eq!(usable, vec!["wss://nos.lol".to_string()]);
}
#[test]
fn unparseable_candidates_are_dropped_rather_than_failing_the_set() {
let usable = signal_relays(
&candidates(&["not a url", "wss://nos.lol"]),
None,
&[],
&pool(&["wss://nos.lol"]),
);
assert_eq!(usable, vec!["wss://nos.lol".to_string()]);
}
#[test]
fn no_shared_relay_yields_an_empty_set_for_the_caller_to_reject() {
let usable = signal_relays(
&candidates(&["wss://temp.iris.to"]),
None,
&[],
&pool(&["wss://nos.lol"]),
);
assert!(
usable.is_empty(),
"with no overlap the caller must see nothing to send to, not a doomed send"
);
}
#[test]
fn signal_relays_merges_all_three_sources_then_filters() {
let usable = signal_relays(
&candidates(&["wss://temp.iris.to", "wss://nos.lol"]),
Some(&candidates(&[
"wss://relay.damus.io",
"wss://unknown.example",
])),
&candidates(&["wss://offchain.pub"]),
&pool(&[
"wss://nos.lol",
"wss://relay.damus.io",
"wss://offchain.pub",
]),
);
assert_eq!(
usable,
vec![
"wss://nos.lol".to_string(),
"wss://relay.damus.io".to_string(),
"wss://offchain.pub".to_string(),
],
"every source must contribute, and only the out-of-pool entries drop out"
);
}
#[test]
fn signal_relays_keeps_our_dm_relays_when_the_peer_shares_nothing() {
let usable = signal_relays(
&candidates(&["wss://temp.iris.to"]),
Some(&candidates(&["wss://also.unknown"])),
&candidates(&["wss://nos.lol"]),
&pool(&["wss://nos.lol"]),
);
assert_eq!(
usable,
vec!["wss://nos.lol".to_string()],
"our own DM relays are always in the pool, so the result is never empty \
while any are configured"
);
}
#[test]
fn signal_relays_without_an_advert_still_resolves() {
let usable = signal_relays(
&candidates(&["wss://nos.lol", "wss://temp.iris.to"]),
None,
&candidates(&["wss://offchain.pub"]),
&pool(&["wss://nos.lol", "wss://offchain.pub"]),
);
assert_eq!(
usable,
vec![
"wss://nos.lol".to_string(),
"wss://offchain.pub".to_string()
],
"the responder path passes no advert and must still produce a target set"
);
}
src/discovery/nostr/traversal.rs
+29 -19
View File
@@ -1,5 +1,5 @@
use std::net::SocketAddr;
use std::sync::{Arc, OnceLock};
use std::sync::Arc;
use std::time::{Duration, Instant, SystemTime, UNIX_EPOCH};
use tokio::net::UdpSocket;
@@ -197,25 +197,35 @@ pub(super) fn nonce() -> String {
format!("{}-{:016x}", now_ms(), rand::random::<u64>())
}
/// Current Unix time in milliseconds, read from the wall clock on every call.
///
/// This deliberately does not cache a start-of-process anchor and advance it
/// with a monotonic `Instant`. A monotonic clock does not advance while the host
/// is suspended, so an anchored value trails real time by the suspend duration
/// for the remaining life of the process. Every expiry computed from it is then
/// published already in the past, the relay drops the event as expired, and
/// traversal signalling fails until the daemon is restarted.
///
/// About half the consumers publish or serialize the value as an absolute
/// timestamp: the NIP-40 expiration tags on adverts and traversal signals, and
/// the `issuedAt`/`expiresAt` fields of offers and answers. The rest compare it
/// against timestamps on the same basis, including the peer-authored, signed
/// `created_at` of a received advert, so they need it to track real time too.
///
/// The interval-shaped consumers survive a step in the wall clock. A forward
/// step, which is what a resume produces, saturates the punch start delay to
/// zero so punching begins immediately; the attempt's own bounds are monotonic
/// `Instant` deadlines, so its length is unaffected. A backward step lengthens
/// that delay instead and can cost a single punch attempt, which retries. Early
/// eviction from the replay window cannot admit a replay under the shipped
/// defaults, because the freshness window a replayed offer would also have to
/// satisfy (`signal_ttl_secs` plus `FRESHNESS_SKEW_TOLERANCE_MS`, 180s) is
/// strictly narrower than the replay window itself (`replay_window_secs`, 300s).
pub(super) fn now_ms() -> u64 {
struct ClockAnchor {
started_at: Instant,
started_unix_ms: u64,
}
static ANCHOR: OnceLock<ClockAnchor> = OnceLock::new();
let anchor = ANCHOR.get_or_init(|| ClockAnchor {
started_at: Instant::now(),
started_unix_ms: SystemTime::now()
.duration_since(UNIX_EPOCH)
.map(|duration| duration.as_millis() as u64)
.unwrap_or(0),
});
anchor
.started_unix_ms
.saturating_add(anchor.started_at.elapsed().as_millis() as u64)
SystemTime::now()
.duration_since(UNIX_EPOCH)
.map(|duration| duration.as_millis() as u64)
.unwrap_or(0)
}
pub(super) fn session_hash(session_id: &str) -> [u8; 16] {
src/identity/auth.rs
+3 -3
View File
@@ -1,7 +1,7 @@
//! Authentication challenge-response protocol.
use rand::Rng;
use secp256k1::{Secp256k1, XOnlyPublicKey};
use secp256k1::XOnlyPublicKey;
use sha2::{Digest, Sha256};
use super::{IdentityError, NodeAddr};
@@ -34,9 +34,9 @@ impl AuthChallenge {
/// Verify a response to this challenge.
pub fn verify(&self, response: &AuthResponse) -> Result<NodeAddr, IdentityError> {
let digest = auth_challenge_digest(&self.0, response.timestamp);
let secp = Secp256k1::new();
secp.verify_schnorr(&response.signature, &digest, &response.pubkey)
super::SECP
.verify_schnorr(&response.signature, &digest, &response.pubkey)
.map_err(|_| IdentityError::SignatureVerificationFailed)?;
Ok(NodeAddr::from_pubkey(&response.pubkey))
src/identity/local.rs
+4 -7
View File
@@ -1,6 +1,6 @@
//! Local node identity with signing capability.
use secp256k1::{Keypair, PublicKey, Secp256k1, SecretKey, XOnlyPublicKey};
use secp256k1::{Keypair, PublicKey, SecretKey, XOnlyPublicKey};
use std::fmt;
use super::auth::{AuthResponse, auth_challenge_digest};
@@ -42,8 +42,7 @@ impl Identity {
/// Create an identity from a secret key.
pub fn from_secret_key(secret_key: SecretKey) -> Self {
let secp = Secp256k1::new();
let keypair = Keypair::from_secret_key(&secp, &secret_key);
let keypair = Keypair::from_secret_key(&super::SECP, &secret_key);
Self::from_keypair(keypair)
}
@@ -93,9 +92,8 @@ impl Identity {
/// Sign arbitrary data with this identity's secret key.
pub fn sign(&self, data: &[u8]) -> secp256k1::schnorr::Signature {
let secp = Secp256k1::new();
let digest = sha256(data);
secp.sign_schnorr(&digest, &self.keypair)
super::SECP.sign_schnorr(&digest, &self.keypair)
}
/// Create an authentication response for a challenge.
@@ -103,8 +101,7 @@ impl Identity {
/// The response signs: SHA256("fips-auth-v1" || challenge || timestamp)
pub fn sign_challenge(&self, challenge: &[u8; 32], timestamp: u64) -> AuthResponse {
let digest = auth_challenge_digest(challenge, timestamp);
let secp = Secp256k1::new();
let signature = secp.sign_schnorr(&digest, &self.keypair);
let signature = super::SECP.sign_schnorr(&digest, &self.keypair);
AuthResponse {
pubkey: self.pubkey(),
timestamp,
src/identity/mod.rs
+12
View File
@@ -11,6 +11,9 @@ mod local;
mod node_addr;
mod peer;
use std::sync::LazyLock;
use secp256k1::{All, Secp256k1};
use sha2::{Digest, Sha256};
use thiserror::Error;
@@ -21,6 +24,15 @@ pub use local::Identity;
pub use node_addr::NodeAddr;
pub use peer::PeerIdentity;
/// Shared secp256k1 context reused across all identity operations.
///
/// `Secp256k1::new()` allocates a `Secp256k1<All>` and runs randomization /
/// blinding table setup; it is designed to be created once and reused rather
/// than rebuilt per sign / verify / key-derive call. This single `All` context
/// serves both signing and verification across the identity module and still
/// performs the standard construction-time blinding.
pub(crate) static SECP: LazyLock<Secp256k1<All>> = LazyLock::new(Secp256k1::new);
/// FIPS address prefix (IPv6 ULA range).
pub const FIPS_ADDRESS_PREFIX: u8 = 0xfd;
src/identity/peer.rs
+3 -3
View File
@@ -1,6 +1,6 @@
//! Remote peer identity (public key only, no signing capability).
use secp256k1::{Parity, PublicKey, Secp256k1, XOnlyPublicKey};
use secp256k1::{Parity, PublicKey, XOnlyPublicKey};
use std::fmt;
use super::encoding::{decode_npub, encode_npub};
@@ -107,9 +107,9 @@ impl PeerIdentity {
/// Verify a signature from this peer.
pub fn verify(&self, data: &[u8], signature: &secp256k1::schnorr::Signature) -> bool {
let secp = Secp256k1::new();
let digest = sha256(data);
secp.verify_schnorr(signature, &digest, &self.pubkey)
super::SECP
.verify_schnorr(signature, &digest, &self.pubkey)
.is_ok()
}
}
src/identity/tests.rs
+4 -5
View File
@@ -1,7 +1,7 @@
use std::collections::HashSet;
use std::net::Ipv6Addr;
use secp256k1::{Keypair, Secp256k1, SecretKey};
use secp256k1::{Keypair, SecretKey};
use super::*;
@@ -161,10 +161,10 @@ fn test_identity_sign() {
let sig = identity.sign(data);
// Verify the signature manually
let secp = secp256k1::Secp256k1::new();
let digest = super::sha256(data);
assert!(
secp.verify_schnorr(&sig, &digest, &identity.pubkey())
super::SECP
.verify_schnorr(&sig, &digest, &identity.pubkey())
.is_ok()
);
}
@@ -580,13 +580,12 @@ fn test_peer_identity_pubkey_full_even_parity_fallback() {
#[test]
fn test_peer_identity_pubkey_full_preserved_parity() {
// Create two identities and find one with odd parity to make this test meaningful
let secp = Secp256k1::new();
let secret_bytes: [u8; 32] = [
0x01, 0x02, 0x03, 0x04, 0x05, 0x06, 0x07, 0x08, 0x09, 0x0a, 0x0b, 0x0c, 0x0d, 0x0e, 0x0f,
0x10, 0x11, 0x12, 0x13, 0x14, 0x15, 0x16, 0x17, 0x18, 0x19, 0x1a, 0x1b, 0x1c, 0x1d, 0x1e,
0x1f, 0x20,
];
let keypair = Keypair::from_seckey_slice(&secp, &secret_bytes).unwrap();
let keypair = Keypair::from_seckey_slice(&super::SECP, &secret_bytes).unwrap();
let full_pubkey = keypair.public_key();
let peer = PeerIdentity::from_pubkey_full(full_pubkey);
src/node/bloom.rs
+24 -16
View File
@@ -29,27 +29,19 @@ impl Node {
filters
}
/// Build a FilterAnnounce for a specific peer.
///
/// The outgoing filter excludes the destination peer's own filter
/// to prevent routing loops (don't tell a peer about destinations
/// reachable only through them).
fn build_filter_announce(&mut self, exclude_peer: &NodeAddr) -> FilterAnnounce {
let peer_filters = self.peer_inbound_filters();
let filter = self
.bloom_state
.compute_outgoing_filter(exclude_peer, &peer_filters);
let sequence = self.bloom_state.next_sequence();
FilterAnnounce::new(filter, sequence)
}
/// Send a FilterAnnounce to a specific peer, respecting debounce.
///
/// `filter` is the outgoing filter for this peer, already computed
/// with the destination peer's own contribution excluded to prevent
/// routing loops (don't tell a peer about destinations reachable
/// only through them).
///
/// If the peer is rate-limited, the update stays pending for
/// delivery on the next tick cycle.
pub(super) async fn send_filter_announce_to_peer(
&mut self,
peer_addr: &NodeAddr,
filter: BloomFilter,
) -> Result<(), NodeError> {
let now_ms = std::time::SystemTime::now()
.duration_since(std::time::UNIX_EPOCH)
@@ -64,7 +56,7 @@ impl Node {
}
// Build and encode
let announce = self.build_filter_announce(peer_addr);
let announce = FilterAnnounce::new(filter, self.bloom_state.next_sequence());
let sent_filter = announce.filter.clone();
let encoded = announce.encode().map_err(|e| NodeError::SendFailed {
node_addr: *peer_addr,
@@ -142,8 +134,24 @@ impl Node {
.copied()
.collect();
if ready.is_empty() {
return;
}
// One snapshot and one union pass for the whole ready set. The
// send path never mutates peer inbound filters or the tree state,
// and the rx loop holds `&mut self` across the awaits, so the
// snapshot cannot go stale mid-loop.
let peer_filters = self.peer_inbound_filters();
let mut outgoing = self
.bloom_state
.compute_outgoing_filters(&ready, &peer_filters);
for peer_addr in ready {
if let Err(e) = self.send_filter_announce_to_peer(&peer_addr).await {
let Some(filter) = outgoing.remove(&peer_addr) else {
continue;
};
if let Err(e) = self.send_filter_announce_to_peer(&peer_addr, filter).await {
debug!(
peer = %self.peer_display_name(&peer_addr),
error = %e,
src/node/handlers/discovery.rs
+26 -11
View File
@@ -240,17 +240,32 @@ impl Node {
// map used by the TUN reader/writer at TCP MSS clamp time.
let fips_addr = crate::FipsAddress::from_node_addr(&target);
match self.path_mtu_lookup.write() {
Ok(mut map) => {
let prior = map.insert(fips_addr, path_mtu);
debug!(
target = %self.peer_display_name(&target),
fips_addr = %fips_addr,
path_mtu = path_mtu,
prior = ?prior,
map_len = map.len(),
"Wrote path_mtu_lookup from discovery LookupResponse"
);
}
Ok(mut map) => match map.get(&fips_addr).copied() {
Some(existing) if existing <= path_mtu => {
// Keep the tighter learned value; never loosen the
// clamp. A reactive MtuExceeded or PathMtuNotification
// tighten takes precedence over a looser discovery
// estimate (cross-carrier keep-tighter).
debug!(
target = %self.peer_display_name(&target),
fips_addr = %fips_addr,
path_mtu = path_mtu,
existing = existing,
"LookupResponse: keeping tighter existing path_mtu_lookup value"
);
}
other => {
map.insert(fips_addr, path_mtu);
debug!(
target = %self.peer_display_name(&target),
fips_addr = %fips_addr,
path_mtu = path_mtu,
prior = ?other,
map_len = map.len(),
"Wrote path_mtu_lookup from discovery LookupResponse"
);
}
},
Err(e) => {
warn!(
target = %self.peer_display_name(&target),
src/node/handlers/forwarding.rs
+38 -27
View File
@@ -1,9 +1,10 @@
//! SessionDatagram forwarding handler.
//!
//! Handles incoming SessionDatagram (0x00) link messages: decodes the
//! envelope, enforces hop limits, performs coordinate cache warming from
//! plaintext session-layer headers, routes to the next hop or delivers
//! locally, and generates error signals on routing failure.
//! envelope, performs coordinate cache warming from plaintext session-layer
//! headers, delivers locally when the datagram is addressed to this node,
//! otherwise enforces the transit hop limit and routes to the next hop, and
//! generates error signals on routing failure.
use crate::NodeAddr;
use crate::node::reject::ForwardingReject;
@@ -43,26 +44,16 @@ impl Node {
}
};
// TTL enforcement: decrement for forwarding and drop only if the
// received datagram was already exhausted.
if datagram_ref.ttl == 0 {
self.metrics()
.forwarding
.record_reject_bytes(ForwardingReject::TtlExhausted, payload.len());
debug!(
src = %datagram_ref.src_addr,
dest = %datagram_ref.dest_addr,
"SessionDatagram TTL exhausted, dropping"
);
return;
}
let forwarded_ttl = datagram_ref.ttl - 1;
// Coordinate cache warming from plaintext session-layer headers
// Coordinate cache warming from plaintext session-layer headers.
// Runs ahead of both the delivery and the TTL decisions: the coords
// a peer put on the wire are equally valid whichever way those go.
self.try_warm_coord_cache_ref(&datagram_ref);
// Local delivery: dispatch to session layer handlers without
// materializing an owned SessionDatagram payload Vec.
// materializing an owned SessionDatagram payload Vec. Delivery to
// the addressed node is *not* TTL-gated — under IP semantics the
// TTL governs forwarding, not delivery to the addressed host — so
// this test precedes the TTL gate below.
if datagram_ref.dest_addr == *self.node_addr() {
self.metrics().forwarding.record_delivered(payload.len());
self.handle_session_payload(
@@ -75,6 +66,24 @@ impl Node {
return;
}
// TTL enforcement on the transit path: decrement first, then drop if
// the datagram would leave with a TTL of zero. `saturating_sub` folds
// the already-exhausted arrival (ttl=0) into the same test as the
// last-hop arrival (ttl=1); neither is transmitted.
let forwarded_ttl = datagram_ref.ttl.saturating_sub(1);
if forwarded_ttl == 0 {
self.metrics()
.forwarding
.record_reject_bytes(ForwardingReject::TtlExhausted, payload.len());
debug!(
src = %datagram_ref.src_addr,
dest = %datagram_ref.dest_addr,
ttl = datagram_ref.ttl,
"SessionDatagram TTL exhausted, dropping"
);
return;
}
let mut datagram = datagram_ref.into_owned();
datagram.ttl = forwarded_ttl;
@@ -151,6 +160,11 @@ impl Node {
}
} else {
self.metrics().forwarding.record_forwarded(encoded.len());
// Classify this transit forward by route class (partition of
// forwarded_packets). Done here, at the data-plane chokepoint, so
// the error-signal routing callers of find_next_hop are excluded.
let class = self.classify_forward(&datagram.dest_addr, &next_hop_addr);
self.metrics().forwarding.record_route_class(class);
if outgoing_ce {
self.metrics().congestion.ce_forwarded.inc();
}
@@ -403,13 +417,10 @@ impl Node {
for (&tid, transport) in &self.transports {
let congestion = transport.congestion();
let state = self.transport_drops.entry(tid).or_default();
if let Some(current) = congestion.recv_drops {
let new_drops = current > state.prev_drops;
if new_drops && !state.dropping {
new_drop_events.push(tid);
}
state.dropping = new_drops;
state.prev_drops = current;
if let Some(current) = congestion.recv_drops
&& state.observe_drops(current)
{
new_drop_events.push(tid);
}
}
for tid in new_drop_events {
src/node/handlers/mmp.rs
-2
View File
@@ -173,7 +173,6 @@ impl Node {
self.coord_cache
.invalidate_via_node(our_identity.node_addr());
self.reset_discovery_backoff();
self.metrics().tree.parent_switched.inc();
self.metrics().tree.parent_switches.inc();
info!(
new_parent = %self.peer_display_name(&new_parent),
@@ -205,7 +204,6 @@ impl Node {
self.coord_cache
.invalidate_other_roots(our_identity.node_addr());
self.reset_discovery_backoff();
self.metrics().tree.parent_switched.inc();
self.metrics().tree.parent_switches.inc();
info!(
new_root = %self.tree_state.root(),
src/node/metrics.rs
+66 -2
View File
@@ -81,6 +81,51 @@ pub struct ForwardingMetrics {
pub drop_send_error_bytes: Counter,
pub originated_packets: Counter,
pub originated_bytes: Counter,
pub route_tree_up: Counter,
pub route_tree_down: Counter,
pub route_tree_down_cross: Counter,
pub route_crosslink_descend: Counter,
pub route_crosslink_ascend: Counter,
pub route_direct_peer: Counter,
}
/// Route class of a transit-forwarded packet, classified from tree
/// coordinates at the forwarding decision point. The six variants
/// partition `forwarded_packets` exactly.
///
/// Two variants are up-and-over forwards (destination not in the chosen
/// peer's subtree); they differ in whether they depend on a child
/// advertising cross-link reach *upward* to its parent:
/// - `TreeDownCross`: the chosen peer is our tree descendant, but the
/// destination is *not* in that child's subtree. The forward only fired
/// because the child advertised cross-link reach upward to us, beyond its
/// own subtree. If children advertised only their subtree upward, this
/// forward would route up instead, so its count measures how much
/// forwarding depends on the upward cross-link advertisement — the
/// dive-to-tree-child cut-through.
/// - `CrosslinkAscend`: the chosen peer is lateral (neither ancestor nor
/// descendant) and the destination is not in its subtree. This is a node
/// using its *own* cross-link, learned via the peer's split-horizon
/// advertisement to its neighbors, so it does not depend on any upward
/// advertisement. Tracked alongside `TreeDownCross` as the lateral
/// up-and-over contrast.
#[derive(Debug, Clone, Copy, PartialEq, Eq)]
pub enum RouteClass {
/// Chosen peer is our ancestor (tree-up).
TreeUp,
/// Chosen peer is our descendant and dest is in its subtree (canonical
/// tree-down).
TreeDown,
/// Chosen peer is our descendant but dest is *not* in its subtree: the
/// dive-to-tree-child cut-through enabled by upward cross-link
/// advertisement.
TreeDownCross,
/// Chosen peer is lateral and dest is in its subtree (subtree entry).
CrosslinkDescend,
/// Chosen peer is lateral and dest is not in its subtree (up-and-over).
CrosslinkAscend,
/// Chosen peer is the destination itself (degenerate direct hop).
DirectPeer,
}
impl ForwardingMetrics {
@@ -112,6 +157,21 @@ impl ForwardingMetrics {
self.originated_bytes.add(bytes as u64);
}
/// Record the route class of a transit-forwarded packet. The five
/// classes partition `forwarded_packets`, so this is called exactly
/// once per `record_forwarded` (transit chokepoint only).
#[inline]
pub fn record_route_class(&self, class: RouteClass) {
match class {
RouteClass::TreeUp => self.route_tree_up.inc(),
RouteClass::TreeDown => self.route_tree_down.inc(),
RouteClass::TreeDownCross => self.route_tree_down_cross.inc(),
RouteClass::CrosslinkDescend => self.route_crosslink_descend.inc(),
RouteClass::CrosslinkAscend => self.route_crosslink_ascend.inc(),
RouteClass::DirectPeer => self.route_direct_peer.inc(),
}
}
/// Mirror of `ForwardingStats::record_reject_bytes`: route a typed
/// forwarding rejection of `bytes` payload to its packet and byte
/// counters.
@@ -163,6 +223,12 @@ impl ForwardingMetrics {
drop_send_error_bytes: self.drop_send_error_bytes.get(),
originated_packets: self.originated_packets.get(),
originated_bytes: self.originated_bytes.get(),
route_tree_up: self.route_tree_up.get(),
route_tree_down: self.route_tree_down.get(),
route_tree_down_cross: self.route_tree_down_cross.get(),
route_crosslink_descend: self.route_crosslink_descend.get(),
route_crosslink_ascend: self.route_crosslink_ascend.get(),
route_direct_peer: self.route_direct_peer.get(),
}
}
}
@@ -251,7 +317,6 @@ pub struct TreeMetrics {
pub stale: Counter,
pub ancestry_invalid: Counter,
pub accepted: Counter,
pub parent_switched: Counter,
pub loop_detected: Counter,
pub ancestry_changed: Counter,
pub sent: Counter,
@@ -285,7 +350,6 @@ impl TreeMetrics {
stale: self.stale.get(),
ancestry_invalid: self.ancestry_invalid.get(),
accepted: self.accepted.get(),
parent_switched: self.parent_switched.get(),
loop_detected: self.loop_detected.get(),
ancestry_changed: self.ancestry_changed.get(),
sent: self.sent.get(),
src/node/mod.rs
+124 -3
View File
@@ -266,6 +266,32 @@ struct TransportDropState {
dropping: bool,
}
impl TransportDropState {
/// Fold a new cumulative `recv_drops` sample into the state and report
/// whether it marks the *transition* into a dropping condition.
///
/// Returns true only on the edge where the cumulative `SO_RXQ_OVFL`
/// counter rose since the previous sample **and** the transport was not
/// already flagged as dropping. That edge is what `kernel_drop_events`
/// counts: a first observation of a new drop burst, not every sample in
/// which the counter happens to be non-zero. A sample with no rise
/// clears the flag, so a later rise counts as a fresh event.
///
/// Pure and sans-IO by design: the tick handler reads the kernel
/// counter from the socket and does the logging, but the detection
/// decision lives here so it can be tested without a socket, a
/// transport, or a running node — which is the only way it can be
/// tested at all, since the kernel drop itself cannot be provoked
/// deterministically.
fn observe_drops(&mut self, current: u64) -> bool {
let rose = current > self.prev_drops;
let new_event = rose && !self.dropping;
self.dropping = rose;
self.prev_drops = current;
new_event
}
}
/// State for a link waiting for transport-level connection establishment.
///
/// For connection-oriented transports (TCP, Tor), the transport connect runs
@@ -292,8 +318,23 @@ struct PendingConnect {
/// 1. **Connection phase** (`connections`): Handshake in progress, indexed by LinkId
/// 2. **Active phase** (`peers`): Authenticated, indexed by NodeAddr
///
/// The `addr_to_link` map enables dispatching incoming packets to the right
/// connection before authentication completes.
/// The `addr_to_link` map is a reverse lookup from `(transport, address)` to
/// link. It is **not** a packet-dispatch path, despite what this comment used
/// to say: `find_link_by_addr` has no callers outside its own tests, and
/// encrypted frames are dispatched by session index. Its live readers are the
/// `should_admit_msg1` fast path and the duplicate-inbound-handshake check in
/// `handle_msg1`.
///
/// **Do not key a peer-identity question on it.** The address form is not
/// canonical: an outbound dial registers the literal configured string, which
/// may be a hostname (`"node-b:2121"`), while an inbound packet carries the
/// resolved form (`"10.128.2.4:2121"`). `TransportAddr` compares byte-wise, so
/// those never match, and a lookup keyed on an inbound address silently returns
/// "no such link" for every hostname-configured peer rather than failing. The
/// entry is also single-valued per key, so an inbound handshake overwrites an
/// outbound dial's entry for the same address. The readers above tolerate this
/// because each compares a key written in the same form it reads; a new reader
/// that does not will be quietly wrong.
// Discovery lookup constants moved to config: node.discovery.attempt_timeouts_secs, node.discovery.ttl
pub struct Node {
// === Immutable Context ===
@@ -1182,7 +1223,7 @@ impl Node {
return name.clone();
}
if let Some(peer) = self.peers.get(addr) {
return peer.identity().short_npub();
return peer.short_npub().to_string();
}
if let Some(entry) = self.sessions.get(addr) {
let (xonly, _) = entry.remote_pubkey().x_only_public_key();
@@ -2666,6 +2707,86 @@ impl Node {
self.peers.get(&next_hop_id).filter(|p| p.can_send())
}
/// Classify a transit forward by route class from tree coordinates.
///
/// Called at the transit chokepoint after `find_next_hop` returns a peer,
/// so the six classes partition `forwarded_packets` exactly. The branch
/// that `find_next_hop` took (bloom vs greedy-tree) is *not* the route
/// class: a peer can be selected by either, so the cut-through splits
/// (`TreeDownCross`, `CrosslinkAscend`) are decided here from coordinates,
/// not from which branch fired.
///
/// Inputs: our coords (`tree_state.my_coords`), the chosen peer's coords
/// (`tree_state.peer_coords`), and the destination coords (re-read from the
/// coord cache, which `find_next_hop` just touched). Both the tree-down and
/// cross-link branches split on whether the destination is in the chosen
/// peer's subtree; when the dest coords are unavailable that test defaults
/// to "not in subtree", i.e. the up-and-over variant (`TreeDownCross` for a
/// descendant peer, `CrosslinkAscend` for a lateral one).
pub(crate) fn classify_forward(
&self,
dest: &NodeAddr,
chosen_peer: &NodeAddr,
) -> metrics::RouteClass {
// Degenerate: the next hop is the destination itself (Branch 2).
if chosen_peer == dest {
return metrics::RouteClass::DirectPeer;
}
let my_addr = self.node_addr();
let my_coords = self.tree_state.my_coords();
// Tree-up: the chosen peer is our ancestor.
if my_coords.has_ancestor(chosen_peer) {
return metrics::RouteClass::TreeUp;
}
// Whether the destination is in the chosen peer's subtree. Both the
// tree-down and cross-link splits below turn on this same test, so it
// is computed once. On the live transit path the dest coords are
// always present here: `find_next_hop` looks them up with an early
// return, so a coord-cache miss yields no next hop to classify (the
// caller signals `CoordsRequired` instead of forwarding). The miss
// branch below is therefore defensive — reachable only by direct
// unit-test calls — and defaults the test to "not in subtree", i.e.
// the up-and-over variant of whichever branch fires (TreeDownCross for
// a descendant peer, CrosslinkAscend for a lateral one), matching the
// original cross-link default-to-ascend.
let now_ms = std::time::SystemTime::now()
.duration_since(std::time::UNIX_EPOCH)
.map(|d| d.as_millis() as u64)
.unwrap_or(0);
let dest_in_peer_subtree = self
.coord_cache
.get(dest, now_ms)
.is_some_and(|dest_coords| dest_coords.has_ancestor(chosen_peer));
// Tree-down: the chosen peer is our descendant (we are its ancestor).
// Split by subtree membership: a dest genuinely below the child is the
// canonical tree-down; a dest *not* below it means we only forwarded
// down because the child advertised cross-link reach upward, beyond its
// own subtree — the dive-to-tree-child cut-through (TreeDownCross).
if let Some(peer_coords) = self.tree_state.peer_coords(chosen_peer)
&& peer_coords.has_ancestor(my_addr)
{
return if dest_in_peer_subtree {
metrics::RouteClass::TreeDown
} else {
metrics::RouteClass::TreeDownCross
};
}
// Cross-link (lateral): split by whether the destination is in the
// chosen peer's subtree. Descend = subtree entry; ascend = up-and-over
// via the node's own cross-link (learned from the peer's split-horizon
// advertisement, independent of any upward advertisement).
if dest_in_peer_subtree {
return metrics::RouteClass::CrosslinkDescend;
}
metrics::RouteClass::CrosslinkAscend
}
/// Select the best peer from a set of bloom filter candidates.
///
/// Uses distance from each candidate's tree coordinates to the destination
src/node/reject.rs
+5 -1
View File
@@ -232,7 +232,11 @@ pub enum ForwardingReject {
/// `SessionDatagramRef::decode` returned an error. Tracked via
/// [`ForwardingStats::decode_error_packets`](crate::node::stats::ForwardingStats).
DecodeError,
/// Datagram arrived with TTL=0 — already exhausted, no forward.
/// Transit datagram whose TTL would reach zero on this hop, so it is
/// dropped rather than forwarded. Charged for an arrival at TTL 1 as
/// well as an already-exhausted arrival at TTL 0. Never charged for a
/// datagram addressed to this node, whose delivery is not TTL-gated and
/// is decided ahead of this test.
/// Tracked via
/// [`ForwardingStats::ttl_exhausted_packets`](crate::node::stats::ForwardingStats).
TtlExhausted,
src/node/retry.rs
+15 -7
View File
@@ -279,7 +279,7 @@ impl Node {
let peer_config = state.peer_config.clone();
// Refresh the peer's overlay advert before retrying. The cache is
// Kick off a refresh of the peer's overlay advert. The cache is
// read-only on hit (see fetch_advert), so every retry without a
// refetch dials the same cached endpoint — and the most common
// reason a peer ended up in retry_pending is that the cached
@@ -289,13 +289,21 @@ impl Node {
//
// refetch_advert_for_stale_check uses the relay's advert as
// ground truth: replaces the cache if there's a newer one,
// evicts if the relay has nothing, otherwise leaves it. Cheap
// (one Filter fetch with 2s timeout) and bounded by the retry
// backoff cadence.
// evicts if the relay has nothing, otherwise leaves it.
//
// Fire-and-forget, NOT awaited: this runs inline on the 1s
// rx-loop tick, and the fetch carries a 2s relay timeout that
// would stall the tick — and every other rx-loop arm with it —
// by up to 2s per due peer, MAX_RETRY_CONNECTIONS_PER_TICK times
// over. So the dial below uses whatever advert is cached now and
// the refreshed one lands for the *next* retry of this peer.
// Retries are backoff-paced, so that defers the benefit by one
// backoff interval rather than losing it.
if let Some(bootstrap) = self.nostr_discovery.clone() {
let _ = bootstrap
.refetch_advert_for_stale_check(&peer_config.npub)
.await;
let npub = peer_config.npub.clone();
tokio::spawn(async move {
let _ = bootstrap.refetch_advert_for_stale_check(&npub).await;
});
}
match self.initiate_peer_connection(&peer_config).await {
src/node/stats.rs
+6 -1
View File
@@ -229,6 +229,12 @@ pub struct ForwardingStatsSnapshot {
pub drop_send_error_bytes: u64,
pub originated_packets: u64,
pub originated_bytes: u64,
pub route_tree_up: u64,
pub route_tree_down: u64,
pub route_tree_down_cross: u64,
pub route_crosslink_descend: u64,
pub route_crosslink_ascend: u64,
pub route_direct_peer: u64,
}
#[derive(Clone, Debug, Default, Serialize)]
@@ -267,7 +273,6 @@ pub struct TreeStatsSnapshot {
pub stale: u64,
pub ancestry_invalid: u64,
pub accepted: u64,
pub parent_switched: u64,
pub loop_detected: u64,
pub ancestry_changed: u64,
pub sent: u64,
src/node/tests/discovery.rs
+39
View File
@@ -912,6 +912,45 @@ async fn test_originator_stores_path_mtu_in_cache() {
);
}
#[tokio::test]
async fn test_originator_lookup_response_keeps_tighter_path_mtu_lookup() {
// Regression: a LookupResponse carrying a looser (larger) path_mtu must
// NOT clobber a tighter (smaller) value already in path_mtu_lookup that a
// reactive MtuExceeded or PathMtuNotification learned. Cross-carrier
// keep-tighter: the clamp must never loosen.
let mut node = make_node();
let from = make_node_addr(0xAA);
let target_identity = Identity::generate();
let target = *target_identity.node_addr();
let root = make_node_addr(0xF0);
let coords = TreeCoordinate::from_addrs(vec![target, root]).unwrap();
node.register_identity(target, target_identity.pubkey_full());
// Pre-seed a tighter value, as if a reactive signal already narrowed it.
let target_fips = crate::FipsAddress::from_node_addr(&target);
node.path_mtu_lookup_insert(target_fips, 1280);
let proof_data = LookupResponse::proof_bytes(800, &target, &coords);
let proof = target_identity.sign(&proof_data);
let mut response = LookupResponse::new(800, target, coords.clone(), proof);
// Looser discovery estimate that must be rejected in favor of the tighter
// existing entry.
response.path_mtu = 1500;
let payload = &response.encode()[1..];
node.handle_lookup_response(&from, payload).await;
assert_eq!(
node.path_mtu_lookup_get(&target_fips),
Some(1280),
"LookupResponse must not loosen a tighter existing path_mtu_lookup value"
);
}
// ============================================================================
// Open-Discovery Sweep — cache-injection unit test
// ============================================================================
src/node/tests/ethernet.rs
-214
View File
@@ -1,214 +0,0 @@
//! Ethernet transport integration tests.
//!
//! Tests that the Ethernet transport works end-to-end using veth pairs.
//! All tests require root or CAP_NET_RAW and are marked `#[ignore]`.
use super::*;
use crate::config::EthernetConfig;
use crate::transport::ethernet::EthernetTransport;
use crate::transport::{TransportAddr, TransportHandle, TransportId, packet_channel};
use spanning_tree::{TestNode, cleanup_nodes, drain_all_packets, initiate_handshake};
use std::process::Command;
use std::sync::atomic::{AtomicU32, Ordering};
/// Atomic counter for unique veth names across tests.
static VETH_COUNTER: AtomicU32 = AtomicU32::new(0);
/// RAII wrapper for a veth pair.
///
/// Creates a pair of connected virtual Ethernet interfaces. Destroying
/// one end automatically destroys the other.
struct VethPair {
name_a: String,
name_b: String,
}
impl VethPair {
/// Create a new veth pair with unique interface names.
///
/// Names are kept under 15 chars (IFNAMSIZ limit). Format: `ftXXa`/`ftXXb`
/// where XX is an atomic counter combined with PID for cross-process uniqueness.
fn create() -> Self {
let id = VETH_COUNTER.fetch_add(1, Ordering::Relaxed);
let pid = std::process::id() % 10000;
let name_a = format!("ft{}{}a", pid, id);
let name_b = format!("ft{}{}b", pid, id);
assert!(name_a.len() <= 15, "veth name too long: {}", name_a);
assert!(name_b.len() <= 15, "veth name too long: {}", name_b);
// Create veth pair
let status = Command::new("ip")
.args([
"link", "add", &name_a, "type", "veth", "peer", "name", &name_b,
])
.status()
.expect("failed to run 'ip link add'");
assert!(status.success(), "failed to create veth pair");
// Bring both ends up
let status = Command::new("ip")
.args(["link", "set", &name_a, "up"])
.status()
.expect("failed to run 'ip link set up'");
assert!(status.success(), "failed to bring up {}", name_a);
let status = Command::new("ip")
.args(["link", "set", &name_b, "up"])
.status()
.expect("failed to run 'ip link set up'");
assert!(status.success(), "failed to bring up {}", name_b);
VethPair { name_a, name_b }
}
}
impl Drop for VethPair {
fn drop(&mut self) {
// Deleting one end destroys both
let _ = Command::new("ip")
.args(["link", "delete", &self.name_a])
.status();
}
}
/// Create a test node with a live Ethernet transport on the given interface.
///
/// Parallel to `make_test_node()` in spanning_tree.rs but uses
/// EthernetTransport instead of UDP.
async fn make_test_node_ethernet(interface: &str) -> TestNode {
let mut node = make_node();
let transport_id = TransportId::new(1);
let config = EthernetConfig {
interface: interface.to_string(),
discovery: Some(false),
announce: Some(false),
accept_connections: Some(true),
..Default::default()
};
let (packet_tx, packet_rx) = packet_channel(256);
let mut transport = EthernetTransport::new(transport_id, None, config, packet_tx);
transport.start_async().await.unwrap();
let mac = transport
.local_mac()
.expect("transport should have MAC after start");
let addr = TransportAddr::from_bytes(&mac);
node.transports
.insert(transport_id, TransportHandle::Ethernet(transport));
TestNode {
node,
transport_id,
packet_rx: spanning_tree::bridge_to_unbounded(packet_rx),
addr,
}
}
/// Two nodes on a veth pair complete a Noise handshake and establish peering.
#[tokio::test]
#[ignore] // Requires root or CAP_NET_RAW
async fn test_ethernet_two_node_handshake() {
let veth = VethPair::create();
let mut nodes = vec![
make_test_node_ethernet(&veth.name_a).await,
make_test_node_ethernet(&veth.name_b).await,
];
// Initiate handshake from node 0 to node 1
initiate_handshake(&mut nodes, 0, 1).await;
// Drain all packets (handshake + tree announce)
let total = drain_all_packets(&mut nodes, false).await;
assert!(total > 0, "should have processed packets");
// Verify bidirectional peering
let addr_0 = *nodes[0].node.node_addr();
let addr_1 = *nodes[1].node.node_addr();
assert!(
nodes[0].node.get_peer(&addr_1).is_some(),
"node 0 should have node 1 as peer"
);
assert!(
nodes[1].node.get_peer(&addr_0).is_some(),
"node 1 should have node 0 as peer"
);
cleanup_nodes(&mut nodes).await;
}
/// Two Ethernet nodes converge to a correct spanning tree (2-node tree).
#[tokio::test]
#[ignore] // Requires root or CAP_NET_RAW
async fn test_ethernet_data_exchange() {
use spanning_tree::verify_tree_convergence;
let veth = VethPair::create();
let mut nodes = vec![
make_test_node_ethernet(&veth.name_a).await,
make_test_node_ethernet(&veth.name_b).await,
];
initiate_handshake(&mut nodes, 0, 1).await;
let total = drain_all_packets(&mut nodes, false).await;
assert!(total > 0);
// Verify spanning tree convergence
verify_tree_convergence(&nodes);
// The root should be the node with the smallest NodeAddr
let expected_root = std::cmp::min(*nodes[0].node.node_addr(), *nodes[1].node.node_addr());
assert_eq!(*nodes[0].node.tree_state().root(), expected_root);
assert_eq!(*nodes[1].node.tree_state().root(), expected_root);
cleanup_nodes(&mut nodes).await;
}
/// Mixed transport: 2 Ethernet nodes + 2 UDP nodes coexist.
///
/// Each transport forms its own connected component. Validates that
/// `process_available_packets()` handles heterogeneous transport types.
#[tokio::test]
#[ignore] // Requires root or CAP_NET_RAW
async fn test_mixed_transport_coexistence() {
use spanning_tree::{make_test_node, verify_tree_convergence_components};
let veth = VethPair::create();
// Create 2 Ethernet nodes and 2 UDP nodes
let eth_0 = make_test_node_ethernet(&veth.name_a).await;
let eth_1 = make_test_node_ethernet(&veth.name_b).await;
let udp_0 = make_test_node().await;
let udp_1 = make_test_node().await;
let mut nodes = vec![eth_0, eth_1, udp_0, udp_1];
// Handshake within each component
initiate_handshake(&mut nodes, 0, 1).await; // Ethernet pair
initiate_handshake(&mut nodes, 2, 3).await; // UDP pair
// Drain all packets across both transports
let total = drain_all_packets(&mut nodes, false).await;
assert!(total > 0);
// Verify each component converges independently
verify_tree_convergence_components(&nodes, &[vec![0, 1], vec![2, 3]]);
// Ethernet component has its own root
let eth_root = std::cmp::min(*nodes[0].node.node_addr(), *nodes[1].node.node_addr());
assert_eq!(*nodes[0].node.tree_state().root(), eth_root);
assert_eq!(*nodes[1].node.tree_state().root(), eth_root);
// UDP component has its own root
let udp_root = std::cmp::min(*nodes[2].node.node_addr(), *nodes[3].node.node_addr());
assert_eq!(*nodes[2].node.tree_state().root(), udp_root);
assert_eq!(*nodes[3].node.tree_state().root(), udp_root);
cleanup_nodes(&mut nodes).await;
}
src/node/tests/forwarding.rs
+212 -9
View File
@@ -40,22 +40,116 @@ async fn test_forwarding_hop_limit_exhausted() {
node.handle_session_datagram(&from, &encoded[1..], false)
.await;
// No panic, no send (node has no peers)
let fwd = &node.metrics().forwarding;
assert_eq!(
fwd.ttl_exhausted_packets.get(),
1,
"transit ttl=0 should be charged to TtlExhausted"
);
assert_eq!(
fwd.drop_no_route_packets.get(),
0,
"transit ttl=0 should never reach the routing step"
);
}
#[tokio::test]
async fn test_forwarding_hop_limit_one_drops_at_transit() {
// ttl=1 means after decrement it becomes 0 — the datagram can
// still be delivered this hop but would be dropped at the next.
// decrement_ttl returns true (1 > 0), so the handler proceeds.
async fn test_forwarding_ttl_one_local_delivery_is_not_gated() {
// dest == self, so this is local delivery, not transit: the TTL gate
// does not apply and the datagram is handed to the session layer.
let mut node = make_node();
let from = make_node_addr(0xAA);
let my_addr = *node.node_addr();
let src = make_node_addr(0x01);
let dg = SessionDatagram::new(src, my_addr, vec![0x10, 0x00, 0x00, 0x00]).with_ttl(1);
let encoded = dg.encode();
// Should succeed — ttl=1 decrements to 0 but packet is still processed
node.handle_session_datagram(&from, &encoded[1..], false)
.await;
let fwd = &node.metrics().forwarding;
assert_eq!(fwd.delivered_packets.get(), 1, "ttl=1 should be delivered");
assert_eq!(fwd.ttl_exhausted_packets.get(), 0);
}
/// Acceptance: a datagram addressed to this node with ttl=0 is delivered
/// locally. The TTL governs forwarding, not delivery to the addressed host,
/// so the gate must sit after the local-delivery test — and the
/// `TtlExhausted` reject must not be charged for a delivered datagram.
#[tokio::test]
async fn test_forwarding_ttl_zero_local_delivery_is_not_gated() {
let mut node = make_node();
let from = make_node_addr(0xAA);
let my_addr = *node.node_addr();
let src = make_node_addr(0x01);
let dg = SessionDatagram::new(src, my_addr, vec![0x10, 0x00, 0x00, 0x00]).with_ttl(0);
let encoded = dg.encode();
node.handle_session_datagram(&from, &encoded[1..], false)
.await;
let fwd = &node.metrics().forwarding;
assert_eq!(
fwd.delivered_packets.get(),
1,
"ttl=0 addressed to this node must still be delivered locally"
);
assert_eq!(
fwd.ttl_exhausted_packets.get(),
0,
"local delivery must not be charged to the TtlExhausted reject"
);
assert_eq!(fwd.drop_no_route_packets.get(), 0);
}
/// Acceptance: a transit datagram arriving with ttl=1 would leave with ttl=0,
/// so it is dropped here rather than transmitted. Reaching the routing step at
/// all (`drop_no_route`) would mean it had been handed to the forwarder.
#[tokio::test]
async fn test_forwarding_ttl_one_transit_dropped_before_routing() {
let mut node = make_node();
let from = make_node_addr(0xAA);
let src = make_node_addr(0x01);
let dest = make_node_addr(0x02);
let dg = SessionDatagram::new(src, dest, vec![0x10, 0x00, 0x00, 0x00]).with_ttl(1);
let encoded = dg.encode();
node.handle_session_datagram(&from, &encoded[1..], false)
.await;
let fwd = &node.metrics().forwarding;
assert_eq!(
fwd.ttl_exhausted_packets.get(),
1,
"transit ttl=1 must be dropped as TTL-exhausted, not forwarded"
);
assert_eq!(
fwd.drop_no_route_packets.get(),
0,
"transit ttl=1 must not reach the routing step"
);
assert_eq!(fwd.forwarded_packets.get(), 0);
assert_eq!(fwd.delivered_packets.get(), 0);
}
/// The other side of the same boundary: ttl=2 clears the gate. This node has
/// no peers, so it fails at the routing step instead — which is the evidence
/// that the TTL gate passed it through.
#[tokio::test]
async fn test_forwarding_ttl_two_transit_clears_the_gate() {
let mut node = make_node();
let from = make_node_addr(0xAA);
let src = make_node_addr(0x01);
let dest = make_node_addr(0x02);
let dg = SessionDatagram::new(src, dest, vec![0x10, 0x00, 0x00, 0x00]).with_ttl(2);
let encoded = dg.encode();
node.handle_session_datagram(&from, &encoded[1..], false)
.await;
let fwd = &node.metrics().forwarding;
assert_eq!(
fwd.ttl_exhausted_packets.get(),
0,
"transit ttl=2 must clear the TTL gate"
);
assert_eq!(
fwd.drop_no_route_packets.get(),
1,
"transit ttl=2 should have reached the routing step and found no route"
);
}
// --- Local delivery ---
@@ -392,9 +486,9 @@ async fn test_forwarding_multi_hop() {
#[tokio::test]
async fn test_forwarding_hop_limit_prevents_infinite_loops() {
// 3-node chain: 0 -- 1 -- 2
// Send a datagram with ttl=1. It should be forwarded by node 1
// (decrement to 0) and delivered at node 2 (local delivery). If node 2
// tried to forward further, the 0 ttl would prevent it.
// Send a datagram with ttl=2. Node 1 forwards it as transit (2 -> 1) and
// node 2 delivers it locally, which is not TTL-gated. Had node 2 been
// transit instead, the arriving ttl=1 would have stopped it there.
let edges = vec![(0, 1), (1, 2)];
let mut nodes = run_tree_test(3, &edges, false).await;
verify_tree_convergence(&nodes);
@@ -409,7 +503,7 @@ async fn test_forwarding_hop_limit_prevents_infinite_loops() {
node2_addr,
vec![0x10, 0x00, 0x04, 0x00, 1, 2, 3, 4],
)
.with_ttl(2); // Enough for 0->1 (decrement to 1) and 1->2 (decrement to 0, local delivery)
.with_ttl(2); // Node 1 forwards with ttl=1; node 2 is the destination
let encoded = dg.encode();
@@ -428,6 +522,115 @@ async fn test_forwarding_hop_limit_prevents_infinite_loops() {
cleanup_nodes(&mut nodes).await;
}
/// Acceptance: a transit datagram arriving with ttl=2 leaves with ttl=1.
///
/// Pinned on a live 3-node chain (0 -- 1 -- 2) by where the datagram stops,
/// since the TTL that leaves node 0 is only observable through what the next
/// hop does with it. Both injections are transit at node 0 (external source,
/// destined for node 2), so node 1 is a forwarder in both.
///
/// - ttl=2 in: node 0 must emit ttl=1, which node 1 (transit) drops. If node 0
/// emitted ttl=2 unchanged, node 1 would forward and node 2 would deliver.
/// - ttl=3 in: node 0 emits 2, node 1 emits 1, node 2 delivers (delivery is
/// not TTL-gated). If either hop decremented by more than one, the datagram
/// would have died at node 1 instead.
///
/// Together the two pin the decrement at exactly one per hop and the drop at
/// would-leave-zero.
#[tokio::test]
async fn test_forwarding_ttl_decrement_is_one_per_hop() {
let edges = vec![(0, 1), (1, 2)];
let mut nodes = run_tree_test(3, &edges, false).await;
verify_tree_convergence(&nodes);
populate_all_coord_caches(&mut nodes);
let node0_addr = *nodes[0].node.node_addr();
let node2_addr = *nodes[2].node.node_addr();
let external_src = make_node_addr(0xEE);
// --- ttl=2: must die at node 1, one hop short of the destination ---
let dg =
SessionDatagram::new(external_src, node2_addr, vec![0x10, 0x00, 0x00, 0x00]).with_ttl(2);
let encoded = dg.encode();
nodes[0]
.node
.handle_session_datagram(&node0_addr, &encoded[1..], false)
.await;
for _ in 0..3 {
tokio::time::sleep(Duration::from_millis(50)).await;
process_available_packets(&mut nodes).await;
}
assert_eq!(
nodes[0].node.metrics().forwarding.forwarded_packets.get(),
1,
"node 0 should have forwarded the ttl=2 datagram"
);
assert_eq!(
nodes[1]
.node
.metrics()
.forwarding
.ttl_exhausted_packets
.get(),
1,
"node 1 should have received ttl=1 and dropped it as TTL-exhausted"
);
assert_eq!(
nodes[1].node.metrics().forwarding.forwarded_packets.get(),
0,
"node 1 must not forward a datagram that would leave with ttl=0"
);
assert_eq!(
nodes[2].node.metrics().forwarding.delivered_packets.get(),
0,
"node 2 must never see the ttl=2 datagram"
);
// --- ttl=3: must survive both transit hops and be delivered at node 2 ---
let dg =
SessionDatagram::new(external_src, node2_addr, vec![0x10, 0x00, 0x00, 0x00]).with_ttl(3);
let encoded = dg.encode();
nodes[0]
.node
.handle_session_datagram(&node0_addr, &encoded[1..], false)
.await;
for _ in 0..3 {
tokio::time::sleep(Duration::from_millis(50)).await;
process_available_packets(&mut nodes).await;
}
assert_eq!(
nodes[0].node.metrics().forwarding.forwarded_packets.get(),
2,
"node 0 should have forwarded the ttl=3 datagram too"
);
assert_eq!(
nodes[1].node.metrics().forwarding.forwarded_packets.get(),
1,
"node 1 should have forwarded the ttl=2 it received"
);
assert_eq!(
nodes[1]
.node
.metrics()
.forwarding
.ttl_exhausted_packets
.get(),
1,
"node 1 should not have dropped the second datagram"
);
assert_eq!(
nodes[2].node.metrics().forwarding.delivered_packets.get(),
1,
"node 2 should have delivered the datagram that arrived with ttl=1"
);
cleanup_nodes(&mut nodes).await;
}
#[tokio::test]
async fn test_forwarding_no_route_generates_error() {
// 2-node network: 0 -- 1
src/node/tests/mod.rs
-2
View File
@@ -13,8 +13,6 @@ mod bootstrap;
mod decrypt_failure;
mod disconnect;
mod discovery;
#[cfg(target_os = "linux")]
mod ethernet;
mod forwarding;
mod handshake;
mod heartbeat;
src/node/tests/routing.rs
+345
View File
@@ -1110,3 +1110,348 @@ async fn test_routing_source_only_coords_100_nodes() {
cleanup_nodes(&mut nodes).await;
}
// === Route-class classification (transit-forward partition) ===
use crate::node::metrics::{ForwardingMetrics, RouteClass};
/// Current epoch millis, matching the cache-insert idiom used above.
fn now_ms() -> u64 {
std::time::SystemTime::now()
.duration_since(std::time::UNIX_EPOCH)
.map(|d| d.as_millis() as u64)
.unwrap_or(0)
}
#[test]
fn test_classify_forward_tree_up() {
// my_coords = [me, parent, root]; the chosen peer is our parent (an
// ancestor in our path) → tree-up.
let mut node = make_node();
let me = *node.node_addr();
let parent = make_node_addr(10);
let root = make_node_addr(1);
node.tree_state_mut()
.set_my_coords_for_test(TreeCoordinate::from_addrs(vec![me, parent, root]).unwrap());
// Destination somewhere above us; routed via the parent.
let dest = make_node_addr(50);
node.coord_cache_mut().insert(
dest,
TreeCoordinate::from_addrs(vec![dest, root]).unwrap(),
now_ms(),
);
assert_eq!(
node.classify_forward(&dest, &parent),
RouteClass::TreeUp,
"chosen peer is our ancestor"
);
}
#[test]
fn test_classify_forward_tree_down() {
// Chosen peer is our descendant: its coords name us as an ancestor.
let mut node = make_node();
let me = *node.node_addr();
let root = make_node_addr(1);
node.tree_state_mut()
.set_my_coords_for_test(TreeCoordinate::from_addrs(vec![me, root]).unwrap());
let child = make_node_addr(20);
node.tree_state_mut().update_peer(
ParentDeclaration::new(child, me, 1, 1000),
TreeCoordinate::from_addrs(vec![child, me, root]).unwrap(),
);
// Destination below the child; routed down to it.
let dest = make_node_addr(60);
node.coord_cache_mut().insert(
dest,
TreeCoordinate::from_addrs(vec![dest, child, me, root]).unwrap(),
now_ms(),
);
assert_eq!(
node.classify_forward(&dest, &child),
RouteClass::TreeDown,
"chosen peer is our descendant, dest in its subtree"
);
}
#[test]
fn test_classify_forward_tree_down_cross() {
// Chosen peer is our descendant (a tree child), but the destination is NOT
// in that child's subtree: we are diving down to the child only because it
// advertised cross-link reach upward, beyond its own subtree. This is the
// dive-to-tree-child cut-through.
let mut node = make_node();
let me = *node.node_addr();
let root = make_node_addr(1);
node.tree_state_mut()
.set_my_coords_for_test(TreeCoordinate::from_addrs(vec![me, root]).unwrap());
let child = make_node_addr(20);
node.tree_state_mut().update_peer(
ParentDeclaration::new(child, me, 1, 1000),
TreeCoordinate::from_addrs(vec![child, me, root]).unwrap(),
);
// Destination lives elsewhere (directly under root), NOT under the child;
// reachable from the child only via a cross-link.
let dest = make_node_addr(60);
node.coord_cache_mut().insert(
dest,
TreeCoordinate::from_addrs(vec![dest, root]).unwrap(),
now_ms(),
);
assert_eq!(
node.classify_forward(&dest, &child),
RouteClass::TreeDownCross,
"descendant peer, dest not in its subtree (dive-to-tree-child cut-through)"
);
}
#[test]
fn test_classify_forward_crosslink_descend() {
// Chosen peer is lateral (not in our path, we are not in its path) and the
// destination is inside the peer's subtree → cross-link descend.
let mut node = make_node();
let me = *node.node_addr();
let root = make_node_addr(1);
let sibling_parent = make_node_addr(2);
node.tree_state_mut()
.set_my_coords_for_test(TreeCoordinate::from_addrs(vec![me, root]).unwrap());
let peer = make_node_addr(30);
node.tree_state_mut().update_peer(
ParentDeclaration::new(peer, sibling_parent, 1, 1000),
TreeCoordinate::from_addrs(vec![peer, sibling_parent, root]).unwrap(),
);
// Destination is under the cross-link peer.
let dest = make_node_addr(70);
node.coord_cache_mut().insert(
dest,
TreeCoordinate::from_addrs(vec![dest, peer, sibling_parent, root]).unwrap(),
now_ms(),
);
assert_eq!(
node.classify_forward(&dest, &peer),
RouteClass::CrosslinkDescend,
"lateral peer, dest in its subtree"
);
}
#[test]
fn test_classify_forward_crosslink_ascend() {
// Chosen peer is lateral and the destination is NOT in its subtree → the
// up-and-over case (the Bloom v2 behavior delta).
let mut node = make_node();
let me = *node.node_addr();
let root = make_node_addr(1);
let sibling_parent = make_node_addr(2);
node.tree_state_mut()
.set_my_coords_for_test(TreeCoordinate::from_addrs(vec![me, root]).unwrap());
let peer = make_node_addr(40);
node.tree_state_mut().update_peer(
ParentDeclaration::new(peer, sibling_parent, 1, 1000),
TreeCoordinate::from_addrs(vec![peer, sibling_parent, root]).unwrap(),
);
// Destination lives elsewhere (under root directly), NOT under the peer.
let dest = make_node_addr(80);
node.coord_cache_mut().insert(
dest,
TreeCoordinate::from_addrs(vec![dest, root]).unwrap(),
now_ms(),
);
assert_eq!(
node.classify_forward(&dest, &peer),
RouteClass::CrosslinkAscend,
"lateral peer, dest not in its subtree"
);
}
#[test]
fn test_classify_forward_direct_peer() {
// Degenerate case: the next hop is the destination itself.
let mut node = make_node();
let me = *node.node_addr();
let root = make_node_addr(1);
node.tree_state_mut()
.set_my_coords_for_test(TreeCoordinate::from_addrs(vec![me, root]).unwrap());
let dest = make_node_addr(90);
assert_eq!(
node.classify_forward(&dest, &dest),
RouteClass::DirectPeer,
"next hop is the destination"
);
}
#[test]
fn test_route_class_partition_sums_to_forwarded() {
// The six route classes partition forwarded_packets: bumping
// record_forwarded once per record_route_class keeps the sum of the class
// counters equal to forwarded_packets.
let m = ForwardingMetrics::default();
let classes = [
RouteClass::TreeUp,
RouteClass::TreeUp,
RouteClass::TreeDown,
RouteClass::TreeDownCross,
RouteClass::TreeDownCross,
RouteClass::CrosslinkDescend,
RouteClass::CrosslinkAscend,
RouteClass::CrosslinkAscend,
RouteClass::CrosslinkAscend,
RouteClass::DirectPeer,
];
for &c in &classes {
m.record_forwarded(100);
m.record_route_class(c);
}
let snap = m.snapshot();
let class_sum = snap.route_tree_up
+ snap.route_tree_down
+ snap.route_tree_down_cross
+ snap.route_crosslink_descend
+ snap.route_crosslink_ascend
+ snap.route_direct_peer;
assert_eq!(
class_sum, snap.forwarded_packets,
"route classes must partition forwarded_packets"
);
assert_eq!(snap.route_tree_up, 2);
assert_eq!(snap.route_tree_down_cross, 2);
assert_eq!(snap.route_crosslink_ascend, 3);
assert_eq!(snap.route_direct_peer, 1);
}
// === Coord-cache invalidation on parent loss ===
//
// Parent-lost-via-peer-removal is a genuine position change and must
// surgically invalidate the coordinate cache like every other such path
// (reparent → invalidate_via_node; self-root → invalidate_other_roots).
// `make_node_addr(0)` is the network minimum, so the node's random identity
// addr is always greater than it — the reparent/child geometry is deterministic.
#[test]
fn test_parent_loss_reparent_invalidates_coord_cache() {
let mut node = make_node();
let my_addr = *node.node_addr();
let root = make_node_addr(0);
let parent = make_node_addr(1);
let alt = make_node_addr(2);
// Current parent and an alternative, both rooted at `root`.
node.tree_state_mut().update_peer(
ParentDeclaration::new(parent, root, 1, 1000),
TreeCoordinate::from_addrs(vec![parent, root]).unwrap(),
);
node.tree_state_mut().update_peer(
ParentDeclaration::new(alt, root, 1, 1000),
TreeCoordinate::from_addrs(vec![alt, root]).unwrap(),
);
// Adopt `parent`; our coords become [my_addr, parent, root], root = `root`.
node.tree_state_mut().set_parent(parent, 1, 1000);
node.tree_state_mut().recompute_coords();
assert!(!node.tree_state().is_root());
assert_eq!(node.tree_state().root(), &root);
let now_ms = std::time::SystemTime::now()
.duration_since(std::time::UNIX_EPOCH)
.map(|d| d.as_millis() as u64)
.unwrap_or(0);
// via-node class: a downstream destination that routes through us.
let downstream = make_node_addr(10);
node.coord_cache_mut().insert(
downstream,
TreeCoordinate::from_addrs(vec![downstream, my_addr, root]).unwrap(),
now_ms,
);
// survivor: same root, does not route through us.
let sibling_dest = make_node_addr(11);
node.coord_cache_mut().insert(
sibling_dest,
TreeCoordinate::from_addrs(vec![sibling_dest, alt, root]).unwrap(),
now_ms,
);
// Parent link drops; node reparents onto `alt` (still rooted at `root`).
let changed = node.handle_peer_removal_tree_cleanup(&parent);
assert!(changed);
assert_eq!(node.tree_state().my_declaration().parent_id(), &alt);
assert_eq!(node.tree_state().root(), &root);
assert!(
!node.coord_cache().contains(&downstream, now_ms),
"entry routing through us must be invalidated after reparent"
);
assert!(
node.coord_cache().contains(&sibling_dest, now_ms),
"same-root entry not routing through us must survive (surgical, not a flush)"
);
}
#[test]
fn test_parent_loss_selfroot_invalidates_coord_cache() {
let mut node = make_node();
let my_addr = *node.node_addr();
let old_root = make_node_addr(0);
let parent = make_node_addr(1);
// Adopt `parent` (rooted at `old_root`); no alternative peers exist, so a
// parent loss self-roots the node.
node.tree_state_mut().update_peer(
ParentDeclaration::new(parent, old_root, 1, 1000),
TreeCoordinate::from_addrs(vec![parent, old_root]).unwrap(),
);
node.tree_state_mut().set_parent(parent, 1, 1000);
node.tree_state_mut().recompute_coords();
assert!(!node.tree_state().is_root());
let now_ms = std::time::SystemTime::now()
.duration_since(std::time::UNIX_EPOCH)
.map(|d| d.as_millis() as u64)
.unwrap_or(0);
// via-node class: routes through us.
let downstream = make_node_addr(10);
node.coord_cache_mut().insert(
downstream,
TreeCoordinate::from_addrs(vec![downstream, my_addr, old_root]).unwrap(),
now_ms,
);
// other-roots class: on the old root, does not route through us.
let foreign = make_node_addr(11);
node.coord_cache_mut().insert(
foreign,
TreeCoordinate::from_addrs(vec![foreign, parent, old_root]).unwrap(),
now_ms,
);
// Parent link drops; no alternative parent → node self-roots.
let changed = node.handle_peer_removal_tree_cleanup(&parent);
assert!(changed);
assert!(node.tree_state().is_root());
assert_eq!(node.tree_state().root(), &my_addr);
assert!(
!node.coord_cache().contains(&downstream, now_ms),
"via-node entry must be invalidated after self-root"
);
assert!(
!node.coord_cache().contains(&foreign, now_ms),
"stale old-root entry must be invalidated after self-root"
);
}
src/node/tests/unit.rs
+170
View File
@@ -1707,6 +1707,93 @@ async fn process_pending_retries_gated_at_capacity() {
);
}
/// A TCP listener that accepts connections and then never speaks. A relay
/// URL pointed at it makes the nostr client's websocket handshake hang, so
/// `refetch_advert_for_stale_check` burns its full 2s fetch timeout without
/// any network egress.
fn spawn_blackhole_relay() -> String {
use std::net::TcpListener;
let listener = TcpListener::bind("127.0.0.1:0").expect("bind blackhole listener");
let port = listener.local_addr().expect("blackhole local addr").port();
std::thread::spawn(move || {
let mut held = Vec::new();
while let Ok((stream, _)) = listener.accept() {
held.push(stream);
}
});
format!("ws://127.0.0.1:{port}")
}
/// The per-tick retry loop must not await the pre-dial advert refetch.
///
/// `process_pending_retries` runs inline on the node's 1s rx-loop tick. Each
/// due peer's refetch carries a 2s relay-fetch timeout, so awaiting it stalls
/// the whole tick by 2s per peer — up to `MAX_RETRY_CONNECTIONS_PER_TICK`
/// times in one tick body. The refresh is fire-and-forget: it exists to make
/// the *next* retry dial a fresh endpoint, and retries are backoff-paced.
///
/// Discriminator: wall-clock duration of one `process_pending_retries` call
/// with several due peers whose refetches all hang. Awaited, the call takes
/// `2s * peers`; spawned, it returns without waiting on any of them.
#[tokio::test]
async fn process_pending_retries_does_not_await_advert_refetch() {
use std::time::Instant;
const DUE_PEERS: usize = 4;
// Awaited: >= 8s (4 x 2s). Spawned: milliseconds. A 3s bound sits far
// from both, so neither machine load nor the 2s timeout's own slack can
// flip the verdict.
const MAX_TICK_MS: u128 = 3_000;
let mut node = make_node_with_max_peers(64);
let mut bootstrap = NostrDiscovery::new_for_test();
bootstrap
.set_advert_relays_for_test(vec![spawn_blackhole_relay()])
.await;
node.nostr_discovery = Some(Arc::new(bootstrap));
let mut queued = Vec::new();
for _ in 0..DUE_PEERS {
let peer_npub = Identity::generate().npub();
let peer_node_addr = *PeerIdentity::from_npub(&peer_npub).unwrap().node_addr();
let mut state = super::super::retry::RetryState::new(crate::config::PeerConfig::new(
peer_npub,
"udp",
"127.0.0.1:9",
));
state.retry_after_ms = 0;
state.reconnect = true;
node.retry_pending.insert(peer_node_addr, state);
queued.push(peer_node_addr);
}
let started = Instant::now();
node.process_pending_retries(1_000).await;
let elapsed = started.elapsed();
assert!(
elapsed.as_millis() < MAX_TICK_MS,
"retry tick must not block on the advert refetch: took {}ms for {} due peers \
(a per-peer 2s relay-fetch timeout awaited inline is the fingerprint)",
elapsed.as_millis(),
DUE_PEERS
);
// The rest of the loop body is unchanged: every due peer was still
// attempted, failed for want of a transport, and was rescheduled.
for addr in &queued {
let state = node
.retry_pending
.get(addr)
.expect("due peer must remain queued after a failed attempt");
assert_eq!(
state.retry_count, 1,
"each due peer must still have been attempted and rescheduled"
);
}
}
#[tokio::test]
async fn poll_nostr_discovery_established_gated_at_capacity() {
use crate::discovery::EstablishedTraversal;
@@ -1995,3 +2082,86 @@ async fn handle_msg1_admits_existing_peer_at_cap() {
"rate limiter must rebalance after the (bypass-admitted) handler returns"
);
}
// ===== Transport kernel-drop detection (sans-IO) =====
//
// The drop-detection edge-detector, tested directly. It replaces the
// congestion-drops docker scenario, which could not provoke SO_RXQ_OVFL
// deterministically (a fresh daemon reader keeps up with container-speed
// traffic, so the kernel never overflows the socket queue). The kernel
// dropping datagrams is not FIPS behaviour to test; the FIPS behaviour is
// reading the SO_RXQ_OVFL counter and firing kernel_drop_events on the
// transition into a new drop burst, which is exactly this decision.
#[test]
fn test_transport_drop_state_fires_on_edge_and_rearms() {
let mut s = TransportDropState::default();
// Cumulative counter still 0: no rise, no event.
assert!(!s.observe_drops(0));
// First rise (0 -> 5): a new drop burst is observed, so it fires.
assert!(s.observe_drops(5));
// Counter keeps rising (5 -> 9) but we are already dropping: this is
// the "first observed" contract, so it must NOT fire again.
assert!(!s.observe_drops(9));
// A sample with no further rise clears the dropping flag (no event).
assert!(!s.observe_drops(9));
// A later rise (9 -> 12) is a fresh burst and fires again.
assert!(s.observe_drops(12));
}
#[test]
fn test_transport_drop_state_steady_counter_fires_once() {
let mut s = TransportDropState::default();
// A cumulative counter that jumps once and then holds steady must
// register exactly one event, not one per sample — otherwise a single
// historical drop burst would report congestion forever.
assert!(s.observe_drops(7));
assert!(!s.observe_drops(7));
assert!(!s.observe_drops(7));
}
#[test]
fn test_peer_display_name_uses_cached_short_npub() {
// Path 3 of `peer_display_name` (no host entry, no alias) reads the
// per-peer cached short npub; it must still equal the value derived
// from the peer's identity.
let mut node = make_node();
let peer_identity_full = Identity::generate();
let peer_addr = *peer_identity_full.node_addr();
let peer_identity = PeerIdentity::from_pubkey(peer_identity_full.pubkey());
node.peers
.insert(peer_addr, ActivePeer::new(peer_identity, LinkId::new(1), 0));
assert_eq!(
node.peer_display_name(&peer_addr),
peer_identity.short_npub()
);
}
#[test]
fn test_peer_display_name_tracks_alias_change() {
// The display name is NOT cached on the peer: `peer_aliases` is a
// runtime-mutable map (`update_peers` inserts and removes entries), so
// a cached name would go stale. Caching only the immutable short npub
// must leave that tracking intact.
let mut node = make_node();
let peer_identity_full = Identity::generate();
let peer_addr = *peer_identity_full.node_addr();
let peer_identity = PeerIdentity::from_pubkey(peer_identity_full.pubkey());
node.peers
.insert(peer_addr, ActivePeer::new(peer_identity, LinkId::new(1), 0));
assert_eq!(
node.peer_display_name(&peer_addr),
peer_identity.short_npub()
);
node.peer_aliases.insert(peer_addr, "gateway".to_string());
assert_eq!(node.peer_display_name(&peer_addr), "gateway");
node.peer_aliases.remove(&peer_addr);
assert_eq!(
node.peer_display_name(&peer_addr),
peer_identity.short_npub()
);
}
src/node/tree.rs
+10 -4
View File
@@ -300,7 +300,6 @@ impl Node {
.invalidate_via_node(our_identity.node_addr());
self.reset_discovery_backoff();
self.metrics().tree.parent_switched.inc();
self.metrics().tree.parent_switches.inc();
info!(
@@ -338,7 +337,6 @@ impl Node {
self.coord_cache
.invalidate_other_roots(our_identity.node_addr());
self.reset_discovery_backoff();
self.metrics().tree.parent_switched.inc();
self.metrics().tree.parent_switches.inc();
info!(
new_root = %self.tree_state.root(),
@@ -524,7 +522,6 @@ impl Node {
.invalidate_via_node(our_identity.node_addr());
self.reset_discovery_backoff();
self.metrics().tree.parent_switched.inc();
self.metrics().tree.parent_switches.inc();
info!(
@@ -560,7 +557,6 @@ impl Node {
self.coord_cache
.invalidate_other_roots(our_identity.node_addr());
self.reset_discovery_backoff();
self.metrics().tree.parent_switched.inc();
self.metrics().tree.parent_switches.inc();
info!(
new_root = %self.tree_state.root(),
@@ -620,6 +616,16 @@ impl Node {
.tree
.record_reject(TreeReject::OutboundSignFailed);
}
// handle_parent_lost may promote to root OR find new parent;
// cover both invalidation classes (same as the loop-detection
// branch above). Without this, cached downstream entries keep
// our now-stale coordinate prefix until TTL — and get_and_touch
// refreshes the TTL on every routing access, so an actively
// routed stale entry never self-expires.
self.coord_cache
.invalidate_via_node(our_identity.node_addr());
self.coord_cache
.invalidate_other_roots(self.tree_state.root());
info!(
new_root = %self.tree_state.root(),
is_root = self.tree_state.is_root(),
src/peer/active.rs
+82 -1
View File
@@ -81,6 +81,16 @@ pub struct ActivePeer {
// === Identity (Verified) ===
/// Cryptographic identity (verified via handshake).
identity: PeerIdentity,
/// Bech32 npub, derived once at construction.
///
/// The npub is a pure function of `identity`'s public key, and
/// `identity` is never mutated after construction, so this can never
/// go stale. Deriving it costs a bech32 encode, which the per-tick
/// stats snapshot was paying once per peer per tick.
npub: String,
/// Shortened npub for log/UI display, derived once at construction.
/// Immutable for the same reason as [`ActivePeer::npub`].
short_npub: String,
// === Connection ===
/// Link used to reach this peer.
@@ -224,6 +234,8 @@ impl ActivePeer {
pub fn new(identity: PeerIdentity, link_id: LinkId, authenticated_at: u64) -> Self {
let now = Instant::now();
Self {
npub: identity.npub(),
short_npub: identity.short_npub(),
identity,
link_id,
connectivity: ConnectivityState::Connected,
@@ -310,6 +322,8 @@ impl ActivePeer {
) -> Self {
let now = Instant::now();
Self {
npub: identity.npub(),
short_npub: identity.short_npub(),
identity,
link_id,
connectivity: ConnectivityState::Connected,
@@ -423,8 +437,21 @@ impl ActivePeer {
}
/// Get the peer's npub string.
///
/// Returns a clone of the value cached at construction; the bech32
/// encode is not repeated.
pub fn npub(&self) -> String {
self.identity.npub()
self.npub.clone()
}
/// Borrow the peer's cached npub without allocating.
pub fn npub_str(&self) -> &str {
&self.npub
}
/// Borrow the peer's cached shortened npub (e.g. `npub1abcd...wxyz`).
pub fn short_npub(&self) -> &str {
&self.short_npub
}
// === Connection Accessors ===
@@ -1219,6 +1246,60 @@ mod tests {
assert!(peer.needs_filter_update()); // New peers need filter
}
#[test]
fn test_npub_cache_matches_identity() {
let identity = make_peer_identity();
let peer = ActivePeer::new(identity, LinkId::new(1), 1000);
assert_eq!(peer.npub(), identity.npub());
assert_eq!(peer.npub_str(), identity.npub());
assert_eq!(peer.short_npub(), identity.short_npub());
}
#[test]
fn test_npub_cache_matches_identity_with_session() {
// `with_session` builds its own struct literal, so it needs its
// own check that the cache is populated from the same identity.
let identity = make_peer_identity();
let (session, _peer_session) = ik_session_pair();
let peer = ActivePeer::with_session(
identity,
LinkId::new(1),
1000,
session,
SessionIndex::new(1),
SessionIndex::new(2),
TransportId::new(1),
TransportAddr::from_string("127.0.0.1:9000"),
LinkStats::new(),
true,
&MmpConfig::default(),
None,
);
assert_eq!(peer.npub(), identity.npub());
assert_eq!(peer.short_npub(), identity.short_npub());
}
#[test]
fn test_npub_is_memoized_not_rederived() {
// The whole point of the fix: the strings are stored on the peer,
// not recomputed per call. A stored string keeps one heap buffer,
// so repeated borrows have a stable address. A per-call bech32
// encode would hand back a fresh allocation each time.
let identity = make_peer_identity();
let peer = ActivePeer::new(identity, LinkId::new(1), 1000);
let first = peer.npub_str().as_ptr();
let second = peer.npub_str().as_ptr();
assert_eq!(first, second);
let short_first = peer.short_npub().as_ptr();
let short_second = peer.short_npub().as_ptr();
assert_eq!(short_first, short_second);
}
#[test]
fn test_connectivity_transitions() {
let identity = make_peer_identity();
src/protocol/link.rs
+57 -9
View File
@@ -337,19 +337,27 @@ impl SessionDatagram {
self
}
/// Decrement TTL, returning false if exhausted.
/// Decrement the TTL for a transit hop, returning whether the result may
/// still be transmitted.
///
/// Follows IP semantics: the decrement happens first, and a datagram that
/// would leave with a TTL of zero is not transmitted. `saturating_sub`
/// folds an already-exhausted arrival (TTL 0) into the same outcome as a
/// last-hop arrival (TTL 1); both leave `ttl` at 0 and return false.
///
/// This governs forwarding only. Delivery to the addressed node is not
/// TTL-gated and must not consult this method.
pub fn decrement_ttl(&mut self) -> bool {
if self.ttl > 0 {
self.ttl -= 1;
true
} else {
false
}
self.ttl = self.ttl.saturating_sub(1);
self.ttl > 0
}
/// Check if the datagram can be forwarded.
/// Check whether this datagram would survive a transit hop.
///
/// True only at TTL 2 or more: at TTL 1 the decrement leaves zero, so the
/// datagram is dropped rather than forwarded.
pub fn can_forward(&self) -> bool {
self.ttl > 0
self.ttl > 1
}
/// Encode as link-layer message (msg_type + ttl + path_mtu + src_addr + dest_addr + payload).
@@ -675,4 +683,44 @@ mod tests {
assert_eq!(decoded.ttl, hop);
}
}
#[test]
fn test_session_datagram_can_forward() {
let dg = SessionDatagram::new(make_node_addr(1), make_node_addr(2), vec![0x42]);
assert!(!dg.clone().with_ttl(0).can_forward());
assert!(
!dg.clone().with_ttl(1).can_forward(),
"ttl=1 would leave at zero, so it is not forwardable"
);
assert!(
dg.clone().with_ttl(2).can_forward(),
"ttl=2 leaves at one, so it is forwardable"
);
assert!(dg.with_ttl(255).can_forward());
}
#[test]
fn test_session_datagram_decrement_ttl() {
let base = SessionDatagram::new(make_node_addr(1), make_node_addr(2), vec![0x42]);
let mut dg = base.clone().with_ttl(0);
assert!(!dg.decrement_ttl(), "ttl=0 is already exhausted");
assert_eq!(dg.ttl, 0, "decrement must saturate rather than wrap");
let mut dg = base.clone().with_ttl(1);
assert!(
!dg.decrement_ttl(),
"ttl=1 leaves at zero, so it is dropped"
);
assert_eq!(dg.ttl, 0);
let mut dg = base.clone().with_ttl(2);
assert!(dg.decrement_ttl());
assert_eq!(dg.ttl, 1);
let mut dg = base.with_ttl(64);
assert!(dg.decrement_ttl());
assert_eq!(dg.ttl, 63);
}
}
src/transport/udp/socket.rs
+1 -1
View File
@@ -853,7 +853,7 @@ mod tests {
/// find N packets already buffered, and one syscall reaps the burst.
#[cfg(any(target_os = "linux", target_os = "macos"))]
#[tokio::test(flavor = "multi_thread", worker_threads = 2)]
#[ignore]
#[ignore = "microbenchmark; run explicitly with --ignored --nocapture"]
async fn bench_udp_recv_amortization() {
use std::sync::Arc;
use std::sync::atomic::{AtomicBool, Ordering};
src/tree/state.rs
+9
View File
@@ -92,6 +92,15 @@ impl TreeState {
&self.my_coords
}
/// Test-only override of this node's coordinates, bypassing the
/// parent/declaration state machine. Lets routing tests place the node at
/// an arbitrary tree position to exercise coordinate-based classification.
#[cfg(test)]
pub(crate) fn set_my_coords_for_test(&mut self, coords: TreeCoordinate) {
self.root = *coords.root_id();
self.my_coords = coords;
}
/// Get the current root.
pub fn root(&self) -> &NodeAddr {
&self.root
src/tree/tests.rs
+166
View File
@@ -536,6 +536,172 @@ fn test_evaluate_parent_picks_loop_free_over_loopy() {
assert_eq!(result, Some(peer2));
}
// ===== Cost-based parent selection =====
//
// These exercise evaluate_parent's MMP-cost path directly, as a sans-IO
// unit test of the exact decision. They replace six Docker chaos
// scenarios — cost-reeval, cost-avoidance, cost-stability, depth-vs-cost,
// mixed-technology and bottleneck-parent — whose subject was this
// decision but which could not test it reliably: the mesh's root is
// whichever node holds the smallest NodeAddr, MMP costs take several
// measurement windows to settle, and hold-down plus hysteresis timing all
// confounded the assertion. Here the peer ancestry, depths and costs are
// constructed directly, so the decision is deterministic and each check
// can fail on a real regression.
#[test]
fn test_evaluate_parent_cost_prefers_cheaper_link_at_equal_depth() {
// mixed-technology / cost-avoidance subject: two candidate parents at
// the SAME depth, one over a cheap (fiber) link and one over an
// expensive (Bluetooth) link. The cheaper link must win.
//
// The cheap peer is given the LARGER NodeAddr on purpose: with cost
// ignored the two candidates tie on depth and the NodeAddr tiebreak
// would pick the expensive, smaller-addr peer. Only a cost-aware
// decision picks the cheaper, larger-addr one, so the assertion
// discriminates.
let my_node = make_node_addr(5);
let mut state = TreeState::new(my_node);
let expensive = make_node_addr(2); // smaller addr, high cost (Bluetooth)
let cheap = make_node_addr(3); // larger addr, low cost (fiber)
let root = make_node_addr(0);
state.update_peer(
ParentDeclaration::new(expensive, root, 1, 1000),
make_coords(&[2, 0]), // depth 1
);
state.update_peer(
ParentDeclaration::new(cheap, root, 1, 1000),
make_coords(&[3, 0]), // depth 1
);
// eff_depth(expensive) = 1 + 4.0 = 5.0; eff_depth(cheap) = 1 + 1.0 = 2.0
let costs = HashMap::from([(expensive, 4.0_f64), (cheap, 1.0_f64)]);
assert_eq!(state.evaluate_parent(&costs), Some(cheap));
}
#[test]
fn test_evaluate_parent_cost_switches_when_link_to_parent_degrades() {
// cost-reeval subject: the node is parented to A over a cheap link;
// that link then degrades so the alternative B is strictly cheaper.
// Re-evaluation must switch to B. This is the periodic-reeval decision,
// taken here without any timer, netem or MMP-measurement latency.
let my_node = make_node_addr(5);
let mut state = TreeState::new(my_node);
let peer_a = make_node_addr(2);
let peer_b = make_node_addr(3);
let root = make_node_addr(0);
state.update_peer(
ParentDeclaration::new(peer_a, root, 1, 1000),
make_coords(&[2, 0]), // depth 1
);
state.update_peer(
ParentDeclaration::new(peer_b, root, 1, 1000),
make_coords(&[3, 0]), // depth 1
);
// Adopt A as parent (both links cheap at first).
state.set_parent(peer_a, 1, 1000);
state.recompute_coords();
assert!(!state.is_root());
// A's link degrades: eff(A) = 1 + 5.0 = 6.0, eff(B) = 1 + 1.0 = 2.0.
// Default hysteresis is zero, so the strictly-cheaper B wins.
let costs = HashMap::from([(peer_a, 5.0_f64), (peer_b, 1.0_f64)]);
assert_eq!(state.evaluate_parent(&costs), Some(peer_b));
}
#[test]
fn test_evaluate_parent_hysteresis_suppresses_marginal_cost_change() {
// cost-stability subject: a cost change smaller than the hysteresis
// band must NOT trigger a reparent. This is the property the scenario
// was named for and could only approximate with a switch-count ceiling.
let my_node = make_node_addr(5);
let mut state = TreeState::new(my_node);
state.set_parent_hysteresis(0.2);
let peer_a = make_node_addr(2);
let peer_b = make_node_addr(3);
let root = make_node_addr(0);
state.update_peer(
ParentDeclaration::new(peer_a, root, 1, 1000),
make_coords(&[2, 0]),
);
state.update_peer(
ParentDeclaration::new(peer_b, root, 1, 1000),
make_coords(&[3, 0]),
);
state.set_parent(peer_a, 1, 1000);
state.recompute_coords();
// eff(A) = 1 + 1.0 = 2.0; eff(B) = 1 + 0.9 = 1.9. B is cheaper, but
// 1.9 is not below 2.0 * (1 - 0.2) = 1.6, so hysteresis holds the parent.
let costs = HashMap::from([(peer_a, 1.0_f64), (peer_b, 0.9_f64)]);
assert_eq!(state.evaluate_parent(&costs), None);
}
#[test]
fn test_evaluate_parent_hysteresis_allows_significant_cost_change() {
// cost-stability healthy-path companion: a change LARGER than the band
// must still switch, so the hysteresis test above is not passing merely
// because the node never reparents.
let my_node = make_node_addr(5);
let mut state = TreeState::new(my_node);
state.set_parent_hysteresis(0.2);
let peer_a = make_node_addr(2);
let peer_b = make_node_addr(3);
let root = make_node_addr(0);
state.update_peer(
ParentDeclaration::new(peer_a, root, 1, 1000),
make_coords(&[2, 0]),
);
state.update_peer(
ParentDeclaration::new(peer_b, root, 1, 1000),
make_coords(&[3, 0]),
);
state.set_parent(peer_a, 1, 1000);
state.recompute_coords();
// eff(A) = 2.0; eff(B) = 1 + 0.3 = 1.3 < 1.6 threshold → switch to B.
let costs = HashMap::from([(peer_a, 1.0_f64), (peer_b, 0.3_f64)]);
assert_eq!(state.evaluate_parent(&costs), Some(peer_b));
}
#[test]
fn test_evaluate_parent_effective_depth_weighs_depth_against_cost() {
// depth-vs-cost / bottleneck-parent subject: a shallow parent reached
// over an expensive (bottleneck) link versus a deeper parent over a
// cheap link. effective_depth = depth + link_cost decides, and here the
// deeper-but-cheaper path wins — the outcome the depth-vs-cost scenario
// named no falsifiable answer for.
let my_node = make_node_addr(5);
let mut state = TreeState::new(my_node);
let shallow = make_node_addr(2); // depth 1, bottleneck link
let deep = make_node_addr(3); // depth 3, cheap link
let root = make_node_addr(0);
state.update_peer(
ParentDeclaration::new(shallow, root, 1, 1000),
make_coords(&[2, 0]), // depth 1
);
state.update_peer(
ParentDeclaration::new(deep, make_node_addr(6), 1, 1000),
make_coords(&[3, 6, 7, 0]), // depth 3
);
// eff(shallow) = 1 + 3.0 = 4.0; eff(deep) = 3 + 0.5 = 3.5 → pick deep.
// A depth-only decision would take the shallow bottleneck instead.
let costs = HashMap::from([(shallow, 3.0_f64), (deep, 0.5_f64)]);
assert_eq!(state.evaluate_parent(&costs), Some(deep));
}
#[test]
fn test_handle_parent_lost_finds_alternative() {
let my_node = make_node_addr(5);
src/upper/tcp_mss.rs
+159 -38
View File
@@ -115,8 +115,8 @@ pub fn clamp_tcp_mss(ipv6_packet: &mut [u8], max_mss: u16) -> bool {
if current_mss > max_mss {
ipv6_packet[i + 2..i + 4].copy_from_slice(&max_mss.to_be_bytes());
// Recalculate TCP checksum
recalculate_tcp_checksum(ipv6_packet, tcp_start);
// Recompute the now-stale checksum over the rewritten header.
recalculate_l4_checksum(ipv6_packet);
modified = true;
}
@@ -129,41 +129,39 @@ pub fn clamp_tcp_mss(ipv6_packet: &mut [u8], max_mss: u16) -> bool {
modified
}
/// Recalculate TCP checksum after modifying the packet.
fn recalculate_tcp_checksum(ipv6_packet: &mut [u8], tcp_start: usize) {
// Zero out existing checksum
ipv6_packet[tcp_start + 16] = 0;
ipv6_packet[tcp_start + 17] = 0;
// Extract addresses
let src = &ipv6_packet[8..24];
let dst = &ipv6_packet[24..40];
// Get TCP segment length
/// Recalculate the TCP or UDP checksum (IPv6 pseudo-header + segment) in place.
///
/// Completes the checksum macOS leaves offloaded on hairpinned self-traffic, and
/// is reused by MSS clamping. No-op for other next-headers (e.g. ICMPv6) and for
/// malformed packets. Assumes the L4 header follows the 40-byte IPv6 header, as
/// all FIPS packets do.
pub fn recalculate_l4_checksum(ipv6_packet: &mut [u8]) {
if ipv6_packet.len() < 40 || ipv6_packet[0] >> 4 != 6 {
return;
}
let payload_len = u16::from_be_bytes([ipv6_packet[4], ipv6_packet[5]]) as usize;
let tcp_segment = &ipv6_packet[tcp_start..tcp_start + payload_len];
if payload_len == 0 || 40 + payload_len > ipv6_packet.len() {
return;
}
// Calculate checksum with pseudo-header
// Transport checksum field offset within the packet: TCP at 16, UDP at 6.
let proto = ipv6_packet[6];
let csum = match proto {
6 if payload_len >= TCP_HEADER_MIN_LEN => 40 + 16,
17 if payload_len >= 8 => 40 + 6,
_ => return,
};
ipv6_packet[csum] = 0;
ipv6_packet[csum + 1] = 0;
// Pseudo-header (src + dst are contiguous at 8..40), length, next header,
let mut sum: u32 = 0;
// Pseudo-header: source address
for chunk in src.chunks(2) {
for chunk in ipv6_packet[8..40].chunks(2) {
sum += u16::from_be_bytes([chunk[0], chunk[1]]) as u32;
}
// Pseudo-header: destination address
for chunk in dst.chunks(2) {
sum += u16::from_be_bytes([chunk[0], chunk[1]]) as u32;
}
// Pseudo-header: TCP length
sum += payload_len as u32;
// Pseudo-header: next header (TCP = 6)
sum += 6;
// TCP segment
for chunk in tcp_segment.chunks(2) {
sum += payload_len as u32 + proto as u32;
// then the transport segment itself (with the checksum field zeroed above).
for chunk in ipv6_packet[40..40 + payload_len].chunks(2) {
let value = if chunk.len() == 2 {
u16::from_be_bytes([chunk[0], chunk[1]])
} else {
@@ -171,15 +169,16 @@ fn recalculate_tcp_checksum(ipv6_packet: &mut [u8], tcp_start: usize) {
};
sum += value as u32;
}
// Fold 32-bit sum to 16 bits
while sum >> 16 != 0 {
sum = (sum & 0xffff) + (sum >> 16);
}
// One's complement
let checksum = !sum as u16;
ipv6_packet[tcp_start + 16..tcp_start + 18].copy_from_slice(&checksum.to_be_bytes());
// IPv6 forbids an all-zero UDP checksum; send 0xffff instead (TCP keeps 0).
let checksum = match (!sum as u16, proto) {
(0, 17) => 0xffff,
(c, _) => c,
};
ipv6_packet[csum..csum + 2].copy_from_slice(&checksum.to_be_bytes());
}
#[cfg(test)]
@@ -216,7 +215,7 @@ mod tests {
packet[tcp_start + 24] = 0;
// Calculate checksum
recalculate_tcp_checksum(&mut packet, tcp_start);
recalculate_l4_checksum(&mut packet);
packet
}
@@ -277,4 +276,126 @@ mod tests {
assert!(!modified);
}
// ========================================================================
// recalculate_l4_checksum — finish macOS's offloaded self-traffic
// checksums (the bug that left ACK/data/FIN segments undeliverable).
// ========================================================================
/// True if the packet's TCP/UDP checksum verifies (folded ones-complement
/// sum over pseudo-header + segment, including the checksum field, == 0xffff).
fn l4_checksum_valid(pkt: &[u8]) -> bool {
let payload_len = u16::from_be_bytes([pkt[4], pkt[5]]) as usize;
let mut sum: u32 = 0;
for chunk in pkt[8..40].chunks(2) {
sum += u16::from_be_bytes([chunk[0], chunk[1]]) as u32;
}
sum += payload_len as u32;
sum += pkt[6] as u32; // next header
for chunk in pkt[40..40 + payload_len].chunks(2) {
let v = if chunk.len() == 2 {
u16::from_be_bytes([chunk[0], chunk[1]])
} else {
u16::from_be_bytes([chunk[0], 0])
};
sum += v as u32;
}
while sum >> 16 != 0 {
sum = (sum & 0xffff) + (sum >> 16);
}
sum as u16 == 0xffff
}
const SELF_ADDR: [u8; 16] = [0xfd, 0x12, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0x55];
/// A self-addressed 20-byte TCP ACK (no options) carrying a deliberately
/// wrong checksum — the shape macOS hands us for hairpinned non-SYN traffic.
fn make_tcp_ack_packet() -> Vec<u8> {
let mut p = vec![0u8; 40 + 20];
p[0] = 0x60;
p[4..6].copy_from_slice(&20u16.to_be_bytes()); // payload length
p[6] = 6; // TCP
p[7] = 64;
p[8..24].copy_from_slice(&SELF_ADDR);
p[24..40].copy_from_slice(&SELF_ADDR);
let t = 40;
p[t..t + 2].copy_from_slice(&52097u16.to_be_bytes());
p[t + 2..t + 4].copy_from_slice(&9999u16.to_be_bytes());
p[t + 4..t + 8].copy_from_slice(&1000u32.to_be_bytes()); // seq
p[t + 8..t + 12].copy_from_slice(&2000u32.to_be_bytes()); // ack
p[t + 12] = 0x50; // data offset = 5 (20-byte header)
p[t + 13] = 0x10; // ACK
p[t + 14..t + 16].copy_from_slice(&2049u16.to_be_bytes()); // window
p[t + 16] = 0x8e; // bogus checksum (macOS pseudo-header partial)
p[t + 17] = 0xce;
p
}
#[test]
fn recompute_fixes_tcp_non_syn_checksum() {
let mut pkt = make_tcp_ack_packet();
assert!(
!l4_checksum_valid(&pkt),
"fixture should start with a bad checksum"
);
recalculate_l4_checksum(&mut pkt);
assert!(
l4_checksum_valid(&pkt),
"TCP checksum must verify after recompute"
);
}
#[test]
fn recompute_fixes_udp_checksum() {
let mut p = vec![0u8; 40 + 12]; // 8-byte UDP header + 4-byte payload
p[0] = 0x60;
p[4..6].copy_from_slice(&12u16.to_be_bytes());
p[6] = 17; // UDP
p[7] = 64;
p[8..24].copy_from_slice(&SELF_ADDR);
p[24..40].copy_from_slice(&SELF_ADDR);
let u = 40;
p[u..u + 2].copy_from_slice(&40000u16.to_be_bytes()); // src port
p[u + 2..u + 4].copy_from_slice(&5354u16.to_be_bytes()); // dst port
p[u + 4..u + 6].copy_from_slice(&12u16.to_be_bytes()); // UDP length
p[u + 6] = 0x8e; // bogus checksum
p[u + 7] = 0xce;
p[u + 8..u + 12].copy_from_slice(&[0xde, 0xad, 0xbe, 0xef]); // payload
assert!(!l4_checksum_valid(&p), "fixture should start invalid");
recalculate_l4_checksum(&mut p);
assert!(
l4_checksum_valid(&p),
"UDP checksum must verify after recompute"
);
assert_ne!(
&p[u + 6..u + 8],
&[0, 0],
"IPv6 UDP checksum must not be zero"
);
}
#[test]
fn recompute_is_noop_for_non_transport() {
// Next-header 59 (No Next Header): nothing to checksum.
let mut pkt = vec![0u8; 60];
pkt[0] = 0x60;
pkt[4..6].copy_from_slice(&20u16.to_be_bytes());
pkt[6] = 59;
let before = pkt.clone();
recalculate_l4_checksum(&mut pkt);
assert_eq!(pkt, before, "non-transport packet must be left untouched");
}
#[test]
fn recompute_ignores_truncated_packet() {
// payload_len claims 20 bytes of TCP but only 10 are present.
let mut pkt = vec![0u8; 40 + 10];
pkt[0] = 0x60;
pkt[4..6].copy_from_slice(&20u16.to_be_bytes());
pkt[6] = 6;
let before = pkt.clone();
recalculate_l4_checksum(&mut pkt); // must not panic
assert_eq!(pkt, before, "truncated packet must be left untouched");
}
}
src/upper/tun.rs
+107 -1
View File
@@ -693,7 +693,7 @@ fn handle_tun_packet(
path_mtu_lookup: &PathMtuLookup,
) -> bool {
use super::icmp::{DestUnreachableCode, build_dest_unreachable, should_send_icmp_error};
use super::tcp_mss::clamp_tcp_mss;
use super::tcp_mss::{clamp_tcp_mss, recalculate_l4_checksum};
log_ipv6_packet(packet);
@@ -704,6 +704,30 @@ fn handle_tun_packet(
// Check if destination is a FIPS address (fd::/8 prefix)
if packet[24] == crate::identity::FIPS_ADDRESS_PREFIX {
// Loopback: a packet to our own mesh address must be delivered
// locally, not pushed into the mesh (we have no session/route to
// ourselves, so it would just be dropped). Hairpin it back to the TUN
// writer for inbound delivery.
//
// Platform note: in practice this branch is reached only on macOS.
// macOS point-to-point `utun` interfaces egress self-addressed traffic
// down the tunnel into this reader, so the daemon has to loop it back
// itself. On Linux the kernel routes traffic to our own bound
// addresses via `lo` before it ever reaches the TUN, so this branch
// never fires there. The check is kept unconditional anyway, both as a
// platform-independent self-delivery invariant and so the path stays
// exercised by the Linux-only CI unit tests.
if packet[24..40] == *our_addr.as_bytes() {
trace!(name = %name, "Hairpinning self-addressed packet back to TUN (loopback)");
// Finish the checksum macOS leaves offloaded on self-traffic, else the
// local stack drops every non-SYN segment. See recalculate_l4_checksum.
recalculate_l4_checksum(packet);
if tun_tx.send(packet.to_vec()).is_err() {
return false; // Channel closed, shutdown
}
return true;
}
// Per-destination clamp: if discovery has learned a smaller path
// MTU for this destination, tighten the ceiling for this flow.
let effective_max_mss = per_flow_max_mss(path_mtu_lookup, &packet[24..40], max_mss);
@@ -1520,6 +1544,88 @@ mod tests {
assert_eq!(per_flow_max_mss(&lookup, b.as_bytes(), 1360), 1315);
}
// ========================================================================
// handle_tun_packet — self-addressed loopback hairpin
//
// A packet destined for our own mesh address must be delivered back to
// the local stack via the TUN writer, never pushed into the mesh (there
// is no session/route to ourselves). On macOS the kernel egresses such
// self-traffic down the utun into the reader, so the daemon has to loop
// it back itself.
// ========================================================================
/// Build a minimal 40-byte IPv6 packet (no upper-layer payload) addressed
/// to `dst`, sourced from a distinct fips address.
fn ipv6_packet_to(dst: &FipsAddress) -> Vec<u8> {
let mut pkt = vec![0u8; 40];
pkt[0] = 0x60; // version 6
pkt[6] = 59; // next header = No Next Header (skips MSS clamp)
pkt[7] = 64; // hop limit
pkt[8] = crate::identity::FIPS_ADDRESS_PREFIX; // src in fd::/8
pkt[24..40].copy_from_slice(dst.as_bytes()); // dst
pkt
}
#[test]
fn self_addressed_packet_is_hairpinned_to_tun() {
let our_addr = fips_addr_with_node_byte(0x55);
let (tun_tx, tun_rx) = mpsc::channel::<Vec<u8>>();
let (outbound_tx, mut outbound_rx) = tokio::sync::mpsc::channel::<Vec<u8>>(4);
let lookup = empty_lookup();
let mut pkt = ipv6_packet_to(&our_addr);
assert!(handle_tun_packet(
&mut pkt,
1360,
"test0",
our_addr,
&tun_tx,
&outbound_tx,
&lookup,
));
// Delivered locally via the TUN writer...
let looped = tun_rx
.try_recv()
.expect("self-addressed packet should be hairpinned to the TUN");
assert_eq!(&looped[24..40], our_addr.as_bytes());
// ...and never handed to the mesh.
assert!(
outbound_rx.try_recv().is_err(),
"self-addressed packet must not be pushed into the mesh"
);
}
#[test]
fn other_fips_packet_goes_to_mesh() {
let our_addr = fips_addr_with_node_byte(0x55);
let peer = fips_addr_with_node_byte(0x66);
let (tun_tx, tun_rx) = mpsc::channel::<Vec<u8>>();
let (outbound_tx, mut outbound_rx) = tokio::sync::mpsc::channel::<Vec<u8>>(4);
let lookup = empty_lookup();
let mut pkt = ipv6_packet_to(&peer);
assert!(handle_tun_packet(
&mut pkt,
1360,
"test0",
our_addr,
&tun_tx,
&outbound_tx,
&lookup,
));
// A non-self fips destination is routed into the mesh, not looped back.
assert!(
outbound_rx.try_recv().is_ok(),
"non-self fips destination should be sent to the mesh"
);
assert!(
tun_rx.try_recv().is_err(),
"non-self fips destination must not be hairpinned"
);
}
// ========================================================================
// macOS utun packet-info header (AF_INET6 4-byte big-endian prefix)
//
testing/README.md
+107 -2
View File
@@ -16,8 +16,6 @@ configurations.
| ----------- | ----- | --------- | -------------------------------- |
| mesh | 5 | UDP | Sparse mesh, 6 links, multi-hop |
| chain | 5 | UDP | Linear chain, max 4-hop paths |
| mesh-public | 5+1 | UDP | Mesh with external public node |
| tcp-chain | 3 | TCP | Linear chain over TCP (port 8443) |
| rekey | 5 | UDP | Rekey integration test topology |
### [tor/](tor/) -- Tor Transport Integration
@@ -72,3 +70,110 @@ and optional trace-level RUST_LOG, capturing per-rep diagnostics and a
mechanism-match summary across the run. Used for statistical reliability
characterization of known flake classes under calibrated stress, not as
a per-commit gate; not part of `ci-local.sh`.
## Running CI locally (`ci-local.sh`)
[`ci-local.sh`](ci-local.sh) runs the full local CI pipeline — build,
clippy, unit tests, and the integration suites (including the chaos
scenarios) — mirroring the GitHub `ci.yml` integration matrix. Run
`./ci-local.sh --help` for the full option list and `--list` for the
available suites. Every run starts with a parity check that verifies the
local suite set covers the same work as the GitHub matrix, per scenario for
chaos and per distro for deb-install; a divergence fails the run. GitHub
runs the same check as its own `ci-parity` job. `--check-parity` runs it
alone (see [check-ci-parity.sh](check-ci-parity.sh)).
### Per-run isolation and the `FIPS_CI_RUN_ID` override
Every invocation derives a **run id** and scopes all of its Docker
resources to it, so two simultaneous runs on the same host (for example,
one per git worktree, or an operator testing by hand while CI is in
flight) never collide:
- **Compose projects** are named `fipsci_<run-id>_<suite>`, so
container, network, and volume names are all prefixed per run.
- **Build images** are tagged `fips-test:<run-id>` and
`fips-test-app:<run-id>`, exported as `FIPS_TEST_IMAGE` /
`FIPS_TEST_APP_IMAGE`, and **every** compose file and suite script reads
those. The run does not write `fips-test:latest` at all: a bridge back to
that shared mutable name would let a consumer that had been missed keep
working while resolving whichever concurrent run wrote the tag last.
`:latest` stays the hand-build name, produced by
`testing/scripts/build.sh`, and remains the default every consumer falls
back to when the variables are unset.
- **The build context** is a per-run copy at `testing/docker-<run-id>/`,
exported as `FIPS_BUILD_CONTEXT`. It is absolute because compose resolves
a relative build context against the compose file's own directory rather
than the working directory. `testing/docker/` is the hand-run context and
a CI run does not write to it. Without this, two runs race on the contents
of one directory and either can build a correctly-per-run-tagged image
from the other's binaries.
- Each parallel chaos child gets a unique, non-overlapping `/24` in
`10.30.x` (via the sim `--subnet` override). `10.30.x` sits outside
Docker's default address pool and the fixed-subnet suites' `172.x`
ranges, so neither a sibling chaos child nor an auto-assigned network
can swallow a pinned subnet.
By default the run id is `<short-git-sha>-<random>` — the SHA portion
records *what code* a container is testing, the random suffix keeps
simultaneous runs of the same SHA disjoint. Override it for a
reproducible, attach-by-name debug session:
```sh
FIPS_CI_RUN_ID=mydebug ./ci-local.sh --only static-mesh
# containers are named fipsci_mydebug_static_fips-node-a, etc.
```
### Preemption-safety and exit codes
`ci-local.sh` is safe to cancel mid-run. A signal trap tears down *every*
compose project the run started (not just the current suite) and reaps
any in-flight parallel chaos children, bounded by a `timeout` so a stuck
`compose down` cannot wedge the trap. Exit codes distinguish a cancelled
run from a failing one:
| Code | Meaning |
| ---- | ------- |
| `0` | all stages passed |
| `1` | one or more stages failed |
| `130` | interrupted by SIGINT — cancelled, not a failure |
| `143` | terminated by SIGTERM — cancelled, not a failure |
A preempting CI worker (the push-triggered, CI-gated build pipeline that
kills an in-flight run when a newer same-branch tip arrives) maps
`130`/`143` → *cancelled* (discard, do not record a failing commit), `0`
→ green, any other non-zero → red.
### Cleaning up leftover resources
Every CI-created container, network, and volume carries the label
`com.corganlabs.fips-ci=1`. If a run is hard-killed (SIGKILL, OOM, crash)
and leaves resources behind, reap them with:
```sh
./ci-local.sh --reap # or: ./ci-cleanup.sh
```
[`ci-cleanup.sh`](ci-cleanup.sh) force-removes everything bearing the CI
label or a `fipsci_` compose-project prefix; it is safe to run when there
is nothing to reap and safe to run repeatedly. Pass `--project-prefix` to
scope the sweep to a single run.
It also removes the chaos simulation's leftover host-namespace veth
interfaces (`vh…a`/`vh…b`), the one resource it touches that is neither a
docker object nor labelled — a host interface can carry neither a label
nor a compose project, so it is matched by name shape alone. That makes
the reach here asymmetric with everything above, and worth stating
plainly:
- A bare `chaos.sh` run's **containers** survive a broad reap. Its
compose project is not `fipsci_`, and the simulation labels only the
network, not the services.
- A bare `chaos.sh` run's **veth interfaces do not.** An unscoped reap
deletes them while they are in use, severing the Ethernet links of a
live simulation and leaving its containers running.
So do not run a broad `--reap` while a bare simulation is up. Scope the
interface sweep with `--veth-suffixes` (which is what `ci-local.sh`'s own
teardown passes) or wait for the simulation to finish. `--project-prefix`
does not help here: it scopes only the compose-project sweep.
testing/acl-allowlist/.gitignore
+1 -1
View File
@@ -1 +1 @@
generated-configs
generated-configs*
testing/acl-allowlist/README.md
+9 -1
View File
@@ -79,6 +79,12 @@ Docker service/container/hostname identifiers in this harness intentionally use
`node-a` through `node-f`. For data-plane checks and operator examples, use the
explicit FIPS names such as `node-a.fips` and `node-d.fips`.
The bridge network requests no subnet, so docker assigns one from its own
address pool and two concurrent runs of this harness never contend for a fixed
range. Consequently no node's IPv4 address is known before startup, and the
generated peer stanzas address each other by docker hostname (`host-a` …
`host-f`), resolved through the container's dnsmasq to docker's embedded DNS.
ACL paths are fixed in this branch:
- `/etc/fips/peers.allow`
@@ -93,7 +99,9 @@ Mounted ACL files in this harness:
Generated fixture location:
- `testing/acl-allowlist/generated-configs/`
- `testing/acl-allowlist/generated-configs/`, or
`generated-configs<suffix>/` when `FIPS_CI_NAME_SUFFIX` is set, which is how
concurrent runs keep their fixtures apart
Inspect peer state:
testing/acl-allowlist/docker-compose.yml
+59 -49
View File
@@ -1,14 +1,30 @@
networks:
# No subnet is requested: docker assigns one from the daemon's address pool.
# A fixed request is honoured verbatim, so two runs asking for the same range
# collide on "Pool overlaps" — which is why the generated peer addresses are
# docker hostnames (host-a … host-f) rather than literal IPs. Docker's
# embedded DNS is per-network, so the same hostname in two concurrent runs
# resolves inside each run's own subnet.
#
# That removes one of the two obstacles to running this suite twice at once,
# not both. The compose project name is still fixed, so two runs that do not
# set COMPOSE_PROJECT_NAME share a project, and the second `up` recreates the
# first's containers while either `down` removes both. Nothing scopes it for
# this suite: it is run by hand only, so the caller has to. Do not read the
# floating subnet as making concurrent bare runs safe.
acl-net:
driver: bridge
ipam:
config:
- subnet: 172.31.0.0/24
labels:
- "com.corganlabs.fips-ci=1"
x-fips-common: &fips-common
build:
context: ../docker
image: fips-test:latest
# The harness scopes its build context per run and passes it here; the
# shared directory is the hand-run default. Compose resolves a relative
# value against THIS file's directory, so the harness must export an
# absolute path.
context: ${FIPS_BUILD_CONTEXT:-../docker}
image: ${FIPS_TEST_IMAGE:-fips-test:latest}
entrypoint: ["/usr/local/bin/entrypoint.sh"]
cap_add:
- NET_ADMIN
@@ -26,86 +42,80 @@ x-fips-common: &fips-common
services:
service-a:
<<: *fips-common
container_name: fips-acl-container-a
container_name: fips-acl-container-a${FIPS_CI_NAME_SUFFIX:-}
hostname: host-a
volumes:
- ../docker/resolv.conf:/etc/resolv.conf:ro
- ./generated-configs/node-a/hosts:/etc/fips/hosts:ro
- ./generated-configs/node-a/peers.allow:/etc/fips/peers.allow:ro
- ./generated-configs/node-a/peers.deny:/etc/fips/peers.deny:ro
- ./generated-configs/node-a/fips.yaml:/etc/fips/fips.yaml:ro
- ./generated-configs/node-a/fips.key:/etc/fips/fips.key:ro
- ./generated-configs${FIPS_CI_NAME_SUFFIX:-}/node-a/hosts:/etc/fips/hosts:ro
- ./generated-configs${FIPS_CI_NAME_SUFFIX:-}/node-a/peers.allow:/etc/fips/peers.allow:ro
- ./generated-configs${FIPS_CI_NAME_SUFFIX:-}/node-a/peers.deny:/etc/fips/peers.deny:ro
- ./generated-configs${FIPS_CI_NAME_SUFFIX:-}/node-a/fips.yaml:/etc/fips/fips.yaml:ro
- ./generated-configs${FIPS_CI_NAME_SUFFIX:-}/node-a/fips.key:/etc/fips/fips.key:ro
networks:
acl-net:
ipv4_address: 172.31.0.10
- acl-net
service-b:
<<: *fips-common
container_name: fips-acl-container-b
container_name: fips-acl-container-b${FIPS_CI_NAME_SUFFIX:-}
hostname: host-b
volumes:
- ../docker/resolv.conf:/etc/resolv.conf:ro
- ./generated-configs/node-b/hosts:/etc/fips/hosts:ro
- ./generated-configs/node-b/peers.allow:/etc/fips/peers.allow:ro
- ./generated-configs/node-b/peers.deny:/etc/fips/peers.deny:ro
- ./generated-configs/node-b/fips.yaml:/etc/fips/fips.yaml:ro
- ./generated-configs/node-b/fips.key:/etc/fips/fips.key:ro
- ./generated-configs${FIPS_CI_NAME_SUFFIX:-}/node-b/hosts:/etc/fips/hosts:ro
- ./generated-configs${FIPS_CI_NAME_SUFFIX:-}/node-b/peers.allow:/etc/fips/peers.allow:ro
- ./generated-configs${FIPS_CI_NAME_SUFFIX:-}/node-b/peers.deny:/etc/fips/peers.deny:ro
- ./generated-configs${FIPS_CI_NAME_SUFFIX:-}/node-b/fips.yaml:/etc/fips/fips.yaml:ro
- ./generated-configs${FIPS_CI_NAME_SUFFIX:-}/node-b/fips.key:/etc/fips/fips.key:ro
networks:
acl-net:
ipv4_address: 172.31.0.11
- acl-net
service-c:
<<: *fips-common
container_name: fips-acl-container-c
container_name: fips-acl-container-c${FIPS_CI_NAME_SUFFIX:-}
hostname: host-c
volumes:
- ../docker/resolv.conf:/etc/resolv.conf:ro
- ./generated-configs/node-c/hosts:/etc/fips/hosts:ro
- ./generated-configs/node-c/peers.allow:/etc/fips/peers.allow:ro
- ./generated-configs/node-c/peers.deny:/etc/fips/peers.deny:ro
- ./generated-configs/node-c/fips.yaml:/etc/fips/fips.yaml:ro
- ./generated-configs/node-c/fips.key:/etc/fips/fips.key:ro
- ./generated-configs${FIPS_CI_NAME_SUFFIX:-}/node-c/hosts:/etc/fips/hosts:ro
- ./generated-configs${FIPS_CI_NAME_SUFFIX:-}/node-c/peers.allow:/etc/fips/peers.allow:ro
- ./generated-configs${FIPS_CI_NAME_SUFFIX:-}/node-c/peers.deny:/etc/fips/peers.deny:ro
- ./generated-configs${FIPS_CI_NAME_SUFFIX:-}/node-c/fips.yaml:/etc/fips/fips.yaml:ro
- ./generated-configs${FIPS_CI_NAME_SUFFIX:-}/node-c/fips.key:/etc/fips/fips.key:ro
networks:
acl-net:
ipv4_address: 172.31.0.12
- acl-net
service-d:
<<: *fips-common
container_name: fips-acl-container-d
container_name: fips-acl-container-d${FIPS_CI_NAME_SUFFIX:-}
hostname: host-d
volumes:
- ../docker/resolv.conf:/etc/resolv.conf:ro
- ./generated-configs/node-d/hosts:/etc/fips/hosts:ro
- ./generated-configs/node-d/peers.allow:/etc/fips/peers.allow:ro
- ./generated-configs/node-d/peers.deny:/etc/fips/peers.deny:ro
- ./generated-configs/node-d/fips.yaml:/etc/fips/fips.yaml:ro
- ./generated-configs/node-d/fips.key:/etc/fips/fips.key:ro
- ./generated-configs${FIPS_CI_NAME_SUFFIX:-}/node-d/hosts:/etc/fips/hosts:ro
- ./generated-configs${FIPS_CI_NAME_SUFFIX:-}/node-d/peers.allow:/etc/fips/peers.allow:ro
- ./generated-configs${FIPS_CI_NAME_SUFFIX:-}/node-d/peers.deny:/etc/fips/peers.deny:ro
- ./generated-configs${FIPS_CI_NAME_SUFFIX:-}/node-d/fips.yaml:/etc/fips/fips.yaml:ro
- ./generated-configs${FIPS_CI_NAME_SUFFIX:-}/node-d/fips.key:/etc/fips/fips.key:ro
networks:
acl-net:
ipv4_address: 172.31.0.13
- acl-net
service-e:
<<: *fips-common
container_name: fips-acl-container-e
container_name: fips-acl-container-e${FIPS_CI_NAME_SUFFIX:-}
hostname: host-e
volumes:
- ../docker/resolv.conf:/etc/resolv.conf:ro
- ./generated-configs/node-e/hosts:/etc/fips/hosts:ro
- ./generated-configs/node-e/fips.yaml:/etc/fips/fips.yaml:ro
- ./generated-configs/node-e/fips.key:/etc/fips/fips.key:ro
- ./generated-configs${FIPS_CI_NAME_SUFFIX:-}/node-e/hosts:/etc/fips/hosts:ro
- ./generated-configs${FIPS_CI_NAME_SUFFIX:-}/node-e/fips.yaml:/etc/fips/fips.yaml:ro
- ./generated-configs${FIPS_CI_NAME_SUFFIX:-}/node-e/fips.key:/etc/fips/fips.key:ro
networks:
acl-net:
ipv4_address: 172.31.0.14
- acl-net
service-f:
<<: *fips-common
container_name: fips-acl-container-f
container_name: fips-acl-container-f${FIPS_CI_NAME_SUFFIX:-}
hostname: host-f
volumes:
- ../docker/resolv.conf:/etc/resolv.conf:ro
- ./generated-configs/node-f/hosts:/etc/fips/hosts:ro
- ./generated-configs/node-f/fips.yaml:/etc/fips/fips.yaml:ro
- ./generated-configs/node-f/fips.key:/etc/fips/fips.key:ro
- ./generated-configs${FIPS_CI_NAME_SUFFIX:-}/node-f/hosts:/etc/fips/hosts:ro
- ./generated-configs${FIPS_CI_NAME_SUFFIX:-}/node-f/fips.yaml:/etc/fips/fips.yaml:ro
- ./generated-configs${FIPS_CI_NAME_SUFFIX:-}/node-f/fips.key:/etc/fips/fips.key:ro
networks:
acl-net:
ipv4_address: 172.31.0.15
- acl-net
testing/acl-allowlist/generate-configs.sh
+22 -13
View File
@@ -3,7 +3,12 @@
set -euo pipefail
SCRIPT_DIR="$(cd "$(dirname "$0")" && pwd)"
GENERATED_DIR="$SCRIPT_DIR/generated-configs"
# Scoped by the per-run suffix because this directory is wiped and rewritten
# below: two runs sharing one output directory would delete each other's
# fixtures out from under running containers. Unset (a bare hand run, or the
# GitHub-hosted path) it collapses to the historical "generated-configs".
GENERATED_DIR="$SCRIPT_DIR/generated-configs${FIPS_CI_NAME_SUFFIX:-}"
write_file() {
local path="$1"
@@ -23,6 +28,10 @@ node-f npub1ytrut7gjncn2zfnhn56c0zgftf0w6p99gf6fu8j73hzw5603zglqc9av6c
EOF
}
# Peers are addressed by the docker hostname the compose file assigns
# (host-a … host-f), not by IP. The network requests no subnet so that two
# concurrent runs cannot collide on one address range, which means no node's
# address is knowable before `docker compose up`.
echo "Generating ACL allowlist fixtures..."
rm -rf "$GENERATED_DIR"
@@ -48,31 +57,31 @@ peers:
alias: "node-b"
addresses:
- transport: udp
addr: "172.31.0.11:2121"
addr: "host-b:2121"
connect_policy: auto_connect
- npub: "npub1cld9yay0u24davpu6c35l4vldrhzvaq66pcqtg9a0j2cnjrn9rtsxx2pe6"
alias: "node-c"
addresses:
- transport: udp
addr: "172.31.0.12:2121"
addr: "host-c:2121"
connect_policy: auto_connect
- npub: "npub1n9lpnv0592cc2ps6nm0ca3qls642vx7yjsv35rkxqzj2vgds52sqgpverl"
alias: "node-d"
addresses:
- transport: udp
addr: "172.31.0.13:2121"
addr: "host-d:2121"
connect_policy: auto_connect
- npub: "npub1x5z9rwzzm26q9verutx4aajhf2zw2pyp34c6whhde2zduxqav40qgq36l6"
alias: "node-e"
addresses:
- transport: udp
addr: "172.31.0.14:2121"
addr: "host-e:2121"
connect_policy: auto_connect
- npub: "npub1ytrut7gjncn2zfnhn56c0zgftf0w6p99gf6fu8j73hzw5603zglqc9av6c"
alias: "node-f"
addresses:
- transport: udp
addr: "172.31.0.15:2121"
addr: "host-f:2121"
connect_policy: auto_connect
EOF
@@ -113,19 +122,19 @@ peers:
alias: "node-a"
addresses:
- transport: udp
addr: "172.31.0.10:2121"
addr: "host-a:2121"
connect_policy: auto_connect
- npub: "npub1cld9yay0u24davpu6c35l4vldrhzvaq66pcqtg9a0j2cnjrn9rtsxx2pe6"
alias: "node-c"
addresses:
- transport: udp
addr: "172.31.0.12:2121"
addr: "host-c:2121"
connect_policy: auto_connect
- npub: "npub1n9lpnv0592cc2ps6nm0ca3qls642vx7yjsv35rkxqzj2vgds52sqgpverl"
alias: "node-d"
addresses:
- transport: udp
addr: "172.31.0.13:2121"
addr: "host-d:2121"
connect_policy: auto_connect
EOF
@@ -166,7 +175,7 @@ peers:
alias: "node-a"
addresses:
- transport: udp
addr: "172.31.0.10:2121"
addr: "host-a:2121"
connect_policy: auto_connect
EOF
@@ -209,7 +218,7 @@ peers:
alias: "node-a"
addresses:
- transport: udp
addr: "172.31.0.10:2121"
addr: "host-a:2121"
connect_policy: auto_connect
EOF
@@ -252,7 +261,7 @@ peers:
alias: "node-a"
addresses:
- transport: udp
addr: "172.31.0.10:2121"
addr: "host-a:2121"
connect_policy: auto_connect
EOF
@@ -282,7 +291,7 @@ peers:
alias: "node-a"
addresses:
- transport: udp
addr: "172.31.0.10:2121"
addr: "host-a:2121"
connect_policy: auto_connect
EOF
testing/acl-allowlist/test.sh
+133 -39
View File
@@ -75,47 +75,103 @@ assert_acl_field() {
echo "PASS: $container ACL field $field matches expected value"
}
# Connected-peer count for a container, or the empty string if it did not
# answer.
#
# Empty is deliberately distinct from a real 0, and here the distinction is
# the whole point: the ACL denial checks below expect exactly 0, so an
# `|| echo 0` fallback lets an unreachable container satisfy them on the first
# iteration and the security property is never observed. Same shape as
# admission-cap-test.sh's read_peer_count.
read_connected_peers() {
local container="$1"
docker exec "$container" fipsctl show peers 2>/dev/null \
| python3 -c 'import json,sys; data=json.load(sys.stdin); print(sum(1 for p in data.get("peers", []) if p.get("connectivity") == "connected"))' 2>/dev/null \
|| true
}
wait_for_peers_exact() {
local container="$1"
local expected_count="$2"
local timeout="${3:-30}"
local count="" answered=false
for _ in $(seq 1 "$timeout"); do
local count
count=$(docker exec "$container" fipsctl show peers 2>/dev/null \
| python3 -c 'import json,sys; data=json.load(sys.stdin); print(sum(1 for p in data.get("peers", []) if p.get("connectivity") == "connected"))' 2>/dev/null || echo 0)
if [ "$count" -eq "$expected_count" ]; then
return 0
count=$(read_connected_peers "$container")
if [ -n "$count" ]; then
answered=true
if [ "$count" -eq "$expected_count" ]; then
return 0
fi
fi
sleep 1
done
echo "FAIL: $container did not reach $expected_count connected peers in ${timeout}s" >&2
if [ "$answered" = false ]; then
echo "FAIL: $container never answered a peer query in ${timeout}s, so a count of $expected_count was never actually observed" >&2
else
echo "FAIL: $container did not reach $expected_count connected peers in ${timeout}s (last answer: $count)" >&2
fi
docker exec "$container" fipsctl show peers >&2 || true
exit 1
}
assert_log_contains() {
# Assert that ONE log line in $container contains every one of the given
# fixed strings.
#
# Single-line matching is the point. Independent whole-log greps for an
# npub and for `decision=denylist match` are jointly satisfied by "this
# npub appears somewhere" plus "somebody was rejected by denylist", which
# is not the property this suite exists to prove. Node-a lists the denied
# peers as auto_connect peers, so it logs their npubs on the outbound
# connect path whether or not a rejection ever happened; the npub has to
# be on the rejection line itself to mean anything.
#
# Strings match in any order, by chaining fixed-string greps over the
# surviving lines, so the assertion does not depend on the order the
# tracing formatter emits a message and its fields in. A grep over empty
# input yields the empty string rather than a value that could satisfy
# the caller, so a container that cannot be read times out and fails
# rather than passing.
#
# Polls rather than reading once: under the XX handshake the
# cross-connection tie-breaker decides which side reaches its ACL check
# first, so an inbound-handshake rejection may not emit until a later
# retry. Same wait-with-timeout shape as wait_for_peers_exact above.
#
# Deliberately NOT registered in check-log-strings.py's SHELL_HELPERS,
# for two reasons, and note that registering it would in fact capture
# nothing: that extractor reads only a helper's FIRST argument and only
# when it is a quoted literal free of `$` (check-log-strings.py:116),
# whereas the first argument here is the unquoted container name
# carrying ${FIPS_CI_NAME_SUFFIX}. Verified by running the extractor
# with this helper added: zero hits. The same is true of the
# `assert_log_contains` this replaced, so nothing left the check's scope.
#
# It should stay out of scope regardless: that check exists for strings
# whose disappearance from src/ would let an assertion silently pass,
# and every string here is a positive requirement, so a missing one
# exhausts the poll and exits 1, which is loud.
assert_log_line_contains_all() {
local container="$1"
local pattern="$2"
local timeout="${3:-15}"
local logs
local timeout="$2"
shift 2
# Poll docker logs instead of one-shot reading: under XX handshake,
# the cross-connection tie-breaker determines which side reaches
# its ACL-check point first, so the inbound-handshake-context
# rejection may not emit until a later retry. Same wait-with-timeout
# shape as wait_for_peers_exact above.
local logs surviving pattern
for _ in $(seq 1 "$timeout"); do
logs="$(docker logs "$container" 2>&1 | python3 -c 'import re,sys; print(re.sub(r"\x1b\[[0-9;]*m", "", sys.stdin.read()), end="")' || true)"
if printf '%s' "$logs" | grep -F "$pattern" >/dev/null; then
echo "PASS: $container logs contain expected ACL rejection"
surviving="$logs"
for pattern in "$@"; do
surviving="$(printf '%s' "$surviving" | grep -F -- "$pattern" || true)"
done
if [ -n "$surviving" ]; then
echo "PASS: $container has a log line matching all of: $*"
return 0
fi
sleep 1
done
echo "FAIL: missing log pattern in $container: $pattern (waited ${timeout}s)" >&2
echo "FAIL: no single log line in $container contains all of: $* (waited ${timeout}s)" >&2
exit 1
}
@@ -129,34 +185,72 @@ log "Generating ACL allowlist fixtures"
log "Starting ACL allowlist harness"
docker compose -f "$COMPOSE_FILE" down >/dev/null 2>&1 || true
docker compose -f "$COMPOSE_FILE" up -d --build
# --build only on the hand path. Under a harness, --skip-build means the caller
# has already built the image this compose file names, and rebuilding it here
# would overwrite that image from whatever the shared build context happens to
# hold — which is how a suite ends up certifying binaries it was never given.
if [ "$SKIP_BUILD" = false ]; then
docker compose -f "$COMPOSE_FILE" up -d --build
else
docker compose -f "$COMPOSE_FILE" up -d
fi
log "Waiting for expected peer convergence"
wait_for_peers_exact fips-acl-container-a 3 40
wait_for_peers_exact fips-acl-container-b 1 40
wait_for_peers_exact fips-acl-container-c 0 5
wait_for_peers_exact fips-acl-container-d 0 5
wait_for_peers_exact fips-acl-container-e 1 40
wait_for_peers_exact fips-acl-container-f 1 40
wait_for_peers_exact fips-acl-container-a${FIPS_CI_NAME_SUFFIX:-} 3 40
wait_for_peers_exact fips-acl-container-b${FIPS_CI_NAME_SUFFIX:-} 1 40
wait_for_peers_exact fips-acl-container-c${FIPS_CI_NAME_SUFFIX:-} 0 5
wait_for_peers_exact fips-acl-container-d${FIPS_CI_NAME_SUFFIX:-} 0 5
wait_for_peers_exact fips-acl-container-e${FIPS_CI_NAME_SUFFIX:-} 1 40
wait_for_peers_exact fips-acl-container-f${FIPS_CI_NAME_SUFFIX:-} 1 40
log "Verifying peer sets"
assert_peer_set fips-acl-container-a "npub1tdwa4vjrjl33pcjdpf2t4p027nl86xrx24g4d3avg4vwvayr3g8qhd84le npub1x5z9rwzzm26q9verutx4aajhf2zw2pyp34c6whhde2zduxqav40qgq36l6 npub1ytrut7gjncn2zfnhn56c0zgftf0w6p99gf6fu8j73hzw5603zglqc9av6c"
assert_peer_set fips-acl-container-b "npub1sjlh2c3x9w7kjsqg2ay080n2lff2uvt325vpan33ke34rn8l5jcqawh57m"
assert_peer_set fips-acl-container-c ""
assert_peer_set fips-acl-container-d ""
assert_peer_set fips-acl-container-e "npub1sjlh2c3x9w7kjsqg2ay080n2lff2uvt325vpan33ke34rn8l5jcqawh57m"
assert_peer_set fips-acl-container-f "npub1sjlh2c3x9w7kjsqg2ay080n2lff2uvt325vpan33ke34rn8l5jcqawh57m"
assert_peer_set fips-acl-container-a${FIPS_CI_NAME_SUFFIX:-} "npub1tdwa4vjrjl33pcjdpf2t4p027nl86xrx24g4d3avg4vwvayr3g8qhd84le npub1x5z9rwzzm26q9verutx4aajhf2zw2pyp34c6whhde2zduxqav40qgq36l6 npub1ytrut7gjncn2zfnhn56c0zgftf0w6p99gf6fu8j73hzw5603zglqc9av6c"
assert_peer_set fips-acl-container-b${FIPS_CI_NAME_SUFFIX:-} "npub1sjlh2c3x9w7kjsqg2ay080n2lff2uvt325vpan33ke34rn8l5jcqawh57m"
assert_peer_set fips-acl-container-c${FIPS_CI_NAME_SUFFIX:-} ""
assert_peer_set fips-acl-container-d${FIPS_CI_NAME_SUFFIX:-} ""
assert_peer_set fips-acl-container-e${FIPS_CI_NAME_SUFFIX:-} "npub1sjlh2c3x9w7kjsqg2ay080n2lff2uvt325vpan33ke34rn8l5jcqawh57m"
assert_peer_set fips-acl-container-f${FIPS_CI_NAME_SUFFIX:-} "npub1sjlh2c3x9w7kjsqg2ay080n2lff2uvt325vpan33ke34rn8l5jcqawh57m"
log "Checking alias-based ACL resolution"
assert_acl_field fips-acl-container-a allow_file_entries "node-a node-b node-e node-f"
assert_acl_field fips-acl-container-a allow_entries "npub1sjlh2c3x9w7kjsqg2ay080n2lff2uvt325vpan33ke34rn8l5jcqawh57m npub1tdwa4vjrjl33pcjdpf2t4p027nl86xrx24g4d3avg4vwvayr3g8qhd84le npub1x5z9rwzzm26q9verutx4aajhf2zw2pyp34c6whhde2zduxqav40qgq36l6 npub1ytrut7gjncn2zfnhn56c0zgftf0w6p99gf6fu8j73hzw5603zglqc9av6c"
assert_acl_field fips-acl-container-c allow_file_entries "node-a node-b node-c node-d node-e node-f"
assert_acl_field fips-acl-container-c allow_entries "npub1cld9yay0u24davpu6c35l4vldrhzvaq66pcqtg9a0j2cnjrn9rtsxx2pe6 npub1n9lpnv0592cc2ps6nm0ca3qls642vx7yjsv35rkxqzj2vgds52sqgpverl npub1sjlh2c3x9w7kjsqg2ay080n2lff2uvt325vpan33ke34rn8l5jcqawh57m npub1tdwa4vjrjl33pcjdpf2t4p027nl86xrx24g4d3avg4vwvayr3g8qhd84le npub1x5z9rwzzm26q9verutx4aajhf2zw2pyp34c6whhde2zduxqav40qgq36l6 npub1ytrut7gjncn2zfnhn56c0zgftf0w6p99gf6fu8j73hzw5603zglqc9av6c"
assert_acl_field fips-acl-container-a${FIPS_CI_NAME_SUFFIX:-} allow_file_entries "node-a node-b node-e node-f"
assert_acl_field fips-acl-container-a${FIPS_CI_NAME_SUFFIX:-} allow_entries "npub1sjlh2c3x9w7kjsqg2ay080n2lff2uvt325vpan33ke34rn8l5jcqawh57m npub1tdwa4vjrjl33pcjdpf2t4p027nl86xrx24g4d3avg4vwvayr3g8qhd84le npub1x5z9rwzzm26q9verutx4aajhf2zw2pyp34c6whhde2zduxqav40qgq36l6 npub1ytrut7gjncn2zfnhn56c0zgftf0w6p99gf6fu8j73hzw5603zglqc9av6c"
assert_acl_field fips-acl-container-c${FIPS_CI_NAME_SUFFIX:-} allow_file_entries "node-a node-b node-c node-d node-e node-f"
assert_acl_field fips-acl-container-c${FIPS_CI_NAME_SUFFIX:-} allow_entries "npub1cld9yay0u24davpu6c35l4vldrhzvaq66pcqtg9a0j2cnjrn9rtsxx2pe6 npub1n9lpnv0592cc2ps6nm0ca3qls642vx7yjsv35rkxqzj2vgds52sqgpverl npub1sjlh2c3x9w7kjsqg2ay080n2lff2uvt325vpan33ke34rn8l5jcqawh57m npub1tdwa4vjrjl33pcjdpf2t4p027nl86xrx24g4d3avg4vwvayr3g8qhd84le npub1x5z9rwzzm26q9verutx4aajhf2zw2pyp34c6whhde2zduxqav40qgq36l6 npub1ytrut7gjncn2zfnhn56c0zgftf0w6p99gf6fu8j73hzw5603zglqc9av6c"
log "Checking ACL rejection logs"
assert_log_contains fips-acl-container-a "npub1cld9yay0u24davpu6c35l4vldrhzvaq66pcqtg9a0j2cnjrn9rtsxx2pe6"
assert_log_contains fips-acl-container-a "npub1n9lpnv0592cc2ps6nm0ca3qls642vx7yjsv35rkxqzj2vgds52sqgpverl"
assert_log_contains fips-acl-container-a "context=inbound_handshake"
assert_log_contains fips-acl-container-a "decision=denylist match"
# One assertion per denied peer, each requiring the real message, that
# peer's npub and the denylist decision on the SAME line. The npub being
# on the rejection line is what carries the weight here: node-a rejected
# THIS peer. The `decision=` conjunct adds no discrimination, since
# PeerAclDecision::allowed() returns early for AllowList and DefaultAllow
# (src/node/acl.rs:44-46), so DenyList is the only value that can reach
# that warn! and every rejection line carries it. It is kept because the
# remediation prescribes it and it documents the expected decision.
#
# Residual, recorded rather than hidden: node-a has auto_connect stanzas
# for both denied peers and authorizes before dialing, so it emits a
# fully-formed rejection line for each on the outbound_connect path.
# These two assertions are therefore satisfiable without the inbound ACL
# check running at all. The suite still catches that, because the denied
# peer would then connect and the peer-count assertions above would time
# out; but these two lines alone do not prove the inbound path.
assert_log_line_contains_all fips-acl-container-a${FIPS_CI_NAME_SUFFIX:-} 15 \
"Rejected peer by ACL" \
"npub1cld9yay0u24davpu6c35l4vldrhzvaq66pcqtg9a0j2cnjrn9rtsxx2pe6" \
"decision=denylist match"
assert_log_line_contains_all fips-acl-container-a${FIPS_CI_NAME_SUFFIX:-} 15 \
"Rejected peer by ACL" \
"npub1n9lpnv0592cc2ps6nm0ca3qls642vx7yjsv35rkxqzj2vgds52sqgpverl" \
"decision=denylist match"
# The outsider-initiated path specifically, asserted separately and not
# per-npub. Node-a carries static stanzas for both denied peers, so each
# can be rejected on the outbound_connect path as well and which context
# a given npub lands in is not deterministic (see README). Requiring an
# inbound rejection of a *named* peer would red on scheduling rather than
# on a regression; requiring that one exists at all does not.
assert_log_line_contains_all fips-acl-container-a${FIPS_CI_NAME_SUFFIX:-} 15 \
"Rejected peer by ACL" \
"context=inbound_handshake" \
"decision=denylist match"
log "ACL allowlist integration test passed"
testing/boringtun/docker-compose.yml
+4 -2
View File
@@ -6,6 +6,8 @@
networks:
bt-net:
driver: bridge
labels:
- "com.corganlabs.fips-ci=1"
ipam:
config:
- subnet: 172.99.0.0/24
@@ -25,7 +27,7 @@ x-boringtun-common: &boringtun-common
services:
alice:
<<: *boringtun-common
container_name: bt-alice
container_name: bt-alice${FIPS_CI_NAME_SUFFIX:-}
hostname: alice
environment:
- ROLE=alice
@@ -36,7 +38,7 @@ services:
bob:
<<: *boringtun-common
container_name: bt-bob
container_name: bt-bob${FIPS_CI_NAME_SUFFIX:-}
hostname: bob
environment:
- ROLE=bob
testing/boringtun/scripts/bench.sh
+2 -2
View File
@@ -10,14 +10,14 @@ PARALLEL="${PARALLEL:-1}"
echo "=== boringtun iperf3 throughput (single TCP stream, ${DURATION}s) ==="
# Run iperf3 server on alice (background), client on bob.
docker exec -d bt-alice iperf3 -s -1 -B 10.99.0.1 -p 5201
docker exec -d bt-alice${FIPS_CI_NAME_SUFFIX:-} iperf3 -s -1 -B 10.99.0.1 -p 5201
sleep 1
# wait for tun handshake to settle (boringtun + WG keepalive)
sleep 2
# Client: bob → alice over WG (10.99.0.1)
OUT=$(docker exec bt-bob iperf3 -c 10.99.0.1 -p 5201 -t "$DURATION" -P "$PARALLEL" -J)
OUT=$(docker exec bt-bob${FIPS_CI_NAME_SUFFIX:-} iperf3 -c 10.99.0.1 -p 5201 -t "$DURATION" -P "$PARALLEL" -J)
# Pull SUM bps.
MBPS=$(echo "$OUT" | python3 -c "import json,sys; d=json.load(sys.stdin); print(f\"{d['end']['sum_received']['bits_per_second'] / 1_000_000:.2f}\")")
testing/chaos/README.md
+63 -46
View File
@@ -18,7 +18,7 @@ Ethernet). Logs are collected and analyzed automatically.
```bash
./testing/chaos/scripts/build.sh
./testing/chaos/scripts/chaos.sh smoke-10
./testing/chaos/scripts/chaos.sh churn-mixed
```
## Available Scenarios
@@ -29,12 +29,10 @@ Random topologies with increasing stressor intensity.
| Scenario | Nodes | Topology | Duration | Netem | Link Flaps | Traffic | Node Churn | Bandwidth |
| -------- | ----- | ---------------- | -------- | ----- | ---------- | ------- | ---------- | --------- |
| smoke-10 | 10 | random_geometric | 60s | -- | -- | -- | -- | -- |
| chaos-10 | 10 | random_geometric | 120s | yes | yes | yes | -- | -- |
| churn-10 | 10 | random_geometric | 600s | yes | yes | yes | yes | -- |
| churn-20 | 20 | erdos_renyi | 600s | yes | yes | yes | yes | yes |
- **smoke-10**: Baseline sanity check. No stressors, just verify tree convergence.
- **chaos-10**: Network degradation (5-50ms delay, 0-2% loss), link flaps (max 2
down, 10-30s), and iperf traffic (max 3 concurrent). Netem mutates 30% of
links every 15-30s between normal and degraded policies.
@@ -44,41 +42,22 @@ Random topologies with increasing stressor intensity.
simultaneously, bandwidth tiers (1/10/100/1000 Mbps), `protect_connectivity`
disabled (partitions allowed).
### Cost-based parent selection
### Cost-based parent selection — retired, now sans-IO unit tests
Explicit topologies with heterogeneous link types (fiber, Bluetooth, WiFi) to
test that the spanning tree selects optimal parents based on link cost.
The cost-selection scenarios (cost-avoidance, depth-vs-cost, bottleneck-parent,
cost-reeval, cost-stability, mixed-technology) were retired on 2026-07-23.
Their subject was the pure `TreeState::evaluate_parent` decision — which parent
wins on `effective_depth = depth + link_cost`, when periodic re-evaluation
switches, and when hysteresis suppresses a flap. A Docker mesh could not test
that reliably: the root is whichever node holds the smallest `NodeAddr`, MMP
costs take several measurement windows to settle, and hold-down plus hysteresis
timing all confound the outcome (a deterministic `link_swap` attempt still
produced zero periodic switches in a full run).
| Scenario | Nodes | Shape | Link types | Duration | What it tests |
| ----------------- | ----- | --------------- | ------------------------ | -------- | ------------------------------------------------------------------- |
| cost-avoidance | 4 | Diamond | Fiber + Bluetooth | 120s | n04 picks fiber parent (n03) over Bluetooth parent (n02) |
| depth-vs-cost | 4 | Linear tree | Fiber + Bluetooth | 120s | Cost tradeoff: depth vs. Bluetooth link quality |
| bottleneck-parent | 10 | Tree with BT | Fiber + Bluetooth | 120s | n06 avoids Bluetooth bottleneck via n02, picks fiber via n03 |
| cost-mixed-7node | 7 | Multi-type tree | Fiber + Bluetooth + WiFi | 180s | n06 prefers fiber (n03) over WiFi (n04) |
| cost-reeval | 4 | Diamond | Fiber (mutated) | 180s | Periodic re-evaluation triggers parent switch (reeval_interval=15s) |
| cost-stability | 4 | Diamond | WiFi (all) | 180s | Hysteresis prevents flapping when costs vary within 20% band |
- **cost-avoidance**, **depth-vs-cost**: Minimal scenarios validating the core
cost formula. Bluetooth (L2CAP) links use 15-40ms delay and 2-8% loss;
fiber uses 1-5ms delay and 0-1% loss.
- **bottleneck-parent**: Larger topology where some nodes have both fiber and
Bluetooth paths to choose from, and one node (n09) is stuck with Bluetooth
(no alternative).
- **cost-mixed-7node**: Three link technologies in one mesh. Traffic enabled.
- **cost-reeval**: Netem mutation (50% fraction, every 12-18s) degrades random
links. FIPS override sets `reeval_interval_secs=15` so periodic re-evaluation
catches cost asymmetry. Look for `trigger=periodic` in logs.
- **cost-stability**: All links are WiFi. Mutation swings costs between
`slightly_better` and `slightly_worse` — within the hysteresis band. Expect
≤ 5 parent switches over 180s.
### Mixed-technology
Larger explicit topologies combining multiple link technologies.
| Scenario | Nodes | Link types | Duration | Netem mutation | What it tests |
| ---------------- | ----- | ------------------------ | -------- | -------------- | ------------------------------------------------ |
| mixed-technology | 10 | Fiber + Bluetooth + WiFi | 180s | 20%/30-60s | Tree convergence across heterogeneous link types |
That logic is now covered by deterministic sans-IO unit tests in
`src/tree/tests.rs` (`test_evaluate_parent_cost_*`, `..._hysteresis_*`,
`..._effective_depth_*`), which run in the cargo quartet on every commit and
can each be shown to fail by breaking the cost or hysteresis logic.
### Transport-specific
@@ -88,19 +67,12 @@ Explicit topologies exercising non-UDP transports.
| ------------- | ----- | -------------- | ----- | -------- | ----- | ---------- | ------------------------------------------ |
| ethernet-only | 4 | Ethernet | Ring | 90s | yes | -- | AF_PACKET transport with beacon discovery |
| ethernet-mesh | 6 | UDP + Ethernet | Mesh | 120s | yes | yes | Mixed UDP/Ethernet, netem mutation + flaps |
| tcp-only | 4 | TCP | Ring | 90s | yes | -- | TCP transport with static peer config |
| tcp-chain | 4 | TCP | Chain | 90s | yes | -- | TCP multi-hop routing through chain |
| tcp-mesh | 6 | UDP + TCP | Mesh | 120s | yes | yes | Mixed UDP/TCP, netem mutation + flaps |
- **ethernet-only**: 4-node ring on raw Ethernet (AF_PACKET). Peers discovered
via beacons, not static config. Minimal netem (1-5ms delay).
- **ethernet-mesh**: Mirrors `tcp-mesh` topology but with Ethernet instead of
TCP. UDP edges use static config; Ethernet edges use beacon discovery.
- **tcp-only**: 4-node ring using TCP on port 8443. Tests connect-on-send,
FMP framing over TCP, and reconnection. Netem enabled (1-10ms delay, 0-1%
loss).
- **tcp-chain**: 4-node linear chain, all TCP. Tests multi-hop routing over
TCP-only mesh.
- **tcp-mesh**: 6-node mesh with 4 UDP and 3 TCP edges. Both transports use
static peer config. Netem mutation (30% fraction, every 20-40s) and link
flaps (1 link max, 10-20s down).
@@ -124,8 +96,14 @@ detection.
- **ecn-ab-on / ecn-ab-off**: Paired scenarios with identical conditions
(6-node tree, 10 Mbps egress, 1000 kbps ingress policing, 10ms link
delay, 8 KB recv buffer) differing only in `ecn.enabled`.
`ecn-ab-test.sh` runs both and compares throughput and congestion
counters. Initial results: +10.2% recv throughput with ECN enabled.
`ecn-ab-compare.sh` runs both and prints a side-by-side of throughput
and congestion counters. It is a manual tool, not a test: it asserts
nothing and no runner invokes it. The "+10.2% recv throughput with ECN
enabled" figure once recorded here is not reproducible from anything on
disk — the script read a fixed `sim-results/ecn-ab-on/` path while the
runner has written timestamped directories since 2026-03-20, and no
ecn-ab result directory survives. The path bug is fixed; the figure is
left out until a run produces one.
### Ingress Traffic Control
@@ -257,12 +235,51 @@ since they use beacon discovery.
Results written to `sim-results/` (configurable via
`logging.output_dir`):
- `status.txt` -- How the run ended, plus the scenario, the seed and the
container names it used; one `key=value` per line
- `analysis.txt` -- Summary: panics, errors, sessions, metrics
- `metadata.txt` -- Seed, node count, edges, adjacency list
- `runner.log` -- Orchestration events (topology, netem, churn, traffic) with timestamps
- `fips-node-nXX.log` -- Per-node log output
Exit code 0 on success, 2 if panics detected.
The `status` field reads:
- `completed` -- ran for its configured duration
- `interrupted` -- a signal cut the run short, so the artifacts are real
but describe less time than the scenario asked for
- `aborted` -- the run raised part way through; same caveat, and
`runner.log` carries the traceback
- `setup-failed` -- the containers never started
- `teardown-failed` -- the mesh ran but its logs or analysis could not be
produced
A `setup-failed` directory holds `runner.log` and `status.txt` and nothing
else. Nothing is harvested, because container names are global to the host
and reading them after a failed setup describes whichever run holds them
now. So `analysis.txt` in a result directory is proof that this scenario's
own mesh existed. A directory with no `status.txt` was written before this
was the case and says nothing either way.
Exit codes:
- `0` -- Ran to completion, no panics, every assertion passed
- `1` -- The scenario file could not be loaded, or a second interrupt
arrived while the first was being handled
- `2` -- Panics found in the collected node logs. Also what the argument
parser exits with when it rejects the command line, before any run starts
- `3` -- A post-run assertion failed
- `4` -- Setup, warmup, the simulation loop or teardown raised, so the run
did not complete; `runner.log` carries the traceback
Codes 2 and 3 describe what a mesh that ran did. Code 4 says there is
nothing to describe, and takes precedence over both. Code 2 is dual-use:
a run that never started cannot have panicked, so read it together with
whether `runner.log` exists.
A run stopped by a signal exits on this same ladder rather than one of its
own: what it collected before stopping is still worth reporting, and
`status.txt` says it was cut short. `chaos.sh` reports 130 for a Ctrl-C of
its own accord.
## Creating Custom Scenarios
testing/chaos/ecn-ab-test.sh → testing/chaos/ecn-ab-compare.sh
+50 -9
View File
@@ -1,10 +1,27 @@
#!/usr/bin/env bash
# ECN A/B Throughput Test
# ECN A/B Throughput Comparison (a manual tool, NOT a test)
#
# Runs two identical chaos scenarios — one with ECN enabled, one disabled —
# and compares iperf3 throughput and congestion counter results.
# and prints a side-by-side of iperf3 throughput and congestion counters.
#
# Usage: ./ecn-ab-test.sh [--seed N] [--duration N]
# Renamed from ecn-ab-test.sh on 2026-07-23. The old name claimed a verdict
# this has never produced: it asserts nothing, applies no threshold, and no
# runner invokes it. Naming it a test made it look like coverage.
#
# It also could not have worked. It read sim-results/ecn-ab-on/... while the
# runner has written sim-results/<timestamp>-<scenario>/ since 2026-03-20, so
# it has found neither input for at least four months, and there is not one
# archived ecn-ab result directory on disk. That path bug is fixed below.
#
# WHAT IS STILL MISSING, and why it was not added: turning this into a real
# test needs a threshold — how much throughput ECN should buy, or how much
# lower the congestion counters should run — and there is no corpus to derive
# one from, precisely because the tool has never produced a kept result. A
# number invented here would assert the author's guess. The prerequisite is a
# calibration run set, and that is a protocol question about what ECN is
# expected to deliver, not a harness one.
#
# Usage: ./ecn-ab-compare.sh [--seed N] [--duration N]
set -euo pipefail
cd "$(dirname "$0")"
@@ -24,12 +41,12 @@ echo ""
# --- Run A: ECN ON ---
echo "--- Phase A: ECN ENABLED ---"
sudo python3 -m sim scenarios/ecn-ab-on.yaml "${EXTRA_ARGS[@]}" || true
sudo python3 -m sim scenarios/ecn-ab-on.yaml "${EXTRA_ARGS[@]}"
echo ""
# --- Run B: ECN OFF ---
echo "--- Phase B: ECN DISABLED ---"
sudo python3 -m sim scenarios/ecn-ab-off.yaml "${EXTRA_ARGS[@]}" || true
sudo python3 -m sim scenarios/ecn-ab-off.yaml "${EXTRA_ARGS[@]}"
echo ""
# --- Compare results ---
@@ -37,7 +54,27 @@ echo "=== Results ==="
echo ""
python3 - <<'PYEOF'
import glob
import json
def latest(scenario, filename):
"""Newest run directory for a scenario, or None.
The runner writes sim-results/<timestamp>-<scenario>/, so a fixed path
such as sim-results/ecn-ab-on/ has never matched anything. Sorting the
glob works because the timestamp is the leading, fixed-width component.
"""
dirs = sorted(glob.glob(f"sim-results/*-{scenario}"))
if not dirs:
print(f" no run directory found for {scenario}")
return None
path = f"{dirs[-1]}/{filename}"
if not glob.glob(path):
print(f" {scenario}: {filename} missing from {dirs[-1]}")
return None
return path
import os
import sys
@@ -92,8 +129,10 @@ def print_sessions(label, sessions):
print(f" {'':>14} completed={n} incomplete={incomplete}")
return total_recv / n
on_results = load_results("sim-results/ecn-ab-on/iperf3-results.json")
off_results = load_results("sim-results/ecn-ab-off/iperf3-results.json")
on_path = latest("ecn-ab-on", "iperf3-results.json")
off_path = latest("ecn-ab-off", "iperf3-results.json")
on_results = load_results(on_path) if on_path else []
off_results = load_results(off_path) if off_path else []
on_sessions = extract_throughput(on_results)
off_sessions = extract_throughput(off_results)
@@ -111,8 +150,10 @@ if avg_on and avg_off:
# Congestion counters
print("Congestion Counters (final snapshot):")
for label, path in [("ECN ON", "sim-results/ecn-ab-on/congestion-snapshot-final.json"),
("ECN OFF", "sim-results/ecn-ab-off/congestion-snapshot-final.json")]:
for label, scenario in [("ECN ON", "ecn-ab-on"), ("ECN OFF", "ecn-ab-off")]:
path = latest(scenario, "congestion-snapshot-final.json")
if path is None:
continue
snap = load_congestion(path)
if not snap:
print(f" {label}: no snapshot")
testing/chaos/scenarios/bottleneck-parent.yaml
-92
View File
@@ -1,92 +0,0 @@
# Bottleneck Parent: 10-node focused Bluetooth bottleneck test
#
# Explicit topology with two Bluetooth (L2CAP) links that create
# bottleneck parent candidates. Tests that nodes avoid choosing
# Bluetooth parents when fiber alternatives exist at the same or
# slightly greater depth.
#
# Topology:
#
# n01 (root)
# / | \ \
# f f f f
# / | \ \
# n02 n03 n04 n05
# | / | \ |
# BT f f BT
# | / | \ |
# n06 n07 n08 n09
# |
# f
# |
# n10
#
# Edges and link types:
# Fiber: n01-n02, n01-n03, n01-n04, n01-n05,
# n03-n06, n03-n07, n04-n08, n08-n10
# Bluetooth: n02-n06, n05-n09
#
# Cross-links: n03-n08 (fiber) — gives n08 a fiber alternative to n04
#
# Test subjects:
# - n06 has Bluetooth (n02) and fiber (n03) at depth 1 — should pick n03
# - n09 has only Bluetooth (n05) — no alternative, stuck with BT parent
scenario:
name: "bottleneck-parent"
seed: 42
duration_secs: 60
topology:
algorithm: explicit
num_nodes: 10
params:
adjacency:
- [n01, n02]
- [n01, n03]
- [n01, n04]
- [n01, n05]
- [n02, n06]
- [n03, n06]
- [n03, n07]
- [n03, n08]
- [n04, n08]
- [n05, n09]
- [n08, n10]
subnet: "172.20.0.0/24"
ip_start: 10
netem:
enabled: true
default_policy:
delay_ms: [1, 5]
jitter_ms: [0, 1]
loss_pct: [0, 0.5]
link_policies:
# Bluetooth (L2CAP) links
- edges: ["n02-n06", "n05-n09"]
policy:
delay_ms: [15, 40]
jitter_ms: [5, 15]
loss_pct: [2, 8]
mutation:
interval_secs: {min: 30, max: 60}
fraction: 0.2
policies:
normal:
delay_ms: [1, 5]
loss_pct: [0, 0.5]
link_flaps:
enabled: false
traffic:
enabled: true
max_concurrent: 2
interval_secs: {min: 15, max: 30}
duration_secs: {min: 5, max: 10}
parallel_streams: 2
logging:
rust_log: "info"
output_dir: "./sim-results"
testing/chaos/scenarios/churn-mixed.yaml
+60
View File
@@ -67,6 +67,66 @@ bandwidth:
enabled: true
tiers_mbps: [1, 10, 100, 1000]
# Baseline: the mesh came up, agreed on a root, and took parents. This
# asserts nothing about which nodes survive the churn; it exists so that
# a run in which the mesh never formed cannot report success, which
# until now it could, because this scenario carried no assertions at
# all.
#
# This one is calibrated rather than derived, because a scenario that
# stops and starts nodes on purpose does not hold a single spanning
# tree.
#
# Calibrated 2026-07-26 against fourteen runs that recorded a baseline
# verdict, read from their assertions.txt files. Every one used this
# file's fixed seed 42 and therefore an identical chaos schedule, so the
# spread below is container timing rather than differing scenarios:
#
# distinct roots 1 2 3 4 5 -> 2 3 5 3 1 runs
# nodes parented 9 8 7 6 5 -> the exact complement, in all 14
# sessions 12 to 20, minimum 12
#
# The previous ceiling of 4 roots sat at roughly the 93rd percentile of
# that distribution: one run in fourteen exceeded it and four sat at or
# above it, so it reddened a share of runs whatever the daemon did. It
# was set from six runs whose observed maximum was 3, which is how a
# threshold one step outside a small sample ends up inside the real one.
#
# The ceiling is now 6, one step beyond the observed maximum of 5, and
# the parented floor is its complement at 4. That still fails a mesh
# that collapsed — seven or more of ten nodes islanded — which is the
# only thing this assertion was ever meant to catch. Do not read a pass
# as convergence: four of the six gating scenarios assert a floor of
# this kind, and it means the mesh formed and nobody errored.
#
# min_sessions stays at 10 against an observed minimum of 12. It has
# never fired and there is no evidence it is mis-set, but the margin is
# thin and a future failure there should be read as calibration before
# it is read as a defect.
#
# Tighten any of these only against a larger sample, and against one
# gathered at the invocation CI actually runs.
#
# READ THIS BEFORE RETUNING: the numbers above describe the invocation CI
# gates on, which is not this file's own defaults. ci-local runs it as
# "churn-mixed --nodes 10 --duration 120", and --nodes sed-patches
# num_nodes in a copy of this file before the scenario is loaded, so the
# gating run is a 10-node 120-second one while a bare `chaos.sh
# churn-mixed` runs the 20-node 600-second scenario written here. This is
# the only scenario CI overrides that way.
#
# The floors hold for both but are calibrated for the smaller. A bare
# 20-node run clears them easily: 20 answering, 1 root, 19 parented, 69
# sessions, measured 2026-07-23. Retuning against numbers like those would
# put them past what the 10-node gating run can reach, and CI would go red
# while a manual run stayed green.
assertions:
baseline:
min_nodes_reporting: 10
max_roots: 6
min_nodes_parented: 4
min_sessions: 10
logging:
rust_log: "debug"
output_dir: "./sim-results"
testing/chaos/scenarios/congestion-stress.yaml
+60 -5
View File
@@ -7,17 +7,22 @@
# Congestion detection signals exercised:
# 1. MMP loss detection: netem loss exceeds the 5% loss_threshold,
# triggering detect_congestion() via MMP metrics on transit nodes.
# 2. Kernel socket drops: small recv_buf_size (8KB) combined with
# traffic saturation causes SO_RXQ_OVFL on the UDP socket,
# triggering the transport drop detection path.
# 3. Ingress policing: tc policer on the receive side drops excess
# 2. Ingress policing: tc policer on the receive side drops excess
# inbound packets, creating bursty arrival patterns.
# 3. ECN CE marking under the shaped bottleneck queue.
#
# The kernel socket-drop signal (SO_RXQ_OVFL) is NOT exercised here. This
# scenario's 1 Mbps cap and ingress policer, which its ECN/MMP signals
# require, make socket overflow impossible. It also cannot be provoked
# deterministically anywhere in Docker — a fresh daemon reader keeps up
# with container-speed traffic — so the FIPS drop-detection logic is
# unit-tested directly in src/node/tests/unit.rs instead. See the note at
# the assertions block below.
#
# ECN is explicitly enabled via fips_overrides.
#
# Success criteria (verified via post-run congestion snapshot):
# - congestion_detected > 0 on at least one forwarding node
# - kernel_drop_events > 0 on at least one node
# - ce_forwarded > 0 on transit nodes
# - ce_received > 0 on destination nodes
#
@@ -91,6 +96,56 @@ traffic:
node_churn:
enabled: false
# Three of the four success criteria above, encoded. Until now all four
# existed only as that comment and were checked by nothing, so the scenario
# could not fail on any of them.
#
# The floors are 1 node each because that is what the criteria say ("on at
# least one forwarding node", "on transit nodes", "on destination nodes").
# They are deliberately not tightened to the observed counts: the criterion
# is the spec, and a floor invented from six runs would assert something
# nobody wrote down.
#
# What the floors are worth, from the archived corpus. The six runs that
# carry a status.txt -- the only ones provably completed, all between
# 2026-07-22 22:14 and 2026-07-23 02:02 -- meet all three with 5 to 9 nodes
# reporting each signal, so the margin over a floor of 1 is comfortable.
# The 176 older runs meet none of them. Those older runs did reach teardown
# (each wrote an analysis.txt), and ECN landed 2026-03-05, before all but
# one of them, so they are valid runs that observed no congestion rather
# than runs that died early. What changed on 2026-07-22 is not established:
# this file has not been touched since it was created, and neither have
# netem.py, traffic.py or control.py. Worth knowing before trusting a green
# result here, and worth its own investigation.
assertions:
congestion_signals:
min_nodes_detected: 1
min_nodes_ce_forwarded: 1
min_nodes_ce_received: 1
# The kernel socket-drop criterion is NOT asserted here and no longer
# belongs to this scenario. The FIPS drop-DETECTION logic is unit-tested in
# src/node/tests/unit.rs (2026-07-23); the kernel dropping datagrams is a
# kernel behaviour, not FIPS's to test, and could not be provoked in Docker.
#
# Why it could never be met here: across all 182 archived runs of this
# scenario, including the six that meet the three signals above, no node
# ever reported a non-zero kernel_drop_events. SO_RXQ_OVFL counts datagrams
# arriving at a FULL receive queue, and the 1 Mbps cap sets the arrival rate
# to 125 kB/s per link. Linux doubles a requested SO_RCVBUF, so the queue is
# 8192 bytes and filling it would take ~65 ms of reader stall (~22 ms even at
# the busiest node's 3 Mbps aggregate). Traffic volume cannot cause the
# overflow because the volume is capped below the rate the buffer drains, and
# the ingress policer discards excess in tc before the socket ever sees it.
# An unshaped attempt (congestion-drops, 2026-07-23) recorded zero raw drops
# on every node even with a 4 KB buffer and heavy iperf: a fresh daemon
# reader keeps up, so the overflow cannot be provoked deterministically. That
# is why the detection edge is tested as a sans-IO unit test.
#
# The recv_buf_size: 4096 override below is kept because the three asserted
# signals were validated with it in place; it is inert for socket overflow
# under this scenario's cap.
logging:
rust_log: "info"
output_dir: "./sim-results"
testing/chaos/scenarios/cost-avoidance.yaml
-66
View File
@@ -1,66 +0,0 @@
# Cost-Based Parent Selection: Bottleneck Avoidance Test
#
# Topology (explicit 4-node diamond):
#
# n01 (root — smallest addr)
# / \
# fiber fiber
# / \
# n02 n03
# \ /
# BT fiber
# \ /
# n04 (test subject)
#
# n04 has two candidate parents at depth 1: n02 (via Bluetooth L2CAP)
# and n03 (via fiber). Cost-based selection should pick n03 because:
# effective_depth(n02) = 1 + ~1.4 (Bluetooth) = ~2.4
# effective_depth(n03) = 1 + ~1.01 (fiber) = ~2.01
#
# Validation: tree snapshot shows n04's parent is n03.
scenario:
name: "cost-avoidance"
seed: 42
duration_secs: 45
topology:
algorithm: explicit
num_nodes: 4
params:
adjacency:
- [n01, n02]
- [n01, n03]
- [n02, n04]
- [n03, n04]
subnet: "172.20.0.0/24"
ip_start: 10
netem:
enabled: true
default_policy:
# Fiber-like
delay_ms: [1, 5]
jitter_ms: [0, 1]
loss_pct: [0, 0.5]
link_policies:
# Bluetooth (L2CAP) link from n02 to n04
- edges: ["n02-n04"]
policy:
delay_ms: [15, 40]
jitter_ms: [5, 15]
loss_pct: [2, 8]
link_flaps:
enabled: false
traffic:
enabled: true
max_concurrent: 2
interval_secs: {min: 15, max: 30}
duration_secs: {min: 5, max: 10}
parallel_streams: 2
logging:
rust_log: "info"
output_dir: "./sim-results"
testing/chaos/scenarios/cost-reeval.yaml
-86
View File
@@ -1,86 +0,0 @@
# Periodic Cost Re-evaluation Test
#
# Topology (explicit 4-node diamond):
#
# n01 (root)
# / \
# fiber fiber
# / \
# n02 n03
# \ /
# fiber fiber
# \ /
# n04 (test subject)
#
# Initial state: All links are fiber. n04 has two candidate parents at
# depth 1: n02 and n03. Both have identical costs (~1.01), so n04 picks
# n02 (smaller NodeAddr, tiebreak rule).
#
# Mutation: Stochastic netem mutation with a single "degraded" policy
# (Bluetooth-like: 15-40ms delay, 2-8% loss). With fraction=0.5, on
# average 2 of 4 edges degrade each round. Over 12 mutation rounds
# (180s / 15s interval), n02-n04 will be degraded in some rounds.
#
# Expected behavior: When n02-n04 is degraded and n03-n04 stays fiber
# (or vice versa), periodic re-evaluation detects the cost asymmetry
# and switches parents. Look for "trigger = periodic" in logs.
#
# FIPS overrides: reeval_interval_secs=15 (vs default 60) to increase
# the chance of catching a cost asymmetry within the mutation window.
#
# Validation: grep n04 logs for "Parent switched via periodic cost
# re-evaluation" (trigger=periodic). If mutation never creates enough
# asymmetry in a particular seed, the test still validates that periodic
# re-eval runs without interference.
scenario:
name: "cost-reeval"
seed: 42
duration_secs: 180
topology:
algorithm: explicit
num_nodes: 4
params:
adjacency:
- [n01, n02]
- [n01, n03]
- [n02, n04]
- [n03, n04]
subnet: "172.20.0.0/24"
ip_start: 10
netem:
enabled: true
default_policy:
# Fiber-like baseline
delay_ms: [1, 5]
jitter_ms: [0, 1]
loss_pct: [0, 0.5]
mutation:
interval_secs: {min: 12, max: 18}
fraction: 0.5
policies:
degraded:
delay_ms: [15, 40]
jitter_ms: [5, 15]
loss_pct: [2, 8]
link_flaps:
enabled: false
traffic:
enabled: true
max_concurrent: 1
interval_secs: {min: 15, max: 30}
duration_secs: {min: 5, max: 10}
parallel_streams: 2
logging:
rust_log: "info"
output_dir: "./sim-results"
fips_overrides:
node:
tree:
reeval_interval_secs: 15
testing/chaos/scenarios/cost-stability.yaml
-72
View File
@@ -1,72 +0,0 @@
# Cost-Based Parent Selection: Hysteresis Stability Test
#
# Topology (explicit 4-node diamond, symmetric):
#
# n01 (root)
# / \
# wifi wifi
# / \
# n02 n03
# \ /
# wifi wifi
# \ /
# n04 (test subject)
#
# All links are WiFi-like with similar characteristics. Aggressive
# netem mutation shifts link qualities every 10-20s, but the changes
# stay within the 20% hysteresis band. n04 should pick one parent
# and mostly stick with it — flapping indicates insufficient hysteresis.
#
# Validation: count "Parent switched" in n04 logs, expect <= 5 switches
# over the full 180s duration.
scenario:
name: "cost-stability"
seed: 42
duration_secs: 180
topology:
algorithm: explicit
num_nodes: 4
params:
adjacency:
- [n01, n02]
- [n01, n03]
- [n02, n04]
- [n03, n04]
subnet: "172.20.0.0/24"
ip_start: 10
netem:
enabled: true
default_policy:
# WiFi baseline
delay_ms: [5, 20]
jitter_ms: [2, 5]
loss_pct: [1, 3]
mutation:
interval_secs: {min: 10, max: 20}
fraction: 1.0
policies:
slightly_better:
delay_ms: [3, 8]
jitter_ms: [1, 3]
loss_pct: [0, 1]
slightly_worse:
delay_ms: [15, 25]
jitter_ms: [3, 8]
loss_pct: [2, 4]
link_flaps:
enabled: false
traffic:
enabled: true
max_concurrent: 2
interval_secs: {min: 15, max: 30}
duration_secs: {min: 5, max: 15}
parallel_streams: 2
logging:
rust_log: "info"
output_dir: "./sim-results"
testing/chaos/scenarios/depth-vs-cost.yaml
-71
View File
@@ -1,71 +0,0 @@
# Cost-Based Parent Selection: Deeper Fiber Beats Shallow Bluetooth
#
# Topology (explicit 4-node):
#
# n01 (root)
# / \
# fiber BT
# / \
# n02 n04 (test subject)
# | /
# fiber fiber
# | /
# n03
#
# n04 has two candidate parents:
# - n01 (root, depth 0) via Bluetooth L2CAP: effective_depth = 0 + ~1.4 = ~1.4
# - n03 (depth 2) via fiber: effective_depth = 2 + ~1.01 = ~3.01
#
# Without cost-based selection, n04 would pick n01 (depth 0 < depth 2).
# With cost-based selection and these Bluetooth impairments, n04 may
# still pick n01 since the Bluetooth cost (~1.4) is modest. This tests
# that the cost formula correctly weighs depth against link quality.
#
# Validation: tree snapshot shows n04's parent selection reflects the
# actual cost tradeoff between depth and link quality.
scenario:
name: "depth-vs-cost"
seed: 42
duration_secs: 45
topology:
algorithm: explicit
num_nodes: 4
params:
adjacency:
- [n01, n02]
- [n02, n03]
- [n03, n04]
- [n01, n04]
subnet: "172.20.0.0/24"
ip_start: 10
netem:
enabled: true
default_policy:
# Fiber-like
delay_ms: [1, 5]
jitter_ms: [0, 1]
loss_pct: [0, 0.5]
link_policies:
# Bluetooth (L2CAP) link from n01 to n04
- edges: ["n01-n04"]
policy:
delay_ms: [15, 40]
jitter_ms: [5, 15]
loss_pct: [2, 8]
link_flaps:
enabled: false
traffic:
enabled: true
max_concurrent: 1
interval_secs: {min: 15, max: 30}
duration_secs: {min: 5, max: 10}
parallel_streams: 2
logging:
rust_log: "info"
output_dir: "./sim-results"
testing/chaos/scenarios/ethernet-mesh.yaml
+15
View File
@@ -62,6 +62,21 @@ link_flaps:
traffic:
enabled: false
# Baseline: the mesh came up, agreed on a root, and took parents. This
# asserts nothing about Ethernet link behaviour under flaps; it exists
# so that a run in which the mesh never formed cannot report success,
# which until now it could, because this scenario carried no assertions
# at all.
#
# Six nodes, one root, five parented in all six provably-completed
# archived runs. The other assertion covering Ethernet transport in
# CI; see ethernet-only for why that matters.
assertions:
baseline:
min_nodes_reporting: 6
max_roots: 1
min_nodes_parented: 5
logging:
rust_log: "info"
output_dir: "./sim-results"
testing/chaos/scenarios/ethernet-only.yaml
+20
View File
@@ -41,6 +41,26 @@ link_flaps:
traffic:
enabled: false
# Baseline: the mesh came up, agreed on a root, and took parents. This
# asserts nothing about what Ethernet transport does; it exists so that
# a run in which the mesh never formed cannot report success, which
# until now it could, because this scenario carried no assertions at
# all.
#
# A 4-node mesh forms a spanning tree: one root and three nodes
# with a parent. All six provably-completed archived runs show exactly
# that, so these are the shape of a converged mesh rather than a
# tolerance fitted to observations. This is the only assertion covering
# Ethernet transport anywhere in CI, and it covers the control plane
# only: traffic is disabled above, so no datagram crosses an Ethernet
# link in any test. Framing, the length field that trims NIC minimum-
# frame padding, and AEAD over Ethernet are all unexercised as a result.
assertions:
baseline:
min_nodes_reporting: 4
max_roots: 1
min_nodes_parented: 3
logging:
rust_log: "info"
output_dir: "./sim-results"
testing/chaos/scenarios/mixed-technology.yaml
-103
View File
@@ -1,103 +0,0 @@
# Mixed Technology: 10-node heterogeneous network
#
# Explicit topology with Bluetooth (L2CAP), WiFi, and fiber links.
# Tests that cost-based parent selection produces a tree favoring
# low-cost paths when multiple link technologies coexist.
#
# Topology:
#
# n01 (root)
# / | \
# f f f
# / | \
# n02 n03 n04
# | \ | \ | \
# f BT f f BT f
# | \ | | \ |
# n05 n06 n07 n08 n09
# \ /
# wifi---wifi
# n10
#
# Edges and link types:
# Fiber: n01-n02, n01-n03, n01-n04, n02-n05, n03-n07, n04-n09
# Bluetooth: n02-n06, n04-n08
# WiFi: n03-n06, n08-n10
# Fiber: n03-n08, n06-n10
#
# Test subjects:
# - n06 has fiber (n03) and Bluetooth (n02) parents — should pick n03
# - n08 has fiber (n03) and Bluetooth (n04) parents — should pick n03
#
# Netem mutation shifts fiber-only links between normal and degraded.
scenario:
name: "mixed-technology"
seed: 42
duration_secs: 90
topology:
algorithm: explicit
num_nodes: 10
params:
adjacency:
- [n01, n02]
- [n01, n03]
- [n01, n04]
- [n02, n05]
- [n02, n06]
- [n03, n06]
- [n03, n07]
- [n03, n08]
- [n04, n08]
- [n04, n09]
- [n06, n10]
- [n08, n10]
subnet: "172.20.0.0/24"
ip_start: 10
netem:
enabled: true
default_policy:
# Fiber-like defaults
delay_ms: [1, 5]
jitter_ms: [0, 1]
loss_pct: [0, 0.5]
link_policies:
# Bluetooth (L2CAP) links
- edges: ["n02-n06", "n04-n08"]
policy:
delay_ms: [15, 40]
jitter_ms: [5, 15]
loss_pct: [2, 8]
# WiFi links (moderate latency, low loss)
- edges: ["n03-n06", "n08-n10"]
policy:
delay_ms: [5, 20]
jitter_ms: [2, 5]
loss_pct: [1, 3]
mutation:
interval_secs: {min: 30, max: 60}
fraction: 0.2
policies:
normal:
delay_ms: [1, 10]
loss_pct: [0, 1]
degraded:
delay_ms: [50, 100]
jitter_ms: [10, 30]
loss_pct: [3, 8]
link_flaps:
enabled: false
traffic:
enabled: true
max_concurrent: 3
interval_secs: {min: 10, max: 30}
duration_secs: {min: 5, max: 15}
parallel_streams: 4
logging:
rust_log: "info"
output_dir: "./sim-results"
testing/chaos/scenarios/smoke-10.yaml
-27
View File
@@ -1,27 +0,0 @@
scenario:
name: "smoke-10"
seed: 42
duration_secs: 30
topology:
num_nodes: 10
algorithm: random_geometric
params:
radius: 0.5
ensure_connected: true
subnet: "172.20.0.0/24"
ip_start: 10
# Phase 2+ features (disabled for MVP)
netem:
enabled: false
link_flaps:
enabled: false
traffic:
enabled: false
logging:
rust_log: "info"
output_dir: "./sim-results"
testing/chaos/scenarios/tcp-mesh.yaml
+13
View File
@@ -61,6 +61,19 @@ link_flaps:
traffic:
enabled: false
# Baseline: the mesh came up, agreed on a root, and took parents. This
# asserts nothing about TCP transport specifically; it exists so that a
# run in which the mesh never formed cannot report success, which until
# now it could, because this scenario carried no assertions at all.
#
# Six nodes, one root, five parented in all six provably-completed
# archived runs.
assertions:
baseline:
min_nodes_reporting: 6
max_roots: 1
min_nodes_parented: 5
logging:
rust_log: "info"
output_dir: "./sim-results"
testing/chaos/scripts/chaos.sh
+9 -1
View File
@@ -65,6 +65,7 @@ VERBOSE=""
SEED=""
DURATION=""
NODES=""
SUBNET=""
while [ $# -gt 0 ]; do
case "$1" in
@@ -72,6 +73,7 @@ while [ $# -gt 0 ]; do
--seed) SEED="$2"; shift 2 ;;
--duration) DURATION="$2"; shift 2 ;;
--nodes) NODES="$2"; shift 2 ;;
--subnet) SUBNET="$2"; shift 2 ;;
--list) list_scenarios ;;
-*) echo "Error: Unknown option '$1'" >&2; usage ;;
*)
@@ -123,7 +125,11 @@ if ! docker info &> /dev/null; then
exit 1
fi
DOCKER_DIR="$CHAOS_DIR/../docker"
# A harness that scopes its build context per run passes it in
# FIPS_BUILD_CONTEXT and stops writing to the shared directory, so checking the
# shared one would either fail on a clean checkout or, worse, pass while
# reading a stale binary that is not the one under test.
DOCKER_DIR="${FIPS_BUILD_CONTEXT:-$CHAOS_DIR/../docker}"
if [ ! -f "$DOCKER_DIR/fips" ]; then
echo "Error: FIPS binary not found at $DOCKER_DIR/fips" >&2
echo "Run testing/scripts/build.sh first" >&2
@@ -135,6 +141,7 @@ PYTHON_ARGS=("$SCENARIO_FILE")
[ -n "$VERBOSE" ] && PYTHON_ARGS+=("$VERBOSE")
[ -n "$SEED" ] && PYTHON_ARGS+=("--seed" "$SEED")
[ -n "$DURATION" ] && PYTHON_ARGS+=("--duration" "$DURATION")
[ -n "$SUBNET" ] && PYTHON_ARGS+=("--subnet" "$SUBNET")
echo "=== FIPS Stochastic Simulation ==="
echo ""
@@ -143,6 +150,7 @@ echo " File: $SCENARIO_FILE"
[ -n "$SEED" ] && echo " Seed: $SEED (override)"
[ -n "$DURATION" ] && echo " Duration: ${DURATION}s (override)"
[ -n "$NODES" ] && echo " Nodes: $NODES (override)"
[ -n "$SUBNET" ] && echo " Subnet: $SUBNET (override)"
echo ""
# If --nodes is specified, create a patched copy of the scenario file
testing/chaos/sim/__main__.py
+17
View File
@@ -25,6 +25,13 @@ def main():
"--duration", type=int, default=None,
help="Override scenario duration in seconds",
)
parser.add_argument(
"--subnet", type=str, default=None,
help="Pin the topology subnet CIDR (e.g. 10.30.0.0/24) instead of "
"claiming a free one. The sim normally claims a range so "
"concurrent runs cannot collide; pin only when you need a known "
"range, and expect a hard failure if it is already taken.",
)
args = parser.parse_args()
level = logging.DEBUG if args.verbose else logging.INFO
@@ -48,10 +55,20 @@ def main():
print("Error: --duration must be >= 1", file=sys.stderr)
sys.exit(1)
scenario.duration_secs = args.duration
if args.subnet is not None:
# Pinning opts out of claiming. The runner still *creates* the network,
# so a range already in use fails loudly here rather than silently
# overlapping a concurrent run.
scenario.topology.pinned_subnet = args.subnet
runner = SimRunner(scenario)
result = runner.run()
# Liveness before content. A simulation that raised on its way up, or part
# way through, has no panic count or assertion outcome worth reporting, and
# saying so is more useful than whatever partial content it did produce.
if runner.aborted:
sys.exit(4)
if result and result.panics:
sys.exit(2)
if runner.assertions_failed:
testing/chaos/sim/assertions.py
+284 -3
View File
@@ -18,7 +18,15 @@ import logging
from dataclasses import dataclass
from .control import snapshot_all_bloom
from .scenario import BloomSendRateAssertion, MinParentSwitchesAssertion
from .scenario import (
BaselineAssertion,
BloomSendRateAssertion,
CongestionSignalsAssertion,
MaxErrorsAssertion,
MaxParentSwitchesAssertion,
MinParentSwitchesAssertion,
TreeParentsAssertion,
)
from .topology import SimTopology
log = logging.getLogger(__name__)
@@ -131,6 +139,278 @@ class BloomSendRateMonitor:
)
def evaluate_max_parent_switches(
cfg: MaxParentSwitchesAssertion,
parent_switch_count: int,
scope: str,
) -> AssertionOutcome:
"""Stability ceiling on parent switches over the run.
``scope`` describes what was counted and appears in the message, so a
reader can tell a per-node result from a mesh-wide one. Resolving a
per-node scope to a real node is the caller's job, and so is failing
loudly when it cannot: an unresolvable node would count zero switches
and sail under any ceiling without having observed anything.
"""
if parent_switch_count <= cfg.max_total:
return AssertionOutcome(
name="max_parent_switches",
passed=True,
detail=(
f"PASS max_parent_switches: {parent_switch_count} switches "
f"({scope}) <= ceiling {cfg.max_total}"
),
)
return AssertionOutcome(
name="max_parent_switches",
passed=False,
detail=(
f"FAIL max_parent_switches: {parent_switch_count} switches "
f"({scope}) > ceiling {cfg.max_total} — the tree is reparenting "
f"more than the hysteresis band should allow. Check whether a "
f"cost change smaller than the hysteresis margin is still "
f"triggering a switch."
),
)
def evaluate_baseline(
cfg: BaselineAssertion,
snapshot: dict | None,
sessions: int,
) -> AssertionOutcome:
"""Floor on the mesh having formed: nodes answered, agreed a root, took parents."""
if not snapshot:
return AssertionOutcome(
name="baseline",
passed=False,
detail=(
"FAIL baseline: no final tree snapshot was taken, so nothing "
"about the mesh was observed. This is a harness failure."
),
)
reporting = len(snapshot)
roots = {v.get("root") for v in snapshot.values() if v.get("root")}
parented = sum(
1 for v in snapshot.values()
if v.get("parent") and v.get("parent") != v.get("my_node_addr")
)
parts, failures = [], []
def note(ok, text):
parts.append(text)
if not ok:
failures.append(text)
if cfg.min_nodes_reporting is not None:
note(reporting >= cfg.min_nodes_reporting,
f"{reporting} node(s) answered (need {cfg.min_nodes_reporting})")
if cfg.max_roots is not None:
note(len(roots) <= cfg.max_roots and len(roots) >= 1,
f"{len(roots)} distinct root(s) (allowed {cfg.max_roots})")
if cfg.min_nodes_parented is not None:
note(parented >= cfg.min_nodes_parented,
f"{parented} node(s) have a parent (need {cfg.min_nodes_parented})")
if cfg.min_sessions is not None:
note(sessions >= cfg.min_sessions,
f"{sessions} session(s) established (need {cfg.min_sessions})")
summary = "; ".join(parts)
if failures:
return AssertionOutcome(
name="baseline",
passed=False,
detail=(
f"FAIL baseline: {'; '.join(failures)}. Full: {summary}"
),
)
return AssertionOutcome(
name="baseline", passed=True, detail=f"PASS baseline: {summary}"
)
def evaluate_tree_parents(
cfg: TreeParentsAssertion,
snapshot: dict | None,
) -> AssertionOutcome:
"""Check each node's parent in the final tree snapshot.
Parents are compared by node address, resolved from the snapshot's own
``my_node_addr`` fields, so the check does not depend on the display
name a node happened to publish.
Every way of not knowing the answer is a failure: no snapshot, the
node absent from it, the expected parent absent from it, or the node
still claiming to be its own root. Each of those produces the same
"no match" that a genuinely wrong parent does, and only saying so
separately keeps a harness problem from reading as a routing verdict.
"""
if not snapshot:
return AssertionOutcome(
name="tree_parents",
passed=False,
detail=(
"FAIL tree_parents: no final tree snapshot was taken, so no "
"node's parent was observed. This is a harness failure, not "
"a statement about the tree."
),
)
addr_of = {
nid: data.get("my_node_addr")
for nid, data in snapshot.items()
if data.get("my_node_addr")
}
id_of = {addr: nid for nid, addr in addr_of.items()}
good, bad = [], []
for child, want_parent in sorted(cfg.expected.items()):
entry = snapshot.get(child)
if entry is None:
bad.append(
f"{child} is absent from the snapshot ({len(snapshot)} node(s) "
f"present: {', '.join(sorted(snapshot))})"
)
continue
want_addr = addr_of.get(want_parent)
if want_addr is None:
bad.append(
f"{child}: expected parent {want_parent} is absent from the "
f"snapshot, so its address cannot be resolved"
)
continue
got_addr = entry.get("parent")
if got_addr == entry.get("my_node_addr"):
bad.append(
f"{child} is its own parent — it still believes it is root, "
f"so the tree never converged around it (wanted {want_parent})"
)
continue
if got_addr == want_addr:
good.append(f"{child}->{want_parent}")
continue
got_id = id_of.get(got_addr) or entry.get("parent_display_name") or got_addr
bad.append(f"{child} chose {got_id}, wanted {want_parent}")
if bad:
detail = f"FAIL tree_parents: {'; '.join(bad)}"
if good:
detail += f". Correct: {', '.join(good)}"
return AssertionOutcome(name="tree_parents", passed=False, detail=detail)
return AssertionOutcome(
name="tree_parents",
passed=True,
detail=f"PASS tree_parents: {', '.join(good)}",
)
_CONGESTION_FLOORS = (
("min_nodes_detected", "congestion_detected"),
("min_nodes_ce_forwarded", "ce_forwarded"),
("min_nodes_ce_received", "ce_received"),
)
def evaluate_congestion_signals(
cfg: CongestionSignalsAssertion,
snapshot: dict | None,
) -> AssertionOutcome:
"""Floors on how many nodes observed each congestion counter.
``snapshot`` is the final congestion snapshot keyed by node id. None
means the snapshot never ran, which fails: a missing snapshot and a
mesh that observed no congestion produce the same zero counts, and
treating them alike is how an assertion comes to pass on an absence
of evidence.
"""
if not snapshot:
return AssertionOutcome(
name="congestion_signals",
passed=False,
detail=(
"FAIL congestion_signals: no final congestion snapshot was "
"taken, so no node was observed at all. This is a harness "
"failure, not a statement about congestion."
),
)
parts, failures = [], []
for attr, counter in _CONGESTION_FLOORS:
floor = getattr(cfg, attr)
if floor is None:
continue
hits = sorted(
nid for nid, data in snapshot.items()
if (data.get("congestion") or {}).get(counter, 0) > 0
)
parts.append(f"{counter}: {len(hits)} node(s) >0 (floor {floor})")
if len(hits) < floor:
failures.append(
f"{counter} non-zero on {len(hits)} node(s), need {floor}"
)
else:
parts[-1] += f" [{', '.join(hits)}]"
summary = "; ".join(parts)
if failures:
return AssertionOutcome(
name="congestion_signals",
passed=False,
detail=(
f"FAIL congestion_signals: {'; '.join(failures)} — across "
f"{len(snapshot)} node(s) sampled. Full counts: {summary}"
),
)
return AssertionOutcome(
name="congestion_signals",
passed=True,
detail=f"PASS congestion_signals: {summary}",
)
def evaluate_max_errors(
cfg: MaxErrorsAssertion,
errors: list[tuple[str, str]],
) -> AssertionOutcome:
"""Ceiling on ERROR-level log lines across the whole mesh.
``errors`` is the ``AnalysisResult.errors`` list of ``(source, line)``
pairs rather than a bare count, so a failure can name the nodes and
quote the lines. A ceiling breach that only reports a number sends the
reader back to the logs it was supposed to save them reading.
"""
count = len(errors)
if count <= cfg.max_total:
return AssertionOutcome(
name="max_errors",
passed=True,
detail=(
f"PASS max_errors: {count} ERROR line(s) mesh-wide <= "
f"ceiling {cfg.max_total}"
),
)
per_node: dict[str, int] = {}
for source, _line in errors:
per_node[source] = per_node.get(source, 0) + 1
worst = sorted(per_node.items(), key=lambda kv: -kv[1])
breakdown = ", ".join(f"{src}={n}" for src, n in worst)
samples = "\n".join(
f" [{src}] {line.strip()}" for src, line in errors[:5]
)
return AssertionOutcome(
name="max_errors",
passed=False,
detail=(
f"FAIL max_errors: {count} ERROR line(s) mesh-wide > ceiling "
f"{cfg.max_total} — per node: {breakdown}. First "
f"{min(5, count)}:\n{samples}"
),
)
def evaluate_min_parent_switches(
cfg: MinParentSwitchesAssertion,
parent_switch_count: int,
@@ -147,7 +427,7 @@ def evaluate_min_parent_switches(
passed=True,
detail=(
f"PASS min_parent_switches: {parent_switch_count} switches "
f">= floor {cfg.min_total}"
f"(mesh-wide) >= floor {cfg.min_total}"
),
)
return AssertionOutcome(
@@ -155,7 +435,8 @@ def evaluate_min_parent_switches(
passed=False,
detail=(
f"FAIL min_parent_switches: {parent_switch_count} switches "
f"< floor {cfg.min_total} — harness did not induce sufficient "
f"(mesh-wide) < floor {cfg.min_total} — harness did not induce "
f"sufficient "
f"parent flapping; bloom-rate assertion would be trivially "
f"true. Check tree-snapshot-warmup.json: did the expected "
f"node win the root election?"
testing/chaos/sim/compose.py
+23 -9
View File
@@ -10,8 +10,13 @@ from .scenario import Scenario
from .topology import SimTopology
# Image name for the pre-built FIPS test image.
# The runner builds this once before starting containers.
FIPS_SIM_IMAGE = "fips-test:latest"
#
# A harness that has already built an image passes it in FIPS_TEST_IMAGE, and
# it is then the caller's image: the runner uses it and must not rebuild it.
# Unset means a bare run, where the shared tag is the right name and the runner
# still builds it. Read at import, which is safe because the simulation always
# starts as a child process with the environment already set.
FIPS_SIM_IMAGE = os.environ.get("FIPS_TEST_IMAGE", "fips-test:latest")
# Jinja2 template for the compose file.
# Uses a pre-built image instead of per-service build to support large topologies.
@@ -19,10 +24,12 @@ _COMPOSE_TEMPLATE = Template(
"""\
networks:
fips-net:
driver: bridge
ipam:
config:
- subnet: {{ subnet }}
# External, and created by the sim rather than by compose. The range has to
# be *claimed* -- attempt-create, advance on docker's own overlap error --
# so that two concurrent runs cannot select the same one, and only the
# process that creates the network can do that. See sim/netclaim.py.
external: true
name: {{ network_name }}
x-fips-common: &fips-common
image: {{ image }}
@@ -45,7 +52,7 @@ services:
{% for node in nodes %}
{{ node.node_id }}:
<<: *fips-common
container_name: fips-node-{{ node.node_id }}
container_name: {{ topology.container_name(node.node_id) }}
hostname: {{ node.node_id }}
volumes:
- ./{{ node.node_id }}.yaml:/etc/fips/fips.yaml:ro
@@ -62,8 +69,14 @@ def generate_compose(
topology: SimTopology,
scenario: Scenario,
output_dir: str,
network_name: str,
) -> str:
"""Render docker-compose.yml and write to output_dir. Returns the file path."""
"""Render docker-compose.yml and write to output_dir. Returns the file path.
``network_name`` is the docker network the sim has already claimed; the
compose file refers to it as external rather than declaring a subnet, so
that the claim and the creation are the same operation.
"""
os.makedirs(output_dir, exist_ok=True)
nodes = [topology.nodes[nid] for nid in sorted(topology.nodes)]
@@ -75,11 +88,12 @@ def generate_compose(
)
content = _COMPOSE_TEMPLATE.render(
subnet=scenario.topology.subnet,
network_name=network_name,
rust_log=scenario.logging.rust_log,
image=FIPS_SIM_IMAGE,
nodes=nodes,
resolv_conf=resolv_conf,
topology=topology,
)
path = os.path.join(output_dir, "docker-compose.yml")
testing/chaos/sim/docker_exec.py
+70 -8
View File
@@ -7,6 +7,34 @@ import logging
log = logging.getLogger(__name__)
# `docker compose` renders its progress UI on stderr, so a failed command can
# have hundreds of lines of captured output behind it. Log the tail: whatever
# the daemon refused is the last thing it wrote.
OUTPUT_TAIL_CHARS = 2000
def _tail(text: str) -> str:
"""Trim captured output to its last few KB for logging."""
text = text.strip()
if not text:
return "(empty)"
if len(text) <= OUTPUT_TAIL_CHARS:
return text
return "...\n" + text[-OUTPUT_TAIL_CHARS:]
def _decode(output) -> str:
"""Render captured output as text.
A timed-out command hands back what it had written as bytes, even when
the call asked for text, so the timeout path cannot assume either.
"""
if output is None:
return ""
if isinstance(output, bytes):
return output.decode(errors="replace")
return output
class DockerExecError(Exception):
def __init__(self, container, cmd, returncode, stderr):
@@ -47,16 +75,50 @@ def docker_compose(
timeout: int = 300,
check: bool = True,
) -> subprocess.CompletedProcess:
"""Run a docker compose command with the given compose file."""
"""Run a docker compose command with the given compose file.
Output is captured, so a failure's stderr is logged here before anything
else sees it. `CalledProcessError` reports only the argv and the exit
status, and a `check=False` caller reads neither, so without this the one
place the daemon says what it objected to -- a container name already in
use, an unusable subnet, a missing image -- is captured and then thrown
away. Nothing about the success path changes.
"""
cmd = ["docker", "compose", "-f", compose_file] + args
log.info("Running: %s", " ".join(cmd))
return subprocess.run(
cmd,
capture_output=True,
text=True,
timeout=timeout,
check=check,
)
try:
result = subprocess.run(
cmd,
capture_output=True,
text=True,
timeout=timeout,
)
except subprocess.TimeoutExpired as e:
# A timeout raises from inside communicate(), so the return-code
# branch below never runs and this is the only chance to say what
# the command had managed to emit. The partials arrive as bytes
# even under text=True.
log.error(
"%s timed out after %ds\nstderr: %s\nstdout: %s",
" ".join(cmd),
timeout,
_tail(_decode(e.stderr)),
_tail(_decode(e.stdout)),
)
raise
if result.returncode != 0:
log.error(
"%s exited %d\nstderr: %s\nstdout: %s",
" ".join(cmd),
result.returncode,
_tail(result.stderr),
_tail(result.stdout),
)
if check:
raise subprocess.CalledProcessError(
result.returncode, cmd, result.stdout, result.stderr
)
return result
def is_container_running(container: str) -> bool:

Some files were not shown because too many files have changed in this diff Show More