The guard that keeps "local green" and "GitHub green" meaning the same thing was invoked by nothing. Its only entry point was an exec behind an explicit --check-parity flag, which by exec-ing could not be combined with an actual run, and no workflow step called it at all. A guard nobody runs is a guard that does not exist, which is how the mixed-profile divergence on next survived unnoticed.
It now runs as the first stage of every local run and as its own job on GitHub. Locally it reports through the same result-recording path as every other stage, so a divergence sets the run's exit status instead of scrolling past. It is deliberately placed above the mode branches, so a divergence also fails --only, --test-only and --build-only runs: whichever subset was asked for, the claim that a local result means what a GitHub result means is what has broken. The GitHub side installs pyyaml explicitly rather than assuming the runner image carries it.
--check-parity keeps working exactly as before. Three comment blocks that described the old folded comparison are corrected here, in the workflow, the local runner and the testing README; they were spread across three files and only two mention the guard by name, so the third is best found by searching for what it claims rather than for the script.
The guard collapsed all thirteen chaos legs to the token "chaos" and all five deb-install legs to "deb-install", so sixteen of the thirty-four integration legs were invisible to it. Deleting twelve chaos legs and four deb-install legs from a copy of the workflow left it still reporting "CI parity OK". The fold existed because the guard compared the cosmetic suite name, which really does differ between the runners; both sides have carried a directly comparable scenario field all along, so the fix is to compare that instead, along with the chaos flags.
The local suite set is now discovered by sweeping ci-local.sh for suite arrays rather than reading a hardcoded list of variable names, and the deb-install distro list is read from the suite's own script. A suite dispatched with no backing array is invisible to either approach, so every run_suite arm is now checked to have one and the guard names any that does not. That is not hypothetical: next dispatches its mixed-profile suite without an array, and the guard could only report it as missing from the local runner without being able to say why.
The guard also now refuses to run rather than failing obscurely when python3 or pyyaml is absent, since a parity check that cannot run must not look like one that passed. Verified by breaking each thing it guards: legs deleted from the workflow, drifted chaos flags, a fabricated suite array, and a dispatch arm whose array was removed. That last case initially did not fire because the arm pattern was pinned to the wrong indentation, which the test caught.
The scenario loader read every section and member through raw.get with a default, so a mistyped key was silently ignored and the block it belonged to stayed default-constructed while the YAML still looked correct on the page. Writing "assertion:" for "assertions:" disarmed a scenario's only assertions and left it unable to return a non-zero exit code; "link_flap:" for "link_flaps:" turned off the churn the scenario existed to inject, and a calm mesh then ran under the name of a chaos test.
Keys are now checked at load time, at the top level and inside every section with a fixed schema, including the netem policy bodies, the {min, max} ranges and the assertion bodies themselves. The error names the offending key, the section it appeared in, and the keys that section does understand. Three mappings are deliberately exempt because their keys are names the author chooses rather than a schema, and two sub-trees are passed through whole; all five are listed at the key sets with the reason.
Adding an assertion type now means registering it, so the next person to add one gets a rejection rather than a silent no-op. That coupling is the point and it is written down where they will see it. Verified against the seventeen scenarios in the tree, all of which still load, and against fixtures for each failure mode: a top-level typo, a nested typo, a typo inside an assertion body, and a section left empty, which previously raised an unrelated AttributeError from the next line of the loader.
The e2e scenarios build fips and fips-gateway in a builder image and extract them with two docker cp calls whose stderr went to /dev/null and whose exit status was never tested. Because build_fips_for_e2e ended in a log call it returned 0 on every path, so the caller's guard could not fire. A failed extraction left the previous run's binaries in the cache, they satisfied the caller's executable check, and the five e2e legs then exercised the previous commit's code and reported green.
The extraction now deletes the cached binaries before it starts, which is what makes the stale case impossible rather than merely detected, and it checks each docker cp independently, reports the failure with docker's own message, verifies each extracted file is non-empty and executable, and returns an explicit status. This mirrors what the deb-install suite already does.
Confirmed by inducing the failure rather than by observing a green run. With both extractions broken and a stale binary in place, the old code logged "Cached fips (26936 bytes)" and went on to build the runtime image around it; the new code reports both docker errors, deletes the stale binaries and fails the suite. Breaking either extraction on its own is caught separately, and an unmodified run passes 7 of 7.
Compose commands run with their output captured and with check set, so a failure raised a CalledProcessError and nothing ever looked at what the daemon had said. That exception reports the argv and the exit status and nothing else, so every scenario that died bringing its containers up left a runner log saying only that docker compose returned non-zero exit status 1, and the reason was captured and then discarded. In the run that prompted this work twelve scenarios failed that way and why they failed can no longer be established at all. Log the captured stderr and stdout before the error goes anywhere, keeping the tail of each so a verbose failure cannot fill the log file. A command that times out raises before its exit status is ever read, so that path records whatever it had emitted too, rather than leaving the same hole one shape over. Reproduced against both of the shapes this is meant to catch: an unusable network prefix now records the daemon's parse error, and a name already held by another container now records the conflict and names the container holding it. The success path logs nothing new, and a completed scenario's runner log is unchanged line for line.
The raising behaviour is deliberately untouched, since the abort flag and the exit code that came with it depend on it. The check is applied here rather than by subprocess so that a caller who passed check false gets the same record: both such callers are the compose down in teardown, which today can leave a mesh behind without saying so.
The same omission was costing the veth path its reasons. Failing ip commands had their stderr logged at debug, which the runner does not emit unless asked for verbose output, so a failed pair creation reached the log as nothing more than the names of the two interfaces it could not create. Raise that to a warning. The deletes issued to clear a stale pair are expected to fail and still say nothing.
Teardown runs from a finally, so it also runs after a failed setup, and it was guarded only on the topology and the compose file, both of which are set well before any container exists. A scenario that died bringing its containers up therefore ran the whole harvest anyway: final snapshots, docker logs, the analysis, the assertions and the metadata, all addressing containers by names that are global to the host. What it left behind was not an empty directory an investigator would notice but a full and plausible one, in the recorded case ten node logs and an analysis reporting two promotions and two parent switches, every byte of it from a different scenario's mesh. Gate the harvest on whether the containers ever started, and leave the compose down outside that gate so a partly successful start still gets cleaned up.
Record the outcome in a status file in every result directory, naming the run as completed, interrupted, aborted, setup-failed or teardown-failed, alongside the scenario, the seed and the container names it used. With the harvest gated, the presence of an analysis file is now itself proof that the scenario's own mesh existed, and the status file says which of the several ways a run can end applies. A run cut short by a signal keeps the existing exit codes rather than gaining one of its own: what it collected before stopping is real and still worth reporting, the status file records that it was truncated, and every wrapper already reports a Ctrl-C of its own.
Log collection never looked at the status of docker logs. It concatenated stdout and stderr unconditionally, so collecting from containers that were not there wrote the daemon's "No such container" reply into each node log and analysed the result as a mesh with no panics, no errors and no sessions, which reads exactly like a clean run and exits zero. Check the return code, and treat a harvest that cannot read every container, or that reads none, as a failed teardown. A teardown that raises now also reports on the same footing as a run that never started, instead of escaping past the exit codes as a bare traceback and being read as a malformed command line.
The simulation caught every exception a scenario raised, logged it, and exited 0, so a scenario that died during setup was recorded as a pass. Local CI discarded the exit code in any case: run_chaos ends in a call whose own last statement is an echo, so it returned 0 whatever the scenario did, and the parallel launcher's wait therefore always succeeded and always recorded a pass. Between them no chaos failure of any kind could turn a local run red, and none could since the script was written.
Carry the abort out of the run and give it its own exit code, ordered ahead of the two content codes so a run that never produced a mesh cannot report that mesh's panic and assertion counts. Return the scenario's status from run_chaos so the launcher sees it. The documented exit codes were stale before this change and are rewritten in full, including that the argument parser rejects a malformed command line with the same code that reports panics.
Expect scenarios that have been reporting green to start reporting red. In the last recorded local run twelve of the thirteen scenarios never got past starting their containers, and all thirteen were counted as passes; those that now execute have not run for months, and their assertions have never been evaluated against a mesh of their own.
Ethernet edges get a veth pair created in the host namespace and then moved into the containers, named from the node ids alone. ethernet-mesh and ethernet-only both connect n01 to n04 and always run together, so one tore down the other's live link. Hash the scenario suffix to four hex characters and insert those after the vh prefix; 15 characters is far too few to carry the suffix itself.
The names are now run-specific and never regenerated, so the delete-before-create no longer reclaims an interface an earlier run abandoned. ci-cleanup.sh takes over, deriving the names from the suffixes a run's teardown hands it. It is the first thing that script removes that is neither a docker object nor labelled, so an unscoped reap can sever a running bare simulation's links; testing/README.md and the --reap help say so.
Chaos scenarios run four at a time under local CI, but every one of them claimed the container names fips-node-nNN and wrote its generated configs and compose file to the same generated-configs/sim directory. Container names are global in Docker and are not scoped by the compose project, so concurrent scenarios collided on both, and a scenario could start containers from a compose file another had overwritten.
Thread the existing FIPS_CI_NAME_SUFFIX into the simulation. run_chaos narrows the run-wide suffix to the scenario, and the sim reads it once when the topology is built, applying it to the container names and to the config directory basename. The compose template renders the name through the topology accessor instead of duplicating the literal, so one expression produces every chaos container name.
The suffix is empty when the variable is unset, so a bare chaos.sh run and the hosted CI jobs render byte-identical names and paths. Verified by rendering every scenario's compose file before and after with the variable unset and diffing.
The forwarding path tested the hop limit before testing whether the datagram was
addressed to this node, and decremented only on the forwarding branch. Two
consequences followed. A datagram addressed here that arrived with a zero hop
limit was dropped rather than delivered. And a forwarder receiving a transit
datagram with one hop left transmitted it with zero for the next hop to discard,
wasting a transmission on every expiring datagram.
Delivery to the addressed node is no longer gated on the hop limit, and the
decrement now happens before the drop decision, so a datagram that would leave
with zero is not sent. Saturating subtraction folds the two arrivals that cannot
be transmitted, already exhausted and last hop, into a single test.
The reachable radius does not change, which is worth stating because it is easy
to assume otherwise. The old behavior refused to deliver at zero but was willing
to send at zero, and the new behavior is the mirror of that, so the two cancel: a
path of h links still delivers for any source value of h or more. What this
actually buys is the wasted transmission, the exhaustion counter charging at the
node that makes the decision rather than the one after it, conformance with the
specified semantics, and one real gain during a rolling upgrade, where an
unupgraded forwarder feeding an upgraded destination delivers one hop further
than either version does on its own.
Coordinate cache warming moved ahead of both decisions, so a transit datagram
later dropped for hop limit still contributes its plaintext coordinates. The only
arrivals this newly warms from are those with a zero hop limit, and every insert
they can make is already achievable with a value of one, so it grants nothing
that was not already available and adds no memory growth the cap does not bound.
Four tests added: delivery at zero to this node, the transit drop at one, the
transit pass at two, and the one per hop decrement across a three node chain,
which brackets the emitted value from both sides since it is only observable
through what the next hop does. One existing test was renamed because its
destination was this node, so it never exercised the transit path its name
claimed, and it asserted nothing at all.
Bump the version to 0.4.2-dev and update the status badge and prose. The
0.4.x patch line continues; v0.4.1 remains the shipped release it points at.
Follows the shape of the v0.4.0 rollover commit. No maint reset was needed
this time: v0.4.1 was cut from maint, so the tag is already on this line and
it simply continues.
Drop the -dev suffix from the package version and refresh the lockfile.
The version is single-sourced from CARGO_PKG_VERSION, so build.rs and
src/version.rs need no change.
Also fix the archived v0.4.0 release notes, which still read
"2026-06-21 (provisional)" while the changelog records the actual
release date of 2026-06-27. The date was confirmed in the changelog at
the time but not in the two release-notes files that carry it, so the
shipped notes have been showing a wrong date and the word provisional
since that release. The root mirror is regenerated for v0.4.1, so only
the archived copy needed correcting.
Back-fill the changelog for the six user-facing changes on this line
since v0.4.0: the antipoison FPR cap default, the bloom probe and
secp256k1 context performance work, the coordinate-cache and path-MTU
fixes, and the removed parent_switched counter. The Unreleased section
was empty, so all six entries are new.
Write the v0.4.1 release notes and mirror them to the root copy. The
notes lead with the FPR cap change and state plainly that this is the
second raise of that default in two releases, that it buys headroom
rather than fixing the underlying fixed-filter constraint, and that the
structural remedy is the v2 filter work. They also flag the rolling
upgrade window, where an upgraded and a not-yet-upgraded node can
disagree about mesh size, and point consumers of parent_switched at
parent_switches.
Bump the README status badge and the roadmap prose together so the two
agree.
The inbound FilterAnnounce cap at 0.10 rejects aggregates that are
legitimately near their operating ceiling, before the network reaches
the fixed-filter capacity limit. Raise the default to 0.20, which
corresponds to fill 0.7248 at k=5, about 2,114 entries on the 1 KB
filter (Swamidass-Baldi).
Also remove the duplicate default definitions in BloomConfig. Each
default was written twice, once in impl Default and once in the serde
default function, with nothing enforcing that they agree, so a config
file that omits the key took a different path from one that sets it.
impl Default now delegates to the serde default functions, leaving a
single source of truth.
The isolation loop and the failure-log dump both built container names
without the run suffix, so under a suffixed run they addressed containers
that do not exist.
The consequence was worse than the visible failure. Two of the three
checks in that loop assert a ping FAILS, and a ping into a non-existent
container fails for the wrong reason - so both passed vacuously, and the
security assertion the loop exists for was silently a no-op. Only the
third check, which expects a ping to succeed, noticed anything was wrong.
Add a guard that fails loudly when the container is absent, so a future
naming slip cannot quietly turn these back into no-ops rather than
surfacing as one confusing failure among two false passes.
These sites were missed because the suite passes when run by hand: with no
suffix set the unsuffixed names are correct, and the defect only appears
under the automation that sets one. Verified this time with the suffix
set, which is the condition that matters: 18 of 18, container names
resolving to real containers.
Making the sidecar network name unique per run exposed a latent problem
rather than causing a new one. With a fixed name, compose found any
existing network and reused it, which quietly masked the fact that the
address range was hardcoded. With a unique name there is nothing to
reuse, so the range is requested every time and the run fails outright if
anything already holds it.
Claim the range instead of assuming it. The suite now tries to create its
network on a candidate range and, if the daemon reports an overlap, moves
to the next candidate. The daemon's own address pool becomes the arbiter,
so two concurrent runs cannot land on the same range at all - as opposed
to a range derived from the run identifier, which only makes a collision
unlikely, and a collision is the precise failure being avoided.
The node addresses are derived from whichever range is claimed rather than
written alongside it, including the gateway address the isolation check
probes. Moved to a range that nothing in the tree claims and that sits
outside the daemon's default pool.
Only an overlap is worth advancing on. Any other creation failure is real
and is reported with the daemon's own message, rather than being retried
sixty-four times and buried - a lesson from a failure elsewhere in the
harness this week that was undiagnosable because its error output was
discarded.
All three nodes now attach to the pre-created network as external, so
teardown removes it explicitly.
Verified: three concurrent claims take three distinct ranges; a
non-overlap error fails fast rather than looping; the suite passes 18/18;
and the claimed gateway address is confirmed pingable when not firewalled,
so the isolation assertions still mean something.
The suite red-ed with an empty final peer_count while every substantive
assertion passed: both denied peers showed inbound handshake attempts and
zero outbound responses, which is the gate this suite exists to verify.
Two pre-existing defects combined. The final count was sampled once, in
the same second the load driver issues its last round of peer restarts, so
the read can land on a daemon busy with those restarts and return nothing.
And the fallback meant to cover that could never fire, because the shell
applies it to the last stage of the pipeline, which succeeds on empty
input - so an unanswered query was indistinguishable from a genuine zero,
and reported as a cap violation.
Poll the final read for a short window instead, and give the two cases
distinct output. A real regression still fails, just after the retry
window, since the loop exits early only on the expected value. Extract the
read the two phases share rather than leaving them to drift.
Two runs on one host destroyed each other's containers, producing
mid-test "No such container" failures that look like real defects. The
automated builder runs a full local CI on the same box every few minutes,
so the machine is contended almost always and this has red-ed both a hand
run and an automated gate.
Two independent causes are fixed here. Container names were hardcoded, and
docker names are global rather than scoped by compose project, so two runs
collided on the same name; every name now takes an optional suffix that
the harness sets from the run id. And the cleanup sweep matched a label
shared by every run, so one run's teardown force-removed another's
containers; resources now also carry a per-run label and the sweep can be
narrowed to it.
A third hazard turned up that was not in the original report: the sidecar
suite passes explicit compose project names, which override the run-scoped
project and put it outside the shared prefix entirely. Its project names,
network, and derived container references are now scoped too.
Both are default-off. With the suffix unset, names render exactly as they
do today and a bare compose invocation is unchanged, which is what keeps
the hosted CI and the documentation correct. A cleanup run with no run id
still reaps everything, which is what a manual "clear the box" wants.
The literal-name sweep was not sufficient: six scripts build container
names dynamically from node labels, and two suites create their own
containers outside compose. Those are handled at their construction sites.
Verified: syntax check on all modified scripts; compose validation on
every modified file with the suffix both set and unset; and a synthetic
two-run reproduction that shows the old cleanup destroying a bystander run
and the new one leaving it alone.
Known gap: subnets are still hardcoded, so two concurrent full runs will
still collide on address-pool overlap. That fix reaches into topology
configs, chaos scenarios, diagrams and production source, so it is left
for its own change rather than half-done here.
The rekey suites' Phase 4 polled until the initiator-cutover count
reached its value at phase entry plus two, while Phase 6 asserts an
absolute minimum of four. On an unloaded host the ping phases can
outrun the jittered first rekey cycle, so Phase 4 can begin with a
single cutover on the books, release at three, and leave Phase 6 to
lose a sub-second race against the remaining first-cycle cutovers.
A rekey-outbound-only run failed exactly this way: five counted
cutover lines by t=32.6s of container life, with the Phase-6 count
snapshot at t~32.4s seeing three.
Floor the Phase-4 wait target at four so a released wait guarantees
the Phase-6 assertion — the docker-logs count is monotone over a
container's lifetime. A genuine rekey stall still times out the
bounded wait and fails Phase 6 with the real count.
Also drop the second-cycle framing from the comment, the phase
banner, and the assertion description: with rekey timers resetting at
each cutover a second cycle cannot begin inside this test's window,
so the counted events are first-cycle cutovers spread across the
topology's links, emitted once or twice per completed link rekey
depending on which side's promotion path logs them.
Phase 4 of the rekey suite waited for only one more initiator cutover
(guaranteeing three) while Phase 6 asserts at least four. The fourth
cutover then had to land in the brief window between the wait returning
and the Phase 6 log snapshot, so host load could push it past the window
and fail the run even though every cutover completed correctly. Wait for
two more cutovers, the full second rekey cycle, matching the Phase 6
threshold, so the asserted count is guaranteed before it is checked.
Every sign/verify/key-derive site built a fresh context via
Secp256k1::new(), which allocates a Secp256k1<All> and runs
randomization/blinding table setup on each call. Introduce one
crate-wide LazyLock<Secp256k1<All>> and reuse it across the local, peer,
and auth sites (and their tests). Behavior-neutral: identical secp256k1
API calls, only the context lifetime changes, and the shared All context
still performs the standard construction-time blinding.
An originator handling a LookupResponse unconditionally overwrote the
cached path_mtu_lookup entry, so a looser (larger) estimate in a later
response could clobber a tighter value already learned from a reactive
MtuExceeded or PathMtuNotification. Read-and-compare before writing and
keep the minimum, so a looser discovery estimate no longer loosens the
clamp. Add a regression test.
parent_switched was incremented on the line immediately before
parent_switches at every site and never independently, so the two
counters were always identical. Drop parent_switched from TreeMetrics,
its snapshot, TreeStatsSnapshot, the show_tree fixture, and the fipstop
render, keeping parent_switches as the sole counter.
BloomFilter derived all k hash functions from one SHA-256 digest but
recomputed that digest inside the per-function loop, so every insert and
contains ran SHA-256 hash_count times (5x at the default) over the same
bytes. Hoist the digest out of the loop: base_hashes() computes it once
and returns (h1, h2), and bit_index() derives each of the k indices with
the same (h1 + k*h2) mod m arithmetic. Bit-for-bit identical output; this
is the hottest path in packet forwarding and mesh-size estimation.
Adds a test pinning the bit indices against the double-hashing formula
recomputed independently, proving the refactor is behavior-neutral.
handle_peer_removal_tree_cleanup reparents or self-roots the node when
its parent link drops, but omitted the coordinate-cache invalidation that
every other position-change path performs. Cached entries for downstream
destinations kept the node's now-stale coordinate prefix, and because
find_next_hop refreshes the TTL on every routing access, an actively
routed stale entry never self-expired — corrected only by a fresh insert.
Mirror the loop-detection branch: inside the parent-loss changed block,
invalidate both classes — invalidate_via_node (reparent) and
invalidate_other_roots (self-root). Add regression tests for a parent-link
removal that reparents (via-node entry dropped, same-root sibling
preserved) and one that self-roots (both via-node and stale old-root
entries dropped).
Add a "Running CI locally" section to testing/README.md covering the
ci-local.sh orchestrator: the per-run FIPS_CI_RUN_ID override and how it
scopes compose projects, image tags, and per-chaos-child subnets so
simultaneous runs on one host never collide; the cancellation exit codes
(130/143 vs 0/1) and how a preempting worker interprets them; and the
label-based cleanup via --reap / ci-cleanup.sh.
A CI worker may preempt an in-flight ci-local.sh run (SIGTERM, then SIGKILL
after a grace period) to restart on a newer commit. For that kill to be safe,
the script must clean up after itself and never let a dying run collide with
its restart. It previously had no signal handling, shared the default compose
project name across runs, and tore down each suite only at the suite end.
- Derive a per-run id (honoring FIPS_CI_RUN_ID, else short-sha+random) and
namespace every docker resource to it: a fipsci_<run>_<suite> compose project
per suite and per parallel chaos child, and per-run image tags
(fips-test:<run>, fips-test-app:<run>) retagged to :latest only after both
builds succeed so :latest never points at a half-built image.
- Install a bounded, idempotent teardown trap on SIGTERM/SIGINT (+ EXIT): reap
parallel chaos children, then force-remove this run's docker resources via
the new ci-cleanup.sh, wrapped in timeout so a stuck down cannot wedge it.
- Exit 143 (SIGTERM) / 130 (SIGINT), distinct from 0 (pass) / 1 (failed), so a
preempting worker tells a cancelled run from a real failure.
- Add ci-cleanup.sh (also ci-local.sh --reap): force-removes leftover CI
resources by the com.corganlabs.fips-ci=1 label and the fipsci_ project
prefix, robust to however a prior run died.
- Label every per-suite docker resource so the label sweep reaps it after a
SIGKILL regardless of network name: direct docker run/network resources, the
sidecar compose services, and every per-suite compose network (acl-allowlist,
boringtun, firewall, nat, static, both tor suites, and the chaos generator
template). Parametrize the static/sidecar compose image refs so the per-run
tags are honored.
- Give each parallel chaos child a unique /24 from 10.30.x (a new --subnet
override on the sim CLI, assigned per-child in ci-local.sh) so parallel
children never collide on a shared docker subnet, and a chaos net can never
span a fixed-subnet suite (sidecar/static in 172.20.x). 10.30.x sits outside
docker's default-address-pool range, so an auto-assigned net cannot land on
it either; node IPs derive from the subnet, so no scenario config changes.
Replace the fixed post-first-rekey and post-second-rekey settle sleeps in
the rekey integration test, and the fixed-deadline baseline wait in the
interop test, with the deterministic wait_until_connected progress-aware
polling helper already used for the initial baseline convergence.
Each converted site fails fast with structured diagnostics when the mesh
is stuck and extends its deadline only while pairwise reachability is
still climbing, removing the wall-clock settle windows that produced
intermittent connectivity failures after a rekey.
A transient timeout uploading the built .ipk to the Blossom CDN
(blossom.primal.net) failed the entire OpenWrt Package run, and because
the GitHub release job depends on the build jobs, a CDN blip would block
every OpenWrt artifact from the release. Blossom/nostr distribution is
supplementary; the package already ships as a GitHub release artifact.
Wrap the upload in a 3-attempt retry with backoff to absorb transient
timeouts, mark the step continue-on-error so a persistent outage no
longer fails the build, and guard the NIP-94 publish on the upload
succeeding so no event is created with an empty URL. Applied to both the
.ipk and .apk build jobs.
Claude-Session: https://claude.ai/code/session_01A2pYfSypNmmG4HyHwZuLex
Prepare the source tree for the v0.4.0 release cut, leaving the version
at 0.4.0-dev (the version bump rides a separate release-candidate
commit).
- CHANGELOG: backfill the missing entry for the route-class transit
counters and fipstop routing-tab reorg, then reorganize the Unreleased
block from a flat per-commit list into topic-grouped subsections
mirroring the 0.3.0 entry (coalescing interim fixes into net-effect
descriptions without dropping technical detail), and stamp it as
[0.4.0] - 2026-06-21 with a fresh empty Unreleased block. The date is
provisional and reconfirmed at the final tag.
- Release notes: refresh both RELEASE-NOTES.md and the versioned archive
(kept byte-identical) to cover the OpenWrt .apk packaging, the Nix
flake, the macOS self-traffic checksum fix (#117), and the route-class
transit counters.
- README: bump the status badge to v0.4.0 so it matches the prose.
Claude-Session: https://claude.ai/code/session_01A2pYfSypNmmG4HyHwZuLex
Add six forwarding counters that partition transit-forwarded packets by their
tree relationship to the chosen next hop: tree-up (peer is our ancestor),
tree-down (peer is our descendant and the destination is within its subtree),
tree-down-cross (peer is our descendant but the destination is outside its
subtree), cross-link descend (lateral peer, destination within its subtree),
cross-link ascend (lateral peer, destination outside its subtree), and
direct-peer. The six classes sum to forwarded_packets, asserted by a unit
test. Classification is computed from tree coordinates at the transit
chokepoint, so the error-signal routing callers are excluded.
The two "outside the chosen peer's subtree" classes are both up-and-over
forwards but differ in what they depend on. Tree-down-cross is the
dive-to-tree-child cut-through: we forward down to our own child for a
destination not beneath it, which is only possible because the child
advertised cross-link reach upward to us, beyond its own subtree. Its count
measures how much forwarding depends on that upward advertisement, i.e. what
would change if cross-link advertisements were narrowed to subtree-entry only.
Cross-link ascend, by contrast, uses the node's own lateral cross-link learned
from a peer's split-horizon advertisement, so it does not depend on any upward
advertisement.
Surface the counters through the forwarding stats snapshot (control socket,
show_routing and show_status) and reorganize the fipstop routing tab so its
two columns separate own/endpoint traffic (received, delivered, originated)
from forwarded/transit traffic (the route-class breakdown and drop reasons),
with the tree-down-cross line visually flagged.
The linux/macos/windows package workflows each publish a
checksums-<platform>.txt sha256 file with their release artifacts, but
the OpenWrt workflow attached only the .ipk files with no checksum
coverage. Generate checksums-openwrt.txt over the .ipk outputs using the
same sha256sum idiom as the other platforms and add it to the release
file set.
The fips build runs bindgen (via the rustables crate that powers the LAN
gateway's nftables bindings), which needs libclang.so at build time.
makepkg -s installs only declared makedepends, so without clang the AUR
build panics with "Unable to find libclang". Add clang to the makedepends
of both PKGBUILD and PKGBUILD-git so makepkg installs it and AUR users
get the dependency. Surfaced by the always-on aur-build CI job.
Bring the source tree to its finished v0.4.0 content state ahead of the
release candidate. Documentation, changelog, release notes, and
packaging metadata only; no code or version-string changes.
CHANGELOG.md: backfill the operator-visible changes that landed since
v0.3.0 (the show_metrics scraper query and fipsctl stats metrics, the
discovery dedup-cache-full counter, the off-rx_loop control read
surface and its new daemon-resolved fields, the fipstop TUI overhaul,
the TCP inbound cap now honoring max_connections, host-map hot-reload,
log-noise demotions, and the net bug fixes), topic-grouped rather than
replayed per commit; intra-cycle fixes folded into their feature
entries; the duplicate Unreleased Fixed section merged into one.
Release notes: author docs/releases/release-notes-v0.4.0.md to the
operator-upgrade bar and mirror it byte-for-byte into the root
RELEASE-NOTES.md. Attribute each feature to its author in Contributors
(Nym transport and the mixnet demo to @oleksky, opt-in mDNS LAN
discovery to @mmalmi).
README.md: add the Nym transport to the support matrix (Linux, macOS,
Windows; OpenWrt pending verification), the multi-transport bullet, and
the "What works today" list; fold mDNS LAN discovery into the existing
Nostr-discovery bullets; refresh the status narrative to v0.4.0 over a
global, public test mesh.
packaging/common/fips.yaml: add commented Nym transport and mDNS LAN
discovery example stanzas with verified field names and defaults.
Cargo.toml: add homepage, keywords, and categories crate metadata.
docs/reference: update the cli-fips version example to the released
form; document the new control-socket output fields and the typed
RejectReason families in control-socket.md; sync cli-fipstop.md with the
overhauled TUI keybindings.
poll_lan_discovery's comment said Noise XX, but LAN-discovered peers dial over
UDP through initiate_connection, which uses Noise IK (IK at FMP). compute_mesh_size's
header comment still described the obsolete sum-of-disjoint-subtrees estimate;
the function OR-unions every connected peer's inbound filter plus self and
estimates cardinality once (matching the body comment). Comment-only, no
behavior change.
Pre-cut documentation pass for the 0.4.0 release, verified against current source.
Corrections:
- fipsctl: stale 'show identities'/'show node' -> 'show status'
(host-a-service, run-as-unprivileged-user)
- mesh address derivation: first 16 bytes of SHA-256(pubkey) with the leading
byte set to 0xfd, not a fixed fd97: prefix (reach-mesh-services,
ipv6-adapter-walkthrough)
- gateway control socket mode 0660 -> 0770 (troubleshoot-gateway)
- Tor example: add advertised_port: 8443 so the published port matches the
prose (enable-nostr-discovery)
- bloom mesh-size estimate rewritten to the OR-union-of-peer-filters algorithm;
plus mtu deep-link, gateway pool wording, and a NAT failure-mode line
- examples: delete orphaned nostr-rs-relay config, accept inbound to the local
8443 TCP listener, fix fd::/8 -> fd00::/8 typos, dotless wireguard alias
Additions:
- new Nym mixnet transport section (fips-transport-layer) and the architecture
transport list
- new LAN/mDNS discovery section (fips-nostr-discovery)
- reference docs: Nym transport, LAN discovery, and new control/stats surfaces;
drop ble from the connect transport list
Run the AUR package through makepkg + namcap on the same triggers as the
other package workflows (pushes to master/maint/next, pull requests,
tags, and manual dispatch), so a broken PKGBUILD is caught continuously
rather than only at release time. The build runs in an Arch container
(makepkg/namcap are not on ubuntu-latest) and packages the checked-out
tree from a local git-archive tarball, so it works for branch/PR builds
and unreleased rc tags that have no published GitHub source archive yet.
makepkg runs with --nocheck since the test suite is already covered by
ci.yml; this job validates packaging.
Publishing to the AUR is unchanged in intent but now gated to a real
(non-prerelease) release tag push, plus the existing manual-dispatch
republish path for packaging-only pkgrel bumps; it depends on the build
job so a package that fails to build or lint is never published. Branch
pushes and pull requests build and lint but never publish.
The PKGBUILD-patching logic is factored into a shared
packaging/aur/patch-pkgbuild.sh used by both jobs; the build path
sanitizes the version for makepkg (which forbids '-' in pkgver).
The GitHub deb-install matrix ran debian12/ubuntu24/ubuntu26, but the
local harness (testing/deb-install/test.sh) runs five distros. Add the
missing debian13 (trixie) and ubuntu22 legs so the cloud gate covers the
same distro set as local CI. Each new leg invokes the existing test.sh
scenario, so per-distro behavior is identical to the local run.
Update the granularity-only parity notes in ci.yml, ci-local.sh, and
check-ci-parity.sh to list the full distro set.
Pull the available point releases that sit within the current version
constraints: tun 0.8.11, tokio-socks 0.5.3, socket2 0.6.4, nostr 0.44.3,
nostr-relay-pool 0.44.1, ratatui 0.30.1, serde_json 1.0.150, simple-dns
0.11.3, mdns-sd 0.19.2, plus the transitive maintenance crowd. Lockfile
only; no Cargo.toml constraint changes.
Defer the out-of-semver crypto and identity majors (secp256k1 0.31, sha2
0.11 with hkdf 0.13, bech32 0.12) and the mdns-sd 0.20 bump to a separate
coordinated refresh with its own handshake, identity, and LAN validation.
The connect_refused stat counter (the Refused line in fipstop) was
defined but never incremented: every SOCKS5 connect failure recorded
socks5_errors instead, so the counter sat at zero and the operator-facing
gauge was permanently misleading. Both the synchronous connect path and
the background connect_async task now count a genuine SOCKS5 REP=0x05
refusal as connect_refused and every other failure as a socks5_error,
distinguishing them precisely via tokio_socks::Error::ConnectionRefused.
Extends the mock SOCKS5 server with a configurable reply code and adds
two tests covering the refused and general-failure paths.
Two independent rendering glitches, both most visible over SSH and inside
tmux:
Startup: ratatui::try_init() enters the alternate screen but never clears
it, and the first terminal.draw() only emits cells that differ from an
assumed-blank internal buffer. On terminals that don't hand back a cleared
alternate buffer (notably tmux, and amplified by SSH latency) the prior
contents show through. Force a full repaint with terminal.clear() before
the first draw.
Quit: the input EventHandler spawned a detached thread that polled stdin in
a loop outliving the main loop, so at quit it kept reading after raw mode
was disabled and stray bytes (a keystroke or a terminal query response)
echoed onto the restored screen. Give the thread a stop flag and join it
before restoring the terminal; poll on a short fixed interval (decoupled
from the refresh tick) so quit stays responsive.
git log --oneline -1
When an FMP msg1 or FSP msg3 rekey retransmission budget is exhausted, the
cycle is abandoned and retried on the next timer. On lossy or high-latency
links this is an expected, self-limiting outcome: the existing session stays
valid and keeps carrying traffic, so the abort is not a failure that warrants
operator-level visibility. Demote both abandon-cycle messages from warn to
debug to cut steady-state log noise on nodes with many flapping peers.
A manual disconnect tore down only the local side and sent the peer nothing, so
the peer kept its session and never re-emitted its tree and filter
announcements; on reconnect it was never re-adopted as a child and its bloom
filter was never recorded. Send the disconnected peer a scoped Disconnect, the
same message graceful shutdown sends to all peers, so both sides tear down and
re-handshake cleanly on the next connection.
Reworks the fipstop TUI across its rendering, the control read surface it
draws from, and its interaction model, on a machine-verified base.
Test infrastructure:
- Add a ratatui TestBackend snapshot harness (testkit + snapshots
modules) that renders any ui::draw_* into an in-memory Buffer from
canned show_* JSON and asserts the text grid plus per-cell style.
Layout, columns, alignment, labels, grouping, and colour are now
checkable under cargo test; every render below ships a snapshot.
Control read surface (each new field emitted byte-identically on the
live and off-loop builders, published once from the tick, with schema
fixtures regenerated and the parity asserts holding):
- show_status: effective persistence (persistent || nsec.is_some());
root and is_root; and a per-configured-transport-type peer-count map
in which idle-but-configured types stay visible at zero.
- show_peers: per-peer effective_depth (depth + link_cost, the value
evaluate_parent ranks on), null when unmeasured or coordless so
fipstop never recomputes it.
- show_tree: root_npub, resolved once daemon-side (self when root, an
attested peer npub, or an identity-cache hit).
- show_bloom: the last-actually-sent uptree filter fill ratio and
subtree estimate, null for a root or before the first announce.
- show_mmp: session-layer srtt, loss, and etx trend labels.
Rendering:
- Display a 6-byte non-UTF-8 TransportAddr as a colon-separated MAC at
the type layer, so daemon logs, fipsctl, and JSON consumers all
benefit; non-6-byte payloads stay bare hex.
- Right-justify the Bloom Peer Filters numerics into aligned fixed-width
columns, render the Routing panes through a kv_lines helper that shares
one value column across a key-value group, and right-justify the Graphs
by-peer summary columns.
- Truncate an over-long peer name (the npub shown when no friendly name
exists) in the Tree, Bloom, and MMP peer lists so it no longer runs
into the next column.
- Group the Peers table by role (parent, then STP children, then other)
and render it as a full grouped view with styled group labels and
blank separators; the selection stays a peer index and the cursor only
ever lands on a peer row. Apply the same role grouping to the Tree and
Bloom peer lists, joining each peer's role from the peers view by node
address.
- Show min in the Graphs plot titles, rest a steady non-zero metric on
the baseline as a row of dots, render a genuine zero as an empty plot,
and keep a distinct no-data placeholder.
- Replace the metric-by-peer grid, which squeezed plots to nothing once
peers overflowed, with a master/detail Graphs view: a scrollable
per-peer summary list that expands (Enter) to a full-pane btop plot,
with up/down to flip peer, n/N to switch statistic, m to cycle mode,
and Esc to return.
- Put inline colored trend arrows on the Link and Session MMP values
(drawn only on a rising or falling trend, with a fixed blank slot when
stable so the value columns stay aligned), via a shared helper.
- Cycle column sorting on the Link MMP, Session MMP, and Graphs by-peer
tables (one key cycles the active column, another toggles direction),
with the active column marked in each table's header.
- Render the new daemon-surfaced fields: the dashboard root line (a
self-is-root marker, otherwise a truncated root hex), a
transports-by-type line, and an "approx. mesh estimate" line; an
effective_depth column and lines on the Peers, peer-detail, and Tree
sites from the single daemon derivation, showing a dash placeholder
when unmeasured rather than a misleading zero; the full Tree root hex
plus an Npub line; and the Bloom uptree fill and subtree-estimate lines.
Interaction model:
- Add a declarative keybinding registry keyed by (Tab, UiMode) that both
the context footer and the ? help overlay render from, so the two
cannot drift; a test asserts every registry key has a dispatch handler.
- Add a modal ? help overlay, and a context-aware footer that shows the
current state's actions first, drops global hints when the terminal is
narrow, and always keeps a Help affordance as the overflow path.
- Generalize per-pane focus and scroll state on App, wired across the
Tree, Filters, Routing, and MMP tabs (f cycles pane focus and the
focused pane scrolls instead of clipping its overflow); on the MMP tab
the column sort acts on the focused pane. Esc deselects the active row
when no detail is open (detail-close still takes priority).
- Add a Del-disconnect confirmation modal naming the peer, the only
state-mutating action, issuing the control-socket disconnect on confirm
and noting that the peer stays disconnected until manually reconnected.
Complete the control-plane read-isolation work: every pure-read show_*
query now renders in the control accept task from published read
snapshots, so none round-trips the data-plane receive loop. Only the
mutating connect/disconnect commands still reach that loop.
Three subsystem snapshots are published via ArcSwap and served through the
read handle's snapshot_dispatch:
- A routing read view (spanning tree, bloom filters, coordinate cache,
identity cache, and the discovery F-queue summary scalars), published
from the tick, serving show_tree/show_bloom/show_cache/show_routing/
show_identity_cache.
- A per-entity read view (peers, sessions, links, connections, transports,
and the MMP link/session views) as Vec<Arc<Row>> tables reconciled
against the prior snapshot so a republish reuses unchanged rows by
pointer and re-allocates only changed or new rows, keeping the per-tick
publish cost bounded as the peer/session count grows. Serves
show_peers/show_sessions/show_links/show_connections/show_transports/
show_mmp.
- The stats snapshot is extended with the peer-ACL status and a per-peer
metadata map (is_active, npub, display name), resolved at publish time,
serving show_acl and the two per-peer stats queries.
Display names and other cross-subsystem fields are resolved at publish
time; time-relative fields are derived at render time from captured
absolute timestamps, so rendered output is byte-identical to the prior
on-loop handlers, which are retained as the equality oracle.
With every read query served off-loop, the show_* branch is removed from
the rx_loop control handler and the now-dead on-loop dispatcher deleted.
The snapshot projections are forward-compatible with the later structural
extraction of the derived-state and session tables: they become thin
views over the extracted types without changing the read-handle interface.
Introduce a read-snapshot plane so pure-snapshot control queries render in
the control-socket task instead of round-tripping the rx_loop, removing the
head-of-line coupling that let a busy or slow rx_loop time out fipsctl and
fipstop observability.
- ControlReadHandle: a cloneable bundle the control accept loop holds, over
the node's already-shared NodeContext and MetricsRegistry plus an
ArcSwap-published StatsSnapshot. A snapshot_dispatch seam serves cut-over
commands off-loop and falls through to the rx_loop for the rest, keeping
the rx_loop's ownership of Node intact.
- StatsSnapshot is published from the tick (the natural and sole mutator of
stats_history), carrying the history rings plus the scalar gauges and
counts show_status reports. Readers serve the latest snapshot
unconditionally, with staleness bounded by the tick interval and no
IO_TIMEOUT-coupled fallback.
- Off-loop now: show_status, show_stats_history, show_stats_all_history,
show_listening_sockets, show_stats_list, and a new counter-only
show_metrics (exposed as fipsctl "stats metrics", the enabler for a
Prometheus scraper at no hot-path cost). Queries that need live per-entity
state (peers, links, sessions, routing, and the per-peer stats variants)
stay on the rx_loop path pending later phases.
Quartet green; forward-merge to next verified clean.
The pool's TTL clock (VirtualIpMapping.last_referenced) advanced only on
DNS re-query, never on traffic, and the mapping-TTL is wired equal to the
DNS TTL, so an in-use mapping was forced to drain at TTL and reclaimed at
the first zero-conntrack tick (a stale drain_start gave no grace effective
protection), breaking long-lived, bursty, or DNS-cached clients.
In tick(), refresh last_referenced whenever conntrack reports sessions > 0
so an actively used mapping never ages out, and recover a Draining mapping
to Active (clearing drain_start) when traffic resumes, so a later drain
gets a fresh grace window instead of a stale one. The Active arm now only
drains an idle mapping. The DNS-TTL / idle-reclaim-TTL wiring is unchanged.
Adds regression tests for continuous-traffic-survives-past-TTL, bursty
drain-then-recover, and fresh-grace-on-redrain.
The per-transport TCP inbound cap was hardwired to 256 and never read
node.limits.max_connections, so raising max_connections was a silent
no-op for inbound TCP. Resolve the effective cap with precedence:
explicit per-transport max_inbound_connections, then node-wide
max_connections, then the built-in default of 256. Established peers
remain bounded node-wide by add_connection, so deriving the per-transport
raw-accept ceiling from max_connections does not admit more real peers
across multiple transports.
Add effective_max_inbound on the TCP transport with a node_max_connections
setter wired from create_transports, plus a precedence unit test.
Estimate the OR-union cardinality over self plus every connected peer's
inbound filter, dropping the parent/child tree gating in
compute_mesh_size. Filter propagation is split-horizon, so cross-links
advertise near-complete mesh views; unioning all peers yields the same
set as the tree-only union in steady state (OR dedups overlap and no
filter can over-count) while damping the node-count flap on parent
switches, since dropping the parent no longer collapses the upward leg.
This also removes the estimate's dependence on tree-declaration cache
freshness.
Rename the debug-log child_count to contributor_count, adapt the two
membership-invariant tests to all-peers semantics, and add a test that
the estimate stays stable across a parent drop when a healthy cross-link
is present.
The inbound FilterAnnounce FPR cap rejects filters whose false-positive
rate (fill^k) exceeds the configured maximum. On the fixed 1 KB / k=5
filter, 0.05 corresponds to fill 0.549 (~1,300 reachable entries), and
the busiest nodes' aggregates were beginning to hit that ceiling as the
mesh grew. Raise the default to 0.10 (fill 0.631, ~1,630 entries) to
restore headroom toward the fixed-filter capacity limit. A saturated or
poisoned filter is ~100% FPR and remains rejected, so the antipoison
gate is not materially weakened.
Updates the config default, the config-reference and bloom-filter
design docs, and the changelog.
The sidecar chain test intends node-a to be a standalone root ("node-a: no
outbound peers"), but started node-a without clearing FIPS_PEER_*, so it
inherited the external peer default from testing/sidecar/.env (a real public
mesh node, test-us01.fips.network) and auto-connected to the live mesh. The
whole a-b-c chain then attached under an external root, inflating tree depth and
breaking test isolation; this also produced spurious multi-hop failures.
Set FIPS_PEER_NPUB/FIPS_PEER_ADDR empty for node-a, matching how node-b and
node-c already get explicit inline peers, so the suite is hermetic regardless of
the .env default. Validated against the unmodified .env: node-a comes up with a
single link to node-b, no external attachment, multi-hop passes 18/18.
A node with a single tree peer has its periodic parent re-evaluation
disabled (it needs at least two peers for a meaningful comparison), so
it depends entirely on its peer pushing a TreeAnnounce for it to attach.
That push happens once at promotion time plus on the parent's slow
periodic no-change re-broadcast. If the one-shot attaching announce is
lost, the single-uplink node falls back to self-root and cannot recover
until the next periodic re-broadcast (reeval_interval_secs later),
stranding it out of the tree and unreachable end-to-end in the interim.
Make tree-position exchange self-healing on the receive path: when an
accepted TreeAnnounce advertises a root strictly worse (higher NodeAddr,
since election is smallest-wins) than our own, echo our current
declaration back to that peer. A stranded self-root node's announce now
provokes its better-rooted peer to re-push its real position immediately,
so the node re-attaches within a round-trip instead of waiting for the
periodic cadence.
Echo only in that one direction. If the peer's root is lower (better)
than ours, we are the stale side: the peer would ignore our worse root
anyway and we converge via the parent re-evaluation that follows, so
echoing back is pure waste and would double announce traffic in the
learning direction during a root change or partition merge. Equal roots
are already converged. The echo is bounded by the existing per-peer
500 ms tree-announce rate limiter and is a no-op once the peer adopts our
root, so it adds no traffic in a converged mesh.
Add a spanning-tree unit test that drives a converged child back to
self-root with its peer-ancestry view of the root cleared (modelling the
lost attaching announce), and asserts the root re-pushes on the
resulting root disagreement and the child re-attaches.
wait_until_connected fail-fasted on stall even when the mesh was all but
converged, abandoning the run with budget still unspent because one hard
pair straggled to come up. On the rekey-outbound-only topology this turned a
rare deep-node timing straggle (stacked discovery backoff + late bloom
propagation, which clears well inside the budget) into a false baseline RED.
Add a near_converged_slack threshold (default 2): when the number of
still-failing pairs is at or below the slack and the stall window elapses,
keep polling toward max_secs instead of bailing. A mesh genuinely far from
convergence still fast-bails on stall, and a never-converging pair still
hits the hard cap, so a real regression is never masked.
Add a self-contained behavioral test (wait-converge-test.sh) covering the
four contract cases: near-converged hold (with a slack=0 contrast that
proves the slack is what saves the run), far-from-converged fast-bail,
never-converging hard cap, and four-argument backward compatibility.
The interop harness only tested a full mesh, where every pair is a direct
one-hop FMP link. That left multi-hop forwarding, routing, coordinate
handling, and mesh-size estimation untested across versions.
Add an opt-in multi-hop topology and three new checks:
- generate-configs.sh: FIPS_INTEROP_EDGES selects an explicit undirected
edge list instead of the full mesh, with symmetric peering, connectivity
validation (no isolated node, graph connected), and per-node degree plus
edge metadata in nodes.env. Unset means full mesh, unchanged.
- interop-test.sh: a --topology flag with a built-in multihop-3v-cycle
(six nodes, two of each version, one cycle, two leaves). The per-node
peer check now expects each node's adjacency degree rather than N-1, and
the all-pairs ping now also exercises cross-version forwarding over
non-adjacent pairs. Adds a control-differential data-plane continuity
stream across the rekey window (Phase 5b) and a strict plus/minus 25
percent mesh-size convergence check (Phase 7).
- interop-stress.sh: forward --topology through to the per-rep driver
(mutually exclusive with a positional node-spec).
Phase 1 makes the strict retrying ping authoritative. The convergence
detector (one fully-clean, no-retry sweep of every directed pair) is now an
advisory settle-wait whose timeout is non-fatal; the strict retrying ping
decides pass/fail, matching how the post-rekey phases already work. Under
packet loss a clean no-retry sweep of a large mesh is statistically unlikely
even when the mesh is healthy (30 pairs at 2 percent loss leaves only about
55 percent of sweeps clean), so the detector otherwise falsely failed runs
whose strict ping reported full reachability. CONVERGENCE_TIMEOUT is now
env-overridable and defaults to 90s for larger meshes under loss.
Full-mesh runs are unaffected. Validated end-to-end with the
multihop-3v-cycle topology across three versions: all 30 directed pairs
reachable including multi-hop routed pairs, zero data-plane loss through
both rekey cutovers, and mesh-size converging to the true node count on
every node.
Brings the maint-line OR-union mesh-size estimator fix and the
per-peer / capacity-cap log-level demotions forward to master.
Conflict resolution: compute_mesh_size resolved to master's config()
accessor form carrying maint's OR-union rewrite (drop the stale summing
`total`); the log-level demotions auto-merged into master's handler
versions. The equivalent master-only log "connected UDP socket installed"
(connected_udp.rs, a file that does not exist on maint) was demoted
info -> debug as part of this merge so the connected-UDP path matches the
rest of the per-peer lifecycle logging.
On a saturated public-mesh node the connection-lifecycle and capacity-cap
events fire continuously and drown out the genuinely notable INFO/WARN
lines. Demote them to debug and drop a redundant duplicate:
- FMP K-bit cutover promotion (encrypted): info -> debug
- "Connection promoted to active peer" (handshake): info -> debug, and
remove the duplicate "Inbound peer promoted to active" line that
shadowed it on the inbound path
- "Peer restart detected" (handshake): info -> debug
- "Peer removed and state cleaned up" (dispatch): info -> debug
- "Rejecting inbound TCP connection (max_inbound_connections reached)"
(tcp): warn -> debug
- "Congestion detected, CE flag set on forwarded packet" (forwarding):
warn -> debug
- "Removing peer: link dead timeout" (mmp): warn -> debug
These are expected, high-frequency conditions on a busy public node (new
and reconnecting peers, ECN CE marking, the inbound connection cap, and
link-dead churn), not operator-actionable signals.
The mesh-size estimator summed the per-filter cardinality of the parent
filter and each child filter, which assumes those filters are perfectly
disjoint. When they overlap -- a stale or oversized parent filter, or a
routing loop -- the sum over-counts and inflates the reported mesh size
to as much as several times the true size.
Estimate the cardinality of the OR-union of the contributing filters
(self + parent + children) once instead. OR is idempotent, so any
overlap is deduplicated: the result equals the old sum in the disjoint
case and stays correct under overlap. The union is seeded from a clone
of a contributing filter so it keeps that filter's size class, and a
filter whose size class does not match is skipped rather than panicking.
The refuse-to-estimate behavior on a saturated or above-cap filter is
preserved.
Add a regression test with overlapping parent and child filters where
the naive sum over-counts and the union estimate tracks the distinct
member count.
The transport layer used Mutex::lock().unwrap() at ten sites across the
UDP, BLE, and Ethernet code. A std mutex poisons if a thread panics
while holding it, after which every lock().unwrap() on that same mutex
also panics, turning one fault into a cascade. These critical sections
only perform short HashMap/Vec operations on locally constructed values
and are not reachable from peer input, but the idiom is fragile against
any future in-section panic. Replace each with
lock().unwrap_or_else(|e| e.into_inner()), which recovers the guarded
data and removes the cascade with no new dependency and no call-graph
change.
Also replace four self.local_addr.unwrap() calls in the UDP start and
adopt paths with a sentinel fallback. The value is provably set just
above each log line today, but the unwrap is brittle against a future
reordering; logging an unbound sentinel is harmless and cannot panic.
Linux source builds pull in rustables, whose build script runs bindgen
to generate nftables bindings for the LAN gateway. bindgen needs
libclang.so on the build host, so a clean source build fails with
'Unable to find libclang' unless libclang-dev (or llvm) is installed.
The prerequisite text in README.md and CONTRIBUTING.md previously
listed only the optional BLE dependencies, and packaging/README.md had
no source-build prerequisite list at all. Document libclang-dev as a
mandatory Linux build dependency, distinct from the optional BLE deps,
and note that it is build-time only so pre-built .deb installs are
unaffected.
Record the changes that landed since the last changelog update: the
dual-auto_connect traversal-session election, the FMP link-layer rekey
reliability fixes (bounded msg1 retransmission with rekey-aware
heartbeat, and authenticate-before-cutover), the in-process loopback
test transport and progress-aware convergence wait that remove CI flake
classes, the local/GitHub CI suite-parity and single-source toolchain
selection, and the packaging change shipping the config as an example
that postinst seeds when absent.
Installing /etc/fips/fips.yaml as a live dpkg conf-file collides with a
configuration-management-rendered or operator-edited config on upgrade:
dpkg either prompts interactively (keep/replace), stalling unattended
upgrades, or clobbers the local file. Ship the default config as
/usr/share/doc/fips/fips.yaml.example (mode 644) and drop it from
conf-files. postinst now seeds /etc/fips/fips.yaml from the example only
when it does not already exist (mode 600), yielding to any existing
config without a prompt or clobber. Add ConditionPathExists for the
config to the service unit so a missing config skips the unit cleanly
rather than crash-looping.
A busy node opens roughly three file descriptors per established UDP peer
(a connect()-ed socket plus a 2-FD drain self-pipe), so the default 1024
soft RLIMIT_NOFILE is exhausted near 320 peers and further peer admission,
handshakes, and discovery fail with EMFILE. Document the FD budget, the
symptom, and the systemd (LimitNOFILE drop-in) and OpenWrt (procd nofile)
procedures to raise it, plus how to verify the per-peer ratio is bounded.
Link the new guide from the how-to index.
Bring the local runner (testing/ci-local.sh) and GitHub CI into agreement
on two axes.
Suite coverage: the admission-cap integration suite ran only locally, so a
regression in it could never turn the GitHub gate red. Add an admission-cap
leg to the integration matrix, add testing/check-ci-parity.sh (wired as
'ci-local.sh --check-parity') to diff the two suite sets and fail on
unexpected drift, and document the deliberate local-only (live-Tor) and
granularity-only differences in a comment block atop both runners.
Toolchain selection: every CI and packaging job installed its toolchain
with dtolnay/rust-toolchain@stable, but the rust-toolchain.toml channel pin
overrode it for all compilation, wasting an install and printing a
misleading rustc version. Switch the stable call sites to
actions-rust-lang/setup-rust-toolchain, which reads rust-toolchain.toml as
the single source of truth. Keep the explicit cache steps (cache: false on
the new action), set rustflags empty so the action does not impose a global
-D warnings the previous setup never applied, fold the macOS cross-compile
target into the action input, and leave the OpenWrt nightly Tier-3 leg on
dtolnay/rust-toolchain@nightly.
When two peers each auto_connect to the other, each runs both an
initiator and a responder NAT-traversal session and binds a separate
UDP socket per session. Each side adopts only the first Established
event and drops the loser session's socket; when the two sides adopt
mismatched sessions, each sends its Noise msg1 to a peer port the peer
has already stopped draining, and both handshakes stall.
Deterministically keep the session initiated by the smaller NodeAddr,
decided on the responder path: decline an incoming offer only when we
also have an in-flight outbound initiator for the same peer and our
NodeAddr is smaller. The peer's redundant initiator then times out,
leaving a single matching socket pair on both ends. Asymmetric
(one-sided) auto_connect has no co-active initiator and is never
suppressed, so connectivity is preserved; an undecidable NodeAddr falls
through to answering.
Reuses the NodeAddr tie-breaker convention already used by the
cross-connection and rekey dual-init paths. Adds a unit test for the
election helper.
Add wait_until_connected to the shared convergence helpers: it polls a
suite's own pairwise pings (the signal it actually asserts on), returns
as soon as every pair is reachable, extends its deadline while the
reachable-pair count is still climbing, and gives up only when progress
stalls.
Use it in the rekey, static-mesh, and sidecar suites in place of the
fixed wall-clock baseline timeout and the blind sleep, which timed out
under concurrent CI load while the mesh was still converging.
Add a Loopback variant to TransportHandle backed by an unbounded
in-process channel and a shared address-to-receiver registry, so
node-level multi-node tests deliver packets directly between nodes
instead of over real localhost UDP sockets. This removes the kernel
UDP receive-buffer overflow that dropped handshake packets when many
tests ran in parallel under CPU contention, and lets the large-network
convergence tests run reliably in the default suite again (their
parallel-load ignore markers are removed).
The new transport and its enum variant are cfg(test)-gated, so the
daemon build is unaffected.
The FMP rekey msg1 resend driver retransmitted indefinitely with no cap
and no abandon, so a rekey that never completed kept resending msg1
forever. Give it a retransmission budget: cap resends at
handshake_max_resends with exponential backoff and abandon the rekey
cycle cleanly once the budget is exhausted, mirroring the FSP session
rekey msg3 driver.
With the cap in place the link-dead heartbeat can safely become
rekey-aware: check_link_heartbeats now suppresses teardown while a rekey
is in progress with msg1 budget remaining, instead of reaping a link
that is still actively carrying rekey-handshake traffic. The suppression
terminates deterministically (the budget abandons on exhaustion, cutover
clears the in-progress flag), so a genuinely dead link is still reaped on
the next cycle.
Adds a rekey_msg1_resend_count counter on ActivePeer reset at every
rekey-clear and cutover site, msg1 resend-budget unit tests, and two-node
heartbeat suppression/resume/regression integration tests.
Remove the duplicated immutable fields (config, identity, startup_epoch,
started_at, is_leaf_only, max_connections/peers/links) from the Node
struct so the Arc<NodeContext> bundle is the single source of truth.
Previously Node owned these fields and a parallel context copy, kept in
lockstep by rebuild_context() at every mutation site — pure overhead that
existed only because of the duplication.
- Replace rebuild_context() with replace_context(): a clone-edit-swap of
the whole Arc. The per-instance context stays immutable; mutation swaps
the Arc. This is the sole runtime mutation path (constructors, leaf_only,
update_peers).
- Add Copy-returning accessors startup_epoch() and max_connections()/
max_peers()/max_links(); migrate the remaining direct field readers onto
the accessors. node_addr()/npub()/Debug now read identity/is_leaf_only
from the context.
- update_peers reads the pre-update peer set from the live context Arc
before building a fresh Config + context and swapping — preserving the
read-before-write ordering its mutation-window test depends on.
- Remove the test-only set_max_* setters; tests set the limits on Config at
construction instead (new make_node_with_max_peers/links helpers).
- Add a ci-local guard that fails if the Node struct re-declares a bundled
field, so the single-store invariant can't silently regress.
cargo test --lib 1291/0; clippy -D warnings and release build clean.
Store node counters in an atomic metric registry read through &self, and
introduce a shared NodeContext bundle holding the effectively-immutable
fields (config, identity, startup epoch, capability limits). Source the
immutable config and identity reads across the receive hot path, the
handshake/session/mmp/encrypted state machines, and the discovery, tree,
bloom, retry, and lifecycle modules through the context accessors rather
than direct field reads. The Node fields and the context are rebuilt in
lockstep at every mutation site.
PeerRecvDrain::drop previously called std::thread::join on the worker
thread synchronously. The worker uses packet_tx.blocking_send on a
tokio mpsc Sender, which internally parks the worker via
tokio::block_on on the same current_thread runtime that drives
rx_loop. Calling join from inside remove_active_peer (which runs on
the runtime thread, the runtime's sole driver) created a circular
wait:
- rx_loop blocks in libc futex via Thread::join
- the worker being joined cannot observe the stop flag because the
runtime that polls it is the very thread now blocked joining it
- all other PeerRecvDrain workers park on the same runtime via
block_on, so a single peer's removal wedges every worker on the
daemon
The /proc snapshot from a production wedge showed exactly this
shape: 107 of 108 threads in futex_do_wait, 101 of them named
fips-peer-drain. fipsctl became unresponsive (EAGAIN on control
socket), SIGTERM was ignored, and Docker SIGKILLed the container
after the 10 s grace period. Two confirmed wedges on the public
test deployment (52 min and 23 min uptime), plus a third on the
admission-gate-Msg2-silent-drop build at 2 min 21 sec — all ending
with the identical "Peer removed and state cleaned up
tree_changed=false" final log line preceding total silence.
Fix: detach the std::thread instead of joining. The stop flag plus
self-pipe write already signal the worker to exit; the worker's
kernel-level libc::poll inside the drain loop sees the wake, checks
the flag, exits, and the OS reclaims the thread state independently
of the JoinHandle being dropped.
The trigger was statistically amplified by aggressive multi-npub-
from-one-NAT peer reconnect patterns at the moment of the 30 s
link-dead-timeout peer-removal, but not bounded to them. Any
peer whose disconnect happens with the per-peer drain worker
parked in block_on can fire the bug. The admission-gate work
that landed earlier in this branch line compressed more handshake
work per rx_loop tick, increasing the rate at which workers are
parked in block_on and so reducing time-to-wedge — but the
underlying bug pre-dated the admission gate and pre-dated this
fix branch.
The deployed wedged daemon is mitigated operationally by blocking
the trigger IP at the host firewall; this commit removes the bug
class entirely.
Bring the refactor-hotpath integration branch into master: explicit
RejectReason counters for previously-silent receive-path drop sites
across the tree, discovery, and handshake handlers, plus the Reloadable
trait and ArcSwap-backed hot-reload consolidation for the host map and
peer ACL. Includes the new discovery dedup-cache-full reject counter.
The discovery request dedup cache (recent_requests) silently dropped
LookupRequests once it reached MAX_RECENT_DISCOVERY_REQUESTS, with no
counter to surface the condition. Add a DiscoveryReject::ReqDedupCacheFull
reject reason backed by a req_dedup_cache_full counter on DiscoveryStats,
mirroring the existing duplicate-request counter, and record it at the
drop site so the rejection is visible in show_routing.
Bring the runtime peer-list refresh and opt-in mDNS LAN discovery work
on master into the receive-path RejectReason / reloadable-config
integration branch. Code files auto-merge clean; the only conflict is
the CHANGELOG Unreleased section, resolved as the union of both sets of
entries.
Move PeerAclReloader onto the Reloadable trait: its ACL snapshot is now
published through an arc_swap::ArcSwap so the authorization hot path reads
it without locking, and the former check_reload becomes the trait's
reload(). The node tick calls self.peer_acl.reload().await.
Wire the host map into the tick as well. The host map snapshot was
previously taken once at construction and never polled; it now hot-reloads
on /etc/fips/hosts mtime changes once per tick, alongside the ACL, so
hostname display reflects edits without a restart.
The path_mtu_lookup cache (event-driven, populated from observed traffic)
and the nostr_discovery subsystem (an async spawned task) are deliberately
left off the trait: neither reloads from a backing file, so a no-op reload()
would be misleading. The rationale is documented on the trait module.
The host map and the ACL's embedded alias reloader still stat /etc/fips/hosts
independently each tick. A single small-file stat per tick is cheap, so the
duplicate is left in place; sharing one mtime observation between the two is
a possible future cleanup.
Tests: a node-level test exercises the host-map tick reload end to end
through peer_display_name; the ACL reloader tests are updated to drive the
async reload().
Add a `Reloadable` trait that normalizes the node's reloadable
configuration/resource pattern onto a single contract built around an
`arc_swap::ArcSwap` snapshot: a lock-free `load()` for the hot read path
and an async `reload()` that re-reads the backing source and atomically
swaps in a fresh snapshot. The trait carries the canonical Arc-wrapper
template documentation (single-writer node tick, many-reader hot path,
whole-snapshot swap so readers never observe a partial update).
Migrate the host map to this trait via a new `HostMapReloadable` that
reuses the existing load/merge/mtime helpers in upper::hosts. The Node
`host_map` field changes from `Arc<HostMap>` to `HostMapReloadable`, and
`peer_display_name` reads through a lock-free guard. The initial snapshot
is byte-identical to the previous construction, so behavior is unchanged.
The host map is still snapshotted once at construction and not polled;
`reload()` is exercised only by unit tests for now. Wiring the periodic
poll into the node tick, and deduplicating the hosts-file stat against
the ACL reloader's embedded copy, is left as a follow-up.
Add `arc-swap` as a dependency. Unit tests cover initial load (base +
file, base only), change/no-change/deletion/creation detection,
base-preserved-on-reload, and equivalence of the initial snapshot to the
pre-migration construction.
Introduce a typed RejectReason enum and a NodeStats::record_reject
dispatch so every receive-path rejection-and-return site bumps a
machine-readable per-subsystem counter while keeping its operator-facing
log line. The top-level variants mirror the existing NodeStats subsystem
split (Tree, Bloom, Discovery, Forwarding) and add Handshake, Session,
Mmp, and Transport categories; HandshakeStats, SessionStats, and MmpStats
are new sub-stats.
Wired clusters: tree and MMP outbound sign-failure; the FSP session
unknown-session and state-machine cluster; the Noise IK handshake
state-machine cluster (msg1/msg2); and the decode / crypto / cap /
semantic tail across bloom, discovery, forwarding, mmp, and tree. The
TreeStats::ancestry_invalid counter, present since the scaffold but never
incremented, is now bumped from the validate_semantics ancestry rejection.
Several handshake, MMP, tree, and discovery paths that previously had no
counter at all are now counted, including the send_lookup_response
no-route drop (DiscoveryStats::resp_no_route).
Existing direct counters at the bloom / discovery / forwarding sites are
retained alongside the new dispatch while the rollout is in progress (the
bloom_poison tests expect the transitional +2 delta); a later change
collapses the duplicate increment.
Raise the in-process backpressure headroom in make_test_node_with_mtu
(request an 8 MiB recv_buf_size on UdpConfig and grow packet_channel from
256 to 8192) to reduce localhost-UDP receive overflow under parallel-CPU
scheduler contention, and mark the large-network convergence tests
#[ignore] so cargo test --lib stays green by default. The ignored tests
remain runnable on demand with --ignored or --test-threads=1.
Add per-direction pool_inbound/pool_outbound counters to TcpStats and
TorStats, updated at every pool-insert, receive-loop-exit, transport-stop,
and send-failure removal site. Compare the max_inbound_connections cap
against pool_inbound rather than the combined pool length, so outbound
connect-on-send connections no longer consume the operator-facing inbound
budget. The configuration field name and operator semantics are preserved;
only the cap-check comparison and accounting change.
Johnathan Corgan