The `mixed-profile` integration suite (Full / NonRouting / Leaf
node-role mix introduced by the forklift-upgrade work) is wired
into `testing/ci-local.sh` but was silently absent from the GitHub
Actions integration matrix on next. Regressions to node-role
behavior could land via GitHub CI without the suite running.
Adds a matrix entry plus the matching step block (mirroring the
rekey pattern: generate configs + inject config + compose up +
test + log-collect on failure + compose down always). master
correctly omits the suite from both ci-local and ci.yml since
mixed-profile is forklift-only, so this change never forward-merges
to master.
The dual-initiation tie-breaker in the rekey arm of handle_msg3 (and
the FSP analogue in handle_session_setup) only fired when
rekey_in_progress was true. With Noise IK (one-message rekey on
master/maint) that is sufficient: msg1 IS the rekey, so both sides
being mid-handshake is the only state where the race can fire.
With Noise XX (three-message rekey on next), set_pending_session runs
when the initiator has processed msg2 and sent msg3, and that call
clears rekey_in_progress. Both sides' set_pending_session can run
before either peer's msg3 has landed. Each side then receives the
peer's msg3 in the post-pending state with rekey_in_progress=false,
falls into the "drop because pending_new_session is set" guard, and
discards the peer's handshake. Each side commits its own initiator
session at K-bit cutover. The two sessions use different Noise key
material, so the link breaks asymmetrically after cutover.
Unify both checks into a single tie-breaker that fires when either
rekey_in_progress() OR pending_new_session().is_some() is true. The
existing smaller-NodeAddr rule applies uniformly; abandon_rekey()
already clears both states and returns whichever index needs
freeing.
Mirrored to the FSP rekey msg1 path in handle_session_setup for the
same race shape.
Logging:
- The previously-silent drop in the rekey arm of handle_msg3 logs at
info as a tie-break decision rather than as a silent drop.
Pending_new_session is added as a log field on the tie-break
win/lose lines so a reader can distinguish which of the two race
states fired.
- Cutover-complete and K-bit-flip log lines gained our_addr and
their_addr fields so the two endpoints' logs of the same handshake
can be correlated.
- "Pending session set, awaiting K-bit cutover" lines remain at
debug per the existing convention of info-for-cutover-completion-
only.
Verification: under the un-fixed handler, both sides log
"rekey-msg3 drop: pending_new_session already set" within 242
microseconds of each other, with rekey_in_progress=false and
different pending session indices. Failure pattern matches: six of
twenty pairs FAILED post-rekey, all involving the single-peer node.
Six consecutive post-fix CI attempts on the same test green across
rekey, rekey-accept-off, and rekey-outbound-only suites.
Three entries under [Unreleased] for the three commits since the
v0.3.0 release tag:
- Sidecar example: FIPS_UDP_MTU env override (commit 32a3b58)
- CONTRIBUTING.md overhaul + new docs/branching.md (commit 538ce07)
- Rekey-test Phase 1 baseline-convergence headroom 36s -> 60s
(commit 6533276)
The previous CONTRIBUTING.md read like a stock Rust contributing
template that mentioned FIPS. Replace it with an entry-point doc
that gives new contributors the FIPS-specific mental model they
need to make a useful first PR: the FMP/FSP layering and why
mesh-level changes need multi-node testing, the three-branch
release model and how to choose a target branch, structured bug
reporting expectations, and PR submission requirements (scope
discipline, the local-CI ladder, separate requirements for feature
PRs vs bug-fix PRs, squash-merge mechanics).
Add a contributor-facing AI coding assistant policy: use is
welcome, but the contributor must do a thorough manual review and
editorial pass before submission. The agent is a tool; the
contributor is accountable for the submission. Review effort
scales with submission effort -- unreviewed agent output will
receive an agent reply in turn, without human review.
Add docs/branching.md as the long-form companion covering the
release workflow, version conventions, and merge-direction
rationale. The new CONTRIBUTING.md is the day-to-day entry point;
docs/branching.md is the reference.
All cross-references resolve against current HEAD. Previous
stale links to fips-intro.md, fips-wire-formats.md, and
fips-configuration.md are gone; the doc points at the current
docs/design/ layout, docs/getting-started.md,
docs/tutorials/join-the-test-mesh.md, and testing/README.md.
No code changes.
The Phase 1 pre-rekey baseline in `wait_for_full_baseline`
occasionally times out on GitHub-hosted runners with one ping pair
failing to converge inside the BASELINE_CONVERGENCE_TIMEOUT window.
Phases 2–6 always pass cleanly when this happens — the rekey itself
is fine, the mesh just hasn't finished spanning-tree + bloom-filter
convergence by the time Phase 1 starts pinging.
The wait loop returns as soon as all 20 pairs converge, so the cost
on the success path is unchanged (typical local CI returns well
under the old 36s). The bump only adds headroom on the failure
path. Operators previously worked around this with
`gh run rerun --failed`; this aims to retire that workaround for
the IK/XK lines.
Distinct from the next-branch XX rekey dual-init race, which is a
real protocol bug tracked separately.
The sidecar entrypoint hardcoded `udp.mtu: 1472`, the Docker-bridge
IPv4 maximum (1500 MTU - 8 UDP - 20 IPv4 header). Promote it to
`FIPS_UDP_MTU` (defaulting to 1472, preserving behavior) so non-Docker
reuses of the example can override without editing the script. Plumb
the env var through `docker-compose.yml` so a host-level setting
reaches the container, and add a row to the README's env-var table.
Also annotate the static-CI node template with a comment explaining
the same 1472 rationale and the daemon's 1280 default. The template
keeps 1472 as the literal value since the CI suite runs on a Docker
bridge where that's correct.
No behavior change unless the host explicitly sets FIPS_UDP_MTU.
The v0.3.0 stable AUR push silently failed: with updpkgsums: true,
makepkg downloaded fips-<ver>.tar.gz into the AUR working tree, where
it was then staged by the deploy action and rejected by AUR's 488 KiB
max-blob hook.
Fetch the upstream source tarball and compute its b2sum in CI, patch
pkgver and the b2sums SKIP placeholder in PKGBUILD in-place, then
publish with updpkgsums: false so the AUR clone stays metadata-only.
Recompute and patch the fips.sysusers / fips.tmpfiles asset b2sums in
the same step so they stay in sync with the local files; this safety
net was previously provided by updpkgsums.
Add aur-publish-git.yml for the VCS fips-git PKGBUILD, triggered on
master pushes that touch PKGBUILD-git or its companion files plus
workflow_dispatch. pkgver is computed at build time by the PKGBUILD's
pkgver() function, so this workflow is not tied to release tags.
Add a workflow_dispatch tag input on the stable workflow so historical
release tags can be re-published manually, and drop continue-on-error:
true so future regressions surface in CI.
Marker merge to record the master dev-line (now v0.4.0-dev, with
maint's v0.3.1-dev open already incorporated) as known on next
without taking master's tree (next is on v0.5.0-dev). Future
forward-merges from master to next land cleanly on top of this base.
Marker merge to record the maint dev-line as known on master
without taking maint's v0.3.1-dev opening commit (master is on
v0.4.0-dev). Future bug-fix forward-merges from maint to master
land cleanly on top of this base.
Master is now the v0.4.0-dev line; next moves up to v0.5.0-dev for
the wire-format-breaking work staged here.
- Cargo: 0.4.0-dev → 0.5.0-dev
- CHANGELOG Breaking-section header retargeted to v0.5.0; baseline
bumped from v0.2.x to v0.3.x peers (master is now the v0.3.1-dev
patch line and v0.4.0-dev minor line, so v0.5.0 breaks against
v0.3.x, not v0.2.x)
- README: badge v0.4.0--dev → v0.5.0--dev, three Status & Roadmap
prose references retargeted
- docs/reference/cli-fips.md: example version 0.4.0-dev → 0.5.0-dev
Reset maint to v0.3.0 to retire the v0.2.x maintenance window and
open a new tracking branch for v0.3.1-dev bug-fix work.
- Cargo: 0.3.0 → 0.3.1-dev
- CHANGELOG: fresh [Unreleased] block above [0.3.0]
- README: badge v0.3.0 → v0.3.1--dev
Brings v0.3.0 release content forward into the next-branch
development line, preserves next's own version state, and updates
operator-facing version references to v0.4.0-dev.
Kept from master:
- docs/releases/release-notes-v0.2.1.md and -v0.3.0.md (cumulative
archive of shipped releases)
- RELEASE-NOTES.md root mirror at v0.3.0 (tracks the most recent
shipped release; will be replaced when the v0.4.0 release cycle
begins on this line)
- CHANGELOG entries: [0.3.0] and [0.2.1] sections inserted under
the existing Breaking and [Unreleased] sections
- All code, config, test, and documentation updates from master
(openwrt yaml resync, doc-config IP placeholders, etc.)
Kept from next (resolved against master's release-prep changes):
- Cargo.toml / Cargo.lock at 0.4.0-dev (next's package version)
- CHANGELOG ## Breaking section (next-specific v0.4.0 wire-format
breaking work) and the empty ## [Unreleased] block for future
v0.4.0 non-breaking work
Updated to v0.4.0-dev for consistency with Cargo.toml:
- README badge (v0.3.0--dev -> v0.4.0--dev) and status-section
prose, rewritten to describe v0.4.0 wire-format-breaking work
on this branch (Noise XX unification, FMP node profiles,
slimmer MMP, extensible bloom-filter encoding) instead of the
v0.3.0 testing-and-polishing narrative that applied while
v0.3.0 was unreleased
- docs/reference/cli-fips.md example version string
Bump Cargo.toml version 0.3.0-dev -> 0.3.0 and resync Cargo.lock,
move the CHANGELOG [Unreleased] block under [0.3.0] - 2026-05-11,
update the README status badge and prose to v0.3.0, and add the
release notes at docs/releases/release-notes-v0.3.0.md with a
mirrored copy at the repo root as RELEASE-NOTES.md.
Uses the 'ours' merge strategy to keep all of master's tree
verbatim with one carve-out: the v0.2.1 release notes archive at
docs/releases/release-notes-v0.2.1.md is retained as a permanent
docs artifact.
Discarded from maint's release-prep commit: Cargo.toml / Cargo.lock
version bumps (master stays on v0.3.0-dev), README v0.2.1 badge and
prose, and the RELEASE-NOTES.md root mirror (master's root mirror
will eventually carry the v0.3.0 release notes when v0.3.0 ships
from this line).
CHANGELOG: the v0.2.1 release section is integrated between
[Unreleased] (v0.3.0-dev work) and the [0.2.0] historical section.
Items that landed on maint during the v0.2.1 cycle are removed
from [Unreleased] so [0.2.1] is the canonical home for those
changes (Linux release artifact and AUR workflows; bloom fill-ratio
validation; seven maint-line bug fixes).
Operator-facing IPs in user-visible configs/docs (examples, tutorials,
packaging, sidecar templates) are now the resolvable hostnames of the
public test fleet (test-us01.fips.network, etc.) so they keep working
without baking specific addresses into examples.
Doc-comment and test fixtures in src/config/transport.rs use RFC 5737
TEST-NET-2 (198.51.100.1) so they cannot accidentally point at a real
host.
Also resyncs the openwrt-ipk fips.yaml with the common reference
(merge from master) and applies the same DNS-name swap there.
Bring the OpenWrt-shipped fips.yaml back into line with
packaging/common/fips.yaml, which had drifted: the OpenWrt copy was
missing the Nostr discovery, BLE, and TCP reference comment blocks, so
operators had no in-config hint that Nostr-mediated discovery existed.
Take common/fips.yaml verbatim and apply just the OpenWrt-specific
active overrides:
- ethernet: uncomment with wan/wwan/lan defaults (eth0, phy0-sta0,
br-lan)
- gateway: uncomment with lan_interface=br-lan and dns.listen on
[::1]:5353 (matches the dnsmasq forwarder the init script wires up)
Identity stays ephemeral by default (no persistent override), matching
common.
Bump Cargo.toml version 0.2.1-dev -> 0.2.1 and resync Cargo.lock,
move the CHANGELOG [Unreleased] block under [0.2.1] - 2026-05-11,
update the README status badge and prose to v0.2.1, and add the
release notes at docs/releases/release-notes-v0.2.1.md with a
mirrored copy at the repo root as RELEASE-NOTES.md.
The host (217.77.8.91:443) was renamed to test-us01 some time ago
(canonical name shared with packaging/common/hosts and the docs);
the test scaffold's alias labels and narrative comments hadn't been
updated. Pure naming cleanup, no behavior change. The continued
dependency on the live external host is tracked separately.
Walk through reviewer feedback on the Nostr-discovery docs and
land 18 items.
Bulk patterns:
- `external_addr` / `public: true` semantics consistently
misdescribed. The advert path is gated on `cfg.is_public()`;
inside that branch the daemon picks an address by precedence
(`external_addr`, non-wildcard `bind_addr`, STUN). The docs
treated `public: true` and `external_addr` as alternatives when
they are stacked: `public: true` is the master switch and
`external_addr` populates the address inside it. Reconciled
across `enable-nostr-discovery.md` and `advertise-your-node.md`:
add `public: true` to the `external_addr` examples; replace
"STUN as a logging cross-check" with "STUN is skipped entirely";
fix "neither flag is needed" for direct public bind (both flags
still required); make the publish-tutorial Step 3 conditional
on the chosen Step 2 path (STUN runs only on the `public: true`
path); rewrite the troubleshooting "wrong public IP advertised"
bullet with two coherent fixes.
- `udp:nat` overpromised as a symmetric-NAT solution. Symmetric
NAT on either side typically defeats the punch. Reframe
`udp:nat` as best-effort hole-punching for nodes without a
directly reachable UDP endpoint in the how-to, the publish
tutorial (intro, callout, section heading rewrite from "If
you're behind symmetric NAT" to "If your direct UDP advert
isn't reachable"), the consume tutorial's "What's next"
pointer, and `tutorials/README.md`. Promote reachability over
named NAT classes: STUN can confirm the public IP but not that
the listener-port mapping is open.
- YAML "silently ignores unknown keys" is wrong. Config parser
rejects unknown fields via `serde(deny_unknown_fields)` on the
per-section structs; misspelled fields refuse the daemon's
start with a parse-error line in the journal. Fixed in the
publish tutorial's troubleshooting and the open-discovery
tutorial's `policy` typo bullet.
Mechanical fixes:
- Repoint stale anchors. `getting-started.md` and
`configuration.md` linked to `#installation` / `#inspect` on
the README; the README has no such headings. Repoint to
`#quick-start` and `cli-fipsctl.md`. Two stale anchors in the
publish tutorial pointing at non-existent sub-scenarios in the
how-to (`#sub-scenario-2c-...`,
`#sub-scenario-2b-tor-onion-node`) repointed to the correct
anchors.
- Drop the `fipsctl show status` claim from the open-discovery
troubleshooting bullet (`show_status` doesn't include
`discovery.nostr.policy`). Replace with daemon startup logs.
- Fix the `advertise: false` parenthetical in the consume-only
tutorial (`default_advertise()` returns `true`; we set `false`
explicitly for the consume-only path).
- Drop the "supplies a relay list" overstatement in two
activation paragraphs (the how-to and the design doc). Default
relay / STUN-server lists ship in the config; both are
optional overrides.
- Add the missing `transports.udp.public` entry to the
open-discovery tutorial's prerequisites checklist. Tutorial
users coming out of advertise-your-node could be on either the
direct-UDP (`public: true`) or `udp:nat` (`public: false`)
path; list both.
Files: docs/getting-started.md, docs/reference/configuration.md,
docs/how-to/enable-nostr-discovery.md, docs/tutorials/README.md,
docs/tutorials/advertise-your-node.md,
docs/tutorials/resolve-peers-via-nostr.md,
docs/tutorials/open-discovery.md,
docs/design/fips-nostr-discovery.md.
Four short prose corrections folded together to align operator-facing
docs with current v0.3.0 reality:
- README Rust prerequisite reconciled with the toolchain pin (1.94.1,
was 1.85+).
- CONTRIBUTING.md bumps the Rust prerequisite and adds the squash-merge
policy note.
- examples/sidecar-nostr-relay/Dockerfile drops stale --features tui
and the paired --no-default-features; the tui/ble/gateway cargo
features were replaced by platform cfg gates in cbc7809.
- README Features list adds two v0.3.0-visible items: the mesh-
interface security baseline (fips.nft conffile, fips.d/ drop-in,
opt-in fips-firewall.service across all packaging formats) and the
fipsctl stats time-series queries plus fipstop inline sparkline
dashboards.
Status badge bump (v0.3.0--dev to v0.3.0) is deferred to tag time per
the release-prep checklist.
The generic systemd install tarball is the catch-all install path
for systemd Linux distros that don't have a per-format package
(Fedora, RHEL/CentOS, openSUSE, Alpine, etc.). It had drifted
behind the .deb and AUR packages and was missing fips-gateway, the
mesh-interface firewall baseline, and the multi-backend DNS helper
in the shipped tarball. Bring it to parity:
- New `packaging/systemd/fips-gateway.service` (clone of the .deb
unit; ExecStart pointed at `/usr/local/bin/fips-gateway`). Not
enabled at install time; operator opt-in.
- New `packaging/systemd/fips-firewall.service` (clone of the .deb
unit; nft path unchanged at `/usr/sbin/nft`). Not enabled at
install time; operator opt-in.
- `build-tarball.sh` now bundles the `fips-gateway` binary, the two
new units, the `fips.nft` baseline conffile, and the
`fips-dns-setup` / `fips-dns-teardown` multi-backend helpers from
`packaging/common/`.
- `install.sh` now installs `fips-gateway` to `/usr/local/bin/`,
installs both new units to `/etc/systemd/system/` (without
enabling them), preserves `/etc/fips/fips.nft` on upgrade like
`fips.yaml`, and creates the `/etc/fips/fips.d/` operator drop-in
directory. Post-install messaging mentions both opt-in services.
- `uninstall.sh` stops and disables the optional services in
dependency order (firewall, gateway, dns, daemon), removes the
new unit files, and removes the gateway binary. `--purge` already
handles `/etc/fips/` removal which covers `fips.nft` and
`fips.d/`.
- `README.install.md` documents all of the above: expanded
"What Gets Installed" table, new sections covering the firewall
baseline and the LAN gateway, refreshed DNS section reflecting
the multi-backend setup helper (systemd dns-delegate /
systemd-resolved drop-in / per-link resolvectl / dnsmasq /
NetworkManager-dnsmasq), and updated Service Management.
Also fixes a latent packaging bug: `install.sh` previously
referenced `${SCRIPT_DIR}/../common/fips-dns-setup`, a path that
exists only in the source-repo layout and not in the extracted
tarball. The script now resolves the helper from the staging
directory first (the tarball case), falling back to the source-repo
relative path. Bug latent since the multi-backend DNS helpers
landed.
CHANGELOG `[Unreleased]` documents the parity bump under Changed
and the path-resolution fix under Fixed.
Closes the longest-standing parity gap for non-Debian / non-Arch
systemd Linux distros installing from the release-distribution
tarball.
Three changes folded together close the AUR-side parity gap with the
.deb packaging:
- PKGBUILD now ships fips.nft baseline and fips-firewall.service, with
fips.nft marked as backup so operator edits survive upgrades.
- PKGBUILD-git mirrors the release PKGBUILD: installs fips-gateway
(was missing entirely), ships fips.nft and fips-firewall.service,
and tracks the same backup() set.
- packaging/aur/README.md documents the gateway, firewall service,
and nft baseline that ship as of this revision.
AUR users now receive the same artifact set as .deb users.
The Noise-session AEAD swap landed as 5cda4a9 + 9b1016f without a
companion CHANGELOG entry; add it under [Unreleased] § Changed
ahead of the rest of the perf-win cluster from PR #81 since it's
the most operator-visible single change of that class.
Investigated the mipsel-unknown-linux-musl build of this branch on a
Linux/x86_64 host. ring 0.17, portable-atomic, and the fips codebase
itself all compile cleanly for that target — the AtomicU64 portability
work is already done. The actual blocker is in the nostr-relay-pool 0.44
transitive dep, which uses std::sync::atomic::AtomicU64 directly in
src/relay/{stats,ping,flags}.rs. Verified fixable with a 6-line
portable-atomic patch via a [patch.crates-io] shim during local testing.
Updating the comment so the next person looking at this matrix has the
right starting point.
No functional change.
The chacha20 crate (RustCrypto) ships SSE2 + soft backends only — on
aarch64 (Apple Silicon, ARM Linux servers, Docker on M-series Macs) it
falls through to a portable software impl at ~600–800 MB/s/core. ring
0.17 wraps BoringSSL's hand-tuned ChaCha20-Poly1305, which dispatches
to NEON on aarch64 and AVX2/AVX-512 on x86_64 — typically 3-5 GB/s/core
on the same hardware.
Same wire format. ChaCha20-Poly1305 is byte-deterministic for a given
(key, nonce, plaintext, aad), so any correct AEAD implementation
produces identical ciphertext. The full noise test suite covers this
implicitly: IK and XK roundtrip handshakes, replay window correctness,
multi-message nonce sequencing, and 100-message stress all pass at
1129/1129 (the lib's full `cargo test` count) — these only succeed if
ring's output matches what the receiver's existing replay-window
decrypt path expects.
Implementation notes:
* `LessSafeKey` (and `UnboundKey`) deliberately do not implement
Clone for safety. `CipherState`'s manual Clone impl rebuilds it
from the retained 32-byte key — cheap for ChaCha20-Poly1305 since
construction is essentially a key copy + a constant-time check.
* The keyed AEAD is now cached in `CipherState.cipher` instead of
being re-derived per packet. This was already a perf win for the
chacha20poly1305 backend (`new_from_slice` per packet was hot in
profiles); for ring it's a bigger win because `LessSafeKey`
construction also derives the Poly1305 key.
* Public `Vec<u8>`-returning API preserved. New module-private
`seal`/`open` helpers wrap ring's `seal_in_place_append_tag` /
`open_in_place` so the per-packet allocation pattern is local to
one place.
* `EndToEndState::Established` triggers `clippy::large_enum_variant`
after the swap (`NoiseSession` grew from ~600 to ~1.5 KB because
ring precomputes the Poly1305 key state at construction). That
precomputation is the win — boxing the variant would re-add an
indirection per packet and work against it. `#[allow]`'d at the
enum decl with a justifying comment.
ring is widely deployed (rustls, hyper-rustls, AWS SDK, …) and a
pure-Rust crate (uses BoringSSL's asm via a vendored build). It
introduces no new C toolchain requirements that aren't already there
for any rustls user.
Bench data from a downstream consumer of this crate (Docker e2e,
DURATION=10, identical hardware before/after, aarch64 Linux on
Apple Silicon):
2-node direct (A↔B):
TCP 1-stream 437 → 1097 Mbps (2.51×)
TCP 4-stream 439 → 1109 Mbps (2.53×)
TCP 8-stream 445 → 1069 Mbps (2.40×)
UDP @1000 Mbit 599/40% loss → 1000 Mbps lossless
ping under load ~0.6 ms (unchanged)
3-node forced transit (A → C → B):
TCP 1-stream 438 → 1019 Mbps (2.33×)
TCP 4-stream 421 → 982 Mbps (2.33×)
TCP 8-stream 443 → 1031 Mbps (2.33×)
UDP @1000 Mbit 475/52% loss → 1000 Mbps lossless
ping under load 7.68 ms / 215 ms max → 0.72 ms / 3.6 ms max
The relay-path lift is the cleanest tell on the bottleneck: the
transit node was crypto-bound (single-threaded soft chacha couldn't
keep up with offered rate), so the queue accumulated under load. With
NEON the relay isn't crypto-bound and the queue stops accumulating —
the 215ms ping-tail collapses to 3.6ms.
Freshly-restarted nodes with policy: open silently lost the historical
event replay that relays send in response to subscribe(). The
broadcast::Receiver was created INSIDE spawn_notify_loop, which the
tokio runtime starts at some indeterminate point after subscribe()
returns. tokio's broadcast channel only delivers messages sent after
the receiver is created; messages dispatched in the gap between
subscribe() issuing the REQ and the spawned task calling
client.notifications() were dropped by external_notification_sender.send
returning Err(SendError) with no subscribers attached.
Symptom on a node with policy: open: non-configured peers were not
discovered until they next re-published their advert (default
advert_refresh_secs = 1800s = 30 min). Configured peers were unaffected
because fetch_advert (relay-fetch path) caches them at startup-sweep
time. The bug has been latent since 34e00b9 added Nostr discovery —
relay-fetch covered the common case for configured-peer setups.
Fix: create the broadcast::Receiver in start() before subscribe() and
pass it into spawn_notify_loop. The receiver now exists when the REQ
replay arrives, so historical events flow through the cache path.
Also handle broadcast::error::RecvError::Lagged separately from
::Closed. The previous `while let Ok(...) = recv().await` exited the
loop on any Err, so a single lag event would silently kill the entire
subscription consumer with no recovery. Lagged now logs a warn (with
the skipped count) and continues; only Closed exits the loop.
Add two info-level log lines for in-field observability of the loop's
liveness. "nostr notify loop entered" fires once at task start; "nostr
notify loop received first event" fires once after the first
successful recv() with elapsed_ms since loop entry. Together these
turn the previous silent-failure shape (zero advert: peer cached
log lines indistinguishable between dead loop and idle channel) into
an immediately greppable startup signal — operators can confirm the
loop is alive and see how long it took to receive its first event,
catching any future regression in the subscription codepath in
seconds rather than waiting one advert_refresh_secs interval.
No public API change; the test fixture (NostrDiscovery::new_for_test)
does not call spawn_notify_loop and is unaffected.
Three Changed entries for the rx-path performance work
(Linux UDP recvmmsg batched receive, run_rx_loop drain batching, and
eager pubkey_full precompute on PeerIdentity construction) and five
Fixed entries: adopted NAT-traversed UDP transports inheriting the
primary listener's MTU and buffer config, TreeAnnounce ancestry on
self-root transitions, unconditional overlay-advert refetch before
each retry, stale overlay-advert eviction on NoTransportForType,
and scheduled retry on startup peer-init failure.
Pure CHANGELOG addition (+117 lines, no edits to existing entries).
Bullets are wrapped at 80 columns and attribute external
contributions to the originating PR and author.
The advert cache inside fetch_advert is read-only on hit — once a peer's
overlay advert is cached, every subsequent lookup returns the same
endpoints regardless of whether they still work. So when a peer rebinds
its NAT (or its STUN-discovered port flaps), connection retries to that
peer dial the same dead address forever, even with exponential backoff
firing at the right cadence.
Observed in deployment: macOS daemon's view of a Linux peer would
"regress" — peer marked rch=False after a brief link-dead window, then
hours of "Retry connection initiation failed: no operational transport
for any of <npub>'s addresses" with no recovery. Manual pause+resume of
the daemon (which restarts the FIPS endpoint and forces fresh advert
fetches) was the only way out.
When initiate_peer_connection / a retry tick returns
NodeError::NoTransportForType, fire-and-forget refetch_advert_for_stale_check
on the peer's npub. This re-fetches kind 37195 from advert_relays; if
the relay has a newer advert it replaces the cached entry, if it has
nothing it evicts the cached entry. Either way the next retry tick goes
to fresh data instead of looping on the same dead endpoint.
Mirrors the existing stale-advert sweep that runs from the
BootstrapEvent::Failed (NAT-traversal-streak) path, but covers the
direct-UDP-retry path which never crosses that streak threshold.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
When initiate_peer_connections() runs at boot, address resolution can
fail for an entire peer (no operational transport for the configured
transport types, all addresses unreachable, NAT rebind invalidated cached
endpoints, etc.). Before this change the failure was logged and silently
forgotten — the peer entry stayed in a dead state forever, accepting
incoming pings but unable to answer them, until the daemon was manually
restarted.
The retry plumbing (schedule_retry / process_pending_retries with
exponential backoff) already exists and is wired into the post-handshake
failure paths (BootstrapEvent::Failed, MMP dead-link timeout, handshake
timeout). The startup loop just wasn't calling it. Mirror the
BootstrapEvent::Failed path: on a startup peer-init error, parse the
peer's npub and call schedule_retry so the peer recovers without
operator intervention.
Includes a regression test that asserts retry_pending is populated when
initiate_peer_connections() fails for a peer with no operational
transport.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
The previous fix (6ebca3e) only refetched the advert when retry returned
NodeError::NoTransportForType (cache returned no addresses at all). But
the much more common stale-cache failure mode is: cache returns an
endpoint that LOOKS valid (the address it had last week, before the
peer's NAT rebound), the dial succeeds at the IP layer, the handshake
times out, MMP fires, schedule_reconnect adds the entry back to
retry_pending, next retry hits the same cached endpoint, dials it
again, times out again. Loop forever — no NoTransportForType ever
fires because the cache has data, just dead data.
Move refetch_advert_for_stale_check to before each retry attempt
unconditionally. Cheap (one Filter query against advert_relays with
a 2s timeout, bounded by the retry backoff cadence), and replaces the
cache only if the relay has a newer advert or evicts if the relay has
nothing. Keeps the retry loop pinned to relay ground truth instead of
whatever the cache happened to learn at startup.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
When a node was the smallest-NodeAddr peer it could see (no smaller
neighbor available as a parent), the spanning-tree state was promoting
it to root. But the ancestry it advertised on the next TreeAnnounce
still referenced its previous parent's path, so receiving peers
rejected the announce with `invalid ancestry: advertised root X is
not the minimum path entry Y`, blocking mesh transit on any path that
needed to traverse this node.
Detect the self-root transition explicitly in `TreeState::become_root`
and rebuild the advertised ancestry to start from self. Also surface
the same path through the MMP receive handler so a stale ancestry
inherited across reconnect is corrected eagerly rather than waiting
for the next observation tick.
Adds 80 unit tests in `tree::tests` covering self-root transitions,
mid-chain ancestor disappearance, and ancestry validation against the
new root, plus a regression in `node::tests::spanning_tree` for a
3-node chain where the middle node's only parent (the smallest-addr
peer) goes away — previously it would advertise an ancestry rejected
by both endpoints; now it self-roots cleanly.
The run_rx_loop's `tokio::select!` was costing one full scheduler hop
+ futex per inbound packet and per outbound TUN packet. Under
sustained load that capped throughput at one event per scheduler
quantum — independent of CPU (which sat near-idle) because every
iteration parked the worker, woke it via futex, processed one event,
then parked again.
After the await on `packet_rx.recv()` / `tun_outbound_rx.recv()`
fires, drain up to 256 additional ready items via `try_recv()` in a
tight inner loop before yielding back to `select!`. `biased` ordering
gives the data-plane branches priority over tick / control / DNS
under sustained load.
The 256 cap is empirically tuned to keep the worker on a busy stream
between yield points (a contiguous burst of ~256 MTU-sized packets
≈ 400 KB of contiguous traffic) while still bounding the inner loop
so a flood on one branch can't starve the periodic tick or control
socket. Lower caps (64) left perf on the table; higher caps (1024+)
delayed tick handling visibly under stress.
Pairs with the recvmmsg(2) change in the previous commit: the kernel
UDP queue now hands packets to `packet_rx` in 32-batches, and the
rx_loop drains them without a per-packet scheduler hop.
The UDP recv loop drained the kernel queue one packet per recvmsg(2).
Each call paid full per-syscall + per-task-wakeup overhead (~50us avg
including a futex-based scheduler hop), so under sustained load the
loop ran at one rx event per scheduler quantum — the dominant cap on
inbound packet rate.
On Linux, switch the steady-state path to recvmmsg(2) with a 32-packet
batch. A single readable() wakeup drains up to 32 datagrams in one
syscall before yielding back to the reactor. Stack-allocated mmsghdr
arrays sized to a module-level `BATCH_SIZE` constant.
`SO_RXQ_OVFL` is sampled once per batch off the cmsg chain of `msgs[0]`
and plumbed through `AsyncUdpSocket::recv_batch` as `(count, drops)`.
The counter is socket-wide and monotonic, so a single sample per batch
gives the 1Hz `sample_transport_congestion()` detector ample fresh
values under load (one batch = up to 32 datagrams). Cost is one
stack-allocated CMSG_SPACE(4) buffer + one CMSG_FIRSTHDR walk per
batch syscall.
macOS / Windows fall through to the per-packet recv_from loop —
recvmmsg is Linux-specific and the per-packet API is fast enough on
those platforms for now (recvmsg_x for Darwin can be added later).
The slice-array build also drops the `MaybeUninit::uninit().assume_init()`
+ `transmute` pair for `std::array::from_fn` over a single shared
`backing.iter_mut()` — same disjoint mutable borrows, no `unsafe`.
`PeerIdentity::pubkey_full()` falls through to
`self.pubkey.public_key(Parity::Even)` whenever the parity-aware full
key wasn't passed at construction (i.e. for every peer constructed
from an npub or x-only key). Underneath, that runs a secp256k1 EC
point parse — `fe_sqrt` + `fe_mul` + `ge_set_xo_var` — which is ~6%
of per-packet CPU on the bulk-data send path for a value that never
changes after construction.
Compute it eagerly. The same EC point parse already runs at
construction inside `NodeAddr::from_pubkey`, so the cost is paid once
where it would be paid anyway.
`Node::adopt_established_traversal` was constructing the adopted UDP
transport with `UdpConfig::default()` — MTU 1280, default recv/send
buffer sizes, default accept/advertise flags. If the operator had
configured a higher MTU on the primary `[transports.udp]` listener
(e.g. 1500 on a path where larger frames are known viable), full-sized
tunnel datagrams sent over the NAT-traversed link would exceed the
adopted socket's MTU and get dropped at the socket layer with no
visibility into why throughput collapsed.
Inherit the primary UDP config (MTU + recv/send buffer sizes + accept
/ advertise flags) and clear the bind / external-address fields since
the adopted socket is already bound. Lookup tries `transport_name`
first so operators with multiple named `[transports.udp.<name>]`
listeners pick up inheritance from the matching listener, and falls
back to the unnamed `Single` listener so single-instance configs work
unchanged.
The previous default of MTU 1280 was deliberately the IPv6 minimum,
the only value guaranteed to survive arbitrary middlebox paths. With
this change, operators who set their primary listener higher (based
on known-clean LAN topology) will have NAT-traversed flows initially
attempting that higher MTU and possibly black-holing on tighter paths
until reactive `MtuExceeded` recovery kicks in. Documented in the
adoption call-site comment so future readers understand why the
conservative default went away.
Discovered in a downstream consumer where a `MESH_TUNNEL_MTU=1320` /
encrypted wire ~1426B produced silent packet drop on every session
that had been promoted onto a NAT-traversed link.
Adds two sibling tests in `src/node/tests/bootstrap.rs` pinning the
new behaviour for the `Single` and `Named` config variants.
The drop-counter sanity check piped `nft list table inet fips`
through `awk '/counter packets/ {print $3; exit}'`. Awk's `exit`
on first match closes the pipe, the upstream `nft list` SIGPIPEs
on its next write, `set -o pipefail` makes the pipeline return
141, and the surrounding command-substitution aborts the script
before it can assign DROP_PKTS or print the section header.
Replaces the early-exit pattern with `/counter packets/ && !seen
{ print $3; seen=1 }` — same first-match output, but awk reads
the full input so nft never SIGPIPEs.
The original form had been latent for as long as the test has
existed; recent CI runs at master tip 53ad528 finally tripped
it (output shows the script dying immediately after the case-(d)
PASS, before "=== Drop counter incremented..." prints).
Verified locally: `bash testing/ci-local.sh --only firewall`
runs all six setup-and-functional steps green and prints
"PASS: drop counter = 5".
Mirrors systemd's RuntimeDirectory=fips so the daemon's
resolve_default_socket() picks /run/fips/control.sock inside
containers, matching production layout and the path that the
chaos sim harness (testing/chaos/sim/control.py) probes.
Without this, the resolver falls through to
/tmp/fips-control.sock (no /run/fips, no XDG_RUNTIME_DIR), the
daemon binds there, and the harness's hardcoded
/run/fips/control.sock probe returns FileNotFoundError on every
node. chaos-bloom-storm then fails its bloom_send_rate assertion
with start=0 nodes, end=0 nodes; other chaos scenarios pass only
because their tolerances absorb the empty samples.
Production hosts always have /run/fips materialized by the
fips.service unit's RuntimeDirectory directive before the daemon
starts, which is why the regression hit only the containerized
test path.
Verified locally with chaos-bloom-storm: max per-node delta 27
<= ceiling 30 over the trailing 30s window.
Surfaces local services reachable from the mesh, paired with their
current `inet fips` baseline filter classification. Lands to the
right of the existing TUN section in the Traffic block.
A new daemon control query `show_listening_sockets` returns IPv6
listeners bound to either `::` (wildcard) or the node's fd00::/8
address, each classified as Accept / Drop / Unknown / NoFirewall
against the running inbound chain. fipstop renders the result as a
table beside the Traffic counters: Accept rows in default White,
Drop / Unknown in DarkGray, a yellow banner above the table when
`fips-firewall.service` is inactive, and a trailing `*` on
wildcard binds to remind the operator the bind is not
fips0-specific.
Daemon side:
- `src/control/listening.rs` walks `/proc/net/tcp6` and
`/proc/net/udp6` via the procfs crate (LISTEN state for TCP,
wildcard remote for UDP), filters to fips0-reachable binds, and
resolves inodes to PID / comm via `/proc/<pid>/fd`.
- `src/control/firewall_state.rs` shells out to
`nft -j list table inet fips` and walks the inbound chain.
Recognises canonical accepts (`tcp/udp dport N accept`,
`dport { ... } accept`, `dport A-B accept`), the iifname-scoping
line, conntrack and icmpv6 lines (skipped). Any rule with
unrecognised matchers (saddr filters, jumps, daddr filters) or
non-terminal verdicts forces Unknown classification for the
ports it references. Eleven unit tests cover the classification
logic; the listening enumerator carries a /proc-parsing test of
its own.
- `show_listening_sockets` emits
`{fips0_addr, firewall_active, sockets[]}` with per-row
`{proto, local_addr, port, pid, process, filter, wildcard_bind}`.
fipstop side:
- `src/bin/fipstop/ui/dashboard.rs` splits the Traffic block into
a 50/50 horizontal layout; the existing TUN + Forwarded panel
occupies the left half.
- `src/bin/fipstop/ui/listening.rs` renders the right half.
- `main.rs` fetches the new query each tick when the Node tab is
active. Errors are non-fatal: an old daemon without the query
leaves the payload at None and the panel renders "loading...".
`Cargo.toml` gains `procfs = "0.18"` on the Linux target. IPv4
listeners are not enumerated — fips0 is IPv6-only.
Folded in: revert the default-socket lookup from writability-probe
back to existence-based selection. The previous tempfile-probe on
`/run/fips` silently steered fipstop / fipsctl onto an XDG path
the daemon never bound for any user in the `fips` group whose
shell session had not yet picked up the supplementary group (no
re-login after `usermod -aG`). `XDG_RUNTIME_DIR` is set on every
modern systemd-managed user session, so this hit the common case.
The kernel checks actual group membership at `connect(2)`, so a
user who genuinely cannot connect now gets a clear `EACCES`
rather than a silent path mismatch. Drops the now-unused
`is_writable_dir` helper. `XDG_RUNTIME_DIR` existence validation
is preserved.
Documentation:
- `docs/reference/cli-fipstop.md` — Node-tab row updated, new
"Listening on fips0 panel" section.
- `docs/reference/control-socket.md` — `show_listening_sockets`
added to the read-only queries table.
- `docs/how-to/enable-mesh-firewall.md` — new "Verify with
fipstop" section.
- `docs/tutorials/host-a-service.md` — fipstop callouts at
Steps 3, 5, 6 + Troubleshooting bullet + wildcard-bind reminder
under "What you've learned".
- `CHANGELOG.md` — new bullet under `Added / Operator Tooling`,
resolver `Fixed` entry rewritten to describe the
existence-based final shape.
Forward-merge of 12 master commits past the previous merge
(823b830, master @ 18019bb): dep-audit bumps (rand,
clap, tun, rtnetlink, windows-service, plus the bump-safe
lockfile batch), bloom-storm chaos scenario, control-socket
resolver consolidation, gateway dns.listen default change,
OpenWrt ipk README refresh, gateway tutorial review, Ethernet
MTU rustdoc fix, dead session-variant drop, CHANGELOG prep.
Conflict resolution:
- CHANGELOG.md: both bullets kept under [Unreleased] / Fixed.
Master's spanning-tree internal-path-propagation fix precedes
next's tree-ancestry-test determinism entry and the
responder-Disconnect XX-handshake entry.
- src/protocol/session.rs: kept next's SessionSetup/SessionAck
variants and rustdoc. Master's drop of those variants suits
v0.3.0's FSP phase-byte dispatch but is undone by next's
v0.4.0 wire format, which retains the variants and uses the
inner msg_type byte for handshake identification.
- src/transport/ethernet/mod.rs: kept next's "interface MTU - 4"
comment. Master corrected the v0.3.0 3-byte rustdoc; next
redesigned the framing to a 4-byte header (type/flags/length)
for shared-media beacons, so master's correction does not
apply to next's format.
Auto-merged cleanly: Cargo.toml (next's 0.4.0-dev + the new dep
pins from master), Cargo.lock, all gateway docs,
docs/reference/configuration.md, packaging files,
.github/workflows/ci.yml, testing/chaos/sim/* and
testing/ci-local.sh (bloom-storm additions), src/config/*.
Local verification: cargo build --release, cargo test (1252
passed, 4 ignored), cargo clippy -D warnings, cargo fmt --check
all green.
Johnathan Corgan
ArjenandJohnathan Corgan
Martti MalmiandJohnathan Corgan
