Two [Unreleased] / Changed entries that should have landed alongside
the originating commits but didn't:
- macOS recvmsg_x batched receive (originally 59225ccf): completes
the Linux-equivalent inbound batching shape on Apple builds. Now
sequenced before the rx zero-copy entry so the section reads as a
coherent receive-path progression.
- Platform-specific test-build warning cleanup (originally 6bd40640,
PR #93): non-behavioral; documents the gating decisions that keep
cross-platform builds warning-clean.
Two Fixed entries appended to [Unreleased]:
- The coord cache surgical invalidation (49bd2104): replaces the
global CoordCache::clear() at parent-switch / become-root /
loop-detection / root-change sites with two targeted methods
(invalidate_via_node, invalidate_other_roots). Preserves cache
entries that remain correct after the topology change.
- The rekey-test strict-ping retry (306e4555): Phase 1 / 3 / 5
per-pair pings now retry up to 4 attempts. Brings the ICMP-noise
miss-floor from ~33% per phase to ~3.2e-6 at 1% loss without
changing the failure-shape signal the asserts target. Test
scaffold only, no daemon code changes.
The non-Linux test build was emitting warnings from code that is
intentionally platform-specific: the nftables firewall parser is
Linux-only, the utun address-family helper is only used in macOS
TUN paths, and one macOS Ethernet test module trips a clippy
layout lint. These warnings made focused test runs noisy and
encouraged bundling unrelated warning fixes into behavioral PRs.
- Gate the firewall parser dead-code allowance to non-Linux
targets, where the parser is compiled but not used.
- Mark the macOS utun helper and long TUN reader entry point with
narrow allowances.
- Rewrite the small MAC-copy loop to satisfy clippy and mark the
macOS Ethernet test module layout explicitly.
No runtime behavior change.
The Phase 1, Phase 3, and Phase 5 strict asserts each fire a
single ping per directed pair. Under low-level packet loss
(e.g. 1% i.i.d. per-direction loss from a CI runner under
pressure), a single-shot round-trip fails at ~2% per pair, so a
20-pair strict assert misses with probability
1 - (0.98)^20 = ~33% per phase from ICMP noise alone, well above
the routing-state signal the asserts are meant to catch.
ping_one gains a max_attempts parameter (default 1, preserving
existing call sites). On failure it retries up to
MAX_PING_ATTEMPTS-1 additional times with PING_RETRY_DELAY
seconds between attempts. Per-pair worst case under the defaults
(4 attempts, 1 s spacing, 5 s ping6 -W timeout) is 4*5 + 3 = 23 s;
per-rep worst case scales with the failing-pair count.
Successful retries log "OK (RTT, attempt N)"; exhausted retries
log "FAIL (after N attempts)".
The retry budget is wired into all three strict asserts:
- Phase 1 final ping_all (after wait_for_full_baseline converges)
- Phase 3 ping_all (post-first-rekey)
- Phase 5 ping_all (post-second-rekey)
The wait_for_full_baseline convergence loop itself stays
single-shot. Its job is to detect when the mesh first sees a
fully clean 20-pair batch, and retries inside the loop would
conflate transient ping loss with still-converging routing
state.
No daemon code changes.
Replaces the unconditional `CoordCache::clear()` calls at parent-switch,
become-root, and loop-detection sites with two targeted invalidation
methods scoped to what actually makes an entry stale:
- `invalidate_via_node(node_addr)`: drop entries whose cached
destination ancestry contains `node_addr`. Used at parent-position-
change sites — our prefix changed, so destinations downstream of
us have stale-prefix coords.
- `invalidate_other_roots(current_root)`: drop entries rooted under
a different root than the current one. Used at root-change sites.
Under the previous global flush, parent switches blanked the cache
across the board, leaving `find_next_hop` returning `None` for every
non-direct-peer destination until the cache passively re-warmed via
incoming TreeAnnounces / SessionSetup. Surgical invalidation
preserves entries that remain correct after the topology change.
The cached coord describes a destination's tree position; that
position only goes stale relative to our own routing decisions when
our own prefix changes (entries we are downstream of) or the root
changes (entries in a different tree). Peer removal does not
invalidate cached coords: `Node::find_next_hop` recomputes the
next-hop decision on every call against the current peer set, bloom
filters, and tree state, and Discovery already triggers on
`no route to destination` errors when a destination becomes
unroutable through us. The peer-removal site retains the original
"no cache invalidation" behavior.
Each method returns the count of entries removed for observability.
Unit tests cover each method against the cases enumerated in the
acceptance criterion.
- Borrowed SessionDatagramRef decoder is used in the forwarding
handler so local delivery and coordinate-cache warming no longer
allocate or copy the session payload. The owned SessionDatagram is
materialized only when re-encoding for the next hop.
- Owned SessionDatagram::decode is reimplemented as Ref::decode +
into_owned, so the two decoders cannot drift.
- recvmmsg / recvmsg_x (Linux + macOS) receive loop moves each filled
slot buffer into ReceivedPacket via mem::replace instead of cloning
it; a fresh empty buffer is installed for the next syscall.
- TransportAddr is formatted directly from the SocketAddr without
going through an intermediate String.
Focused decode bench: ref 1.6 ns/op vs owned 34.7 ns/op (21.4x).
End-to-end iperf is neutral as expected for a ~30 ns saving per
packet.
Unit tests added:
- test_session_datagram_ref_decode_borrows_payload (verifies the
payload slice pointer equals the input slice's offset 35, a real
zero-copy invariant guard against accidental future to_vec)
- bench_session_datagram_decode_owned_vs_ref (ignored, run with
--ignored --nocapture)
- test_transport_addr_from_socket_addr
The Linux recv path drains up to 32 datagrams per kernel wakeup via
recvmmsg(2), amortising the per-syscall + per-task-wakeup cost across
the burst. macOS still fell through to single-packet recv_from, so
the same overhead capped inbound rate on Apple builds.
Add an equivalent batch path for Darwin using recvmsg_x(2). It is a
xnu-private syscall (not in the public SDK) but is the canonical
amortisation primitive on macOS — same shape used by quinn-udp for
the same reason. ABI is the public msghdr layout plus a trailing
msg_datalen (per-datagram bytes-received output), declared via
`unsafe extern "C"` against a local repr(C) `msghdr_x`.
Same `(count, kernel_drops)` contract as the Linux `recv_batch`. macOS
has no SO_RXQ_OVFL equivalent, so `kernel_drops` is always 0 — the
1Hz `sample_transport_congestion()` detector simply sees no kernel
drop signal on Apple hosts (it already tolerates that, since the
field has been 0 there pre-batching too).
cmsg buffer is intentionally null: we never consume ancillary data on
this path, and quinn-udp documents that `recvmsg_x` does not overwrite
`msg_controllen` on macOS 10.15+ (zeroed init is the only safe state).
The udp_receive_loop dispatch widens from cfg(linux) to
cfg(any(linux, macos)); the per-packet recv_from path is now used
only on the remaining unix targets (BSDs etc.) and Windows.
Add test_burst_recv_batch exercising 10 in-flight datagrams to
verify per-datagram boundaries and arrival order across the batch.
Add an ignored bench_udp_recv_amortization measuring recv-side
syscall amortization across 1/2/4/8 sender threads on dedicated
blocking std threads (kernel rx queue stays saturated regardless of
tokio scheduling). Sample numbers on aarch64-apple-darwin (100B
payloads, 3s windows):
senders=1: recv_from 398k pps recv_batch 432k pps 1.09x
senders=2: recv_from 353k pps recv_batch 608k pps 1.72x
senders=4: recv_from 322k pps recv_batch 503k pps 1.56x
senders=8: recv_from 353k pps recv_batch 515k pps 1.46x
Gate the Linux-only IpAddr import in control::listening behind a
cfg(target_os = "linux") so the macOS test build is warning-clean
now that test code paths there compile.
New testing/boringtun/ harness runs two Cloudflare BoringTun userspace
WireGuard containers with iperf3 between them, giving a single-hop
userspace tunnel baseline for comparison against FIPS throughput
numbers. Local WG key generation runs through the harness image so the
host needs no wireguard-tools.
New testing/static/scripts/iperf-compare-refs.sh builds two git refs
into separate fips-test:* images via git worktree and runs the same
static iperf topology against both, with RUNS-based repetition and
aggregate avg/min/max reporting.
testing/static/scripts/iperf-test.sh gains DURATION, PARALLEL,
SETTLE_SECONDS, IPERF_TIMEOUT env knobs and a per-path iperf timeout.
testing/static/docker-compose.yml selects the image under test via
FIPS_TEST_IMAGE; testing/scripts/build.sh respects CARGO_TARGET_DIR.
Author benchmark on aarch64 Docker Desktop:
boringtun bob -> alice : 1000.13 Mbits/sec
Skip BootstrapEvent::Established and BootstrapEvent::Failed dispatch
in poll_nostr_discovery for peers that are already connected or
actively handshaking. Without these guards, stale traversal events
arriving after a peer connected through a different path would
either attempt to adopt a redundant socket against the live
connection (Established) or poison the per-peer failure-state
cooldown and trigger redundant retraversal via schedule_retry /
try_peer_addresses (Failed).
The four guard sites use a new is_connecting_to_peer helper extracted
from the existing closure inside initiate_peer_connection; the helper
checks for an in-flight outbound handshake state. adopt_established_traversal
gains a defense-in-depth check returning PeerAlreadyExists when called
against an already-promoted peer, so the invariant holds if a future
caller bypasses the outer dispatch guard.
Side benefit: narrows a cooldown-poisoning vector previously available
to an attacker injecting stale failure events for an active peer.
Test coverage for the new behavior:
- test_try_peer_addresses_skips_connected_peer
- test_try_peer_addresses_skips_connecting_peer
- test_nostr_traversal_failure_skips_connected_peer (Failed-arm event
injection)
- test_nostr_traversal_established_skips_connected_peer (Established-arm
event injection, mirror of the Failed test)
- test_adopted_traversal_skips_already_connected_peer
(adopt_established_traversal defense-in-depth)
CHANGELOG entry under [Unreleased] / Fixed.
Closes#87
Bring [Unreleased] into sync with all maint commits since v0.3.0:
- Add a Fixed entry for the acl-allowlist test-script poll-assertion
conversion (commit e9dd316) that was previously missing.
- Add the AUR-publish workflow rewrite and new fips-git VCS workflow
(commit 9bf9701) which had no entry, and merge them with the
ci.yml cancel-in-progress block under a single "CI and
release-publish workflows hardened" entry so the three workflow
changes read as one operational theme.
Pure changelog content reshuffle. No code touched.
Add a top-level concurrency block to ci.yml keyed on
(workflow, ref) with cancel-in-progress: true. Pushes (including
force-pushes) to the same ref now retire any in-flight run for
that ref rather than letting the superseded and current-tip runs
both burn runner minutes.
Motivated by a 2026-05-14 force-push experience where two CI runs
ran concurrently against a feature branch — the original push at
c76ec99 continued for ~17 minutes alongside the amended push at
927ef47, despite only the latter being the live tip.
Scope deliberately limited to ci.yml. Tag-triggered release-build
workflows (package-*.yml, aur-publish-*.yml) are untouched — they
operate on per-tag refs that already form distinct concurrency
groups and release artifact builds generally should not be
cancellable by unrelated activity.
Phase 5's per-pair connectivity check ran immediately after the second
rekey cycle, with no settle for routing reconvergence. Under
GitHub-runner CPU contention, post-rekey parent-switches and
coord-cache flushes can take longer than the per-ping 5s timeout for
a small fraction of pairs (1-3 of 20 typically), even though the
rekey mechanism itself completes cleanly. Phase 6 log analysis stays
all-green on these failed runs; the failure is purely connectivity
timing.
Mirror Phase 3's existing 12-second settle pattern: reuse REKEY_SETTLE
and emit the same banner shape. Two-line change. Cost on the success
path is a fixed 12s per suite run.
Convert assert_log_contains from a one-shot grep snapshot into a
bounded poll that retries until the pattern appears or the timeout
elapses (default 15s). Same wait-with-timeout shape as
wait_for_peers_exact above it in the file.
The pre-existing flake on next-branch CI is structural: under XX
handshake, the cross-connection tie-breaker selects which side
reaches its ACL-check point first. When container-a wins the
tie-breaker on the first attempt against c and d, only the
outbound-handshake-context rejection fires immediately, and the
inbound-handshake-context rejection only emits on a later retry
when c or d's msg1 lands while a has no pending outbound. On one
2026-05-14 run the inbound rejection appeared 63ms after the test
had given up. The race window is small but real.
Polling the log instead of one-shot reading absorbs the
millisecond-to-second variance without slowing the success path
(the helper returns as soon as the pattern appears).
Add a per-session signed jitter offset (uniform [-15, +15] seconds)
to the rekey timer triggers in check_rekey (FMP) and check_session_rekey
(FSP). The configured `node.rekey.after_secs` becomes the nominal
interval rather than a floor; mean is preserved. Desynchronizes
both endpoints in symmetric-start meshes so the dual-initiation
race stops occurring rather than being resolved after the fact by
the smaller-NodeAddr tie-breaker.
Per-session storage means each rekey cutover reconstructs the
session and redraws the jitter naturally — successive cycles get
independent offsets, preventing drift back into sync.
Three entries under [Unreleased] for the three commits since the
v0.3.0 release tag:
- Sidecar example: FIPS_UDP_MTU env override (commit 32a3b58)
- CONTRIBUTING.md overhaul + new docs/branching.md (commit 538ce07)
- Rekey-test Phase 1 baseline-convergence headroom 36s -> 60s
(commit 6533276)
The previous CONTRIBUTING.md read like a stock Rust contributing
template that mentioned FIPS. Replace it with an entry-point doc
that gives new contributors the FIPS-specific mental model they
need to make a useful first PR: the FMP/FSP layering and why
mesh-level changes need multi-node testing, the three-branch
release model and how to choose a target branch, structured bug
reporting expectations, and PR submission requirements (scope
discipline, the local-CI ladder, separate requirements for feature
PRs vs bug-fix PRs, squash-merge mechanics).
Add a contributor-facing AI coding assistant policy: use is
welcome, but the contributor must do a thorough manual review and
editorial pass before submission. The agent is a tool; the
contributor is accountable for the submission. Review effort
scales with submission effort -- unreviewed agent output will
receive an agent reply in turn, without human review.
Add docs/branching.md as the long-form companion covering the
release workflow, version conventions, and merge-direction
rationale. The new CONTRIBUTING.md is the day-to-day entry point;
docs/branching.md is the reference.
All cross-references resolve against current HEAD. Previous
stale links to fips-intro.md, fips-wire-formats.md, and
fips-configuration.md are gone; the doc points at the current
docs/design/ layout, docs/getting-started.md,
docs/tutorials/join-the-test-mesh.md, and testing/README.md.
No code changes.
The Phase 1 pre-rekey baseline in `wait_for_full_baseline`
occasionally times out on GitHub-hosted runners with one ping pair
failing to converge inside the BASELINE_CONVERGENCE_TIMEOUT window.
Phases 2–6 always pass cleanly when this happens — the rekey itself
is fine, the mesh just hasn't finished spanning-tree + bloom-filter
convergence by the time Phase 1 starts pinging.
The wait loop returns as soon as all 20 pairs converge, so the cost
on the success path is unchanged (typical local CI returns well
under the old 36s). The bump only adds headroom on the failure
path. Operators previously worked around this with
`gh run rerun --failed`; this aims to retire that workaround for
the IK/XK lines.
Distinct from the next-branch XX rekey dual-init race, which is a
real protocol bug tracked separately.
The sidecar entrypoint hardcoded `udp.mtu: 1472`, the Docker-bridge
IPv4 maximum (1500 MTU - 8 UDP - 20 IPv4 header). Promote it to
`FIPS_UDP_MTU` (defaulting to 1472, preserving behavior) so non-Docker
reuses of the example can override without editing the script. Plumb
the env var through `docker-compose.yml` so a host-level setting
reaches the container, and add a row to the README's env-var table.
Also annotate the static-CI node template with a comment explaining
the same 1472 rationale and the daemon's 1280 default. The template
keeps 1472 as the literal value since the CI suite runs on a Docker
bridge where that's correct.
No behavior change unless the host explicitly sets FIPS_UDP_MTU.
The v0.3.0 stable AUR push silently failed: with updpkgsums: true,
makepkg downloaded fips-<ver>.tar.gz into the AUR working tree, where
it was then staged by the deploy action and rejected by AUR's 488 KiB
max-blob hook.
Fetch the upstream source tarball and compute its b2sum in CI, patch
pkgver and the b2sums SKIP placeholder in PKGBUILD in-place, then
publish with updpkgsums: false so the AUR clone stays metadata-only.
Recompute and patch the fips.sysusers / fips.tmpfiles asset b2sums in
the same step so they stay in sync with the local files; this safety
net was previously provided by updpkgsums.
Add aur-publish-git.yml for the VCS fips-git PKGBUILD, triggered on
master pushes that touch PKGBUILD-git or its companion files plus
workflow_dispatch. pkgver is computed at build time by the PKGBUILD's
pkgver() function, so this workflow is not tied to release tags.
Add a workflow_dispatch tag input on the stable workflow so historical
release tags can be re-published manually, and drop continue-on-error:
true so future regressions surface in CI.
Marker merge to record the maint dev-line as known on master
without taking maint's v0.3.1-dev opening commit (master is on
v0.4.0-dev). Future bug-fix forward-merges from maint to master
land cleanly on top of this base.
Reset maint to v0.3.0 to retire the v0.2.x maintenance window and
open a new tracking branch for v0.3.1-dev bug-fix work.
- Cargo: 0.3.0 → 0.3.1-dev
- CHANGELOG: fresh [Unreleased] block above [0.3.0]
- README: badge v0.3.0 → v0.3.1--dev
Bump Cargo.toml version 0.3.0-dev -> 0.3.0 and resync Cargo.lock,
move the CHANGELOG [Unreleased] block under [0.3.0] - 2026-05-11,
update the README status badge and prose to v0.3.0, and add the
release notes at docs/releases/release-notes-v0.3.0.md with a
mirrored copy at the repo root as RELEASE-NOTES.md.
Uses the 'ours' merge strategy to keep all of master's tree
verbatim with one carve-out: the v0.2.1 release notes archive at
docs/releases/release-notes-v0.2.1.md is retained as a permanent
docs artifact.
Discarded from maint's release-prep commit: Cargo.toml / Cargo.lock
version bumps (master stays on v0.3.0-dev), README v0.2.1 badge and
prose, and the RELEASE-NOTES.md root mirror (master's root mirror
will eventually carry the v0.3.0 release notes when v0.3.0 ships
from this line).
CHANGELOG: the v0.2.1 release section is integrated between
[Unreleased] (v0.3.0-dev work) and the [0.2.0] historical section.
Items that landed on maint during the v0.2.1 cycle are removed
from [Unreleased] so [0.2.1] is the canonical home for those
changes (Linux release artifact and AUR workflows; bloom fill-ratio
validation; seven maint-line bug fixes).
Operator-facing IPs in user-visible configs/docs (examples, tutorials,
packaging, sidecar templates) are now the resolvable hostnames of the
public test fleet (test-us01.fips.network, etc.) so they keep working
without baking specific addresses into examples.
Doc-comment and test fixtures in src/config/transport.rs use RFC 5737
TEST-NET-2 (198.51.100.1) so they cannot accidentally point at a real
host.
Also resyncs the openwrt-ipk fips.yaml with the common reference
(merge from master) and applies the same DNS-name swap there.
Bring the OpenWrt-shipped fips.yaml back into line with
packaging/common/fips.yaml, which had drifted: the OpenWrt copy was
missing the Nostr discovery, BLE, and TCP reference comment blocks, so
operators had no in-config hint that Nostr-mediated discovery existed.
Take common/fips.yaml verbatim and apply just the OpenWrt-specific
active overrides:
- ethernet: uncomment with wan/wwan/lan defaults (eth0, phy0-sta0,
br-lan)
- gateway: uncomment with lan_interface=br-lan and dns.listen on
[::1]:5353 (matches the dnsmasq forwarder the init script wires up)
Identity stays ephemeral by default (no persistent override), matching
common.
Bump Cargo.toml version 0.2.1-dev -> 0.2.1 and resync Cargo.lock,
move the CHANGELOG [Unreleased] block under [0.2.1] - 2026-05-11,
update the README status badge and prose to v0.2.1, and add the
release notes at docs/releases/release-notes-v0.2.1.md with a
mirrored copy at the repo root as RELEASE-NOTES.md.
The host (217.77.8.91:443) was renamed to test-us01 some time ago
(canonical name shared with packaging/common/hosts and the docs);
the test scaffold's alias labels and narrative comments hadn't been
updated. Pure naming cleanup, no behavior change. The continued
dependency on the live external host is tracked separately.
Walk through reviewer feedback on the Nostr-discovery docs and
land 18 items.
Bulk patterns:
- `external_addr` / `public: true` semantics consistently
misdescribed. The advert path is gated on `cfg.is_public()`;
inside that branch the daemon picks an address by precedence
(`external_addr`, non-wildcard `bind_addr`, STUN). The docs
treated `public: true` and `external_addr` as alternatives when
they are stacked: `public: true` is the master switch and
`external_addr` populates the address inside it. Reconciled
across `enable-nostr-discovery.md` and `advertise-your-node.md`:
add `public: true` to the `external_addr` examples; replace
"STUN as a logging cross-check" with "STUN is skipped entirely";
fix "neither flag is needed" for direct public bind (both flags
still required); make the publish-tutorial Step 3 conditional
on the chosen Step 2 path (STUN runs only on the `public: true`
path); rewrite the troubleshooting "wrong public IP advertised"
bullet with two coherent fixes.
- `udp:nat` overpromised as a symmetric-NAT solution. Symmetric
NAT on either side typically defeats the punch. Reframe
`udp:nat` as best-effort hole-punching for nodes without a
directly reachable UDP endpoint in the how-to, the publish
tutorial (intro, callout, section heading rewrite from "If
you're behind symmetric NAT" to "If your direct UDP advert
isn't reachable"), the consume tutorial's "What's next"
pointer, and `tutorials/README.md`. Promote reachability over
named NAT classes: STUN can confirm the public IP but not that
the listener-port mapping is open.
- YAML "silently ignores unknown keys" is wrong. Config parser
rejects unknown fields via `serde(deny_unknown_fields)` on the
per-section structs; misspelled fields refuse the daemon's
start with a parse-error line in the journal. Fixed in the
publish tutorial's troubleshooting and the open-discovery
tutorial's `policy` typo bullet.
Mechanical fixes:
- Repoint stale anchors. `getting-started.md` and
`configuration.md` linked to `#installation` / `#inspect` on
the README; the README has no such headings. Repoint to
`#quick-start` and `cli-fipsctl.md`. Two stale anchors in the
publish tutorial pointing at non-existent sub-scenarios in the
how-to (`#sub-scenario-2c-...`,
`#sub-scenario-2b-tor-onion-node`) repointed to the correct
anchors.
- Drop the `fipsctl show status` claim from the open-discovery
troubleshooting bullet (`show_status` doesn't include
`discovery.nostr.policy`). Replace with daemon startup logs.
- Fix the `advertise: false` parenthetical in the consume-only
tutorial (`default_advertise()` returns `true`; we set `false`
explicitly for the consume-only path).
- Drop the "supplies a relay list" overstatement in two
activation paragraphs (the how-to and the design doc). Default
relay / STUN-server lists ship in the config; both are
optional overrides.
- Add the missing `transports.udp.public` entry to the
open-discovery tutorial's prerequisites checklist. Tutorial
users coming out of advertise-your-node could be on either the
direct-UDP (`public: true`) or `udp:nat` (`public: false`)
path; list both.
Files: docs/getting-started.md, docs/reference/configuration.md,
docs/how-to/enable-nostr-discovery.md, docs/tutorials/README.md,
docs/tutorials/advertise-your-node.md,
docs/tutorials/resolve-peers-via-nostr.md,
docs/tutorials/open-discovery.md,
docs/design/fips-nostr-discovery.md.
Four short prose corrections folded together to align operator-facing
docs with current v0.3.0 reality:
- README Rust prerequisite reconciled with the toolchain pin (1.94.1,
was 1.85+).
- CONTRIBUTING.md bumps the Rust prerequisite and adds the squash-merge
policy note.
- examples/sidecar-nostr-relay/Dockerfile drops stale --features tui
and the paired --no-default-features; the tui/ble/gateway cargo
features were replaced by platform cfg gates in cbc7809.
- README Features list adds two v0.3.0-visible items: the mesh-
interface security baseline (fips.nft conffile, fips.d/ drop-in,
opt-in fips-firewall.service across all packaging formats) and the
fipsctl stats time-series queries plus fipstop inline sparkline
dashboards.
Status badge bump (v0.3.0--dev to v0.3.0) is deferred to tag time per
the release-prep checklist.
The generic systemd install tarball is the catch-all install path
for systemd Linux distros that don't have a per-format package
(Fedora, RHEL/CentOS, openSUSE, Alpine, etc.). It had drifted
behind the .deb and AUR packages and was missing fips-gateway, the
mesh-interface firewall baseline, and the multi-backend DNS helper
in the shipped tarball. Bring it to parity:
- New `packaging/systemd/fips-gateway.service` (clone of the .deb
unit; ExecStart pointed at `/usr/local/bin/fips-gateway`). Not
enabled at install time; operator opt-in.
- New `packaging/systemd/fips-firewall.service` (clone of the .deb
unit; nft path unchanged at `/usr/sbin/nft`). Not enabled at
install time; operator opt-in.
- `build-tarball.sh` now bundles the `fips-gateway` binary, the two
new units, the `fips.nft` baseline conffile, and the
`fips-dns-setup` / `fips-dns-teardown` multi-backend helpers from
`packaging/common/`.
- `install.sh` now installs `fips-gateway` to `/usr/local/bin/`,
installs both new units to `/etc/systemd/system/` (without
enabling them), preserves `/etc/fips/fips.nft` on upgrade like
`fips.yaml`, and creates the `/etc/fips/fips.d/` operator drop-in
directory. Post-install messaging mentions both opt-in services.
- `uninstall.sh` stops and disables the optional services in
dependency order (firewall, gateway, dns, daemon), removes the
new unit files, and removes the gateway binary. `--purge` already
handles `/etc/fips/` removal which covers `fips.nft` and
`fips.d/`.
- `README.install.md` documents all of the above: expanded
"What Gets Installed" table, new sections covering the firewall
baseline and the LAN gateway, refreshed DNS section reflecting
the multi-backend setup helper (systemd dns-delegate /
systemd-resolved drop-in / per-link resolvectl / dnsmasq /
NetworkManager-dnsmasq), and updated Service Management.
Also fixes a latent packaging bug: `install.sh` previously
referenced `${SCRIPT_DIR}/../common/fips-dns-setup`, a path that
exists only in the source-repo layout and not in the extracted
tarball. The script now resolves the helper from the staging
directory first (the tarball case), falling back to the source-repo
relative path. Bug latent since the multi-backend DNS helpers
landed.
CHANGELOG `[Unreleased]` documents the parity bump under Changed
and the path-resolution fix under Fixed.
Closes the longest-standing parity gap for non-Debian / non-Arch
systemd Linux distros installing from the release-distribution
tarball.
Three changes folded together close the AUR-side parity gap with the
.deb packaging:
- PKGBUILD now ships fips.nft baseline and fips-firewall.service, with
fips.nft marked as backup so operator edits survive upgrades.
- PKGBUILD-git mirrors the release PKGBUILD: installs fips-gateway
(was missing entirely), ships fips.nft and fips-firewall.service,
and tracks the same backup() set.
- packaging/aur/README.md documents the gateway, firewall service,
and nft baseline that ship as of this revision.
AUR users now receive the same artifact set as .deb users.
The Noise-session AEAD swap landed as 5cda4a9 + 9b1016f without a
companion CHANGELOG entry; add it under [Unreleased] § Changed
ahead of the rest of the perf-win cluster from PR #81 since it's
the most operator-visible single change of that class.
Investigated the mipsel-unknown-linux-musl build of this branch on a
Linux/x86_64 host. ring 0.17, portable-atomic, and the fips codebase
itself all compile cleanly for that target — the AtomicU64 portability
work is already done. The actual blocker is in the nostr-relay-pool 0.44
transitive dep, which uses std::sync::atomic::AtomicU64 directly in
src/relay/{stats,ping,flags}.rs. Verified fixable with a 6-line
portable-atomic patch via a [patch.crates-io] shim during local testing.
Updating the comment so the next person looking at this matrix has the
right starting point.
No functional change.
The chacha20 crate (RustCrypto) ships SSE2 + soft backends only — on
aarch64 (Apple Silicon, ARM Linux servers, Docker on M-series Macs) it
falls through to a portable software impl at ~600–800 MB/s/core. ring
0.17 wraps BoringSSL's hand-tuned ChaCha20-Poly1305, which dispatches
to NEON on aarch64 and AVX2/AVX-512 on x86_64 — typically 3-5 GB/s/core
on the same hardware.
Same wire format. ChaCha20-Poly1305 is byte-deterministic for a given
(key, nonce, plaintext, aad), so any correct AEAD implementation
produces identical ciphertext. The full noise test suite covers this
implicitly: IK and XK roundtrip handshakes, replay window correctness,
multi-message nonce sequencing, and 100-message stress all pass at
1129/1129 (the lib's full `cargo test` count) — these only succeed if
ring's output matches what the receiver's existing replay-window
decrypt path expects.
Implementation notes:
* `LessSafeKey` (and `UnboundKey`) deliberately do not implement
Clone for safety. `CipherState`'s manual Clone impl rebuilds it
from the retained 32-byte key — cheap for ChaCha20-Poly1305 since
construction is essentially a key copy + a constant-time check.
* The keyed AEAD is now cached in `CipherState.cipher` instead of
being re-derived per packet. This was already a perf win for the
chacha20poly1305 backend (`new_from_slice` per packet was hot in
profiles); for ring it's a bigger win because `LessSafeKey`
construction also derives the Poly1305 key.
* Public `Vec<u8>`-returning API preserved. New module-private
`seal`/`open` helpers wrap ring's `seal_in_place_append_tag` /
`open_in_place` so the per-packet allocation pattern is local to
one place.
* `EndToEndState::Established` triggers `clippy::large_enum_variant`
after the swap (`NoiseSession` grew from ~600 to ~1.5 KB because
ring precomputes the Poly1305 key state at construction). That
precomputation is the win — boxing the variant would re-add an
indirection per packet and work against it. `#[allow]`'d at the
enum decl with a justifying comment.
ring is widely deployed (rustls, hyper-rustls, AWS SDK, …) and a
pure-Rust crate (uses BoringSSL's asm via a vendored build). It
introduces no new C toolchain requirements that aren't already there
for any rustls user.
Bench data from a downstream consumer of this crate (Docker e2e,
DURATION=10, identical hardware before/after, aarch64 Linux on
Apple Silicon):
2-node direct (A↔B):
TCP 1-stream 437 → 1097 Mbps (2.51×)
TCP 4-stream 439 → 1109 Mbps (2.53×)
TCP 8-stream 445 → 1069 Mbps (2.40×)
UDP @1000 Mbit 599/40% loss → 1000 Mbps lossless
ping under load ~0.6 ms (unchanged)
3-node forced transit (A → C → B):
TCP 1-stream 438 → 1019 Mbps (2.33×)
TCP 4-stream 421 → 982 Mbps (2.33×)
TCP 8-stream 443 → 1031 Mbps (2.33×)
UDP @1000 Mbit 475/52% loss → 1000 Mbps lossless
ping under load 7.68 ms / 215 ms max → 0.72 ms / 3.6 ms max
The relay-path lift is the cleanest tell on the bottleneck: the
transit node was crypto-bound (single-threaded soft chacha couldn't
keep up with offered rate), so the queue accumulated under load. With
NEON the relay isn't crypto-bound and the queue stops accumulating —
the 215ms ping-tail collapses to 3.6ms.
Freshly-restarted nodes with policy: open silently lost the historical
event replay that relays send in response to subscribe(). The
broadcast::Receiver was created INSIDE spawn_notify_loop, which the
tokio runtime starts at some indeterminate point after subscribe()
returns. tokio's broadcast channel only delivers messages sent after
the receiver is created; messages dispatched in the gap between
subscribe() issuing the REQ and the spawned task calling
client.notifications() were dropped by external_notification_sender.send
returning Err(SendError) with no subscribers attached.
Symptom on a node with policy: open: non-configured peers were not
discovered until they next re-published their advert (default
advert_refresh_secs = 1800s = 30 min). Configured peers were unaffected
because fetch_advert (relay-fetch path) caches them at startup-sweep
time. The bug has been latent since 34e00b9 added Nostr discovery —
relay-fetch covered the common case for configured-peer setups.
Fix: create the broadcast::Receiver in start() before subscribe() and
pass it into spawn_notify_loop. The receiver now exists when the REQ
replay arrives, so historical events flow through the cache path.
Also handle broadcast::error::RecvError::Lagged separately from
::Closed. The previous `while let Ok(...) = recv().await` exited the
loop on any Err, so a single lag event would silently kill the entire
subscription consumer with no recovery. Lagged now logs a warn (with
the skipped count) and continues; only Closed exits the loop.
Add two info-level log lines for in-field observability of the loop's
liveness. "nostr notify loop entered" fires once at task start; "nostr
notify loop received first event" fires once after the first
successful recv() with elapsed_ms since loop entry. Together these
turn the previous silent-failure shape (zero advert: peer cached
log lines indistinguishable between dead loop and idle channel) into
an immediately greppable startup signal — operators can confirm the
loop is alive and see how long it took to receive its first event,
catching any future regression in the subscription codepath in
seconds rather than waiting one advert_refresh_secs interval.
No public API change; the test fixture (NostrDiscovery::new_for_test)
does not call spawn_notify_loop and is unaffected.
Three Changed entries for the rx-path performance work
(Linux UDP recvmmsg batched receive, run_rx_loop drain batching, and
eager pubkey_full precompute on PeerIdentity construction) and five
Fixed entries: adopted NAT-traversed UDP transports inheriting the
primary listener's MTU and buffer config, TreeAnnounce ancestry on
self-root transitions, unconditional overlay-advert refetch before
each retry, stale overlay-advert eviction on NoTransportForType,
and scheduled retry on startup peer-init failure.
Pure CHANGELOG addition (+117 lines, no edits to existing entries).
Bullets are wrapped at 80 columns and attribute external
contributions to the originating PR and author.
The advert cache inside fetch_advert is read-only on hit — once a peer's
overlay advert is cached, every subsequent lookup returns the same
endpoints regardless of whether they still work. So when a peer rebinds
its NAT (or its STUN-discovered port flaps), connection retries to that
peer dial the same dead address forever, even with exponential backoff
firing at the right cadence.
Observed in deployment: macOS daemon's view of a Linux peer would
"regress" — peer marked rch=False after a brief link-dead window, then
hours of "Retry connection initiation failed: no operational transport
for any of <npub>'s addresses" with no recovery. Manual pause+resume of
the daemon (which restarts the FIPS endpoint and forces fresh advert
fetches) was the only way out.
When initiate_peer_connection / a retry tick returns
NodeError::NoTransportForType, fire-and-forget refetch_advert_for_stale_check
on the peer's npub. This re-fetches kind 37195 from advert_relays; if
the relay has a newer advert it replaces the cached entry, if it has
nothing it evicts the cached entry. Either way the next retry tick goes
to fresh data instead of looping on the same dead endpoint.
Mirrors the existing stale-advert sweep that runs from the
BootstrapEvent::Failed (NAT-traversal-streak) path, but covers the
direct-UDP-retry path which never crosses that streak threshold.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
When initiate_peer_connections() runs at boot, address resolution can
fail for an entire peer (no operational transport for the configured
transport types, all addresses unreachable, NAT rebind invalidated cached
endpoints, etc.). Before this change the failure was logged and silently
forgotten — the peer entry stayed in a dead state forever, accepting
incoming pings but unable to answer them, until the daemon was manually
restarted.
The retry plumbing (schedule_retry / process_pending_retries with
exponential backoff) already exists and is wired into the post-handshake
failure paths (BootstrapEvent::Failed, MMP dead-link timeout, handshake
timeout). The startup loop just wasn't calling it. Mirror the
BootstrapEvent::Failed path: on a startup peer-init error, parse the
peer's npub and call schedule_retry so the peer recovers without
operator intervention.
Includes a regression test that asserts retry_pending is populated when
initiate_peer_connections() fails for a peer with no operational
transport.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
The previous fix (6ebca3e) only refetched the advert when retry returned
NodeError::NoTransportForType (cache returned no addresses at all). But
the much more common stale-cache failure mode is: cache returns an
endpoint that LOOKS valid (the address it had last week, before the
peer's NAT rebound), the dial succeeds at the IP layer, the handshake
times out, MMP fires, schedule_reconnect adds the entry back to
retry_pending, next retry hits the same cached endpoint, dials it
again, times out again. Loop forever — no NoTransportForType ever
fires because the cache has data, just dead data.
Move refetch_advert_for_stale_check to before each retry attempt
unconditionally. Cheap (one Filter query against advert_relays with
a 2s timeout, bounded by the retry backoff cadence), and replaces the
cache only if the relay has a newer advert or evicts if the relay has
nothing. Keeps the retry loop pinned to relay ground truth instead of
whatever the cache happened to learn at startup.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
When a node was the smallest-NodeAddr peer it could see (no smaller
neighbor available as a parent), the spanning-tree state was promoting
it to root. But the ancestry it advertised on the next TreeAnnounce
still referenced its previous parent's path, so receiving peers
rejected the announce with `invalid ancestry: advertised root X is
not the minimum path entry Y`, blocking mesh transit on any path that
needed to traverse this node.
Detect the self-root transition explicitly in `TreeState::become_root`
and rebuild the advertised ancestry to start from self. Also surface
the same path through the MMP receive handler so a stale ancestry
inherited across reconnect is corrected eagerly rather than waiting
for the next observation tick.
Adds 80 unit tests in `tree::tests` covering self-root transitions,
mid-chain ancestor disappearance, and ancestry validation against the
new root, plus a regression in `node::tests::spanning_tree` for a
3-node chain where the middle node's only parent (the smallest-addr
peer) goes away — previously it would advertise an ancestry rejected
by both endpoints; now it self-roots cleanly.
The run_rx_loop's `tokio::select!` was costing one full scheduler hop
+ futex per inbound packet and per outbound TUN packet. Under
sustained load that capped throughput at one event per scheduler
quantum — independent of CPU (which sat near-idle) because every
iteration parked the worker, woke it via futex, processed one event,
then parked again.
After the await on `packet_rx.recv()` / `tun_outbound_rx.recv()`
fires, drain up to 256 additional ready items via `try_recv()` in a
tight inner loop before yielding back to `select!`. `biased` ordering
gives the data-plane branches priority over tick / control / DNS
under sustained load.
The 256 cap is empirically tuned to keep the worker on a busy stream
between yield points (a contiguous burst of ~256 MTU-sized packets
≈ 400 KB of contiguous traffic) while still bounding the inner loop
so a flood on one branch can't starve the periodic tick or control
socket. Lower caps (64) left perf on the table; higher caps (1024+)
delayed tick handling visibly under stress.
Pairs with the recvmmsg(2) change in the previous commit: the kernel
UDP queue now hands packets to `packet_rx` in 32-batches, and the
rx_loop drains them without a per-packet scheduler hop.
The UDP recv loop drained the kernel queue one packet per recvmsg(2).
Each call paid full per-syscall + per-task-wakeup overhead (~50us avg
including a futex-based scheduler hop), so under sustained load the
loop ran at one rx event per scheduler quantum — the dominant cap on
inbound packet rate.
On Linux, switch the steady-state path to recvmmsg(2) with a 32-packet
batch. A single readable() wakeup drains up to 32 datagrams in one
syscall before yielding back to the reactor. Stack-allocated mmsghdr
arrays sized to a module-level `BATCH_SIZE` constant.
`SO_RXQ_OVFL` is sampled once per batch off the cmsg chain of `msgs[0]`
and plumbed through `AsyncUdpSocket::recv_batch` as `(count, drops)`.
The counter is socket-wide and monotonic, so a single sample per batch
gives the 1Hz `sample_transport_congestion()` detector ample fresh
values under load (one batch = up to 32 datagrams). Cost is one
stack-allocated CMSG_SPACE(4) buffer + one CMSG_FIRSTHDR walk per
batch syscall.
macOS / Windows fall through to the per-packet recv_from loop —
recvmmsg is Linux-specific and the per-packet API is fast enough on
those platforms for now (recvmsg_x for Darwin can be added later).
The slice-array build also drops the `MaybeUninit::uninit().assume_init()`
+ `transmute` pair for `std::array::from_fn` over a single shared
`backing.iter_mut()` — same disjoint mutable borrows, no `unsafe`.
`PeerIdentity::pubkey_full()` falls through to
`self.pubkey.public_key(Parity::Even)` whenever the parity-aware full
key wasn't passed at construction (i.e. for every peer constructed
from an npub or x-only key). Underneath, that runs a secp256k1 EC
point parse — `fe_sqrt` + `fe_mul` + `ge_set_xo_var` — which is ~6%
of per-packet CPU on the bulk-data send path for a value that never
changes after construction.
Compute it eagerly. The same EC point parse already runs at
construction inside `NodeAddr::from_pubkey`, so the cost is paid once
where it would be paid anyway.
Johnathan Corgan
Martti MalmiandJohnathan Corgan
ArjenandJohnathan Corgan
