35 Commits
Author SHA1 Message Date
Arjen 081855e8e2 fix(ble): reframe inbound L2CAP stream so packets survive non-SeqPacket backends
BLE delivery was unreliable on every backend except BlueZ. The receive
path assumed one `recv()` returned exactly one whole FIPS packet, which
only holds for BlueZ's SOCK_SEQPACKET (boundary-preserving). Android's
BluetoothSocket input stream and macOS CoreBluetooth are byte-stream
oriented: a read can return a fragment of a packet or several packets
coalesced. Under the old loop a fragment was shipped up as a runt packet
(rejected by FMP/Noise) and a coalesced tail was silently truncated and
dropped — so packets were lost and the transport thrashed.

Recover packet boundaries from the byte stream instead of trusting the
OS to preserve them. FIPS packets are self-delimiting via the 4-byte FMP
common prefix, so this reuses the exact length-prefixed framer TCP already
uses (`tcp::stream::read_fmp_packet`) — kept transport-agnostic for this
reason. A new `BleStreamRead` adapter turns the datagram-shaped `BleStream`
into the `AsyncRead` that framer expects, buffering leftover bytes across
reads. The recv future owns its scratch buffer and returns an owned Vec so
it is `'static` and storable across `poll_read` calls.

One reader is threaded through the pubkey exchange and the receive loop per
connection, so bytes a peer coalesces after the 33-byte pubkey stay buffered
rather than being lost at the handoff. The pubkey exchange now reads via
`read_exact` (reassembles a fragmented pubkey) instead of a brittle single
`recv()` with an exact-length check.

On BlueZ this is a transparent pass-through (one recv already equals one
packet); on stream backends it reassembles. Either way the layer above sees
one whole packet per read, identically on every platform.

Also bump the Android outbound SEND_QUEUE_CAP from 8 to 32. Swept against
the peer speedtest: 8 starved the radio's connection events (~half
throughput), 64 bufferbloated TCP, 32 was best (~200/500 kbps up/down). The
sweep was noisy and non-monotonic — run-to-run BLE variance (RF, 2M PHY /
connection-priority grants) rivals the knob's effect — so 32 is the
best-observed value pending re-validation with PHY/interval instrumentation,
not a proven optimum.
2026-07-14 13:53:57 +02:00
Arjen 3d81f4b1f5 style(ble): rustfmt android_io and psm
Import ordering and multi-line wrapping rustfmt would enforce; android_io
is cfg(target_os = "android") so it never compiled on the host CI matrix
and the drift went unnoticed until the mobile build was wired up.
2026-07-14 13:53:57 +02:00
Arjen 0450b11d5f docs: list Android as a supported platform
Android lands as an embedded library: the host app owns the TUN (e.g. an
Android VpnService) and FIPS does no system-TUN ops. Document where the
docs enumerate platforms as a set.

- README transport matrix: add an Android column (UDP/TCP/BLE supported;
  Ethernet has no raw sockets; Tor/Nym need an external proxy not run on
  Android). Reword the intro to distinguish standalone-daemon hosts from
  the embedded-library Android target.
- transport-layer status: BLE is now implemented on Linux/glibc and
  Android (Linux via BlueZ, Android via the embedder radio bridge).
- set-up-bluetooth-peer how-to: add Android to the BLE platform table.
2026-07-14 13:53:57 +02:00
Arjen 4afa27f056 perf(ble): shallow, backpressured outbound queue to fix bufferbloat
The per-stream outbound byte queue was 256 deep (~384 KB) on a link whose
bandwidth-delay product is ~1 packet. FSP/MMP filled it, RTT ballooned to several
seconds, and TCP above couldn't ramp. A shallow tail-drop queue is no better — it
sheds packets and collapses TCP throughput.

Give the outbound queue its own shallow cap (SEND_QUEUE_CAP=8) and make
BleStream::send backpressure — await a free slot rather than try_send-dropping —
so flow control propagates up through FSP/MMP to TCP. On-device this cut RTT from
~5s to ~1.2s with throughput preserved and 0% loss.
2026-07-14 13:53:57 +02:00
Arjen 0aa50d245c fix(ble): dial the last-learned PSM on a per-peer lookup miss
Per-peer PSM discovery keys the learned PSM by BLE address, but RPAs rotate
between the scan that learns a peer's PSM and the dial, so resolve() missed and
fell back to the legacy default 0x0085 — which no node listens on. Every outbound
L2CAP connect was rejected and the link ran inbound-only at a fraction of its rate.

A node advertises exactly one OS-assigned listener PSM, so on an exact-address
miss, dial the most-recently-learned PSM instead of the fixed default; a wrong
guess only costs a dial-retry.
2026-07-14 13:53:57 +02:00
Arjen abb0701048 feat(node): app-owned TUN seam — embedder owns the fd, FIPS uses channels
Node::enable_app_owned_tun() lets an embedder that owns the TUN fd (e.g. an
Android VpnService) exchange IPv6 packet bytes with FIPS over channels instead of
FIPS creating a system TUN device. It returns (app_outbound_tx, app_inbound_rx):
the embedder pushes packets read from its fd into the outbound sender (app ->
mesh) and pulls packets destined for its fd from the inbound receiver (mesh ->
app). Called after Node::new and before start(), mirroring control_read_handle().

start() now gates TunDevice::create on tun_tx being unset, so when the app-owned
channels are pre-installed it skips system-TUN creation and does no system-TUN
ops. The inbound IPv6-shim delivery already writes to tun_tx, and run_rx_loop
already drains tun_outbound_rx into handle_tun_outbound, so both directions reuse
the existing wiring.

Packets entering via app_outbound_tx bypass the system-TUN reader's
handle_tun_packet, so the embedder must push only fd::/8-destined packets (FIPS no
longer filters the destination) and clamp TCP MSS on outbound SYNs; the rustdoc
and the IPv6-adapter design doc spell this out.

Tests: app_owned_tun_seam_wires_channels covers the channel round-trip and the
Active state; start_skips_system_tun_when_app_owned runs start() and asserts no
named system device is created (tun_name stays unset).
2026-07-14 13:53:57 +02:00
Arjen 5c02d33ad8 feat(ble): record scan adverts (PSM + RSSI) for the developer UI
deliver_scan now carries RSSI as well, and records the latest advert per address
(PSM + RSSI) into a map exposed via advert_views() -> Vec<AdvertView>. Cleared
each scan cycle (addresses rotate with MAC privacy). This is the radio-level
view, distinct from the node's mesh-level peer table.
2026-07-14 13:53:57 +02:00
Arjen 5083a09223 fix(ble): safe AndroidBleBridge teardown + replaceable injection
- next_send: clone the per-channel receiver out before blocking (no longer holds
  the channels lock across recv_timeout), and treat a dropped sender
  (Disconnected) as closed so the Kotlin writer thread exits when the stream is
  gone.
- channel_open: also reports a closed (dropped-stream) channel, not just a
  removed one — so writers stop promptly.
- set_android_ble_bridge: replaceable (Mutex<Option>) so a stop→start cycle
  re-injects a fresh bridge for the rebuilt node.
2026-07-14 13:53:57 +02:00
Arjen 16ead248d0 feat(ble): public peer-view read API for embedders running run_rx_loop
Expose ControlReadHandle (was pub(crate)) + make Node::control_read_handle()
public, and add ControlReadHandle::peer_views() -> Vec<PeerView>
{ node_addr_hex, npub, connected }, read lock-free from the tick-published stats
snapshot (peer_meta). This lets an embedder run run_rx_loop on a background task
(which exclusively borrows &mut Node) and still poll live peer state from a
cloned handle — no &Node access needed. Used by the Myco app's developer UI.
2026-07-14 13:53:57 +02:00
Arjen 9e59659073 feat(ble): AndroidBleBridge::channel_open — let next_send tell closed from timeout 2026-07-14 13:53:57 +02:00
Arjen 1611282047 feat(ble): Android backend — BleIo over a Kotlin-radio byte-bridge
The Android BLE radio lives in Kotlin, so AndroidIo/Stream/Acceptor/Scanner
implement BleIo by delegating to AndroidBleBridge — the channel machinery shared
with the JNI layer in the embedder. The AndroidRadio trait is the object-safe
command surface Kotlin implements (listen/connect/advertise/scan/close).

- Inbound bytes/events are pushed non-blocking into tokio channels
  (deliver_recv/inbound/scan/connect_result); outbound bytes are pulled by a
  per-channel Kotlin writer thread via next_send. BleStream::send never calls
  JNI — the byte hot path is pure channel push.
- Per-peer PSM: connect() substitutes the learned PSM; advertising emits the
  OS-assigned listener PSM (16-bit LE service-data).
- Wired as DefaultBleTransport on Android, plus a node construction arm that
  reads the embedder-injected bridge (set_android_ble_bridge).

Pure Rust (no JNI here — that's in myco-core), so the channel logic unit-tests
on the host with a mock radio. Cross-compiles clean for arm64-android.
2026-07-14 13:53:57 +02:00
Arjen 786758e8c3 feat(ble): per-peer PSM discovery core + compile BLE on macOS/Android
Foundation for "BLE v2" universal per-peer PSM discovery, shared by every
BleIo backend.

- ble/psm.rs: a 16-bit little-endian service-data PSM codec + a short-lived
  BleAddr->PSM map (learn/lookup/resolve/clear), with unit tests. Every node
  advertises its OS-assigned listener PSM and every dialer reads a peer's
  advertised PSM before connect(), replacing the fixed DEFAULT_PSM (0x0085)
  assumption that only BlueZ could satisfy (Android/macOS get OS-assigned
  listener PSMs).
- Un-gate the BLE transport module from linux-only to a new `ble_available`
  cfg (linux/macos/android). The pool/discovery/psm logic and the generic
  BleTransport<I> are platform-agnostic; only the concrete BleIo backend is
  platform-specific (BluerIo on linux-glibc, else the MockBleIo fallback).
  The macOS BluestIo and Android AndroidIo backends, and the per-backend
  advertise/scan/connect wiring of the PSM map, land in follow-ups.
2026-07-14 13:53:57 +02:00
Arjen 0cf035a0e1 feat(mobile): gate desktop transports/TUN by target_os, not features
A plain `cargo build` now compiles correctly for every target with no
flags: the compiler selects platform code by `target_os`, enabling what
each platform supports and disabling what it doesn't. Android no longer
needs `--no-default-features`.

- ethernet: gated `any(target_os = "linux", target_os = "macos")` (raw
  AF_PACKET / BPF). Android is `target_os = "android"` — not "linux" — so
  the raw-socket transport self-excludes there, as it already did on Windows.
- system-tun: real platform ops gated per `target_os = "linux"`/`"macos"`;
  Windows keeps its own path. Android gets a no-op stub — the TUN is
  app-owned by the embedder (e.g. an Android VpnService).
- dns: ipi6_ifindex is i32 on Android, u32 on macOS — cast.
- node: gate the unix-only ESTABLISHED_HEADER_SIZE import so the Windows
  build is warning-clean.

No Cargo features are introduced; desktop builds are unchanged.
2026-07-14 13:53:57 +02:00
Johnathan Corgan 1dbfefc9d0 Merge branch 'maint' 2026-07-02 22:03:41 +00:00
Johnathan Corgan 1d277e67c7 fix(tree): invalidate coord cache on parent-lost via peer removal
handle_peer_removal_tree_cleanup reparents or self-roots the node when
its parent link drops, but omitted the coordinate-cache invalidation that
every other position-change path performs. Cached entries for downstream
destinations kept the node's now-stale coordinate prefix, and because
find_next_hop refreshes the TTL on every routing access, an actively
routed stale entry never self-expired — corrected only by a fresh insert.

Mirror the loop-detection branch: inside the parent-loss changed block,
invalidate both classes — invalidate_via_node (reparent) and
invalidate_other_roots (self-root). Add regression tests for a parent-link
removal that reparents (via-node entry dropped, same-root sibling
preserved) and one that self-roots (both via-node and stale old-root
entries dropped).
2026-07-02 22:03:36 +00:00
Johnathan Corgan 793f844448 Merge branch 'maint' 2026-06-29 16:47:43 +00:00
Johnathan Corgan 2491091868 docs(testing): document ci-local.sh run isolation, preemption, and cleanup
Add a "Running CI locally" section to testing/README.md covering the
ci-local.sh orchestrator: the per-run FIPS_CI_RUN_ID override and how it
scopes compose projects, image tags, and per-chaos-child subnets so
simultaneous runs on one host never collide; the cancellation exit codes
(130/143 vs 0/1) and how a preempting worker interprets them; and the
label-based cleanup via --reap / ci-cleanup.sh.
2026-06-29 16:47:37 +00:00
Johnathan Corgan 243bd7985a Merge branch 'maint' 2026-06-29 14:23:57 +00:00
Johnathan Corgan 965de26239 testing: make ci-local.sh preemption-safe with per-run docker isolation
A CI worker may preempt an in-flight ci-local.sh run (SIGTERM, then SIGKILL
after a grace period) to restart on a newer commit. For that kill to be safe,
the script must clean up after itself and never let a dying run collide with
its restart. It previously had no signal handling, shared the default compose
project name across runs, and tore down each suite only at the suite end.

- Derive a per-run id (honoring FIPS_CI_RUN_ID, else short-sha+random) and
  namespace every docker resource to it: a fipsci_<run>_<suite> compose project
  per suite and per parallel chaos child, and per-run image tags
  (fips-test:<run>, fips-test-app:<run>) retagged to :latest only after both
  builds succeed so :latest never points at a half-built image.
- Install a bounded, idempotent teardown trap on SIGTERM/SIGINT (+ EXIT): reap
  parallel chaos children, then force-remove this run's docker resources via
  the new ci-cleanup.sh, wrapped in timeout so a stuck down cannot wedge it.
- Exit 143 (SIGTERM) / 130 (SIGINT), distinct from 0 (pass) / 1 (failed), so a
  preempting worker tells a cancelled run from a real failure.
- Add ci-cleanup.sh (also ci-local.sh --reap): force-removes leftover CI
  resources by the com.corganlabs.fips-ci=1 label and the fipsci_ project
  prefix, robust to however a prior run died.
- Label every per-suite docker resource so the label sweep reaps it after a
  SIGKILL regardless of network name: direct docker run/network resources, the
  sidecar compose services, and every per-suite compose network (acl-allowlist,
  boringtun, firewall, nat, static, both tor suites, and the chaos generator
  template). Parametrize the static/sidecar compose image refs so the per-run
  tags are honored.
- Give each parallel chaos child a unique /24 from 10.30.x (a new --subnet
  override on the sim CLI, assigned per-child in ci-local.sh) so parallel
  children never collide on a shared docker subnet, and a chaos net can never
  span a fixed-subnet suite (sidecar/static in 172.20.x). 10.30.x sits outside
  docker's default-address-pool range, so an auto-assigned net cannot land on
  it either; node IPs derive from the subnet, so no scenario config changes.
2026-06-29 14:23:46 +00:00
Johnathan Corgan ab915d0479 testing: convert remaining fixed-sleep settles to progress-aware polling
Replace the fixed post-first-rekey and post-second-rekey settle sleeps in
the rekey integration test, and the fixed-deadline baseline wait in the
interop test, with the deterministic wait_until_connected progress-aware
polling helper already used for the initial baseline convergence.

Each converted site fails fast with structured diagnostics when the mesh
is stuck and extends its deadline only while pairwise reachability is
still climbing, removing the wall-clock settle windows that produced
intermittent connectivity failures after a rekey.
2026-06-29 00:40:06 +00:00
Johnathan Corgan 30c5808e09 Merge maint into master after the v0.4.0 rollover
maint carries only its 0.4.1-dev version bump; master keeps its own
0.5.0-dev development version. Recorded as a linkage merge so later
forward-merges of real fixes land cleanly.
2026-06-27 18:35:41 +00:00
Johnathan Corgan 8f30924fc7 Open 0.4.1-dev cycle on maint after v0.4.0 release
Fast-forward maint to the v0.4.0 release and open the 0.4.x patch
series: bump the version to 0.4.1-dev and update the status badge.
2026-06-27 18:29:34 +00:00
Johnathan Corgan 3c9a629ad4 Open 0.5.0-dev cycle on master after v0.4.0 release
Bump the version to 0.5.0-dev and update the status badge.
2026-06-27 18:28:52 +00:00
Johnathan Corgan d5ee526f0e Release v0.4.0: bump version and finalize CHANGELOG date
Bump the crate version from 0.4.0-dev to 0.4.0 and stamp the 0.4.0
CHANGELOG heading with the release date.
2026-06-27 15:57:58 +00:00
ArjenandJohnathan Corgan 22a5b3e5c6 packaging(openwrt): default .apk WAN port to DSA name 'wan'
The shared fips.yaml ships ethernet.wan.interface: "eth0", the default
WAN port on OpenWrt 24 and earlier. OpenWrt 25 (DSA) boards, which the
.apk package targets, name the WAN port "wan" instead. Rewrite the
staged copy at build time so the as-installed config binds the Ethernet
transport to the right port out of the box, without maintaining a second
copy of the config file. The .ipk package keeps "eth0".

Update the openwrt README to document eth0 (24) vs wan (25/DSA) and add
a CHANGELOG entry.
2026-06-26 03:31:17 +00:00
Johnathan Corgan 3ea7ca1fd1 ci(openwrt): retry Blossom upload and skip nostr event on failure
A transient timeout uploading the built .ipk to the Blossom CDN
(blossom.primal.net) failed the entire OpenWrt Package run, and because
the GitHub release job depends on the build jobs, a CDN blip would block
every OpenWrt artifact from the release. Blossom/nostr distribution is
supplementary; the package already ships as a GitHub release artifact.

Wrap the upload in a 3-attempt retry with backoff to absorb transient
timeouts, mark the step continue-on-error so a persistent outage no
longer fails the build, and guard the NIP-94 publish on the upload
succeeding so no event is created with an empty URL. Applied to both the
.ipk and .apk build jobs.

Claude-Session: https://claude.ai/code/session_01A2pYfSypNmmG4HyHwZuLex
2026-06-21 14:40:50 +00:00
Johnathan Corgan 262d98a8eb Finalize v0.4.0 release content: CHANGELOG, release notes, README
Prepare the source tree for the v0.4.0 release cut, leaving the version
at 0.4.0-dev (the version bump rides a separate release-candidate
commit).

- CHANGELOG: backfill the missing entry for the route-class transit
  counters and fipstop routing-tab reorg, then reorganize the Unreleased
  block from a flat per-commit list into topic-grouped subsections
  mirroring the 0.3.0 entry (coalescing interim fixes into net-effect
  descriptions without dropping technical detail), and stamp it as
  [0.4.0] - 2026-06-21 with a fresh empty Unreleased block. The date is
  provisional and reconfirmed at the final tag.
- Release notes: refresh both RELEASE-NOTES.md and the versioned archive
  (kept byte-identical) to cover the OpenWrt .apk packaging, the Nix
  flake, the macOS self-traffic checksum fix (#117), and the route-class
  transit counters.
- README: bump the status badge to v0.4.0 so it matches the prose.

Claude-Session: https://claude.ai/code/session_01A2pYfSypNmmG4HyHwZuLex
2026-06-21 13:45:19 +00:00
Johnathan Corgan 274b09d4ff node: classify transit forwards by route class; regroup fipstop routing tab
Add six forwarding counters that partition transit-forwarded packets by their
tree relationship to the chosen next hop: tree-up (peer is our ancestor),
tree-down (peer is our descendant and the destination is within its subtree),
tree-down-cross (peer is our descendant but the destination is outside its
subtree), cross-link descend (lateral peer, destination within its subtree),
cross-link ascend (lateral peer, destination outside its subtree), and
direct-peer. The six classes sum to forwarded_packets, asserted by a unit
test. Classification is computed from tree coordinates at the transit
chokepoint, so the error-signal routing callers are excluded.

The two "outside the chosen peer's subtree" classes are both up-and-over
forwards but differ in what they depend on. Tree-down-cross is the
dive-to-tree-child cut-through: we forward down to our own child for a
destination not beneath it, which is only possible because the child
advertised cross-link reach upward to us, beyond its own subtree. Its count
measures how much forwarding depends on that upward advertisement, i.e. what
would change if cross-link advertisements were narrowed to subtree-entry only.
Cross-link ascend, by contrast, uses the node's own lateral cross-link learned
from a peer's split-horizon advertisement, so it does not depend on any upward
advertisement.

Surface the counters through the forwarding stats snapshot (control socket,
show_routing and show_status) and reorganize the fipstop routing tab so its
two columns separate own/endpoint traffic (received, delivered, originated)
from forwarded/transit traffic (the route-class breakdown and drop reasons),
with the tree-down-cross line visually flagged.
2026-06-20 14:56:54 +00:00
ArjenandJohnathan Corgan 0f1fd18c25 packaging(openwrt): SDK-free .apk build for OpenWrt 25+ and control-socket fix
Add OpenWrt .apk packaging for OpenWrt 25+, where apk-tools is the
mandatory package manager. The existing .ipk continues to cover OpenWrt
24.x and earlier. Built SDK-free like the .ipk: it reuses the
cargo-zigbuild cross-compile and the shared installed-filesystem payload
under openwrt-ipk/files, and assembles the ADB container with the
official `apk mkpkg` applet from apk-tools 3.0.5 built from source, so no
OpenWrt SDK image is needed.

The package-openwrt workflow is refactored so a single compile-binaries
job cross-compiles and strips each arch once; both the .ipk and .apk
packagers consume the binaries via a new --bin-dir flag instead of each
recompiling. A build-apk job (aarch64, x86_64) builds apk-tools,
packages, and structurally verifies the .apk with `apk adbdump`. Releases
now publish .apk artifacts and checksums alongside .ipk; the release
download is scoped to fips_* so the shared raw-binary artifacts are not
swept into the published release. apk-version.sh maps a release tag or
commit height to an apk-tools-valid version, covered by a case-table test.
Packages are unsigned, installed with `apk add --allow-untrusted`,
matching the .ipk posture.

Also fix the OpenWrt control socket: the init script now pre-creates
/run/fips before starting the daemon, the procd equivalent of the systemd
unit's RuntimeDirectory=fips. Without it, on a fresh boot the daemon
resolves its control socket to /tmp (since /run/fips does not yet exist),
while fips-gateway later creates /run/fips for its own gateway.sock,
leaving fipsctl/fipstop resolving a /run/fips/control.sock the daemon
never bound.
2026-06-20 14:44:05 +00:00
ArjenandGitHub 225fab29ab fix(tun): complete L4 checksum on hairpinned self-traffic (macOS) (#117)
Self-addressed TCP/UDP connections to a node's own <npub>.fips address
half-opened and hung on macOS. macOS routes self-traffic as loopback (a
LOCAL route via lo0), which defers the transport TX checksum, but the
point-to-point utun then egresses the packet into the daemon with only
the pseudo-header partial checksum present. The hairpin path added in
9a9e90a re-injected these verbatim, so the local stack dropped every
segment whose checksum MSS clamping didn't happen to rewrite: the
SYN/SYN-ACK got through (clamping recomputes them) but the bare ACK,
data, and FIN were dropped for a bad checksum, leaving the listener
stuck in SYN_RCVD.

Recompute the TCP/UDP checksum for self-addressed packets on the hairpin
path before re-injection, completing the self-delivery 9a9e90a started
(which only covered ICMP and the TCP handshake). Linux is unaffected: it
loops self-traffic via lo before the TUN, so the hairpin branch never
fires and checksums are already valid.

Confirmed on macOS: a self-connect that previously timed out now
completes in ~9ms with payload delivered.
2026-06-18 08:36:57 -07:00
ArjenandJohnathan Corgan effd69bd53 ci(openwrt): bump Node 20 actions to Node 24 majors
checkout v4->v6, cache v4->v5, upload-artifact v4->v7,
download-artifact v4->v8 to clear the Node 20 runner deprecation.
2026-06-17 17:58:37 +00:00
Johnathan Corgan 3749853716 packaging(openwrt): publish checksums-openwrt.txt alongside .ipk artifacts
The linux/macos/windows package workflows each publish a
checksums-<platform>.txt sha256 file with their release artifacts, but
the OpenWrt workflow attached only the .ipk files with no checksum
coverage. Generate checksums-openwrt.txt over the .ipk outputs using the
same sha256sum idiom as the other platforms and add it to the release
file set.
2026-06-17 17:58:37 +00:00
ArjenandJohnathan Corgan 289e5f8571 ci: bump Node 20 actions to Node 24 majors
Clear the GitHub Node 20 runner deprecation across CI, AUR, and the
Linux/macOS/Windows packaging workflows:

  actions/checkout          v4 -> v6
  actions/cache             v4 -> v5
  actions/upload-artifact   v4 -> v7
  actions/download-artifact v4 -> v8

package-openwrt.yml is handled separately on fix/openwrt-checksums.
2026-06-17 17:58:37 +00:00
sandwichandJohnathan Corgan 3733349d33 packaging(nix): add a Nix flake for reproducible from-source builds
Add a flake that builds all four binaries (fips, fipsctl, fips-gateway,
fipstop) on Linux and macOS, pinning the exact toolchain from
rust-toolchain.toml (1.94.1 + rustfmt/clippy) via fenix so Nix builds
match CI and the AUR/Debian packaging.

Wire up the build-time native deps the source tree needs: pkg-config and
bindgenHook (libclang) for the rustables/libdbus-sys bindgen step, and
dbus for bluer's BLE support. autoPatchelfHook rewrites the binary RPATHs
so the daemon resolves libdbus-1.so.3 and libgcc_s.so.1 from the Nix store
at runtime — without it `fips` fails to load on NixOS, which has no global
/usr/lib.

Outputs: packages.{default,fips}, apps for each binary, checks.fips, and a
devShell with the pinned toolchain plus cargo-edit. Tests are skipped in
the package build since they exercise TUN devices, raw sockets, and mDNS
that aren't available in the sandbox, mirroring the AUR/Debian packaging.

Verified end-to-end in a pure Nix store (nixos/nix container): nix build,
nix flake check, and running all four binaries succeed against the
committed flake.lock.

Documented across the install and developer docs: a Nix / NixOS section in
packaging/README.md, the from-source guide in docs/getting-started.md
(noting the flake produces binaries only, with NixOS system integration
through the system configuration rather than the installer), the CHANGELOG,
README, CONTRIBUTING, and the v0.4.0 release notes.

Co-authored-by: Johnathan Corgan <johnathan@corganlabs.com>
2026-06-17 16:36:38 +00:00
ArjenandJohnathan Corgan 9a9e90a32c fix(tun): loop back self-addressed packets for local delivery
A packet destined for our own mesh address reached the TUN reader on
macOS (utun egresses self-traffic into the daemon despite the lo0 host
route) and was pushed onto the mesh outbound path, where it was dropped
for lack of a session/route to self. Hairpin self-addressed packets back
to the TUN writer instead, so ping6 and connections to our own
<npub>.fips address are delivered locally.

On Linux the kernel already loops self-traffic via `lo` before it reaches
the TUN, so the branch never fires there; the check is kept unconditional
as a daemon-level delivery invariant and to keep it covered by Linux CI.
2026-06-17 14:32:17 +00:00
73 changed files with 4737 additions and 645 deletions
+1 -1
View File
@@ -17,7 +17,7 @@ jobs:
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v4
- uses: actions/checkout@v6
- name: Patch PKGBUILD-git b2sums for local assets
run: |
+2 -2
View File
@@ -41,7 +41,7 @@ jobs:
set -euo pipefail
pacman -Sy --noconfirm --needed base-devel namcap git curl
- uses: actions/checkout@v4
- uses: actions/checkout@v6
- name: Resolve package version
id: ver
@@ -176,7 +176,7 @@ jobs:
echo "version=${TAG#v}" >> "$GITHUB_OUTPUT"
echo "pkgrel=${PKGREL}" >> "$GITHUB_OUTPUT"
- uses: actions/checkout@v4
- uses: actions/checkout@v6
with:
ref: ${{ steps.tag.outputs.tag }}
+16 -16
View File
@@ -56,7 +56,7 @@ jobs:
name: Format check
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v4
- uses: actions/checkout@v6
- uses: actions-rust-lang/setup-rust-toolchain@v1
with:
components: rustfmt
@@ -68,7 +68,7 @@ jobs:
name: Clippy
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v4
- uses: actions/checkout@v6
- name: Install system dependencies
run: sudo apt-get update && sudo apt-get install -y libdbus-1-dev
- uses: actions-rust-lang/setup-rust-toolchain@v1
@@ -77,7 +77,7 @@ jobs:
cache: false
rustflags: ''
- name: Cache Cargo registry + build
uses: actions/cache@v4
uses: actions/cache@v5
with:
path: |
~/.cargo/registry
@@ -102,7 +102,7 @@ jobs:
- os: windows-latest
steps:
- uses: actions/checkout@v4
- uses: actions/checkout@v6
- name: Set SOURCE_DATE_EPOCH from git (Unix)
if: runner.os != 'Windows'
@@ -130,7 +130,7 @@ jobs:
rustflags: ''
- name: Cache Cargo registry + build
uses: actions/cache@v4
uses: actions/cache@v5
with:
path: |
~/.cargo/registry
@@ -159,7 +159,7 @@ jobs:
# Upload the Linux binary so integration jobs can use it without rebuilding
- name: Upload Linux binary
if: matrix.os == 'ubuntu-latest'
uses: actions/upload-artifact@v4
uses: actions/upload-artifact@v7
with:
name: fips-linux
path: |
@@ -180,7 +180,7 @@ jobs:
runs-on: ubuntu-latest
needs: [build]
steps:
- uses: actions/checkout@v4
- uses: actions/checkout@v6
- name: Set SOURCE_DATE_EPOCH from git
run: echo "SOURCE_DATE_EPOCH=$(git log -1 --format=%ct)" >> "$GITHUB_ENV"
@@ -195,7 +195,7 @@ jobs:
rustflags: ''
- name: Cache Cargo registry + build
uses: actions/cache@v4
uses: actions/cache@v5
with:
path: |
~/.cargo/registry
@@ -236,7 +236,7 @@ jobs:
runs-on: macos-latest
needs: [build]
steps:
- uses: actions/checkout@v4
- uses: actions/checkout@v6
- name: Set SOURCE_DATE_EPOCH from git
run: echo "SOURCE_DATE_EPOCH=$(git log -1 --format=%ct)" >> "$GITHUB_ENV"
@@ -248,7 +248,7 @@ jobs:
rustflags: ''
- name: Cache Cargo registry + build
uses: actions/cache@v4
uses: actions/cache@v5
with:
path: |
~/.cargo/registry
@@ -271,7 +271,7 @@ jobs:
name: Unit tests (Windows)
runs-on: windows-latest
steps:
- uses: actions/checkout@v4
- uses: actions/checkout@v6
- name: Install Rust toolchain
uses: actions-rust-lang/setup-rust-toolchain@v1
@@ -280,7 +280,7 @@ jobs:
rustflags: ''
- name: Cache Cargo registry + build
uses: actions/cache@v4
uses: actions/cache@v5
with:
path: |
~/.cargo/registry
@@ -309,7 +309,7 @@ jobs:
name: PowerShell lint (Windows packaging)
runs-on: windows-latest
steps:
- uses: actions/checkout@v4
- uses: actions/checkout@v6
- name: Run PSScriptAnalyzer
shell: pwsh
@@ -478,11 +478,11 @@ jobs:
type: dns-resolver
steps:
- uses: actions/checkout@v4
- uses: actions/checkout@v6
# Fetch the pre-built Linux binary from job 1
- name: Download Linux binary
uses: actions/download-artifact@v4
uses: actions/download-artifact@v8
with:
name: fips-linux
path: _bin
@@ -668,7 +668,7 @@ jobs:
- name: Upload sim results on failure (chaos)
if: matrix.type == 'chaos' && failure()
uses: actions/upload-artifact@v4
uses: actions/upload-artifact@v7
with:
name: sim-results-${{ matrix.scenario }}
path: testing/chaos/sim-results/
+5 -5
View File
@@ -19,7 +19,7 @@ jobs:
outputs:
linux_package_version: ${{ steps.linux_version.outputs.linux_package_version }}
steps:
- uses: actions/checkout@v4
- uses: actions/checkout@v6
with:
fetch-depth: 0
@@ -61,7 +61,7 @@ jobs:
deb_arch: arm64
steps:
- uses: actions/checkout@v4
- uses: actions/checkout@v6
with:
fetch-depth: 0
@@ -79,7 +79,7 @@ jobs:
- name: Cache Cargo registry + build
if: ${{ env.ACT != 'true' }}
uses: actions/cache@v4
uses: actions/cache@v5
with:
path: |
~/.cargo/registry
@@ -140,7 +140,7 @@ jobs:
- name: Upload artifact (GitHub only)
if: ${{ env.ACT != 'true' }}
uses: actions/upload-artifact@v4
uses: actions/upload-artifact@v7
with:
name: fips_${{ needs.determine-versioning.outputs.linux_package_version }}_${{ matrix.artifact_arch }}_linux
path: |
@@ -164,7 +164,7 @@ jobs:
steps:
- name: Download Linux artifacts
uses: actions/download-artifact@v4
uses: actions/download-artifact@v8
with:
path: dist
merge-multiple: true
+6 -6
View File
@@ -19,7 +19,7 @@ jobs:
outputs:
macos_package_version: ${{ steps.macos_version.outputs.macos_package_version }}
steps:
- uses: actions/checkout@v4
- uses: actions/checkout@v6
with:
fetch-depth: 0
@@ -61,7 +61,7 @@ jobs:
target: x86_64-apple-darwin
steps:
- uses: actions/checkout@v4
- uses: actions/checkout@v6
with:
fetch-depth: 0
@@ -76,7 +76,7 @@ jobs:
rustflags: ''
- name: Cache Cargo registry + build
uses: actions/cache@v4
uses: actions/cache@v5
with:
path: |
~/.cargo/registry
@@ -209,7 +209,7 @@ jobs:
( cd "$(dirname "$PKG")" && shasum -a 256 "$(basename "$PKG")" | tee "$(basename "$PKG").sha256" )
- name: Upload artifact
uses: actions/upload-artifact@v4
uses: actions/upload-artifact@v7
with:
name: fips_${{ needs.determine-versioning.outputs.macos_package_version }}_${{ matrix.arch }}_macos
path: |
@@ -229,7 +229,7 @@ jobs:
steps:
- name: Download macOS artifacts
uses: actions/download-artifact@v4
uses: actions/download-artifact@v8
with:
path: dist
merge-multiple: true
@@ -283,7 +283,7 @@ jobs:
steps:
- name: Download macOS artifacts
uses: actions/download-artifact@v4
uses: actions/download-artifact@v8
with:
path: dist
merge-multiple: true
+431 -30
View File
@@ -24,9 +24,10 @@ jobs:
runs-on: ubuntu-latest
outputs:
package_version: ${{ steps.version.outputs.package_version }}
apk_version: ${{ steps.version.outputs.apk_version }}
release_channel: ${{ steps.channel.outputs.release_channel }}
steps:
- uses: actions/checkout@v4
- uses: actions/checkout@v6
with:
fetch-depth: 0
@@ -35,13 +36,20 @@ jobs:
shell: bash
run: |
: ${GITHUB_OUTPUT:=/tmp/github_output}
# package_version is the human-readable label used in artifact
# filenames; apk_version is the apk-tools-compatible string embedded
# in the .apk metadata. apk_version is built directly from the same
# structured inputs (tag, or commit height) — no reparse of the
# flattened package_version. See packaging/openwrt-apk/apk-version.sh.
if [[ "$GITHUB_REF" == refs/tags/* ]]; then
echo "package_version=${GITHUB_REF_NAME}" >> "$GITHUB_OUTPUT"
echo "apk_version=$(sh packaging/openwrt-apk/apk-version.sh tag "${GITHUB_REF_NAME}")" >> "$GITHUB_OUTPUT"
else
BRANCH=$(echo "$GITHUB_REF_NAME" | sed 's|/|-|g')
HEIGHT=$(git rev-list --count HEAD)
HASH=$(git rev-parse --short HEAD)
echo "package_version=${BRANCH}.${HEIGHT}.${HASH}" >> "$GITHUB_OUTPUT"
echo "apk_version=$(sh packaging/openwrt-apk/apk-version.sh dev "${HEIGHT}")" >> "$GITHUB_OUTPUT"
fi
- name: Determine release channel
@@ -64,8 +72,8 @@ jobs:
echo "release_channel=dev" >> "$GITHUB_OUTPUT"
fi
build:
name: Build .ipk (${{ matrix.openwrt_arch }})
compile-binaries:
name: Cross-compile (${{ matrix.openwrt_arch }})
runs-on: ubuntu-latest
needs: determine-versioning
@@ -96,18 +104,10 @@ jobs:
# x86 routers / VMs
steps:
- uses: actions/checkout@v4
- uses: actions/checkout@v6
with:
fetch-depth: 0
- name: Set SOURCE_DATE_EPOCH from git
run: echo "SOURCE_DATE_EPOCH=$(git log -1 --format=%ct)" >> "$GITHUB_ENV"
- name: Initialize
run: |
PACKAGE_FILENAME=${{ env.PACKAGE_NAME }}_${{ needs.determine-versioning.outputs.package_version }}_${{ matrix.openwrt_arch }}.ipk
echo "PACKAGE_FILENAME=$PACKAGE_FILENAME" >> $GITHUB_ENV
- name: Install Rust toolchain (stable)
if: matrix.rust_channel == 'stable'
uses: actions-rust-lang/setup-rust-toolchain@v1
@@ -124,7 +124,7 @@ jobs:
- name: Cache Cargo registry + build
if: ${{ env.ACT != 'true' }}
uses: actions/cache@v4
uses: actions/cache@v5
with:
path: |
~/.cargo/registry
@@ -153,6 +153,64 @@ jobs:
- name: Install llvm-strip
run: sudo apt-get update && sudo apt-get install -y --no-install-recommends llvm
# Cross-compile + strip once; both the .ipk and .apk packagers consume
# these artifacts via --bin-dir, so the Rust build runs a single time
# per architecture instead of once per package format.
- name: Cross-compile and strip binaries
run: |
set -euo pipefail
cargo zigbuild --release --target ${{ matrix.rust_target }} \
--bin fips --bin fipsctl --bin fipstop --bin fips-gateway
RELEASE_DIR="target/${{ matrix.rust_target }}/release"
mkdir -p out
for b in fips fipsctl fipstop fips-gateway; do
llvm-strip "$RELEASE_DIR/$b" 2>/dev/null || true
cp "$RELEASE_DIR/$b" "out/$b"
done
ls -lh out/
- name: Upload binaries artifact
uses: actions/upload-artifact@v7
with:
name: fips-bins-${{ matrix.openwrt_arch }}
path: out/
retention-days: 1
build:
name: Build .ipk (${{ matrix.openwrt_arch }})
runs-on: ubuntu-latest
needs: [determine-versioning, compile-binaries]
strategy:
fail-fast: false
matrix:
# Must be a subset of compile-binaries' arches (this job consumes those
# binary artifacts). Currently both ship aarch64 + x86_64.
include:
- build_arch: aarch64
openwrt_arch: aarch64_cortex-a53
- build_arch: x86_64
openwrt_arch: x86_64
steps:
- uses: actions/checkout@v6
with:
fetch-depth: 0
- name: Set SOURCE_DATE_EPOCH from git
run: echo "SOURCE_DATE_EPOCH=$(git log -1 --format=%ct)" >> "$GITHUB_ENV"
- name: Initialize
run: |
PACKAGE_FILENAME=${{ env.PACKAGE_NAME }}_${{ needs.determine-versioning.outputs.package_version }}_${{ matrix.openwrt_arch }}.ipk
echo "PACKAGE_FILENAME=$PACKAGE_FILENAME" >> $GITHUB_ENV
- name: Download prebuilt binaries
uses: actions/download-artifact@v8
with:
name: fips-bins-${{ matrix.openwrt_arch }}
path: bins
- name: Install nak
shell: bash
run: |
@@ -201,8 +259,7 @@ jobs:
- name: Build .ipk
env:
PKG_VERSION: ${{ needs.determine-versioning.outputs.package_version }}
LLVM_STRIP: llvm-strip
run: ./packaging/openwrt-ipk/build-ipk.sh --arch ${{ matrix.build_arch }}
run: ./packaging/openwrt-ipk/build-ipk.sh --arch ${{ matrix.build_arch }} --bin-dir "$GITHUB_WORKSPACE/bins"
- name: Install shellcheck (if missing)
shell: bash
@@ -389,13 +446,13 @@ jobs:
- name: SHA-256 hashes
run: |
echo "==> Binaries:"
sha256sum target/${{ matrix.rust_target }}/release/fips target/${{ matrix.rust_target }}/release/fipsctl target/${{ matrix.rust_target }}/release/fipstop
sha256sum bins/fips bins/fipsctl bins/fipstop
echo "==> Package:"
sha256sum dist/${{ env.PACKAGE_FILENAME }}
- name: Upload artifact (GitHub only)
if: ${{ env.ACT != 'true' }}
uses: actions/upload-artifact@v4
uses: actions/upload-artifact@v7
with:
name: ${{ env.PACKAGE_FILENAME }}
path: dist/${{ env.PACKAGE_FILENAME }}
@@ -403,6 +460,7 @@ jobs:
- name: Upload to Blossom
id: blossom_upload
continue-on-error: true
shell: bash
env:
BLOSSOM_SERVER: "https://blossom.primal.net"
@@ -410,17 +468,30 @@ jobs:
run: |
: ${GITHUB_OUTPUT:=/tmp/github_output}
UPLOAD_RESPONSE=$(nak blossom upload \
--server "$BLOSSOM_SERVER" \
--sec "$NSEC" \
"dist/${{ env.PACKAGE_FILENAME }}" < /dev/null)
FILE_HASH=""
for attempt in 1 2 3; do
if UPLOAD_RESPONSE=$(nak blossom upload \
--server "$BLOSSOM_SERVER" \
--sec "$NSEC" \
"dist/${{ env.PACKAGE_FILENAME }}" < /dev/null); then
echo "Upload response (attempt $attempt):"
echo "$UPLOAD_RESPONSE"
FILE_HASH=$(echo "$UPLOAD_RESPONSE" | jq -r '.sha256')
if [ -n "$FILE_HASH" ] && [ "$FILE_HASH" != "null" ]; then
break
fi
echo "Upload response had no sha256 (attempt $attempt)"
else
echo "Blossom upload timed out or failed (attempt $attempt)"
fi
FILE_HASH=""
[ "$attempt" -lt 3 ] && sleep $((attempt * 10))
done
echo "Upload response:"
echo "$UPLOAD_RESPONSE"
FILE_HASH=$(echo "$UPLOAD_RESPONSE" | jq -r '.sha256')
if [ -z "$FILE_HASH" ] || [ "$FILE_HASH" = "null" ]; then
echo "Failed to extract hash from upload response"
echo "Blossom upload did not succeed after 3 attempts; non-fatal."
echo "The package still ships as a GitHub release artifact; only the"
echo "supplementary Blossom/nostr distribution is skipped this run."
exit 1
fi
@@ -431,6 +502,7 @@ jobs:
- name: Publish NIP-94 release event
id: publish
if: steps.blossom_upload.outcome == 'success'
shell: bash
env:
RELAYS: "wss://relay.damus.io wss://nos.lol wss://nostr.mom wss://offchain.pub"
@@ -452,6 +524,7 @@ jobs:
--tag A="${{ matrix.openwrt_arch }}" \
--tag v="$VERSION" \
--tag n="${{ env.PACKAGE_NAME }}" \
--tag format="ipk" \
--tag compression="none" \
> event.json 2> event.err
@@ -509,23 +582,351 @@ jobs:
echo " Release EventId: ${{ steps.publish.outputs.eventId }}"
echo " Blossom URL: ${{ steps.blossom_upload.outputs.url }}"
build-apk:
name: Build .apk (${{ matrix.openwrt_arch }})
runs-on: ubuntu-latest
needs: [determine-versioning, compile-binaries]
strategy:
fail-fast: false
matrix:
# Must be a subset of compile-binaries' arches.
include:
- build_arch: aarch64
openwrt_arch: aarch64_cortex-a53
# MT3000, MT6000, Flint 2, RPi 3/4/5 on OpenWrt 25+
- build_arch: x86_64
openwrt_arch: x86_64
# x86 routers / VMs on OpenWrt 25+
env:
# apk-tools commit OpenWrt pins for the .apk (ADB) format. Keep in sync
# with package/system/apk/Makefile in the targeted OpenWrt release so the
# packages we produce are readable by the apk on the device.
APK_TOOLS_VERSION: "3.0.5"
APK_TOOLS_COMMIT: "b5a31c0d865342ad80be10d68f1bb3d3ad9b0866"
steps:
- uses: actions/checkout@v6
with:
fetch-depth: 0
- name: Set SOURCE_DATE_EPOCH from git
run: echo "SOURCE_DATE_EPOCH=$(git log -1 --format=%ct)" >> "$GITHUB_ENV"
- name: Initialize
run: |
PACKAGE_FILENAME=${{ env.PACKAGE_NAME }}_${{ needs.determine-versioning.outputs.package_version }}_${{ matrix.openwrt_arch }}.apk
echo "PACKAGE_FILENAME=$PACKAGE_FILENAME" >> $GITHUB_ENV
- name: Download prebuilt binaries
uses: actions/download-artifact@v8
with:
name: fips-bins-${{ matrix.openwrt_arch }}
path: bins
- name: Install fakeroot
run: sudo apt-get update && sudo apt-get install -y --no-install-recommends fakeroot
# apk mkpkg lives in apk-tools v3, which is not packaged for Ubuntu, so we
# build the pinned release from source. This is the SDK-free equivalent of
# how the .ipk path uses plain tar — one small C tool, no OpenWrt SDK.
- name: Build apk-tools (${{ env.APK_TOOLS_VERSION }}) from source
run: |
set -euo pipefail
sudo apt-get install -y --no-install-recommends \
git ca-certificates build-essential meson ninja-build pkg-config \
zlib1g-dev libssl-dev libzstd-dev liblzma-dev lua5.4-dev scdoc
git clone --quiet https://gitlab.alpinelinux.org/alpine/apk-tools.git /tmp/apk-tools
cd /tmp/apk-tools
git checkout --quiet "${APK_TOOLS_COMMIT}"
meson setup build
ninja -C build src/apk
APK_BIN=/tmp/apk-tools/build/src/apk
"$APK_BIN" --version 2>/dev/null || "$APK_BIN" version 2>/dev/null || true
echo "APK_BIN=$APK_BIN" >> "$GITHUB_ENV"
- name: Build .apk
env:
PKG_VERSION: ${{ needs.determine-versioning.outputs.package_version }}
APK_VERSION: ${{ needs.determine-versioning.outputs.apk_version }}
run: ./packaging/openwrt-apk/build-apk.sh --arch ${{ matrix.build_arch }} --bin-dir "$GITHUB_WORKSPACE/bins"
- name: Verify apk structural integrity
shell: bash
run: |
set -euo pipefail
APK="dist/${{ env.PACKAGE_FILENAME }}"
if [ ! -s "$APK" ]; then
echo "FAIL: produced apk not found or empty at $APK"
exit 1
fi
echo "==> file type:"; file "$APK"
# apk v3 packages are ADB containers; dump the whole manifest with the
# apk-tools we just built. Print it in full so the exact schema is
# always visible in the log if an assertion needs adjusting.
DUMP=$(mktemp)
"$APK_BIN" adbdump "$APK" > "$DUMP" 2>/dev/null || {
echo "FAIL: 'apk adbdump' could not read $APK"; exit 1; }
echo "==> full adbdump:"; cat "$DUMP"
fail=0
# Package metadata (flat keys under info:).
for needle in "name: fips" "version: ${{ needs.determine-versioning.outputs.apk_version }}" "arch: ${{ matrix.openwrt_arch }}"; do
if grep -qF "$needle" "$DUMP"; then
echo " PASS meta: $needle"
else
echo " FAIL meta: missing '$needle'"; fail=1
fi
done
# installed-size reflects the bundled binaries (4 stripped Rust
# binaries, several MB). A payload regression that drops them shows up
# here regardless of how the path tree is formatted.
SIZE=$(awk '/^[[:space:]]*installed-size:/ {print $2; exit}' "$DUMP")
echo " installed-size: ${SIZE:-unknown}"
if [ -z "${SIZE:-}" ] || [ "$SIZE" -lt 1000000 ]; then
echo " FAIL: installed-size implausibly small (binaries missing?)"; fail=1
else
echo " PASS: installed-size >= 1MB"
fi
# The adbdump paths: block is hierarchical. Each directory is a
# top-level list item "- name: <full relative dir>"; its files are
# "- name: <basename>" nested one indent level deeper under "files:".
# (There are no "path:" keys.) Reconstruct full file paths by keying
# off the indentation of the directory-level list items.
RECON=$(awk '
/^paths:/ {p=1; diri=-1; next}
p && /^[^ #-]/ {p=0} # a new top-level key ends paths:
!p {next}
match($0, /^ *- /) {
ind=RLENGTH; rest=substr($0, RLENGTH+1)
if (diri==-1) diri=ind # first list item = directory indent
if (ind==diri) { # directory entry (or the root acl: entry)
if (rest ~ /^name: /) { dir=rest; sub(/^name: /,"",dir) } else dir=""
next
}
if (rest ~ /^name: /) { # deeper item = a file under files:
f=rest; sub(/^name: /,"",f); print (dir==""?f:dir"/"f)
}
}
' "$DUMP")
echo "==> reconstructed paths:"; printf '%s\n' "$RECON"
for path in \
usr/bin/fips usr/bin/fipsctl usr/bin/fipstop usr/bin/fips-gateway \
etc/init.d/fips etc/init.d/fips-gateway \
etc/fips/fips.yaml etc/fips/firewall.sh etc/dnsmasq.d/fips.conf \
etc/sysctl.d/fips-gateway.conf etc/sysctl.d/fips-bridge.conf \
etc/hotplug.d/net/99-fips etc/uci-defaults/90-fips-setup \
lib/upgrade/keep.d/fips; do
if printf '%s\n' "$RECON" | grep -qxF "$path"; then
echo " PASS path: $path"
else
echo " FAIL path: missing $path"; fail=1
fi
done
if [ "$fail" -ne 0 ]; then
echo "apk structural verification FAILED"
exit 1
fi
echo "apk structural verification PASS"
- name: SHA-256 hashes
run: |
echo "==> Binaries:"
sha256sum bins/fips bins/fipsctl bins/fipstop
echo "==> Package:"
sha256sum dist/${{ env.PACKAGE_FILENAME }}
- name: Upload artifact (GitHub only)
if: ${{ env.ACT != 'true' }}
uses: actions/upload-artifact@v7
with:
name: ${{ env.PACKAGE_FILENAME }}
path: dist/${{ env.PACKAGE_FILENAME }}
retention-days: 30
- name: Install nak
shell: bash
run: |
NAK_VERSION="0.16.2"
ARCH=$(uname -m)
case "$ARCH" in
x86_64|amd64) NAK_ARCH="amd64" ;;
aarch64|arm64) NAK_ARCH="arm64" ;;
*) echo "Unsupported architecture: $ARCH"; exit 1 ;;
esac
curl -fsSL "https://github.com/fiatjaf/nak/releases/download/v${NAK_VERSION}/nak-v${NAK_VERSION}-linux-${NAK_ARCH}" \
-o /usr/local/bin/nak
chmod +x /usr/local/bin/nak
nak --version
- name: Install jq
run: |
if ! command -v jq &>/dev/null; then
sudo apt-get update && sudo apt-get install -y jq
fi
# Priority: HIVE_CI_NSEC from env (loom job) > repo secret > generate ephemeral
- name: Resolve signing key
id: keys
shell: bash
env:
SECRET_NSEC: ${{ secrets.HIVE_CI_NSEC }}
run: |
: ${GITHUB_OUTPUT:=/tmp/github_output}
if [ -n "${HIVE_CI_NSEC:-}" ]; then
echo "Using HIVE_CI_NSEC from loom job environment"
NSEC="$HIVE_CI_NSEC"
elif [ -n "$SECRET_NSEC" ]; then
echo "Using HIVE_CI_NSEC from repository secrets"
NSEC="$SECRET_NSEC"
else
echo "No nsec provided -- generating ephemeral keypair"
NSEC=$(nak key generate)
fi
PUBKEY=$(echo "$NSEC" | nak key public)
echo "::add-mask::$NSEC"
echo "nsec=$NSEC" >> "$GITHUB_OUTPUT"
echo "pubkey=$PUBKEY" >> "$GITHUB_OUTPUT"
echo "Publisher pubkey (hex): $PUBKEY"
- name: Upload to Blossom
id: blossom_upload
continue-on-error: true
shell: bash
env:
BLOSSOM_SERVER: "https://blossom.primal.net"
NSEC: ${{ steps.keys.outputs.nsec }}
run: |
: ${GITHUB_OUTPUT:=/tmp/github_output}
FILE_HASH=""
for attempt in 1 2 3; do
if UPLOAD_RESPONSE=$(nak blossom upload \
--server "$BLOSSOM_SERVER" \
--sec "$NSEC" \
"dist/${{ env.PACKAGE_FILENAME }}" < /dev/null); then
echo "Upload response (attempt $attempt):"
echo "$UPLOAD_RESPONSE"
FILE_HASH=$(echo "$UPLOAD_RESPONSE" | jq -r '.sha256')
if [ -n "$FILE_HASH" ] && [ "$FILE_HASH" != "null" ]; then
break
fi
echo "Upload response had no sha256 (attempt $attempt)"
else
echo "Blossom upload timed out or failed (attempt $attempt)"
fi
FILE_HASH=""
[ "$attempt" -lt 3 ] && sleep $((attempt * 10))
done
if [ -z "$FILE_HASH" ] || [ "$FILE_HASH" = "null" ]; then
echo "Blossom upload did not succeed after 3 attempts; non-fatal."
echo "The package still ships as a GitHub release artifact; only the"
echo "supplementary Blossom/nostr distribution is skipped this run."
exit 1
fi
BLOSSOM_URL="${BLOSSOM_SERVER}/${FILE_HASH}"
echo "url=$BLOSSOM_URL" >> "$GITHUB_OUTPUT"
echo "hash=$FILE_HASH" >> "$GITHUB_OUTPUT"
echo "Uploaded to Blossom: $BLOSSOM_URL"
- name: Publish NIP-94 release event
id: publish
if: steps.blossom_upload.outcome == 'success'
shell: bash
env:
RELAYS: "wss://relay.damus.io wss://nos.lol wss://nostr.mom wss://offchain.pub"
NSEC: ${{ steps.keys.outputs.nsec }}
run: |
: ${GITHUB_OUTPUT:=/tmp/github_output}
set -e
VERSION="${{ needs.determine-versioning.outputs.package_version }}"
CHANNEL="${{ needs.determine-versioning.outputs.release_channel }}"
nak event --sec "$NSEC" -k 1063 \
-c "FIPS Package: ${{ env.PACKAGE_NAME }} for ${{ matrix.openwrt_arch }} (apk)" \
--tag url="${{ steps.blossom_upload.outputs.url }}" \
--tag m="application/octet-stream" \
--tag x="${{ steps.blossom_upload.outputs.hash }}" \
--tag ox="${{ steps.blossom_upload.outputs.hash }}" \
--tag filename="${{ env.PACKAGE_FILENAME }}" \
--tag A="${{ matrix.openwrt_arch }}" \
--tag v="$VERSION" \
--tag n="${{ env.PACKAGE_NAME }}" \
--tag format="apk" \
--tag compression="none" \
> event.json 2> event.err
if [ ! -s event.json ]; then
echo "Failed to create event"
cat event.err 2>/dev/null || true
exit 1
fi
echo "=== Event JSON ==="
cat event.json
echo "=================="
EVENT_ID=$(jq -r '.id' event.json)
if [ -z "$EVENT_ID" ] || [ "$EVENT_ID" = "null" ]; then
echo "Failed to extract event ID"
exit 1
fi
# Publish to relays
cat event.json | nak event $RELAYS 2>&1
echo "eventId=$EVENT_ID" >> "$GITHUB_OUTPUT"
echo "Published NIP-94 event: $EVENT_ID"
- name: Build Summary
run: |
echo "Build Summary for ${{ matrix.openwrt_arch }} (apk):"
echo " Package: ${{ env.PACKAGE_FILENAME }}"
echo " apk version: ${{ needs.determine-versioning.outputs.apk_version }}"
echo " Release EventId: ${{ steps.publish.outputs.eventId }}"
echo " Blossom URL: ${{ steps.blossom_upload.outputs.url }}"
release:
name: Publish GitHub Release (github only)
runs-on: ubuntu-latest
needs: build
needs: [build, build-apk]
if: startsWith(github.ref, 'refs/tags/')
permissions:
contents: write
steps:
- name: Download all .ipk artifacts
uses: actions/download-artifact@v4
- name: Download package artifacts
uses: actions/download-artifact@v8
with:
# Only the .ipk/.apk packages (named fips_<ver>_<arch>.*), not the
# fips-bins-* raw-binary artifacts shared between the build jobs.
pattern: fips_*
path: dist
merge-multiple: true
- name: Generate OpenWrt release checksums
run: |
cd dist
find . -maxdepth 1 -type f \( -name '*.ipk' -o -name '*.apk' \) -printf '%P\n' \
| LC_ALL=C sort \
| xargs sha256sum \
> checksums-openwrt.txt
- name: Create release
uses: softprops/action-gh-release@v2
with:
files: dist/*.ipk
files: |
dist/*.ipk
dist/*.apk
dist/checksums-openwrt.txt
generate_release_notes: true
+5 -5
View File
@@ -19,7 +19,7 @@ jobs:
outputs:
package_version: ${{ steps.version.outputs.package_version }}
steps:
- uses: actions/checkout@v4
- uses: actions/checkout@v6
with:
fetch-depth: 0
@@ -52,7 +52,7 @@ jobs:
needs: determine-versioning
steps:
- uses: actions/checkout@v4
- uses: actions/checkout@v6
with:
fetch-depth: 0
@@ -69,7 +69,7 @@ jobs:
rustflags: ''
- name: Cache Cargo registry + build
uses: actions/cache@v4
uses: actions/cache@v5
with:
path: |
~/.cargo/registry
@@ -146,7 +146,7 @@ jobs:
}
- name: Upload artifact
uses: actions/upload-artifact@v4
uses: actions/upload-artifact@v7
with:
name: fips_${{ needs.determine-versioning.outputs.package_version }}_x86_64_windows
path: deploy/fips-*-windows-*.zip
@@ -170,7 +170,7 @@ jobs:
steps:
- name: Download Windows artifacts
uses: actions/download-artifact@v4
uses: actions/download-artifact@v8
with:
path: dist
merge-multiple: true
+405 -295
View File
@@ -9,6 +9,16 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
### Added
### Changed
### Fixed
## [0.4.0] - 2026-06-27
### Added
#### Transports (Nym, mDNS LAN discovery)
- Nym mixnet transport (`transports.nym`) for outbound peer links
tunneled through a local `nym-socks5-client` SOCKS5 proxy into the
Nym mixnet, as a privacy transport alongside Tor. Outbound-only and
@@ -16,6 +26,26 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
crate dependencies. A single-container example
(`examples/sidecar-nostr-mixnet-relay/`) demonstrates FIPS peering
across the mixnet end to end.
- Opt-in mDNS / DNS-SD LAN discovery for sub-second pairing of peers on
the same local link, without a relay or NAT-traversal roundtrip.
Disabled by default; operators enable it with
`node.discovery.lan.enabled: true`. Configurable service type and an
optional `node.discovery.lan.scope` that isolates discovery to peers
sharing the same private-network scope. The advertised UDP port is
chosen from a non-bootstrap operational UDP transport using a stable
selector, so it is deterministic across restarts.
#### Admission / peer-list management
- `Node::update_peers` for runtime peer-list refresh, returning an
`UpdatePeersOutcome` summarizing added, removed, and retained peers.
Re-derives active peer connections from a new peer configuration
without dropping links to peers that remain in the set.
`PeerAddress` gains a `seen_at_ms` recency field (with
`with_seen_at_ms`) used to prefer more recently observed addresses.
#### Data-plane / metrics / observability
- Typed `RejectReason` classification for receive-path silent-rejection
sites across the node. Each rejection-and-return path now passes a
typed reason to `NodeStats::record_reject`, which routes it to a
@@ -28,47 +58,73 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
site that was previously silent. Several handshake, MMP, tree, and
discovery rejection paths that had no counter at all are now counted,
including the `send_lookup_response` no-route drop
(`DiscoveryStats::resp_no_route`). Existing
direct counters at the bloom / discovery / forwarding sites are
retained alongside the new dispatch while the rollout is in progress;
a later change collapses the duplicate increment.
(`DiscoveryStats::resp_no_route`).
- Internal atomic metric registry (`Arc<MetricsRegistry>`) that shadows
the plain-`u64` `NodeStats` counters, written alongside them and
validated by a whole-struct debug-build parity check. Covers the
forwarding receive counters, the full discovery counter family, and the
tree, bloom, congestion, and error-signal counter families so far, with
tree, bloom, congestion, and error-signal counter families, with
the hottest counters cache-line padded. Behavior-neutral:
`NodeStats` remains the serving path. Groundwork for sampling metrics
without contending the receive loop.
- `Node::update_peers` for runtime peer-list refresh, returning an
`UpdatePeersOutcome` summarizing added, removed, and retained peers.
Re-derives active peer connections from a new peer configuration
without dropping links to peers that remain in the set.
`PeerAddress` gains a `seen_at_ms` recency field (with
`with_seen_at_ms`) used to prefer more recently observed addresses.
- Opt-in mDNS / DNS-SD LAN discovery for sub-second pairing of peers on
the same local link, without a relay or NAT-traversal roundtrip.
Disabled by default; operators enable it with
`node.discovery.lan.enabled: true`. Configurable service type and an
optional `node.discovery.lan.scope` that isolates discovery to peers
sharing the same private-network scope. The advertised UDP port is
chosen from a non-bootstrap operational UDP transport using a stable
selector, so it is deterministic across restarts.
- `fipsctl stats metrics`, backed by a new counter-only `show_metrics`
control query that dumps the atomic metric registry as flat counter
name/value pairs. Serves a Prometheus-style scraper that samples node
counters without contending the receive loop.
- `pool_inbound` and `pool_outbound` counters on the TCP and Tor
transport stats (`TcpStats`, `TorStats`). Per-direction accounting
is updated at every pool-insert and receive-loop-exit site, plus on
transport stop and on send-failure-driven removal. Surfaces through
`TcpStatsSnapshot` and `TorStatsSnapshot` for `show_transports`.
- `fipsctl stats metrics`, backed by a new counter-only `show_metrics`
control query that dumps the atomic metric registry as flat counter
name/value pairs. Serves a Prometheus-style scraper that samples node
counters without contending the receive loop.
#### Spanning-tree / mesh-size / routing
- Six route-class transit counters that partition transit-forwarded
packets by their tree relationship to the chosen next hop: tree-up
(peer is our ancestor), tree-down (peer is our descendant and the
destination is within its subtree), tree-down-cross (peer is our
descendant but the destination is outside its subtree), cross-link
descend (lateral peer, destination within its subtree), cross-link
ascend (lateral peer, destination outside its subtree), and
direct-peer. The six classes sum to `forwarded_packets` (asserted by a
unit test) and are computed from tree coordinates at the transit
chokepoint, so error-signal routing callers are excluded. They surface
through the forwarding stats snapshot via `show_routing` and
`show_status`.
- Discovery now counts `LookupRequest`s dropped when the dedup cache is
full. A saturated `recent_requests` cache
(`MAX_RECENT_DISCOVERY_REQUESTS`) previously dropped requests
silently; a new `DiscoveryStats::req_dedup_cache_full` counter (typed
reject reason `DiscoveryReject::ReqDedupCacheFull`) makes the drop
visible through `show_routing`.
#### Packaging & deployment
- OpenWrt `.apk` packaging (`packaging/openwrt-apk/`, `make apk`) for
OpenWrt 25+, where apk-tools is the mandatory package manager (the
existing `.ipk` continues to cover OpenWrt 24.x and earlier). Built
SDK-free: it reuses the `.ipk` cross-compile (`cargo-zigbuild`) and the
shared installed-filesystem payload, and assembles the package with
`apk mkpkg` from apk-tools 3.0.5 built from source — no OpenWrt SDK
image. A `build-apk` CI job (aarch64, x86_64) builds and structurally
verifies the package; releases now publish `.apk` artifacts and
checksums alongside `.ipk`. Packages are unsigned, installed with
`apk add --allow-untrusted`, matching the `.ipk` posture.
- Nix flake (`flake.nix` at the project root) for reproducible
from-source builds on Nix/NixOS. Builds all four binaries (`fips`,
`fipsctl`, `fips-gateway`, `fipstop`), pins the exact toolchain from
`rust-toolchain.toml` via fenix, and wires the build-time native
dependencies (`libclang` for `bindgen`, plus `dbus` and `pkg-config`),
so it needs no host setup beyond Nix with flakes enabled. Flake inputs
are lock-pinned (`flake.lock` committed) for reproducibility, and the
flake exposes `nix build`, `nix run`, a `nix develop` dev shell with the
pinned toolchain, and `nix flake check`. The flake produces binaries
(and a NixOS `packages.<system>.fips` output); the systemd/service
integration that the `.deb`/tarball installers provide is handled
through the NixOS configuration instead.
#### Docs & contributor tooling
- [`PR-REVIEW.md`](PR-REVIEW.md) — the 13-criteria PR review checklist
the maintainer runs against every incoming PR, published at the
repo root so contributors can run the same pass on their own change
@@ -90,17 +146,14 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
### Changed
#### FMP/FSP rekey reliability
- `complete_rekey_msg2` now returns the remote peer's startup epoch
alongside the new Noise session, so the rekey path can detect a peer
restart and clear stale session state.
- Active-peer path selection now sorts address candidates by recency
(`seen_at_ms`), preferring the most recently observed address when
racing concurrent path probes.
- Per-tick work budgets bound the connection churn done in a single
node tick: `MAX_DISCOVERY_CONNECTS_PER_TICK`,
`MAX_RETRY_CONNECTIONS_PER_TICK`, and
`MAX_PARALLEL_PATH_CANDIDATES_PER_PEER`. Work beyond a tick's budget
is deferred to the next tick rather than discarded.
#### NAT traversal / Nostr discovery
- Nostr discovery startup is now non-blocking. `Node::start` no
longer waits for relay connect, subscribe, or initial advert
publish before returning. A slow or unreachable relay no longer
@@ -109,6 +162,20 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
Subscribe retries with exponential backoff (2 s base, 60 s cap),
publish attempts time out at 10 s, and the new tasks are aborted
cleanly on `Node::stop`.
#### Spanning-tree / mesh-size / routing
- Active-peer path selection now sorts address candidates by recency
(`seen_at_ms`), preferring the most recently observed address when
racing concurrent path probes.
- Per-tick work budgets bound the connection churn done in a single
node tick: `MAX_DISCOVERY_CONNECTS_PER_TICK`,
`MAX_RETRY_CONNECTIONS_PER_TICK`, and
`MAX_PARALLEL_PATH_CANDIDATES_PER_PEER`. Work beyond a tick's budget
is deferred to the next tick rather than discarded.
#### Admission / peer caps
- `node.bloom.max_inbound_fpr` default raised from `0.05` to `0.10`. The
cap rejects inbound `FilterAnnounce` whose FPR (`fill^k`) exceeds it. On
the fixed 1 KB / k=5 filter, `0.05` corresponds to fill 0.549 (~1,300
@@ -124,6 +191,9 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
per-transport `max_inbound_connections`, then node-wide
`max_connections`, then the built-in default of 256. Established peers
remain bounded node-wide by `add_connection`.
#### Data-plane / worker-pool / metrics / observability
- The control-socket read surface is now served off the `rx_loop`.
Every pure-read `show_*` query — `show_status`, the `show_stats_*`
family, `show_listening_sockets`, the new `show_metrics`,
@@ -148,11 +218,6 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
snapshot fields above. Built on a ratatui `TestBackend`
render-snapshot harness that asserts the text grid and per-cell
style of every `ui::draw_*` against canned `show_*` JSON.
- Static host aliases in `/etc/fips/hosts` now hot-reload on mtime
change instead of only at daemon startup, so `fipsctl`/`fipstop`
display names reflect edits without a restart. The peer ACL and host
map both reload once per node tick through a new lock-free
`Reloadable` snapshot.
- Steady-state log noise reduced on saturated public-mesh nodes.
Routine per-peer connection-lifecycle and capacity-cap events are
demoted from info/warn to debug — FMP K-bit cutover promotion,
@@ -163,43 +228,6 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
are no longer drowned out. An exhausted FMP-msg1 / FSP-msg3 rekey
retransmission-budget abort (an expected, self-limiting outcome on
lossy or high-latency links) is likewise demoted from warn to debug.
- Sidecar example (`examples/sidecar-nostr-relay`): `udp.mtu` is now
overridable via the `FIPS_UDP_MTU` environment variable, defaulting to
1472 (preserving prior behavior). Plumbed through `docker-compose.yml`
and documented in the README env-var table. Annotated the static-CI
node template `mtu: 1472` literal with the same Docker-bridge
rationale and a pointer at the daemon's 1280 default.
- Overhauled `CONTRIBUTING.md`: replaced generic Rust-template framing
with a FIPS-specific entry point covering the four-layer
architecture, branch model and PR-target selection, structured bug
reporting, scope discipline and local-CI requirements, an AI coding
assistant policy, and project communication channels. Added
`docs/branching.md` as the long-form companion covering the release
workflow, version conventions, and merge-direction rationale.
- CI and release-publish workflows hardened:
- `ci.yml` declares a top-level `concurrency` block keyed on
`(workflow, ref)` with `cancel-in-progress: true`. Force-pushes
and rapid successive pushes to the same ref now retire any
in-flight run rather than letting superseded and current-tip runs
both burn runner minutes.
- `aur-publish.yml` rewritten to fetch the upstream source tarball
and compute its `b2sum` in CI, then patch `pkgver` and the
`b2sums` SKIP placeholder in `PKGBUILD` in-place. Previously
`updpkgsums: true` downloaded the tarball into the AUR working
tree, where it was rejected by AUR's 488 KiB max-blob hook —
silently no-op'ing the v0.3.0 stable AUR push. `fips.sysusers` /
`fips.tmpfiles` asset b2sums are recomputed in the same step to
stay in sync with the local files. `workflow_dispatch` gains a
tag input so historical release tags can be re-published
manually, and `continue-on-error: true` is dropped so future
regressions surface in CI.
- New `aur-publish-git.yml` workflow for the `fips-git` VCS
PKGBUILD, triggered on master pushes touching `PKGBUILD-git` or
companion files plus `workflow_dispatch`. `pkgver` is computed at
build time by the PKGBUILD's `pkgver()` function, so this workflow
is not tied to release tags.
- Tag-triggered `package-*` release-build workflows remain
untouched.
- macOS UDP receive path now batches up to 32 datagrams per kernel
wakeup via `recvmsg_x(2)`, matching the Linux `recvmmsg(2)`
amortization shape introduced in v0.3.0. Previously macOS fell
@@ -272,6 +300,22 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
and the connected-UDP drain no longer busy-spins on a poll error
(#106).
#### Transports & config
- Static host aliases in `/etc/fips/hosts` now hot-reload on mtime
change instead of only at daemon startup, so `fipsctl`/`fipstop`
display names reflect edits without a restart. The peer ACL and host
map both reload once per node tick through a new lock-free
`Reloadable` snapshot.
- Sidecar example (`examples/sidecar-nostr-relay`): `udp.mtu` is now
overridable via the `FIPS_UDP_MTU` environment variable, defaulting to
1472 (preserving prior behavior). Plumbed through `docker-compose.yml`
and documented in the README env-var table. Annotated the static-CI
node template `mtu: 1472` literal with the same Docker-bridge
rationale and a pointer at the daemon's 1280 default.
#### Packaging & deployment
- The Debian package no longer ships `/etc/fips/fips.yaml` as a dpkg
conf-file. The default configuration is installed as an example at
`/usr/share/fips/fips.yaml.example`, and `postinst` seeds
@@ -283,6 +327,36 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
example is placed under `/usr/share/fips`, deliberately outside
`/usr/share/doc`, which minimal and container installs path-exclude
(so the install-time seed source is never dropped).
- openwrt: the `.apk` package now defaults `ethernet.wan` to the
OpenWrt 25 DSA port name `wan`; the `.ipk` package keeps `eth0` for
OpenWrt 24 and earlier.
#### CI & test-harness reliability
- CI and release-publish workflows hardened:
- `ci.yml` declares a top-level `concurrency` block keyed on
`(workflow, ref)` with `cancel-in-progress: true`. Force-pushes
and rapid successive pushes to the same ref now retire any
in-flight run rather than letting superseded and current-tip runs
both burn runner minutes.
- `aur-publish.yml` rewritten to fetch the upstream source tarball
and compute its `b2sum` in CI, then patch `pkgver` and the
`b2sums` SKIP placeholder in `PKGBUILD` in-place. Previously
`updpkgsums: true` downloaded the tarball into the AUR working
tree, where it was rejected by AUR's 488 KiB max-blob hook —
silently no-op'ing the v0.3.0 stable AUR push. `fips.sysusers` /
`fips.tmpfiles` asset b2sums are recomputed in the same step to
stay in sync with the local files. `workflow_dispatch` gains a
tag input so historical release tags can be re-published
manually, and `continue-on-error: true` is dropped so future
regressions surface in CI.
- New `aur-publish-git.yml` workflow for the `fips-git` VCS
PKGBUILD, triggered on master pushes touching `PKGBUILD-git` or
companion files plus `workflow_dispatch`. `pkgver` is computed at
build time by the PKGBUILD's `pkgver()` function, so this workflow
is not tied to release tags.
- Tag-triggered `package-*` release-build workflows remain
untouched.
- Local and GitHub CI integration coverage brought into parity, and
the Rust toolchain selection given a single source of truth:
- The `admission-cap` integration suite, previously run only by
@@ -305,39 +379,70 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
`-D warnings` is newly imposed. The OpenWrt nightly Tier-3 leg
keeps `@nightly`.
#### Docs & contributor tooling
- Overhauled `CONTRIBUTING.md`: replaced generic Rust-template framing
with a FIPS-specific entry point covering the four-layer
architecture, branch model and PR-target selection, structured bug
reporting, scope discipline and local-CI requirements, an AI coding
assistant policy, and project communication channels. Added
`docs/branching.md` as the long-form companion covering the release
workflow, version conventions, and merge-direction rationale.
### Fixed
- The Tor transport now increments its `connect_refused` statistic (the
"Refused" line in fipstop) when a SOCKS5 connection is actively
refused, instead of recording every connect failure as a generic
SOCKS5 error. The counter previously stayed at zero.
- MMP sender metrics now ignore duplicate or regressed receiver reports
before updating RTT, loss, goodput, or ETX. Receiver reports also
suppress timestamp echo when dwell time overflows, so stale reports
cannot inflate SRTT.
- Six discovery counters (`req_decode_error`, `req_duplicate`,
`req_ttl_exhausted`, `resp_decode_error`, `resp_identity_miss`,
`resp_proof_failed`) no longer double-count. Each was incremented both
by a direct bump and again through the typed reject dispatch; the
redundant direct increment is removed, so each counts once per event.
- Six bloom counters (`decode_error`, `invalid`, `non_v1`,
`unknown_peer`, `stale`, `fill_exceeded`) no longer double-count. Each
was incremented both by a direct bump and again through the typed
reject dispatch; the redundant direct increment is removed, so each
counts once per event.
- Five forwarding reject packet counters (`decode_error_packets`,
`ttl_exhausted_packets`, `drop_no_route_packets`,
`drop_mtu_exceeded_packets`, `drop_send_error_packets`) no longer
double-count. Each was incremented both by the byte-aware outcome
recorder and again through the typed reject dispatch; the two calls are
collapsed into a single byte-aware reject entry point, so packets and
bytes each count once per event.
#### FMP/FSP rekey reliability
- FMP link-layer rekey is now reliable under packet loss, bringing it up
to the FSP session layer's rekey discipline. The rekey msg1
retransmission driver was previously uncapped and never abandoned, so a
rekey that never completed resent msg1 forever; it now uses a bounded
retransmission budget (`handshake_max_resends` with exponential
backoff) and abandons the rekey cycle cleanly once the budget is
exhausted, mirroring the FSP rekey msg3 driver. With the cap in place
the link-dead heartbeat is rekey-aware: `check_link_heartbeats` no
longer reaps a link that is still actively carrying rekey-handshake
traffic, while a genuinely dead link is still reaped once the budget
abandons. At the K-bit cutover the receiver now authenticates an
inbound frame against the pending session before promoting it, instead
of promoting on the bare header K-bit; under jitter a node could
otherwise promote a stale pending session, leaving the two endpoints on
different keys and silently dropping traffic until the link died — the
same failure class already closed on FSP, now closed on FMP.
- FSP session rekey is now hitless under packet loss and reordering.
Previously, a rekey could leave the two endpoints holding different
key sets for a brief window — if a handshake message was lost in
transit one side rotated keys while the other did not, and traffic
sealed in one key epoch reached a peer still on the other epoch and
failed to decrypt, producing bursts of AEAD decryption failures and
dropped connectivity until a later rekey reconverged the pair. The
receive path now trial-decrypts each frame against every live key
epoch (current, pending, and the draining previous session) for the
duration of the rekey transition, so no rotation ordering and no
packet reordering can cause a decryption failure. The previous-epoch
slot is retained as long as the peer keeps using it, with its drain
deadline anchored on the last frame the peer authenticates against
it rather than a fixed wall-clock timer, so a peer that did not
receive the new keys is not stranded by a silent permanent decrypt
failure. The lost-handshake case is closed by retransmitting the
third rekey handshake message until the peer is confirmed on the
new keys, with a bounded retry budget after which the rekey cycle
is cleanly abandoned and retried. There are no FSP decryption
failures across a rekey under lossy, jittery links.
- ±15s symmetric jitter is applied per session to the FMP and FSP rekey
timer trigger, eliminating the steady-state dual-initiation race in
symmetric-start meshes (previously the smaller-NodeAddr tie-breaker
resolved correctness only after every cycle's collision).
`node.rekey.after_secs` becomes the nominal interval rather than a
floor; the mean is preserved.
- A stale FSP (session-layer) session is now cleared when a peer
restart is detected during FMP rekey or cross-connection promotion.
Previously the old session could linger after the peer came back
with a new startup epoch, leaving the session-layer map out of sync
with the freshly promoted peer.
#### NAT traversal / Nostr discovery
- Two nodes that each `auto_connect` to the other no longer stall their
Nostr-mediated NAT-traversal handshake. Each side ran both an
initiator and a responder traversal session, binding a separate UDP
@@ -350,43 +455,55 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
its own NodeAddr is smaller — so one matching socket pair survives on
both ends and the peer's redundant initiator times out harmlessly.
One-sided (asymmetric) `auto_connect` has no co-active initiator and is
never suppressed, so connectivity is preserved. (Distinct from the
cross-init adoption tie-breaker below, which dedups two simultaneous
punches but does not stop each node from running redundant
initiator + responder sessions.)
- FMP link-layer rekey is now reliable under packet loss, bringing it up
to the FSP session layer's rekey discipline:
- The rekey msg1 retransmission driver was uncapped and never
abandoned, so a rekey that never completed resent msg1 forever. It
now uses a bounded retransmission budget (`handshake_max_resends`
with exponential backoff) and abandons the rekey cycle cleanly once
the budget is exhausted, mirroring the FSP rekey msg3 driver. With
the cap in place the link-dead heartbeat is rekey-aware:
`check_link_heartbeats` no longer reaps a link that is still
actively carrying rekey-handshake traffic, while a genuinely dead
link is still reaped once the budget abandons.
- At the K-bit cutover the receiver now authenticates an inbound frame
against the pending session before promoting it, instead of
promoting on the bare header K-bit. Under jitter a node could
otherwise promote a stale pending session, leaving the two endpoints
on different keys and silently dropping traffic until the link died
— the same failure class already closed on FSP, now closed on FMP.
- Node-level multi-node tests no longer flake under parallel CPU load.
They previously delivered handshake packets over real localhost UDP,
whose kernel receive buffer could overflow and drop a packet when many
tests ran concurrently, panicking the large-network convergence tests.
A `cfg(test)`-only loopback `TransportHandle` variant now delivers
packets directly between nodes over an unbounded in-process channel, so
there is no socket buffer to overflow, and the previously-quarantined
large-network tests run in the default suite again. The shipping daemon
build is unaffected (the variant is test-gated).
- Integration suites that wait for the mesh to converge no longer
false-fail under concurrent CI load. The rekey, static-mesh, and
sidecar suites replace a fixed wall-clock baseline timeout (and a blind
sleep) with a progress-aware wait that polls the suite's own pairwise
pings, returns as soon as every pair is reachable, extends its deadline
while the reachable-pair count is still climbing, and gives up only
when progress stalls.
never suppressed, so connectivity is preserved.
- NAT-traversal cross-init adoption is now deterministic under
simultaneous dual-initiation. Previously, when two peers'
Nostr-mediated UDP punches completed within the same scheduling
window, each side's bootstrap-completion event arrived with an
in-flight handshake already recorded against the other peer (each
side had received an inbound msg1 from the other's pre-punch
outbound attempt). The deduplication skip then fired on both
sides, neither installed the fresh traversal socket as canonical,
and the 45-second peer-adoption budget expired with both nodes
stuck waiting for an adoption that never happened. The handler now
applies the same deterministic NodeAddr tie-breaker the codebase
already uses for rekey dual-initiation and cross-connection
resolution: the smaller NodeAddr wins as adopter, tears down its
in-flight handshake state, and proceeds with adoption; the larger
NodeAddr keeps the skip semantics, and its in-flight outbound is
reconciled by the cross-connection logic when the winner's fresh
msg1 arrives over the adopted socket. The dual cross-init stall is
eliminated; cross-init NAT-traversal completes in well under a
second even under host CPU contention.
- Nostr-discovered NAT-traversal events (`BootstrapEvent::Established`
and `BootstrapEvent::Failed`) for peers that are already connected
or actively handshaking are now short-circuited at the
`poll_nostr_discovery` dispatch sites before any cooldown
bookkeeping or fallback retry scheduling runs. Stale `Failed` events
previously poisoned the per-peer failure-state cooldown of healthy
peers and could trigger redundant retraversal attempts via
`schedule_retry` / `try_peer_addresses`; stale `Established`
handoffs could attempt to adopt a second socket against a live
connection. A defense-in-depth guard was added to
`adopt_established_traversal` so the same invariant holds if a
future caller bypasses the outer dispatch check. As a side benefit,
narrows a cooldown-poisoning vector previously available to an
attacker injecting stale failure events for an active peer.
- Nostr discovery now filters unroutable direct UDP/TCP advert
endpoints. Publisher and validator retain only endpoints that parse as
concrete socket addresses with routable IPs and nonzero ports;
`udp:nat` rendezvous endpoints and Tor endpoints pass through
unchanged. Adverts that collapse to zero usable endpoints after
filtering are rejected with a clear "missing publicly routable
endpoints" error. Before this change, misconfigured nodes could
publish RFC1918, loopback, link-local, CGNAT 100.64/10, IPv6 ULA,
or IPv6 link-local endpoints into Nostr discovery, and consumers
would cache and dial them; in mixed LAN/VPN/NAT environments, that
could prefer a misleading one-way private path over the intended
`udp:nat` bootstrap.
#### Admission / peer caps
- TCP and Tor `max_inbound_connections` admission cap is now compared
against the per-direction inbound count (`pool_inbound`) rather than
the combined pool size. Outbound connect-on-send connections share
@@ -415,10 +532,13 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
reconnect / cross-connection bypass for known peers still fires) and
before index allocation + Msg2 wire send. The late cap check inside
`promote_connection` is intentionally retained as
defense-in-depth. Wire savings observed in a 45 s ops tcpdump at
defense-in-depth. Wire savings observed in a 45 s tcpdump at
saturation: ~3.6 cap-denials/s × Msg2 (~104 B + AEAD compute) each.
Bigger win is cleaner peer-side semantics — no fake-completed
handshake whose subsequent data frames fail decryption on this side.
#### Spanning-tree / mesh-size / routing
- The mesh-size estimator (`compute_mesh_size`) no longer over-counts
under filter overlap. It previously summed the per-filter cardinality
of the parent and each child filter, which assumes the filters are
@@ -429,12 +549,11 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
nearly-but-not-exactly doubling during rebalancing). The estimator now
computes the cardinality of the OR-union over self plus every
connected peer's inbound filter, dropping the parent/child tree gating
entirely.
OR is idempotent, so any overlap is deduplicated — the result equals
the old sum in the disjoint case, stays correct under overlap, damps
the parent-switch flap, and removes the estimate's dependence on
tree-declaration cache freshness. The per-peer 500 ms rate-limiter and
overall recompute cadence are unchanged.
entirely. OR is idempotent, so any overlap is deduplicated — the
result equals the old sum in the disjoint case, stays correct under
overlap, damps the parent-switch flap, and removes the estimate's
dependence on tree-declaration cache freshness. The per-peer 500 ms
rate-limiter and overall recompute cadence are unchanged.
- Spanning-tree state distribution is now eventually-consistent.
Previously every `send_tree_announce_to_all` call site fired only
on a local state-change event (parent switch, self-root promotion,
@@ -472,7 +591,20 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
that peer, provoking the better-rooted peer to re-push its real
position immediately. The echo fires only in that one direction and is
bounded by the existing per-peer rate limiter.
- Coord cache invalidation made surgical at parent-position-change
and root-change sites. Replaces the previous unconditional
`CoordCache::clear()` calls with two targeted methods:
`invalidate_via_node(node_addr)` (drops entries whose cached
ancestry contains the changed node, used at parent-switch /
become-root / loop-detection sites) and `invalidate_other_roots`
(drops entries from a different tree, used at root-change sites).
The previous global flush left `find_next_hop` returning `None`
for every non-direct-peer destination after every parent switch
until the cache passively re-warmed; surgical invalidation
preserves entries that remain correct across the topology change.
Peer-removal retains the original "no invalidation" behavior
(`find_next_hop` already recomputes against the current peer set
every call, and Discovery handles "no route" on demand).
- `rx_loop` tick-arm stall under convergence-phase mesh pressure
is eliminated. Previously, the tick body's per-peer `check_*`
loops (heartbeats, bloom announces, MMP reports, tree announces)
@@ -504,45 +636,133 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
synchronous connect (e.g., explicit operator-driven
`fipsctl connect`); the tick path just no longer trips it.
- NAT-traversal cross-init adoption is now deterministic under
simultaneous dual-initiation. Previously, when two peers'
Nostr-mediated UDP punches completed within the same scheduling
window, each side's bootstrap-completion event arrived with an
in-flight handshake already recorded against the other peer (each
side had received an inbound msg1 from the other's pre-punch
outbound attempt). The deduplication skip then fired on both
sides, neither installed the fresh traversal socket as canonical,
and the 45-second peer-adoption budget expired with both nodes
stuck waiting for an adoption that never happened. The handler now
applies the same deterministic NodeAddr tie-breaker the codebase
already uses for rekey dual-initiation and cross-connection
resolution: the smaller NodeAddr wins as adopter, tears down its
in-flight handshake state, and proceeds with adoption; the larger
NodeAddr keeps the skip semantics, and its in-flight outbound is
reconciled by the cross-connection logic when the winner's fresh
msg1 arrives over the adopted socket. The dual cross-init stall is
eliminated; cross-init NAT-traversal completes in well under a
second even under host CPU contention.
- FSP session rekey is now hitless under packet loss and reordering.
Previously, a rekey could leave the two endpoints holding different
key sets for a brief window — if a handshake message was lost in
transit one side rotated keys while the other did not, and traffic
sealed in one key epoch reached a peer still on the other epoch and
failed to decrypt, producing bursts of AEAD decryption failures and
dropped connectivity until a later rekey reconverged the pair. The
receive path now trial-decrypts each frame against every live key
epoch (current, pending, and the draining previous session) for the
duration of the rekey transition, so no rotation ordering and no
packet reordering can cause a decryption failure. The previous-epoch
slot is retained as long as the peer keeps using it, with its drain
deadline anchored on the last frame the peer authenticates against
it rather than a fixed wall-clock timer, so a peer that did not
receive the new keys is not stranded by a silent permanent decrypt
failure. The lost-handshake case is closed by retransmitting the
third rekey handshake message until the peer is confirmed on the
new keys, with a bounded retry budget after which the rekey cycle
is cleanly abandoned and retried. There are no FSP decryption
failures across a rekey under lossy, jittery links.
#### Data-plane / metrics / observability
- The Tor transport now increments its `connect_refused` statistic (the
"Refused" line in fipstop) when a SOCKS5 connection is actively
refused, instead of recording every connect failure as a generic
SOCKS5 error. The counter previously stayed at zero.
- MMP sender metrics now ignore duplicate or regressed receiver reports
before updating RTT, loss, goodput, or ETX. Receiver reports also
suppress timestamp echo when dwell time overflows, so stale reports
cannot inflate SRTT.
- Reject-reason counters no longer double-count now that the rollout's
interim direct increments are removed. Six discovery counters
(`req_decode_error`, `req_duplicate`, `req_ttl_exhausted`,
`resp_decode_error`, `resp_identity_miss`, `resp_proof_failed`), six
bloom counters (`decode_error`, `invalid`, `non_v1`, `unknown_peer`,
`stale`, `fill_exceeded`), and five forwarding reject packet counters
(`decode_error_packets`, `ttl_exhausted_packets`,
`drop_no_route_packets`, `drop_mtu_exceeded_packets`,
`drop_send_error_packets`) were each incremented both by a direct bump
and again through the typed reject dispatch. The redundant direct
increments are removed — for the forwarding family the two calls are
collapsed into a single byte-aware reject entry point — so each counter
(and, for forwarding, its byte tally) counts once per event.
- Transport-layer mutex poisoning no longer cascades. Ten
`Mutex::lock().unwrap()` sites across the UDP, BLE, and Ethernet
transports would turn a single panic (poisoning the mutex) into a
cascade of panics on every subsequent lock. Each is replaced with
`lock().unwrap_or_else(|e| e.into_inner())`, recovering the guarded
data with no new dependency and no call-graph change; four
`local_addr.unwrap()` calls on the UDP start/adopt paths get a
provably-safe sentinel fallback. The critical sections are short,
locally-scoped, and not reachable from peer input, so this is
robustness hardening, not a remotely-triggerable fix.
#### Peer lifecycle / gateway
- A manual `fipsctl disconnect` now notifies the peer so teardown is
symmetric. Previously a manual disconnect tore down only the local
side and sent the peer nothing, so the peer kept its session and never
re-emitted its tree and filter announcements; on reconnect it was
never re-adopted as a child and its bloom filter was never recorded.
The local side now sends the disconnected peer a scoped `Disconnect`
(the same message graceful shutdown sends), so both ends tear down and
re-handshake cleanly on the next connection.
- `fips-gateway` no longer drops long-lived or DNS-cached client
mappings while traffic is still flowing. The virtual-IP pool's TTL
clock advanced only on DNS re-query, never on traffic, and the mapping
TTL is wired equal to the DNS TTL, so an in-use mapping was forced to
drain at TTL and reclaimed at the first zero-conntrack tick — breaking
long-lived, bursty, or DNS-cached clients. The tick now refreshes the
mapping's last-referenced time whenever conntrack reports active
sessions, and recovers a draining mapping to active (with a fresh
grace window) when traffic resumes; only genuinely idle mappings
drain.
#### macOS self-traffic / resolver
- Self-addressed mesh traffic is now delivered locally on macOS instead
of being dropped, for both `ping6` and full TCP/UDP. The point-to-point
`utun` interface egresses self-addressed traffic into the daemon, which
previously pushed it onto the mesh outbound path where it was dropped
for lack of a route to self; such packets are now hairpinned back to
the TUN for inbound delivery. macOS first routes self-addressed packets
as loopback (a `LOCAL` route via `lo0`), which leaves their transport
TX checksum offloaded and unfinished, so re-injecting them verbatim
made the local stack drop every segment whose checksum MSS clamping did
not happen to rewrite (the SYN and SYN-ACK got through, but the bare
ACK, data, and FIN were dropped, so connections to a node's own
`<npub>.fips` service half-opened and hung). The hairpin path now
recomputes the TCP/UDP checksum before re-injection, so full
self-connections — not just `ping6` — to a node's own `<npub>.fips`
address work. Linux was unaffected (the kernel already loops
self-traffic via `lo`). (#117)
- macOS `.fips` name resolution now works on a fresh install: the
shipped resolver shim points at `::1`, matching the daemon's default
IPv6 DNS listener, instead of `127.0.0.1`. The mismatched shim
(`nameserver 127.0.0.1` while the daemon listens on `::1`) broke
`getaddrinfo` for `.fips` on every macOS install since the resolver
was introduced.
#### CI & test-harness reliability
- Node-level multi-node tests no longer flake under parallel CPU load.
They previously delivered handshake packets over real localhost UDP,
whose kernel receive buffer could overflow and drop a packet when many
tests ran concurrently, panicking the large-network convergence tests.
A `cfg(test)`-only loopback `TransportHandle` variant now delivers
packets directly between nodes over an unbounded in-process channel, so
there is no socket buffer to overflow, and the previously-quarantined
large-network tests run in the default suite again. The shipping daemon
build is unaffected (the variant is test-gated).
- Integration suites that wait for the mesh to converge no longer
false-fail under concurrent CI load. The rekey, static-mesh, and
sidecar suites replace a fixed wall-clock baseline timeout (and a blind
sleep) with a progress-aware wait that polls the suite's own pairwise
pings, returns as soon as every pair is reachable, extends its deadline
while the reachable-pair count is still climbing, and gives up only
when progress stalls.
- Rekey integration test (`testing/static/scripts/rekey-test.sh`) no
longer false-fails on GitHub runners under packet loss and CPU
contention. Phase 1, Phase 3, and Phase 5 strict per-pair pings retry
up to 4 attempts (configurable via `MAX_PING_ATTEMPTS` /
`PING_RETRY_DELAY`) — under 1% per-direction loss, single-shot 20-pair
ping_all misses ~33% per phase from ICMP noise alone, and the
4-attempt retry brings that floor to ~3.2e-6 per phase; the
`wait_for_full_baseline` convergence loop stays single-shot so retries
there cannot conflate transient ping loss with still-converging routing
state. Phase 1 baseline-convergence headroom is bumped from 36s to 60s
to eliminate the intermittent Phase 1 timeout that previously required
a `gh run rerun --failed`, and a post-second-rekey settle window is
added in Phase 5 (mirroring Phase 3's 12-second pattern) to close the
post-rekey per-pair-ping flake from convergence exceeding the per-ping
5-second timeout. Test scaffold only; no daemon code changes, and the
success path is unchanged because the wait loops return as soon as all
20 pairs converge.
- ACL-allowlist integration test (`testing/acl-allowlist/test.sh`):
converted `assert_log_contains` from a one-shot `docker logs | grep`
snapshot into a bounded poll with the same wait-with-timeout shape
as `wait_for_peers_exact`. Absorbs the millisecond-to-second
variance in the XX-handshake cross-connection tie-breaker: the
inbound-handshake-context rejection can land tens of milliseconds
after the test's previous one-shot grep gave up, producing a
pre-existing flake on CI. Success-path cost is unchanged — the helper
returns as soon as the pattern appears.
#### Packaging & deployment
- AUR packaging: the `fips` and `fips-git` PKGBUILDs now install the
`fips-dns-setup` and `fips-dns-teardown` helpers into
`/usr/lib/fips/`, matching the Debian package. The AUR `package()`
@@ -564,113 +784,9 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
the arch-named file is present and carries a SHA-256 integrity
chain from the build runner through to `gh release upload`, so a
recurrence fails CI instead of publishing.
- Nostr discovery: filter unroutable direct UDP/TCP advert endpoints.
Publisher and validator now retain only endpoints that parse as
concrete socket addresses with routable IPs and nonzero ports.
`udp:nat` rendezvous endpoints and Tor endpoints pass through
unchanged. Adverts that collapse to zero usable endpoints after
filtering are rejected with a clear "missing publicly routable
endpoints" error. Before this change, misconfigured nodes could
publish RFC1918, loopback, link-local, CGNAT 100.64/10, IPv6 ULA,
or IPv6 link-local endpoints into Nostr discovery, and consumers
would cache and dial them; in mixed LAN/VPN/NAT environments, that
could prefer a misleading one-way private path over the intended
`udp:nat` bootstrap.
- Coord cache invalidation made surgical at parent-position-change
and root-change sites. Replaces the previous unconditional
`CoordCache::clear()` calls with two targeted methods:
`invalidate_via_node(node_addr)` (drops entries whose cached
ancestry contains the changed node, used at parent-switch /
become-root / loop-detection sites) and `invalidate_other_roots`
(drops entries from a different tree, used at root-change sites).
The previous global flush left `find_next_hop` returning `None`
for every non-direct-peer destination after every parent switch
until the cache passively re-warmed; surgical invalidation
preserves entries that remain correct across the topology change.
Peer-removal retains the original "no invalidation" behavior
(`find_next_hop` already recomputes against the current peer set
every call, and Discovery handles "no route" on demand).
- Rekey integration test (`testing/static/scripts/rekey-test.sh`):
Phase 1, Phase 3, and Phase 5 strict per-pair pings now retry up
to 4 attempts (configurable via `MAX_PING_ATTEMPTS` /
`PING_RETRY_DELAY`). Under low-level packet loss (1% per
direction), single-shot 20-pair ping_all misses with probability
~33% per phase from ICMP noise alone, masking the routing-state
signal the asserts target. The 4-attempt retry brings that floor
to ~3.2e-6 per phase. The `wait_for_full_baseline` convergence
loop itself stays single-shot — retries there would conflate
transient ping loss with still-converging routing state. Test
scaffold only; no daemon code changes.
- Apply ±15s symmetric jitter per session to the FMP and FSP rekey
timer trigger. Eliminates the steady-state dual-initiation race
in symmetric-start meshes; previously the smaller-NodeAddr
tie-breaker resolved correctness after every cycle's collision.
`node.rekey.after_secs` becomes the nominal interval rather than
a floor; mean is preserved.
- Rekey integration test (`testing/static/scripts/rekey-test.sh`):
bumped Phase 1 baseline-convergence headroom from 36s to 60s.
Eliminates the intermittent GitHub-runner Phase 1 timeout that
previously required `gh run rerun --failed`. Cost on the success
path is unchanged because the wait loop returns as soon as all 20
pairs converge.
- Rekey integration test (`testing/static/scripts/rekey-test.sh`):
added a post-second-rekey settle window in Phase 5, mirroring
Phase 3's existing 12-second pattern. Closes the intermittent
GitHub-runner Phase 5 per-pair-ping flake caused by post-rekey
routing convergence exceeding the per-ping 5-second timeout under
runner CPU contention. Cost on the success path is a fixed 12s per
suite run.
- ACL-allowlist integration test (`testing/acl-allowlist/test.sh`):
converted `assert_log_contains` from a one-shot `docker logs | grep`
snapshot into a bounded poll with the same wait-with-timeout shape
as `wait_for_peers_exact`. Absorbs the millisecond-to-second
variance in the XX-handshake cross-connection tie-breaker: the
inbound-handshake-context rejection can land tens of milliseconds
after the test's previous one-shot grep gave up, producing a
pre-existing flake on next-branch CI. Success-path cost is
unchanged — the helper returns as soon as the pattern appears.
- Nostr-discovered NAT-traversal events (`BootstrapEvent::Established`
and `BootstrapEvent::Failed`) for peers that are already connected
or actively handshaking are now short-circuited at the
`poll_nostr_discovery` dispatch sites before any cooldown
bookkeeping or fallback retry scheduling runs. Stale `Failed` events
previously poisoned the per-peer failure-state cooldown of healthy
peers and could trigger redundant retraversal attempts via
`schedule_retry` / `try_peer_addresses`; stale `Established`
handoffs could attempt to adopt a second socket against a live
connection. A defense-in-depth guard was added to
`adopt_established_traversal` so the same invariant holds if a
future caller bypasses the outer dispatch check. As a side benefit,
narrows a cooldown-poisoning vector previously available to an
attacker injecting stale failure events for an active peer.
- A manual `fipsctl disconnect` now notifies the peer so teardown is
symmetric. Previously a manual disconnect tore down only the local
side and sent the peer nothing, so the peer kept its session and never
re-emitted its tree and filter announcements; on reconnect it was
never re-adopted as a child and its bloom filter was never recorded.
The local side now sends the disconnected peer a scoped `Disconnect`
(the same message graceful shutdown sends), so both ends tear down and
re-handshake cleanly on the next connection.
- `fips-gateway` no longer drops long-lived or DNS-cached client
mappings while traffic is still flowing. The virtual-IP pool's TTL
clock advanced only on DNS re-query, never on traffic, and the mapping
TTL is wired equal to the DNS TTL, so an in-use mapping was forced to
drain at TTL and reclaimed at the first zero-conntrack tick — breaking
long-lived, bursty, or DNS-cached clients. The tick now refreshes the
mapping's last-referenced time whenever conntrack reports active
sessions, and recovers a draining mapping to active (with a fresh
grace window) when traffic resumes; only genuinely idle mappings
drain.
- Transport-layer mutex poisoning no longer cascades. Ten
`Mutex::lock().unwrap()` sites across the UDP, BLE, and Ethernet
transports would turn a single panic (poisoning the mutex) into a
cascade of panics on every subsequent lock. Each is replaced with
`lock().unwrap_or_else(|e| e.into_inner())`, recovering the guarded
data with no new dependency and no call-graph change; four
`local_addr.unwrap()` calls on the UDP start/adopt paths get a
provably-safe sentinel fallback. The critical sections are short,
locally-scoped, and not reachable from peer input, so this is
robustness hardening, not a remotely-triggerable fix.
#### fipstop
- `fipstop` no longer renders a garbled screen on startup or leaves
stray bytes on quit, most visible over SSH and inside tmux. Startup
forces a full repaint (`terminal.clear()`) before the first draw so
@@ -678,12 +794,6 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
stdin-poll thread a stop flag and joins it before restoring the
terminal, so post-raw-mode keystrokes or terminal query responses no
longer echo onto the restored screen.
- macOS `.fips` name resolution now works on a fresh install: the
shipped resolver shim points at `::1`, matching the daemon's default
IPv6 DNS listener, instead of `127.0.0.1`. The mismatched shim
(`nameserver 127.0.0.1` while the daemon listens on `::1`) broke
`getaddrinfo` for `.fips` on every macOS install since the resolver
was introduced.
## [0.3.0] - 2026-05-11
+4
View File
@@ -42,6 +42,10 @@ and fail without it. BLE-capable builds additionally need `bluez`,
`libdbus-1-dev`, and `pkg-config` installed; the default build picks
up BLE if those are present and skips it cleanly if not.
On Nix, `nix develop` provides the pinned toolchain and all of these
build prerequisites without any manual install; see the Nix / NixOS
section of [packaging/README.md](packaging/README.md).
For multi-node integration runs, Docker is required. The harness
under [testing/](testing/) starts containerized topologies and
exercises real mesh behavior; see [testing/README.md](testing/README.md)
Generated
+1 -1
View File
@@ -1074,7 +1074,7 @@ checksum = "9844ddc3a6e533d62bba727eb6c28b5d360921d5175e9ff0f1e621a5c590a4d5"
[[package]]
name = "fips"
version = "0.4.0-dev"
version = "0.5.0-dev"
dependencies = [
"arc-swap",
"bech32",
+1 -1
View File
@@ -1,6 +1,6 @@
[package]
name = "fips"
version = "0.4.0-dev"
version = "0.5.0-dev"
edition = "2024"
description = "A distributed, decentralized network routing protocol for mesh nodes connecting over arbitrary transports"
license = "MIT"
+29 -19
View File
@@ -3,7 +3,7 @@
![banner](docs/logos/fips_banner.png)
[![License: MIT](https://img.shields.io/badge/license-MIT-blue.svg)](LICENSE)
[![Rust](https://img.shields.io/badge/rust-1.85%2B-orange.svg)](https://www.rust-lang.org/)
[![Status](https://img.shields.io/badge/status-v0.4.0--dev-green.svg)](#status--roadmap)
[![Status](https://img.shields.io/badge/status-v0.5.0--dev-green.svg)](#status--roadmap)
A self-organizing encrypted mesh network built on Nostr identities,
capable of operating over arbitrary transports without central
@@ -98,9 +98,9 @@ This installs the daemon, CLI tools (`fipsctl`, `fipstop`), the
optional `fips-gateway` service, systemd units, and a default
`/etc/fips/fips.yaml` you can edit before starting.
For macOS, Windows, OpenWrt, the systemd tarball, or a from-source
build, see [docs/getting-started.md](docs/getting-started.md) for
the full multi-platform installation guide.
For macOS, Windows, OpenWrt, the systemd tarball, a Nix flake, or a
from-source build, see [docs/getting-started.md](docs/getting-started.md)
for the full multi-platform installation guide.
To join a live mesh and reach your first peer, follow the new-user
tutorial progression starting at
@@ -112,17 +112,19 @@ tutorial progression starting at
cargo build --release
```
Requires Rust 1.94.1+ (edition 2024). Linux, macOS, and Windows are
supported; transport availability varies by platform.
Requires Rust 1.94.1+ (edition 2024). Linux, macOS, and Windows run as
standalone daemons; Android is supported as an embedded library (the host
app owns the TUN, e.g. a `VpnService`). Transport availability varies by
platform.
| Transport | Linux | macOS | Windows | OpenWrt |
|-----------|:-----:|:-----:|:-------:|:-------:|
| UDP | ✅ | ✅ | ✅ | ✅ |
| TCP | ✅ | ✅ | ✅ | ✅ |
| Ethernet | ✅ | ✅ | ❌ | ✅ |
| Tor | ✅ | ✅ | ✅ | ✅ |
| Nym | ✅ | ✅ | ✅ | ❌ |
| BLE | ✅ | ❌ | ❌ | ❌ |
| Transport | Linux | macOS | Windows | Android | OpenWrt |
|-----------|:-----:|:-----:|:-------:|:-------:|:-------:|
| UDP | ✅ | ✅ | ✅ | ✅ | ✅ |
| TCP | ✅ | ✅ | ✅ | ✅ | ✅ |
| Ethernet | ✅ | ✅ | ❌ | ❌ | ✅ |
| Tor | ✅ | ✅ | ✅ | ❌ | ✅ |
| Nym | ✅ | ✅ | ✅ | ❌ | ❌ |
| BLE | ✅ | ❌ | ❌ | ✅ | ❌ |
On Linux, a source build requires `libclang` — the LAN gateway's
nftables bindings are generated by `bindgen` at build time, which
@@ -143,6 +145,12 @@ Nym (mixnet) transport builds on all desktop platforms. The OpenWrt
availability on the target; it will flip to ✅ only if confirmed
buildable there.
Alternatively, the repo ships a [Nix flake](flake.nix): `nix develop`
drops you into a shell with the pinned toolchain and every build
prerequisite (libclang, dbus, pkg-config) already provided, and
`nix build .#fips` builds all four binaries with no host setup. See the
Nix / NixOS section of [packaging/README.md](packaging/README.md).
## Documentation
`docs/` is organised by reader purpose:
@@ -204,12 +212,14 @@ testing/ Docker-based integration test harnesses + chaos simulation
## Status & roadmap
FIPS is at **v0.4.0**. The core protocol works end-to-end over
FIPS is at **v0.5.0-dev** on the `master` branch.
[v0.4.0](https://github.com/jmcorgan/fips/releases/tag/v0.4.0) has
shipped; this development line continues the testing-and-polishing
track toward v0.5.0. The core protocol works end-to-end over
UDP, TCP, Ethernet, Tor, Nym, and Bluetooth on a global, public test
mesh of thousands of nodes. v0.4.0 builds on the v0.3.0 testing-and-polishing
track, adding the Nym mixnet transport and mDNS LAN discovery
alongside the existing Nostr-mediated peer discovery, UDP NAT
traversal, peer ACL, and packaging hardening. New wire-format work
mesh of thousands of nodes. v0.4.0 added the Nym mixnet transport and
mDNS LAN discovery alongside the existing Nostr-mediated peer discovery,
UDP NAT traversal, peer ACL, and packaging hardening. New wire-format work
continues to be staged on the `next` branch for the subsequent
release line.
+44 -2
View File
@@ -1,6 +1,6 @@
# FIPS v0.4.0
**Released**: 2026-06-DD (provisional)
**Released**: 2026-06-21 (provisional)
v0.4.0 is the throughput-and-observability release on the v0.3.x wire
format. It adds two new ways for nodes to find and reach each other (the
@@ -31,6 +31,11 @@ across a mesh in any order.
hot-path cost.
- Reworked `fipstop` TUI on a machine-verified render-snapshot base.
- Rekey is now hitless under loss and reordering in both directions.
- New packaging targets: an OpenWrt `.apk` for OpenWrt 25+ and a Nix
flake for reproducible from-source builds on Nix/NixOS.
- Six route-class transit counters partition forwarded traffic by its
tree relationship to the next hop, visible via `show_routing` and
`show_status`.
## What's new
@@ -115,6 +120,14 @@ returns a counter-only snapshot of every metric family. It is the
enabler for a Prometheus scraper that pulls node counters at no hot-path
cost.
Six **route-class transit counters** partition transit-forwarded packets
by their tree relationship to the chosen next hop — tree-up, tree-down,
tree-down-cross, cross-link descend, cross-link ascend, and direct-peer
— and the six classes sum to `forwarded_packets`. They surface through
`show_routing` and `show_status`, and the `fipstop` routing tab is
reorganized so its two columns separate own/endpoint traffic from
forwarded/transit traffic with the tree-down-cross line visually flagged.
### Reworked fipstop TUI
`fipstop` gets a rendering, navigation, and read-surface overhaul on a
@@ -155,6 +168,22 @@ The net operator takeaway: rekey completes cleanly without dropping
traffic, even on lossy or high-latency links, and the log no longer
cries wolf when a rekey gives up and retries.
### New packaging targets
- **OpenWrt `.apk`.** A new `.apk` package targets OpenWrt 25+, where
apk-tools is the mandatory package manager; the existing `.ipk`
continues to cover OpenWrt 24.x and earlier. It is built SDK-free,
reusing the `.ipk` cross-compile and installed-filesystem payload, and
releases publish `.apk` artifacts and checksums alongside `.ipk`. Like
the `.ipk`, the package is unsigned and installed with
`apk add --allow-untrusted`.
- **Nix flake.** A `flake.nix` at the project root builds all four
binaries (`fips`, `fipsctl`, `fips-gateway`, `fipstop`) from source on
Nix/NixOS, pinning the exact toolchain and wiring the native build
dependencies so no host setup is needed beyond Nix with flakes
enabled. It exposes `nix build`, `nix run`, a `nix develop` dev shell,
and `nix flake check`, with `flake.lock` committed for reproducibility.
## Behavior changes worth flagging
These affect operators on upgrade.
@@ -223,6 +252,15 @@ subset of fixes for behavior that shipped in v0.3.0.
advertising a strictly worse root echoes its own declaration back,
provoking the better-rooted peer to re-push its real position
immediately.
- **macOS self-connections work end to end (#117).** Traffic a macOS
node sends to its own `<npub>.fips` address is now delivered locally
for full TCP/UDP, not just `ping6`. The point-to-point `utun` egresses
self-addressed packets into the daemon with an unfinished transport
checksum (macOS offloads it on the `lo0` loopback route), so
re-injecting them verbatim made the local stack drop every segment the
MSS-clamp rewrite did not happen to fix and self-connections
half-opened and hung. The hairpin path now recomputes the TCP/UDP
checksum before re-injection. Linux was unaffected.
## Upgrade notes
@@ -253,10 +291,14 @@ Operator-actionable items moving from v0.3.0 to v0.4.0:
- **Arch Linux**: `fips` from the AUR.
- **macOS**: `.pkg` at the v0.4.0 release page.
- **Windows**: ZIP at the v0.4.0 release page.
- **OpenWrt**: `.ipk` at the v0.4.0 release page.
- **OpenWrt**: `.ipk` (OpenWrt 24.x and earlier) or `.apk` (OpenWrt 25+)
at the v0.4.0 release page.
- **From source**: `cargo build --release` from a checkout of the v0.4.0
tag (Rust 1.94.1 per `rust-toolchain.toml`; `libclang-dev` is a
required Linux build prerequisite).
- **Nix / NixOS**: `nix build .#fips` from a checkout of the v0.4.0 tag
builds the binaries from source with the pinned toolchain and no manual
prerequisites (see the Nix section of `packaging/README.md`).
The full per-commit changelog lives in
[`CHANGELOG.md`](../../CHANGELOG.md). Issues and discussion at
+9
View File
@@ -41,9 +41,18 @@ fn main() {
// satisfy libdbus-sys's pkg-config cross-compile requirement, and musl
// router targets don't run BlueZ by default anyway.
println!("cargo:rustc-check-cfg=cfg(bluer_available)");
// `ble_available` gates the platform-agnostic BLE transport module (pool,
// discovery, per-peer PSM, the generic `BleTransport`). The module compiles
// on every platform that has — or will have — a concrete `BleIo` backend;
// the backend is selected per-platform (BluerIo on linux-glibc, BluestIo on
// macOS, AndroidIo on Android, else the in-memory MockBleIo fallback).
println!("cargo:rustc-check-cfg=cfg(ble_available)");
let target_os = std::env::var("CARGO_CFG_TARGET_OS").unwrap_or_default();
let target_env = std::env::var("CARGO_CFG_TARGET_ENV").unwrap_or_default();
if target_os == "linux" && target_env != "musl" {
println!("cargo:rustc-cfg=bluer_available");
}
if matches!(target_os.as_str(), "linux" | "macos" | "android") {
println!("cargo:rustc-cfg=ble_available");
}
}
+28
View File
@@ -302,6 +302,34 @@ alternative — running under a dedicated unprivileged service
account with the capability granted on the binary — see
[../how-to/run-as-unprivileged-user.md](../how-to/run-as-unprivileged-user.md).
### App-Owned TUN (embedded hosts)
On platforms where FIPS is embedded rather than run as a daemon — notably
Android, where the `VpnService` owns the TUN fd and the app has no
`CAP_NET_ADMIN` — FIPS does not create `fips0` itself. Instead the embedder owns
the fd and exchanges IPv6 packet bytes with FIPS over channels.
`Node::enable_app_owned_tun()` sets this up. It is called after `Node::new` and
before `start()` (and before the node is moved into a background task), mirroring
`control_read_handle()`, and returns two app-side channel ends:
- **app → mesh** — the embedder pushes IPv6 packets read from its fd into
`app_outbound_tx`. These are drained by `run_rx_loop` into `handle_tun_outbound`
and routed exactly as the Reader Thread's output would be.
- **mesh → app** — inbound mesh traffic on port 256 is reconstructed and written
to the node's `tun_tx` (the same sink the Writer Thread reads); the embedder
pulls from `app_inbound_rx` and writes to its fd.
With the channels installed, `start()` skips system-TUN creation (it gates on
`tun_tx` being unset), so FIPS does no `CAP_NET_ADMIN` operations.
Because packets enter via `app_outbound_tx` rather than the Reader Thread, they
**bypass `handle_tun_packet`** — the `fd00::/8` destination filter, the ICMPv6
Destination Unreachable for off-mesh dests (see [Reader Thread](#reader-thread)),
and the [TUN-Side TCP MSS Clamping](#tun-side-tcp-mss-clamping). The embedder is
therefore responsible for routing only `fd00::/8` to its TUN (so only mesh-bound
packets arrive) and for clamping TCP MSS on outbound SYNs.
## Implementation Status
| Feature | Status |
+1 -1
View File
@@ -899,7 +899,7 @@ transitions through `Starting` to `Up` (operational). `stop()` moves to
| WiFi | **Implemented** (via Ethernet transport, infrastructure mode) | mac80211 translates 802.11↔802.3; broadcast beacons unreliable through APs |
| Tor | **Implemented** | Outbound SOCKS5, inbound via onion service, .onion and clearnet addressing |
| Nym | **Implemented** | Outbound-only SOCKS5 through nym-socks5-client, mixnet anonymity, IP/hostname addressing |
| BLE | **Implemented** (Linux/glibc only; experimental) | L2CAP CoC, ATT_MTU negotiation, per-link MTU; musl/macOS/Windows skip |
| BLE | **Implemented** (Linux/glibc and Android; experimental) | L2CAP CoC, ATT_MTU negotiation, per-link MTU; Linux via BlueZ, Android via the embedder radio bridge; macOS/Windows/musl skip |
| Radio | Future direction | Constrained MTU (51222 bytes) |
| Serial | Future direction | SLIP/COBS framing, point-to-point |
+17
View File
@@ -88,6 +88,23 @@ See [packaging/README.md](../packaging/README.md) for per-format
build details, cross-target options, and the full `make` target
list.
### With Nix (flake)
On Nix/NixOS, a [flake](../flake.nix) at the project root builds the
binaries from source with the pinned toolchain and no manual
prerequisite install:
```sh
nix build .#fips # all four binaries, into ./result/bin
nix develop # dev shell with the toolchain + build deps
```
This path produces binaries only — it does not run the installer, so
there are no systemd units, no `fips` group, and no default `fips.yaml`.
On NixOS, wire the daemon in through your system configuration using the
flake's `packages.<system>.fips` output instead. See the Nix / NixOS
section of [packaging/README.md](../packaging/README.md).
## What's installed and running
Here's what the installer leaves on your machine, what's
+1
View File
@@ -44,6 +44,7 @@ crate accordingly).
| -------- | -------------- |
| Linux (glibc) | Supported. |
| Linux (musl, OpenWrt) | Disabled at build time. |
| Android | Supported (native Android BLE, via the embedder's radio bridge). |
| macOS | Not supported. |
| Windows | Not supported. |
+44 -2
View File
@@ -1,6 +1,6 @@
# FIPS v0.4.0
**Released**: 2026-06-DD (provisional)
**Released**: 2026-06-21 (provisional)
v0.4.0 is the throughput-and-observability release on the v0.3.x wire
format. It adds two new ways for nodes to find and reach each other (the
@@ -31,6 +31,11 @@ across a mesh in any order.
hot-path cost.
- Reworked `fipstop` TUI on a machine-verified render-snapshot base.
- Rekey is now hitless under loss and reordering in both directions.
- New packaging targets: an OpenWrt `.apk` for OpenWrt 25+ and a Nix
flake for reproducible from-source builds on Nix/NixOS.
- Six route-class transit counters partition forwarded traffic by its
tree relationship to the next hop, visible via `show_routing` and
`show_status`.
## What's new
@@ -115,6 +120,14 @@ returns a counter-only snapshot of every metric family. It is the
enabler for a Prometheus scraper that pulls node counters at no hot-path
cost.
Six **route-class transit counters** partition transit-forwarded packets
by their tree relationship to the chosen next hop — tree-up, tree-down,
tree-down-cross, cross-link descend, cross-link ascend, and direct-peer
— and the six classes sum to `forwarded_packets`. They surface through
`show_routing` and `show_status`, and the `fipstop` routing tab is
reorganized so its two columns separate own/endpoint traffic from
forwarded/transit traffic with the tree-down-cross line visually flagged.
### Reworked fipstop TUI
`fipstop` gets a rendering, navigation, and read-surface overhaul on a
@@ -155,6 +168,22 @@ The net operator takeaway: rekey completes cleanly without dropping
traffic, even on lossy or high-latency links, and the log no longer
cries wolf when a rekey gives up and retries.
### New packaging targets
- **OpenWrt `.apk`.** A new `.apk` package targets OpenWrt 25+, where
apk-tools is the mandatory package manager; the existing `.ipk`
continues to cover OpenWrt 24.x and earlier. It is built SDK-free,
reusing the `.ipk` cross-compile and installed-filesystem payload, and
releases publish `.apk` artifacts and checksums alongside `.ipk`. Like
the `.ipk`, the package is unsigned and installed with
`apk add --allow-untrusted`.
- **Nix flake.** A `flake.nix` at the project root builds all four
binaries (`fips`, `fipsctl`, `fips-gateway`, `fipstop`) from source on
Nix/NixOS, pinning the exact toolchain and wiring the native build
dependencies so no host setup is needed beyond Nix with flakes
enabled. It exposes `nix build`, `nix run`, a `nix develop` dev shell,
and `nix flake check`, with `flake.lock` committed for reproducibility.
## Behavior changes worth flagging
These affect operators on upgrade.
@@ -223,6 +252,15 @@ subset of fixes for behavior that shipped in v0.3.0.
advertising a strictly worse root echoes its own declaration back,
provoking the better-rooted peer to re-push its real position
immediately.
- **macOS self-connections work end to end (#117).** Traffic a macOS
node sends to its own `<npub>.fips` address is now delivered locally
for full TCP/UDP, not just `ping6`. The point-to-point `utun` egresses
self-addressed packets into the daemon with an unfinished transport
checksum (macOS offloads it on the `lo0` loopback route), so
re-injecting them verbatim made the local stack drop every segment the
MSS-clamp rewrite did not happen to fix and self-connections
half-opened and hung. The hairpin path now recomputes the TCP/UDP
checksum before re-injection. Linux was unaffected.
## Upgrade notes
@@ -253,10 +291,14 @@ Operator-actionable items moving from v0.3.0 to v0.4.0:
- **Arch Linux**: `fips` from the AUR.
- **macOS**: `.pkg` at the v0.4.0 release page.
- **Windows**: ZIP at the v0.4.0 release page.
- **OpenWrt**: `.ipk` at the v0.4.0 release page.
- **OpenWrt**: `.ipk` (OpenWrt 24.x and earlier) or `.apk` (OpenWrt 25+)
at the v0.4.0 release page.
- **From source**: `cargo build --release` from a checkout of the v0.4.0
tag (Rust 1.94.1 per `rust-toolchain.toml`; `libclang-dev` is a
required Linux build prerequisite).
- **Nix / NixOS**: `nix build .#fips` from a checkout of the v0.4.0 tag
builds the binaries from source with the pinned toolchain and no manual
prerequisites (see the Nix section of `packaging/README.md`).
The full per-commit changelog lives in
[`CHANGELOG.md`](../../CHANGELOG.md). Issues and discussion at
Generated
+100
View File
@@ -0,0 +1,100 @@
{
"nodes": {
"fenix": {
"inputs": {
"nixpkgs": [
"nixpkgs"
],
"rust-analyzer-src": "rust-analyzer-src"
},
"locked": {
"lastModified": 1781527054,
"narHash": "sha256-1fX9ev2Fh5QoKQ41G9dYutjo5j/jywu6tZse5Eb1Ck4=",
"owner": "nix-community",
"repo": "fenix",
"rev": "8c2e51dffefc040a21975da7abf6f252c8c9b783",
"type": "github"
},
"original": {
"owner": "nix-community",
"repo": "fenix",
"type": "github"
}
},
"flake-utils": {
"inputs": {
"systems": "systems"
},
"locked": {
"lastModified": 1731533236,
"narHash": "sha256-l0KFg5HjrsfsO/JpG+r7fRrqm12kzFHyUHqHCVpMMbI=",
"owner": "numtide",
"repo": "flake-utils",
"rev": "11707dc2f618dd54ca8739b309ec4fc024de578b",
"type": "github"
},
"original": {
"owner": "numtide",
"repo": "flake-utils",
"type": "github"
}
},
"nixpkgs": {
"locked": {
"lastModified": 1781074563,
"narHash": "sha256-md8WlXOlfnIeHeOScMTTHFyf2d6iaTwPl2apR5EQ3P4=",
"owner": "NixOS",
"repo": "nixpkgs",
"rev": "9ae611a455b90cf061d8f332b977e387bda8e1ca",
"type": "github"
},
"original": {
"owner": "NixOS",
"ref": "nixos-unstable",
"repo": "nixpkgs",
"type": "github"
}
},
"root": {
"inputs": {
"fenix": "fenix",
"flake-utils": "flake-utils",
"nixpkgs": "nixpkgs"
}
},
"rust-analyzer-src": {
"flake": false,
"locked": {
"lastModified": 1781453968,
"narHash": "sha256-+V3nK4pCngbmgyVGXY6Kkrlevp4ocPkJJLf2aqwkDNA=",
"owner": "rust-lang",
"repo": "rust-analyzer",
"rev": "cc272809a173c2c11d0e479d639c811c1eacf049",
"type": "github"
},
"original": {
"owner": "rust-lang",
"ref": "nightly",
"repo": "rust-analyzer",
"type": "github"
}
},
"systems": {
"locked": {
"lastModified": 1681028828,
"narHash": "sha256-Vy1rq5AaRuLzOxct8nz4T6wlgyUR7zLU309k9mBC768=",
"owner": "nix-systems",
"repo": "default",
"rev": "da67096a3b9bf56a91d16901293e51ba5b49a27e",
"type": "github"
},
"original": {
"owner": "nix-systems",
"repo": "default",
"type": "github"
}
}
},
"root": "root",
"version": 7
}
+128
View File
@@ -0,0 +1,128 @@
{
description = "FIPS a distributed, decentralized network routing protocol for mesh nodes connecting over arbitrary transports";
inputs = {
nixpkgs.url = "github:NixOS/nixpkgs/nixos-unstable";
flake-utils.url = "github:numtide/flake-utils";
fenix = {
url = "github:nix-community/fenix";
inputs.nixpkgs.follows = "nixpkgs";
};
};
outputs =
{
self,
nixpkgs,
flake-utils,
fenix,
}:
flake-utils.lib.eachDefaultSystem (
system:
let
pkgs = import nixpkgs { inherit system; };
# Honor the toolchain the repo pins in rust-toolchain.toml
# (channel 1.94.1 + rustfmt, clippy) so Nix builds match CI and the
# AUR/Debian packaging exactly, including the edition-2024 frontend.
rustToolchain = fenix.packages.${system}.fromToolchainFile {
file = ./rust-toolchain.toml;
sha256 = "sha256-zC8E38iDVJ1oPIzCqTk/Ujo9+9kx9dXq7wAwPMpkpg0=";
};
rustPlatform = pkgs.makeRustPlatform {
cargo = rustToolchain;
rustc = rustToolchain;
};
cargoToml = pkgs.lib.importTOML ./Cargo.toml;
# libdbus-sys (pulled in transitively by `bluer`, Linux/glibc only)
# runs `bindgen` against the system D-Bus headers at build time.
nativeBuildInputs = [
pkgs.pkg-config
rustPlatform.bindgenHook # sets LIBCLANG_PATH + clang for bindgen
];
buildInputs = pkgs.lib.optionals pkgs.stdenv.isLinux [
pkgs.dbus # libdbus-1.so.3, linked via bluer→libdbus-sys
pkgs.stdenv.cc.cc.lib # libgcc_s.so.1, needed by every Rust binary
];
fips = rustPlatform.buildRustPackage {
pname = "fips";
version = cargoToml.package.version;
src = pkgs.lib.cleanSourceWith {
src = ./.;
# Drop the build dir and the usual editor/VCS noise so the source
# hash is stable and unrelated edits don't trigger rebuilds.
filter =
path: type:
(pkgs.lib.cleanSourceFilter path type) && (baseNameOf path != "target");
};
cargoLock.lockFile = ./Cargo.lock;
inherit buildInputs;
# autoPatchelfHook rewrites the RPATH of the built binaries so the
# daemon finds libdbus-1.so.3 (linked via bluer→libdbus-sys) in the
# Nix store at runtime — without it the `fips` binary fails to load
# on NixOS where there is no global /usr/lib.
nativeBuildInputs =
nativeBuildInputs ++ pkgs.lib.optionals pkgs.stdenv.isLinux [ pkgs.autoPatchelfHook ];
# The test suite exercises TUN devices, raw sockets and mDNS, none of
# which exist in the build sandbox. The AUR/Debian packaging likewise
# ships the release binaries without running the integration tests
# here, so keep the package build hermetic and skip them.
doCheck = false;
meta = {
description = cargoToml.package.description;
homepage = cargoToml.package.homepage;
license = pkgs.lib.licenses.mit;
mainProgram = "fips";
platforms = pkgs.lib.platforms.linux ++ pkgs.lib.platforms.darwin;
};
};
mkApp = name: {
type = "app";
program = "${fips}/bin/${name}";
meta.description = "Run the ${name} binary from the FIPS package";
};
in
{
packages = {
default = fips;
fips = fips;
};
apps = {
default = mkApp "fips";
fips = mkApp "fips";
fipsctl = mkApp "fipsctl";
fips-gateway = mkApp "fips-gateway";
fipstop = mkApp "fipstop";
};
# `nix flake check` builds the package (and thus validates the flake on
# the current system).
checks.fips = fips;
devShells.default = pkgs.mkShell {
inherit buildInputs;
nativeBuildInputs = nativeBuildInputs ++ [
rustToolchain
pkgs.cargo-edit
];
# Point rust-analyzer at the matching std sources.
RUST_SRC_PATH = "${rustToolchain}/lib/rustlib/src/rust/library";
};
formatter = pkgs.nixfmt;
}
);
}
+6 -2
View File
@@ -6,7 +6,8 @@
# Usage:
# make deb Build a Debian/Ubuntu .deb package
# make tarball Build a systemd install tarball
# make ipk Build an OpenWrt .ipk package
# make ipk Build an OpenWrt .ipk package (opkg, OpenWrt 24.x and earlier)
# make apk Build an OpenWrt .apk package (apk-tools, mandatory on OpenWrt 25+)
# make aur Build fips-git AUR package and validate with namcap
# make pkg Build a macOS .pkg installer
# make zip Build a Windows .zip package
@@ -17,7 +18,7 @@ SHELL := /bin/bash
PACKAGING_DIR := $(dir $(abspath $(lastword $(MAKEFILE_LIST))))
PROJECT_ROOT := $(abspath $(PACKAGING_DIR)/..)
.PHONY: all deb tarball ipk aur pkg zip clean
.PHONY: all deb tarball ipk apk aur pkg zip clean
all: deb tarball
@@ -30,6 +31,9 @@ tarball:
ipk:
@bash $(PACKAGING_DIR)/openwrt-ipk/build-ipk.sh
apk:
@bash $(PACKAGING_DIR)/openwrt-apk/build-apk.sh
aur:
@bash $(PACKAGING_DIR)/aur/build-aur.sh
+49 -5
View File
@@ -8,7 +8,8 @@ All build outputs go to `deploy/` at the project root.
```sh
make deb # Debian/Ubuntu .deb
make tarball # systemd install tarball
make ipk # OpenWrt .ipk
make ipk # OpenWrt .ipk (opkg, OpenWrt 24.x and earlier)
make apk # OpenWrt .apk (apk-tools, mandatory on OpenWrt 25+)
make aur # Arch Linux AUR package (fips-git, local build + namcap)
make pkg # macOS .pkg installer
make zip # Windows .zip package
@@ -46,7 +47,8 @@ packaging/
debian/ Debian/Ubuntu .deb packaging via cargo-deb
macos/ macOS .pkg installer via pkgbuild
systemd/ Generic Linux systemd tarball packaging
openwrt/ OpenWrt .ipk packaging via cargo-zigbuild
openwrt-ipk/ OpenWrt .ipk packaging via cargo-zigbuild (opkg)
openwrt-apk/ OpenWrt .apk packaging via cargo-zigbuild + apk mkpkg
windows/ Windows .zip package with service scripts
```
@@ -100,7 +102,7 @@ sudo ./fips-<version>-linux-<arch>/install.sh
See [systemd/README.install.md](systemd/README.install.md) for full
installation and configuration instructions.
### OpenWrt (`.ipk`)
### OpenWrt (`.ipk`, opkg — OpenWrt 24.x and earlier)
Cross-compiled with cargo-zigbuild and assembled as a standard `.ipk`
archive. Supports aarch64, mipsel, mips, arm, and x86\_64 targets.
@@ -110,12 +112,33 @@ archive. Supports aarch64, mipsel, mips, arm, and x86\_64 targets.
make ipk
# Build for a specific architecture
bash packaging/openwrt/build-ipk.sh --arch mipsel
bash packaging/openwrt-ipk/build-ipk.sh --arch mipsel
```
See [openwrt/README.md](openwrt/README.md) for router-specific
See [openwrt-ipk/README.md](openwrt-ipk/README.md) for router-specific
installation instructions.
### OpenWrt (`.apk`, apk-tools — mandatory on OpenWrt 25+)
OpenWrt 25 makes apk-tools the mandatory package manager (it is opt-in on
24.10). Same SDK-free approach
(cargo-zigbuild), but the `.apk` container is assembled by `apk mkpkg`
rather than hand-rolled, so the build additionally needs an apk-tools v3
`apk` binary built from source. The installed-filesystem payload is shared
with the `.ipk` package.
```sh
# Build (default: aarch64; also x86_64)
make apk
# Build for a specific architecture
bash packaging/openwrt-apk/build-apk.sh --arch x86_64
```
Packages are unsigned; install with `apk add --allow-untrusted`. See
[openwrt-apk/README.md](openwrt-apk/README.md) for building apk-tools and
router-specific installation.
### macOS (`.pkg`)
Built with `pkgbuild` (included with Xcode command-line tools). Installs
@@ -177,6 +200,27 @@ yay -S fips # release build from latest tag
See [aur/README.md](aur/README.md) for AUR publication instructions
and maintainer guide.
### Nix / NixOS (flake)
A [flake](../flake.nix) at the project root builds all four binaries
(`fips`, `fipsctl`, `fips-gateway`, `fipstop`) from source. It pins the
exact toolchain from `rust-toolchain.toml` via
[fenix](https://github.com/nix-community/fenix) and wires up the
build-time native dependencies (`libclang` for `bindgen`, plus `dbus`
and `pkg-config` for BLE), so it needs no system setup beyond Nix with
flakes enabled.
```sh
nix build .#fips # build the package (all four binaries)
nix run .#fips -- --help # run a binary directly
nix run .#fipsctl -- status
nix develop # dev shell with the pinned toolchain + cargo-edit
nix flake check # build + validate the flake
```
Add to a NixOS configuration via the flake's `packages.<system>.fips`
output, e.g. `environment.systemPackages = [ fips.packages.${system}.default ];`.
## Shared Assets
`common/` contains assets used across packaging formats:
+98
View File
@@ -0,0 +1,98 @@
# FIPS OpenWrt Package (apk)
Builds a FIPS `.apk` for **OpenWrt 25+**, where apk-tools is the mandatory
package manager. apk is also available opt-in on **24.10** (where opkg remains
the default). For OpenWrt 24.x and earlier, the `.ipk` package in
[`../openwrt-ipk/`](../openwrt-ipk/) still works.
Like the `.ipk` build, this is **SDK-free**: it cross-compiles with
`cargo-zigbuild` and assembles the package directly — no OpenWrt SDK image. The
`.ipk` format is a plain tar.gz we can hand-roll, but the `.apk` (apk-tools v3
ADB) container is not, so we drive the official `apk mkpkg` applet — the same
tool OpenWrt's own [`include/package-pack.mk`](https://github.com/openwrt/openwrt/blob/main/include/package-pack.mk)
calls. The only extra requirement over the `.ipk` build is the `apk` binary.
## Layout
| File | Purpose |
|---|---|
| `build-apk.sh` | Cross-compile + assemble the `.apk` via `apk mkpkg` |
| `apk-version.sh` | Map a release tag / commit height to an apk-tools-valid version |
| `apk-version.test.sh` | Case-table test for `apk-version.sh` (`sh apk-version.test.sh`) |
The installed-filesystem payload (init scripts, `fips.yaml`, sysctl drop-ins,
hotplug, uci-defaults, …) is **shared** with the `.ipk` package — there is one
canonical copy in [`../openwrt-ipk/files/`](../openwrt-ipk/files/). `build-apk.sh`
stages from there, so the two packages always ship the same files. Keep the
staging block in `build-apk.sh` in sync with `../openwrt-ipk/build-ipk.sh`.
## Versioning
apk-tools enforces a strict version grammar
(`<digit>(.<digit>)*(_<suffix><digit>*)*(-r<N>)`). `apk-version.sh` builds a
valid version from structured inputs rather than rewriting an already-flattened
string:
| Input | apk version |
|---|---|
| `tag v1.2.3` | `1.2.3-r0` |
| `tag v1.2.3-rc1` | `1.2.3_rc1-r0` |
| `dev 1234` (commit height) | `0.0.0_git1234-r0` |
The human-readable version (`v1.2.3`, `master.123.abcdef0`) is still used for the
artifact filename; only the metadata embedded in the package is normalized.
## Building
### Prerequisites
| Requirement | Notes |
|---|---|
| `cargo install cargo-zigbuild` + `zig` | Rust musl cross-compilation (as for `.ipk`) |
| apk-tools v3 `apk` binary | Provides `apk mkpkg`; not packaged for most distros — build from source |
| `fakeroot` | Optional; makes packaged files root-owned on an unprivileged build host |
apk-tools is not in Debian/Ubuntu repos, so build the pinned release from source.
Pin the same commit the targeted OpenWrt release ships (see
`package/system/apk/Makefile` upstream) so the `.apk` is readable by the device's
`apk`. CI builds **3.0.5** (`b5a31c0d…`):
```bash
sudo apt-get install -y build-essential meson ninja-build pkg-config \
zlib1g-dev libssl-dev libzstd-dev liblzma-dev lua5.4-dev scdoc
git clone https://gitlab.alpinelinux.org/alpine/apk-tools.git
cd apk-tools && git checkout b5a31c0d865342ad80be10d68f1bb3d3ad9b0866
meson setup build && ninja -C build src/apk
export APK_BIN="$PWD/build/src/apk"
```
### Build the package
```bash
# from the repo root
./packaging/openwrt-apk/build-apk.sh --arch aarch64 # or x86_64, mipsel, mips, arm
```
Output: `dist/fips_<version>_<openwrt-arch>.apk`. Override the version with
`PKG_VERSION` (filename) and `APK_VERSION` (embedded metadata); otherwise both are
derived from git.
## Installing on the router
Packages are **unsigned** (the same posture as our `.ipk`), so install with
`--allow-untrusted`:
```bash
scp -O dist/fips_<version>_<arch>.apk root@192.168.1.1:/tmp/
ssh root@192.168.1.1 apk add --allow-untrusted /tmp/fips_<version>_<arch>.apk
```
On OpenWrt 25.x, installing from a *signed repository* requires the publisher's
key; a single `--allow-untrusted` package install does not. If we ever publish an
apk feed, add ECDSA (prime256v1) signing via `apk mkpkg --sign` and distribute the
public key to `/etc/apk/keys/`.
`/etc/fips/fips.yaml` is marked as a config file (via
`/lib/apk/packages/fips.conffiles`), so apk preserves local edits across upgrades,
and `/lib/upgrade/keep.d/fips` preserves `/etc/fips/` across `sysupgrade` — the
same guarantees as the `.ipk` package.
+74
View File
@@ -0,0 +1,74 @@
#!/bin/sh
# Emit an apk-tools-compatible version string for FIPS.
#
# apk-tools enforces a strict version grammar:
# <digit>(.<digit>)*(_<suffix><digit>*)*(-r<N>)
# where <suffix> is a recognised pre-release/post-release token
# (alpha, beta, pre, rc, cvs, svn, git, hg, p).
#
# Unlike a regex rewrite of an already-flattened version string, this
# helper builds the apk version directly from the *structured* inputs the
# caller already has (a release tag, or a commit height). There is no
# parsing-back-out of a "branch.height.hash" blob, so there is no fragile
# reparse step to get wrong.
#
# Usage:
# apk-version.sh tag <git-tag> # e.g. v1.2.3, v1.2.3-rc1
# apk-version.sh dev <height> # e.g. 1234 (git rev-list --count HEAD)
# apk-version.sh auto # derive from the current git checkout
#
# Examples:
# apk-version.sh tag v1.2.3 -> 1.2.3-r0
# apk-version.sh tag v1.2.3-rc1 -> 1.2.3_rc1-r0
# apk-version.sh dev 1234 -> 0.0.0_git1234-r0
set -eu
mode="${1:-auto}"
case "$mode" in
tag) raw_tag="${2:?tag mode requires a tag argument}"; height="" ;;
dev) raw_tag=""; height="${2:?dev mode requires a height argument}" ;;
auto)
if raw_tag="$(git describe --exact-match --tags 2>/dev/null)"; then
height=""
else
raw_tag=""
height="$(git rev-list --count HEAD 2>/dev/null || echo 0)"
fi
;;
*)
echo "usage: $0 [auto | tag <git-tag> | dev <height>]" >&2
exit 2
;;
esac
if [ -n "$raw_tag" ]; then
# Release tag: vX.Y.Z or vX.Y.Z-<pre>. Strip the leading 'v', split the
# core (X.Y.Z) from the pre-release token, and map our hyphen separator
# to apk's '_' pre-release marker.
body="${raw_tag#v}"
core="${body%%-*}"
case "$body" in
*-*) pre="${body#*-}" ;;
*) pre="" ;;
esac
case "$pre" in
"") suffix="" ;;
alpha*|beta*|pre*|rc*) suffix="_${pre}" ;;
*)
# Unknown pre-release token: apk would reject or misorder it, so
# drop it rather than emit an invalid version. The human-readable
# PACKAGE_VERSION (the raw tag) is still used for the filename.
suffix=""
;;
esac
printf '%s%s-r0\n' "$core" "$suffix"
else
# Untagged build: no meaningful semver, so anchor at 0.0.0 and encode the
# monotonic commit height as a _git pre-release component. This keeps apk's
# ordering sane across dev builds without smuggling the hash/branch into a
# field that cannot represent them.
printf '0.0.0_git%s-r0\n' "${height:-0}"
fi
+45
View File
@@ -0,0 +1,45 @@
#!/bin/sh
# Case-table test for apk-version.sh. Run: sh apk-version.test.sh
set -eu
HERE="$(cd "$(dirname "$0")" && pwd)"
SUT="$HERE/apk-version.sh"
fail=0
check() {
# check <expected> <args...>
expected="$1"; shift
actual="$(sh "$SUT" "$@")"
if [ "$actual" = "$expected" ]; then
printf ' PASS %-22s -> %s\n' "$*" "$actual"
else
printf ' FAIL %-22s -> %s (expected %s)\n' "$*" "$actual" "$expected"
fail=1
fi
}
echo "== apk-version.sh =="
# Plain release tags.
check "1.2.3-r0" tag v1.2.3
check "0.4.0-r0" tag v0.4.0
check "10.20.30-r0" tag v10.20.30
# Pre-release tags: hyphen separator becomes apk's '_' marker.
check "1.2.3_rc1-r0" tag v1.2.3-rc1
check "1.2.3_alpha1-r0" tag v1.2.3-alpha1
check "1.2.3_beta2-r0" tag v1.2.3-beta2
check "1.2.3_pre1-r0" tag v1.2.3-pre1
# Unknown pre-release token is dropped (apk cannot represent it).
check "1.2.3-r0" tag v1.2.3-weird9
# Dev builds: monotonic commit height as a _git component.
check "0.0.0_git1234-r0" dev 1234
check "0.0.0_git0-r0" dev 0
if [ "$fail" -ne 0 ]; then
echo "FAILED"
exit 1
fi
echo "OK"
+296
View File
@@ -0,0 +1,296 @@
#!/bin/bash
# Build a FIPS .apk package for OpenWrt without the OpenWrt SDK.
#
# apk-tools (.apk) is the mandatory package manager from OpenWrt 25 onward; it
# is also available opt-in on 24.10, where opkg (.ipk) remains the default. The
# .ipk package in ../openwrt-ipk/ still covers OpenWrt 24.x and earlier; this
# .apk package is what you need on 25+. Unlike the .ipk format (a plain tar.gz
# of tarballs that we
# assemble by hand in ../openwrt-ipk/build-ipk.sh), the .apk container is the
# apk-tools v3 ADB format, which is impractical to hand-roll. Instead we drive
# the official `apk mkpkg` applet — the same tool OpenWrt's build system calls
# in include/package-pack.mk — so no SDK is required, only the `apk` binary.
#
# Usage:
# ./packaging/openwrt-apk/build-apk.sh [--arch <name>]
#
# Architectures (--arch): aarch64 [default], x86_64, mipsel, mips, arm
# (the apk CI matrix ships aarch64 + x86_64; the rest are buildable locally).
#
# Output: dist/fips_<version>_<openwrt-arch>.apk
#
# Prerequisites:
# cargo install cargo-zigbuild (Rust musl cross-compilation)
# apk-tools v3 `apk` binary on PATH, or pointed at via APK_BIN=/path/to/apk
# (build from source — see README.md; CI builds apk-tools 3.0.5).
# fakeroot (optional but recommended; makes packaged files root-owned).
#
# Install on a router (packages are unsigned, like our .ipk):
# scp -O dist/fips_<version>_<arch>.apk root@192.168.1.1:/tmp/
# ssh root@192.168.1.1 apk add --allow-untrusted /tmp/fips_<version>_<arch>.apk
set -euo pipefail
# ---------------------------------------------------------------------------
# Arguments
# ---------------------------------------------------------------------------
ARCH="aarch64"
BIN_DIR="" # if set, use prebuilt binaries from here instead of compiling
while [[ $# -gt 0 ]]; do
case "$1" in
--arch) ARCH="$2"; shift 2 ;;
--arch=*) ARCH="${1#*=}"; shift ;;
--bin-dir) BIN_DIR="$2"; shift 2 ;;
--bin-dir=*) BIN_DIR="${1#*=}"; shift ;;
*) echo "Unknown argument: $1" >&2; exit 1 ;;
esac
done
# ---------------------------------------------------------------------------
# Architecture mapping
#
# RUST_TARGET — passed to cargo --target
# OPENWRT_ARCH — apk "arch:" field and the package filename
#
# Kept in sync with ../openwrt-ipk/build-ipk.sh (same target table).
# ---------------------------------------------------------------------------
case "$ARCH" in
aarch64)
RUST_TARGET="aarch64-unknown-linux-musl"
OPENWRT_ARCH="aarch64_cortex-a53"
;;
mipsel)
RUST_TARGET="mipsel-unknown-linux-musl"
OPENWRT_ARCH="mipsel_24kc"
;;
mips)
RUST_TARGET="mips-unknown-linux-musl"
OPENWRT_ARCH="mips_24kc"
;;
arm)
RUST_TARGET="arm-unknown-linux-musleabihf"
OPENWRT_ARCH="arm_cortex-a7"
;;
x86_64)
RUST_TARGET="x86_64-unknown-linux-musl"
OPENWRT_ARCH="x86_64"
;;
*)
echo "Unknown arch: $ARCH" >&2
echo "Valid: aarch64, mipsel, mips, arm, x86_64" >&2
exit 1
;;
esac
# ---------------------------------------------------------------------------
# Paths
# ---------------------------------------------------------------------------
SCRIPT_DIR="$(cd "$(dirname "$0")" && pwd)"
PROJECT_ROOT="$(cd "$SCRIPT_DIR/../.." && pwd)"
# The installed-filesystem payload (init scripts, config, sysctl, etc.) is
# shared with the .ipk package; there is one canonical copy in openwrt-ipk/.
FILES_DIR="$PROJECT_ROOT/packaging/openwrt-ipk/files"
DIST_DIR="$PROJECT_ROOT/dist"
PKG_NAME="fips"
# Human-readable version for the filename (e.g. v0.4.0 or master.123.abcdef0),
# mirroring the .ipk artifacts and the CI/NIP-94 plumbing.
PKG_VERSION="${PKG_VERSION:-$(cd "$PROJECT_ROOT" && git describe --tags --always --dirty 2>/dev/null || echo "0.1.0")}"
# apk-tools-compatible version embedded inside the package metadata.
APK_VERSION="${APK_VERSION:-$(cd "$PROJECT_ROOT" && sh "$SCRIPT_DIR/apk-version.sh" auto)}"
APK_BIN="${APK_BIN:-apk}"
if ! command -v "$APK_BIN" >/dev/null 2>&1; then
echo "Error: apk-tools binary not found (looked for '$APK_BIN')." >&2
echo " Build apk-tools v3 from source or set APK_BIN=/path/to/apk." >&2
echo " See packaging/openwrt-apk/README.md." >&2
exit 1
fi
echo "==> Building $PKG_NAME $PKG_VERSION (apk version $APK_VERSION) for $OPENWRT_ARCH ($RUST_TARGET)"
# ---------------------------------------------------------------------------
# 1. Obtain binaries
#
# Either use a directory of prebuilt binaries (--bin-dir; CI cross-compiles
# once in a shared job and hands them to both the .ipk and .apk packagers), or
# compile from source here for a self-contained local build.
# ---------------------------------------------------------------------------
if [ -n "$BIN_DIR" ]; then
RELEASE_DIR="$BIN_DIR"
echo "==> Using prebuilt binaries from $RELEASE_DIR"
for bin in fips fipsctl fipstop fips-gateway; do
[ -f "$RELEASE_DIR/$bin" ] || {
echo "Error: prebuilt binary not found: $RELEASE_DIR/$bin" >&2
exit 1
}
done
else
if ! command -v cargo-zigbuild &>/dev/null; then
echo "Error: cargo-zigbuild not found." >&2
echo " Install: cargo install cargo-zigbuild" >&2
exit 1
fi
if ! rustup target list --installed | grep -q "^$RUST_TARGET$"; then
echo "==> Adding Rust target $RUST_TARGET..."
rustup target add "$RUST_TARGET"
fi
echo "==> Compiling..."
cd "$PROJECT_ROOT"
cargo zigbuild \
--release \
--target "$RUST_TARGET" \
--bin fips \
--bin fipsctl \
--bin fipstop \
--bin fips-gateway
RELEASE_DIR="$PROJECT_ROOT/target/$RUST_TARGET/release"
echo "==> Stripping binaries..."
STRIP="${LLVM_STRIP:-strip}"
for bin in fips fipsctl fipstop fips-gateway; do
"$STRIP" "$RELEASE_DIR/$bin" 2>/dev/null || true
done
fi
SIZE=$(du -sh "$RELEASE_DIR/fips" | cut -f1)
echo " fips: $SIZE"
# ---------------------------------------------------------------------------
# 2. Stage the installed filesystem tree (--files root for apk mkpkg)
# ---------------------------------------------------------------------------
# This block is the same payload as ../openwrt-ipk/build-ipk.sh; keep the two
# in sync. The CI apk structural check asserts every path below is present.
WORK_DIR="$(mktemp -d)"
trap 'rm -rf "$WORK_DIR"' EXIT
STAGE_DIR="$WORK_DIR/root" # becomes the package's filesystem
SCRIPTS_DIR="$WORK_DIR/scripts" # maintainer scripts (metadata, not payload)
mkdir -p "$STAGE_DIR" "$SCRIPTS_DIR"
install -d "$STAGE_DIR/usr/bin"
install -m 0755 "$RELEASE_DIR/fips" "$STAGE_DIR/usr/bin/fips"
install -m 0755 "$RELEASE_DIR/fipsctl" "$STAGE_DIR/usr/bin/fipsctl"
install -m 0755 "$RELEASE_DIR/fipstop" "$STAGE_DIR/usr/bin/fipstop"
install -m 0755 "$RELEASE_DIR/fips-gateway" "$STAGE_DIR/usr/bin/fips-gateway"
install -d "$STAGE_DIR/etc/init.d"
install -m 0755 "$FILES_DIR/etc/init.d/fips" "$STAGE_DIR/etc/init.d/fips"
install -m 0755 "$FILES_DIR/etc/init.d/fips-gateway" "$STAGE_DIR/etc/init.d/fips-gateway"
install -d "$STAGE_DIR/etc/fips"
install -m 0600 "$FILES_DIR/etc/fips/fips.yaml" "$STAGE_DIR/etc/fips/fips.yaml"
install -m 0755 "$FILES_DIR/etc/fips/firewall.sh" "$STAGE_DIR/etc/fips/firewall.sh"
# The shared fips.yaml ships ethernet.wan.interface: "eth0", the OpenWrt 24
# default. This .apk package targets OpenWrt 25+ (DSA), where the WAN port is
# named "wan", so ship "wan" as the default. Patching the staged copy keeps the
# as-installed config correct for the platform without maintaining a second copy
# of the file; operators can still edit /etc/fips/fips.yaml for non-standard boards.
sed -i 's|interface: "eth0"|interface: "wan"|' "$STAGE_DIR/etc/fips/fips.yaml"
install -d "$STAGE_DIR/etc/dnsmasq.d"
install -m 0644 "$FILES_DIR/etc/dnsmasq.d/fips.conf" "$STAGE_DIR/etc/dnsmasq.d/fips.conf"
install -d "$STAGE_DIR/etc/sysctl.d"
install -m 0644 "$FILES_DIR/etc/sysctl.d/fips-bridge.conf" "$STAGE_DIR/etc/sysctl.d/fips-bridge.conf"
install -m 0644 "$FILES_DIR/etc/sysctl.d/fips-gateway.conf" "$STAGE_DIR/etc/sysctl.d/fips-gateway.conf"
install -d "$STAGE_DIR/etc/hotplug.d/net"
install -m 0755 "$FILES_DIR/etc/hotplug.d/net/99-fips" "$STAGE_DIR/etc/hotplug.d/net/99-fips"
install -d "$STAGE_DIR/etc/uci-defaults"
install -m 0755 "$FILES_DIR/etc/uci-defaults/90-fips-setup" "$STAGE_DIR/etc/uci-defaults/90-fips-setup"
install -d "$STAGE_DIR/lib/upgrade/keep.d"
install -m 0644 "$FILES_DIR/lib/upgrade/keep.d/fips" "$STAGE_DIR/lib/upgrade/keep.d/fips"
# ---- conffiles ----
# apk mkpkg discovers config files from /lib/apk/packages/<name>.conffiles
# inside the --files tree (same mechanism OpenWrt's package-pack.mk uses).
# Listing fips.yaml here makes apk preserve user edits across upgrades, the
# apk equivalent of opkg's conffiles handling.
install -d "$STAGE_DIR/lib/apk/packages"
cat > "$STAGE_DIR/lib/apk/packages/${PKG_NAME}.conffiles" <<'EOF'
/etc/fips/fips.yaml
EOF
# ---- maintainer scripts ----
# Map our opkg maintainer scripts onto apk's lifecycle phases:
# opkg postinst -> apk post-install (enable + start services)
# opkg prerm -> apk pre-deinstall (stop + disable services)
cat > "$SCRIPTS_DIR/post-install" <<'EOF'
#!/bin/sh
# Run first-boot UCI setup (the script deletes itself when done).
if [ -x /etc/uci-defaults/90-fips-setup ]; then
/etc/uci-defaults/90-fips-setup && rm -f /etc/uci-defaults/90-fips-setup
fi
/etc/init.d/fips enable
/etc/init.d/fips start
/etc/init.d/fips-gateway enable
/etc/init.d/fips-gateway start
exit 0
EOF
cat > "$SCRIPTS_DIR/pre-deinstall" <<'EOF'
#!/bin/sh
/etc/init.d/fips-gateway stop 2>/dev/null || true
/etc/init.d/fips-gateway disable 2>/dev/null || true
/etc/init.d/fips stop 2>/dev/null || true
/etc/init.d/fips disable 2>/dev/null || true
exit 0
EOF
chmod 0755 "$SCRIPTS_DIR/post-install" "$SCRIPTS_DIR/pre-deinstall"
# ---------------------------------------------------------------------------
# 3. Assemble the .apk via apk mkpkg
# ---------------------------------------------------------------------------
# fakeroot makes the packaged files root-owned even though CI runs unprivileged.
DESCRIPTION="FIPS Mesh Network Daemon. Distributed, decentralized mesh networking over UDP, TCP, and raw Ethernet, with a TUN interface (fips0), ULA IPv6 addressing, and a .fips DNS responder."
DEPENDS="kmod-tun kmod-br-netfilter kmod-nft-nat kmod-nf-conntrack ip-full"
PKG_FILENAME="${PKG_NAME}_${PKG_VERSION}_${OPENWRT_ARCH}.apk"
mkdir -p "$DIST_DIR"
FAKEROOT=""
if command -v fakeroot >/dev/null 2>&1; then
FAKEROOT="fakeroot"
else
echo "Warning: fakeroot not found — packaged files will be owned by the build user." >&2
fi
$FAKEROOT "$APK_BIN" mkpkg \
--info "name:$PKG_NAME" \
--info "version:$APK_VERSION" \
--info "description:$DESCRIPTION" \
--info "arch:$OPENWRT_ARCH" \
--info "license:MIT" \
--info "origin:$PKG_NAME" \
--info "url:https://github.com/jmcorgan/fips" \
--info "maintainer:FIPS Network" \
--info "depends:$DEPENDS" \
--script "post-install:$SCRIPTS_DIR/post-install" \
--script "pre-deinstall:$SCRIPTS_DIR/pre-deinstall" \
--files "$STAGE_DIR" \
--output "$DIST_DIR/$PKG_FILENAME"
echo ""
echo "==> Done: dist/$PKG_FILENAME"
echo " $(du -sh "$DIST_DIR/$PKG_FILENAME" | cut -f1)"
echo ""
echo "Install on router (OpenWrt 25+, or 24.10 with apk enabled):"
echo " scp -O dist/$PKG_FILENAME root@192.168.1.1:/tmp/"
echo " ssh root@192.168.1.1 apk add --allow-untrusted /tmp/$PKG_FILENAME"
+4 -1
View File
@@ -132,7 +132,10 @@ The default config enables:
For Ethernet transport, uncomment the `ethernet:` section and set the correct
physical interface names for your router. **Always use physical port names
(`eth0`, `eth1`), never bridge names (`br-lan`).** See
(`eth0`, `eth1`, or DSA port names like `wan`/`lan1`), never bridge names
(`br-lan`).** The shipped default WAN port is `eth0` (OpenWrt 24); on OpenWrt
25 (DSA) boards the WAN port is named `wan` — the `.apk` package ships that
default. Run `ip link show` to confirm the names on your board. See
[`deploy/native/README.md`](../../deploy/native/README.md) for details.
## Service management
+47 -33
View File
@@ -27,11 +27,14 @@ set -euo pipefail
# ---------------------------------------------------------------------------
ARCH="aarch64"
BIN_DIR="" # if set, use prebuilt binaries from here instead of compiling
while [[ $# -gt 0 ]]; do
case "$1" in
--arch) ARCH="$2"; shift 2 ;;
--arch=*) ARCH="${1#*=}"; shift ;;
--bin-dir) BIN_DIR="$2"; shift 2 ;;
--bin-dir=*) BIN_DIR="${1#*=}"; shift ;;
*) echo "Unknown argument: $1" >&2; exit 1 ;;
esac
done
@@ -86,44 +89,55 @@ PKG_VERSION="${PKG_VERSION:-$(cd "$PROJECT_ROOT" && git describe --tags --always
echo "==> Building $PKG_NAME $PKG_VERSION for $OPENWRT_ARCH ($RUST_TARGET)"
# ---------------------------------------------------------------------------
# Prerequisites
# 1. Obtain binaries
#
# Either use a directory of prebuilt binaries (--bin-dir; CI cross-compiles
# once in a shared job and hands them to both the .ipk and .apk packagers), or
# compile from source here for a self-contained local build.
# ---------------------------------------------------------------------------
if ! command -v cargo-zigbuild &>/dev/null; then
echo "Error: cargo-zigbuild not found." >&2
echo " Install: cargo install cargo-zigbuild" >&2
exit 1
if [ -n "$BIN_DIR" ]; then
RELEASE_DIR="$BIN_DIR"
echo "==> Using prebuilt binaries from $RELEASE_DIR"
for bin in fips fipsctl fipstop fips-gateway; do
[ -f "$RELEASE_DIR/$bin" ] || {
echo "Error: prebuilt binary not found: $RELEASE_DIR/$bin" >&2
exit 1
}
done
else
if ! command -v cargo-zigbuild &>/dev/null; then
echo "Error: cargo-zigbuild not found." >&2
echo " Install: cargo install cargo-zigbuild" >&2
exit 1
fi
if ! rustup target list --installed | grep -q "^$RUST_TARGET$"; then
echo "==> Adding Rust target $RUST_TARGET..."
rustup target add "$RUST_TARGET"
fi
echo "==> Compiling..."
cd "$PROJECT_ROOT"
cargo zigbuild \
--release \
--target "$RUST_TARGET" \
--bin fips \
--bin fipsctl \
--bin fipstop \
--bin fips-gateway
RELEASE_DIR="$PROJECT_ROOT/target/$RUST_TARGET/release"
echo "==> Stripping binaries..."
STRIP="${LLVM_STRIP:-strip}"
for bin in fips fipsctl fipstop fips-gateway; do
"$STRIP" "$RELEASE_DIR/$bin" 2>/dev/null || true
done
fi
if ! rustup target list --installed | grep -q "^$RUST_TARGET$"; then
echo "==> Adding Rust target $RUST_TARGET..."
rustup target add "$RUST_TARGET"
fi
# ---------------------------------------------------------------------------
# 1. Build
# ---------------------------------------------------------------------------
echo "==> Compiling..."
cd "$PROJECT_ROOT"
cargo zigbuild \
--release \
--target "$RUST_TARGET" \
--bin fips \
--bin fipsctl \
--bin fipstop \
--bin fips-gateway
RELEASE_DIR="$PROJECT_ROOT/target/$RUST_TARGET/release"
echo "==> Stripping binaries..."
STRIP="${LLVM_STRIP:-strip}"
for bin in fips fipsctl fipstop fips-gateway; do
"$STRIP" "$RELEASE_DIR/$bin" 2>/dev/null || true
done
SIZE=$(du -sh "$RELEASE_DIR/fips" | cut -f1)
echo " fips: $SIZE after strip"
echo " fips: $SIZE"
# ---------------------------------------------------------------------------
# 2. Assemble .ipk
@@ -12,6 +12,17 @@ start_service() {
# Ensure TUN module is loaded before starting the daemon.
modprobe tun 2>/dev/null || true
# Pre-create the control-socket runtime directory so the daemon binds the
# canonical /run/fips/control.sock instead of falling back to /tmp. This is
# the procd equivalent of the systemd unit's RuntimeDirectory=fips (and the
# fips.tmpfiles "d /run/fips 0750 root fips" entry); OpenWrt was the only
# platform missing it. Without it, fips-gateway — which creates /run/fips
# for its own gateway.sock — makes fipsctl/fipstop resolve a control socket
# under /run/fips that the daemon actually bound under /tmp.
mkdir -p /run/fips
chmod 0750 /run/fips
chgrp fips /run/fips 2>/dev/null || true
procd_open_instance
procd_set_param command "$PROG" --config "$CONFIG"
# Respawn: restart after 5 s, give up after 5 consecutive failures within
+131 -38
View File
@@ -83,6 +83,35 @@ fn fwd_value(data: &serde_json::Value, pkt_key: &str, byte_key: &str) -> String
format!("{} pkts ({})", pkts, helpers::format_bytes(bytes))
}
/// Read a raw forwarding counter as a u64 (0 if missing), for arithmetic
/// (percentages, derived totals) that the string-returning helpers can't do.
fn fwd_count(data: &serde_json::Value, key: &str) -> u64 {
data.get("forwarding")
.and_then(|f| f.get(key))
.and_then(|v| v.as_u64())
.unwrap_or(0)
}
/// Total mesh egress = locally-originated + transit-forwarded, formatted as
/// "N pkts (B)". There is no single daemon counter for everything this node
/// transmits to peers, so it is derived from its two contributors.
fn mesh_tx_value(data: &serde_json::Value) -> String {
let pkts = fwd_count(data, "originated_packets") + fwd_count(data, "forwarded_packets");
let bytes = fwd_count(data, "originated_bytes") + fwd_count(data, "forwarded_bytes");
format!("{} pkts ({})", pkts, helpers::format_bytes(bytes))
}
/// Format a route-class count as "N (xx.x%)" where the percentage is the class's
/// share of total forwarded (transit) packets. Zero forwarded yields "0.0%".
fn route_class_value(count: u64, total_forwarded: u64) -> String {
let pct = if total_forwarded > 0 {
count as f64 / total_forwarded as f64 * 100.0
} else {
0.0
};
format!("{count} ({pct:.1}%)")
}
/// Build a section: a styled header line followed by the kv pairs rendered
/// through the group helper so the section's values share a left edge.
fn section(title: &str, pairs: &[(&str, String)]) -> Vec<Line<'static>> {
@@ -110,49 +139,39 @@ fn draw_routing_stats(
let err = |key: &str| helpers::nested_u64(data, "error_signals", key);
let cong = |key: &str| helpers::nested_u64(data, "congestion", key);
// Left column: Forwarding + Discovery. Each section's values share a left
// edge via the kv_lines group helper.
// The node is an interface adapter between the local host stack and the
// mesh; the left column reads each side as a Transmitted/Received pair.
//
// Local Stack — traffic crossing the TUN / local-origination boundary:
// Transmitted is what the host injects into the mesh (originated), Received
// is what the mesh hands up to the host (delivered).
let mut left = section(
"Forwarding",
"Local Stack",
&[
(
"Transmitted",
fwd_value(data, "originated_packets", "originated_bytes"),
),
(
"Received",
fwd_value(data, "delivered_packets", "delivered_bytes"),
),
],
);
left.push(Line::from(""));
// Mesh — traffic crossing the peer-link boundary: Transmitted is everything
// this node puts on the wire (originated + forwarded, derived), Received is
// the ingress aggregate from peers (own-delivered + transit + drops).
left.extend(section(
"Mesh",
&[
("Transmitted", mesh_tx_value(data)),
(
"Received",
fwd_value(data, "received_packets", "received_bytes"),
),
(
"Delivered",
fwd_value(data, "delivered_packets", "delivered_bytes"),
),
(
"Forwarded",
fwd_value(data, "forwarded_packets", "forwarded_bytes"),
),
(
"Originated",
fwd_value(data, "originated_packets", "originated_bytes"),
),
(
"Decode Error",
fwd_value(data, "decode_error_packets", "decode_error_bytes"),
),
(
"TTL Exhausted",
fwd_value(data, "ttl_exhausted_packets", "ttl_exhausted_bytes"),
),
(
"No Route",
fwd_value(data, "drop_no_route_packets", "drop_no_route_bytes"),
),
(
"MTU Exceeded",
fwd_value(data, "drop_mtu_exceeded_packets", "drop_mtu_exceeded_bytes"),
),
(
"Send Error",
fwd_value(data, "drop_send_error_packets", "drop_send_error_bytes"),
),
],
);
));
left.push(Line::from(""));
left.extend(section(
"Discovery Requests",
@@ -184,15 +203,89 @@ fn draw_routing_stats(
],
));
// Right column: Error Signals + Congestion
// Right column — "Forwarded" (transit / routed through this node).
// Forwarded total, then the route-class breakdown (a percentage partition
// of the total), then the transit-path drop reasons.
let fwd_total = fwd_count(data, "forwarded_packets");
let mut right = section(
"Forwarded",
&[(
"Forwarded",
fwd_value(data, "forwarded_packets", "forwarded_bytes"),
)],
);
// Blank separator after the Forwarded total, matching the spacing between
// every other section pair; the total and its route-class breakdown read
// as two distinct groups.
right.push(Line::from(""));
// Route-class breakdown: a partition of Forwarded, each line annotated with
// its share of the total. Tree-down cross — the dive-to-tree-child
// cut-through — is the last class; Tree-down + Tree-down cross sum to the
// pre-split tree-down total.
right.extend(section(
"Route Class",
&[
(
"Direct Peer",
route_class_value(fwd_count(data, "route_direct_peer"), fwd_total),
),
(
"Tree-down",
route_class_value(fwd_count(data, "route_tree_down"), fwd_total),
),
(
"Tree-up",
route_class_value(fwd_count(data, "route_tree_up"), fwd_total),
),
(
"Cross-link descend",
route_class_value(fwd_count(data, "route_crosslink_descend"), fwd_total),
),
(
"Cross-link ascend",
route_class_value(fwd_count(data, "route_crosslink_ascend"), fwd_total),
),
(
"Tree-down cross",
route_class_value(fwd_count(data, "route_tree_down_cross"), fwd_total),
),
],
));
right.push(Line::from(""));
right.extend(section(
"Dropped",
&[
(
"No Route",
fwd_value(data, "drop_no_route_packets", "drop_no_route_bytes"),
),
(
"TTL Exhausted",
fwd_value(data, "ttl_exhausted_packets", "ttl_exhausted_bytes"),
),
(
"Decode Error",
fwd_value(data, "decode_error_packets", "decode_error_bytes"),
),
(
"MTU Exceeded",
fwd_value(data, "drop_mtu_exceeded_packets", "drop_mtu_exceeded_bytes"),
),
(
"Send Error",
fwd_value(data, "drop_send_error_packets", "drop_send_error_bytes"),
),
],
));
right.push(Line::from(""));
right.extend(section(
"Error Signals",
&[
("Coords Required", err("coords_required")),
("Path Broken", err("path_broken")),
("MTU Exceeded", err("mtu_exceeded")),
],
);
));
right.push(Line::from(""));
right.extend(section(
"Congestion",
+6 -1
View File
@@ -1134,7 +1134,12 @@ fn routing_focused_pane_scrolls() {
let mut app1 = app_with(Tab::Routing, data);
app1.data.insert(Tab::Cache, json!({}));
app1.focused_pane.insert(Tab::Routing, 2);
app1.scroll_offsets.insert((Tab::Routing, 2), 6);
// Congestion is the last section of the right ("Forwarded") column, below
// the route-class breakdown, Dropped, and Error Signals groups. The left
// column is the taller of the two, so scrolling fully to the bottom would
// over-scroll the right column past Congestion; this offset lands the
// Congestion region inside the short window instead.
app1.scroll_offsets.insert((Tab::Routing, 2), 24);
let buf1 = testkit::render(100, 20, |frame, area| {
super::routing::draw(frame, &app1, area);
});
+35 -1
View File
@@ -41,7 +41,7 @@ use super::snapshot::{EntitySnapshot, RoutingSnapshot, StatsSnapshot};
/// starting R1 as `show_*` queries cut over to off-loop rendering; until then
/// they are wired but unread.
#[derive(Clone)]
pub(crate) struct ControlReadHandle {
pub struct ControlReadHandle {
/// Effectively-immutable node context (config, identity, limits).
context: Arc<NodeContext>,
/// Metrics registry (counters / gauges) for `show_stats_*`.
@@ -108,6 +108,40 @@ impl ControlReadHandle {
}
}
/// A minimal, public view of one peer for embedders (e.g. an app UI), read
/// lock-free from the tick-published snapshot. See [`ControlReadHandle::peer_views`].
#[derive(Debug, Clone)]
pub struct PeerView {
/// The peer's `node_addr`, hex-encoded.
pub node_addr_hex: String,
/// Resolved npub (or the `node_addr` hex when not yet resolved to a peer).
pub npub: String,
/// Whether the peer is currently in the live authenticated-peer table.
pub connected: bool,
}
impl ControlReadHandle {
/// A lock-free snapshot of known peers (node_addr / npub / connected),
/// read from the tick-published stats snapshot.
///
/// Intended for embedders that run [`crate::Node::run_rx_loop`] on a
/// background task (so the node is exclusively borrowed there) and poll peer
/// state from a clone of this handle — the read touches only an `ArcSwap`
/// load, never the `Node`. See the Myco app for the reference embedding.
pub fn peer_views(&self) -> Vec<PeerView> {
self.stats
.load()
.peer_meta
.iter()
.map(|(addr, meta)| PeerView {
node_addr_hex: addr.to_string(),
npub: meta.npub.clone(),
connected: meta.is_active,
})
.collect()
}
}
/// Attempt to serve a request entirely from the read handle, off the rx_loop.
///
/// Returns `Some(response)` when the command is a pure-snapshot query that has
+6
View File
@@ -53,6 +53,12 @@
"originated_packets": 0,
"received_bytes": 0,
"received_packets": 0,
"route_crosslink_ascend": 0,
"route_crosslink_descend": 0,
"route_direct_peer": 0,
"route_tree_down": 0,
"route_tree_down_cross": 0,
"route_tree_up": 0,
"ttl_exhausted_bytes": 0,
"ttl_exhausted_packets": 0
},
+6
View File
@@ -22,6 +22,12 @@
"originated_packets": 0,
"received_bytes": 0,
"received_packets": 0,
"route_crosslink_ascend": 0,
"route_crosslink_descend": 0,
"route_direct_peer": 0,
"route_tree_down": 0,
"route_tree_down_cross": 0,
"route_tree_up": 0,
"ttl_exhausted_bytes": 0,
"ttl_exhausted_packets": 0
},
+5
View File
@@ -151,6 +151,11 @@ impl Node {
}
} else {
self.metrics().forwarding.record_forwarded(encoded.len());
// Classify this transit forward by route class (partition of
// forwarded_packets). Done here, at the data-plane chokepoint, so
// the error-signal routing callers of find_next_hop are excluded.
let class = self.classify_forward(&datagram.dest_addr, &next_hop_addr);
self.metrics().forwarding.record_route_class(class);
if outgoing_ce {
self.metrics().congestion.ce_forwarded.inc();
}
+4 -2
View File
@@ -1201,8 +1201,10 @@ impl Node {
// This allows handshake messages to be sent before we start accepting packets
self.initiate_peer_connections().await;
// Initialize TUN interface last, after transports and peers are ready
if self.config().tun.enabled {
// Initialize TUN interface last, after transports and peers are ready.
// Skip when the TUN is app-owned (the embedder pre-set `tun_tx` via
// `enable_app_owned_tun`) — then FIPS does no system-TUN ops.
if self.config().tun.enabled && self.tun_tx.is_none() {
let address = *self.identity().address();
match TunDevice::create(&self.config().tun, address).await {
Ok(device) => {
+66
View File
@@ -81,6 +81,51 @@ pub struct ForwardingMetrics {
pub drop_send_error_bytes: Counter,
pub originated_packets: Counter,
pub originated_bytes: Counter,
pub route_tree_up: Counter,
pub route_tree_down: Counter,
pub route_tree_down_cross: Counter,
pub route_crosslink_descend: Counter,
pub route_crosslink_ascend: Counter,
pub route_direct_peer: Counter,
}
/// Route class of a transit-forwarded packet, classified from tree
/// coordinates at the forwarding decision point. The six variants
/// partition `forwarded_packets` exactly.
///
/// Two variants are up-and-over forwards (destination not in the chosen
/// peer's subtree); they differ in whether they depend on a child
/// advertising cross-link reach *upward* to its parent:
/// - `TreeDownCross`: the chosen peer is our tree descendant, but the
/// destination is *not* in that child's subtree. The forward only fired
/// because the child advertised cross-link reach upward to us, beyond its
/// own subtree. If children advertised only their subtree upward, this
/// forward would route up instead, so its count measures how much
/// forwarding depends on the upward cross-link advertisement — the
/// dive-to-tree-child cut-through.
/// - `CrosslinkAscend`: the chosen peer is lateral (neither ancestor nor
/// descendant) and the destination is not in its subtree. This is a node
/// using its *own* cross-link, learned via the peer's split-horizon
/// advertisement to its neighbors, so it does not depend on any upward
/// advertisement. Tracked alongside `TreeDownCross` as the lateral
/// up-and-over contrast.
#[derive(Debug, Clone, Copy, PartialEq, Eq)]
pub enum RouteClass {
/// Chosen peer is our ancestor (tree-up).
TreeUp,
/// Chosen peer is our descendant and dest is in its subtree (canonical
/// tree-down).
TreeDown,
/// Chosen peer is our descendant but dest is *not* in its subtree: the
/// dive-to-tree-child cut-through enabled by upward cross-link
/// advertisement.
TreeDownCross,
/// Chosen peer is lateral and dest is in its subtree (subtree entry).
CrosslinkDescend,
/// Chosen peer is lateral and dest is not in its subtree (up-and-over).
CrosslinkAscend,
/// Chosen peer is the destination itself (degenerate direct hop).
DirectPeer,
}
impl ForwardingMetrics {
@@ -112,6 +157,21 @@ impl ForwardingMetrics {
self.originated_bytes.add(bytes as u64);
}
/// Record the route class of a transit-forwarded packet. The five
/// classes partition `forwarded_packets`, so this is called exactly
/// once per `record_forwarded` (transit chokepoint only).
#[inline]
pub fn record_route_class(&self, class: RouteClass) {
match class {
RouteClass::TreeUp => self.route_tree_up.inc(),
RouteClass::TreeDown => self.route_tree_down.inc(),
RouteClass::TreeDownCross => self.route_tree_down_cross.inc(),
RouteClass::CrosslinkDescend => self.route_crosslink_descend.inc(),
RouteClass::CrosslinkAscend => self.route_crosslink_ascend.inc(),
RouteClass::DirectPeer => self.route_direct_peer.inc(),
}
}
/// Mirror of `ForwardingStats::record_reject_bytes`: route a typed
/// forwarding rejection of `bytes` payload to its packet and byte
/// counters.
@@ -163,6 +223,12 @@ impl ForwardingMetrics {
drop_send_error_bytes: self.drop_send_error_bytes.get(),
originated_packets: self.originated_packets.get(),
originated_bytes: self.originated_bytes.get(),
route_tree_up: self.route_tree_up.get(),
route_tree_down: self.route_tree_down.get(),
route_tree_down_cross: self.route_tree_down_cross.get(),
route_crosslink_descend: self.route_crosslink_descend.get(),
route_crosslink_ascend: self.route_crosslink_ascend.get(),
route_direct_peer: self.route_direct_peer.get(),
}
}
}
+163 -9
View File
@@ -41,14 +41,18 @@ use self::routing_error_rate_limit::RoutingErrorRateLimiter;
/// `node.rekey.after_secs` remains the nominal interval (mean preserved).
pub(crate) const REKEY_JITTER_SECS: i64 = 15;
use self::wire::{
ESTABLISHED_HEADER_SIZE, FLAG_CE, FLAG_KEY_EPOCH, FLAG_SP, build_encrypted,
build_established_header, prepend_inner_header,
FLAG_CE, FLAG_KEY_EPOCH, FLAG_SP, build_encrypted, build_established_header,
prepend_inner_header,
};
// Only referenced by the unix UDP fast-path block below; on Windows the wire
// buffer is sized through build_encrypted, leaving this import otherwise unused.
#[cfg(unix)]
use self::wire::ESTABLISHED_HEADER_SIZE;
use crate::bloom::{BloomFilter, BloomState};
use crate::cache::CoordCache;
use crate::node::session::SessionEntry;
use crate::peer::{ActivePeer, PeerConnection};
#[cfg(unix)]
#[cfg(any(target_os = "linux", target_os = "macos"))]
use crate::transport::ethernet::EthernetTransport;
use crate::transport::nym::NymTransport;
use crate::transport::tcp::TcpTransport;
@@ -61,7 +65,7 @@ use crate::transport::{
use crate::tree::TreeState;
use crate::upper::hosts::HostMap;
use crate::upper::icmp_rate_limit::IcmpRateLimiter;
use crate::upper::tun::{TunError, TunOutboundRx, TunState, TunTx};
use crate::upper::tun::{TunError, TunOutboundRx, TunOutboundTx, TunState, TunTx};
use crate::utils::index::IndexAllocator;
use crate::{Config, ConfigError, Identity, IdentityError, NodeAddr, PeerIdentity};
use rand::Rng;
@@ -928,7 +932,7 @@ impl Node {
}
// Create Ethernet transport instances (Unix only — requires raw sockets)
#[cfg(unix)]
#[cfg(any(target_os = "linux", target_os = "macos"))]
{
let eth_instances: Vec<_> = self
.config()
@@ -1039,6 +1043,43 @@ impl Node {
}
}
// Android BLE: the radio lives in Kotlin; build AndroidIo over the bridge
// injected by the embedder (see ble::android_io::set_android_ble_bridge).
#[cfg(all(target_os = "android", not(test)))]
{
let ble_instances: Vec<_> = self
.config()
.transports
.ble
.iter()
.map(|(name, config)| (name.map(|s| s.to_string()), config.clone()))
.collect();
match crate::transport::ble::android_io::android_ble_bridge() {
Some(bridge) => {
for (name, ble_config) in ble_instances {
let transport_id = self.allocate_transport_id();
let io = crate::transport::ble::android_io::AndroidIo::new(
std::sync::Arc::clone(&bridge),
);
let mut ble = crate::transport::ble::BleTransport::new(
transport_id,
name,
ble_config,
io,
packet_tx.clone(),
);
ble.set_local_pubkey(self.identity().pubkey().serialize());
transports.push(TransportHandle::Ble(ble));
}
}
None => {
if !ble_instances.is_empty() {
tracing::warn!("BLE configured but no Android radio bridge injected");
}
}
}
}
transports
}
@@ -1062,7 +1103,7 @@ impl Node {
&self,
addr_str: &str,
) -> Result<(TransportId, TransportAddr), NodeError> {
#[cfg(unix)]
#[cfg(any(target_os = "linux", target_os = "macos"))]
{
let (iface, mac_str) = addr_str.split_once('/').ok_or_else(|| {
NodeError::NoTransportForType(format!(
@@ -1094,7 +1135,7 @@ impl Node {
Ok((transport_id, TransportAddr::from_bytes(&mac)))
}
#[cfg(not(unix))]
#[cfg(not(any(target_os = "linux", target_os = "macos")))]
{
Err(NodeError::NoTransportForType(
"Ethernet transport is not supported on this platform".to_string(),
@@ -1435,8 +1476,10 @@ impl Node {
/// over this node's already-shared `NodeContext` and `MetricsRegistry`.
///
/// Used at control-socket spawn time so pure-snapshot `show_*` queries
/// render off the rx_loop. Cloneable; cheap (all `Arc` clones).
pub(crate) fn control_read_handle(&self) -> crate::control::read_handle::ControlReadHandle {
/// render off the rx_loop. Cloneable; cheap (all `Arc` clones). Also the
/// public seam for embedders that run [`Self::run_rx_loop`] on a background
/// task and poll peer state via [`crate::control::read_handle::ControlReadHandle::peer_views`].
pub fn control_read_handle(&self) -> crate::control::read_handle::ControlReadHandle {
crate::control::read_handle::ControlReadHandle::new(
self.context.clone(),
self.metrics.clone(),
@@ -2666,6 +2709,86 @@ impl Node {
self.peers.get(&next_hop_id).filter(|p| p.can_send())
}
/// Classify a transit forward by route class from tree coordinates.
///
/// Called at the transit chokepoint after `find_next_hop` returns a peer,
/// so the six classes partition `forwarded_packets` exactly. The branch
/// that `find_next_hop` took (bloom vs greedy-tree) is *not* the route
/// class: a peer can be selected by either, so the cut-through splits
/// (`TreeDownCross`, `CrosslinkAscend`) are decided here from coordinates,
/// not from which branch fired.
///
/// Inputs: our coords (`tree_state.my_coords`), the chosen peer's coords
/// (`tree_state.peer_coords`), and the destination coords (re-read from the
/// coord cache, which `find_next_hop` just touched). Both the tree-down and
/// cross-link branches split on whether the destination is in the chosen
/// peer's subtree; when the dest coords are unavailable that test defaults
/// to "not in subtree", i.e. the up-and-over variant (`TreeDownCross` for a
/// descendant peer, `CrosslinkAscend` for a lateral one).
pub(crate) fn classify_forward(
&self,
dest: &NodeAddr,
chosen_peer: &NodeAddr,
) -> metrics::RouteClass {
// Degenerate: the next hop is the destination itself (Branch 2).
if chosen_peer == dest {
return metrics::RouteClass::DirectPeer;
}
let my_addr = self.node_addr();
let my_coords = self.tree_state.my_coords();
// Tree-up: the chosen peer is our ancestor.
if my_coords.has_ancestor(chosen_peer) {
return metrics::RouteClass::TreeUp;
}
// Whether the destination is in the chosen peer's subtree. Both the
// tree-down and cross-link splits below turn on this same test, so it
// is computed once. On the live transit path the dest coords are
// always present here: `find_next_hop` looks them up with an early
// return, so a coord-cache miss yields no next hop to classify (the
// caller signals `CoordsRequired` instead of forwarding). The miss
// branch below is therefore defensive — reachable only by direct
// unit-test calls — and defaults the test to "not in subtree", i.e.
// the up-and-over variant of whichever branch fires (TreeDownCross for
// a descendant peer, CrosslinkAscend for a lateral one), matching the
// original cross-link default-to-ascend.
let now_ms = std::time::SystemTime::now()
.duration_since(std::time::UNIX_EPOCH)
.map(|d| d.as_millis() as u64)
.unwrap_or(0);
let dest_in_peer_subtree = self
.coord_cache
.get(dest, now_ms)
.is_some_and(|dest_coords| dest_coords.has_ancestor(chosen_peer));
// Tree-down: the chosen peer is our descendant (we are its ancestor).
// Split by subtree membership: a dest genuinely below the child is the
// canonical tree-down; a dest *not* below it means we only forwarded
// down because the child advertised cross-link reach upward, beyond its
// own subtree — the dive-to-tree-child cut-through (TreeDownCross).
if let Some(peer_coords) = self.tree_state.peer_coords(chosen_peer)
&& peer_coords.has_ancestor(my_addr)
{
return if dest_in_peer_subtree {
metrics::RouteClass::TreeDown
} else {
metrics::RouteClass::TreeDownCross
};
}
// Cross-link (lateral): split by whether the destination is in the
// chosen peer's subtree. Descend = subtree entry; ascend = up-and-over
// via the node's own cross-link (learned from the peer's split-horizon
// advertisement, independent of any upward advertisement).
if dest_in_peer_subtree {
return metrics::RouteClass::CrosslinkDescend;
}
metrics::RouteClass::CrosslinkAscend
}
/// Select the best peer from a set of bloom filter candidates.
///
/// Uses distance from each candidate's tree coordinates to the destination
@@ -2733,6 +2856,37 @@ impl Node {
self.tun_tx.as_ref()
}
/// Set up an **app-owned TUN**: rather than FIPS creating a system TUN
/// device, the embedder (e.g. an Android `VpnService`) owns the fd and
/// exchanges IPv6 packet bytes with FIPS over the returned channels. Call
/// this after [`Node::new`] and **before** [`Self::start`] — and before
/// moving the node into a background task — exactly like
/// [`Self::control_read_handle`].
///
/// Returns `(app_outbound_tx, app_inbound_rx)`:
/// - push IPv6 packets read from the app's TUN fd into `app_outbound_tx`
/// (app → mesh); FIPS routes them to the destination node.
/// - pull IPv6 packets destined for the app's TUN fd from `app_inbound_rx`
/// (mesh → app) and write them to the fd (`recv_timeout` for clean stop).
///
/// With this set, [`Self::start`] skips system-TUN creation (it gates on
/// `tun_tx` being unset). Packets pushed into `app_outbound_tx` bypass the
/// system-TUN reader's `handle_tun_packet`, so the embedder must do what that
/// path otherwise would: push only `fd::/8`-destined IPv6 packets — FIPS no
/// longer filters the destination or emits ICMPv6 unreachable for off-mesh
/// dests — and clamp TCP MSS on outbound SYNs.
pub fn enable_app_owned_tun(&mut self) -> (TunOutboundTx, std::sync::mpsc::Receiver<Vec<u8>>) {
let tun_channel_size = self.config().node.buffers.tun_channel;
// app → mesh: the app pushes; `run_rx_loop` drains `tun_outbound_rx`.
let (outbound_tx, outbound_rx) = tokio::sync::mpsc::channel(tun_channel_size);
// mesh → app: the node writes inbound packets to `tun_tx`; the app pulls.
let (tun_tx, tun_rx) = std::sync::mpsc::channel();
self.tun_tx = Some(tun_tx);
self.tun_outbound_rx = Some(outbound_rx);
self.tun_state = TunState::Active;
(outbound_tx, tun_rx)
}
// === Sending ===
/// Encrypt and send a link-layer message to an authenticated peer.
+6
View File
@@ -229,6 +229,12 @@ pub struct ForwardingStatsSnapshot {
pub drop_send_error_bytes: u64,
pub originated_packets: u64,
pub originated_bytes: u64,
pub route_tree_up: u64,
pub route_tree_down: u64,
pub route_tree_down_cross: u64,
pub route_crosslink_descend: u64,
pub route_crosslink_ascend: u64,
pub route_direct_peer: u64,
}
#[derive(Clone, Debug, Default, Serialize)]
+345
View File
@@ -1110,3 +1110,348 @@ async fn test_routing_source_only_coords_100_nodes() {
cleanup_nodes(&mut nodes).await;
}
// === Route-class classification (transit-forward partition) ===
use crate::node::metrics::{ForwardingMetrics, RouteClass};
/// Current epoch millis, matching the cache-insert idiom used above.
fn now_ms() -> u64 {
std::time::SystemTime::now()
.duration_since(std::time::UNIX_EPOCH)
.map(|d| d.as_millis() as u64)
.unwrap_or(0)
}
#[test]
fn test_classify_forward_tree_up() {
// my_coords = [me, parent, root]; the chosen peer is our parent (an
// ancestor in our path) → tree-up.
let mut node = make_node();
let me = *node.node_addr();
let parent = make_node_addr(10);
let root = make_node_addr(1);
node.tree_state_mut()
.set_my_coords_for_test(TreeCoordinate::from_addrs(vec![me, parent, root]).unwrap());
// Destination somewhere above us; routed via the parent.
let dest = make_node_addr(50);
node.coord_cache_mut().insert(
dest,
TreeCoordinate::from_addrs(vec![dest, root]).unwrap(),
now_ms(),
);
assert_eq!(
node.classify_forward(&dest, &parent),
RouteClass::TreeUp,
"chosen peer is our ancestor"
);
}
#[test]
fn test_classify_forward_tree_down() {
// Chosen peer is our descendant: its coords name us as an ancestor.
let mut node = make_node();
let me = *node.node_addr();
let root = make_node_addr(1);
node.tree_state_mut()
.set_my_coords_for_test(TreeCoordinate::from_addrs(vec![me, root]).unwrap());
let child = make_node_addr(20);
node.tree_state_mut().update_peer(
ParentDeclaration::new(child, me, 1, 1000),
TreeCoordinate::from_addrs(vec![child, me, root]).unwrap(),
);
// Destination below the child; routed down to it.
let dest = make_node_addr(60);
node.coord_cache_mut().insert(
dest,
TreeCoordinate::from_addrs(vec![dest, child, me, root]).unwrap(),
now_ms(),
);
assert_eq!(
node.classify_forward(&dest, &child),
RouteClass::TreeDown,
"chosen peer is our descendant, dest in its subtree"
);
}
#[test]
fn test_classify_forward_tree_down_cross() {
// Chosen peer is our descendant (a tree child), but the destination is NOT
// in that child's subtree: we are diving down to the child only because it
// advertised cross-link reach upward, beyond its own subtree. This is the
// dive-to-tree-child cut-through.
let mut node = make_node();
let me = *node.node_addr();
let root = make_node_addr(1);
node.tree_state_mut()
.set_my_coords_for_test(TreeCoordinate::from_addrs(vec![me, root]).unwrap());
let child = make_node_addr(20);
node.tree_state_mut().update_peer(
ParentDeclaration::new(child, me, 1, 1000),
TreeCoordinate::from_addrs(vec![child, me, root]).unwrap(),
);
// Destination lives elsewhere (directly under root), NOT under the child;
// reachable from the child only via a cross-link.
let dest = make_node_addr(60);
node.coord_cache_mut().insert(
dest,
TreeCoordinate::from_addrs(vec![dest, root]).unwrap(),
now_ms(),
);
assert_eq!(
node.classify_forward(&dest, &child),
RouteClass::TreeDownCross,
"descendant peer, dest not in its subtree (dive-to-tree-child cut-through)"
);
}
#[test]
fn test_classify_forward_crosslink_descend() {
// Chosen peer is lateral (not in our path, we are not in its path) and the
// destination is inside the peer's subtree → cross-link descend.
let mut node = make_node();
let me = *node.node_addr();
let root = make_node_addr(1);
let sibling_parent = make_node_addr(2);
node.tree_state_mut()
.set_my_coords_for_test(TreeCoordinate::from_addrs(vec![me, root]).unwrap());
let peer = make_node_addr(30);
node.tree_state_mut().update_peer(
ParentDeclaration::new(peer, sibling_parent, 1, 1000),
TreeCoordinate::from_addrs(vec![peer, sibling_parent, root]).unwrap(),
);
// Destination is under the cross-link peer.
let dest = make_node_addr(70);
node.coord_cache_mut().insert(
dest,
TreeCoordinate::from_addrs(vec![dest, peer, sibling_parent, root]).unwrap(),
now_ms(),
);
assert_eq!(
node.classify_forward(&dest, &peer),
RouteClass::CrosslinkDescend,
"lateral peer, dest in its subtree"
);
}
#[test]
fn test_classify_forward_crosslink_ascend() {
// Chosen peer is lateral and the destination is NOT in its subtree → the
// up-and-over case (the Bloom v2 behavior delta).
let mut node = make_node();
let me = *node.node_addr();
let root = make_node_addr(1);
let sibling_parent = make_node_addr(2);
node.tree_state_mut()
.set_my_coords_for_test(TreeCoordinate::from_addrs(vec![me, root]).unwrap());
let peer = make_node_addr(40);
node.tree_state_mut().update_peer(
ParentDeclaration::new(peer, sibling_parent, 1, 1000),
TreeCoordinate::from_addrs(vec![peer, sibling_parent, root]).unwrap(),
);
// Destination lives elsewhere (under root directly), NOT under the peer.
let dest = make_node_addr(80);
node.coord_cache_mut().insert(
dest,
TreeCoordinate::from_addrs(vec![dest, root]).unwrap(),
now_ms(),
);
assert_eq!(
node.classify_forward(&dest, &peer),
RouteClass::CrosslinkAscend,
"lateral peer, dest not in its subtree"
);
}
#[test]
fn test_classify_forward_direct_peer() {
// Degenerate case: the next hop is the destination itself.
let mut node = make_node();
let me = *node.node_addr();
let root = make_node_addr(1);
node.tree_state_mut()
.set_my_coords_for_test(TreeCoordinate::from_addrs(vec![me, root]).unwrap());
let dest = make_node_addr(90);
assert_eq!(
node.classify_forward(&dest, &dest),
RouteClass::DirectPeer,
"next hop is the destination"
);
}
#[test]
fn test_route_class_partition_sums_to_forwarded() {
// The six route classes partition forwarded_packets: bumping
// record_forwarded once per record_route_class keeps the sum of the class
// counters equal to forwarded_packets.
let m = ForwardingMetrics::default();
let classes = [
RouteClass::TreeUp,
RouteClass::TreeUp,
RouteClass::TreeDown,
RouteClass::TreeDownCross,
RouteClass::TreeDownCross,
RouteClass::CrosslinkDescend,
RouteClass::CrosslinkAscend,
RouteClass::CrosslinkAscend,
RouteClass::CrosslinkAscend,
RouteClass::DirectPeer,
];
for &c in &classes {
m.record_forwarded(100);
m.record_route_class(c);
}
let snap = m.snapshot();
let class_sum = snap.route_tree_up
+ snap.route_tree_down
+ snap.route_tree_down_cross
+ snap.route_crosslink_descend
+ snap.route_crosslink_ascend
+ snap.route_direct_peer;
assert_eq!(
class_sum, snap.forwarded_packets,
"route classes must partition forwarded_packets"
);
assert_eq!(snap.route_tree_up, 2);
assert_eq!(snap.route_tree_down_cross, 2);
assert_eq!(snap.route_crosslink_ascend, 3);
assert_eq!(snap.route_direct_peer, 1);
}
// === Coord-cache invalidation on parent loss ===
//
// Parent-lost-via-peer-removal is a genuine position change and must
// surgically invalidate the coordinate cache like every other such path
// (reparent → invalidate_via_node; self-root → invalidate_other_roots).
// `make_node_addr(0)` is the network minimum, so the node's random identity
// addr is always greater than it — the reparent/child geometry is deterministic.
#[test]
fn test_parent_loss_reparent_invalidates_coord_cache() {
let mut node = make_node();
let my_addr = *node.node_addr();
let root = make_node_addr(0);
let parent = make_node_addr(1);
let alt = make_node_addr(2);
// Current parent and an alternative, both rooted at `root`.
node.tree_state_mut().update_peer(
ParentDeclaration::new(parent, root, 1, 1000),
TreeCoordinate::from_addrs(vec![parent, root]).unwrap(),
);
node.tree_state_mut().update_peer(
ParentDeclaration::new(alt, root, 1, 1000),
TreeCoordinate::from_addrs(vec![alt, root]).unwrap(),
);
// Adopt `parent`; our coords become [my_addr, parent, root], root = `root`.
node.tree_state_mut().set_parent(parent, 1, 1000);
node.tree_state_mut().recompute_coords();
assert!(!node.tree_state().is_root());
assert_eq!(node.tree_state().root(), &root);
let now_ms = std::time::SystemTime::now()
.duration_since(std::time::UNIX_EPOCH)
.map(|d| d.as_millis() as u64)
.unwrap_or(0);
// via-node class: a downstream destination that routes through us.
let downstream = make_node_addr(10);
node.coord_cache_mut().insert(
downstream,
TreeCoordinate::from_addrs(vec![downstream, my_addr, root]).unwrap(),
now_ms,
);
// survivor: same root, does not route through us.
let sibling_dest = make_node_addr(11);
node.coord_cache_mut().insert(
sibling_dest,
TreeCoordinate::from_addrs(vec![sibling_dest, alt, root]).unwrap(),
now_ms,
);
// Parent link drops; node reparents onto `alt` (still rooted at `root`).
let changed = node.handle_peer_removal_tree_cleanup(&parent);
assert!(changed);
assert_eq!(node.tree_state().my_declaration().parent_id(), &alt);
assert_eq!(node.tree_state().root(), &root);
assert!(
!node.coord_cache().contains(&downstream, now_ms),
"entry routing through us must be invalidated after reparent"
);
assert!(
node.coord_cache().contains(&sibling_dest, now_ms),
"same-root entry not routing through us must survive (surgical, not a flush)"
);
}
#[test]
fn test_parent_loss_selfroot_invalidates_coord_cache() {
let mut node = make_node();
let my_addr = *node.node_addr();
let old_root = make_node_addr(0);
let parent = make_node_addr(1);
// Adopt `parent` (rooted at `old_root`); no alternative peers exist, so a
// parent loss self-roots the node.
node.tree_state_mut().update_peer(
ParentDeclaration::new(parent, old_root, 1, 1000),
TreeCoordinate::from_addrs(vec![parent, old_root]).unwrap(),
);
node.tree_state_mut().set_parent(parent, 1, 1000);
node.tree_state_mut().recompute_coords();
assert!(!node.tree_state().is_root());
let now_ms = std::time::SystemTime::now()
.duration_since(std::time::UNIX_EPOCH)
.map(|d| d.as_millis() as u64)
.unwrap_or(0);
// via-node class: routes through us.
let downstream = make_node_addr(10);
node.coord_cache_mut().insert(
downstream,
TreeCoordinate::from_addrs(vec![downstream, my_addr, old_root]).unwrap(),
now_ms,
);
// other-roots class: on the old root, does not route through us.
let foreign = make_node_addr(11);
node.coord_cache_mut().insert(
foreign,
TreeCoordinate::from_addrs(vec![foreign, parent, old_root]).unwrap(),
now_ms,
);
// Parent link drops; no alternative parent → node self-roots.
let changed = node.handle_peer_removal_tree_cleanup(&parent);
assert!(changed);
assert!(node.tree_state().is_root());
assert_eq!(node.tree_state().root(), &my_addr);
assert!(
!node.coord_cache().contains(&downstream, now_ms),
"via-node entry must be invalidated after self-root"
);
assert!(
!node.coord_cache().contains(&foreign, now_ms),
"stale old-root entry must be invalidated after self-root"
);
}
+55
View File
@@ -1995,3 +1995,58 @@ async fn handle_msg1_admits_existing_peer_at_cap() {
"rate limiter must rebalance after the (bypass-admitted) handler returns"
);
}
/// App-owned TUN seam: `enable_app_owned_tun` wires the embedder's packet
/// channels (an Android `VpnService` owns the fd) and marks the TUN active so
/// `start()` skips system-TUN creation.
#[test]
fn app_owned_tun_seam_wires_channels() {
let mut config = crate::Config::new();
config.tun.enabled = true;
let mut node = make_node_with(config);
let (outbound_tx, tun_rx) = node.enable_app_owned_tun();
// TUN is active and the inbound (mesh→app) sender is installed, so `start()`
// will skip `TunDevice::create` (it gates on `tun_tx.is_none()`).
assert_eq!(node.tun_state(), crate::upper::tun::TunState::Active);
assert!(node.tun_tx().is_some(), "inbound sender installed");
// mesh → app: a packet the node delivers to its `tun_tx` reaches the app's rx.
let pkt = vec![0x60u8, 0, 0, 0, 0, 0];
node.tun_tx().unwrap().send(pkt.clone()).unwrap();
assert_eq!(
tun_rx
.recv_timeout(std::time::Duration::from_millis(200))
.unwrap(),
pkt,
"the app pulls the same bytes the node wrote",
);
// app → mesh: the returned sender is live (its matching rx is held by the node
// and drained by `run_rx_loop` → `handle_tun_outbound`).
assert!(outbound_tx.try_send(vec![0x60]).is_ok());
}
/// With an app-owned TUN configured, `start()` must NOT create a system TUN
/// device: it leaves `tun_name` unset (a real device records its interface name)
/// and keeps the TUN `Active` with the app-owned channels.
#[tokio::test]
async fn start_skips_system_tun_when_app_owned() {
let mut config = crate::Config::new();
config.tun.enabled = true;
let mut node = make_node_with(config);
let (_outbound_tx, _tun_rx) = node.enable_app_owned_tun();
node.start().await.unwrap();
// No system device was created (that path records the interface name); the
// app-owned TUN stayed active.
assert!(
node.tun_name().is_none(),
"app-owned TUN must not create a named system device",
);
assert_eq!(node.tun_state(), crate::upper::tun::TunState::Active);
node.stop().await.unwrap();
}
+10
View File
@@ -620,6 +620,16 @@ impl Node {
.tree
.record_reject(TreeReject::OutboundSignFailed);
}
// handle_parent_lost may promote to root OR find new parent;
// cover both invalidation classes (same as the loop-detection
// branch above). Without this, cached downstream entries keep
// our now-stale coordinate prefix until TTL — and get_and_touch
// refreshes the TTL on every routing access, so an actively
// routed stale entry never self-expires.
self.coord_cache
.invalidate_via_node(our_identity.node_addr());
self.coord_cache
.invalidate_other_roots(self.tree_state.root());
info!(
new_root = %self.tree_state.root(),
is_root = self.tree_state.is_root(),
+734
View File
@@ -0,0 +1,734 @@
//! Android BLE backend: a [`BleIo`] whose radio lives in Kotlin.
//!
//! Android's BLE APIs are Java-only, so Kotlin owns the radio (scan, advertise,
//! L2CAP listen/connect, socket read/write) and exchanges **raw bytes** with this
//! Rust backend over a byte-bridge — symmetric to how nostr-vpn's `MobileTunnel`
//! exchanges TUN packet bytes across the FFI. FIPS keeps everything above the
//! `BleIo` trait (the pool, the cross-probe tiebreaker, the pubkey exchange,
//! Noise); this backend only moves bytes and surfaces adverts.
//!
//! ## Layering
//!
//! FIPS cannot depend on the app crate (`myco-core`), so the split is:
//!
//! - [`AndroidRadio`] — an object-safe trait for the few **commands** the radio
//! must run (listen/connect/advertise/scan/close). `myco-core` implements it
//! via JNI calls into the Kotlin radio object.
//! - [`AndroidBleBridge`] — the channel machinery shared by this backend and the
//! JNI layer. `myco-core` constructs it, injects it via
//! [`set_android_ble_bridge`], and drives its `deliver_*` / `next_send`
//! methods from its `Java_..._NativeCore_*` exports.
//! - [`AndroidIo`] / [`AndroidStream`] / [`AndroidAcceptor`] / [`AndroidScanner`]
//! — the `BleIo` impl, delegating to the bridge.
//!
//! ## Direction of blocking (matches nostr-vpn's MobileTunnel)
//!
//! - **Inbound** bytes/events (Kotlin → Rust) are **pushed** non-blocking into
//! tokio channels (`deliver_recv`, `deliver_inbound`, `deliver_scan`,
//! `deliver_connect_result`); the awaiting FIPS task wakes.
//! - **Outbound** bytes (Rust → Kotlin) are **pulled, blocking with timeout**, by
//! a per-channel Kotlin writer thread via [`AndroidBleBridge::next_send`].
//! `BleStream::send` only pushes into a std channel — it never calls JNI — so
//! the byte hot path never blocks a tokio worker on a JNI upcall.
//!
//! This module is platform-agnostic Rust (no JNI here — that lives in
//! `myco-core`), so it compiles and unit-tests on the host with a mock radio.
use std::collections::HashMap;
use std::sync::atomic::{AtomicBool, AtomicI64, AtomicU16, Ordering};
use std::sync::{Arc, Mutex};
use std::time::Duration;
use tokio::sync::Mutex as AsyncMutex;
use tokio::sync::mpsc;
use tokio::sync::oneshot;
use crate::transport::TransportError;
use super::DEFAULT_PSM;
use super::addr::BleAddr;
use super::io::{BleAcceptor, BleIo, BleScanner, BleStream};
use super::psm::PsmMap;
/// Synthetic adapter label (Android does not expose a BlueZ-style adapter name;
/// identity is the pubkey, never the MAC — see ble-interop.md).
const ANDROID_ADAPTER: &str = "ble0";
/// Bound on a per-channel inbound queue and the accept/scan fan-in. Generous so
/// control events (accept/scan) are not dropped under burst; L2CAP data drops are
/// tolerable since FMP/Noise above retransmits.
const CHANNEL_CAP: usize = 256;
/// Bound on the **outbound byte** queue (Rust → Kotlin writer), fixed at channel
/// creation. A bounded queue makes `BleStream::send` backpressure (the `SyncSender`
/// blocks), propagating flow control up through FSP/MMP to the TUN and TCP rather
/// than letting an unbounded queue bufferbloat the link.
///
/// 32 is empirical, swept against the peer speedtest: 8 starved the radio's
/// connection events (~half throughput), 64 bufferbloated TCP (regressed), and 32
/// was best (~200/500 kbps up/down). The sweep was noisy and non-monotonic across
/// single runs, though — run-to-run BLE variance (RF, and whether the OS grants 2M
/// PHY / high connection priority that session) rivals the effect of this knob — so
/// 32 is the best-observed working value, to be re-validated with repeated runs +
/// PHY/interval instrumentation, not a proven optimum.
const SEND_QUEUE_CAP: usize = 32;
/// Transport default MTU, used when the OS reports an unknown (0) channel MTU.
/// Matches `DEFAULT_BLE_MTU` in `config/transport.rs`.
const DEFAULT_BLE_MTU: u16 = 2048;
// ============================================================================
// AndroidRadio — the Kotlin-implemented command surface
// ============================================================================
/// The radio commands the bridge issues to the platform. `myco-core` implements
/// this via JNI `call_method` on the Kotlin `BleRadio` object. Object-safe so the
/// bridge can hold `Arc<dyn AndroidRadio>`.
///
/// These are the **control** plane only — never the byte hot path. Outbound bytes
/// are pulled by Kotlin via [`AndroidBleBridge::next_send`]; inbound bytes are
/// pushed by Kotlin via [`AndroidBleBridge::deliver_recv`].
pub trait AndroidRadio: Send + Sync {
/// Open an insecure L2CAP listener and return the OS-assigned PSM (0 = failure).
fn listen(&self) -> u16;
/// Begin dialing `addr` at `psm`. The outcome is delivered asynchronously via
/// [`AndroidBleBridge::deliver_connect_result`] keyed by `connect_id`.
fn connect(&self, connect_id: i64, addr: &BleAddr, psm: u16);
/// Advertise the FIPS service UUID plus our listener `psm` (16-bit LE
/// service-data — see [`super::psm`]).
fn start_advertising(&self, psm: u16);
fn stop_advertising(&self);
/// Scan for the FIPS UUID; deliver hits via [`AndroidBleBridge::deliver_scan`].
fn start_scanning(&self);
fn stop_scanning(&self);
/// Close the L2CAP socket for `ch_id` (called when FIPS drops the stream).
fn close_channel(&self, ch_id: i64);
}
/// One discovered scan advert (address / learned PSM / RSSI), for the developer
/// UI's "discovered devices" list.
#[derive(Debug, Clone)]
pub struct AdvertView {
/// `BleAddr` string (`adapter/AA:BB:..`); the MAC rotates with privacy.
pub addr: String,
/// The peer's advertised listener PSM (0 if not present in the advert).
pub psm: u16,
/// Signal strength in dBm (negative; closer ≈ less negative).
pub rssi: i32,
}
// ============================================================================
// AndroidBleBridge — the shared channel machinery
// ============================================================================
/// The half of a channel kept by the bridge (the JNI-facing ends).
struct ChannelState {
/// Kotlin-pushed inbound bytes land here; the stream's `recv` awaits them.
recv_tx: mpsc::Sender<Vec<u8>>,
/// `BleStream::send` pushes here; the Kotlin writer thread pulls via `next_send`.
/// `Arc` so `next_send` can clone it out and release the channels lock before
/// blocking on `recv_timeout`.
send_rx: Arc<Mutex<std::sync::mpsc::Receiver<Vec<u8>>>>,
closed: Arc<AtomicBool>,
}
/// The half of a channel handed to the `BleStream` (the FIPS-facing ends).
struct StreamEndpoints {
ch_id: i64,
remote: BleAddr,
send_mtu: u16,
recv_mtu: u16,
recv_rx: mpsc::Receiver<Vec<u8>>,
send_tx: std::sync::mpsc::SyncSender<Vec<u8>>,
closed: Arc<AtomicBool>,
}
/// Channel machinery shared between [`AndroidIo`] and the JNI layer in `myco-core`.
///
/// Constructed by `myco-core` with a concrete [`AndroidRadio`], injected via
/// [`set_android_ble_bridge`], and driven by its `deliver_*` / `next_send`
/// methods from the JNI exports.
pub struct AndroidBleBridge {
radio: Arc<dyn AndroidRadio>,
next_id: AtomicI64,
/// Our own OS-assigned listener PSM, learned from `radio.listen()`.
local_psm: AtomicU16,
/// Learned peer PSMs (advert service-data), consulted on `connect`.
psm_map: PsmMap,
/// Latest scan advert per address (PSM + RSSI), for the developer UI's
/// "discovered" list. Re-learned each scan cycle (addresses rotate).
adverts: Mutex<HashMap<BleAddr, (u16, i32)>>,
channels: Mutex<HashMap<i64, ChannelState>>,
/// connect_id → result slot for an in-flight outbound dial.
connects: Mutex<HashMap<i64, oneshot::Sender<StreamEndpoints>>>,
/// Inbound-accept fan-in; the acceptor takes the receiver once.
accept_tx: mpsc::Sender<StreamEndpoints>,
accept_rx: Mutex<Option<mpsc::Receiver<StreamEndpoints>>>,
/// Scan fan-in; the scanner takes the receiver once.
scan_tx: mpsc::Sender<BleAddr>,
scan_rx: Mutex<Option<mpsc::Receiver<BleAddr>>>,
}
impl AndroidBleBridge {
/// Build a bridge over a concrete radio.
pub fn new(radio: Arc<dyn AndroidRadio>) -> Arc<Self> {
let (accept_tx, accept_rx) = mpsc::channel(CHANNEL_CAP);
let (scan_tx, scan_rx) = mpsc::channel(CHANNEL_CAP);
Arc::new(Self {
radio,
next_id: AtomicI64::new(1),
local_psm: AtomicU16::new(DEFAULT_PSM),
psm_map: PsmMap::new(),
adverts: Mutex::new(HashMap::new()),
channels: Mutex::new(HashMap::new()),
connects: Mutex::new(HashMap::new()),
accept_tx,
accept_rx: Mutex::new(Some(accept_rx)),
scan_tx,
scan_rx: Mutex::new(Some(scan_rx)),
})
}
fn lock_channels(&self) -> std::sync::MutexGuard<'_, HashMap<i64, ChannelState>> {
self.channels.lock().unwrap_or_else(|e| e.into_inner())
}
/// Allocate a channel id and wire its two halves, registering the
/// bridge-facing half and returning the FIPS-facing half.
fn make_channel(&self, remote: BleAddr, send_mtu: u16, recv_mtu: u16) -> StreamEndpoints {
let ch_id = self.next_id.fetch_add(1, Ordering::Relaxed);
let (recv_tx, recv_rx) = mpsc::channel(CHANNEL_CAP);
let (send_tx, send_rx) = std::sync::mpsc::sync_channel(SEND_QUEUE_CAP);
let closed = Arc::new(AtomicBool::new(false));
self.lock_channels().insert(
ch_id,
ChannelState {
recv_tx,
send_rx: Arc::new(Mutex::new(send_rx)),
closed: Arc::clone(&closed),
},
);
StreamEndpoints {
ch_id,
remote,
send_mtu: if send_mtu == 0 {
DEFAULT_BLE_MTU
} else {
send_mtu
},
recv_mtu: if recv_mtu == 0 {
DEFAULT_BLE_MTU
} else {
recv_mtu
},
recv_rx,
send_tx,
closed,
}
}
// --- JNI-facing push/pull surface (called by myco-core's exports) ---
/// Kotlin accepted a new inbound L2CAP channel. Returns the allocated `ch_id`.
pub fn deliver_inbound(&self, remote: BleAddr, send_mtu: u16, recv_mtu: u16) -> i64 {
let ep = self.make_channel(remote, send_mtu, recv_mtu);
let ch_id = ep.ch_id;
if self.accept_tx.try_send(ep).is_err() {
// Acceptor gone or saturated: reclaim the half-registered channel.
self.lock_channels().remove(&ch_id);
return 0;
}
ch_id
}
/// Kotlin finished (or failed) an outbound dial started by `radio.connect`.
/// Returns the allocated `ch_id` on success, else 0.
pub fn deliver_connect_result(
&self,
connect_id: i64,
ok: bool,
remote: BleAddr,
send_mtu: u16,
recv_mtu: u16,
) -> i64 {
let waiter = self
.connects
.lock()
.unwrap_or_else(|e| e.into_inner())
.remove(&connect_id);
let Some(tx) = waiter else { return 0 };
if !ok {
drop(tx); // dropping the sender wakes the awaiting connect() as an error
return 0;
}
let ep = self.make_channel(remote, send_mtu, recv_mtu);
let ch_id = ep.ch_id;
if tx.send(ep).is_err() {
self.lock_channels().remove(&ch_id);
return 0;
}
ch_id
}
/// Kotlin discovered a FIPS peer advertising `psm` (its OS-assigned listener
/// PSM) at signal strength `rssi` (dBm). Learns the per-peer PSM, records the
/// advert for the developer UI, and surfaces the address to the scanner.
pub fn deliver_scan(&self, addr: BleAddr, psm: u16, rssi: i32) {
if psm != 0 {
self.psm_map.learn(&addr, psm);
}
self.adverts
.lock()
.unwrap_or_else(|e| e.into_inner())
.insert(addr.clone(), (psm, rssi));
let _ = self.scan_tx.try_send(addr);
}
/// Snapshot of the current scan adverts (address / PSM / RSSI) for the
/// developer UI.
pub fn advert_views(&self) -> Vec<AdvertView> {
self.adverts
.lock()
.unwrap_or_else(|e| e.into_inner())
.iter()
.map(|(addr, (psm, rssi))| AdvertView {
addr: addr.to_string_repr(),
psm: *psm,
rssi: *rssi,
})
.collect()
}
/// Drop recorded adverts (called at the start of a scan cycle).
pub fn clear_adverts(&self) {
self.adverts
.lock()
.unwrap_or_else(|e| e.into_inner())
.clear();
}
/// Kotlin read one L2CAP packet for `ch_id`. Returns false if the channel is
/// unknown/closed (Kotlin should then stop its reader).
pub fn deliver_recv(&self, ch_id: i64, data: &[u8]) -> bool {
let tx = self.lock_channels().get(&ch_id).map(|c| c.recv_tx.clone());
match tx {
Some(tx) => tx.try_send(data.to_vec()).is_ok(),
None => false,
}
}
/// Kotlin's per-channel writer thread pulls the next outbound packet, blocking
/// up to `timeout`. `None` = timed out (Kotlin loops) or the channel is gone.
pub fn next_send(&self, ch_id: i64, timeout: Duration) -> Option<Vec<u8>> {
// Clone the per-channel receiver Arc + closed flag, then release the
// channels lock before blocking on recv_timeout (so close/create on other
// channels aren't stalled for up to `timeout`).
let (send_rx, closed) = {
let guard = self.lock_channels();
let state = guard.get(&ch_id)?;
(Arc::clone(&state.send_rx), Arc::clone(&state.closed))
};
let rx = send_rx.lock().unwrap_or_else(|e| e.into_inner());
match rx.recv_timeout(timeout) {
Ok(bytes) => Some(bytes),
Err(std::sync::mpsc::RecvTimeoutError::Timeout) => None,
// The send half (the BleStream) was dropped — the stream is gone
// (e.g. the node stopped). Mark closed so the next channel_open()
// returns false and the Kotlin writer thread exits.
Err(std::sync::mpsc::RecvTimeoutError::Disconnected) => {
closed.store(true, Ordering::Relaxed);
None
}
}
}
/// Kotlin reports `ch_id` closed (EOF / socket gone). Wakes the stream's
/// `recv` with a zero-length read (FIPS treats that as peer-closed).
pub fn channel_closed(&self, ch_id: i64) {
if let Some(state) = self.lock_channels().remove(&ch_id) {
state.closed.store(true, Ordering::Relaxed);
// Dropping recv_tx closes the stream's recv_rx → recv() returns Ok(0).
drop(state);
}
}
/// Whether `ch_id` is still open (registered and not marked closed). The JNI
/// `next_send` export uses this to tell a timeout (loop again) from a closed
/// channel (stop the writer thread).
pub fn channel_open(&self, ch_id: i64) -> bool {
self.lock_channels()
.get(&ch_id)
.map(|s| !s.closed.load(Ordering::Relaxed))
.unwrap_or(false)
}
}
// ============================================================================
// Global injection seam
// ============================================================================
static BRIDGE: Mutex<Option<Arc<AndroidBleBridge>>> = Mutex::new(None);
/// Inject the process-wide bridge before `Node::new` / start (one radio per
/// process). Replaceable so a stop-then-start cycle (BLE toggled off then on)
/// can inject a fresh bridge for the rebuilt node.
pub fn set_android_ble_bridge(bridge: Arc<AndroidBleBridge>) {
*BRIDGE.lock().unwrap_or_else(|e| e.into_inner()) = Some(bridge);
}
/// The injected bridge, if any. The node's BLE construction arm reads this.
pub fn android_ble_bridge() -> Option<Arc<AndroidBleBridge>> {
BRIDGE.lock().unwrap_or_else(|e| e.into_inner()).clone()
}
// ============================================================================
// BleIo implementation
// ============================================================================
/// macOS/Android-style external-radio backend over [`AndroidBleBridge`].
pub struct AndroidIo {
bridge: Arc<AndroidBleBridge>,
}
impl AndroidIo {
pub fn new(bridge: Arc<AndroidBleBridge>) -> Self {
Self { bridge }
}
}
/// One live L2CAP channel.
pub struct AndroidStream {
ch_id: i64,
remote: BleAddr,
send_mtu: u16,
recv_mtu: u16,
recv_rx: AsyncMutex<mpsc::Receiver<Vec<u8>>>,
send_tx: std::sync::mpsc::SyncSender<Vec<u8>>,
closed: Arc<AtomicBool>,
radio: Arc<dyn AndroidRadio>,
}
impl AndroidStream {
fn from_endpoints(ep: StreamEndpoints, radio: Arc<dyn AndroidRadio>) -> Self {
Self {
ch_id: ep.ch_id,
remote: ep.remote,
send_mtu: ep.send_mtu,
recv_mtu: ep.recv_mtu,
recv_rx: AsyncMutex::new(ep.recv_rx),
send_tx: ep.send_tx,
closed: ep.closed,
radio,
}
}
}
impl Drop for AndroidStream {
fn drop(&mut self) {
self.closed.store(true, Ordering::Relaxed);
self.radio.close_channel(self.ch_id);
}
}
impl BleStream for AndroidStream {
async fn send(&self, data: &[u8]) -> Result<(), TransportError> {
if self.closed.load(Ordering::Relaxed) {
return Err(TransportError::Io(std::io::Error::other(
"BLE channel closed",
)));
}
// Pure channel push — no JNI on the hot path. The Kotlin writer thread
// pulls this via the bridge's next_send and writes the socket.
//
// Backpressure rather than drop on a full queue. The queue is deliberately
// shallow (SEND_QUEUE_CAP) so it can't bufferbloat the BLE link; awaiting a
// free slot here propagates flow control up through FSP/MMP to the TUN and
// TCP — keeping RTT low *without* the loss-driven throughput collapse that a
// shallow tail-drop queue causes.
let mut payload = data.to_vec();
loop {
if self.closed.load(Ordering::Relaxed) {
return Err(TransportError::Io(std::io::Error::other(
"BLE channel closed",
)));
}
match self.send_tx.try_send(payload) {
Ok(()) => return Ok(()),
Err(std::sync::mpsc::TrySendError::Full(p)) => {
payload = p;
tokio::time::sleep(std::time::Duration::from_millis(2)).await;
}
Err(std::sync::mpsc::TrySendError::Disconnected(_)) => {
return Err(TransportError::Io(std::io::Error::other(
"BLE send: peer closed",
)));
}
}
}
}
async fn recv(&self, buf: &mut [u8]) -> Result<usize, TransportError> {
match self.recv_rx.lock().await.recv().await {
Some(packet) => {
let n = packet.len().min(buf.len());
buf[..n].copy_from_slice(&packet[..n]);
Ok(n)
}
// Sender dropped (channel closed) → peer-closed, per the BleStream
// contract (a zero-length recv means the peer closed).
None => Ok(0),
}
}
fn send_mtu(&self) -> u16 {
self.send_mtu
}
fn recv_mtu(&self) -> u16 {
self.recv_mtu
}
fn remote_addr(&self) -> &BleAddr {
&self.remote
}
}
/// Yields inbound channels Kotlin accepted.
pub struct AndroidAcceptor {
rx: Option<mpsc::Receiver<StreamEndpoints>>,
radio: Arc<dyn AndroidRadio>,
}
impl BleAcceptor for AndroidAcceptor {
type Stream = AndroidStream;
async fn accept(&mut self) -> Result<AndroidStream, TransportError> {
match self.rx.as_mut() {
Some(rx) => match rx.recv().await {
Some(ep) => Ok(AndroidStream::from_endpoints(ep, Arc::clone(&self.radio))),
None => std::future::pending().await, // fan-in closed; idle
},
None => std::future::pending().await, // acceptor already consumed
}
}
}
/// Yields discovered FIPS peers (the learned PSM is captured into the bridge map).
pub struct AndroidScanner {
rx: Option<mpsc::Receiver<BleAddr>>,
}
impl BleScanner for AndroidScanner {
async fn next(&mut self) -> Option<BleAddr> {
match self.rx.as_mut() {
Some(rx) => rx.recv().await,
None => None,
}
}
}
impl BleIo for AndroidIo {
type Stream = AndroidStream;
type Acceptor = AndroidAcceptor;
type Scanner = AndroidScanner;
async fn listen(&self, _psm: u16) -> Result<AndroidAcceptor, TransportError> {
// Android assigns the listener PSM; the `psm` arg from FIPS is ignored.
let os_psm = self.bridge.radio.listen();
if os_psm != 0 {
self.bridge.local_psm.store(os_psm, Ordering::Relaxed);
}
let rx = self
.bridge
.accept_rx
.lock()
.unwrap_or_else(|e| e.into_inner())
.take();
Ok(AndroidAcceptor {
rx,
radio: Arc::clone(&self.bridge.radio),
})
}
async fn connect(&self, addr: &BleAddr, psm: u16) -> Result<AndroidStream, TransportError> {
// Substitute the learned per-peer PSM for this address, if known.
let dial_psm = self.bridge.psm_map.resolve(addr, psm);
let connect_id = self.bridge.next_id.fetch_add(1, Ordering::Relaxed);
let (tx, rx) = oneshot::channel();
self.bridge
.connects
.lock()
.unwrap_or_else(|e| e.into_inner())
.insert(connect_id, tx);
self.bridge.radio.connect(connect_id, addr, dial_psm);
// FIPS already wraps connect() in a timeout, so we just await the result.
match rx.await {
Ok(ep) => Ok(AndroidStream::from_endpoints(
ep,
Arc::clone(&self.bridge.radio),
)),
Err(_) => {
self.bridge
.connects
.lock()
.unwrap_or_else(|e| e.into_inner())
.remove(&connect_id);
Err(TransportError::Io(std::io::Error::other(format!(
"BLE connect to {addr} failed"
))))
}
}
}
async fn start_advertising(&self) -> Result<(), TransportError> {
self.bridge
.radio
.start_advertising(self.bridge.local_psm.load(Ordering::Relaxed));
Ok(())
}
async fn stop_advertising(&self) -> Result<(), TransportError> {
self.bridge.radio.stop_advertising();
Ok(())
}
async fn start_scanning(&self) -> Result<AndroidScanner, TransportError> {
// Re-learn PSMs / adverts each scan cycle (addresses rotate with MAC).
self.bridge.psm_map.clear();
self.bridge.clear_adverts();
self.bridge.radio.start_scanning();
let rx = self
.bridge
.scan_rx
.lock()
.unwrap_or_else(|e| e.into_inner())
.take();
Ok(AndroidScanner { rx })
}
fn local_addr(&self) -> Result<BleAddr, TransportError> {
Ok(BleAddr {
adapter: ANDROID_ADAPTER.to_string(),
device: [0, 0, 0, 0, 0, 0],
})
}
fn adapter_name(&self) -> &str {
ANDROID_ADAPTER
}
}
#[cfg(test)]
mod tests {
use super::*;
use std::sync::atomic::AtomicU16 as TestAtomicU16;
/// A mock radio that records commands and lets the test drive the bridge.
#[derive(Default)]
struct MockRadio {
listen_psm: TestAtomicU16,
scanning: AtomicBool,
advertising_psm: TestAtomicU16,
}
impl AndroidRadio for MockRadio {
fn listen(&self) -> u16 {
self.listen_psm.load(Ordering::Relaxed)
}
fn connect(&self, _connect_id: i64, _addr: &BleAddr, _psm: u16) {}
fn start_advertising(&self, psm: u16) {
self.advertising_psm.store(psm, Ordering::Relaxed);
}
fn stop_advertising(&self) {}
fn start_scanning(&self) {
self.scanning.store(true, Ordering::Relaxed);
}
fn stop_scanning(&self) {}
fn close_channel(&self, _ch_id: i64) {}
}
fn addr(n: u8) -> BleAddr {
BleAddr {
adapter: ANDROID_ADAPTER.to_string(),
device: [0xAA, 0xBB, 0xCC, 0xDD, 0xEE, n],
}
}
#[tokio::test]
async fn inbound_channel_recv_and_close() {
let radio = Arc::new(MockRadio::default());
let bridge = AndroidBleBridge::new(radio.clone());
let io = AndroidIo::new(Arc::clone(&bridge));
let mut acceptor = io.listen(0).await.unwrap();
// Kotlin accepts an inbound channel, then pushes a packet.
let ch_id = bridge.deliver_inbound(addr(1), 512, 512);
assert!(ch_id > 0);
assert!(bridge.deliver_recv(ch_id, b"hello"));
let stream = acceptor.accept().await.unwrap();
assert_eq!(stream.remote_addr(), &addr(1));
assert_eq!(stream.send_mtu(), 512);
let mut buf = [0u8; 64];
let n = stream.recv(&mut buf).await.unwrap();
assert_eq!(&buf[..n], b"hello");
// Closing the channel makes the next recv return 0 (peer closed).
bridge.channel_closed(ch_id);
let n = stream.recv(&mut buf).await.unwrap();
assert_eq!(n, 0);
}
#[tokio::test]
async fn outbound_send_is_pulled_by_next_send() {
let radio = Arc::new(MockRadio::default());
let bridge = AndroidBleBridge::new(radio.clone());
// Simulate an accepted channel and grab its stream.
let mut acceptor = AndroidIo::new(Arc::clone(&bridge)).listen(0).await.unwrap();
let ch_id = bridge.deliver_inbound(addr(2), 0, 0);
let stream = acceptor.accept().await.unwrap();
// 0 MTU falls back to the transport default.
assert!(stream.send_mtu() > 0);
stream.send(b"out").await.unwrap();
let pulled = bridge.next_send(ch_id, Duration::from_millis(100)).unwrap();
assert_eq!(pulled, b"out");
// Nothing more queued → times out (None).
assert!(bridge.next_send(ch_id, Duration::from_millis(10)).is_none());
}
#[tokio::test]
async fn scan_learns_psm_and_connect_substitutes_it() {
let radio = Arc::new(MockRadio::default());
let bridge = AndroidBleBridge::new(radio.clone());
let io = AndroidIo::new(Arc::clone(&bridge));
let mut scanner = io.start_scanning().await.unwrap();
assert!(radio.scanning.load(Ordering::Relaxed));
bridge.deliver_scan(addr(3), 0x00C1, -50);
assert_eq!(scanner.next().await, Some(addr(3)));
// The advert is also recorded for the developer UI.
let adverts = bridge.advert_views();
assert_eq!(adverts.len(), 1);
assert_eq!(adverts[0].psm, 0x00C1);
assert_eq!(adverts[0].rssi, -50);
// The learned PSM is what a later dial would use (over FIPS's default).
assert_eq!(bridge.psm_map.resolve(&addr(3), DEFAULT_PSM), 0x00C1);
}
#[tokio::test]
async fn advertise_uses_os_assigned_listen_psm() {
let radio = Arc::new(MockRadio::default());
radio.listen_psm.store(0x0099, Ordering::Relaxed);
let bridge = AndroidBleBridge::new(radio.clone());
let io = AndroidIo::new(Arc::clone(&bridge));
let _ = io.listen(0).await.unwrap(); // learns the OS PSM
io.start_advertising().await.unwrap();
assert_eq!(radio.advertising_psm.load(Ordering::Relaxed), 0x0099);
}
}
+94 -50
View File
@@ -1,9 +1,11 @@
//! BLE L2CAP Transport Implementation
//!
//! Provides BLE-based transport for FIPS peer communication using L2CAP
//! Connection-Oriented Channels (CoC) in SeqPacket mode. L2CAP CoC
//! preserves message boundaries (unlike TCP byte streams), so no FMP
//! framing is needed — each send/recv is one FIPS packet.
//! Connection-Oriented Channels (CoC). BlueZ (SeqPacket) preserves message
//! boundaries, but stream-oriented backends (Android `BluetoothSocket`,
//! CoreBluetooth) do not, so the receive path recovers FIPS packet boundaries
//! with the shared FMP framer (`tcp::stream::read_fmp_packet`) over a
//! [`stream_read::BleStreamRead`] adapter — reliable on every platform.
//!
//! ## Architecture
//!
@@ -22,7 +24,15 @@ pub mod addr;
pub mod discovery;
pub mod io;
pub mod pool;
pub mod psm;
pub mod stats;
pub mod stream_read;
// The Android backend (radio in Kotlin, bytes over a bridge). Compiled on
// Android, and under `cfg(test)` on any host so its channel logic is unit-tested
// without a device. Not built into non-test desktop builds.
#[cfg(any(target_os = "android", test))]
pub mod android_io;
use super::{
ConnectionState, DiscoveredPeer, PacketTx, ReceivedPacket, Transport, TransportAddr,
@@ -35,6 +45,9 @@ use discovery::DiscoveryBuffer;
use io::{BleIo, BleScanner, BleStream};
use pool::{BleConnection, ConnectionPool};
use stats::BleStats;
use stream_read::BleStreamRead;
use crate::transport::tcp::stream::{StreamError, read_fmp_packet};
use secp256k1::XOnlyPublicKey;
use std::collections::HashMap;
@@ -55,7 +68,12 @@ pub const DEFAULT_PSM: u16 = 0x0085;
#[cfg(all(bluer_available, not(test)))]
pub type DefaultBleTransport = BleTransport<io::BluerIo>;
#[cfg(any(not(bluer_available), test))]
// Android: the Kotlin-radio backend over the byte-bridge.
#[cfg(all(target_os = "android", not(test)))]
pub type DefaultBleTransport = BleTransport<android_io::AndroidIo>;
// Everything else (macOS/musl/host) and all test builds: the in-memory mock.
#[cfg(any(all(not(bluer_available), not(target_os = "android")), test))]
pub type DefaultBleTransport = BleTransport<io::MockBleIo>;
// ============================================================================
@@ -364,9 +382,15 @@ impl<I: BleIo> BleTransport<I> {
}
};
// One buffering reader per connection, shared by the pubkey exchange
// and the receive loop so coalesced bytes are never dropped.
let recv_mtu = stream.recv_mtu();
let stream = Arc::new(stream);
let mut reader = BleStreamRead::new(Arc::clone(&stream), recv_mtu);
// Pre-handshake pubkey exchange (temporary, pre-XX)
if let Some(ref our_pubkey) = self.local_pubkey {
match pubkey_exchange(&stream, our_pubkey).await {
match pubkey_exchange(&mut reader, our_pubkey).await {
Ok(peer_pubkey) => {
debug!(addr = %addr, "BLE outbound pubkey exchange complete");
self.discovery_buffer
@@ -379,24 +403,26 @@ impl<I: BleIo> BleTransport<I> {
}
}
self.promote_connection(addr, &ble_addr, stream).await
self.promote_connection(addr, &ble_addr, stream, reader)
.await
}
/// Promote a newly established stream into the connection pool.
///
/// Spawns the receive loop and inserts into the pool with eviction.
/// Spawns the receive loop (driven by `reader`) and inserts into the pool
/// with eviction.
async fn promote_connection(
&self,
addr: &TransportAddr,
ble_addr: &BleAddr,
stream: I::Stream,
stream: Arc<I::Stream>,
reader: BleStreamRead<I::Stream>,
) -> Result<(), TransportError> {
let send_mtu = stream.send_mtu();
let recv_mtu = stream.recv_mtu();
let stream = Arc::new(stream);
let recv_task = tokio::spawn(receive_loop(
Arc::clone(&stream),
reader,
addr.clone(),
Arc::clone(&self.pool),
self.packet_tx.clone(),
@@ -484,9 +510,16 @@ impl<I: BleIo> BleTransport<I> {
match result {
Ok(Ok(stream)) => {
let send_mtu = stream.send_mtu();
let recv_mtu = stream.recv_mtu();
let stream = Arc::new(stream);
// Shared reader: pubkey exchange + receive loop read the
// same buffer so coalesced bytes survive the handoff.
let mut reader = BleStreamRead::new(Arc::clone(&stream), recv_mtu);
// Pre-handshake pubkey exchange (temporary, pre-XX)
if let Some(ref our_pubkey) = local_pubkey {
match pubkey_exchange(&stream, our_pubkey).await {
match pubkey_exchange(&mut reader, our_pubkey).await {
Ok(peer_pubkey) => {
debug!(addr = %addr_clone, "BLE outbound pubkey exchange complete");
discovery_buffer.add_peer_with_pubkey(&ble_addr, peer_pubkey);
@@ -501,12 +534,8 @@ impl<I: BleIo> BleTransport<I> {
}
}
let send_mtu = stream.send_mtu();
let recv_mtu = stream.recv_mtu();
let stream = Arc::new(stream);
let recv_task = tokio::spawn(receive_loop(
Arc::clone(&stream),
reader,
addr_clone.clone(),
Arc::clone(&pool),
packet_tx,
@@ -677,31 +706,38 @@ const PUBKEY_EXCHANGE_TIMEOUT_SECS: u64 = 5;
/// Exchange public keys over a newly established L2CAP connection.
///
/// Both sides send `[0x00][our_pubkey:32]` and receive the peer's.
/// Both sides send `[0x00][our_pubkey:32]` and receive the peer's. Reads go
/// through the connection's [`BleStreamRead`] buffer so a peer that fragments
/// the 33-byte message (stream-oriented backends) is read correctly, and any
/// bytes it coalesced after the pubkey stay buffered for the receive loop.
/// Returns the peer's XOnlyPublicKey on success.
async fn pubkey_exchange<S: BleStream>(
stream: &S,
reader: &mut BleStreamRead<S>,
local_pubkey: &[u8; 32],
) -> Result<XOnlyPublicKey, TransportError> {
use tokio::io::AsyncReadExt;
// Send our pubkey
let mut msg = [0u8; PUBKEY_EXCHANGE_SIZE];
msg[0] = PUBKEY_EXCHANGE_PREFIX;
msg[1..].copy_from_slice(local_pubkey);
stream.send(&msg).await?;
reader.stream().send(&msg).await?;
// Receive peer's pubkey (with timeout to prevent indefinite blocking)
// Receive peer's pubkey (with timeout to prevent indefinite blocking).
// read_exact reassembles across fragmented recvs and leaves any trailing
// bytes in the reader's buffer.
let mut buf = [0u8; PUBKEY_EXCHANGE_SIZE];
let timeout = std::time::Duration::from_secs(PUBKEY_EXCHANGE_TIMEOUT_SECS);
let n = match tokio::time::timeout(timeout, stream.recv(&mut buf)).await {
Ok(result) => result?,
match tokio::time::timeout(timeout, reader.read_exact(&mut buf)).await {
Ok(Ok(_)) => {}
Ok(Err(e)) => {
return Err(TransportError::RecvFailed(format!(
"pubkey exchange: {}",
e
)));
}
Err(_) => return Err(TransportError::Timeout),
};
if n != PUBKEY_EXCHANGE_SIZE {
return Err(TransportError::RecvFailed(format!(
"pubkey exchange: expected {} bytes, got {}",
PUBKEY_EXCHANGE_SIZE, n
)));
}
if buf[0] != PUBKEY_EXCHANGE_PREFIX {
return Err(TransportError::RecvFailed(format!(
"pubkey exchange: bad prefix 0x{:02X}",
@@ -751,10 +787,13 @@ async fn accept_loop<A>(
let send_mtu = stream.send_mtu();
let recv_mtu = stream.recv_mtu();
let stream = Arc::new(stream);
// Shared reader for pubkey exchange + receive loop.
let mut reader = BleStreamRead::new(Arc::clone(&stream), recv_mtu);
// Pre-handshake pubkey exchange (temporary, pre-XX)
if let Some(ref our_pubkey) = local_pubkey {
match pubkey_exchange(&stream, our_pubkey).await {
match pubkey_exchange(&mut reader, our_pubkey).await {
Ok(peer_pubkey) => {
debug!(addr = %ta, "BLE inbound pubkey exchange complete");
discovery_buffer.add_peer_with_pubkey(&addr, peer_pubkey);
@@ -780,11 +819,9 @@ async fn accept_loop<A>(
}
}
let stream = Arc::new(stream);
// Spawn receive loop
let recv_task = tokio::spawn(receive_loop(
Arc::clone(&stream),
reader,
ta.clone(),
Arc::clone(&pool),
packet_tx.clone(),
@@ -829,8 +866,14 @@ async fn accept_loop<A>(
}
/// Receive loop: reads packets from a BLE stream and delivers to node.
async fn receive_loop<S: BleStream>(
stream: Arc<S>,
///
/// Recovers FIPS packet boundaries from the byte stream via the shared FMP
/// framer ([`read_fmp_packet`]) rather than assuming one `recv` is one packet.
/// L2CAP only preserves SDU boundaries on BlueZ (SeqPacket); stream-oriented
/// backends (Android, CoreBluetooth) fragment and coalesce, so the framer's
/// length-prefixed reads are what make delivery reliable across platforms.
async fn receive_loop<S: BleStream + 'static>(
mut reader: BleStreamRead<S>,
addr: TransportAddr,
pool: Arc<Mutex<ConnectionPool<Arc<S>>>>,
packet_tx: PacketTx,
@@ -838,21 +881,21 @@ async fn receive_loop<S: BleStream>(
stats: Arc<BleStats>,
recv_mtu: u16,
) {
let mut buf = vec![0u8; recv_mtu as usize];
loop {
match stream.recv(&mut buf).await {
Ok(0) => {
debug!(addr = %addr, "BLE connection closed by peer");
break;
}
Ok(n) => {
stats.record_recv(n);
let packet = ReceivedPacket::new(transport_id, addr.clone(), buf[..n].to_vec());
if packet_tx.send(packet).await.is_err() {
match read_fmp_packet(&mut reader, recv_mtu).await {
Ok(packet) => {
stats.record_recv(packet.len());
let received = ReceivedPacket::new(transport_id, addr.clone(), packet);
if packet_tx.send(received).await.is_err() {
trace!("BLE packet_tx closed, stopping receive loop");
break;
}
}
// Clean peer close (EOF at a packet boundary) is expected, not an error.
Err(StreamError::Io(e)) if e.kind() == std::io::ErrorKind::UnexpectedEof => {
debug!(addr = %addr, "BLE connection closed by peer");
break;
}
Err(e) => {
debug!(addr = %addr, error = %e, "BLE receive error");
stats.record_recv_error();
@@ -986,7 +1029,12 @@ async fn scan_probe_loop<I: io::BleIo>(
// Pubkey exchange, then promote connection to pool
let ta = addr.to_transport_addr();
match pubkey_exchange(&stream, &our_pubkey).await {
let send_mtu = stream.send_mtu();
let recv_mtu = stream.recv_mtu();
let stream = Arc::new(stream);
// Shared reader for pubkey exchange + receive loop.
let mut reader = BleStreamRead::new(Arc::clone(&stream), recv_mtu);
match pubkey_exchange(&mut reader, &our_pubkey).await {
Ok(peer_pubkey) => {
debug!(addr = %addr, "BLE probe complete");
@@ -1005,12 +1053,8 @@ async fn scan_probe_loop<I: io::BleIo>(
}
// Promote connection to pool — no second L2CAP connect needed
let send_mtu = stream.send_mtu();
let recv_mtu = stream.recv_mtu();
let stream = Arc::new(stream);
let recv_task = tokio::spawn(receive_loop(
Arc::clone(&stream),
reader,
ta.clone(),
Arc::clone(&pool),
packet_tx.clone(),
+192
View File
@@ -0,0 +1,192 @@
//! Per-peer PSM discovery (BLE v2).
//!
//! On the platforms that matter the L2CAP listener PSM is **OS-assigned**, not
//! chosen by the app (Android `listenUsingInsecureL2capChannel`, macOS
//! `CBPeripheralManager.publishL2CAPChannel`); only BlueZ can bind a fixed one.
//! So rather than relying on the fixed `DEFAULT_PSM` (0x0085), every node
//! **advertises its own listener PSM** as a 16-bit little-endian value in the
//! service-data field keyed on the FIPS service UUID, and every dialer **reads a
//! peer's advertised PSM before `connect()`**. The fixed-PSM assumption was a
//! BlueZ quirk; this makes discovery symmetric across all backends.
//!
//! This module holds the platform-agnostic pieces shared by every `BleIo`
//! backend (`BluerIo`, `BluestIo`, `AndroidIo`): the service-data **codec** and
//! the short-lived **`BleAddr → PSM` map**. The per-backend advertise/scan/dial
//! wiring lives in the backends. See
//! [`docs/reference/ble-wire.md`](../../../../docs/reference/ble-wire.md) and
//! [`docs/design/ble-interop.md`](../../../../docs/design/ble-interop.md).
use std::collections::HashMap;
use std::sync::Mutex;
use std::sync::atomic::{AtomicU16, Ordering};
use super::addr::BleAddr;
/// Encode a listener PSM as the 2-byte little-endian service-data payload
/// advertised under the FIPS service UUID.
///
/// The legacy advertising PDU caps at ~31 bytes, so a 128-bit UUID + this
/// 2-byte value is the tight, legacy-safe layout (see ble-wire.md).
pub fn encode_psm(psm: u16) -> [u8; 2] {
psm.to_le_bytes()
}
/// Decode a peer's advertised PSM from its FIPS service-data payload.
///
/// Returns `None` when fewer than 2 bytes are present — e.g. a legacy,
/// UUID-only advert that carries no PSM, for which the dialer falls back to
/// [`DEFAULT_PSM`](super::DEFAULT_PSM). Any bytes beyond the first two are
/// ignored, so the encoding can grow without breaking older readers.
pub fn decode_psm(data: &[u8]) -> Option<u16> {
match data {
[lo, hi, ..] => Some(u16::from_le_bytes([*lo, *hi])),
_ => None,
}
}
/// Short-lived map of discovered peer addresses to their advertised listener
/// PSM, populated by the scan loop and consulted by `connect()`.
///
/// Keyed on [`BleAddr`], which **rotates** with MAC randomization, so entries
/// are transient: they are re-learned each scan cycle rather than cached
/// durably. A stale entry merely causes one dial failure and a re-probe on the
/// next scan tick (see "Why MAC randomization is harmless" in ble-interop.md).
#[derive(Debug, Default)]
pub struct PsmMap {
inner: Mutex<HashMap<BleAddr, u16>>,
/// The most-recently-learned PSM across all peers. A node advertises exactly
/// one OS-assigned listener PSM, so when an exact-address lookup misses — the
/// common case, since the peer's RPA rotates between the scan that learned the
/// PSM and the dial — this is a far better guess than the fixed legacy default,
/// which no one listens on. 0 = unset.
last: AtomicU16,
}
impl PsmMap {
/// Create an empty map.
pub fn new() -> Self {
Self::default()
}
/// Record a peer's advertised PSM, learned from a scan advert. A later
/// advert for the same address overwrites the earlier value.
pub fn learn(&self, addr: &BleAddr, psm: u16) {
self.lock().insert(addr.clone(), psm);
if psm != 0 {
self.last.store(psm, Ordering::Relaxed);
}
}
/// Look up a peer's learned PSM, if one has been seen this scan cycle.
pub fn lookup(&self, addr: &BleAddr) -> Option<u16> {
self.lock().get(addr).copied()
}
/// Resolve the PSM to dial for `addr`: the learned per-peer PSM if known,
/// otherwise `fallback` (the configured PSM, else the legacy
/// [`DEFAULT_PSM`](super::DEFAULT_PSM)).
pub fn resolve(&self, addr: &BleAddr, fallback: u16) -> u16 {
if let Some(psm) = self.lookup(addr) {
return psm;
}
// Exact-address miss — almost always because the peer's MAC rotated
// between the scan that learned its PSM and this dial. Use the most
// recently learned PSM rather than the legacy default (never a real
// listener); a wrong guess only costs a dial-retry and a re-probe.
match self.last.load(Ordering::Relaxed) {
0 => fallback,
last => last,
}
}
/// Forget a single learned entry (e.g. after a dial failure).
pub fn forget(&self, addr: &BleAddr) {
self.lock().remove(addr);
}
/// Drop all learned entries. Called at the start of a scan cycle, since
/// addresses rotate and a dropped PSM only costs a dial-retry.
pub fn clear(&self) {
self.lock().clear();
}
fn lock(&self) -> std::sync::MutexGuard<'_, HashMap<BleAddr, u16>> {
self.inner.lock().unwrap_or_else(|e| e.into_inner())
}
}
#[cfg(test)]
mod tests {
use super::*;
use crate::transport::ble::DEFAULT_PSM;
fn addr(n: u8) -> BleAddr {
BleAddr {
adapter: "ble0".to_string(),
device: [0xAA, 0xBB, 0xCC, 0xDD, 0xEE, n],
}
}
#[test]
fn codec_roundtrips_little_endian() {
// 0x0085 -> [0x85, 0x00]; explicit LE byte order matches the wire spec.
assert_eq!(encode_psm(0x0085), [0x85, 0x00]);
assert_eq!(encode_psm(0x1234), [0x34, 0x12]);
for psm in [0u16, 1, 0x0085, 0x0080, 0x00FF, 0x1234, u16::MAX] {
assert_eq!(decode_psm(&encode_psm(psm)), Some(psm));
}
}
#[test]
fn decode_rejects_short_payload_but_ignores_trailing() {
assert_eq!(decode_psm(&[]), None); // legacy UUID-only advert
assert_eq!(decode_psm(&[0x85]), None); // truncated
// Forward-compatible: trailing bytes beyond the PSM are ignored.
assert_eq!(decode_psm(&[0x85, 0x00, 0xFF, 0xFF]), Some(0x0085));
}
#[test]
fn learn_lookup_and_overwrite() {
let map = PsmMap::new();
assert_eq!(map.lookup(&addr(1)), None);
map.learn(&addr(1), 0x0091);
assert_eq!(map.lookup(&addr(1)), Some(0x0091));
// A later advert for the same address overwrites.
map.learn(&addr(1), 0x00A0);
assert_eq!(map.lookup(&addr(1)), Some(0x00A0));
// Distinct addresses are independent.
map.learn(&addr(2), 0x00B0);
assert_eq!(map.lookup(&addr(1)), Some(0x00A0));
assert_eq!(map.lookup(&addr(2)), Some(0x00B0));
}
#[test]
fn resolve_prefers_learned_then_falls_back() {
let map = PsmMap::new();
// No learned PSM yet -> legacy default.
assert_eq!(map.resolve(&addr(1), DEFAULT_PSM), DEFAULT_PSM);
map.learn(&addr(1), 0x0091);
assert_eq!(map.resolve(&addr(1), DEFAULT_PSM), 0x0091);
// An unseen address (e.g. the peer's MAC rotated since we learned its PSM)
// dials the most-recently-learned PSM, not the legacy default.
assert_eq!(map.resolve(&addr(9), DEFAULT_PSM), 0x0091);
}
#[test]
fn forget_and_clear_drop_entries() {
let map = PsmMap::new();
map.learn(&addr(1), 0x0091);
map.learn(&addr(2), 0x0092);
map.forget(&addr(1));
assert_eq!(map.lookup(&addr(1)), None);
assert_eq!(map.lookup(&addr(2)), Some(0x0092));
map.clear();
assert_eq!(map.lookup(&addr(2)), None);
}
}
+175
View File
@@ -0,0 +1,175 @@
//! `AsyncRead` adapter over a datagram-shaped [`BleStream`].
//!
//! L2CAP delivers different boundary guarantees per platform:
//!
//! - **BlueZ** (`SOCK_SEQPACKET`) preserves SDU boundaries — one `recv` is
//! exactly one FIPS packet.
//! - **Android** (`BluetoothSocket` input stream) and **CoreBluetooth** are
//! byte-stream oriented — a `recv` may return a fragment of a packet, or
//! several packets coalesced.
//!
//! FIPS packets are self-delimiting via the 4-byte FMP common prefix, so the
//! shared framer [`crate::transport::tcp::stream::read_fmp_packet`] can recover
//! boundaries from any byte stream. This adapter turns a [`BleStream`] (whose
//! `recv` fills a `&mut [u8]`) into the [`AsyncRead`] that framer expects. On a
//! SeqPacket backend it's a no-op pass-through; on a stream backend it
//! reassembles. Either way the layer above sees one whole packet per read.
use std::future::Future;
use std::pin::Pin;
use std::sync::Arc;
use std::task::{Context, Poll};
use tokio::io::{AsyncRead, ReadBuf};
use super::io::BleStream;
/// An in-flight `recv`: owns its scratch buffer and yields an owned `Vec`, so
/// the future is `'static` and can live across `poll_read` calls.
type PendingRecv = Pin<Box<dyn Future<Output = std::io::Result<Vec<u8>>> + Send>>;
/// Buffers a [`BleStream`] into an [`AsyncRead`] byte stream.
pub struct BleStreamRead<S: BleStream + 'static> {
stream: Arc<S>,
/// Per-`recv` scratch size; also the framer's MTU bound.
mtu: u16,
/// Bytes from the last `recv` not yet consumed by the framer.
leftover: Vec<u8>,
/// Read cursor into `leftover`.
pos: usize,
/// In-flight `recv`, if one is underway.
pending: Option<PendingRecv>,
}
impl<S: BleStream + 'static> BleStreamRead<S> {
/// Wrap a stream. `mtu` is the per-`recv` scratch size (use the channel's
/// recv MTU).
pub fn new(stream: Arc<S>, mtu: u16) -> Self {
Self {
stream,
mtu: mtu.max(1),
leftover: Vec::new(),
pos: 0,
pending: None,
}
}
/// The wrapped stream, for sending on the same channel (e.g. the pubkey
/// exchange writes here while reads come through the buffer).
pub fn stream(&self) -> &S {
&self.stream
}
}
impl<S: BleStream + 'static> AsyncRead for BleStreamRead<S> {
fn poll_read(
self: Pin<&mut Self>,
cx: &mut Context<'_>,
dst: &mut ReadBuf<'_>,
) -> Poll<std::io::Result<()>> {
// BleStreamRead is Unpin (all fields are), so get_mut is sound.
let this = self.get_mut();
loop {
// Serve buffered bytes first.
if this.pos < this.leftover.len() {
let avail = &this.leftover[this.pos..];
let n = avail.len().min(dst.remaining());
dst.put_slice(&avail[..n]);
this.pos += n;
return Poll::Ready(Ok(()));
}
// Buffer drained: pull the next datagram. The future owns its
// scratch and returns it truncated, so it captures only `Arc<S>`.
if this.pending.is_none() {
let stream = Arc::clone(&this.stream);
let mtu = this.mtu as usize;
this.pending = Some(Box::pin(async move {
let mut scratch = vec![0u8; mtu];
let n = stream
.recv(&mut scratch)
.await
.map_err(|e| std::io::Error::other(e.to_string()))?;
scratch.truncate(n);
Ok(scratch)
}));
}
match this.pending.as_mut().unwrap().as_mut().poll(cx) {
Poll::Ready(Ok(buf)) => {
this.pending = None;
// A zero-length recv is the BleStream peer-closed signal;
// leaving `dst` unfilled surfaces as EOF to the framer.
if buf.is_empty() {
return Poll::Ready(Ok(()));
}
this.leftover = buf;
this.pos = 0;
// loop to copy out
}
Poll::Ready(Err(e)) => {
this.pending = None;
return Poll::Ready(Err(e));
}
Poll::Pending => return Poll::Pending,
}
}
}
}
#[cfg(test)]
mod tests {
use super::*;
use crate::transport::ble::addr::BleAddr;
use crate::transport::ble::io::MockBleStream;
use tokio::io::AsyncReadExt;
fn addr(n: u8) -> BleAddr {
BleAddr {
adapter: "hci0".to_string(),
device: [0xAA, 0xBB, 0xCC, 0xDD, 0xEE, n],
}
}
/// Several small recvs reassemble into one read_exact.
#[tokio::test]
async fn reassembles_fragmented_recv() {
let (a, b) = MockBleStream::pair(addr(1), addr(2), 2048);
// Peer sends three fragments that together form one 10-byte message.
a.send(b"abc").await.unwrap();
a.send(b"defg").await.unwrap();
a.send(b"hij").await.unwrap();
let mut reader = BleStreamRead::new(Arc::new(b), 2048);
let mut out = [0u8; 10];
reader.read_exact(&mut out).await.unwrap();
assert_eq!(&out, b"abcdefghij");
}
/// A recv carrying several packets is served across multiple reads without
/// dropping the tail (the coalescing case).
#[tokio::test]
async fn serves_coalesced_recv_in_pieces() {
let (a, b) = MockBleStream::pair(addr(1), addr(2), 2048);
a.send(b"0123456789").await.unwrap();
let mut reader = BleStreamRead::new(Arc::new(b), 2048);
let mut first = [0u8; 4];
reader.read_exact(&mut first).await.unwrap();
assert_eq!(&first, b"0123");
let mut rest = [0u8; 6];
reader.read_exact(&mut rest).await.unwrap();
assert_eq!(&rest, b"456789");
}
/// A closed stream surfaces as EOF (read_exact errors with UnexpectedEof).
#[tokio::test]
async fn closed_stream_is_eof() {
let (a, b) = MockBleStream::pair(addr(1), addr(2), 2048);
drop(a); // peer closes
let mut reader = BleStreamRead::new(Arc::new(b), 2048);
let mut out = [0u8; 4];
let err = reader.read_exact(&mut out).await.unwrap_err();
assert_eq!(err.kind(), std::io::ErrorKind::UnexpectedEof);
}
}
+44 -44
View File
@@ -11,15 +11,15 @@ pub mod tcp;
pub mod tor;
pub mod udp;
#[cfg(unix)]
#[cfg(any(target_os = "linux", target_os = "macos"))]
pub mod ethernet;
#[cfg(target_os = "linux")]
#[cfg(ble_available)]
pub mod ble;
#[cfg(target_os = "linux")]
#[cfg(ble_available)]
use ble::DefaultBleTransport;
#[cfg(unix)]
#[cfg(any(target_os = "linux", target_os = "macos"))]
use ethernet::EthernetTransport;
#[cfg(test)]
use loopback::LoopbackTransport;
@@ -894,7 +894,7 @@ pub enum TransportHandle {
/// UDP/IP transport.
Udp(UdpTransport),
/// Raw Ethernet transport.
#[cfg(unix)]
#[cfg(any(target_os = "linux", target_os = "macos"))]
Ethernet(EthernetTransport),
/// TCP/IP transport.
Tcp(TcpTransport),
@@ -903,7 +903,7 @@ pub enum TransportHandle {
/// Nym mixnet transport (via SOCKS5).
Nym(NymTransport),
/// BLE L2CAP transport.
#[cfg(target_os = "linux")]
#[cfg(ble_available)]
Ble(DefaultBleTransport),
/// In-process loopback transport (test harness only).
#[cfg(test)]
@@ -915,12 +915,12 @@ impl TransportHandle {
pub async fn start(&mut self) -> Result<(), TransportError> {
match self {
TransportHandle::Udp(t) => t.start_async().await,
#[cfg(unix)]
#[cfg(any(target_os = "linux", target_os = "macos"))]
TransportHandle::Ethernet(t) => t.start_async().await,
TransportHandle::Tcp(t) => t.start_async().await,
TransportHandle::Tor(t) => t.start_async().await,
TransportHandle::Nym(t) => t.start_async().await,
#[cfg(target_os = "linux")]
#[cfg(ble_available)]
TransportHandle::Ble(t) => t.start_async().await,
#[cfg(test)]
TransportHandle::Loopback(t) => t.start_async().await,
@@ -931,12 +931,12 @@ impl TransportHandle {
pub async fn stop(&mut self) -> Result<(), TransportError> {
match self {
TransportHandle::Udp(t) => t.stop_async().await,
#[cfg(unix)]
#[cfg(any(target_os = "linux", target_os = "macos"))]
TransportHandle::Ethernet(t) => t.stop_async().await,
TransportHandle::Tcp(t) => t.stop_async().await,
TransportHandle::Tor(t) => t.stop_async().await,
TransportHandle::Nym(t) => t.stop_async().await,
#[cfg(target_os = "linux")]
#[cfg(ble_available)]
TransportHandle::Ble(t) => t.stop_async().await,
#[cfg(test)]
TransportHandle::Loopback(t) => t.stop_async().await,
@@ -947,12 +947,12 @@ impl TransportHandle {
pub async fn send(&self, addr: &TransportAddr, data: &[u8]) -> Result<usize, TransportError> {
match self {
TransportHandle::Udp(t) => t.send_async(addr, data).await,
#[cfg(unix)]
#[cfg(any(target_os = "linux", target_os = "macos"))]
TransportHandle::Ethernet(t) => t.send_async(addr, data).await,
TransportHandle::Tcp(t) => t.send_async(addr, data).await,
TransportHandle::Tor(t) => t.send_async(addr, data).await,
TransportHandle::Nym(t) => t.send_async(addr, data).await,
#[cfg(target_os = "linux")]
#[cfg(ble_available)]
TransportHandle::Ble(t) => t.send_async(addr, data).await,
#[cfg(test)]
TransportHandle::Loopback(t) => t.send_async(addr, data).await,
@@ -963,12 +963,12 @@ impl TransportHandle {
pub fn transport_id(&self) -> TransportId {
match self {
TransportHandle::Udp(t) => t.transport_id(),
#[cfg(unix)]
#[cfg(any(target_os = "linux", target_os = "macos"))]
TransportHandle::Ethernet(t) => t.transport_id(),
TransportHandle::Tcp(t) => t.transport_id(),
TransportHandle::Tor(t) => t.transport_id(),
TransportHandle::Nym(t) => t.transport_id(),
#[cfg(target_os = "linux")]
#[cfg(ble_available)]
TransportHandle::Ble(t) => t.transport_id(),
#[cfg(test)]
TransportHandle::Loopback(t) => t.transport_id(),
@@ -979,12 +979,12 @@ impl TransportHandle {
pub fn name(&self) -> Option<&str> {
match self {
TransportHandle::Udp(t) => t.name(),
#[cfg(unix)]
#[cfg(any(target_os = "linux", target_os = "macos"))]
TransportHandle::Ethernet(t) => t.name(),
TransportHandle::Tcp(t) => t.name(),
TransportHandle::Tor(t) => t.name(),
TransportHandle::Nym(t) => t.name(),
#[cfg(target_os = "linux")]
#[cfg(ble_available)]
TransportHandle::Ble(t) => t.name(),
#[cfg(test)]
TransportHandle::Loopback(_) => None,
@@ -995,12 +995,12 @@ impl TransportHandle {
pub fn transport_type(&self) -> &TransportType {
match self {
TransportHandle::Udp(t) => t.transport_type(),
#[cfg(unix)]
#[cfg(any(target_os = "linux", target_os = "macos"))]
TransportHandle::Ethernet(t) => t.transport_type(),
TransportHandle::Tcp(t) => t.transport_type(),
TransportHandle::Tor(t) => t.transport_type(),
TransportHandle::Nym(t) => t.transport_type(),
#[cfg(target_os = "linux")]
#[cfg(ble_available)]
TransportHandle::Ble(t) => t.transport_type(),
#[cfg(test)]
TransportHandle::Loopback(t) => t.transport_type(),
@@ -1011,12 +1011,12 @@ impl TransportHandle {
pub fn state(&self) -> TransportState {
match self {
TransportHandle::Udp(t) => t.state(),
#[cfg(unix)]
#[cfg(any(target_os = "linux", target_os = "macos"))]
TransportHandle::Ethernet(t) => t.state(),
TransportHandle::Tcp(t) => t.state(),
TransportHandle::Tor(t) => t.state(),
TransportHandle::Nym(t) => t.state(),
#[cfg(target_os = "linux")]
#[cfg(ble_available)]
TransportHandle::Ble(t) => t.state(),
#[cfg(test)]
TransportHandle::Loopback(t) => t.state(),
@@ -1027,12 +1027,12 @@ impl TransportHandle {
pub fn mtu(&self) -> u16 {
match self {
TransportHandle::Udp(t) => t.mtu(),
#[cfg(unix)]
#[cfg(any(target_os = "linux", target_os = "macos"))]
TransportHandle::Ethernet(t) => t.mtu(),
TransportHandle::Tcp(t) => t.mtu(),
TransportHandle::Tor(t) => t.mtu(),
TransportHandle::Nym(t) => t.mtu(),
#[cfg(target_os = "linux")]
#[cfg(ble_available)]
TransportHandle::Ble(t) => t.mtu(),
#[cfg(test)]
TransportHandle::Loopback(t) => t.mtu(),
@@ -1046,12 +1046,12 @@ impl TransportHandle {
pub fn link_mtu(&self, addr: &TransportAddr) -> u16 {
match self {
TransportHandle::Udp(t) => t.link_mtu(addr),
#[cfg(unix)]
#[cfg(any(target_os = "linux", target_os = "macos"))]
TransportHandle::Ethernet(t) => t.link_mtu(addr),
TransportHandle::Tcp(t) => t.link_mtu(addr),
TransportHandle::Tor(t) => t.link_mtu(addr),
TransportHandle::Nym(t) => t.link_mtu(addr),
#[cfg(target_os = "linux")]
#[cfg(ble_available)]
TransportHandle::Ble(t) => t.link_mtu(addr),
#[cfg(test)]
TransportHandle::Loopback(t) => t.link_mtu(addr),
@@ -1062,12 +1062,12 @@ impl TransportHandle {
pub fn local_addr(&self) -> Option<std::net::SocketAddr> {
match self {
TransportHandle::Udp(t) => t.local_addr(),
#[cfg(unix)]
#[cfg(any(target_os = "linux", target_os = "macos"))]
TransportHandle::Ethernet(_) => None,
TransportHandle::Tcp(t) => t.local_addr(),
TransportHandle::Tor(_) => None,
TransportHandle::Nym(_) => None,
#[cfg(target_os = "linux")]
#[cfg(ble_available)]
TransportHandle::Ble(_) => None,
#[cfg(test)]
TransportHandle::Loopback(_) => None,
@@ -1078,12 +1078,12 @@ impl TransportHandle {
pub fn interface_name(&self) -> Option<&str> {
match self {
TransportHandle::Udp(_) => None,
#[cfg(unix)]
#[cfg(any(target_os = "linux", target_os = "macos"))]
TransportHandle::Ethernet(t) => Some(t.interface_name()),
TransportHandle::Tcp(_) => None,
TransportHandle::Tor(_) => None,
TransportHandle::Nym(_) => None,
#[cfg(target_os = "linux")]
#[cfg(ble_available)]
TransportHandle::Ble(_) => None,
#[cfg(test)]
TransportHandle::Loopback(_) => None,
@@ -1118,12 +1118,12 @@ impl TransportHandle {
pub fn discover(&self) -> Result<Vec<DiscoveredPeer>, TransportError> {
match self {
TransportHandle::Udp(t) => t.discover(),
#[cfg(unix)]
#[cfg(any(target_os = "linux", target_os = "macos"))]
TransportHandle::Ethernet(t) => t.discover(),
TransportHandle::Tcp(t) => t.discover(),
TransportHandle::Tor(t) => t.discover(),
TransportHandle::Nym(t) => t.discover(),
#[cfg(target_os = "linux")]
#[cfg(ble_available)]
TransportHandle::Ble(t) => t.discover(),
#[cfg(test)]
TransportHandle::Loopback(t) => t.discover(),
@@ -1134,12 +1134,12 @@ impl TransportHandle {
pub fn auto_connect(&self) -> bool {
match self {
TransportHandle::Udp(t) => t.auto_connect(),
#[cfg(unix)]
#[cfg(any(target_os = "linux", target_os = "macos"))]
TransportHandle::Ethernet(t) => t.auto_connect(),
TransportHandle::Tcp(t) => t.auto_connect(),
TransportHandle::Tor(t) => t.auto_connect(),
TransportHandle::Nym(t) => t.auto_connect(),
#[cfg(target_os = "linux")]
#[cfg(ble_available)]
TransportHandle::Ble(t) => t.auto_connect(),
#[cfg(test)]
TransportHandle::Loopback(t) => t.auto_connect(),
@@ -1150,12 +1150,12 @@ impl TransportHandle {
pub fn accept_connections(&self) -> bool {
match self {
TransportHandle::Udp(t) => t.accept_connections(),
#[cfg(unix)]
#[cfg(any(target_os = "linux", target_os = "macos"))]
TransportHandle::Ethernet(t) => t.accept_connections(),
TransportHandle::Tcp(t) => t.accept_connections(),
TransportHandle::Tor(t) => t.accept_connections(),
TransportHandle::Nym(t) => t.accept_connections(),
#[cfg(target_os = "linux")]
#[cfg(ble_available)]
TransportHandle::Ble(t) => t.accept_connections(),
#[cfg(test)]
TransportHandle::Loopback(t) => t.accept_connections(),
@@ -1172,12 +1172,12 @@ impl TransportHandle {
pub async fn connect(&self, addr: &TransportAddr) -> Result<(), TransportError> {
match self {
TransportHandle::Udp(_) => Ok(()), // connectionless
#[cfg(unix)]
#[cfg(any(target_os = "linux", target_os = "macos"))]
TransportHandle::Ethernet(_) => Ok(()), // connectionless
TransportHandle::Tcp(t) => t.connect_async(addr).await,
TransportHandle::Tor(t) => t.connect_async(addr).await,
TransportHandle::Nym(t) => t.connect_async(addr).await,
#[cfg(target_os = "linux")]
#[cfg(ble_available)]
TransportHandle::Ble(t) => t.connect_async(addr).await,
#[cfg(test)]
TransportHandle::Loopback(_) => Ok(()), // connectionless
@@ -1192,12 +1192,12 @@ impl TransportHandle {
pub fn connection_state(&self, addr: &TransportAddr) -> ConnectionState {
match self {
TransportHandle::Udp(_) => ConnectionState::Connected,
#[cfg(unix)]
#[cfg(any(target_os = "linux", target_os = "macos"))]
TransportHandle::Ethernet(_) => ConnectionState::Connected,
TransportHandle::Tcp(t) => t.connection_state_sync(addr),
TransportHandle::Tor(t) => t.connection_state_sync(addr),
TransportHandle::Nym(t) => t.connection_state_sync(addr),
#[cfg(target_os = "linux")]
#[cfg(ble_available)]
TransportHandle::Ble(t) => t.connection_state_sync(addr),
#[cfg(test)]
TransportHandle::Loopback(_) => ConnectionState::Connected,
@@ -1211,12 +1211,12 @@ impl TransportHandle {
pub async fn close_connection(&self, addr: &TransportAddr) {
match self {
TransportHandle::Udp(t) => t.close_connection(addr),
#[cfg(unix)]
#[cfg(any(target_os = "linux", target_os = "macos"))]
TransportHandle::Ethernet(t) => t.close_connection(addr),
TransportHandle::Tcp(t) => t.close_connection_async(addr).await,
TransportHandle::Tor(t) => t.close_connection_async(addr).await,
TransportHandle::Nym(t) => t.close_connection_async(addr).await,
#[cfg(target_os = "linux")]
#[cfg(ble_available)]
TransportHandle::Ble(t) => t.close_connection_async(addr).await,
#[cfg(test)]
TransportHandle::Loopback(_) => {} // connectionless no-op
@@ -1236,12 +1236,12 @@ impl TransportHandle {
pub fn congestion(&self) -> TransportCongestion {
match self {
TransportHandle::Udp(t) => t.congestion(),
#[cfg(unix)]
#[cfg(any(target_os = "linux", target_os = "macos"))]
TransportHandle::Ethernet(_) => TransportCongestion::default(),
TransportHandle::Tcp(_) => TransportCongestion::default(),
TransportHandle::Tor(_) => TransportCongestion::default(),
TransportHandle::Nym(_) => TransportCongestion::default(),
#[cfg(target_os = "linux")]
#[cfg(ble_available)]
TransportHandle::Ble(_) => TransportCongestion::default(),
#[cfg(test)]
TransportHandle::Loopback(_) => TransportCongestion::default(),
@@ -1256,7 +1256,7 @@ impl TransportHandle {
TransportHandle::Udp(t) => {
serde_json::to_value(t.stats().snapshot()).unwrap_or_default()
}
#[cfg(unix)]
#[cfg(any(target_os = "linux", target_os = "macos"))]
TransportHandle::Ethernet(t) => {
let snap = t.stats().snapshot();
serde_json::json!({
@@ -1281,7 +1281,7 @@ impl TransportHandle {
TransportHandle::Nym(t) => {
serde_json::to_value(t.stats().snapshot()).unwrap_or_default()
}
#[cfg(target_os = "linux")]
#[cfg(ble_available)]
TransportHandle::Ble(t) => {
serde_json::to_value(t.stats().snapshot()).unwrap_or_default()
}
+9
View File
@@ -92,6 +92,15 @@ impl TreeState {
&self.my_coords
}
/// Test-only override of this node's coordinates, bypassing the
/// parent/declaration state machine. Lets routing tests place the node at
/// an arbitrary tree position to exercise coordinate-based classification.
#[cfg(test)]
pub(crate) fn set_my_coords_for_test(&mut self, coords: TreeCoordinate) {
self.root = *coords.root_id();
self.my_coords = coords;
}
/// Get the current root.
pub fn root(&self) -> &NodeAddr {
&self.root
+1 -1
View File
@@ -301,7 +301,7 @@ fn extract_pktinfo_ifindex(msg: &libc::msghdr) -> Option<u32> {
if cmsg.cmsg_level == libc::IPPROTO_IPV6 && cmsg.cmsg_type == libc::IPV6_PKTINFO {
let data_ptr = unsafe { libc::CMSG_DATA(cmsg_ptr) } as *const libc::in6_pktinfo;
let pktinfo: libc::in6_pktinfo = unsafe { std::ptr::read_unaligned(data_ptr) };
return Some(pktinfo.ipi6_ifindex);
return Some(pktinfo.ipi6_ifindex as u32);
}
cmsg_ptr = unsafe { libc::CMSG_NXTHDR(msg, cmsg_ptr) };
}
+159 -38
View File
@@ -115,8 +115,8 @@ pub fn clamp_tcp_mss(ipv6_packet: &mut [u8], max_mss: u16) -> bool {
if current_mss > max_mss {
ipv6_packet[i + 2..i + 4].copy_from_slice(&max_mss.to_be_bytes());
// Recalculate TCP checksum
recalculate_tcp_checksum(ipv6_packet, tcp_start);
// Recompute the now-stale checksum over the rewritten header.
recalculate_l4_checksum(ipv6_packet);
modified = true;
}
@@ -129,41 +129,39 @@ pub fn clamp_tcp_mss(ipv6_packet: &mut [u8], max_mss: u16) -> bool {
modified
}
/// Recalculate TCP checksum after modifying the packet.
fn recalculate_tcp_checksum(ipv6_packet: &mut [u8], tcp_start: usize) {
// Zero out existing checksum
ipv6_packet[tcp_start + 16] = 0;
ipv6_packet[tcp_start + 17] = 0;
// Extract addresses
let src = &ipv6_packet[8..24];
let dst = &ipv6_packet[24..40];
// Get TCP segment length
/// Recalculate the TCP or UDP checksum (IPv6 pseudo-header + segment) in place.
///
/// Completes the checksum macOS leaves offloaded on hairpinned self-traffic, and
/// is reused by MSS clamping. No-op for other next-headers (e.g. ICMPv6) and for
/// malformed packets. Assumes the L4 header follows the 40-byte IPv6 header, as
/// all FIPS packets do.
pub fn recalculate_l4_checksum(ipv6_packet: &mut [u8]) {
if ipv6_packet.len() < 40 || ipv6_packet[0] >> 4 != 6 {
return;
}
let payload_len = u16::from_be_bytes([ipv6_packet[4], ipv6_packet[5]]) as usize;
let tcp_segment = &ipv6_packet[tcp_start..tcp_start + payload_len];
if payload_len == 0 || 40 + payload_len > ipv6_packet.len() {
return;
}
// Calculate checksum with pseudo-header
// Transport checksum field offset within the packet: TCP at 16, UDP at 6.
let proto = ipv6_packet[6];
let csum = match proto {
6 if payload_len >= TCP_HEADER_MIN_LEN => 40 + 16,
17 if payload_len >= 8 => 40 + 6,
_ => return,
};
ipv6_packet[csum] = 0;
ipv6_packet[csum + 1] = 0;
// Pseudo-header (src + dst are contiguous at 8..40), length, next header,
let mut sum: u32 = 0;
// Pseudo-header: source address
for chunk in src.chunks(2) {
for chunk in ipv6_packet[8..40].chunks(2) {
sum += u16::from_be_bytes([chunk[0], chunk[1]]) as u32;
}
// Pseudo-header: destination address
for chunk in dst.chunks(2) {
sum += u16::from_be_bytes([chunk[0], chunk[1]]) as u32;
}
// Pseudo-header: TCP length
sum += payload_len as u32;
// Pseudo-header: next header (TCP = 6)
sum += 6;
// TCP segment
for chunk in tcp_segment.chunks(2) {
sum += payload_len as u32 + proto as u32;
// then the transport segment itself (with the checksum field zeroed above).
for chunk in ipv6_packet[40..40 + payload_len].chunks(2) {
let value = if chunk.len() == 2 {
u16::from_be_bytes([chunk[0], chunk[1]])
} else {
@@ -171,15 +169,16 @@ fn recalculate_tcp_checksum(ipv6_packet: &mut [u8], tcp_start: usize) {
};
sum += value as u32;
}
// Fold 32-bit sum to 16 bits
while sum >> 16 != 0 {
sum = (sum & 0xffff) + (sum >> 16);
}
// One's complement
let checksum = !sum as u16;
ipv6_packet[tcp_start + 16..tcp_start + 18].copy_from_slice(&checksum.to_be_bytes());
// IPv6 forbids an all-zero UDP checksum; send 0xffff instead (TCP keeps 0).
let checksum = match (!sum as u16, proto) {
(0, 17) => 0xffff,
(c, _) => c,
};
ipv6_packet[csum..csum + 2].copy_from_slice(&checksum.to_be_bytes());
}
#[cfg(test)]
@@ -216,7 +215,7 @@ mod tests {
packet[tcp_start + 24] = 0;
// Calculate checksum
recalculate_tcp_checksum(&mut packet, tcp_start);
recalculate_l4_checksum(&mut packet);
packet
}
@@ -277,4 +276,126 @@ mod tests {
assert!(!modified);
}
// ========================================================================
// recalculate_l4_checksum — finish macOS's offloaded self-traffic
// checksums (the bug that left ACK/data/FIN segments undeliverable).
// ========================================================================
/// True if the packet's TCP/UDP checksum verifies (folded ones-complement
/// sum over pseudo-header + segment, including the checksum field, == 0xffff).
fn l4_checksum_valid(pkt: &[u8]) -> bool {
let payload_len = u16::from_be_bytes([pkt[4], pkt[5]]) as usize;
let mut sum: u32 = 0;
for chunk in pkt[8..40].chunks(2) {
sum += u16::from_be_bytes([chunk[0], chunk[1]]) as u32;
}
sum += payload_len as u32;
sum += pkt[6] as u32; // next header
for chunk in pkt[40..40 + payload_len].chunks(2) {
let v = if chunk.len() == 2 {
u16::from_be_bytes([chunk[0], chunk[1]])
} else {
u16::from_be_bytes([chunk[0], 0])
};
sum += v as u32;
}
while sum >> 16 != 0 {
sum = (sum & 0xffff) + (sum >> 16);
}
sum as u16 == 0xffff
}
const SELF_ADDR: [u8; 16] = [0xfd, 0x12, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0x55];
/// A self-addressed 20-byte TCP ACK (no options) carrying a deliberately
/// wrong checksum — the shape macOS hands us for hairpinned non-SYN traffic.
fn make_tcp_ack_packet() -> Vec<u8> {
let mut p = vec![0u8; 40 + 20];
p[0] = 0x60;
p[4..6].copy_from_slice(&20u16.to_be_bytes()); // payload length
p[6] = 6; // TCP
p[7] = 64;
p[8..24].copy_from_slice(&SELF_ADDR);
p[24..40].copy_from_slice(&SELF_ADDR);
let t = 40;
p[t..t + 2].copy_from_slice(&52097u16.to_be_bytes());
p[t + 2..t + 4].copy_from_slice(&9999u16.to_be_bytes());
p[t + 4..t + 8].copy_from_slice(&1000u32.to_be_bytes()); // seq
p[t + 8..t + 12].copy_from_slice(&2000u32.to_be_bytes()); // ack
p[t + 12] = 0x50; // data offset = 5 (20-byte header)
p[t + 13] = 0x10; // ACK
p[t + 14..t + 16].copy_from_slice(&2049u16.to_be_bytes()); // window
p[t + 16] = 0x8e; // bogus checksum (macOS pseudo-header partial)
p[t + 17] = 0xce;
p
}
#[test]
fn recompute_fixes_tcp_non_syn_checksum() {
let mut pkt = make_tcp_ack_packet();
assert!(
!l4_checksum_valid(&pkt),
"fixture should start with a bad checksum"
);
recalculate_l4_checksum(&mut pkt);
assert!(
l4_checksum_valid(&pkt),
"TCP checksum must verify after recompute"
);
}
#[test]
fn recompute_fixes_udp_checksum() {
let mut p = vec![0u8; 40 + 12]; // 8-byte UDP header + 4-byte payload
p[0] = 0x60;
p[4..6].copy_from_slice(&12u16.to_be_bytes());
p[6] = 17; // UDP
p[7] = 64;
p[8..24].copy_from_slice(&SELF_ADDR);
p[24..40].copy_from_slice(&SELF_ADDR);
let u = 40;
p[u..u + 2].copy_from_slice(&40000u16.to_be_bytes()); // src port
p[u + 2..u + 4].copy_from_slice(&5354u16.to_be_bytes()); // dst port
p[u + 4..u + 6].copy_from_slice(&12u16.to_be_bytes()); // UDP length
p[u + 6] = 0x8e; // bogus checksum
p[u + 7] = 0xce;
p[u + 8..u + 12].copy_from_slice(&[0xde, 0xad, 0xbe, 0xef]); // payload
assert!(!l4_checksum_valid(&p), "fixture should start invalid");
recalculate_l4_checksum(&mut p);
assert!(
l4_checksum_valid(&p),
"UDP checksum must verify after recompute"
);
assert_ne!(
&p[u + 6..u + 8],
&[0, 0],
"IPv6 UDP checksum must not be zero"
);
}
#[test]
fn recompute_is_noop_for_non_transport() {
// Next-header 59 (No Next Header): nothing to checksum.
let mut pkt = vec![0u8; 60];
pkt[0] = 0x60;
pkt[4..6].copy_from_slice(&20u16.to_be_bytes());
pkt[6] = 59;
let before = pkt.clone();
recalculate_l4_checksum(&mut pkt);
assert_eq!(pkt, before, "non-transport packet must be left untouched");
}
#[test]
fn recompute_ignores_truncated_packet() {
// payload_len claims 20 bytes of TCP but only 10 are present.
let mut pkt = vec![0u8; 40 + 10];
pkt[0] = 0x60;
pkt[4..6].copy_from_slice(&20u16.to_be_bytes());
pkt[6] = 6;
let before = pkt.clone();
recalculate_l4_checksum(&mut pkt); // must not panic
assert_eq!(pkt, before, "truncated packet must be left untouched");
}
}
+133 -1
View File
@@ -693,7 +693,7 @@ fn handle_tun_packet(
path_mtu_lookup: &PathMtuLookup,
) -> bool {
use super::icmp::{DestUnreachableCode, build_dest_unreachable, should_send_icmp_error};
use super::tcp_mss::clamp_tcp_mss;
use super::tcp_mss::{clamp_tcp_mss, recalculate_l4_checksum};
log_ipv6_packet(packet);
@@ -704,6 +704,30 @@ fn handle_tun_packet(
// Check if destination is a FIPS address (fd::/8 prefix)
if packet[24] == crate::identity::FIPS_ADDRESS_PREFIX {
// Loopback: a packet to our own mesh address must be delivered
// locally, not pushed into the mesh (we have no session/route to
// ourselves, so it would just be dropped). Hairpin it back to the TUN
// writer for inbound delivery.
//
// Platform note: in practice this branch is reached only on macOS.
// macOS point-to-point `utun` interfaces egress self-addressed traffic
// down the tunnel into this reader, so the daemon has to loop it back
// itself. On Linux the kernel routes traffic to our own bound
// addresses via `lo` before it ever reaches the TUN, so this branch
// never fires there. The check is kept unconditional anyway, both as a
// platform-independent self-delivery invariant and so the path stays
// exercised by the Linux-only CI unit tests.
if packet[24..40] == *our_addr.as_bytes() {
trace!(name = %name, "Hairpinning self-addressed packet back to TUN (loopback)");
// Finish the checksum macOS leaves offloaded on self-traffic, else the
// local stack drops every non-SYN segment. See recalculate_l4_checksum.
recalculate_l4_checksum(packet);
if tun_tx.send(packet.to_vec()).is_err() {
return false; // Channel closed, shutdown
}
return true;
}
// Per-destination clamp: if discovery has learned a smaller path
// MTU for this destination, tighten the ceiling for this flow.
let effective_max_mss = per_flow_max_mss(path_mtu_lookup, &packet[24..40], max_mss);
@@ -1200,6 +1224,32 @@ mod windows_tun {
#[cfg(windows)]
pub use windows_tun::{TunDevice, TunWriter, run_tun_reader, shutdown_tun_interface};
// Android uses an app-owned TUN (the embedder owns the fd, e.g. an Android
// VpnService); FIPS never creates or configures a system TUN here. These no-op
// stubs stand in for the platform ops so the shared TunDevice code compiles.
#[cfg(target_os = "android")]
mod platform {
use super::TunError;
use std::net::Ipv6Addr;
pub fn is_ipv6_disabled() -> bool {
false
}
pub async fn interface_exists(_name: &str) -> bool {
false
}
pub async fn delete_interface(_name: &str) -> Result<(), TunError> {
Ok(())
}
pub async fn configure_interface(
_name: &str,
_addr: Ipv6Addr,
_mtu: u16,
) -> Result<(), TunError> {
Ok(())
}
}
#[cfg(target_os = "linux")]
mod platform {
use super::TunError;
@@ -1520,6 +1570,88 @@ mod tests {
assert_eq!(per_flow_max_mss(&lookup, b.as_bytes(), 1360), 1315);
}
// ========================================================================
// handle_tun_packet — self-addressed loopback hairpin
//
// A packet destined for our own mesh address must be delivered back to
// the local stack via the TUN writer, never pushed into the mesh (there
// is no session/route to ourselves). On macOS the kernel egresses such
// self-traffic down the utun into the reader, so the daemon has to loop
// it back itself.
// ========================================================================
/// Build a minimal 40-byte IPv6 packet (no upper-layer payload) addressed
/// to `dst`, sourced from a distinct fips address.
fn ipv6_packet_to(dst: &FipsAddress) -> Vec<u8> {
let mut pkt = vec![0u8; 40];
pkt[0] = 0x60; // version 6
pkt[6] = 59; // next header = No Next Header (skips MSS clamp)
pkt[7] = 64; // hop limit
pkt[8] = crate::identity::FIPS_ADDRESS_PREFIX; // src in fd::/8
pkt[24..40].copy_from_slice(dst.as_bytes()); // dst
pkt
}
#[test]
fn self_addressed_packet_is_hairpinned_to_tun() {
let our_addr = fips_addr_with_node_byte(0x55);
let (tun_tx, tun_rx) = mpsc::channel::<Vec<u8>>();
let (outbound_tx, mut outbound_rx) = tokio::sync::mpsc::channel::<Vec<u8>>(4);
let lookup = empty_lookup();
let mut pkt = ipv6_packet_to(&our_addr);
assert!(handle_tun_packet(
&mut pkt,
1360,
"test0",
our_addr,
&tun_tx,
&outbound_tx,
&lookup,
));
// Delivered locally via the TUN writer...
let looped = tun_rx
.try_recv()
.expect("self-addressed packet should be hairpinned to the TUN");
assert_eq!(&looped[24..40], our_addr.as_bytes());
// ...and never handed to the mesh.
assert!(
outbound_rx.try_recv().is_err(),
"self-addressed packet must not be pushed into the mesh"
);
}
#[test]
fn other_fips_packet_goes_to_mesh() {
let our_addr = fips_addr_with_node_byte(0x55);
let peer = fips_addr_with_node_byte(0x66);
let (tun_tx, tun_rx) = mpsc::channel::<Vec<u8>>();
let (outbound_tx, mut outbound_rx) = tokio::sync::mpsc::channel::<Vec<u8>>(4);
let lookup = empty_lookup();
let mut pkt = ipv6_packet_to(&peer);
assert!(handle_tun_packet(
&mut pkt,
1360,
"test0",
our_addr,
&tun_tx,
&outbound_tx,
&lookup,
));
// A non-self fips destination is routed into the mesh, not looped back.
assert!(
outbound_rx.try_recv().is_ok(),
"non-self fips destination should be sent to the mesh"
);
assert!(
tun_rx.try_recv().is_err(),
"non-self fips destination must not be hairpinned"
);
}
// ========================================================================
// macOS utun packet-info header (AF_INET6 4-byte big-endian prefix)
//
+72
View File
@@ -72,3 +72,75 @@ and optional trace-level RUST_LOG, capturing per-rep diagnostics and a
mechanism-match summary across the run. Used for statistical reliability
characterization of known flake classes under calibrated stress, not as
a per-commit gate; not part of `ci-local.sh`.
## Running CI locally (`ci-local.sh`)
[`ci-local.sh`](ci-local.sh) runs the full local CI pipeline — build,
clippy, unit tests, and the integration suites (including the chaos
scenarios) — mirroring the GitHub `ci.yml` integration matrix. Run
`./ci-local.sh --help` for the full option list and `--list` for the
available suites. `--check-parity` verifies the local suite set matches
the GitHub matrix (see [check-ci-parity.sh](check-ci-parity.sh)).
### Per-run isolation and the `FIPS_CI_RUN_ID` override
Every invocation derives a **run id** and scopes all of its Docker
resources to it, so two simultaneous runs on the same host (for example,
one per git worktree, or an operator testing by hand while CI is in
flight) never collide:
- **Compose projects** are named `fipsci_<run-id>_<suite>`, so
container, network, and volume names are all prefixed per run.
- **Build images** are tagged `fips-test:<run-id>` and
`fips-test-app:<run-id>` (exported as `FIPS_TEST_IMAGE` /
`FIPS_TEST_APP_IMAGE` for the compose consumers).
- Each parallel chaos child gets a unique, non-overlapping `/24` in
`10.30.x` (via the sim `--subnet` override). `10.30.x` sits outside
Docker's default address pool and the fixed-subnet suites' `172.x`
ranges, so neither a sibling chaos child nor an auto-assigned network
can swallow a pinned subnet.
By default the run id is `<short-git-sha>-<random>` — the SHA portion
records *what code* a container is testing, the random suffix keeps
simultaneous runs of the same SHA disjoint. Override it for a
reproducible, attach-by-name debug session:
```sh
FIPS_CI_RUN_ID=mydebug ./ci-local.sh --only static-mesh
# containers are named fipsci_mydebug_static_fips-node-a, etc.
```
### Preemption-safety and exit codes
`ci-local.sh` is safe to cancel mid-run. A signal trap tears down *every*
compose project the run started (not just the current suite) and reaps
any in-flight parallel chaos children, bounded by a `timeout` so a stuck
`compose down` cannot wedge the trap. Exit codes distinguish a cancelled
run from a failing one:
| Code | Meaning |
| ---- | ------- |
| `0` | all stages passed |
| `1` | one or more stages failed |
| `130` | interrupted by SIGINT — cancelled, not a failure |
| `143` | terminated by SIGTERM — cancelled, not a failure |
A preempting CI worker (the push-triggered, CI-gated build pipeline that
kills an in-flight run when a newer same-branch tip arrives) maps
`130`/`143`*cancelled* (discard, do not record a failing commit), `0`
→ green, any other non-zero → red.
### Cleaning up leftover resources
Every CI-created container, network, and volume carries the label
`com.corganlabs.fips-ci=1`. If a run is hard-killed (SIGKILL, OOM, crash)
and leaves resources behind, reap them with:
```sh
./ci-local.sh --reap # or: ./ci-cleanup.sh
```
[`ci-cleanup.sh`](ci-cleanup.sh) force-removes everything bearing the CI
label or a `fipsci_` compose-project prefix; it is safe to run when there
is nothing to reap and safe to run repeatedly. Pass `--project-prefix` to
scope the sweep to a single run.
+2
View File
@@ -1,6 +1,8 @@
networks:
acl-net:
driver: bridge
labels:
- "com.corganlabs.fips-ci=1"
ipam:
config:
- subnet: 172.31.0.0/24
+2
View File
@@ -6,6 +6,8 @@
networks:
bt-net:
driver: bridge
labels:
- "com.corganlabs.fips-ci=1"
ipam:
config:
- subnet: 172.99.0.0/24
+4
View File
@@ -65,6 +65,7 @@ VERBOSE=""
SEED=""
DURATION=""
NODES=""
SUBNET=""
while [ $# -gt 0 ]; do
case "$1" in
@@ -72,6 +73,7 @@ while [ $# -gt 0 ]; do
--seed) SEED="$2"; shift 2 ;;
--duration) DURATION="$2"; shift 2 ;;
--nodes) NODES="$2"; shift 2 ;;
--subnet) SUBNET="$2"; shift 2 ;;
--list) list_scenarios ;;
-*) echo "Error: Unknown option '$1'" >&2; usage ;;
*)
@@ -135,6 +137,7 @@ PYTHON_ARGS=("$SCENARIO_FILE")
[ -n "$VERBOSE" ] && PYTHON_ARGS+=("$VERBOSE")
[ -n "$SEED" ] && PYTHON_ARGS+=("--seed" "$SEED")
[ -n "$DURATION" ] && PYTHON_ARGS+=("--duration" "$DURATION")
[ -n "$SUBNET" ] && PYTHON_ARGS+=("--subnet" "$SUBNET")
echo "=== FIPS Stochastic Simulation ==="
echo ""
@@ -143,6 +146,7 @@ echo " File: $SCENARIO_FILE"
[ -n "$SEED" ] && echo " Seed: $SEED (override)"
[ -n "$DURATION" ] && echo " Duration: ${DURATION}s (override)"
[ -n "$NODES" ] && echo " Nodes: $NODES (override)"
[ -n "$SUBNET" ] && echo " Subnet: $SUBNET (override)"
echo ""
# If --nodes is specified, create a patched copy of the scenario file
+8
View File
@@ -25,6 +25,12 @@ def main():
"--duration", type=int, default=None,
help="Override scenario duration in seconds",
)
parser.add_argument(
"--subnet", type=str, default=None,
help="Override topology subnet CIDR (e.g. 10.30.0.0/24); node IPs "
"derive from it. Used by CI to give each parallel run a "
"non-overlapping network.",
)
args = parser.parse_args()
level = logging.DEBUG if args.verbose else logging.INFO
@@ -48,6 +54,8 @@ def main():
print("Error: --duration must be >= 1", file=sys.stderr)
sys.exit(1)
scenario.duration_secs = args.duration
if args.subnet is not None:
scenario.topology.subnet = args.subnet
runner = SimRunner(scenario)
result = runner.run()
+2
View File
@@ -20,6 +20,8 @@ _COMPOSE_TEMPLATE = Template(
networks:
fips-net:
driver: bridge
labels:
- "com.corganlabs.fips-ci=1"
ipam:
config:
- subnet: {{ subnet }}
+106
View File
@@ -0,0 +1,106 @@
#!/bin/bash
# Reap FIPS CI docker resources: containers, networks, volumes, images.
#
# Force-removes everything created by ci-local.sh that is still around —
# whether a run finished cleanly, was preempted (SIGTERM/SIGKILL), OOM-killed,
# or crashed. Two complementary selectors make this robust no matter how a
# prior run died:
#
# 1. The CI label com.corganlabs.fips-ci=1 (attached to every direct
# `docker run`/network/volume ci-local drives).
# 2. The compose project-name prefix fipsci_ (every compose project ci-local
# starts is named fipsci_<run-id>_<suite>, so its containers/networks/
# volumes all carry com.docker.compose.project=fipsci_... and are named
# with that prefix).
#
# Usage:
# ci-cleanup.sh Reap ALL fips-ci resources (any run)
# ci-cleanup.sh --project-prefix P Restrict the compose-project sweep to
# names starting with P (scopes the reap
# to a single run; the label sweep still
# runs)
# ci-cleanup.sh --label L Override the CI label (default above)
# ci-cleanup.sh --images "a b,c" Also `docker rmi -f` these image tags
# (space- or comma-separated)
#
# Safe to run when there is nothing to reap, and safe to run repeatedly.
# Also reachable as `ci-local.sh --reap`.
set -uo pipefail
LABEL="com.corganlabs.fips-ci=1"
PROJECT_PREFIX="fipsci_" # broad default: every CI run
IMAGES=""
while [[ $# -gt 0 ]]; do
case "$1" in
--label) LABEL="$2"; shift 2 ;;
--project-prefix) PROJECT_PREFIX="$2"; shift 2 ;;
--images) IMAGES="$2"; shift 2 ;;
-h|--help) sed -n '2,/^set /{ /^set /d; s/^# \?//; p }' "$0"; exit 0 ;;
*) echo "Unknown option: $1" >&2; exit 2 ;;
esac
done
if ! command -v docker >/dev/null 2>&1; then
# No docker, nothing to reap.
exit 0
fi
if ! docker info >/dev/null 2>&1; then
# Daemon unreachable; treat as nothing to reap rather than wedging a caller.
exit 0
fi
# Each docker mutation is wrapped in `timeout` so a stuck daemon/resource can
# never wedge a caller (ci-local's signal trap relies on this being bounded).
TMO=30
# Distinct compose project names (read off container labels) that start with
# the configured prefix.
ci_projects() {
docker ps -a --format '{{.Label "com.docker.compose.project"}}' 2>/dev/null \
| grep -E "^${PROJECT_PREFIX}" | sort -u
}
reap_containers() {
# By CI label.
docker ps -aq --filter "label=${LABEL}" 2>/dev/null \
| xargs -r timeout "$TMO" docker rm -f >/dev/null 2>&1 || true
# By compose project (carried even when container_name is explicit).
local p
for p in $(ci_projects); do
docker ps -aq --filter "label=com.docker.compose.project=${p}" 2>/dev/null \
| xargs -r timeout "$TMO" docker rm -f >/dev/null 2>&1 || true
done
}
reap_networks() {
docker network ls -q --filter "label=${LABEL}" 2>/dev/null \
| xargs -r timeout "$TMO" docker network rm >/dev/null 2>&1 || true
# Compose networks are named <project>_<net> → match by name prefix so
# orphaned networks (whose containers are already gone) are still caught.
docker network ls --format '{{.Name}}' 2>/dev/null | grep -E "^${PROJECT_PREFIX}" \
| xargs -r timeout "$TMO" docker network rm >/dev/null 2>&1 || true
}
reap_volumes() {
docker volume ls -q --filter "label=${LABEL}" 2>/dev/null \
| xargs -r timeout "$TMO" docker volume rm >/dev/null 2>&1 || true
docker volume ls --format '{{.Name}}' 2>/dev/null | grep -E "^${PROJECT_PREFIX}" \
| xargs -r timeout "$TMO" docker volume rm >/dev/null 2>&1 || true
}
reap_images() {
[[ -z "$IMAGES" ]] && return 0
local imgs
read -ra imgs <<< "${IMAGES//,/ }"
[[ ${#imgs[@]} -eq 0 ]] && return 0
timeout "$TMO" docker rmi -f "${imgs[@]}" >/dev/null 2>&1 || true
}
# Order matters: containers reference networks/volumes, so drop them first.
reap_containers
reap_networks
reap_volumes
reap_images
exit 0
+154 -6
View File
@@ -14,6 +14,9 @@
# --list List available integration suites
# --check-parity Verify this suite set matches ci.yml's integration
# matrix (see testing/check-ci-parity.sh), then exit
# --reap Force-remove all leftover FIPS CI docker resources
# (containers/networks/volumes carrying the CI label or a
# fipsci_ compose project), then exit. See ci-cleanup.sh.
# -h, --help Show this help
#
# Integration suites (default coverage):
@@ -32,8 +35,13 @@
# tor-socks5, tor-directory
#
# Exit codes:
# 0 — all stages passed
# 1 — one or more stages failed
# 0 — all stages passed
# 1 — one or more stages failed
# 130 — interrupted by SIGINT (128 + 2; run was cancelled, not a failure)
# 143 — terminated by SIGTERM (128 + 15; run was cancelled, not a failure)
#
# A preempting CI worker maps 130/143 → "cancelled" (discard, do not record a
# failing commit), 0 → green, any other non-zero → red.
#
# ── CI parity invariant ─────────────────────────────────────────────────────
# This local default suite set and the GitHub integration matrix
@@ -193,6 +201,7 @@ while [[ $# -gt 0 ]]; do
-j|--jobs) PARALLEL_JOBS="$2"; shift 2 ;;
--list) list_suites ;;
--check-parity) exec "$SCRIPT_DIR/check-ci-parity.sh" ;;
--reap) exec "$SCRIPT_DIR/ci-cleanup.sh" ;;
-h|--help) usage ;;
*) echo "Unknown option: $1"; usage ;;
esac
@@ -214,6 +223,104 @@ record() {
fi
}
# ── Per-run isolation + signal-safe teardown ───────────────────────────────
#
# This script may be preempted (a CI worker sends SIGTERM, waits ~30s, then
# SIGKILL) so it can restart on a newer tip. To make that safe:
# * every docker resource is namespaced to THIS run (compose project prefix
# + per-run image tags) so a restart never collides with a dying run;
# * a trap tears down everything this run created on signal/exit, bounded by
# `timeout` so a stuck `down` cannot wedge the trap (SIGKILL is the backstop).
# Derive a run id: honor the worker's $FIPS_CI_RUN_ID, else <short-sha>-<rand>,
# else $$-<timestamp>. Sanitize to a valid compose project / image-tag token.
if [[ -n "${FIPS_CI_RUN_ID:-}" ]]; then
CI_RUN_ID="$FIPS_CI_RUN_ID"
else
_ci_sha="$(git -C "$PROJECT_ROOT" rev-parse --short HEAD 2>/dev/null || true)"
if [[ -n "$_ci_sha" ]]; then
CI_RUN_ID="${_ci_sha}-${RANDOM}${RANDOM}"
else
CI_RUN_ID="$$-$(date +%s)"
fi
fi
CI_RUN_ID="$(printf '%s' "$CI_RUN_ID" | tr '[:upper:]' '[:lower:]' | tr -c 'a-z0-9_-' '-')"
# A docker image tag and a compose project name must both START with an
# alphanumeric, so strip any leading -/_ left by sanitization.
CI_RUN_ID="$(printf '%s' "$CI_RUN_ID" | sed -E 's/^[^a-z0-9]+//')"
[[ -z "$CI_RUN_ID" ]] && CI_RUN_ID="r$$"
CI_PROJECT_PREFIX="fipsci_${CI_RUN_ID}"
CI_IMAGE_TEST="fips-test:${CI_RUN_ID}"
CI_IMAGE_APP="fips-test-app:${CI_RUN_ID}"
CI_LABEL="com.corganlabs.fips-ci=1"
# Exported so child suite scripts + their compose/`docker run` invocations
# inherit the run identity. Compose files read FIPS_TEST_IMAGE/FIPS_TEST_APP_IMAGE
# (default :latest when unset, preserving manual `docker compose` use).
export FIPS_CI_RUN_ID="$CI_RUN_ID"
export FIPS_TEST_IMAGE="$CI_IMAGE_TEST"
export FIPS_TEST_APP_IMAGE="$CI_IMAGE_APP"
# Per-suite compose project name: ${prefix}_<suite-or-compose-basename>. Keeps
# today's intra-run distinctness (one project per compose file / chaos child)
# while adding the cross-run prefix that scopes the reap.
ci_project() { printf '%s_%s' "$CI_PROJECT_PREFIX" "$1"; }
# PIDs of in-flight parallel chaos children (subshells). The trap signals these.
CI_CHAOS_PIDS=()
CI_CLEANED=0
# Best-effort, BOUNDED teardown of every docker resource THIS run may have
# created. Idempotent (guarded), so the signal and EXIT paths don't double-run.
ci_teardown() {
[[ $CI_CLEANED -eq 1 ]] && return 0
CI_CLEANED=1
# 1. Propagate to parallel chaos children and reap them (bounded).
if [[ ${#CI_CHAOS_PIDS[@]} -gt 0 ]]; then
kill -TERM "${CI_CHAOS_PIDS[@]}" 2>/dev/null || true
local _end=$(( SECONDS + 10 )) _p
for _p in "${CI_CHAOS_PIDS[@]}"; do
while kill -0 "$_p" 2>/dev/null && (( SECONDS < _end )); do
sleep 0.3
done
done
kill -KILL "${CI_CHAOS_PIDS[@]}" 2>/dev/null || true
wait "${CI_CHAOS_PIDS[@]}" 2>/dev/null || true
fi
# 2. Remove all compose projects + direct-run resources + per-run images
# for this run. ci-cleanup.sh wraps each docker op in `timeout`; bound
# the whole sweep too so the trap can never wedge.
timeout 150 bash "$SCRIPT_DIR/ci-cleanup.sh" \
--label "$CI_LABEL" \
--project-prefix "$CI_PROJECT_PREFIX" \
--images "$CI_IMAGE_TEST $CI_IMAGE_APP" >/dev/null 2>&1 || true
}
on_signal() {
local sig="$1"
# Block re-entry and stop the EXIT trap from overriding the signal code.
trap '' TERM INT EXIT
echo "" >&2
fail "Received SIG$sig — cancelling run, tearing down ${CI_PROJECT_PREFIX}"
ci_teardown
# 128 + signal number: distinct from 0 (green) / 1 (stage failed).
if [[ "$sig" == "TERM" ]]; then exit 143; else exit 130; fi
}
on_exit() {
local code=$?
trap '' TERM INT EXIT
ci_teardown
exit "$code"
}
trap 'on_signal TERM' TERM
trap 'on_signal INT' INT
trap 'on_exit' EXIT
# ── Stage 1: Build ─────────────────────────────────────────────────────────
run_build() {
@@ -317,6 +424,7 @@ run_static() {
local topology="$1"
local compose="testing/static/docker-compose.yml"
local rc=0
export COMPOSE_PROJECT_NAME="$(ci_project static)"
info "[$topology] Generating configs"
bash testing/static/scripts/generate-configs.sh "$topology" || { record "static-$topology" 1; return; }
@@ -341,6 +449,7 @@ run_static() {
run_rekey() {
local compose="testing/static/docker-compose.yml"
local rc=0
export COMPOSE_PROJECT_NAME="$(ci_project static)"
info "[rekey] Generating configs"
bash testing/static/scripts/generate-configs.sh rekey || { record "rekey" 1; return; }
@@ -369,6 +478,7 @@ run_rekey() {
run_admission_cap() {
local compose="testing/static/docker-compose.yml"
local rc=0
export COMPOSE_PROJECT_NAME="$(ci_project static)"
info "[admission-cap] Generating configs"
bash testing/static/scripts/generate-configs.sh mesh || { record "admission-cap" 1; return; }
@@ -395,6 +505,9 @@ run_chaos() {
local name="$1"
shift
local rc=0
# Distinct project per scenario (chaos children run in parallel). When
# invoked from a background subshell this export is local to that child.
export COMPOSE_PROJECT_NAME="$(ci_project "chaos-$name")"
info "[chaos/$name] Running simulation"
if bash testing/chaos/scripts/chaos.sh "$@" 2>&1; then
@@ -410,6 +523,7 @@ run_chaos() {
run_gateway() {
local compose="testing/static/docker-compose.yml"
local rc=0
export COMPOSE_PROJECT_NAME="$(ci_project static)"
info "[gateway] Generating configs"
bash testing/static/scripts/generate-configs.sh gateway gateway-test || { record "gateway" 1; return; }
@@ -434,6 +548,7 @@ run_gateway() {
# Run sidecar test
run_sidecar() {
local rc=0
export COMPOSE_PROJECT_NAME="$(ci_project sidecar)"
info "[sidecar] Running integration test"
if bash testing/sidecar/scripts/test-sidecar.sh --skip-build 2>&1; then
@@ -450,6 +565,7 @@ run_sidecar() {
run_rekey_accept_off() {
local compose="testing/static/docker-compose.yml"
local rc=0
export COMPOSE_PROJECT_NAME="$(ci_project static)"
info "[rekey-accept-off] Generating configs"
bash testing/static/scripts/generate-configs.sh rekey-accept-off || \
@@ -484,6 +600,7 @@ run_rekey_accept_off() {
run_rekey_outbound_only() {
local compose="testing/static/docker-compose.yml"
local rc=0
export COMPOSE_PROJECT_NAME="$(ci_project static)"
info "[rekey-outbound-only] Generating configs"
bash testing/static/scripts/generate-configs.sh rekey-outbound-only || \
@@ -512,6 +629,7 @@ run_rekey_outbound_only() {
# Run ACL allowlist integration test
run_acl_allowlist() {
export COMPOSE_PROJECT_NAME="$(ci_project acl)"
info "[acl-allowlist] Running integration test"
if bash testing/acl-allowlist/test.sh --skip-build 2>&1; then
record "acl-allowlist" 0
@@ -522,6 +640,7 @@ run_acl_allowlist() {
# Run firewall baseline integration test
run_firewall() {
export COMPOSE_PROJECT_NAME="$(ci_project firewall)"
info "[firewall] Running integration test"
if bash testing/firewall/test.sh --skip-build 2>&1; then
record "firewall" 0
@@ -533,6 +652,7 @@ run_firewall() {
# Run a NAT scenario (cone, symmetric, lan)
run_nat() {
local scenario="$1"
export COMPOSE_PROJECT_NAME="$(ci_project nat)"
info "[nat-$scenario] Running NAT lab"
if bash testing/nat/scripts/nat-test.sh "$scenario" 2>&1; then
record "nat-$scenario" 0
@@ -546,6 +666,7 @@ run_nat() {
# (A→B publish/consume), Phase 2 (B→A reverse), and Phase 3 (malformed
# advert injected directly to the relay; consumer-liveness assertion).
run_nostr_publish_consume() {
export COMPOSE_PROJECT_NAME="$(ci_project nat)"
info "[nostr-publish-consume] Running Nostr publish/consume test"
if bash testing/nat/scripts/nostr-relay-test.sh 2>&1; then
record "nostr-publish-consume" 0
@@ -560,6 +681,7 @@ run_nostr_publish_consume() {
# kill. Asserts the daemon detects each fault, recovers from delay, and
# never panics.
run_stun_faults() {
export COMPOSE_PROJECT_NAME="$(ci_project nat)"
info "[stun-faults] Running STUN fault-injection test"
if bash testing/nat/scripts/stun-faults-test.sh 2>&1; then
record "stun-faults" 0
@@ -590,6 +712,7 @@ run_deb_install() {
# Run Tor SOCKS5 outbound test (live Tor network)
run_tor_socks5() {
export COMPOSE_PROJECT_NAME="$(ci_project tor-socks5)"
info "[tor-socks5] Running Tor SOCKS5 outbound test (live Tor)"
if bash testing/tor/socks5-outbound/scripts/tor-test.sh 2>&1; then
record "tor-socks5" 0
@@ -600,6 +723,7 @@ run_tor_socks5() {
# Run Tor directory-mode test (live Tor network)
run_tor_directory() {
export COMPOSE_PROJECT_NAME="$(ci_project tor-directory)"
info "[tor-directory] Running Tor directory-mode test (live Tor)"
if bash testing/tor/directory-mode/scripts/directory-test.sh 2>&1; then
record "tor-directory" 0
@@ -616,10 +740,20 @@ run_integration() {
info "Installing release binaries"
install_binaries testing/docker
# Build unified test image once (used by all harnesses)
info "Building fips-test Docker image"
docker build -t fips-test:latest testing/docker --quiet || { record "docker-build" 1; return; }
docker build -t fips-test-app:latest -f testing/docker/Dockerfile.app testing/docker --quiet || { record "docker-build-app" 1; return; }
# Build unified test image once (used by all harnesses). Tag per-run
# (fips-test:${run}) so a build killed mid-flight never wedges the next
# run's rebuild, and concurrent runs never clobber each other's image.
# Then retag :latest for the compose files / harness scripts that still
# reference fips-test:latest directly; the retag happens only after BOTH
# builds succeed, so :latest never points at a half-built image.
info "Building $CI_IMAGE_TEST Docker image"
docker build -t "$CI_IMAGE_TEST" --label "$CI_LABEL" testing/docker --quiet \
|| { record "docker-build" 1; return; }
docker build -t "$CI_IMAGE_APP" --label "$CI_LABEL" \
-f testing/docker/Dockerfile.app testing/docker --quiet \
|| { record "docker-build-app" 1; return; }
docker tag "$CI_IMAGE_TEST" fips-test:latest
docker tag "$CI_IMAGE_APP" fips-test-app:latest
# Single suite mode
if [[ -n "$ONLY_SUITE" ]]; then
@@ -673,6 +807,7 @@ run_integration() {
local pids=()
local suite_names=()
local running=0
local chaos_idx=0
for entry in "${CHAOS_SUITES[@]}"; do
# Parse: "display-name scenario [flags...]"
@@ -680,6 +815,15 @@ run_integration() {
local name="${parts[0]}"
local args=("${parts[@]:1}")
# Give each chaos child a unique, non-overlapping /24 in 10.30.x so
# parallel children never collide with each other, and so a chaos
# net can never swallow a fixed-subnet suite (sidecar/static/nat in
# 172.x). 10.30.x sits outside docker's default-address-pool range
# (172.17-31 / 192.168), so auto-assigned nets can't land on it
# either. Node IPs derive from this subnet inside the sim.
args+=("--subnet" "10.30.${chaos_idx}.0/24")
chaos_idx=$((chaos_idx + 1))
# Throttle: wait for a slot
while [[ $running -ge $PARALLEL_JOBS ]]; do
wait -n -p done_pid 2>/dev/null || true
@@ -693,6 +837,7 @@ run_integration() {
run_chaos "$name" "${args[@]}" >"$logfile" 2>&1
) &
pids+=($!)
CI_CHAOS_PIDS+=("$!")
suite_names+=("$name:$logfile")
running=$((running + 1))
done
@@ -715,6 +860,9 @@ run_integration() {
fi
rm -f "$logfile"
done
# All chaos children have been waited on; clear so a later signal does
# not try to kill already-reaped PIDs.
CI_CHAOS_PIDS=()
fi
# Sidecar
+1
View File
@@ -65,6 +65,7 @@ start_systemd_container_with_tun() {
local name="$1" image="$2"
cleanup_container "$name"
docker run -d --name "$name" \
--label com.corganlabs.fips-ci=1 \
--privileged \
--cgroupns=host \
--device /dev/net/tun \
+2
View File
@@ -67,6 +67,7 @@ start_systemd_container() {
local name="$1" image="$2"
cleanup_container "$name"
docker run -d --name "$name" \
--label com.corganlabs.fips-ci=1 \
--privileged \
--cgroupns=host \
-v /sys/fs/cgroup:/sys/fs/cgroup:rw \
@@ -79,6 +80,7 @@ start_systemd_container_with_tun() {
local name="$1" image="$2"
cleanup_container "$name"
docker run -d --name "$name" \
--label com.corganlabs.fips-ci=1 \
--privileged \
--cgroupns=host \
--device /dev/net/tun \
+2
View File
@@ -1,6 +1,8 @@
networks:
fw-net:
driver: bridge
labels:
- "com.corganlabs.fips-ci=1"
ipam:
config:
- subnet: 172.32.0.0/24
+30 -14
View File
@@ -162,6 +162,13 @@ REKEY_SETTLE=12 # FSP-cutover settle budget (Phase 6)
# strict assertion sweep runs. A genuinely stuck pair still fails — the
# poll times out and the recording sweep captures it.
POST_REKEY_TIMEOUT=45
# Progress-aware stall budget for the convergence detector
# (wait_for_full_baseline → wait_until_connected). If no additional pair
# becomes reachable for this long while more than the near-converged
# slack of pairs is still down, the detector gives up early instead of
# burning the whole convergence/post-rekey window; any progress resets
# the clock, so a slow-but-converging mesh under netem keeps polling.
RECONVERGE_STALL=15
LOG_POLL_INTERVAL=2
# Data-plane continuity stream (control-differential). Streams run a
@@ -434,25 +441,27 @@ ping_all_pairs() {
done
}
# Convergence detector probe: one full all-pairs ping sweep in the
# "convergence" context (which ping_all_pairs deliberately does NOT
# record as a failure), setting PASSED/FAILED for wait_until_connected.
_baseline_probe() {
ping_all_pairs quiet 1 "convergence"
}
# Poll until every directed pair pings clean, or until timeout. This is a
# convergence DETECTOR — used at establishment (Phase 1) and after each
# rekey (Phases 3/5). It pings with the "convergence" context, which
# ping_all_pairs deliberately does not record as a failure; the caller
# runs a separate strict assertion sweep afterwards.
#
# Delegates to the shared progress-aware wait_until_connected so the
# deadline extends while more pairs are still coming up and gives up fast
# on a genuine stall, instead of the prior fixed-deadline poll that could
# false-time-out under heavy CI contention even while still converging.
# Returns 0 once every pair is reachable, 1 on stall/timeout.
wait_for_full_baseline() {
local timeout="$1"
local start=$SECONDS
local best_passed=0 best_failed="$NUM_DIRECTED"
while (( SECONDS - start < timeout )); do
ping_all_pairs quiet 1 "convergence"
if [ "$PASSED" -gt "$best_passed" ]; then
best_passed="$PASSED"; best_failed="$FAILED"
fi
[ "$FAILED" -eq 0 ] && return 0
sleep 1
done
PASSED="$best_passed"; FAILED="$best_failed"
return 1
wait_until_connected _baseline_probe "$timeout" "$RECONVERGE_STALL"
}
phase_result() {
@@ -759,8 +768,15 @@ echo ""
# ── Phase 4: second rekey cycle ──────────────────────────────────────
echo "Phase 4: Second rekey cycle (waiting ${SECOND_REKEY_WAIT}s)"
sleep "$SECOND_REKEY_WAIT"
echo "Phase 4: Second rekey cycle (waiting up to ${SECOND_REKEY_WAIT}s for the next cutover)"
# Poll for the next FMP cutover beyond what Phases 2/3 already saw, using
# the same pre/post cutover-count delta convention as the control window
# (Phase 1b), instead of a blind sleep. Bounded by SECOND_REKEY_WAIT so a
# stalled rekey falls through to the strict Phase 5/6 assertions.
fmp_cutovers_before="$(count_log_pattern 'Rekey cutover complete \(initiator\), K-bit flipped')"
wait_for_log_pattern_count \
"Rekey cutover complete \(initiator\), K-bit flipped" \
"$((fmp_cutovers_before + 1))" "$SECOND_REKEY_WAIT" || true
echo ""
echo "Phase 5: Post-second-rekey connectivity (reconverge within ${POST_REKEY_TIMEOUT}s)"
+4
View File
@@ -1,11 +1,15 @@
networks:
wan:
driver: bridge
labels:
- "com.corganlabs.fips-ci=1"
ipam:
config:
- subnet: 172.31.254.0/24
shared-lan:
driver: bridge
labels:
- "com.corganlabs.fips-ci=1"
ipam:
config:
- subnet: 172.31.10.0/24
+1 -1
View File
@@ -202,7 +202,7 @@ dump_stun_udp_probe() {
local capture_file
capture_file="$(mktemp)"
docker run --rm --net=container:fips-nat-stun --cap-add NET_ADMIN --cap-add NET_RAW \
docker run --rm --label com.corganlabs.fips-ci=1 --net=container:fips-nat-stun --cap-add NET_ADMIN --cap-add NET_RAW \
--entrypoint sh "$helper_image" \
-lc "timeout 8 tcpdump -ni any 'udp and not port 53' -c 80" \
>"$capture_file" 2>&1 &
+1
View File
@@ -57,6 +57,7 @@ run_host_ip() {
local image="$1"
shift
docker run --rm \
--label com.corganlabs.fips-ci=1 \
--privileged \
--net=host \
--pid=host \
+10
View File
@@ -2,6 +2,12 @@ networks:
fips-net:
name: ${FIPS_NETWORK:-fips-sidecar-net}
driver: bridge
# Reap label: this harness uses fixed -p sidecar-{a,b,c} project names
# (the test execs containers by those derived names), so its resources are
# NOT covered by ci-local's fipsci_ project prefix. Label them directly so
# ci-cleanup.sh can still reap them after a preemption/SIGKILL.
labels:
- "com.corganlabs.fips-ci=1"
ipam:
config:
- subnet: ${FIPS_SUBNET:-172.20.1.0/24}
@@ -10,6 +16,8 @@ services:
fips:
image: fips-test:latest
hostname: fips-sidecar
labels:
- "com.corganlabs.fips-ci=1"
cap_add:
- NET_ADMIN
devices:
@@ -33,6 +41,8 @@ services:
app:
image: fips-test-app:latest
labels:
- "com.corganlabs.fips-ci=1"
network_mode: "service:fips"
depends_on:
- fips
+6 -2
View File
@@ -1,11 +1,15 @@
networks:
fips-net:
driver: bridge
labels:
- "com.corganlabs.fips-ci=1"
ipam:
config:
- subnet: 172.20.0.0/24
gateway-lan:
driver: bridge
labels:
- "com.corganlabs.fips-ci=1"
enable_ipv6: true
ipam:
config:
@@ -534,7 +538,7 @@ services:
ipv4_address: 172.20.0.12
gw-client:
image: fips-test-app:latest
image: ${FIPS_TEST_APP_IMAGE:-fips-test-app:latest}
profiles: ["gateway"]
container_name: fips-gw-client
hostname: gw-client
@@ -556,7 +560,7 @@ services:
# Same image and gateway-lan attachment as
# gw-client; the gateway must allocate a distinct virtual IP for it.
gw-client-2:
image: fips-test-app:latest
image: ${FIPS_TEST_APP_IMAGE:-fips-test-app:latest}
profiles: ["gateway"]
container_name: fips-gw-client-2
hostname: gw-client-2
+1 -1
View File
@@ -149,7 +149,7 @@ HELPER_IMAGE=$(docker inspect -f '{{.Config.Image}}' fips-node-$CAP_NODE 2>/dev/
LOAD_PID=$!
# Foreground: tcpdump capture for CAPTURE_SECS
docker run --rm --net=container:fips-node-$CAP_NODE \
docker run --rm --label com.corganlabs.fips-ci=1 --net=container:fips-node-$CAP_NODE \
--cap-add NET_ADMIN --cap-add NET_RAW \
--entrypoint sh "$HELPER_IMAGE" \
-c "timeout $CAPTURE_SECS tcpdump -nn -i any 'udp port 2121' -l 2>&1 || true" \
+38 -9
View File
@@ -158,9 +158,20 @@ REKEY_SETTLE=12 # > DRAIN_WINDOW_SECS (10) so post-rekey samples are off
# fully converged. Keep this bounded to preserve a meaningful scheduling check
# while still allowing for log visibility at the timeout edge.
FIRST_REKEY_TIMEOUT=$((REKEY_AFTER_SECS + 15))
SECOND_REKEY_WAIT=40 # wait for second cycle
SECOND_REKEY_WAIT=40 # upper bound for observing the second rekey cutover
LOG_EVENT_POLL_INTERVAL=1
# Post-rekey data-plane reconvergence is polled, not fixed-slept.
# wait_until_connected drives the same all-pairs ping sweep the strict
# per-phase assert depends on: it fails fast when reachability stops
# improving (no new reachable pair for RECONVERGE_STALL seconds while
# more than the near-converged slack of pairs is still down) and extends
# its deadline up to RECONVERGE_TIMEOUT only while pairs keep coming up,
# so a slow-but-converging post-rekey mesh under CI load passes instead
# of false-failing on a fixed settle window.
RECONVERGE_TIMEOUT=45
RECONVERGE_STALL=10
TIMEOUT=5
CONVERGENCE_PING_TIMEOUT=1
# Strict-ping retry policy for the per-phase ping_all asserts. Under 1%
@@ -392,20 +403,38 @@ assert_min_count "Rekey cutover complete (initiator), K-bit flipped" 1 \
phase_result "FMP rekey events"
echo ""
# Verify connectivity after first rekey (strict — no failures allowed)
echo "Phase 3: Post-rekey connectivity (settling ${REKEY_SETTLE}s)"
sleep "$REKEY_SETTLE"
# Verify connectivity after first rekey (strict — no failures allowed).
# Wait for the data plane to reconverge with a progress-aware deadline
# (the same all-pairs ping sweep the strict assert below depends on)
# instead of a fixed settle: fail fast on a genuinely stuck pair, extend
# only while reachability is still climbing. The strict ping_all is the
# actual assertion, run only after the reconverge poll.
echo "Phase 3: Post-rekey connectivity (reconverging up to ${RECONVERGE_TIMEOUT}s)"
wait_until_connected _baseline_ping "$RECONVERGE_TIMEOUT" "$RECONVERGE_STALL" || true
ping_all "" "$TIMEOUT" "$MAX_PING_ATTEMPTS"
phase_result "Post-first-rekey (all 20 pairs)"
echo ""
# ── Phase 4: Wait for second rekey cycle ──────────────────────────────
echo "Phase 4: Second rekey cycle (waiting ${SECOND_REKEY_WAIT}s)"
sleep "$SECOND_REKEY_WAIT"
# Poll for the next FMP rekey cutover instead of blind-sleeping: capture
# the cutover count reached so far, then wait until at least one more
# cutover lands — that increment is the second rekey cycle firing.
# Bounded by SECOND_REKEY_WAIT so a stalled rekey still falls through to
# the strict Phase 5/6 assertions rather than hanging.
echo "Phase 4: Second rekey cycle (waiting up to ${SECOND_REKEY_WAIT}s for the next cutover)"
fmp_cutovers_before=$(count_log_pattern "Rekey cutover complete (initiator), K-bit flipped")
wait_for_log_pattern_count \
"Rekey cutover complete (initiator), K-bit flipped" \
"$((fmp_cutovers_before + 1))" "$SECOND_REKEY_WAIT" || true
# Verify connectivity after second rekey (back-to-back)
echo "Phase 5: Post-second-rekey connectivity (settling ${REKEY_SETTLE}s)"
sleep "$REKEY_SETTLE"
# Verify connectivity after second rekey (back-to-back). This is the
# site of the recurring post-second-rekey straggler-pair flake: wait for
# the data plane to reconverge with a progress-aware deadline (the same
# all-pairs ping sweep the strict assert below depends on) rather than a
# fixed settle window — fail fast if reachability stops improving, extend
# only while pairs are still coming up.
echo "Phase 5: Post-second-rekey connectivity (reconverging up to ${RECONVERGE_TIMEOUT}s)"
wait_until_connected _baseline_ping "$RECONVERGE_TIMEOUT" "$RECONVERGE_STALL" || true
ping_all "" "$TIMEOUT" "$MAX_PING_ATTEMPTS"
phase_result "Post-second-rekey (all 20 pairs)"
echo ""
@@ -11,6 +11,8 @@
networks:
dir-test:
driver: bridge
labels:
- "com.corganlabs.fips-ci=1"
services:
fips-a:
@@ -10,6 +10,8 @@
networks:
tor-test:
driver: bridge
labels:
- "com.corganlabs.fips-ci=1"
services:
tor-daemon: