mirror of
https://github.com/jmcorgan/fips.git
synced 2026-10-05 11:08:25 +00:00
ArjenandJohnathan Corgan
abe3f0b40a
An interface-bound transport is a long-lived object that is *sometimes bound*. The interface it names may not exist when the daemon starts, may appear minutes later, may vanish and return mid-operation, and may never appear at all. Until now the first observation was final: an interface missing at start was logged once and skipped for the life of the process, and one that disappeared at runtime published a health change but was never rebound. On OpenWrt that is a live bug. procd starts fips before wifi has created fips-mesh0 / fips-ap0, both transports are skipped and never retried, and the 802.11s peer link forms anyway (that is mac80211, not the daemon) — so the node looks healthy and reaches nothing. Give every interface-bound transport presence state and a binder task. start_async now returns Ok with the transport ABSENT rather than failing; the binder binds when the interface appears, tears the socket down when it goes away, and rebinds when it returns. Start-time absence and runtime detach are one transition and one code path. TransportError::InterfaceUnavailable makes absence branchable: a missing interface and a typo'd interface name were the same flat StartFailed (String), so nothing downstream could tell a state from a fault. A bind failure that is *not* absence — no CAP_NET_RAW, no readable /dev/bpf* — still fails the start, because retrying a socket that can never open behind a Degraded nobody is watching is worse than dying loudly at boot. **Presence is IFF_UP, not IFF_UP|IFF_RUNNING.** Carrier and bindability are different questions and only the second belongs in a bind gate. An AF_PACKET socket on a carrier-less bridge is valid and starts carrying traffic the instant a member port comes up, with no rebind. Gating on carrier would report a wifi-only router Degraded forever for an empty br-lan, turn every carrier flap the socket would have survived into an unbind/rebind cycle, and deadlock an 802.11s interface that reports RUNNING only once it has peered — peering needs beacons, beacons need a bound socket. Carrier is reported beside presence in show_transports instead. **A name is not a device.** Both backends bind by device: AF_PACKET stores sll_ifindex, a BPF descriptor follows the interface it was attached to. A netdev deleted and recreated under the same name leaves the socket attached to something gone while the name resolves perfectly well, and nothing notices — a stale AF_PACKET socket never becomes readable, so the receive loop neither errors nor exits, and send failures go to the caller rather than the binder. A listen-only node (announce: false, no beacon sender to fail) sat present and deaf indefinitely after a wifi reload. The bound index is captured at bind and compared on every poll. A name reappearing with a different MAC is different hardware: cached neighbors are dropped rather than resumed onto. **Detection is event-driven** where the kernel offers a source — netlink RTNLGRP_LINK on Linux, PF_ROUTE on macOS and FreeBSD — with a 1 s getifaddrs poll underneath as a backstop. Link-event payloads are not parsed: an event is a hint to re-run the probe, which is cheap and authoritative, and a parser's bugs would be presence bugs. Probes are coalesced to ten a second because PF_ROUTE has no group filter and delivers every routing message on the host; a persistently failing event source is logged once, backed off, and abandoned for the poll after five errors rather than spinning a core silently on ENOBUFS. **Degraded becomes a level, not a latch.** The supervisor's reason set was monotonic, which was correct while no child could recover; with recovery it would have come to mean "something broke at some point since boot" rather than "something is broken now". Absence lives in its own reversible set, health is recomputed on every transition in both directions, and an absent transport still counts as up so a single-interface node that boots before its wifi degrades rather than exiting on NoTransports. There is deliberately no restart action in the FSM: rebinding belongs next to the file descriptor, and a supervisor-authored retry would be a second mechanism racing the first for the same socket. **The node's egress MTU follows the bound set.** transport_mtu filters on is_bound(), not is_operational() — an interface-bound transport is operational from the moment it starts, so the weaker predicate let hardware that had never appeared clamp the whole node's IPv6 MTU. Because that minimum now moves at runtime, the TUN reader and writer read the TCP MSS ceiling from a shared atomic instead of a u16 captured at spawn; every other consumer (show_status, the snapshot, the session-layer fragmentation check) already read it live, so the clamp was the one place the daemon could report one effective MTU and enforce another. It moves in both directions: a narrow interface appearing tightens it, its departure releases it. MSS is negotiated per connection, so a change binds connections opened after it. **Logging is per edge, never per attempt**, with one deadline after it. The edge itself is not an error — info at boot, warn on a runtime detach, since an interface bound a moment later is the ordinary case this mechanism exists to absorb and crying error at t=0 then "recovered" at t=0.2s is the failure the rule exists to prevent. Ten seconds is the whole grace: past it, absence is no longer a race against a radio or a container, so a required interface still missing is reported once at error. Start-time absence and a runtime detach share that one deadline, as they share everything else here. Said once, not repeated: how long an absence has lasted is state, published as interface.since_secs and as Degraded, and a monitor can threshold it per deployment rather than the daemon compiling a schedule in. Successful rebinds are damped. Backoff covers failed binds; the nastier case is binds that keep succeeding into a socket that dies moments later, which a receive loop giving up on a persistent error while the interface stays UP produces once per second forever. Consecutive bindings dying inside ten seconds back off on the 1 s → 30 s curve, and past three the binder stops announcing each bind as a recovery until one lasts. The receive loop backs off and exits on a dead socket instead of spinning on Err with a warn per iteration, and the ad-hoc ENXIO socket reopen in the beacon sender is gone: both hand recovery to the presence machine, one mechanism for every cause rather than one hack per symptom. Beacons pause while absent because the task simply does not exist then. transports.ethernet.*.optional (default false) decides how loudly absence is reported. Naming an interface is a statement that you expect it, so the default complains: Degraded, and the error at the deadline. optional: true is silent — info on the edge, no health impact, no error — for hardware legitimately not always there. It describes the interface's presence, not the transport's importance, and no value of it makes a missing interface fatal at startup. show_transports grows an `interface` block — presence, carrier, policy, how long the phase has been held, bind count, failed attempts. The original boot race was expensive precisely because nothing an operator could see said the node was deaf. ci: exercise the presence probe against musl and an address-less interface Two legs, both about the assumption the presence probe rests on. OpenWrt — the platform dynamic interface binding exists for — is musl, and musl reimplements getifaddrs independently of glibc. `interface_present` reads ifa_flags out of it, and the interfaces this feature exists for (fips-mesh0, fips-ap0) are unbridged with no IP address at all, which is exactly where getifaddrs implementations differ. Every other leg is glibc, so without this one the probe was asserted on a libc no test had ever run it against, on the target it was written for. Building for the musl target rather than inside an Alpine container keeps the leg to a cross-build plus a run, without a second toolchain image to maintain. The address-less interface is created on both this leg and the glibc one. Pinning the contract on both libcs is what turns "glibc and musl agree about getifaddrs" from an assumption into a checked fact, and makes the glibc leg fail first if glibc is the one that changes. Loopback cannot stand in: it has 127.0.0.1, so probing it asks whether getifaddrs works rather than whether it reports this. Deliberately not tolerant of failure — a guard that quietly does not run is worse than no guard. The iface-binding suite is deliberately not added to the integration matrix here. Its files and its testing/ci-local.sh registration arrive in the next commit, and the GitHub workflow's matrix entry now goes with them, so both runners gain the suite together and testing/check-ci-parity.sh holds at every point in this history. docs: document dynamic interface binding The transport-layer design gains an Interface Presence section: the three deployment scenarios that motivated it, the presence machine and its two invariants, why presence is IFF_UP and not IFF_UP|IFF_RUNNING, interface identity and the recreated-netdev case, the detection sources and their two rate limits, the optional policy, health as a level rather than a latch, the egress-MTU consequence, the logging rules, and what the mechanism retires. It records why the error deadline is a single ten-second window rather than a repeating severity ladder, so the ladder does not come back. The control-socket and fipsctl references described show_transports without its `interface` block, and the transport-layer state machine still implied that Up meant bound. Both now say what Up and presence each describe and how they differ, since "transport up, interface absent" is a normal state an operator will meet and would otherwise read as a contradiction. The configuration reference documents `optional` with the absence, log and retry behaviour of each setting, and the ground-up tutorial's dongle example gains optional: true, which is what that example is actually for. fix(transport): pair every presence edge the binder publishes Six defects, all in the wiring around the presence machine rather than in the machine itself. The state machines were well tested; what went untested was how the binder feeds them, and every one of these lives there. **The first detach after a clean start never reached node health.** start_async binds inline and publishes that edge itself, before binder_loop exists. The loop then built a fresh ChurnGuard, which had therefore recorded no bind — and detached() takes `announced` to decide whether an edge is owed, so it asked for no retraction. A node that booted with its interface present and then lost it kept reporting Full for as long as the interface stayed away, while show_transports said absent and the log said detached. stabilized() could not repair it either: it early-returns while bound_at is None, which it is for a bind the loop did not perform, so the guard stayed unseeded until a second detach happened to fix it. The loop now seeds the guard from the binding it inherits. This is not the cable-unplug case. Presence is IFF_UP, so a carrier loss is correctly not a detach at all; it takes an admin down, a netdev delete, or a device removal — a wifi reload, a hostapd restart, a dongle pulled. **An optional interface published nothing, and the MTU floor rode the same channel.** Filtering the edge at the source conflated two questions that happen to share a transport: whether node health should move, and whether the bound set changed. Only the first is policy. The second determines the node's egress MTU floor, and refresh_tun_mss_ceiling has no other trigger — so an optional transport binding or unbinding at runtime left the TUN MSS clamp derived from a bound set that no longer existed, reporting one effective MTU and clamping to another. That is the defect the MssCeiling work was written to remove, reintroduced for exactly the transports the shipped OpenWrt config marks optional: five of seven. TransportPresence gains health_relevant; the edge always goes out and only health is filtered. **A permanent bind fault went unlogged if any absence race preceded it.** record_attempt ran ahead of the InterfaceUnavailable arm, so a probe that won a race the bind then lost burned the counter that gates the only error a real fault ever gets — emitted on attempts == 1. A flapping interface followed by CAP_NET_RAW being dropped or /dev/bpf* exhausting therefore reported nothing at all, for the life of the process, and the operator got report_sustained_absence's "still missing" instead: wrong, for an interface that is present. An absence race is not a bind attempt and no longer counts as one. **The watcher's give-up did not stick.** Abandoning the event source parked on pending() *inside the changed() future*, and that future is constructed fresh on every pass of the binder's select! and dropped whenever the poll ticker wins. So the next pass re-read the dead socket, re-counted the error and re-logged "not recoverable" — once per wake-up, forever, which at a 1 s tick across the shipped seven-transport config is seven warnings and seven failing syscalls a second on flash-backed logging. The 100 ms backoff lived in the dropped future too and never applied across passes. Abandonment is now state on the watcher. **A zero-length read livelocked the binder.** try_io clears readiness only on WouldBlock, which the sibling error arm handles by hand and this one did not — so breaking out left readable() instantly ready with nothing to read, and the loop never returned Pending. That starves the select! of its ticker entirely and takes presence detection down with it. Cleared and treated as a fault so the give-up path applies. **A failed presence probe read as an absent interface.** getifaddrs is a netlink dump and fails for reasons that have nothing to do with the interface: ENOBUFS under memory pressure, EMFILE or ENFILE under fd exhaustion, since it opens a socket of its own. Answering "not present" there tore down a working socket and degraded health over a transient syscall failure, undiagnosably, and under fd exhaustion the rebind could not have succeeded anyway. interface_has_flags now distinguishes the two; the detach gate holds its binding when the kernel will not answer, while the bind gate still treats it as absence and retries. **On macOS the reader thread could not die, so a dead socket read as live.** Any read() failure other than EBADF — ENXIO being the one that matters, which is what BPF answers once the interface it was attached to is torn away — reset the parse buffer and continued. The thread never returned, so the channel never closed, so recv_from never failed, so the tokio task never exited, so tasks_alive() reported a dead socket as a live one. Detach detection on macOS reduced to the name and the index, and an interface reset in place left the transport present and deaf. It now gives up after a streak, and the return closes the channel the binder is actually watching. Two smaller pairings while here: stop_async retracts the edge it would otherwise leave standing for a socket that is gone, and start_async hands a refused edge to the binder to retry rather than dropping the only edge either consumer will see until the interface next moves. The absence deadline is stamped at start rather than at construction, so a transport staged for longer than the bring-up window no longer reports sustained absence on its first tick having given the interface no window at all. feat(config): reject impossible ethernet interface names at load Config::validate never inspected transports.ethernet, so an empty interface name, a name past the kernel's 15-byte limit, and two transports naming the same netdev all loaded cleanly and failed only at runtime — the first two as a permanent absence indistinguishable from an interface that has not been created yet. That indistinguishability is deliberate and worth keeping: waiting is the right answer for an interface the operator has not made yet, and the daemon cannot know which of the two it is looking at. Which is exactly why the syntactic gate earns its place. A name that is *impossible* is the one case still separable from "not there yet", and without the check a typo costs a permanently Degraded node whose only symptom is an interface that never arrives — the failure mode the presence machine exists to make legible, reintroduced one level up. Syntax only. Whether a well-formed name exists stays the binder's question, asked once a second, forever. The duplicate check is a different fault: two transports on one netdev means two sockets on the same device at the same ethertype, each receiving every frame the other does. test(transport): close the vacuous and uncovered branches Six tests that asserted nothing, or asserted less than they claimed. `a_bind_fault_still_fails_the_start` returned early whenever the socket *could* be opened, so it was vacuous as root and on any developer machine with a group-readable /dev/bpf* — the fail-fast path it is named for went unchecked exactly where someone was most likely to run it. It now asserts in both halves, and the privileged half is worth more than the fix: a present, bindable interface binding inline is the ordinary case on a booted router, and no other unit test reaches it, because every other one here names an interface that does not exist. The bind-success path had no unit coverage at all. `an_interface_with_no_addresses_is_still_present` returned early when the fixture was absent — no fixture, pass. It still has to skip on a machine with no address-less interface, so the guard is the runner declaring that it has fixtures: CI now sets FIPS_TEST_REQUIRE_FIXTURES beside FIPS_TEST_ADDRLESS_IFACE, and the test fails rather than skips if the fixture step is ever removed or renamed. Its `let _ = interface_carrier(...)` is now asserted too: if presence and carrier ever collapsed into one read, a carrier-less bridge would report absent and the whole IFF_UP-not-IFF_RUNNING decision would be silently undone. `policy_labels` checked `Required.as_str()` and not `Optional.as_str()`, so a swapped pair would paint every expected interface as the tolerated kind and stay green. `Presence::as_str` had no test at all — `binding` was never observed by anything, anywhere. Three binder branches had no coverage: a transport restarting (the second `start_async` clearing the previous run's stop flag — only a second *stop* was tested, so a transport that could never restart passed everything), the episode clock being restamped at start rather than at construction, and the refused-edge retry actually delivering. The last one matters most: the existing test filled the channel and dropped the receiver, so a slot that captured an edge and never re-sent it would pass while health sat on a stale level forever. And the hardware-change boundary `record_bind` returns, which the neighbour flush hangs off: false on a first bind (or every clean start would drop a cache it had just built) and true once, not stickily, on a MAC change. Each new test was verified against the defect it guards — comment out the `shutdown.store(false)`, the `mark_starting()`, or the seeded `unpublished`, and the corresponding test goes red while the rest stay green. fix(test): correct the dummy-carrier assertion, and pin Darwin's presence probe The carrier assertion added ind6240698was wrong and would have failed both Linux legs. A `dummy` interface brought up reports `<BROADCAST,NOARP,UP, LOWER_UP>` — `IFF_RUNNING` is set, so it *has* carrier. Verified against the exact fixture CI builds, `addrgenmode none` and all, rather than against the comment: the original code discarded the result and its comment claimed "up but not running", which is what made asserting it look safe. So the fixture pins address-less *presence* and cannot demonstrate the presence-vs-carrier split at all — an interface up with no carrier is a bridge with nothing plugged in, which no fixture here creates. `carrier_is_reported_separately_from_presence` pins that split from the other side. The corrected assertion is Linux-only, because the expected answer is a property of the fixture device rather than of the code. That is also what lets the macOS fixture land. The Linux legs pin the address-less contract on glibc and musl, but the BSD-derived `getifaddrs` the macOS backend actually calls had no coverage — the test skipped itself silently on that runner, which is precisely the shape the previous commit was removing. `feth` is macOS's fake-Ethernet pseudo-interface and is created address-less; the step fails the leg rather than testing the wrong thing if the runner hands it an address anyway, mirroring why the Linux step needs `addrgenmode none`. Both branches of the fixture guard were exercised: unset skips and passes, and declared-but-missing fails loudly. The corrected test was run against a real Linux dummy inside a container, not reasoned about. fix(ci): put the macOS presence fixture on the macOS job The `feth` fixture step added in6b9faa0clanded on the Linux `test` job, not on `test-macos`, and has failed CI ever since: create: Host name lookup failure ifconfig: `--help' gives usage information. That is Linux net-tools `ifconfig`, which has no `create` subcommand. So the Linux job ran two address-less-fixture steps — its own correct `ip link add type dummy` one, then a macOS one that cannot work there — while `test-macos` had none at all, leaving the Darwin `getifaddrs` path exactly as uncovered as before. Cause was a pattern-anchored edit: `Install cargo-nextest` followed by `Run unit tests` appears in three jobs, and the insert hit the first match. Moving it then hit the *last* match, which is `test-windows`. It is now placed by job boundary rather than by pattern, and verified per job: `test` and `test-musl` carry the `ip link`/dummy fixture, `test-macos` carries `ifconfig`/`feth`, and `feth` appears exactly once in the file. The verification that missed this was counting steps in `test-macos` and reading 6 as confirmation. Six was the count *before* the insert; seven is what a successful insert looks like. Now asserted by job and by which tool each fixture step uses, so a step in the wrong place fails the check rather than matching a total. test(transport): cover the detach branch and the stop-race check Two of the three untested branches, by two different routes. **The detach decision is now a pure function.** `classify_detach(gone, replaced, dead)` replaces the inline three-way `if` in the binder loop, and all eight input combinations are asserted, plus the precedence between them and the `reason=` labels the integration suite and operators grep for. This does not make `Replaced` or `SocketDied` reachable from a test — both need a bind that succeeded and then a specific external event, and the integration suite cannot arrange the recreate deterministically either, for the reason recorded in reference/notes.md: the netlink event from a delete is acted on within microseconds, so `Gone` wins that race in practice. What it does is split the untested thing in two. The three inputs each already had tests (`interface_present`, `device_replaced`, `tasks_alive`); the branch between them did not, and that half is now total and exhaustive. Precedence is asserted rather than assumed: an interface that has gone away has also trivially been "replaced" and its socket is also dead, so the most specific true statement has to win, and callers pass `replaced` already masked by `!gone` — the function no longer depends on them having masked correctly. **The post-store shutdown check is now tested directly.** It needs a bind that *succeeds*, which is why `a_stop_racing_a_bind_leaves_nothing_behind` could never reach it: that test's interface does not exist, so `bind_and_spawn` refuses at the presence probe several steps earlier. Binding loopback as root reaches it, and the assertion is that a bind completing after a stop undoes its own socket, its own loops, and its own presence. Unprivileged runners skip it, but loudly: a runner declaring `FIPS_TEST_PRIVILEGED` and unable to open a raw socket fails instead of skipping, the same guard shape as `FIPS_TEST_REQUIRE_FIXTURES`. No CI leg sets that yet — unit tests run unprivileged — so today it exercises the branch only under a root container. Verified there, including the negative control: with the post-store check removed the test fails, and with it present it passes. test(transport): run the bind-success half in CI, and cover the replaced device Adds the privileged unit-test step the rest of this depends on, then uses it. **The privileged step.** Every unit-test leg runs unprivileged, so `PacketSocket::open` cannot succeed on any of them and everything past a successful bind runs nowhere in CI: the post-store shutdown check, the `Present` arm of the binder loop, `bind_now` itself. The step builds as the runner user and executes only the test binary under sudo — running `cargo` as root would use root's CARGO_HOME and discard the cache the job just restored. The binary-path extraction was verified locally before being written into the workflow. `FIPS_TEST_PRIVILEGED` is what makes it honest. Tests needing a raw socket skip quietly without it; with it set, a test that cannot open one fails and says so. A runner that stops granting the capability shows up as a red leg rather than as silence. **The replaced device.** `"interface replaced"` — a netdev recreated under the same name, the `wifi reload` case in #125 — could not be reached by the integration suite: with link events live the kernel's `RTM_DELLINK` is acted on within microseconds, so the binder observes "gone" first and takes the branch already covered. Attempting it there passed about one run in three. Rather than race the binder, the test asserts the *inputs*: after a real delete-and-recreate the name still resolves and the bound index no longer matches. Paired with the exhaustive classifier test, which pins that `(gone: false, replaced: true)` maps to `Replaced`, the path is covered without depending on scheduling. A watcher-disable hook was written for the racing approach and removed once this one made it unnecessary. **Two smaller gaps.** `report_sustained_absence` had no direct test — the integration suite infers it from counting ERROR lines, and ten seconds of real time is how a deadline ends up asserted by proxy. A test-only `backdate_for_test` ages the episode clock instead. And the probe floor was asserted only as a relation between two constants; it now has behaviour, via a `probe_delay` helper extracted from the loop. That extraction was not neutral, and its own test caught it: `checked_sub` yields `Some(0)` at exact equality where the loop used a strict `<`, so a zero-length sleep would have replaced no sleep at all. Harmless in effect, wrong in meaning, and fixed. Every new test was run against the defect it guards, in a root container: remove the post-store shutdown check, or make `device_replaced` always answer false, and the corresponding test fails. Mark publish_presence must_use. The defect this commit fixes was a discarded return value, so the fix is made self-guarding: a future call site that drops the pending edge instead of storing it now fails the lint rather than silently reintroducing the bug. Option is not must_use in std, unlike Result, so the attribute has to be explicit.
FIPS: Free Internetworking Peering System
A self-organizing encrypted mesh network built on Nostr identities, capable of operating over arbitrary transports without central infrastructure.
FIPS is under active development. The protocol and APIs are not yet stable. See Status & roadmap below.
What FIPS does
A machine running FIPS becomes a node in the mesh with a self-generated cryptographic identity, tunneling existing IPv6 traffic over the mesh or bypassing IP altogether and letting natively written applications communicate directly with each other. In either case all traffic between nodes is end-to-end encrypted and authenticated.
The mesh is self-organizing and permissionless. Any node can join and reach any other node without a central address registry, routing configuration, or coordination server. Peering between nodes can be manually configured or use auto-discovery.
There are two equally-supported deployment modes.
As an overlay on top of existing IP networks, FIPS lets your node reach any other FIPS node wherever it sits: behind a NAT, on a different ISP, on a phone over cellular, on a laptop with only Bluetooth in range, or behind a Tor onion.
Ground up over raw Ethernet, WiFi, or Bluetooth, FIPS provides a complete permissionless network without any pre-existing IP infrastructure, ISP, or DNS. Any node that joins the link gets routable IPv6 addresses, peer discovery, and a path to every other node automatically. Support exists in OpenWrt for turning a router radio into a backhaul link and for creating an open access SSID so a phone or laptop can join without any configuration.
Either way, existing networking software runs over it unchanged — SSH, HTTP servers, file transfer, anything IPv6-native works the same way it would on a local network. Applications written to the FIPS native API skip that layer entirely and address each other by public key, with no IPv6 emulation.
Features
The mesh
- Self-organizing mesh routing. Spanning-tree coordinates with bloom-filter-guided discovery; no global routing tables, no flooding.
- Multi-transport. UDP, TCP, Ethernet, Tor, Nym, and Bluetooth (BLE L2CAP) ship today; transports compose on a single mesh and a node may run several at once.
- Self-assigned cryptographic identity. secp256k1 / schnorr keypairs as node addresses; no registration, no central authority.
- Two-layer encryption. Noise IK between peers (hop-by-hop) and Noise XK between mesh endpoints (independent end-to-end), with periodic rekey for forward secrecy.
- (Optional) Nostr-mediated discovery and NAT traversal. Peers may publish endpoint adverts on public Nostr relays, exchange peering candidates, and establish direct paths through NATs using STUN-assisted hole punching. On the local network, mDNS LAN discovery finds peers directly without relays.
Getting traffic onto it
- IPv6 adapter. A TUN interface maps each remote npub to an
fd00::/8address, so unmodified IPv6 software reaches mesh peers as<npub>.fips. Built-in.fipsDNS resolver, with optional static name mapping via/etc/fips/hosts. - Native datagram API. A local program moves bytes between two
public keys over the mesh, addressing a peer as
npub:portwith no IPv6 emulation and no TUN device in the path.connectandbindtake a key and a port, and from there it is ordinary socket calls. - LAN gateway. Optional
fips-gatewayservice folds an entire unmodified LAN into the mesh: outbound (LAN clients reach mesh destinations through a DNS-allocated virtual IPv6 pool and nftables NAT) and inbound (LAN-side services exposed to the mesh through 1:1 port forwards). - OpenWrt support. FIPS ships as an OpenWrt package. Routers run
802.11s between themselves as a bare L2 link, with FIPS supplying the
encryption, authentication and routing over it. A second helper brings
up an open
!FIPSSSID, the same on every router, which a FIPS client joins over WiFi without configuration.
Running a node
- Operator visibility.
fipsctlCLI for control and inspection with time-series stats history queryable for any metric,fipstopTUI for live status with inline sparkline dashboards, and a JSON-line control socket on each binary for direct programmatic access. - Per-link metrics. RTT, loss, jitter, and goodput on every hop, plus mesh-size estimation, via the Metrics Measurement Protocol.
- ECN congestion signaling. Hop-by-hop CE-flag relay with RFC 3168 IPv6 marking and transport kernel-drop detection.
- Mesh-interface security baseline. Optional default-deny
nftables policy for
fips0shipped as a packaged conffile (/etc/fips/fips.nft) with an operator drop-in directory (/etc/fips/fips.d/) and a disabled-by-defaultfips-firewall.service. The baseline polices only the mesh interface, leaving Docker, Tor, and the host firewall untouched. - Reproducible builds with toolchain pinning and
SOURCE_DATE_EPOCH.
Quick start
Start from a released package. Every packaged platform in the table below gets an installer built and published per release, with checksums, on the releases page. Building from source produces the same artifacts and the same post-install state, so it is the path to take when you want to modify FIPS rather than run it.
On Debian or Ubuntu, download fips_<version>_amd64.deb (or
_arm64.deb) and install it:
sudo dpkg -i fips_<version>_amd64.deb
sudo systemctl start fips fips-dns
This installs the daemon, CLI tools (fipsctl, fipstop), the
fips-dns service that wires .fips name resolution into the host
resolver, the optional fips-gateway service, systemd units, and a
default /etc/fips/fips.yaml you can edit before starting. The package
enables fips and fips-dns but starts neither, which is why the
second command is there.
For macOS, Windows, FreeBSD, OpenWrt, the systemd tarball or a Nix flake, see docs/getting-started.md for the full multi-platform installation guide.
To join a live mesh and reach your first peer, follow the new-user tutorial progression starting at docs/tutorials/join-the-test-mesh.md.
Building from source
To build the Debian package yourself rather than downloading it:
git clone https://github.com/jmcorgan/fips.git
cd fips
cargo install cargo-deb
cargo deb
sudo dpkg -i target/debian/fips_*.deb
For the binaries alone, without an installer:
cargo build --release
Requires Rust 1.94.1+ (edition 2024). Linux, macOS, FreeBSD, and Windows
run as standalone daemons. FreeBSD is packaged for x86_64 only;
no aarch64 FreeBSD artifact is built or tested. Android is supported as
an embedded crate rather than as a standalone daemon: a
compile-gated library surface where the host app owns the TUN (a
VpnService, for example) and reaches the built-in resolver through
Node::dns_local_addr(). There is no Android daemon artifact and no
host-app guide. Transport and feature availability varies by platform.
| Feature | Debian/Ubuntu | Arch | NixOS | macOS | OpenWrt | FreeBSD | Android | Windows |
|---|---|---|---|---|---|---|---|---|
| UDP | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ |
| TCP | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ |
| Tor | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ❌ | ✅ |
| Nym | ✅ | ✅ | ✅ | ✅ | ❌ | ✅ | ❌ | ✅ |
| Ethernet | ✅ | ✅ | ✅ | ✅ | ✅ | ❌ | ❌ | ❌ |
| BLE | ✅ | ✅ | ✅ | ❌ | ❌ | ❌ | ✅ | ❌ |
| Native API | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ❌ | ❌ |
| Package format | .deb |
AUR | flake | .pkg |
.ipk / .apk |
.pkg |
❌ | ZIP |
A column records what builds and runs in a packaged daemon, FreeBSD on
x86_64 only. Native API is the native datagram API, which is off by
default; Windows cannot carry it, because it has no SCM_RIGHTS with
which to pass a descriptor. Package format names the artifact you install,
and a ❌ there means the platform ships none. Windows is the odd one:
its ZIP is an archive you unpack yourself rather than a package an
installer consumes, and there is no MSI.
Five of these columns are Linux: Debian/Ubuntu, Arch, NixOS, OpenWrt
and Android. Linux is not one target. Debian, Ubuntu, Arch and NixOS
are the same glibc build, and what
differs is the packaging: Debian and Ubuntu take the same .deb, Arch
takes fips from the AUR, and NixOS uses the Nix flake described
below. Only the .deb is exercised by an install test, by the
deb-install suite across debian12, debian13, ubuntu22, ubuntu24 and
ubuntu26; neither the AUR package nor the flake is. That suite runs on
every push and pull request, on x86_64, against a .deb built by the same
pinned container as the released one. It does not run at a tag, and the
arm64 package is install-tested by nothing: no workflow installs a published
artifact, so the released packages are checked by
hand. OpenWrt is a musl
target rather than glibc, and it takes an .ipk on 24.x and earlier or
an .apk on 25 and later; both carry the fips-mesh-setup and
fips-ap-setup helpers.
Android records what compiles for aarch64-linux-android under the
CI cross-check and nothing more: no transport in that column is
exercised on a device or an emulator, so read it as "compiles", not
"verified here". Being an embedded crate rather than a daemon platform, it
has nothing to
install, which is what its ❌ package format records. The BLE cell is
narrower still: the transport compiles, but the radio behind it is
supplied by the embedding application rather than by FIPS, and no part
of that path is device-tested.
On Linux, a source build requires libclang — the LAN gateway's
nftables bindings are generated by bindgen at build time, which
needs libclang.so on the build host. Install it before building
(sudo apt install libclang-dev on Debian / Ubuntu); without it the
build fails inside the rustables crate with an "Unable to find
libclang" error. This is a build-time prerequisite only — it is not a
runtime dependency, and the pre-built .deb artifacts do not need it.
BLE compiles on every glibc Linux target and on Android, and is
excluded on musl. On glibc Linux, libdbus is a hard build prerequisite
(sudo apt install libdbus-1-dev pkg-config on Debian / Ubuntu) —
without it the build fails inside libdbus-sys rather than skipping
BLE. The BlueZ daemon itself is a runtime dependency, not a build one.
The OpenWrt ipk is a musl target, so it omits BLE.
Nym (mixnet) transport builds on all desktop platforms. The OpenWrt
❌ is provisional, pending verification of nym-socks5-client
availability on the target; it will flip to ✅ only if confirmed
buildable there.
Alternatively, the repo ships a Nix flake: nix develop
drops you into a shell with the pinned toolchain and every build
prerequisite (libclang, dbus, pkg-config) already provided, and
nix build .#fips builds all four binaries with no host setup. See the
Nix / NixOS section of packaging/README.md.
Documentation
docs/ is organised by reader purpose:
- Tutorials — hand-held walk-throughs from a fresh install through to a participating mesh node, plus advanced deployments (gateway on OpenWrt, hosting services, ground-up two-device mesh).
- How-to guides — operator recipes for specific tasks: firewall activation, Nostr discovery, Tor onion service, Bluetooth peering, 802.11s mesh backhaul and the open access SSID on OpenWrt, LAN gateway deployment and troubleshooting, MTU diagnostics, host aliases, persistent identity, unprivileged-user setup, UDP buffer tuning.
- Reference —
fips.yamlconfiguration, wire formats, control-socket protocol, CLI references for each binary, security posture matrix, Nostr events catalog, transport statistics inventory. - Design — protocol-level architecture and layer specifications. Start with fips-concepts.md for the framing, then fips-architecture.md for the protocol stack.
- Release notes — per-version notes, including v0.5.1.
If you want to contribute, see CONTRIBUTING.md and testing/README.md.
Examples
- examples/sidecar-nostr-relay/ — Run a strfry Nostr relay reachable exclusively over the FIPS mesh. The relay container shares the FIPS sidecar's network namespace and is isolated from the host network.
- examples/sidecar-nostr-mixnet-relay/ — Single-container demo of FIPS peering through a mixnet (implemented with Nym): the FIPS daemon, the mixnet proxy, and a strfry Nostr relay all in one isolated container, with the direct route to the peer firewalled off so traffic provably crosses the mixnet.
- examples/k8s-sidecar/ — Run FIPS as
a Kubernetes Pod sidecar. The sidecar creates
fips0in the Pod's shared network namespace so every other container in the Pod gets mesh access without modification. - examples/wireguard-sidecar-macos/ —
Reach the FIPS mesh from a macOS host through a local Docker
container over a WireGuard tunnel. Only traffic destined for
fd00::/8transits the sidecar; regular internet traffic continues to use the host network.
Project structure
src/ Rust source: library + fips, fipsctl, fipstop, fips-gateway binaries
docs/ Documentation: tutorials, how-to, reference, design
packaging/ Debian, AUR, systemd tarball, OpenWrt ipk/apk,
macOS .pkg, FreeBSD .pkg, Windows ZIP
examples/ Deployment examples (Nostr relay, K8s sidecar, macOS WireGuard)
testing/ Docker-based integration test harnesses + chaos simulation
Status & roadmap
FIPS is at v0.6.0-dev on the master branch.
v0.5.1 is the
current release, a maintenance release on the v0.5.x line that makes the
Linux packages install and run on Debian 12 and Ubuntu 22.04, where every
artifact from v0.3.0 through v0.5.0 installed and then could not start.
v0.5.0 was the last
feature release; this development line continues the testing-and-polishing
track toward v0.6.0. The core protocol works end-to-end over
UDP, TCP, Ethernet, Tor, Nym, and Bluetooth on a global, public test
mesh of thousands of nodes. v0.5.0 added FreeBSD as a packaged platform,
OpenWrt setup helpers for an 802.11s mesh backhaul and an open client
SSID, an Android embedding interface, a native datagram API addressed by
public key, and published node health with a bounded shutdown drain.
New wire-format work continues to be staged on the next branch for the
subsequent release line.
What works today
- Spanning-tree construction with greedy coordinate routing.
- Bloom-filter-guided destination discovery (no flooding, single-path with retry).
- Two-layer Noise encryption (IK at the link, XK at the session) with periodic hitless rekey for forward secrecy at both layers.
- Persistent or ephemeral node identity with key-file management.
- IPv6 TUN adapter with built-in
.fipsDNS resolver and multi-backend auto-configuration (systemd dns-delegate, systemd-resolved, dnsmasq, NetworkManager). - Native datagram API for FIPS-aware applications (npub:port addressing without the IPv6-shim path): off by default, with a surface that may still change.
- Static hostname mapping (
/etc/fips/hosts) with auto-reload. - Per-link metrics (RTT, loss, jitter, goodput) and mesh size estimation.
- ECN congestion signaling (hop-by-hop CE relay, IPv6 CE marking, kernel-drop detection).
- UDP, TCP, Ethernet, Tor, Nym (mixnet), and BLE transports (BLE via L2CAP CoC with per-link MTU negotiation).
- Nostr-mediated overlay endpoint discovery and UDP hole punching for NAT traversal, plus mDNS LAN discovery for local peers.
- LAN gateway (
fips-gateway) with both outbound (LAN-to-mesh) and inbound (mesh-to-LAN port-forwarding) modes. - Peer ACL: per-npub allow / deny admission control at the link
layer; opt-in mesh-firewall baseline at
fips0ingress. - Runtime inspection and peer management via
fipsctl(includingfipsctl probefor reachability diagnosis andfipsctl addressfor mesh-address derivation) andfipstop. - Reproducible builds with toolchain pinning and
SOURCE_DATE_EPOCH. - Node lifecycle and health reporting (
Starting,Running,Degraded,Failed,Draining) with a fatal start when no transport comes up and a bounded shutdown drain window. - OpenWrt setup helpers for an 802.11s mesh between routers
(
fips-mesh-setup) and for the open!FIPSclient SSID (fips-ap-setup). - Linux (Debian, systemd tarball, OpenWrt
.ipkand.apk, AUR), macOS (.pkg), FreeBSD (.pkg, x86_64 only), and Windows (ZIP, service) packaging. - Docker-based integration and chaos testing.
Near-term priorities
- Security audit of the cryptographic protocols.
Longer-term
- Packaged mobile applications: an Android host app, and iOS. The Android embedding interface ships today (see Building from source); what is absent is a packaged app on either platform.
- Bandwidth-aware routing and QoS.
- Protocol stability and a versioned wire format.
- Published crate.
License
MIT — see LICENSE.