mirror of
https://github.com/jmcorgan/fips.git
synced 2026-10-05 19:18:25 +00:00
Johnathan Corgan
3a789370b9
A client process opens a flow to a peer's public key on a chosen port and sends and receives datagrams on a file descriptor the daemon hands it. No IPv6 emulation, no TUN device, no DNS: a datagram travels from key to key. The feature is off by default and is not a stable interface. The wire needs no change and gets none. Every FSP data packet has carried a port pair inside its AEAD envelope since v0.2.0, and port 256 is simply the IPv6 shim. What was missing was a way for a program to ask for a port of its own and be handed the traffic. Addressing is the part worth reading twice, because the obvious design is wrong. The x-only public key is the address. An npub is that key written in bech32, so converting between them is a local encoding rather than a lookup or a name service. The 16-byte node address that travels on the wire is the first half of a SHA-256 of the key: it is a truncated hash, it does not invert, and it appears nowhere a client can see. An earlier iteration of this work reported a peer by that hash and could supply a key only sometimes, which is what treating a wire identifier as an identity produces. An accepted flow therefore always knows its peer. The key is captured where the peer is authenticated rather than looked up when a report is rendered: every inbound datagram passes one call site inside a handler that refuses anything whose session is not established, and the responder has already rejected the session unless the claimed address derives from the key it proved. Reaching for the identity cache instead gives a best-effort answer from a structure that evicts. A listener is a descriptor. The daemon writes one message per arrival to it, carrying the new flow's descriptor and the peer's address, so poll, select and epoll work on a listener and accepting is a recvmsg. That is what lets the API be used from a program that already has an event loop, which a command-and-reply listener could not support: an arrival could not be waited on beside anything else. There is no accept command and no reject command. Refusing a flow is closing the descriptor you were handed. The Rust surface mirrors std::net. FipsStream::connect, FipsListener::bind, incoming, accept, io::Result and an errno mapping rather than a bespoke error type. An address is given as an npub, as a key, or as a pair, through one parameter, the way ToSocketAddrs takes several spellings of one thing. Each type holds its descriptor and copies of what setup told it and nothing else, so a stream that outlives its setup connection is not representable. set_nonblocking, AsFd and the four deadline methods carry the names and signatures std::net uses for the same jobs. They were asked for by a user integrating the API with tokio: AsyncFd requires a non-blocking descriptor, and anything receiving from a peer needs a bounded wait. AsFd is the better of the two descriptor accessors, because the borrow cannot outlive the value that owns the descriptor, so a reactor cannot hold a registration for a descriptor that has since been closed and its number reused by the next open. The non-blocking flag is read, modified and written back rather than assigned, since the flag word carries more than that one bit and a caller may have set O_ASYNC. A zero timeout is refused with EINVAL, because the kernel reads a zero timeval as "wait for ever", which inverts what a caller passing zero means; std::net refuses it for the same reason. The two directions are separate options and stay that way. FipsListener gets no timeout methods, matching TcpListener: bounding an accept is set_nonblocking plus the caller's own poll, which the reactor how-to builds. A flow taken from accept is blocking whatever the listener was set to, because the two are separate sockets and the daemon hands over a fresh one. One rule has no counterpart in Berkeley sockets and a client author must know it: the v1 wire carries no half-close, so nothing peer-driven ever closes a flow. A server written to read until the flow ends waits for a signal that cannot arrive, holding a thread and a flow per peer until its process exits. A program decides its own termination, and the example serves one datagram per flow. The tests reach a live daemon rather than a stand-in. Every public item had a unit test against a hand-written stand-in with canned replies, and the five entry points a program actually calls first, connect, connect_from, connect_at, bind and the SOCKET constant, had no coverage of any kind, because the tests that appear to cover them build a Wire over a socket pair and hand it to the private open and hold, so nothing ever resolved a socket path or mapped its errors. examples/native-surface.rs walks all thirty-eight items against a running daemon and reports the number of assertions it made. The count is read from the recorder rather than written as a literal, and the harness asserts the exit status, the completion marker and the count together, so deleting an assertion fails the check rather than quietly shrinking it. Watchdogs turn a hang into a named failure, which several of the walked behaviours would otherwise produce. The shared Docker image is built once for every integration leg, so the new binary is staged at all ten places the existing one is, the interop builder included, which gets a stub because those images exercise the wire between daemon versions and older refs do not carry the example. The platform gating was tested rather than reasoned about: flipping all eleven gates so the native API is excluded leaves the crate compiling clean across the workspace, every target and the profiling feature. The shipped docs tree gains what only the LaTeX manual under design/ had, which is not published with the daemon. A reference entry covers the whole surface: addressing and the port tiers, the Berkeley mapping, every method on FipsAddr, FipsStream, FipsListener and Incoming, the errno table, the ceilings, the four places data disappears with nothing reported, the line protocol and the command reference. The errno table gives names rather than numbers, since the client maps each name onto the libc constant for the platform it was built for and the supported platforms disagree on the numbers. A tutorial side trip stands up two throwaway nodes on one machine, peered over loopback UDP with no TUN and no DNS, then writes a listening program and a connecting program against them; it needs neither the public mesh nor root, because the native path is the one that does not go through the IPv6 adapter. The obligations a client in another language carries are a how-to of their own, since they are a task rather than a description: reading the setup connection with recvmsg, associating a descriptor with the last complete line, telling an empty datagram from a close, and six others. Serving many peers from one poll loop is another, with the whole program, because the straightforward listener spawns a thread per flow and that is wrong at the node's ceiling of 256. The drop causes are a table mapping each of the seven texts DropReason::as_str produces to the counter it increments, with drop_oversize called out as the ninth counter that is not in the table. What a daemon restart costs is a section of its own: every flow and listener ends, descriptors do not survive, there is no resumption, and datagrams sent but not yet forwarded are lost through a window nothing bounds. A stack comparison diagram places the interface against the stack a reader already knows: the same application over HTTP, TLS, TCP, IP and Ethernet on one side, and over its own format, FSP, FMP and a FIPS transport on the other, aligned so each row is one concern. The two columns are not alternatives and are not drawn as such. An unmodified IPv6 program's packets reach fips0, and the adapter hands each one to FSP as a payload, so the left stack runs inside the right one; the left column ends at a fork, eth0 for the ordinary internet and fips0 for the mesh, and an arrow leaves fips0 and runs back up into FSP's input. The row where TCP would be is empty on purpose and names Reliable Object Delivery, which is where that capability is expected to land. ROD is a v2 capability, the box is dashed because none of it exists yet, and the design entry says the part a reader needs most: nothing on the surface anticipates it, so a program written today should assume it does not exist. Both endpoints carry a scheme and a worked port, https://<npub>.fips:443 and fips://<npub>:443, with a footnote saying the two ports are not the same kind of thing, a TCP port inside the tunnel on the left and an FSP port on the right. The fips:// form is a coinage: nothing in the tree parses it, nothing registers the scheme, and the API takes a key and a port as separate arguments rather than a URL. The diagram also says where the right column stops, since FIPS over UDP still rides IP and Ethernet beneath. It appears in fips-concepts.md and fips-ipv6-adapter.md, which were making its argument in prose without a picture, and deliberately not in fips-architecture.md, which already carries the OSI mapping and makes the same point about the transport row. The gateway's control socket moves onto the same bind policy this API uses, which is the one change here that touches deployed behaviour: fips-gateway now tightens /run/fips to 0750. That is unreachable under the packaged deployment, where fips.service has already created the directory at that mode, and reachable for a source build or a container that starts the gateway alone. One changelog entry under Added, describing the released state: what a client opens and reads, the addressing and why the node address is not it, the listener being a descriptor, the std::net shape of the Rust surface, and the one rule Berkeley sockets have no counterpart for. It says in as many words that the wire is unchanged.
274 lines
12 KiB
Markdown
274 lines
12 KiB
Markdown
# FIPS Architecture
|
|
|
|
The protocol architecture, identity system, and two-layer encryption
|
|
model. For the higher-level "what is FIPS and why" framing, see
|
|
[fips-concepts.md](fips-concepts.md). For prior art and academic
|
|
citations, see [fips-prior-work.md](fips-prior-work.md).
|
|
|
|
## Protocol Architecture
|
|
|
|
FIPS is organized in three protocol layers, each with distinct
|
|
responsibilities and clean service boundaries. No layer depends on
|
|
the specifics of the layers above or below it — transport plugins
|
|
know nothing about sessions, the routing layer knows nothing about
|
|
application addressing, and applications know nothing about which
|
|
physical media carry their traffic. This separation means new
|
|
transports, protocol features, and application interfaces can be
|
|
added independently.
|
|
|
|
|
|
|
|
### Mapping to Traditional Networking
|
|
|
|
Readers familiar with the OSI model or TCP/IP networking may find it
|
|
helpful to see how FIPS concepts relate to traditional layers:
|
|
|
|
|
|
|
|
Note that FMP spans what would traditionally be separate link and
|
|
network layers. This is intentional — in a self-organizing mesh, the
|
|
same layer that authenticates peers also makes routing decisions,
|
|
because routing depends on authenticated peer state (spanning tree
|
|
positions, bloom filters).
|
|
|
|
### Layer Responsibilities
|
|
|
|
**Transport layer**: Delivers datagrams between endpoints over a
|
|
specific medium. Each transport type (UDP socket, Ethernet interface,
|
|
radio modem) implements the same abstract interface: send and receive
|
|
datagrams, report MTU. The transport layer knows nothing about FIPS
|
|
identities, routing, or encryption. It provides raw datagram delivery
|
|
to FMP above.
|
|
|
|
See [fips-transport-layer.md](fips-transport-layer.md) for the
|
|
transport layer specification.
|
|
|
|
**FIPS Mesh Protocol (FMP)**: Manages peer connections, authenticates
|
|
peers via Noise IK handshakes, and encrypts all traffic on each link.
|
|
FMP is where the mesh organizes itself — nodes exchange spanning tree
|
|
announcements and bloom filters with their direct peers, and FMP
|
|
makes forwarding decisions for transit traffic. FMP provides
|
|
authenticated, encrypted forwarding to FSP above.
|
|
|
|
See [fips-mesh-layer.md](fips-mesh-layer.md) for the FMP specification
|
|
and [fips-mesh-operation.md](fips-mesh-operation.md) for how FMP's
|
|
routing and self-organization work in practice.
|
|
|
|
**FIPS Session Protocol (FSP)**: Provides end-to-end authenticated
|
|
encryption between any two nodes, regardless of how many intermediate
|
|
hops separate them. FSP manages session lifecycle (setup, data
|
|
transfer, teardown), caches destination coordinates for efficient
|
|
routing, and handles the warmup strategy that keeps transit node
|
|
caches populated. Session dispatch uses index-based routing inspired
|
|
by [WireGuard](https://www.wireguard.com/), enabling O(1) packet
|
|
demultiplexing. FSP provides a datagram service to applications above.
|
|
|
|
See [fips-session-layer.md](fips-session-layer.md) for the FSP
|
|
specification.
|
|
|
|
**IPv6 adaptation layer**: Sits above FSP as a service on port 256,
|
|
adapting the FIPS datagram service for unmodified IPv6 applications.
|
|
Provides DNS resolution (npub → fd00::/8 address), identity cache
|
|
management, IPv6 header compression, MTU enforcement, and a TUN
|
|
interface. This is the primary way existing applications use the FIPS
|
|
mesh.
|
|
|
|
See [fips-ipv6-adapter.md](fips-ipv6-adapter.md) for the IPv6 adapter.
|
|
|
|
### Node Architecture
|
|
|
|
Application services sit at the top of the stack, dispatched by FSP
|
|
port number: the IPv6 TUN adapter (port 256) maps npubs to `fd00::/8`
|
|
addresses with header compression so unmodified IP applications can
|
|
use the network transparently, while the native datagram API
|
|
addresses destinations directly by npub.
|
|
|
|
The native datagram API is **experimental** and off by default. It is
|
|
not a stable API surface and carries no compatibility promise. See
|
|
[../how-to/use-the-native-datagram-api.md](../how-to/use-the-native-datagram-api.md)
|
|
for enabling it and writing against it, and
|
|
[../reference/security.md](../reference/security.md#native-datagram-api)
|
|
for what enabling it grants to the `fips` group.
|
|
|
|
|
|
|
|
The mesh routes application traffic across heterogeneous transports
|
|
transparently. A packet may traverse WiFi, Ethernet, UDP/IP, and Tor
|
|
links on its way from source to destination — the application never
|
|
needs to know which transports are involved. Each hop is independently
|
|
encrypted at the link layer, while a single end-to-end session
|
|
protects the payload across the entire path.
|
|
|
|
|
|
|
|
|
|
|
|
## Identity System
|
|
|
|
FIPS uses [Nostr](https://github.com/nostr-protocol/nips) keypairs
|
|
(secp256k1) as node identities. The public key identifies the node;
|
|
the private key signs protocol messages and establishes encrypted
|
|
sessions.
|
|
|
|
The public key (or its bech32-encoded npub form) is the primary means
|
|
for application-layer software to identify communication endpoints.
|
|
Internally, the protocol derives a `node_addr` (a 16-byte SHA-256 hash
|
|
of the pubkey) used as the routing identifier in packet headers, and
|
|
an IPv6 address derived from the node_addr for the TUN adapter.
|
|
Applications use the pubkey or npub; the routing layer uses node_addr;
|
|
unmodified IPv6 applications use the derived `fd00::/8` address. All
|
|
three are deterministically derived from the same keypair.
|
|
|
|
### FIPS Identity Handling
|
|
|
|
|
|
|
|
The pubkey is the node's cryptographic identity, used in Noise
|
|
handshakes for both link encryption (IK) and session encryption (XK).
|
|
It is never exposed beyond the endpoints of an encrypted channel. The node_addr, a one-way
|
|
SHA-256 hash truncated to 16 bytes, serves as the routing identifier
|
|
in packet headers and bloom filters. Intermediate routers see only
|
|
node_addrs — they can forward traffic without learning the Nostr
|
|
identities of the endpoints. An observer can verify "does this
|
|
node_addr belong to pubkey X?" if they already know the pubkey, but
|
|
cannot enumerate communicating identities by inspecting traffic. The
|
|
IPv6 address prepends `fd` to the first 15 bytes of the node_addr,
|
|
providing a ULA overlay address for unmodified IP applications via the
|
|
TUN interface.
|
|
|
|
Below the FIPS identity layer, each transport uses its own native
|
|
addressing — IP:port or hostname:port addresses, MAC addresses,
|
|
.onion identifiers. These **link addresses** are opaque to everything
|
|
above FMP and discarded once link authentication completes.
|
|
|
|
### Identity Verification
|
|
|
|
The Noise Protocol Framework mutually authenticates both peer-to-peer
|
|
link connections (at FMP) and end-to-end session traffic (at FSP),
|
|
proving each party controls the private key for their claimed
|
|
identity.
|
|
|
|
See [fips-mesh-layer.md](fips-mesh-layer.md) for peer authentication
|
|
and [fips-session-layer.md](fips-session-layer.md) for end-to-end
|
|
session establishment.
|
|
|
|
Key rotation changes the node's identity — a new keypair produces a
|
|
new node_addr and IPv6 address, requiring all sessions to be
|
|
re-established. Migration mechanisms that allow a node to announce a
|
|
successor key are a future consideration.
|
|
|
|
## Two-Layer Encryption
|
|
|
|
FIPS uses independent encryption at two protocol layers:
|
|
|
|
| Layer | Scope | Pattern | Purpose |
|
|
| ----- | ----- | ------- | ------- |
|
|
| **FMP (Mesh)** | Hop-by-hop | Noise IK | Encrypt all traffic on each peer link |
|
|
| **FSP (Session)** | End-to-end | Noise XK | Encrypt application payload between endpoints |
|
|
|
|
### Link Layer (Hop-by-Hop)
|
|
|
|
When two nodes establish a direct connection, they perform a [Noise
|
|
IK](https://noiseprotocol.org/) handshake. This authenticates both
|
|
parties and establishes symmetric keys for encrypting all traffic on
|
|
that link. Every packet between direct peers is encrypted — gossip
|
|
messages, routing queries, and forwarded session datagrams alike.
|
|
|
|
The IK pattern is used because outbound connections know the peer's
|
|
npub from configuration, while inbound connections learn the
|
|
initiator's identity from the first handshake message.
|
|
|
|
### Session Layer (End-to-End)
|
|
|
|
FIPS establishes end-to-end encrypted sessions between any two
|
|
communicating nodes using Noise XK, regardless of how many hops
|
|
separate them. The initiator knows the destination's npub (required
|
|
for XK's pre-message); the responder learns the initiator's identity
|
|
from the third handshake message. Unlike the link-layer IK pattern
|
|
where the initiator's identity is revealed in msg1, XK delays
|
|
identity disclosure until msg3, providing stronger initiator identity
|
|
protection for traffic traversing untrusted intermediate nodes.
|
|
|
|
A packet from A to D through intermediate nodes B and C:
|
|
|
|
1. A encrypts payload with A↔D session key (FSP)
|
|
2. A wraps in SessionDatagram, encrypts with A↔B link key (FMP),
|
|
sends to B
|
|
3. B decrypts link layer, reads destination node_addr, re-encrypts
|
|
with B↔C link key, forwards to C
|
|
4. C decrypts link layer, re-encrypts with C↔D link key, forwards
|
|
to D
|
|
5. D decrypts link layer, then decrypts session layer to get payload
|
|
|
|
Intermediate nodes route based on destination node_addr but cannot
|
|
read session-layer payloads. Each hop strips one link encryption and
|
|
applies the next — the session-layer ciphertext passes through
|
|
untouched.
|
|
|
|
Both layers always apply, even between adjacent peers — a packet to a
|
|
direct neighbor is still encrypted twice. This uniform model means no
|
|
special cases for local vs remote destinations, and topology changes
|
|
(a direct peer becomes reachable only through intermediaries) don't
|
|
affect existing sessions.
|
|
|
|
See [fips-mesh-layer.md](fips-mesh-layer.md) for link encryption and
|
|
[fips-session-layer.md](fips-session-layer.md) for session encryption.
|
|
|
|
## Routing and Mesh Operation
|
|
|
|
Forwarding decisions are local. Each node combines spanning-tree
|
|
coordinates with peer bloom filters to choose a next hop, falling back
|
|
to greedy tree routing when bloom filters have not converged. Discovery
|
|
warms transit node caches with destination coordinates, and three
|
|
explicit error signals (CoordsRequired, PathBroken, MtuExceeded) drive
|
|
recovery when forwarding fails. The full routing decision process,
|
|
discovery protocol, and error-recovery integration view live in
|
|
[fips-mesh-operation.md](fips-mesh-operation.md).
|
|
|
|
## Transport Abstraction
|
|
|
|
FIPS treats the communication medium as a pluggable component. UDP,
|
|
TCP, raw Ethernet, Tor, BLE, and Nym all implement the same small
|
|
datagram interface (send, receive, report MTU) and feed peers into a
|
|
single FMP routing layer; radio and serial transports are in the
|
|
planned set. Nym (an outbound-only mixnet transport) and Tor are
|
|
privacy-oriented deployment modes rather than failover paths.
|
|
Multi-transport nodes bridge between networks transparently. The
|
|
transport-layer specification — including per-transport categories,
|
|
the trait surface, the connection model, and implementation status —
|
|
is in [fips-transport-layer.md](fips-transport-layer.md).
|
|
|
|
## Security
|
|
|
|
FIPS defends against four adversary classes (transport observers,
|
|
active transport attackers, intermediate routers, and adversarial
|
|
mesh nodes) through layered controls: hop-by-hop FMP link encryption,
|
|
end-to-end FSP session encryption with stronger initiator identity
|
|
protection, signed and replay-protected gossip, and rate-limited
|
|
handshake processing. The threat-model details and per-layer
|
|
mitigations are in [fips-mesh-layer.md](fips-mesh-layer.md), and the
|
|
operator-facing controls (default-deny baseline, peer ACLs,
|
|
filesystem permissions, cryptographic primitives) are consolidated in
|
|
[fips-security.md](fips-security.md) and
|
|
[../reference/security.md](../reference/security.md).
|
|
|
|
## MTU as a Cross-Cutting Concern
|
|
|
|
MTU is not owned by any single layer. The transport layer reports
|
|
per-link MTU, FMP carries `path_mtu` in SessionDatagram and
|
|
LookupResponse to track the minimum along a path, FSP echoes the
|
|
observed forward-path MTU back to the source, and the IPv6 adapter
|
|
enforces the resulting effective MTU at the TUN with ICMP Packet Too
|
|
Big and TCP MSS clamping. The unified design — encapsulation overhead
|
|
budget, proactive PMTUD, reactive MtuExceeded, and per-destination
|
|
storage — is in [fips-mtu.md](fips-mtu.md).
|
|
|
|
## Approaches Considered but Rejected
|
|
|
|
One design alternative evaluated and ruled out during the architecture
|
|
pass was onion routing, rejected because it requires the sender to
|
|
know the full path upfront (incompatible with self-organizing
|
|
routing) and prevents per-hop error feedback (incompatible with
|
|
CoordsRequired/PathBroken recovery). The canonical mention lives in
|
|
[fips-mesh-operation.md](fips-mesh-operation.md#privacy-considerations).
|