mirror of https://github.com/jmcorgan/fips.git synced 2026-10-05 19:18:25 +00:00

Files

T

Johnathan Corgan 3a789370b9 Add an experimental native datagram API addressed by public key

A client process opens a flow to a peer's public key on a chosen port and
sends and receives datagrams on a file descriptor the daemon hands it. No
IPv6 emulation, no TUN device, no DNS: a datagram travels from key to key.
The feature is off by default and is not a stable interface.

The wire needs no change and gets none. Every FSP data packet has carried a
port pair inside its AEAD envelope since v0.2.0, and port 256 is simply the
IPv6 shim. What was missing was a way for a program to ask for a port of its
own and be handed the traffic.

Addressing is the part worth reading twice, because the obvious design is
wrong. The x-only public key is the address. An npub is that key written in
bech32, so converting between them is a local encoding rather than a lookup
or a name service. The 16-byte node address that travels on the wire is the
first half of a SHA-256 of the key: it is a truncated hash, it does not
invert, and it appears nowhere a client can see. An earlier iteration of this
work reported a peer by that hash and could supply a key only sometimes,
which is what treating a wire identifier as an identity produces.

An accepted flow therefore always knows its peer. The key is captured where
the peer is authenticated rather than looked up when a report is rendered:
every inbound datagram passes one call site inside a handler that refuses
anything whose session is not established, and the responder has already
rejected the session unless the claimed address derives from the key it
proved. Reaching for the identity cache instead gives a best-effort answer
from a structure that evicts.

A listener is a descriptor. The daemon writes one message per arrival to it,
carrying the new flow's descriptor and the peer's address, so poll, select
and epoll work on a listener and accepting is a recvmsg. That is what lets
the API be used from a program that already has an event loop, which a
command-and-reply listener could not support: an arrival could not be waited
on beside anything else. There is no accept command and no reject command.
Refusing a flow is closing the descriptor you were handed.

The Rust surface mirrors std::net. FipsStream::connect, FipsListener::bind,
incoming, accept, io::Result and an errno mapping rather than a bespoke
error type. An address is given as an npub, as a key, or as a pair, through
one parameter, the way ToSocketAddrs takes several spellings of one thing.
Each type holds its descriptor and copies of what setup told it and nothing
else, so a stream that outlives its setup connection is not representable.

set_nonblocking, AsFd and the four deadline methods carry the names and
signatures std::net uses for the same jobs. They were asked for by a user
integrating the API with tokio: AsyncFd requires a non-blocking descriptor,
and anything receiving from a peer needs a bounded wait. AsFd is the better
of the two descriptor accessors, because the borrow cannot outlive the value
that owns the descriptor, so a reactor cannot hold a registration for a
descriptor that has since been closed and its number reused by the next
open. The non-blocking flag is read, modified and written back rather than
assigned, since the flag word carries more than that one bit and a caller may
have set O_ASYNC. A zero timeout is refused with EINVAL, because the kernel
reads a zero timeval as "wait for ever", which inverts what a caller passing
zero means; std::net refuses it for the same reason. The two directions are
separate options and stay that way. FipsListener gets no timeout methods,
matching TcpListener: bounding an accept is set_nonblocking plus the caller's
own poll, which the reactor how-to builds. A flow taken from accept is
blocking whatever the listener was set to, because the two are separate
sockets and the daemon hands over a fresh one.

One rule has no counterpart in Berkeley sockets and a client author must know
it: the v1 wire carries no half-close, so nothing peer-driven ever closes a
flow. A server written to read until the flow ends waits for a signal that
cannot arrive, holding a thread and a flow per peer until its process exits.
A program decides its own termination, and the example serves one datagram
per flow.

The tests reach a live daemon rather than a stand-in. Every public item had a
unit test against a hand-written stand-in with canned replies, and the five
entry points a program actually calls first, connect, connect_from,
connect_at, bind and the SOCKET constant, had no coverage of any kind,
because the tests that appear to cover them build a Wire over a socket pair
and hand it to the private open and hold, so nothing ever resolved a socket
path or mapped its errors. examples/native-surface.rs walks all thirty-eight
items against a running daemon and reports the number of assertions it made.
The count is read from the recorder rather than written as a literal, and the
harness asserts the exit status, the completion marker and the count
together, so deleting an assertion fails the check rather than quietly
shrinking it. Watchdogs turn a hang into a named failure, which several of
the walked behaviours would otherwise produce. The shared Docker image is
built once for every integration leg, so the new binary is staged at all ten
places the existing one is, the interop builder included, which gets a stub
because those images exercise the wire between daemon versions and older refs
do not carry the example. The platform gating was tested rather than reasoned
about: flipping all eleven gates so the native API is excluded leaves the
crate compiling clean across the workspace, every target and the profiling
feature.

The shipped docs tree gains what only the LaTeX manual under design/ had,
which is not published with the daemon. A reference entry covers the whole
surface: addressing and the port tiers, the Berkeley mapping, every method on
FipsAddr, FipsStream, FipsListener and Incoming, the errno table, the
ceilings, the four places data disappears with nothing reported, the line
protocol and the command reference. The errno table gives names rather than
numbers, since the client maps each name onto the libc constant for the
platform it was built for and the supported platforms disagree on the
numbers. A tutorial side trip stands up two throwaway nodes on one machine,
peered over loopback UDP with no TUN and no DNS, then writes a listening
program and a connecting program against them; it needs neither the public
mesh nor root, because the native path is the one that does not go through
the IPv6 adapter. The obligations a client in another language carries are a
how-to of their own, since they are a task rather than a description:
reading the setup connection with recvmsg, associating a descriptor with the
last complete line, telling an empty datagram from a close, and six others.
Serving many peers from one poll loop is another, with the whole program,
because the straightforward listener spawns a thread per flow and that is
wrong at the node's ceiling of 256. The drop causes are a table mapping each
of the seven texts DropReason::as_str produces to the counter it increments,
with drop_oversize called out as the ninth counter that is not in the table.
What a daemon restart costs is a section of its own: every flow and listener
ends, descriptors do not survive, there is no resumption, and datagrams sent
but not yet forwarded are lost through a window nothing bounds.

A stack comparison diagram places the interface against the stack a reader
already knows: the same application over HTTP, TLS, TCP, IP and Ethernet on
one side, and over its own format, FSP, FMP and a FIPS transport on the
other, aligned so each row is one concern. The two columns are not
alternatives and are not drawn as such. An unmodified IPv6 program's packets
reach fips0, and the adapter hands each one to FSP as a payload, so the left
stack runs inside the right one; the left column ends at a fork, eth0 for the
ordinary internet and fips0 for the mesh, and an arrow leaves fips0 and runs
back up into FSP's input. The row where TCP would be is empty on purpose and
names Reliable Object Delivery, which is where that capability is expected to
land. ROD is a v2 capability, the box is dashed because none of it exists
yet, and the design entry says the part a reader needs most: nothing on the
surface anticipates it, so a program written today should assume it does not
exist. Both endpoints carry a scheme and a worked port,
https://<npub>.fips:443 and fips://<npub>:443, with a footnote saying the two
ports are not the same kind of thing, a TCP port inside the tunnel on the
left and an FSP port on the right. The fips:// form is a coinage: nothing in
the tree parses it, nothing registers the scheme, and the API takes a key and
a port as separate arguments rather than a URL. The diagram also says where
the right column stops, since FIPS over UDP still rides IP and Ethernet
beneath. It appears in fips-concepts.md and fips-ipv6-adapter.md, which were
making its argument in prose without a picture, and deliberately not in
fips-architecture.md, which already carries the OSI mapping and makes the
same point about the transport row.

The gateway's control socket moves onto the same bind policy this API uses,
which is the one change here that touches deployed behaviour: fips-gateway
now tightens /run/fips to 0750. That is unreachable under the packaged
deployment, where fips.service has already created the directory at that
mode, and reachable for a source build or a container that starts the gateway
alone.

One changelog entry under Added, describing the released state: what a
client opens and reads, the addressing and why the node address is not it,
the listener being a descriptor, the std::net shape of the Rust surface,
and the one rule Berkeley sockets have no counterpart for. It says in as
many words that the wire is unchanged.

2026-08-21 05:48:23 +00:00

16 KiB

Raw Blame History

Security Reference

Consolidated security reference covering the nftables baseline, peer ACL file format, cryptographic primitives, rekey defaults, replay window, filesystem permissions, threat-resistance matrix, and default network exposures per transport. For the threat-model design and rationale, see ../design/fips-security.md. For the operator activation steps and drop-in recipes, see ../how-to/enable-mesh-firewall.md.

nftables Baseline

The shipped baseline is /etc/fips/fips.nft. It defines a single nftables table inet fips with one chain hooked at input, structured as follows:

Step	Rule	Effect
1	`iifname != "fips0" return`	Match only traffic arriving on `fips0`; everything else short-circuits.
2	`ct state established,related accept`	Allow conntrack replies and related ICMPv6 errors.
3	`icmpv6 type echo-request accept`	Allow IPv6 echo (ping6 reachability).
4	`include "/etc/fips/fips.d/*.nft"`	Splice in operator drop-ins (empty matches nothing).
5	`counter drop`	Default-deny everything else; counter increments on every drop.

Outbound from fips0 is unrestricted. The baseline is a documented dpkg conffile — operator edits to /etc/fips/fips.nft are preserved across upgrades.

The systemd unit is fips-firewall.service (oneshot). It is not enabled by default; activation is an explicit operator gesture documented in ../how-to/enable-mesh-firewall.md.

Drop-In File Format

Operator extensions live under /etc/fips/fips.d/ with the .nft suffix. Each file is included inline into the inbound chain at the marked point and may contain any nftables rule lines valid in that context.

Naming convention: <purpose>-from-<source>.nft keeps drop-ins easy to scan. Examples shipped in the design discussion:

ssh-from-bastion.nft — accept TCP/22 from a single mesh-node address
http-from-cluster.nft — accept TCP/80 from a /64 mesh-address prefix
dns-public.nft — accept UDP/53 and TCP/53 from any mesh node
git-from-trusted.nft — accept TCP/9418 from a set of mesh-node addresses

After editing, reload via sudo systemctl reload-or-restart fips-firewall.service (or equivalently sudo nft -f /etc/fips/fips.nft since the file is idempotent).

Cryptographic Primitives

Component	Choice	Where Used
Curve	secp256k1	FMP IK, FSP XK, Schnorr signatures
Diffie-Hellman	ECDH on secp256k1 (x-only normalized)	Noise IK, Noise XK
AEAD	ChaCha20-Poly1305	FMP link encryption, FSP session encryption
Hash	SHA-256	NodeAddr derivation, Noise key schedule
Key derivation	HKDF-SHA256	Noise key schedule
Signatures	secp256k1 Schnorr	TreeAnnounce, LookupResponse proof, Nostr adverts
Noise pattern (link)	`Noise_IK_secp256k1_ChaChaPoly_SHA256`, with the deviation below	FMP link layer (IK with epoch payload)
Noise pattern (session)	`Noise_XK_secp256k1_ChaChaPoly_SHA256`, with the deviation below	FSP session layer (XK with epoch payload)

These choices align with the Nostr cryptographic stack (secp256k1 + ChaCha20-Poly1305 + SHA-256) and the NIP-44 encrypted messaging standard.

Deviation: Empty Associated Data in the Handshake AEAD

Both Noise patterns above deviate from the standard construction in one respect. The handshake AEAD uses an empty associated-data field where standard Noise EncryptAndHash uses the handshake hash h.

The choice was deliberate. Using secp256k1 rather than 25519 already put the construction outside standard Noise, so no standard-Noise peer could be confused with it, and the transcript hash bought no distinguishing value.

That argument is about domain separation, and on those grounds it holds. It does not cover transcript binding, which is the property actually absent. Domain separation and DH binding survive through the chaining key ck, which mix_key chains from ck = h, seeded from the protocol name in SymmetricState::initialize (src/noise/handshake.rs). The handshake hash h is maintained at every step and is never fed to the AEAD, so it binds nothing.

Rekey Defaults

Both link-layer and session-layer Noise sessions rekey under one of two triggers, configurable under node.rekey.*:

Parameter	Default	Description
`enabled`	`true`	Master switch.
`after_secs`	`120`	Time-based rekey threshold.
`after_messages`	`65536`	Message-count rekey threshold.

In addition to the configurable triggers, the daemon retains the old session keys for a fixed 10-second drain window after each cutover (compile-time constant DRAIN_WINDOW_SECS in src/node/handlers/rekey.rs). Rekey rotates the Noise key schedule and the session indices; old session keys are kept in previous_session for the drain window so in-flight packets encrypted under the old keys still decrypt.

Replay Window

Both layers use explicit per-packet counters with a sliding bitmap window for replay protection. The bitmap is 2048 entries at both layers — large enough to accommodate UDP reordering and packet loss without false-positive replay rejection. Counters older than the window are rejected. The same ReplayWindow and decrypt_with_replay_check() implementation is used at both the FMP and FSP layers.

Peer ACL

Mesh-level ACL files at /etc/fips/peers.allow and /etc/fips/peers.deny give the operator allowlist/blocklist control over which npubs may complete the FMP Noise IK link handshake.

File format:

One entry per line. An entry is either a bech32 npub1..., an alias defined in /etc/fips/hosts, or the literal ALL wildcard (case-insensitive).
Lines beginning with # are comments.
Blank lines are ignored.

Evaluation order (first match wins, default-allow on no match):

peers.allow — if the peer matches an entry here (or ALL is in peers.allow), the handshake is admitted, regardless of any peers.deny entry.
peers.deny — if the peer matches an entry here (or ALL is in peers.deny), the handshake is refused.
Otherwise the peer is admitted.

peers.allow is not an exclusive gate on its own: an unlisted peer falls through to step 3 and is admitted unless it appears in peers.deny. To turn peers.allow into a strict allowlist, place ALL in peers.deny so every unlisted peer is rejected at step 2.

The ALL wildcard makes the operator's posture explicit:

ALL in peers.allow admits every peer (same effect as the default-allow behavior, but documented in the file).
ALL in peers.deny blocks every peer except those listed in peers.allow — the "allowlist-strict" posture.

In practice this collapses to a few common postures:

Default-allow with denylist: leave peers.allow empty; populate peers.deny. All npubs may peer except those listed.
Allowlist-strict: populate peers.allow and put ALL in peers.deny. Only the listed npubs may peer; everyone else is rejected at step 2.

A populated peers.allow with an empty peers.deny is not a strict allowlist — it is equivalent to default-allow plus an explicit "always-admit" set. The strict variant requires ALL in peers.deny.

Aliases are resolved through /etc/fips/hosts at file-load time. If peers.allow lists core-vm and /etc/fips/hosts maps core-vm to a specific npub, that npub is admitted. If core-vm is later remapped to a different npub, the ACL re-resolves on the next mtime change. Operators should be aware that ACL semantics follow the hosts-file aliasing, not just the literal npubs visible in the file.

Both files are reloaded automatically when their mtime changes — no daemon restart or signal is needed. ACL evaluation runs after msg1 decryption but before any further peer-state mutation; rate-limited msg1s never reach the ACL.

Filesystem Permissions

Path	Owner	Mode	Purpose
`/etc/fips/fips.key`	root:root	`0600`	Persistent identity private key (sensitive).
`/etc/fips/fips.pub`	root:root	`0644`	Public key (npub).
`/etc/fips/fips.yaml`	root:root	`0644`	Daemon configuration (dpkg conffile).
`/etc/fips/fips.nft`	root:root	`0644`	nftables baseline (dpkg conffile).
`/etc/fips/fips.d/`	root:root	`0755`	Operator drop-in directory.
`/etc/fips/hosts`	root:root	`0644`	Optional hostname → npub map (dpkg conffile).
`/etc/fips/peers.allow`	root:root	`0644`	Optional peer allowlist.
`/etc/fips/peers.deny`	root:root	`0644`	Optional peer denylist.
`/run/fips/control.sock`	root:fips	`0770`	Control socket (members of `fips` group can use `fipsctl`).
`/run/fips/api.sock`	root:fips	`0770`	Native datagram API socket, when `node.native_api.enabled` is set (experimental; absent otherwise).
`/run/fips/`	root:fips	`0750`	Socket parent directory.

Adding a user to the fips group grants fipsctl access without requiring root. The daemon chowns the control socket and its parent directory at bind time, and does the same for the native API socket when that is enabled.

Native Datagram API

Experimental. Disabled by default (node.native_api.enabled, default false), and built on Linux and FreeBSD only. It is not a stable API surface, not a reliability layer, and not the v2 external process API. No compatibility promise is made about it.

Any user in the fips group can impersonate the node on the mesh. The API socket is created at mode 0770 owned by group fips, and that is the entire authorization model. A process that can open it can:

send datagrams under this node's identity to any peer it names, which peers authenticate as coming from this node;
hold any port from 1024 upward and receive mesh traffic addressed to this node on it, including traffic another local program expected;
do both without authenticating, without a capability check, and without any record beyond the daemon's own logs.

Group membership is therefore equivalent to possession of the node's identity for the purpose of sending on the mesh. On a node with the native API enabled, treat membership of the fips group exactly as you would treat /etc/fips/fips.key. Grant it to the accounts that are trusted to speak as the node and to no others, and review it before enabling the API on a shared machine.

The file descriptor carries the grant, not the connection. A setup call hands the client a socket descriptor and the connection it was made on is then closed; the flow or the held port lives until that descriptor is closed. A descriptor is an ordinary kernel object, so it survives fork, survives exec unless the client asked for it close-on-exec when it received it, and can be handed to another process over SCM_RIGHTS. A process holding one can send as this node on that flow, or receive on that port, without ever opening the API socket and without being in the fips group. Nothing revokes a descriptor already handed out. Restarting the daemon closes its own halves and ends every flow and listener at once, and that is the only revocation there is.

Two consequences follow for fipsctl access. First, the fips group is already the control-socket group, so enabling the native API silently upgrades every existing fipsctl user from "can read node state and manage peers" to "can send as the node". Second, an operator who wants the two audiences separated must not enable the API on a node whose fips group has been handed out for monitoring.

node.native_api.debug_commands (default false) is a second, independent gate. It admits three commands (inject, stats, arrive) that exist for the test harness: arrive makes the daemon dispatch a datagram as though a peer had sent it, reaching any listener on this node under any peer identity the caller names. Leave it off outside a test harness; a packaged node does not enable it.

The socket is local only. It is not reachable over the network, and nothing about it changes the mesh's own authentication: a peer still verifies the node's signature, which is precisely why a local caller that can send through this socket is indistinguishable from the node itself.

See configuration.md for the key list and ../how-to/use-the-native-datagram-api.md for the client.

Threat-Resistance Matrix

The link layer's threat-resistance matrix is consolidated here from the FMP design document:

Threat	Mitigation
Connection exhaustion	Token-bucket rate limit + connection count limit
CPU exhaustion (msg1 flood)	Rate limit before crypto operations
Replay attacks	Counter-based nonces with sliding window (2048 entries)
State confusion	Strict handshake state machine validation
Spoofed encrypted packets	Index lookup + AEAD verification
Spoofed msg2	Index lookup + Noise ephemeral key binding
Address spoofing	Cryptographic authority, not address-based
Session correlation	Index rotation on rekey
Inbound exposure on `fips0`	Default-deny nftables baseline (operator opt-in)
Sybil identities	Discretionary peering + handshake rate limiting + optional peer ACL
Eclipse attack	Diverse peering across independent operators and transports
Unauthorized peer admission	Optional `peers.allow` allowlist consulted before handshake
Local impersonation via the native datagram API	API disabled by default; when enabled, `fips` group membership is the only gate and must be treated as key access

See ../design/fips-mesh-layer.md for the unauthenticated-attack-surface analysis (only handshake msg1 is reachable by unauthenticated parties), and ../design/fips-mesh-operation.md for the metadata-privacy model and the rejection of onion routing.

Default Network Exposures by Transport

Transport	Default Inbound	Default Bind	Opt-in
UDP	None until `bind_addr` set	`0.0.0.0:2121` typical	Operator sets `transports.udp.bind_addr`
TCP	None until `bind_addr` set	None — outbound-only without bind	Operator sets `transports.tcp.bind_addr`
Ethernet	Listens on configured interface (raw `AF_PACKET`)	EtherType 0x2121 on selected interface	Per-flag `listen`, `announce`, `auto_connect`, `accept_connections`
Tor	None until `directory_service` configured	`127.0.0.1:8443` (loopback only)	Operator sets `transports.tor.directory_service` and configures `HiddenServiceDir` in `torrc`
BLE	Off by default	n/a	Operator enables `transports.ble.*`
Nostr discovery	Off by default	n/a (relay client, not a listener)	Operator sets `node.discovery.nostr.enabled: true`

The mesh-layer fips0 interface is reachable from any mesh node that can route to you, not only direct peers — your direct peers forward traffic from any reachable mesh node onto your fips0. The default-deny nftables baseline (operator opt-in) is the recommended way to restrict inbound traffic on fips0. See ../how-to/enable-mesh-firewall.md.

16 KiB Raw Blame History