mirror of https://github.com/jmcorgan/fips.git synced 2026-10-05 19:18:25 +00:00

Files

T

Johnathan Corgan 3a789370b9 Add an experimental native datagram API addressed by public key

A client process opens a flow to a peer's public key on a chosen port and
sends and receives datagrams on a file descriptor the daemon hands it. No
IPv6 emulation, no TUN device, no DNS: a datagram travels from key to key.
The feature is off by default and is not a stable interface.

The wire needs no change and gets none. Every FSP data packet has carried a
port pair inside its AEAD envelope since v0.2.0, and port 256 is simply the
IPv6 shim. What was missing was a way for a program to ask for a port of its
own and be handed the traffic.

Addressing is the part worth reading twice, because the obvious design is
wrong. The x-only public key is the address. An npub is that key written in
bech32, so converting between them is a local encoding rather than a lookup
or a name service. The 16-byte node address that travels on the wire is the
first half of a SHA-256 of the key: it is a truncated hash, it does not
invert, and it appears nowhere a client can see. An earlier iteration of this
work reported a peer by that hash and could supply a key only sometimes,
which is what treating a wire identifier as an identity produces.

An accepted flow therefore always knows its peer. The key is captured where
the peer is authenticated rather than looked up when a report is rendered:
every inbound datagram passes one call site inside a handler that refuses
anything whose session is not established, and the responder has already
rejected the session unless the claimed address derives from the key it
proved. Reaching for the identity cache instead gives a best-effort answer
from a structure that evicts.

A listener is a descriptor. The daemon writes one message per arrival to it,
carrying the new flow's descriptor and the peer's address, so poll, select
and epoll work on a listener and accepting is a recvmsg. That is what lets
the API be used from a program that already has an event loop, which a
command-and-reply listener could not support: an arrival could not be waited
on beside anything else. There is no accept command and no reject command.
Refusing a flow is closing the descriptor you were handed.

The Rust surface mirrors std::net. FipsStream::connect, FipsListener::bind,
incoming, accept, io::Result and an errno mapping rather than a bespoke
error type. An address is given as an npub, as a key, or as a pair, through
one parameter, the way ToSocketAddrs takes several spellings of one thing.
Each type holds its descriptor and copies of what setup told it and nothing
else, so a stream that outlives its setup connection is not representable.

set_nonblocking, AsFd and the four deadline methods carry the names and
signatures std::net uses for the same jobs. They were asked for by a user
integrating the API with tokio: AsyncFd requires a non-blocking descriptor,
and anything receiving from a peer needs a bounded wait. AsFd is the better
of the two descriptor accessors, because the borrow cannot outlive the value
that owns the descriptor, so a reactor cannot hold a registration for a
descriptor that has since been closed and its number reused by the next
open. The non-blocking flag is read, modified and written back rather than
assigned, since the flag word carries more than that one bit and a caller may
have set O_ASYNC. A zero timeout is refused with EINVAL, because the kernel
reads a zero timeval as "wait for ever", which inverts what a caller passing
zero means; std::net refuses it for the same reason. The two directions are
separate options and stay that way. FipsListener gets no timeout methods,
matching TcpListener: bounding an accept is set_nonblocking plus the caller's
own poll, which the reactor how-to builds. A flow taken from accept is
blocking whatever the listener was set to, because the two are separate
sockets and the daemon hands over a fresh one.

One rule has no counterpart in Berkeley sockets and a client author must know
it: the v1 wire carries no half-close, so nothing peer-driven ever closes a
flow. A server written to read until the flow ends waits for a signal that
cannot arrive, holding a thread and a flow per peer until its process exits.
A program decides its own termination, and the example serves one datagram
per flow.

The tests reach a live daemon rather than a stand-in. Every public item had a
unit test against a hand-written stand-in with canned replies, and the five
entry points a program actually calls first, connect, connect_from,
connect_at, bind and the SOCKET constant, had no coverage of any kind,
because the tests that appear to cover them build a Wire over a socket pair
and hand it to the private open and hold, so nothing ever resolved a socket
path or mapped its errors. examples/native-surface.rs walks all thirty-eight
items against a running daemon and reports the number of assertions it made.
The count is read from the recorder rather than written as a literal, and the
harness asserts the exit status, the completion marker and the count
together, so deleting an assertion fails the check rather than quietly
shrinking it. Watchdogs turn a hang into a named failure, which several of
the walked behaviours would otherwise produce. The shared Docker image is
built once for every integration leg, so the new binary is staged at all ten
places the existing one is, the interop builder included, which gets a stub
because those images exercise the wire between daemon versions and older refs
do not carry the example. The platform gating was tested rather than reasoned
about: flipping all eleven gates so the native API is excluded leaves the
crate compiling clean across the workspace, every target and the profiling
feature.

The shipped docs tree gains what only the LaTeX manual under design/ had,
which is not published with the daemon. A reference entry covers the whole
surface: addressing and the port tiers, the Berkeley mapping, every method on
FipsAddr, FipsStream, FipsListener and Incoming, the errno table, the
ceilings, the four places data disappears with nothing reported, the line
protocol and the command reference. The errno table gives names rather than
numbers, since the client maps each name onto the libc constant for the
platform it was built for and the supported platforms disagree on the
numbers. A tutorial side trip stands up two throwaway nodes on one machine,
peered over loopback UDP with no TUN and no DNS, then writes a listening
program and a connecting program against them; it needs neither the public
mesh nor root, because the native path is the one that does not go through
the IPv6 adapter. The obligations a client in another language carries are a
how-to of their own, since they are a task rather than a description:
reading the setup connection with recvmsg, associating a descriptor with the
last complete line, telling an empty datagram from a close, and six others.
Serving many peers from one poll loop is another, with the whole program,
because the straightforward listener spawns a thread per flow and that is
wrong at the node's ceiling of 256. The drop causes are a table mapping each
of the seven texts DropReason::as_str produces to the counter it increments,
with drop_oversize called out as the ninth counter that is not in the table.
What a daemon restart costs is a section of its own: every flow and listener
ends, descriptors do not survive, there is no resumption, and datagrams sent
but not yet forwarded are lost through a window nothing bounds.

A stack comparison diagram places the interface against the stack a reader
already knows: the same application over HTTP, TLS, TCP, IP and Ethernet on
one side, and over its own format, FSP, FMP and a FIPS transport on the
other, aligned so each row is one concern. The two columns are not
alternatives and are not drawn as such. An unmodified IPv6 program's packets
reach fips0, and the adapter hands each one to FSP as a payload, so the left
stack runs inside the right one; the left column ends at a fork, eth0 for the
ordinary internet and fips0 for the mesh, and an arrow leaves fips0 and runs
back up into FSP's input. The row where TCP would be is empty on purpose and
names Reliable Object Delivery, which is where that capability is expected to
land. ROD is a v2 capability, the box is dashed because none of it exists
yet, and the design entry says the part a reader needs most: nothing on the
surface anticipates it, so a program written today should assume it does not
exist. Both endpoints carry a scheme and a worked port,
https://<npub>.fips:443 and fips://<npub>:443, with a footnote saying the two
ports are not the same kind of thing, a TCP port inside the tunnel on the
left and an FSP port on the right. The fips:// form is a coinage: nothing in
the tree parses it, nothing registers the scheme, and the API takes a key and
a port as separate arguments rather than a URL. The diagram also says where
the right column stops, since FIPS over UDP still rides IP and Ethernet
beneath. It appears in fips-concepts.md and fips-ipv6-adapter.md, which were
making its argument in prose without a picture, and deliberately not in
fips-architecture.md, which already carries the OSI mapping and makes the
same point about the transport row.

The gateway's control socket moves onto the same bind policy this API uses,
which is the one change here that touches deployed behaviour: fips-gateway
now tightens /run/fips to 0750. That is unreachable under the packaged
deployment, where fips.service has already created the directory at that
mode, and reachable for a source build or a container that starts the gateway
alone.

One changelog entry under Added, describing the released state: what a
client opens and reads, the addressing and why the node address is not it,
the listener being a descriptor, the std::net shape of the Rust surface,
and the one rule Berkeley sockets have no counterpart for. It says in as
many words that the wire is unchanged.

2026-08-21 05:48:23 +00:00

7.7 KiB

Raw Blame History

Native Datagram API

The native datagram API lets a local program move bytes between two public keys over FSP, with no IPv6 emulation and no TUN device in the path. A program calls connect for a flow to a public key and a port, or bind for a port to receive flows on, and from then on uses ordinary socket calls.

This document explains what the interface is for and where its edges are. For the surface itself — every type, method, errno and command — see ../reference/native-api.md. For the steps to enable it and write a program, see ../how-to/use-the-native-datagram-api.md.

Where it sits

The two endpoints at the top are the same node reached two ways. The fips:// form is illustrative: no code in this repository parses it, nothing registers the scheme, and the API takes a key and a port as separate arguments rather than a URL. It is drawn because it is the shape an address takes on that side, against a .fips name the adapter's DNS really does resolve.

Read row by row, the native path replaces three layers and declines to replace a fourth. FSP takes TLS's place and anchors trust in the key rather than in a certificate authority. FMP takes IPv6's place and routes by spanning tree and bloom filter rather than by address prefix, with the address derived from the key. The transport layer takes the medium's place and can be several media at once.

There is nothing where TCP was, and on the native path that is the single most consequential row today. No acknowledgement, no retransmission, no ordering and no flow control: a program that needs any of them builds it into its own payload.

That row is marked ROD — Reliable Object Delivery, which is where the capability is expected to land. ROD is a v2 capability and is not in v1; it may be pulled forward. Until it is, treat the row as empty and design around it, because a program written against a reliability layer that is not there yet fails in the ways this document's "not a reliability layer" section describes.

The two paths are not alternatives at the bottom. They converge. An unmodified IPv6 program does not stop at a wire: its packets reach fips0, and the adapter hands each one to FSP as a payload. That is the arrow running up the middle of the diagram, and it is why the left stack is drawn ending at an interface rather than at Ethernet.

So the whole left column runs inside the right one. TCP included — which is the practical answer to the empty row above it. A program that needs a reliable ordered stream over the mesh already has one: run it over fips0 and let TCP do what TCP does, inside FSP's encryption. What the native API offers instead is the same mesh with four layers of machinery removed, for a program willing to do without them.

The bottom of the diagram is not always the bottom of the stack either. When FIPS overlays an existing network its transport is UDP, which still rides IP and Ethernet beneath; when the mesh is the network, a transport sits on a link directly.

What it is instead of

The fastest way to place the interface is by contrast with the TUN device, which is the other way a program gets FIPS traffic.

	TUN interface	Native datagram API
Addressing	IPv6 address	public key, written as an npub
Name resolution	DNS over the mesh	none: the program supplies the key
Kernel object	TUN device, routes	a `FipsStream` per peer
Encapsulation	IPv6 emulated over FSP	FSP port pair, no IP layer
Program sees	an IP network	a `FipsStream`
Privilege	`CAP_NET_ADMIN` to create the device	membership of group `fips`
Demultiplexing	by address and port	by flow, one stream each

The IPv6 emulation is not removed by this interface. It continues to run beside it on FSP port 256, which is why that port and the tier around it are refused to a program. What the native API removes is a program's dependence on it: a program that wants to move bytes between two known public keys no longer has to acquire an IPv6 address, resolve a name, and hand its payload to a protocol stack that will encapsulate it again.

Both paths reach the same place. A native datagram and an emulated IPv6 packet are both FSP payloads with a port pair, carried in the same encrypted session to the same peer. The difference is entirely on the local side of the daemon.

Status

The wire is connected: a datagram sent on a flow leaves the node over FSP, and one arriving on a held port reaches its flow.

The interface around it is experimental. It is not versioned, it has no compatibility promise, and three of its five commands exist only to let the daemon's own checks drive the receive path without a peer. It is Linux and FreeBSD only, and it is off by default.

What this is not

Not a stable interface. It is an experiment on the v1 wire. Names, fields, reply shapes and the command set may change without a deprecation cycle.

Not the v2 process API. The v2 external process API is a separate and later design, which retires ports entirely in favour of a listener, connection and stream model. Nothing here governs it and nothing there governs this. The one thing this interface takes from that work is the FSP port tiers, because port 256 already carries the IPv6 shim on the deployed wire and a new service must not collide with it.

Not a reliability layer. There is no acknowledgement, no retransmission, no ordering guarantee and no flow control between the two ends. A datagram is carried or it is dropped. Some drops are counted inside the daemon and none are reported to a program for real traffic. A program that needs delivery guarantees builds them itself, on top, in the payload — or runs over fips0 and lets TCP provide them.

Reliable Object Delivery (ROD) is the v2 capability intended to fill this gap, and it may be pulled forward into v1. Nothing here anticipates it: no field, reply shape or command on this surface is reserved for it, and a program written today should assume it does not exist.

Not an authorization boundary. The socket's group ownership is the whole of the access control. Any process that can open it can send as this node's identity and can receive mesh traffic on any port it can claim, and there is no per-program separation beyond the port registry. Because the descriptor carries the flow, a process handed one over SCM_RIGHTS can send as this node on that flow without ever opening the socket. See ../reference/security.md.

Not multi-tenant. max_flows is node-wide with no per-program share, so one program can exhaust it, and every other program then sees EMFILE on connect and silent drops on its listeners.

Not a connection in the TCP sense. A successful connect is a local registration and contacts no peer. There is no handshake, no keepalive and no notification that a peer went away. A flow ends when its descriptor closes, and in no other way. In particular a peer cannot end your flow: it has no close to send. That single fact shapes every program written against this interface, and the consequences are drawn out in ../how-to/use-the-native-datagram-api.md.

7.7 KiB Raw Blame History