mirror of
https://github.com/jmcorgan/fips.git
synced 2026-10-05 19:18:25 +00:00
Johnathan Corgan
3a789370b9
A client process opens a flow to a peer's public key on a chosen port and sends and receives datagrams on a file descriptor the daemon hands it. No IPv6 emulation, no TUN device, no DNS: a datagram travels from key to key. The feature is off by default and is not a stable interface. The wire needs no change and gets none. Every FSP data packet has carried a port pair inside its AEAD envelope since v0.2.0, and port 256 is simply the IPv6 shim. What was missing was a way for a program to ask for a port of its own and be handed the traffic. Addressing is the part worth reading twice, because the obvious design is wrong. The x-only public key is the address. An npub is that key written in bech32, so converting between them is a local encoding rather than a lookup or a name service. The 16-byte node address that travels on the wire is the first half of a SHA-256 of the key: it is a truncated hash, it does not invert, and it appears nowhere a client can see. An earlier iteration of this work reported a peer by that hash and could supply a key only sometimes, which is what treating a wire identifier as an identity produces. An accepted flow therefore always knows its peer. The key is captured where the peer is authenticated rather than looked up when a report is rendered: every inbound datagram passes one call site inside a handler that refuses anything whose session is not established, and the responder has already rejected the session unless the claimed address derives from the key it proved. Reaching for the identity cache instead gives a best-effort answer from a structure that evicts. A listener is a descriptor. The daemon writes one message per arrival to it, carrying the new flow's descriptor and the peer's address, so poll, select and epoll work on a listener and accepting is a recvmsg. That is what lets the API be used from a program that already has an event loop, which a command-and-reply listener could not support: an arrival could not be waited on beside anything else. There is no accept command and no reject command. Refusing a flow is closing the descriptor you were handed. The Rust surface mirrors std::net. FipsStream::connect, FipsListener::bind, incoming, accept, io::Result and an errno mapping rather than a bespoke error type. An address is given as an npub, as a key, or as a pair, through one parameter, the way ToSocketAddrs takes several spellings of one thing. Each type holds its descriptor and copies of what setup told it and nothing else, so a stream that outlives its setup connection is not representable. set_nonblocking, AsFd and the four deadline methods carry the names and signatures std::net uses for the same jobs. They were asked for by a user integrating the API with tokio: AsyncFd requires a non-blocking descriptor, and anything receiving from a peer needs a bounded wait. AsFd is the better of the two descriptor accessors, because the borrow cannot outlive the value that owns the descriptor, so a reactor cannot hold a registration for a descriptor that has since been closed and its number reused by the next open. The non-blocking flag is read, modified and written back rather than assigned, since the flag word carries more than that one bit and a caller may have set O_ASYNC. A zero timeout is refused with EINVAL, because the kernel reads a zero timeval as "wait for ever", which inverts what a caller passing zero means; std::net refuses it for the same reason. The two directions are separate options and stay that way. FipsListener gets no timeout methods, matching TcpListener: bounding an accept is set_nonblocking plus the caller's own poll, which the reactor how-to builds. A flow taken from accept is blocking whatever the listener was set to, because the two are separate sockets and the daemon hands over a fresh one. One rule has no counterpart in Berkeley sockets and a client author must know it: the v1 wire carries no half-close, so nothing peer-driven ever closes a flow. A server written to read until the flow ends waits for a signal that cannot arrive, holding a thread and a flow per peer until its process exits. A program decides its own termination, and the example serves one datagram per flow. The tests reach a live daemon rather than a stand-in. Every public item had a unit test against a hand-written stand-in with canned replies, and the five entry points a program actually calls first, connect, connect_from, connect_at, bind and the SOCKET constant, had no coverage of any kind, because the tests that appear to cover them build a Wire over a socket pair and hand it to the private open and hold, so nothing ever resolved a socket path or mapped its errors. examples/native-surface.rs walks all thirty-eight items against a running daemon and reports the number of assertions it made. The count is read from the recorder rather than written as a literal, and the harness asserts the exit status, the completion marker and the count together, so deleting an assertion fails the check rather than quietly shrinking it. Watchdogs turn a hang into a named failure, which several of the walked behaviours would otherwise produce. The shared Docker image is built once for every integration leg, so the new binary is staged at all ten places the existing one is, the interop builder included, which gets a stub because those images exercise the wire between daemon versions and older refs do not carry the example. The platform gating was tested rather than reasoned about: flipping all eleven gates so the native API is excluded leaves the crate compiling clean across the workspace, every target and the profiling feature. The shipped docs tree gains what only the LaTeX manual under design/ had, which is not published with the daemon. A reference entry covers the whole surface: addressing and the port tiers, the Berkeley mapping, every method on FipsAddr, FipsStream, FipsListener and Incoming, the errno table, the ceilings, the four places data disappears with nothing reported, the line protocol and the command reference. The errno table gives names rather than numbers, since the client maps each name onto the libc constant for the platform it was built for and the supported platforms disagree on the numbers. A tutorial side trip stands up two throwaway nodes on one machine, peered over loopback UDP with no TUN and no DNS, then writes a listening program and a connecting program against them; it needs neither the public mesh nor root, because the native path is the one that does not go through the IPv6 adapter. The obligations a client in another language carries are a how-to of their own, since they are a task rather than a description: reading the setup connection with recvmsg, associating a descriptor with the last complete line, telling an empty datagram from a close, and six others. Serving many peers from one poll loop is another, with the whole program, because the straightforward listener spawns a thread per flow and that is wrong at the node's ceiling of 256. The drop causes are a table mapping each of the seven texts DropReason::as_str produces to the counter it increments, with drop_oversize called out as the ninth counter that is not in the table. What a daemon restart costs is a section of its own: every flow and listener ends, descriptors do not survive, there is no resumption, and datagrams sent but not yet forwarded are lost through a window nothing bounds. A stack comparison diagram places the interface against the stack a reader already knows: the same application over HTTP, TLS, TCP, IP and Ethernet on one side, and over its own format, FSP, FMP and a FIPS transport on the other, aligned so each row is one concern. The two columns are not alternatives and are not drawn as such. An unmodified IPv6 program's packets reach fips0, and the adapter hands each one to FSP as a payload, so the left stack runs inside the right one; the left column ends at a fork, eth0 for the ordinary internet and fips0 for the mesh, and an arrow leaves fips0 and runs back up into FSP's input. The row where TCP would be is empty on purpose and names Reliable Object Delivery, which is where that capability is expected to land. ROD is a v2 capability, the box is dashed because none of it exists yet, and the design entry says the part a reader needs most: nothing on the surface anticipates it, so a program written today should assume it does not exist. Both endpoints carry a scheme and a worked port, https://<npub>.fips:443 and fips://<npub>:443, with a footnote saying the two ports are not the same kind of thing, a TCP port inside the tunnel on the left and an FSP port on the right. The fips:// form is a coinage: nothing in the tree parses it, nothing registers the scheme, and the API takes a key and a port as separate arguments rather than a URL. The diagram also says where the right column stops, since FIPS over UDP still rides IP and Ethernet beneath. It appears in fips-concepts.md and fips-ipv6-adapter.md, which were making its argument in prose without a picture, and deliberately not in fips-architecture.md, which already carries the OSI mapping and makes the same point about the transport row. The gateway's control socket moves onto the same bind policy this API uses, which is the one change here that touches deployed behaviour: fips-gateway now tightens /run/fips to 0750. That is unreachable under the packaged deployment, where fips.service has already created the directory at that mode, and reachable for a source build or a container that starts the gateway alone. One changelog entry under Added, describing the released state: what a client opens and reads, the addressing and why the node address is not it, the listener being a descriptor, the std::net shape of the Rust surface, and the one rule Berkeley sockets have no counterpart for. It says in as many words that the wire is unchanged.
189 lines
8.7 KiB
Markdown
189 lines
8.7 KiB
Markdown
# Write a Native API Client in Another Language
|
|
|
|
**Goal:** speak the native datagram API's line protocol directly, from C,
|
|
Python, Go or anything else, without the Rust client module.
|
|
|
|
Everything here is something the shipped Rust library already does. It is
|
|
written out so an author working where there is no such library knows what they
|
|
are reproducing. **Each of these was a real defect before it was a rule, and
|
|
each fails intermittently rather than outright.**
|
|
|
|
If you are writing Rust, you do not need this guide. Use
|
|
`fips::native::client` and see
|
|
[use-the-native-datagram-api.md](use-the-native-datagram-api.md).
|
|
|
|
For the protocol itself — the framing, the reply shapes, the commands and their
|
|
refusals — see
|
|
[../reference/native-api.md](../reference/native-api.md#the-line-protocol).
|
|
|
|
## Step 1: Read the setup connection with recvmsg, never with a buffered reader
|
|
|
|
**Every read on the setup connection is a `recvmsg` with an ancillary buffer.**
|
|
A plain `read` consumes a descriptor-bearing message's bytes with no control
|
|
buffer, and the kernel then closes the descriptor rather than queueing it. The
|
|
reply looks perfectly correct and the flow is silently gone.
|
|
|
|
This applies to both setup commands: a `listen` reply carries a descriptor as
|
|
much as a `connect` reply does. In any language it means no buffered reader, no
|
|
`BufReader`, no `readline`, and no library that wraps the socket in a stream
|
|
abstraction. Keep the line buffering in your own code, over `recvmsg`.
|
|
|
|
Read a listener's own descriptor the same way, for the ancillary data and the
|
|
close-on-exec flag. One message there is one arrival carrying exactly its own
|
|
descriptor, so there is nothing to associate.
|
|
|
|
## Step 2: Attach a descriptor to the last complete line of its read
|
|
|
|
**A descriptor belongs to the last complete line of the read that carried it,
|
|
never to the next line the reader assembles.** A `recvmsg` returning ancillary
|
|
data ends exactly at the end of the `sendmsg` that carried it, but it may begin
|
|
with any amount of data written before it.
|
|
|
|
A client that sends one command per connection reads one line and cannot hit
|
|
this. A client that pipelines two setup commands on one connection can: the
|
|
first reply and the second, descriptor-bearing reply arrive as one read, and a
|
|
reader that attached the descriptor to the first would hand the flow to the
|
|
wrong caller.
|
|
|
|
Two corollaries:
|
|
|
|
- A read that carries a descriptor and completes no line must be **reported**
|
|
rather than held. Holding it means guessing which later line it belongs to.
|
|
- A descriptor that arrives with a line you are going to discard must still be
|
|
**closed**, or the flow leaks.
|
|
|
|
Neither can happen while the daemon writes exactly one whole line per `sendmsg`
|
|
and treats a short write as an error, which it does. That is an invariant of
|
|
two programs, though, not of the socket type.
|
|
|
|
## Step 3: Treat a zero-byte read as end of file only when POLLHUP is set
|
|
|
|
**An empty datagram and a closed peer both produce a zero-byte read, and
|
|
`MSG_EOR` does not tell them apart.** On Linux 6.8, `recvmsg` on an `AF_UNIX`
|
|
`SOCK_SEQPACKET` socket returns `msg_flags == 0` for a normal message, an empty
|
|
message and end of file alike, so the flag carries no information.
|
|
|
|
`POLLHUP` does discriminate. After a zero-byte read, a queued empty datagram
|
|
leaves the socket with no events pending, while a closed peer leaves `POLLHUP`
|
|
set and latched. Poll with an events mask of **zero**, because `POLLHUP` is
|
|
reported in `revents` whether or not it was requested. The poll costs nothing:
|
|
it runs only on the zero-byte path and does not block.
|
|
|
|
Both directions of the mistake are real. Reading an empty datagram as a close
|
|
lets a peer tear down a live flow by sending nothing, and presents as a
|
|
spurious disconnect. Reading a close as an empty datagram leaves the caller
|
|
spinning on a dead flow.
|
|
|
|
Note what a close here means: the daemon went away, never a peer finishing.
|
|
|
|
## Step 4: Send with MSG_NOSIGNAL
|
|
|
|
A datagram written to a flow whose daemon half has gone, or a command written
|
|
to a daemon that has exited, raises `SIGPIPE`, whose default disposition kills
|
|
the process.
|
|
|
|
Rust ignores the signal at startup, and CPython sets it to `SIG_IGN`, so a
|
|
program in either language sees `EPIPE`. **A C or C++ client that has not
|
|
changed the disposition simply dies.** The daemon and the shipped client pass
|
|
the flag on every send.
|
|
|
|
## Step 5: Set a deadline on the setup socket
|
|
|
|
Set `SO_RCVTIMEO` on the setup connection and rewrite the resulting would-block
|
|
into `ETIMEDOUT`. The shipped client uses five seconds.
|
|
|
|
Without it, a daemon that accepted your connection and then stopped answering
|
|
blocks the setup call forever. This is also what keeps `ETIMEDOUT` to exactly
|
|
one producer on the surface, which is what lets a caller read it.
|
|
|
|
## Step 6: Keep descriptor hygiene
|
|
|
|
Five rules. Each one leaks a flow or loses one when broken.
|
|
|
|
**Request close-on-exec** with `MSG_CMSG_CLOEXEC` on the `recvmsg`, rather than
|
|
setting it afterwards. Without it the descriptor survives an `exec` into a
|
|
child, the child's reference holds the flow open after this process closes its
|
|
own, and the flow keeps its slot against the node's ceiling until the child
|
|
exits.
|
|
|
|
**Walk the whole control buffer**, not only the first header. Close extra
|
|
descriptors rather than dropping them on the floor.
|
|
|
|
**Check for truncation after taking the descriptors, not before.** A
|
|
`MSG_CTRUNC` test that returns early leaks whatever did arrive.
|
|
|
|
**Lift the descriptor out of an arrival you cannot parse** before discarding
|
|
the message. Refusing a flow is closing its descriptor; discarding the message
|
|
without taking it leaks the flow instead.
|
|
|
|
**Bound the partial line.** A daemon that stopped sending newlines would
|
|
otherwise grow your buffer without end. The shipped client caps it at 64 KiB,
|
|
well above any reply.
|
|
|
|
## Step 7: Read the errno name, never the message
|
|
|
|
The refusal's `data.errno` is the contract. The `message` is for an operator
|
|
reading a log.
|
|
|
|
A client that matched on English would break on a wording change. The shipped
|
|
client discards the message entirely so that no caller can come to depend on
|
|
it, and the daemon's own match over its error types is exhaustive precisely so
|
|
a new refusal cannot reach a client without a code.
|
|
|
|
A reply carrying no `errno` at all should be read as `ECONNREFUSED`, which
|
|
covers a daemon older than the field. The errno table is in
|
|
[../reference/native-api.md](../reference/native-api.md#the-errno-table).
|
|
|
|
## Step 8: Size the receive buffer, and add no framing
|
|
|
|
Size every receive buffer at the flow's `max_payload`, read from the reply and
|
|
never computed.
|
|
|
|
`SOCK_SEQPACKET` truncates a longer datagram, discards the remainder and
|
|
reports success. It is detectable: `recvmsg` sets `MSG_TRUNC` in `msg_flags`
|
|
when it dropped part of a message. A plain `recv` discards `msg_flags` and so
|
|
sees none of it, which is where the belief that truncation is silent comes
|
|
from. Test `MSG_TRUNC` as well, and a stale or misread `max_payload` is caught
|
|
rather than quietly corrupting a payload.
|
|
|
|
Do not add framing. There is no header and no length prefix in either
|
|
direction. One send is one datagram.
|
|
|
|
## Step 9: Decide when a flow is over, because nothing else will
|
|
|
|
Everything above is the library's job. **The termination condition is not**, in
|
|
any language.
|
|
|
|
The v1 wire carries no half-close. Nothing peer-driven closes the daemon's half
|
|
of a live flow, so a loop that reads until the flow ends does not terminate.
|
|
Your program decides when a flow is over, or nothing does.
|
|
|
|
The two shapes that work are a bounded exchange, where the program serves a
|
|
known number of datagrams per flow and then drops it, and an idle deadline,
|
|
where the program sets a read timeout and treats its expiry as the end. A
|
|
server that reads "until the flow closes" holds a thread per peer forever and
|
|
holds every flow against the node's `max_flows` ceiling.
|
|
|
|
## Verify it
|
|
|
|
The repository's test harness drives the line protocol from Python and is the
|
|
closest thing to a second implementation:
|
|
|
|
- `testing/native-api/client.py` — a thin RPC client that runs a script of
|
|
steps over one connection and checks the replies. It implements every rule
|
|
above.
|
|
- `testing/native-api/control.py` — reads `show_native_flows` back over the
|
|
control socket while a flow is open.
|
|
|
|
Check what the node actually holds with `fipsctl show native-flows`, and read
|
|
the per-cause drop counters with `fipsctl stats metrics` under `native`.
|
|
|
|
## See also
|
|
|
|
- [use-the-native-datagram-api.md](use-the-native-datagram-api.md) — the Rust
|
|
path, where none of this is your problem
|
|
- [../reference/native-api.md](../reference/native-api.md) — the surface, the
|
|
line protocol, the command reference and the errno table
|
|
- [../reference/control-socket.md](../reference/control-socket.md) — the same
|
|
line framing, for the control socket
|