Files
fips/docs/reference/native-api.md
T
Johnathan Corgan 3a789370b9 Add an experimental native datagram API addressed by public key
A client process opens a flow to a peer's public key on a chosen port and
sends and receives datagrams on a file descriptor the daemon hands it. No
IPv6 emulation, no TUN device, no DNS: a datagram travels from key to key.
The feature is off by default and is not a stable interface.

The wire needs no change and gets none. Every FSP data packet has carried a
port pair inside its AEAD envelope since v0.2.0, and port 256 is simply the
IPv6 shim. What was missing was a way for a program to ask for a port of its
own and be handed the traffic.

Addressing is the part worth reading twice, because the obvious design is
wrong. The x-only public key is the address. An npub is that key written in
bech32, so converting between them is a local encoding rather than a lookup
or a name service. The 16-byte node address that travels on the wire is the
first half of a SHA-256 of the key: it is a truncated hash, it does not
invert, and it appears nowhere a client can see. An earlier iteration of this
work reported a peer by that hash and could supply a key only sometimes,
which is what treating a wire identifier as an identity produces.

An accepted flow therefore always knows its peer. The key is captured where
the peer is authenticated rather than looked up when a report is rendered:
every inbound datagram passes one call site inside a handler that refuses
anything whose session is not established, and the responder has already
rejected the session unless the claimed address derives from the key it
proved. Reaching for the identity cache instead gives a best-effort answer
from a structure that evicts.

A listener is a descriptor. The daemon writes one message per arrival to it,
carrying the new flow's descriptor and the peer's address, so poll, select
and epoll work on a listener and accepting is a recvmsg. That is what lets
the API be used from a program that already has an event loop, which a
command-and-reply listener could not support: an arrival could not be waited
on beside anything else. There is no accept command and no reject command.
Refusing a flow is closing the descriptor you were handed.

The Rust surface mirrors std::net. FipsStream::connect, FipsListener::bind,
incoming, accept, io::Result and an errno mapping rather than a bespoke
error type. An address is given as an npub, as a key, or as a pair, through
one parameter, the way ToSocketAddrs takes several spellings of one thing.
Each type holds its descriptor and copies of what setup told it and nothing
else, so a stream that outlives its setup connection is not representable.

set_nonblocking, AsFd and the four deadline methods carry the names and
signatures std::net uses for the same jobs. They were asked for by a user
integrating the API with tokio: AsyncFd requires a non-blocking descriptor,
and anything receiving from a peer needs a bounded wait. AsFd is the better
of the two descriptor accessors, because the borrow cannot outlive the value
that owns the descriptor, so a reactor cannot hold a registration for a
descriptor that has since been closed and its number reused by the next
open. The non-blocking flag is read, modified and written back rather than
assigned, since the flag word carries more than that one bit and a caller may
have set O_ASYNC. A zero timeout is refused with EINVAL, because the kernel
reads a zero timeval as "wait for ever", which inverts what a caller passing
zero means; std::net refuses it for the same reason. The two directions are
separate options and stay that way. FipsListener gets no timeout methods,
matching TcpListener: bounding an accept is set_nonblocking plus the caller's
own poll, which the reactor how-to builds. A flow taken from accept is
blocking whatever the listener was set to, because the two are separate
sockets and the daemon hands over a fresh one.

One rule has no counterpart in Berkeley sockets and a client author must know
it: the v1 wire carries no half-close, so nothing peer-driven ever closes a
flow. A server written to read until the flow ends waits for a signal that
cannot arrive, holding a thread and a flow per peer until its process exits.
A program decides its own termination, and the example serves one datagram
per flow.

The tests reach a live daemon rather than a stand-in. Every public item had a
unit test against a hand-written stand-in with canned replies, and the five
entry points a program actually calls first, connect, connect_from,
connect_at, bind and the SOCKET constant, had no coverage of any kind,
because the tests that appear to cover them build a Wire over a socket pair
and hand it to the private open and hold, so nothing ever resolved a socket
path or mapped its errors. examples/native-surface.rs walks all thirty-eight
items against a running daemon and reports the number of assertions it made.
The count is read from the recorder rather than written as a literal, and the
harness asserts the exit status, the completion marker and the count
together, so deleting an assertion fails the check rather than quietly
shrinking it. Watchdogs turn a hang into a named failure, which several of
the walked behaviours would otherwise produce. The shared Docker image is
built once for every integration leg, so the new binary is staged at all ten
places the existing one is, the interop builder included, which gets a stub
because those images exercise the wire between daemon versions and older refs
do not carry the example. The platform gating was tested rather than reasoned
about: flipping all eleven gates so the native API is excluded leaves the
crate compiling clean across the workspace, every target and the profiling
feature.

The shipped docs tree gains what only the LaTeX manual under design/ had,
which is not published with the daemon. A reference entry covers the whole
surface: addressing and the port tiers, the Berkeley mapping, every method on
FipsAddr, FipsStream, FipsListener and Incoming, the errno table, the
ceilings, the four places data disappears with nothing reported, the line
protocol and the command reference. The errno table gives names rather than
numbers, since the client maps each name onto the libc constant for the
platform it was built for and the supported platforms disagree on the
numbers. A tutorial side trip stands up two throwaway nodes on one machine,
peered over loopback UDP with no TUN and no DNS, then writes a listening
program and a connecting program against them; it needs neither the public
mesh nor root, because the native path is the one that does not go through
the IPv6 adapter. The obligations a client in another language carries are a
how-to of their own, since they are a task rather than a description:
reading the setup connection with recvmsg, associating a descriptor with the
last complete line, telling an empty datagram from a close, and six others.
Serving many peers from one poll loop is another, with the whole program,
because the straightforward listener spawns a thread per flow and that is
wrong at the node's ceiling of 256. The drop causes are a table mapping each
of the seven texts DropReason::as_str produces to the counter it increments,
with drop_oversize called out as the ninth counter that is not in the table.
What a daemon restart costs is a section of its own: every flow and listener
ends, descriptors do not survive, there is no resumption, and datagrams sent
but not yet forwarded are lost through a window nothing bounds.

A stack comparison diagram places the interface against the stack a reader
already knows: the same application over HTTP, TLS, TCP, IP and Ethernet on
one side, and over its own format, FSP, FMP and a FIPS transport on the
other, aligned so each row is one concern. The two columns are not
alternatives and are not drawn as such. An unmodified IPv6 program's packets
reach fips0, and the adapter hands each one to FSP as a payload, so the left
stack runs inside the right one; the left column ends at a fork, eth0 for the
ordinary internet and fips0 for the mesh, and an arrow leaves fips0 and runs
back up into FSP's input. The row where TCP would be is empty on purpose and
names Reliable Object Delivery, which is where that capability is expected to
land. ROD is a v2 capability, the box is dashed because none of it exists
yet, and the design entry says the part a reader needs most: nothing on the
surface anticipates it, so a program written today should assume it does not
exist. Both endpoints carry a scheme and a worked port,
https://<npub>.fips:443 and fips://<npub>:443, with a footnote saying the two
ports are not the same kind of thing, a TCP port inside the tunnel on the
left and an FSP port on the right. The fips:// form is a coinage: nothing in
the tree parses it, nothing registers the scheme, and the API takes a key and
a port as separate arguments rather than a URL. The diagram also says where
the right column stops, since FIPS over UDP still rides IP and Ethernet
beneath. It appears in fips-concepts.md and fips-ipv6-adapter.md, which were
making its argument in prose without a picture, and deliberately not in
fips-architecture.md, which already carries the OSI mapping and makes the
same point about the transport row.

The gateway's control socket moves onto the same bind policy this API uses,
which is the one change here that touches deployed behaviour: fips-gateway
now tightens /run/fips to 0750. That is unreachable under the packaged
deployment, where fips.service has already created the directory at that
mode, and reachable for a source build or a container that starts the gateway
alone.

One changelog entry under Added, describing the released state: what a
client opens and reads, the addressing and why the node address is not it,
the listener being a descriptor, the std::net shape of the Rust surface,
and the one rule Berkeley sockets have no counterpart for. It says in as
many words that the wire is unchanged.
2026-08-21 05:48:23 +00:00

37 KiB

Native Datagram API

The native datagram API lets a program send and receive datagrams over the mesh addressed by public key and port, with no IPv6 emulation and no TUN device. A program opens a flow to a peer, receives a socket descriptor, and uses ordinary socket calls on it.

This document describes the surface. For the steps to enable it and write a first program, see ../how-to/use-the-native-datagram-api.md.

Status: experimental. The Rust surface and the line protocol may both change. The API is off by default and is Unix only.

Enabling it

Every bound is a key under node.native_api. The defaults are the values compiled into src/config/node.rs.

Key Default What it bounds
enabled false Whether the socket is bound at all.
socket_path resolved See Socket path.
pending_per_flow 16 Datagrams held for one flow, whether it awaits hand-off or its program is slow to read. Ceiling 64, floor 1, both applied when the configuration loads.
backlog 16 Flows announced on one listener and not yet taken. Floor 1.
max_flows 256 Flows this node holds at once, across every program. Established and pending flows both count.
debug_commands false Whether the three debug commands are answered.

There is no cap on flows per program, and no notion of a program to hang one on. One program can reach max_flows by itself, and the ceiling is shared with every other program on that node.

See configuration.md for the full YAML reference, and security.md for what fips group membership grants once the API is on.

Socket path

node.native_api.socket_path has a default resolved at startup rather than a fixed string. The resolver takes the first of these that applies and appends api.sock:

  1. /run/fips/api.sock, when /run/fips exists as a directory. This is the packaged Linux convention and the constant the shipped Rust client compiles in.
  2. /var/run/fips/api.sock on FreeBSD, whose service scripts create that directory. Linux skips this step.
  3. $XDG_RUNTIME_DIR/fips/api.sock, when that variable names an existing directory. A development run usually gets this one.
  4. /tmp/fips-api.sock, the last resort.

Selection is by existence of the directory, not by whether the client can write to it. A client should therefore take the path as an argument with these as its defaults, rather than compile one in.

Addressing

An address is an x-only secp256k1 public key and a port. Nothing else identifies an end. There is no node identifier, no address family, and no scope.

The npub is the key written down. Conversion between the two is bech32 and nothing else: no lookup, no resolution, no name service. The conversion is local, needs no daemon, and fails identically whether or not one runs.

The 16-byte node address that the wire carries never appears on this surface. It is a truncated hash of the public key, it does not invert, and no call here accepts or returns one.

Ports

Ports are FSP ports, carried inside the encrypted FSP envelope. They are tiered.

Range Use What a program may do
0-255 Protocol use Refused.
256-1023 FIPS standard services Refused. Port 256 is the IPv6 shim.
1024-49151 Application, well known Name it explicitly.
49152-65535 Application, ephemeral Name it explicitly, or ask for 0 and be given one from this range.

Nothing reserves the upper range against explicit use. A service that wants a fixed port above 49151 may hold one, and ephemeral allocation will not take it away, because the sweep skips a port already held.

The refusal applies to the remote port as well as the local one. A program cannot send to a peer's port 256 and inject into its IPv6 plane. By the same rule, a peer listening below 1024 is unreachable from here: the connect is refused with EADDRNOTAVAIL before any node state is touched.

Port 0 means "any port", for both calls, and it is the only spelling of that. This matches bind(2) with port 0. local_addr() afterwards reports the port actually held, which is getsockname.

A local port has exactly one owner, a listener or one connected flow, never both. A connected flow owns its local port and frees it when the flow closes. A flow accepted from a listener shares the listener's port and does not own it, so closing an accepted flow frees no port; the port returns when the listener drops. Ephemeral allocation sweeps forward from the last claim and wraps once, so a port is not immediately reused after release.

The Rust surface

The crate module is fips::native::client. It is gated to Unix targets.

use fips::native::client::{FipsAddr, FipsListener, FipsStream, ToFipsAddr};

SOCKET is a &str constant holding /run/fips/api.sock, the packaged path. XOnlyPublicKey is re-exported from secp256k1 so a caller needs no direct dependency on that crate.

The Berkeley mapping

The descriptor is a real socket. Everything a program does with a flow after setup is the operating system, not this API.

Berkeley Here Notes
socket() none There is no unbound object to make. connect and bind each return one already bound.
bind() FipsListener::bind One setup call. Port 0 asks the daemon to choose.
listen() none Folded into bind. The backlog is node configuration, not a caller's argument.
connect() FipsStream::connect One setup call. A local registration that contacts no peer.
accept() FipsListener::accept Blocks until a flow arrives. One recvmsg, no round trip to the daemon.
send() FipsStream::send One datagram, whole or not at all. One local size check, then one syscall.
recv() FipsStream::recv One datagram. Ok(0) is an empty one and only that.
poll(), select(), epoll the same calls Both types are AsFd and AsRawFd.
fcntl(O_NONBLOCK) set_nonblocking() On both types. Read-modify-write, so a flag the caller set survives.
close() drop Dropping releases the flow or unbinds the port. There is no method.
getsockname() local_addr() A field read of what setup reported. It cannot fail.
getpeername() peer_addr() The same, for the far end.
SO_RCVTIMEO set_read_timeout() On a flow, with set_write_timeout and both getters.
SO_RCVBUF, others via AsFd setsockopt on the descriptor. The surface exposes no option beyond the deadlines.
inet_pton() npub decode bech32, locally, with no lookup.
inet_ntop() npub encode Its exact inverse.

Four differences from Berkeley remain. There is no socket() and no listen(), because a descriptor cannot exist before the daemon has agreed to make one. A successful connect proves nothing about the peer. Ok(0) from recv is an empty datagram rather than end of file. And there is a payload ceiling, reported per flow, which TCP has no equivalent of.

FipsAddr

One end of a flow. The fields are private, so an address in hand always names a valid key.

Copy, Debug, Clone, PartialEq, Eq. It derives neither Hash nor Ord, so it cannot key a HashMap or a BTreeMap. Key on addr.key().serialize() or on addr.to_string() instead. std::net::SocketAddr derives both, so the omission is a surprise rather than a convention.

Call Returns, and how it fails
new(key, port) The address. Infallible: both arguments are already well typed. Port 0 and the reserved tiers are representable and are refused later, by the daemon, at setup.
key() The public key, by value. Infallible.
port() The port. Infallible.
Display Writes npub1...:4242. Pure bech32, no daemon, no lookup.
FromStr Reads that back. EINVAL for no colon, a port that is not a u16, or a head that is not an npub. An nsec is refused despite being bech32 of the right length.

Display and FromStr are exact inverses. Parsing splits on the last colon and is purely local, so it fails identically with no daemon running. The colon is unambiguous because the bech32 character set does not contain one.

ToFipsAddr

The mirror of std::net::ToSocketAddrs. One generic parameter takes every spelling of one address. It is taken by &self, so a caller never gives up ownership, and no implementation opens a socket or contacts the daemon. The only error any of them produces is EINVAL.

Implementation Notes
FipsAddr Returns a copy. Structurally infallible.
(XOnlyPublicKey, u16) The binary address, already parsed. Structurally infallible.
([u8; 32], u16) 32 raw bytes. The one binary form that can be rejected: EINVAL if the bytes are not a valid x-only public key, which [0u8; 32] is not.
(&str, u16) An npub and a port. EINVAL if the npub does not decode. The port is taken as given and range-checked later.
(String, u16) The same, for a string from argv or a config file.
str The whole address as "npub1...:4242". Delegates to FromStr.
String The same.
&T where T: ToFipsAddr A blanket implementation over references, so &addr and &&str work at all.

There is no associated iterator type, unlike ToSocketAddrs. An address here resolves to exactly one endpoint.

The concrete implementations are on the unsized str rather than on &str. That is what lets the blanket reference implementation cover &str and &&str alike. It makes no difference at a call site.

FipsStream

One datagram flow, and the descriptor it rides on. A flow is an exact match of both ends and both ports. The descriptor is the flow: it lives while a process holds that descriptor and ends when the last one closes it.

Send + Sync + 'static, with no Arc and no borrow. There is no Clone and no try_clone. Because send and recv both take &self, a shared borrow across two scoped threads is a full-duplex reader and writer. A program wanting an owned writer handle reaches for Arc<FipsStream>.

Constructors

Call What it does
connect(addr) A flow to addr from an ephemeral local port, at the packaged socket path.
connect_from(local, addr) The same from a named local port. connect_from(0, addr) is exactly connect(addr).
connect_at(sock, local, addr) The full form, naming the daemon's socket. What a development node and a test harness need.

All three resolve the address first, so a bad address is EINVAL with no socket opened. Then, in this order: whatever UnixStream::connect gives for the socket path (NotFound when no daemon runs or the API is disabled, PermissionDenied on socket permissions, ECONNREFUSED when the path exists but nothing is accepting); EPIPE if the daemon vanishes mid-write; ETIMEDOUT if no answer arrives inside the five-second setup deadline; then the daemon's own refusals; then InvalidData for a reply this client cannot parse.

A stream does not outlive its setup connection, because the connection is not an object at all. It is a local binding inside the constructor, dropped before the call returns. What comes back holds an owned descriptor and four Copy scalars.

Methods

send(&self, buf: &[u8]) -> io::Result<()> sends one datagram. There is no byte count, because a SOCK_SEQPACKET message is delivered whole or not at all. buf.len() > max_payload() is EMSGSIZE before any syscall, so a caller learns which datagram was too large rather than finding a gap at the far end. EPIPE means the daemon has gone. Takes &self, so it can be called concurrently with recv from another thread.

The library sends with MSG_NOSIGNAL rather than writing to the descriptor. A Rust binary ignores SIGPIPE at startup anyway, but a C program that has loaded this crate would otherwise be killed by a write to a departed daemon rather than told about it. EINTR is retried internally.

Ok(()) means only that the local socket pair accepted the bytes. See Where data disappears.

recv(&self, buf: &mut [u8]) -> io::Result<usize> receives one datagram and returns its length. It blocks by default: the descriptor arrives blocking and no timeout is set on it. set_nonblocking(true) makes it return WouldBlock instead. set_read_timeout bounds a single wait without going non-blocking. A datagram longer than buf is truncated and the remainder discarded, which is SOCK_SEQPACKET behaviour and is not reported; size buf at max_payload() and it cannot happen. EINTR is retried.

Ok(0) is an empty datagram and only that. An empty datagram and a closed peer both produce a zero-byte read, and MSG_EOR does not tell them apart. On the zero-byte path only, the library polls the descriptor with an events mask of zero and reads POLLHUP from revents, returning EPIPE for a genuine close and Ok(0) otherwise. Without that step a peer could tear down a live flow by sending nothing. The EPIPE that recv returns means the daemon went away, never that a peer finished.

peer_addr() and local_addr() return FipsAddr, not io::Result<FipsAddr>, unlike their TcpStream counterparts. These are field reads of what the setup reply already carried. local_addr() carries the node's own public key, which on an accepted stream comes from the arrival message rather than from the listener, so a node holding more than one identity still answers correctly.

max_payload() returns the largest datagram this flow will carry. Infallible. It is a snapshot taken when the flow opened and does not track a later MTU change.

set_read_timeout(Option<Duration>) and set_write_timeout bound one recv or one send, as their TcpStream counterparts do. None clears. A zero duration is EINVAL, because the kernel reads a zero timeout as "wait for ever" and a caller passing zero means the opposite. An expiring deadline reports WouldBlock, the same answer a non-blocking descriptor gives. read_timeout() and write_timeout() read them back, None when unset. The two directions are separate options, and setting one leaves the other alone.

set_nonblocking(bool) puts the flow in or out of non-blocking mode. In that mode recv returns WouldBlock rather than waiting, and so does send when the daemon is not draining the flow fast enough. A flow from accept is blocking however its listener was set: they are separate sockets and the daemon hands over a fresh one.

AsFd and AsRawFd both return the flow's descriptor. Prefer AsFd: its borrow cannot outlive the stream, so a reactor cannot hold a registration for a descriptor that has since been closed and its number reissued. The stream keeps ownership either way. Do not close the descriptor and do not wrap it in anything that takes ownership, because the stream's own drop is what releases the flow. This is the route to SO_RCVBUF and every other socket option the surface does not expose.

A descriptor that survives an exec into a child holds the flow open after this process closes its own copy, and the flow keeps its slot against the node's ceiling until the child exits. The library requests close-on-exec when it takes the descriptor, so this happens only if a program deliberately clears the flag.

FipsListener

A held local port, and the descriptor arriving flows are delivered on. The listener is a descriptor, so it joins an existing poll, select or epoll loop with no new mechanism, and accept is one recvmsg on it.

Send + Sync + 'static, no Clone, no try_clone, no explicit Drop.

bind(port) holds port, or an ephemeral one when port is 0, at the packaged path. bind_at(sock, port) is the same, naming the daemon's socket. Failures are the socket-path and setup-deadline rows above, plus EADDRINUSE when a listener or a flow already holds the port, and EADDRNOTAVAIL for a reserved tier or an exhausted ephemeral range.

accept() -> io::Result<(FipsStream, FipsAddr)> blocks until a flow arrives. One recvmsg, one arrival: a SOCK_SEQPACKET message carries exactly its own descriptor. Takes &self, so several threads may accept on one listener concurrently. It does not map a would-block to ETIMEDOUT, so a caller that put the listener in non-blocking mode sees a raw WouldBlock. Failures: EPIPE when the message carried no descriptor, meaning the daemon closed its half; InvalidData for an arrival that is not JSON or is missing a field; and io::Error::other if the control buffer overflowed and a descriptor was lost.

Whatever the peer sent before accept returned is already queued on the returned stream. Refusing a flow is dropping the stream, because there is no other way to refuse one. An unparseable arrival is therefore reported only after the descriptor it carried has been taken into ownership, so a parse failure refuses the flow rather than leaking it.

incoming() returns an Incoming<'_>, which borrows the listener for the iterator's lifetime, so the listener cannot be moved or dropped mid-iteration.

local_addr() returns the port actually held with the node's own key, not an io::Result.

set_nonblocking(bool) puts the listener in or out of non-blocking mode. In that mode accept returns WouldBlock when no flow has arrived, and so does every incoming item, which makes that iterator spin unless the caller waits on the descriptor between items. It does not reach the flows the listener yields: each arrives blocking.

AsFd and AsRawFd return the listener's descriptor. poll on it reports readable exactly when accept would not block.

There is no set_read_timeout here, following TcpListener, which has none either. A bounded accept is set_nonblocking plus a wait of the caller's own on the descriptor.

Dropping closes the descriptor and unbinds the port. Flows already accepted from it are untouched; flows still pending on it go with it.

Incoming

The iterator incoming() returns. Its item is io::Result<FipsStream>: it calls accept and discards the peer address, which is recoverable as stream.peer_addr().

It never returns None. A listener has no last flow, and a failed accept is yielded as an Err item rather than ending the iteration, so a for loop over it never falls through. A reader who writes let flow = flow?; inside the loop exits it on the first transient error, which is a different shape from TcpListener habits.

Errors

There is no bespoke error type. Every call returns io::Result, so the surface matches std::net and a future C binding can return the number directly.

Match on err.raw_os_error() against libc constants, never on err.kind(). EMFILE and EMSGSIZE both carry ErrorKind::Uncategorized, which is #[non_exhaustive] and cannot be named in a match arm, so kind() cannot distinguish the node's flow ceiling or an oversize datagram from anything else.

The errno table

Compare against the libc constant, never against a literal. The client maps each name onto libc::<NAME> for the platform it was built for, so the number differs between Linux and FreeBSD.

errno kind() What causes it
EADDRINUSE AddrInUse The local port is held, or a flow already exists between those two ends on those two ports.
EADDRNOTAVAIL AddrNotAvailable A port in a reserved tier, local or remote, or the ephemeral range exhausted. Reserved rather than EACCES, because no program however privileged may hold port 256.
EMFILE Uncategorized The node is at max_flows, or the socket pair could not be made. Unmatchable by kind.
EINVAL InvalidInput An address that does not resolve locally, or a malformed command.
EMSGSIZE Uncategorized A send above max_payload(). Raised locally, before the syscall. Unmatchable by kind.
EPIPE BrokenPipe The daemon went away: it exited, the node shut down, or its own read failed. Never a peer finishing.
ETIMEDOUT TimedOut The five-second setup deadline, and nothing else on this surface.
ECONNREFUSED ConnectionRefused The node is shutting down, a debug command is disabled, nothing is accepting on the socket path, or the reply carried an errno name this client has no row for.

The last row is the catch-all, so ECONNREFUSED is the one code that does not narrow the cause much. The daemon sends errno names rather than numbers, because a number belongs to the platform the program was built for and the daemon is not it. An unknown name is read as ECONNREFUSED for the same reason a missing one is. The daemon also sends a human-readable message, and the client discards it, so no caller can come to depend on prose.

The errors that carry no errno

Three shapes have no raw_os_error() at all.

ErrorKind::InvalidData and io::Error::other both mean this daemon is not speaking the protocol: a reply or an arrival that is not JSON, an unknown or missing status, a missing or malformed field, a port outside u16, a reply carrying no descriptor, a descriptor arriving on a read that completed no line, more than 64 KiB with no newline, or a truncated control message. None is worth retrying, and none is a condition a correct daemon produces.

ErrorKind::NotFound on a setup call means no daemon is running, or the native API is disabled, which is the default.

What is worth retrying

EMFILE and EADDRNOTAVAIL from an exhausted ephemeral range are load conditions and may clear. A retry with a backoff is reasonable, and a program that retries without one contributes to the exhaustion. ETIMEDOUT and EPIPE mean the daemon is unhealthy or gone, so the useful retry is the whole setup sequence and not the one call. EADDRINUSE, EADDRNOTAVAIL from a reserved tier, EINVAL and EMSGSIZE are decisions about the arguments and produce the same answer every time.

Where data disappears

The API is datagram-unreliable. Nothing on this surface confirms delivery. There is no acknowledgement, no retransmission, no ordering guarantee and no flow control between the two ends. A program that needs confirmation gets it from the peer, in the payload.

Four places lose data with nothing reported to the client.

A full per-flow queue. Inbound datagrams beyond pending_per_flow are dropped with a trace log and no client-visible signal. A program that stops reading a flow loses datagrams and is never told.

A listener that does not accept fast enough. Whole arriving flows are discarded, counted inside the daemon as ListenerNotReading. The bound on how many flows a stalled listener holds is not the backlog: the send buffer on a listener's pair is sized generously, because the approximation must err toward accepting an arrival a program would have read. A program that binds a listener and stops reading it accumulates flows on the order of max_flows rather than of backlog.

An outbound datagram sent before a session exists. The first send on a new flow almost always takes this path, because connect contacts no peer and leaves no FSP session behind it. The node holds the datagram and starts a handshake. What holds it is bounded twice: at node.session.pending_packets_per_dest datagrams for one destination, past which a further datagram evicts the oldest one held, and at node.session.pending_max_destinations destinations, past which a new destination's datagram is dropped outright. The defaults are 16 and 256. Neither eviction reaches the caller.

An outbound datagram after the MTU has fallen. max_payload() is a snapshot taken at setup. The daemon re-checks each outbound datagram against the node's current limit and drops it silently if the transport MTU has since fallen.

The drop causes, and what they mean

An inbound datagram can be refused for eight reasons, which render as seven texts: a pending flow's full queue and an established flow's are distinct to a counter and alike to a client. The texts are what DropReason::as_str produces; the counter names are what fipsctl stats metrics reports under native.

Text Counter Condition
no listener or flow on that port drop_no_port Nothing holds the destination port.
listener backlog full drop_backlog_full A listener holds the port but will not hold another pending flow.
node flow ceiling reached drop_too_many_flows The node is at max_flows.
queue full drop_pending_queue_full, drop_flow_queue_full A flow's queue is full, whether it is pending or established with a client that is not reading. Two counters, one text.
arrival queue full drop_arrival_queue_full The daemon's own queue to a listener's task is full.
listener not reading arrivals drop_listener_not_reading A listener's client is not reading its descriptor, so the arrival could not be written to it.
listener closed drop_listener_gone A listener's client closed its descriptor between the arrival being taken off the queue and being written.

None of these reaches a client for real traffic. There is no drop event and no reply reports one. The texts are visible only through the debug arrive command, which reports dropped: <text> as its outcome; the counters are readable at any time through the control socket.

drop_oversize is a ninth counter and is not in this table, because it is not a dispatch refusal: it counts an outbound datagram the daemon discarded when the transport MTU had fallen below it, which is the fourth case above.

No framing, in either direction

There is no header and no length prefix. One send is one datagram and one receive is one datagram. A program that adds its own length prefix on top of a message-boundary-preserving transport is paying for something it already has.

Seeing what a node holds

Neither type reports the node's state, and the client module exposes no statistics. fipsctl show native-flows is the only way to see what a node holds, and fipsctl stats metrics under native carries the per-cause drop counters that no program is told about. See cli-fipsctl.md and control-socket.md.

What a daemon restart costs

Every flow and every listener ends when the daemon does, and descriptors do not survive it. The daemon's halves close with the process, so a program reading one gets EPIPE and a program writing one gets the same. There is no reconnection and no resumption: a program that must survive a restart re-runs its setup calls and gets new descriptors.

Datagrams already sent and not yet forwarded are lost. At an orderly flow close there is nothing in flight, because a flow's reader forwards what the client wrote and only then notices the flow is over. At daemon exit every reader stops at once with no such notice. The window is small and nothing bounds it.

A program that needs to know its last datagram reached a peer needs an acknowledgement from that peer. Neither this API nor the wire beneath it has one to offer.

The line protocol

The Rust client hides all of this. It is documented because a client in another language has to implement it. For the obligations such a client carries, see ../how-to/write-a-native-api-client.md.

Framing

The encoding is the control socket's, so a client that speaks one speaks both. One JSON object per line, terminated by \n.

A request is:

{"command": "connect", "params": {"peer": "npub1...", "remote_port": 4242}}

Every command requires params. A request without the key is refused with command '<name>' requires params.

A command line is capped at 8192 bytes. A longer one ends the connection with native API command too large. That is the only condition that ends the connection instead of producing a reply. The cap is applied whether or not the chunk in hand holds the newline, so a well formed 9000-byte line ends the connection exactly as a runaway one with no newline does. The connection is dropped with nothing written back, so a client sees end of file and never a message.

Two kinds of line come back, and only two. A success:

{"status": "ok", "data": {}}

And a refusal, which carries the code a client acts on and prose it must not match against:

{"status": "error", "data": {"errno": "EADDRNOTAVAIL"},
 "message": "port 256 is reserved for FIPS standard services"}

There is no third kind. No event is pushed on this connection and nothing arrives on it unsolicited, so a reader that sends a command and takes the next complete line as its answer is correct. A refusal is a normal reply line and never drops the connection. A reply carrying no errno at all is read as ECONNREFUSED, which covers a daemon older than the field.

The arrival message

An arriving flow is not a line. It is one SOCK_SEQPACKET message on the listener's own descriptor, with no trailing newline: the message boundary is the framing, and a newline would offer a client a second framing to rely on.

Its seven fields are flow_id, peer and node as npubs, local_port, remote_port, max_payload and held. The node field is carried so an accepted flow can answer local_addr without consulting the listener that produced it. flow_id is the same identifier a connect reply carries and the name the debug commands take.

held is the one field a client cannot infer. It counts the datagrams the daemon has already written onto the flow's descriptor before this message, because the hand-off writes every held datagram first and the arrival last. It says exactly how many recv calls a client may make on a newly accepted flow without blocking. The Rust client reads neither held nor flow_id, because neither names anything a caller can name. A client in another language may ignore both on the same reasoning; it cannot ignore that they are there.

Passing the descriptor

The daemon builds an AF_UNIX SOCK_SEQPACKET socket pair with SOCK_CLOEXEC, keeps one half, and sends the other over SCM_RIGHTS in the ancillary data of the same sendmsg that carries the reply line. It makes its own half non-blocking and leaves the client's half blocking. SOCK_SEQPACKET is what preserves message boundaries in both directions, which is why the payload needs no framing.

A refused connect leaves the port free: the socket pair is built before the port is claimed, so a failure to build it needs no rollback.

The setup call

One AF_UNIX SOCK_STREAM connection to the socket path, one command line written, one reply line read. Replies come back in command order, and several commands are permitted on one connection, so a client may pipeline. The shipped Rust client does not: it opens a connection per setup call and drops it before returning.

The connection owns nothing. Closing it releases no flow and no listener, and a descriptor kept across the close keeps working. What owns the flow is the descriptor.

Command reference

Five commands. Two are the interface; three are debug scaffolding, off by default.

Command Carries FD What it does
connect yes Open a flow to a peer named by npub. Returns the flow's descriptor.
listen yes Hold a local port. Returns the listener's descriptor.
stats no Debug. Report what the daemon received on a flow.
inject no Debug. Write bytes into a flow from the daemon's side.
arrive no Debug. Dispatch a datagram as though a peer had sent it.

There is no close command, no accept command, no reject command, and no command to enumerate flows. Closing a descriptor does the first three, and fipsctl show native-flows does the fourth from the control socket.

connect

Parameter Type Meaning
peer string The far end, as an npub.
remote_port u16 The far end's port. Required.
local_port u16 Optional. Absent, null and 0 all mean an ephemeral port.

Reply data: flow_id, local_port, remote_port, peer (the daemon's own re-encode of the key, not an echo of what the caller wrote), node (this node's own npub), and max_payload. Carries the flow's descriptor.

Refusals, in the order they are decided, which is what lets a client read an EADDRNOTAVAIL as a tier refusal rather than an exhaustion:

  1. EADDRNOTAVAIL: the port is in a reserved tier, either port. Decided before any node state is touched.
  2. EINVAL: the npub does not decode.
  3. EMFILE: the socket pair could not be built, or the node holds its maximum flows.
  4. EADDRINUSE: the named local port is held, or a flow to that peer between those two ports already exists.
  5. EADDRNOTAVAIL: no ephemeral port is free.
  6. ECONNREFUSED: the node is shutting down.

listen

One parameter, local_port (u16), where absent, null and 0 all mean an ephemeral port. Reply data: local_port, node, backlog. Carries the listener's descriptor, and a reply that carried none is a daemon that is not this one.

backlog reports the depth the daemon will hold for this listener, so an operator can size a client's reader against it. Nothing on the Rust surface takes a depth as a parameter or exposes the reported one.

Refusals: EADDRNOTAVAIL for a reserved tier or an exhausted ephemeral range; EADDRINUSE when the port is held, whether by another listener or by a connected flow; EMFILE when the socket pair could not be built; ECONNREFUSED when the node is shutting down.

A flow announced on a listener but never taken is discarded after five seconds. This is not an accept timeout and a client cannot reach it: the window it bounds is a hand-off between two tasks inside the daemon, not a client's round trip.

The debug commands

stats, inject and arrive exist so the daemon's own checks can drive the receive and dispatch paths without a peer. They are answered only where node.native_api.debug_commands is set, which is off by default and which no packaged node sets. A node with the key unset refuses all three by name, with ECONNREFUSED and a message naming the key that would admit them, so a client can tell "this node will not" from "this build cannot".

Do not build a client on them.

stats takes flow_id and reports flow_id, local_port, rx_datagrams, rx_bytes and closed. The counters are the daemon's view of what the client wrote into the descriptor, independent of whether any of it then reached a peer.

inject takes flow_id, data (a hex string) and an optional repeat (default 1, maximum 64). It writes repeat separate datagrams of those bytes onto the flow's descriptor from the daemon's side. The flow table it names into is the node's, not the connection's, so a caller that can reach this command can write into any flow on the node.

arrive takes peer (npub), src_port, dst_port and data (hex), and drives the same delivery decision the real receive path uses. It replies ok whenever the node answers at all:

{"status": "ok", "data": {"outcome": "announced", "flow_id": 9}}

outcome is delivered, announced, held, or dropped: <cause>. flow_id is non-null only for announced and held. The peer's key is decoded from the npub the caller named, which makes it client-asserted rather than authenticated. Port tiers are not checked here.

Compiled-in bounds

Three bounds are not configurable: a command line of 8192 bytes, a per-arrival send-buffer allowance of 4096 bytes on a listener's pair, and a repeat of 64 on the debug inject. The daemon reads each descriptor with a 65535-byte buffer, which is above anything the wire carries.

pending_per_flow has a compiled ceiling of 64, checked when the configuration loads rather than at the first arrival: the whole held batch is written onto a socket pair whose client half has not been sent yet, so a larger value could leave a listener's task with a write it cannot complete. Both it and backlog have a floor of 1. A backlog of zero admits no flow at all. A pending_per_flow of zero is worse, because the arrival is announced and the datagram that caused it is then refused, so a peer's opening message vanishes with no refusal a client or an operator can see.

See also