fips/docs/how-to/write-a-native-api-client.md

# Write a Native API Client in Another Language

**Goal:** speak the native datagram API's line protocol directly, from C,
Python, Go or anything else, without the Rust client module.

Everything here is something the shipped Rust library already does. It is
written out so an author working where there is no such library knows what they
are reproducing. **Each of these was a real defect before it was a rule, and
each fails intermittently rather than outright.**

If you are writing Rust, you do not need this guide. Use
`fips::native::client` and see
[use-the-native-datagram-api.md](use-the-native-datagram-api.md).

For the protocol itself — the framing, the reply shapes, the commands and their
refusals — see
[../reference/native-api.md](../reference/native-api.md#the-line-protocol).

## Step 1: Read the setup connection with recvmsg, never with a buffered reader

**Every read on the setup connection is a `recvmsg` with an ancillary buffer.**
A plain `read` consumes a descriptor-bearing message's bytes with no control
buffer, and the kernel then closes the descriptor rather than queueing it. The
reply looks perfectly correct and the flow is silently gone.

This applies to both setup commands: a `listen` reply carries a descriptor as
much as a `connect` reply does. In any language it means no buffered reader, no
`BufReader`, no `readline`, and no library that wraps the socket in a stream
abstraction. Keep the line buffering in your own code, over `recvmsg`.

Read a listener's own descriptor the same way, for the ancillary data and the
close-on-exec flag. One message there is one arrival carrying exactly its own
descriptor, so there is nothing to associate.

## Step 2: Attach a descriptor to the last complete line of its read

**A descriptor belongs to the last complete line of the read that carried it,
never to the next line the reader assembles.** A `recvmsg` returning ancillary
data ends exactly at the end of the `sendmsg` that carried it, but it may begin
with any amount of data written before it.

A client that sends one command per connection reads one line and cannot hit
this. A client that pipelines two setup commands on one connection can: the
first reply and the second, descriptor-bearing reply arrive as one read, and a
reader that attached the descriptor to the first would hand the flow to the
wrong caller.

Two corollaries:

- A read that carries a descriptor and completes no line must be **reported**
  rather than held. Holding it means guessing which later line it belongs to.
- A descriptor that arrives with a line you are going to discard must still be
  **closed**, or the flow leaks.

Neither can happen while the daemon writes exactly one whole line per `sendmsg`
and treats a short write as an error, which it does. That is an invariant of
two programs, though, not of the socket type.

## Step 3: Decide end of file from POLLHUP and an empty queue

**An empty datagram and a closed peer both produce a zero-byte read, and
`MSG_EOR` does not tell them apart.** On Linux 6.8, `recvmsg` on an `AF_UNIX`
`SOCK_SEQPACKET` socket returns `msg_flags == 0` for a normal message, an empty
message and end of file alike, so the flag carries no information.

`POLLHUP` narrows the question and does not answer it. A live peer never sets
it, so a zero-byte read without `POLLHUP` is an empty datagram and nothing else.
A closed peer does set it, **and it latches while messages are still queued**.
Measured on Linux 6.8: a socket holding one empty datagram from a peer that has
since closed reports exactly what a drained socket reports, in `revents`, in
`FIONREAD`, under `MSG_PEEK` and in the `recvmsg` return alike.

Poll with an events mask of **`POLLIN`**. Linux reports `POLLHUP` in `revents`
whether or not it was requested, but a poll that requests nothing registers no
filter on Darwin and returns zero events for a peer that has in fact closed.
`POLLIN` costs nothing on either platform, because you mask the result to
`POLLHUP` regardless. The poll runs only on the zero-byte path and does not
block.

**When `POLLHUP` is set, ask whether anything is still queued before you call it
end of file.** `ioctl(fd, FIONREAD, &n)` reports the bytes the receive queue
holds. A non-zero `n` proves a further message is waiting, so the zero-byte read
you just took was an empty datagram: deliver it and read on. This is the case
that costs a real payload if you get it wrong. A client that sends an empty
datagram, then a message, then closes, leaves both queued, and a reader that
trusts `POLLHUP` alone discards the message.

**One case has no answer on Linux, where the pair is `SOCK_SEQPACKET`, and you
should design around it rather than solve it.** A zero-length datagram that is
the last message before a close is indistinguishable from the close: reading it
drains the queue, and `FIONREAD` then reports zero because a zero-length message
contributes no bytes. If your protocol gives a zero-length payload a meaning, do
not send it as a zero-length socket message. Carry a one-byte discriminator, and
keep the zero-byte read for end of file alone. On macOS and FreeBSD the pair is
`SOCK_DGRAM`: the empty datagram reads as zero bytes, and the close is reported
by the read after it.

Both directions of the mistake are real. Reading an empty datagram as a close
lets a peer tear down a live flow by sending nothing, and presents as a
spurious disconnect. Reading a close as an empty datagram leaves the caller
spinning on a dead flow.

Note what a close here means: the daemon went away, never a peer finishing.

## Step 4: Send with MSG_NOSIGNAL

A datagram written to a flow whose daemon half has gone, or a command written
to a daemon that has exited, raises `SIGPIPE`, whose default disposition kills
the process.

Rust ignores the signal at startup, and CPython sets it to `SIG_IGN`, so a
program in either language sees `EPIPE`. **A C or C++ client that has not
changed the disposition simply dies.** The daemon and the shipped client pass
the flag on every send.

## Step 5: Set a deadline on the setup socket

Set `SO_RCVTIMEO` on the setup connection and rewrite the resulting would-block
into `ETIMEDOUT`. The shipped client uses five seconds.

Without it, a daemon that accepted your connection and then stopped answering
blocks the setup call forever. This is also what keeps `ETIMEDOUT` to exactly
one producer on the surface, which is what lets a caller read it.

## Step 6: Keep descriptor hygiene

Five rules. Each one leaks a flow or loses one when broken.

**Request close-on-exec** with `MSG_CMSG_CLOEXEC` on the `recvmsg`, rather than
setting it afterwards. Without it the descriptor survives an `exec` into a
child, the child's reference holds the flow open after this process closes its
own, and the flow keeps its slot against the node's ceiling until the child
exits.

**Walk the whole control buffer**, not only the first header. Close extra
descriptors rather than dropping them on the floor.

**Check for truncation after taking the descriptors, not before.** A
`MSG_CTRUNC` test that returns early leaks whatever did arrive.

**Lift the descriptor out of an arrival you cannot parse** before discarding
the message. Refusing a flow is closing its descriptor; discarding the message
without taking it leaks the flow instead.

**Bound the partial line.** A daemon that stopped sending newlines would
otherwise grow your buffer without end. The shipped client caps it at 64 KiB,
well above any reply.

## Step 7: Read the errno name, never the message

The refusal's `data.errno` is the contract. The `message` is for an operator
reading a log.

A client that matched on English would break on a wording change. The shipped
client discards the message entirely so that no caller can come to depend on
it, and the daemon's own match over its error types is exhaustive precisely so
a new refusal cannot reach a client without a code.

A reply carrying no `errno` at all should be read as `ECONNREFUSED`, which
covers a daemon older than the field. The errno table is in
[../reference/native-api.md](../reference/native-api.md#the-errno-table).

## Step 8: Size the receive buffer, and add no framing

Size every receive buffer at the flow's `max_payload`, read from the reply and
never computed.

`SOCK_SEQPACKET` truncates a longer datagram, discards the remainder and
reports success. It is detectable: `recvmsg` sets `MSG_TRUNC` in `msg_flags`
when it dropped part of a message. A plain `recv` discards `msg_flags` and so
sees none of it, which is where the belief that truncation is silent comes
from. Test `MSG_TRUNC` as well, and a stale or misread `max_payload` is caught
rather than quietly corrupting a payload.

Do not add framing. There is no header and no length prefix in either
direction. One send is one datagram.

## Step 9: Decide when a flow is over, because nothing else will

Everything above is the library's job. **The termination condition is not**, in
any language.

The v1 wire carries no half-close. Nothing peer-driven closes the daemon's half
of a live flow, so a loop that reads until the flow ends does not terminate.
Your program decides when a flow is over, or nothing does.

The two shapes that work are a bounded exchange, where the program serves a
known number of datagrams per flow and then drops it, and an idle deadline,
where the program sets a read timeout and treats its expiry as the end. A
server that reads "until the flow closes" holds a thread per peer forever and
holds every flow against the node's `max_flows` ceiling.

## Verify it

The repository's test harness drives the line protocol from Python and is the
closest thing to a second implementation:

- `testing/native-api/client.py` — a thin RPC client that runs a script of
  steps over one connection and checks the replies. It implements every rule
  above.
- `testing/native-api/control.py` — reads `show_native_flows` back over the
  control socket while a flow is open.

Check what the node actually holds with `fipsctl show native-flows`, and read
the per-cause drop counters with `fipsctl stats metrics` under `native`.

## See also

- [use-the-native-datagram-api.md](use-the-native-datagram-api.md) — the Rust
  path, where none of this is your problem
- [../reference/native-api.md](../reference/native-api.md) — the surface, the
  line protocol, the command reference and the errno table
- [../reference/control-socket.md](../reference/control-socket.md) — the same
  line framing, for the control socket