mirror of
https://github.com/jmcorgan/fips.git
synced 2026-10-05 11:08:25 +00:00
Johnathan Corgan
8b396b662d
A native API flow's descriptor reaches the client inside a message: an arrival for a flow a listener accepts, or a connect or listen reply. The daemon closed its own copy once the message was written, so until the client read it, the message was the only reference to the socket. xnu's descriptor collector flushes a socket in that state, and the client then receives a flow that reads as end of file with the datagrams the daemon held for it gone. That is the intermittent macOS failure of the two listener tests. A listener now keeps the daemon's copy of each flow it hands over until the client's first write on the flow, the listener's close, or the flow's end, on every platform the listener builds on. The flow is recorded before its reader starts, so a write already queued cannot race the record. The connection's serving loop, now a method on the connection so a test can run it over a real socket, keeps the copy sent in a connect or listen reply until the client's next command on that connection or the connection's end of file. A client sends its next command only after reading the reply, so either event means the descriptor has left the message. The shipped client closes the connection as soon as it has the reply, so it sees no change. The cost is accepted and documented: a flow a client accepts and closes without ever writing stays open, holding its port and a flow slot, until the listener closes, so a server that refuses flows by dropping them pays for each one until then; a client speaking the protocol directly that leaves its setup connection open sees a flow or listener it closes stay open until its next command or the connection's close. The reference and how-to pages, the client rustdoc on FipsStream, FipsListener and accept, and the design note say so, and the security reference records that a remote peer opening flows from many source ports to such a server can exhaust the node-wide max_flows ceiling. Tests cover an arrival surviving a provoked collection (deterministic on macOS), a held flow outliving its dropped descriptor until the listener closes, a client's write releasing it, a flow its client still holds working after the listener has closed and let its copy go, and a reply's copy kept until the next command and let go when the connection ends. The native API harness asserts the new lifetime of a refused flow. On macOS and FreeBSD the daemon notices a client's close only when a reader retries its read, up to a quarter second later, so the test helpers that wait for a close (forgotten, rebind, settle_closed) retry for up to five seconds by the clock rather than for a count of yields, and still_open waits two retry intervals there before asserting a flow is still open.
223 lines
11 KiB
Markdown
223 lines
11 KiB
Markdown
# Write a Native API Client in Another Language
|
|
|
|
**Goal:** speak the native datagram API's line protocol directly, from C,
|
|
Python, Go or anything else, without the Rust client module.
|
|
|
|
Everything here is something the shipped Rust library already does. It is
|
|
written out so an author working where there is no such library knows what they
|
|
are reproducing. **Each of these was a real defect before it was a rule, and
|
|
each fails intermittently rather than outright.**
|
|
|
|
If you are writing Rust, you do not need this guide. Use
|
|
`fips::native::client` and see
|
|
[use-the-native-datagram-api.md](use-the-native-datagram-api.md).
|
|
|
|
For the protocol itself — the framing, the reply shapes, the commands and their
|
|
refusals — see
|
|
[../reference/native-api.md](../reference/native-api.md#the-line-protocol).
|
|
|
|
## Step 1: Read the setup connection with recvmsg, never with a buffered reader
|
|
|
|
**Every read on the setup connection is a `recvmsg` with an ancillary buffer.**
|
|
A plain `read` consumes a descriptor-bearing message's bytes with no control
|
|
buffer, and the kernel then closes the descriptor rather than queueing it. The
|
|
reply looks perfectly correct and the flow is silently gone.
|
|
|
|
This applies to both setup commands: a `listen` reply carries a descriptor as
|
|
much as a `connect` reply does. In any language it means no buffered reader, no
|
|
`BufReader`, no `readline`, and no library that wraps the socket in a stream
|
|
abstraction. Keep the line buffering in your own code, over `recvmsg`.
|
|
|
|
Read a listener's own descriptor the same way, for the ancillary data and the
|
|
close-on-exec flag. One message there is one arrival carrying exactly its own
|
|
descriptor, so there is nothing to associate.
|
|
|
|
## Step 2: Attach a descriptor to the last complete line of its read
|
|
|
|
**A descriptor belongs to the last complete line of the read that carried it,
|
|
never to the next line the reader assembles.** A `recvmsg` returning ancillary
|
|
data ends exactly at the end of the `sendmsg` that carried it, but it may begin
|
|
with any amount of data written before it.
|
|
|
|
A client that sends one command per connection reads one line and cannot hit
|
|
this. A client that pipelines two setup commands on one connection can: the
|
|
first reply and the second, descriptor-bearing reply arrive as one read, and a
|
|
reader that attached the descriptor to the first would hand the flow to the
|
|
wrong caller.
|
|
|
|
Two corollaries:
|
|
|
|
- A read that carries a descriptor and completes no line must be **reported**
|
|
rather than held. Holding it means guessing which later line it belongs to.
|
|
- A descriptor that arrives with a line you are going to discard must still be
|
|
**closed**, or the flow leaks.
|
|
|
|
Neither can happen while the daemon writes exactly one whole line per `sendmsg`
|
|
and treats a short write as an error, which it does. That is an invariant of
|
|
two programs, though, not of the socket type.
|
|
|
|
## Step 3: Decide end of file from POLLHUP and an empty queue
|
|
|
|
**An empty datagram and a closed peer both produce a zero-byte read, and
|
|
`MSG_EOR` does not tell them apart.** On Linux 6.8, `recvmsg` on an `AF_UNIX`
|
|
`SOCK_SEQPACKET` socket returns `msg_flags == 0` for a normal message, an empty
|
|
message and end of file alike, so the flag carries no information.
|
|
|
|
`POLLHUP` narrows the question and does not answer it. A live peer never sets
|
|
it, so a zero-byte read without `POLLHUP` is an empty datagram and nothing else.
|
|
A closed peer does set it, **and it latches while messages are still queued**.
|
|
Measured on Linux 6.8: a socket holding one empty datagram from a peer that has
|
|
since closed reports exactly what a drained socket reports, in `revents`, in
|
|
`FIONREAD`, under `MSG_PEEK` and in the `recvmsg` return alike.
|
|
|
|
Poll with an events mask of **`POLLIN`**. Linux reports `POLLHUP` in `revents`
|
|
whether or not it was requested, but a poll that requests nothing registers no
|
|
filter on Darwin and returns zero events for a peer that has in fact closed.
|
|
`POLLIN` costs nothing on either platform, because you mask the result to
|
|
`POLLHUP` regardless. The poll runs only on the zero-byte path and does not
|
|
block.
|
|
|
|
**When `POLLHUP` is set, ask whether anything is still queued before you call it
|
|
end of file.** `ioctl(fd, FIONREAD, &n)` reports the bytes the receive queue
|
|
holds. A non-zero `n` proves a further message is waiting, so the zero-byte read
|
|
you just took was an empty datagram: deliver it and read on. This is the case
|
|
that costs a real payload if you get it wrong. A client that sends an empty
|
|
datagram, then a message, then closes, leaves both queued, and a reader that
|
|
trusts `POLLHUP` alone discards the message.
|
|
|
|
**One case has no answer on Linux, where the pair is `SOCK_SEQPACKET`, and you
|
|
should design around it rather than solve it.** A zero-length datagram that is
|
|
the last message before a close is indistinguishable from the close: reading it
|
|
drains the queue, and `FIONREAD` then reports zero because a zero-length message
|
|
contributes no bytes. If your protocol gives a zero-length payload a meaning, do
|
|
not send it as a zero-length socket message. Carry a one-byte discriminator, and
|
|
keep the zero-byte read for end of file alone. On macOS and FreeBSD the pair is
|
|
`SOCK_DGRAM`: the empty datagram reads as zero bytes, and the close is reported
|
|
by the read after it.
|
|
|
|
Both directions of the mistake are real. Reading an empty datagram as a close
|
|
lets a peer tear down a live flow by sending nothing, and presents as a
|
|
spurious disconnect. Reading a close as an empty datagram leaves the caller
|
|
spinning on a dead flow.
|
|
|
|
Note what a close here means: the daemon went away, never a peer finishing.
|
|
|
|
## Step 4: Send with MSG_NOSIGNAL
|
|
|
|
A datagram written to a flow whose daemon half has gone, or a command written
|
|
to a daemon that has exited, raises `SIGPIPE`, whose default disposition kills
|
|
the process.
|
|
|
|
Rust ignores the signal at startup, and CPython sets it to `SIG_IGN`, so a
|
|
program in either language sees `EPIPE`. **A C or C++ client that has not
|
|
changed the disposition simply dies.** The daemon and the shipped client pass
|
|
the flag on every send.
|
|
|
|
## Step 5: Set a deadline on the setup socket
|
|
|
|
Set `SO_RCVTIMEO` on the setup connection and rewrite the resulting would-block
|
|
into `ETIMEDOUT`. The shipped client uses five seconds.
|
|
|
|
Without it, a daemon that accepted your connection and then stopped answering
|
|
blocks the setup call forever. This is also what keeps `ETIMEDOUT` to exactly
|
|
one producer on the surface, which is what lets a caller read it.
|
|
|
|
## Step 6: Keep descriptor hygiene
|
|
|
|
Six rules. Each one leaks a flow or loses one when broken.
|
|
|
|
**Request close-on-exec** with `MSG_CMSG_CLOEXEC` on the `recvmsg`, rather than
|
|
setting it afterwards. Without it the descriptor survives an `exec` into a
|
|
child, the child's reference holds the flow open after this process closes its
|
|
own, and the flow keeps its slot against the node's ceiling until the child
|
|
exits.
|
|
|
|
**Walk the whole control buffer**, not only the first header. Close extra
|
|
descriptors rather than dropping them on the floor.
|
|
|
|
**Check for truncation after taking the descriptors, not before.** A
|
|
`MSG_CTRUNC` test that returns early leaks whatever did arrive.
|
|
|
|
**Lift the descriptor out of an arrival you cannot parse** before discarding
|
|
the message. Refusing a flow is closing its descriptor; discarding the message
|
|
without taking it leaks the flow instead. A refused flow ends only when the
|
|
listener closes, though, unless you wrote on it first: the daemon keeps its
|
|
own copy of an accepted flow's descriptor until your first write or the
|
|
listener's close.
|
|
|
|
**Close the setup connection once you have the reply.** The daemon keeps its
|
|
own copy of the descriptor in its last reply until your next command on that
|
|
connection or the connection's close. A flow or listener you close while the
|
|
connection sits idle stays open until one of those happens.
|
|
|
|
**Bound the partial line.** A daemon that stopped sending newlines would
|
|
otherwise grow your buffer without end. The shipped client caps it at 64 KiB,
|
|
well above any reply.
|
|
|
|
## Step 7: Read the errno name, never the message
|
|
|
|
The refusal's `data.errno` is the contract. The `message` is for an operator
|
|
reading a log.
|
|
|
|
A client that matched on English would break on a wording change. The shipped
|
|
client discards the message entirely so that no caller can come to depend on
|
|
it, and the daemon's own match over its error types is exhaustive precisely so
|
|
a new refusal cannot reach a client without a code.
|
|
|
|
A reply carrying no `errno` at all should be read as `ECONNREFUSED`, which
|
|
covers a daemon older than the field. The errno table is in
|
|
[../reference/native-api.md](../reference/native-api.md#the-errno-table).
|
|
|
|
## Step 8: Size the receive buffer, and add no framing
|
|
|
|
Size every receive buffer at the flow's `max_payload`, read from the reply and
|
|
never computed.
|
|
|
|
`SOCK_SEQPACKET` truncates a longer datagram, discards the remainder and
|
|
reports success. It is detectable: `recvmsg` sets `MSG_TRUNC` in `msg_flags`
|
|
when it dropped part of a message. A plain `recv` discards `msg_flags` and so
|
|
sees none of it, which is where the belief that truncation is silent comes
|
|
from. Test `MSG_TRUNC` as well, and a stale or misread `max_payload` is caught
|
|
rather than quietly corrupting a payload.
|
|
|
|
Do not add framing. There is no header and no length prefix in either
|
|
direction. One send is one datagram.
|
|
|
|
## Step 9: Decide when a flow is over, because nothing else will
|
|
|
|
Everything above is the library's job. **The termination condition is not**, in
|
|
any language.
|
|
|
|
The v1 wire carries no half-close. Nothing peer-driven closes the daemon's half
|
|
of a live flow, so a loop that reads until the flow ends does not terminate.
|
|
Your program decides when a flow is over, or nothing does.
|
|
|
|
The two shapes that work are a bounded exchange, where the program serves a
|
|
known number of datagrams per flow and then drops it, and an idle deadline,
|
|
where the program sets a read timeout and treats its expiry as the end. A
|
|
server that reads "until the flow closes" holds a thread per peer forever and
|
|
holds every flow against the node's `max_flows` ceiling.
|
|
|
|
## Verify it
|
|
|
|
The repository's test harness drives the line protocol from Python and is the
|
|
closest thing to a second implementation:
|
|
|
|
- `testing/native-api/client.py` — a thin RPC client that runs a script of
|
|
steps over one connection and checks the replies. It implements every rule
|
|
above.
|
|
- `testing/native-api/control.py` — reads `show_native_flows` back over the
|
|
control socket while a flow is open.
|
|
|
|
Check what the node actually holds with `fipsctl show native-flows`, and read
|
|
the per-cause drop counters with `fipsctl stats metrics` under `native`.
|
|
|
|
## See also
|
|
|
|
- [use-the-native-datagram-api.md](use-the-native-datagram-api.md) — the Rust
|
|
path, where none of this is your problem
|
|
- [../reference/native-api.md](../reference/native-api.md) — the surface, the
|
|
line protocol, the command reference and the errno table
|
|
- [../reference/control-socket.md](../reference/control-socket.md) — the same
|
|
line framing, for the control socket
|