laantungir/fips
1
0
Fork 0
mirror of https://github.com/jmcorgan/fips.git synced 2026-10-05 19:18:25 +00:00
Code Issues Packages Projects Releases Wiki Activity
Files
f466ecdf810b0767ff10b053671ba27b30172761
fips/testing/native-api/README.md
T
Johnathan Corgan 3a789370b9 Add an experimental native datagram API addressed by public key 
A client process opens a flow to a peer's public key on a chosen port and
sends and receives datagrams on a file descriptor the daemon hands it. No
IPv6 emulation, no TUN device, no DNS: a datagram travels from key to key.
The feature is off by default and is not a stable interface.

The wire needs no change and gets none. Every FSP data packet has carried a
port pair inside its AEAD envelope since v0.2.0, and port 256 is simply the
IPv6 shim. What was missing was a way for a program to ask for a port of its
own and be handed the traffic.

Addressing is the part worth reading twice, because the obvious design is
wrong. The x-only public key is the address. An npub is that key written in
bech32, so converting between them is a local encoding rather than a lookup
or a name service. The 16-byte node address that travels on the wire is the
first half of a SHA-256 of the key: it is a truncated hash, it does not
invert, and it appears nowhere a client can see. An earlier iteration of this
work reported a peer by that hash and could supply a key only sometimes,
which is what treating a wire identifier as an identity produces.

An accepted flow therefore always knows its peer. The key is captured where
the peer is authenticated rather than looked up when a report is rendered:
every inbound datagram passes one call site inside a handler that refuses
anything whose session is not established, and the responder has already
rejected the session unless the claimed address derives from the key it
proved. Reaching for the identity cache instead gives a best-effort answer
from a structure that evicts.

A listener is a descriptor. The daemon writes one message per arrival to it,
carrying the new flow's descriptor and the peer's address, so poll, select
and epoll work on a listener and accepting is a recvmsg. That is what lets
the API be used from a program that already has an event loop, which a
command-and-reply listener could not support: an arrival could not be waited
on beside anything else. There is no accept command and no reject command.
Refusing a flow is closing the descriptor you were handed.

The Rust surface mirrors std::net. FipsStream::connect, FipsListener::bind,
incoming, accept, io::Result and an errno mapping rather than a bespoke
error type. An address is given as an npub, as a key, or as a pair, through
one parameter, the way ToSocketAddrs takes several spellings of one thing.
Each type holds its descriptor and copies of what setup told it and nothing
else, so a stream that outlives its setup connection is not representable.

set_nonblocking, AsFd and the four deadline methods carry the names and
signatures std::net uses for the same jobs. They were asked for by a user
integrating the API with tokio: AsyncFd requires a non-blocking descriptor,
and anything receiving from a peer needs a bounded wait. AsFd is the better
of the two descriptor accessors, because the borrow cannot outlive the value
that owns the descriptor, so a reactor cannot hold a registration for a
descriptor that has since been closed and its number reused by the next
open. The non-blocking flag is read, modified and written back rather than
assigned, since the flag word carries more than that one bit and a caller may
have set O_ASYNC. A zero timeout is refused with EINVAL, because the kernel
reads a zero timeval as "wait for ever", which inverts what a caller passing
zero means; std::net refuses it for the same reason. The two directions are
separate options and stay that way. FipsListener gets no timeout methods,
matching TcpListener: bounding an accept is set_nonblocking plus the caller's
own poll, which the reactor how-to builds. A flow taken from accept is
blocking whatever the listener was set to, because the two are separate
sockets and the daemon hands over a fresh one.

One rule has no counterpart in Berkeley sockets and a client author must know
it: the v1 wire carries no half-close, so nothing peer-driven ever closes a
flow. A server written to read until the flow ends waits for a signal that
cannot arrive, holding a thread and a flow per peer until its process exits.
A program decides its own termination, and the example serves one datagram
per flow.

The tests reach a live daemon rather than a stand-in. Every public item had a
unit test against a hand-written stand-in with canned replies, and the five
entry points a program actually calls first, connect, connect_from,
connect_at, bind and the SOCKET constant, had no coverage of any kind,
because the tests that appear to cover them build a Wire over a socket pair
and hand it to the private open and hold, so nothing ever resolved a socket
path or mapped its errors. examples/native-surface.rs walks all thirty-eight
items against a running daemon and reports the number of assertions it made.
The count is read from the recorder rather than written as a literal, and the
harness asserts the exit status, the completion marker and the count
together, so deleting an assertion fails the check rather than quietly
shrinking it. Watchdogs turn a hang into a named failure, which several of
the walked behaviours would otherwise produce. The shared Docker image is
built once for every integration leg, so the new binary is staged at all ten
places the existing one is, the interop builder included, which gets a stub
because those images exercise the wire between daemon versions and older refs
do not carry the example. The platform gating was tested rather than reasoned
about: flipping all eleven gates so the native API is excluded leaves the
crate compiling clean across the workspace, every target and the profiling
feature.

The shipped docs tree gains what only the LaTeX manual under design/ had,
which is not published with the daemon. A reference entry covers the whole
surface: addressing and the port tiers, the Berkeley mapping, every method on
FipsAddr, FipsStream, FipsListener and Incoming, the errno table, the
ceilings, the four places data disappears with nothing reported, the line
protocol and the command reference. The errno table gives names rather than
numbers, since the client maps each name onto the libc constant for the
platform it was built for and the supported platforms disagree on the
numbers. A tutorial side trip stands up two throwaway nodes on one machine,
peered over loopback UDP with no TUN and no DNS, then writes a listening
program and a connecting program against them; it needs neither the public
mesh nor root, because the native path is the one that does not go through
the IPv6 adapter. The obligations a client in another language carries are a
how-to of their own, since they are a task rather than a description:
reading the setup connection with recvmsg, associating a descriptor with the
last complete line, telling an empty datagram from a close, and six others.
Serving many peers from one poll loop is another, with the whole program,
because the straightforward listener spawns a thread per flow and that is
wrong at the node's ceiling of 256. The drop causes are a table mapping each
of the seven texts DropReason::as_str produces to the counter it increments,
with drop_oversize called out as the ninth counter that is not in the table.
What a daemon restart costs is a section of its own: every flow and listener
ends, descriptors do not survive, there is no resumption, and datagrams sent
but not yet forwarded are lost through a window nothing bounds.

A stack comparison diagram places the interface against the stack a reader
already knows: the same application over HTTP, TLS, TCP, IP and Ethernet on
one side, and over its own format, FSP, FMP and a FIPS transport on the
other, aligned so each row is one concern. The two columns are not
alternatives and are not drawn as such. An unmodified IPv6 program's packets
reach fips0, and the adapter hands each one to FSP as a payload, so the left
stack runs inside the right one; the left column ends at a fork, eth0 for the
ordinary internet and fips0 for the mesh, and an arrow leaves fips0 and runs
back up into FSP's input. The row where TCP would be is empty on purpose and
names Reliable Object Delivery, which is where that capability is expected to
land. ROD is a v2 capability, the box is dashed because none of it exists
yet, and the design entry says the part a reader needs most: nothing on the
surface anticipates it, so a program written today should assume it does not
exist. Both endpoints carry a scheme and a worked port,
https://<npub>.fips:443 and fips://<npub>:443, with a footnote saying the two
ports are not the same kind of thing, a TCP port inside the tunnel on the
left and an FSP port on the right. The fips:// form is a coinage: nothing in
the tree parses it, nothing registers the scheme, and the API takes a key and
a port as separate arguments rather than a URL. The diagram also says where
the right column stops, since FIPS over UDP still rides IP and Ethernet
beneath. It appears in fips-concepts.md and fips-ipv6-adapter.md, which were
making its argument in prose without a picture, and deliberately not in
fips-architecture.md, which already carries the OSI mapping and makes the
same point about the transport row.

The gateway's control socket moves onto the same bind policy this API uses,
which is the one change here that touches deployed behaviour: fips-gateway
now tightens /run/fips to 0750. That is unreachable under the packaged
deployment, where fips.service has already created the directory at that
mode, and reachable for a source build or a container that starts the gateway
alone.

One changelog entry under Added, describing the released state: what a
client opens and reads, the addressing and why the node address is not it,
the listener being a descriptor, the std::net shape of the Rust surface,
and the one rule Berkeley sockets have no counterpart for. It says in as
many words that the wire is unchanged.
2026-08-21 05:48:23 +00:00

183 lines
10 KiB
Markdown
Raw Blame History

# Native Datagram API Harness
Checks for the experimental native datagram API: a client process opens a flow
to a remote pubkey over a Unix socket, receives a file descriptor, and sends and
receives datagrams on it with no IPv6 emulation and no TUN device.
Design of record: `design/native-api/v1-datagram-experiment.md` in the project
workspace, which is a separate tree from this repository. The feature is off by
default and Unix only.
## Shape
The client runs in **its own container**, reaching the daemon through a
bind-mounted `/run/fips`. That is the real deployment shape — a separate process
with its own filesystem opening the socket — rather than a test speaking to the
daemon from inside the daemon's container. It also makes the access policy
observable: the host sees the socket file and reads its mode directly.
The step scripts are Python rather than Rust so a check changes without
rebuilding the daemon, which is what keeps the outside-in loop fast. That buys
speed at the cost of covering nothing of the Rust surface a caller links
against, so two compiled programs run here as well, both built on
`fips::native::client`: `examples/native-echo.rs`, which arrived with A5 and
serves the echo check, and `examples/native-surface.rs`, which walks the whole
public surface against a live daemon.
The table covers this directory and the two example programs the driver runs.
| File | What it is |
| ---- | ---------- |
| `test.sh` | The driver. Holds the scenarios and the pass/fail accounting. |
| `client.py` | A thin RPC client. Runs a script of steps over one connection and checks the replies. |
| `control.py` | A thin control-socket client, used to read `show_native_flows` back while a flow is open. |
| `node.yaml` | One node with the API enabled, no TUN, no DNS, no peers. Turns the debug commands on. |
| `node-api-off.yaml` | The same node with the API disabled, for the default-off check. |
| `node-debug-off.yaml` | The API enabled and the debug commands left at their default, for the gate check. |
| `../../examples/native-echo.rs` | The echo server for `check_echo_round_trip`. A program shape to copy. |
| `../../examples/native-surface.rs` | The surface walk for `check_surface_walk`. An assertion harness, not a shape to copy. |
## Running
```bash
cargo build --release --bins --examples # the driver refuses a stale binary
./testing/native-api/test.sh
```
`FIPS_TEST_IMAGE` is used when set, which is how `ci-local.sh` passes its
per-run image. There is deliberately no `fips-test:latest` to fall back on, so a
consumer that stops reading the variable fails loudly. Without it the driver
builds a minimal image from the locally compiled binary.
The driver **refuses to run against a stale binary**. Three binaries are built
or read from this tree, the daemon and the two examples, and each is probed
against what it is actually built from: `src/` plus `Cargo.toml` for all three,
this directory's `*.py` because the harness client is bind-mounted live rather
than built in, and, for an example, its own `.rs` and no other. A guard rooted
only at `src/` would let a stale example pass a check written about new code.
A stale binary is the worst outcome available here: the checks would run and
report a verdict about code that is not the working tree's.
An example is probed against its own source rather than all of `examples/`
because cargo does not relink `target/release/fips` when only an example
changes. Probing the daemon against every example would leave it permanently
older than a just-edited one, and the rebuild the refusal prescribes would not
clear the condition.
All three binaries must come from one profile directory. `resolve_image` refuses
a profile that holds only the daemon, which is what a bare
`cargo build --release` leaves behind.
## Increments
The API is built outside-in, and this harness grows with it. Each increment's
checks must pass before the next one starts.
| # | What it covers | State |
| - | -------------- | ----- |
| A1 | The socket, its access mode, the line framing, the command validation, the reserved-port refusals, and that the API is off by default | present |
| A2 | Descriptor passing over `SCM_RIGHTS`, message boundaries, `poll` readability, close reaching the daemon, flow isolation | present |
| A3 | Port ownership across clients, listening and accepting, the dispatch order, and reclaim when a descriptor closes | present |
| A4 | The end-to-end path between two nodes, and that a queued datagram is not IPv6-compressed | present |
| A5 | Counters, `show_native_flows` read back over the control socket, the Rust client module and echo example, and the debug-command gate | present |
| A6 | Every public item of `fips::native::client` walked against a live daemon: the five setup entry points, all eight `ToFipsAddr` spellings, the deadlines, non-blocking mode, the descriptor traits, and the payload limit | present |
**The `"stub": true` marker is gone.** It meant "this flow reaches no peer",
and after A4 every flow does. `max_payload` is now the real limit — the
transport MTU less the FIPS encapsulation and the port header, 1362 bytes on a
1472-byte transport — and the end-to-end check asserts that number rather than
accepting whatever is reported.
The tightening it existed for happened three times. A1's `connect` checks failed
the moment A2 began returning a descriptor, because `client.py` treats an
unannounced descriptor as a defect rather than ignoring it. A1's `accept` and
`reject` checks failed when A3 gave those commands a real registry, since a flow
no listener announced became a refusal. And the remaining `stub` assertions
failed at A4 when the field disappeared. Checks that had quietly kept passing
would have been worth nothing.
**The `accept` and `reject` commands are gone**, and so is the `incoming` event.
A listener now returns its own descriptor, so it is pollable, accepting is one
`recvmsg` on it that carries the arriving flow's descriptor, and refusing a flow
is closing that descriptor. The command socket carries replies only, in command
order. A step names a listener descriptor with `keep_listener` and takes flows
off it with an `accept` step; every descriptor a reply carries must be named, or
the run fails rather than dropping a flow silently.
**The backlog is no longer the bound a client sees.** It bounds arrivals the
daemon has announced and not yet wired, and the daemon drains that queue itself,
so a listener that never accepts is bounded by its send buffer and by
`node.native_api.max_flows` instead. `check_backlog_is_not_the_clients_bound`
asserts the change; the drop paths behind the new bound are covered by the
daemon's own tests, because neither is a number a shell check can produce.
**Flow identifiers are assigned by the node and keep counting up for its
lifetime.** A check must capture one with `keep_flow` rather than assume a
literal, or it holds only for the first flow the daemon ever made.
## The surface walk
`check_surface_walk` runs `examples/native-surface.rs` against the shared
single node, last among the single-node checks. Its subject is the Rust surface
rather than the wire: until it existed, `FipsStream::connect`, `connect_from`,
`connect_at`, `FipsListener::bind` and `bind_at` had no coverage of any kind,
and every other public item was exercised only against the hand-written stand-in
daemon in the crate's unit tests. That stand-in has already hidden a real defect
once, by being kinder than the daemon, which is why the walk talks to the real
one.
It runs last because `check_ephemeral_allocation` asserts 49152, 49153 and 49154
as the first three ports the node ever hands out and the allocator is a
forward-only cursor. The walk therefore asserts only that its own ephemeral
ports are `>= 49152`, and takes its named ports from the otherwise unused
4800-4809 band.
**The check asserts three things, not one:** that the container exited 0, that
its completion line is there, and that the count in that line equals
`SURFACE_ASSERTIONS` in `test.sh`. The third is the anti-silence measure. The
binary prints the recorder's own counter rather than a literal, so an assertion
block that stopped running — a `#[cfg]` gate that no longer matches, an early
return — still exits 0 and still prints the line, and only the count betrays it.
The number is deliberately brittle: adding an assertion must force an edit in
`test.sh`, so the two cannot drift apart quietly.
**A hang has to become a red, and has to name itself.** The walk's own subjects
fail by blocking forever: a read deadline never applied to the descriptor, a
`set_nonblocking` that did nothing. The binary arms a 30-second watchdog that
prints the assertion it was in and exits 1, and `run_surface_at` bounds the
container at 60 seconds as a backstop for a wedge before that thread is armed.
`timeout 60 docker run` is **not** that backstop, which a break-check measured
rather than a reading of the manual. `timeout` signals the docker client, the
client proxies SIGTERM to the container, and the walk is PID 1 there with no
handler for it, so the kernel discards the signal: the container was still up
five minutes after the bound passed and `docker run` never returned. The helper
runs the container detached, polls its state, and removes it by force, since
`docker rm -f` is a SIGKILL and PID 1 cannot discard that.
## The two-node check
`check_end_to_end` is the only check that runs more than one node. It derives
two identities with `testing/lib/derive_keys.py`, brings both up on their own
docker network peered by npub, and sends a datagram from a client on one to a
client on the other.
Three orderings are waited on explicitly rather than assumed, each because
assuming it produced an intermittent failure:
- **The link forms** before either client runs, watched for by the spanning tree
adopting a parent. Not by a peer-promotion log line: on this path — a
configured peer, dialled outbound — that line is never emitted.
- **The listener has bound its port** before the sender starts, watched for in
the listener's own output. Launching it first is not the same as it having
registered.
- **The client runs unbuffered** (`python3 -u`). Without it the marker above
never reaches the log file, so the wait cannot see it and every run fails at
the gate meant to make the check reliable.
The payload is deliberately not a valid IPv6 packet, and it is sent before any
session exists so it goes through the native pending queue. If a native datagram
were ever routed through the TUN pending queue it would be handed to the IPv6
compressor, which would refuse it, and this check would fail. The trap is
asserted rather than trusted.
Reference in New Issue View Git Blame Copy Permalink