Johnathan Corgan
3a789370b9
A client process opens a flow to a peer's public key on a chosen port and sends and receives datagrams on a file descriptor the daemon hands it. No IPv6 emulation, no TUN device, no DNS: a datagram travels from key to key. The feature is off by default and is not a stable interface. The wire needs no change and gets none. Every FSP data packet has carried a port pair inside its AEAD envelope since v0.2.0, and port 256 is simply the IPv6 shim. What was missing was a way for a program to ask for a port of its own and be handed the traffic. Addressing is the part worth reading twice, because the obvious design is wrong. The x-only public key is the address. An npub is that key written in bech32, so converting between them is a local encoding rather than a lookup or a name service. The 16-byte node address that travels on the wire is the first half of a SHA-256 of the key: it is a truncated hash, it does not invert, and it appears nowhere a client can see. An earlier iteration of this work reported a peer by that hash and could supply a key only sometimes, which is what treating a wire identifier as an identity produces. An accepted flow therefore always knows its peer. The key is captured where the peer is authenticated rather than looked up when a report is rendered: every inbound datagram passes one call site inside a handler that refuses anything whose session is not established, and the responder has already rejected the session unless the claimed address derives from the key it proved. Reaching for the identity cache instead gives a best-effort answer from a structure that evicts. A listener is a descriptor. The daemon writes one message per arrival to it, carrying the new flow's descriptor and the peer's address, so poll, select and epoll work on a listener and accepting is a recvmsg. That is what lets the API be used from a program that already has an event loop, which a command-and-reply listener could not support: an arrival could not be waited on beside anything else. There is no accept command and no reject command. Refusing a flow is closing the descriptor you were handed. The Rust surface mirrors std::net. FipsStream::connect, FipsListener::bind, incoming, accept, io::Result and an errno mapping rather than a bespoke error type. An address is given as an npub, as a key, or as a pair, through one parameter, the way ToSocketAddrs takes several spellings of one thing. Each type holds its descriptor and copies of what setup told it and nothing else, so a stream that outlives its setup connection is not representable. set_nonblocking, AsFd and the four deadline methods carry the names and signatures std::net uses for the same jobs. They were asked for by a user integrating the API with tokio: AsyncFd requires a non-blocking descriptor, and anything receiving from a peer needs a bounded wait. AsFd is the better of the two descriptor accessors, because the borrow cannot outlive the value that owns the descriptor, so a reactor cannot hold a registration for a descriptor that has since been closed and its number reused by the next open. The non-blocking flag is read, modified and written back rather than assigned, since the flag word carries more than that one bit and a caller may have set O_ASYNC. A zero timeout is refused with EINVAL, because the kernel reads a zero timeval as "wait for ever", which inverts what a caller passing zero means; std::net refuses it for the same reason. The two directions are separate options and stay that way. FipsListener gets no timeout methods, matching TcpListener: bounding an accept is set_nonblocking plus the caller's own poll, which the reactor how-to builds. A flow taken from accept is blocking whatever the listener was set to, because the two are separate sockets and the daemon hands over a fresh one. One rule has no counterpart in Berkeley sockets and a client author must know it: the v1 wire carries no half-close, so nothing peer-driven ever closes a flow. A server written to read until the flow ends waits for a signal that cannot arrive, holding a thread and a flow per peer until its process exits. A program decides its own termination, and the example serves one datagram per flow. The tests reach a live daemon rather than a stand-in. Every public item had a unit test against a hand-written stand-in with canned replies, and the five entry points a program actually calls first, connect, connect_from, connect_at, bind and the SOCKET constant, had no coverage of any kind, because the tests that appear to cover them build a Wire over a socket pair and hand it to the private open and hold, so nothing ever resolved a socket path or mapped its errors. examples/native-surface.rs walks all thirty-eight items against a running daemon and reports the number of assertions it made. The count is read from the recorder rather than written as a literal, and the harness asserts the exit status, the completion marker and the count together, so deleting an assertion fails the check rather than quietly shrinking it. Watchdogs turn a hang into a named failure, which several of the walked behaviours would otherwise produce. The shared Docker image is built once for every integration leg, so the new binary is staged at all ten places the existing one is, the interop builder included, which gets a stub because those images exercise the wire between daemon versions and older refs do not carry the example. The platform gating was tested rather than reasoned about: flipping all eleven gates so the native API is excluded leaves the crate compiling clean across the workspace, every target and the profiling feature. The shipped docs tree gains what only the LaTeX manual under design/ had, which is not published with the daemon. A reference entry covers the whole surface: addressing and the port tiers, the Berkeley mapping, every method on FipsAddr, FipsStream, FipsListener and Incoming, the errno table, the ceilings, the four places data disappears with nothing reported, the line protocol and the command reference. The errno table gives names rather than numbers, since the client maps each name onto the libc constant for the platform it was built for and the supported platforms disagree on the numbers. A tutorial side trip stands up two throwaway nodes on one machine, peered over loopback UDP with no TUN and no DNS, then writes a listening program and a connecting program against them; it needs neither the public mesh nor root, because the native path is the one that does not go through the IPv6 adapter. The obligations a client in another language carries are a how-to of their own, since they are a task rather than a description: reading the setup connection with recvmsg, associating a descriptor with the last complete line, telling an empty datagram from a close, and six others. Serving many peers from one poll loop is another, with the whole program, because the straightforward listener spawns a thread per flow and that is wrong at the node's ceiling of 256. The drop causes are a table mapping each of the seven texts DropReason::as_str produces to the counter it increments, with drop_oversize called out as the ninth counter that is not in the table. What a daemon restart costs is a section of its own: every flow and listener ends, descriptors do not survive, there is no resumption, and datagrams sent but not yet forwarded are lost through a window nothing bounds. A stack comparison diagram places the interface against the stack a reader already knows: the same application over HTTP, TLS, TCP, IP and Ethernet on one side, and over its own format, FSP, FMP and a FIPS transport on the other, aligned so each row is one concern. The two columns are not alternatives and are not drawn as such. An unmodified IPv6 program's packets reach fips0, and the adapter hands each one to FSP as a payload, so the left stack runs inside the right one; the left column ends at a fork, eth0 for the ordinary internet and fips0 for the mesh, and an arrow leaves fips0 and runs back up into FSP's input. The row where TCP would be is empty on purpose and names Reliable Object Delivery, which is where that capability is expected to land. ROD is a v2 capability, the box is dashed because none of it exists yet, and the design entry says the part a reader needs most: nothing on the surface anticipates it, so a program written today should assume it does not exist. Both endpoints carry a scheme and a worked port, https://<npub>.fips:443 and fips://<npub>:443, with a footnote saying the two ports are not the same kind of thing, a TCP port inside the tunnel on the left and an FSP port on the right. The fips:// form is a coinage: nothing in the tree parses it, nothing registers the scheme, and the API takes a key and a port as separate arguments rather than a URL. The diagram also says where the right column stops, since FIPS over UDP still rides IP and Ethernet beneath. It appears in fips-concepts.md and fips-ipv6-adapter.md, which were making its argument in prose without a picture, and deliberately not in fips-architecture.md, which already carries the OSI mapping and makes the same point about the transport row. The gateway's control socket moves onto the same bind policy this API uses, which is the one change here that touches deployed behaviour: fips-gateway now tightens /run/fips to 0750. That is unreachable under the packaged deployment, where fips.service has already created the directory at that mode, and reachable for a source build or a container that starts the gateway alone. One changelog entry under Added, describing the released state: what a client opens and reads, the addressing and why the node address is not it, the listener being a descriptor, the std::net shape of the Rust surface, and the one rule Berkeley sockets have no counterpart for. It says in as many words that the wire is unchanged.
Mixed-Version Interop Test Harness
A CI lab harness for mixed-version interoperability testing. It runs an N-node full mesh where nodes run different builds of the FIPS daemon, and checks that every pair of versions interoperates — FMP link, FSP session, connectivity, and rekey survival — without a failure a same-version pair would not have.
What it tests
The static/rekey suites bake one binary set for all nodes, so they only ever test a version against itself. This harness breaks that assumption: each node runs an image built from its own git ref. The harness then looks for interop regressions — places where two different versions fail to interoperate:
- FMP handshake failures across versions
unknown FMP versiondrops- FSP / FMP AEAD decrypt failures
- replay storms / excessive-decrypt-failure removals
- link or session teardowns
- asymmetric connectivity drops
- rekey (FMP link + FSP session) that completes within a version but stalls or breaks across versions
Every failure is attributed to a specific version pair, classified same-version vs MIXED. The summary states whether failures are mixed-version-only (a genuine interop regression), both mixed and same (general instability), or same-version-only (a build is unstable even against itself).
The node-spec
The harness is parameterized by a node-spec: a multiset of image slots,
of size >= 2. Each slot is one of a, b, c, and the same slot may appear
more than once. The slots resolve to the three images built by
build-images.sh:
| Slot | Role | Intended ref |
|---|---|---|
a |
version under test | the branch tip / commit to vet |
b |
parent / comparison | parent commit on the same branch |
c |
release baseline | latest release tag (v0.3.0) |
build-images.sh is unchanged — it always builds exactly three images
from three refs. A node-spec like a a b c resolves to the same three
images; only the mesh topology grows.
Node identity vs image slot
Node identity and image slot are separate:
- A spec entry is a slot letter.
- Node id =
<slot><ordinal>, ordinal counting occurrences of that slot, 1-based. - Container name =
fips-interop-<nodeid>. - IPv4 =
172.30.0.1<index>, index = 0-based position in the spec (.10,.11,.12, ...). - Each node maps to its image slot:
a1,a2→fips-interop:a. - A pair is same-version iff the two nodes' slots resolve to the same
built SHA (read from
.build/refs.env); else MIXED.
Two same-slot nodes get distinct identities — derive_keys.py is keyed
by node id, so a1 and a2 are different npubs even though they run the
same binary.
Example specs
| Node-spec | Node ids | Pairs |
|---|---|---|
a b c |
a1 b1 c1 |
3 pairs, all MIXED — today's triangle (default). |
a a b c |
a1 a2 b1 c1 |
6 pairs: a1↔a2 same + 5 MIXED. |
a a a |
a1 a2 a3 |
3 pairs, all same — a same-version flake rig. |
Why the a a b c control topology matters
The triangle a b c has no same-version pair — every pair is mixed, so
under packet loss you cannot tell an interop regression from generic loss
noise. The a a b c spec adds a control arm: the a1↔a2 pair runs
identical binaries. Under a netem stress loop:
- a failure on a mixed pair the control pair does not share → an interop regression;
- a failure both the mixed pairs and the control pair share → loss-induced instability, not version-specific.
That control pair is what makes a stress run interpretable. It is the
default node-spec for interop-stress.sh.
The a a a spec is the degenerate case: all-same-version, used purely as a
flake rig — exercising a single build against itself under loss to find
loss-induced instability with no version variable at all.
Files
| File | Purpose |
|---|---|
build-images.sh |
Build fips-interop:a/b/c, one Docker image per git ref. |
generate-configs.sh |
Generate per-node configs, the generated compose, manifests. |
interop-test.sh |
Test driver: bring up, converge, rekey, analyze, attribute. |
interop-stress.sh |
Netem stress loop: N reps, pass rate, mixed-vs-same attribution. |
README.md |
This document. |
Generated at runtime: generated-configs/ (per-node configs +
docker-compose.generated.yml + nodes.env + npubs.env), .build/,
.stress-runs/. The root for these three is selected by the
FIPS_INTEROP_RUNS_DIR environment variable — see
Scratch directory location below.
The static docker-compose.yml is gone — the compose file is now generated
per node-spec into generated-configs/docker-compose.generated.yml.
How per-node images work
build-images.sh <ref-a> <ref-b> <ref-c> (unchanged — always three refs):
- For each ref,
git worktree add --detacha temp checkout of the repo. cargo build --releasethe four binaries (fips,fipsctl,fipstop,fips-gateway) in that worktree.- Copy the binaries into a build context alongside the shared
testing/docker/Dockerfile(andentrypoint.sh,resolv.conf). docker buildit, taggingfips-interop:<slot>and labelling the image with its ref + short SHA.- Remove the temp worktree (done per-ref so peak disk stays at one worktree).
It also writes .build/refs.env, recording each slot's ref and SHA. The
driver reads it to know which pairs are mixed-version. (If absent, it falls
back to the image labels.)
How to run it
Build the images (once per ref set)
cd /dpool/src/clabs/nostr/fips
bash testing/interop/build-images.sh <ref-a> <ref-b> <ref-c>
Example: A = tip of fix/fsp-rekey-overlapping-epoch, B = maint,
C = release tag v0.3.0:
bash testing/interop/build-images.sh fix/fsp-rekey-overlapping-epoch maint v0.3.0
Run a single mesh
bash testing/interop/interop-test.sh [node-spec...]
node-spec defaults to a b c (the original triangle). Examples:
bash testing/interop/interop-test.sh # a b c — 3-node triangle
bash testing/interop/interop-test.sh a a b c # 4-node, one control pair
bash testing/interop/interop-test.sh a a a # 3-node same-version flake rig
The driver regenerates configs automatically whenever the requested
node-spec differs from the one on disk, so changing the spec just works.
A run takes a few minutes (driven by REKEY_AFTER_SECS, default 35, times
two rekey cycles).
Run the netem stress loop
FIPS_INTEROP_NETEM="delay 10ms 5ms 25% loss 2%" \
bash testing/interop/interop-stress.sh [--reps N] [node-spec...]
--reps N— repetitions (default 10).node-spec— defaulta a b c(the control topology).- Reps run serially — the harness uses fixed container names and a fixed Docker network, so two reps must never overlap.
- If
FIPS_INTEROP_NETEMis unset the script warns (a stress run normally wants netem) but still runs a clean baseline loop.
Each rep invokes interop-test.sh with netem set and captures its full
output and exit code; a rep passes iff interop-test.sh exits 0. Artifacts
land in testing/interop/.stress-runs/<UTC-timestamp>/:
rep-NN/driver.log— full driver output for every rep.rep-NN/docker-<container>.log— per-containerdocker logsfor failed reps only.summary.txt— the aggregate report.
The aggregate report gives reps run, passed/failed counts and an integer pass rate, then tallies — across all failed reps — connectivity failures by pair kind (mixed vs same), and a verdict:
- mixed pairs only, never the control pair → interop-regression signal;
- both mixed and same pairs → loss-induced general instability;
- same-version control pair only → the build is unstable against itself.
interop-stress.sh exits non-zero only for the interop-regression
signal. A sub-100% pass rate under loss is expected and is not by itself a
failure, so every other outcome exits 0.
Options
| Variable | Effect |
|---|---|
FIPS_INTEROP_NETEM |
tc-netem string applied to each container's eth0, e.g. "delay 10ms 5ms 25% loss 1%". Passed through interop-stress.sh to interop-test.sh. |
REKEY_AFTER_SECS |
Rekey interval for generated configs (default 35). |
FIPS_INTEROP_KEEP_UP |
1 = leave containers running after the test. |
FIPS_INTEROP_KEEP_WORKTREES |
1 = keep build-images.sh worktrees (debug). |
FIPS_INTEROP_RUNS_DIR |
Root for the three scratch dirs — see Scratch directory location. |
The netem hook reuses the flake-lab mechanism (docker exec ... tc qdisc on
each container's eth0) — host-side bridge qdisc does not shape
inter-container port-to-port traffic, so the impairment must live inside the
containers.
Scratch directory location
The three scratch dirs the harness writes — .build/ (per-ref build
contexts and refs.env), generated-configs/ (per-node configs +
generated compose + manifests), and .stress-runs/ (stress-loop
artefacts) — are rooted under FIPS_INTEROP_RUNS_DIR when that
environment variable is set. With
export FIPS_INTEROP_RUNS_DIR=/var/lib/fips-interop
all three land under /var/lib/fips-interop/, and the source tree
stays clean.
When FIPS_INTEROP_RUNS_DIR is unset, each harness script falls back
to writing under testing/interop/ itself and prints a one-line
stderr warning naming the variable. The in-tree paths are
.gitignored, so accidentally running without the variable does not
dirty the checkout — but pointing the variable outside the source
tree is recommended, so lab runs do not interleave with the source
working copy at all.
How to read the output
The driver runs six phases:
| Phase | Check |
|---|---|
| 0 | Bring up the mesh (+ optional netem). |
| 1 | All nodes reach N-1 authenticated peers; all directed pairs ping over fips0 (the definitive FSP-session check). |
| 2 | First FMP rekey cutover completes within the timeout. |
| 3 | All pairs still ping after the first rekey. |
| 4 | Wait out a second rekey cycle. |
| 5 | All pairs still ping after the second rekey. |
| 6 | Per-node / per-pair interop log analysis. |
Phase 6 is the interop-specific part. It reports:
- Global health — panics,
ERRORlines,unknown FMP versiondrops, link teardowns, decrypt failures, handshake failures, rekey-msg2 failures. Any non-zero count is broken down per node, attributed to a specific build. - Rekey machinery exercised — both FMP and FSP rekey cutovers fired.
- Per-pair interop summary — each unordered pair, classified same-version vs MIXED, with whether it stayed healthy through the run.
The final verdict lists every failure attributed to a specific
x[ref@sha] <-> y[ref@sha] pair or build, then states the attribution:
- mixed-version only → a genuine interop regression.
- both mixed and same → general instability, not version-specific.
- same-version only → a build is unstable even against itself.
Exit code is 0 only if every check passed and no per-pair failure was
recorded; non-zero otherwise, with a diagnostic dump (peer/link snapshots and
interop-relevant log tails for all nodes).
CI integration
The harness is self-contained and slots into .github/workflows/ci.yml
alongside the existing integration matrix suites. A future matrix entry
would, per push to a PR branch:
- Resolve the three refs — e.g.
A = github.sha,B = git rev-parse github.sha^,C = $(git describe --tags --abbrev=0)or a pinned release tag. bash testing/interop/build-images.sh "$A" "$B" "$C".bash testing/interop/interop-test.sh a a b cfor the control topology, orinterop-stress.shfor a loss sweep.- On failure, upload the diagnostic dump (or
.stress-runs/) as an artifact.
The three cargo build --release passes are the cost driver; on a CI runner
this suite is heavier than the single-image suites. Reasonable options are to
run it only on release branches / tags, gate it behind a label, or cache the
fips-interop:c (release) image since the release tag rarely moves.
Until then the harness runs on demand locally — the same way the flake-lab is used today.