mirror of https://github.com/jmcorgan/fips.git synced 2026-10-05 19:18:25 +00:00

Files

T

Johnathan Corgan 3a789370b9 Add an experimental native datagram API addressed by public key

A client process opens a flow to a peer's public key on a chosen port and
sends and receives datagrams on a file descriptor the daemon hands it. No
IPv6 emulation, no TUN device, no DNS: a datagram travels from key to key.
The feature is off by default and is not a stable interface.

The wire needs no change and gets none. Every FSP data packet has carried a
port pair inside its AEAD envelope since v0.2.0, and port 256 is simply the
IPv6 shim. What was missing was a way for a program to ask for a port of its
own and be handed the traffic.

Addressing is the part worth reading twice, because the obvious design is
wrong. The x-only public key is the address. An npub is that key written in
bech32, so converting between them is a local encoding rather than a lookup
or a name service. The 16-byte node address that travels on the wire is the
first half of a SHA-256 of the key: it is a truncated hash, it does not
invert, and it appears nowhere a client can see. An earlier iteration of this
work reported a peer by that hash and could supply a key only sometimes,
which is what treating a wire identifier as an identity produces.

An accepted flow therefore always knows its peer. The key is captured where
the peer is authenticated rather than looked up when a report is rendered:
every inbound datagram passes one call site inside a handler that refuses
anything whose session is not established, and the responder has already
rejected the session unless the claimed address derives from the key it
proved. Reaching for the identity cache instead gives a best-effort answer
from a structure that evicts.

A listener is a descriptor. The daemon writes one message per arrival to it,
carrying the new flow's descriptor and the peer's address, so poll, select
and epoll work on a listener and accepting is a recvmsg. That is what lets
the API be used from a program that already has an event loop, which a
command-and-reply listener could not support: an arrival could not be waited
on beside anything else. There is no accept command and no reject command.
Refusing a flow is closing the descriptor you were handed.

The Rust surface mirrors std::net. FipsStream::connect, FipsListener::bind,
incoming, accept, io::Result and an errno mapping rather than a bespoke
error type. An address is given as an npub, as a key, or as a pair, through
one parameter, the way ToSocketAddrs takes several spellings of one thing.
Each type holds its descriptor and copies of what setup told it and nothing
else, so a stream that outlives its setup connection is not representable.

set_nonblocking, AsFd and the four deadline methods carry the names and
signatures std::net uses for the same jobs. They were asked for by a user
integrating the API with tokio: AsyncFd requires a non-blocking descriptor,
and anything receiving from a peer needs a bounded wait. AsFd is the better
of the two descriptor accessors, because the borrow cannot outlive the value
that owns the descriptor, so a reactor cannot hold a registration for a
descriptor that has since been closed and its number reused by the next
open. The non-blocking flag is read, modified and written back rather than
assigned, since the flag word carries more than that one bit and a caller may
have set O_ASYNC. A zero timeout is refused with EINVAL, because the kernel
reads a zero timeval as "wait for ever", which inverts what a caller passing
zero means; std::net refuses it for the same reason. The two directions are
separate options and stay that way. FipsListener gets no timeout methods,
matching TcpListener: bounding an accept is set_nonblocking plus the caller's
own poll, which the reactor how-to builds. A flow taken from accept is
blocking whatever the listener was set to, because the two are separate
sockets and the daemon hands over a fresh one.

One rule has no counterpart in Berkeley sockets and a client author must know
it: the v1 wire carries no half-close, so nothing peer-driven ever closes a
flow. A server written to read until the flow ends waits for a signal that
cannot arrive, holding a thread and a flow per peer until its process exits.
A program decides its own termination, and the example serves one datagram
per flow.

The tests reach a live daemon rather than a stand-in. Every public item had a
unit test against a hand-written stand-in with canned replies, and the five
entry points a program actually calls first, connect, connect_from,
connect_at, bind and the SOCKET constant, had no coverage of any kind,
because the tests that appear to cover them build a Wire over a socket pair
and hand it to the private open and hold, so nothing ever resolved a socket
path or mapped its errors. examples/native-surface.rs walks all thirty-eight
items against a running daemon and reports the number of assertions it made.
The count is read from the recorder rather than written as a literal, and the
harness asserts the exit status, the completion marker and the count
together, so deleting an assertion fails the check rather than quietly
shrinking it. Watchdogs turn a hang into a named failure, which several of
the walked behaviours would otherwise produce. The shared Docker image is
built once for every integration leg, so the new binary is staged at all ten
places the existing one is, the interop builder included, which gets a stub
because those images exercise the wire between daemon versions and older refs
do not carry the example. The platform gating was tested rather than reasoned
about: flipping all eleven gates so the native API is excluded leaves the
crate compiling clean across the workspace, every target and the profiling
feature.

The shipped docs tree gains what only the LaTeX manual under design/ had,
which is not published with the daemon. A reference entry covers the whole
surface: addressing and the port tiers, the Berkeley mapping, every method on
FipsAddr, FipsStream, FipsListener and Incoming, the errno table, the
ceilings, the four places data disappears with nothing reported, the line
protocol and the command reference. The errno table gives names rather than
numbers, since the client maps each name onto the libc constant for the
platform it was built for and the supported platforms disagree on the
numbers. A tutorial side trip stands up two throwaway nodes on one machine,
peered over loopback UDP with no TUN and no DNS, then writes a listening
program and a connecting program against them; it needs neither the public
mesh nor root, because the native path is the one that does not go through
the IPv6 adapter. The obligations a client in another language carries are a
how-to of their own, since they are a task rather than a description:
reading the setup connection with recvmsg, associating a descriptor with the
last complete line, telling an empty datagram from a close, and six others.
Serving many peers from one poll loop is another, with the whole program,
because the straightforward listener spawns a thread per flow and that is
wrong at the node's ceiling of 256. The drop causes are a table mapping each
of the seven texts DropReason::as_str produces to the counter it increments,
with drop_oversize called out as the ninth counter that is not in the table.
What a daemon restart costs is a section of its own: every flow and listener
ends, descriptors do not survive, there is no resumption, and datagrams sent
but not yet forwarded are lost through a window nothing bounds.

A stack comparison diagram places the interface against the stack a reader
already knows: the same application over HTTP, TLS, TCP, IP and Ethernet on
one side, and over its own format, FSP, FMP and a FIPS transport on the
other, aligned so each row is one concern. The two columns are not
alternatives and are not drawn as such. An unmodified IPv6 program's packets
reach fips0, and the adapter hands each one to FSP as a payload, so the left
stack runs inside the right one; the left column ends at a fork, eth0 for the
ordinary internet and fips0 for the mesh, and an arrow leaves fips0 and runs
back up into FSP's input. The row where TCP would be is empty on purpose and
names Reliable Object Delivery, which is where that capability is expected to
land. ROD is a v2 capability, the box is dashed because none of it exists
yet, and the design entry says the part a reader needs most: nothing on the
surface anticipates it, so a program written today should assume it does not
exist. Both endpoints carry a scheme and a worked port,
https://<npub>.fips:443 and fips://<npub>:443, with a footnote saying the two
ports are not the same kind of thing, a TCP port inside the tunnel on the
left and an FSP port on the right. The fips:// form is a coinage: nothing in
the tree parses it, nothing registers the scheme, and the API takes a key and
a port as separate arguments rather than a URL. The diagram also says where
the right column stops, since FIPS over UDP still rides IP and Ethernet
beneath. It appears in fips-concepts.md and fips-ipv6-adapter.md, which were
making its argument in prose without a picture, and deliberately not in
fips-architecture.md, which already carries the OSI mapping and makes the
same point about the transport row.

The gateway's control socket moves onto the same bind policy this API uses,
which is the one change here that touches deployed behaviour: fips-gateway
now tightens /run/fips to 0750. That is unreachable under the packaged
deployment, where fips.service has already created the directory at that
mode, and reachable for a source build or a container that starts the gateway
alone.

One changelog entry under Added, describing the released state: what a
client opens and reads, the addressing and why the node address is not it,
the listener being a descriptor, the std::net shape of the Rust surface,
and the one rule Berkeley sockets have no counterpart for. It says in as
many words that the wire is unchanged.

2026-08-21 05:48:23 +00:00

18 KiB

Raw Blame History

FIPS IPv6 Adapter

The IPv6 adapter sits above the FIPS Session Protocol (FSP) and adapts the FIPS datagram service for unmodified IPv6 applications. It presents each FIPS node as an IPv6 endpoint, so standard socket applications (SSH, HTTP, SCP) can communicate over the mesh without modification.

Role

The adapter bridges two worlds: IPv6 applications that address destinations by IP address, and the FIPS mesh that addresses destinations by public key (npub). The adapter handles the translation: DNS resolution from npub to fd00::/8 ULA (Unique Local Address) address, identity cache management so FIPS can route IPv6 packets, MTU enforcement so packets fit through the mesh, and the TUN interface that connects to the kernel's IPv6 stack.

Applications that are FIPS-aware can bypass the adapter entirely and use the native FIPS datagram API, addressing destinations directly by npub.

The adapter is the fips0 box, and the arrow leaving it is the point: an IPv6 packet arriving there does not go out to a wire. It becomes an FSP payload, so the whole IP stack above runs inside the mesh's own stack — TCP included, which is what lets an unmodified ssh or curl work across the mesh. The right-hand column is the same mesh reached without the adapter, and fips-native-api.md covers that path.

DNS Integration

The Problem

IPv6 addresses in the fd00::/8 range are derived from public keys via a one-way hash (SHA-256). Given only an IPv6 address, the public key cannot be recovered — and without the public key, FIPS cannot compute the node_addr needed for routing.

The identity cache must be populated before packets arrive at the TUN interface, or they cannot be routed.

DNS as Entry Point

DNS resolution serves as the "routing intent" signal. When an application resolves npub1xxx...xxx.fips, the FIPS DNS service:

Extracts the npub from the .fips domain name
Derives the fd00::/8 IPv6 address from the public key
Primes the identity cache with the mapping (IPv6 address prefix ↔ NodeAddr ↔ PublicKey)
Returns the IPv6 address to the application

When the application subsequently sends packets to that address, the identity cache already contains the mapping needed for routing.

DNS Name Format

npub1xxxxxx...xxxxx.fips

The FIPS DNS server recognizes names ending in .fips and extracts the npub for address derivation.

Traffic Without Prior DNS Lookup

A packet may arrive at the TUN for an fd00::/8 destination without a prior DNS lookup — cached address, manual configuration, etc. Since the address derivation is one-way, the npub cannot be recovered from the address alone.

FIPS returns ICMPv6 Destination Unreachable (Code 0: No route to destination) for packets to unknown addresses. The identity cache must be populated before traffic can be routed.

Known cache population mechanisms:

DNS lookup: The primary path
Inbound traffic: Authenticated sessions from other nodes populate the cache with their identity information

Mesh-Interface Query Filter

The DNS responder is intended for local applications resolving .fips names; queries arriving over the mesh interface itself are dropped. The daemon records the index of the TUN interface at startup and compares it against the arrival interface of each incoming UDP DNS query. When they match — meaning the query came from another mesh node, not from a local socket — the responder discards the query without replying.

The check is implemented in is_mesh_interface_query and prevents two classes of misbehaviour: a peer asking the daemon to resolve .fips names on its behalf (which would let one node use another as an identity-cache priming proxy), and accidental query loops where a misconfigured resolver forwards .fips queries back into the mesh. Local applications binding to the host's loopback or non-mesh interfaces are unaffected.

IPv6 Address Derivation

FIPS addresses use the IPv6 Unique Local Address (ULA) prefix fd00::/8:

Public Key (32 bytes)
    │
    ▼
SHA-256 → node_addr (16 bytes, truncated)
    │
    ▼
fd + node_addr[0..15] → IPv6 address (16 bytes)

The fd prefix places FIPS addresses in the IPv6 Unique Local Address (ULA) space defined by RFC 4193. ULAs are the IPv6 equivalent of RFC 1918 private addresses (10.x, 172.16.x, 192.168.x) — they are reserved for local use and are not routable over the public Internet. This means FIPS overlay addresses cannot conflict with native IPv6 traffic that may be present on the same host or network, and they will not leak beyond the local system even if routing is misconfigured. These are overlay identifiers — they appear in the TUN interface for application compatibility but have no meaning outside the FIPS mesh.

Identity Cache

The derivation from public key to NodeAddr and IPv6 address is one-way (SHA-256 truncation). Given a destination IPv6 address from an outbound packet on the TUN interface, the adapter cannot recover the public key or NodeAddr needed for FIPS routing. The identity cache provides the reverse lookup: it maps the FIPS address prefix (15 bytes — the IPv6 address minus the fd prefix) back to (NodeAddr, PublicKey), allowing the adapter to route IPv6 traffic into the mesh. This cache is needed only when using the IPv6 adapter; the native FIPS API provides the public key directly.

Eviction Policy

The mapping is deterministic (derived from the public key) and never becomes stale. The cache uses LRU-only eviction bounded by a configurable size (default 10K entries). There is no TTL — entries are evicted only when the cache is full and space is needed for a new entry. LRU-only eviction is necessary because there is no other way for the FIPS router to recover the routing identity from an IPv6 address, and IPv6 traffic for a destination may arrive an arbitrarily long time after the DNS resolution that populated the cache entry.

Relationship to DNS TTL

The identity cache timeout must be longer than the DNS TTL to ensure that while an application believes its DNS resolution is valid, the corresponding routing entry remains present. The DNS TTL (default 300s) governs when applications re-query; the identity cache (LRU, no TTL) is always available as long as the entry hasn't been evicted by memory pressure.

MTU Enforcement

The adapter sits at the boundary between the host's IPv6 stack and the FIPS encapsulation budget. Its job is to keep IPv6 packets small enough that they fit through the FIPS protocol envelope on every link along the path. The cross-cutting MTU model — proactive SessionDatagram path_mtu annotation, reactive MtuExceeded signals, end-to-end PathMtuNotification echo, and per-destination MTU storage — is documented in fips-mtu.md. What the adapter contributes is the IPv6-specific overhead accounting and the TUN-side enforcement integration.

IPv6-Specific Overhead

For IPv6 traffic, FSP port multiplexing adds 4 bytes (port header) while IPv6 header compression saves 33 bytes (40-byte header → 7-byte format + residual), yielding a net FIPS_IPV6_OVERHEAD of 77 bytes on top of the base FIPS_OVERHEAD (106 bytes) protocol envelope. The full encapsulation breakdown lives in fips-mtu.md.

Effective IPv6 MTU

The effective IPv6 MTU visible to applications is:

effective_ipv6_mtu = transport_mtu - FIPS_IPV6_OVERHEAD

For typical deployments:

Transport MTU	Effective IPv6 MTU	Notes
1472 (UDP/Ethernet)	1395	Standard deployment
1280 (UDP minimum)	1203	Below IPv6 minimum

IPv6 mandates that every link support at least 1280 bytes. The minimum transport path MTU for the IPv6 adapter is therefore:

1280 + 77 = 1357 bytes

Transports with smaller MTUs (radio at ~250 bytes, serial at 256 bytes) cannot support the IPv6 adapter without some form of internal fragmentation and reassembly. Otherwise, applications on those transports must use the native FIPS datagram API.

TUN-Side ICMP Packet Too Big

When an outbound packet at the TUN exceeds the effective IPv6 MTU, the adapter generates an ICMPv6 Packet Too Big message and delivers it back to the application via the TUN. This triggers the kernel's Path MTU Discovery mechanism, which adjusts TCP segment sizes for subsequent transmissions.

ICMP Packet Too Big generation is rate-limited per source address (100ms interval) to prevent storms from applications sending many oversized packets. The ICMP response is delivered locally back through the TUN; no network traversal is needed, so delivery is reliable.

TUN-Side TCP MSS Clamping

The adapter intercepts TCP SYN and SYN-ACK packets at the TUN interface and clamps the Maximum Segment Size (MSS) option:

clamped_mss = effective_ipv6_mtu - 40 (IPv6 header) - 20 (TCP header)

Clamping is applied in two places:

TUN reader (outbound): Clamps MSS on outbound SYN packets
TUN writer (inbound): Clamps MSS on inbound SYN-ACK packets

Together, these ensure both directions of a TCP connection use appropriately sized segments from the start, avoiding the initial oversized packet loss that would occur with ICMP Packet Too Big alone. The conditional clamp (per-flow lookup with cold-flow fallback) and the rationale for max_mss semantics are in fips-mtu.md.

ICMP Rate Limiting

ICMPv6 error generation is rate-limited per source address using a token bucket (100ms interval). This matches the standard ICMP rate limiting approach and prevents amplification when an application sends a burst of oversized packets.

TUN Interface

The TUN device (fips0) is the mechanism that connects the adapter to the kernel's IPv6 stack. It is an implementation detail of the adapter, not its defining feature.

Architecture

Applications (sockets using fd00::/8 addresses)
       │
       ▼
Kernel IPv6 Stack (routing: fd00::/8 → fips0)
       │
       ▼
TUN Device (fips0)
    ├── Reader Thread (blocking I/O → packet processing)
    └── Writer Thread (mpsc queue → TUN writes)

Reader Thread

The TUN reader receives raw IPv6 packets from applications and processes them:

Validate IPv6 header
Extract destination fd00::/8 address
Look up identity cache — miss returns ICMPv6 Destination Unreachable
Retrieve NodeAddr and PublicKey from cache
Look up or establish FSP session
Compress IPv6 header: strip addresses and payload length, build format 0x00 payload with residual fields (traffic class, flow label, next header, hop limit)
Prepend port header (src_port=256, dst_port=256)
Encrypt with session keys
Route through FMP toward destination

Writer Thread

A single writer thread services an mpsc queue of outbound packets:

Inbound mesh traffic on port 256 (IPv6 header reconstructed from session context + residual fields, then delivered as complete IPv6 packets)
ICMPv6 error responses (Packet Too Big, Destination Unreachable)
TCP MSS-clamped SYN-ACK packets

The queue-based design eliminates contention on TUN writes and cleanly separates concerns. New packet sources can be added by cloning the sender handle.

Local Address Guarantee

The Linux kernel routing table processes rules in priority order:

Local table: Intercepts traffic to addresses assigned to this machine
Main table: Routes fd00::/8 to the TUN device

This means every packet arriving at the TUN reader is guaranteed to be for a remote FIPS destination. No "is this for me?" check is needed on the read path.

Configuration

The TUN block (tun.*) is documented in ../reference/configuration.md.

Privileges

TUN device creation requires CAP_NET_ADMIN. The shipped Debian systemd unit runs the daemon as root by default; for the alternative — running under a dedicated unprivileged service account with the capability granted on the binary — see ../how-to/run-as-unprivileged-user.md.

App-Owned TUN (embedded hosts)

On platforms where FIPS is embedded rather than run as a daemon — notably Android, where the VpnService owns the TUN fd and the app has no CAP_NET_ADMIN — FIPS does not create fips0 itself. Instead the embedder owns the fd and exchanges IPv6 packet bytes with FIPS over channels.

Node::enable_app_owned_tun() sets this up. It is called after Node::new and before start() (and before the node is moved into a background task), mirroring control_read_handle(), and returns two app-side channel ends:

app → mesh — the embedder pushes IPv6 packets read from its fd into app_outbound_tx. These are drained by run_rx_loop into handle_tun_outbound and routed exactly as the Reader Thread's output would be.
mesh → app — inbound mesh traffic on port 256 is reconstructed and written to the node's tun_tx (the same sink the Writer Thread reads); the embedder pulls from app_inbound_rx and writes to its fd.

With the channels installed, start() skips system-TUN creation (it gates on tun_tx being unset), so FIPS does no CAP_NET_ADMIN operations.

Because packets enter via app_outbound_tx rather than the Reader Thread, they bypass handle_tun_packet — the fd00::/8 destination filter, the ICMPv6 Destination Unreachable for off-mesh dests (see Reader Thread), and the TUN-Side TCP MSS Clamping. The embedder is therefore responsible for routing only fd00::/8 to its TUN (so only mesh-bound packets arrive) and for clamping TCP MSS on outbound SYNs.

App-Owned DNS Path (embedded hosts)

An embedded host that owns the TUN fd generally has no system DNS socket to aim at the responder either. On Android, VpnService.Builder.addDnsServer() takes an address with no port — the resolver always uses 53, which an unprivileged app UID cannot bind — and it points the resolver into the tunnel, so .fips queries surface as IPv6/UDP packets on the app's own fd rather than at any socket FIPS holds. The DNS responder itself needs no changes for this: its bind is a plain UDP socket, and with no system TUN the mesh-interface filter self-disables because the interface name does not resolve.

Node::dns_local_addr() closes the gap. It reports the address read back off the bound socket — so a dns.port = 0 config yields the port the kernel assigned — and is None when no responder came up. The embedder lifts the DNS payload out of the packet it read, sends it to that address over an ordinary UDP socket of its own, and splices the answer back into a reply packet.

It is a one-shot read taken after start() returns and before the node is moved into a background task, because run_rx_loop then borrows the node exclusively for its whole lifetime and no &Node remains to call it on. By that point the value is settled: the responder is either up for the rest of the node's life or it never came up.

Proxying to the responder rather than resolving in the app is what keeps the identity cache warm. Answering a <npub>.fips query is what registers that peer's public key, and a FIPS address is a truncated hash of a hash of the pubkey — the key cannot be recovered from the IPv6 address alone. An app that resolves the AAAA itself leaves the cache empty, and the first packet to the resolved name is rejected with ICMPv6 Destination Unreachable. Direct neighbours mask the omission, since their identity arrives with the Noise handshake and never needed resolving.

The address is retracted on stop(). A retraction hook for a responder that exits on its own at runtime is wired on the consuming side, but is dormant: run_dns_responder never returns, so nothing produces the Child::Dns exit event it consumes. Watching a responder that dies mid-run is therefore not something an embedder can do today, and would in any case need a way to read live node state from a backgrounded run_rx_loop — a general gap rather than a DNS-specific one.

Implementation Status

Feature	Status
TUN device creation and configuration	Implemented
IPv6 address assignment (netlink)	Implemented
TUN reader/writer threads	Implemented
ICMPv6 Destination Unreachable	Implemented
ICMPv6 Packet Too Big	Implemented
ICMP rate limiting (per-source)	Implemented
TCP MSS clamping (SYN + SYN-ACK)	Implemented
DNS service (.fips domain)	Implemented
DNS responder mesh-interface filter	Implemented
App-owned TUN (`Node::enable_app_owned_tun`)	Implemented
App-owned DNS path (`Node::dns_local_addr`)	Implemented
Port-based service multiplexing (port 256)	Implemented
IPv6 header compression (format 0x00)	Implemented
Per-destination route MTU (netlink)	Planned
Transit MTU error signal	Implemented
Path MTU tracking (SessionDatagram field)	Implemented
Path MTU notification (end-to-end echo)	Implemented
Endpoint fragmentation/reassembly	Transport drivers

Design Considerations

Path MTU Discovery and No-Fragmentation Policy

Path MTU Discovery (proactive path_mtu annotation, reactive MtuExceeded, end-to-end PathMtuNotification) and the no-fragmentation policy that drives the design both live in the unified MTU treatment at fips-mtu.md. The adapter is a consumer of that model — its job is to enforce the resulting effective IPv6 MTU at the TUN with ICMP Packet Too Big and TCP MSS clamping.

References

fips-concepts.md — Protocol overview
fips-architecture.md — Layer architecture and identity model
fips-session-layer.md — FSP (below the adapter)
fips-mtu.md — Unified path MTU model (proactive, reactive, hysteresis, no-fragmentation)
../reference/wire-formats.md — FSP and SessionDatagram wire formats
../reference/configuration.md — TUN configuration parameters
../how-to/run-as-unprivileged-user.md — privilege options for the daemon, including the unprivileged service-account path

18 KiB Raw Blame History