Raise the in-process backpressure headroom in make_test_node_with_mtu
(request an 8 MiB recv_buf_size on UdpConfig and grow packet_channel from
256 to 8192) to reduce localhost-UDP receive overflow under parallel-CPU
scheduler contention, and mark the large-network convergence tests
#[ignore] so cargo test --lib stays green by default. The ignored tests
remain runnable on demand with --ignored or --test-threads=1.
Forward-merges PR #91 (off-task encrypt/decrypt worker pools, GSO,
connected UDP), the macOS #102 package-integrity fix, and the AUR
#98 fips-dns fix from master.
Six textual conflicts resolved: wire/noise imports, the K-bit-flip
and rekey-cutover handlers, ActivePeer fields, and an advert test
fixture. One semantic conflict: PR #91's worker-pool code on master
referenced FLAG_SP and MmpPeerState.spin_bit, which next removed in
4aded9a2 (spin bit superseded by the MMP receiver-report timestamp
echo; FMP flags bit 2 reclaimed). The worker acceleration and its
fmp_flags round-trip test are kept; the spin-bit references are
dropped to honor next's wire-format change.
Moves both AEAD layers (ChaCha20-Poly1305, one round per layer per
packet) plus the sendmsg syscall off the rx_loop task onto a per-shard
worker pool, adds per-peer connect(2)-ed UDP with SO_REUSEPORT, and
uses Linux UDP GSO (sendmsg+UDP_SEGMENT — kernel splits one super-skb
into N on-the-wire datagrams in a single TX-stack walk) when packets
in a batch are uniform-size. Same kernel primitive WireGuard's
in-kernel module and BoringTun use to hit 2.5–3.2 Gbps single-stream.
Single TCP stream on a 5-node docker-bridge mesh, 5 x 15 s x P=1:
A→D: 1379 → 2708 Mbps (1.96x, RTT +0.12 ms)
A→E: 1394 → 2663 Mbps (1.91x, RTT +0.11 ms)
E→A: 1406 → 2624 Mbps (1.87x, RTT +0.19 ms)
Static-peer pairs only — every CoV under 3%, 0 outliers, 0% ICMP
loss. The ~+100 µs RTT is the worker queue handoff cost; AEAD +
sendmmsg now run on a separate core in exchange.
What lands:
- src/node/encrypt_worker.rs: std::thread + crossbeam_channel
workers; hash-by-destination dispatch pins a TCP flow to one
worker so wire ordering is preserved; per-worker sendmmsg(2)
batching up to 32; Linux uses sendmsg(2)+UDP_SEGMENT when
packets in a group are uniform-size.
- src/node/decrypt_worker.rs: receive-side mirror. Each shard owns
its session's recv cipher + replay window in a thread-local
HashMap (no shared RwLock/Mutex). Sessions are handed off at
promote_connection and re-registered on K-bit flip / rekey
cutover.
- src/node/handlers/session.rs try_send_session_data_pipelined:
FSP+FMP both seal in-place in the worker on one wire-buffer
alloc; no intermediate inner_plaintext / fsp_payload Vecs.
- src/transport/udp/connected_peer.rs + peer_drain.rs: per-peer
connect(2)-ed UDP socket with SO_REUSEPORT (set on the listen
socket too — without that, EADDRINUSE on activation and every
packet falls back to the wildcard path); the worker sends with
msg_name=NULL and the kernel uses its cached 5-tuple. Tick-
driven activation in handlers/connected_udp.rs, idempotent.
- src/transport/udp/mod.rs: mem::replace the recvmmsg backing buffer
instead of buf.to_vec() per packet — single pointer swap, no
MTU-sized memcpy.
- src/protocol/link.rs SessionDatagramRef: zero-copy borrowed view
used by handle_session_datagram for the bulk local-delivery
path; handle_session_payload takes the borrowed payload
directly (no payload[35..].to_vec()).
- src/transport/mod.rs TransportAddr::from_socket_addr: collapses
the two-alloc from_string(addr.to_string()) pattern to one.
- src/node/handlers/rx_loop.rs: decrypt-fallback drain promoted
ahead of packet_rx in the select! (TCP ACK starvation fix);
interleaved fallback drain every 32 packets inside the rx burst
loop.
- noise::Session: send_cipher_clone / recv_cipher_clone /
recv_replay_snapshot_owned / take_send_counter / accept_replay
so off-task workers can hold a cloned cipher + reserved counter
while the dispatcher keeps replay/counter sequencing serial.
CipherState::cipher_clone returns a refcount-bumped LessSafeKey.
AsyncUdpSocket: AsRawFd so workers issue raw sendmmsg / sendmsg
without going through the tokio reactor.
- Worker pool sizing: both default to num_cpus, overridable via
FIPS_ENCRYPT_WORKERS=N / FIPS_DECRYPT_WORKERS=N. Per-peer
connected UDP can be disabled via FIPS_CONNECTED_UDP=0.
- src/perf_profile.rs: optional per-stage timing reporter under
FIPS_PERF=1 (or FIPS_PIPELINE_TRACE=1). Off by default; zero
overhead when disabled.
- All cfg(unix)-gated. Windows continues on the existing tokio-
based send/recv.
Decrypt worker session lifecycle:
- Node::unregister_decrypt_worker_session mirrors the existing
register helper. Wired at the two natural sites that already
iterate peers_by_index: the rekey drain-completion block in
handlers/rekey.rs (drops the worker entry for the old our_index
once the drain window has expired and the cache_key is
unreachable to any in-flight OLD-K packet), and remove_active_peer
in handlers/dispatch.rs (drops the worker entry for each of the
four index slots: current, rekey, pending, previous). Only
our_index is normally registered; unregister_session is fire-
and-forget for missing entries, so calling unconditionally on
all four slots is correct and bounds the cleanup without per-
slot accounting. Without these callers the per-worker sessions
HashMap and the Node's decrypt_registered_sessions set would
grow monotonically per rekey on long-lived peers.
Testing:
- testing/static/scripts/bench-multirun.sh: multi-run iperf3 +
ping bench. N reruns (default 5), median / min / max / CoV % /
per-run outlier flag, avg ping RTT, ICMP loss %, TCP retransmit
total. Plain client→dest labels + topology header. Pre-bench
peer-convergence check (FIPS_BENCH_CONVERGE_SECS, default 15);
per-path route verification via stats.bytes_sent deltas — fails
fast if traffic exits via a non-static-peer link.
- testing/static/docker-compose.yml: passes FIPS_ENCRYPT_WORKERS /
FIPS_DECRYPT_WORKERS / FIPS_PERF through to containers for A/B
benchmarking without rebuilds.
- testing/static/scripts/iperf-test.sh: same plain client→dest
labels + topology header (was multihop/direct/N hop, which
conflated topology distance with on-wire path).
- .config/nextest.toml: synthetic UDP node tests serialized
through a max-threads=1 test group. Localhost handshakes drop
on shared CI runners under parallel load; one-at-a-time keeps
assertions reliable.
- src/node/tests/spanning_tree.rs: repair_missing_edge_handshakes
— retries up to 5 times for synthetic edges whose msg1 was
dropped, with a drain after each edge retry instead of after
each attempt's full burst.
- src/node/decrypt_worker.rs::tests: two unit tests asserting
WorkerMsg::UnregisterSession removes the worker-thread session
HashMap entry (handle_msg_unregister_session_removes_entry) and
is a no-op for never-seen cache_keys
(handle_msg_unregister_session_idempotent_on_unknown_key), which
is the safety invariant the unconditional unregister calls at
the four index slots in remove_active_peer rely on.
- src/node/encrypt_worker.rs::unix_tests
pipelined_send_wire_layout_roundtrips_canonical_decoders: mirrors
the encoder geometry of try_send_session_data_pipelined (no
coords, the common established-session path), runs the worker's
real seal + send via flush_direct_batch_sync, and decodes the
resulting wire packet using only canonical receive-side decoders
(EncryptedHeader::parse, SessionDatagramRef::decode, FSP header
parse, noise::open). Any divergence between the hand-rolled
encoder offsets (fsp_aad_offset, fsp_plaintext_offset) and the
decoders fails at one of the parse / open / decode steps before
the inner-plaintext assertion fires. Complements the existing
fsp_preseal_runs_before_outer_fmp_seal test which covers the
seal-ordering invariant with synthetic headers but does not
exercise the wire-layout invariant.
CHANGELOG.md [Unreleased] # Changed entry added describing the
worker-pool threading model, hash-by-destination dispatch,
sendmmsg/UDP_GSO, per-peer connected UDP, the operator-facing env
vars, and the bench numbers above.
Cherry-picks from mmalmi/master (paths translated from
crates/fips-core/src/ to src/): 9b7c723, 0deb5cb, 13f7339, e036c0e,
3740a68, 3792f83, 8510193, 4910b07, e53f545, e4e2896, 5fe4af5,
1d01ada, 8c37008, e12469e, 6eb2860.
Co-authored-by: Johnathan Corgan <johnathan@corganlabs.com>
Implement TCP Wrappers-style peer access control using
/etc/fips/peers.allow and /etc/fips/peers.deny files. Evaluation
order: allow overrides deny, default permit when no files exist.
Three enforcement points: outbound connect (before dialing), inbound
handshake (msg1 receipt, after restart/rekey classification), and
outbound handshake completion (msg2, before peer promotion). Files
support npub, hex pubkey, host alias, and ALL wildcard entries with
automatic mtime-based reload.
Adds fipsctl acl show query, 954-line acl module with unit tests,
and a 6-node Docker integration harness (testing/acl-allowlist/)
exercising insider, outsider, and allowed-remote scenarios. CI
matrix entry included.
Closes#50
Co-authored-by: Johnathan Corgan <johnathan@corganlabs.com>
Adds TreeAnnounce::validate_semantics() called from handle_tree_announce
before any tree-state mutation. Enforces that the ancestry accompanying
a parent declaration conforms to the spanning tree rules:
- first ancestry entry matches the signed sender
- is_root declarations carry a single-entry ancestry
- non-root declarations include the signed parent as the second entry
- the advertised root is the minimum node_addr in the ancestry
Non-conforming announcements are rejected with a warn log and no state
change. Adds unit tests for each rejected shape plus an integration
test covering the full receive path in a two-node tree.
Co-authored-by: Johnathan Corgan <johnathan@corganlabs.com>
Replace the 2-message IK handshake with a 3-message XX handshake for
FMP link establishment. XX requires no prior knowledge of the peer's
static key — both identities are revealed during the handshake
(responder in msg2, initiator in msg3). This is the foundation for
the forklift upgrade that enables rolling protocol upgrades.
Changes:
- Noise XX state machine alongside IK/XK (8 unit tests)
- Protocol negotiation payload codec: format byte, packed version
min/max, 64-bit feature bitfield, TLV extensions (11 unit tests)
- FMP wire format version 0→1, msg3 header/builder, TCP stream framing
- FMP handshake switched to XX: PeerConnection 3-message flow,
handle_msg1 simplified (no identity), handle_msg2 sends msg3 and
promotes initiator, new handle_msg3 promotes responder with
restart/rekey/cross-connection detection
- Rekey handshake switched to XX with negotiation payload hash chain
fix (decrypt-and-discard in complete_rekey_msg2/msg3)
- Negotiation payload in msg2/msg3 (FMP version [1,1], features=0)
- Debug logging for handshake promotion paths
- Integration test convergence timeouts adjusted for extra round-trip
Squashed commits:
- Add Noise XX state machine alongside IK/XK
- Add protocol negotiation payload codec
- FMP wire format prep: version 1, msg3 header support
- Switch FMP handshake from Noise IK to XX
- Increase convergence timeouts for XX 3-message handshake
- Fix negotiation hash chain desync in rekey handshake
Two bugs prevented Path MTU Discovery from working across heterogeneous
links (e.g., ethernet→UDP boundary):
1. ICMPv6 Packet Too Big wrote the MTU as u16 into a 32-bit field
(RFC 4443 §3.2), causing the kernel to read an inflated value.
Fixed by changing build_packet_too_big() to use u32 and writing
all 4 bytes.
2. handle_tun_outbound() only checked the local transport MTU, not
the per-destination PathMtuState updated by MtuExceeded signals.
Added a PathMtuState check after session lookup so subsequent
oversized packets generate ICMPv6 PTB on TUN instead of being
forwarded and dropped at the bottleneck hop.
Added integration test exercising the full PMTUD loop across a 3-node
chain with heterogeneous MTUs: oversized packet → forwarding failure →
MtuExceeded signal → PathMtuState update → ICMPv6 PTB on TUN.
Bump rand (0.8→0.10), rtnetlink (0.14→0.20), tun (0.7→0.8),
simple-dns (0.9→0.11), socket2 (0.5→0.6), and criterion (0.5→0.8).
Migrate all rand call sites: thread_rng()→rng(), gen()→random(),
gen_range()→random_range(), RngCore→Rng trait. Work around secp256k1
0.30 requiring rand 0.8 by generating random bytes directly and
constructing SecretKey from slice.
Migrate rtnetlink to builder-based API: LinkSetRequest replaced with
LinkUnspec builder + change(), RouteAddRequest replaced with
RouteMessageBuilder.
Remove bloom benchmark (criterion 0.8 incompatible with old harness
config).
Each node generates a random 8-byte startup epoch, encrypted inside
both Noise IK handshake messages (msg1 and msg2). When a peer's msg1
arrives with a different epoch than the stored value, the node tears
down the stale session and processes the msg1 as a new connection,
enabling near-instant restart detection instead of the 30-second
dead timeout.
Wire format impact:
- msg1: 82 -> 106 bytes (added 24-byte encrypted epoch after ss DH)
- msg2: 33 -> 57 bytes (added 24-byte encrypted epoch after se DH)
- Wire msg1: 90 -> 114 bytes, wire msg2: 45 -> 69 bytes
Rename FIPS Link Protocol (FLP) to FIPS Mesh Protocol (FMP)
The "Link Protocol" name understated the layer's scope — spanning tree
construction, bloom filter routing, greedy forwarding, and mesh-wide
coordination go well beyond link-level concerns. Rename fips-link-layer.md
to fips-mesh-layer.md, update FLP→FMP throughout docs and source code
(FLP_VERSION→FMP_VERSION, wire.rs, rx_loop.rs, spanning_tree.rs).
New SVG illustrations
- Protocol stack: color-coded layer diagram replacing ASCII art
- OSI mapping: side-by-side comparison with traditional networking layers
- Bloom filter propagation: 6-node tree with sender-colored filter boxes
showing split-horizon computation per link
- Routing decision flowchart: 5-step priority chain with candidate ranking
by tree distance and link performance
- Coordinate discovery: sequence diagram showing LookupRequest propagation,
response caching, and SessionSetup cache warming
Redesigned existing SVGs
- Architecture overview: uniform node layout, U-shaped encrypted link
connectors, separate end-to-end session line
- Node architecture: split Router Core into FSP and FMP layers, reorganize
transports into Overlay/Shared Medium/Point-to-Point categories
- Identity derivation: wider boxes, visible encode arrow, dashed npub line
fips-intro.md revisions
- Add inline references to prior work: Yggdrasil/Ironwood for coordinate
routing, Noise Protocol Framework for IK handshakes, WireGuard for
index-based session dispatch, Wikipedia for bloom filters, split-horizon,
and greedy embedding
- Add explanatory paragraphs after bloom filter diagram describing
split-horizon filter computation and candidate selection behavior
- Simplify transport abstraction language, remove I2P/LoRa references
- Fix LookupRequest wording ("propagates" not "floods"), note intermediate
node coordinate caching on lookup responses
- Rewrite architecture overview prose to match redesigned diagrams
Under high-throughput forwarding, the kernel default 212KB receive
buffer overflows, silently dropping ~11% of UDP datagrams at transit
nodes. Add configurable recv_buf_size and send_buf_size to UdpConfig
(default 2MB each) using socket2 for pre-bind buffer configuration.
Startup log now reports actual buffer sizes granted by the kernel.
Requires net.core.rmem_max >= 2097152 on the host for the full 2MB
to take effect; otherwise the kernel silently clamps.
Split handlers.rs (986 lines) into handlers/ with 5 subfiles organized
by responsibility: rx_loop, encrypted, handshake, dispatch, timeout.
Split tests.rs (2350 lines) into tests/ with 4 subfiles: unit tests,
handshake integration, spanning tree convergence, and bloom filter tests.
Shared test helpers extracted to tests/mod.rs.
Visibility adjusted from pub(super) to pub(in crate::node) for handler
methods now two levels deep. Unused imports cleaned up in node/mod.rs.
All 316 tests pass, zero warnings.
Johnathan Corgan
Martti MalmiandJohnathan Corgan
Alex XieandJohnathan Corgan
Sats And SportsandJohnathan Corgan