Johnathan Corgan
5abf9a9325
Restructures /docs/ by reader purpose (tutorials, how-to, reference, design), adds the new-user-progression and operator-recipe content the prior layout lacked, runs an accuracy pass against current source across the pre-existing design docs, and rewrites the gateway feature-set documentation end-to-end around its actual operational profile (a niche feature designed for systems already serving DHCP/DNS to a LAN, with two independent halves — outbound LAN→mesh, inbound mesh→LAN — sharing one nftables table, one binary, and one control socket). Top-level README and getting-started rewritten around two equally-weighted deployment modes (overlay on existing IP networks; ground-up over non-IP transports). ## Additions - 11 new tutorials in docs/tutorials/: an 8-step new-user progression from single-daemon test-mesh peering through to a ground-up two-device mesh, an IPv6-adapter side-trip walkthrough, an Advanced Tutorials index, and a hand-held OpenWrt walk-through for fips-gateway deployment that exercises both halves of the feature. - 12 new how-tos in docs/how-to/: firewall activation, Nostr discovery (resolve / advertise / open across five scenarios), Tor onion (directory + control_port modes), UDP buffer tuning, unprivileged-user setup, persistent identity, host aliases, Bluetooth LE peering, MTU diagnostics, manual Linux-host gateway deployment (covers both halves), gateway troubleshooting (organised by half), and a section index. - 9 new reference docs in docs/reference/: configuration, wire formats, control-socket protocol, four CLI references (fips, fipsctl, fipstop, fips-gateway), security posture matrix, and Nostr events catalog. Configuration and wire-formats are renamed-and-extended from prior design/ versions; the other seven are net-new. - 6 new design docs: fips-concepts, fips-architecture, and fips-prior-work split out of the deleted fips-intro.md; consolidated fips-mmp and fips-mtu aggregations; and a new generic port-advertisement-and-nat-traversal doc (Nostr-signaled port advertisement plus UDP NAT-traversal protocol, FIPS as an example implementation, suitable for eventual NIP submission). - Top-level docs/getting-started.md walking through the binary-installer-only Install story. - packaging/common/hosts pre-populated with the eight public test-mesh nodes so shortnames resolve out of the box on every fresh install. ## Changes - 23 wire-format diagrams relocated to reference/diagrams/ alongside the wire-formats move. - 4 design diagrams corrected against source code (fips-protocol-stack, fips-identity-derivation, fips-coordinate-discovery, fips-routing-decision). - 10 pre-existing design docs reconciled with current source. Numeric corrections: stale link-MMP report bounds (now [1s, 5s] with 200 ms cold-start floor); UDP default MTU (now 1280, IPv6 minimum); node_addr formula (SHA-256(pubkey)[..16]); Noise patterns (IK at link, XK at session); peer-ACL semantics (strict allowlist requires ALL in peers.deny); daemon DNS upstream ([::1]:5354); on-the-wire bloom-filter size (1,071 bytes); obsolete Cargo-feature references (PR #79 dropped them) removed. - Transport framing tightened across the docs: TCP is for UDP-filtered networks (not NAT traversal); Tor is a deployment mode (not failover); WebSocket dropped (not a shipped FIPS transport); WiFi promoted to Implemented via Ethernet in infrastructure mode; classic-Bluetooth row removed (BLE is the only Bluetooth-mode transport). - docs/design/fips-gateway.md rewritten end-to-end to lead with the niche-feature framing and the two-halves structure. Title moved from "FIPS Outbound LAN Gateway" to "FIPS Gateway"; architecture section describes the common machinery (the fips-gateway service, the nftables table, the control socket) before splitting into separate "Outbound Half" and "Inbound Half" sections of equal weight; security considerations split per-half; no Future Work section (speculative directions live in the project tracker, not in protocol design docs). Inbound port forwarding is a first-class half rather than a buried "Implemented Extensions" subsection. - Gateway terminology unified across all gateway docs as a separate Linux service running alongside the fips daemon (its own systemd unit / OpenWrt init script). Container- pattern terms (sidecar) are reserved for the Docker/Kubernetes sidecar deployment examples — the testing/sidecar/ tree, examples/k8s-sidecar/, examples/sidecar-nostr-relay/, examples/wireguard-sidecar-macos/, and the related CHANGELOG / top-level README entries — where the term carries its standard container meaning. - Net-new design body content: rekey section in fips-mesh-layer (Noise IK msg1/msg2 over the established link, K-bit cutover, drain window, smaller-NodeAddr-wins tie-breaker on dual-init); Mesh Size Estimation and Antipoison FPR Cap sections in fips-bloom-filters; Mesh-Interface Query Filter subsection in fips-ipv6-adapter; failure-suppression knobs and clock- skew tolerance in fips-nostr-discovery; loop-rejection and mid-chain ancestor swap added to spanning-tree propagation / stability rules; Priority Chain in fips-mesh-operation renumbered to match the routing-decision diagram. - Top-level README: dropped the stale nostr-discovery cargo-feature parenthetical. docs/README.md and the four section READMEs (tutorials, how-to, reference, design) refreshed for the new structure; index rows reflect both halves of the gateway feature and the new fips-gateway CLI reference. - Cargo.toml [package.metadata.deb] assets path updated for the fips-security.md move; .gitignore /reference/ rule anchored to repo root so docs/reference/ is trackable. - packaging/openwrt-ipk/files/etc/fips/fips.yaml configuration-doc URL updated to the new docs/reference/configuration.md location. ## Deletions - docs/design/fips-intro.md (split into the three new intro design docs). - docs/design/document-relationships.svg (orphan, no longer referenced). - docs/proposals/ tree removed; the only proposal it contained (the Nostr UDP hole-punch protocol) was rewritten as the new generic design/port-advertisement-and-nat-traversal.md.
4.1 KiB
Tune Host UDP Socket Buffers for FIPS
The FIPS UDP transport requests larger send and receive socket
buffers (default 2 MB each, doubled by the kernel to 4 MB actual)
than the Linux defaults provide. The kernel silently clamps the
request to net.core.rmem_max and net.core.wmem_max if those
sysctls are smaller than the requested size — which causes silent
packet drops under high throughput. For the design context (why FIPS
requests larger buffers and how SO_RXQ_OVFL feeds ECN congestion
detection), see
../design/fips-transport-layer.md.
This guide covers the host-side sysctl setup needed before deploying a high-throughput FIPS node.
Why this matters
The default Linux UDP receive buffer (net.core.rmem_default,
typically 212 KB) fills in roughly 2.5 ms at ~85 MB/s. Any stall in
the FIPS receive loop (decryption, routing, forwarding) causes the
kernel to drop incoming datagrams without notification — they don't
appear in recv errors, they don't trigger any application-visible
event. The drops show up only in SO_RXQ_OVFL on subsequent
packets, where FIPS surfaces them as congestion-detection events.
Setting rmem_max and wmem_max to at least the requested buffer
size prevents the kernel clamp and the silent drop loss it causes.
Step 1: Check current limits
sysctl net.core.rmem_max net.core.wmem_max
Typical defaults on stock Linux distributions are 212992 bytes (212 KB). FIPS requests 2 MB by default, which the kernel doubles internally to 4 MB; for the request to succeed without clamping, both sysctls must be at least 4194304 (4 MB).
Step 2: Set the limits temporarily
sudo sysctl -w net.core.rmem_max=4194304
sudo sysctl -w net.core.wmem_max=4194304
Verify:
sysctl net.core.rmem_max net.core.wmem_max
These changes take effect immediately for new socket binds but do not survive a reboot.
Step 3: Make the limits persistent
Drop a file under /etc/sysctl.d/:
sudo tee /etc/sysctl.d/60-fips.conf <<'EOF'
# FIPS UDP transport requests 2 MB socket buffers, kernel doubles to 4 MB.
# Avoid silent receive-buffer drops under load.
net.core.rmem_max = 4194304
net.core.wmem_max = 4194304
EOF
Apply:
sudo sysctl --system
The drop-in is loaded automatically on every boot.
Step 4: Restart FIPS and verify the actual buffer size
After raising the host limits, restart the FIPS daemon so the next socket bind picks up the new ceiling:
sudo systemctl restart fips
The daemon logs the actual buffer sizes at startup:
UDP transport started local_addr=0.0.0.0:2121 recv_buf=4194304 send_buf=4194304
If recv_buf or send_buf shows a smaller number than expected, the
host sysctl is still clamping. Recheck sysctl net.core.rmem_max net.core.wmem_max and confirm the drop-in file is being loaded
(sudo sysctl --system prints the loaded files).
Docker and other container hosts
Containers share the host kernel, so sysctls apply to the host, not
the container. If you run FIPS inside Docker, set
net.core.rmem_max / net.core.wmem_max on the Docker host, not
inside the container. Container privileges (cap_sys_admin) and
--sysctl flags do not let you raise these particular limits from
inside a container — they are global to the host network namespace.
For Kubernetes deployments, the host-level sysctl tuning is the same;
node-level configuration (DaemonSet with privileged: true, or a
node-init script) is the typical mechanism.
Tuning higher
The 4 MB ceiling is a conservative starting point. For very high
throughput (multi-gigabit per second), raise both sysctls and the
corresponding transports.udp.recv_buf_size /
transports.udp.send_buf_size config values together. Setting a config value
larger than the host ceiling silently clamps to the ceiling, so both
must move in lockstep.
See also
- ../design/fips-transport-layer.md
— UDP transport design, why FIPS requests larger buffers,
SO_RXQ_OVFLand ECN integration - ../reference/configuration.md —
transports.udp.recv_buf_sizeandsend_buf_sizedefaults