mirror of
https://github.com/jmcorgan/fips.git
synced 2026-07-30 19:46:15 +00:00
docs: four-section /docs/ restructure with new-user content, accuracy pass, and gateway feature-set rewrite
Restructures /docs/ by reader purpose (tutorials, how-to, reference, design), adds the new-user-progression and operator-recipe content the prior layout lacked, runs an accuracy pass against current source across the pre-existing design docs, and rewrites the gateway feature-set documentation end-to-end around its actual operational profile (a niche feature designed for systems already serving DHCP/DNS to a LAN, with two independent halves — outbound LAN→mesh, inbound mesh→LAN — sharing one nftables table, one binary, and one control socket). Top-level README and getting-started rewritten around two equally-weighted deployment modes (overlay on existing IP networks; ground-up over non-IP transports). ## Additions - 11 new tutorials in docs/tutorials/: an 8-step new-user progression from single-daemon test-mesh peering through to a ground-up two-device mesh, an IPv6-adapter side-trip walkthrough, an Advanced Tutorials index, and a hand-held OpenWrt walk-through for fips-gateway deployment that exercises both halves of the feature. - 12 new how-tos in docs/how-to/: firewall activation, Nostr discovery (resolve / advertise / open across five scenarios), Tor onion (directory + control_port modes), UDP buffer tuning, unprivileged-user setup, persistent identity, host aliases, Bluetooth LE peering, MTU diagnostics, manual Linux-host gateway deployment (covers both halves), gateway troubleshooting (organised by half), and a section index. - 9 new reference docs in docs/reference/: configuration, wire formats, control-socket protocol, four CLI references (fips, fipsctl, fipstop, fips-gateway), security posture matrix, and Nostr events catalog. Configuration and wire-formats are renamed-and-extended from prior design/ versions; the other seven are net-new. - 6 new design docs: fips-concepts, fips-architecture, and fips-prior-work split out of the deleted fips-intro.md; consolidated fips-mmp and fips-mtu aggregations; and a new generic port-advertisement-and-nat-traversal doc (Nostr-signaled port advertisement plus UDP NAT-traversal protocol, FIPS as an example implementation, suitable for eventual NIP submission). - Top-level docs/getting-started.md walking through the binary-installer-only Install story. - packaging/common/hosts pre-populated with the eight public test-mesh nodes so shortnames resolve out of the box on every fresh install. ## Changes - 23 wire-format diagrams relocated to reference/diagrams/ alongside the wire-formats move. - 4 design diagrams corrected against source code (fips-protocol-stack, fips-identity-derivation, fips-coordinate-discovery, fips-routing-decision). - 10 pre-existing design docs reconciled with current source. Numeric corrections: stale link-MMP report bounds (now [1s, 5s] with 200 ms cold-start floor); UDP default MTU (now 1280, IPv6 minimum); node_addr formula (SHA-256(pubkey)[..16]); Noise patterns (IK at link, XK at session); peer-ACL semantics (strict allowlist requires ALL in peers.deny); daemon DNS upstream ([::1]:5354); on-the-wire bloom-filter size (1,071 bytes); obsolete Cargo-feature references (PR #79 dropped them) removed. - Transport framing tightened across the docs: TCP is for UDP-filtered networks (not NAT traversal); Tor is a deployment mode (not failover); WebSocket dropped (not a shipped FIPS transport); WiFi promoted to Implemented via Ethernet in infrastructure mode; classic-Bluetooth row removed (BLE is the only Bluetooth-mode transport). - docs/design/fips-gateway.md rewritten end-to-end to lead with the niche-feature framing and the two-halves structure. Title moved from "FIPS Outbound LAN Gateway" to "FIPS Gateway"; architecture section describes the common machinery (the fips-gateway service, the nftables table, the control socket) before splitting into separate "Outbound Half" and "Inbound Half" sections of equal weight; security considerations split per-half; no Future Work section (speculative directions live in the project tracker, not in protocol design docs). Inbound port forwarding is a first-class half rather than a buried "Implemented Extensions" subsection. - Gateway terminology unified across all gateway docs as a separate Linux service running alongside the fips daemon (its own systemd unit / OpenWrt init script). Container- pattern terms (sidecar) are reserved for the Docker/Kubernetes sidecar deployment examples — the testing/sidecar/ tree, examples/k8s-sidecar/, examples/sidecar-nostr-relay/, examples/wireguard-sidecar-macos/, and the related CHANGELOG / top-level README entries — where the term carries its standard container meaning. - Net-new design body content: rekey section in fips-mesh-layer (Noise IK msg1/msg2 over the established link, K-bit cutover, drain window, smaller-NodeAddr-wins tie-breaker on dual-init); Mesh Size Estimation and Antipoison FPR Cap sections in fips-bloom-filters; Mesh-Interface Query Filter subsection in fips-ipv6-adapter; failure-suppression knobs and clock- skew tolerance in fips-nostr-discovery; loop-rejection and mid-chain ancestor swap added to spanning-tree propagation / stability rules; Priority Chain in fips-mesh-operation renumbered to match the routing-decision diagram. - Top-level README: dropped the stale nostr-discovery cargo-feature parenthetical. docs/README.md and the four section READMEs (tutorials, how-to, reference, design) refreshed for the new structure; index rows reflect both halves of the gateway feature and the new fips-gateway CLI reference. - Cargo.toml [package.metadata.deb] assets path updated for the fips-security.md move; .gitignore /reference/ rule anchored to repo root so docs/reference/ is trackable. - packaging/openwrt-ipk/files/etc/fips/fips.yaml configuration-doc URL updated to the new docs/reference/configuration.md location. ## Deletions - docs/design/fips-intro.md (split into the three new intro design docs). - docs/design/document-relationships.svg (orphan, no longer referenced). - docs/proposals/ tree removed; the only proposal it contained (the Nostr UDP hole-punch protocol) was rewritten as the new generic design/port-advertisement-and-nat-traversal.md.
This commit is contained in:
Johnathan Corgan
@@ -0,0 +1,27 @@
|
||||
# How-To Guides
|
||||
|
||||
Task-oriented, step-by-step recipes for operators with a specific
|
||||
goal in mind. Each guide assumes the reader already knows what FIPS
|
||||
is and wants to get a particular thing done — enable a feature,
|
||||
deploy a component, troubleshoot a class of problem.
|
||||
|
||||
How-to guides do not teach concepts (that is the role of design/)
|
||||
and do not enumerate options (that is the role of reference/). They
|
||||
take the reader along the shortest correct path from "I want to do
|
||||
X" to "X is done".
|
||||
|
||||
## Available Guides
|
||||
|
||||
| Guide | Goal |
|
||||
| ----- | ---- |
|
||||
| [enable-mesh-firewall.md](enable-mesh-firewall.md) | Activate the default-deny nftables baseline on `fips0` |
|
||||
| [enable-nostr-discovery.md](enable-nostr-discovery.md) | Turn on Nostr-mediated discovery (3 capabilities — resolve, advertise, open — across 5 scenarios) |
|
||||
| [deploy-tor-onion.md](deploy-tor-onion.md) | Run a Tor onion service for inbound FIPS connections |
|
||||
| [tune-udp-buffers.md](tune-udp-buffers.md) | Set host sysctls so FIPS UDP sockets don't get clamped |
|
||||
| [run-as-unprivileged-user.md](run-as-unprivileged-user.md) | Run the daemon under a dedicated unprivileged service account (drops the default-root posture) |
|
||||
| [deploy-gateway.md](deploy-gateway.md) | Manually deploy `fips-gateway` on a non-OpenWrt Linux host (LAN-to-mesh outbound + mesh-to-LAN inbound port-forwards). For the OpenWrt path, see the gateway tutorial. |
|
||||
| [troubleshoot-gateway.md](troubleshoot-gateway.md) | Diagnostic recipes for the gateway, organised by half (outbound, inbound, common) |
|
||||
| [persistent-identity.md](persistent-identity.md) | Provision a stable Nostr keypair so the node keeps the same npub across restarts |
|
||||
| [host-aliases.md](host-aliases.md) | Use shortnames (`test-us01.fips`, `my-laptop.fips`) instead of full npubs by editing `/etc/fips/hosts` or setting peer aliases |
|
||||
| [set-up-bluetooth-peer.md](set-up-bluetooth-peer.md) | Configure a Bluetooth Low Energy peer link |
|
||||
| [diagnose-mtu-issues.md](diagnose-mtu-issues.md) | Triage MTU-shaped failures and rule out their imposters (bufferbloat, transport saturation) |
|
||||
@@ -0,0 +1,450 @@
|
||||
# Deploy `fips-gateway` (Manual Linux-Host Setup)
|
||||
|
||||
`fips-gateway` is a separate service that runs alongside the FIPS
|
||||
daemon and bridges a non-FIPS LAN to the FIPS mesh in two
|
||||
independent directions: **outbound** (LAN clients reach mesh
|
||||
services through DNS proxy + virtual-IP NAT) and **inbound** (mesh
|
||||
peers reach LAN services through 1:1 port forwards on `fips0`).
|
||||
This guide covers the **manual Linux-host** deployment path —
|
||||
wiring DNS forwarding, route distribution, and firewall integration
|
||||
on a server or non-OpenWrt router by hand.
|
||||
|
||||
> **Running OpenWrt?** Use the
|
||||
> [tutorial](../tutorials/deploy-fips-gateway.md) instead. The OpenWrt
|
||||
> ipk ships with the `gateway:` block pre-populated and the init
|
||||
> script automates dnsmasq forwarding, RA route distribution, and the
|
||||
> global IPv6 prefix on `br-lan`. The OpenWrt path is the canonical
|
||||
> deployment of this feature; this how-to is the secondary path for
|
||||
> operators with a different LAN-edge box (a Linux server already
|
||||
> serving DHCP/DNS, a custom router distribution, etc.).
|
||||
|
||||
For the gateway design (NAT pipeline, virtual IP pool lifecycle, DNS
|
||||
resolution flow), see [../design/fips-gateway.md](../design/fips-gateway.md).
|
||||
For the full `gateway.*` configuration block, see the
|
||||
[Gateway section](../reference/configuration.md#gateway-gateway) of
|
||||
the configuration reference. For the `fips-gateway` binary's CLI
|
||||
flags, see [../reference/cli-fips-gateway.md](../reference/cli-fips-gateway.md).
|
||||
|
||||
## The two halves
|
||||
|
||||
The gateway exposes two independent features that share a common
|
||||
control plane (the same binary, the same nftables table `inet
|
||||
fips_gateway`, the same control socket `/run/fips/gateway.sock`, the
|
||||
same `gateway.*` config block). You can configure either half on its
|
||||
own or both together.
|
||||
|
||||
- **Outbound gateway** (LAN → mesh). Non-FIPS LAN workstations resolve
|
||||
`<npub>.fips` names against the gateway's DNS listener and receive
|
||||
AAAA answers from the gateway's virtual-IP pool. Outbound traffic
|
||||
to those addresses is DNAT'd to the real mesh address and SNAT'd
|
||||
(masqueraded) onto `fips0` under the gateway's mesh identity. The
|
||||
audience is unmodified LAN clients.
|
||||
|
||||
- **Inbound gateway** (mesh → LAN). A static `(listen_port, proto)
|
||||
→ [target_addr]:target_port` table — configured in
|
||||
`gateway.port_forwards[]` — exposes selected LAN services to the
|
||||
mesh as `<gateway-npub>.fips:<listen_port>`. Mesh peers connect to
|
||||
the gateway's mesh address; the gateway DNATs to the LAN target
|
||||
and masquerades on the LAN side so return traffic flows through
|
||||
conntrack. The audience is mesh peers reaching a service that
|
||||
happens to live on this LAN.
|
||||
|
||||
The two halves are independent. Configure the outbound half if you
|
||||
want LAN clients to *reach* the mesh; configure the inbound half if
|
||||
you want mesh peers to *reach into* the LAN; configure both if you
|
||||
want both.
|
||||
|
||||
## Common gateway-host setup
|
||||
|
||||
Both halves require the same host preparation. Work through this
|
||||
section first, then jump to whichever half (or both) you need.
|
||||
|
||||
### FIPS daemon prerequisites
|
||||
|
||||
The gateway runs alongside a `fips` daemon on the same host:
|
||||
|
||||
- The daemon must be running with the TUN adapter enabled (the
|
||||
`fips0` interface must exist).
|
||||
- The daemon's DNS resolver must be enabled (`dns.enabled: true`,
|
||||
default) and reachable from `fips-gateway`. By default that means
|
||||
`[::1]:5354` (IPv6 loopback). The gateway's default
|
||||
`dns.upstream` matches this; a v4 upstream like `127.0.0.1:5354`
|
||||
cannot reach a daemon bound on `[::1]:5354` because Linux IPv6
|
||||
sockets bound to explicit `::1` do not accept v4-mapped traffic.
|
||||
|
||||
If the daemon is not yet running with these features, set up the
|
||||
daemon first — see [persistent-identity.md](persistent-identity.md)
|
||||
and [../reference/configuration.md](../reference/configuration.md).
|
||||
|
||||
### Kernel sysctls
|
||||
|
||||
```sh
|
||||
sudo sysctl -w net.ipv6.conf.all.forwarding=1
|
||||
sudo sysctl -w net.ipv6.conf.all.proxy_ndp=1
|
||||
```
|
||||
|
||||
`forwarding` lets the host route IPv6 packets between the LAN
|
||||
interface and `fips0`. `proxy_ndp` lets the gateway answer Neighbor
|
||||
Solicitation requests for virtual-pool addresses so LAN clients can
|
||||
resolve their link-layer addresses (only relevant for the outbound
|
||||
half, but harmless if you only run the inbound half).
|
||||
|
||||
Persist via a drop-in:
|
||||
|
||||
```sh
|
||||
sudo tee /etc/sysctl.d/60-fips-gateway.conf <<'EOF'
|
||||
net.ipv6.conf.all.forwarding = 1
|
||||
net.ipv6.conf.all.proxy_ndp = 1
|
||||
EOF
|
||||
sudo sysctl --system
|
||||
```
|
||||
|
||||
### Capability
|
||||
|
||||
`fips-gateway` requires `CAP_NET_ADMIN` to manage its nftables table
|
||||
(`inet fips_gateway`) and proxy-NDP entries. The packaged systemd
|
||||
unit (`fips-gateway.service`) runs as root, which satisfies this. For
|
||||
non-package installs, set the file capability:
|
||||
|
||||
```sh
|
||||
sudo setcap cap_net_admin+ep /usr/bin/fips-gateway
|
||||
```
|
||||
|
||||
### Pool route
|
||||
|
||||
At startup `fips-gateway` adds `local <pool-cidr> dev lo` to the
|
||||
local routing table. This tells the kernel to accept packets
|
||||
destined for pool addresses as locally-owned, enabling the NAT
|
||||
processing path. The route is cleaned up on shutdown. You do not
|
||||
need to install it manually; if you see "destination unreachable"
|
||||
errors for pool addresses on the gateway host, verify the route is
|
||||
present:
|
||||
|
||||
```sh
|
||||
ip -6 route show table local | grep <pool-cidr>
|
||||
```
|
||||
|
||||
### Minimum configuration
|
||||
|
||||
In `/etc/fips/fips.yaml`, populate the `gateway` block with at minimum
|
||||
`enabled: true`, `pool`, and `lan_interface`:
|
||||
|
||||
```yaml
|
||||
gateway:
|
||||
enabled: true
|
||||
pool: "fd01::/112"
|
||||
lan_interface: "enp3s0"
|
||||
```
|
||||
|
||||
Pick a pool CIDR that does **not** overlap with any address space in
|
||||
use on the LAN or in the mesh (the FIPS mesh occupies `fd00::/8`;
|
||||
pick a different `fdXX::/N`). The `/112` size yields 65 536 virtual
|
||||
IPs, which is the gateway's hard cap regardless of CIDR width.
|
||||
|
||||
This minimum config is enough to start the gateway. The `dns.*` block
|
||||
is optional and defaults to `listen: "[::]:53"` and
|
||||
`upstream: "[::1]:5354"`. The full block — including `dns.*`,
|
||||
`pool_grace_period`, `conntrack.*`, and `port_forwards[]` — is
|
||||
documented in
|
||||
[../reference/configuration.md#gateway-gateway](../reference/configuration.md#gateway-gateway).
|
||||
|
||||
### Start the service
|
||||
|
||||
```sh
|
||||
sudo systemctl enable --now fips-gateway
|
||||
```
|
||||
|
||||
Verify the unit came up:
|
||||
|
||||
```sh
|
||||
sudo systemctl status fips-gateway
|
||||
sudo journalctl -u fips-gateway -e
|
||||
```
|
||||
|
||||
The startup log will report `Gateway config loaded`,
|
||||
`DNS upstream is reachable`, `Created nftables table 'fips_gateway'`,
|
||||
and finally `fips-gateway running`. The unit's `ExecStartPre` waits up
|
||||
to 30 s for `fips0` to appear, which covers the cold-boot race where
|
||||
the daemon is still bringing up its TUN.
|
||||
|
||||
## Configure the outbound half
|
||||
|
||||
The outbound half lets LAN clients resolve `.fips` names and reach
|
||||
mesh destinations. Three operator decisions are involved: pool CIDR,
|
||||
DNS listen address, and how LAN clients learn the route to the pool
|
||||
and the resolver address.
|
||||
|
||||
### Choose the pool CIDR
|
||||
|
||||
```yaml
|
||||
gateway:
|
||||
pool: "fd01::/112"
|
||||
```
|
||||
|
||||
Constraints:
|
||||
|
||||
- Must not overlap with `fd00::/8` (the FIPS mesh address space).
|
||||
- Must not overlap with any LAN-side IPv6 prefix already in use.
|
||||
- `/112` is the practical width — wider just wastes address space
|
||||
because the pool is hard-capped at 65 536 entries. Narrower is
|
||||
fine if you want a smaller pool, but you'll reject DNS lookups
|
||||
faster under churn.
|
||||
|
||||
### Choose the DNS listen address
|
||||
|
||||
```yaml
|
||||
gateway:
|
||||
dns:
|
||||
listen: "[::]:53"
|
||||
upstream: "[::1]:5354"
|
||||
ttl: 60
|
||||
```
|
||||
|
||||
Common cases:
|
||||
|
||||
- **No other resolver on the host:** `listen: "[::]:53"` is the
|
||||
default and works.
|
||||
- **systemd-resolved is on port 53:** either disable its stub
|
||||
listener (`DNSStubListener=no` in
|
||||
`/etc/systemd/resolved.conf`) or move the gateway to a different
|
||||
port (e.g., `[::]:5353`) and put a forwarder on 53 that delegates
|
||||
`.fips` to the gateway. See
|
||||
[troubleshoot-gateway.md](troubleshoot-gateway.md#port-conflict-on-the-dns-listen-port).
|
||||
- **Bind on the LAN address only:** `listen: "192.168.1.1:53"`
|
||||
exposes the resolver only to LAN clients, not loopback.
|
||||
|
||||
The gateway returns `REFUSED` for any non-`.fips` query — clients
|
||||
that point at it directly need a fallback resolver, or you should
|
||||
front it with a stub forwarder.
|
||||
|
||||
### Distribute the route to LAN clients
|
||||
|
||||
Each LAN client must route the gateway's pool CIDR to the gateway's
|
||||
LAN-side IPv6 address. Three options, in order of preference for
|
||||
production:
|
||||
|
||||
- **RA Route Information Option** (RFC 4191). If the LAN's RA daemon
|
||||
(`radvd`, `dnsmasq --enable-ra`, OpenWrt's `odhcpd`) supports
|
||||
publishing route options, configure it to advertise the pool CIDR
|
||||
with the gateway as next-hop. Clients pick this up automatically.
|
||||
|
||||
- **Static route on the LAN router**. If clients route through a
|
||||
central LAN router, add a static route entry there — the router
|
||||
then handles forwarding to the gateway. The exact syntax depends
|
||||
on the router OS.
|
||||
|
||||
- **Per-host static route** (testing or single-client deployments):
|
||||
|
||||
```sh
|
||||
sudo ip -6 route add fd01::/112 via fe80::<gateway-link-local>%<iface>
|
||||
# or, if the gateway has a stable global LAN address:
|
||||
sudo ip -6 route add fd01::/112 via <gateway-lan-addr>
|
||||
```
|
||||
|
||||
### Distribute the resolver to LAN clients
|
||||
|
||||
LAN clients also need to send `.fips` queries to the gateway. Two
|
||||
patterns:
|
||||
|
||||
- **Forward `.fips` from the LAN's main resolver.** If the LAN runs
|
||||
Pi-hole, Unbound, dnsmasq, or systemd-resolved as the central
|
||||
resolver, configure a conditional forward for `fips.`. Unbound
|
||||
example:
|
||||
|
||||
```text
|
||||
forward-zone:
|
||||
name: "fips."
|
||||
forward-addr: <gateway-lan-addr>@53
|
||||
```
|
||||
|
||||
dnsmasq example:
|
||||
|
||||
```text
|
||||
server=/fips/<gateway-lan-addr>
|
||||
```
|
||||
|
||||
Clients keep their existing DNS settings; only `.fips` queries are
|
||||
diverted.
|
||||
|
||||
- **Point clients directly at the gateway.** Simpler for testing,
|
||||
but the gateway returns `REFUSED` for non-`.fips` queries, so each
|
||||
client must also have a fallback resolver configured.
|
||||
|
||||
### Verify the outbound path
|
||||
|
||||
From a LAN client:
|
||||
|
||||
```sh
|
||||
dig @<gateway-lan-addr> hostname.fips AAAA
|
||||
# Expect an AAAA from the pool CIDR
|
||||
|
||||
ping6 hostname.fips
|
||||
# Should succeed via the gateway
|
||||
```
|
||||
|
||||
If either step fails, see
|
||||
[troubleshoot-gateway.md](troubleshoot-gateway.md#outbound-half-diagnostics).
|
||||
|
||||
## Configure the inbound half
|
||||
|
||||
The inbound half exposes a LAN-side service to mesh peers. Configured
|
||||
under `gateway.port_forwards[]`:
|
||||
|
||||
```yaml
|
||||
gateway:
|
||||
port_forwards:
|
||||
- listen_port: 8080
|
||||
proto: tcp
|
||||
target: "[fd12:3456::10]:80"
|
||||
- listen_port: 2222
|
||||
proto: tcp
|
||||
target: "[fd12:3456::20]:22"
|
||||
- listen_port: 5353
|
||||
proto: udp
|
||||
target: "[fd12:3456::10]:53"
|
||||
```
|
||||
|
||||
Field reference:
|
||||
|
||||
- `listen_port` — port on the gateway's `fips0` mesh-side address
|
||||
that mesh peers connect to. Must be non-zero. Each
|
||||
`(listen_port, proto)` pair must be unique across the list (the
|
||||
same port on TCP and UDP is allowed; the same port twice on the
|
||||
same proto is rejected at config-load time).
|
||||
- `proto` — `tcp` or `udp`.
|
||||
- `target` — IPv6 LAN destination as `[addr]:port`. IPv4 targets are
|
||||
rejected at parse time by the YAML deserializer (the field is
|
||||
typed `SocketAddrV6`). If the LAN host is reachable only by IPv4,
|
||||
put a small IPv6-aware reverse proxy in front of it on the gateway
|
||||
itself.
|
||||
|
||||
### Worked example: HTTP and DNS
|
||||
|
||||
Suppose the gateway runs on a LAN with an HTTP server at
|
||||
`[fd12:3456::10]:80` and a recursive resolver at
|
||||
`[fd12:3456::10]:53`, and you want mesh peers to reach them as
|
||||
`<gateway-npub>.fips:8080` (HTTP) and `<gateway-npub>.fips:5353`
|
||||
(DNS). Add to the gateway's `fips.yaml`:
|
||||
|
||||
```yaml
|
||||
gateway:
|
||||
port_forwards:
|
||||
- listen_port: 8080
|
||||
proto: tcp
|
||||
target: "[fd12:3456::10]:80"
|
||||
- listen_port: 5353
|
||||
proto: udp
|
||||
target: "[fd12:3456::10]:53"
|
||||
```
|
||||
|
||||
Reload:
|
||||
|
||||
```sh
|
||||
sudo systemctl restart fips-gateway
|
||||
```
|
||||
|
||||
From any mesh peer (the host name `gateway` is whatever the gateway's
|
||||
npub maps to in the local `hosts` file or via Nostr advert):
|
||||
|
||||
```sh
|
||||
curl http://gateway.fips:8080/
|
||||
dig @gateway.fips -p 5353 example.com A
|
||||
```
|
||||
|
||||
Each mesh-side request enters `fips0` on the listen port, gets DNAT'd
|
||||
to the LAN target, and the LAN-side masquerade rule rewrites the
|
||||
source to the gateway's LAN address so return traffic flows back
|
||||
through conntrack.
|
||||
|
||||
### Compose with the mesh firewall
|
||||
|
||||
`gateway.port_forwards[]` opens *mesh-side* listeners on `fips0`. If
|
||||
the host's mesh firewall is enabled (see
|
||||
[enable-mesh-firewall.md](enable-mesh-firewall.md)), inbound TCP/UDP
|
||||
on `fips0` for these ports must be permitted in the baseline or via
|
||||
a drop-in. The default baseline allows established/related and
|
||||
ICMPv6 only, so without an explicit allow rule, mesh peers will see
|
||||
TCP RSTs or silent drops on the listen port.
|
||||
|
||||
A typical drop-in for the worked example:
|
||||
|
||||
```nft
|
||||
# /etc/fips/fips.d/gateway-inbound.nft
|
||||
tcp dport 8080 accept
|
||||
udp dport 5353 accept
|
||||
```
|
||||
|
||||
Reload the firewall:
|
||||
|
||||
```sh
|
||||
sudo systemctl reload-or-restart fips-firewall.service
|
||||
```
|
||||
|
||||
If the inbound half doesn't need access control beyond the listen
|
||||
port itself, no source filter is needed. To restrict to specific
|
||||
mesh peers, follow the `ip6 saddr <addr> tcp dport <port> accept`
|
||||
pattern from the firewall guide.
|
||||
|
||||
### Verify the inbound path
|
||||
|
||||
From a mesh peer (any FIPS node):
|
||||
|
||||
```sh
|
||||
curl -v http://<gateway-npub>.fips:8080/
|
||||
```
|
||||
|
||||
A successful response confirms the full path: mesh ingress on
|
||||
`fips0`, DNAT to the LAN target, LAN-side masquerade, and conntrack-
|
||||
tracked return. If it fails, see
|
||||
[troubleshoot-gateway.md](troubleshoot-gateway.md#inbound-half-diagnostics).
|
||||
|
||||
## Operate and verify
|
||||
|
||||
`fips-gateway` exposes its own control socket at
|
||||
`/run/fips/gateway.sock`, separate from the daemon's
|
||||
`/run/fips/control.sock`. There is no `fipsctl gateway` subcommand —
|
||||
talk to it directly:
|
||||
|
||||
```sh
|
||||
echo '{"command":"show_gateway"}' | sudo nc -U /run/fips/gateway.sock
|
||||
echo '{"command":"show_mappings"}' | sudo nc -U /run/fips/gateway.sock
|
||||
```
|
||||
|
||||
`show_gateway` returns pool counters (`pool_total`, `pool_allocated`,
|
||||
`pool_active`, `pool_draining`, `pool_free`), `nat_mappings`,
|
||||
`dns_listen`, `uptime_secs`, and the active config snapshot.
|
||||
`show_mappings` returns the per-allocation list with virtual IP, mesh
|
||||
address, npub-derived `node_addr`, dns name, state (`Allocated`,
|
||||
`Active`, `Draining`), session count, and ages. For the full schema
|
||||
see [../reference/control-socket.md#gateway-command-catalog](../reference/control-socket.md#gateway-command-catalog).
|
||||
|
||||
The journal is the other primary signal:
|
||||
|
||||
```sh
|
||||
sudo systemctl status fips-gateway
|
||||
sudo journalctl -u fips-gateway -e
|
||||
```
|
||||
|
||||
Expect `MappingCreated`/`MappingRemoved` debug lines as DNS-driven
|
||||
allocations come and go (run with `--log-level debug` to see them),
|
||||
and `Final pool status` on shutdown. Errors in adding NAT rules or
|
||||
proxy-NDP entries surface here.
|
||||
|
||||
## See also
|
||||
|
||||
- [../tutorials/deploy-fips-gateway.md](../tutorials/deploy-fips-gateway.md) —
|
||||
the canonical, package-driven OpenWrt deployment path.
|
||||
- [../design/fips-gateway.md](../design/fips-gateway.md) — gateway
|
||||
design, NAT pipeline, virtual IP pool lifecycle, security
|
||||
considerations.
|
||||
- [Gateway section](../reference/configuration.md#gateway-gateway) of
|
||||
the configuration reference — full `gateway.*` block.
|
||||
- [../reference/cli-fips-gateway.md](../reference/cli-fips-gateway.md) —
|
||||
`fips-gateway` binary CLI flags.
|
||||
- [Gateway command catalog](../reference/control-socket.md#gateway-command-catalog)
|
||||
in the control-socket reference — JSON schema for `show_gateway`
|
||||
and `show_mappings`.
|
||||
- [troubleshoot-gateway.md](troubleshoot-gateway.md) — diagnostic
|
||||
recipes grouped by half.
|
||||
- [enable-mesh-firewall.md](enable-mesh-firewall.md) — mesh-firewall
|
||||
baseline and drop-ins (needed when exposing inbound ports).
|
||||
@@ -0,0 +1,217 @@
|
||||
# Deploy a Tor Onion Service for FIPS
|
||||
|
||||
This guide covers running a Tor onion service that accepts inbound
|
||||
FIPS peer connections.
|
||||
|
||||
For the Tor transport's design and the bridge-node pattern (running
|
||||
Tor and UDP simultaneously), see
|
||||
[../design/fips-transport-layer.md](../design/fips-transport-layer.md).
|
||||
For the full `transports.tor.*` config knob inventory, see
|
||||
[../reference/configuration.md](../reference/configuration.md).
|
||||
|
||||
## Inbound modes
|
||||
|
||||
FIPS supports two inbound Tor modes. (A third mode, `socks5`, is
|
||||
outbound-only and not covered here.)
|
||||
|
||||
- **`directory` mode** *(recommended)*. Tor manages the onion
|
||||
service via `HiddenServiceDir` and `HiddenServicePort` directives
|
||||
in `torrc`. FIPS reads the resulting `.onion` hostname from a
|
||||
file and binds a local TCP listener for Tor to forward inbound
|
||||
connections to. No control-port interaction is required, which
|
||||
makes this mode compatible with Tor's `Sandbox 1` seccomp-bpf
|
||||
hardening.
|
||||
- **`torrc` requires:** `HiddenServiceDir` + `HiddenServicePort`.
|
||||
- **`control_port` mode**. FIPS speaks to Tor's control port to
|
||||
create an ephemeral onion service at startup (`ADD_ONION`). The
|
||||
onion key lives only for the lifetime of the FIPS daemon's
|
||||
control-port session. This mode is **incompatible** with
|
||||
`Sandbox 1` — the sandbox forbids control-port-driven onion
|
||||
service management.
|
||||
- **`torrc` requires:** `ControlPort` (typically the Unix socket
|
||||
`/run/tor/control`) and a usable auth method
|
||||
(`CookieAuthentication 1` is the common choice).
|
||||
|
||||
Pick `directory` unless you have a specific reason to prefer
|
||||
`control_port`. The rest of this guide covers `directory` mode
|
||||
end-to-end.
|
||||
|
||||
## Prerequisites
|
||||
|
||||
- Tor daemon installed and running (Debian/Ubuntu: `apt install tor`)
|
||||
- FIPS daemon configured and able to start
|
||||
- Operator access to `/etc/tor/torrc` (or a drop-in under
|
||||
`/etc/tor/torrc.d/`)
|
||||
|
||||
## Step 1: Configure Tor's HiddenServiceDir
|
||||
|
||||
Add the following to `/etc/tor/torrc`:
|
||||
|
||||
```text
|
||||
HiddenServiceDir /var/lib/tor/fips
|
||||
HiddenServicePort 8443 127.0.0.1:8444
|
||||
```
|
||||
|
||||
`HiddenServiceDir` tells Tor where to store the onion service's
|
||||
private key and `hostname` file. `HiddenServicePort` declares that
|
||||
inbound TCP traffic to port 8443 of the onion address should be
|
||||
forwarded to `127.0.0.1:8444` on the local host — that is where FIPS
|
||||
will bind its listener.
|
||||
|
||||
The external port (`8443` here) is what peers will connect to over
|
||||
Tor; the internal target (`127.0.0.1:8444`) is purely local and is
|
||||
not directly reachable from the network.
|
||||
|
||||
## Step 2: Reload Tor and read the onion hostname
|
||||
|
||||
```sh
|
||||
sudo systemctl reload tor@default # or `tor` on systems without instance support
|
||||
```
|
||||
|
||||
After Tor processes the new config, the hostname file appears:
|
||||
|
||||
```sh
|
||||
sudo cat /var/lib/tor/fips/hostname
|
||||
# xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx.onion
|
||||
```
|
||||
|
||||
Tor regenerates the onion key only on first run (or if you remove
|
||||
`HiddenServiceDir`). The `hostname` value is stable across daemon
|
||||
restarts as long as `HiddenServiceDir` is preserved.
|
||||
|
||||
## Step 3: Verify HiddenServiceDir permissions
|
||||
|
||||
The directory must be readable only by the Tor user (Tor refuses to
|
||||
start otherwise):
|
||||
|
||||
```sh
|
||||
ls -la /var/lib/tor/fips
|
||||
# drwx------ debian-tor debian-tor ...
|
||||
```
|
||||
|
||||
With the shipped Debian systemd unit, FIPS runs as root and reads
|
||||
the `hostname` file directly — no permission adjustment is needed.
|
||||
|
||||
### Non-default deployments
|
||||
|
||||
If you run FIPS as an unprivileged user (custom packaging,
|
||||
hardened deployment, etc.), the FIPS daemon user needs read access
|
||||
to `hostname`. Options:
|
||||
|
||||
- Add the FIPS user to the `debian-tor` group and loosen group
|
||||
read on `HiddenServiceDir` (Tor still requires the directory
|
||||
itself to be `0700`, so this typically means making `hostname`
|
||||
itself group-readable rather than the directory).
|
||||
- Read `hostname` once at startup as root, then drop privileges.
|
||||
- Copy the hostname into a path the FIPS user can read, refreshed
|
||||
whenever the onion key changes.
|
||||
|
||||
## Step 4: Configure the FIPS Tor transport
|
||||
|
||||
In `/etc/fips/fips.yaml`, configure `transports.tor` with `mode:
|
||||
directory`:
|
||||
|
||||
```yaml
|
||||
transports:
|
||||
tor:
|
||||
mode: directory
|
||||
socks5_addr: "127.0.0.1:9050"
|
||||
connect_timeout_ms: 120000
|
||||
mtu: 1400
|
||||
advertised_port: 8443
|
||||
directory_service:
|
||||
hostname_file: "/var/lib/tor/fips/hostname"
|
||||
bind_addr: "127.0.0.1:8444"
|
||||
```
|
||||
|
||||
The `bind_addr` must match the *target* of the `HiddenServicePort`
|
||||
directive in `torrc`. The `hostname_file` path must match
|
||||
`HiddenServiceDir` plus `/hostname`.
|
||||
|
||||
`advertised_port` is the *virtual* onion port peers dial — i.e. the
|
||||
first number on the `HiddenServicePort` line, **not** the local
|
||||
target. The default is `443`; this guide uses `8443` on both sides
|
||||
to match the `HiddenServicePort 8443 127.0.0.1:8444` example
|
||||
above. Setting this explicitly is important if you ever flip
|
||||
`advertise_on_nostr: true`: the published advert otherwise
|
||||
defaults to `tor:<hash>.onion:443`, which won't match the actual
|
||||
onion port.
|
||||
|
||||
The `socks5_addr` is the Tor SOCKS5 proxy used for *outbound*
|
||||
connections to other onion services or clearnet endpoints (separate
|
||||
from inbound onion service handling).
|
||||
|
||||
Optional monitoring knobs: `control_addr` and `control_auth` (e.g.
|
||||
`/run/tor/control` and `cookie`) let the daemon read Tor's status
|
||||
through the control port even in `directory` mode. They are
|
||||
non-fatal on failure — the onion service still works without them.
|
||||
See [../reference/configuration.md](../reference/configuration.md)
|
||||
for the full key list and examples.
|
||||
|
||||
## Step 5: Reload the FIPS daemon
|
||||
|
||||
```sh
|
||||
sudo systemctl reload-or-restart fips
|
||||
```
|
||||
|
||||
At startup the daemon reads the `.onion` hostname from
|
||||
`hostname_file`, binds `127.0.0.1:8444`, and announces the onion
|
||||
endpoint internally. From this point inbound connections to
|
||||
`<your-onion>.onion:8443` arrive at FIPS over Tor.
|
||||
|
||||
## Step 6: Verify
|
||||
|
||||
Check that the FIPS daemon log shows the onion endpoint at startup:
|
||||
|
||||
```sh
|
||||
sudo journalctl -u fips -e | grep -i 'onion\|directory'
|
||||
```
|
||||
|
||||
You should see a line indicating the onion address FIPS will accept
|
||||
inbound connections on, and that the local bind on `127.0.0.1:8444`
|
||||
succeeded.
|
||||
|
||||
From another node configured with the Tor transport in `socks5` or
|
||||
`directory` mode, attempt to dial:
|
||||
|
||||
```sh
|
||||
fipsctl connect <peer-npub-or-hostname> <your-onion>.onion:8443 tor
|
||||
```
|
||||
|
||||
A successful `fipsctl show peers` afterwards on the inbound side
|
||||
shows the new peer with `transport=tor`.
|
||||
|
||||
## Optional: advertise the onion endpoint via Nostr discovery
|
||||
|
||||
If `node.discovery.nostr.enabled: true`, set
|
||||
`transports.tor.advertise_on_nostr: true` so the onion endpoint
|
||||
appears in this node's published advert. See
|
||||
[enable-nostr-discovery.md](enable-nostr-discovery.md) Scenario 2.
|
||||
|
||||
## Troubleshooting
|
||||
|
||||
- **Tor refuses to start with `Sandbox 1` and onion-service errors.**
|
||||
`Sandbox 1` requires `directory` mode and forbids creating onion
|
||||
services through the control port. Verify your `torrc` uses
|
||||
`HiddenServiceDir` (this guide), not `ADD_ONION` via control port.
|
||||
- **FIPS daemon fails to bind `127.0.0.1:8444`.** Another process is
|
||||
already bound to that port. Either stop the conflicting process or
|
||||
pick a different port and update both `torrc`'s
|
||||
`HiddenServicePort` target and `fips.yaml`'s `bind_addr` to match.
|
||||
- **Onion hostname is empty or missing.** Check `journalctl -u tor`
|
||||
for permission errors on `HiddenServiceDir`. The directory must be
|
||||
owned by the Tor user with mode `0700`.
|
||||
- **FIPS daemon cannot read `hostname_file`.** File is owned by the
|
||||
Tor user and not readable by the FIPS daemon user. Adjust
|
||||
permissions, or copy the hostname into a path the FIPS user can
|
||||
read.
|
||||
|
||||
## See also
|
||||
|
||||
- [../design/fips-transport-layer.md](../design/fips-transport-layer.md)
|
||||
— Tor transport design, three modes (`socks5`, `control_port`,
|
||||
`directory`), bridge-node pattern
|
||||
- [../reference/configuration.md](../reference/configuration.md) —
|
||||
full `transports.tor.*` configuration knob table
|
||||
- [enable-nostr-discovery.md](enable-nostr-discovery.md) — Scenario 2
|
||||
for advertising the onion endpoint to peers via Nostr
|
||||
@@ -0,0 +1,211 @@
|
||||
# Diagnose MTU Issues
|
||||
|
||||
MTU symptoms in FIPS look like ordinary network failures: handshakes
|
||||
succeed but bulk transfers hang, ssh connects but stalls after the
|
||||
banner, an HTTP request times out on the first response. This guide
|
||||
walks through the diagnostic surfaces that FIPS exposes so you can
|
||||
distinguish a real MTU problem from its frequent imposters
|
||||
(bufferbloat, transport saturation, transient packet loss).
|
||||
|
||||
For the underlying model — encapsulation overhead, proactive vs
|
||||
reactive PMTUD, the per-destination MTU storage layout — read
|
||||
[../design/fips-mtu.md](../design/fips-mtu.md) first.
|
||||
|
||||
## Symptom map
|
||||
|
||||
| Application symptom | Likely cause |
|
||||
| ------------------- | ------------ |
|
||||
| `iperf3 -c <host.fips>` control socket closes immediately after `Connecting to host`. | Forward-path MTU smaller than the negotiated MSS on the control connection. |
|
||||
| `ssh user@<host.fips>` shows the SSH banner then hangs forever. | First post-banner exchange exceeds the path MTU; SYN MSS clamp did not engage in time, or the path narrowed mid-session. |
|
||||
| `curl http://<host.fips>/` connects, then times out before the first response byte. | Same shape as the SSH-banner case, applied to the first server-to-client large packet. |
|
||||
| Throughput bursts then drops to zero, recovers, drops again, in seconds-long cycles. | Bufferbloat masquerading as MTU failure — usually the upload of the underlay link is saturated. See [Distinguishing bufferbloat](#distinguishing-bufferbloat-from-mtu-drops). |
|
||||
| `MtuExceeded` counters tick up under topology change but settle in seconds. | Normal: the reactive MTU mechanism doing its job. No action needed. |
|
||||
| `MtuExceeded` counters tick continuously under steady state. | Forward-path MTU smaller than what the source learned via `path_mtu` echo. After `mmp.path_mtu` has settled, this is a bug — see [File a bug](#file-a-bug). |
|
||||
|
||||
The first three are MTU candidates; the fourth is usually not. The
|
||||
fifth is benign. The sixth is the bug shape worth filing.
|
||||
|
||||
## Diagnostic toolkit
|
||||
|
||||
### `fipsctl show sessions`
|
||||
|
||||
The authoritative end-to-end MTU for an established session:
|
||||
|
||||
```sh
|
||||
fipsctl show sessions | jq '.sessions[] | {display_name, state, mmp: .mmp.path_mtu}'
|
||||
```
|
||||
|
||||
`mmp.path_mtu` is the value the session-layer MMP currently believes
|
||||
is in force end-to-end. It updates on each PathMtuNotification echo
|
||||
from the destination — immediately on decrease, with hysteresis on
|
||||
increase. A field that starts at `1280` (the IPv6 floor) and then
|
||||
climbs to a higher value as echoes arrive is healthy; one that
|
||||
oscillates between two values may indicate a flapping path.
|
||||
|
||||
### `fipsctl show transports`
|
||||
|
||||
Per-transport MTU. The `mtu` field reports the transport-wide
|
||||
default; for BLE, individual links may have a smaller negotiated
|
||||
ATT_MTU.
|
||||
|
||||
```sh
|
||||
fipsctl show transports | jq '.transports[] | {type, mtu}'
|
||||
```
|
||||
|
||||
### `fipsctl show cache`
|
||||
|
||||
The coordinate cache carries reverse-path-annotated MTU per
|
||||
destination — the freshest "what fit on the way back from the
|
||||
discovery target" estimate, consulted before the session has any
|
||||
PathMtuNotification feedback.
|
||||
|
||||
```sh
|
||||
fipsctl show cache | jq '.entries[] | {display_name, depth, path_mtu}'
|
||||
```
|
||||
|
||||
Entries without a `path_mtu` field are pre-discovery or were
|
||||
populated through a path that did not annotate the MTU.
|
||||
|
||||
### `fipsctl show peers`
|
||||
|
||||
Per-peer link state, including the link-layer MMP metrics. Useful
|
||||
mostly for ruling out underlying loss (loss rate near zero, SRTT
|
||||
sane) before chasing an MTU explanation.
|
||||
|
||||
```sh
|
||||
fipsctl show peers | jq '.peers[] | {display_name, mmp: .mmp}'
|
||||
```
|
||||
|
||||
### Trace logging
|
||||
|
||||
Module-scoped trace logging on the TUN reader and the MMP handler
|
||||
shows the per-packet decisions. The `tracing` macros default the
|
||||
target to the emitting module path, so the filter targets are the
|
||||
fully-qualified module paths under the `fips` crate.
|
||||
|
||||
```sh
|
||||
sudo systemctl edit fips
|
||||
# Add:
|
||||
# [Service]
|
||||
# Environment=RUST_LOG=info,fips::upper::tun=trace,fips::node::handlers::mmp=debug
|
||||
sudo systemctl restart fips
|
||||
sudo journalctl -u fips -f
|
||||
```
|
||||
|
||||
### tcpdump on `fips0`
|
||||
|
||||
Capturing on the TUN reveals the IPv6 packets the daemon hands the
|
||||
kernel and vice-versa. Two important caveats live in the design doc
|
||||
and are worth restating here:
|
||||
|
||||
- TX direction (outbound from a local app): tcpdump sees the packet
|
||||
**before** the daemon's TCP MSS clamp at the TUN boundary. The
|
||||
packet may be larger than the daemon will let leave the node.
|
||||
- RX direction (inbound to a local app): tcpdump sees the packet
|
||||
**after** the daemon's MSS clamp on inbound SYN-ACKs. The clamp
|
||||
fires only when `max_mss < kernel-natural-MSS`; otherwise it is a
|
||||
silent no-op.
|
||||
|
||||
```sh
|
||||
sudo tcpdump -ni fips0 -w /tmp/fips0.pcap port 22 or port 80
|
||||
# in another terminal, reproduce the symptom, then Ctrl-C
|
||||
```
|
||||
|
||||
Open the pcap in Wireshark and check segment sizes against what the
|
||||
session's `path_mtu` reports.
|
||||
|
||||
## Distinguishing bufferbloat from MTU drops
|
||||
|
||||
WAN bufferbloat (sustained upload saturation on a cable or DSL link)
|
||||
produces a retransmit signature that looks remarkably like
|
||||
oversized-packet drops. Both manifest as long stalls in TCP flows,
|
||||
both clear when you stop pushing data, both can ramp the loss-rate
|
||||
counter without obvious cause.
|
||||
|
||||
Two ways to disambiguate:
|
||||
|
||||
1. **Saturate the underlay first.** Run a reference upload outside
|
||||
FIPS (`iperf3 -c <internet-target>`) until it stabilises, then
|
||||
measure latency to the underlay's first hop with a separate `ping`.
|
||||
If RTT shoots up by hundreds of ms during the upload, the
|
||||
underlay buffer is the culprit, not FIPS MTU. Apply CAKE / fq_codel
|
||||
on the underlay router before continuing.
|
||||
|
||||
2. **Watch the FIPS counters during the symptom.** A real MTU
|
||||
problem ticks `MtuExceeded` (visible in `fipsctl show routing`'s
|
||||
`error_signals` block) and shifts the session's `mmp.path_mtu`
|
||||
downward. Bufferbloat ticks loss rate and RTT but leaves
|
||||
`path_mtu` and `MtuExceeded` alone.
|
||||
|
||||
If both signatures fire together, you have both problems.
|
||||
|
||||
## Cold-flow first-SYN
|
||||
|
||||
The MMP echo populates path-MTU state only after the first
|
||||
end-to-end exchange, but the TUN reader has to size the very first
|
||||
SYN before any echo has arrived. The cold-flow ceiling is the
|
||||
1143-byte conservative fallback derived from the 1280-byte IPv6
|
||||
floor. The first SYN may therefore be smaller than what the path
|
||||
ultimately supports; once MMP echoes arrive, subsequent flows use
|
||||
the larger learned value.
|
||||
|
||||
If the first SYN of a flow is still oversized relative to the path,
|
||||
the receiving transit node generates an `MtuExceeded`, the source
|
||||
shrinks immediately, and the next packet of the flow fits. This is
|
||||
expected for one round trip; it becomes a problem only if it
|
||||
persists.
|
||||
|
||||
## Fixes
|
||||
|
||||
The operator's choices, in rough order of preference:
|
||||
|
||||
### Pin a per-transport MTU floor in config
|
||||
|
||||
If a known link in the path has a small MTU that discovery does not
|
||||
pick up promptly (e.g., a Tor hop with an unusually tight cap), set
|
||||
a transport-level MTU floor on the relevant `transports.*` block.
|
||||
See [../reference/configuration.md](../reference/configuration.md)
|
||||
for the per-transport MTU keys.
|
||||
|
||||
### Tune host UDP buffers
|
||||
|
||||
For UDP transports specifically, undersized kernel buffers can drop
|
||||
oversized datagrams in a way that looks identical to MTU failure.
|
||||
See [tune-udp-buffers.md](tune-udp-buffers.md).
|
||||
|
||||
### Accept the floor on intrinsically small links
|
||||
|
||||
Tor and BLE link MTUs are properties of the medium, not tunables.
|
||||
For sessions that cross those links, the path MTU will be small; the
|
||||
fix is to design applications around it (smaller TCP windows, fewer
|
||||
large RTTs) rather than fight the transport.
|
||||
|
||||
### File a bug
|
||||
|
||||
The bug shape worth filing is session `mmp.path_mtu` itself
|
||||
oscillating, or `MtuExceeded` ticking *within* an established
|
||||
session after `mmp.path_mtu` has settled. The TCP-clamp mirror
|
||||
(`path_mtu_lookup`) is now updated on every successful proactive
|
||||
`PathMtuNotification` apply (tighter-only) as well as by the
|
||||
reactive `MtuExceeded` handler, so a steady-state divergence
|
||||
between the per-session `mmp.path_mtu` and the mirror used for
|
||||
new TCP flows is itself a defect, not an expected behavior.
|
||||
|
||||
Capture `fipsctl show sessions`, `fipsctl show cache`, `fipsctl
|
||||
show routing` (for the `error_signals` block), and a tcpdump from
|
||||
`fips0` covering the symptom window. See
|
||||
[../design/fips-mtu.md](../design/fips-mtu.md#per-destination-mtu-storage)
|
||||
for the per-destination MTU storage layout.
|
||||
|
||||
## See also
|
||||
|
||||
- [../design/fips-mtu.md](../design/fips-mtu.md) — encapsulation
|
||||
overhead, the proactive `path_mtu` field, the reactive
|
||||
`MtuExceeded` mechanism, MSS clamping, the no-fragmentation
|
||||
policy.
|
||||
- [../design/fips-mmp.md](../design/fips-mmp.md) — what the MMP
|
||||
metrics mean and how they are computed.
|
||||
- [../design/fips-ipv6-adapter.md](../design/fips-ipv6-adapter.md) —
|
||||
TUN-side ICMPv6 PTB generation and the MSS clamp.
|
||||
- [tune-udp-buffers.md](tune-udp-buffers.md) — host sysctl recipes
|
||||
that rule out kernel-buffer drops as a confounder.
|
||||
@@ -0,0 +1,158 @@
|
||||
# Enable the Mesh-Interface Firewall
|
||||
|
||||
FIPS ships a default-deny nftables baseline at `/etc/fips/fips.nft` that
|
||||
restricts inbound traffic on the `fips0` mesh interface to conntrack
|
||||
replies and ICMPv6 echo. The baseline is **not** enabled by default — see
|
||||
[../design/fips-security.md](../design/fips-security.md) for the threat
|
||||
model and the rationale behind keeping activation explicit. This guide
|
||||
covers the operator steps to load the baseline, extend it with per-host
|
||||
allowances, and inspect drops.
|
||||
|
||||
## Activate the baseline
|
||||
|
||||
The package ships `fips-firewall.service`, a systemd oneshot that runs
|
||||
`nft -f /etc/fips/fips.nft` on start and removes the `inet fips` table
|
||||
on stop. To activate:
|
||||
|
||||
```sh
|
||||
sudo systemctl enable --now fips-firewall.service
|
||||
```
|
||||
|
||||
This loads the table now and arranges for it to load on every subsequent
|
||||
boot. To disable and tear it down:
|
||||
|
||||
```sh
|
||||
sudo systemctl disable --now fips-firewall.service
|
||||
```
|
||||
|
||||
To reload after editing `/etc/fips/fips.nft` or adding a drop-in under
|
||||
`/etc/fips/fips.d/`:
|
||||
|
||||
```sh
|
||||
sudo systemctl reload-or-restart fips-firewall.service
|
||||
```
|
||||
|
||||
The file is idempotent — it begins with `add table inet fips; flush
|
||||
table inet fips;` so re-running it replaces the live ruleset atomically.
|
||||
Equivalently:
|
||||
|
||||
```sh
|
||||
sudo nft -f /etc/fips/fips.nft
|
||||
```
|
||||
|
||||
## Folding the baseline into the host's main nftables
|
||||
|
||||
If you prefer to load the baseline from your existing
|
||||
`/etc/nftables.conf` rather than via the systemd unit, include it
|
||||
directly:
|
||||
|
||||
```nft
|
||||
# in /etc/nftables.conf
|
||||
include "/etc/fips/fips.nft"
|
||||
```
|
||||
|
||||
In that case do **not** enable `fips-firewall.service` — the host's main
|
||||
nftables setup owns the loading. The two paths are mutually exclusive.
|
||||
|
||||
## Extend with per-host allowances via drop-ins
|
||||
|
||||
The baseline drops everything inbound on `fips0` except conntrack
|
||||
replies and ICMPv6 echo. To open specific services to specific mesh
|
||||
nodes, drop a file into `/etc/fips/fips.d/` ending in `.nft`. Each
|
||||
file is included inline into the `inbound` chain at the marked point
|
||||
and may contain any nftables rule lines valid in that context.
|
||||
|
||||
Reload after editing:
|
||||
|
||||
```sh
|
||||
sudo systemctl reload-or-restart fips-firewall.service
|
||||
# or: sudo nft -f /etc/fips/fips.nft
|
||||
```
|
||||
|
||||
### Allow inbound SSH from a specific mesh node
|
||||
|
||||
```nft
|
||||
# /etc/fips/fips.d/ssh-from-bastion.nft
|
||||
ip6 saddr fd97:1234:5678:9abc:def0:1234:5678:9abc tcp dport 22 accept
|
||||
```
|
||||
|
||||
The source filter is the node's mesh address. To find a node's mesh
|
||||
address, look in their `fips.pub` (which contains the npub) and derive
|
||||
the `fd97:...` address from it, or query the running daemon:
|
||||
|
||||
```sh
|
||||
fipsctl show identity-cache
|
||||
fipsctl show peers
|
||||
```
|
||||
|
||||
### Allow inbound DNS broadly
|
||||
|
||||
Some services need to be reachable from any mesh node (a public DNS
|
||||
resolver, a public bootstrap node):
|
||||
|
||||
```nft
|
||||
# /etc/fips/fips.d/dns-public.nft
|
||||
udp dport 53 accept
|
||||
tcp dport 53 accept
|
||||
```
|
||||
|
||||
Omit the source filter only when the service is intended to be
|
||||
universally reachable on the mesh. The baseline's purpose is to make
|
||||
"universally reachable" an explicit decision rather than the default.
|
||||
|
||||
### Multiple nodes, one service
|
||||
|
||||
```nft
|
||||
# /etc/fips/fips.d/git-from-trusted.nft
|
||||
ip6 saddr {
|
||||
fd97:1111:2222:3333:4444:5555:6666:7777,
|
||||
fd97:8888:9999:aaaa:bbbb:cccc:dddd:eeee
|
||||
} tcp dport 9418 accept
|
||||
```
|
||||
|
||||
Set syntax keeps multi-node rules readable and is more efficient than a
|
||||
chain of individual rules.
|
||||
|
||||
## Inspect drops
|
||||
|
||||
The baseline counter increments on every dropped packet. Inspect it:
|
||||
|
||||
```sh
|
||||
sudo nft list table inet fips
|
||||
```
|
||||
|
||||
Look for the `counter packets N bytes M drop` line at the bottom of the
|
||||
`inbound` chain. A non-zero counter means mesh nodes are sending
|
||||
traffic that hits the default-deny — usually benign (probes, neighbor
|
||||
discovery) but occasionally a misconfigured drop-in.
|
||||
|
||||
To see which packets are being dropped, uncomment the `log` line near
|
||||
the bottom of `/etc/fips/fips.nft`:
|
||||
|
||||
```nft
|
||||
log prefix "fips drop: " level info limit rate 10/minute
|
||||
```
|
||||
|
||||
Reload:
|
||||
|
||||
```sh
|
||||
sudo nft -f /etc/fips/fips.nft
|
||||
```
|
||||
|
||||
Then tail the kernel log:
|
||||
|
||||
```sh
|
||||
sudo journalctl -k -f -g "fips drop:"
|
||||
```
|
||||
|
||||
The rate-limit prevents flooding the journal under sustained probing.
|
||||
Adjust the rate, log level, or prefix as needed for the situation.
|
||||
Re-comment the rule when you are done; production hosts do not need
|
||||
the log line on by default.
|
||||
|
||||
## See also
|
||||
|
||||
- [../design/fips-security.md](../design/fips-security.md) — threat
|
||||
model, baseline design, and coexistence with other firewalls
|
||||
- [../reference/security.md](../reference/security.md) — consolidated
|
||||
security reference
|
||||
@@ -0,0 +1,347 @@
|
||||
# Enable Nostr-Mediated Discovery and NAT Traversal
|
||||
|
||||
Nostr-mediated discovery lets FIPS nodes find each other (and punch
|
||||
through UDP NAT) using public Nostr relays as the signaling channel.
|
||||
The feature ships in every stock packaging artifact but is **off by
|
||||
default** — it activates when an operator sets
|
||||
`node.discovery.nostr.enabled: true` and provides a relay list. See
|
||||
[../design/fips-nostr-discovery.md](../design/fips-nostr-discovery.md)
|
||||
for the design and rationale; see
|
||||
[../reference/configuration.md](../reference/configuration.md) for the
|
||||
full knob inventory.
|
||||
|
||||
Nostr discovery provides three independent capabilities. They can be
|
||||
enabled separately; most deployments end up using two or three of
|
||||
them together.
|
||||
|
||||
1. **Resolve a known peer's address by npub.** Your daemon consumes
|
||||
adverts from the relays to look up the current network endpoint
|
||||
for a peer you have configured by npub. You don't have to know
|
||||
their IP / port / transport in advance.
|
||||
2. **Publish your own endpoint so others can resolve you.** Your
|
||||
daemon publishes a signed advert listing the transports it will
|
||||
accept connections on. Has two sub-shapes depending on your
|
||||
network topology: UDP (using NAT traversal if needed) or TCP.
|
||||
Running a Tor onion service is a separate deployment mode,
|
||||
covered in its own section below.
|
||||
3. **Discover peers without prior configuration.** Your daemon
|
||||
subscribes to all adverts on a chosen application namespace and
|
||||
treats any publisher as a connection candidate. The most
|
||||
permissive posture; useful for ambient mesh participation.
|
||||
|
||||
Each capability is covered below as one or more scenarios with the
|
||||
minimal YAML fragment that enables it. Only keys relevant to Nostr
|
||||
discovery are shown; surrounding node, transport, TUN, DNS, and peer
|
||||
configuration follows the usual shape.
|
||||
|
||||
All scenarios assume `node.identity` is set to a persistent key — an
|
||||
ephemeral identity would invalidate any advert the moment the node
|
||||
restarts. See [persistent-identity.md](persistent-identity.md) for
|
||||
the persistent-key setup.
|
||||
|
||||
For hand-held walkthroughs of each capability, see the
|
||||
[resolve-peers-via-nostr](../tutorials/resolve-peers-via-nostr.md),
|
||||
[advertise-your-node](../tutorials/advertise-your-node.md),
|
||||
and [open-discovery](../tutorials/open-discovery.md)
|
||||
tutorials.
|
||||
|
||||
## Capability 1: Resolve a known peer's address by npub
|
||||
|
||||
The node does not publish any advert of its own. It only consumes
|
||||
adverts for peers it has explicitly listed with `via_nostr: true`.
|
||||
This is the right shape for a client that wants Nostr-mediated
|
||||
resolution without becoming a rendezvous target itself.
|
||||
|
||||
```yaml
|
||||
node:
|
||||
identity:
|
||||
persistent: true
|
||||
discovery:
|
||||
nostr:
|
||||
enabled: true
|
||||
advertise: false
|
||||
policy: configured_only
|
||||
|
||||
transports:
|
||||
udp:
|
||||
bind_addr: "0.0.0.0:2121"
|
||||
|
||||
peers:
|
||||
- npub: "npub1peer..."
|
||||
alias: "remote-node"
|
||||
via_nostr: true
|
||||
connect_policy: auto_connect
|
||||
```
|
||||
|
||||
What this achieves: dial endpoints for this peer are taken from
|
||||
the peer's published Nostr advert. `configured_only` is the
|
||||
default — it is shown here for clarity.
|
||||
|
||||
> **Note:** You can also supply a static address alongside
|
||||
> `via_nostr: true` (for example, while testing, or as a
|
||||
> known-good fallback if the advert is stale). Add an `addresses`
|
||||
> block to the peer entry; static addresses are tried first on
|
||||
> dial and Nostr-resolved endpoints are appended as additional
|
||||
> candidates.
|
||||
|
||||
## Capability 2: Publish your own endpoint so others can resolve you
|
||||
|
||||
This capability has three sub-scenarios depending on the network
|
||||
shape your node sits behind.
|
||||
|
||||
### Sub-scenario 2a: UDP (using NAT traversal if needed)
|
||||
|
||||
The node has a public IP (or a stable port-forward) and binds UDP on
|
||||
a known port. It publishes `udp:host:port` to the advert relays. Any
|
||||
peer that knows this node's npub and has Nostr discovery enabled can
|
||||
dial it without knowing the address out-of-band.
|
||||
|
||||
When UDP is wildcard-bound (`0.0.0.0:2121`, the default), the daemon
|
||||
needs help knowing what IP to put in the advert. There are two ways:
|
||||
STUN auto-discovery (`public: true`) or an explicit override
|
||||
(`external_addr`). Both are first-class options; pick the one that
|
||||
fits the deployment.
|
||||
|
||||
```yaml
|
||||
node:
|
||||
identity:
|
||||
persistent: true
|
||||
discovery:
|
||||
nostr:
|
||||
enabled: true
|
||||
advertise: true
|
||||
|
||||
transports:
|
||||
udp:
|
||||
bind_addr: "0.0.0.0:2121"
|
||||
advertise_on_nostr: true
|
||||
public: true # ← STUN auto-discovery
|
||||
```
|
||||
|
||||
Or, when the public IP is known up front (static residential IP,
|
||||
cloud Elastic IP behind 1:1 NAT, etc.):
|
||||
|
||||
```yaml
|
||||
transports:
|
||||
udp:
|
||||
bind_addr: "0.0.0.0:2121"
|
||||
advertise_on_nostr: true
|
||||
external_addr: "203.0.113.45:2121" # ← explicit
|
||||
```
|
||||
|
||||
`external_addr` accepts a bare IP (combined with the bind port) or a
|
||||
full `host:port`. Setting both `public: true` and `external_addr`
|
||||
together is allowed — the explicit override wins, with STUN as a
|
||||
logging cross-check. If UDP is bound directly to a public IP rather
|
||||
than to a wildcard, neither flag is needed; the daemon advertises
|
||||
the bound endpoint.
|
||||
|
||||
What this achieves: the node publishes a single
|
||||
`udp:<public-ip>:2121` endpoint to the three default advert relays
|
||||
(`wss://relay.damus.io`, `wss://nos.lol`, `wss://offchain.pub`).
|
||||
|
||||
What the other side needs: either a static `addresses` entry for this
|
||||
peer, or a peer entry with `via_nostr: true` and an empty (or
|
||||
omitted) `addresses` list — the advert-resolved endpoint will be used
|
||||
at dial time. Static and Nostr-resolved addresses can also be
|
||||
combined: when both are present, static addresses are tried first and
|
||||
Nostr-resolved endpoints are appended as fallback.
|
||||
|
||||
#### When the node is behind NAT
|
||||
|
||||
If this node doesn't have a stable public UDP endpoint, advertise
|
||||
`udp:nat`. The daemon runs the STUN + offer/answer exchange with
|
||||
the peer and punches through the NAT to establish a direct UDP
|
||||
link. The peer can either have a public endpoint of its own or
|
||||
also be behind NAT — both shapes work, as long as at least one
|
||||
side has a NAT type compatible with hole-punching.
|
||||
|
||||
```yaml
|
||||
node:
|
||||
identity:
|
||||
persistent: true
|
||||
discovery:
|
||||
nostr:
|
||||
enabled: true
|
||||
advertise: true
|
||||
dm_relays: # overrides the default three-relay
|
||||
- "wss://relay.damus.io" # set with two for demonstration;
|
||||
- "wss://nos.lol" # omit this block to keep the defaults
|
||||
stun_servers:
|
||||
- "stun:stun.l.google.com:19302"
|
||||
- "stun:stun.cloudflare.com:3478"
|
||||
|
||||
transports:
|
||||
udp:
|
||||
bind_addr: "0.0.0.0:2121"
|
||||
advertise_on_nostr: true
|
||||
public: false
|
||||
|
||||
peers:
|
||||
- npub: "npub1peer..."
|
||||
alias: "nat-peer"
|
||||
addresses:
|
||||
- transport: udp
|
||||
addr: "nat"
|
||||
via_nostr: true
|
||||
connect_policy: auto_connect
|
||||
```
|
||||
|
||||
What this achieves: the node publishes a `udp:nat` endpoint plus its
|
||||
signaling relays in the advert. When either side initiates, an
|
||||
encrypted offer is sealed to the peer's npub, a matching answer
|
||||
comes back, and both sides punch at the negotiated time. On success,
|
||||
the punch socket is adopted as an FMP UDP transport and Noise IK
|
||||
proceeds normally.
|
||||
|
||||
> **Validation:** `advertise_on_nostr: true` with `public: false` on
|
||||
> UDP requires `dm_relays` and `stun_servers` to be non-empty. Both
|
||||
> ship with non-empty defaults (three relays and three STUN servers
|
||||
> respectively), so the default config passes. The node fails
|
||||
> startup only if the operator has explicitly emptied either list —
|
||||
> a `udp:nat` advert without signaling relays or STUN servers is
|
||||
> unreachable by construction.
|
||||
|
||||
Works best with full-cone NAT on at least one side. Symmetric NAT on
|
||||
both sides is not reliably traversable with this protocol and will
|
||||
time out after `punch_duration_ms`.
|
||||
|
||||
### Sub-scenario 2b: TCP
|
||||
|
||||
The node has a public IP (or a stable port-forward) and accepts
|
||||
inbound TCP. It publishes `tcp:host:port` to the advert relays.
|
||||
|
||||
TCP endpoints exist to serve peers whose networks filter outbound
|
||||
UDP (corporate LANs, restrictive guest WiFi). NAT traversal does
|
||||
not apply: the publishing node is publicly reachable on TCP, and
|
||||
the dialing peer's network only needs to permit outbound TCP to
|
||||
the advertised port.
|
||||
|
||||
```yaml
|
||||
node:
|
||||
identity:
|
||||
persistent: true
|
||||
discovery:
|
||||
nostr:
|
||||
enabled: true
|
||||
advertise: true
|
||||
|
||||
transports:
|
||||
tcp:
|
||||
bind_addr: "0.0.0.0:8443"
|
||||
advertise_on_nostr: true
|
||||
external_addr: "203.0.113.45:8443"
|
||||
```
|
||||
|
||||
`external_addr` is typically required on cloud setups (AWS Elastic
|
||||
IP, etc.) where binding directly to the public IP returns
|
||||
`EADDRNOTAVAIL`. When TCP is bound directly to a public IP, the
|
||||
override is unnecessary.
|
||||
|
||||
What this achieves: the node publishes a `tcp:<public-ip>:8443`
|
||||
endpoint to the advert relays. Peers with Nostr discovery enabled
|
||||
dial by npub without out-of-band address exchange.
|
||||
|
||||
### Tor onion node
|
||||
|
||||
A separate deployment mode for nodes that want anonymity and
|
||||
censorship-resistance properties on the data plane. Functionally
|
||||
this still uses Capability 2 (publishing an endpoint to advert
|
||||
relays) — the difference is that the published endpoint is a Tor
|
||||
hidden service rather than a public IP.
|
||||
|
||||
The node runs a Tor onion service in directory mode (Tor-managed
|
||||
`HiddenServiceDir`) and advertises the `.onion` address. Peers dial
|
||||
via their local Tor SOCKS5 proxy without ever knowing the onion
|
||||
string out-of-band. For the Tor daemon side of this setup, including the inbound-mode
|
||||
trade-offs and the `torrc` directives each requires, see
|
||||
[deploy-tor-onion.md](deploy-tor-onion.md).
|
||||
|
||||
```yaml
|
||||
node:
|
||||
identity:
|
||||
persistent: true
|
||||
discovery:
|
||||
nostr:
|
||||
enabled: true
|
||||
advertise: true
|
||||
|
||||
transports:
|
||||
tor:
|
||||
mode: directory
|
||||
socks5_addr: "127.0.0.1:9050"
|
||||
directory_service:
|
||||
hostname_file: "/var/lib/tor/fips/hostname"
|
||||
bind_addr: "127.0.0.1:8444"
|
||||
advertise_on_nostr: true
|
||||
```
|
||||
|
||||
What this achieves: the node publishes a `tor:<hash>.onion:8443`
|
||||
endpoint alongside any other advertised transports. The advert itself
|
||||
is still published over clearnet WebSocket relays — Tor protects the
|
||||
data plane, not the discovery plane. See the security and threat
|
||||
model section in
|
||||
[../design/fips-nostr-discovery.md](../design/fips-nostr-discovery.md#security-and-threat-model)
|
||||
for the trade-off and how to route relay traffic through Tor as well.
|
||||
|
||||
## Capability 3: Discover peers without prior configuration
|
||||
|
||||
Under `policy: open`, any node that publishes an advert under the
|
||||
same `app` namespace becomes a candidate. Discovered peers are queued
|
||||
for connection attempts subject to `open_discovery_max_pending`.
|
||||
|
||||
```yaml
|
||||
node:
|
||||
identity:
|
||||
persistent: true
|
||||
discovery:
|
||||
nostr:
|
||||
enabled: true
|
||||
advertise: true
|
||||
policy: open
|
||||
open_discovery_max_pending: 64
|
||||
app: "my-experiment.v1"
|
||||
|
||||
transports:
|
||||
udp:
|
||||
bind_addr: "0.0.0.0:2121"
|
||||
advertise_on_nostr: true
|
||||
public: true
|
||||
|
||||
peers: []
|
||||
```
|
||||
|
||||
What this achieves: peers are discovered entirely through ambient
|
||||
advert traffic on the configured relays. Setting a non-default `app`
|
||||
value (replacing `fips-overlay-v1`) scopes the discovery set to
|
||||
participants who opt into the same experiment and avoids being joined
|
||||
to unrelated overlays that happen to share the default namespace.
|
||||
|
||||
> **Scope warning:** Open discovery is an admission-free mode. Any
|
||||
> node that publishes on the same `app` name and passes the peer-ACL
|
||||
> check becomes a connection candidate. If you rely on peer ACLs for
|
||||
> admission control, verify that list is set correctly before
|
||||
> enabling this mode. See
|
||||
> [../reference/security.md](../reference/security.md) for the peer
|
||||
> ACL format.
|
||||
|
||||
## See also
|
||||
|
||||
- [../tutorials/resolve-peers-via-nostr.md](../tutorials/resolve-peers-via-nostr.md)
|
||||
— hand-held walkthrough of capability 1
|
||||
- [../tutorials/advertise-your-node.md](../tutorials/advertise-your-node.md)
|
||||
— hand-held walkthrough of capability 2 (publish, plus a
|
||||
short section on `udp:nat` NAT traversal)
|
||||
- [../tutorials/open-discovery.md](../tutorials/open-discovery.md)
|
||||
— hand-held walkthrough of capability 3 (open ambient
|
||||
discovery, the additive policy: open mode)
|
||||
- [../design/fips-nostr-discovery.md](../design/fips-nostr-discovery.md)
|
||||
— discovery runtime design, security model
|
||||
- [../reference/configuration.md](../reference/configuration.md) —
|
||||
full `node.discovery.nostr.*` and per-transport
|
||||
`advertise_on_nostr`/`public` table
|
||||
- [../reference/nostr-events.md](../reference/nostr-events.md) — Kind
|
||||
37195 advert format, Kind 21059 traversal signaling, Kind 10050
|
||||
inbox relay list
|
||||
- [deploy-tor-onion.md](deploy-tor-onion.md) — Tor daemon-side setup
|
||||
for advertising onion endpoints
|
||||
@@ -0,0 +1,173 @@
|
||||
# Use Shortnames Instead of Long Npubs
|
||||
|
||||
A FIPS node's canonical address is `<npub>.fips`. The npub is
|
||||
63 characters of bech32 — fine for the daemon, awkward to type
|
||||
or fit in a docs example. The local DNS resolver consults a
|
||||
host map before falling back to direct-npub resolution, so
|
||||
short names like `test-us01.fips` work as substitutes wherever
|
||||
`<npub>.fips` would.
|
||||
|
||||
This guide covers the two ways to populate that map and when
|
||||
to use which.
|
||||
|
||||
## When to use which
|
||||
|
||||
Two independent mechanisms feed the same DNS responder:
|
||||
|
||||
| Mechanism | Source | Scope | Reload |
|
||||
|-----------|--------|-------|--------|
|
||||
| Hosts file | `/etc/fips/hosts` | Node-local, intended for shared rosters | Auto on mtime change |
|
||||
| Peer alias | `alias:` field on a `peers:` entry | Node-local, scoped to configured peers | Daemon restart |
|
||||
|
||||
Pick the hosts file when:
|
||||
|
||||
- The shortname refers to a peer your operator-team agrees to
|
||||
call by that name across machines (the public test mesh
|
||||
ships this way).
|
||||
- You want the destination's name to resolve in DNS or appear
|
||||
in `fipsctl show peers` display even though it isn't in your
|
||||
`peers:` block — e.g., a mesh node you reach transitively
|
||||
through your direct peers. The hosts-file entry is for name
|
||||
resolution and display only; it does not stand in for the
|
||||
npub a peer-config entry requires.
|
||||
|
||||
Pick the peer alias when:
|
||||
|
||||
- The shortname is just a label *you* use locally for a peer
|
||||
that's already in your `peers:` block.
|
||||
- You want the alias to live with the rest of the peer config
|
||||
(one place to look) rather than in a separate file.
|
||||
|
||||
The two coexist. If both reference the same shortname,
|
||||
`/etc/fips/hosts` wins — the file is treated as the
|
||||
authoritative shared roster.
|
||||
|
||||
## What ships in the default `/etc/fips/hosts`
|
||||
|
||||
The installer drops `/etc/fips/hosts` populated with the
|
||||
public test mesh roster:
|
||||
|
||||
```text
|
||||
test-us01 npub1qmc3cvfz0yu2hx96nq3gp55zdan2qclealn7xshgr448d3nh6lks7zel98
|
||||
test-us02 npub10yffd020a4ag8zcy75f9pruq3rnghvvhd5hphl9s62zgp35s560qrksp9u
|
||||
test-us03 npub136yqae6na688fs75g95ppps3lxe07fvxefj77938zf47uhm6074sxw8ctm
|
||||
test-us03-next npub15m6c4ghuegx4pcde6tra8f7smn8vfv2wundyxwhkjynuerkrzmgsy09sh3
|
||||
test-us04 npub1gd7ye2qp2lphhzx75fynnjzaxx4dqanddecet0wtt5ss5ek8h9ps62wdkf
|
||||
test-de01 npub1260n42s06vzc7796w0fh3ny7zcpw6tlk4gq3940gmfrzl5c9pv2s3657q8
|
||||
test-es01 npub17lpmzulpc98d8ff727k6e98atxn3phzupzsqqwe54ytduym747ws4tw5zm
|
||||
test-uk01 npub1u0z26dc4qeneu5rvwvmpfhtwh3522ed6rlgxr9jarrfnjrc6ew4qxjysrs
|
||||
```
|
||||
|
||||
These resolve out of the box — `ping6 test-us01.fips` works
|
||||
even before you've added any peer to your config, as long as
|
||||
the destination is reachable through your mesh links.
|
||||
|
||||
If you don't intend to interact with the public test mesh,
|
||||
the entries are safe to comment out or delete. They are
|
||||
plain hosts-file lines, not protocol participants — removing
|
||||
them only changes name resolution on your machine.
|
||||
|
||||
## Add an entry to `/etc/fips/hosts`
|
||||
|
||||
Append a line to `/etc/fips/hosts`:
|
||||
|
||||
```text
|
||||
my-laptop npub1abc...xyz
|
||||
```
|
||||
|
||||
Format rules:
|
||||
|
||||
- One hostname and one npub per line, separated by
|
||||
whitespace.
|
||||
- Hostnames are lowercase letters, digits, and hyphens; max
|
||||
63 characters.
|
||||
- Comments start with `#` and continue to end of line; blank
|
||||
lines are ignored.
|
||||
- On duplicate hostnames, the last entry wins.
|
||||
|
||||
The daemon picks up the change on the next DNS query — no
|
||||
restart required (the file's mtime is checked on each query).
|
||||
Verify:
|
||||
|
||||
```sh
|
||||
dig my-laptop.fips AAAA +short
|
||||
```
|
||||
|
||||
Expect one `fd97:...` AAAA record.
|
||||
|
||||
`/etc/fips/hosts` is shipped as a dpkg conffile (and the AUR
|
||||
equivalent), so package upgrades preserve your edits. The
|
||||
file is `0644 root:root` — readable by anyone, writable by
|
||||
root.
|
||||
|
||||
## Add a peer alias
|
||||
|
||||
In `/etc/fips/fips.yaml`, set the `alias:` field on the peer
|
||||
entry:
|
||||
|
||||
```yaml
|
||||
peers:
|
||||
- npub: "npub1abc...xyz"
|
||||
alias: "my-laptop"
|
||||
addresses:
|
||||
- transport: udp
|
||||
addr: "192.0.2.10:2121"
|
||||
connect_policy: auto_connect
|
||||
```
|
||||
|
||||
Restart the daemon for the alias to take effect:
|
||||
|
||||
```sh
|
||||
sudo systemctl restart fips
|
||||
dig my-laptop.fips AAAA +short
|
||||
```
|
||||
|
||||
The alias also shows up in `fipsctl show peers` `display_name`
|
||||
column, so log entries and CLI output reference the peer by
|
||||
shortname instead of truncated npub.
|
||||
|
||||
## Resolution order
|
||||
|
||||
When the DNS responder receives a query for `<name>.fips`:
|
||||
|
||||
1. **Hosts file lookup.** If `<name>` matches an entry in
|
||||
`/etc/fips/hosts`, the daemon returns the AAAA record
|
||||
derived from that entry's npub.
|
||||
2. **Peer alias lookup.** If `<name>` matches the `alias`
|
||||
field on a configured peer, return that peer's AAAA.
|
||||
3. **Direct npub resolution.** If `<name>` is itself a valid
|
||||
bech32 npub (the canonical 63-char `npub1...` form), the
|
||||
daemon returns the AAAA derived from that npub directly.
|
||||
4. **NXDOMAIN.** If none of the above match, the query
|
||||
returns no answer.
|
||||
|
||||
The order means the hosts file overrides peer aliases on
|
||||
conflict. That's deliberate: the file represents
|
||||
operator-shared naming, the peer alias is a node-local label.
|
||||
|
||||
## Cross-references and ACLs
|
||||
|
||||
Aliases interact with the peer ACL — if you maintain
|
||||
`peers.allow` or `peers.deny` lists keyed on hostnames rather
|
||||
than npubs, those names go through the same hosts-file
|
||||
resolution. See
|
||||
[../reference/security.md](../reference/security.md) for the
|
||||
ACL format and the alias-resolution semantics.
|
||||
|
||||
`fipsctl connect` and `fipsctl disconnect` accept a shortname
|
||||
where they expect an npub. Resolution for these commands goes
|
||||
through `/etc/fips/hosts` only — peer-config `alias:` entries
|
||||
are not loaded by `fipsctl`, so a shortname that exists only as
|
||||
a peer alias must still be referenced by full npub on the CLI.
|
||||
See [../reference/cli-fipsctl.md](../reference/cli-fipsctl.md).
|
||||
|
||||
## See also
|
||||
|
||||
- [../reference/configuration.md § Host Mapping](../reference/configuration.md#host-mapping)
|
||||
— minimal reference entry for the host-map mechanism.
|
||||
- [../reference/cli-fipsctl.md](../reference/cli-fipsctl.md)
|
||||
— `fipsctl` arguments that accept shortnames.
|
||||
- [../reference/security.md](../reference/security.md)
|
||||
— peer ACL semantics with aliased entries.
|
||||
- [../design/fips-ipv6-adapter.md](../design/fips-ipv6-adapter.md)
|
||||
— the DNS resolver design and the npub-to-IPv6 derivation.
|
||||
@@ -0,0 +1,232 @@
|
||||
# Provision a Persistent Identity
|
||||
|
||||
A FIPS node's identity is a Nostr keypair. Its public key (npub)
|
||||
determines the node's `fd00::/8` mesh address; peers and configs
|
||||
reference the node by that npub. Out of the box the daemon generates
|
||||
a fresh identity on every start (`node.identity.persistent: false`),
|
||||
which is fine for one-off testing but useless when other nodes need
|
||||
to refer to this one across restarts.
|
||||
|
||||
This guide covers the three ways to give a node a stable identity.
|
||||
For the configuration keys involved, see
|
||||
[../reference/configuration.md](../reference/configuration.md).
|
||||
|
||||
> **First time?** If you have just installed FIPS and want a
|
||||
> hand-held walkthrough of the package-default path (set
|
||||
> `persistent: true`, restart, observe the keys land), the
|
||||
> [persistent-identity tutorial](../tutorials/persistent-identity.md)
|
||||
> is the gentler entry point. This guide assumes an operator
|
||||
> picking among Options A/B/C for a deployment.
|
||||
|
||||
## When to use
|
||||
|
||||
Use a persistent identity for any node that:
|
||||
|
||||
- Other operators reference by npub (in their `peers` lists, `hosts`
|
||||
files, or ACL allow-lists).
|
||||
- Acts as a discoverable bootstrap or rendezvous (Nostr advert,
|
||||
static peer entry, gateway).
|
||||
- Is expected to keep its `fd00::/8` mesh address across restarts.
|
||||
|
||||
Stay with the ephemeral default for throw-away clients, sandbox
|
||||
nodes, and tests where you actively want a fresh identity per run.
|
||||
|
||||
## Option A: Let the package do it
|
||||
|
||||
The Debian/Ubuntu `.deb` and the Arch `fips` AUR package both ship a
|
||||
default `/etc/fips/fips.yaml` with `node.identity.persistent` left as
|
||||
the upstream default (false), so the daemon writes a fresh keypair to
|
||||
`/etc/fips/fips.{key,pub}` on every start until you set
|
||||
`persistent: true`. To pin the current keypair:
|
||||
|
||||
1. Install the package and start the daemon once so it generates
|
||||
`fips.key` / `fips.pub`:
|
||||
|
||||
```sh
|
||||
sudo systemctl start fips
|
||||
sudo systemctl status fips # confirm it came up
|
||||
```
|
||||
|
||||
2. Edit `/etc/fips/fips.yaml` and set:
|
||||
|
||||
```yaml
|
||||
node:
|
||||
identity:
|
||||
persistent: true
|
||||
```
|
||||
|
||||
3. Restart the daemon and verify the identity is reused:
|
||||
|
||||
```sh
|
||||
sudo systemctl restart fips
|
||||
fipsctl show status | grep -E '"npub"|"node_addr"'
|
||||
cat /etc/fips/fips.pub
|
||||
```
|
||||
|
||||
The npub printed by `fipsctl show status` should match
|
||||
`/etc/fips/fips.pub` and remain stable across subsequent restarts.
|
||||
|
||||
The package's `postinst` script does **not** generate the keypair —
|
||||
the daemon does, on first start. This means the keypair is only
|
||||
present after the first successful daemon start. If the daemon never
|
||||
came up cleanly (config error, permission problem), the key files
|
||||
will be missing.
|
||||
|
||||
### File layout and permissions
|
||||
|
||||
| Path | Mode | Owner | Contents |
|
||||
| ---- | ---- | ----- | -------- |
|
||||
| `/etc/fips/fips.key` | `0600` | `root:root` | Bech32 `nsec` (one line). |
|
||||
| `/etc/fips/fips.pub` | `0644` | `root:root` | Bech32 `npub` (one line). |
|
||||
|
||||
Both files live next to the highest-priority `fips.yaml` the daemon
|
||||
loaded. For non-systemd installs that use a different config path,
|
||||
the key files are placed in that config's directory.
|
||||
|
||||
## Option B: Generate manually
|
||||
|
||||
For from-source installs, custom config paths, or any deployment
|
||||
where you want to mint the keypair before the daemon ever runs.
|
||||
|
||||
### With `fipsctl keygen`
|
||||
|
||||
```sh
|
||||
sudo fipsctl keygen --dir /etc/fips
|
||||
```
|
||||
|
||||
This writes `/etc/fips/fips.key` (mode `0600`) and
|
||||
`/etc/fips/fips.pub` (mode `0644`), prints the new npub on stderr,
|
||||
and reminds you to set `persistent: true`. Add `--force` to overwrite
|
||||
an existing `fips.key`. Add `--stdout` to print `nsec` then `npub`
|
||||
to stdout instead of writing files.
|
||||
|
||||
To put the keypair in a non-default directory (e.g., a per-deployment
|
||||
config tree), pass `--dir` and point your `fips.yaml` search at the
|
||||
matching directory.
|
||||
|
||||
### Without the daemon installed
|
||||
|
||||
If you cannot run `fipsctl` (e.g., scripting on a build host), any
|
||||
nostr-tools-equivalent that emits a bech32 `nsec` works. Write the
|
||||
nsec to `fips.key` (mode `0600`) and the corresponding `npub` to
|
||||
`fips.pub` (mode `0644`).
|
||||
|
||||
### Hooking the keypair into the config
|
||||
|
||||
```yaml
|
||||
node:
|
||||
identity:
|
||||
persistent: true
|
||||
```
|
||||
|
||||
`persistent: true` plus a `fips.key` next to the loaded config is the
|
||||
intended steady-state setup.
|
||||
|
||||
## Option C: Provision from an existing nsec
|
||||
|
||||
To migrate an existing Nostr identity into a FIPS node — for example,
|
||||
re-using a personal npub for a node you operate.
|
||||
|
||||
1. Obtain the bech32 `nsec` for the identity.
|
||||
2. Write it to the config-adjacent key file:
|
||||
|
||||
```sh
|
||||
sudo install -m 0600 -o root -g root /dev/null /etc/fips/fips.key
|
||||
sudo bash -c 'printf "%s\n" nsec1... > /etc/fips/fips.key'
|
||||
```
|
||||
|
||||
3. Derive the matching `npub` and write `fips.pub`:
|
||||
|
||||
```sh
|
||||
# compute the npub with any nostr tool, then:
|
||||
sudo bash -c 'printf "%s\n" npub1... > /etc/fips/fips.pub'
|
||||
sudo chmod 0644 /etc/fips/fips.pub
|
||||
```
|
||||
|
||||
4. Set `persistent: true` and restart:
|
||||
|
||||
```yaml
|
||||
node:
|
||||
identity:
|
||||
persistent: true
|
||||
```
|
||||
|
||||
```sh
|
||||
sudo systemctl restart fips
|
||||
fipsctl show status | grep '"npub"'
|
||||
```
|
||||
|
||||
The reported npub should match the one you wrote to `fips.pub`.
|
||||
|
||||
## Verifying
|
||||
|
||||
The daemon prints the resolved identity at startup; the same value is
|
||||
queryable via the control socket:
|
||||
|
||||
```sh
|
||||
fipsctl show status | jq '{npub, node_addr, ipv6_addr}'
|
||||
cat /etc/fips/fips.pub
|
||||
```
|
||||
|
||||
The `npub` field of `show status` and the contents of `fips.pub`
|
||||
should match. The `node_addr` is the SHA-256 prefix used internally
|
||||
by FMP/FSP; the `ipv6_addr` is the routable `fd00::/8` mesh address
|
||||
derived from the node addr. Together they are stable for the lifetime
|
||||
of the keypair.
|
||||
|
||||
The journal also records the source on every start:
|
||||
|
||||
```text
|
||||
INFO Loaded persistent identity from key file path=/etc/fips/fips.key
|
||||
```
|
||||
|
||||
(`Generated persistent identity, saved to key file` on the first
|
||||
start; `Using ephemeral identity (new keypair each start)` when
|
||||
persistence is off.)
|
||||
|
||||
## Rotating
|
||||
|
||||
Key rotation is a destructive operation: every cached
|
||||
`(node_addr → npub)` mapping on every other node points at the old
|
||||
key, every Nostr advert and every static peer entry references the
|
||||
old npub, and every existing FSP session was authenticated under the
|
||||
old keypair. There is no in-protocol "key change" message.
|
||||
|
||||
To rotate:
|
||||
|
||||
1. Stop the daemon.
|
||||
|
||||
```sh
|
||||
sudo systemctl stop fips
|
||||
```
|
||||
|
||||
2. Remove the existing key files.
|
||||
|
||||
```sh
|
||||
sudo rm /etc/fips/fips.key /etc/fips/fips.pub
|
||||
```
|
||||
|
||||
3. Start the daemon. With `persistent: true`, the daemon generates a
|
||||
new keypair and writes new `fips.key` / `fips.pub`.
|
||||
|
||||
```sh
|
||||
sudo systemctl start fips
|
||||
cat /etc/fips/fips.pub # the new npub
|
||||
```
|
||||
|
||||
4. Update every downstream reference: peer configs that name this
|
||||
node by npub, `hosts` files, ACL allow-lists, Nostr adverts
|
||||
pinned by other operators.
|
||||
|
||||
There is no recovery from a lost `fips.key` — the npub is gone with
|
||||
the secret. Treat key rotation as a coordinated event; do not rotate
|
||||
production identities ad hoc.
|
||||
|
||||
## See also
|
||||
|
||||
- [../reference/configuration.md](../reference/configuration.md) —
|
||||
`node.identity.*` keys.
|
||||
- [../reference/cli-fipsctl.md](../reference/cli-fipsctl.md) —
|
||||
`fipsctl keygen`.
|
||||
- [../design/fips-architecture.md](../design/fips-architecture.md) —
|
||||
identity model, npub-to-NodeAddr derivation.
|
||||
@@ -0,0 +1,195 @@
|
||||
# Run the FIPS Daemon as an Unprivileged User
|
||||
|
||||
By default, the FIPS daemon runs as `root` — the shipped Debian
|
||||
systemd unit configures this, and no further setup is required.
|
||||
The trade-off is that the daemon has full root authority,
|
||||
including outside its actual network needs. Acceptable for
|
||||
single-purpose hosts; less desirable for shared hosts.
|
||||
|
||||
This guide covers the alternative: drop privileges and run the
|
||||
daemon under a dedicated unprivileged user account. The TUN
|
||||
device that the FIPS IPv6 adapter creates requires
|
||||
`CAP_NET_ADMIN` on Linux; the recipe below grants that privilege
|
||||
via a file capability on the binary, plus everything else the
|
||||
daemon needs to keep working without root: a service user
|
||||
account, file permissions on the config directory, and a systemd
|
||||
unit override to drop privileges.
|
||||
|
||||
For the design context (why the adapter needs a TUN, how the
|
||||
adapter integrates with the kernel routing table), see
|
||||
[../design/fips-ipv6-adapter.md](../design/fips-ipv6-adapter.md).
|
||||
|
||||
## Prerequisites
|
||||
|
||||
- FIPS package installed (the postinst already creates the `fips`
|
||||
system group used for control-socket access).
|
||||
- `setcap` available (`apt install libcap2-bin` on Debian/Ubuntu;
|
||||
it is a standard utility on most distributions).
|
||||
- Operator access to systemd unit overrides (`systemctl edit`).
|
||||
|
||||
## Step 1: Create a `fips` system user
|
||||
|
||||
The package creates a `fips` system *group* but no matching user.
|
||||
Add a system user that belongs to the `fips` group:
|
||||
|
||||
```sh
|
||||
sudo useradd --system --gid fips --no-create-home --shell /usr/sbin/nologin fips
|
||||
```
|
||||
|
||||
The user has no home directory and no login shell — this account
|
||||
exists only to run the daemon.
|
||||
|
||||
## Step 2: Grant `CAP_NET_ADMIN` to the binary
|
||||
|
||||
Apply the file capability so the daemon can create the TUN device
|
||||
without root authority:
|
||||
|
||||
```sh
|
||||
sudo setcap cap_net_admin+ep /usr/bin/fips
|
||||
```
|
||||
|
||||
Verify:
|
||||
|
||||
```sh
|
||||
getcap /usr/bin/fips
|
||||
# /usr/bin/fips cap_net_admin=ep
|
||||
```
|
||||
|
||||
The binary can now create TUN devices when run by any user.
|
||||
|
||||
**File-capability caveats:**
|
||||
|
||||
- The capability is attached to the binary file. **Re-applying
|
||||
the capability after every package upgrade is required**,
|
||||
because package upgrades replace the binary file and lose the
|
||||
cap. The systemd override in Step 4 includes an `ExecStartPre`
|
||||
line that automates this.
|
||||
- File capabilities are stripped when the binary is copied across
|
||||
most filesystems and when it is downloaded via web tooling. If
|
||||
you build from source and install manually, remember to
|
||||
re-`setcap` after each rebuild.
|
||||
- `LD_LIBRARY_PATH` and similar environment-driven loader
|
||||
controls are stripped at exec time when file capabilities are
|
||||
present; this is normally what you want, but development
|
||||
workflows that rely on custom library paths may be surprised.
|
||||
|
||||
## Step 3: Adjust config-file permissions
|
||||
|
||||
The shipped `/etc/fips/fips.yaml` is mode `0600` and owned by
|
||||
`root:root`. The daemon needs to read it and, if persistent
|
||||
identity is enabled, write `/etc/fips/fips.key` into the same
|
||||
directory.
|
||||
|
||||
```sh
|
||||
sudo chown -R fips:fips /etc/fips
|
||||
sudo chmod 0640 /etc/fips/fips.yaml
|
||||
```
|
||||
|
||||
If `node.identity.persistent: true` is set and `fips.key` does
|
||||
not exist yet, leave `/etc/fips` itself writable by the `fips`
|
||||
user so the daemon can create it on first start. After the key
|
||||
file exists, you can tighten further:
|
||||
|
||||
```sh
|
||||
sudo chmod 0600 /etc/fips/fips.key
|
||||
```
|
||||
|
||||
## Step 4: Drop privileges in the systemd unit
|
||||
|
||||
Create an override:
|
||||
|
||||
```sh
|
||||
sudo systemctl edit fips.service
|
||||
```
|
||||
|
||||
Add:
|
||||
|
||||
```ini
|
||||
[Service]
|
||||
User=fips
|
||||
Group=fips
|
||||
AmbientCapabilities=CAP_NET_ADMIN
|
||||
NoNewPrivileges=no
|
||||
ExecStartPre=/sbin/setcap cap_net_admin+ep /usr/bin/fips
|
||||
```
|
||||
|
||||
`User=` / `Group=` set the service identity.
|
||||
`AmbientCapabilities=` ensures the file capability granted in
|
||||
Step 2 actually carries into the daemon's process tree.
|
||||
`NoNewPrivileges=no` is required for file-capability execution
|
||||
to work — systemd defaults this to `yes` for hardened units,
|
||||
which would block the `setcap` from taking effect.
|
||||
`ExecStartPre=` re-applies the capability before each start,
|
||||
which makes the package-upgrade path self-heal.
|
||||
|
||||
The unit's `RuntimeDirectory=fips` directive already arranges
|
||||
for `/run/fips/` to be created with the right ownership at
|
||||
service start, now as `fips:fips 0750` instead of
|
||||
`root:fips 0750`.
|
||||
|
||||
Reload and restart:
|
||||
|
||||
```sh
|
||||
sudo systemctl daemon-reload
|
||||
sudo systemctl restart fips
|
||||
```
|
||||
|
||||
## Step 5: Verify
|
||||
|
||||
Confirm the daemon is running as `fips`:
|
||||
|
||||
```sh
|
||||
ps -eo user,cmd | grep '[/]usr/bin/fips'
|
||||
# fips /usr/bin/fips --config /etc/fips/fips.yaml
|
||||
```
|
||||
|
||||
Confirm the TUN device came up (the `setcap` worked):
|
||||
|
||||
```sh
|
||||
ip link show fips0
|
||||
# fips0: <POINTOPOINT,UP,...> mtu 1280 ...
|
||||
```
|
||||
|
||||
Confirm the control socket is bound and accessible to the `fips`
|
||||
group:
|
||||
|
||||
```sh
|
||||
ls -la /run/fips/control.sock
|
||||
# srwxrwx--- 1 fips fips ... /run/fips/control.sock
|
||||
```
|
||||
|
||||
Add yourself to the `fips` group so you can use `fipsctl` /
|
||||
`fipstop` without `sudo`:
|
||||
|
||||
```sh
|
||||
sudo usermod -aG fips $USER
|
||||
# log out and back in for the group change to take effect
|
||||
```
|
||||
|
||||
Then:
|
||||
|
||||
```sh
|
||||
fipsctl show node
|
||||
```
|
||||
|
||||
## Caveats
|
||||
|
||||
- **`fips-firewall.service` still runs as root.** Loading nftables
|
||||
rules into the kernel requires root regardless. The firewall
|
||||
unit is intentionally separate from the daemon unit.
|
||||
- **Bluetooth peers (`transports.ble.*`)** require additional
|
||||
privileges the `CAP_NET_ADMIN` setcap doesn't cover. If you use
|
||||
the BLE transport, you'll likely need to keep running as root
|
||||
or layer additional capability/D-Bus configuration; that path
|
||||
is not covered here.
|
||||
|
||||
## See also
|
||||
|
||||
- [persistent-identity.md](persistent-identity.md) — how the
|
||||
daemon manages `/etc/fips/fips.key`
|
||||
- [../design/fips-ipv6-adapter.md](../design/fips-ipv6-adapter.md)
|
||||
— IPv6 adapter design, TUN interface architecture
|
||||
- [../reference/security.md](../reference/security.md) —
|
||||
consolidated security surface
|
||||
- [../reference/configuration.md](../reference/configuration.md)
|
||||
— `tun.*` configuration block
|
||||
@@ -0,0 +1,299 @@
|
||||
# Set Up a Bluetooth (BLE) Peer Link
|
||||
|
||||
FIPS supports Bluetooth Low Energy as a transport for short-range
|
||||
mesh extension — same room, same building, no IP infrastructure
|
||||
between the two endpoints. The BLE transport runs as L2CAP
|
||||
Connection-Oriented Channels on a configurable PSM and reports
|
||||
per-link MTU back to the mesh layer for path-MTU computation.
|
||||
|
||||
For the design rationale and per-link MTU model, see
|
||||
[../design/fips-transport-layer.md](../design/fips-transport-layer.md).
|
||||
For all `transports.ble.*` configuration keys, see
|
||||
[../reference/configuration.md](../reference/configuration.md).
|
||||
|
||||
> **Experimental.** The BLE transport works but is still maturing.
|
||||
> Expect rougher edges than UDP or TCP — particularly around link
|
||||
> stability under interference and MTU negotiation on older
|
||||
> controllers. Treat it as you would any experimental transport in a
|
||||
> production deployment.
|
||||
|
||||
## When to use
|
||||
|
||||
BLE is the right transport when:
|
||||
|
||||
- Two nodes are within roughly 10 metres line-of-sight (more with
|
||||
external antennas, less through walls).
|
||||
- You want a self-contained mesh segment with no shared WiFi or
|
||||
Ethernet between the participants.
|
||||
- You can work within practical L2CAP CoC throughput (1-2 Mbps in
|
||||
good conditions, often substantially less under interference or
|
||||
at range) and the higher latency variance compared to WiFi.
|
||||
|
||||
It is **not** the right transport for backbone links between rooms
|
||||
where WiFi or Ethernet exists, for high-throughput data, or for any
|
||||
deployment where range matters more than infrastructure-freedom.
|
||||
|
||||
## Platform support
|
||||
|
||||
The BLE transport is **Linux-only** in the current implementation.
|
||||
The runtime depends on BlueZ via the `bluer` crate, which in turn
|
||||
needs `glibc` (musl builds skip BLE; the build script gates the
|
||||
crate accordingly).
|
||||
|
||||
| Platform | BLE transport |
|
||||
| -------- | -------------- |
|
||||
| Linux (glibc) | Supported. |
|
||||
| Linux (musl, OpenWrt) | Disabled at build time. |
|
||||
| macOS | Not supported. |
|
||||
| Windows | Not supported. |
|
||||
|
||||
The Debian package `Recommends: bluez`; install it explicitly if you
|
||||
opted out:
|
||||
|
||||
```sh
|
||||
sudo apt install bluez
|
||||
```
|
||||
|
||||
## Prerequisites
|
||||
|
||||
Both endpoints need:
|
||||
|
||||
1. A BLE-capable HCI adapter visible to BlueZ. Confirm with:
|
||||
|
||||
```sh
|
||||
sudo bluetoothctl show
|
||||
```
|
||||
|
||||
Note the controller name (typically `hci0`).
|
||||
|
||||
2. The `bluetoothd` service running and the adapter powered on:
|
||||
|
||||
```sh
|
||||
sudo systemctl enable --now bluetooth
|
||||
sudo bluetoothctl power on
|
||||
```
|
||||
|
||||
3. Sufficient privileges for the FIPS daemon. There are two
|
||||
independent privilege concerns; the BLE-only deployment case
|
||||
(mesh router with `tun.enabled: false`) needs only the second.
|
||||
|
||||
- **TUN adapter (always required when `tun.enabled: true`).**
|
||||
The daemon needs `CAP_NET_ADMIN` to create and configure the
|
||||
TUN device. The shipped systemd unit handles this by running
|
||||
as root; if you prefer to drop privileges, see
|
||||
[run-as-unprivileged-user.md](run-as-unprivileged-user.md).
|
||||
|
||||
- **BLE access (required for this how-to).** BlueZ exposes
|
||||
L2CAP and D-Bus paths under either group membership or
|
||||
`CAP_NET_RAW`. Pick one:
|
||||
|
||||
- Run the daemon as root. The shipped systemd unit takes
|
||||
this route.
|
||||
- Run as an unprivileged user that is a member of the
|
||||
`bluetooth` group. No additional capability is needed for
|
||||
the BLE side.
|
||||
- Run as an unprivileged user with no group membership, and
|
||||
grant the binary `CAP_NET_RAW`:
|
||||
|
||||
```sh
|
||||
sudo setcap cap_net_raw+ep $(which fips)
|
||||
```
|
||||
|
||||
This bypasses BlueZ's polkit/group check by holding
|
||||
`CAP_NET_RAW` directly. If you also need `CAP_NET_ADMIN`
|
||||
for TUN, combine them:
|
||||
|
||||
```sh
|
||||
sudo setcap cap_net_admin,cap_net_raw+ep $(which fips)
|
||||
```
|
||||
|
||||
4. The same L2CAP PSM on both endpoints. The default is `0x0085`
|
||||
(133); override only if you need to coexist with another L2CAP
|
||||
service on that PSM.
|
||||
|
||||
## Configuration
|
||||
|
||||
Add a `ble` block under `transports` in `fips.yaml`. A minimum BLE-
|
||||
active node looks like this:
|
||||
|
||||
```yaml
|
||||
transports:
|
||||
ble:
|
||||
adapter: "hci0"
|
||||
advertise: true
|
||||
scan: true
|
||||
auto_connect: true
|
||||
accept_connections: true
|
||||
```
|
||||
|
||||
Note: `auto_connect: true` is intentionally non-default (the default
|
||||
is `false`). For a symmetric ground-up discovery flow where either
|
||||
side may dial, both ends must opt in explicitly.
|
||||
|
||||
| Key | Purpose |
|
||||
| --- | ------- |
|
||||
| `adapter` | HCI controller name. Default: `hci0`. |
|
||||
| `psm` | L2CAP PSM. Default: `0x0085` (must match on both ends). |
|
||||
| `mtu` | Default L2CAP CoC MTU. Default: `2048`. The kernel may negotiate lower per link. |
|
||||
| `max_connections` | Concurrent BLE connections. Default: `7` (Bluetooth controllers typically support up to ~7 simultaneous L2CAP CoCs). |
|
||||
| `advertise` | Broadcast our BLE adverts so other FIPS nodes discover us. Default: `true`. |
|
||||
| `scan` | Listen for other FIPS nodes' BLE adverts. Default: `true`. |
|
||||
| `auto_connect` | Initiate a BLE connection to discovered FIPS adverts. Default: `false`. |
|
||||
| `accept_connections` | Accept inbound L2CAP connections. Default: `true`. |
|
||||
| `connect_timeout_ms` | Outbound L2CAP connect timeout. Default: `10000`. |
|
||||
| `probe_cooldown_secs` | After probing a BD_ADDR (success or failure), wait this long before probing it again. Default: `30`. |
|
||||
|
||||
Two pairing patterns are common:
|
||||
|
||||
**Symmetric auto-discovery.** Both nodes advertise, scan, and
|
||||
auto-connect. Whichever side completes the L2CAP connection first
|
||||
wins; the other side aborts its in-flight attempt. This is the
|
||||
"toss two devices in the same room" setup.
|
||||
|
||||
```yaml
|
||||
# Both nodes
|
||||
transports:
|
||||
ble:
|
||||
adapter: "hci0"
|
||||
advertise: true
|
||||
scan: true
|
||||
auto_connect: true
|
||||
accept_connections: true
|
||||
```
|
||||
|
||||
**Asymmetric peripheral / central.** One node only listens
|
||||
(peripheral), the other actively dials (central). Useful when one
|
||||
endpoint is a dedicated bootstrap and the other is mobile.
|
||||
|
||||
```yaml
|
||||
# Listener
|
||||
transports:
|
||||
ble:
|
||||
adapter: "hci0"
|
||||
advertise: true
|
||||
scan: false
|
||||
auto_connect: false
|
||||
accept_connections: true
|
||||
```
|
||||
|
||||
```yaml
|
||||
# Dialer
|
||||
transports:
|
||||
ble:
|
||||
adapter: "hci0"
|
||||
advertise: false
|
||||
scan: true
|
||||
auto_connect: true
|
||||
accept_connections: false
|
||||
```
|
||||
|
||||
After editing, restart the daemon on each side:
|
||||
|
||||
```sh
|
||||
sudo systemctl restart fips
|
||||
```
|
||||
|
||||
## Verify
|
||||
|
||||
On each endpoint, confirm the transport came up:
|
||||
|
||||
```sh
|
||||
fipsctl show transports
|
||||
```
|
||||
|
||||
Look for an entry of type `ble` in the `state: Running` (or
|
||||
equivalent) state. The `mtu` field reports the configured default;
|
||||
per-link MTU is reported separately.
|
||||
|
||||
Confirm the link is established:
|
||||
|
||||
```sh
|
||||
fipsctl show peers
|
||||
```
|
||||
|
||||
The peer entry for the BLE-attached neighbour should report
|
||||
`transport_type: "ble"` and a non-zero `last_seen_ms`.
|
||||
|
||||
BLE peering is auto-discovery only: there is no `fipsctl connect`
|
||||
path for BLE (the command accepts `udp`, `tcp`, `tor`, and
|
||||
`ethernet` only). Links come up via advert/scan; if you don't see
|
||||
the peer here, the configuration above is the only knob.
|
||||
|
||||
To watch the link in real time, use `fipstop`'s **Peers** and
|
||||
**Transports** tabs:
|
||||
|
||||
```sh
|
||||
fipstop
|
||||
```
|
||||
|
||||
The Performance tab reports the per-link MMP metrics — SRTT, loss
|
||||
rate, ETX — which on BLE typically run an order of magnitude worse
|
||||
than over UDP, with much higher jitter.
|
||||
|
||||
## Troubleshooting
|
||||
|
||||
### Transport never comes up
|
||||
|
||||
Check the BlueZ side first:
|
||||
|
||||
```sh
|
||||
systemctl status bluetooth
|
||||
sudo bluetoothctl show
|
||||
```
|
||||
|
||||
If `bluetoothctl show` reports `Powered: no`, fix that before
|
||||
debugging FIPS. The FIPS daemon will log a warning if it cannot
|
||||
acquire the adapter.
|
||||
|
||||
If the FIPS log contains `bluer` D-Bus errors, the daemon usually
|
||||
lacks permission. Run as root or grant `CAP_NET_ADMIN` and add the
|
||||
fips user to the `bluetooth` group.
|
||||
|
||||
### Peers see each other but never connect
|
||||
|
||||
Verify `accept_connections` is true on at least one side and
|
||||
`auto_connect` is true on at least one side. Two listen-only nodes
|
||||
will discover each other but never establish an L2CAP connection.
|
||||
|
||||
Check `psm` matches on both ends. A mismatch presents as adverts
|
||||
visible (in `fipstop` discovery counters) but every connect attempt
|
||||
fails.
|
||||
|
||||
### Link comes up but throughput is poor
|
||||
|
||||
Practical L2CAP CoC throughput in good conditions reaches
|
||||
1-2 Mbps, but interference, range, and controller capability all
|
||||
push it lower. If throughput is well below that range, check the
|
||||
negotiated ATT_MTU — a small ATT_MTU (default 23 bytes when
|
||||
extended ATT MTU is not negotiated) caps per-PDU payload
|
||||
regardless of radio conditions. The per-link MTU reported in
|
||||
`fipsctl show transports` reveals what was negotiated.
|
||||
|
||||
If MTU is unexpectedly low, both endpoints must support and have
|
||||
negotiated the BlueZ L2CAP `cocmode=2` extension. Older Bluetooth
|
||||
controllers cap MTU regardless.
|
||||
|
||||
### Unstable links / repeated reconnects
|
||||
|
||||
Bluetooth in busy 2.4 GHz environments suffers from WiFi
|
||||
interference. Switch the adapter to a less crowded channel (kernel
|
||||
side, not configurable from FIPS) or add an external antenna. The
|
||||
`probe_cooldown_secs` tunable backs off retry attempts; raise it if
|
||||
the daemon log shows many short-lived probes.
|
||||
|
||||
### Permission errors on socket open
|
||||
|
||||
Most modern systemd installs do not allow non-root processes to
|
||||
open raw L2CAP sockets without an explicit policy. Run the daemon
|
||||
as root (the shipped systemd unit does this) or add a `polkit`
|
||||
rule for the `bluetooth` group.
|
||||
|
||||
## See also
|
||||
|
||||
- [../design/fips-transport-layer.md](../design/fips-transport-layer.md)
|
||||
— per-transport MTU reporting and the BLE row of the supported-
|
||||
transports table.
|
||||
- [../reference/configuration.md](../reference/configuration.md) —
|
||||
full `transports.ble.*` reference.
|
||||
- [run-as-unprivileged-user.md](run-as-unprivileged-user.md) —
|
||||
adjacent privilege handling for the daemon process.
|
||||
@@ -0,0 +1,458 @@
|
||||
# Troubleshoot `fips-gateway`
|
||||
|
||||
Diagnostic recipes for `fips-gateway`, grouped by which half of the
|
||||
gateway is failing. For gateway design and deployment, see
|
||||
[../design/fips-gateway.md](../design/fips-gateway.md) and
|
||||
[deploy-gateway.md](deploy-gateway.md). For OpenWrt-specific
|
||||
deployment problems, see the
|
||||
[OpenWrt deployment tutorial](../tutorials/deploy-fips-gateway.md);
|
||||
most of the recipes below apply on OpenWrt as well, but paths and
|
||||
service names differ.
|
||||
|
||||
## Inspect gateway state via the control socket
|
||||
|
||||
Before digging into nftables or conntrack, ask the gateway directly
|
||||
whether it has the mapping or session you expect. `fips-gateway`
|
||||
exposes a separate control socket (`/run/fips/gateway.sock`) with its
|
||||
own command set; there is no `fipsctl gateway` subcommand — talk to
|
||||
the socket directly with `nc -U`. Each request is a single line of
|
||||
JSON terminated with a newline; the connection is closed after one
|
||||
response.
|
||||
|
||||
Pool summary, listen address, NAT counters, uptime, and the loaded
|
||||
config snapshot:
|
||||
|
||||
```sh
|
||||
echo '{"command":"show_gateway"}' | sudo nc -U /run/fips/gateway.sock
|
||||
```
|
||||
|
||||
Per-mapping virtual-IP state (allocated, active, draining):
|
||||
|
||||
```sh
|
||||
echo '{"command":"show_mappings"}' | sudo nc -U /run/fips/gateway.sock
|
||||
```
|
||||
|
||||
If either command returns `gateway not yet initialized`, the gateway
|
||||
is still in early startup; wait a moment and retry. If a mapping you
|
||||
expect is not in the list, the DNS path didn't allocate it — fall
|
||||
through to the outbound DNS recipes below. If the mapping exists in
|
||||
`state: Active` but mesh traffic still fails, the problem is
|
||||
downstream of the allocation (firewall, route, masquerade); see the
|
||||
recipes that follow.
|
||||
|
||||
For the full command catalog and JSON shapes, see
|
||||
[../reference/control-socket.md#gateway-command-catalog](../reference/control-socket.md#gateway-command-catalog).
|
||||
|
||||
## Common (either-half) issues
|
||||
|
||||
These break both halves at once because they affect the gateway
|
||||
process itself or the shared NAT machinery.
|
||||
|
||||
### "No gateway section in configuration"
|
||||
|
||||
`fips-gateway` is normally launched by the systemd unit shipped with
|
||||
the package:
|
||||
|
||||
```sh
|
||||
sudo systemctl restart fips-gateway
|
||||
sudo journalctl -u fips-gateway -e
|
||||
```
|
||||
|
||||
The unit reads the standard FIPS config search paths (typically
|
||||
`/etc/fips/fips.yaml`). If the unit logs "no gateway section in
|
||||
configuration" or "Gateway section exists but is not enabled", confirm
|
||||
the section is present and `enabled: true`:
|
||||
|
||||
```sh
|
||||
grep -A1 '^gateway:' /etc/fips/fips.yaml
|
||||
```
|
||||
|
||||
For one-off debugging outside systemd, run the binary directly and
|
||||
point it at a specific config file:
|
||||
|
||||
```sh
|
||||
sudo fips-gateway --config /etc/fips/fips.yaml --log-level debug
|
||||
```
|
||||
|
||||
This is useful to capture stderr in a terminal, but the systemd unit
|
||||
is the supported entry point in production. See
|
||||
[../reference/cli-fips-gateway.md](../reference/cli-fips-gateway.md)
|
||||
for the full flag list.
|
||||
|
||||
### Port conflict on the DNS listen port
|
||||
|
||||
Symptom: gateway fails to start with "address already in use" on
|
||||
port 53 (or whatever `gateway.dns.listen` is set to).
|
||||
|
||||
Another DNS server (systemd-resolved, dnsmasq, BIND) is bound to
|
||||
the port. Identify it:
|
||||
|
||||
```sh
|
||||
sudo ss -tulnp | grep ':53'
|
||||
```
|
||||
|
||||
Two options:
|
||||
|
||||
- **Use an alternate listen address.** Pick a non-conflicting port
|
||||
and update the gateway config:
|
||||
|
||||
```yaml
|
||||
gateway:
|
||||
dns:
|
||||
listen: "192.168.1.1:5353"
|
||||
```
|
||||
|
||||
Then either point LAN clients at the alternate port directly, or
|
||||
run a forwarding stub on port 53 that delegates `.fips` queries to
|
||||
the gateway.
|
||||
|
||||
- **Relocate the conflicting resolver.** Move it to a different port
|
||||
(or disable it if not needed) and let the gateway bind 53.
|
||||
Practical for systemd-resolved (set `DNSStubListener=no` in
|
||||
`/etc/systemd/resolved.conf`); rarely worth it for production
|
||||
resolvers.
|
||||
|
||||
### IPv6 forwarding disabled
|
||||
|
||||
Symptom: gateway exits at startup with
|
||||
"IPv6 forwarding is disabled. Enable with: sysctl -w
|
||||
net.ipv6.conf.all.forwarding=1".
|
||||
|
||||
The gateway is completely non-functional without forwarding — packets
|
||||
cannot traverse the NAT pipeline. Enable it:
|
||||
|
||||
```sh
|
||||
sudo sysctl -w net.ipv6.conf.all.forwarding=1
|
||||
```
|
||||
|
||||
Persist via the drop-in shown in
|
||||
[deploy-gateway.md](deploy-gateway.md#kernel-sysctls). The same
|
||||
section lists `proxy_ndp`, which is also required for the outbound
|
||||
half.
|
||||
|
||||
### nftables table missing or not loaded
|
||||
|
||||
Symptom: `show_gateway` reports an active gateway but
|
||||
`nft list table inet fips_gateway` errors with "No such file or
|
||||
directory".
|
||||
|
||||
The table is created by the gateway at startup and rebuilt atomically
|
||||
on every mapping change and on every `set_port_forwards` call. If the
|
||||
table is missing while the gateway claims to be running, something
|
||||
else (a host firewall script, a `nft flush ruleset` from another
|
||||
service) deleted it after creation. Restart the gateway to recreate
|
||||
it:
|
||||
|
||||
```sh
|
||||
sudo systemctl restart fips-gateway
|
||||
```
|
||||
|
||||
If a peer service is repeatedly clobbering the table, switch that
|
||||
service to use `add table` / `flush table <name>` for its own table
|
||||
rather than `flush ruleset`, which destroys every table on the host.
|
||||
|
||||
### Control socket permission errors
|
||||
|
||||
Symptom: `nc -U /run/fips/gateway.sock` fails with "Permission
|
||||
denied" or "No such file or directory".
|
||||
|
||||
The socket is owned by root with mode `0660` (group `fips`). Either
|
||||
run `nc` as root (`sudo nc -U ...`) or add your user to the `fips`
|
||||
group and re-login. If the file does not exist at all, the gateway
|
||||
either failed to start (check `journalctl -u fips-gateway`) or
|
||||
failed to bind the socket and continued without it (the warning
|
||||
`Failed to bind gateway control socket — continuing without it` is
|
||||
in the journal in that case).
|
||||
|
||||
## Outbound-half diagnostics
|
||||
|
||||
Symptoms in this section all involve a LAN client trying to reach a
|
||||
mesh destination through the gateway and failing.
|
||||
|
||||
### DNS queries fail
|
||||
|
||||
Symptom: LAN clients get `SERVFAIL` or no response when querying
|
||||
`.fips` names; or the gateway log shows DNS upstream timeouts.
|
||||
|
||||
**Step 1.** Verify the daemon resolver is running and reachable from
|
||||
the gateway host:
|
||||
|
||||
```sh
|
||||
dig @::1 -p 5354 hostname.fips AAAA
|
||||
```
|
||||
|
||||
If this returns no answer or fails, the FIPS daemon's DNS resolver is
|
||||
not running or not enabled. Check that the daemon config has
|
||||
`dns.enabled: true` (the default) and the daemon is healthy:
|
||||
`fipsctl show status`.
|
||||
|
||||
**Step 2.** Verify the gateway is listening on its DNS port:
|
||||
|
||||
```sh
|
||||
sudo ss -tulnp | grep -E ':53\b'
|
||||
```
|
||||
|
||||
If nothing is listening on the configured `dns.listen` address, the
|
||||
gateway either failed to start or is bound to a different address.
|
||||
Check the gateway log: `sudo journalctl -u fips-gateway -e`.
|
||||
|
||||
**Step 3.** Verify the LAN client can reach the gateway's DNS port:
|
||||
|
||||
```sh
|
||||
# from the LAN client
|
||||
dig @<gateway-lan-addr> hostname.fips AAAA
|
||||
```
|
||||
|
||||
If this hangs, the LAN-side firewall is blocking DNS, or the LAN
|
||||
route to the gateway is missing.
|
||||
|
||||
### Ping works but TCP does not
|
||||
|
||||
Symptom: `ping6 <virtual-ip>` succeeds from a LAN client, but TCP
|
||||
connections (SSH, HTTP) hang or time out.
|
||||
|
||||
This usually means the `fips0`-side masquerade rule is missing or
|
||||
misconfigured. Inspect the gateway's nftables table:
|
||||
|
||||
```sh
|
||||
sudo nft list table inet fips_gateway
|
||||
```
|
||||
|
||||
In the `postrouting` chain, look for a rule matching
|
||||
`oifname "fips0"` with a `masquerade` verdict. Without masquerade,
|
||||
the destination mesh node sees a source address (from the virtual
|
||||
pool) it cannot route replies to, and return packets are
|
||||
black-holed.
|
||||
|
||||
If the rule is missing, restart the gateway — the table is rebuilt
|
||||
atomically on every mapping change and on startup.
|
||||
|
||||
### Connection timeout to a virtual IP
|
||||
|
||||
Symptom: any traffic to a virtual pool address times out, including
|
||||
ping.
|
||||
|
||||
**Step 1.** Verify IPv6 forwarding is still enabled:
|
||||
|
||||
```sh
|
||||
sysctl net.ipv6.conf.all.forwarding
|
||||
# Expect: net.ipv6.conf.all.forwarding = 1
|
||||
```
|
||||
|
||||
**Step 2.** Verify the pool route exists:
|
||||
|
||||
```sh
|
||||
ip -6 route show table local | grep <pool-cidr>
|
||||
```
|
||||
|
||||
If the route is missing, the kernel does not recognize pool
|
||||
addresses as locally-owned and drops the packets before NAT can
|
||||
process them. The gateway adds this route at startup; if it's
|
||||
missing, check the gateway log for startup errors.
|
||||
|
||||
**Step 3.** Verify the destination mesh address actually exists in
|
||||
the FIPS daemon's identity cache:
|
||||
|
||||
```sh
|
||||
fipsctl show identity-cache | grep <fd00-mesh-addr>
|
||||
```
|
||||
|
||||
If the entry is missing, the DNS-side mapping never primed the
|
||||
identity cache, which means the daemon resolver did not actually
|
||||
resolve the `.fips` name. Re-test the DNS path:
|
||||
|
||||
```sh
|
||||
dig @::1 -p 5354 hostname.fips AAAA
|
||||
```
|
||||
|
||||
### Virtual IP unreachable from a LAN client
|
||||
|
||||
Symptom: client cannot reach the virtual IP at all (no ping, no
|
||||
ARP/ND response).
|
||||
|
||||
**Step 1.** Verify the client has a route to the pool via the
|
||||
gateway:
|
||||
|
||||
```sh
|
||||
# from the LAN client
|
||||
ip -6 route get <virtual-ip>
|
||||
```
|
||||
|
||||
The output should show the gateway as the next-hop. If it shows
|
||||
something else (or "unreachable"), fix the LAN-side route — see
|
||||
[deploy-gateway.md](deploy-gateway.md#distribute-the-route-to-lan-clients).
|
||||
|
||||
**Step 2.** On the gateway, verify proxy NDP entries exist for
|
||||
allocated virtual IPs:
|
||||
|
||||
```sh
|
||||
ip -6 neigh show proxy
|
||||
```
|
||||
|
||||
If proxy NDP entries are missing, the gateway cannot answer Neighbor
|
||||
Solicitation requests for virtual IPs on the LAN, so clients cannot
|
||||
resolve the link-layer address and packets never leave the client's
|
||||
NIC.
|
||||
|
||||
The gateway adds these entries when a mapping is created (i.e., when
|
||||
a `.fips` DNS query allocates a virtual IP). If they're absent,
|
||||
trigger a DNS query first:
|
||||
|
||||
```sh
|
||||
dig @<gateway-lan-addr> hostname.fips AAAA
|
||||
```
|
||||
|
||||
Then re-check `ip -6 neigh show proxy`.
|
||||
|
||||
**Step 3.** Verify `proxy_ndp` is enabled in the kernel:
|
||||
|
||||
```sh
|
||||
sysctl net.ipv6.conf.all.proxy_ndp
|
||||
# Expect: net.ipv6.conf.all.proxy_ndp = 1
|
||||
```
|
||||
|
||||
If 0, enable it (see
|
||||
[deploy-gateway.md](deploy-gateway.md#kernel-sysctls)).
|
||||
|
||||
## Inbound-half diagnostics
|
||||
|
||||
Symptoms in this section all involve a mesh peer trying to reach a
|
||||
LAN-side service through the gateway and failing.
|
||||
|
||||
### Mesh peer can't reach `<gateway-npub>.fips:<listen_port>`
|
||||
|
||||
Walk the path from the mesh-side ingress to the LAN target:
|
||||
|
||||
**Step 1.** Verify the port-forward rule is loaded. On the gateway:
|
||||
|
||||
```sh
|
||||
sudo nft list table inet fips_gateway
|
||||
```
|
||||
|
||||
Look in the `prerouting` chain for a rule of the form
|
||||
|
||||
```text
|
||||
iif "fips0" meta nfproto ipv6 meta l4proto <tcp|udp> \
|
||||
<th> dport <listen_port> dnat ip6 to [<target_addr>]:<target_port>
|
||||
```
|
||||
|
||||
and, in the `postrouting` chain, a rule of the form
|
||||
|
||||
```text
|
||||
iif "fips0" oif "<lan_interface>" meta nfproto ipv6 masquerade
|
||||
```
|
||||
|
||||
The port-forward DNAT and the LAN-side masquerade come from
|
||||
`gateway.port_forwards[]` and the active `lan_interface` setting.
|
||||
The masquerade is emitted only when at least one port-forward exists.
|
||||
If either rule is missing, restart the gateway — the table is rebuilt
|
||||
atomically on config load.
|
||||
|
||||
**Step 2.** Verify the mesh firewall is not blocking the listen
|
||||
port. If `fips-firewall.service` is enabled, the default baseline
|
||||
drops everything inbound on `fips0` except established/related and
|
||||
ICMPv6. Add an explicit allow rule under `/etc/fips/fips.d/`:
|
||||
|
||||
```nft
|
||||
# /etc/fips/fips.d/gateway-inbound.nft
|
||||
tcp dport <listen_port> accept
|
||||
```
|
||||
|
||||
(See [enable-mesh-firewall.md](enable-mesh-firewall.md) for the
|
||||
full drop-in pattern, including source-address restrictions.)
|
||||
Without an allow rule, mesh peers see TCP RSTs (the firewall drops
|
||||
on the way in) or silent UDP loss.
|
||||
|
||||
**Step 3.** Verify the LAN target is reachable from the gateway
|
||||
itself:
|
||||
|
||||
```sh
|
||||
ping6 <target_addr>
|
||||
curl -v http://[<target_addr>]:<target_port>/ # for TCP HTTP
|
||||
```
|
||||
|
||||
If the target is unreachable from the gateway, the DNAT rule will
|
||||
fire but the inner connection attempt will fail. Fix LAN-side
|
||||
routing or the target service before going further.
|
||||
|
||||
**Step 4.** Verify conntrack is tracking the inbound flow. Try the
|
||||
connection from a mesh peer once:
|
||||
|
||||
```sh
|
||||
curl -v http://<gateway-npub>.fips:<listen_port>/
|
||||
```
|
||||
|
||||
Then on the gateway:
|
||||
|
||||
```sh
|
||||
sudo conntrack -L | grep -E '<listen_port>|<target_port>'
|
||||
```
|
||||
|
||||
You should see a flow tuple in both directions (orig and reply) with
|
||||
the mesh peer's source on `fips0` and the gateway's LAN address as
|
||||
the masqueraded source on the LAN side. No conntrack entry suggests
|
||||
the prerouting DNAT didn't match — recheck step 1.
|
||||
|
||||
**Step 5.** Check the gateway log for nftables or rule install
|
||||
errors:
|
||||
|
||||
```sh
|
||||
sudo journalctl -u fips-gateway -e | grep -E 'port_forward|nftables'
|
||||
```
|
||||
|
||||
A "Failed to install port-forward rules" log line at startup means
|
||||
the rule batch was rejected by netlink — usually a transient
|
||||
condition during a config edit, but persistent failures warrant
|
||||
inspecting the rule with `nft -d`.
|
||||
|
||||
### Config rejected: IPv4 target
|
||||
|
||||
Symptom: `fips-gateway` exits at startup with a deserialization
|
||||
error referencing `port_forwards[N].target` and an invalid IPv6
|
||||
literal.
|
||||
|
||||
The `target` field is typed as `SocketAddrV6` and rejects IPv4
|
||||
literals at parse time:
|
||||
|
||||
```yaml
|
||||
# fails at config load
|
||||
- listen_port: 8080
|
||||
proto: tcp
|
||||
target: "192.168.1.10:80"
|
||||
```
|
||||
|
||||
Either re-address the LAN service to be reachable on IPv6, or front
|
||||
it with a small IPv6-aware reverse proxy on the gateway and point
|
||||
the `target` at that proxy.
|
||||
|
||||
### Config rejected: zero or duplicate listen_port
|
||||
|
||||
Symptom: `fips-gateway` exits at startup with
|
||||
"Invalid gateway.port_forwards: …".
|
||||
|
||||
`validate_port_forwards()` enforces:
|
||||
|
||||
- `listen_port` must be non-zero.
|
||||
- The pair `(listen_port, proto)` must be unique across the list
|
||||
(the same port on TCP and UDP simultaneously is allowed; the same
|
||||
port twice on the same proto is not).
|
||||
|
||||
Fix the offending entry and reload.
|
||||
|
||||
## See also
|
||||
|
||||
- [../tutorials/deploy-fips-gateway.md](../tutorials/deploy-fips-gateway.md) —
|
||||
canonical OpenWrt deployment.
|
||||
- [../design/fips-gateway.md](../design/fips-gateway.md) — gateway
|
||||
design, NAT pipeline, virtual IP pool lifecycle, security
|
||||
considerations.
|
||||
- [deploy-gateway.md](deploy-gateway.md) — manual Linux-host setup.
|
||||
- [Gateway section](../reference/configuration.md#gateway-gateway) of
|
||||
the configuration reference — full `gateway.*` block.
|
||||
- [../reference/cli-fips-gateway.md](../reference/cli-fips-gateway.md) —
|
||||
`fips-gateway` binary CLI flags.
|
||||
- [Gateway command catalog](../reference/control-socket.md#gateway-command-catalog)
|
||||
in the control-socket reference — JSON schema for `show_gateway`
|
||||
and `show_mappings`.
|
||||
- [enable-mesh-firewall.md](enable-mesh-firewall.md) — mesh-firewall
|
||||
baseline and drop-ins.
|
||||
@@ -0,0 +1,125 @@
|
||||
# Tune Host UDP Socket Buffers for FIPS
|
||||
|
||||
The FIPS UDP transport requests larger send and receive socket
|
||||
buffers (default 2 MB each, doubled by the kernel to 4 MB actual)
|
||||
than the Linux defaults provide. The kernel silently clamps the
|
||||
request to `net.core.rmem_max` and `net.core.wmem_max` if those
|
||||
sysctls are smaller than the requested size — which causes silent
|
||||
packet drops under high throughput. For the design context (why FIPS
|
||||
requests larger buffers and how `SO_RXQ_OVFL` feeds ECN congestion
|
||||
detection), see
|
||||
[../design/fips-transport-layer.md](../design/fips-transport-layer.md#socket-buffer-sizing).
|
||||
|
||||
This guide covers the host-side sysctl setup needed before deploying
|
||||
a high-throughput FIPS node.
|
||||
|
||||
## Why this matters
|
||||
|
||||
The default Linux UDP receive buffer (`net.core.rmem_default`,
|
||||
typically 212 KB) fills in roughly 2.5 ms at ~85 MB/s. Any stall in
|
||||
the FIPS receive loop (decryption, routing, forwarding) causes the
|
||||
kernel to drop incoming datagrams without notification — they don't
|
||||
appear in `recv` errors, they don't trigger any application-visible
|
||||
event. The drops show up only in `SO_RXQ_OVFL` on subsequent
|
||||
packets, where FIPS surfaces them as congestion-detection events.
|
||||
|
||||
Setting `rmem_max` and `wmem_max` to at least the requested buffer
|
||||
size prevents the kernel clamp and the silent drop loss it causes.
|
||||
|
||||
## Step 1: Check current limits
|
||||
|
||||
```sh
|
||||
sysctl net.core.rmem_max net.core.wmem_max
|
||||
```
|
||||
|
||||
Typical defaults on stock Linux distributions are 212992 bytes
|
||||
(212 KB). FIPS requests 2 MB by default, which the kernel doubles
|
||||
internally to 4 MB; for the request to succeed without clamping, both
|
||||
sysctls must be at least 4194304 (4 MB).
|
||||
|
||||
## Step 2: Set the limits temporarily
|
||||
|
||||
```sh
|
||||
sudo sysctl -w net.core.rmem_max=4194304
|
||||
sudo sysctl -w net.core.wmem_max=4194304
|
||||
```
|
||||
|
||||
Verify:
|
||||
|
||||
```sh
|
||||
sysctl net.core.rmem_max net.core.wmem_max
|
||||
```
|
||||
|
||||
These changes take effect immediately for new socket binds but do
|
||||
not survive a reboot.
|
||||
|
||||
## Step 3: Make the limits persistent
|
||||
|
||||
Drop a file under `/etc/sysctl.d/`:
|
||||
|
||||
```sh
|
||||
sudo tee /etc/sysctl.d/60-fips.conf <<'EOF'
|
||||
# FIPS UDP transport requests 2 MB socket buffers, kernel doubles to 4 MB.
|
||||
# Avoid silent receive-buffer drops under load.
|
||||
net.core.rmem_max = 4194304
|
||||
net.core.wmem_max = 4194304
|
||||
EOF
|
||||
```
|
||||
|
||||
Apply:
|
||||
|
||||
```sh
|
||||
sudo sysctl --system
|
||||
```
|
||||
|
||||
The drop-in is loaded automatically on every boot.
|
||||
|
||||
## Step 4: Restart FIPS and verify the actual buffer size
|
||||
|
||||
After raising the host limits, restart the FIPS daemon so the next
|
||||
socket bind picks up the new ceiling:
|
||||
|
||||
```sh
|
||||
sudo systemctl restart fips
|
||||
```
|
||||
|
||||
The daemon logs the actual buffer sizes at startup:
|
||||
|
||||
```text
|
||||
UDP transport started local_addr=0.0.0.0:2121 recv_buf=4194304 send_buf=4194304
|
||||
```
|
||||
|
||||
If `recv_buf` or `send_buf` shows a smaller number than expected, the
|
||||
host sysctl is still clamping. Recheck `sysctl net.core.rmem_max
|
||||
net.core.wmem_max` and confirm the drop-in file is being loaded
|
||||
(`sudo sysctl --system` prints the loaded files).
|
||||
|
||||
## Docker and other container hosts
|
||||
|
||||
Containers share the host kernel, so sysctls apply to the host, not
|
||||
the container. If you run FIPS inside Docker, set
|
||||
`net.core.rmem_max` / `net.core.wmem_max` on the **Docker host**, not
|
||||
inside the container. Container privileges (cap_sys_admin) and
|
||||
`--sysctl` flags do not let you raise these particular limits from
|
||||
inside a container — they are global to the host network namespace.
|
||||
|
||||
For Kubernetes deployments, the host-level sysctl tuning is the same;
|
||||
node-level configuration (DaemonSet with `privileged: true`, or a
|
||||
node-init script) is the typical mechanism.
|
||||
|
||||
## Tuning higher
|
||||
|
||||
The 4 MB ceiling is a conservative starting point. For very high
|
||||
throughput (multi-gigabit per second), raise both sysctls and the
|
||||
corresponding `transports.udp.recv_buf_size` /
|
||||
`transports.udp.send_buf_size` config values together. Setting a config value
|
||||
larger than the host ceiling silently clamps to the ceiling, so both
|
||||
must move in lockstep.
|
||||
|
||||
## See also
|
||||
|
||||
- [../design/fips-transport-layer.md](../design/fips-transport-layer.md)
|
||||
— UDP transport design, why FIPS requests larger buffers,
|
||||
`SO_RXQ_OVFL` and ECN integration
|
||||
- [../reference/configuration.md](../reference/configuration.md) —
|
||||
`transports.udp.recv_buf_size` and `send_buf_size` defaults
|
||||
Reference in New Issue
Block a user