laantungir/fips
1
0
Fork 0
mirror of https://github.com/jmcorgan/fips.git synced 2026-07-30 19:46:15 +00:00
Code Issues Packages Projects Releases Wiki Activity

docs: four-section /docs/ restructure with new-user content, accuracy pass, and gateway feature-set rewrite

Browse Source 
Restructures /docs/ by reader purpose (tutorials, how-to,
reference, design), adds the new-user-progression and
operator-recipe content the prior layout lacked, runs an
accuracy pass against current source across the pre-existing
design docs, and rewrites the gateway feature-set documentation
end-to-end around its actual operational profile (a niche
feature designed for systems already serving DHCP/DNS to a
LAN, with two independent halves — outbound LAN→mesh, inbound
mesh→LAN — sharing one nftables table, one binary, and one
control socket). Top-level README and getting-started rewritten
around two equally-weighted deployment modes (overlay on
existing IP networks; ground-up over non-IP transports).

## Additions

- 11 new tutorials in docs/tutorials/: an 8-step new-user
  progression from single-daemon test-mesh peering through
  to a ground-up two-device mesh, an IPv6-adapter side-trip
  walkthrough, an Advanced Tutorials index, and a hand-held
  OpenWrt walk-through for fips-gateway deployment that
  exercises both halves of the feature.
- 12 new how-tos in docs/how-to/: firewall activation,
  Nostr discovery (resolve / advertise / open across five
  scenarios), Tor onion (directory + control_port modes),
  UDP buffer tuning, unprivileged-user setup, persistent
  identity, host aliases, Bluetooth LE peering, MTU
  diagnostics, manual Linux-host gateway deployment (covers
  both halves), gateway troubleshooting (organised by half),
  and a section index.
- 9 new reference docs in docs/reference/: configuration,
  wire formats, control-socket protocol, four CLI references
  (fips, fipsctl, fipstop, fips-gateway), security posture
  matrix, and Nostr events catalog. Configuration and
  wire-formats are renamed-and-extended from prior design/
  versions; the other seven are net-new.
- 6 new design docs: fips-concepts, fips-architecture, and
  fips-prior-work split out of the deleted fips-intro.md;
  consolidated fips-mmp and fips-mtu aggregations; and a
  new generic port-advertisement-and-nat-traversal doc
  (Nostr-signaled port advertisement plus UDP NAT-traversal
  protocol, FIPS as an example implementation, suitable for
  eventual NIP submission).
- Top-level docs/getting-started.md walking through the
  binary-installer-only Install story.
- packaging/common/hosts pre-populated with the eight public
  test-mesh nodes so shortnames resolve out of the box on
  every fresh install.

## Changes

- 23 wire-format diagrams relocated to reference/diagrams/
  alongside the wire-formats move.
- 4 design diagrams corrected against source code
  (fips-protocol-stack, fips-identity-derivation,
  fips-coordinate-discovery, fips-routing-decision).
- 10 pre-existing design docs reconciled with current
  source. Numeric corrections: stale link-MMP report bounds
  (now [1s, 5s] with 200 ms cold-start floor); UDP default
  MTU (now 1280, IPv6 minimum); node_addr formula
  (SHA-256(pubkey)[..16]); Noise patterns (IK at link, XK
  at session); peer-ACL semantics (strict allowlist requires
  ALL in peers.deny); daemon DNS upstream ([::1]:5354);
  on-the-wire bloom-filter size (1,071 bytes); obsolete
  Cargo-feature references (PR #79 dropped them) removed.
- Transport framing tightened across the docs: TCP is for
  UDP-filtered networks (not NAT traversal); Tor is a
  deployment mode (not failover); WebSocket dropped (not a
  shipped FIPS transport); WiFi promoted to Implemented via
  Ethernet in infrastructure mode; classic-Bluetooth row
  removed (BLE is the only Bluetooth-mode transport).
- docs/design/fips-gateway.md rewritten end-to-end to lead
  with the niche-feature framing and the two-halves
  structure. Title moved from "FIPS Outbound LAN Gateway"
  to "FIPS Gateway"; architecture section describes the
  common machinery (the fips-gateway service, the nftables
  table, the control socket) before splitting into separate
  "Outbound Half" and "Inbound Half" sections of equal
  weight; security considerations split per-half; no Future
  Work section (speculative directions live in the project
  tracker, not in protocol design docs). Inbound port
  forwarding is a first-class half rather than a buried
  "Implemented Extensions" subsection.
- Gateway terminology unified across all gateway docs as a
  separate Linux service running alongside the fips daemon
  (its own systemd unit / OpenWrt init script). Container-
  pattern terms (sidecar) are reserved for the
  Docker/Kubernetes sidecar deployment examples — the
  testing/sidecar/ tree, examples/k8s-sidecar/,
  examples/sidecar-nostr-relay/,
  examples/wireguard-sidecar-macos/, and the related
  CHANGELOG / top-level README entries — where the term
  carries its standard container meaning.
- Net-new design body content: rekey section in
  fips-mesh-layer (Noise IK msg1/msg2 over the established
  link, K-bit cutover, drain window, smaller-NodeAddr-wins
  tie-breaker on dual-init); Mesh Size Estimation and
  Antipoison FPR Cap sections in fips-bloom-filters;
  Mesh-Interface Query Filter subsection in
  fips-ipv6-adapter; failure-suppression knobs and clock-
  skew tolerance in fips-nostr-discovery; loop-rejection
  and mid-chain ancestor swap added to spanning-tree
  propagation / stability rules; Priority Chain in
  fips-mesh-operation renumbered to match the
  routing-decision diagram.
- Top-level README: dropped the stale nostr-discovery
  cargo-feature parenthetical. docs/README.md and the four
  section READMEs (tutorials, how-to, reference, design)
  refreshed for the new structure; index rows reflect both
  halves of the gateway feature and the new fips-gateway
  CLI reference.
- Cargo.toml [package.metadata.deb] assets path updated for
  the fips-security.md move; .gitignore /reference/ rule
  anchored to repo root so docs/reference/ is trackable.
- packaging/openwrt-ipk/files/etc/fips/fips.yaml
  configuration-doc URL updated to the new
  docs/reference/configuration.md location.

## Deletions

- docs/design/fips-intro.md (split into the three new intro
  design docs).
- docs/design/document-relationships.svg (orphan, no longer
  referenced).
- docs/proposals/ tree removed; the only proposal it
  contained (the Nostr UDP hole-punch protocol) was
  rewritten as the new generic
  design/port-advertisement-and-nat-traversal.md.
This commit is contained in:
Johnathan Corgan
2026-05-08 03:02:12 +00:00
parent 4cdf382038
commit 5abf9a9325
91 changed files with 11474 additions and 3278 deletions
Download Patch File Download Diff File Expand all files Collapse all files
docs/design/README.md
+38 -25
View File
@@ -1,52 +1,65 @@
# FIPS Design Documents
# FIPS Design
Protocol design specifications for the Federated Interoperable Peering
System — a self-organizing encrypted mesh network built on Nostr identities.
Architectural and protocol-level explanations for FIPS — the *why*
and the *how* behind the wire and the system. For wire formats and
configuration keys, see [reference/](../reference/). For task
recipes, see [how-to/](../how-to/). For end-to-end lessons, see
[tutorials/](../tutorials/).
## Reading Order
Start with the introduction, then follow the protocol stack from bottom to
top. After the stack, the mesh operation document explains how all the
pieces work together. Supporting references provide deeper dives into
specific topics.
Start with [fips-concepts.md](fips-concepts.md) for the
novice-friendly framing of what FIPS is and why, then move to
[fips-architecture.md](fips-architecture.md) for the protocol stack,
identity model, and two-layer encryption walkthrough. From there,
follow the protocol stack from bottom to top. After the stack,
[fips-mesh-operation.md](fips-mesh-operation.md) explains how the
pieces work together at runtime. Cross-cutting and supporting
documents cover specific subsystems in detail.
### Foundations
| Document | Description |
| -------- | ----------- |
| [fips-concepts.md](fips-concepts.md) | What FIPS is, why it exists, mental model |
| [fips-architecture.md](fips-architecture.md) | Protocol stack, identity, two-layer encryption |
| [fips-prior-work.md](fips-prior-work.md) | Designs and protocols FIPS builds on |
### Protocol Stack
| Document | Description |
| -------- | ----------- |
| [fips-intro.md](fips-intro.md) | Protocol introduction: goals, architecture, layer model |
| [fips-transport-layer.md](fips-transport-layer.md) | Transport layer: datagram delivery over arbitrary media |
| [fips-mesh-layer.md](fips-mesh-layer.md) | FIPS Mesh Protocol (FMP): peer authentication, link encryption, forwarding |
| [fips-session-layer.md](fips-session-layer.md) | FIPS Session Protocol (FSP): end-to-end encryption, sessions |
| [fips-ipv6-adapter.md](fips-ipv6-adapter.md) | IPv6 adaptation: TUN interface, DNS, MTU enforcement |
### Cross-Cutting
| Document | Description |
| -------- | ----------- |
| [fips-mmp.md](fips-mmp.md) | Metrics Measurement Protocol (link + session) |
| [fips-mtu.md](fips-mtu.md) | Path MTU model, encapsulation overhead, PMTUD |
| [fips-security.md](fips-security.md) | `fips0` interface threat model and default-deny baseline |
### Mesh Behavior
| Document | Description |
| -------- | ----------- |
| [fips-mesh-operation.md](fips-mesh-operation.md) | How the mesh operates: routing, discovery, error recovery |
| [fips-wire-formats.md](fips-wire-formats.md) | Wire format reference for all message types |
| [fips-nostr-discovery.md](fips-nostr-discovery.md) | Optional Nostr-mediated peer discovery and UDP NAT hole-punch (behind `nostr-discovery` feature) |
| [fips-nostr-discovery.md](fips-nostr-discovery.md) | Optional Nostr-mediated peer discovery and UDP NAT hole-punch |
| [port-advertisement-and-nat-traversal.md](port-advertisement-and-nat-traversal.md) | Nostr-signaled port advertisement and UDP NAT-traversal protocol; generic, with FIPS as an example implementation |
### Supporting References
### Deeper Dives
| Document | Description |
| -------- | ----------- |
| [fips-spanning-tree.md](fips-spanning-tree.md) | Spanning tree algorithms: root discovery, parent selection, coordinates |
| [fips-bloom-filters.md](fips-bloom-filters.md) | Bloom filter math: FPR analysis, size classes, split-horizon |
### Implementation
| Document | Description |
| -------- | ----------- |
| [fips-configuration.md](fips-configuration.md) | YAML configuration reference |
### Supplemental
| Document | Description |
| -------- | ----------- |
| [fips-bloom-filters.md](fips-bloom-filters.md) | Bloom filter properties: FPR analysis, size classes, split-horizon |
| [spanning-tree-dynamics.md](spanning-tree-dynamics.md) | Spanning tree walkthroughs: convergence scenarios, worked examples |
## Document Relationships
### Adjacent Components
![Document relationships](document-relationships.svg)
| Document | Description |
| -------- | ----------- |
| [fips-gateway.md](fips-gateway.md) | `fips-gateway` service: outbound (LAN-to-mesh) DNS-proxy + virtual-IP NAT and inbound (mesh-to-LAN) port-forwarding, sharing one nftables table |
docs/design/diagrams/ancestry-entry.svg
-28
View File
@@ -1,28 +0,0 @@
<svg xmlns="http://www.w3.org/2000/svg" viewBox="0 0 580 192" font-family="monospace" font-size="13">
<!-- Background -->
<rect width="580" height="192" fill="#1a1a2e" rx="4"/>
<!-- Title -->
<text x="310" y="26" fill="#e0e0e0" text-anchor="middle" font-size="14" font-weight="bold">AncestryEntry (32 bytes)</text>
<!-- Row 0 (0–15): node_addr -->
<text x="50" y="62" fill="#666" font-size="10" text-anchor="end">0–15</text>
<rect x="55" y="36" width="520" height="52" fill="#2d5a5a" stroke="#4ad9d9" stroke-width="1.5" rx="3"/>
<text x="315" y="58" fill="#e0e0e0" text-anchor="middle" font-size="12" font-weight="bold">node_addr</text>
<text x="315" y="76" fill="#8af8f8" text-anchor="middle" font-size="10">16 bytes — routing identifier</text>
<!-- Row 1 (16–23): sequence -->
<text x="50" y="114" fill="#666" font-size="10" text-anchor="end">16–23</text>
<rect x="55" y="88" width="520" height="52" fill="#5a3d2d" stroke="#d9904a" stroke-width="1.5" rx="3"/>
<text x="315" y="110" fill="#e0e0e0" text-anchor="middle" font-size="12" font-weight="bold">sequence</text>
<text x="315" y="128" fill="#f8c88a" text-anchor="middle" font-size="10">8 bytes LE</text>
<!-- Row 2 (24–31): timestamp -->
<text x="50" y="166" fill="#666" font-size="10" text-anchor="end">24–31</text>
<rect x="55" y="140" width="520" height="52" fill="#4a2d5a" stroke="#b04ad9" stroke-width="1.5" rx="3"/>
<text x="315" y="162" fill="#e0e0e0" text-anchor="middle" font-size="12" font-weight="bold">timestamp</text>
<text x="315" y="180" fill="#d8a0f8" text-anchor="middle" font-size="10">8 bytes LE — Unix seconds</text>
</svg>
Side by Side

Before

Width:  |  Height:  |  Size: 1.6 KiB

docs/design/diagrams/common-prefix.svg
-39
View File
@@ -1,39 +0,0 @@
<svg xmlns="http://www.w3.org/2000/svg" viewBox="0 0 580 160" font-family="monospace" font-size="13">
<!-- Background -->
<rect width="580" height="160" fill="#1a1a2e" rx="4"/>
<!-- Title -->
<text x="290" y="26" fill="#e0e0e0" text-anchor="middle" font-size="14" font-weight="bold">FMP/FSP Common Prefix (4 bytes)</text>
<!-- Bit ruler -->
<text x="65" y="50" fill="#888" font-size="10" text-anchor="middle">0</text>
<text x="130" y="50" fill="#888" font-size="10" text-anchor="middle">4</text>
<text x="195" y="50" fill="#888" font-size="10" text-anchor="middle">8</text>
<text x="325" y="50" fill="#888" font-size="10" text-anchor="middle">16</text>
<text x="530" y="50" fill="#888" font-size="10" text-anchor="middle">32</text>
<!-- Byte 0: version (high nibble) -->
<rect x="35" y="56" width="95" height="52" fill="#2d4a7a" stroke="#4a90d9" stroke-width="1.5" rx="3"/>
<text x="82" y="80" fill="#e0e0e0" text-anchor="middle" font-size="12" font-weight="bold">ver</text>
<text x="82" y="98" fill="#8ab4f8" text-anchor="middle" font-size="10">4 bits</text>
<!-- Byte 0: phase (low nibble) -->
<rect x="130" y="56" width="95" height="52" fill="#2d5a4a" stroke="#4ad99a" stroke-width="1.5" rx="3"/>
<text x="177" y="80" fill="#e0e0e0" text-anchor="middle" font-size="12" font-weight="bold">phase</text>
<text x="177" y="98" fill="#8af8c8" text-anchor="middle" font-size="10">4 bits</text>
<!-- Byte 1: flags -->
<rect x="225" y="56" width="130" height="52" fill="#5a3d2d" stroke="#d9904a" stroke-width="1.5" rx="3"/>
<text x="290" y="80" fill="#e0e0e0" text-anchor="middle" font-size="12" font-weight="bold">flags</text>
<text x="290" y="98" fill="#f8c88a" text-anchor="middle" font-size="10">1 byte</text>
<!-- Bytes 2-3: payload_len -->
<rect x="355" y="56" width="200" height="52" fill="#4a2d5a" stroke="#b04ad9" stroke-width="1.5" rx="3"/>
<text x="455" y="80" fill="#e0e0e0" text-anchor="middle" font-size="12" font-weight="bold">payload_len</text>
<text x="455" y="98" fill="#d8a0f8" text-anchor="middle" font-size="10">2 bytes LE</text>
<!-- Byte offset labels -->
<text x="82" y="130" fill="#666" font-size="10" text-anchor="middle">byte 0</text>
<text x="290" y="130" fill="#666" font-size="10" text-anchor="middle">byte 1</text>
<text x="455" y="130" fill="#666" font-size="10" text-anchor="middle">bytes 2-3</text>
</svg>
Side by Side

Before

Width:  |  Height:  |  Size: 2.3 KiB

docs/design/diagrams/coords-required.svg
-34
View File
@@ -1,34 +0,0 @@
<svg xmlns="http://www.w3.org/2000/svg" viewBox="0 0 580 218" font-family="monospace" font-size="13">
<!-- Background -->
<rect width="580" height="218" fill="#1a1a2e" rx="4"/>
<!-- Title -->
<text x="310" y="26" fill="#e0e0e0" text-anchor="middle" font-size="14" font-weight="bold">CoordsRequired (0x20) — 34 bytes payload</text>
<!-- Row 0 (0–1): msg_type + flags -->
<text x="50" y="62" fill="#666" font-size="10" text-anchor="end">0–1</text>
<rect x="55" y="36" width="260" height="52" fill="#2d4a7a" stroke="#4a90d9" stroke-width="1.5" rx="3"/>
<text x="185" y="58" fill="#e0e0e0" text-anchor="middle" font-size="12" font-weight="bold">msg_type</text>
<text x="185" y="76" fill="#8ab4f8" text-anchor="middle" font-size="10">0x20</text>
<rect x="315" y="36" width="260" height="52" fill="#1f1f3a" stroke="#444" stroke-width="1" rx="3"/>
<text x="445" y="62" fill="#666" text-anchor="middle" font-size="12">flags (reserved)</text>
<!-- Row 1 (2–17): dest_addr -->
<text x="50" y="114" fill="#666" font-size="10" text-anchor="end">2–17</text>
<rect x="55" y="88" width="520" height="52" fill="#4a2d5a" stroke="#b04ad9" stroke-width="1.5" rx="3"/>
<text x="315" y="110" fill="#e0e0e0" text-anchor="middle" font-size="12" font-weight="bold">dest_addr</text>
<text x="315" y="128" fill="#d8a0f8" text-anchor="middle" font-size="10">16 bytes — NodeAddr we couldn't route to</text>
<!-- Row 2 (18–33): reporter -->
<text x="50" y="166" fill="#666" font-size="10" text-anchor="end">18–33</text>
<rect x="55" y="140" width="520" height="52" fill="#2d5a5a" stroke="#4ad9d9" stroke-width="1.5" rx="3"/>
<text x="315" y="162" fill="#e0e0e0" text-anchor="middle" font-size="12" font-weight="bold">reporter</text>
<text x="315" y="180" fill="#8af8f8" text-anchor="middle" font-size="10">16 bytes — NodeAddr of reporting router</text>
<!-- Note -->
<text x="310" y="212" fill="#777" font-size="10" text-anchor="middle">plaintext error signal (U flag set) · total on wire: 38 bytes (4 prefix + 34 payload)</text>
</svg>
Side by Side

Before

Width:  |  Height:  |  Size: 2.0 KiB

docs/design/diagrams/disconnect.svg
-18
View File
@@ -1,18 +0,0 @@
<svg xmlns="http://www.w3.org/2000/svg" viewBox="0 0 580 114" font-family="monospace" font-size="13">
<!-- Background -->
<rect width="580" height="114" fill="#1a1a2e" rx="4"/>
<!-- Title -->
<text x="310" y="26" fill="#e0e0e0" text-anchor="middle" font-size="14" font-weight="bold">Disconnect (0x50) — 2 bytes</text>
<!-- Row 0 (0–1): msg_type + reason -->
<text x="50" y="62" fill="#666" font-size="10" text-anchor="end">0–1</text>
<rect x="55" y="36" width="260" height="52" fill="#2d4a7a" stroke="#4a90d9" stroke-width="1.5" rx="3"/>
<text x="185" y="60" fill="#e0e0e0" text-anchor="middle" font-size="12" font-weight="bold">msg_type</text>
<text x="185" y="78" fill="#8ab4f8" text-anchor="middle" font-size="10">0x50</text>
<rect x="315" y="36" width="260" height="52" fill="#5a2d3d" stroke="#d94a6a" stroke-width="1.5" rx="3"/>
<text x="445" y="60" fill="#e0e0e0" text-anchor="middle" font-size="12" font-weight="bold">reason</text>
<text x="445" y="78" fill="#f88aaa" text-anchor="middle" font-size="10">1 byte</text>
</svg>
Side by Side

Before

Width:  |  Height:  |  Size: 1.0 KiB

docs/design/diagrams/established-complete-frame.svg
-91
View File
@@ -1,91 +0,0 @@
<svg xmlns="http://www.w3.org/2000/svg" viewBox="0 0 580 498" font-family="monospace" font-size="13">
<!-- Background -->
<rect width="580" height="498" fill="#1a1a2e" rx="4"/>
<!-- Title -->
<text x="310" y="26" fill="#e0e0e0" text-anchor="middle" font-size="14" font-weight="bold">Complete Encrypted Frame (37 + body bytes)</text>
<!-- Bit ruler -->
<text x="85" y="50" fill="#888" font-size="10" text-anchor="middle">0</text>
<text x="150" y="50" fill="#888" font-size="10" text-anchor="middle">4</text>
<text x="215" y="50" fill="#888" font-size="10" text-anchor="middle">8</text>
<text x="345" y="50" fill="#888" font-size="10" text-anchor="middle">16</text>
<text x="550" y="50" fill="#888" font-size="10" text-anchor="middle">32</text>
<!-- Row 0: Common prefix (bytes 0-3) -->
<text x="50" y="82" fill="#666" font-size="10" text-anchor="end">0–3</text>
<rect x="55" y="56" width="95" height="52" fill="#2d4a7a" stroke="#4a90d9" stroke-width="1.5" rx="3"/>
<text x="102" y="80" fill="#e0e0e0" text-anchor="middle" font-size="12" font-weight="bold">ver</text>
<text x="102" y="98" fill="#8ab4f8" text-anchor="middle" font-size="10">4 bits</text>
<rect x="150" y="56" width="95" height="52" fill="#2d5a4a" stroke="#4ad99a" stroke-width="1.5" rx="3"/>
<text x="197" y="80" fill="#e0e0e0" text-anchor="middle" font-size="12" font-weight="bold">phase</text>
<text x="197" y="98" fill="#8af8c8" text-anchor="middle" font-size="10">4 bits</text>
<rect x="245" y="56" width="130" height="52" fill="#5a3d2d" stroke="#d9904a" stroke-width="1.5" rx="3"/>
<text x="310" y="80" fill="#e0e0e0" text-anchor="middle" font-size="12" font-weight="bold">flags</text>
<text x="310" y="98" fill="#f8c88a" text-anchor="middle" font-size="10">1 byte</text>
<rect x="375" y="56" width="200" height="52" fill="#4a2d5a" stroke="#b04ad9" stroke-width="1.5" rx="3"/>
<text x="475" y="80" fill="#e0e0e0" text-anchor="middle" font-size="12" font-weight="bold">payload_len</text>
<text x="475" y="98" fill="#d8a0f8" text-anchor="middle" font-size="10">2 bytes LE</text>
<!-- Row 1: receiver_idx (bytes 4-7) -->
<text x="50" y="134" fill="#666" font-size="10" text-anchor="end">4–7</text>
<rect x="55" y="108" width="520" height="52" fill="#2d5a5a" stroke="#4ad9d9" stroke-width="1.5" rx="3"/>
<text x="315" y="132" fill="#e0e0e0" text-anchor="middle" font-size="12" font-weight="bold">receiver_idx</text>
<text x="315" y="150" fill="#8af8f8" text-anchor="middle" font-size="10">4 bytes LE</text>
<!-- Row 2: counter low (bytes 8-11) -->
<text x="50" y="186" fill="#666" font-size="10" text-anchor="end">8–11</text>
<rect x="55" y="160" width="520" height="52" fill="#5a2d3d" stroke="#d94a6a" stroke-width="1.5" rx="3"/>
<text x="315" y="190" fill="#e0e0e0" text-anchor="middle" font-size="12" font-weight="bold">counter / lower 4 bytes LE</text>
<!-- Row 3: counter high (bytes 12-15) -->
<text x="50" y="238" fill="#666" font-size="10" text-anchor="end">12–15</text>
<rect x="55" y="212" width="520" height="52" fill="#5a2d3d" stroke="#d94a6a" stroke-width="1.5" rx="3"/>
<text x="315" y="242" fill="#e0e0e0" text-anchor="middle" font-size="12" font-weight="bold">counter / upper 4 bytes LE</text>
<!-- AAD bracket -->
<text x="10" y="166" fill="#777" font-size="10" text-anchor="middle" transform="rotate(-90, 10, 166)">AAD</text>
<line x1="18" y1="56" x2="18" y2="264" stroke="#555" stroke-width="1"/>
<line x1="18" y1="56" x2="23" y2="56" stroke="#555" stroke-width="1"/>
<line x1="18" y1="264" x2="23" y2="264" stroke="#555" stroke-width="1"/>
<!-- Separator: cleartext above / ciphertext below -->
<line x1="55" y1="272" x2="575" y2="272" stroke="#555" stroke-width="1" stroke-dasharray="6,4"/>
<text x="315" y="288" fill="#777" font-size="10" text-anchor="middle">── ciphertext (encrypted with ChaCha20-Poly1305) ──</text>
<!-- Row 4: timestamp (bytes 16-19, encrypted) -->
<text x="50" y="322" fill="#666" font-size="10" text-anchor="end">16–19</text>
<rect x="55" y="296" width="520" height="52" fill="#3d4a2d" stroke="#8ad94a" stroke-width="1.5" rx="3"/>
<text x="315" y="320" fill="#e0e0e0" text-anchor="middle" font-size="12" font-weight="bold">timestamp</text>
<text x="315" y="338" fill="#c8f88a" text-anchor="middle" font-size="10">4 bytes LE</text>
<!-- Row 5: msg_type (1 byte) + body starts -->
<text x="50" y="374" fill="#666" font-size="10" text-anchor="end">20</text>
<rect x="55" y="348" width="130" height="52" fill="#4a3d2d" stroke="#d9b04a" stroke-width="1.5" rx="3"/>
<text x="120" y="372" fill="#e0e0e0" text-anchor="middle" font-size="12" font-weight="bold">msg_type</text>
<text x="120" y="390" fill="#f8d88a" text-anchor="middle" font-size="10">1 byte</text>
<rect x="185" y="348" width="390" height="52" fill="#1f1f3a" stroke="#444" stroke-width="1" stroke-dasharray="4,3" rx="3"/>
<text x="380" y="374" fill="#666" text-anchor="middle" font-size="11">message body ...</text>
<!-- Row 6: body variable -->
<text x="50" y="426" fill="#666" font-size="10" text-anchor="end">...</text>
<rect x="55" y="400" width="520" height="52" fill="#1f1f3a" stroke="#444" stroke-width="1" stroke-dasharray="4,3" rx="3"/>
<text x="315" y="430" fill="#666" text-anchor="middle" font-size="11">message body (variable length)</text>
<!-- AEAD tag -->
<text x="50" y="478" fill="#666" font-size="10" text-anchor="end">+16</text>
<rect x="55" y="452" width="520" height="30" fill="#4a2d4a" stroke="#b04ab0" stroke-width="1.5" rx="3"/>
<text x="315" y="472" fill="#e0e0e0" text-anchor="middle" font-size="11" font-weight="bold">Poly1305 AEAD tag (16 bytes)</text>
</svg>
Side by Side

Before

Width:  |  Height:  |  Size: 5.6 KiB

docs/design/diagrams/established-inner-header.svg
-31
View File
@@ -1,31 +0,0 @@
<svg xmlns="http://www.w3.org/2000/svg" viewBox="0 0 580 186" font-family="monospace" font-size="13">
<!-- Background -->
<rect width="580" height="186" fill="#1a1a2e" rx="4"/>
<!-- Title -->
<text x="310" y="26" fill="#e0e0e0" text-anchor="middle" font-size="14" font-weight="bold">Established Frame Inner Header (5 bytes, plaintext after decrypt)</text>
<!-- Bit ruler -->
<text x="85" y="50" fill="#888" font-size="10" text-anchor="middle">0</text>
<text x="215" y="50" fill="#888" font-size="10" text-anchor="middle">8</text>
<text x="345" y="50" fill="#888" font-size="10" text-anchor="middle">16</text>
<text x="550" y="50" fill="#888" font-size="10" text-anchor="middle">32</text>
<!-- Row 0: timestamp (bytes 0-3) -->
<text x="50" y="82" fill="#666" font-size="10" text-anchor="end">0–3</text>
<rect x="55" y="56" width="520" height="52" fill="#2d5a4a" stroke="#4ad99a" stroke-width="1.5" rx="3"/>
<text x="315" y="80" fill="#e0e0e0" text-anchor="middle" font-size="12" font-weight="bold">timestamp</text>
<text x="315" y="98" fill="#8af8c8" text-anchor="middle" font-size="10">4 bytes LE</text>
<!-- Row 1: msg_type (byte 4) -->
<text x="50" y="134" fill="#666" font-size="10" text-anchor="end">4</text>
<rect x="55" y="108" width="130" height="52" fill="#5a3d2d" stroke="#d9904a" stroke-width="1.5" rx="3"/>
<text x="120" y="132" fill="#e0e0e0" text-anchor="middle" font-size="12" font-weight="bold">msg_type</text>
<text x="120" y="150" fill="#f8c88a" text-anchor="middle" font-size="10">1 byte</text>
<!-- Row 1: body starts -->
<rect x="185" y="108" width="390" height="52" fill="#1f1f3a" stroke="#444" stroke-width="1" stroke-dasharray="4,3" rx="3"/>
<text x="380" y="134" fill="#666" text-anchor="middle" font-size="11">message body ...</text>
</svg>
Side by Side

Before

Width:  |  Height:  |  Size: 1.8 KiB

docs/design/diagrams/established-outer-header.svg
-52
View File
@@ -1,52 +0,0 @@
<svg xmlns="http://www.w3.org/2000/svg" viewBox="0 0 580 290" font-family="monospace" font-size="13">
<!-- Background -->
<rect width="580" height="290" fill="#1a1a2e" rx="4"/>
<!-- Title -->
<text x="310" y="26" fill="#e0e0e0" text-anchor="middle" font-size="14" font-weight="bold">Established Frame Outer Header (16 bytes, AEAD AAD)</text>
<!-- Bit ruler -->
<text x="85" y="50" fill="#888" font-size="10" text-anchor="middle">0</text>
<text x="150" y="50" fill="#888" font-size="10" text-anchor="middle">4</text>
<text x="215" y="50" fill="#888" font-size="10" text-anchor="middle">8</text>
<text x="345" y="50" fill="#888" font-size="10" text-anchor="middle">16</text>
<text x="550" y="50" fill="#888" font-size="10" text-anchor="middle">32</text>
<!-- Row 0: Common prefix (bytes 0-3) -->
<text x="50" y="82" fill="#666" font-size="10" text-anchor="end">0–3</text>
<rect x="55" y="56" width="95" height="52" fill="#2d4a7a" stroke="#4a90d9" stroke-width="1.5" rx="3"/>
<text x="102" y="80" fill="#e0e0e0" text-anchor="middle" font-size="12" font-weight="bold">ver</text>
<text x="102" y="98" fill="#8ab4f8" text-anchor="middle" font-size="10">4 bits</text>
<rect x="150" y="56" width="95" height="52" fill="#2d5a4a" stroke="#4ad99a" stroke-width="1.5" rx="3"/>
<text x="197" y="80" fill="#e0e0e0" text-anchor="middle" font-size="12" font-weight="bold">phase</text>
<text x="197" y="98" fill="#8af8c8" text-anchor="middle" font-size="10">4 bits</text>
<rect x="245" y="56" width="130" height="52" fill="#5a3d2d" stroke="#d9904a" stroke-width="1.5" rx="3"/>
<text x="310" y="80" fill="#e0e0e0" text-anchor="middle" font-size="12" font-weight="bold">flags</text>
<text x="310" y="98" fill="#f8c88a" text-anchor="middle" font-size="10">1 byte</text>
<rect x="375" y="56" width="200" height="52" fill="#4a2d5a" stroke="#b04ad9" stroke-width="1.5" rx="3"/>
<text x="475" y="80" fill="#e0e0e0" text-anchor="middle" font-size="12" font-weight="bold">payload_len</text>
<text x="475" y="98" fill="#d8a0f8" text-anchor="middle" font-size="10">2 bytes LE</text>
<!-- Row 1: receiver_idx (bytes 4-7) -->
<text x="50" y="134" fill="#666" font-size="10" text-anchor="end">4–7</text>
<rect x="55" y="108" width="520" height="52" fill="#2d5a5a" stroke="#4ad9d9" stroke-width="1.5" rx="3"/>
<text x="315" y="132" fill="#e0e0e0" text-anchor="middle" font-size="12" font-weight="bold">receiver_idx</text>
<text x="315" y="150" fill="#8af8f8" text-anchor="middle" font-size="10">4 bytes LE</text>
<!-- Row 2: counter low (bytes 8-11) -->
<text x="50" y="186" fill="#666" font-size="10" text-anchor="end">8–11</text>
<rect x="55" y="160" width="520" height="52" fill="#5a2d3d" stroke="#d94a6a" stroke-width="1.5" rx="3"/>
<text x="315" y="190" fill="#e0e0e0" text-anchor="middle" font-size="12" font-weight="bold">counter / lower 4 bytes LE</text>
<!-- Row 3: counter high (bytes 12-15) -->
<text x="50" y="238" fill="#666" font-size="10" text-anchor="end">12–15</text>
<rect x="55" y="212" width="520" height="52" fill="#5a2d3d" stroke="#d94a6a" stroke-width="1.5" rx="3"/>
<text x="315" y="242" fill="#e0e0e0" text-anchor="middle" font-size="12" font-weight="bold">counter / upper 4 bytes LE</text>
</svg>
Side by Side

Before

Width:  |  Height:  |  Size: 3.2 KiB

docs/design/diagrams/filter-announce.svg
-43
View File
@@ -1,43 +0,0 @@
<svg xmlns="http://www.w3.org/2000/svg" viewBox="0 0 580 270" font-family="monospace" font-size="13">
<!-- Background -->
<rect width="580" height="270" fill="#1a1a2e" rx="4"/>
<!-- Title -->
<text x="310" y="26" fill="#e0e0e0" text-anchor="middle" font-size="14" font-weight="bold">FilterAnnounce (0x20) — 11 + filter bytes</text>
<!-- Row 0 (0–3): msg_type(1) + sequence starts -->
<text x="50" y="62" fill="#666" font-size="10" text-anchor="end">0–3</text>
<rect x="55" y="36" width="130" height="52" fill="#2d4a7a" stroke="#4a90d9" stroke-width="1.5" rx="3"/>
<text x="120" y="60" fill="#e0e0e0" text-anchor="middle" font-size="12" font-weight="bold">msg_type</text>
<text x="120" y="78" fill="#8ab4f8" text-anchor="middle" font-size="10">0x20</text>
<rect x="185" y="36" width="390" height="52" fill="#5a3d2d" stroke="#d9904a" stroke-width="1.5" rx="3"/>
<text x="380" y="66" fill="#e0e0e0" text-anchor="middle" font-size="12" font-weight="bold">sequence</text>
<!-- Row 1 (4–8): sequence continued -->
<text x="50" y="114" fill="#666" font-size="10" text-anchor="end">4–8</text>
<rect x="55" y="88" width="520" height="52" fill="#5a3d2d" stroke="#d9904a" stroke-width="1.5" rx="3"/>
<text x="315" y="118" fill="#f8c88a" text-anchor="middle" font-size="10">8 bytes LE — monotonic counter</text>
<!-- Row 2 (9–10): hash_count + size_class -->
<text x="50" y="166" fill="#666" font-size="10" text-anchor="end">9–10</text>
<rect x="55" y="140" width="260" height="52" fill="#2d5a4a" stroke="#4ad99a" stroke-width="1.5" rx="3"/>
<text x="185" y="162" fill="#e0e0e0" text-anchor="middle" font-size="12" font-weight="bold">hash_count</text>
<text x="185" y="180" fill="#8af8c8" text-anchor="middle" font-size="10">1 byte</text>
<rect x="315" y="140" width="260" height="52" fill="#4a2d5a" stroke="#b04ad9" stroke-width="1.5" rx="3"/>
<text x="445" y="162" fill="#e0e0e0" text-anchor="middle" font-size="12" font-weight="bold">size_class</text>
<text x="445" y="180" fill="#d8a0f8" text-anchor="middle" font-size="10">1 byte</text>
<!-- Row 3 (11–): filter_bits -->
<text x="50" y="218" fill="#666" font-size="10" text-anchor="end">11–</text>
<rect x="55" y="198" width="520" height="40" fill="#1f1f3a" stroke="#4ad9d9" stroke-width="1" stroke-dasharray="4,3" rx="3"/>
<text x="315" y="222" fill="#8af8f8" text-anchor="middle" font-size="11">filter_bits (variable, 512 &lt;&lt; size_class bytes)</text>
<!-- Total -->
<text x="310" y="258" fill="#777" font-size="10" text-anchor="middle">v1 payload: 1,035 bytes (11 header + 1,024 filter)</text>
</svg>
Side by Side

Before

Width:  |  Height:  |  Size: 2.6 KiB

docs/design/diagrams/fips-coordinate-discovery.svg
+30 -32
View File
@@ -82,7 +82,7 @@
<text x="610" y="223" text-anchor="middle" class="alabel" fill="#5080c0">LookupRequest</text>
<!-- Bloom filter annotation -->
<text x="430" y="260" text-anchor="middle" class="annot">guided by bloom filters at each hop</text>
<text x="430" y="260" text-anchor="middle" class="annot">guided by bloom filters at each hop; transits do not cache</text>
<!-- ═══ Phase separator ═══ -->
<line x1="70" y1="285" x2="830" y2="285" class="sep"/>
@@ -99,23 +99,19 @@
<polygon points="534,335 524,340 534,345" class="resp"/>
<text x="610" y="333" text-anchor="middle" class="alabel" fill="#d0a040">LookupResponse + coords</text>
<!-- Cache box at C -->
<rect x="490" y="358" width="60" height="22" class="cache"/>
<text x="520" y="373" text-anchor="middle" class="clabel">cache D</text>
<!-- C → B LookupResponse (transit forwards without caching) -->
<line x1="510" y1="380" x2="352" y2="380" class="resp"/>
<polygon points="354,375 344,380 354,385" class="resp"/>
<text x="430" y="373" text-anchor="middle" class="alabel" fill="#d0a040">LookupResponse + coords</text>
<!-- C → B LookupResponse -->
<line x1="510" y1="395" x2="352" y2="395" class="resp"/>
<polygon points="354,390 344,395 354,400" class="resp"/>
<text x="430" y="388" text-anchor="middle" class="alabel" fill="#d0a040">LookupResponse + coords</text>
<!-- B → A LookupResponse (transit forwards without caching) -->
<line x1="330" y1="420" x2="172" y2="420" class="resp"/>
<polygon points="174,415 164,420 174,425" class="resp"/>
<text x="250" y="413" text-anchor="middle" class="alabel" fill="#d0a040">LookupResponse + coords</text>
<!-- Cache box at B -->
<rect x="310" y="413" width="60" height="22" class="cache"/>
<text x="340" y="428" text-anchor="middle" class="clabel">cache D</text>
<!-- B → A LookupResponse -->
<line x1="330" y1="448" x2="172" y2="448" class="resp"/>
<polygon points="174,443 164,448 174,453" class="resp"/>
<text x="250" y="441" text-anchor="middle" class="alabel" fill="#d0a040">LookupResponse + coords</text>
<!-- Cache box at A — only the originator caches on LookupResponse -->
<rect x="130" y="438" width="60" height="22" class="cache"/>
<text x="160" y="453" text-anchor="middle" class="clabel">cache D</text>
<!-- ═══ Phase separator ═══ -->
<line x1="70" y1="475" x2="830" y2="475" class="sep"/>
@@ -125,32 +121,34 @@
<!-- ═══════════════════════════════════════════════ -->
<text x="40" y="508" text-anchor="middle" class="plabel">Phase 3</text>
<text x="40" y="524" text-anchor="middle" class="annot">Routing</text>
<text x="40" y="524" text-anchor="middle" class="annot">Data flow</text>
<!-- A → B Data -->
<!-- A → B Data with coords (CP flag / SessionSetup) -->
<line x1="170" y1="530" x2="328" y2="530" class="data"/>
<polygon points="326,525 336,530 326,535" class="data"/>
<text x="250" y="523" text-anchor="middle" class="alabel" fill="#40a060">Data</text>
<text x="250" y="523" text-anchor="middle" class="alabel" fill="#40a060">Data + coords</text>
<!-- Cached coords note at B -->
<text x="340" y="553" text-anchor="middle" class="annot">cached coords</text>
<!-- Cache box at B — warmed in-band from CP-flagged data -->
<rect x="310" y="540" width="60" height="22" class="cache"/>
<text x="340" y="555" text-anchor="middle" class="clabel">cache D</text>
<!-- B → C Data -->
<line x1="350" y1="565" x2="508" y2="565" class="data"/>
<polygon points="506,560 516,565 506,570" class="data"/>
<text x="430" y="558" text-anchor="middle" class="alabel" fill="#40a060">Data</text>
<!-- B → C Data with coords -->
<line x1="350" y1="572" x2="508" y2="572" class="data"/>
<polygon points="506,567 516,572 506,577" class="data"/>
<text x="430" y="565" text-anchor="middle" class="alabel" fill="#40a060">Data + coords</text>
<!-- Cached coords note at C -->
<text x="520" y="588" text-anchor="middle" class="annot">cached coords</text>
<!-- Cache box at C — warmed in-band -->
<rect x="490" y="582" width="60" height="22" class="cache"/>
<text x="520" y="597" text-anchor="middle" class="clabel">cache D</text>
<!-- C → D Data -->
<line x1="530" y1="600" x2="688" y2="600" class="data"/>
<polygon points="686,595 696,600 686,605" class="data"/>
<text x="610" y="593" text-anchor="middle" class="alabel" fill="#40a060">Data</text>
<line x1="530" y1="614" x2="688" y2="614" class="data"/>
<polygon points="686,609 696,614 686,619" class="data"/>
<text x="610" y="607" text-anchor="middle" class="alabel" fill="#40a060">Data + coords</text>
<!-- Efficient forwarding annotation -->
<text x="430" y="622" text-anchor="middle" class="annot">cached coords enable efficient forwarding — no re-discovery needed</text>
<text x="430" y="636" text-anchor="middle" class="annot">transits cache coords from in-flight data; subsequent traffic forwards without re-discovery</text>
<!-- ═══ Caption ═══ -->
<text x="430" y="660" text-anchor="middle" class="caption">Each transit node caches coordinates from the LookupResponse return path</text>
<text x="430" y="668" text-anchor="middle" class="caption">LookupResponse caches coords at the originator only; transit caches warm during the subsequent data flow</text>
</svg>
Side by Side Swipe Overlay

Before

Width:  |  Height:  |  Size: 7.8 KiB

After

Width:  |  Height:  |  Size: 8.1 KiB

docs/design/diagrams/fips-identity-derivation.svg
+2 -2
View File
@@ -71,7 +71,7 @@
<!-- Arrow down from pubkey -->
<line x1="400" y1="120" x2="400" y2="170" class="derive"/>
<polygon points="396,168 400,176 404,168" class="dhead"/>
<text x="416" y="148" class="op">one-way hash</text>
<text x="416" y="148" class="op">SHA-256, truncate to 16 bytes</text>
<!-- node_addr box -->
<rect x="280" y="176" width="240" height="50" class="box derived"/>
@@ -95,7 +95,7 @@
<!-- Arrow down from node_addr -->
<line x1="400" y1="226" x2="400" y2="276" class="derive"/>
<polygon points="396,274 400,282 404,274" class="dhead"/>
<text x="416" y="254" class="op">add fd00::/8 prefix</text>
<text x="416" y="254" class="op">0xfd + node_addr[0..15]</text>
<!-- IPv6 address box -->
<rect x="280" y="282" width="240" height="50" class="box compat"/>
Side by Side Swipe Overlay

Before

Width:  |  Height:  |  Size: 6.2 KiB

After

Width:  |  Height:  |  Size: 6.3 KiB

docs/design/diagrams/fips-protocol-stack.svg
+1 -1
View File
@@ -19,7 +19,7 @@
<rect x="50" y="25" width="775" height="100" class="layer app"/>
<text x="75" y="58" class="name">Application Layer Interface</text>
<text x="75" y="78" class="desc">Native FIPS API — for FIPS-aware applications</text>
<text x="75" y="98" class="desc">IPv6 Shim — for traditional IP application backward compatibility</text>
<text x="75" y="98" class="desc">IPv6 adapter — for traditional IP application backward compatibility</text>
<!-- FSP layer -->
<rect x="50" y="150" width="775" height="120" class="layer fsp"/>
Side by Side Swipe Overlay

Before

Width:  |  Height:  |  Size: 2.8 KiB

After

Width:  |  Height:  |  Size: 2.8 KiB

docs/design/diagrams/fips-routing-decision.svg
+26 -26
View File
@@ -76,54 +76,54 @@
<text x="372" y="364" class="branch">No</text>
<!-- ============================================================ -->
<!-- STEP 3: Bloom filter hit? -->
<!-- STEP 3: Coords known? -->
<!-- ============================================================ -->
<text x="240" y="395" text-anchor="end" class="step">3</text>
<polygon points="360,380 440,420 360,460 280,420" class="diamond"/>
<text x="360" y="416" text-anchor="middle" class="decision">Bloom filter</text>
<text x="360" y="430" text-anchor="middle" class="decision">hit?</text>
<text x="360" y="416" text-anchor="middle" class="decision">Coords</text>
<text x="360" y="430" text-anchor="middle" class="decision">known?</text>
<!-- Yes → right to interim step -->
<!-- No → right to error outcome -->
<line x1="440" y1="420" x2="520" y2="420" class="arrow" marker-end="url(#arrowhead)"/>
<text x="475" y="412" text-anchor="middle" class="branch">Yes</text>
<rect x="520" y="396" width="176" height="48" class="interim"/>
<text x="608" y="412" text-anchor="middle" class="action">Rank candidates by</text>
<text x="608" y="426" text-anchor="middle" class="action">tree distance and</text>
<text x="608" y="440" text-anchor="middle" class="action">link performance</text>
<text x="475" y="412" text-anchor="middle" class="branch">No</text>
<rect x="528" y="402" width="160" height="36" class="error"/>
<text x="608" y="425" text-anchor="middle" class="action">No route → error signal</text>
<!-- Arrow from interim → final outcome -->
<line x1="608" y1="444" x2="608" y2="470" class="arrow" marker-end="url(#arrowhead)"/>
<rect x="528" y="478" width="160" height="36" class="outcome"/>
<text x="608" y="501" text-anchor="middle" class="action">Forward to 'best'</text>
<!-- No → down -->
<!-- Yes → down -->
<line x1="360" y1="460" x2="360" y2="530" class="arrow" marker-end="url(#arrowhead)"/>
<text x="372" y="500" class="branch">No</text>
<text x="372" y="500" class="branch">Yes</text>
<!-- ============================================================ -->
<!-- STEP 4: Coords known? -->
<!-- STEP 4: Bloom filter hit? -->
<!-- ============================================================ -->
<text x="240" y="545" text-anchor="end" class="step">4</text>
<polygon points="360,530 440,570 360,610 280,570" class="diamond"/>
<text x="360" y="566" text-anchor="middle" class="decision">Coords</text>
<text x="360" y="580" text-anchor="middle" class="decision">known?</text>
<text x="360" y="566" text-anchor="middle" class="decision">Bloom filter</text>
<text x="360" y="580" text-anchor="middle" class="decision">hit?</text>
<!-- Yes → right to outcome -->
<!-- Yes → right to interim step -->
<line x1="440" y1="570" x2="520" y2="570" class="arrow" marker-end="url(#arrowhead)"/>
<text x="475" y="562" text-anchor="middle" class="branch">Yes</text>
<rect x="528" y="552" width="160" height="36" class="outcome"/>
<text x="608" y="575" text-anchor="middle" class="action">Greedy tree forward</text>
<rect x="520" y="546" width="176" height="48" class="interim"/>
<text x="608" y="562" text-anchor="middle" class="action">Rank candidates by</text>
<text x="608" y="576" text-anchor="middle" class="action">tree distance and</text>
<text x="608" y="590" text-anchor="middle" class="action">link performance</text>
<!-- Arrow from interim → final outcome -->
<line x1="608" y1="594" x2="608" y2="620" class="arrow" marker-end="url(#arrowhead)"/>
<rect x="528" y="628" width="160" height="36" class="outcome"/>
<text x="608" y="651" text-anchor="middle" class="action">Forward to 'best'</text>
<!-- No → down -->
<line x1="360" y1="610" x2="360" y2="660" class="arrow" marker-end="url(#arrowhead)"/>
<text x="372" y="640" class="branch">No</text>
<!-- ============================================================ -->
<!-- STEP 5: No route → error signal -->
<!-- STEP 5: Greedy tree forward -->
<!-- ============================================================ -->
<text x="240" y="683" text-anchor="end" class="step">5</text>
<rect x="260" y="668" width="200" height="36" class="error"/>
<text x="360" y="691" text-anchor="middle" class="action">No route → error signal</text>
<rect x="260" y="668" width="200" height="36" class="outcome"/>
<text x="360" y="691" text-anchor="middle" class="action">Greedy tree forward</text>
<!-- ============================================================ -->
<!-- Legend -->
@@ -149,5 +149,5 @@
<text x="442" y="835" font-size="11" fill="#a0a0b0">Control flow direction</text>
<!-- Caption -->
<text x="360" y="896" text-anchor="middle" class="caption">Each hop evaluates destinations in priority order 1–4, falling through on miss</text>
<text x="360" y="896" text-anchor="middle" class="caption">Each hop checks 1–4 in priority order, falling through to greedy tree (5) when bloom yields no candidate; missing coords is the only error path</text>
</svg>
Side by Side Swipe Overlay

Before

Width:  |  Height:  |  Size: 8.5 KiB

After

Width:  |  Height:  |  Size: 8.6 KiB

docs/design/diagrams/fsp-complete-message.svg
-97
View File
@@ -1,97 +0,0 @@
<svg xmlns="http://www.w3.org/2000/svg" viewBox="0 0 580 524" font-family="monospace" font-size="13">
<!-- Background -->
<rect width="580" height="524" fill="#1a1a2e" rx="4"/>
<!-- Title -->
<text x="310" y="26" fill="#e0e0e0" text-anchor="middle" font-size="14" font-weight="bold">FSP Complete Encrypted Message (34 + body bytes)</text>
<!-- Bit ruler -->
<text x="85" y="50" fill="#888" font-size="10" text-anchor="middle">0</text>
<text x="150" y="50" fill="#888" font-size="10" text-anchor="middle">4</text>
<text x="215" y="50" fill="#888" font-size="10" text-anchor="middle">8</text>
<text x="345" y="50" fill="#888" font-size="10" text-anchor="middle">16</text>
<text x="550" y="50" fill="#888" font-size="10" text-anchor="middle">32</text>
<!-- Row 0: Common prefix (bytes 0-3) -->
<text x="50" y="82" fill="#666" font-size="10" text-anchor="end">0–3</text>
<rect x="55" y="56" width="95" height="52" fill="#2d4a7a" stroke="#4a90d9" stroke-width="1.5" rx="3"/>
<text x="102" y="80" fill="#e0e0e0" text-anchor="middle" font-size="12" font-weight="bold">ver</text>
<text x="102" y="98" fill="#8ab4f8" text-anchor="middle" font-size="10">4 bits</text>
<rect x="150" y="56" width="95" height="52" fill="#2d5a4a" stroke="#4ad99a" stroke-width="1.5" rx="3"/>
<text x="197" y="80" fill="#e0e0e0" text-anchor="middle" font-size="12" font-weight="bold">phase</text>
<text x="197" y="98" fill="#8af8c8" text-anchor="middle" font-size="10">4 bits</text>
<rect x="245" y="56" width="130" height="52" fill="#5a3d2d" stroke="#d9904a" stroke-width="1.5" rx="3"/>
<text x="310" y="80" fill="#e0e0e0" text-anchor="middle" font-size="12" font-weight="bold">flags</text>
<text x="310" y="98" fill="#f8c88a" text-anchor="middle" font-size="10">1 byte</text>
<rect x="375" y="56" width="200" height="52" fill="#4a2d5a" stroke="#b04ad9" stroke-width="1.5" rx="3"/>
<text x="475" y="80" fill="#e0e0e0" text-anchor="middle" font-size="12" font-weight="bold">payload_len</text>
<text x="475" y="98" fill="#d8a0f8" text-anchor="middle" font-size="10">2 bytes LE</text>
<!-- Row 1: counter low (bytes 4-7) -->
<text x="50" y="134" fill="#666" font-size="10" text-anchor="end">4–7</text>
<rect x="55" y="108" width="520" height="52" fill="#5a2d3d" stroke="#d94a6a" stroke-width="1.5" rx="3"/>
<text x="315" y="138" fill="#e0e0e0" text-anchor="middle" font-size="12" font-weight="bold">counter / lower 4 bytes LE</text>
<!-- Row 2: counter high (bytes 8-11) -->
<text x="50" y="186" fill="#666" font-size="10" text-anchor="end">8–11</text>
<rect x="55" y="160" width="520" height="52" fill="#5a2d3d" stroke="#d94a6a" stroke-width="1.5" rx="3"/>
<text x="315" y="190" fill="#e0e0e0" text-anchor="middle" font-size="12" font-weight="bold">counter / upper 4 bytes LE</text>
<!-- AAD bracket -->
<text x="10" y="134" fill="#777" font-size="10" text-anchor="middle" transform="rotate(-90, 10, 134)">AAD</text>
<line x1="18" y1="56" x2="18" y2="212" stroke="#555" stroke-width="1"/>
<line x1="18" y1="56" x2="23" y2="56" stroke="#555" stroke-width="1"/>
<line x1="18" y1="212" x2="23" y2="212" stroke="#555" stroke-width="1"/>
<!-- Optional coords -->
<text x="50" y="238" fill="#666" font-size="10" text-anchor="end">opt</text>
<rect x="55" y="220" width="520" height="40" fill="#1f1f3a" stroke="#4ad99a" stroke-width="1" stroke-dasharray="4,3" rx="3"/>
<text x="315" y="244" fill="#8af8c8" text-anchor="middle" font-size="11">cleartext coordinates (if CP flag set, variable length)</text>
<!-- Separator: ciphertext below -->
<line x1="55" y1="268" x2="575" y2="268" stroke="#555" stroke-width="1" stroke-dasharray="6,4"/>
<text x="315" y="284" fill="#777" font-size="10" text-anchor="middle">── ciphertext (encrypted with ChaCha20-Poly1305) ──</text>
<!-- Row 3: timestamp (4 bytes, encrypted) -->
<text x="50" y="318" fill="#666" font-size="10" text-anchor="end">+0–3</text>
<rect x="55" y="292" width="520" height="52" fill="#3d4a2d" stroke="#8ad94a" stroke-width="1.5" rx="3"/>
<text x="315" y="316" fill="#e0e0e0" text-anchor="middle" font-size="12" font-weight="bold">timestamp</text>
<text x="315" y="334" fill="#c8f88a" text-anchor="middle" font-size="10">4 bytes LE</text>
<!-- Row 4: msg_type (1) + inner_flags (1) + body starts -->
<text x="50" y="370" fill="#666" font-size="10" text-anchor="end">+4</text>
<rect x="55" y="344" width="130" height="52" fill="#4a3d2d" stroke="#d9b04a" stroke-width="1.5" rx="3"/>
<text x="120" y="368" fill="#e0e0e0" text-anchor="middle" font-size="12" font-weight="bold">msg_type</text>
<text x="120" y="386" fill="#f8d88a" text-anchor="middle" font-size="10">1 byte</text>
<rect x="185" y="344" width="130" height="52" fill="#3d3d2d" stroke="#a0a04a" stroke-width="1.5" rx="3"/>
<text x="250" y="368" fill="#e0e0e0" text-anchor="middle" font-size="12" font-weight="bold">inner_flags</text>
<text x="250" y="386" fill="#d8d88a" text-anchor="middle" font-size="10">1 byte</text>
<rect x="315" y="344" width="260" height="52" fill="#1f1f3a" stroke="#444" stroke-width="1" stroke-dasharray="4,3" rx="3"/>
<text x="445" y="374" fill="#666" text-anchor="middle" font-size="11">body ...</text>
<!-- Row 5: body variable -->
<text x="50" y="422" fill="#666" font-size="10" text-anchor="end">...</text>
<rect x="55" y="396" width="520" height="52" fill="#1f1f3a" stroke="#444" stroke-width="1" stroke-dasharray="4,3" rx="3"/>
<text x="315" y="426" fill="#666" text-anchor="middle" font-size="11">message body (variable length)</text>
<!-- AEAD tag -->
<text x="50" y="468" fill="#666" font-size="10" text-anchor="end">+16</text>
<rect x="55" y="452" width="520" height="30" fill="#4a2d4a" stroke="#b04ab0" stroke-width="1.5" rx="3"/>
<text x="315" y="472" fill="#e0e0e0" text-anchor="middle" font-size="11" font-weight="bold">Poly1305 AEAD tag (16 bytes)</text>
<!-- Totals -->
<text x="310" y="510" fill="#777" font-size="10" text-anchor="middle">overhead: 34 bytes (12 header + 6 inner + 16 tag) · coords add variable length</text>
</svg>
Side by Side

Before

Width:  |  Height:  |  Size: 6.0 KiB

docs/design/diagrams/handshake-flow.svg
-66
View File
@@ -1,66 +0,0 @@
<svg xmlns="http://www.w3.org/2000/svg" viewBox="0 0 620 500" font-family="monospace" font-size="13">
<defs>
<marker id="arrowR" markerWidth="10" markerHeight="8" refX="9" refY="4" orient="auto" markerUnits="userSpaceOnUse">
<polygon points="0,0 10,4 0,8" fill="#e0e0e0"/>
</marker>
<marker id="arrowL" markerWidth="10" markerHeight="8" refX="1" refY="4" orient="auto" markerUnits="userSpaceOnUse">
<polygon points="10,0 0,4 10,8" fill="#e0e0e0"/>
</marker>
</defs>
<!-- Background -->
<rect width="620" height="500" fill="#1a1a2e" rx="4"/>
<!-- Title -->
<text x="310" y="26" fill="#e0e0e0" text-anchor="middle" font-size="14" font-weight="bold">FMP Handshake Flow (Noise IK)</text>
<!-- Column headers -->
<text x="100" y="54" fill="#8ab4f8" text-anchor="middle" font-size="12" font-weight="bold">Initiator</text>
<text x="520" y="54" fill="#8ab4f8" text-anchor="middle" font-size="12" font-weight="bold">Responder</text>
<!-- Lifelines -->
<line x1="100" y1="62" x2="100" y2="480" stroke="#333" stroke-width="1" stroke-dasharray="4,4"/>
<line x1="520" y1="62" x2="520" y2="480" stroke="#333" stroke-width="1" stroke-dasharray="4,4"/>
<!-- Step 1: Initiator prepares -->
<text x="20" y="88" fill="#8af8c8" font-size="10">generates sender_idx</text>
<text x="20" y="102" fill="#8af8c8" font-size="10">generates ephemeral keypair</text>
<!-- Arrow 1: msg1 (initiator -> responder) -->
<line x1="100" y1="120" x2="520" y2="120" stroke="#4a90d9" stroke-width="1.5" marker-end="url(#arrowR)"/>
<rect x="130" y="128" width="360" height="24" fill="#2d4a7a" stroke="#4a90d9" stroke-width="1" rx="3"/>
<text x="310" y="144" fill="#e0e0e0" text-anchor="middle" font-size="10">[0x01|flags=0|len] | sender_idx | noise_msg1</text>
<text x="310" y="166" fill="#666" text-anchor="middle" font-size="9">phase 0x1 — 114 bytes</text>
<!-- Step 2: Responder processes (right-aligned to stay in bounds) -->
<text x="600" y="192" fill="#8af8c8" font-size="10" text-anchor="end">validates msg1</text>
<text x="600" y="206" fill="#8af8c8" font-size="10" text-anchor="end">learns initiator's static key</text>
<text x="600" y="220" fill="#8af8c8" font-size="10" text-anchor="end">generates sender_idx</text>
<text x="600" y="234" fill="#8af8c8" font-size="10" text-anchor="end">generates ephemeral keypair</text>
<!-- Arrow 2: msg2 (responder -> initiator) -->
<!-- Draw line right-to-left so orient="auto" points the arrowhead left -->
<line x1="520" y1="256" x2="100" y2="256" stroke="#4ad99a" stroke-width="1.5" marker-end="url(#arrowR)"/>
<rect x="130" y="264" width="360" height="24" fill="#2d5a4a" stroke="#4ad99a" stroke-width="1" rx="3"/>
<text x="310" y="280" fill="#e0e0e0" text-anchor="middle" font-size="9">[0x02|flags=0|len] | sender_idx | receiver_idx | noise_msg2</text>
<text x="310" y="300" fill="#666" text-anchor="middle" font-size="9">phase 0x2 — 69 bytes</text>
<!-- Step 3: Initiator completes -->
<text x="20" y="324" fill="#8af8c8" font-size="10">validates msg2</text>
<text x="20" y="338" fill="#8af8c8" font-size="10">derives session keys</text>
<!-- Handshake complete separator -->
<line x1="30" y1="360" x2="230" y2="360" stroke="#d9904a" stroke-width="1.5"/>
<text x="310" y="364" fill="#d9904a" text-anchor="middle" font-size="11" font-weight="bold">HANDSHAKE COMPLETE</text>
<line x1="390" y1="360" x2="590" y2="360" stroke="#d9904a" stroke-width="1.5"/>
<!-- Arrow 3: first encrypted frame -->
<text x="20" y="392" fill="#777" font-size="10">first encrypted frame:</text>
<line x1="100" y1="406" x2="520" y2="406" stroke="#d94a6a" stroke-width="1.5" marker-end="url(#arrowR)"/>
<rect x="115" y="414" width="390" height="24" fill="#5a2d3d" stroke="#d94a6a" stroke-width="1" rx="3"/>
<text x="310" y="430" fill="#e0e0e0" text-anchor="middle" font-size="9">[0x00|flags|len] | receiver_idx | counter=0 | ciphertext+tag</text>
<text x="310" y="450" fill="#666" text-anchor="middle" font-size="9">phase 0x0 — established frame</text>
<!-- Legend -->
<text x="310" y="478" fill="#555" text-anchor="middle" font-size="9">Both parties hold identical symmetric keys. Epoch exchange enables restart detection.</text>
</svg>
Side by Side

Before

Width:  |  Height:  |  Size: 4.2 KiB

docs/design/diagrams/lookup-request.svg
-63
View File
@@ -1,63 +0,0 @@
<svg xmlns="http://www.w3.org/2000/svg" viewBox="0 0 580 430" font-family="monospace" font-size="13">
<!-- Background -->
<rect width="580" height="430" fill="#1a1a2e" rx="4"/>
<!-- Title -->
<text x="310" y="26" fill="#e0e0e0" text-anchor="middle" font-size="14" font-weight="bold">LookupRequest (0x30) — 46 + 16n bytes</text>
<!-- Row 0 (0–3): msg_type + request_id starts -->
<text x="50" y="62" fill="#666" font-size="10" text-anchor="end">0–3</text>
<rect x="55" y="36" width="130" height="52" fill="#2d4a7a" stroke="#4a90d9" stroke-width="1.5" rx="3"/>
<text x="120" y="60" fill="#e0e0e0" text-anchor="middle" font-size="12" font-weight="bold">msg_type</text>
<text x="120" y="78" fill="#8ab4f8" text-anchor="middle" font-size="10">0x30</text>
<rect x="185" y="36" width="390" height="52" fill="#5a3d2d" stroke="#d9904a" stroke-width="1.5" rx="3"/>
<text x="380" y="66" fill="#e0e0e0" text-anchor="middle" font-size="12" font-weight="bold">request_id</text>
<!-- Row 1 (4–8): request_id continued -->
<text x="50" y="114" fill="#666" font-size="10" text-anchor="end">4–8</text>
<rect x="55" y="88" width="520" height="52" fill="#5a3d2d" stroke="#d9904a" stroke-width="1.5" rx="3"/>
<text x="315" y="118" fill="#f8c88a" text-anchor="middle" font-size="10">8 bytes LE — unique random identifier</text>
<!-- Row 2 (9–24): target -->
<text x="50" y="166" fill="#666" font-size="10" text-anchor="end">9–24</text>
<rect x="55" y="140" width="520" height="52" fill="#2d5a5a" stroke="#4ad9d9" stroke-width="1.5" rx="3"/>
<text x="315" y="162" fill="#e0e0e0" text-anchor="middle" font-size="12" font-weight="bold">target</text>
<text x="315" y="180" fill="#8af8f8" text-anchor="middle" font-size="10">16 bytes — NodeAddr being sought</text>
<!-- Row 3 (25–40): origin -->
<text x="50" y="218" fill="#666" font-size="10" text-anchor="end">25–40</text>
<rect x="55" y="192" width="520" height="52" fill="#4a2d5a" stroke="#b04ad9" stroke-width="1.5" rx="3"/>
<text x="315" y="214" fill="#e0e0e0" text-anchor="middle" font-size="12" font-weight="bold">origin</text>
<text x="315" y="232" fill="#d8a0f8" text-anchor="middle" font-size="10">16 bytes — requester's NodeAddr</text>
<!-- Row 4 (41–43): ttl + min_mtu -->
<text x="50" y="270" fill="#666" font-size="10" text-anchor="end">41–43</text>
<rect x="55" y="244" width="130" height="52" fill="#2d5a4a" stroke="#4ad99a" stroke-width="1.5" rx="3"/>
<text x="120" y="268" fill="#e0e0e0" text-anchor="middle" font-size="12" font-weight="bold">ttl</text>
<text x="120" y="286" fill="#8af8c8" text-anchor="middle" font-size="10">1 byte</text>
<rect x="185" y="244" width="390" height="52" fill="#5a2d3d" stroke="#d94a6a" stroke-width="1.5" rx="3"/>
<text x="380" y="268" fill="#e0e0e0" text-anchor="middle" font-size="12" font-weight="bold">min_mtu</text>
<text x="380" y="286" fill="#f88aaa" text-anchor="middle" font-size="10">2 bytes LE</text>
<!-- Row 5 (44–45): origin_coords_cnt -->
<text x="50" y="322" fill="#666" font-size="10" text-anchor="end">44–45</text>
<rect x="55" y="296" width="260" height="52" fill="#4a3d2d" stroke="#d9b04a" stroke-width="1.5" rx="3"/>
<text x="185" y="326" fill="#e0e0e0" text-anchor="middle" font-size="12" font-weight="bold">origin_coords_cnt</text>
<!-- Row 6 (46–): origin_coords -->
<text x="50" y="370" fill="#666" font-size="10" text-anchor="end">46–</text>
<rect x="55" y="348" width="520" height="40" fill="#1f1f3a" stroke="#4ad9d9" stroke-width="1" stroke-dasharray="4,3" rx="3"/>
<text x="315" y="372" fill="#8af8f8" text-anchor="middle" font-size="11">origin_coords × n (16 bytes each — NodeAddr only)</text>
<!-- Total -->
<text x="310" y="414" fill="#777" font-size="10" text-anchor="middle">total: 46 + (n × 16) bytes, where n = origin depth + 1</text>
</svg>
Side by Side

Before

Width:  |  Height:  |  Size: 3.8 KiB

docs/design/diagrams/lookup-response.svg
-56
View File
@@ -1,56 +0,0 @@
<svg xmlns="http://www.w3.org/2000/svg" viewBox="0 0 580 374" font-family="monospace" font-size="13">
<!-- Background -->
<rect width="580" height="374" fill="#1a1a2e" rx="4"/>
<!-- Title -->
<text x="310" y="26" fill="#e0e0e0" text-anchor="middle" font-size="14" font-weight="bold">LookupResponse (0x31) — 93 + 16n bytes</text>
<!-- Row 0 (0–3): msg_type + request_id starts -->
<text x="50" y="62" fill="#666" font-size="10" text-anchor="end">0–3</text>
<rect x="55" y="36" width="130" height="52" fill="#2d4a7a" stroke="#4a90d9" stroke-width="1.5" rx="3"/>
<text x="120" y="60" fill="#e0e0e0" text-anchor="middle" font-size="12" font-weight="bold">msg_type</text>
<text x="120" y="78" fill="#8ab4f8" text-anchor="middle" font-size="10">0x31</text>
<rect x="185" y="36" width="390" height="52" fill="#5a3d2d" stroke="#d9904a" stroke-width="1.5" rx="3"/>
<text x="380" y="66" fill="#e0e0e0" text-anchor="middle" font-size="12" font-weight="bold">request_id</text>
<!-- Row 1 (4–8): request_id continued -->
<text x="50" y="114" fill="#666" font-size="10" text-anchor="end">4–8</text>
<rect x="55" y="88" width="520" height="52" fill="#5a3d2d" stroke="#d9904a" stroke-width="1.5" rx="3"/>
<text x="315" y="118" fill="#f8c88a" text-anchor="middle" font-size="10">8 bytes LE — echoes the request's ID</text>
<!-- Row 2 (9–24): target -->
<text x="50" y="166" fill="#666" font-size="10" text-anchor="end">9–24</text>
<rect x="55" y="140" width="520" height="52" fill="#2d5a5a" stroke="#4ad9d9" stroke-width="1.5" rx="3"/>
<text x="315" y="162" fill="#e0e0e0" text-anchor="middle" font-size="12" font-weight="bold">target</text>
<text x="315" y="180" fill="#8af8f8" text-anchor="middle" font-size="10">16 bytes — NodeAddr that was found</text>
<!-- Row 3 (25–28): path_mtu + target_coords_cnt -->
<text x="50" y="218" fill="#666" font-size="10" text-anchor="end">25–28</text>
<rect x="55" y="192" width="260" height="52" fill="#5a2d3d" stroke="#d94a6a" stroke-width="1.5" rx="3"/>
<text x="185" y="214" fill="#e0e0e0" text-anchor="middle" font-size="12" font-weight="bold">path_mtu</text>
<text x="185" y="232" fill="#f88aaa" text-anchor="middle" font-size="10">2 bytes LE</text>
<rect x="315" y="192" width="260" height="52" fill="#4a3d2d" stroke="#d9b04a" stroke-width="1.5" rx="3"/>
<text x="445" y="214" fill="#e0e0e0" text-anchor="middle" font-size="12" font-weight="bold">target_coords_cnt</text>
<text x="445" y="232" fill="#f8d88a" text-anchor="middle" font-size="10">2 bytes LE</text>
<!-- Row 4 (29–): target_coords -->
<text x="50" y="270" fill="#666" font-size="10" text-anchor="end">29–</text>
<rect x="55" y="250" width="520" height="40" fill="#1f1f3a" stroke="#4ad9d9" stroke-width="1" stroke-dasharray="4,3" rx="3"/>
<text x="315" y="274" fill="#8af8f8" text-anchor="middle" font-size="11">target_coords × n (16 bytes each — NodeAddr only)</text>
<!-- Row 5: proof -->
<text x="50" y="316" fill="#666" font-size="10" text-anchor="end">+64</text>
<rect x="55" y="296" width="520" height="40" fill="#3d4a2d" stroke="#8ad94a" stroke-width="1.5" rx="3"/>
<text x="315" y="320" fill="#e0e0e0" text-anchor="middle" font-size="11" font-weight="bold">Schnorr proof (64 bytes)</text>
<!-- Total -->
<text x="310" y="358" fill="#777" font-size="10" text-anchor="middle">total: 93 + (n × 16) bytes · proof signs (request_id || target || target_coords)</text>
</svg>
Side by Side

Before

Width:  |  Height:  |  Size: 3.4 KiB

docs/design/diagrams/mtu-exceeded.svg
-41
View File
@@ -1,41 +0,0 @@
<svg xmlns="http://www.w3.org/2000/svg" viewBox="0 0 580 270" font-family="monospace" font-size="13">
<!-- Background -->
<rect width="580" height="270" fill="#1a1a2e" rx="4"/>
<!-- Title -->
<text x="310" y="26" fill="#e0e0e0" text-anchor="middle" font-size="14" font-weight="bold">MtuExceeded (0x22) — 36 bytes payload</text>
<!-- Row 0 (0–1): msg_type + flags -->
<text x="50" y="62" fill="#666" font-size="10" text-anchor="end">0–1</text>
<rect x="55" y="36" width="260" height="52" fill="#2d4a7a" stroke="#4a90d9" stroke-width="1.5" rx="3"/>
<text x="185" y="58" fill="#e0e0e0" text-anchor="middle" font-size="12" font-weight="bold">msg_type</text>
<text x="185" y="76" fill="#8ab4f8" text-anchor="middle" font-size="10">0x22</text>
<rect x="315" y="36" width="260" height="52" fill="#1f1f3a" stroke="#444" stroke-width="1" rx="3"/>
<text x="445" y="62" fill="#666" text-anchor="middle" font-size="12">flags (reserved)</text>
<!-- Row 1 (2–17): dest_addr -->
<text x="50" y="114" fill="#666" font-size="10" text-anchor="end">2–17</text>
<rect x="55" y="88" width="520" height="52" fill="#4a2d5a" stroke="#b04ad9" stroke-width="1.5" rx="3"/>
<text x="315" y="110" fill="#e0e0e0" text-anchor="middle" font-size="12" font-weight="bold">dest_addr</text>
<text x="315" y="128" fill="#d8a0f8" text-anchor="middle" font-size="10">16 bytes — destination being forwarded to</text>
<!-- Row 2 (18–33): reporter -->
<text x="50" y="166" fill="#666" font-size="10" text-anchor="end">18–33</text>
<rect x="55" y="140" width="520" height="52" fill="#2d5a5a" stroke="#4ad9d9" stroke-width="1.5" rx="3"/>
<text x="315" y="162" fill="#e0e0e0" text-anchor="middle" font-size="12" font-weight="bold">reporter</text>
<text x="315" y="180" fill="#8af8f8" text-anchor="middle" font-size="10">16 bytes — NodeAddr of reporting router</text>
<!-- Row 3 (34–35): mtu -->
<text x="50" y="218" fill="#666" font-size="10" text-anchor="end">34–35</text>
<rect x="55" y="192" width="260" height="52" fill="#5a2d3d" stroke="#d94a6a" stroke-width="1.5" rx="3"/>
<text x="185" y="214" fill="#e0e0e0" text-anchor="middle" font-size="12" font-weight="bold">mtu</text>
<text x="185" y="232" fill="#f88aaa" text-anchor="middle" font-size="10">2 bytes LE — bottleneck MTU</text>
<!-- Note -->
<text x="310" y="264" fill="#777" font-size="10" text-anchor="middle">plaintext error signal (U flag set) · total on wire: 40 bytes (4 prefix + 36 payload)</text>
</svg>
Side by Side

Before

Width:  |  Height:  |  Size: 2.5 KiB

docs/design/diagrams/noise-ik-msg1.svg
-67
View File
@@ -1,67 +0,0 @@
<svg xmlns="http://www.w3.org/2000/svg" viewBox="0 0 580 426" font-family="monospace" font-size="13">
<!-- Background -->
<rect width="580" height="426" fill="#1a1a2e" rx="4"/>
<!-- Title -->
<text x="310" y="26" fill="#e0e0e0" text-anchor="middle" font-size="14" font-weight="bold">Noise IK Message 1 — phase 0x1 (114 bytes)</text>
<!-- Row 0: Common prefix (bytes 0-3) -->
<text x="50" y="62" fill="#666" font-size="10" text-anchor="end">0–3</text>
<rect x="55" y="36" width="95" height="52" fill="#2d4a7a" stroke="#4a90d9" stroke-width="1.5" rx="3"/>
<text x="102" y="60" fill="#e0e0e0" text-anchor="middle" font-size="12" font-weight="bold">ver</text>
<text x="102" y="78" fill="#8ab4f8" text-anchor="middle" font-size="10">4 bits</text>
<rect x="150" y="36" width="95" height="52" fill="#2d5a4a" stroke="#4ad99a" stroke-width="1.5" rx="3"/>
<text x="197" y="60" fill="#e0e0e0" text-anchor="middle" font-size="12" font-weight="bold">phase</text>
<text x="197" y="78" fill="#8af8c8" text-anchor="middle" font-size="10">4 bits</text>
<rect x="245" y="36" width="130" height="52" fill="#5a3d2d" stroke="#d9904a" stroke-width="1.5" rx="3"/>
<text x="310" y="60" fill="#e0e0e0" text-anchor="middle" font-size="12" font-weight="bold">flags</text>
<text x="310" y="78" fill="#f8c88a" text-anchor="middle" font-size="10">1 byte</text>
<rect x="375" y="36" width="200" height="52" fill="#4a2d5a" stroke="#b04ad9" stroke-width="1.5" rx="3"/>
<text x="475" y="60" fill="#e0e0e0" text-anchor="middle" font-size="12" font-weight="bold">payload_len</text>
<text x="475" y="78" fill="#d8a0f8" text-anchor="middle" font-size="10">2 bytes LE</text>
<!-- Row 1: sender_idx (bytes 4-7) -->
<text x="50" y="114" fill="#666" font-size="10" text-anchor="end">4–7</text>
<rect x="55" y="88" width="520" height="52" fill="#2d5a5a" stroke="#4ad9d9" stroke-width="1.5" rx="3"/>
<text x="315" y="112" fill="#e0e0e0" text-anchor="middle" font-size="12" font-weight="bold">sender_idx</text>
<text x="315" y="130" fill="#8af8f8" text-anchor="middle" font-size="10">4 bytes LE</text>
<!-- Noise IK msg1 label -->
<text x="10" y="250" fill="#777" font-size="10" text-anchor="middle" transform="rotate(-90, 10, 250)">Noise IK msg1 (106 bytes)</text>
<line x1="18" y1="140" x2="18" y2="348" stroke="#555" stroke-width="1"/>
<line x1="18" y1="140" x2="23" y2="140" stroke="#555" stroke-width="1"/>
<line x1="18" y1="348" x2="23" y2="348" stroke="#555" stroke-width="1"/>
<!-- Rows 2-3: ephemeral_pubkey (bytes 8-40, 33 bytes) -->
<text x="50" y="166" fill="#666" font-size="10" text-anchor="end">8–40</text>
<rect x="55" y="140" width="520" height="52" fill="#5a3d2d" stroke="#d9904a" stroke-width="1.5" rx="3"/>
<text x="315" y="170" fill="#e0e0e0" text-anchor="middle" font-size="12" font-weight="bold">ephemeral_pubkey</text>
<rect x="55" y="192" width="520" height="52" fill="#5a3d2d" stroke="#d9904a" stroke-width="1.5" rx="3"/>
<text x="315" y="222" fill="#f8c88a" text-anchor="middle" font-size="10">33 bytes (compressed secp256k1)</text>
<!-- Rows 4-5: encrypted_static (bytes 41-89, 49 bytes) -->
<text x="50" y="270" fill="#666" font-size="10" text-anchor="end">41–89</text>
<rect x="55" y="244" width="520" height="52" fill="#5a2d3d" stroke="#d94a6a" stroke-width="1.5" rx="3"/>
<text x="315" y="274" fill="#e0e0e0" text-anchor="middle" font-size="12" font-weight="bold">encrypted_static</text>
<rect x="55" y="296" width="520" height="52" fill="#5a2d3d" stroke="#d94a6a" stroke-width="1.5" rx="3"/>
<text x="315" y="326" fill="#f88aaa" text-anchor="middle" font-size="10">49 bytes (static key 33 + AEAD tag 16)</text>
<!-- Row 6: encrypted_epoch (bytes 90-113, 24 bytes) -->
<text x="50" y="374" fill="#666" font-size="10" text-anchor="end">90–113</text>
<rect x="55" y="348" width="520" height="52" fill="#4a2d5a" stroke="#b04ad9" stroke-width="1.5" rx="3"/>
<text x="315" y="372" fill="#e0e0e0" text-anchor="middle" font-size="12" font-weight="bold">encrypted_epoch</text>
<text x="315" y="390" fill="#d8a0f8" text-anchor="middle" font-size="10">24 bytes (epoch 8 + AEAD tag 16)</text>
<!-- Total -->
<text x="310" y="418" fill="#777" font-size="10" text-anchor="middle">total: 114 bytes · payload_len = 110 · pattern: e, es, s, ss</text>
</svg>
Side by Side

Before

Width:  |  Height:  |  Size: 4.3 KiB

docs/design/diagrams/noise-ik-msg2.svg
-65
View File
@@ -1,65 +0,0 @@
<svg xmlns="http://www.w3.org/2000/svg" viewBox="0 0 580 374" font-family="monospace" font-size="13">
<!-- Background -->
<rect width="580" height="374" fill="#1a1a2e" rx="4"/>
<!-- Title -->
<text x="310" y="26" fill="#e0e0e0" text-anchor="middle" font-size="14" font-weight="bold">Noise IK Message 2 — phase 0x2 (69 bytes)</text>
<!-- Row 0: Common prefix (bytes 0-3) -->
<text x="50" y="62" fill="#666" font-size="10" text-anchor="end">0–3</text>
<rect x="55" y="36" width="95" height="52" fill="#2d4a7a" stroke="#4a90d9" stroke-width="1.5" rx="3"/>
<text x="102" y="60" fill="#e0e0e0" text-anchor="middle" font-size="12" font-weight="bold">ver</text>
<text x="102" y="78" fill="#8ab4f8" text-anchor="middle" font-size="10">4 bits</text>
<rect x="150" y="36" width="95" height="52" fill="#2d5a4a" stroke="#4ad99a" stroke-width="1.5" rx="3"/>
<text x="197" y="60" fill="#e0e0e0" text-anchor="middle" font-size="12" font-weight="bold">phase</text>
<text x="197" y="78" fill="#8af8c8" text-anchor="middle" font-size="10">4 bits</text>
<rect x="245" y="36" width="130" height="52" fill="#5a3d2d" stroke="#d9904a" stroke-width="1.5" rx="3"/>
<text x="310" y="60" fill="#e0e0e0" text-anchor="middle" font-size="12" font-weight="bold">flags</text>
<text x="310" y="78" fill="#f8c88a" text-anchor="middle" font-size="10">1 byte</text>
<rect x="375" y="36" width="200" height="52" fill="#4a2d5a" stroke="#b04ad9" stroke-width="1.5" rx="3"/>
<text x="475" y="60" fill="#e0e0e0" text-anchor="middle" font-size="12" font-weight="bold">payload_len</text>
<text x="475" y="78" fill="#d8a0f8" text-anchor="middle" font-size="10">2 bytes LE</text>
<!-- Row 1: sender_idx (bytes 4-7) -->
<text x="50" y="114" fill="#666" font-size="10" text-anchor="end">4–7</text>
<rect x="55" y="88" width="520" height="52" fill="#2d5a5a" stroke="#4ad9d9" stroke-width="1.5" rx="3"/>
<text x="315" y="112" fill="#e0e0e0" text-anchor="middle" font-size="12" font-weight="bold">sender_idx</text>
<text x="315" y="130" fill="#8af8f8" text-anchor="middle" font-size="10">4 bytes LE</text>
<!-- Row 2: receiver_idx (bytes 8-11) -->
<text x="50" y="166" fill="#666" font-size="10" text-anchor="end">8–11</text>
<rect x="55" y="140" width="520" height="52" fill="#2d5a4a" stroke="#4ad99a" stroke-width="1.5" rx="3"/>
<text x="315" y="164" fill="#e0e0e0" text-anchor="middle" font-size="12" font-weight="bold">receiver_idx</text>
<text x="315" y="182" fill="#8af8c8" text-anchor="middle" font-size="10">4 bytes LE</text>
<!-- Noise IK msg2 bracket -->
<text x="10" y="270" fill="#777" font-size="10" text-anchor="middle" transform="rotate(-90, 10, 270)">Noise IK msg2 (57 bytes)</text>
<line x1="18" y1="192" x2="18" y2="348" stroke="#555" stroke-width="1"/>
<line x1="18" y1="192" x2="23" y2="192" stroke="#555" stroke-width="1"/>
<line x1="18" y1="348" x2="23" y2="348" stroke="#555" stroke-width="1"/>
<!-- Rows 3-4: ephemeral_pubkey (bytes 12-44, 33 bytes) -->
<text x="50" y="218" fill="#666" font-size="10" text-anchor="end">12–44</text>
<rect x="55" y="192" width="520" height="52" fill="#5a3d2d" stroke="#d9904a" stroke-width="1.5" rx="3"/>
<text x="315" y="222" fill="#e0e0e0" text-anchor="middle" font-size="12" font-weight="bold">ephemeral_pubkey</text>
<rect x="55" y="244" width="520" height="52" fill="#5a3d2d" stroke="#d9904a" stroke-width="1.5" rx="3"/>
<text x="315" y="274" fill="#f8c88a" text-anchor="middle" font-size="10">33 bytes (compressed secp256k1)</text>
<!-- Row 5: encrypted_epoch (bytes 45-68, 24 bytes) -->
<text x="50" y="322" fill="#666" font-size="10" text-anchor="end">45–68</text>
<rect x="55" y="296" width="520" height="52" fill="#4a2d5a" stroke="#b04ad9" stroke-width="1.5" rx="3"/>
<text x="315" y="320" fill="#e0e0e0" text-anchor="middle" font-size="12" font-weight="bold">encrypted_epoch</text>
<text x="315" y="338" fill="#d8a0f8" text-anchor="middle" font-size="10">24 bytes (epoch 8 + AEAD tag 16)</text>
<!-- Total -->
<text x="310" y="366" fill="#777" font-size="10" text-anchor="middle">total: 69 bytes · payload_len = 65 · pattern: e, ee, se</text>
</svg>
Side by Side

Before

Width:  |  Height:  |  Size: 4.1 KiB

docs/design/diagrams/path-broken.svg
-43
View File
@@ -1,43 +0,0 @@
<svg xmlns="http://www.w3.org/2000/svg" viewBox="0 0 580 296" font-family="monospace" font-size="13">
<!-- Background -->
<rect width="580" height="296" fill="#1a1a2e" rx="4"/>
<!-- Title -->
<text x="310" y="26" fill="#e0e0e0" text-anchor="middle" font-size="14" font-weight="bold">PathBroken (0x21) — 36 + 16n bytes payload</text>
<!-- Row 0 (0–1): msg_type + flags -->
<text x="50" y="62" fill="#666" font-size="10" text-anchor="end">0–1</text>
<rect x="55" y="36" width="260" height="52" fill="#2d4a7a" stroke="#4a90d9" stroke-width="1.5" rx="3"/>
<text x="185" y="58" fill="#e0e0e0" text-anchor="middle" font-size="12" font-weight="bold">msg_type</text>
<text x="185" y="76" fill="#8ab4f8" text-anchor="middle" font-size="10">0x21</text>
<rect x="315" y="36" width="260" height="52" fill="#1f1f3a" stroke="#444" stroke-width="1" rx="3"/>
<text x="445" y="62" fill="#666" text-anchor="middle" font-size="12">flags (reserved)</text>
<!-- Row 1 (2–17): dest_addr -->
<text x="50" y="114" fill="#666" font-size="10" text-anchor="end">2–17</text>
<rect x="55" y="88" width="520" height="52" fill="#4a2d5a" stroke="#b04ad9" stroke-width="1.5" rx="3"/>
<text x="315" y="110" fill="#e0e0e0" text-anchor="middle" font-size="12" font-weight="bold">dest_addr</text>
<text x="315" y="128" fill="#d8a0f8" text-anchor="middle" font-size="10">16 bytes — unreachable NodeAddr</text>
<!-- Row 2 (18–33): reporter -->
<text x="50" y="166" fill="#666" font-size="10" text-anchor="end">18–33</text>
<rect x="55" y="140" width="520" height="52" fill="#2d5a5a" stroke="#4ad9d9" stroke-width="1.5" rx="3"/>
<text x="315" y="162" fill="#e0e0e0" text-anchor="middle" font-size="12" font-weight="bold">reporter</text>
<text x="315" y="180" fill="#8af8f8" text-anchor="middle" font-size="10">16 bytes — NodeAddr of reporting router</text>
<!-- Row 3 (34–35): last_coords_count -->
<text x="50" y="218" fill="#666" font-size="10" text-anchor="end">34–35</text>
<rect x="55" y="192" width="260" height="52" fill="#4a3d2d" stroke="#d9b04a" stroke-width="1.5" rx="3"/>
<text x="185" y="222" fill="#e0e0e0" text-anchor="middle" font-size="12" font-weight="bold">last_coords_count</text>
<!-- Row 4 (36–): last_known_coords -->
<text x="50" y="264" fill="#666" font-size="10" text-anchor="end">36–</text>
<rect x="55" y="248" width="520" height="40" fill="#1f1f3a" stroke="#d9904a" stroke-width="1" stroke-dasharray="4,3" rx="3"/>
<text x="315" y="272" fill="#f8c88a" text-anchor="middle" font-size="11">last_known_coords × n (16 bytes each — stale coords)</text>
</svg>
Side by Side

Before

Width:  |  Height:  |  Size: 2.6 KiB

docs/design/diagrams/path-mtu-notification.svg
-17
View File
@@ -1,17 +0,0 @@
<svg xmlns="http://www.w3.org/2000/svg" viewBox="0 0 580 114" font-family="monospace" font-size="13">
<!-- Background -->
<rect width="580" height="114" fill="#1a1a2e" rx="4"/>
<!-- Title -->
<text x="310" y="26" fill="#e0e0e0" text-anchor="middle" font-size="14" font-weight="bold">PathMtuNotification (0x13) — 2 bytes body</text>
<!-- Row 0 (0–1): path_mtu -->
<text x="50" y="62" fill="#666" font-size="10" text-anchor="end">0–1</text>
<rect x="55" y="36" width="520" height="52" fill="#5a2d3d" stroke="#d94a6a" stroke-width="1.5" rx="3"/>
<text x="315" y="58" fill="#e0e0e0" text-anchor="middle" font-size="12" font-weight="bold">path_mtu</text>
<text x="315" y="76" fill="#f88aaa" text-anchor="middle" font-size="10">2 bytes LE — minimum MTU observed along forward path</text>
<!-- Note -->
<text x="310" y="108" fill="#777" font-size="10" text-anchor="middle">body only — carried inside FSP encrypted message (prefix + inner header + AEAD tag)</text>
</svg>
Side by Side

Before

Width:  |  Height:  |  Size: 1000 B

docs/design/diagrams/receiver-report.svg
-106
View File
@@ -1,106 +0,0 @@
<svg xmlns="http://www.w3.org/2000/svg" viewBox="0 0 580 634" font-family="monospace" font-size="13">
<!-- Background -->
<rect width="580" height="634" fill="#1a1a2e" rx="4"/>
<!-- Title -->
<text x="310" y="26" fill="#e0e0e0" text-anchor="middle" font-size="14" font-weight="bold">ReceiverReport (0x02) — 68 bytes</text>
<!-- Row 0 (0–3): msg_type + reserved -->
<text x="50" y="62" fill="#666" font-size="10" text-anchor="end">0–3</text>
<rect x="55" y="36" width="130" height="52" fill="#2d4a7a" stroke="#4a90d9" stroke-width="1.5" rx="3"/>
<text x="120" y="60" fill="#e0e0e0" text-anchor="middle" font-size="12" font-weight="bold">msg_type</text>
<text x="120" y="78" fill="#8ab4f8" text-anchor="middle" font-size="10">0x02</text>
<rect x="185" y="36" width="390" height="52" fill="#1f1f3a" stroke="#444" stroke-width="1" rx="3"/>
<text x="380" y="66" fill="#666" text-anchor="middle" font-size="12">reserved (3 bytes zero)</text>
<!-- Row 1 (4–11): highest_counter -->
<text x="50" y="114" fill="#666" font-size="10" text-anchor="end">4–11</text>
<rect x="55" y="88" width="520" height="52" fill="#5a3d2d" stroke="#d9904a" stroke-width="1.5" rx="3"/>
<text x="315" y="110" fill="#e0e0e0" text-anchor="middle" font-size="12" font-weight="bold">highest_counter</text>
<text x="315" y="128" fill="#f8c88a" text-anchor="middle" font-size="10">8 bytes LE</text>
<!-- Row 2 (12–19): cumulative_packets_recv -->
<text x="50" y="166" fill="#666" font-size="10" text-anchor="end">12–19</text>
<rect x="55" y="140" width="520" height="52" fill="#2d5a4a" stroke="#4ad99a" stroke-width="1.5" rx="3"/>
<text x="315" y="162" fill="#e0e0e0" text-anchor="middle" font-size="12" font-weight="bold">cumulative_packets_recv</text>
<text x="315" y="180" fill="#8af8c8" text-anchor="middle" font-size="10">8 bytes LE</text>
<!-- Row 3 (20–27): cumulative_bytes_recv -->
<text x="50" y="218" fill="#666" font-size="10" text-anchor="end">20–27</text>
<rect x="55" y="192" width="520" height="52" fill="#2d5a4a" stroke="#4ad99a" stroke-width="1.5" rx="3"/>
<text x="315" y="214" fill="#e0e0e0" text-anchor="middle" font-size="12" font-weight="bold">cumulative_bytes_recv</text>
<text x="315" y="232" fill="#8af8c8" text-anchor="middle" font-size="10">8 bytes LE</text>
<!-- Row 4 (28–33): timestamp_echo + dwell_time -->
<text x="50" y="270" fill="#666" font-size="10" text-anchor="end">28–33</text>
<rect x="55" y="244" width="260" height="52" fill="#4a2d5a" stroke="#b04ad9" stroke-width="1.5" rx="3"/>
<text x="185" y="266" fill="#e0e0e0" text-anchor="middle" font-size="12" font-weight="bold">timestamp_echo</text>
<text x="185" y="284" fill="#d8a0f8" text-anchor="middle" font-size="10">4 bytes LE</text>
<rect x="315" y="244" width="260" height="52" fill="#2d5a5a" stroke="#4ad9d9" stroke-width="1.5" rx="3"/>
<text x="445" y="266" fill="#e0e0e0" text-anchor="middle" font-size="12" font-weight="bold">dwell_time</text>
<text x="445" y="284" fill="#8af8f8" text-anchor="middle" font-size="10">2 bytes LE ms</text>
<!-- Row 5 (34–39): max_burst_loss + mean_burst_loss + reserved -->
<text x="50" y="322" fill="#666" font-size="10" text-anchor="end">34–39</text>
<rect x="55" y="296" width="173" height="52" fill="#5a2d3d" stroke="#d94a6a" stroke-width="1.5" rx="3"/>
<text x="141" y="318" fill="#e0e0e0" text-anchor="middle" font-size="11" font-weight="bold">max_burst</text>
<text x="141" y="336" fill="#f88aaa" text-anchor="middle" font-size="10">2 bytes LE</text>
<rect x="228" y="296" width="174" height="52" fill="#5a2d3d" stroke="#d94a6a" stroke-width="1.5" rx="3"/>
<text x="315" y="318" fill="#e0e0e0" text-anchor="middle" font-size="11" font-weight="bold">mean_burst</text>
<text x="315" y="336" fill="#f88aaa" text-anchor="middle" font-size="10">2 bytes u8.8</text>
<rect x="402" y="296" width="173" height="52" fill="#1f1f3a" stroke="#444" stroke-width="1" rx="3"/>
<text x="488" y="322" fill="#666" text-anchor="middle" font-size="10">reserved</text>
<!-- Row 6 (40–47): jitter + ecn_ce_count -->
<text x="50" y="374" fill="#666" font-size="10" text-anchor="end">40–47</text>
<rect x="55" y="348" width="260" height="52" fill="#4a3d2d" stroke="#d9b04a" stroke-width="1.5" rx="3"/>
<text x="185" y="370" fill="#e0e0e0" text-anchor="middle" font-size="12" font-weight="bold">jitter</text>
<text x="185" y="388" fill="#f8d88a" text-anchor="middle" font-size="10">4 bytes LE µs</text>
<rect x="315" y="348" width="260" height="52" fill="#4a3d2d" stroke="#d9b04a" stroke-width="1.5" rx="3"/>
<text x="445" y="370" fill="#e0e0e0" text-anchor="middle" font-size="12" font-weight="bold">ecn_ce_count</text>
<text x="445" y="388" fill="#f8d88a" text-anchor="middle" font-size="10">4 bytes LE</text>
<!-- Row 7 (48–55): owd_trend + burst_loss_count -->
<text x="50" y="426" fill="#666" font-size="10" text-anchor="end">48–55</text>
<rect x="55" y="400" width="260" height="52" fill="#3d4a2d" stroke="#8ad94a" stroke-width="1.5" rx="3"/>
<text x="185" y="422" fill="#e0e0e0" text-anchor="middle" font-size="12" font-weight="bold">owd_trend</text>
<text x="185" y="440" fill="#c8f88a" text-anchor="middle" font-size="10">4 bytes i32 LE µs/s</text>
<rect x="315" y="400" width="260" height="52" fill="#3d4a2d" stroke="#8ad94a" stroke-width="1.5" rx="3"/>
<text x="445" y="422" fill="#e0e0e0" text-anchor="middle" font-size="12" font-weight="bold">burst_loss_count</text>
<text x="445" y="440" fill="#c8f88a" text-anchor="middle" font-size="10">4 bytes LE</text>
<!-- Row 8 (56–63): cumulative_reorder_count + interval_packets_recv -->
<text x="50" y="478" fill="#666" font-size="10" text-anchor="end">56–63</text>
<rect x="55" y="452" width="260" height="52" fill="#2d5a5a" stroke="#4ad9d9" stroke-width="1.5" rx="3"/>
<text x="185" y="474" fill="#e0e0e0" text-anchor="middle" font-size="11" font-weight="bold">reorder_count</text>
<text x="185" y="492" fill="#8af8f8" text-anchor="middle" font-size="10">4 bytes LE cumulative</text>
<rect x="315" y="452" width="260" height="52" fill="#2d5a5a" stroke="#4ad9d9" stroke-width="1.5" rx="3"/>
<text x="445" y="474" fill="#e0e0e0" text-anchor="middle" font-size="11" font-weight="bold">interval_pkts_recv</text>
<text x="445" y="492" fill="#8af8f8" text-anchor="middle" font-size="10">4 bytes LE</text>
<!-- Row 9 (64–67): interval_bytes_recv -->
<text x="50" y="530" fill="#666" font-size="10" text-anchor="end">64–67</text>
<rect x="55" y="504" width="260" height="52" fill="#2d5a5a" stroke="#4ad9d9" stroke-width="1.5" rx="3"/>
<text x="185" y="526" fill="#e0e0e0" text-anchor="middle" font-size="11" font-weight="bold">interval_bytes_recv</text>
<text x="185" y="544" fill="#8af8f8" text-anchor="middle" font-size="10">4 bytes LE</text>
<!-- Total -->
<text x="310" y="586" fill="#777" font-size="10" text-anchor="middle">total: 68 bytes</text>
</svg>
Side by Side

Before

Width:  |  Height:  |  Size: 6.8 KiB

docs/design/diagrams/sender-report.svg
-66
View File
@@ -1,66 +0,0 @@
<svg xmlns="http://www.w3.org/2000/svg" viewBox="0 0 580 478" font-family="monospace" font-size="13">
<!-- Background -->
<rect width="580" height="478" fill="#1a1a2e" rx="4"/>
<!-- Title -->
<text x="310" y="26" fill="#e0e0e0" text-anchor="middle" font-size="14" font-weight="bold">SenderReport (0x01) — 48 bytes</text>
<!-- Row 0 (0–3): msg_type + reserved -->
<text x="50" y="62" fill="#666" font-size="10" text-anchor="end">0–3</text>
<rect x="55" y="36" width="130" height="52" fill="#2d4a7a" stroke="#4a90d9" stroke-width="1.5" rx="3"/>
<text x="120" y="60" fill="#e0e0e0" text-anchor="middle" font-size="12" font-weight="bold">msg_type</text>
<text x="120" y="78" fill="#8ab4f8" text-anchor="middle" font-size="10">0x01</text>
<rect x="185" y="36" width="390" height="52" fill="#1f1f3a" stroke="#444" stroke-width="1" rx="3"/>
<text x="380" y="66" fill="#666" text-anchor="middle" font-size="12">reserved (3 bytes zero)</text>
<!-- Row 1 (4–11): interval_start_counter -->
<text x="50" y="114" fill="#666" font-size="10" text-anchor="end">4–11</text>
<rect x="55" y="88" width="520" height="52" fill="#5a3d2d" stroke="#d9904a" stroke-width="1.5" rx="3"/>
<text x="315" y="110" fill="#e0e0e0" text-anchor="middle" font-size="12" font-weight="bold">interval_start_counter</text>
<text x="315" y="128" fill="#f8c88a" text-anchor="middle" font-size="10">8 bytes LE</text>
<!-- Row 2 (12–19): interval_end_counter -->
<text x="50" y="166" fill="#666" font-size="10" text-anchor="end">12–19</text>
<rect x="55" y="140" width="520" height="52" fill="#5a3d2d" stroke="#d9904a" stroke-width="1.5" rx="3"/>
<text x="315" y="162" fill="#e0e0e0" text-anchor="middle" font-size="12" font-weight="bold">interval_end_counter</text>
<text x="315" y="180" fill="#f8c88a" text-anchor="middle" font-size="10">8 bytes LE</text>
<!-- Row 3 (20–27): interval_start_timestamp + interval_end_timestamp -->
<text x="50" y="218" fill="#666" font-size="10" text-anchor="end">20–27</text>
<rect x="55" y="192" width="260" height="52" fill="#4a2d5a" stroke="#b04ad9" stroke-width="1.5" rx="3"/>
<text x="185" y="214" fill="#e0e0e0" text-anchor="middle" font-size="12" font-weight="bold">interval_start_ts</text>
<text x="185" y="232" fill="#d8a0f8" text-anchor="middle" font-size="10">4 bytes LE</text>
<rect x="315" y="192" width="260" height="52" fill="#4a2d5a" stroke="#b04ad9" stroke-width="1.5" rx="3"/>
<text x="445" y="214" fill="#e0e0e0" text-anchor="middle" font-size="12" font-weight="bold">interval_end_ts</text>
<text x="445" y="232" fill="#d8a0f8" text-anchor="middle" font-size="10">4 bytes LE</text>
<!-- Row 4 (28–31): interval_bytes_sent -->
<text x="50" y="270" fill="#666" font-size="10" text-anchor="end">28–31</text>
<rect x="55" y="244" width="520" height="52" fill="#2d5a5a" stroke="#4ad9d9" stroke-width="1.5" rx="3"/>
<text x="315" y="266" fill="#e0e0e0" text-anchor="middle" font-size="12" font-weight="bold">interval_bytes_sent</text>
<text x="315" y="284" fill="#8af8f8" text-anchor="middle" font-size="10">4 bytes LE</text>
<!-- Row 5 (32–39): cumulative_packets_sent -->
<text x="50" y="322" fill="#666" font-size="10" text-anchor="end">32–39</text>
<rect x="55" y="296" width="520" height="52" fill="#2d5a4a" stroke="#4ad99a" stroke-width="1.5" rx="3"/>
<text x="315" y="318" fill="#e0e0e0" text-anchor="middle" font-size="12" font-weight="bold">cumulative_packets_sent</text>
<text x="315" y="336" fill="#8af8c8" text-anchor="middle" font-size="10">8 bytes LE</text>
<!-- Row 6 (40–47): cumulative_bytes_sent -->
<text x="50" y="374" fill="#666" font-size="10" text-anchor="end">40–47</text>
<rect x="55" y="348" width="520" height="52" fill="#2d5a4a" stroke="#4ad99a" stroke-width="1.5" rx="3"/>
<text x="315" y="370" fill="#e0e0e0" text-anchor="middle" font-size="12" font-weight="bold">cumulative_bytes_sent</text>
<text x="315" y="388" fill="#8af8c8" text-anchor="middle" font-size="10">8 bytes LE</text>
<!-- Total -->
<text x="310" y="422" fill="#777" font-size="10" text-anchor="middle">total: 48 bytes</text>
</svg>
Side by Side

Before

Width:  |  Height:  |  Size: 4.1 KiB

docs/design/diagrams/session-ack.svg
-47
View File
@@ -1,47 +0,0 @@
<svg xmlns="http://www.w3.org/2000/svg" viewBox="0 0 580 322" font-family="monospace" font-size="13">
<!-- Background -->
<rect width="580" height="322" fill="#1a1a2e" rx="4"/>
<!-- Title -->
<text x="310" y="26" fill="#e0e0e0" text-anchor="middle" font-size="14" font-weight="bold">SessionAck (phase 0x2) — Noise XK msg2</text>
<!-- Row 0 (0–3): FSP prefix -->
<text x="50" y="62" fill="#666" font-size="10" text-anchor="end">0–3</text>
<rect x="55" y="36" width="520" height="52" fill="#2d4a7a" stroke="#4a90d9" stroke-width="1.5" rx="3"/>
<text x="315" y="58" fill="#e0e0e0" text-anchor="middle" font-size="12" font-weight="bold">FSP common prefix</text>
<text x="315" y="76" fill="#8ab4f8" text-anchor="middle" font-size="10">ver=0, phase=0x2, flags=0, payload_len</text>
<!-- Row 1 (4): flags -->
<text x="50" y="114" fill="#666" font-size="10" text-anchor="end">4</text>
<rect x="55" y="88" width="520" height="52" fill="#5a3d2d" stroke="#d9904a" stroke-width="1.5" rx="3"/>
<text x="315" y="110" fill="#e0e0e0" text-anchor="middle" font-size="12" font-weight="bold">flags</text>
<text x="315" y="128" fill="#f8c88a" text-anchor="middle" font-size="10">1 byte — reserved</text>
<!-- Row 2 (5–): src_coords (acknowledger) -->
<text x="50" y="166" fill="#666" font-size="10" text-anchor="end">5–</text>
<rect x="55" y="140" width="520" height="40" fill="#1f1f3a" stroke="#4ad9d9" stroke-width="1" stroke-dasharray="4,3" rx="3"/>
<text x="315" y="164" fill="#8af8f8" text-anchor="middle" font-size="11">src_coords_count(2) + src_coords(16 × n) — acknowledger</text>
<!-- Row 3: dest_coords (initiator) -->
<text x="50" y="206" fill="#666" font-size="10" text-anchor="end">...</text>
<rect x="55" y="186" width="520" height="40" fill="#1f1f3a" stroke="#b04ad9" stroke-width="1" stroke-dasharray="4,3" rx="3"/>
<text x="315" y="210" fill="#d8a0f8" text-anchor="middle" font-size="11">dest_coords_count(2) + dest_coords(16 × m) — initiator</text>
<!-- Row 4: handshake_len + handshake_payload -->
<text x="50" y="252" fill="#666" font-size="10" text-anchor="end">...</text>
<rect x="55" y="232" width="130" height="52" fill="#4a3d2d" stroke="#d9b04a" stroke-width="1.5" rx="3"/>
<text x="120" y="254" fill="#e0e0e0" text-anchor="middle" font-size="11" font-weight="bold">hs_len</text>
<text x="120" y="272" fill="#f8d88a" text-anchor="middle" font-size="10">2 bytes LE</text>
<rect x="185" y="232" width="390" height="52" fill="#2d5a4a" stroke="#4ad99a" stroke-width="1.5" rx="3"/>
<text x="380" y="254" fill="#e0e0e0" text-anchor="middle" font-size="12" font-weight="bold">handshake_payload</text>
<text x="380" y="272" fill="#8af8c8" text-anchor="middle" font-size="10">Noise XK msg2 (57 bytes — ephemeral + epoch)</text>
<!-- Total -->
<text x="310" y="308" fill="#777" font-size="10" text-anchor="middle">typical ~190 bytes (depth-dependent, carries both endpoints' coords)</text>
</svg>
Side by Side

Before

Width:  |  Height:  |  Size: 2.9 KiB

docs/design/diagrams/session-datagram.svg
-46
View File
@@ -1,46 +0,0 @@
<svg xmlns="http://www.w3.org/2000/svg" viewBox="0 0 580 374" font-family="monospace" font-size="13">
<!-- Background -->
<rect width="580" height="374" fill="#1a1a2e" rx="4"/>
<!-- Title -->
<text x="310" y="26" fill="#e0e0e0" text-anchor="middle" font-size="14" font-weight="bold">SessionDatagram (0x00) — 36 + payload bytes</text>
<!-- Row 0 (0–3): msg_type + ttl + path_mtu -->
<text x="50" y="62" fill="#666" font-size="10" text-anchor="end">0–3</text>
<rect x="55" y="36" width="130" height="52" fill="#2d4a7a" stroke="#4a90d9" stroke-width="1.5" rx="3"/>
<text x="120" y="60" fill="#e0e0e0" text-anchor="middle" font-size="12" font-weight="bold">msg_type</text>
<text x="120" y="78" fill="#8ab4f8" text-anchor="middle" font-size="10">0x00</text>
<rect x="185" y="36" width="130" height="52" fill="#2d5a4a" stroke="#4ad99a" stroke-width="1.5" rx="3"/>
<text x="250" y="60" fill="#e0e0e0" text-anchor="middle" font-size="12" font-weight="bold">ttl</text>
<text x="250" y="78" fill="#8af8c8" text-anchor="middle" font-size="10">1 byte</text>
<rect x="315" y="36" width="260" height="52" fill="#5a3d2d" stroke="#d9904a" stroke-width="1.5" rx="3"/>
<text x="445" y="60" fill="#e0e0e0" text-anchor="middle" font-size="12" font-weight="bold">path_mtu</text>
<text x="445" y="78" fill="#f8c88a" text-anchor="middle" font-size="10">2 bytes LE</text>
<!-- Row 1-4 (4–19): src_addr -->
<text x="50" y="114" fill="#666" font-size="10" text-anchor="end">4–19</text>
<rect x="55" y="88" width="520" height="52" fill="#2d5a5a" stroke="#4ad9d9" stroke-width="1.5" rx="3"/>
<text x="315" y="110" fill="#e0e0e0" text-anchor="middle" font-size="12" font-weight="bold">src_addr</text>
<text x="315" y="128" fill="#8af8f8" text-anchor="middle" font-size="10">16 bytes — source NodeAddr</text>
<!-- Row 5-8 (20–35): dest_addr -->
<text x="50" y="166" fill="#666" font-size="10" text-anchor="end">20–35</text>
<rect x="55" y="140" width="520" height="52" fill="#4a2d5a" stroke="#b04ad9" stroke-width="1.5" rx="3"/>
<text x="315" y="162" fill="#e0e0e0" text-anchor="middle" font-size="12" font-weight="bold">dest_addr</text>
<text x="315" y="180" fill="#d8a0f8" text-anchor="middle" font-size="10">16 bytes — destination NodeAddr</text>
<!-- Row 9 (36–): payload -->
<text x="50" y="218" fill="#666" font-size="10" text-anchor="end">36–</text>
<rect x="55" y="198" width="520" height="40" fill="#1f1f3a" stroke="#444" stroke-width="1" stroke-dasharray="4,3" rx="3"/>
<text x="315" y="222" fill="#666" text-anchor="middle" font-size="11">payload (variable — session-layer message)</text>
<!-- Notes -->
<text x="310" y="262" fill="#777" font-size="10" text-anchor="middle">fixed header: 36 bytes · payload opaque to transit nodes</text>
<text x="310" y="278" fill="#777" font-size="10" text-anchor="middle">ttl decremented and path_mtu min'd at each forwarding hop</text>
</svg>
Side by Side

Before

Width:  |  Height:  |  Size: 2.9 KiB

docs/design/diagrams/session-msg3.svg
-35
View File
@@ -1,35 +0,0 @@
<svg xmlns="http://www.w3.org/2000/svg" viewBox="0 0 580 244" font-family="monospace" font-size="13">
<!-- Background -->
<rect width="580" height="244" fill="#1a1a2e" rx="4"/>
<!-- Title -->
<text x="310" y="26" fill="#e0e0e0" text-anchor="middle" font-size="14" font-weight="bold">SessionMsg3 (phase 0x3) — Noise XK msg3</text>
<!-- Row 0 (0–3): FSP prefix -->
<text x="50" y="62" fill="#666" font-size="10" text-anchor="end">0–3</text>
<rect x="55" y="36" width="520" height="52" fill="#2d4a7a" stroke="#4a90d9" stroke-width="1.5" rx="3"/>
<text x="315" y="58" fill="#e0e0e0" text-anchor="middle" font-size="12" font-weight="bold">FSP common prefix</text>
<text x="315" y="76" fill="#8ab4f8" text-anchor="middle" font-size="10">ver=0, phase=0x3, flags=0, payload_len</text>
<!-- Row 1 (4–6): flags + handshake_len -->
<text x="50" y="114" fill="#666" font-size="10" text-anchor="end">4–6</text>
<rect x="55" y="88" width="130" height="52" fill="#5a3d2d" stroke="#d9904a" stroke-width="1.5" rx="3"/>
<text x="120" y="110" fill="#e0e0e0" text-anchor="middle" font-size="12" font-weight="bold">flags</text>
<text x="120" y="128" fill="#f8c88a" text-anchor="middle" font-size="10">1 byte</text>
<rect x="185" y="88" width="390" height="52" fill="#4a3d2d" stroke="#d9b04a" stroke-width="1.5" rx="3"/>
<text x="380" y="110" fill="#e0e0e0" text-anchor="middle" font-size="12" font-weight="bold">handshake_len</text>
<text x="380" y="128" fill="#f8d88a" text-anchor="middle" font-size="10">2 bytes LE</text>
<!-- Row 2 (7–): handshake_payload -->
<text x="50" y="166" fill="#666" font-size="10" text-anchor="end">7–</text>
<rect x="55" y="140" width="520" height="52" fill="#2d5a4a" stroke="#4ad99a" stroke-width="1.5" rx="3"/>
<text x="315" y="162" fill="#e0e0e0" text-anchor="middle" font-size="12" font-weight="bold">handshake_payload</text>
<text x="315" y="180" fill="#8af8c8" text-anchor="middle" font-size="10">Noise XK msg3 (73 bytes — encrypted static + encrypted epoch)</text>
<!-- Total -->
<text x="310" y="218" fill="#777" font-size="10" text-anchor="middle">~80 bytes · no coordinates (both endpoints already have them)</text>
</svg>
Side by Side

Before

Width:  |  Height:  |  Size: 2.2 KiB

docs/design/diagrams/session-setup.svg
-47
View File
@@ -1,47 +0,0 @@
<svg xmlns="http://www.w3.org/2000/svg" viewBox="0 0 580 322" font-family="monospace" font-size="13">
<!-- Background -->
<rect width="580" height="322" fill="#1a1a2e" rx="4"/>
<!-- Title -->
<text x="310" y="26" fill="#e0e0e0" text-anchor="middle" font-size="14" font-weight="bold">SessionSetup (phase 0x1) — Noise XK msg1</text>
<!-- Row 0 (0–3): FSP prefix -->
<text x="50" y="62" fill="#666" font-size="10" text-anchor="end">0–3</text>
<rect x="55" y="36" width="520" height="52" fill="#2d4a7a" stroke="#4a90d9" stroke-width="1.5" rx="3"/>
<text x="315" y="58" fill="#e0e0e0" text-anchor="middle" font-size="12" font-weight="bold">FSP common prefix</text>
<text x="315" y="76" fill="#8ab4f8" text-anchor="middle" font-size="10">ver=0, phase=0x1, flags=0, payload_len</text>
<!-- Row 1 (4): flags -->
<text x="50" y="114" fill="#666" font-size="10" text-anchor="end">4</text>
<rect x="55" y="88" width="520" height="52" fill="#5a3d2d" stroke="#d9904a" stroke-width="1.5" rx="3"/>
<text x="315" y="110" fill="#e0e0e0" text-anchor="middle" font-size="12" font-weight="bold">flags</text>
<text x="315" y="128" fill="#f8c88a" text-anchor="middle" font-size="10">1 byte — bit 0: REQUEST_ACK, bit 1: BIDIRECTIONAL</text>
<!-- Row 2 (5–): src_coords -->
<text x="50" y="166" fill="#666" font-size="10" text-anchor="end">5–</text>
<rect x="55" y="140" width="520" height="40" fill="#1f1f3a" stroke="#4ad9d9" stroke-width="1" stroke-dasharray="4,3" rx="3"/>
<text x="315" y="164" fill="#8af8f8" text-anchor="middle" font-size="11">src_coords_count(2) + src_coords(16 × n)</text>
<!-- Row 3: dest_coords -->
<text x="50" y="206" fill="#666" font-size="10" text-anchor="end">...</text>
<rect x="55" y="186" width="520" height="40" fill="#1f1f3a" stroke="#b04ad9" stroke-width="1" stroke-dasharray="4,3" rx="3"/>
<text x="315" y="210" fill="#d8a0f8" text-anchor="middle" font-size="11">dest_coords_count(2) + dest_coords(16 × m)</text>
<!-- Row 4: handshake_len + handshake_payload -->
<text x="50" y="252" fill="#666" font-size="10" text-anchor="end">...</text>
<rect x="55" y="232" width="130" height="52" fill="#4a3d2d" stroke="#d9b04a" stroke-width="1.5" rx="3"/>
<text x="120" y="254" fill="#e0e0e0" text-anchor="middle" font-size="11" font-weight="bold">hs_len</text>
<text x="120" y="272" fill="#f8d88a" text-anchor="middle" font-size="10">2 bytes LE</text>
<rect x="185" y="232" width="390" height="52" fill="#2d5a4a" stroke="#4ad99a" stroke-width="1.5" rx="3"/>
<text x="380" y="254" fill="#e0e0e0" text-anchor="middle" font-size="12" font-weight="bold">handshake_payload</text>
<text x="380" y="272" fill="#8af8c8" text-anchor="middle" font-size="10">Noise XK msg1 (33 bytes — ephemeral key)</text>
<!-- Total -->
<text x="310" y="308" fill="#777" font-size="10" text-anchor="middle">typical ~170 bytes (depth-dependent)</text>
</svg>
Side by Side

Before

Width:  |  Height:  |  Size: 2.9 KiB

docs/design/diagrams/tree-announce.svg
-59
View File
@@ -1,59 +0,0 @@
<svg xmlns="http://www.w3.org/2000/svg" viewBox="0 0 580 374" font-family="monospace" font-size="13">
<!-- Background -->
<rect width="580" height="374" fill="#1a1a2e" rx="4"/>
<!-- Title -->
<text x="310" y="26" fill="#e0e0e0" text-anchor="middle" font-size="14" font-weight="bold">TreeAnnounce (0x10) — 100 + 32n bytes</text>
<!-- Row 0 (0–3): msg_type(1) + version(1) + sequence starts -->
<text x="50" y="62" fill="#666" font-size="10" text-anchor="end">0–3</text>
<rect x="55" y="36" width="130" height="52" fill="#2d4a7a" stroke="#4a90d9" stroke-width="1.5" rx="3"/>
<text x="120" y="60" fill="#e0e0e0" text-anchor="middle" font-size="12" font-weight="bold">msg_type</text>
<text x="120" y="78" fill="#8ab4f8" text-anchor="middle" font-size="10">0x10</text>
<rect x="185" y="36" width="130" height="52" fill="#2d5a4a" stroke="#4ad99a" stroke-width="1.5" rx="3"/>
<text x="250" y="60" fill="#e0e0e0" text-anchor="middle" font-size="12" font-weight="bold">version</text>
<text x="250" y="78" fill="#8af8c8" text-anchor="middle" font-size="10">0x01</text>
<rect x="315" y="36" width="260" height="52" fill="#5a3d2d" stroke="#d9904a" stroke-width="1.5" rx="3"/>
<text x="445" y="66" fill="#e0e0e0" text-anchor="middle" font-size="12" font-weight="bold">sequence</text>
<!-- Row 1 (4–9): sequence continued -->
<text x="50" y="114" fill="#666" font-size="10" text-anchor="end">4–9</text>
<rect x="55" y="88" width="520" height="52" fill="#5a3d2d" stroke="#d9904a" stroke-width="1.5" rx="3"/>
<text x="315" y="118" fill="#f8c88a" text-anchor="middle" font-size="10">8 bytes LE — monotonic counter</text>
<!-- Row 2 (10–17): timestamp -->
<text x="50" y="166" fill="#666" font-size="10" text-anchor="end">10–17</text>
<rect x="55" y="140" width="520" height="52" fill="#4a2d5a" stroke="#b04ad9" stroke-width="1.5" rx="3"/>
<text x="315" y="162" fill="#e0e0e0" text-anchor="middle" font-size="12" font-weight="bold">timestamp</text>
<text x="315" y="180" fill="#d8a0f8" text-anchor="middle" font-size="10">8 bytes LE — Unix seconds</text>
<!-- Row 3 (18–33): parent -->
<text x="50" y="218" fill="#666" font-size="10" text-anchor="end">18–33</text>
<rect x="55" y="192" width="520" height="52" fill="#2d5a5a" stroke="#4ad9d9" stroke-width="1.5" rx="3"/>
<text x="315" y="214" fill="#e0e0e0" text-anchor="middle" font-size="12" font-weight="bold">parent</text>
<text x="315" y="232" fill="#8af8f8" text-anchor="middle" font-size="10">16 bytes — NodeAddr (self = root)</text>
<!-- Row 4 (34–35): ancestry_count -->
<text x="50" y="270" fill="#666" font-size="10" text-anchor="end">34–35</text>
<rect x="55" y="250" width="260" height="40" fill="#5a2d3d" stroke="#d94a6a" stroke-width="1.5" rx="3"/>
<text x="185" y="274" fill="#e0e0e0" text-anchor="middle" font-size="12" font-weight="bold">ancestry_count</text>
<!-- Row 5 (36–): ancestry entries -->
<text x="50" y="316" fill="#666" font-size="10" text-anchor="end">36–</text>
<rect x="55" y="296" width="520" height="40" fill="#1f1f3a" stroke="#d9904a" stroke-width="1" stroke-dasharray="4,3" rx="3"/>
<text x="315" y="320" fill="#f8c88a" text-anchor="middle" font-size="11">AncestryEntry × n (32 bytes each)</text>
<!-- Signature -->
<text x="50" y="360" fill="#666" font-size="10" text-anchor="end">+64</text>
<rect x="55" y="342" width="520" height="26" fill="#3d4a2d" stroke="#8ad94a" stroke-width="1.5" rx="3"/>
<text x="315" y="360" fill="#e0e0e0" text-anchor="middle" font-size="10" font-weight="bold">Schnorr signature (64 bytes)</text>
</svg>
Side by Side

Before

Width:  |  Height:  |  Size: 3.5 KiB

docs/design/document-relationships.svg
-100
View File
@@ -1,100 +0,0 @@
<svg xmlns="http://www.w3.org/2000/svg" viewBox="0 0 720 520" font-family="system-ui, -apple-system, sans-serif" font-size="13">
<defs>
<marker id="arrow" viewBox="0 0 10 10" refX="10" refY="5" markerWidth="8" markerHeight="8" orient="auto-start-reverse">
<path d="M 0 0 L 10 5 L 0 10 z" fill="#555"/>
</marker>
</defs>
<!-- Background -->
<rect width="720" height="520" rx="8" fill="#fafafa" stroke="#ddd" stroke-width="1"/>
<!-- Title -->
<text x="360" y="32" text-anchor="middle" font-size="16" font-weight="600" fill="#333">Document Relationships</text>
<!-- fips-intro -->
<rect x="260" y="50" width="200" height="32" rx="6" fill="#e3f2fd" stroke="#90caf9"/>
<text x="360" y="71" text-anchor="middle" font-weight="500" fill="#1565c0">fips-intro.md</text>
<!-- Arrows from intro -->
<line x1="310" y1="82" x2="130" y2="120" stroke="#555" marker-end="url(#arrow)"/>
<line x1="360" y1="82" x2="360" y2="120" stroke="#555" marker-end="url(#arrow)"/>
<!-- fips-transport-layer -->
<rect x="30" y="120" width="200" height="32" rx="6" fill="#e8f5e9" stroke="#a5d6a7"/>
<text x="130" y="141" text-anchor="middle" font-weight="500" fill="#2e7d32">fips-transport-layer.md</text>
<!-- fips-mesh-operation -->
<rect x="270" y="120" width="200" height="32" rx="6" fill="#fff3e0" stroke="#ffcc80"/>
<text x="370" y="141" text-anchor="middle" font-weight="500" fill="#e65100">fips-mesh-operation.md</text>
<!-- Arrow: transport -> mesh-layer -->
<line x1="130" y1="152" x2="130" y2="190" stroke="#555" marker-end="url(#arrow)"/>
<!-- fips-mesh-layer -->
<rect x="30" y="190" width="200" height="32" rx="6" fill="#e8f5e9" stroke="#a5d6a7"/>
<text x="130" y="211" text-anchor="middle" font-weight="500" fill="#2e7d32">fips-mesh-layer.md</text>
<!-- Arrow: mesh-operation -> mesh-layer -->
<line x1="270" y1="145" x2="230" y2="200" stroke="#555" marker-end="url(#arrow)"/>
<!-- Arrow: mesh-layer -> session-layer -->
<line x1="130" y1="222" x2="130" y2="260" stroke="#555" marker-end="url(#arrow)"/>
<!-- fips-session-layer -->
<rect x="30" y="260" width="200" height="32" rx="6" fill="#e8f5e9" stroke="#a5d6a7"/>
<text x="130" y="281" text-anchor="middle" font-weight="500" fill="#2e7d32">fips-session-layer.md</text>
<!-- Arrow: session-layer -> ipv6-adapter -->
<line x1="130" y1="292" x2="130" y2="330" stroke="#555" marker-end="url(#arrow)"/>
<!-- fips-ipv6-adapter -->
<rect x="30" y="330" width="200" height="32" rx="6" fill="#e8f5e9" stroke="#a5d6a7"/>
<text x="130" y="351" text-anchor="middle" font-weight="500" fill="#2e7d32">fips-ipv6-adapter.md</text>
<!-- Arrow: mesh-operation -> spanning-tree -->
<line x1="470" y1="145" x2="550" y2="190" stroke="#555" marker-end="url(#arrow)"/>
<!-- Arrow: mesh-operation -> bloom-filters -->
<line x1="470" y1="148" x2="550" y2="260" stroke="#555" marker-end="url(#arrow)"/>
<!-- fips-spanning-tree -->
<rect x="500" y="190" width="200" height="32" rx="6" fill="#f3e5f5" stroke="#ce93d8"/>
<text x="600" y="211" text-anchor="middle" font-weight="500" fill="#7b1fa2">fips-spanning-tree.md</text>
<!-- Arrow: intro -> spanning-tree -->
<line x1="460" y1="72" x2="560" y2="190" stroke="#555" marker-end="url(#arrow)"/>
<!-- fips-bloom-filters -->
<rect x="500" y="260" width="200" height="32" rx="6" fill="#f3e5f5" stroke="#ce93d8"/>
<text x="600" y="281" text-anchor="middle" font-weight="500" fill="#7b1fa2">fips-bloom-filters.md</text>
<!-- Arrow: spanning-tree -> bloom (dependency) -->
<line x1="600" y1="222" x2="600" y2="260" stroke="#999" stroke-dasharray="4,3" marker-end="url(#arrow)"/>
<!-- fips-wire-formats -->
<rect x="260" y="400" width="200" height="32" rx="6" fill="#fce4ec" stroke="#ef9a9a"/>
<text x="360" y="421" text-anchor="middle" font-weight="500" fill="#c62828">fips-wire-formats.md</text>
<text x="360" y="448" text-anchor="middle" font-size="11" fill="#888">(referenced by all layer docs)</text>
<!-- fips-configuration -->
<rect x="30" y="400" width="200" height="32" rx="6" fill="#f5f5f5" stroke="#bdbdbd"/>
<text x="130" y="421" text-anchor="middle" font-weight="500" fill="#424242">fips-configuration.md</text>
<text x="130" y="448" text-anchor="middle" font-size="11" fill="#888">(standalone reference)</text>
<!-- spanning-tree-dynamics -->
<rect x="500" y="330" width="200" height="32" rx="6" fill="#f3e5f5" stroke="#ce93d8"/>
<text x="600" y="351" text-anchor="middle" font-weight="500" fill="#7b1fa2">spanning-tree-dynamics.md</text>
<text x="600" y="378" text-anchor="middle" font-size="11" fill="#888">(companion to fips-spanning-tree)</text>
<!-- Legend -->
<rect x="30" y="472" width="14" height="14" rx="3" fill="#e8f5e9" stroke="#a5d6a7"/>
<text x="50" y="484" font-size="11" fill="#666">Protocol stack</text>
<rect x="155" y="472" width="14" height="14" rx="3" fill="#fff3e0" stroke="#ffcc80"/>
<text x="175" y="484" font-size="11" fill="#666">Mesh behavior</text>
<rect x="295" y="472" width="14" height="14" rx="3" fill="#f3e5f5" stroke="#ce93d8"/>
<text x="315" y="484" font-size="11" fill="#666">Supporting references</text>
<rect x="460" y="472" width="14" height="14" rx="3" fill="#fce4ec" stroke="#ef9a9a"/>
<text x="480" y="484" font-size="11" fill="#666">Wire formats</text>
<rect x="580" y="472" width="14" height="14" rx="3" fill="#f5f5f5" stroke="#bdbdbd"/>
<text x="600" y="484" font-size="11" fill="#666">Implementation</text>
</svg>
Side by Side

Before

Width:  |  Height:  |  Size: 5.4 KiB

docs/design/fips-architecture.md
+264
View File
@@ -0,0 +1,264 @@
# FIPS Architecture
The protocol architecture, identity system, and two-layer encryption
model. For the higher-level "what is FIPS and why" framing, see
[fips-concepts.md](fips-concepts.md). For prior art and academic
citations, see [fips-prior-work.md](fips-prior-work.md).
## Protocol Architecture
FIPS is organized in three protocol layers, each with distinct
responsibilities and clean service boundaries. No layer depends on
the specifics of the layers above or below it — transport plugins
know nothing about sessions, the routing layer knows nothing about
application addressing, and applications know nothing about which
physical media carry their traffic. This separation means new
transports, protocol features, and application interfaces can be
added independently.
![Protocol Stack](diagrams/fips-protocol-stack.svg)
### Mapping to Traditional Networking
Readers familiar with the OSI model or TCP/IP networking may find it
helpful to see how FIPS concepts relate to traditional layers:
![OSI Mapping](diagrams/fips-osi-mapping.svg)
Note that FMP spans what would traditionally be separate link and
network layers. This is intentional — in a self-organizing mesh, the
same layer that authenticates peers also makes routing decisions,
because routing depends on authenticated peer state (spanning tree
positions, bloom filters).
### Layer Responsibilities
**Transport layer**: Delivers datagrams between endpoints over a
specific medium. Each transport type (UDP socket, Ethernet interface,
radio modem) implements the same abstract interface: send and receive
datagrams, report MTU. The transport layer knows nothing about FIPS
identities, routing, or encryption. It provides raw datagram delivery
to FMP above.
See [fips-transport-layer.md](fips-transport-layer.md) for the
transport layer specification.
**FIPS Mesh Protocol (FMP)**: Manages peer connections, authenticates
peers via Noise IK handshakes, and encrypts all traffic on each link.
FMP is where the mesh organizes itself — nodes exchange spanning tree
announcements and bloom filters with their direct peers, and FMP
makes forwarding decisions for transit traffic. FMP provides
authenticated, encrypted forwarding to FSP above.
See [fips-mesh-layer.md](fips-mesh-layer.md) for the FMP specification
and [fips-mesh-operation.md](fips-mesh-operation.md) for how FMP's
routing and self-organization work in practice.
**FIPS Session Protocol (FSP)**: Provides end-to-end authenticated
encryption between any two nodes, regardless of how many intermediate
hops separate them. FSP manages session lifecycle (setup, data
transfer, teardown), caches destination coordinates for efficient
routing, and handles the warmup strategy that keeps transit node
caches populated. Session dispatch uses index-based routing inspired
by [WireGuard](https://www.wireguard.com/), enabling O(1) packet
demultiplexing. FSP provides a datagram service to applications above.
See [fips-session-layer.md](fips-session-layer.md) for the FSP
specification.
**IPv6 adaptation layer**: Sits above FSP as a service on port 256,
adapting the FIPS datagram service for unmodified IPv6 applications.
Provides DNS resolution (npub → fd00::/8 address), identity cache
management, IPv6 header compression, MTU enforcement, and a TUN
interface. This is the primary way existing applications use the FIPS
mesh.
See [fips-ipv6-adapter.md](fips-ipv6-adapter.md) for the IPv6 adapter.
### Node Architecture
Application services sit at the top of the stack, dispatched by FSP
port number: the IPv6 TUN adapter (port 256) maps npubs to `fd00::/8`
addresses with header compression so unmodified IP applications can
use the network transparently, while the native datagram API
addresses destinations directly by npub.
![Node Architecture](diagrams/fips-node-architecture.svg)
The mesh routes application traffic across heterogeneous transports
transparently. A packet may traverse WiFi, Ethernet, UDP/IP, and Tor
links on its way from source to destination — the application never
needs to know which transports are involved. Each hop is independently
encrypted at the link layer, while a single end-to-end session
protects the payload across the entire path.
![Architecture Overview](diagrams/fips-architecture-overview.svg)
![Mesh Topology](diagrams/fips-mesh-topology.svg)
## Identity System
FIPS uses [Nostr](https://github.com/nostr-protocol/nips) keypairs
(secp256k1) as node identities. The public key identifies the node;
the private key signs protocol messages and establishes encrypted
sessions.
The public key (or its bech32-encoded npub form) is the primary means
for application-layer software to identify communication endpoints.
Internally, the protocol derives a `node_addr` (a 16-byte SHA-256 hash
of the pubkey) used as the routing identifier in packet headers, and
an IPv6 address derived from the node_addr for the TUN adapter.
Applications use the pubkey or npub; the routing layer uses node_addr;
unmodified IPv6 applications use the derived `fd00::/8` address. All
three are deterministically derived from the same keypair.
### FIPS Identity Handling
![Identity Derivation](diagrams/fips-identity-derivation.svg)
The pubkey is the node's cryptographic identity, used in Noise
handshakes for both link encryption (IK) and session encryption (XK).
It is never exposed beyond the endpoints of an encrypted channel. The node_addr, a one-way
SHA-256 hash truncated to 16 bytes, serves as the routing identifier
in packet headers and bloom filters. Intermediate routers see only
node_addrs — they can forward traffic without learning the Nostr
identities of the endpoints. An observer can verify "does this
node_addr belong to pubkey X?" if they already know the pubkey, but
cannot enumerate communicating identities by inspecting traffic. The
IPv6 address prepends `fd` to the first 15 bytes of the node_addr,
providing a ULA overlay address for unmodified IP applications via the
TUN interface.
Below the FIPS identity layer, each transport uses its own native
addressing — IP:port or hostname:port addresses, MAC addresses,
.onion identifiers. These **link addresses** are opaque to everything
above FMP and discarded once link authentication completes.
### Identity Verification
The Noise Protocol Framework mutually authenticates both peer-to-peer
link connections (at FMP) and end-to-end session traffic (at FSP),
proving each party controls the private key for their claimed
identity.
See [fips-mesh-layer.md](fips-mesh-layer.md) for peer authentication
and [fips-session-layer.md](fips-session-layer.md) for end-to-end
session establishment.
Key rotation changes the node's identity — a new keypair produces a
new node_addr and IPv6 address, requiring all sessions to be
re-established. Migration mechanisms that allow a node to announce a
successor key are a future consideration.
## Two-Layer Encryption
FIPS uses independent encryption at two protocol layers:
| Layer | Scope | Pattern | Purpose |
| ----- | ----- | ------- | ------- |
| **FMP (Mesh)** | Hop-by-hop | Noise IK | Encrypt all traffic on each peer link |
| **FSP (Session)** | End-to-end | Noise XK | Encrypt application payload between endpoints |
### Link Layer (Hop-by-Hop)
When two nodes establish a direct connection, they perform a [Noise
IK](https://noiseprotocol.org/) handshake. This authenticates both
parties and establishes symmetric keys for encrypting all traffic on
that link. Every packet between direct peers is encrypted — gossip
messages, routing queries, and forwarded session datagrams alike.
The IK pattern is used because outbound connections know the peer's
npub from configuration, while inbound connections learn the
initiator's identity from the first handshake message.
### Session Layer (End-to-End)
FIPS establishes end-to-end encrypted sessions between any two
communicating nodes using Noise XK, regardless of how many hops
separate them. The initiator knows the destination's npub (required
for XK's pre-message); the responder learns the initiator's identity
from the third handshake message. Unlike the link-layer IK pattern
where the initiator's identity is revealed in msg1, XK delays
identity disclosure until msg3, providing stronger initiator identity
protection for traffic traversing untrusted intermediate nodes.
A packet from A to D through intermediate nodes B and C:
1. A encrypts payload with A↔D session key (FSP)
2. A wraps in SessionDatagram, encrypts with A↔B link key (FMP),
sends to B
3. B decrypts link layer, reads destination node_addr, re-encrypts
with B↔C link key, forwards to C
4. C decrypts link layer, re-encrypts with C↔D link key, forwards
to D
5. D decrypts link layer, then decrypts session layer to get payload
Intermediate nodes route based on destination node_addr but cannot
read session-layer payloads. Each hop strips one link encryption and
applies the next — the session-layer ciphertext passes through
untouched.
Both layers always apply, even between adjacent peers — a packet to a
direct neighbor is still encrypted twice. This uniform model means no
special cases for local vs remote destinations, and topology changes
(a direct peer becomes reachable only through intermediaries) don't
affect existing sessions.
See [fips-mesh-layer.md](fips-mesh-layer.md) for link encryption and
[fips-session-layer.md](fips-session-layer.md) for session encryption.
## Routing and Mesh Operation
Forwarding decisions are local. Each node combines spanning-tree
coordinates with peer bloom filters to choose a next hop, falling back
to greedy tree routing when bloom filters have not converged. Discovery
warms transit node caches with destination coordinates, and three
explicit error signals (CoordsRequired, PathBroken, MtuExceeded) drive
recovery when forwarding fails. The full routing decision process,
discovery protocol, and error-recovery integration view live in
[fips-mesh-operation.md](fips-mesh-operation.md).
## Transport Abstraction
FIPS treats the communication medium as a pluggable component. UDP,
TCP, raw Ethernet, Tor, and BLE all implement the same small datagram
interface (send, receive, report MTU) and feed peers into a single FMP
routing layer; radio and serial transports are in the planned set.
Multi-transport nodes bridge between networks transparently. The
transport-layer specification — including per-transport categories,
the trait surface, the connection model, and implementation status —
is in [fips-transport-layer.md](fips-transport-layer.md).
## Security
FIPS defends against four adversary classes (transport observers,
active transport attackers, intermediate routers, and adversarial
mesh nodes) through layered controls: hop-by-hop FMP link encryption,
end-to-end FSP session encryption with stronger initiator identity
protection, signed and replay-protected gossip, and rate-limited
handshake processing. The threat-model details and per-layer
mitigations are in [fips-mesh-layer.md](fips-mesh-layer.md), and the
operator-facing controls (default-deny baseline, peer ACLs,
filesystem permissions, cryptographic primitives) are consolidated in
[fips-security.md](fips-security.md) and
[../reference/security.md](../reference/security.md).
## MTU as a Cross-Cutting Concern
MTU is not owned by any single layer. The transport layer reports
per-link MTU, FMP carries `path_mtu` in SessionDatagram and
LookupResponse to track the minimum along a path, FSP echoes the
observed forward-path MTU back to the source, and the IPv6 adapter
enforces the resulting effective MTU at the TUN with ICMP Packet Too
Big and TCP MSS clamping. The unified design — encapsulation overhead
budget, proactive PMTUD, reactive MtuExceeded, and per-destination
storage — is in [fips-mtu.md](fips-mtu.md).
## Approaches Considered but Rejected
One design alternative evaluated and ruled out during the architecture
pass was onion routing, rejected because it requires the sender to
know the full path upfront (incompatible with self-organizing
routing) and prevents per-hop error feedback (incompatible with
CoordsRequired/PathBroken recovery). The canonical mention lives in
[fips-mesh-operation.md](fips-mesh-operation.md#privacy-considerations).
docs/design/fips-bloom-filters.md
+54 -22
View File
@@ -153,6 +153,15 @@ this node, and this node thinks it can reach the same destination through Q.
Split-horizon is computed per-peer: the outbound filter for peer Q merges
all tree peer inbound filters except Q's.
### Filter Propagation Diagram
![Bloom filter propagation on a spanning tree](diagrams/fips-bloom-propagation.svg)
The outbound filter for peer Q merges this node's identity with tree
peer inbound filters except Q's (split-horizon exclusion). Upward
filters (child → parent) contain the child's subtree, while downward
filters (parent → child) contain the complement.
### Directional Asymmetry
Because merge is restricted to tree peers, outgoing filters exhibit
@@ -202,9 +211,10 @@ Filter updates are event-driven, not periodic:
### Rate Limiting
Updates are rate-limited at 500ms minimum interval per peer to prevent
storms during topology changes. Multiple pending changes within the
cooldown period are coalesced into a single announcement.
Updates are rate-limited at a 500ms minimum interval per peer
(`node.bloom.update_debounce_ms`) to prevent storms during topology
changes. Multiple pending changes within the cooldown period are
coalesced into a single announcement.
### Propagation Scope
@@ -249,21 +259,14 @@ Where `filter_bits = 8 × (512 << size_class)` — 8,192 for v1.
## Wire Format
FilterAnnounce messages are carried inside encrypted link-layer frames:
| Offset | Field | Size | Description |
| ------ | ----- | ---- | ----------- |
| 0 | msg_type | 1 byte | 0x20 |
| 1 | sequence | 8 bytes LE | Monotonic counter for freshness |
| 9 | hash_count | 1 byte | Number of hash functions (5 in v1) |
| 10 | size_class | 1 byte | Filter size: `512 << size_class` bytes |
| 11 | filter_bits | 1,024 bytes | Bloom filter bit array (v1) |
**v1 total**: 1,035 bytes payload, 1,064 bytes with link encryption
overhead.
See [fips-wire-formats.md](fips-wire-formats.md) for the complete wire
format reference.
The FilterAnnounce byte layout (`msg_type 0x20`, sequence, hash_count,
size_class, filter_bits) lives in
[../reference/wire-formats.md](../reference/wire-formats.md). The
v1 plaintext payload is 1,035 bytes (11-byte header + 1,024-byte
filter); link encryption adds 36 bytes of FMP framing (16-byte outer
header + 4-byte inner timestamp + 16-byte AEAD tag), bringing the
on-the-wire size to roughly 1,071 bytes before the underlying
transport's per-packet overhead.
## Scale and Size Classes
@@ -320,9 +323,8 @@ The envisioned approach is that hub nodes near the root — which carry the
largest downward filters — would use larger size classes, while leaf nodes
and resource-constrained nodes continue with smaller filters. A node
receiving a filter larger than its own size class folds it down locally.
The mechanism by which heterogeneous filter sizes propagate through the
tree is a future design direction not specified in v1. See
[IDEA-0043](../../ideas/IDEA-0043-heterogeneous-filter-propagation.md).
The mechanism by which heterogeneous filter sizes propagate through
the tree is a future design direction not specified in v1.
### Folding
@@ -335,6 +337,33 @@ The hash function design supports folding: membership tests at a smaller
size use `hash(item, i) % smaller_bit_count`, which maps to the same bit
positions that folding produces.
## Mesh Size Estimation
Each filter's saturation can be inverted into an estimated entry count
via the standard formula `n ≈ -(m/k) · ln(1 − X/m)`, where `m` is the
filter size in bits, `k` is the hash count, and `X` is the population
count. Combining the parent's inbound filter with the children's
inbound filters gives an estimate of the whole network: parent + each
child's subtree are disjoint by construction, and adding 1 for the
node itself yields the total. The result is cached on the node and
exposed through the control socket and `fipstop` dashboard.
The estimator refuses to produce a value when any contributing filter
is above the antipoison FPR cap (`node.bloom.max_inbound_fpr`,
default `0.05`); a partial aggregate would silently underestimate.
Consumers handle the resulting `None` by displaying an "unknown"
state rather than a misleading number.
## Antipoison: Inbound FPR Cap
Inbound `FilterAnnounce` payloads are checked against
`node.bloom.max_inbound_fpr` (default `0.05`). Filters whose
estimated false positive rate exceeds the cap are dropped silently
(no NACK on the wire) — they would otherwise inflate downstream
candidate evaluation cost without contributing useful discrimination.
The cap also gates filters from feeding into mesh size estimation,
as described above.
## Implementation Status
| Feature | Status |
@@ -349,6 +378,8 @@ positions that folding produces.
| 500ms rate limiting | **Implemented** |
| FilterAnnounce gossip (all peers) | **Implemented** |
| Filter cardinality logging | **Implemented** |
| Mesh size estimation (parent + children + 1) | **Implemented** |
| Inbound FPR cap (antipoison) | **Implemented** |
| Size class negotiation | Future direction |
| Folding support | Future direction |
| Adaptive filter sizing | Future direction |
@@ -357,6 +388,7 @@ positions that folding produces.
- [fips-mesh-operation.md](fips-mesh-operation.md) — How bloom filters fit
into routing
- [fips-wire-formats.md](fips-wire-formats.md) — FilterAnnounce wire format
- [../reference/wire-formats.md](../reference/wire-formats.md) —
FilterAnnounce wire format
- [fips-spanning-tree.md](fips-spanning-tree.md) — The coordinate system
that bloom filter candidates are ranked by
docs/design/fips-concepts.md
+123
View File
@@ -0,0 +1,123 @@
# FIPS Concepts
A novice-friendly introduction to what FIPS is, why it exists, and the
mental model behind a self-organizing mesh. For the protocol stack,
identity system, and encryption walkthrough, see
[fips-architecture.md](fips-architecture.md). For prior art and
academic citations, see [fips-prior-work.md](fips-prior-work.md).
## What is FIPS?
FIPS is a self-organizing mesh network that can operate natively over a
variety of physical and logical media, such as local area networks,
Bluetooth, serial links, or the existing internet as an overlay. The
long-term goal is infrastructure that can function alongside or
ultimately replace dependence on the Internet itself. Systems running
FIPS establish peer connections, authenticate each other, and route
traffic for each other without any central authority or global topology
knowledge, and allow end-to-end encrypted sessions between any two
nodes regardless of how many hops separate them.
Nodes in the mesh route traffic for each other using Nostr identities
(npubs) as network addresses. Applications can access the mesh through
a native FIPS datagram service, or through an IPv6 adaptation layer
that presents each node as an IPv6 endpoint for compatibility with
existing IP-based applications.
## Why FIPS?
**Self-sovereign identity**: FIPS nodes generate their own addresses,
node IDs, and security credentials without coordination with any
central authority. These identities can be long-term fixed or may be
ephemeral, changed at any time. These identities are not visible to
the FIPS network itself — they are used only at the application layer
and for end-to-end session encryption.
**Infrastructure independence**: The internet depends on centralized
infrastructure — ISPs, backbone providers, DNS, certificate
authorities. FIPS works over any transport that can carry packets: a
serial connection, onion-routed connections through Tor, local area
networking, radio links between remote sites, or the existing internet
as an overlay. When the internet is unavailable, unreliable, or
untrusted, the mesh still works.
**Privacy by design**: FIPS provides secure, authenticated, and
encrypted communication between any two nodes in the mesh, independent
of the mix of transports used along the routed path between them.
Furthermore, the mesh itself is designed to minimize metadata exposure
— intermediate nodes route packets without learning the identities of
the endpoints.
**Zero configuration**: Nodes discover each other and build routing
automatically. Connect to one peer and you can reach the entire mesh.
The network self-heals around failures and adapts to changing topology.
## A Self-Organizing Mesh
Traditional networks are built top-down. A central authority assigns
addresses, configures routing tables, provisions hardware, and manages
the topology. If the authority disappears or the infrastructure fails,
the network fails with it. Nodes cannot reach each other without
infrastructure mediating the connection.
FIPS inverts this model. There is no central authority, no address
assignment service, no routing table pushed from above. Each node
generates its own identity from a cryptographic keypair. Each node
independently decides which peers to connect to and which transports
to use. From these local decisions alone, the network self-organizes:
- A **spanning tree** forms through distributed parent selection,
giving every node a coordinate in the network without any node
knowing the full topology
- **Bloom filters** propagate through gossip, so each node learns
which peers can reach which destinations — again without global
knowledge
- **Routing decisions** are made locally at each hop, using only the
node's immediate peers and cached coordinate information
Each peer link and end-to-end session actively measures RTT, loss,
jitter, and goodput through a lightweight in-band Metrics Measurement
Protocol (MMP), providing operator visibility and a foundation for
quality-aware routing.
The result is a network that builds itself from the bottom up, heals
around failures automatically, and scales without central coordination.
Adding a node is as simple as connecting to one existing peer — the
network integrates the new node through its normal mesh protocols.
## Specific Design Goals
- **Nostr-native identity and cryptography** — Use Nostr keypairs as
node identities and leverage secp256k1, Schnorr signatures, and
SHA-256
- **Transport agnostic** — Support overlay, shared medium, and
point-to-point transports transparently
- **Self-organizing** — Automatic topology discovery and route
optimization
- **Privacy preserving** — Minimize metadata leakage across untrusted
links
- **Resilient** — Self-healing with graceful degradation
Non-goals include:
- **Reliable delivery** — FIPS provides a best-effort datagram
service; retransmission and ordering are left to applications or
higher-layer protocols
- **Anonymity** — Direct peers learn each other's identity; FIPS
minimizes metadata exposure but is not an anonymity network like Tor
- **Congestion control** — FIPS measures link quality but does not
implement flow control or congestion avoidance at the mesh layer
## Where to Read Next
- [fips-architecture.md](fips-architecture.md) — protocol stack,
identity system, two-layer encryption, MTU as a cross-cutting
concern
- [fips-spanning-tree.md](fips-spanning-tree.md) — how the tree forms
and reconverges
- [fips-bloom-filters.md](fips-bloom-filters.md) — how reachability
information propagates
- [fips-mesh-operation.md](fips-mesh-operation.md) — how the pieces
work together at runtime
- [fips-prior-work.md](fips-prior-work.md) — designs and protocols
FIPS builds on
docs/design/fips-configuration.md
-886
View File
@@ -1,886 +0,0 @@
# FIPS Configuration
FIPS uses YAML-based configuration with a cascading multi-file priority system.
All parameters have sensible defaults; a node can run with no configuration file
at all (it will generate an ephemeral identity and listen on default addresses).
## Configuration Loading
### Search Paths
When started without the `-c` flag, FIPS searches for `fips.yaml` in these
locations, lowest to highest priority:
| Priority | Path | Purpose |
|----------|------|---------|
| 1 (lowest) | `/etc/fips/fips.yaml` | System-wide defaults |
| 2 | `~/.config/fips/fips.yaml` | User preferences |
| 3 | `~/.fips.yaml` | Legacy user config |
| 4 (highest) | `./fips.yaml` | Deployment-specific overrides |
All found files are loaded and merged in priority order. Values from higher
priority files override those from lower priority files. This allows a system
administrator to set site-wide defaults in `/etc/fips/fips.yaml` while
individual deployments override specific values in `./fips.yaml`.
### CLI Option
```text
fips -c /path/to/config.yaml
```
When `-c` is specified, only that file is loaded (search paths are skipped).
### Partial Configuration
Every field has a built-in default. A configuration file only needs to specify
values that differ from defaults. For example, a minimal config might contain
only the identity and peer list, inheriting all other defaults.
## YAML Structure
The configuration is organized into five top-level sections:
```yaml
node: # Node behavior, protocol parameters, and tuning
tun: # TUN virtual interface
dns: # DNS responder for .fips domain
transports: # Network transports (UDP, Ethernet, Bluetooth, Tor, ...)
peers: # Static peer list
```
### Control Socket (`node.control.*`)
| Parameter | Type | Default | Description |
|-----------|------|---------|-------------|
| `node.control.enabled` | bool | `true` | Enable the control socket |
| `node.control.socket_path` | string | *(auto)* | **Linux:** Socket file path. Default: `$XDG_RUNTIME_DIR/fips/control.sock`, then `/run/fips/control.sock` (if root), then `/tmp/fips-control.sock`. **Windows:** TCP port number (default: `21210`); the control socket listens on `localhost` at this port. |
The control socket provides access to node state and runtime management
via the `fipsctl` command-line tool. In addition to read-only status
queries, `fipsctl connect` and `fipsctl disconnect` enable runtime peer
management. See the project [README](../../README.md#inspect) for the
command list.
On Linux, the control socket is a Unix domain socket with filesystem
permissions (mode 0770, group `fips`). On Windows, it is a TCP listener
on localhost. TCP does not provide filesystem-level ACLs, so any local
user can connect to the control port.
> **Security note (Windows):** The TCP control socket on Windows is a
> known limitation. Any process running on the local machine can connect
> to the control port and issue commands, including `disconnect`,
> `connect`, and `inject-config`. This is acceptable for single-user
> workstations but may be inappropriate for shared machines. Future
> improvements may include named pipe support (with Windows ACLs) or an
> authentication token mechanism. On shared Windows systems, consider
> using firewall rules to restrict access to the control port.
All tunable protocol parameters live under `node.*`, organized as sysctl-style
dotted paths. The top-level sections (`tun`, `dns`, `transports`, `peers`)
handle infrastructure concerns only.
## Node Parameters (`node.*`)
### Identity (`node.identity.*`)
| Parameter | Type | Default | Description |
|-----------|------|---------|-------------|
| `node.identity.nsec` | string | *(none)* | Secret key in nsec (bech32) or hex format. If omitted, behavior depends on `persistent`. |
| `node.identity.persistent` | bool | `false` | Persist identity across restarts via key file. |
Identity resolution follows a three-tier priority:
1. **Explicit `nsec`** in config — always used when present, regardless of `persistent`
2. **Persistent key file** — when `persistent: true` and no `nsec`, loads from `fips.key`
adjacent to the config file; if no key file exists, generates a new keypair and saves it
3. **Ephemeral** — when `persistent: false` (default) and no `nsec`, generates a fresh
keypair on each start
Key files (`fips.key` with mode 0600, `fips.pub` with mode 0644) are written adjacent
to the highest-priority config file for operator visibility, even in ephemeral mode.
### General
| Parameter | Type | Default | Description |
|-----------|------|---------|-------------|
| `node.leaf_only` | bool | `false` | Leaf-only mode: node does not forward traffic or participate in routing |
| `node.tick_interval_secs` | u64 | `1` | Periodic maintenance tick interval (retry checks, timeout cleanup, tree refresh) |
| `node.base_rtt_ms` | u64 | `100` | Initial RTT estimate for new links before measurements converge |
| `node.heartbeat_interval_secs` | u64 | `10` | Heartbeat send interval per peer for liveness detection |
| `node.link_dead_timeout_secs` | u64 | `30` | No-traffic timeout before a peer is declared dead and removed |
### Resource Limits (`node.limits.*`)
Controls capacity for connections, peers, and links.
| Parameter | Type | Default | Description |
|-----------|------|---------|-------------|
| `node.limits.max_connections` | usize | `256` | Max handshake-phase connections |
| `node.limits.max_peers` | usize | `128` | Max authenticated peers |
| `node.limits.max_links` | usize | `256` | Max active links |
| `node.limits.max_pending_inbound` | usize | `1000` | Max pending inbound handshakes |
### Rate Limiting (`node.rate_limit.*`)
Handshake rate limiting protects against DoS on the Noise IK handshake path.
| Parameter | Type | Default | Description |
|-----------|------|---------|-------------|
| `node.rate_limit.handshake_burst` | u32 | `100` | Token bucket burst capacity |
| `node.rate_limit.handshake_rate` | f64 | `10.0` | Tokens per second refill rate |
| `node.rate_limit.handshake_timeout_secs` | u64 | `30` | Stale handshake cleanup timeout |
| `node.rate_limit.handshake_resend_interval_ms` | u64 | `1000` | Initial handshake message resend interval |
| `node.rate_limit.handshake_resend_backoff` | f64 | `2.0` | Resend backoff multiplier (1s, 2s, 4s, 8s, 16s with defaults) |
| `node.rate_limit.handshake_max_resends` | u32 | `5` | Max resends per handshake attempt |
### Retry / Backoff (`node.retry.*`)
Connection retry with exponential backoff.
| Parameter | Type | Default | Description |
|-----------|------|---------|-------------|
| `node.retry.max_retries` | u32 | `5` | Max connection retry attempts |
| `node.retry.base_interval_secs` | u64 | `5` | Base backoff interval |
| `node.retry.max_backoff_secs` | u64 | `300` | Cap on exponential backoff (5 minutes) |
Auto-reconnect (triggered by MMP link-dead removal) uses the same backoff
parameters but bypasses `max_retries`, retrying indefinitely. See
`peers[].auto_reconnect` below.
### Cache Parameters (`node.cache.*`)
Controls caching of tree coordinates and identity mappings.
| Parameter | Type | Default | Description |
|-----------|------|---------|-------------|
| `node.cache.coord_size` | usize | `50000` | Max entries in coordinate cache |
| `node.cache.coord_ttl_secs` | u64 | `300` | Coordinate cache entry TTL (5 minutes) |
| `node.cache.identity_size` | usize | `10000` | Max entries in identity cache (LRU, no TTL) |
### Discovery Protocol (`node.discovery.*`)
Controls bloom-guided node discovery (LookupRequest/LookupResponse).
| Parameter | Type | Default | Description |
|-----------|------|---------|-------------|
| `node.discovery.ttl` | u8 | `64` | Hop limit for LookupRequest forwarding |
| `node.discovery.attempt_timeouts_secs` | array&lt;u64&gt; | `[1, 2, 4, 8]` | Per-attempt timeouts. Each entry is the deadline for one `LookupRequest` before sending the next attempt with a fresh `request_id`. Length determines total attempt count; default gives 4 attempts and a 15s total budget |
| `node.discovery.recent_expiry_secs` | u64 | `10` | Dedup cache expiry for recent request IDs |
| `node.discovery.backoff_base_secs` | u64 | `0` | Optional post-failure suppression base in seconds; doubles per consecutive failure. `0` disables (default) — the per-attempt sequence is the only retry pacing |
| `node.discovery.backoff_max_secs` | u64 | `0` | Cap on optional post-failure backoff |
| `node.discovery.forward_min_interval_secs` | u64 | `2` | Transit-side rate limiting: minimum interval between forwarded lookups for the same target |
#### Nostr Overlay Discovery (`node.discovery.nostr.*`)
Optional Nostr-mediated overlay discovery. This layer publishes replaceable
endpoint adverts (`fips-overlay-v1`), consumes advert-derived endpoint
fallbacks for configured peers, and can optionally discover non-configured
peers (`policy: open`). `udp:nat` remains the trigger for NAT traversal
offer/answer + punch-through, after which the established UDP socket is handed
into the normal FIPS transport/session stack.
Inbox-relay discovery falls back to the local DM relay list if remote relay
metadata cannot be fetched.
This support is compiled behind the crate feature `nostr-discovery`; builds
without that feature ignore `udp:nat` bootstrap configuration.
| Parameter | Type | Default | Description |
|-----------|------|---------|-------------|
| `node.discovery.nostr.enabled` | bool | `false` | Enable Nostr-mediated overlay discovery |
| `node.discovery.nostr.policy` | string | `"configured_only"` | Advert discovery policy: `disabled`, `configured_only`, `open` |
| `node.discovery.nostr.open_discovery_max_pending` | usize | `64` | Max open-discovery peers queued in outbound retry/connection state at once |
| `node.discovery.nostr.max_concurrent_incoming_offers` | usize | `16` | Max concurrent inbound traversal offers processed at once (rate limit against offer spam) |
| `node.discovery.nostr.advert_cache_max_entries` | usize | `2048` | Max cached overlay adverts retained from relay traffic |
| `node.discovery.nostr.seen_sessions_max_entries` | usize | `2048` | Max seen-session IDs retained for replay detection |
| `node.discovery.nostr.advertise` | bool | `true` | Publish local endpoint adverts |
| `node.discovery.nostr.advert_relays` | list[string] | `["wss://relay.damus.io", "wss://nos.lol", "wss://offchain.pub"]` | Relays used for service adverts |
| `node.discovery.nostr.dm_relays` | list[string] | `["wss://relay.damus.io", "wss://nos.lol", "wss://offchain.pub"]` | Relays used for encrypted signaling events |
| `node.discovery.nostr.stun_servers` | list[string] | `["stun:stun.l.google.com:19302", "stun:stun.cloudflare.com:3478", "stun:global.stun.twilio.com:3478"]` | STUN servers used for local reflexive address discovery |
| `node.discovery.nostr.app` | string | `"fips-overlay-v1"` | Traversal application namespace and advert identifier suffix |
| `node.discovery.nostr.signal_ttl_secs` | u64 | `120` | Signaling TTL in seconds |
| `node.discovery.nostr.attempt_timeout_secs` | u64 | `10` | Overall traversal attempt timeout in seconds |
| `node.discovery.nostr.replay_window_secs` | u64 | `300` | Replay tracking retention window in seconds |
| `node.discovery.nostr.punch_start_delay_ms` | u64 | `2000` | Delay before punch traffic starts |
| `node.discovery.nostr.punch_interval_ms` | u64 | `200` | Interval between punch packets |
| `node.discovery.nostr.punch_duration_ms` | u64 | `10000` | How long to keep punching before failure |
| `node.discovery.nostr.advert_ttl_secs` | u64 | `3600` | Advert TTL in seconds |
| `node.discovery.nostr.advert_refresh_secs` | u64 | `1800` | How often adverts are refreshed in seconds |
| `node.discovery.nostr.startup_sweep_delay_secs` | u64 | `5` | Settle delay after Nostr discovery starts before the one-shot startup advert sweep runs (only used under `policy: open`). Allows the relay subscription backlog to populate the in-memory advert cache before the sweep fires |
| `node.discovery.nostr.startup_sweep_max_age_secs` | u64 | `3600` | Maximum advert age (`now - created_at`) considered by the one-shot startup sweep (only used under `policy: open`). Adverts older than this are skipped on startup; the per-tick sweep still considers them up to `valid_until_ms` |
If `stun_servers` is omitted, the built-in default list above is used. If it is
specified in YAML, the configured list fully overrides the defaults.
Initiators use only this local list for outbound STUN queries; peer-advertised
STUN values are published for diagnostics/interoperability but are not used as
arbitrary egress targets.
The built-in advert and DM relay defaults point at widely-operated public
relays (Damus, nos.lol, Primal) as best-effort endpoints; operators are
encouraged to override them with their own relay preferences for production
deployments.
Advert freshness is enforced semantically: events with expired NIP-40
`expiration` tags are dropped, and adverts are also bounded by a created-at
staleness window derived from `advert_ttl_secs` (with a grace multiplier).
The current in-tree STUN parser handles IPv4 and IPv6 mapped-address
attributes. Local traversal candidates include active non-loopback private
interface addresses (RFC1918 IPv4 and IPv6 ULA) plus probed local egress
addresses for the punch socket port.
During punching, compatible private-subnet candidates and reflexive candidates
are attempted in parallel; the first successful path wins.
### Spanning Tree (`node.tree.*`)
Controls tree construction and parent selection.
| Parameter | Type | Default | Description |
|----------------------------------------|-------|---------|--------------------------------------------------|
| `node.tree.announce_min_interval_ms` | u64 | `500` | Per-peer TreeAnnounce rate limit |
| `node.tree.parent_hysteresis` | f64 | `0.2` | Cost improvement fraction required for same-root parent switch (0.0–1.0) |
| `node.tree.hold_down_secs` | u64 | `30` | Suppress non-mandatory re-evaluation after parent switch |
| `node.tree.reeval_interval_secs` | u64 | `60` | Periodic cost-based parent re-evaluation interval (0 = disabled) |
| `node.tree.flap_threshold` | u32 | `4` | Parent switches in window before dampening engages |
| `node.tree.flap_window_secs` | u64 | `60` | Sliding window for counting parent switches |
| `node.tree.flap_dampening_secs` | u64 | `120` | Extended hold-down duration when flap threshold exceeded |
### Bloom Filter (`node.bloom.*`)
| Parameter | Type | Default | Description |
|-----------|------|---------|-------------|
| `node.bloom.update_debounce_ms` | u64 | `500` | Debounce interval for filter update propagation |
Bloom filter size (1 KB), hash count (5), and size classes are protocol
constants and not configurable.
### ECN Signaling (`node.ecn.*`)
Controls hop-by-hop ECN (Explicit Congestion Notification) signaling. When
enabled, transit nodes detect congestion on outgoing links (via MMP loss/ETX
metrics or kernel buffer drops) and set the CE flag on forwarded FMP frames.
Destination nodes mark ECN-capable IPv6 packets with CE before TUN delivery
per RFC 3168, enabling end-host TCP congestion control to react.
| Parameter | Type | Default | Description |
|-----------|------|---------|-------------|
| `node.ecn.enabled` | bool | `true` | Enable ECN congestion signaling (CE flag relay and local congestion detection) |
| `node.ecn.loss_threshold` | f64 | `0.05` | MMP loss rate threshold for CE marking (0.0–1.0). When the outgoing link's loss rate meets or exceeds this value, forwarded packets are CE-marked. |
| `node.ecn.etx_threshold` | f64 | `3.0` | MMP ETX threshold for CE marking (≥1.0). When the outgoing link's ETX meets or exceeds this value, forwarded packets are CE-marked. |
Congestion detection triggers on any of: outgoing link loss ≥ `loss_threshold`,
outgoing link ETX ≥ `etx_threshold`, or kernel receive buffer drops detected on
any local transport. CE is relayed hop-by-hop: once set on any hop, the flag
stays set for all subsequent hops to the destination.
### Rekey (`node.rekey.*`)
Controls periodic Noise rekey for forward secrecy. When enabled, both FMP
(link-layer IK) and FSP (session-layer XK) sessions perform fresh Diffie-Hellman
key exchanges after a time or message count threshold, whichever comes first.
A 10-second drain window keeps the old session active for decryption during
cutover.
| Parameter | Type | Default | Description |
|-----------|------|---------|-------------|
| `node.rekey.enabled` | bool | `true` | Enable periodic Noise rekey on all links and sessions |
| `node.rekey.after_secs` | u64 | `120` | Initiate rekey after this many seconds on a session |
| `node.rekey.after_messages` | u64 | `65536` | Initiate rekey after this many messages sent on a session |
### Session / Data Plane (`node.session.*`)
Controls end-to-end session behavior and packet queuing.
| Parameter | Type | Default | Description |
|-----------|------|---------|-------------|
| `node.session.default_ttl` | u8 | `64` | Default SessionDatagram TTL |
| `node.session.pending_packets_per_dest` | usize | `16` | Queue depth per destination during session establishment |
| `node.session.pending_max_destinations` | usize | `256` | Max destinations with pending packets |
| `node.session.idle_timeout_secs` | u64 | `90` | Idle session timeout; established sessions with no application data for this duration are removed. MMP reports (SenderReport, ReceiverReport, PathMtuNotification) do not count as activity |
| `node.session.coords_warmup_packets` | u8 | `5` | Number of initial data packets per session that include the CP flag for transit cache warmup; also the reset count on CoordsRequired/PathBroken receipt |
| `node.session.coords_response_interval_ms` | u64 | `2000` | Minimum interval (ms) between standalone CoordsWarmup responses to CoordsRequired/PathBroken signals per destination |
The anti-replay window size (2048 packets) is a compile-time constant and not
configurable.
### Link-Layer MMP (`node.mmp.*`)
Metrics Measurement Protocol for per-peer link measurement. See
[fips-mesh-layer.md](fips-mesh-layer.md) for behavioral details.
| Parameter | Type | Default | Description |
|-----------|------|---------|-------------|
| `node.mmp.mode` | string | `"full"` | Operating mode: `full` (sender + receiver reports), `lightweight` (receiver reports only), or `minimal` (spin bit + CE echo only, no reports) |
| `node.mmp.log_interval_secs` | u64 | `30` | Periodic operator log interval for link metrics |
| `node.mmp.owd_window_size` | usize | `32` | One-way delay trend ring buffer size |
### Session-Layer MMP (`node.session_mmp.*`)
Metrics Measurement Protocol for end-to-end session measurement. Configured
independently from link-layer MMP because session reports are routed through
every transit link, consuming bandwidth proportional to path length.
| Parameter | Type | Default | Description |
|-----------|------|---------|-------------|
| `node.session_mmp.mode` | string | `"full"` | Operating mode: `full`, `lightweight`, or `minimal` |
| `node.session_mmp.log_interval_secs` | u64 | `30` | Periodic operator log interval for session metrics |
| `node.session_mmp.owd_window_size` | usize | `32` | One-way delay trend ring buffer size |
### Internal Buffers (`node.buffers.*`)
Channel sizes affecting throughput and memory. Primarily useful for performance
tuning under high load or on memory-constrained devices.
| Parameter | Type | Default | Description |
|-----------|------|---------|-------------|
| `node.buffers.packet_channel` | usize | `1024` | Transport to Node packet channel capacity |
| `node.buffers.tun_channel` | usize | `1024` | TUN to Node outbound channel capacity |
| `node.buffers.dns_channel` | usize | `64` | DNS to Node identity channel capacity |
## TUN Interface (`tun.*`)
| Parameter | Type | Default | Description |
|-----------|------|---------|-------------|
| `tun.enabled` | bool | `false` | Enable TUN virtual interface |
| `tun.name` | string | `"fips0"` | Interface name |
| `tun.mtu` | u16 | `1280` | Interface MTU (IPv6 minimum) |
## DNS Responder (`dns.*`)
Resolves `<npub>.fips` queries to FIPS IPv6 addresses. Resolution is pure
computation (npub to public key to address); resolved identities are registered
with the node for routing.
| Parameter | Type | Default | Description |
|-----------|------|---------|-------------|
| `dns.enabled` | bool | `true` | Enable DNS responder |
| `dns.bind_addr` | string | `"127.0.0.1"` | Bind address |
| `dns.port` | u16 | `5354` | Listen port |
| `dns.ttl` | u32 | `300` | AAAA record TTL in seconds |
The `dns.ttl` value should not exceed `node.cache.coord_ttl_secs` to avoid
stale address mappings.
### Host Mapping
The DNS resolver checks a host map before falling back to direct npub
resolution, enabling names like `gateway.fips` instead of `npub1...fips`.
The host map is populated from two sources:
1. **Peer aliases** — the `alias` field on configured peers in `peers:`.
2. **Hosts file** — `/etc/fips/hosts`, one `hostname npub1...` per line.
Blank lines and `#` comments are allowed.
The hosts file is auto-reloaded on modification (mtime change) without
restarting the daemon. Hostnames are case-insensitive.
## Transports (`transports.*`)
### UDP (`transports.udp.*`)
| Parameter | Type | Default | Description |
|-----------|------|---------|-------------|
| `transports.udp.bind_addr` | string | `"0.0.0.0:2121"` | UDP bind address and port |
| `transports.udp.mtu` | u16 | `1280` | Transport MTU |
| `transports.udp.recv_buf_size` | usize | `2097152` | UDP socket receive buffer size in bytes (2 MB). Linux kernel doubles the requested value internally. Host `net.core.rmem_max` must be >= this value. |
| `transports.udp.send_buf_size` | usize | `2097152` | UDP socket send buffer size in bytes (2 MB). Host `net.core.wmem_max` must be >= this value. |
| `transports.udp.advertise_on_nostr` | bool | `false` | Include this UDP transport in Nostr endpoint adverts |
| `transports.udp.public` | bool | `false` | If advertised: `true` publishes direct `host:port`; `false` publishes `udp:nat` rendezvous |
### Ethernet (`transports.ethernet.*`)
Ethernet transport sends raw frames via AF_PACKET SOCK_DGRAM sockets.
Requires `CAP_NET_RAW` or running as root. Linux only.
| Parameter | Type | Default | Description |
|-----------|------|---------|-------------|
| `interface` | string | *(required)* | Network interface name (e.g., `"eth0"`, `"enp3s0"`) |
| `ethertype` | u16 | `0x2121` | EtherType |
| `mtu` | u16 | *(auto)* | Override MTU. Default: interface MTU minus 3 (for frame type + length prefix) |
| `recv_buf_size` | usize | `2097152` | Socket receive buffer size in bytes (2 MB) |
| `send_buf_size` | usize | `2097152` | Socket send buffer size in bytes (2 MB) |
| `discovery` | bool | `true` | Listen for discovery beacons from other nodes |
| `announce` | bool | `false` | Broadcast announcement beacons on the LAN |
| `auto_connect` | bool | `false` | Auto-connect to discovered peers |
| `accept_connections` | bool | `false` | Accept incoming connection attempts from discovered peers |
| `beacon_interval_secs` | u64 | `30` | Announcement beacon interval in seconds (minimum 10) |
**Named instances.** Multiple Ethernet interfaces can be configured by
using named sub-keys instead of flat parameters:
```yaml
transports:
ethernet:
lan:
interface: "eth0"
discovery: true
announce: true
backbone:
interface: "eth1"
announce: false
```
Each named instance operates independently with its own socket and
discovery state. The instance name is used in log messages and the
`name()` method on the Transport trait.
### TCP (`transports.tcp.*`)
TCP transport enables firewall traversal on networks that block UDP but
allow TCP (e.g., port 443). Uses FMP header-based framing with zero
overhead.
| Parameter | Type | Default | Description |
|-----------|------|---------|-------------|
| `transports.tcp.bind_addr` | string | *(none)* | Listen address (e.g., `"0.0.0.0:8443"`). If omitted, outbound-only mode. |
| `transports.tcp.mtu` | u16 | `1400` | Default MTU. Per-connection MTU derived from `TCP_MAXSEG` when available. |
| `transports.tcp.connect_timeout_ms` | u64 | `5000` | Outbound connect timeout in milliseconds |
| `transports.tcp.nodelay` | bool | `true` | `TCP_NODELAY` (disable Nagle for low latency) |
| `transports.tcp.keepalive_secs` | u64 | `30` | TCP keepalive interval in seconds (0 = disabled) |
| `transports.tcp.recv_buf_size` | usize | `2097152` | Socket receive buffer size in bytes (2 MB) |
| `transports.tcp.send_buf_size` | usize | `2097152` | Socket send buffer size in bytes (2 MB) |
| `transports.tcp.max_inbound_connections` | usize | `256` | Maximum simultaneous inbound connections |
**Named instances.** Like other transports, multiple TCP instances can
be configured with named sub-keys:
```yaml
transports:
tcp:
public:
bind_addr: "0.0.0.0:443"
internal:
bind_addr: "10.0.0.1:8443"
max_inbound_connections: 64
```
### Tor (`transports.tor.*`)
Tor transport routes FIPS traffic through the Tor network for anonymity.
Requires an external Tor daemon providing a SOCKS5 proxy. Three modes:
`socks5` for outbound-only, `control_port` for outbound + monitoring,
`directory` for outbound + inbound via Tor-managed onion service.
| Parameter | Type | Default | Description |
|-----------|------|---------|-------------|
| `transports.tor.mode` | string | `"socks5"` | Tor access mode: `socks5` (outbound only), `control_port` (outbound + monitoring), or `directory` (outbound + inbound onion service) |
| `transports.tor.socks5_addr` | string | `"127.0.0.1:9050"` | SOCKS5 proxy address (host:port) |
| `transports.tor.connect_timeout_ms` | u64 | `120000` | Connect timeout in milliseconds. Tor circuits take 10–60s. |
| `transports.tor.mtu` | u16 | `1400` | Default MTU |
| `transports.tor.control_addr` | string | `"/run/tor/control"` | Tor control port address: Unix socket path or host:port. Used in `control_port` mode; optional in `directory` mode for monitoring. |
| `transports.tor.control_auth` | string | `"cookie"` | Control port authentication: `"cookie"`, `"cookie:/path/to/cookie"`, or `"password:<secret>"`. |
| `transports.tor.cookie_path` | string | `"/var/run/tor/control.authcookie"` | Path to Tor control cookie file. Used when `control_auth` is `"cookie"`. |
| `transports.tor.max_inbound_connections` | usize | `64` | Maximum inbound connections via onion service. |
| `transports.tor.directory_service.hostname_file` | string | `"/var/lib/tor/fips_onion_service/hostname"` | Path to Tor-managed hostname file containing the `.onion` address. |
| `transports.tor.directory_service.bind_addr` | string | `"127.0.0.1:8443"` | Local bind address for the listener that Tor forwards inbound connections to. Must match `HiddenServicePort` target in `torrc`. |
| `transports.tor.advertised_port` | u16 | `443` | Public-facing onion port published in Nostr overlay adverts. Must match the virtual port in torrc's `HiddenServicePort <port> 127.0.0.1:<bind_port>` directive — that is the port other peers will use to reach this onion. |
**Named instances.** Like other transports, multiple Tor instances can
be configured with named sub-keys for different SOCKS5 proxy endpoints.
**Directory mode** (recommended for production). Tor manages the onion
service via `HiddenServiceDir` in `torrc`. FIPS reads the `.onion`
address from the hostname file and binds a local TCP listener. This
enables Tor's `Sandbox 1` (seccomp-bpf). If `control_addr` is also
set, the transport connects to the control port for daemon monitoring
(non-fatal on failure).
**Control port mode.** Connects to the Tor daemon's control port for
monitoring only (bootstrap status, circuit health, traffic stats).
No inbound connections. Both `control_addr` and `control_auth` are
required.
### UDP + Tor Bridge Example
A node bridging clearnet (UDP) and anonymous (Tor) portions of the mesh:
```yaml
node:
identity:
persistent: true
tun:
enabled: true
transports:
udp:
bind_addr: "0.0.0.0:2121"
mtu: 1472
tor:
socks5_addr: "127.0.0.1:9050"
peers:
- npub: "npub1abc..."
alias: "clearnet-peer"
addresses:
- transport: udp
addr: "203.0.113.5:2121"
- npub: "npub1def..."
alias: "anonymous-peer"
addresses:
- transport: tor
addr: "abc123...xyz.onion:2121"
```
### Tor Directory Mode Example
A node accepting inbound connections via Tor-managed onion service
(recommended for production — enables Sandbox 1):
```yaml
node:
identity:
persistent: true
tun:
enabled: true
transports:
tor:
mode: "directory"
socks5_addr: "127.0.0.1:9050"
control_addr: "/run/tor/control" # optional, for monitoring
control_auth: "cookie"
directory_service:
hostname_file: "/var/lib/tor/fips/hostname"
bind_addr: "127.0.0.1:8444"
peers:
- npub: "npub1abc..."
alias: "tor-peer"
addresses:
- transport: tor
addr: "abcdef...xyz.onion:8443"
```
Requires a corresponding `torrc`:
```text
HiddenServiceDir /var/lib/tor/fips
HiddenServicePort 8443 127.0.0.1:8444
```
### BLE (`transports.ble.*`)
Bluetooth Low Energy transport using L2CAP Connection-Oriented Channels.
Requires BlueZ and the `ble` Cargo feature flag (default-on). Linux only;
guarded by `#[cfg(target_os = "linux")]`. Communicates with BlueZ via D-Bus
using the `bluer` crate.
| Parameter | Type | Default | Description |
|-----------|------|---------|-------------|
| `transports.ble.adapter` | string | `"hci0"` | HCI adapter name |
| `transports.ble.psm` | u16 | `0x0085` (133) | L2CAP Protocol/Service Multiplexer |
| `transports.ble.mtu` | u16 | `2048` | Default MTU. Actual MTU is negotiated per-link during L2CAP connection setup. |
| `transports.ble.max_connections` | usize | `7` | Maximum concurrent BLE connections |
| `transports.ble.connect_timeout_ms` | u64 | `10000` | Outbound connect timeout in milliseconds |
| `transports.ble.advertise` | bool | `true` | Broadcast BLE beacon advertisements for peer discovery |
| `transports.ble.scan` | bool | `true` | Listen for BLE beacon advertisements from other nodes |
| `transports.ble.auto_connect` | bool | `false` | Automatically connect to discovered peers |
| `transports.ble.accept_connections` | bool | `true` | Accept incoming L2CAP connections |
| `transports.ble.probe_cooldown_secs` | u64 | `30` | Cooldown before re-probing the same BLE address |
**Address format.** BLE peer addresses use the form
`"adapter/device_address"` — for example, `"hci0/AA:BB:CC:DD:EE:FF"`.
**Advertising and scanning.** When `advertise` is enabled, the transport
advertises the FIPS service UUID continuously so that nearby nodes can
discover and connect via L2CAP. When `scan` is enabled, the transport
continuously scans for other FIPS nodes' advertisements. Discovered
peers are probed immediately (L2CAP connect + pubkey exchange) with a
cooldown (`probe_cooldown_secs`) to prevent rapid re-probing of the same
address. If two nodes probe each other at the same time (cross-probe),
a deterministic tie-breaker based on NodeAddr comparison ensures only
one connection is established.
**Connection pool.** The `max_connections` parameter limits the number of
concurrent BLE connections. When the pool is full, the least-recently-used
connection is evicted to make room for new connections.
### BLE Example
A node using BLE for local mesh discovery alongside UDP for internet peers:
```yaml
node:
identity:
persistent: true
tun:
enabled: true
transports:
udp:
bind_addr: "0.0.0.0:2121"
ble:
adapter: "hci0"
advertise: true
scan: true
auto_connect: true
accept_connections: true
peers:
- npub: "npub1abc..."
alias: "internet-peer"
addresses:
- transport: udp
addr: "203.0.113.5:2121"
connect_policy: auto_connect
```
BLE peers on the local radio range are discovered automatically via
beacons — no static peer entries needed. Internet peers still require
explicit configuration.
## Peers (`peers[]`)
Static peer list. Each entry defines a peer to connect to.
| Parameter | Type | Default | Description |
|-----------|------|---------|-------------|
| `peers[].npub` | string | *(required)* | Peer's Nostr public key (npub-encoded) |
| `peers[].alias` | string | *(none)* | Human-readable name for logging |
| `peers[].addresses[].transport` | string | *(required)* | Transport type: `udp`, `tcp`, `ethernet`, `tor`, or `ble` |
| `peers[].addresses[].addr` | string | *(required)* | Transport address. UDP/TCP: `"host:port"` (IP or DNS hostname). Ethernet: `"interface/mac"` (e.g., `"eth0/aa:bb:cc:dd:ee:ff"`). BLE: `"adapter/device_address"` (e.g., `"hci0/AA:BB:CC:DD:EE:FF"`). Tor: `".onion:port"` or `"host:port"` |
| `peers[].addresses[].priority` | u8 | `100` | Address priority (lower = preferred) |
| `peers[].connect_policy` | string | `"auto_connect"` | Connection policy: `auto_connect`, `on_demand`, or `manual` |
| `peers[].auto_reconnect` | bool | `true` | Automatically reconnect after MMP link-dead removal (exponential backoff, unlimited retries) |
| `peers[].via_nostr` | bool | `false` | Append Nostr advert-derived endpoints after static addresses for this peer |
## Minimal Example
A typical node configuration enabling TUN, DNS, and a single peer:
```yaml
node:
identity:
nsec: "0102030405060708090a0b0c0d0e0f101112131415161718191a1b1c1d1e1f20"
tun:
enabled: true
name: fips0
mtu: 1280
dns:
enabled: true
bind_addr: "127.0.0.1"
port: 53
transports:
udp:
bind_addr: "0.0.0.0:2121"
mtu: 1472
peers:
- npub: "npub1tdwa4vjrjl33pcjdpf2t4p027nl86xrx24g4d3avg4vwvayr3g8qhd84le"
alias: "node-b"
addresses:
- transport: udp
addr: "172.20.0.11:2121"
connect_policy: auto_connect
```
### Mixed UDP + Ethernet Example
A node bridging internet peers (UDP) and a local Ethernet segment with
beacon discovery:
```yaml
node:
identity:
nsec: "0102030405060708090a0b0c0d0e0f101112131415161718191a1b1c1d1e1f20"
tun:
enabled: true
transports:
udp:
bind_addr: "0.0.0.0:2121"
mtu: 1472
ethernet:
interface: "eth0"
discovery: true
announce: true
auto_connect: true
accept_connections: true
peers:
- npub: "npub1tdwa4vjrjl33pcjdpf2t4p027nl86xrx24g4d3avg4vwvayr3g8qhd84le"
alias: "internet-peer"
addresses:
- transport: udp
addr: "203.0.113.5:2121"
connect_policy: auto_connect
```
Ethernet peers on the local segment are discovered automatically via
beacons — no static peer entries needed. Internet peers still require
explicit configuration.
All `node.*` parameters use their defaults. To override specific values, add
only the relevant sections:
```yaml
node:
identity:
nsec: "..."
limits:
max_peers: 64
retry:
max_retries: 10
max_backoff_secs: 600
cache:
coord_size: 100000
```
## Complete Reference
The full YAML structure with all defaults:
```yaml
node:
identity:
nsec: null # secret key in nsec or hex (null = depends on persistent)
persistent: false # true = load/save fips.key; false = ephemeral each start
leaf_only: false
tick_interval_secs: 1
base_rtt_ms: 100
heartbeat_interval_secs: 10
link_dead_timeout_secs: 30
limits:
max_connections: 256
max_peers: 128
max_links: 256
max_pending_inbound: 1000
rate_limit:
handshake_burst: 100
handshake_rate: 10.0
handshake_timeout_secs: 30
handshake_resend_interval_ms: 1000
handshake_resend_backoff: 2.0
handshake_max_resends: 5
retry:
max_retries: 5
base_interval_secs: 5
max_backoff_secs: 300
cache:
coord_size: 50000
coord_ttl_secs: 300
identity_size: 10000
discovery:
ttl: 64
attempt_timeouts_secs: [1, 2, 4, 8]
recent_expiry_secs: 10
backoff_base_secs: 0
backoff_max_secs: 0
forward_min_interval_secs: 2
tree:
announce_min_interval_ms: 500
parent_hysteresis: 0.2 # cost improvement fraction for parent switch
hold_down_secs: 30 # suppress re-evaluation after switch
reeval_interval_secs: 60 # periodic cost-based re-evaluation (0 = disabled)
flap_threshold: 4 # parent switches before dampening
flap_window_secs: 60 # sliding window for flap detection
flap_dampening_secs: 120 # extended hold-down on flap
bloom:
update_debounce_ms: 500
session:
default_ttl: 64
pending_packets_per_dest: 16
pending_max_destinations: 256
idle_timeout_secs: 90
coords_warmup_packets: 5
coords_response_interval_ms: 2000
mmp:
mode: full # full | lightweight | minimal
log_interval_secs: 30
owd_window_size: 32
session_mmp:
mode: full # full | lightweight | minimal
log_interval_secs: 30
owd_window_size: 32
ecn:
enabled: true # ECN congestion signaling (CE flag relay)
loss_threshold: 0.05 # MMP loss rate threshold for CE marking (5%)
etx_threshold: 3.0 # MMP ETX threshold for CE marking
rekey:
enabled: true # periodic Noise rekey for forward secrecy
after_secs: 120 # rekey interval (seconds)
after_messages: 65536 # rekey after N messages sent
control:
enabled: true
socket_path: null # null = auto ($XDG_RUNTIME_DIR → /run/fips → /tmp fallback)
buffers:
packet_channel: 1024
tun_channel: 1024
dns_channel: 64
tun:
enabled: false
name: "fips0"
mtu: 1280
dns:
enabled: true
bind_addr: "127.0.0.1"
port: 5354
ttl: 300
transports:
udp:
bind_addr: "0.0.0.0:2121"
mtu: 1280
recv_buf_size: 2097152 # 2 MB (kernel doubles to 4 MB actual)
send_buf_size: 2097152 # 2 MB
# ethernet: # uncomment to enable (requires CAP_NET_RAW)
# interface: "eth0" # required: network interface name
# ethertype: 0x2121 # default EtherType
# mtu: null # null = interface MTU - 3 (typically 1497)
# recv_buf_size: 2097152 # 2 MB
# send_buf_size: 2097152 # 2 MB
# discovery: true # listen for beacons
# announce: false # broadcast beacons
# auto_connect: false # connect to discovered peers
# accept_connections: false # accept inbound handshakes
# beacon_interval_secs: 30 # beacon interval (min 10)
# tcp: # uncomment to enable TCP transport
# bind_addr: "0.0.0.0:8443" # listen address (omit for outbound-only)
# mtu: 1400 # default MTU
# connect_timeout_ms: 5000 # outbound connect timeout
# nodelay: true # TCP_NODELAY
# keepalive_secs: 30 # keepalive interval (0 = disabled)
# recv_buf_size: 2097152 # 2 MB
# send_buf_size: 2097152 # 2 MB
# max_inbound_connections: 256 # resource protection limit
# tor: # uncomment to enable Tor transport
# mode: "socks5" # "socks5", "control_port", or "directory"
# socks5_addr: "127.0.0.1:9050" # SOCKS5 proxy address
# connect_timeout_ms: 120000 # connect timeout (120s for Tor circuits)
# mtu: 1400 # default MTU
# # monitoring (control_port mode, or optional in directory mode):
# # control_addr: "/run/tor/control" # Unix socket or host:port
# # control_auth: "cookie" # "cookie" or "password:<secret>"
# # cookie_path: "/var/run/tor/control.authcookie"
# # directory mode (inbound via Tor-managed onion service):
# # directory_service:
# # hostname_file: "/var/lib/tor/fips/hostname"
# # bind_addr: "127.0.0.1:8444"
# # max_inbound_connections: 64
# ble: # uncomment to enable BLE transport (Linux only, requires BlueZ)
# adapter: "hci0" # HCI adapter name
# psm: 0x0085 # L2CAP PSM (133)
# mtu: 2048 # default MTU (negotiated per-link)
# max_connections: 7 # max concurrent BLE connections
# connect_timeout_ms: 10000 # outbound connect timeout
# advertise: true # broadcast BLE beacons
# scan: true # listen for BLE beacons
# auto_connect: false # connect to discovered peers
# accept_connections: true # accept incoming L2CAP connections
# probe_cooldown_secs: 30 # cooldown before re-probing same address
peers: # static peer list
# - npub: "npub1..."
# alias: "node-b"
# addresses:
# - transport: udp
# addr: "10.0.0.2:2121"
# priority: 100
# connect_policy: auto_connect
# auto_reconnect: true # reconnect after link-dead removal
```
docs/design/fips-gateway.md
+467 -329
View File
@@ -1,398 +1,536 @@
# FIPS Outbound LAN Gateway
# FIPS Gateway
`fips-gateway` is a sidecar binary that runs alongside the FIPS daemon, enabling
unmodified LAN hosts to reach mesh destinations. It provides DNS resolution of
`.fips` names to virtual IPv6 addresses from a managed pool and configures
kernel nftables NAT rules for traffic forwarding through the fips0 TUN
interface. LAN clients need no FIPS software — any device that can resolve DNS
and send IPv6 packets can use the gateway.
The FIPS gateway lets unmodified IPv6 hosts on a LAN exchange traffic
with the mesh without running any FIPS software themselves. It is a
niche feature — most operators will never enable it. The gateway
runs most conveniently on a system that is already providing network
services (DHCP, DNS, RA) to a LAN segment, since hosts on that
segment already get IP assignment and a default route from that box.
The canonical example is an OpenWrt-based WiFi access point: every
client that associates with the AP already has the AP as default
router and DNS server, which is exactly the placement the gateway
needs. The OpenWrt ipk ships with the `gateway:` block of
`/etc/fips/fips.yaml` pre-populated and the integration glue
(dnsmasq forwarding, RA route for the virtual pool, global-scope
IPv6 prefix on `br-lan`) automated by the init script —
[`packaging/openwrt-ipk/files/etc/init.d/fips-gateway`](https://github.com/jmcorgan/fips/blob/master/packaging/openwrt-ipk/files/etc/init.d/fips-gateway).
The operator only needs to enable and start the service. Running the
gateway on a non-OpenWrt LAN-edge host (a Linux router/server, for
example) is technically possible but requires manual integration:
distributing a route to the virtual-IP pool, wiring DNS forwarding so
LAN clients send `.fips` queries to the gateway, configuring sysctls
and capabilities. That path is supported but tedious; it is the
secondary path.
The feature has two halves that share common machinery and have
their own unique parts.
The **outbound half** carries traffic from LAN to mesh. A non-FIPS
LAN workstation resolves `<npub>.fips` (or a `.fips` host alias) via
the gateway's DNS proxy, which returns a virtual IPv6 address from a
managed pool. The kernel routes the LAN packet to that virtual IP via
a route to the pool CIDR (RA-advertised, statically distributed, or
on-link via the default route). The gateway runs nftables NAT so the
packet appears on the mesh as if it had originated from the gateway's
own FIPS identity: prerouting DNAT rewrites the destination from the
virtual IP to the real `fd00::/8` mesh address, and postrouting
masquerade rewrites the source from the LAN host's address to the
gateway's `fips0` address. Return traffic follows the conntrack
reverse path back to the originating LAN host, with postrouting SNAT
restoring the virtual IP as source so the client sees a response from
the address it connected to.
The **inbound half** carries traffic from mesh to LAN. A
configuration entry in `gateway.port_forwards[]` exposes a LAN
service (`host:port`) on a port of the gateway's mesh-side `fips0`
address. Mesh peers reach it as `<gateway-npub>.fips:<listen_port>`.
A prerouting DNAT rule keyed on `(iif=fips0, l4proto, dport)`
rewrites the destination to the LAN target; a LAN-side masquerade in
postrouting rewrites the mesh peer's source so the LAN target sees a
reachable LAN address and conntrack steers replies back through the
gateway. This is the inverse of port-forwarding on a conventional NAT
router.
The two halves are independent and can be configured separately.
Inbound port-forwards work without any outbound configuration (just
a port-forward list and the table); outbound works without any
inbound forwards. They share the same nftables table, the same
binary, the same control socket, and the same atomic-rebuild
strategy. That shared machinery is what makes them halves of one
feature rather than two separate features.
## Architecture
The gateway is a separate binary (`fips-gateway`), not part of the FIPS daemon.
It connects to the daemon indirectly: `.fips` DNS queries are forwarded to the
daemon's built-in resolver (localhost:5354), which resolves names to mesh
addresses and primes its identity cache as a side effect. The gateway then
allocates a virtual IP, installs NAT rules, and returns the virtual IP to the
LAN client.
### The `fips-gateway` Service
The gateway is a separate binary, [`fips-gateway`](https://github.com/jmcorgan/fips/blob/master/src/bin/fips-gateway.rs),
not part of the FIPS daemon. It reads the same `/etc/fips/fips.yaml`
the daemon reads (via `--config`, or the standard search path), but
acts on the `gateway.*` block. It needs `CAP_NET_ADMIN` to install
nftables rules, manage proxy NDP entries, and add the pool route.
The CLI is documented in
[../reference/cli-fips-gateway.md](../reference/cli-fips-gateway.md).
The gateway connects to the daemon indirectly. The outbound half
forwards `.fips` DNS queries to the daemon's built-in resolver
(default `[::1]:5354`); the daemon resolves the name to a mesh
address and primes its identity cache as a side effect. The inbound
half does not require any daemon plumbing at all — packets that
arrive on `fips0` after the daemon's TUN injection path are matched
by the nftables rules on `fips0` ingress. There is no shared memory,
no IPC channel, and no startup ordering coupling beyond "the daemon's
DNS responder must be reachable before the gateway starts serving
LAN queries", which the gateway enforces with a bounded reachability
probe at startup.
### nftables Table Layout
All gateway rules live in a single nftables table, `inet
fips_gateway`, with two chains:
- `prerouting` — `type nat hook prerouting priority dstnat (-100)`,
for both LAN→mesh DNAT (per virtual-IP mapping) and mesh→LAN DNAT
(per port-forward).
- `postrouting` — `type nat hook postrouting priority srcnat (100)`,
for both the always-on `oifname fips0` masquerade, the per-mapping
return-path SNAT, and (when any port-forward is configured) the
LAN-side masquerade for inbound traffic.
The table is rebuilt atomically on every change. The rebuild
sequence — delete the existing table (ignore `ENOENT` on first
call), then create a new table with chains and the full rule set in
a single netlink batch — avoids reliance on kernel rule-handle
tracking, which the rustables crate does not expose. The table stays
small (one always-on masquerade plus two rules per active outbound
mapping plus one rule per inbound forward, with one extra masquerade
when any forward is present), so rebuilds are cheap.
### Control Socket
`fips-gateway` exposes a Unix-domain control socket at
`/run/fips/gateway.sock` (`root:fips`, mode `0770`) with two
commands: `show_gateway` and `show_mappings`. The protocol is the
same line-delimited JSON used by the daemon's control socket. The
shapes are documented in the
[Gateway command catalog](../reference/control-socket.md#gateway-command-catalog).
There is no `fipsctl gateway` subcommand; clients (including
`fipstop`'s gateway view) talk to the socket directly.
### Diagram
```text
LAN Client
|
DNS query (.fips)
|
v
+---------------------+
| DNS Proxy | Listens on [::]:53
| (dns.rs) |
+---------------------+
| |
.fips query non-.fips → REFUSED
|
v
FIPS Daemon Resolver
(localhost:5354)
|
mesh address (fd00::/8)
|
v
+---------------------+
| Virtual IP Pool | Allocates from pool CIDR
| (pool.rs) |
+---------------------+
|
pool event (new/removed mapping)
|
v
+----------+----------+
| |
v v
+---------------+ +-----------------+
| NAT Manager | | Network Setup |
| (nat.rs) | | (net.rs) |
| DNAT/SNAT/ | | Proxy NDP, |
| masquerade | | pool route |
+---------------+ +-----------------+
| |
v v
nftables rules ip -6 neigh proxy
(inet fips_gateway) ip -6 route local
LAN clients
│
DNS query (.fips) │ IPv6 packet
for outbound │ to virtual IP
│ or mesh peer
▼
┌───────────────────────────────────┐
│ fips-gateway │
│ │
│ ┌──────────────┐ ┌───────────┐ │
│ │ DNS proxy │ │ Virtual │ │
│ │ ([::]:53) │─▶│ IP pool │ │
│ │ .fips only │ │ (state │ │
│ └──────┬───────┘ │ machine) │ │
│ │ └─────┬─────┘ │
│ │ │ │
│ forward to │ pool │
│ daemon resolver │ events │
│ ([::1]:5354) ▼ │
│ │ ┌───────────┐ │
│ │ │ NAT │ │
│ │ │ manager │ │
│ │ │ (rebuild │ │
│ │ │ inet │ │
│ │ │ fips_ │ │
│ │ │ gateway) │ │
│ │ └─────┬─────┘ │
│ │ │ │
│ │ ┌─────▼─────┐ │
│ │ │ net │ │
│ │ │ setup │ │
│ │ │ (proxy │ │
│ │ │ NDP, lo │ │
│ │ │ route) │ │
│ │ └───────────┘ │
│ │ │
│ │ control socket │
│ │ /run/fips/ │
│ │ gateway.sock │
└─────────┼─────────────────────────┘
│
▼
FIPS daemon resolver
([::1]:5354)
│
▼
fips0 TUN interface
│
▼
the mesh
```
### Data Flow
The DNS proxy and the virtual IP pool are exclusive to the outbound
half. The NAT manager and the kernel-side machinery (nftables table,
`fips0` and LAN interfaces, conntrack) are shared. The inbound half
contributes per-port-forward rules to the same table without
involving the DNS proxy or the pool.
1. LAN client queries `hostname.fips` via DNS
2. Gateway forwards to daemon resolver (localhost:5354)
3. Daemon resolves name to mesh address (fd00::/8), primes identity cache
4. Gateway allocates virtual IP from pool, creates DNAT/SNAT rules and proxy NDP
entry
5. Gateway returns AAAA record with virtual IP to client
6. Client sends traffic to virtual IP
7. Kernel DNAT rewrites destination to mesh address, masquerade rewrites source
to gateway's fips0 address
8. Traffic flows through fips0 into the mesh
9. Return traffic follows the reverse path via conntrack
## The Outbound Half (LAN → Mesh)
## NAT Pipeline
### DNS Resolution Flow
The gateway manages a dedicated nftables table (`inet fips_gateway`) containing
two chains with rules that translate between virtual IPs and mesh addresses.
1. A LAN client sends a DNS query to the gateway's listener (default
`[::]:53`, configurable via `gateway.dns.listen`).
2. If the question is not for a `.fips` domain, the gateway replies
`REFUSED`. The proxy is intentionally narrow — it does not resolve
public DNS, and the LAN's primary resolver should hold port 53 on
the gateway host (the OpenWrt init script wires this up by binding
the gateway listener to a non-conflicting port and configuring
dnsmasq to forward `.fips` queries there).
3. The gateway forwards the query to the daemon resolver
(`gateway.dns.upstream`, default `[::1]:5354`). The daemon must
match: an IPv6 socket bound to `[::1]` does not accept v4-mapped
traffic, so a `127.0.0.1:5354` upstream cannot reach a daemon
bound on `[::1]:5354`.
4. If the daemon is unreachable or times out (5 s), the gateway
replies `SERVFAIL`. If the daemon returns `NXDOMAIN` or a
non-`AAAA` answer, the gateway forwards the response unchanged.
5. The gateway extracts the AAAA (`fd00::/8`) record from the
daemon's response. This resolution primes the daemon's identity
cache as a side effect — a prerequisite for `fips0` routing,
because the daemon needs the cache entry to map the mesh address
back to a `NodeAddr` for forwarding.
6. The gateway allocates a virtual IP from the pool for that mesh
address (idempotent: an existing mapping is reused and its TTL
refreshed).
7. If a new mapping was created, the pool emits `MappingCreated`,
which the main loop turns into `add_mapping` calls on the NAT
manager and `add_proxy_ndp` on the network setup.
8. The gateway returns an `AAAA` response containing the virtual IP,
with the configured TTL (default 60 s).
### Prerouting DNAT
### Virtual IP Pool
A per-mapping rule in the `prerouting` chain (priority dstnat / -100) rewrites
the destination address from the virtual IP to the corresponding fd00::/8 mesh
address:
The pool allocates IPv6 addresses from a configured CIDR (default
`fd01::/112`). Each address maps to one mesh destination, keyed by
`NodeAddr` rather than by hostname — different `.fips` aliases for
the same node share a virtual IP. Address 0 (the network-equivalent)
is reserved; the rest are allocatable. The pool is capped at 2^16
addresses regardless of prefix length, to bound memory.
The pool tracks state per address:
```text
match: ip6 daddr == <virtual_ip>
Allocated ──→ Active ──→ Draining ──→ Free
│ ▲
└──────────────────────────────────┘
(TTL expired, no sessions)
```
| State | Meaning |
| ----- | ------- |
| Allocated | DNS query created the mapping; no NAT sessions yet. |
| Active | Conntrack reports at least one session for this virtual IP. |
| Draining | TTL has expired; sessions may still be in progress, or grace period is running after sessions ended. |
| Free | Reclaimed and available for new allocations. |
Transitions:
- **Allocated → Active**: conntrack sessions count goes above zero.
- **Allocated → Free**: TTL expires before any session is ever
observed.
- **Active → Draining**: TTL expires (sessions may or may not still
be present).
- **Draining → Free**: session count is zero and the grace period
has elapsed since draining began.
Timing:
- **TTL** (`gateway.dns.ttl`, default 60 s) is both the DNS TTL
returned to the client and the mapping's idle lifetime. Repeated
DNS queries for the same destination refresh the
`last_referenced` timestamp.
- **Grace period** (`gateway.pool_grace_period`, default 60 s) is
the dwell time after the last session ends before the address is
recycled. It prevents immediate reuse from confusing hosts with
cached DNS responses.
- **Tick interval**: the pool re-evaluates state every 10 s.
Active session counts come from `/proc/net/nf_conntrack`: an entry
counts as a session if its original destination is the virtual IP.
If the pool is exhausted, new DNS queries return `SERVFAIL`.
Existing mappings are never evicted prematurely — the correctness of
in-flight sessions takes precedence over fresh allocations.
### NAT Pipeline (Outbound)
Three rule classes in `inet fips_gateway` together implement the
LAN→mesh path:
**Prerouting DNAT (per mapping)** rewrites the destination from the
virtual IP to the corresponding mesh address:
```text
match: nfproto ipv6 && ip6 daddr == <virtual_ip>
action: dnat to <mesh_addr>
```
After DNAT, the kernel routes the packet through fips0 via the standard routing
table.
After DNAT, the kernel routes the packet through `fips0` via the
standard routing table.
### Postrouting Masquerade
A single masquerade rule in the `postrouting` chain (priority srcnat / 100)
rewrites the source address of all traffic exiting via fips0 to the gateway's
own fips0 address:
**Postrouting masquerade (`oifname fips0`)** rewrites the source of
all traffic exiting via `fips0` to the gateway's own `fips0` address:
```text
match: oifname == "fips0"
match: oifname == "fips0"
action: masquerade
```
This is critical. Without masquerade, LAN client source addresses (e.g.,
`fd01::5` from the virtual pool) would appear as the source on the mesh. These
addresses are meaningless to mesh peers, so return traffic would be black-holed.
Masquerade ensures all mesh traffic appears to originate from the gateway's own
FIPS identity.
This rule is critical. Without it, LAN client source addresses (for
example `fd02::20` from the LAN's RA-advertised prefix, or virtual
addresses from another forwarding domain) would appear as the source
on the mesh. Those addresses are meaningless to mesh nodes, so
return traffic would be black-holed. Masquerade ensures all mesh
traffic appears to originate from the gateway's own FIPS identity.
### Postrouting SNAT
A per-mapping rule in the `postrouting` chain rewrites the source address of
return traffic from the mesh address back to the virtual IP:
**Postrouting SNAT (per mapping)** rewrites the source of return
traffic from the mesh address back to the virtual IP:
```text
match: ip6 saddr == <mesh_addr>
match: nfproto ipv6 && ip6 saddr == <mesh_addr>
action: snat to <virtual_ip>
```
This ensures LAN hosts see responses from the virtual IP they connected to,
not from the raw fd00::/8 mesh address.
Without it, the LAN client would see replies from the raw
`fd00::/8` mesh address rather than from the virtual IP it had
originally connected to, breaking application-layer assumptions about
the destination address.
### Atomic Table Rebuild
### Network Requirements (Outbound)
The entire nftables table is rebuilt atomically on every mapping change. The
rebuild sequence is: delete the existing table (ignore ENOENT on first call),
then create a new table with all chains, the masquerade rule, and all
per-mapping DNAT/SNAT rules in a single netlink batch.
The gateway host needs IPv6 forwarding enabled
(`net.ipv6.conf.all.forwarding=1`), proxy NDP enabled on the LAN
interface, `CAP_NET_ADMIN` for `fips-gateway`, and a `local
<pool-cidr> dev lo` route so the kernel accepts packets to the pool
as locally owned and runs them through the NAT chains. LAN clients
need a route to the pool via the gateway and DNS resolution that
forwards `.fips` queries there. On OpenWrt the init script handles
all of this; on other Linux hosts the operator handles it manually.
Full setup is documented in
[../how-to/deploy-gateway.md](../how-to/deploy-gateway.md).
This approach avoids relying on kernel rule handle tracking, which the rustables
crate does not expose. The table is small — one masquerade rule plus two rules
per active mapping — so rebuilding is cheap.
## The Inbound Half (Mesh → LAN)
## Virtual IP Pool Lifecycle
### Configuration Shape
The pool allocates IPv6 addresses from a configured CIDR range (e.g.,
`fd01::/112`). Each address maps to one FIPS mesh destination (keyed by
NodeAddr, not hostname). Address 0 (network equivalent) is reserved; the
remaining addresses are available for allocation.
Inbound port-forwards live in `gateway.port_forwards[]`. Each entry
is a triple:
### State Machine
| Field | Type | Notes |
| ----- | ---- | ----- |
| `listen_port` | `u16` | Port on the gateway's `fips0` address. Must be non-zero. |
| `proto` | `tcp` \| `udp` | Match protocol. |
| `target` | `[ipv6]:port` | LAN destination. IPv4 targets are rejected at parse time by `SocketAddrV6`. |
Validation runs at startup and on every config reload:
`(listen_port, proto)` must be unique across the list, and zero
listen ports are rejected. Forwards are independent of outbound
configuration: a gateway with no `pool` consumers can still expose
inbound services (the pool route and DNS proxy still run, since they
are part of the same binary, but they sit idle).
### NAT Pipeline (Inbound)
For each port-forward, a single prerouting DNAT rule matches
mesh-originated traffic landing on the gateway's `fips0` address
and rewrites it to the LAN target:
```text
ALLOCATED ──→ ACTIVE ──→ DRAINING ──→ FREE
│ ↑
└───────────────────────────────────┘
(TTL expired, no sessions)
match: iifname == "fips0" && nfproto ipv6
&& l4proto == <tcp|udp> && th dport == <listen_port>
action: dnat to <target_ip>:<target_port>
```
| State | Description |
| ----- | ----------- |
| Allocated | DNS query created the mapping. No NAT sessions yet. |
| Active | Conntrack reports at least one active session. |
| Draining | TTL expired but sessions remain, or sessions ended and grace period is running. |
| Free | Reclaimed. Virtual IP returned to the available pool. |
The match clause is deliberately narrow:
### Transitions
- **`iifname == "fips0"`** restricts the rule to traffic that
arrived from the mesh. LAN-side ingress is never subject to
inbound forwarding.
- **`nfproto ipv6`** is enforced both here and at config-load time
(`SocketAddrV6` rejects IPv4 targets); FIPS is IPv6-only end to
end.
- **`l4proto + dport`** narrows the match to one
`(listen_port, proto)` pair per rule. Unique-tuple validation
ensures no two rules contend for the same packet.
- **Allocated to Active**: Conntrack reports sessions > 0.
- **Allocated to Free**: TTL expired with no sessions ever created.
- **Active to Draining**: TTL expired (sessions may or may not remain).
- **Draining to Free**: Sessions drop to zero and the grace period elapses.
When *any* port-forward is configured, a single LAN-side masquerade
is added to postrouting:
### Timing
```text
match: iifname == "fips0" && oifname == <lan_interface>
&& nfproto ipv6
action: masquerade
```
- **TTL**: Default 60 seconds (matches DNS TTL). Repeated DNS queries for the
same destination reset the `last_referenced` timestamp.
- **Grace period**: Default 60 seconds after draining begins with zero sessions.
Prevents immediate reuse that could confuse hosts with cached DNS responses.
- **Tick interval**: The pool evaluates state transitions every 10 seconds.
Without this rule, the LAN target would attempt to reply directly to
the mesh peer's `fd00::/8` source address, which is not reachable on
the LAN. Masquerade rewrites the source to the gateway's LAN-side
address so the target sees a reachable peer and conntrack routes
the reply back through the gateway.
### Conntrack Integration
This LAN-side masquerade is independent of the `oifname fips0`
masquerade in the outbound pipeline; the two have disjoint match
clauses (different `iifname`/`oifname` combinations) and coexist
without interaction when both directions are active.
The pool queries `/proc/net/nf_conntrack` to count active sessions per virtual
IP. A session is counted if any conntrack entry's original destination matches
the virtual IP address.
### Independence From Outbound
### Pool Exhaustion
The inbound half does not require:
If no addresses are available, new DNS queries return SERVFAIL. Existing
mappings are never evicted prematurely — correctness of active sessions takes
priority over new allocations. The pool is capped at 2^16 addresses regardless
of CIDR prefix length to prevent excessive memory allocation.
- A virtual-IP pool. Mesh peers connect directly to the gateway's
own `fips0` address, which the FIPS daemon already owns.
- DNS resolution. Mesh peers reach the gateway as
`<gateway-npub>.fips:<port>` using their own resolver (or a
numeric mesh address); the gateway's DNS proxy is not in the path.
- A daemon-side identity cache for the LAN target. The target is a
LAN-side IPv6 address, not a mesh address; no `fd00::/8` lookup
happens for it.
## DNS Resolution Flow
A gateway configured with port-forwards but with no LAN clients ever
issuing `.fips` DNS queries will have an empty pool and zero
outbound mappings, but its inbound forwards work normally. The
inverse is also true: a gateway that serves only outbound LAN→mesh
traffic has zero entries in the port-forwards list and no LAN-side
masquerade.
1. Gateway listens on configured address (default `[::]:53`).
2. Client sends DNS query.
3. If the query is not for a `.fips` domain, return `REFUSED`.
4. Forward the query to the daemon resolver at `127.0.0.1:5354` (configurable).
5. If the daemon is unreachable or times out (5 seconds), return `SERVFAIL`.
6. If the daemon returns NXDOMAIN or an error, forward the response as-is.
7. Extract the AAAA record (fd00::/8 mesh address) from the daemon's response.
8. Allocate a virtual IP from the pool for this destination (idempotent — if a
mapping already exists, reuse it and refresh the TTL).
9. If a new mapping was created, emit a `MappingCreated` event to install NAT
rules and proxy NDP entry.
10. Build and return an AAAA response containing the virtual IP with the
configured TTL.
## Atomic Table Rebuild (Common)
The daemon's resolver populates its identity cache as a side effect of
resolution. This is required for fips0 routing to work — without the cache
entry, the daemon cannot map the fd00::/8 address back to a NodeAddr for mesh
routing.
Both halves contribute rules to the same `inet fips_gateway` table,
and that table is rebuilt as one unit on every state change —
mapping added, mapping removed, port-forwards updated. The rebuild
sequence is:
## Network Requirements
1. Delete the existing table in its own batch (ignore `ENOENT`).
2. In a fresh batch: add the table; add the `prerouting` and
`postrouting` chains; add the always-on `oifname fips0`
masquerade; add per-mapping DNAT/SNAT rules for every active
pool entry; add per-port-forward DNAT rules; add the LAN-side
masquerade if any port-forwards exist.
3. Send the batch as a single netlink transaction.
### Gateway Host
The following must be true on the machine running `fips-gateway`:
- **FIPS daemon running** with TUN enabled (fips0 interface must exist) and DNS
resolver on port 5354
- **IPv6 forwarding enabled**: `sysctl -w net.ipv6.conf.all.forwarding=1`
- **Proxy NDP enabled**: `sysctl -w net.ipv6.conf.all.proxy_ndp=1`
- **CAP_NET_ADMIN**: Required for nftables table management and proxy NDP
manipulation (run as root or set the capability)
- **Pool route**: The gateway adds `local <pool-cidr> dev lo` at startup, which
tells the kernel to accept packets destined for pool addresses as
locally-owned, enabling NAT processing. This route is cleaned up on shutdown.
### LAN Clients
LAN clients need no FIPS software. They require:
- **Route to virtual IP pool**: `ip -6 route add <pool-cidr> via
<gateway-lan-addr>`. This can be pushed via DHCP, configured on the LAN
router, or set per-host.
- **DNS resolution**: Either configure the LAN's main DNS server to forward
`.fips` queries to the gateway, or point individual hosts at the gateway for
DNS (noting that non-`.fips` queries will get `REFUSED`).
The rustables crate does not expose rule-handle tracking, so
incremental update of individual rules is not available. Atomic
rebuild was chosen for simplicity and correctness: it eliminates an
entire class of partial-update inconsistency bugs at the cost of
repeating the (cheap) rule construction on every change. The total
rule count is bounded by the pool capacity (2 per mapping, capped
at 2^16) and the port-forward count, both of which are small in
practice.
## Configuration Reference
All configuration lives under the `gateway` key in `fips.yaml`:
The full `gateway.*` block — pool CIDR, LAN interface, DNS
listen/upstream/TTL, pool grace period, conntrack timeouts, and
inbound port-forwards — is documented in the
[Gateway section](../reference/configuration.md#gateway-gateway)
of the configuration reference. The same block governs both halves;
fields specific to one half (`pool`, `dns.*` for outbound;
`port_forwards[]` for inbound) are simply unused when the other
half is not in play.
```yaml
gateway:
enabled: true
pool: "fd01::/112"
lan_interface: "enp3s0"
dns:
listen: "[::]:53"
upstream: "127.0.0.1:5354"
ttl: 60
pool_grace_period: 60
conntrack:
tcp_established: 432000
udp_timeout: 30
udp_assured: 180
icmp_timeout: 30
```
## Operations and Troubleshooting
| Field | Type | Default | Description |
| ----- | ---- | ------- | ----------- |
| `enabled` | bool | `false` | Enable the gateway. Must be `true` for `fips-gateway` to start. |
| `pool` | string (CIDR) | required | Virtual IP pool range (e.g., `fd01::/112`). |
| `lan_interface` | string | required | LAN-facing interface for proxy NDP entries. |
| `dns.listen` | string | `[::]:53` | Address and port for the gateway DNS listener. |
| `dns.upstream` | string | `127.0.0.1:5354` | FIPS daemon DNS resolver address. |
| `dns.ttl` | u32 | `60` | DNS response TTL in seconds. Also governs mapping TTL. |
| `pool_grace_period` | u64 | `60` | Seconds after last session before a mapping is reclaimed. |
| `conntrack.tcp_established` | u64 | `432000` | TCP established timeout (seconds). 5 days. |
| `conntrack.udp_timeout` | u64 | `30` | UDP unreplied timeout (seconds). |
| `conntrack.udp_assured` | u64 | `180` | UDP bidirectional (assured) timeout (seconds). |
| `conntrack.icmp_timeout` | u64 | `30` | ICMP timeout (seconds). |
## Troubleshooting
### "No gateway section in configuration"
The `fips-gateway` binary loads the same config file as the daemon. If it
cannot find a `gateway:` section, use the `--config` flag to point at the
correct file:
```bash
fips-gateway --config /etc/fips/fips.yaml
```
### DNS Queries Fail
Verify the daemon resolver is running and reachable:
```bash
dig @127.0.0.1 -p 5354 hostname.fips AAAA
```
If this fails, the daemon is not running or its DNS resolver is not enabled.
Check that the daemon config has `dns.enabled: true` (enabled by default).
### Ping Works But TCP Does Not
This usually means the masquerade rule is missing or misconfigured. Inspect the
nftables table:
```bash
nft list table inet fips_gateway
```
Verify the postrouting chain contains a masquerade rule matching `oifname
"fips0"`. Without masquerade, the mesh peer sees a source address it cannot
route replies to.
### Connection Timeout
Check that IPv6 forwarding is enabled:
```bash
sysctl net.ipv6.conf.all.forwarding
```
Verify the pool route exists:
```bash
ip -6 route show table local | grep <pool-cidr>
```
If the route is missing, the kernel does not recognize pool addresses as local
and drops the packets before NAT can process them.
### Virtual IP Unreachable From Client
Verify the client has a route to the pool via the gateway:
```bash
ip -6 route get <virtual-ip>
```
On the gateway, verify proxy NDP entries exist for allocated virtual IPs:
```bash
ip -6 neigh show proxy
```
If proxy NDP entries are missing, the gateway cannot answer Neighbor Solicitation
requests for virtual IPs on the LAN, so clients cannot resolve the link-layer
address.
### Port 53 Conflict
If another DNS server (systemd-resolved, dnsmasq) is using port 53, the
gateway cannot bind. Options:
```bash
# Check what is using port 53
ss -tulnp | grep :53
# Use an alternate listen address
dns:
listen: "192.168.1.1:5353"
```
Then configure LAN clients to query the alternate port, or run a forwarding
stub on port 53 that delegates `.fips` queries to the gateway.
- [../tutorials/deploy-fips-gateway.md](../tutorials/deploy-fips-gateway.md)
— end-to-end walkthrough on OpenWrt.
- [../how-to/deploy-gateway.md](../how-to/deploy-gateway.md) —
recipe for non-OpenWrt Linux hosts and inbound-port-forwarding
configuration.
- [../how-to/troubleshoot-gateway.md](../how-to/troubleshoot-gateway.md)
— diagnostic recipes (DNS failures, ping working but TCP not,
conntrack inspection, pool exhaustion, port-53 conflicts,
port-forward verification).
- [../reference/cli-fips-gateway.md](../reference/cli-fips-gateway.md)
— command-line interface.
- [../reference/control-socket.md](../reference/control-socket.md#gateway-command-catalog)
— `show_gateway` and `show_mappings` commands.
## Security Considerations
- **LAN trust boundary**: The gateway DNS listener is accessible to any host on
the LAN. Any host that can reach the DNS port and route to the virtual IP pool
can access mesh destinations through the gateway. Access restriction must be
enforced at the network level (firewall rules on the LAN interface).
### Outbound
- **Identity masking**: All LAN traffic appears on the mesh under the gateway's
own FIPS identity. Mesh peers cannot determine which LAN host originated a
connection. This provides privacy for LAN hosts but means the gateway's
reputation covers all its clients.
- **LAN trust boundary.** The DNS listener and the virtual-IP pool
are reachable by every host on the LAN. Any LAN host that can
resolve `.fips` and route to the pool CIDR can reach mesh
destinations. There is no per-client authentication; access
restriction is a network-level concern, enforced with firewall
rules on the LAN interface or on the gateway host itself.
- **Identity masking.** All outbound LAN traffic appears on the
mesh under the gateway's own FIPS identity. Mesh nodes cannot
determine which LAN host originated a connection. This provides
privacy for LAN hosts but means the gateway's reputation covers
all of its clients — and that abusive behavior from one LAN host
is attributed to the gateway, not to the host.
- **Plaintext between client and gateway.** Traffic between the LAN
client and the gateway is unencrypted at the IP layer. FIPS
encryption (FSP) protects the segment between the gateway and the
destination mesh node; application-layer encryption (TLS, SSH,
Noise) is the only thing that provides true end-to-end protection
through the gateway.
- **Pool addresses are ephemeral.** Virtual IPs are allocated
dynamically and recycled. They are not authenticated and not
bound to client identity — a LAN host connecting to a virtual IP
is trusting the gateway's recent DNS response.
- **DNS upstream trust.** The outbound half's correctness depends
on the FIPS daemon's resolver returning honest `fd00::/8`
answers; a compromised daemon could redirect LAN clients to
arbitrary mesh nodes.
- **Plaintext at the gateway**: Traffic between LAN hosts and the gateway is
unencrypted at the IP layer. FIPS encryption (FSP) protects traffic between
the gateway and the destination mesh peer. Application-layer encryption (TLS,
SSH) provides end-to-end protection through the gateway.
### Inbound
- **Pool addresses are ephemeral**: Virtual IPs are allocated dynamically and
recycled. They are not authenticated or bound to client identity. A LAN host
connecting to a virtual IP is trusting the gateway's DNS response.
- **Port exposure.** Each entry in `port_forwards[]` exposes the
matched `(listen_port, proto)` on the gateway's mesh-side
address to every reachable mesh peer. Inbound port-forwards are
not gated by any peer ACL beyond what FMP normally enforces;
treat them with the same care as a public-internet port forward.
- **Mesh peer trust.** The LAN target sees connections that have
been masqueraded to the gateway's LAN address. The target cannot
distinguish one mesh peer from another, and there is no
authenticated peer identity available to the LAN target — any
application-layer authentication or rate-limiting must run on
the target itself.
- **Return-path masquerade exposes the gateway's LAN address.**
The LAN-side masquerade rewrites the mesh peer's source to the
gateway's LAN address. A malicious or buggy LAN target can use
this to send unsolicited traffic back at the gateway, or to
probe other LAN hosts via the gateway's network position; LAN
segmentation (VLANs, host firewalls) is the right control.
- **No client identity verification**: The gateway does not authenticate LAN
clients. Any host that can send packets is served.
### Common
## Future Work
- **IPv4 pool support**: NAT46 translation via TAYGA or Jool, allowing LAN
hosts to use IPv4 virtual addresses while the mesh remains IPv6.
- **Inbound gateway**: Exposing LAN services to mesh peers (mesh to LAN
direction), requiring port-forwarding or reverse-proxy configuration.
- **fipstop Gateway tab**: Monitoring integration showing pool utilization,
active mappings, and NAT session counts.
- **Gateway control socket**: Status queries via `fipsctl gateway status` and
`fipsctl gateway mappings` for operational visibility.
- **No client identity verification.** The gateway authenticates
neither LAN clients nor mesh peers beyond what the underlying
layers already do — `fips0` ingress carries an FSP-authenticated
payload, the LAN side is whoever the LAN admits.
## References
- [fips-ipv6-adapter.md](fips-ipv6-adapter.md) — IPv6 adapter and TUN interface
design
- [fips-configuration.md](fips-configuration.md) — Configuration reference
- [fips-intro.md](fips-intro.md) — Protocol overview and architecture
- [fips-ipv6-adapter.md](fips-ipv6-adapter.md) — IPv6 adapter and
TUN interface design.
- [fips-architecture.md](fips-architecture.md) — protocol layer
architecture.
- [fips-concepts.md](fips-concepts.md) — protocol overview.
- [../reference/configuration.md](../reference/configuration.md) —
configuration reference.
- [../reference/cli-fips-gateway.md](../reference/cli-fips-gateway.md)
— `fips-gateway` CLI.
- [../reference/control-socket.md](../reference/control-socket.md) —
control-socket protocol and command catalog.
- [../how-to/deploy-gateway.md](../how-to/deploy-gateway.md) —
gateway host and LAN client setup.
- [../how-to/troubleshoot-gateway.md](../how-to/troubleshoot-gateway.md)
— diagnostic recipes.
- [../tutorials/deploy-fips-gateway.md](../tutorials/deploy-fips-gateway.md)
— OpenWrt walkthrough.
docs/design/fips-intro.md
-822
View File
@@ -1,822 +0,0 @@
# FIPS: Free Internetworking Peering System
## What is FIPS?
FIPS is a self-organizing mesh network that can operate natively over a
variety of physical and logical media, such as local area networks,
Bluetooth, serial links, or the existing internet as an overlay. The
long-term goal is infrastructure that can function alongside or ultimately
replace dependence on the Internet itself. Systems running FIPS establish
peer connections, authenticate each other, and route traffic for each other
without any central authority or global topology knowledge, and allow
end-to-end encrypted sessions between any two nodes regardless of how many
hops separate them.
Nodes in the mesh route traffic for each other using Nostr identities
(npubs) as network addresses. Applications can access the mesh through a
native FIPS datagram service, or through an IPv6 adaptation layer that
presents each node as an IPv6 endpoint for compatibility with existing
IP-based applications.
## Why FIPS?
**Self-sovereign identity**: FIPS nodes generate their own addresses, node
IDs, and security credentials without coordination with any central
authority. These identities can be long-term fixed or may be ephemeral,
changed at any time. These identities are not visible to the FIPS network
itself — they are used only at the application layer and for end-to-end
session encryption.
**Infrastructure independence**: The internet depends on centralized
infrastructure — ISPs, backbone providers, DNS, certificate authorities.
FIPS works over any transport that can carry packets: a serial connection,
onion-routed connections through Tor, local area networking, radio links
between remote sites, or the existing internet as an overlay. When the
internet is unavailable, unreliable, or untrusted, the mesh still works.
**Privacy by design**: FIPS provides secure, authenticated, and encrypted
communication between any two nodes in the mesh, independent of the mix of
transports used along the routed path between them. Furthermore, the mesh
itself is designed to minimize metadata exposure — intermediate nodes route
packets without learning the identities of the endpoints.
**Zero configuration**: Nodes discover each other and build routing
automatically. Connect to one peer and you can reach the entire mesh. The
network self-heals around failures and adapts to changing topology.
## A Self-Organizing Mesh
Traditional networks are built top-down. A central authority assigns
addresses, configures routing tables, provisions hardware, and manages the
topology. If the authority disappears or the infrastructure fails, the
network fails with it. Nodes cannot reach each other without infrastructure
mediating the connection.
FIPS inverts this model. There is no central authority, no address
assignment service, no routing table pushed from above. Each node generates
its own identity from a cryptographic keypair. Each node independently
decides which peers to connect to and which transports to use. From these
local decisions alone, the network self-organizes:
- A **spanning tree** forms through distributed parent selection, giving
every node a coordinate in the network without any node knowing the full
topology
- **Bloom filters** propagate through gossip, so each node learns which
peers can reach which destinations — again without global knowledge
- **Routing decisions** are made locally at each hop, using only the node's
immediate peers and cached coordinate information
Each peer link and end-to-end session actively measures RTT, loss, jitter,
and goodput through a lightweight in-band Metrics Measurement Protocol
(MMP), providing operator visibility and a foundation for quality-aware
routing.
The result is a network that builds itself from the bottom up, heals around
failures automatically, and scales without central coordination. Adding a
node is as simple as connecting to one existing peer — the network
integrates the new node through its normal mesh protocols.
## Specific Design Goals
- **Nostr-native identity and cryptography** — Use Nostr keypairs as node
identities and leverage secp256k1, Schnorr signatures, and SHA-256
- **Transport agnostic** — Support overlay, shared medium, and
point-to-point transports transparently
- **Self-organizing** — Automatic topology discovery and route optimization
- **Privacy preserving** — Minimize metadata leakage across untrusted links
- **Resilient** — Self-healing with graceful degradation
Non-goals include:
- **Reliable delivery** — FIPS provides a best-effort datagram service;
retransmission and ordering are left to applications or higher-layer
protocols
- **Anonymity** — Direct peers learn each other's identity; FIPS minimizes
metadata exposure but is not an anonymity network like Tor
- **Congestion control** — FIPS measures link quality but does not implement
flow control or congestion avoidance at the mesh layer
---
## Protocol Architecture
FIPS is organized in three protocol layers, each with distinct
responsibilities and clean service boundaries. No layer depends on the
specifics of the layers above or below it — transport plugins know nothing
about sessions, the routing layer knows nothing about application addressing,
and applications know nothing about which physical media carry their traffic.
This separation means new transports, protocol features, and application
interfaces can be added independently.
![Protocol Stack](diagrams/fips-protocol-stack.svg)
### Mapping to Traditional Networking
Readers familiar with the OSI model or TCP/IP networking may find it helpful
to see how FIPS concepts relate to traditional layers:
![OSI Mapping](diagrams/fips-osi-mapping.svg)
Note that FMP spans what would traditionally be separate link and network
layers. This is intentional — in a self-organizing mesh, the same layer that
authenticates peers also makes routing decisions, because routing depends on
authenticated peer state (spanning tree positions, bloom filters).
### Layer Responsibilities
**Transport layer**: Delivers datagrams between endpoints over a specific
medium. Each transport type (UDP socket, Ethernet interface, radio modem)
implements the same abstract interface: send and receive datagrams, report
MTU. The transport layer knows nothing about FIPS identities, routing, or
encryption. It provides raw datagram delivery to FMP above.
See [fips-transport-layer.md](fips-transport-layer.md) for the transport layer
specification.
**FIPS Mesh Protocol (FMP)**: Manages peer connections, authenticates peers
via Noise IK handshakes, and encrypts all traffic on each link. FMP is where
the mesh organizes itself — nodes exchange spanning tree announcements and
bloom filters with their direct peers, and FMP makes forwarding decisions
for transit traffic. FMP provides authenticated, encrypted forwarding to FSP
above.
See [fips-mesh-layer.md](fips-mesh-layer.md) for the FMP specification and
[fips-mesh-operation.md](fips-mesh-operation.md) for how FMP's routing and
self-organization work in practice.
**FIPS Session Protocol (FSP)**: Provides end-to-end authenticated
encryption between any two nodes, regardless of how many intermediate hops
separate them. FSP manages session lifecycle (setup, data transfer,
teardown), caches destination coordinates for efficient routing, and handles
the warmup strategy that keeps transit node caches populated. Session
dispatch uses index-based routing inspired by
[WireGuard](https://www.wireguard.com/), enabling O(1) packet
demultiplexing. FSP provides a datagram service to applications above.
See [fips-session-layer.md](fips-session-layer.md) for the FSP specification.
**IPv6 adaptation layer**: Sits above FSP as a service on port 256, adapting
the FIPS datagram service for unmodified IPv6 applications. Provides DNS
resolution (npub → fd00::/8 address), identity cache management, IPv6 header
compression, MTU enforcement, and a TUN interface. This is the primary way
existing applications use the FIPS mesh.
See [fips-ipv6-adapter.md](fips-ipv6-adapter.md) for the IPv6 adapter.
### Node Architecture
Application services sit at the top of the stack, dispatched by FSP port
number: the IPv6 TUN adapter (port 256) maps npubs to `fd00::/8` addresses
with header compression so unmodified IP applications can use the network
transparently, while the native datagram API addresses destinations directly
by npub.
![Node Architecture](diagrams/fips-node-architecture.svg)
The mesh routes application traffic across heterogeneous transports
transparently. A packet may traverse WiFi, Ethernet, UDP/IP, and Tor links
on its way from source to destination — the application never needs to know
which transports are involved. Each hop is independently encrypted at the
link layer, while a single end-to-end session protects the payload across
the entire path.
![Architecture Overview](diagrams/fips-architecture-overview.svg)
---
## Identity System
FIPS uses [Nostr](https://github.com/nostr-protocol/nips) keypairs
(secp256k1) as node identities. The public key identifies the node; the
private key signs protocol messages and establishes encrypted sessions.
The public key (or its bech32-encoded npub form) is the primary means for
application-layer software to identify communication endpoints. Internally,
the protocol derives a `node_addr` (a 16-byte SHA-256 hash of the pubkey)
used as the routing identifier in packet headers, and an IPv6 address derived
from the node_addr for the TUN adapter. Applications use the pubkey or npub;
the routing layer uses node_addr; unmodified IPv6 applications use the
derived `fd00::/8` address. All three are deterministically derived from the
same keypair.
### FIPS Identity Handling
![Identity Derivation](diagrams/fips-identity-derivation.svg)
The pubkey is the node's cryptographic identity, used in Noise IK handshakes
for both link and session encryption. It is never exposed beyond the
endpoints of an encrypted channel. The node_addr, a one-way SHA-256 hash
truncated to 16 bytes, serves as the routing identifier in packet headers
and bloom filters. Intermediate routers see only node_addrs — they can
forward traffic without learning the Nostr identities of the endpoints. An
observer can verify "does this node_addr belong to pubkey X?" if they already
know the pubkey, but cannot enumerate communicating identities by inspecting
traffic. The IPv6
address prepends `fd` to the first 15 bytes of the node_addr, providing a
ULA overlay address for unmodified IP applications via the TUN interface.
Below the FIPS identity layer, each transport uses its own native addressing
— IP:port or hostname:port addresses, MAC addresses, .onion identifiers. These **link
addresses** are opaque to everything above FMP and discarded once link
authentication completes.
### Identity Verification
The Noise Protocol Framework mutually authenticates both peer-to-peer link
connections (at FMP) and end-to-end session traffic (at FSP), proving each
party controls the private key for their claimed identity.
See [fips-mesh-layer.md](fips-mesh-layer.md) for peer authentication and
[fips-session-layer.md](fips-session-layer.md) for end-to-end session
establishment.
Key rotation changes the node's identity — a new keypair produces a new
node_addr and IPv6 address, requiring all sessions to be re-established.
Migration mechanisms that allow a node to announce a successor key are a
future consideration.
---
## Two-Layer Encryption
FIPS uses independent encryption at two protocol layers:
| Layer | Scope | Pattern | Purpose |
| ----- | ----- | ------- | ------- |
| **FMP (Mesh)** | Hop-by-hop | Noise IK | Encrypt all traffic on each peer link |
| **FSP (Session)** | End-to-end | Noise XK | Encrypt application payload between endpoints |
### Link Layer (Hop-by-Hop)
When two nodes establish a direct connection, they perform a [Noise
IK](https://noiseprotocol.org/) handshake. This authenticates both parties
and establishes symmetric keys for encrypting all traffic on that link.
Every packet between direct peers is encrypted — gossip messages, routing
queries, and forwarded session datagrams alike.
The IK pattern is used because outbound connections know the peer's npub
from configuration, while inbound connections learn the initiator's identity
from the first handshake message.
### Session Layer (End-to-End)
FIPS establishes end-to-end encrypted sessions between any two communicating
nodes using Noise XK, regardless of how many hops separate them. The
initiator knows the destination's npub (required for XK's pre-message);
the responder learns the initiator's identity from the third handshake
message. Unlike the link-layer IK pattern where the initiator's identity
is revealed in msg1, XK delays identity disclosure until msg3, providing
stronger initiator identity protection for traffic traversing untrusted
intermediate nodes.
A packet from A to D through intermediate nodes B and C:
1. A encrypts payload with A↔D session key (FSP)
2. A wraps in SessionDatagram, encrypts with A↔B link key (FMP), sends to B
3. B decrypts link layer, reads destination node_addr, re-encrypts with B↔C
link key, forwards to C
4. C decrypts link layer, re-encrypts with C↔D link key, forwards to D
5. D decrypts link layer, then decrypts session layer to get payload
Intermediate nodes route based on destination node_addr but cannot read
session-layer payloads. Each hop strips one link encryption and applies the
next — the session-layer ciphertext passes through untouched.
Both layers always apply, even between adjacent peers — a packet to a direct
neighbor is still encrypted twice. This uniform model means no special cases
for local vs remote destinations, and topology changes (a direct peer
becomes reachable only through intermediaries) don't affect existing
sessions.
See [fips-mesh-layer.md](fips-mesh-layer.md) for link encryption and
[fips-session-layer.md](fips-session-layer.md) for session encryption.
---
## Routing and Mesh Operation
Each node makes forwarding decisions using only local information — its
immediate peers, their bloom filters, and cached coordinates — rather than
centrally distributed routing tables or global topology knowledge. Two
complementary mechanisms provide the information each node needs.
### Spanning Tree: The Coordinate System
![Mesh Topology](diagrams/fips-mesh-topology.svg)
Nodes self-organize into a spanning tree through gossip — each node
exchanges announcements with its direct peers and independently selects a
parent. Because every node applies the same rule (prefer the root with the
smallest node_addr), the network converges on a single agreed-upon root
without any voting or coordination. This is the same principle behind the
[Spanning Tree
Protocol](https://en.wikipedia.org/wiki/Spanning_Tree_Protocol) used in
Ethernet bridging since the 1980s: purely local decisions that converge to
consistent global state. The resulting tree gives every node a
**coordinate** — its path from itself to the root. Using tree coordinates
for routing is adapted from
[Yggdrasil](https://yggdrasil-network.github.io/)'s
[Ironwood](https://github.com/Arceliar/ironwood) routing library.
These coordinates enable distance calculations between any two nodes: the
distance is the number of hops from each node to their lowest common
ancestor in the tree. This provides a metric for routing decisions without
any node needing to know the full network topology.
The tree maintains itself through gossip — nodes exchange TreeAnnounce
messages with their peers, propagating parent selections and ancestry
chains. Changes cascade through the tree proportional to depth, not network
size. If the network partitions, each segment converges to its own new root
through the same process and reconverges automatically when segments rejoin.
See [fips-spanning-tree.md](fips-spanning-tree.md) for the tree algorithms
and [spanning-tree-dynamics.md](spanning-tree-dynamics.md) for detailed
convergence walkthroughs.
### Bloom Filters: Candidate Selection
The spanning tree provides a coordinate system for distance-based routing,
but on its own each node would only know about its immediate neighbors.
Bloom filters complement the tree by distributing reachability knowledge
across the entire mesh — each node learns which destinations are reachable
through which peers, without any node needing a complete view of the
network.
Each node's peer-advertised [bloom
filter](https://en.wikipedia.org/wiki/Bloom_filter) is a compact, fixed-size
data structure that answers one question: "can this peer possibly reach
destination D?" The answer is either "no" (definitive) or "maybe"
(probabilistic — false positives are possible). Because the filter size is
constant regardless of how many destinations it represents, bloom filters
scale efficiently as the network grows. This is candidate selection for
routing — bloom filters narrow the set of peers worth considering, and the
actual forwarding decision ranks those candidates by tree distance and link
quality.
Filters propagate transitively through tree edges, with each node computing
outbound filters by merging the filters received from its tree peers (parent
and children) using a
[split-horizon](https://en.wikipedia.org/wiki/Split_horizon_route_advertisement)
technique borrowed from distance-vector routing. All peers — including
non-tree mesh shortcuts — receive FilterAnnounce messages, but only tree
peers' filters are merged into outgoing computation. This prevents filter
saturation where mesh shortcuts would cause every filter to converge toward
the full network.
See [fips-bloom-filters.md](fips-bloom-filters.md) for filter parameters and
mathematical properties.
![Bloom filter propagation on a spanning tree](diagrams/fips-bloom-propagation.svg)
The outbound filter for peer Q merges this node's identity with tree peer
inbound filters except Q's (split-horizon exclusion). This creates
directional asymmetry: upward filters (child → parent) contain the child's
subtree, while downward filters (parent → child) contain the complement.
Mesh peers receive filters but their inbound filters are not merged
transitively — they provide single-hop shortcut visibility only.
A node with multiple peers receives genuinely different filters from each.
In the diagram, R receives {B, D, E} from B and {C, F} from C — two disjoint
subtrees. When R needs to reach F, only C's filter matches. This is where
bloom filters provide real candidate selection: a node with several peers
can narrow the forwarding choice before consulting tree coordinates. Leaf
nodes like D have only one peer, so their single inbound filter is
necessarily near-complete (everything except themselves) and offers no
selection — but leaf nodes have no choice to make anyway.
Bloom filter sizing (bit count and hash functions) requires further analysis
based on actual deployment scenarios. The FMP wire format is versioned to
accommodate future parameter changes as operational experience accumulates.
### Routing Decisions
At each hop, FMP makes a local forwarding decision using the following
priority chain:
1. **Local delivery** — the destination is this node
2. **Direct peer** — the destination is an authenticated neighbor
3. **Bloom-guided candidate selection** — bloom filters identify peers that
can reach the destination; tree coordinates rank them by distance and
link quality
4. **[Greedy routing](https://en.wikipedia.org/wiki/Greedy_embedding)** —
fallback when bloom filters haven't converged; forward to the peer that
minimizes tree distance to the destination
5. **No route** — destination unreachable; send error signal to source
All multi-hop routing depends on knowing the destination's tree coordinates.
These are cached at each node after being learned through discovery
(LookupRequest/LookupResponse) or session establishment (SessionSetup). The
coordinate cache is the critical piece that enables efficient forwarding.
![Per-hop routing decision flowchart](diagrams/fips-routing-decision.svg)
### Coordinate Caching and Discovery
When a node first needs to reach an unknown destination, it sends a
LookupRequest that propagates through the network guided by bloom filters
and loop prevention. The destination responds with its coordinates, which
the source and intermediate nodes along the return path cache. Subsequent
traffic routes efficiently using the cached coordinates.
Session establishment (SessionSetup) also carries coordinates, warming
transit node caches along the path so that data packets can be forwarded
without individual discovery at each hop.
![Coordinate discovery and cache warming sequence](diagrams/fips-coordinate-discovery.svg)
### Error Recovery
When routing fails — because cached coordinates are stale, a path has
broken, or a packet exceeds a link's MTU — transit nodes signal the source:
- **CoordsRequired**: A transit node lacks the destination's coordinates.
The source re-initiates discovery and resets its coordinate warmup
strategy.
- **PathBroken**: Greedy routing reached a dead end. The source re-discovers
the destination's current coordinates.
- **MtuExceeded**: A transit node cannot forward a packet because it exceeds
the next-hop link MTU. The source adjusts its path MTU estimate.
All three signals trigger active recovery, and are rate-limited to prevent
storms during topology changes.
See [fips-mesh-operation.md](fips-mesh-operation.md) for the complete
routing and mesh behavior description.
### Metrics Measurement Protocol (MMP)
Each peer link runs an instance of the Metrics Measurement Protocol, which
measures link quality through in-band report exchange. MMP computes smoothed
round-trip time (SRTT), packet loss rate, interarrival jitter, goodput, and
one-way delay trend — all derived from counter and timestamp fields already
present in the FMP wire format, with no additional probing traffic required.
MMP operates in three modes. **Full** mode exchanges both SenderReports and
ReceiverReports to compute all metrics including RTT. **Lightweight** mode
exchanges only ReceiverReports, providing loss and jitter but not RTT — useful
for constrained links. **Minimal** mode disables reports entirely, relying
only on spin bit and congestion echo flags in the frame header.
Reports are sent at RTT-adaptive intervals (clamped to 100 ms–2 s), so
high-latency links don't generate excessive measurement traffic while
low-latency links converge quickly. Each metric carries both short-term and
long-term exponentially weighted moving averages, enabling detection of
quality changes against a stable baseline.
MMP serves dual roles: operator visibility and cost-based parent selection.
Periodic log lines report per-link RTT, loss, jitter, and goodput. MMP
computes an Expected Transmission Count (ETX) from bidirectional delivery
ratios, which feeds into cost-based parent selection where each node
evaluates `effective_depth = depth + link_cost` using
`link_cost = etx * (1.0 + srtt_ms / 100.0)`. ETX is not yet used in
`find_next_hop()` candidate ranking for data forwarding.
See [fips-mesh-layer.md](fips-mesh-layer.md) for MMP operating modes, report
scheduling, and the spin bit design.
---
## Transport Abstraction
FIPS treats the communication medium as a pluggable component. Every transport
— whether a UDP socket, an Ethernet interface, a Tor circuit, or a radio modem
— implements the same simple interface: send a datagram to an address, receive
datagrams, and report the link MTU. The rest of the protocol stack sees no
difference between them.
A **transport** is a driver for a particular medium. A **link** is a peer
connection established over a transport. Transport addresses (IP:port, MAC
address, .onion) are opaque to all layers above FMP — they exist only to
deliver datagrams and are discarded once FMP has authenticated the peer via
the Noise IK handshake. From that point on, the peer is identified solely by
its cryptographic identity.
Transports fall into three categories based on their connectivity model:
| Category | Examples | Characteristics |
| -------- | -------- | --------------- |
| Overlay | UDP/IP, Tor | Tunnels FIPS over existing networks |
| Shared medium | Ethernet, WiFi, Bluetooth, Radio | Local broadcast, peer discovery |
| Point-to-point | Serial, dialup | Fixed connections, no discovery |
These categories differ in addressing, MTU, reliability, and whether they
support local discovery, but FMP handles all of them uniformly. A node
running multiple transports simultaneously bridges between those networks
automatically — peers from all transports feed into a single spanning tree,
and the router selects the best path regardless of which medium carries it.
If one transport fails, traffic reroutes through alternatives without
application involvement.
Some transports support an optional discovery capability — the ability to
broadcast and listen for announcements indicating the availability of FIPS
endpoints on the local medium. Shared media like Ethernet, WiFi, Bluetooth,
and radio are natural fits for this, as they can reach nearby devices without
prior configuration. When discovery is available, nodes can automatically
find and peer with other FIPS nodes on the same medium. Transports that
lack discovery (such as configured UDP endpoints) simply skip this step and
connect directly to configured addresses. For internet-reachable nodes,
endpoint discovery via signed Nostr events allows nodes to publish and
consume transport addresses through public relays — available behind the
`nostr-discovery` cargo feature.
NAT traversal for internet-connected nodes is supported via STUN-assisted
UDP hole punching, with offer/answer signaling carried over Nostr gift-wrap
events. Once a direct UDP path is established, the punched socket is handed
into the standard FIPS transport/session stack. Nodes that cannot establish
a direct path remain reachable through port forwarding, a publicly addressed
peer, or relay through other mesh nodes. The Nostr-mediated discovery and
NAT traversal paths are gated by the `nostr-discovery` cargo feature and
configured under `node.discovery.nostr.*`.
> **Implementation status**: UDP/IP, TCP/IP, Ethernet, Tor
> (SOCKS5 outbound + directory-mode inbound via onion service),
> and Bluetooth (BLE L2CAP CoC) transports are implemented.
> All others are future directions.
See [fips-transport-layer.md](fips-transport-layer.md) for the full transport
layer specification.
---
## Security
FIPS is designed around four classes of adversary, each addressed by a
different layer of the protocol.
### Transport Observers
A passive observer on the underlying transport — someone monitoring a WiFi
network, tapping an Ethernet segment, or inspecting UDP traffic — sees only
encrypted packets. The FMP link-layer Noise IK session encrypts all traffic
between direct peers, including routing gossip and forwarded session
datagrams. The observer can infer timing, packet sizes, and which transport
endpoints are exchanging traffic, but cannot read content or determine
FIPS-level node identities from the encrypted packets. Traffic analysis —
correlating timing and volume patterns across multiple vantage points to
infer communication relationships — is not defended against (see
[Specific Design Goals](#specific-design-goals)).
### Active Attackers on the Transport
An adversary who can inject, modify, drop, or replay packets on the
transport is also defeated by the FMP link-layer Noise IK session. Mutual
authentication prevents impersonation, AEAD encryption detects tampering,
and counter-based nonces with a sliding replay window reject replayed
packets.
### Other FIPS Nodes (Intermediate Routers)
The most important adversary class is the operators of other nodes in the
mesh — the peers that forward your traffic. FIPS treats every intermediate
router as potentially adversarial. The FSP session layer establishes a
completely independent Noise XK session between the communicating endpoints,
so intermediate nodes cannot read application payloads even though they
decrypt and re-encrypt the link-layer envelope at each hop.
Routing headers expose only the destination's node_addr — an opaque
SHA-256 hash of the actual public key. Intermediate routers can forward
traffic without learning which Nostr identities are communicating. An
observer can verify "does this node_addr belong to pubkey X?" if they
already know the pubkey, but cannot enumerate communicating identities by
inspecting routed traffic.
| Entity | Can See |
| ------ | ------- |
| Transport observer | Encrypted packets, timing, packet sizes |
| Direct peer | Your npub, traffic volume, timing |
| Intermediate router | Source and destination node_addrs, packet size |
| Destination | Your npub, payload content |
### Adversarial Nodes Disrupting the Mesh
Beyond passive observation, a malicious node could attempt to disrupt
routing by injecting false spanning tree announcements, advertising bogus
bloom filters, or claiming invalid tree positions. FMP mitigates these
through signed TreeAnnounce messages verified by direct peers, transitive
ancestry chain validation, replay protection via sequence numbers, and
discretionary peering — node operators choose who to peer with, so an
attacker with many identities still needs real nodes to accept their
connections. Handshake rate limiting further constrains how fast an attacker
can establish new links. In fully open networks with automatic peer
discovery, Sybil resistance relies primarily on rate limiting; discretionary
peering provides stronger resistance in curated deployments where operators
vet their peers. An attacker who controls all of a target node's direct
peers can completely control its view of the network (an eclipse attack);
diverse peering across independent operators and transports is the primary
mitigation.
---
## Prior Work
FIPS builds on proven designs rather than inventing new cryptography or routing
algorithms. Nearly every major design decision has deployed precedent.
### Spanning Tree Self-Organization
The idea that distributed nodes can build a spanning tree through purely local
decisions — each node selecting a parent based on announcements from its
neighbors — dates to the
[IEEE 802.1D Spanning Tree Protocol](https://en.wikipedia.org/wiki/Spanning_Tree_Protocol)
(STP, 1985). STP demonstrated that a network-wide tree emerges from a simple
deterministic rule (lowest bridge ID wins root election) applied independently
at each node. FIPS uses the same principle — lowest node address determines the
root — adapted from an Ethernet bridging context to a general-purpose overlay
mesh.
### Tree Coordinate Routing
The spanning tree coordinates, bloom filter candidate selection, and greedy
routing algorithms are adapted from
[Yggdrasil v0.5](https://yggdrasil-network.github.io/2023/10/22/upcoming-v05-release.html)
and its [Ironwood](https://github.com/Arceliar/ironwood) routing library.
Yggdrasil's key insight was using the tree path from root to node as a
routable coordinate, enabling greedy forwarding without global routing tables.
FIPS adapts these algorithms for multi-transport operation, Nostr identity
integration, and constrained MTU environments.
The theoretical foundation for greedy routing on tree embeddings draws on
[Kleinberg's work](https://www.cs.cornell.edu/home/kleinber/swn.pdf) on
navigable small-world networks, which showed that greedy forwarding succeeds
in O(log² n) steps when the network has hierarchical structure. Thorup-Zwick
compact routing schemes separately demonstrated that sublinear routing state
is achievable with bounded stretch, motivating the use of tree coordinates
rather than full routing tables.
### Split-Horizon Bloom Filter Propagation
FIPS distributes reachability information using bloom filters computed with a
split-horizon rule: when advertising to a peer, exclude that peer's own
contributions. This technique is borrowed from distance-vector routing
protocols — [RIP](https://en.wikipedia.org/wiki/Routing_Information_Protocol)
(1988) and [Babel](https://www.irif.fr/~jch/software/babel/) use split-horizon
to prevent routing loops by not advertising a route back to the neighbor it was
learned from. FIPS applies the same principle to probabilistic set
advertisements rather than distance-vector tables.
### Cryptographic Identity as Network Address
FIPS nodes are identified by their Nostr public keys (secp256k1). The network
address *is* the cryptographic identity — there is no separate address
assignment or registration step.
[CJDNS](https://github.com/cjdelisle/cjdns) pioneered this approach in
overlay meshes, deriving IPv6 addresses from the double-SHA-512 of each node's
public key. Tor [.onion addresses](https://spec.torproject.org/rend-spec-v3)
and the IETF
[Host Identity Protocol](https://en.wikipedia.org/wiki/Host_Identity_Protocol)
(HIP) follow the same principle. FIPS uses Nostr's existing key infrastructure
rather than introducing a new identity scheme.
### Dual-Layer Encryption
FIPS encrypts traffic twice: FMP provides hop-by-hop link encryption
(protecting against transport-layer observers), while FSP provides independent
end-to-end session encryption (protecting against intermediate FIPS nodes).
This layered approach mirrors [Tor](https://www.torproject.org/), where each
relay peels one layer of encryption (hop-by-hop) while the innermost layer
protects end-to-end payload. [I2P](https://geti2p.net/) uses a similar
garlic routing scheme with tunnel-layer and end-to-end encryption. Unlike Tor
and I2P, FIPS does not provide anonymity — its dual encryption protects
confidentiality and integrity rather than hiding traffic patterns.
### Noise Protocol Framework
FIPS uses the [Noise Protocol Framework](https://noiseprotocol.org/) at both
protocol layers, with different handshake patterns chosen for each layer's
threat model. FMP link encryption uses **Noise IK**, providing mutual
authentication with a single round trip where the initiator knows the
responder's static key in advance.
[WireGuard](https://www.wireguard.com/) uses the same IK base pattern
(extended with a pre-shared key as IKpsk2) for VPN tunnels. FSP session
encryption uses **Noise XK**, the same pattern used by the
[Lightning Network](https://github.com/lightning/bolts/blob/master/08-transport.md),
where the initiator's static key is transmitted in a third message rather
than the first. XK provides stronger initiator identity hiding at the cost
of an additional round trip — a worthwhile tradeoff for session-layer traffic
that traverses untrusted intermediate nodes. At the link layer, where both
peers are configured and directly connected, IK's single round trip is
preferred.
### Index-Based Session Dispatch
FIPS uses locally-assigned 32-bit session indices to demultiplex incoming
packets to the correct cryptographic session in O(1) time, without parsing
source addresses or performing expensive lookups. This directly follows
[WireGuard's](https://www.wireguard.com/papers/wireguard.pdf) receiver index
approach, where each peer assigns a random index during handshake and the
remote side includes it in every packet header.
### Transport-Agnostic Overlay Mesh
FIPS is designed to operate over any datagram-capable transport — UDP, raw
Ethernet, Bluetooth, radio, serial — through a uniform transport abstraction.
Several mesh overlays have demonstrated transport-agnostic design:
[CJDNS](https://github.com/cjdelisle/cjdns) runs over UDP and Ethernet,
[Yggdrasil](https://yggdrasil-network.github.io/) supports TCP and TLS
transports, and [Tor](https://www.torproject.org/) can use pluggable
transports to tunnel through various media. FIPS extends this pattern to
shared-medium transports (radio, BLE) with per-transport MTU and discovery
capabilities.
### Metrics Measurement Protocol
MMP's design assembles well-established measurement techniques into a unified
per-link protocol. The SenderReport/ReceiverReport exchange structure follows
[RTCP](https://www.rfc-editor.org/rfc/rfc3550) (RFC 3550), which uses the
same report pairing for media stream quality monitoring in RTP sessions. MMP's
jitter computation uses the RTCP interarrival jitter algorithm directly.
The smoothed RTT estimator uses the Jacobson/Karels algorithm
([RFC 6298](https://www.rfc-editor.org/rfc/rfc6298)), the same SRTT
computation used in TCP for retransmission timeout calculation since 1988.
MMP derives RTT from timestamp-echo in ReceiverReports with dwell-time
compensation, rather than from packet round-trips.
The spin bit in the FMP frame header follows the
[QUIC](https://www.rfc-editor.org/rfc/rfc9000) spin bit
([RFC 9312](https://www.rfc-editor.org/rfc/rfc9312)) — a single bit that
alternates each round trip, enabling passive latency measurement. FIPS
implements the spin bit state machine but relies on timestamp-echo for SRTT,
as irregular mesh traffic makes spin bit RTT unreliable.
The Expected Transmission Count (ETX) metric, computed from bidirectional
delivery ratios, was introduced by
[De Couto et al. (2003)](https://pdos.csail.mit.edu/papers/grid:mobicom03/paper.pdf)
for wireless mesh routing and is used in protocols including
[OLSR](https://en.wikipedia.org/wiki/Optimized_Link_State_Routing_Protocol)
and [Babel](https://www.irif.fr/~jch/software/babel/). FIPS computes ETX
per-link from MMP loss measurements for future use in candidate ranking.
The CE (Congestion Experienced) echo flag provides hop-by-hop
[ECN](https://en.wikipedia.org/wiki/Explicit_Congestion_Notification)
signaling, following the TCP/IP ECN echo pattern (RFC 3168). Transit nodes
detect congestion via MMP loss/ETX metrics or kernel buffer drops and set
the CE flag on forwarded frames; destination nodes mark ECN-capable IPv6
packets accordingly.
### Cryptographic Primitives
FIPS reuses [Nostr's](https://github.com/nostr-protocol/nips) cryptographic
stack — secp256k1 for identity keys, Schnorr signatures for authentication,
SHA-256 for hashing, and ChaCha20-Poly1305 for authenticated encryption. This
is the same primitive set used across Bitcoin, Nostr, and a growing ecosystem
of self-sovereign identity systems. No novel cryptography is introduced.
---
## Further Reading
### Protocol Layers
| Document | Description |
| -------- | ----------- |
| [fips-transport-layer.md](fips-transport-layer.md) | Transport layer: abstraction, types, services provided to FMP |
| [fips-mesh-layer.md](fips-mesh-layer.md) | FMP: peer authentication, link encryption, forwarding |
| [fips-session-layer.md](fips-session-layer.md) | FSP: end-to-end encryption, session lifecycle |
| [fips-ipv6-adapter.md](fips-ipv6-adapter.md) | IPv6 adaptation: DNS, TUN interface, MTU enforcement |
### Mesh Behavior and Wire Formats
| Document | Description |
| -------- | ----------- |
| [fips-mesh-operation.md](fips-mesh-operation.md) | How the mesh operates: routing, discovery, error recovery |
| [fips-wire-formats.md](fips-wire-formats.md) | Complete wire format reference for all protocol layers |
### Supporting References
| Document | Description |
| -------- | ----------- |
| [fips-spanning-tree.md](fips-spanning-tree.md) | Spanning tree algorithms and data structures |
| [fips-bloom-filters.md](fips-bloom-filters.md) | Bloom filter parameters, math, and computation |
| [spanning-tree-dynamics.md](spanning-tree-dynamics.md) | Scenario walkthroughs: convergence, partitions, recovery |
### Implementation
| Document | Description |
| -------- | ----------- |
| [fips-configuration.md](fips-configuration.md) | YAML configuration reference |
### External References
- [IEEE 802.1D Spanning Tree Protocol](https://en.wikipedia.org/wiki/Spanning_Tree_Protocol)
- [Yggdrasil Network](https://yggdrasil-network.github.io/)
- [Yggdrasil v0.5 Release Notes](https://yggdrasil-network.github.io/2023/10/22/upcoming-v05-release.html)
- [Ironwood Routing Library](https://github.com/Arceliar/ironwood)
- [Kleinberg — The Small-World Phenomenon](https://www.cs.cornell.edu/home/kleinber/swn.pdf)
- [CJDNS](https://github.com/cjdelisle/cjdns)
- [Tor Project](https://www.torproject.org/)
- [I2P](https://geti2p.net/)
- [Host Identity Protocol (HIP)](https://en.wikipedia.org/wiki/Host_Identity_Protocol)
- [Babel Routing Protocol](https://www.irif.fr/~jch/software/babel/)
- [Noise Protocol Framework](https://noiseprotocol.org/)
- [WireGuard](https://www.wireguard.com/)
- [WireGuard Whitepaper](https://www.wireguard.com/papers/wireguard.pdf)
- [Lightning Network BOLT #8 — Transport](https://github.com/lightning/bolts/blob/master/08-transport.md)
- [QUIC (RFC 9000)](https://www.rfc-editor.org/rfc/rfc9000)
- [QUIC Spin Bit (RFC 9312)](https://www.rfc-editor.org/rfc/rfc9312)
- [RTCP (RFC 3550)](https://www.rfc-editor.org/rfc/rfc3550)
- [TCP SRTT / RTO (RFC 6298)](https://www.rfc-editor.org/rfc/rfc6298)
- [ECN (RFC 3168)](https://www.rfc-editor.org/rfc/rfc3168)
- [ETX — De Couto et al. 2003](https://pdos.csail.mit.edu/papers/grid:mobicom03/paper.pdf)
- [OLSR](https://en.wikipedia.org/wiki/Optimized_Link_State_Routing_Protocol)
- [Nostr Protocol](https://github.com/nostr-protocol/nips)
docs/design/fips-ipv6-adapter.md
+89 -99
View File
@@ -69,6 +69,24 @@ Known cache population mechanisms:
- **Inbound traffic**: Authenticated sessions from other nodes populate the
cache with their identity information
### Mesh-Interface Query Filter
The DNS responder is intended for local applications resolving `.fips`
names; queries arriving over the mesh interface itself are dropped. The
daemon records the index of the TUN interface at startup and compares
it against the arrival interface of each incoming UDP DNS query. When
they match — meaning the query came from another mesh node, not from a
local socket — the responder discards the query without replying.
The check is implemented in
[`is_mesh_interface_query`](../../src/upper/dns.rs) and prevents two
classes of misbehaviour: a peer asking the daemon to resolve `.fips`
names on its behalf (which would let one node use another as an
identity-cache priming proxy), and accidental query loops where a
misconfigured resolver forwards `.fips` queries back into the mesh.
Local applications binding to the host's loopback or non-mesh
interfaces are unaffected.
## IPv6 Address Derivation
FIPS addresses use the IPv6 Unique Local Address (ULA) prefix `fd00::/8`:
@@ -125,41 +143,24 @@ entry hasn't been evicted by memory pressure.
## MTU Enforcement
FIPS does not provide fragmentation or reassembly at the session or mesh
protocol layers — every datagram must fit in a single transport-layer packet.
Some transports may perform fragmentation and reassembly internally (e.g., BLE
L2CAP) and can advertise a larger virtual MTU than the physical medium
supports, but this is transparent to FIPS. The mesh layer provides two
facilities to manage MTU across heterogeneous paths: route discovery can
constrain results to paths that support a required minimum MTU, and transit
nodes that cannot forward an oversized datagram send an MtuExceeded error
signal back to the source. The adapter must ensure that IPv6 packets from
applications fit within the FIPS encapsulation budget after all layers of
wrapping.
The adapter sits at the boundary between the host's IPv6 stack and the
FIPS encapsulation budget. Its job is to keep IPv6 packets small
enough that they fit through the FIPS protocol envelope on every link
along the path. The cross-cutting MTU model — proactive
SessionDatagram `path_mtu` annotation, reactive MtuExceeded signals,
end-to-end PathMtuNotification echo, and per-destination MTU storage
— is documented in [fips-mtu.md](fips-mtu.md). What the adapter
contributes is the IPv6-specific overhead accounting and the TUN-side
enforcement integration.
### Encapsulation Overhead
### IPv6-Specific Overhead
| Layer | Overhead | Purpose |
| ----- | -------- | ------- |
| Link encryption | 37 bytes | 16-byte outer header + 5-byte inner header (timestamp + msg_type) + 16-byte AEAD tag |
| SessionDatagram body | 35 bytes | ttl + path_mtu + src_addr + dest_addr (msg_type counted in inner header) |
| FSP header | 12 bytes | 4-byte prefix + 8-byte counter (used as AEAD AAD) |
| FSP inner header | 6 bytes | 4-byte timestamp + 1-byte msg_type + 1-byte inner_flags (inside AEAD) |
| Session AEAD tag | 16 bytes | ChaCha20-Poly1305 tag on session-encrypted payload |
| **Protocol envelope** | **106 bytes** | `FIPS_OVERHEAD` constant |
| Port header | 4 bytes | src_port + dst_port (DataPacket service dispatch) |
| IPv6 compression | −33 bytes | 40-byte IPv6 header → 7-byte format + residual |
| **IPv6 data path total** | **77 bytes** | `FIPS_IPV6_OVERHEAD` constant |
Coordinate piggybacking (CP flag) adds variable overhead: `2 + entries × 16`
per coordinate, with both src and dst coords sent. The send path skips the
CP flag if adding coords would exceed the transport MTU.
The `FIPS_OVERHEAD` constant (106 bytes) represents the base protocol
envelope overhead (link encryption + routing + session encryption). For IPv6
traffic, FSP port multiplexing adds 4 bytes (port header) while IPv6 header
compression saves 33 bytes (40-byte header → 7-byte format + residual),
yielding a net `FIPS_IPV6_OVERHEAD` of 77 bytes.
For IPv6 traffic, FSP port multiplexing adds 4 bytes (port header)
while IPv6 header compression saves 33 bytes (40-byte header →
7-byte format + residual), yielding a net `FIPS_IPV6_OVERHEAD` of
77 bytes on top of the base `FIPS_OVERHEAD` (106 bytes) protocol
envelope. The full encapsulation breakdown lives in
[fips-mtu.md](fips-mtu.md#encapsulation-overhead).
### Effective IPv6 MTU
@@ -183,48 +184,51 @@ transport path MTU for the IPv6 adapter is therefore:
1280 + 77 = 1357 bytes
```
Transports with smaller MTUs (radio at ~250 bytes, serial at 256 bytes) cannot
support the IPv6 adapter without some form of internal fragmentation and
reassembly. Otherwise, applications on those transports must use the native
FIPS datagram API.
Transports with smaller MTUs (radio at ~250 bytes, serial at 256
bytes) cannot support the IPv6 adapter without some form of internal
fragmentation and reassembly. Otherwise, applications on those
transports must use the native FIPS datagram API.
### ICMP Packet Too Big
### TUN-Side ICMP Packet Too Big
When an outbound packet at the TUN exceeds the effective IPv6 MTU, the adapter
generates an ICMPv6 Packet Too Big message and delivers it back to the
application via the TUN. This triggers the kernel's Path MTU Discovery (PMTUD)
mechanism, which adjusts TCP segment sizes for subsequent transmissions.
When an outbound packet at the TUN exceeds the effective IPv6 MTU,
the adapter generates an ICMPv6 Packet Too Big message and delivers
it back to the application via the TUN. This triggers the kernel's
Path MTU Discovery mechanism, which adjusts TCP segment sizes for
subsequent transmissions.
ICMP Packet Too Big generation is rate-limited per source address (100ms
interval) to prevent storms from applications sending many oversized packets.
ICMP Packet Too Big generation is rate-limited per source address
(100ms interval) to prevent storms from applications sending many
oversized packets. The ICMP response is delivered locally back through
the TUN; no network traversal is needed, so delivery is reliable.
The ICMP response is delivered locally (back through the TUN to the kernel) —
no network traversal is needed, so delivery is reliable.
### TUN-Side TCP MSS Clamping
### TCP MSS Clamping
The adapter intercepts TCP SYN and SYN-ACK packets at the TUN interface and
clamps the Maximum Segment Size (MSS) option:
The adapter intercepts TCP SYN and SYN-ACK packets at the TUN
interface and clamps the Maximum Segment Size (MSS) option:
```text
clamped_mss = effective_ipv6_mtu - 40 (IPv6 header) - 20 (TCP header)
```
This prevents TCP connections from negotiating segment sizes that would exceed
the FIPS path MTU. Clamping is applied in two places:
Clamping is applied in two places:
- **TUN reader** (outbound): Clamps MSS on outbound SYN packets
- **TUN writer** (inbound): Clamps MSS on inbound SYN-ACK packets
Together, these ensure both directions of a TCP connection use appropriately
sized segments from the start, avoiding the initial oversized packet loss
that would occur with ICMP Packet Too Big alone.
Together, these ensure both directions of a TCP connection use
appropriately sized segments from the start, avoiding the initial
oversized packet loss that would occur with ICMP Packet Too Big
alone. The conditional clamp (per-flow lookup with cold-flow
fallback) and the rationale for `max_mss` semantics are in
[fips-mtu.md](fips-mtu.md#tcp-mss-clamping).
### ICMP Rate Limiting
ICMPv6 error generation is rate-limited per source address using a token bucket
(100ms interval). This matches the standard ICMP rate limiting approach and
prevents amplification when an application sends a burst of oversized packets.
ICMPv6 error generation is rate-limited per source address using a
token bucket (100ms interval). This matches the standard ICMP rate
limiting approach and prevents amplification when an application sends
a burst of oversized packets.
## TUN Interface
@@ -287,20 +291,16 @@ path.
### Configuration
```yaml
tun:
enabled: true
name: fips0
mtu: 1280
```
The TUN block (`tun.*`) is documented in
[../reference/configuration.md](../reference/configuration.md).
### Privileges
TUN device creation requires `CAP_NET_ADMIN`. Options:
- Run as root
- Set capability: `sudo setcap cap_net_admin+ep ./target/debug/fips`
- Pre-created persistent TUN device
TUN device creation requires `CAP_NET_ADMIN`. The shipped Debian
systemd unit runs the daemon as `root` by default; for the
alternative — running under a dedicated unprivileged service
account with the capability granted on the binary — see
[../how-to/run-as-unprivileged-user.md](../how-to/run-as-unprivileged-user.md).
## Implementation Status
@@ -314,6 +314,7 @@ TUN device creation requires `CAP_NET_ADMIN`. Options:
| ICMP rate limiting (per-source) | **Implemented** |
| TCP MSS clamping (SYN + SYN-ACK) | **Implemented** |
| DNS service (.fips domain) | **Implemented** |
| DNS responder mesh-interface filter | **Implemented** |
| Port-based service multiplexing (port 256) | **Implemented** |
| IPv6 header compression (format 0x00) | **Implemented** |
| Per-destination route MTU (netlink) | Planned |
@@ -324,38 +325,27 @@ TUN device creation requires `CAP_NET_ADMIN`. Options:
## Design Considerations
### Path MTU Discovery
### Path MTU Discovery and No-Fragmentation Policy
Two complementary mechanisms support full PMTUD:
1. **Proactive**: The `path_mtu` field (2 bytes) in the SessionDatagram envelope
is implemented at the FMP level. The source sets it to its outbound link MTU
minus overhead; each transit node applies
`min(current, own_outbound_mtu - overhead)`. The destination receives the
forward-path minimum. PathMtuNotification is handled at the session layer;
the destination sends the observed forward-path MTU back to the source,
which applies it with decrease-immediate / increase-requires-3-consecutive
hysteresis.
2. **Reactive**: When a transit node cannot forward a packet (MTU exceeded), it
sends an error signal back to the source. This handles the in-flight gap
between a path MTU decrease and the source learning via the echo.
Both are needed: proactive handles steady state; reactive handles the transient
window when oversized packets hit a new bottleneck before the source adapts.
### No Fragmentation
FIPS remains a pure datagram service with no fragmentation at transit nodes.
Session-layer encryption is end-to-end — the AEAD tag authenticates the entire
plaintext. Fragmenting encrypted datagrams would require either exposing
plaintext structure to transit nodes (unacceptable) or reassembly before
decryption (opens attack surface).
Path MTU Discovery (proactive `path_mtu` annotation, reactive
MtuExceeded, end-to-end PathMtuNotification) and the no-fragmentation
policy that drives the design both live in the unified MTU treatment
at [fips-mtu.md](fips-mtu.md). The adapter is a consumer of that
model — its job is to enforce the resulting effective IPv6 MTU at the
TUN with ICMP Packet Too Big and TCP MSS clamping.
## References
- [fips-intro.md](fips-intro.md) — Protocol overview and architecture
- [fips-concepts.md](fips-concepts.md) — Protocol overview
- [fips-architecture.md](fips-architecture.md) — Layer architecture and
identity model
- [fips-session-layer.md](fips-session-layer.md) — FSP (below the adapter)
- [fips-wire-formats.md](fips-wire-formats.md) — FSP and SessionDatagram wire
formats
- [fips-configuration.md](fips-configuration.md) — TUN configuration parameters
- [fips-mtu.md](fips-mtu.md) — Unified path MTU model (proactive,
reactive, hysteresis, no-fragmentation)
- [../reference/wire-formats.md](../reference/wire-formats.md) — FSP and
SessionDatagram wire formats
- [../reference/configuration.md](../reference/configuration.md) — TUN
configuration parameters
- [../how-to/run-as-unprivileged-user.md](../how-to/run-as-unprivileged-user.md)
— privilege options for the daemon, including the unprivileged
service-account path
docs/design/fips-mesh-layer.md
+135 -145
View File
@@ -215,8 +215,8 @@ The plaintext inside the encrypted frame begins with a 5-byte inner header
(4-byte session-relative timestamp followed by a message type byte), then the
message-specific payload.
See [fips-wire-formats.md](fips-wire-formats.md) for the complete wire format
specification.
See [../reference/wire-formats.md](../reference/wire-formats.md) for
the complete wire format specification.
### What Encryption Provides
@@ -296,7 +296,7 @@ Roaming addresses *mid-session* NAT rebinding. Establishing the initial UDP
path through NAT is a separate concern, addressed by the optional
Nostr-mediated overlay discovery and STUN-assisted hole punching feature
(see [fips-transport-layer.md](fips-transport-layer.md) and
[fips-configuration.md](fips-configuration.md)).
[../reference/configuration.md](../reference/configuration.md)).
## Replay Protection
@@ -341,7 +341,9 @@ Additional protections:
memory usage
- **Handshake timeout**: Stale pending handshakes are cleaned up after a
configurable timeout
- **Allowlist/blocklist**: Optional peer filtering before handshake processing
- **Peer ACL**: Optional allowlist / denylist filtering of peer npubs
before handshake processing (loaded from `/etc/fips/peers.allow` and
`/etc/fips/peers.deny`, mtime-watched and reloaded automatically)
## Disconnect
@@ -361,6 +363,72 @@ links.
On node shutdown, Disconnect is sent to all active peers before transports are
stopped.
## Rekey
FMP periodically negotiates a fresh Noise session over each established link
to bound forward-secrecy exposure: limiting AEAD nonce reuse risk, bounding
the volume of ciphertext recoverable from a stolen long-term static key, and
rotating the session indices that addressed packets carry on the wire.
A rekey is initiated when either threshold is reached on the link's current
session: `node.rekey.after_secs` (default 120) elapsed since the link came
up or last rekeyed, or `node.rekey.after_messages` (default 65536) frames
sent. Either side can be the initiator independently. Rekey is on by
default and can be disabled via `node.rekey.enabled: false` (the
configuration tree is documented in
[../reference/configuration.md](../reference/configuration.md)).
### Mechanism
A rekey reuses the Noise IK pattern of the initial handshake, but the two
messages travel over the existing link as ordinary encrypted FMP frames
rather than as plaintext bootstrap packets. The initiator builds a fresh
`HandshakeState`, generates msg1, and sends it through the current session;
the responder consumes msg1, builds msg2, and replies. After both sides
have exchanged messages and finalised the new keys, traffic transitions
from the old session to the new one.
Cutover is signalled in-band by the **K-bit** in the FMP flags byte. Each
side starts emitting frames under the new session with K set; on receipt
of the first K-marked frame the peer accepts the cutover and follows
suit. A new pair of session indices is allocated as part of the new
session, replacing the old indices on subsequent frames (see
[Index Properties](#index-properties)).
### Drain Window
To absorb in-flight reordering across the cutover, the old session is not
discarded immediately. Each peer retains it in a `previous_session` slot
on the active-peer state for `DRAIN_WINDOW_SECS = 10` seconds (a
compile-time constant in `src/node/handlers/rekey.rs`). During the
window, decrypt attempts fall back to `previous_session` when the new
session rejects a frame, so a packet sent under the old keys that
arrives a few hundred milliseconds late still decrypts. After the
window expires, the old session is dropped.
### Dual-Initiation Race
On high-latency links, both sides' rekey timers can fire close enough
together that each peer's msg1 crosses the other in flight. Without
arbitration, each side would act as both initiator and responder, end
up with two different Noise sessions, and lose connectivity at cutover.
FMP arbitrates with a deterministic tie-breaker: the peer with the
**numerically smaller `NodeAddr`** wins the role of initiator and
discards any inbound msg1 it sees during the race; the larger-`NodeAddr`
peer abandons its own initiation and processes the inbound msg1 as
responder. The same tie-breaker is applied to cross-connection races
during initial handshake.
### Operator Visibility
Successful cutover is reported at INFO level on the K-bit observation;
intermediate steps (handshake start, msg1/msg2 exchange, drain-window
fallback decrypts) log at DEBUG/TRACE. Failures (handshake error,
drain-window expiry without cutover) log at WARN.
The end-to-end rekey at the session layer follows a parallel design;
see [fips-session-layer.md](fips-session-layer.md).
## Liveness Detection
FMP detects link liveness through a combination of explicit heartbeats and
@@ -368,8 +436,8 @@ traffic observation.
### Heartbeat
A Heartbeat message (0x51) is sent to each active peer every
`node.heartbeat_interval_secs` (default 10s). The heartbeat is a minimal
A Heartbeat message (0x51) is sent to each active peer at a configurable
interval (`node.heartbeat_interval_secs`). The heartbeat is a minimal
encrypted frame with no payload beyond the standard inner header (timestamp +
message type). Any successfully decrypted frame — data, gossip, MMP report,
or heartbeat — resets the peer's last-receive timestamp tracked by the MMP
@@ -377,8 +445,8 @@ receiver.
### Dead Timeout
When no traffic (of any kind) is received from a peer for
`node.link_dead_timeout_secs` (default 30s), the peer is declared dead and
When no traffic (of any kind) is received from a peer for the
configured `node.link_dead_timeout_secs` window, the peer is declared dead and
removed via `remove_active_peer()`. This triggers the full teardown cascade:
spanning tree parent reselection (if the dead peer was the parent),
TreeAnnounce propagation, coordinate cache flush, and bloom filter recompute.
@@ -386,156 +454,74 @@ TreeAnnounce propagation, coordinate cache flush, and bloom filter recompute.
If the dead peer is eligible for auto-reconnect (see [Auto-Reconnect]
(#auto-reconnect)), reconnection is scheduled immediately after removal.
The heartbeat is independent of MMP — it is needed because idle links in
Lightweight MMP mode have no guaranteed periodic traffic (gossip is
event-driven, and MMP reports require at least one side running Full mode).
The heartbeat is independent of MMP. Gossip is event-driven, Lightweight
produces receiver reports only when traffic arrives, and Minimal emits no
reports at all — so on a fully idle link no MMP-mode combination
guarantees periodic activity. The heartbeat is the always-on liveness
signal.
## Link Message Types
FMP defines eight message types carried inside encrypted frames:
FMP defines several encrypted message types carried inside the
established-frame envelope. They group naturally by purpose:
| Type | Name | Purpose |
| ---- | ---- | ------- |
| 0x10 | TreeAnnounce | Spanning tree state announcements between peers |
| 0x20 | FilterAnnounce | Bloom filter reachability updates |
| 0x30 | LookupRequest | Coordinate discovery — flood toward destination |
| 0x31 | LookupResponse | Coordinate discovery — response with coordinates |
| 0x00 | SessionDatagram | Encapsulated session-layer payload for forwarding |
| 0x01 | SenderReport | MMP sender-side metrics report |
| 0x02 | ReceiverReport | MMP receiver-side metrics report |
| 0x50 | Disconnect | Orderly link teardown with reason code |
| 0x51 | Heartbeat | Link liveness probe |
- **Routing gossip**: TreeAnnounce carries spanning-tree announcements
between direct peers; FilterAnnounce carries bloom-filter
reachability updates between direct peers. Both are peer-to-peer
(not forwarded).
- **Discovery**: LookupRequest is forwarded through tree peers under
bloom-filter guidance to find a destination's coordinates;
LookupResponse routes back to the requester via reverse-path lookup
in `recent_requests`.
- **Forwarded payload**: SessionDatagram carries a session-layer
payload hop-by-hop toward the destination.
- **Metrics**: SenderReport and ReceiverReport carry the link-layer
MMP report stream peer-to-peer.
- **Liveness and lifecycle**: Heartbeat is a minimal frame sent
peer-to-peer to keep the link alive; Disconnect carries an orderly
teardown reason code peer-to-peer.
Additionally, handshake messages (phase 0x1 msg1, phase 0x2 msg2) are sent
unencrypted before the link session is established.
Handshake messages (phase 0x1 msg1, phase 0x2 msg2) travel before
encryption is established and are identified by the FMP common-prefix
`phase` field rather than a `msg_type` byte.
TreeAnnounce and FilterAnnounce are exchanged between direct peers only — they
are not forwarded. LookupRequest and LookupResponse are forwarded through the
mesh (flooded with deduplication). SessionDatagram is forwarded hop-by-hop
toward the destination. Disconnect is peer-to-peer.
See [fips-mesh-operation.md](fips-mesh-operation.md) for how these messages
work together to build and maintain the mesh, and
[fips-wire-formats.md](fips-wire-formats.md) for byte-level message layouts.
See [../reference/wire-formats.md](../reference/wire-formats.md) for
byte-level message layouts and the canonical FMP message type
catalog, and [fips-mesh-operation.md](fips-mesh-operation.md) for how
these messages work together to build and maintain the mesh.
## Metrics Measurement Protocol (MMP)
Each active peer link runs an instance of the Metrics Measurement Protocol,
providing per-link quality metrics to the operator and to the spanning tree
layer for cost-based parent selection.
MMP runs on every active link to provide per-link quality metrics
(SRTT, loss, jitter, goodput, OWD trend, ETX) to the operator and to
the spanning tree layer for cost-based parent selection. Reports are
exchanged peer-to-peer between direct neighbors at RTT-adaptive
intervals clamped to `[1s, 5s]`, with a 200 ms cold-start floor for
the first five SRTT samples.
### Metrics Tracked
The CE (Congestion Experienced) bit in the FMP flags byte carries
hop-by-hop ECN signaling: transit nodes detect congestion on outgoing
links (via MMP loss/ETX or `SO_RXQ_OVFL` kernel drops) and set CE on
forwarded packets, which the destination then mirrors to the IPv6
Traffic Class for ECN-capable flows.
MMP computes the following metrics from the per-frame counter and timestamp
fields in the FMP wire format:
- **SRTT** — Smoothed round-trip time (Jacobson/RFC 6298, α=1/8). Derived
from timestamp-echo in ReceiverReports with dwell-time compensation.
- **Loss rate** — Bidirectional loss inferred from counter gaps. Tracked as
both instantaneous (per-interval) and long-term EWMA.
- **Jitter** — Interarrival jitter (RFC 3550 algorithm) in microseconds.
- **Goodput** — Bytes per second of payload data (excludes MMP reports).
- **OWD trend** — One-way delay trend (µs/s, signed). Indicates congestion
buildup before loss occurs.
- **ETX** — Expected Transmission Count, computed from bidirectional delivery
ratios. Used in cost-based parent selection via
`link_cost = etx * (1.0 + srtt_ms / 100.0)`; not yet used in
`find_next_hop()` candidate ranking.
- **Dual EWMA trends** — Short-term (α=1/4) and long-term (α=1/32) trend
indicators for both RTT and loss, enabling change detection.
### Operating Modes
MMP supports three modes, configured via `node.mmp.mode`:
| Mode | Reports Exchanged | Metrics Available |
| ---- | ----------------- | ----------------- |
| **Full** (default) | SenderReport + ReceiverReport | All metrics including RTT, loss, jitter, goodput, OWD trend |
| **Lightweight** | ReceiverReport only | Loss (from counter gaps), jitter, OWD trend. No RTT. |
| **Minimal** | None | Spin bit and CE echo flags only. No computed metrics. |
### Report Scheduling
Reports are sent at RTT-adaptive intervals, clamped to [100ms, 2s]. A
cold-start interval of 500ms is used before SRTT converges. The interval
formula is `clamp(2 × SRTT, 100ms, 2000ms)`.
### Spin Bit and RTT
The SP (spin bit) flag in the FMP inner header follows the QUIC spin bit
pattern: reflected on receive, toggled on send when the reflected value
matches the last sent value. The spin bit state machine runs for TX
reflection, but **RTT samples from the spin bit are discarded**. In a mesh
protocol where frames are sent irregularly (tree announces, bloom filters,
MMP reports on different timers), inter-frame processing delays inflate spin
bit RTT measurements unpredictably. Timestamp-echo from ReceiverReports
(with dwell-time compensation) is the sole SRTT source.
### ECN Congestion Signaling
The CE (Congestion Experienced) flag (bit 1 in the FMP flags byte) provides
hop-by-hop congestion signaling through the mesh. Transit nodes detect
congestion on outgoing links and set CE on forwarded packets; once set, the
flag stays set for all subsequent hops to the destination.
**Congestion detection** (`detect_congestion()`) triggers on any of:
- Outgoing link MMP loss rate ≥ `node.ecn.loss_threshold` (default 5%)
- Outgoing link MMP ETX ≥ `node.ecn.etx_threshold` (default 3.0)
- Kernel receive buffer drops detected on any local transport (via
`SO_RXQ_OVFL` on UDP)
**CE relay**: The forwarding path computes `outgoing_ce = incoming_ce ||
local_congestion`. The `send_encrypted_link_message_with_ce()` method ORs
`FLAG_CE` into the FMP header flags when ce is true. The original
`send_encrypted_link_message()` delegates with `ce_flag=false`, leaving the
20+ existing call sites unchanged.
**IPv6 ECN-CE marking**: When a CE-flagged DataPacket arrives at its final
destination, the IPv6 Traffic Class ECN bits are marked CE (0b11) before
TUN delivery — but only for ECN-capable packets (ECT(0) or ECT(1)). Not-ECT
packets are never marked per RFC 3168. The host TCP stack then echoes ECE in
ACKs, triggering sender cwnd reduction through standard congestion control.
**Session-layer tracking**: The `ecn_ce_count` field in MMP ReceiverReports
tracks CE-flagged packets received per link, providing end-to-end visibility
into congestion propagation.
**Monitoring**: `CongestionStats` tracks four counters — `ce_forwarded`,
`ce_received`, `congestion_detected`, and `kernel_drop_events` — exposed via
`fipsctl show routing` (congestion block) and `fipstop` (routing tab).
Rate-limited warn logging (5s interval) alerts on congestion detection events.
See `node.ecn.*` in
[fips-configuration.md](fips-configuration.md#ecn-signaling-nodeecn) for
tuning parameters.
### Operator Logging
MMP emits periodic link metrics at info level (configurable via
`node.mmp.log_interval_secs`, default 30s):
```text
MMP link metrics peer=node-b rtt=2.3ms loss=0.2% jitter=0.1ms goodput=76.0MB/s tx_pkts=1234 rx_pkts=5678
```
Teardown logs include final SRTT, loss rate, jitter, ETX, goodput, and
cumulative tx/rx packet and byte counts.
For the full MMP design — operating modes, report scheduling, spin
bit interaction, ECN, and the algorithmic details shared with
session-layer MMP — see [fips-mmp.md](fips-mmp.md). For the
SenderReport and ReceiverReport byte layouts, see
[../reference/wire-formats.md](../reference/wire-formats.md).
Configuration knobs live under `node.mmp.*` and `node.ecn.*` in
[../reference/configuration.md](../reference/configuration.md).
## Security Properties
### Threat Resistance
| Threat | Mitigation |
| ------ | ---------- |
| Connection exhaustion | Token bucket rate limit + connection count limit |
| CPU exhaustion (msg1 flood) | Rate limit before crypto operations |
| Replay attacks | Counter-based nonces with sliding window |
| State confusion | Strict handshake state machine validation |
| Spoofed encrypted packets | Index lookup + AEAD verification |
| Spoofed msg2 | Index lookup + Noise ephemeral key binding |
| Address spoofing | Cryptographic authority, not address-based |
| Session correlation | Index rotation on rekey |
The link-layer threat-resistance matrix (connection exhaustion, CPU
exhaustion, replay, state confusion, spoofing variants, address
spoofing, session correlation) is consolidated in
[../reference/security.md](../reference/security.md) along with the
session-layer matrix and operator-facing controls.
### Unauthenticated Attack Surface
@@ -580,13 +566,17 @@ an attacker sends invalid packets to elicit responses.
| Metrics Measurement Protocol (MMP) | **Implemented** |
| ECN congestion signaling (CE relay, IPv6 marking) | **Implemented** |
| Rekey with index rotation | **Implemented** |
| Allowlist/blocklist | Planned |
| Peer ACL (allowlist / denylist) | **Implemented** |
## References
- [fips-intro.md](fips-intro.md) — Protocol overview and architecture
- [fips-concepts.md](fips-concepts.md) — Protocol overview
- [fips-architecture.md](fips-architecture.md) — Layer architecture and
identity model
- [fips-transport-layer.md](fips-transport-layer.md) — Transport layer (below FMP)
- [fips-session-layer.md](fips-session-layer.md) — FSP (above FMP)
- [fips-mmp.md](fips-mmp.md) — Metrics Measurement Protocol (link + session)
- [fips-mesh-operation.md](fips-mesh-operation.md) — How FMP's routing and
self-organization work in practice
- [fips-wire-formats.md](fips-wire-formats.md) — Byte-level wire format reference
- [../reference/wire-formats.md](../reference/wire-formats.md) — Byte-level
wire format reference
docs/design/fips-mesh-operation.md
+99 -310
View File
@@ -31,183 +31,51 @@ and self-healing.
## Spanning Tree Formation and Maintenance
### What the Spanning Tree Provides
For routing purposes, the spanning tree provides each node with a
coordinate (its ancestry path from itself to the root) plus a way to
compute distance between any two nodes (hops to their lowest common
ancestor). The strictly-decreasing distance invariant gives greedy
forwarding its loop-freedom.
The spanning tree gives each node a **coordinate**: its ancestry path from
itself to the root, expressed as a sequence of node_addrs. These coordinates
enable:
The tree forms through distributed parent selection — root is the
smallest node_addr (no election), and each node picks the peer with
the lowest `effective_depth = depth + link_cost`. Cost-aware parent
selection lets the tree trade hop count for link quality once MMP has
accumulated SRTT and ETX metrics. Hysteresis (20% improvement
required to switch) and hold-down (suppress non-mandatory
re-evaluation after a switch) keep the tree stable under metric
noise. Partitions self-resolve — each segment converges to its own
root and reconverges to the smallest reachable root when segments
rejoin.
- **Distance calculation**: The tree distance between two nodes is the number
of hops from each to their lowest common ancestor (LCA). This provides a
routing metric without any node knowing the full topology.
- **Greedy routing**: At each hop, forward to the peer that minimizes tree
distance to the destination. The strictly-decreasing distance invariant
guarantees loop-free forwarding.
Liveness is detected via FMP heartbeats; dead-peer removal triggers
tree reconvergence and bloom filter recomputation for the affected
subtree. The heartbeat and dead-timeout mechanism lives at the link
layer; see [fips-mesh-layer.md](fips-mesh-layer.md#liveness-detection).
### How the Tree Forms
Nodes self-organize into a spanning tree through distributed parent selection:
1. **Root discovery**: The node with the smallest node_addr becomes the root.
No election protocol — this is a consequence of each node independently
preferring lower-addressed roots.
2. **Parent selection**: Each node selects a single parent from among its
direct peers based on which offers the lowest effective depth (tree depth
weighted by local link cost).
3. **Coordinate computation**: Once a node has a parent, its coordinate is
computed from its ancestry path.
### How the Tree Maintains Itself
Nodes exchange **TreeAnnounce** messages with their direct peers (not
forwarded — peer-to-peer only). Each TreeAnnounce carries the sender's
current ancestry chain and a sequence number.
Changes cascade through the tree:
- A node that changes its parent recomputes its coordinates and announces to
all peers
- Each receiving peer evaluates whether the change affects its own parent
selection
- Only nodes that actually change their coordinates (root or depth changed)
propagate further
TreeAnnounce propagation is rate-limited at 500ms minimum interval per peer.
A tree of depth D reconverges in roughly D×0.5s to D×1.0s.
### How the Tree Adapts to Link Quality
The initial tree forms based on hop count alone — all links default to a
cost of 1.0 before measurements are available. As the Metrics Measurement
Protocol (MMP) accumulates bidirectional delivery ratios and round-trip
time estimates, each node computes a per-link cost:
```text
link_cost = ETX × (1.0 + SRTT_ms / 100.0)
```
ETX (Expected Transmission Count) captures loss — a perfect link has
ETX = 1.0, while 10% loss in each direction yields ETX ≈ 1.23. The SRTT
term weights latency so that a low-loss but high-latency link (e.g., a
satellite hop) costs more than a low-loss, low-latency link.
Parent selection uses **effective depth** rather than raw hop count:
```text
effective_depth = peer.depth + link_cost_to_peer
```
This allows a node to trade a shorter but lossy path for a longer but
higher-quality one. A node two hops from the root over clean links
(effective depth ≈ 3.0) is preferred over a node one hop away over a
degraded link (effective depth ≈ 4.5).
Parent reselection is triggered by three paths:
1. **TreeAnnounce**: When a peer announces a new tree position, the node
re-evaluates using current link costs
2. **Periodic re-evaluation**: Every 60s (configurable), the node
re-evaluates its parent choice using the latest MMP metrics, catching
gradual link degradation that doesn't trigger TreeAnnounce
3. **Parent loss**: When the current parent is removed, the node
immediately selects the best alternative
To prevent oscillation from metric noise, parent switches are subject to
**hysteresis**: a candidate must offer an effective depth at least 20%
better than the current parent to trigger a switch. A **hold-down period**
(default 30s) suppresses non-mandatory re-evaluation after a switch,
allowing MMP metrics to stabilize on the new link before reconsidering.
### Flap Dampening
Unstable links that repeatedly connect and disconnect can cause cascading
tree reconvergence. The spanning tree uses flap dampening with hysteresis
and hold-down periods to suppress rapid parent oscillation. Links that flap
above a configurable threshold are temporarily penalized, preventing them
from being selected as parent until the link stabilizes.
### Link Liveness
Each node sends a dedicated **Heartbeat** message (0x51, 1 byte, no
payload) to every peer at a fixed interval (default 10s). Any
authenticated encrypted frame — heartbeat, MMP report, TreeAnnounce,
data packet — resets the peer's liveness timer. On an idle link with no
application data or topology changes, the heartbeat is the only traffic
that keeps the link alive.
Peers that are silent for a configurable dead timeout (default 30s) are
considered dead and removed from the peer table. With the default 10s
heartbeat interval, a peer must miss three consecutive heartbeats before
removal. This triggers tree reconvergence and bloom filter recomputation
for the affected subtree.
### Partition Handling
If the network partitions, each segment independently rediscovers its own
root (the smallest node_addr in the segment) and reconverges. When segments
rejoin, nodes discover the globally-smallest root through TreeAnnounce
exchange and reconverge to a single tree.
See [fips-spanning-tree.md](fips-spanning-tree.md) for algorithm details
and [spanning-tree-dynamics.md](spanning-tree-dynamics.md) for convergence
walkthroughs.
For the parent-selection algorithm, hold-down/hysteresis details, and
the convergence walkthroughs, see
[fips-spanning-tree.md](fips-spanning-tree.md) and
[spanning-tree-dynamics.md](spanning-tree-dynamics.md).
## Bloom Filter Gossip and Propagation
### What Bloom Filters Provide
For routing purposes, each node maintains a bloom filter per peer
that answers "can peer P possibly reach destination D?" — either "no"
(definitive) or "maybe" (probabilistic). Because filters propagate
along tree edges with split-horizon exclusion, a bloom hit on a tree
peer reliably indicates which subtree contains the destination, and
tree-coordinate distance ranks competing matches.
Each node maintains a bloom filter per peer, answering: "can peer P possibly
reach destination D?" The answer is either "no" (definitive) or "maybe"
(probabilistic — false positives are possible).
FilterAnnounce updates are event-driven (peer changes, tree
restructuring, local identity changes) and rate-limited to prevent
storms. False positives at large scale never cause loops — the
self-distance check at each hop guarantees forward progress, and
mismatched bloom matches fall through to greedy tree routing.
Because filters propagate along tree edges with split-horizon exclusion,
they encode directional reachability: a bloom hit on a tree peer reliably
indicates which subtree contains the destination. When multiple peers match,
tree coordinate distance ranks them.
### How Filters Propagate
Nodes exchange **FilterAnnounce** messages with all direct peers. Each
FilterAnnounce replaces the previous filter for that peer — there is no
incremental update.
Filter computation uses **tree-only merge with split-horizon exclusion**:
the outbound filter for peer Q is computed by merging the local node's own
identity, its leaf-only dependents (if any), and the inbound filters from
tree peers (parent and children) *except* Q. Filters from non-tree mesh
peers are stored locally for routing queries but are not merged into
outgoing filters. This prevents saturation where mesh shortcuts cause
filters to converge toward the full network.
The restriction creates **directional asymmetry**: upward filters
(child → parent) contain the child's subtree, while downward filters
(parent → child) contain the complement. Together they cover the entire
network.
Filters propagate transitively through tree edges. At steady state, every
reachable destination appears in at least one tree peer's filter.
### Update Triggers
Filter updates are event-driven, not periodic:
- Peer connects or disconnects
- A peer's incoming filter changes (triggers recomputation for other peers)
- Tree relationship changes (new parent, new child, parent switch)
- Local state changes (new identity, leaf-only dependent changes)
Updates are rate-limited at 500ms to prevent storms during topology changes.
### Scale Properties
At moderate network sizes, bloom filters are highly accurate. At larger
scales (~1M nodes), hub nodes with many peers may see elevated false positive
rates (7–15% for nodes with 20+ peers). False positives may cause a packet
to be forwarded toward the wrong subtree, but the self-distance check at
each hop prevents loops and the packet falls through to greedy tree routing.
See [fips-bloom-filters.md](fips-bloom-filters.md) for filter parameters,
FPR calculations, and size class folding.
For the filter computation, split-horizon merge rules, FPR analysis,
size classes, and folding, see
[fips-bloom-filters.md](fips-bloom-filters.md).
## Routing Decision Process
@@ -222,34 +90,33 @@ priority chain. This is the core routing algorithm.
2. **Direct peer** — The destination is an authenticated neighbor. Forward
directly. No coordinates or bloom filters needed.
3. **Bloom-guided routing** — One or more peers' bloom filters contain the
3. **Coordinate cache check** — Multi-hop forwarding requires the
destination's tree coordinates to be in the local cache. On miss,
`find_next_hop()` returns None immediately — bloom filters are never
consulted — and the source receives a CoordsRequired error signal.
4. **Bloom-guided routing** — One or more peers' bloom filters contain the
destination. Select the best peer by composite key:
`(link_cost, tree_distance, node_addr)`. This requires the destination's
tree coordinates to be in the local coordinate cache.
`(link_cost, tree_distance, node_addr)`.
4. **Greedy tree routing** — Fallback when bloom filters haven't converged
for this destination. Forward to the peer that minimizes tree distance.
Also requires destination coordinates.
5. **Greedy tree routing** — Fall-through when bloom yields no candidate.
Forward to the peer that minimizes tree distance. If the tree has no
next hop closer to the destination, the source receives a PathBroken
error signal.
5. **No route** — Destination unreachable. Generate an error signal
(CoordsRequired or PathBroken) back to the source.
### Convergence Requirements
### The Coordinate Requirement
All multi-hop routing (steps 3–4) requires the destination's tree coordinates
to be in the local coordinate cache. Without coordinates, `find_next_hop()`
returns None immediately — bloom filters are never even consulted.
This creates two simultaneous convergence requirements for multi-hop routing:
Multi-hop routing depends on two propagation processes that must run
to convergence simultaneously:
1. **Bloom convergence**: Filters must propagate so peers advertise
reachability
2. **Coordinate availability**: Destination coordinates must be cached at
every transit node on the path
Both must be satisfied simultaneously. Bloom convergence without coordinates
causes a coordinate cache miss. Coordinates without bloom convergence falls
through to greedy tree routing (functional but suboptimal).
Bloom convergence without coordinates trips step 3 (coord-cache miss →
CoordsRequired). Coordinates without bloom convergence falls through to
greedy tree routing — functional but suboptimal.
### Candidate Ranking
@@ -270,6 +137,10 @@ A peer with a bloom filter hit but no entry in the peer ancestry table
(missing TreeAnnounce) defaults to maximum distance and is effectively
invisible to routing.
### Routing Decision Flowchart
![Per-hop routing decision flowchart](diagrams/fips-routing-decision.svg)
### Loop Prevention
The routing decision enforces strict progress: a packet is only forwarded
@@ -284,48 +155,12 @@ PathBroken error.
## Coordinate Caching
The coordinate cache maps `NodeAddr → TreeCoordinate` and is the critical
data structure for multi-hop routing. Without it, forwarding decisions cannot
be made.
### Unified Cache
The coordinate cache is a single unified cache. All sources — SessionSetup
transit, CP-flagged data packets, LookupResponse — write to the same cache.
### Population Sources
| Source | When | What |
| ------ | ---- | ---- |
| SessionSetup transit | Session establishment | Both src and dest coordinates |
| SessionAck transit | Session establishment | Both src and dest coordinates |
| CP-flagged data packet | Warmup or recovery | Both src and dest coordinates (cleartext) |
| LookupResponse | Discovery | Target's coordinates |
### Eviction
- **TTL-based**: Entries expire after 300s (configurable)
- **Refresh on use**: Active routing refreshes the TTL, keeping hot entries
alive
- **LRU**: When full, least recently used entries are evicted first
- **Flush on parent change**: When the local node's tree parent changes, the
entire cache is flushed. Parent changes mean the node's own coordinates
have changed, making relative distance calculations with cached coordinates
potentially invalid. Flushing is preferred over stale routing: the cost of
re-discovery is lower than routing packets to dead ends.
### Cache and Session Timer Ordering
Timer values are ordered so that idle sessions tear down before transit
caches expire:
| Timer | Default | Purpose |
| ----- | ------- | ------- |
| Session idle | 90s | Session teardown |
| Coordinate cache TTL | 300s | Coordinate expiration |
When traffic stops, the session tears down at 90s. When traffic resumes, a
fresh SessionSetup re-warms transit caches (still within their 300s TTL).
The coordinate cache maps `NodeAddr → TreeCoordinate` and is the
critical data structure for multi-hop routing. The session layer owns
this cache (its eviction policy, TTL/refresh semantics, parent-change
flush, and timer ordering with session idle timeout); see
[fips-session-layer.md](fips-session-layer.md#coordinate-cache) for
the canonical treatment.
## Discovery Protocol
@@ -375,28 +210,17 @@ where a request might arrive via both tree and fallback paths.
Single-path forwarding is more fragile than flooding — if any transit node
on the path has a stale bloom filter or loses a link, the request fails.
To compensate, the originator retries:
To compensate, each discovery is a sequence of attempts with growing
per-attempt timeouts. The default sequence is `[1s, 2s, 4s, 8s]`
(configurable via `node.discovery.attempt_timeouts_secs`); the destination
is declared unreachable only after the full sequence is exhausted (15s
total at default).
- **T=0**: Initial lookup sent
- **T=5s**: Retry if no response (configurable via `retry_interval_secs`)
- **T=10s**: Timeout, fail (configurable via `timeout_secs`)
The default `max_attempts` is 2 (initial + one retry). Each retry generates
a fresh `request_id` and re-evaluates bloom filter matches, so it can take
a different path if the tree has restructured.
### Per-Attempt Timeouts
Each discovery is a sequence of attempts with growing per-attempt timeouts.
Default sequence is `[1s, 2s, 4s, 8s]` (configurable via
`node.discovery.attempt_timeouts_secs`). When the current attempt's deadline
elapses without a `LookupResponse`, the originator sends another
`LookupRequest` with a **fresh `request_id`** and the next entry in the
sequence as its deadline. Fresh request_ids let each attempt take a
different forwarding path as the bloom and tree state evolve, which is
particularly useful during cold-start convergence. The destination is
declared unreachable only after the full sequence is exhausted (15s total
with the default).
When the current attempt's deadline elapses without a `LookupResponse`,
the originator sends another `LookupRequest` with a **fresh `request_id`**
and the next entry in the sequence as its deadline. Fresh `request_id`s
let each attempt take a different forwarding path as the bloom and tree
state evolve, which is particularly useful during cold-start convergence.
### Originator Backoff (optional, off by default)
@@ -462,6 +286,10 @@ verification at the source confirms the target holds the claimed position.
The `path_mtu` field is excluded from the proof because it is a transit
annotation modified at each hop.
### Coordinate Discovery Sequence
![Coordinate discovery and cache warming sequence](diagrams/fips-coordinate-discovery.svg)
### Discovery Outcome
On receiving a verified LookupResponse, the source caches the target's
@@ -473,50 +301,19 @@ If discovery times out (no response after all retry attempts), queued
packets receive ICMPv6 Destination Unreachable and the target enters
backoff.
## SessionSetup Self-Bootstrapping
## Coordinate Cache Warming
SessionSetup is the mechanism that warms transit node coordinate caches
along a path, enabling subsequent data packets to route efficiently.
### How It Works
SessionSetup carries plaintext coordinates (outside the Noise handshake
payload, visible to transit nodes):
- **src_coords**: Source's current tree coordinates
- **dest_coords**: Destination's tree coordinates (learned from discovery)
As the SessionSetup transits each intermediate node:
1. The transit node extracts both coordinate sets
2. Caches `src_addr → src_coords` and `dest_addr → dest_coords` in its
coordinate cache
3. Forwards the message using the cached destination coordinates
SessionAck returns along the reverse path, carrying both the responder's
and initiator's coordinates and warming caches in the other direction. This
ensures return-path transit nodes can route even when the reverse path
diverges from the forward path (e.g., after tree reconvergence).
### Result
After the handshake completes, the entire forward and reverse paths have
cached coordinates for both endpoints. Subsequent data packets use minimal
headers (no coordinates) and route efficiently through the warmed caches.
## Hybrid Coordinate Warmup (CP + CoordsWarmup)
The CP flag in the FSP common prefix and the standalone CoordsWarmup message
(0x14) together provide a hybrid cache-warming mechanism that complements
SessionSetup. See [fips-session-layer.md](fips-session-layer.md) for the
full warmup strategy.
Transit nodes parse the CP flag from the FSP header and extract source and
destination coordinates from the cleartext section between the header and
ciphertext — no decryption needed. This is the same caching operation
performed for SessionSetup coordinates. CoordsWarmup messages use the same
CP-flag format and are handled identically by transit nodes via the existing
`try_warm_coord_cache()` path.
SessionSetup carries plaintext source and destination coordinates,
which transit nodes cache as the message travels — warming the
forward path. SessionAck carries them back along the reverse path,
warming return-path caches. Steady-state data packets piggyback
coordinates via the FSP CP flag during the warmup window, falling
back to standalone CoordsWarmup messages when piggybacking would
exceed the transport MTU. See
[fips-session-layer.md](fips-session-layer.md#hybrid-coordinate-warmup-strategy)
for the canonical hybrid-warmup design (SessionSetup
self-bootstrapping plus CP-flag piggyback plus standalone
CoordsWarmup).
## Error Recovery
@@ -716,21 +513,10 @@ routing decisions but retains its own end-to-end encryption and identity.
## Packet Type Summary
| Message | Typical Size | When | Forwarded? |
| ------- | ------------ | ---- | ---------- |
| TreeAnnounce | Variable (depth-dependent) | Topology changes | No (peer-to-peer) |
| FilterAnnounce | ~1 KB | Topology changes | No (peer-to-peer) |
| LookupRequest | ~300 bytes | First contact, recovery | Yes (bloom-guided tree) |
| LookupResponse | ~400 bytes | Response to discovery | Yes (greedy routed) |
| SessionDatagram + SessionSetup | ~232–402 bytes | Session establishment | Yes (routed) |
| SessionDatagram + SessionAck | ~170 bytes | Session confirmation | Yes (routed) |
| SessionDatagram + Data (minimal) | 77 bytes + IPv6 payload | Bulk IPv6 traffic (compressed) | Yes (routed) |
| SessionDatagram + Data (with CP) | 77 + coords + IPv6 payload | Warmup/recovery (compressed) | Yes (routed) |
| SessionDatagram + CoordsRequired | 70 bytes | Cache miss error | Yes (routed) |
| SessionDatagram + PathBroken | 70+ bytes | Dead-end error | Yes (routed) |
| Disconnect | 2 bytes | Link teardown | No (peer-to-peer) |
See [fips-wire-formats.md](fips-wire-formats.md) for byte-level layouts.
For typical sizes, forwarding category, and the byte-level layouts
of each FMP and FSP message type, see
[../reference/wire-formats.md](../reference/wire-formats.md). The
canonical Packet Type Summary table lives there.
## Privacy Considerations
@@ -784,11 +570,14 @@ recovery).
## References
- [fips-intro.md](fips-intro.md) — Protocol overview
- [fips-concepts.md](fips-concepts.md) — Protocol overview
- [fips-architecture.md](fips-architecture.md) — Layer architecture and
identity model
- [fips-mesh-layer.md](fips-mesh-layer.md) — FMP specification
- [fips-spanning-tree.md](fips-spanning-tree.md) — Tree algorithms and data
structures
- [fips-bloom-filters.md](fips-bloom-filters.md) — Filter parameters and math
- [fips-wire-formats.md](fips-wire-formats.md) — Wire format reference
- [../reference/wire-formats.md](../reference/wire-formats.md) — Wire
format reference
- [spanning-tree-dynamics.md](spanning-tree-dynamics.md) — Convergence
walkthroughs
docs/design/fips-mmp.md
+218
View File
@@ -0,0 +1,218 @@
# Metrics Measurement Protocol (MMP)
The Metrics Measurement Protocol provides per-link and per-session
quality metrics — SRTT, loss, jitter, goodput, ETX, and one-way delay
trend — using only counter and timestamp fields already present in
the FMP and FSP wire formats. No additional probing traffic is
required. The same algorithms and report message format are used at
both layers; only the routing scope and configuration namespace
differ.
This document is the canonical home for the MMP design. For the
link-layer instance's role inside FMP, see
[fips-mesh-layer.md](fips-mesh-layer.md). For the session-layer
instance's role inside FSP, see
[fips-session-layer.md](fips-session-layer.md). For the byte-level
SenderReport and ReceiverReport layouts, see
[../reference/wire-formats.md](../reference/wire-formats.md).
## Two Layers, One Protocol
MMP runs at two layers:
- **Link-layer MMP**: One instance per active FMP peer link. Reports
are exchanged peer-to-peer between direct neighbors and measure the
quality of that single hop.
- **Session-layer MMP**: One instance per established FSP session.
Reports are encrypted end-to-end and forwarded through every transit
link, measuring end-to-end quality independent of hop count.
The algorithms (SRTT estimation, jitter computation, loss inference,
ETX) are identical at both layers. The differences are configuration
namespace, report intervals, and routing scope. See
[Layer Differences](#layer-differences) below.
## Metrics Tracked
MMP computes the following metrics from the per-frame counter and
timestamp fields:
- **SRTT** — Smoothed round-trip time (Jacobson/RFC 6298, α=1/8).
Derived from timestamp-echo in ReceiverReports with dwell-time
compensation.
- **Loss rate** — Bidirectional loss inferred from counter gaps.
Tracked as both instantaneous (per-interval) and long-term EWMA.
- **Jitter** — Interarrival jitter (RFC 3550 algorithm) in
microseconds.
- **Goodput** — Bytes per second of payload data (excludes MMP
reports).
- **OWD trend** — One-way delay trend (µs/s, signed). Indicates
congestion buildup before loss occurs.
- **ETX** — Expected Transmission Count, computed from bidirectional
delivery ratios. Used in cost-based parent selection via
`link_cost = etx * (1.0 + srtt_ms / 100.0)`, and in bloom-filter
candidate ranking inside `find_next_hop()` (the same `link_cost`
is the primary key when choosing among bloom-filter peers, with
tree distance as the tie-breaker).
- **Dual EWMA trends** — Short-term (α=1/4) and long-term (α=1/32)
trend indicators for both RTT and loss, enabling change detection.
Session-layer MMP additionally tracks the observed forward-path MTU;
see [fips-mtu.md](fips-mtu.md) for the end-to-end path-MTU mechanism.
## Operating Modes
MMP supports three modes:
| Mode | Reports Exchanged | Metrics Available |
| ---- | ----------------- | ----------------- |
| **Full** (default) | SenderReport + ReceiverReport | All metrics including RTT, loss, jitter, goodput, OWD trend |
| **Lightweight** | ReceiverReport only | Loss (from counter gaps), jitter, OWD trend. No RTT. |
| **Minimal** | None | Spin bit and CE echo flags only. No computed metrics. |
The mode is configured per layer (`node.mmp.mode` and
`node.session_mmp.mode`).
## Report Scheduling
Reports are sent at RTT-adaptive intervals computed as
`clamp(2 × SRTT, low, high)`. A cold-start interval is used until SRTT
has converged.
| Layer | Adaptive bounds | Cold-start |
| ----- | --------------- | ---------- |
| Link | `[1s, 5s]` | 200 ms (first 5 samples) |
| Session | `[500ms, 10s]` | 1 s |
The session-layer bounds are higher because session reports are
encrypted and forwarded through every transit link, so bandwidth cost
is proportional to path length.
## Spin Bit and RTT
The SP (spin bit) flag in the FMP inner header follows the QUIC spin
bit pattern: reflected on receive, toggled on send when the reflected
value matches the last sent value. The spin bit state machine runs
for TX reflection, but **RTT samples from the spin bit are
discarded**. In a mesh protocol where frames are sent irregularly
(tree announces, bloom filters, MMP reports on different timers),
inter-frame processing delays inflate spin bit RTT measurements
unpredictably. Timestamp-echo from ReceiverReports (with dwell-time
compensation) is the sole SRTT source.
The spin bit lives in the link-layer FMP inner header, so this
mechanism applies to link-layer MMP only. Session-layer MMP carries
its spin bit in the FSP encrypted inner header but uses it the same
way: reflected for diagnostic visibility, not used for SRTT.
## ECN Congestion Signaling
The CE (Congestion Experienced) flag (bit 1 in the FMP flags byte)
provides hop-by-hop congestion signaling through the mesh. Transit
nodes detect congestion on outgoing links and set CE on forwarded
packets; once set, the flag stays set for all subsequent hops to the
destination.
**Congestion detection** triggers on any of:
- Outgoing link MMP loss rate ≥ `node.ecn.loss_threshold` (default 5%)
- Outgoing link MMP ETX ≥ `node.ecn.etx_threshold` (default 3.0)
- Kernel receive buffer drops detected on any local transport (via
`SO_RXQ_OVFL` on UDP)
**CE relay**: The forwarding path computes
`outgoing_ce = incoming_ce || local_congestion`. Once CE is set on a
packet, it remains set for the rest of the forward path.
**IPv6 ECN-CE marking**: When a CE-flagged DataPacket arrives at its
final destination, the IPv6 Traffic Class ECN bits are marked CE
(0b11) before TUN delivery — but only for ECN-capable packets (ECT(0)
or ECT(1)). Not-ECT packets are never marked per RFC 3168. The host
TCP stack then echoes ECE in ACKs, triggering sender cwnd reduction
through standard congestion control.
**Session-layer tracking**: The `ecn_ce_count` field in MMP
ReceiverReports tracks CE-flagged packets received per link, providing
end-to-end visibility into congestion propagation.
ECN signaling is a link-layer mechanism. Session-layer MMP only
observes the CE counter as part of the report stream; CE marking is
not generated end-to-end. Tuning parameters live under `node.ecn.*`
in [../reference/configuration.md](../reference/configuration.md).
## Send Failure Backoff (Session Layer Only)
When a session MMP report cannot be delivered (destination unreachable,
no route), the sender applies exponential backoff to the probe
interval — a standard distributed-systems pattern for transient
failure handling:
- Each consecutive failure doubles the interval: 2x, 4x, 8x, 16x, 32x
- Backoff caps at 32x the base interval (5 consecutive failures)
- A successful send resets to the normal SRTT-based interval
- Debug logging is suppressed after 3 consecutive failures; a summary
is logged when the destination becomes reachable again
This prevents wasted CPU and log noise when a session's remote
endpoint has departed the network but the local session has not yet
timed out. Link-layer MMP has no equivalent — link-layer reports are
peer-to-peer over an authenticated link, so delivery failure is
indistinguishable from link death and the link-liveness mechanism
takes over.
## Layer Differences
| Aspect | Link layer | Session layer |
| ------ | ---------- | ------------- |
| Routing scope | Peer-to-peer (one hop) | End-to-end (forwarded through every hop) |
| Configuration namespace | `node.mmp.*` | `node.session_mmp.*` |
| Report bounds | `[1s, 5s]` | `[500ms, 10s]` |
| Cold-start interval | 200 ms (first 5 samples) | 1 s |
| Bandwidth cost | One link | Proportional to path length |
| Send-failure backoff | Not applicable | Yes |
| Path-MTU echo | Not applicable | PathMtuNotification (see [fips-mtu.md](fips-mtu.md)) |
| Idle-timeout interaction | None | Reports do **not** reset session idle timer |
## Idle Timeout Interaction (Session Layer Only)
MMP reports (SenderReport, ReceiverReport) and PathMtuNotification do
**not** reset the session idle timer. Only application data
(DataPacket, type 0x10) resets `last_activity`. This ensures sessions
with no application traffic tear down after
`node.session.idle_timeout_secs` (default 90s), while MMP continues
providing measurement data up to the teardown moment.
## Operator Logging
Both layers emit periodic metrics at info level. The interval is
`node.mmp.log_interval_secs` for link-layer (default 30s) and
`node.session_mmp.log_interval_secs` for session-layer (default 30s).
Link-layer:
```text
MMP link metrics peer=node-b rtt=2.3ms loss=0.2% jitter=0.1ms goodput=76.0MB/s tx_pkts=1234 rx_pkts=5678
```
Session-layer:
```text
MMP session metrics session=npub1tdwa...84le rtt=4.3ms loss=0.6% jitter=0.2ms goodput=71.3MB/s mtu=1472 tx_pkts=1234 rx_pkts=5678
```
Teardown logs include final SRTT, loss rate, jitter, ETX, goodput,
and cumulative tx/rx packet and byte counts.
## See also
- [fips-mesh-layer.md](fips-mesh-layer.md) — link-layer MMP integration
inside FMP
- [fips-session-layer.md](fips-session-layer.md) — session-layer MMP
integration inside FSP
- [fips-mtu.md](fips-mtu.md) — PathMtuNotification, the session-only
end-to-end path-MTU echo
- [../reference/wire-formats.md](../reference/wire-formats.md) —
SenderReport (0x01 / 0x11) and ReceiverReport (0x02 / 0x12) byte
layouts
- [../reference/configuration.md](../reference/configuration.md) —
full `node.mmp.*`, `node.session_mmp.*`, and `node.ecn.*` knob tables
docs/design/fips-mtu.md
+316
View File
@@ -0,0 +1,316 @@
# FIPS Path MTU and Encapsulation Overhead
MTU is a cross-cutting concern in FIPS. No single layer owns it: the
transport reports per-link MTU, FMP propagates `path_mtu` along
forward and reverse paths, FSP echoes the observed path MTU end-to-end
back to the source, and the IPv6 adapter enforces the resulting
effective MTU at the TUN interface. This document is the canonical
home for the unified MTU model.
For operator-facing diagnostic recipes (interpreting `MtuExceeded`
counters, tuning IPv6 application MSS, troubleshooting cold-flow
oversize), see the relevant how-to under `docs/how-to/`.
## The MTU Problem in FIPS
A FIPS path can traverse heterogeneous link types — UDP/IP (1280
default, IPv6 minimum), Ethernet (interface MTU − 3, typically 1497),
BLE (negotiated ATT_MTU per link), Tor stream (1400 default), radio
(51–222) — within a single end-to-end session.
The minimum MTU along the path determines the largest datagram a
session can deliver. Several properties make this harder than in
classic IP networks:
- **No fragmentation.** FIPS does not fragment at transit nodes (see
[No fragmentation policy](#no-fragmentation-policy)). A datagram
that exceeds the next-hop link MTU is dropped, and the source is
signaled.
- **Forward/reverse path asymmetry.** After tree reconvergence the
return path may diverge from the forward path, so the bottleneck
on each direction can differ.
- **First-flow race.** The very first SessionDatagram races
destination discovery — the source has not yet learned the path MTU
but must pick a payload size for the queued packet.
- **Variable per-link MTU.** Some transports (BLE, TCP via
`TCP_MAXSEG`) report different MTUs for different links rather than
a single transport-wide value.
The unified MTU model below combines proactive and reactive
mechanisms to converge on a working effective MTU within the first
few packets of a session, then maintain it across topology changes.
## Encapsulation Overhead
The byte budget for a FIPS-encapsulated packet:
| Layer | Overhead | Purpose |
| ----- | -------- | ------- |
| Link encryption | 37 bytes | 16-byte outer header + 5-byte inner header (timestamp + msg_type) + 16-byte AEAD tag |
| SessionDatagram body | 35 bytes | ttl + path_mtu + src_addr + dest_addr (msg_type counted in inner header) |
| FSP header | 12 bytes | 4-byte prefix + 8-byte counter (used as AEAD AAD) |
| FSP inner header | 6 bytes | 4-byte timestamp + 1-byte msg_type + 1-byte inner_flags (inside AEAD) |
| Session AEAD tag | 16 bytes | ChaCha20-Poly1305 tag on session-encrypted payload |
| **Protocol envelope** | **106 bytes** | `FIPS_OVERHEAD` constant — the base payload budget for any service |
`FIPS_OVERHEAD = 106` is the constant the rest of the system reasons
about. Coordinate piggybacking via the CP flag adds variable extra
overhead — `2 + entries × 16` bytes per coordinate, with both source
and destination coordinates carried — and the send path skips the CP
flag if adding coords would exceed the transport MTU.
Service-specific overheads layer on top of `FIPS_OVERHEAD`:
| Service | Overhead | Note |
| ------- | -------- | ---- |
| DataPacket port header | +4 bytes | Always present for port-multiplexed services |
| IPv6 compression | −33 bytes | 40-byte IPv6 header → 7-byte format + residual |
| **IPv6 effective overhead** | **77 bytes** | `FIPS_IPV6_OVERHEAD` constant |
See [fips-ipv6-adapter.md](fips-ipv6-adapter.md) for the IPv6
compression scheme that lets the adapter reach `FIPS_IPV6_OVERHEAD`.
## Per-Link MTU Reporting
Each transport implements two MTU methods on its trait:
- `mtu() -> u16` — Transport-wide default MTU.
- `link_mtu(addr: &TransportAddr) -> u16` — Per-link MTU for a
specific remote address. The default implementation falls back to
`mtu()`, so transports with uniform MTU (UDP, raw Ethernet) need
not override it.
FMP uses `link_mtu()` when it needs to reason about a specific
outbound link — typically for `path_mtu` annotation in
SessionDatagram and LookupResponse. Per-transport defaults:
| Transport | Default MTU | Per-link MTU source |
| --------- | ----------- | ------------------- |
| UDP | 1280 (IPv6 minimum) | uniform (`mtu()` fallback) |
| Ethernet | interface MTU − 3 (typically 1497) | uniform |
| TCP | 1400 | derived from `TCP_MAXSEG` per connection |
| Tor | 1400 | uniform |
| BLE | 2048 default; negotiated ATT_MTU per link | per-link (overrides `mtu()`) |
For TCP, the per-connection `TCP_MAXSEG` query lets FMP discover the
actual MSS the kernel negotiated for each connection, rather than
assuming a single value across all TCP peers.
## Proactive PMTUD: SessionDatagram path_mtu
Every SessionDatagram and LookupResponse carries a 2-byte `path_mtu`
field. The source initializes it to its outbound link MTU; each
transit node applies `min(current, link_mtu(next_hop))` before
forwarding. The destination receives the forward-path minimum.
For SessionDatagram, the receiver of the forward-path minimum is the
session-layer destination, which then echoes the value back to the
source via PathMtuNotification (see
[End-to-end echo](#end-to-end-echo-pathmtunotification)).
For LookupResponse, the receiver is the original requester, and the
annotation is reverse-path-only: the LookupResponse path is the
return path of the lookup, so the annotated `path_mtu` reflects what
the requester can use to reach the discovered destination over the
discovered path.
Because the field is initialized by the source and mins as it travels,
it converges to the bottleneck without any additional probing. The
first SessionDatagram on a fresh session may carry an over-estimate
(the source has not yet been told a smaller min), which is what makes
the reactive MtuExceeded path necessary.
## Reactive PMTUD: MtuExceeded
When a transit node receives a SessionDatagram whose total wire size
exceeds the next-hop `link_mtu`, it cannot forward without
fragmentation. Instead:
1. The transit node generates a SessionDatagram addressed back to the
source carrying an `MtuExceeded` payload (msg_type 0x22). The
payload identifies the destination, the reporting router, and the
bottleneck MTU.
2. The error is routed via `find_next_hop(src_addr)`. If the source
is also unreachable, the error is dropped silently (no cascading
errors).
3. The original oversized packet is dropped.
The source's FSP layer applies the reported bottleneck immediately —
unlike the increase case (see hysteresis below), decrease is always
take-the-lower-value because the original packet has already been
dropped. The source can then reduce payload sizes on subsequent
SessionDatagrams.
MtuExceeded is the reactive complement to the proactive `path_mtu`
field. The proactive field tracks the minimum along the forward path
under steady-state convergence; MtuExceeded handles the in-flight gap
when an oversized packet hits a new bottleneck (forward path shifted,
peer's outbound MTU dropped, BLE renegotiated) before the source has
adapted.
Error generation is rate-limited at 100ms per destination at the
transit node to prevent storms during topology changes.
## End-to-End Echo: PathMtuNotification
PathMtuNotification (msg_type 0x13, session-layer) provides
end-to-end path MTU feedback, adapting RFC 1191 Path MTU Discovery
for overlay networks — the transit-node `min()` propagation replaces
ICMP Packet Too Big.
Mechanism:
1. The source sets `path_mtu` in each SessionDatagram envelope to its
outbound link MTU.
2. Each transit node applies `min(current, transport.link_mtu(addr))`
before forwarding.
3. The destination receives the forward-path minimum and sends a
PathMtuNotification (2-byte body: `u16 LE path_mtu`) back to the
source.
4. The source applies the notification with hysteresis:
- **Decrease**: immediate (take lower value).
- **Increase**: requires 3 consecutive higher-value notifications
spanning at least 2 × notification interval.
5. Notifications are sent on first measurement, on any decrease, and
periodically at `max(10s, 5 × SRTT)`.
The hysteresis on increase prevents oscillation when the path MTU
fluctuates around a boundary; the immediate decrease prevents
delivering oversized packets after a path has narrowed.
PathMtuNotification is wrapped in a session-layer encrypted message
and travels back to the source via the session's normal forwarding
path. It is part of the session-layer MMP report stream's traffic
budget and (along with SenderReport and ReceiverReport) does not
reset the session idle timer.
## Per-Destination MTU Storage
Two storage locations track per-destination MTU, serving different
consumers:
- **Session-canonical** (`MmpSessionState.path_mtu`, type
`PathMtuState`). Holds the running end-to-end path MTU for an
established FSP session. Updated by both `PathMtuNotification`
(proactive, end-to-end echo) and reactive `MtuExceeded` from
transit routers. Read by the session layer when constructing
outbound `SessionDatagram` envelopes.
- **TCP-clamp mirror** (`path_mtu_lookup`, a
`HashMap<FipsAddress, u16>` on the Node). Read by the
TUN-side TCP MSS clamp (`per_flow_max_mss` in
`src/upper/tun.rs`) at first-SYN time so outbound TCP flows
are clamped to the per-destination MTU rather than a generic
ceiling. Written from four sites, all using tighter-only
semantics — the clamp is never loosened:
- Discovery's `LookupResponse` handler — reverse-path
annotated value carried back by the discovery target.
- `seed_path_mtu_for_link_peer` when a peer is promoted to
an active link, seeding with the new link's `link_mtu`
so traffic to that peer immediately uses the per-link
value rather than a generic default.
- The reactive `MtuExceeded` handler, mirroring the
bottleneck reported by a transit router.
- The proactive `PathMtuNotification` handler, mirroring
the new effective end-to-end value so a fresh TCP flow
benefits immediately from PMTU knowledge the session has
already acquired.
All four writers apply the same tighter-only rule, so the mirror
converges to the smallest MTU any signal has reported for that
destination and a subsequent looser observation cannot widen it.
## TCP MSS Clamping
The IPv6 adapter intercepts TCP SYN and SYN-ACK packets at the TUN
interface and clamps the Maximum Segment Size (MSS) option to:
```text
clamped_mss = effective_ipv6_mtu - 40 (IPv6 header) - 20 (TCP header)
```
Clamping is applied in two places:
- **TUN reader** (outbound): clamps MSS on outbound SYN packets
- **TUN writer** (inbound): clamps MSS on inbound SYN-ACK packets
Together these ensure both directions of a TCP connection use
appropriately-sized segments from the start, avoiding the initial
oversized-packet loss that would occur if the adapter relied on ICMP
Packet Too Big alone.
Clamping is **conditional**: when `per_flow_max_mss` already has an
entry for the flow, that entry is used; otherwise the clamp falls
back to a ceiling derived from the most pessimistic effective IPv6
MTU the adapter knows about (1143 with the typical 1280 transport
floor). The fallback handles cold-flow first-SYN traffic — the very
first SYN of a flow may arrive before the MMP path-MTU echo and any
per-flow lookup has been populated, so the conservative ceiling
prevents the SYN-ACK chain from negotiating a too-large MSS that
would later drop.
The adapter integrates with the MTU subsystem rather than owning it.
The "why we clamp and what `max_mss` means" lives here in the MTU
design; the "how the clamp is implemented at the TUN" lives in the
[IPv6 adapter](fips-ipv6-adapter.md#tcp-mss-clamping) doc.
## ICMP Packet Too Big
When an outbound packet at the TUN exceeds the effective IPv6 MTU,
the adapter generates an ICMPv6 Packet Too Big message and delivers
it back to the application via the TUN. This triggers the kernel's
Path MTU Discovery mechanism for non-TCP traffic and for any TCP flow
where MSS clamping was insufficient.
ICMPv6 Packet Too Big generation is rate-limited per source address
(100ms interval) to prevent storms from applications sending many
oversized packets. The ICMP response is delivered locally back
through the TUN; no network traversal is needed, so delivery is
reliable.
## No Fragmentation Policy
FIPS does not perform fragmentation at transit nodes:
- **Why no transit fragmentation.** Session-layer encryption is
end-to-end — the AEAD tag authenticates the entire plaintext.
Fragmenting an encrypted SessionDatagram would require either
exposing plaintext structure to transit nodes (unacceptable) or
reassembling before decryption (opens an attack surface — a transit
node could replay or withhold fragments to influence reassembly).
- **Why no source-side fragmentation.** The source doesn't need
fragmentation because the proactive `path_mtu` field plus the
reactive MtuExceeded signal converge on a working size within the
first few packets. Applications that need oversized payloads run
TCP over the IPv6 adapter, which has its own segmentation under
MSS clamping.
Some transports may perform fragmentation and reassembly internally
(e.g., BLE L2CAP) and can advertise a larger virtual MTU than the
physical medium supports — this is transparent to FIPS.
## Operational Considerations
Diagnosing MTU-related symptoms (handshakes succeed but bulk
transfers stall, ssh hangs after `Welcome` banner, sporadic
`MtuExceeded` spikes during topology changes) requires inspecting
per-link MTU, per-session MTU, and the per-destination
`path_mtu_lookup` table. See
[../how-to/diagnose-mtu-issues.md](../how-to/diagnose-mtu-issues.md)
for the operator recipes. The relevant control-socket queries are
`fipsctl show sessions` (per-session MTU), `fipsctl show transports`
(per-link MTU), and `fipsctl show identity-cache` (with adapter MTU
context).
## See also
- [fips-transport-layer.md](fips-transport-layer.md) — the `mtu()` /
`link_mtu()` trait surface and per-transport defaults
- [fips-mesh-layer.md](fips-mesh-layer.md) — SessionDatagram and the
MtuExceeded error signal
- [fips-session-layer.md](fips-session-layer.md) — session-layer
PathMtuNotification echo, applied with hysteresis
- [fips-ipv6-adapter.md](fips-ipv6-adapter.md) — TUN-side ICMPv6 PTB
generation, MSS clamping integration, IPv6-specific overhead table
- [../reference/wire-formats.md](../reference/wire-formats.md) —
SessionDatagram, LookupResponse, MtuExceeded, PathMtuNotification
byte layouts
docs/design/fips-nostr-discovery.md
+86 -339
View File
@@ -9,15 +9,14 @@ For peers behind UDP NAT, the same relay channel carries an encrypted
offer/answer exchange, and STUN supplies the reflexive address used for
a coordinated hole-punch.
The feature is compiled into FIPS by default on all supported platforms
(Linux, macOS, Windows) and ships in every stock packaging artifact
Nostr discovery is unconditionally compiled into the `fips` binary on
every supported platform and ships in every stock packaging artifact
(`.deb`, AUR, systemd tarball, OpenWrt `.ipk`, macOS `.pkg`, Windows
`.zip`). It is runtime-opt-in: the YAML configuration defaults to
disabled, so shipping the feature is a no-op until an operator enables
it. When disabled, nodes behave exactly as before: only the static
`peers[]` addresses are used. See
[Build configuration](#build-configuration) for details on opting out
at build time.
disabled (`node.discovery.nostr.enabled: false`), so the discovery
runtime stays dormant — and opens no relay connections — until an
operator flips the flag and supplies a relay list. When disabled, nodes
behave exactly as before: only the static `peers[]` addresses are used.
## Role
@@ -59,301 +58,27 @@ relay dependencies, STUN round-trips for NAT cases, and a small ambient
background of relay traffic; none of that is useful when you already
know where peers are.
## Build configuration
`nostr-discovery` is a default Cargo feature. Plain `cargo build
--release` produces a binary with the feature compiled in, and every
stock packaging artifact under `packaging/` ships with it enabled.
There is no extra `--features` flag to remember, on any platform.
Shipping the feature is runtime-safe: Nostr discovery is **off by
default in the YAML configuration**
(`node.discovery.nostr.enabled: false` in every stock config). An
operator opts in per-node by flipping the flag and providing a relay
list; until then the feature is dormant and does not open connections
to any relay.
To build a binary **without** the feature — for example, to reduce
the dependency footprint on a minimal build — use
`--no-default-features`:
```bash
cargo build --release --no-default-features
```
The `nostr` and `nostr-sdk` crates are then omitted from the
dependency tree entirely, and `node.discovery.nostr` config blocks
fail at startup validation.
## Scenarios and configuration
Each scenario below gives the minimal YAML fragment that enables it.
Only keys relevant to Nostr discovery are shown; surrounding node,
transport, TUN, DNS, and peer configuration follows the usual shape
described in [fips-configuration.md](fips-configuration.md).
All scenarios assume `node.identity` is set to a persistent key — an
ephemeral identity would invalidate any advert the moment the node
restarts.
### Scenario 1: Advertise a directly-reachable UDP node
The node has a public IP (or a stable port-forward) and binds UDP on a
known port. It publishes `udp:host:port` to the advert relays. Any peer
that knows this node's npub and has Nostr discovery enabled can dial it
without knowing the address out-of-band.
```yaml
node:
identity:
persistent: true
discovery:
nostr:
enabled: true
advertise: true
transports:
udp:
bind_addr: "0.0.0.0:2121"
advertise_on_nostr: true
public: true
```
What this achieves: the node publishes a single `udp:<public-ip>:2121`
endpoint to the three default advert relays
(`wss://relay.damus.io`, `wss://nos.lol`, `wss://offchain.pub`).
What the other side needs: either a static `addresses` entry for this
peer, or a peer entry with `via_nostr: true` and an empty (or omitted)
`addresses` list — the advert-resolved endpoint will be used at dial
time. Static and Nostr-resolved addresses can also be combined: when
both are present, static addresses are tried first and Nostr-resolved
endpoints are appended as fallback.
### Scenario 2: Advertise a Tor onion node
The node runs a Tor onion service in directory mode (Tor-managed
`HiddenServiceDir`) and advertises the `.onion` address. Peers dial via
their local Tor SOCKS5 proxy without ever knowing the onion string
out-of-band.
```yaml
node:
identity:
persistent: true
discovery:
nostr:
enabled: true
advertise: true
transports:
tor:
mode: directory
socks5_addr: "127.0.0.1:9050"
directory_service:
hostname_file: "/var/lib/tor/fips/hostname"
bind_addr: "127.0.0.1:8444"
advertise_on_nostr: true
```
What this achieves: the node publishes a `tor:<hash>.onion:8443`
endpoint alongside any other advertised transports. The advert itself
is still published over clearnet WebSocket relays — Tor protects the
data plane, not the discovery plane. See
[Security and threat model](#security-and-threat-model) for the trade-off.
### Scenario 3: Lookup a configured peer by npub (no advertising)
The node does not publish any advert of its own. It only consumes
adverts for peers it has explicitly listed with `via_nostr: true`. This
is the right shape for a client that wants Nostr-mediated resolution
without becoming a rendezvous target itself.
```yaml
node:
identity:
persistent: true
discovery:
nostr:
enabled: true
advertise: false
policy: configured_only
transports:
udp:
bind_addr: "0.0.0.0:2121"
peers:
- npub: "npub1peer..."
alias: "remote-node"
addresses:
- transport: udp
addr: "203.0.113.45:2121"
priority: 10
via_nostr: true
connect_policy: auto_connect
```
What this achieves: on dial, the static address is tried first; if the
peer has published a newer advert (for example, its public IP has
changed), those addresses are appended as additional candidates.
`configured_only` is the default — it is shown here for clarity.
If you have no static address for the peer at all, omit `addresses`
entirely (or leave it empty) — `via_nostr: true` is sufficient on its
own and dial endpoints are taken from the advert.
### Scenario 4: UDP NAT hole-punch with a configured peer
Neither side has a stable public UDP endpoint. Both sides advertise
`udp:nat`, run the STUN + offer/answer exchange, and punch through
their NATs to establish a direct UDP link. This is the full
NAT-traversal path.
```yaml
node:
identity:
persistent: true
discovery:
nostr:
enabled: true
advertise: true
dm_relays:
- "wss://relay.damus.io"
- "wss://nos.lol"
stun_servers:
- "stun:stun.l.google.com:19302"
- "stun:stun.cloudflare.com:3478"
transports:
udp:
bind_addr: "0.0.0.0:2121"
advertise_on_nostr: true
public: false
peers:
- npub: "npub1peer..."
alias: "nat-peer"
addresses:
- transport: udp
addr: "nat"
priority: 1
via_nostr: true
connect_policy: auto_connect
auto_reconnect: true
```
What this achieves: the node publishes a `udp:nat` endpoint plus its
signaling relays and STUN server list in the advert. The peer side runs
the same configuration. When either side initiates, an encrypted offer
is sealed to the peer's npub, a matching answer comes back, and both
sides punch at the negotiated time. On success, the punch socket is
adopted as an FMP UDP transport and Noise IK proceeds normally.
> **Validation:** `advertise_on_nostr: true` with `public: false` on UDP
> requires both `dm_relays` and `stun_servers` to be non-empty. The
> node fails startup with a config validation error if either list is
> empty. This is enforced because a `udp:nat` advert without signaling
> relays or STUN servers is unreachable by construction.
Works best with full-cone NAT on at least one side. Symmetric NAT on
both sides is not reliably traversable with this protocol and will time
out after `punch_duration_ms`; fall back to a Tor or TCP transport in
that case.
### Scenario 5: Open discovery — no pre-configured peers
Under `policy: open`, any node that publishes an advert under the same
`app` namespace becomes a candidate. Discovered peers are queued for
connection attempts subject to `open_discovery_max_pending`.
```yaml
node:
identity:
persistent: true
discovery:
nostr:
enabled: true
advertise: true
policy: open
open_discovery_max_pending: 32
app: "my-experiment.v1"
transports:
udp:
bind_addr: "0.0.0.0:2121"
advertise_on_nostr: true
public: true
peers: []
```
What this achieves: peers are discovered entirely through ambient advert
traffic on the configured relays. Setting a non-default `app` value
(replacing `fips-overlay-v1`) scopes the discovery set to participants
who opt into the same experiment and avoids being joined to unrelated
overlays that happen to share the default namespace.
> **Scope warning:** Open discovery is an admission-free mode. Any node
> that publishes on the same `app` name and passes the peer-ACL check
> becomes a connection candidate. If you rely on peer ACLs for admission
> control, verify that list is set correctly before enabling this mode.
## Operational knobs
All fields below live under `node.discovery.nostr.*`. Defaults are
defined in `src/config/node.rs`.
| Field | Type | Default | Purpose |
| --- | --- | --- | --- |
| `enabled` | bool | `false` | Master switch. When false, the discovery runtime is not started. |
| `advertise` | bool | `true` | If true, publish this node's own overlay advert. |
| `advert_relays` | list | `["wss://relay.damus.io", "wss://nos.lol", "wss://offchain.pub"]` | Relays used to publish and fetch overlay adverts (kind 37195). |
| `dm_relays` | list | same as `advert_relays` | Relays used for encrypted offer/answer signaling (kind 21059). |
| `stun_servers` | list | `["stun:stun.l.google.com:19302", "stun:stun.cloudflare.com:3478", "stun:global.stun.twilio.com:3478"]` | STUN servers used to observe the local reflexive address before a punch. Peer-advertised STUN values are not used. |
| `share_local_candidates` | bool | `false` | If true, include this node's RFC 1918 / ULA interface addresses as host candidates in the traversal offer. Off by default — sharing private host candidates is only useful when peers are on the same physical LAN, and tends to cause misleading punch successes when an asymmetric L3 path (corporate VPN, Tailscale subnet route, overlapping address space) makes a peer's private IP one-way reachable. Enable per-node only when same-LAN punching is wanted. |
| `app` | string | `"fips-overlay-v1"` | Application namespace. Included in the advert identifier; only peers with the same value cross-resolve. |
| `policy` | enum | `configured_only` | Advert consumption policy: `disabled`, `configured_only`, or `open`. |
| `signal_ttl_secs` | u64 | `120` | TTL on the encrypted offer/answer events. Also caps the wait for an answer. |
| `advert_ttl_secs` | u64 | `3600` | NIP-40 expiration set on this node's published advert. |
| `advert_refresh_secs` | u64 | `1800` | Interval between re-publishes. Must be less than `advert_ttl_secs`. |
| `attempt_timeout_secs` | u64 | `10` | Overall timeout for a single punch attempt (STUN + signal + punch). |
| `punch_start_delay_ms` | u64 | `2000` | Delay between receiving the answer and sending the first punch packet. Gives the remote side time to arrive at the same point. |
| `punch_interval_ms` | u64 | `200` | Gap between successive punch probes. |
| `punch_duration_ms` | u64 | `10000` | How long to keep probing before declaring the attempt failed. |
| `replay_window_secs` | u64 | `300` | How long a session id stays in the replay-detection cache. |
| `max_concurrent_incoming_offers` | usize | `16` | Semaphore cap on inbound offers being processed simultaneously. Excess offers are dropped with a warn log. |
| `advert_cache_max_entries` | usize | `2048` | Max cached peer adverts (LRU by expiry). |
| `seen_sessions_max_entries` | usize | `2048` | Max tracked session ids for replay detection. |
| `open_discovery_max_pending` | usize | `64` | Max peers queued for connection attempts under `policy: open`. |
The per-transport keys are:
| Key | Type | Where | Default | Purpose |
| --- | --- | --- | --- | --- |
| `advertise_on_nostr` | bool | `transports.{udp,tcp,tor}` | `false` | Include this transport's endpoint in the overlay advert. |
| `public` | bool | `transports.udp` | `false` | When `advertise_on_nostr` is true: `true` publishes `udp:host:port`, `false` publishes `udp:nat`. |
| `via_nostr` | bool | `peers[]` | `false` | Append advert-resolved endpoints to this peer's dial list. |
## Validation rules at startup
The following combinations are rejected with `ConfigError::Validation`:
- Any transport sets `advertise_on_nostr: true` while
`node.discovery.nostr.enabled` is `false` or absent.
- Any peer sets `via_nostr: true` while
`node.discovery.nostr.enabled` is `false` or absent.
- A UDP transport sets `advertise_on_nostr: true` with `public: false`
(a `udp:nat` advert) but `dm_relays` is empty.
- A UDP transport sets `advertise_on_nostr: true` with `public: false`
but `stun_servers` is empty.
For end-to-end operator recipes — each of the five activation scenarios
(advertise a directly-reachable UDP node, advertise a Tor onion node,
look up a configured peer by npub without advertising, NAT hole-punch
between two configured peers, and open discovery within an `app`
namespace) — see
[../how-to/enable-nostr-discovery.md](../how-to/enable-nostr-discovery.md).
The full configuration knob tables, per-transport keys, and startup
validation rules live in
[../reference/configuration.md](../reference/configuration.md) under
`node.discovery.nostr.*`. The Kind 37195 advert event format is in
[../reference/nostr-events.md](../reference/nostr-events.md). The rest
of this document covers the design of the discovery runtime itself.
## Under the covers
The rest of this document describes how the feature works inside the
node. For the on-the-wire event format and NIP references, see the
protocol reference at
[../proposals/nostr-udp-hole-punch-protocol.md](../proposals/nostr-udp-hole-punch-protocol.md).
node. For the generic protocol shape (event tags, NIP usage, on-the-
wire offer/answer schema, failure-suppression machinery), see
[port-advertisement-and-nat-traversal.md](port-advertisement-and-nat-traversal.md).
### Overview
@@ -403,32 +128,18 @@ configuration and by connection attempts made by the rest of the node.
Adverts are published as Nostr kind `37195` parameterized replaceable
events (FIPS-specific, in the application-defined replaceable range
`30000–39999`; the digits visually spell `FIPS` — 7=F, 1=I, 9=P, 5=S).
The `d` tag is set to the `app` value (default `fips-overlay-v1`), so
each node has a single, in-place-updatable advert under its identity.
The event is signed with the node's FIPS identity key; there is no
separate Nostr key. A NIP-40 `expiration` tag is set to now +
`advert_ttl_secs`.
The advert content is a JSON document shaped as `OverlayAdvert`:
```json
{
"identifier": "fips-overlay-v1",
"version": 1,
"endpoints": [
{"transport": "udp", "addr": "203.0.113.45:2121"},
{"transport": "tor", "addr": "xxxxx.onion:8443"},
{"transport": "udp", "addr": "nat"}
],
"signalRelays": ["wss://relay.damus.io", "wss://nos.lol"],
"stunServers": ["stun:stun.l.google.com:19302"]
}
```
`signalRelays` and `stunServers` are only present when at least one
endpoint is `udp:nat`; for advert shapes that cannot involve punching
they are omitted to reduce advert size and keep the relay and STUN
lists private to the nodes that need them.
The `d` tag is hardcoded to the wire-format identifier
`fips-overlay-v1` (or `fips-overlay-v1-next` on the `next` branch),
so each node has a single, in-place-updatable advert under its
identity. The configurable `app` value populates a separate
`protocol` tag, which scopes adverts within a relay set without
splitting them across multiple `d`-tag streams. The event is signed
with the node's FIPS identity key; there is no separate Nostr key. A
NIP-40 `expiration` tag is set to now + `advert_ttl_secs`, and a
`version` tag carries the protocol version. The advert content is a
JSON document shaped as `OverlayAdvert` (see
[../reference/nostr-events.md](../reference/nostr-events.md) for the
schema).
Publication happens on startup, again whenever the set of advertised
endpoints changes (for example, when a Tor onion hostname first
@@ -438,13 +149,23 @@ deleted using a NIP-9 kind 5 delete event. Advert publication is
fan-out: the same event is sent to every relay in `advert_relays` with
no explicit failover — relay redundancy is implicit.
For a UDP or TCP transport with `public: true`, the address advertised
follows a fixed precedence: an operator-supplied `external_addr` wins;
otherwise a non-wildcard bound `local_addr` is used directly;
otherwise — only for UDP — the runtime asks `stun_servers` for the
reflexive address of the bound socket and advertises that. TCP has no
STUN equivalent, so wildcard-bound TCP without `external_addr`
produces a loud WARN and the endpoint is omitted from the advert.
### Phase 2 — Lookup
When the node decides to dial a peer that is eligible for Nostr
resolution (a `via_nostr` peer, or any peer under `policy: open`), it
issues a Nostr REQ filtered by `author = peer_pubkey`, `kind = 37195`,
`#d = <app>`. The fetch is time-bounded (~2 s) and runs against all
configured `advert_relays` in parallel. The first valid advert wins.
`#d = fips-overlay-v1`. The fetch is time-bounded (~2 s) and runs
against all configured `advert_relays` in parallel. The first valid
advert wins; adverts whose `protocol` tag does not match the local
`app` value are rejected at validation.
Results are kept in an in-memory cache keyed by author npub. Cache
entries carry the advert's expiration time; a periodic prune drops
@@ -477,7 +198,7 @@ The initiator performs STUN first (see Phase 4), then builds a
The offer is sealed to the recipient's npub and published to the peer's
preferred signaling relays — the node first tries to resolve the peer's
NIP-65 inbox relay list (kind 10002), and falls back to `dm_relays` if
NIP-17 DM relay list (kind 10050), and falls back to `dm_relays` if
the inbox-relays fetch fails. Each side also publishes its own inbox
relay list on startup so dialers can discover it.
@@ -580,11 +301,22 @@ machinery:
| Signal TTL (`signal_ttl_secs`) | 120 s | Indefinite in-flight offers on relays. | Expired offers rejected at validation. |
| Open discovery queue (`open_discovery_max_pending`) | 64 | Unbounded retry queue under ambient advert load. | New candidates skipped until the queue drains. |
| Punch window (`punch_duration_ms`) | 10 s | Endless probe traffic after one side has given up. | Attempt declared failed; sockets discarded. |
| Failure-streak threshold (`failure_streak_threshold`) | 5 | Repeated traversal attempts against a peer that keeps failing. | Peer enters extended cooldown. |
| Extended cooldown (`extended_cooldown_secs`) | 1800 s | Tight retry loops after a failure streak. | Per-peer suppression for the cooldown window. |
| WARN log throttle (`warn_log_interval_secs`) | 300 s | Log floods from a peer that fails on every attempt. | One WARN per peer per interval; the rest demote to debug. |
| Failure-state cap (`failure_state_max_entries`) | 4096 | Memory growth from per-peer failure tracking. | LRU eviction. |
Only one of these (`max_concurrent_incoming_offers`) is a load-shedding
mechanism — the rest are capacity bounds. The load-shedding threshold
is deliberately conservative so that a misbehaving relay cannot flood
the node with offers fast enough to starve legitimate traffic.
The load-shedding mechanisms (`max_concurrent_incoming_offers` and the
failure-streak / extended-cooldown pair) are deliberately conservative
so that a misbehaving relay cannot flood the node with offers and a
chronically unreachable peer cannot keep the traversal pipeline
saturated. The remaining rows are capacity bounds.
Adverts also undergo a stale-advert sweep: cached entries whose
`expiresAt` has passed are evicted on the periodic prune tick. Inbound
signaling tolerates ±60 s of clock skew between sender and receiver,
and the runtime maintains an NTP-style skew estimate per remote so
that consistently-skewed relays don't trip the freshness check.
### Relay model
@@ -595,12 +327,12 @@ relay selection. Redundancy is implicit — a downed relay simply means
its copy of the advert or signal is unavailable, while other relays
still serve the same data.
For signaling specifically, the node prefers the recipient's NIP-65
inbox relays when available (the recipient publishes its inbox list as
a kind 10002 event to its own DM relays on startup) and falls back to
For signaling specifically, the node prefers the recipient's NIP-17
DM relays when available (the recipient publishes its DM relay list as
a kind 10050 event to its own DM relays on startup) and falls back to
the local `dm_relays` list otherwise. This keeps the common case
off the sender's DM relays when those are different from the
recipient's, at the cost of one extra NIP-65 fetch per offer.
recipient's, at the cost of one extra NIP-17 fetch per offer.
There is no per-relay rate limiting or health check. The relay model
assumes that an operator chooses relays they trust to be best-effort
@@ -646,14 +378,29 @@ semaphore and replay-cache layers downstream.
## See also
- [fips-configuration.md](fips-configuration.md) — full configuration
reference, including all surrounding keys elided from the scenarios
above.
- [../how-to/enable-nostr-discovery.md](../how-to/enable-nostr-discovery.md)
— operator activation recipes grouped under three capabilities
(resolve, advertise, open) across five scenarios.
- [../tutorials/resolve-peers-via-nostr.md](../tutorials/resolve-peers-via-nostr.md),
[../tutorials/advertise-your-node.md](../tutorials/advertise-your-node.md),
and [../tutorials/open-discovery.md](../tutorials/open-discovery.md)
— hand-held walkthroughs of the three capabilities, in
pedagogical order.
- [../reference/configuration.md](../reference/configuration.md) — full
configuration reference, including all surrounding keys elided from
the scenarios above.
- [../reference/nostr-events.md](../reference/nostr-events.md) — Kind
37195 (overlay advert), Kind 21059 (gift-wrapped traversal
signaling), Kind 10050 (NIP-17 inbox relay list).
- [../reference/security.md](../reference/security.md) — consolidated
security reference, including how the FIPS identity key signs both
adverts and Noise handshakes.
- [fips-transport-layer.md](fips-transport-layer.md) — UDP, TCP, and
Tor transport mechanics; the punch socket is adopted as a normal
UDP transport after handoff.
- [fips-mesh-layer.md](fips-mesh-layer.md) — FMP Noise IK handshake
that runs on the adopted socket.
- [../proposals/nostr-udp-hole-punch-protocol.md](../proposals/nostr-udp-hole-punch-protocol.md)
— protocol-level reference for event tags, NIP usage, and the
on-the-wire offer/answer schema.
- [port-advertisement-and-nat-traversal.md](port-advertisement-and-nat-traversal.md)
— generic protocol reference (event tags, NIP usage, on-the-wire
offer/answer schema, failure-suppression machinery), with the
FIPS-specific values called out as worked examples.
docs/design/fips-prior-work.md
+341
View File
@@ -0,0 +1,341 @@
# FIPS Prior Work and References
FIPS builds on proven designs rather than inventing new cryptography or
routing algorithms. Nearly every major design decision has deployed
precedent. This document collects the relevant prior art, organized by
the FIPS subsystem that draws on it, and gathers the academic and
standards references cited from the per-subsystem design docs.
## Spanning Tree Self-Organization
The idea that distributed nodes can build a spanning tree through
purely local decisions — each node selecting a parent based on
announcements from its neighbors — dates to the
[IEEE 802.1D Spanning Tree Protocol](https://en.wikipedia.org/wiki/Spanning_Tree_Protocol)
(STP, 1985). STP demonstrated that a network-wide tree emerges from a
simple deterministic rule (lowest bridge ID wins root election)
applied independently at each node. FIPS uses the same principle —
lowest node address determines the root — adapted from an Ethernet
bridging context to a general-purpose overlay mesh.
## Tree Coordinate Routing
The spanning tree coordinates, bloom filter candidate selection, and
greedy routing algorithms are adapted from
[Yggdrasil v0.5](https://yggdrasil-network.github.io/2023/10/22/upcoming-v05-release.html)
and its [Ironwood](https://github.com/Arceliar/ironwood) routing
library. Yggdrasil's key insight was using the tree path from root to
node as a routable coordinate, enabling greedy forwarding without
global routing tables. FIPS adapts these algorithms for
multi-transport operation, Nostr identity integration, and constrained
MTU environments.
The theoretical foundation for greedy routing on tree embeddings draws
on [Kleinberg's work](https://www.cs.cornell.edu/home/kleinber/swn.pdf)
on navigable small-world networks, which showed that greedy forwarding
succeeds in O(log² n) steps when the network has hierarchical
structure. Thorup-Zwick compact routing schemes separately demonstrated
that sublinear routing state is achievable with bounded stretch,
motivating the use of tree coordinates rather than full routing tables.
## Split-Horizon Bloom Filter Propagation
FIPS distributes reachability information using bloom filters computed
with a split-horizon rule: when advertising to a peer, exclude that
peer's own contributions. This technique is borrowed from
distance-vector routing protocols —
[RIP](https://en.wikipedia.org/wiki/Routing_Information_Protocol)
(1988) and [Babel](https://www.irif.fr/~jch/software/babel/) use
split-horizon to prevent routing loops by not advertising a route back
to the neighbor it was learned from. FIPS applies the same principle
to probabilistic set advertisements rather than distance-vector tables.
## Cryptographic Identity as Network Address
FIPS nodes are identified by their Nostr public keys (secp256k1). The
network address *is* the cryptographic identity — there is no separate
address assignment or registration step.
[CJDNS](https://github.com/cjdelisle/cjdns) pioneered this approach in
overlay meshes, deriving IPv6 addresses from the double-SHA-512 of
each node's public key. Tor [.onion
addresses](https://spec.torproject.org/rend-spec-v3) and the IETF
[Host Identity Protocol](https://en.wikipedia.org/wiki/Host_Identity_Protocol)
(HIP) follow the same principle. FIPS uses Nostr's existing key
infrastructure rather than introducing a new identity scheme.
## Dual-Layer Encryption
FIPS encrypts traffic twice: FMP provides hop-by-hop link encryption
(protecting against transport-layer observers), while FSP provides
independent end-to-end session encryption (protecting against
intermediate FIPS nodes). This layered approach mirrors
[Tor](https://www.torproject.org/), where each relay peels one layer
of encryption (hop-by-hop) while the innermost layer protects
end-to-end payload. [I2P](https://geti2p.net/) uses a similar garlic
routing scheme with tunnel-layer and end-to-end encryption. Unlike Tor
and I2P, FIPS does not provide anonymity — its dual encryption
protects confidentiality and integrity rather than hiding traffic
patterns.
## Noise Protocol Framework
FIPS uses the [Noise Protocol Framework](https://noiseprotocol.org/)
at both protocol layers, with different handshake patterns chosen for
each layer's threat model. FMP link encryption uses **Noise IK**,
providing mutual authentication with a single round trip where the
initiator knows the responder's static key in advance.
[WireGuard](https://www.wireguard.com/) uses the same IK base pattern
(extended with a pre-shared key as IKpsk2) for VPN tunnels. FSP
session encryption uses **Noise XK**, the same pattern used by the
[Lightning Network](https://github.com/lightning/bolts/blob/master/08-transport.md),
where the initiator's static key is transmitted in a third message
rather than the first. XK provides stronger initiator identity hiding
at the cost of an additional round trip — a worthwhile tradeoff for
session-layer traffic that traverses untrusted intermediate nodes. At
the link layer, where both peers are configured and directly
connected, IK's single round trip is preferred.
Specific Noise references and adapted constructions:
- Perrin, T. ["The Noise Protocol Framework"](https://noiseprotocol.org/noise.html).
Revision 34, 2018. *Framework for building crypto protocols using
Diffie-Hellman key agreement and AEAD ciphers. FSP uses the XK
handshake pattern.*
- Donenfeld, J.A. ["WireGuard: Next Generation Kernel Network Tunnel"](https://www.wireguard.com/papers/wireguard.pdf).
NDSS 2017. *Transport-independent cryptographic sessions bound to
identity keys rather than network addresses; AEAD-only authentication
model.*
## Index-Based Session Dispatch
FIPS uses locally-assigned 32-bit session indices to demultiplex
incoming packets to the correct cryptographic session in O(1) time,
without parsing source addresses or performing expensive lookups.
This directly follows
[WireGuard's](https://www.wireguard.com/papers/wireguard.pdf) receiver
index approach, where each peer assigns a random index during
handshake and the remote side includes it in every packet header.
## Replay Protection Over Unreliable Transports
FSP and FMP both use explicit per-packet counters with a sliding
bitmap window for replay protection — the standard DTLS approach,
chosen because implicit nonce counters desynchronize permanently under
UDP packet loss or reordering.
- Rescorla, E., Modadugu, N. [RFC 6347](https://datatracker.ietf.org/doc/html/rfc6347):
"Datagram Transport Layer Security Version 1.2". 2012. *Explicit
sequence numbers with sliding bitmap window for replay protection
over unreliable transports.*
## Transport-Agnostic Overlay Mesh
FIPS is designed to operate over any datagram-capable transport — UDP,
raw Ethernet, Bluetooth, radio, serial — through a uniform transport
abstraction. Several mesh overlays have demonstrated transport-agnostic
design: [CJDNS](https://github.com/cjdelisle/cjdns) runs over UDP and
Ethernet, [Yggdrasil](https://yggdrasil-network.github.io/) supports
TCP and TLS transports, and [Tor](https://www.torproject.org/) can use
pluggable transports to tunnel through various media. FIPS extends
this pattern to shared-medium transports (radio, BLE) with
per-transport MTU and discovery capabilities.
## Metrics Measurement Protocol
MMP's design assembles well-established measurement techniques into a
unified per-link protocol. The SenderReport/ReceiverReport exchange
structure follows [RTCP](https://www.rfc-editor.org/rfc/rfc3550)
(RFC 3550), which uses the same report pairing for media stream
quality monitoring in RTP sessions. MMP's jitter computation uses the
RTCP interarrival jitter algorithm directly.
The smoothed RTT estimator uses the Jacobson/Karels algorithm
([RFC 6298](https://www.rfc-editor.org/rfc/rfc6298)), the same SRTT
computation used in TCP for retransmission timeout calculation since
1988. MMP derives RTT from timestamp-echo in ReceiverReports with
dwell-time compensation, rather than from packet round-trips.
The spin bit in the FMP frame header follows the
[QUIC](https://www.rfc-editor.org/rfc/rfc9000) spin bit
([RFC 9312](https://www.rfc-editor.org/rfc/rfc9312)) — a single bit
that alternates each round trip, enabling passive latency measurement.
FIPS implements the spin bit state machine but relies on
timestamp-echo for SRTT, as irregular mesh traffic makes spin bit RTT
unreliable.
The Expected Transmission Count (ETX) metric, computed from
bidirectional delivery ratios, was introduced by
[De Couto et al. (2003)](https://pdos.csail.mit.edu/papers/grid:mobicom03/paper.pdf)
for wireless mesh routing and is used in protocols including
[OLSR](https://en.wikipedia.org/wiki/Optimized_Link_State_Routing_Protocol)
and [Babel](https://www.irif.fr/~jch/software/babel/). FIPS computes
ETX per-link from MMP loss measurements for future use in candidate
ranking.
The CE (Congestion Experienced) echo flag provides hop-by-hop
[ECN](https://en.wikipedia.org/wiki/Explicit_Congestion_Notification)
signaling, following the TCP/IP ECN echo pattern (RFC 3168). Transit
nodes detect congestion via MMP loss/ETX metrics or kernel buffer
drops and set the CE flag on forwarded frames; destination nodes mark
ECN-capable IPv6 packets accordingly.
## Path MTU Discovery
FSP adapts RFC 1191 Path MTU Discovery for overlay networks. The
classic ICMP Packet Too Big mechanism is replaced by a transit-node
`min()` propagation in SessionDatagram and LookupResponse plus an
end-to-end PathMtuNotification echo back to the source.
- Mogul, J., Deering, S. [RFC 1191](https://datatracker.ietf.org/doc/html/rfc1191):
"Path MTU Discovery". 1990. *End-to-end path MTU discovery; FSP
adapts this for overlay networks using transit-node min()
propagation.*
## Session Restart and Simultaneous Initiation
FSP's epoch-based peer restart detection mirrors IKEv2's
INITIAL_CONTACT notification, and its lowest-address-wins
simultaneous-initiation tie-breaker mirrors IKEv2's resolution rule.
- Kaufman, C., Hoffman, P., Nir, Y., Eronen, P., Kivinen, T.
[RFC 7296](https://datatracker.ietf.org/doc/html/rfc7296):
"Internet Key Exchange Protocol Version 2 (IKEv2)". 2014.
*Simultaneous initiation resolution (§2.8) and INITIAL_CONTACT peer
restart detection (§2.4).*
## Hybrid Coordinate Warmup
FSP's hybrid coordinate warmup (CP flag piggybacking + standalone
CoordsWarmup) draws on Yggdrasil's approach of embedding coordinates
in session traffic to keep transit caches populated.
- [Yggdrasil Network](https://yggdrasil-network.github.io/).
*Coordinate-based overlay routing with session traffic used to warm
transit node coordinate caches.*
## Cryptographic Primitives
FIPS reuses [Nostr's](https://github.com/nostr-protocol/nips)
cryptographic stack — secp256k1 for identity keys, Schnorr signatures
for authentication, SHA-256 for hashing, and ChaCha20-Poly1305 for
authenticated encryption. This is the same primitive set used across
Bitcoin, Nostr, and a growing ecosystem of self-sovereign identity
systems. No novel cryptography is introduced.
## Spanning-Tree Dynamics: Foundations
The CRDT framing, gossip dissemination, failure detection, link
metrics, and route stability mechanisms in
[spanning-tree-dynamics.md](spanning-tree-dynamics.md) draw on a body
of academic and standards work, summarized below.
### Virtual Coordinate Routing
- Rao, A., Ratnasamy, S., Papadimitriou, C., Shenker, S., Stoica, I.
["Geographic Routing without Location Information"](https://people.eecs.berkeley.edu/~sylvia/papers/p327-rao.pdf).
MobiCom 2003. *Established virtual coordinate routing using network
topology.*
### Greedy Embedding Theory
- Kleinberg, R.
["Geographic Routing Using Hyperbolic Space"](https://www.semanticscholar.org/paper/Geographic-Routing-Using-Hyperbolic-Space-Kleinberg/f506b2ddb142d2ec539400297ba53383d958abef).
IEEE INFOCOM 2007. *Proved every connected graph has a greedy
embedding in hyperbolic space; showed spanning trees enable
coordinate assignment.*
- Cvetkovski, A., Crovella, M.
["Hyperbolic Embedding and Routing for Dynamic Graphs"](https://www.cs.bu.edu/faculty/crovella/paper-archive/infocom09-hyperbolic.pdf).
IEEE INFOCOM 2009. *Dynamic embedding for nodes joining/leaving;
introduced Gravity-Pressure routing for failure recovery.*
- Crovella, M. et al.
["On the Choice of a Spanning Tree for Greedy Embedding"](https://www.cs.bu.edu/faculty/crovella/paper-archive/networking-science13.pdf).
Networking Science 2013. *Analysis of how tree structure affects
routing stretch.*
- Bläsius, T. et al.
["Hyperbolic Embeddings for Near-Optimal Greedy Routing"](https://dl.acm.org/doi/10.1145/3381751).
ACM Journal of Experimental Algorithmics 2020. *Achieved 100%
success ratio with 6% stretch on Internet graph.*
### Link Metrics
- De Couto, D., Aguayo, D., Bicket, J., Morris, R.
"A High-Throughput Path Metric for Multi-Hop Wireless Routing".
MobiCom 2003. *Introduced ETX (Expected Transmission Count) as a
link quality metric for wireless mesh networks.*
### Routing Protocol Stability
- IEEE 802.1D. "IEEE Standard for Local and Metropolitan Area
Networks: Media Access Control (MAC) Bridges". *Spanning Tree
Protocol (STP) — root election via bridge ID, BPDU exchange.*
- Moy, J. [RFC 2328](https://datatracker.ietf.org/doc/html/rfc2328):
"OSPF Version 2". 1998. *Link-state routing with cumulative path
costs and SPF computation. FIPS's local-only cost approach is
contrasted with OSPF's cumulative model in
[spanning-tree-dynamics.md §8](spanning-tree-dynamics.md#8-parent-selection).*
### Distributed Systems Primitives
- Shapiro, M., Preguiça, N., Baquero, C., Zawirski, M.
"Conflict-free Replicated Data Types". SSS 2011. *Formal definition
of CRDTs enabling coordination-free consistency.*
- Das, A., Gupta, I., Motivala, A.
["SWIM: Scalable Weakly-consistent Infection-style Process Group Membership"](https://www.cs.cornell.edu/projects/Quicksilver/public_pdfs/SWIM.pdf).
IPDPS 2002. *O(1) failure detection, O(log N) dissemination via
gossip.*
- Kermarrec, A-M.
["Gossiping in Distributed Systems"](https://www.distributed-systems.net/my-data/papers/2007.osr.pdf).
ACM SIGOPS Operating Systems Review 2007. *Framework for
gossip-based protocols achieving O(log N) propagation.*
## FIPS Contributions
The protocol builds on these foundations and adds several new elements:
- Cost-aware parent selection using local-only link metrics
(`effective_depth = depth + link_cost`), replacing Yggdrasil's
depth-only selection
- Combined ETX + SRTT link cost formula with MMP-measured components
- Flap dampening with mandatory switch bypass
- Announcement suppression for transient state changes
- Tree-only bloom filter merge with split-horizon exclusion
- Hybrid coordinate warmup (CP flag piggybacking plus standalone
CoordsWarmup) layered on top of SessionSetup self-bootstrapping
- Bloom-guided tree routing for discovery (vs. flooding)
- Reverse-path routing for LookupResponse via `recent_requests`
## External Reference Index
| Reference | Used by |
| --------- | ------- |
| [IEEE 802.1D STP](https://en.wikipedia.org/wiki/Spanning_Tree_Protocol) | spanning tree, root election |
| [Yggdrasil v0.5](https://yggdrasil-network.github.io/2023/10/22/upcoming-v05-release.html) | tree coordinates, greedy routing |
| [Ironwood](https://github.com/Arceliar/ironwood) | tree coordinates, candidate ranking |
| [Kleinberg, Small-world](https://www.cs.cornell.edu/home/kleinber/swn.pdf) | greedy routing on tree embeddings |
| [CJDNS](https://github.com/cjdelisle/cjdns) | cryptographic-identity-as-address |
| [Tor](https://www.torproject.org/) | onion address scheme, dual-layer encryption |
| [I2P](https://geti2p.net/) | dual-layer encryption (garlic routing) |
| [HIP](https://en.wikipedia.org/wiki/Host_Identity_Protocol) | identity-as-address |
| [Babel](https://www.irif.fr/~jch/software/babel/) | split-horizon, ETX |
| [RIP](https://en.wikipedia.org/wiki/Routing_Information_Protocol) | split-horizon |
| [Noise Framework](https://noiseprotocol.org/) | FMP IK, FSP XK |
| [WireGuard](https://www.wireguard.com/) | IK pattern, receiver-index dispatch, identity-bound sessions |
| [Lightning BOLT #8](https://github.com/lightning/bolts/blob/master/08-transport.md) | XK pattern |
| [QUIC (RFC 9000)](https://www.rfc-editor.org/rfc/rfc9000) | spin bit, transport design |
| [QUIC Spin Bit (RFC 9312)](https://www.rfc-editor.org/rfc/rfc9312) | passive RTT measurement |
| [RTCP (RFC 3550)](https://www.rfc-editor.org/rfc/rfc3550) | sender/receiver report structure, jitter algorithm |
| [TCP SRTT/RTO (RFC 6298)](https://www.rfc-editor.org/rfc/rfc6298) | Jacobson/Karels SRTT |
| [ECN (RFC 3168)](https://www.rfc-editor.org/rfc/rfc3168) | CE echo |
| [DTLS 1.2 (RFC 6347)](https://datatracker.ietf.org/doc/html/rfc6347) | replay window |
| [IKEv2 (RFC 7296)](https://datatracker.ietf.org/doc/html/rfc7296) | INITIAL_CONTACT, simultaneous-initiation tie-breaker |
| [PMTUD (RFC 1191)](https://datatracker.ietf.org/doc/html/rfc1191) | adapted PMTUD |
| [ETX paper, De Couto et al.](https://pdos.csail.mit.edu/papers/grid:mobicom03/paper.pdf) | ETX metric |
| [OLSR](https://en.wikipedia.org/wiki/Optimized_Link_State_Routing_Protocol) | ETX in mesh routing |
| [Nostr](https://github.com/nostr-protocol/nips) | identity stack |
docs/design/fips-security.md
+215
View File
@@ -0,0 +1,215 @@
# FIPS Mesh-Interface Security
This document describes the threat model and design rationale for the
operator-facing security posture of the `fips0` mesh interface on Linux.
The default-deny nftables baseline shipped as `/etc/fips/fips.nft` is the
artifact discussed below; for the operator activation steps and drop-in
extension recipes, see [enable-mesh-firewall.md](../how-to/enable-mesh-firewall.md).
The baseline is a documented operator conffile, not an auto-loaded
package side-effect. Activation is an explicit one-liner. The
rationale for that design follows.
## Threat Model for `fips0`
The mesh is a flat layer-3 segment. Every mesh node that can route to
you can deliver packets to your `fips0` address — your direct peers
forward traffic from non-peer mesh nodes onto your `fips0` the same
way any router forwards transit traffic. Identity on the mesh is the
originating node's npub — the FMP link layer authenticates direct
peers with Noise IK and the FSP session layer authenticates session
endpoints with Noise XK — but identity is **not** authorization.
Knowing who sent a packet does not, by itself, decide whether the
local host should accept it.
That means: any service on a mesh host that binds to a wildcard
address (`0.0.0.0`, `[::]`, or any IPv6 address that includes the
`fips0` interface in its scope) is reachable from every mesh node
that can route to you by default, not only from your direct peers.
There is no NAT, no perimeter firewall, no "local-only" address
space between you and an arbitrary mesh node. The mesh is closer
to a shared LAN than to the public internet.
Compare to the corresponding internet trust assumptions:
| Surface | Public internet | FIPS mesh (no baseline) |
|---|---|---|
| Reachability from arbitrary mesh node | Mediated by NAT, firewalls, ISPs | Direct |
| Default identity | None | Originating node's npub (authenticated) |
| Default authorization | None | None |
| Accidental exposure cost | Low (NAT hides you) | High (every mesh node sees you) |
The third row is the gap this document closes. The default-deny
baseline removes "accidental exposure" from the failure modes an
operator has to think about.
## The Default-Deny Baseline
The shipped baseline is `/etc/fips/fips.nft`. It defines a single
nftables table, `inet fips`, with one chain hooked at `input`. The
chain:
1. Returns immediately for any packet not arriving on `fips0`. This
makes the table a no-op for every other interface — Docker, Tor,
the host's main filter table, OPNsense, anything.
2. Accepts packets that conntrack identifies as `established` or
`related`. Replies to outbound flows initiated from the mesh host
come back; ICMPv6 errors related to existing flows (Packet Too
Big, Destination Unreachable) come back.
3. Accepts ICMPv6 echo-request, so `ping6` reachability tests work.
4. Includes operator drop-ins from `/etc/fips/fips.d/*.nft`. An empty
directory is fine — the include glob simply matches nothing.
5. Falls through to `counter drop`. Every dropped packet increments
the counter, visible via `nft list table inet fips`.
Outbound from `fips0` is unrestricted. The baseline is concerned only
with what the mesh host accepts, not what it sends.
The file is a documented dpkg conffile. Operator edits to
`/etc/fips/fips.nft` are preserved across upgrades, the same way
edits to `/etc/fips/fips.yaml` and `/etc/fips/hosts` are preserved.
If the packaged baseline is ever updated upstream, dpkg prompts the
operator on upgrade rather than silently overwriting local changes.
The canonical artifact is the file itself; read it for the inline
documentation that the rest of this document references.
## Why no auto-load on package install
The `postinst` script does **not** enable `fips-firewall.service`.
This is deliberate. Quietly mutating host firewall state on package
install is hostile on every axis that matters: it surprises operators
who already have their own nftables ruleset, it can collide with
podman/Docker/OPNsense integrations even though the early-return
makes it technically safe, and it converts an explicit security
decision into an invisible one. The mesh-interface filter belongs to
the operator, not to the package's `postinst`.
The activation gesture is one short, well-formed command. The
rationale is documented in the file's inline header and in this
document. That is enough; auto-loading would trade discoverability
for no real gain.
## Coexistence with other firewalls
The `inet fips` table only matches packets arriving on `fips0`.
Anything else returns from the chain on the first rule. Specifically:
- **Docker / containerd** install nftables rules in the `ip` and `ip6`
families and operate on `docker0`, `br-*`, and `veth*`
interfaces. They do not touch `fips0`. The two tables coexist
without interference.
- **Tor** runs in user space and does not install firewall rules. The
baseline is independent of Tor's onion-service and SOCKS listeners.
- **OPNsense** is an upstream perimeter device. The baseline runs on
the local host and applies only to traffic that has already reached
the host's `fips0` interface. They do not interact.
- **The host's main `/etc/nftables.conf`** typically defines a
separate `inet filter` table. nftables allows multiple tables in
the same family to coexist; both run in parallel at hook
`input`/priority 0 and the `iifname != "fips0" return` rule keeps
the `inet fips` table from interfering with anything outside the
mesh interface.
- **`inet fips_gateway`**, when `fips-gateway` is running, manages
DNAT/SNAT on the LAN-facing interface to translate virtual IPs to
mesh addresses. It is a separate concern owned by the gateway
binary and is unrelated to this baseline. See the section below.
## Coexistence with `inet fips_gateway`
When `fips-gateway` is running, it manages a separate nftables
table, `inet fips_gateway`, containing the DNAT and masquerade rules
that translate between the gateway's virtual-IP pool and mesh
addresses on the LAN-facing interface. That table is created and
torn down by the gateway binary at runtime and is not an operator
artifact in the same sense as `inet fips`.
The two tables do not interfere:
- `inet fips` filters inbound on `fips0`.
- `inet fips_gateway` performs NAT on the LAN interface.
They operate on different interfaces and at different hook points
(`input` filter vs. `prerouting`/`postrouting` NAT). Both can be
loaded simultaneously on a gateway host, and that is the intended
deployment shape. See [fips-gateway.md](fips-gateway.md) for the
gateway table's structure.
## What the Baseline Does Not Cover
The baseline is one half of a defense-in-depth posture. It is
explicitly not:
- **Outbound filtering.** Anything the mesh host originates on
`fips0` is unrestricted. If you need to constrain what the host
can send to the mesh, add rules to a separate chain hooked at
`output` — out of scope for the baseline.
- **Application-layer authorization.** The baseline decides whether
a packet reaches a service. It does not decide whether the
originating mesh node's npub is allowed to use that service. That
is the application's responsibility (e.g., an `authorized_keys`
file for SSH, an ACL in the application's configuration).
- **ACL on the mesh handshake.** The FMP Noise IK handshake
authenticates the peer's npub and, on both inbound and outbound
paths, consults the peer ACL (`peers.allow` / `peers.deny`) before
promoting the connection. The ACL evaluates in TCP-Wrappers order:
an `allow` match permits, otherwise a `deny` match rejects,
otherwise the connection is permitted. A strict allowlist posture
therefore requires an explicit `ALL` entry in `peers.deny`; a
populated `peers.allow` alone does not turn the ACL into a strict
allowlist. Mesh-level ACLs are a separate concern from the inbound
packet filter described here; see the peer ACL section in
[../reference/security.md](../reference/security.md).
- **Compromised peers.** A peer whose key has been stolen or whose
host has been taken over is, by mesh-level identity, still that
peer. Source-address filtering in drop-ins operates on the source
mesh address of inbound traffic regardless of whether that source
is a direct peer or a multi-hop mesh node, and so can limit damage
from a known-compromised mesh address; but the baseline cannot
revoke trust on its own.
Treat the baseline as removing the "wide-open by default" failure
mode. Higher-layer authorization decisions are the operator's and
the application's, the same as on any other shared network.
## Future Work
The current baseline is Linux-only. Parallel work for other targets:
- **macOS PF baseline.** macOS uses Packet Filter (PF), inherited
from OpenBSD. PF maps cleanly onto the same conceptual model as
nftables: stateful inspection (`keep state` ≈ `ct state
established,related`), default policy, anchor-based modular rule
loading. A `packaging/macos/fips.pf` will land alongside the
Linux baseline with the same posture: documented asset, no
auto-load, operator opts in via launchd. The macOS interface name
is `utunN` rather than `fips0`, so the rule template needs runtime
substitution or a PF interface group assigned at TUN bring-up;
this is being worked through with the macOS port.
- **OpenWrt fw4 path.** OpenWrt's fw4 already drives nftables under
the hood, but rules go into `/etc/nftables.d/` includes or UCI
entries in `/etc/config/firewall`, not a free-standing
`fips.nft`. The ipk will ship a layout-compatible variant or
document the operator setup separately, decided when the OpenWrt
packaging is updated.
- **Cross-OS gateway abstraction.** `fips-gateway` is currently
Linux-only because `src/gateway/nat.rs` uses the `rustables`
netlink API directly. macOS gateway support requires a PF-backed
equivalent behind a shared backend trait. This is a larger lift
than the static baseline and is tracked separately under the same
cross-OS thread.
When those land, this document will grow per-OS sections describing
each baseline's load mechanism and extension points. The threat
model and the operator-extension principle are the same on every OS;
only the filter syntax and the activation gesture differ.
## See also
- [enable-mesh-firewall.md](../how-to/enable-mesh-firewall.md) — operator
activation steps, drop-in recipes, drop visibility and debugging
- [../reference/security.md](../reference/security.md) — consolidated
security reference (cryptographic primitives, peer ACL format,
filesystem permissions, default network exposures)
- [fips-gateway.md](fips-gateway.md) — `fips-gateway` service and the
separate `inet fips_gateway` table
docs/design/fips-session-layer.md
+57 -139
View File
@@ -1,10 +1,10 @@
# FIPS Session Protocol (FSP)
The FIPS Session Protocol is the top protocol layer in the FIPS stack. It sits
above the FIPS Mesh Protocol (FMP) and below applications (native FIPS API or
IPv6 adapter). FSP provides end-to-end authenticated, encrypted datagram
delivery between any two FIPS nodes, regardless of how many intermediate hops
separate them.
The FIPS Session Protocol is the topmost layer of the FIPS protocol stack.
It sits above the FIPS Mesh Protocol (FMP) and below applications (native
FIPS API or IPv6 adapter). FSP provides end-to-end authenticated, encrypted
datagram delivery between any two FIPS nodes, regardless of how many
intermediate hops separate them.
## Role
@@ -99,9 +99,9 @@ FMP signals routing failures asynchronously:
a standalone CoordsWarmup (rate-limited), re-discovering the destination's
current coordinates, and resetting the warmup counter.
- **MtuExceeded**: A transit node cannot forward a SessionDatagram because
the packet exceeds the next-hop link MTU. FSP uses the reported bottleneck
MTU to adjust its session-layer path MTU estimate. MtuExceeded is the
reactive complement to the proactive `path_mtu` field in SessionDatagram.
the packet exceeds the next-hop link MTU; FSP adjusts its
session-layer path MTU estimate from the reported bottleneck. See
[fips-mtu.md](fips-mtu.md) for the full forward/reverse MTU model.
All three signals are generated by transit nodes (not the destination) and
travel back to the source inside a new SessionDatagram. They are plaintext
@@ -123,9 +123,10 @@ a destination with no existing session.
FSP uses Noise XK for session key agreement (Noise Protocol Framework;
Perrin 2018). The initiator knows the destination's npub (required for
XK's pre-message `s` token); the responder learns the initiator's
identity from msg3 (not msg1, unlike IK at the link layer). This provides stronger initiator identity hiding
— the initiator's static key is encrypted under the established shared
secret rather than under only the responder's static key.
identity from msg3 (not msg1, unlike IK at the link layer). This
provides stronger initiator identity hiding — the initiator's static
key is encrypted under the established shared secret rather than under
only the responder's static key.
The handshake is a three-message flow carried in SessionSetup, SessionAck,
and SessionMsg3:
@@ -244,17 +245,9 @@ operations) rather than under only the responder's static key.
### Cryptographic Primitives
| Component | Choice | Notes |
| --------- | ------ | ----- |
| Curve | secp256k1 | Nostr-native |
| DH | ECDH on secp256k1 | Standard EC Diffie-Hellman |
| Cipher | ChaCha20-Poly1305 | AEAD, same as NIP-44 |
| Hash | SHA-256 | Nostr-native |
| Key derivation | HKDF-SHA256 | Standard Noise KDF |
These choices prioritize compatibility with the Nostr cryptographic stack —
secp256k1 + ChaCha20-Poly1305 + SHA-256 aligns with the NIP-44 encrypted
messaging standard.
FSP uses ChaCha20-Poly1305 with secp256k1 ECDH; see
[../reference/security.md](../reference/security.md) for the full
primitive table shared with the link layer.
### secp256k1 Parity Normalization
@@ -391,22 +384,10 @@ signal generation (100ms per destination).
## Identity Cache
The identity cache maps FIPS address prefix (15 bytes, the `fd00::/8` IPv6
address minus the `fd` prefix) to `(NodeAddr, PublicKey)`. This cache is
needed only when using the IPv6 adapter — the native FIPS API provides the
public key directly.
The mapping is deterministic (derived from the public key via SHA-256) and
never becomes stale. The cache uses LRU-only eviction bounded by a
configurable size (default 10K entries). There is no TTL — entries are evicted
only when the cache is full and space is needed for a new entry.
Cache population mechanisms:
- **DNS lookup**: The primary path. Resolving `npub1xxx...xxx.fips` derives
the IPv6 address and populates the identity cache.
- **Inbound traffic**: Authenticated sessions from other nodes populate the
cache with their identity information.
The IPv6 adapter requires an identity cache to map `fd00::/8` addresses
back to `(NodeAddr, PublicKey)` for routing; see
[fips-ipv6-adapter.md](fips-ipv6-adapter.md#identity-cache) for the
cache rationale, eviction policy, and population mechanics.
## Coordinate Cache
@@ -451,84 +432,30 @@ node caches (still within their 300s TTL) are re-warmed.
## Session-Layer MMP
Each established session runs its own Metrics Measurement Protocol instance,
providing end-to-end quality metrics independent of the number of hops.
FSP runs an MMP instance per established session for end-to-end metrics
independent of hop count. Reports are encrypted and forwarded through
every transit link, so bandwidth cost is proportional to path length;
the session-layer report intervals are correspondingly higher than the
link-layer intervals (clamped to `[500ms, 10s]` vs. `[1s, 5s]`).
### Relationship to Link-Layer MMP
The session-layer instance shares its algorithms (SRTT, jitter, loss,
ETX) and report wire format with link-layer MMP. The differences —
configuration namespace (`node.session_mmp.*`), routing scope,
send-failure backoff, idle-timeout interaction, and the
PathMtuNotification mechanism — are documented in the unified MMP
treatment at [fips-mmp.md](fips-mmp.md). For the end-to-end path-MTU
echo specifically, see [fips-mtu.md](fips-mtu.md). Reports and
PathMtuNotification do **not** reset the session idle timer, so a
session carrying only MMP traffic still tears down at the configured
idle threshold.
Session-layer MMP uses the same report wire format (SenderReport 0x11,
ReceiverReport 0x12) and identical algorithms as link-layer MMP, but with
two key differences:
### MtuExceeded Handling
1. **End-to-end routing**: Session reports are encrypted and forwarded through
every transit link. A 3-hop session generates report traffic on all 3 links,
making bandwidth cost proportional to path length.
2. **Independent configuration**: The `node.session_mmp.*` parameters are
separate from `node.mmp.*`, allowing operators to run a lighter mode for
sessions (e.g., Lightweight) while keeping Full mode on links.
### Metrics
The same metrics as link-layer MMP: SRTT, loss rate, jitter, goodput, OWD
trend, ETX, and dual EWMA trends. Session MMP additionally tracks observed
path MTU.
### Report Intervals
Session-layer report intervals are higher than link-layer to account for
bandwidth cost: clamped to [500ms, 10s] with a cold-start interval of 1s
(vs. link-layer [100ms, 2s] with 500ms cold-start).
### Path MTU Tracking
PathMtuNotification (message type 0x13) provides end-to-end path MTU
feedback, adapting RFC 1191 Path MTU Discovery for overlay networks — the
transit-node `min()` propagation replaces ICMP Packet Too Big:
1. The source sets `path_mtu` in each SessionDatagram envelope to its
outbound link MTU.
2. Each transit node applies `min(current, transport.link_mtu(addr))` before
forwarding.
3. The destination receives the forward-path minimum and sends a
PathMtuNotification (2-byte body: u16 LE path_mtu) back to the source.
4. The source applies the notification with hysteresis:
- **Decrease**: immediate (take lower value).
- **Increase**: requires 3 consecutive higher-value notifications spanning
at least 2 × notification interval.
5. Notifications are sent on first measurement, on any decrease, and
periodically at `max(10s, 5 × SRTT)`.
### Send Failure Backoff
When a session MMP report cannot be delivered (destination unreachable, no
route), the sender applies exponential backoff to the probe interval (a
standard distributed systems pattern for transient failure handling):
- Each consecutive failure doubles the interval: 2x, 4x, 8x, 16x, 32x
- Backoff caps at 32x the base interval (5 consecutive failures)
- A successful send resets to the normal SRTT-based interval
- Debug logging is suppressed after 3 consecutive failures; a summary is
logged when the destination becomes reachable again
This prevents wasted CPU and log noise when a session's remote endpoint has
departed the network but the local session has not yet timed out.
### Idle Timeout Interaction
MMP reports (SenderReport, ReceiverReport) and PathMtuNotification do **not**
reset the session idle timer. Only application data (DataPacket, type 0x10)
resets `last_activity`. This ensures sessions with no application traffic
tear down after `node.session.idle_timeout_secs` (default 90s), while MMP
continues providing measurement data up to the teardown moment.
### Operator Logging
Session metrics are logged at info level (configurable via
`node.session_mmp.log_interval_secs`, default 30s):
```text
MMP session metrics session=npub1tdwa...84le rtt=4.3ms loss=0.6% jitter=0.2ms goodput=71.3MB/s mtu=1472 tx_pkts=1234 rx_pkts=5678
```
When FMP signals MtuExceeded (a transit node could not forward a
SessionDatagram because it exceeded the next-hop link MTU), FSP uses
the reported bottleneck MTU to adjust its session-layer path MTU
estimate immediately. See [fips-mtu.md](fips-mtu.md) for the full
reactive PMTUD mechanism.
## Implementation Status
@@ -559,36 +486,27 @@ MMP session metrics session=npub1tdwa...84le rtt=4.3ms loss=0.6% jitter=0.2ms go
### FIPS Internal Documentation
- [fips-intro.md](fips-intro.md) — Protocol overview and architecture
- [fips-concepts.md](fips-concepts.md) — Protocol overview
- [fips-architecture.md](fips-architecture.md) — Layer architecture and
identity model
- [fips-mesh-layer.md](fips-mesh-layer.md) — FMP specification (below FSP)
- [fips-ipv6-adapter.md](fips-ipv6-adapter.md) — IPv6 adaptation layer (above FSP)
- [fips-mesh-operation.md](fips-mesh-operation.md) — Routing, discovery, and
error recovery
- [fips-wire-formats.md](fips-wire-formats.md) — Wire format reference for all
session message types
- [fips-ipv6-adapter.md](fips-ipv6-adapter.md) — IPv6 adaptation layer
(above FSP)
- [fips-mesh-operation.md](fips-mesh-operation.md) — Routing, discovery,
and error recovery
- [fips-mmp.md](fips-mmp.md) — Metrics Measurement Protocol (link + session)
- [fips-mtu.md](fips-mtu.md) — Path MTU model (PathMtuNotification,
MtuExceeded, hysteresis)
- [fips-prior-work.md](fips-prior-work.md) — Noise XK, WireGuard,
DTLS replay window, IKEv2 simultaneous initiation, hybrid coordinate
warmup citations
- [../reference/wire-formats.md](../reference/wire-formats.md) — Wire
format reference for all session message types
- [../reference/security.md](../reference/security.md) — Cryptographic
primitives and rekey defaults
### External References
- Perrin, T. ["The Noise Protocol Framework"](https://noiseprotocol.org/noise.html).
Revision 34, 2018. *Framework for building crypto protocols using Diffie-Hellman
key agreement and AEAD ciphers. FSP uses the XK handshake pattern.*
- Donenfeld, J.A. ["WireGuard: Next Generation Kernel Network Tunnel"](https://www.wireguard.com/papers/wireguard.pdf).
NDSS 2017. *Transport-independent cryptographic sessions bound to identity keys
rather than network addresses; AEAD-only authentication model.*
- Rescorla, E., Modadugu, N. [RFC 6347](https://datatracker.ietf.org/doc/html/rfc6347):
"Datagram Transport Layer Security Version 1.2". 2012. *Explicit sequence numbers
with sliding bitmap window for replay protection over unreliable transports.*
- Kaufman, C., Hoffman, P., Nir, Y., Eronen, P., Kivinen, T.
[RFC 7296](https://datatracker.ietf.org/doc/html/rfc7296):
"Internet Key Exchange Protocol Version 2 (IKEv2)". 2014. *Simultaneous
initiation resolution (§2.8) and INITIAL_CONTACT peer restart detection (§2.4).*
- Mogul, J., Deering, S. [RFC 1191](https://datatracker.ietf.org/doc/html/rfc1191):
"Path MTU Discovery". 1990. *End-to-end path MTU discovery; FSP adapts this for
overlay networks using transit-node min() propagation.*
- [Yggdrasil Network](https://yggdrasil-network.github.io/). *Coordinate-based
overlay routing with session traffic used to warm transit node coordinate caches.*
docs/design/fips-spanning-tree.md
+19 -5
View File
@@ -71,8 +71,11 @@ quality.
2. Compute **effective depth** for each candidate peer:
`effective_depth = peer.depth + link_cost`, where
`link_cost = etx * (1.0 + srtt_ms / 100.0)` using locally measured MMP
metrics. When MMP metrics have not yet converged, `link_cost` defaults to
1.0, preserving pure depth-based behavior as a graceful fallback.
metrics. During cold start (no peer has MMP data yet), candidates without
measurements default to `link_cost = 1.0`, preserving pure depth-based
behavior. Once any peer has MMP data, unmeasured candidates are excluded
so that a freshly connected peer cannot win parent selection on the
default cost alone.
3. Apply **hysteresis**: switch parents only when the best candidate's
effective depth is significantly better than the current parent's:
`best_eff_depth < current_eff_depth * (1.0 - parent_hysteresis)`
@@ -109,6 +112,9 @@ immediate parent reselection:
(ETX and SRTT). No cumulative path costs are propagated, avoiding
the trust problems inherent in self-reported cost metrics in a
permissionless network.
- **Loop rejection**: Candidates whose advertised ancestry already contains
the local node are skipped, preventing two nodes from selecting each
other as parent and entering an alternating coordinate loop.
### After Parent Change
@@ -180,9 +186,14 @@ A node re-announces (propagates) only when its own state changes:
- **Root changed**: Always propagate — this is a significant topology event
- **Depth changed**: Always propagate — affects routing distance calculations
- **Mid-chain ancestor swap**: A reroute that replaces an interior ancestor
without changing the root or the path length still alters the node's
coordinate path, so it propagates. Without this, downstream peers would
route into a phantom intermediate that no longer appears on the parent's
tree.
- **Sequence-only refresh**: Does NOT propagate beyond depth 1 — peers that
receive a sequence-only update do not re-announce, because their own root
and depth have not changed
receive a sequence-only update do not re-announce, because their own root,
depth, and address path have not changed
This means TreeAnnounce cascades through the tree proportional to depth,
not network size. A change at depth D affects at most D nodes along the
@@ -303,12 +314,15 @@ Example: In a 1000-node network with depth 10 and 5 peers, a node stores
| Rate limiting (500ms per peer) | **Implemented** |
| Coord cache flush on parent change | **Implemented** |
| Flap dampening (extended hold-down on rapid switches) | **Implemented** |
| Loop rejection (ancestry self-check in `evaluate_parent`) | **Implemented** |
| Mid-chain ancestor swap propagation | **Implemented** |
| Per-ancestry-entry signatures | Future direction |
## References
- [fips-mesh-operation.md](fips-mesh-operation.md) — How the spanning tree
fits into mesh routing
- [fips-wire-formats.md](fips-wire-formats.md) — TreeAnnounce wire format
- [../reference/wire-formats.md](../reference/wire-formats.md) —
TreeAnnounce wire format
- [spanning-tree-dynamics.md](spanning-tree-dynamics.md) — Convergence
scenario walkthroughs
docs/design/fips-transport-layer.md
+96 -148
View File
@@ -1,5 +1,7 @@
# FIPS Transport Layer
<!-- markdownlint-disable MD024 -->
The transport layer is the bottom of the FIPS protocol stack. It delivers
datagrams between transport-specific endpoints over arbitrary physical or
logical media. Everything above — peer authentication, routing, encryption,
@@ -50,7 +52,7 @@ determine how much payload can fit in a single packet after link-layer
encryption overhead.
MTU is fundamentally a per-link property. A transport with a fixed MTU
(Ethernet: 1500, UDP configured at 1472) returns the same value for every
(Ethernet effective 1499, UDP default 1280) returns the same value for every
link — this is the degenerate case. Transports that negotiate MTU
per-connection (e.g., BLE ATT_MTU) report the negotiated value for each
link individually.
@@ -68,7 +70,7 @@ forwarding and LookupResponse transit annotation.
### Connection Lifecycle
For connection-oriented transports, manage the underlying connection: TCP
handshake, Tor circuit establishment, Bluetooth pairing. FMP cannot begin
handshake, Tor circuit establishment, BLE pairing. FMP cannot begin
the Noise IK link handshake until the transport-layer connection is
established.
@@ -117,7 +119,6 @@ for internet connectivity:
| --------- | ---------- | --- | ----------- | ----- |
| UDP/IP | host:port | 1280–1472 | Unreliable | Primary internet transport |
| TCP/IP | host:port | Stream | Reliable | Requires length-prefix framing |
| WebSocket | URL | Stream | Reliable | Browser-compatible |
| Tor | .onion | Stream | Reliable | High latency, strong anonymity |
**Shared medium transports** operate over broadcast- or multicast-capable
@@ -127,7 +128,6 @@ media:
| --------- | ---------- | --- | ----------- | ----- |
| Ethernet | MAC | 1500 | Unreliable | Raw AF_PACKET frames |
| WiFi | MAC | 1500 | Unreliable | Infrastructure mode = Ethernet |
| Bluetooth | BD_ADDR | 672–64K | Reliable | L2CAP |
| BLE | BD_ADDR | 23–517 | Reliable | Negotiated ATT_MTU |
| Radio | Device addr | 51–222 | Unreliable | Low bandwidth, long range |
@@ -157,9 +157,9 @@ require connection setup before FMP can begin the Noise IK link handshake,
adding startup latency.
**Stream vs. datagram**: Datagram transports have natural packet boundaries.
Stream transports (TCP, WebSocket, Tor) require framing to delineate FIPS
packets within the byte stream. The FMP common prefix includes a payload
length field that provides this framing directly, replacing the need for a
Stream transports (TCP, Tor) require framing to delineate FIPS packets
within the byte stream. The FMP common prefix includes a payload length
field that provides this framing directly, replacing the need for a
separate length-prefix layer.
**Addressing opacity**: Transport addresses are opaque byte vectors. FMP
@@ -189,9 +189,7 @@ proceed.
| Transport | Connection Setup |
| --------- | ---------------- |
| TCP/IP | TCP three-way handshake |
| WebSocket | HTTP upgrade + TCP |
| Tor | Circuit establishment (500ms–5s) |
| Bluetooth | L2CAP connection |
| Tor | Circuit establishment (typically 10–60s, default timeout 120s) |
| BLE | L2CAP CoC or GATT connection |
| Serial | Physical connection (static) |
@@ -203,9 +201,9 @@ Connected → Disconnected. Failure can occur during connection setup, adding
error handling paths that connectionless transports don't have.
**Startup latency**: Connection-oriented transports add delay before a peer
becomes usable. This ranges from milliseconds (TCP) to seconds (Tor
circuit). Peer timeout configuration must account for transport-specific
setup times.
becomes usable. This ranges from milliseconds (TCP) to tens of seconds
(Tor circuit). Peer timeout configuration must account for
transport-specific setup times.
**Framing**: Stream transports must delimit FIPS packets within the byte
stream. The FMP common prefix includes a payload length field that provides
@@ -229,45 +227,29 @@ NAT devices and firewalls, limiting deployment to networks without NAT.
### Socket Buffer Sizing
The default Linux UDP receive buffer (`net.core.rmem_default`, typically
212 KB) is insufficient for high-throughput forwarding. At ~85 MB/s, a 212 KB
buffer fills in ~2.5 ms; any stall in the async receive loop (decryption,
routing, forwarding overhead) causes the kernel to silently drop incoming
datagrams.
The default Linux UDP receive buffer (`net.core.rmem_default`,
typically 212 KB) is insufficient for high-throughput forwarding. At
~85 MB/s, a 212 KB buffer fills in ~2.5 ms; any stall in the async
receive loop (decryption, routing, forwarding overhead) causes the
kernel to silently drop incoming datagrams.
FIPS uses `socket2::Socket` wrapped in `tokio::io::unix::AsyncFd` for the
UDP receive path. This replaces `tokio::UdpSocket` and enables direct
`libc::recvmsg()` calls with ancillary data parsing — specifically the
`SO_RXQ_OVFL` socket option, which delivers a cumulative kernel receive
buffer drop counter on every received packet. The drop counter feeds into
the ECN congestion detection system (see
[fips-mesh-layer.md](fips-mesh-layer.md#ecn-congestion-signaling)).
FIPS uses `socket2::Socket` wrapped in `tokio::io::unix::AsyncFd` for
the UDP receive path. This replaces `tokio::UdpSocket` and enables
direct `libc::recvmsg()` calls with ancillary data parsing —
specifically the `SO_RXQ_OVFL` socket option, which delivers a
cumulative kernel receive buffer drop counter on every received
packet. The drop counter feeds into the ECN congestion detection
system (see [fips-mmp.md](fips-mmp.md#ecn-congestion-signaling)).
Socket buffers are configured at bind time via `socket2`:
| Parameter | Default | Description |
| ---------------- | ------- | ------------------------------------ |
| `recv_buf_size` | 2 MB | `SO_RCVBUF` — kernel receive buffer |
| `send_buf_size` | 2 MB | `SO_SNDBUF` — kernel send buffer |
Linux internally doubles the requested value (to account for kernel
bookkeeping overhead), so requesting 2 MB yields 4 MB actual buffer space.
The kernel silently clamps to `net.core.rmem_max` if the request exceeds it.
**Host requirement**: `net.core.rmem_max` and `net.core.wmem_max` must be
set to at least the requested buffer size on the host. For Docker containers,
this must be configured on the Docker host (containers share the host kernel).
Verify with:
```text
sysctl net.core.rmem_max net.core.wmem_max
```
Actual buffer sizes are logged at startup:
```text
UDP transport started local_addr=0.0.0.0:2121 recv_buf=4194304 send_buf=4194304
```
Socket buffers (`recv_buf_size`, `send_buf_size`) are configured at
bind time via `socket2`. Linux internally doubles the requested value
(to account for kernel bookkeeping overhead) and silently clamps to
`net.core.rmem_max` / `net.core.wmem_max` if the request exceeds the
host kernel limits. The full UDP transport configuration is in
[../reference/configuration.md](../reference/configuration.md). The
host-side sysctl requirements and how to set them persistently live
in
[../how-to/tune-udp-buffers.md](../how-to/tune-udp-buffers.md).
## Ethernet: The Local Network Transport
@@ -317,18 +299,16 @@ x-only public key. Receiving nodes extract the MAC source address from the
frame and the public key from the payload, then report the discovered peer
to FMP.
Four configuration flags control discovery behavior:
Four configuration flags control discovery behavior — `discovery`
(listen for beacons), `announce` (broadcast beacons), `auto_connect`
(initiate handshakes to discovered peers), and `accept_connections`
(accept inbound handshakes). The flag table and per-flag defaults
live in [../reference/configuration.md](../reference/configuration.md)
under `transports.ethernet.*`.
| Flag | Default | Description |
| ---- | ------- | ----------- |
| `discovery` | true | Listen for beacons from other nodes |
| `announce` | false | Broadcast beacons periodically |
| `auto_connect` | false | Initiate handshakes to discovered peers |
| `accept_connections` | false | Accept inbound handshake attempts |
A typical discoverable node sets `announce: true`, `auto_connect: true`, and
`accept_connections: true`. A passive listener uses just `discovery: true` to
observe the network without announcing itself.
A typical discoverable node sets `announce`, `auto_connect`, and
`accept_connections` all true. A passive listener uses just
`discovery: true` to observe the network without announcing itself.
### WiFi Compatibility
@@ -343,10 +323,12 @@ Startup logging:
Ethernet transport started name=eth0 interface=eth0 mac=aa:bb:cc:dd:ee:ff mtu=1499 if_mtu=1500
```
## TCP/IP: Firewall Traversal Transport
## TCP/IP: Transport for UDP-Filtered Networks
For networks where UDP is blocked but TCP port 443 is open, the TCP
transport provides an alternative path.
For peers whose networks filter outbound UDP, the TCP transport
provides an alternative datagram path between public endpoints. TCP
is not a NAT-traversal mechanism — there is no `tcp:nat` analogue to
the UDP hole-punch flow.
FIPS protocols (FMP, FSP, MMP) are all unreliable datagrams. Running them
over TCP introduces head-of-line blocking, which adds latency jitter. MMP
@@ -425,21 +407,12 @@ removes it from the pool and aborts its receive task.
### Configuration
```yaml
transports:
tcp:
bind_addr: "0.0.0.0:8443" # Listen address (omit for outbound-only)
mtu: 1400 # Default MTU
connect_timeout_ms: 5000 # Outbound connect timeout
nodelay: true # TCP_NODELAY (disable Nagle)
keepalive_secs: 30 # TCP keepalive interval (0 = disabled)
recv_buf_size: 2097152 # SO_RCVBUF (2 MB)
send_buf_size: 2097152 # SO_SNDBUF (2 MB)
max_inbound_connections: 256 # Resource protection limit
```
If `bind_addr` is configured, the transport accepts inbound connections.
Without it, the transport operates in outbound-only mode (no listener
The TCP transport configuration block (`transports.tcp.*` — bind
address, MTU, connect timeout, TCP_NODELAY, keepalive, socket buffer
sizes, max inbound connections) is documented in
[../reference/configuration.md](../reference/configuration.md). If
`bind_addr` is configured, the transport accepts inbound connections;
without it, the transport operates in outbound-only mode (no listener
socket is created).
## Tor: The Anonymity Transport
@@ -531,22 +504,13 @@ connections arrive from `127.0.0.1` (Tor daemon's local forwarding); peer
identity is resolved during the Noise IK handshake, not from the transport
address.
Configuration requires coordinating `torrc` and `fips.yaml`:
```text
# torrc
HiddenServiceDir /var/lib/tor/fips
HiddenServicePort 8443 127.0.0.1:8444
# fips.yaml tor section
mode: "directory"
directory_service:
hostname_file: "/var/lib/tor/fips/hostname"
bind_addr: "127.0.0.1:8444"
```
The `HiddenServicePort` external port (8443) is what peers connect to.
The bind_addr must match the `HiddenServicePort` target address.
Configuration requires coordinating `torrc` and `fips.yaml`. The
operator setup — torrc directives, `fips.yaml` `tor` section,
HiddenServiceDir permissions, and `Sandbox 1` notes — is in
[../how-to/deploy-tor-onion.md](../how-to/deploy-tor-onion.md). In
brief: the `HiddenServicePort` external port is what peers connect
to, and `tor.directory_service.bind_addr` must match the
`HiddenServicePort` target address.
### Session Independence
@@ -569,10 +533,9 @@ anonymous node's IP.
### Latency Characteristics
Tor adds 200ms–2s RTT per circuit. First-packet latency after connection
is higher (~2.8s) due to circuit warm-up. MMP measures this elevated
latency, and cost-based parent selection penalizes Tor links (high SRTT
→ high link cost). ETX is 1.0 since TCP handles retransmission.
Tor adds 200ms–2s RTT per circuit. MMP measures this elevated latency,
and cost-based parent selection penalizes Tor links (high SRTT → high
link cost). ETX is 1.0 since TCP handles retransmission.
Tor throughput is typically 1–5 Mbps — adequate for control plane and
moderate data transfer, not for bulk transfer.
@@ -600,37 +563,22 @@ connections (`/run/tor/control`) are preferred over TCP for security.
### Configuration
```yaml
transports:
tor:
mode: "socks5" # "socks5", "control_port", or "directory"
socks5_addr: "127.0.0.1:9050" # SOCKS5 proxy address
connect_timeout_ms: 120000 # Connect timeout (120s for Tor circuits)
mtu: 1400 # Default MTU
# control_port mode: monitoring via Tor control port (no inbound)
# control_addr: "/run/tor/control" # Unix socket (preferred) or host:port
# control_auth: "cookie" # "cookie" or "password:<secret>"
# cookie_path: "/var/run/tor/control.authcookie"
# directory mode: inbound via Tor-managed HiddenServiceDir
# directory_service:
# hostname_file: "/var/lib/tor/fips/hostname"
# bind_addr: "127.0.0.1:8444"
# max_inbound_connections: 64
```
Three modes are available:
The Tor transport block (`transports.tor.*`) is documented in
[../reference/configuration.md](../reference/configuration.md). Three
modes are available:
- **`socks5`** (default): Outbound-only through a SOCKS5 proxy. No
control port, no inbound connections.
- **`control_port`**: Outbound via SOCKS5 plus control port connection
for Tor daemon monitoring. No inbound connections.
- **`directory`** (recommended for inbound): Outbound via SOCKS5 plus
inbound via Tor-managed `HiddenServiceDir` onion service. Optionally
connects to the control port for monitoring when `control_addr` is set.
Enables Tor's `Sandbox 1` for maximum security.
inbound via Tor-managed `HiddenServiceDir` onion service.
Optionally connects to the control port for monitoring when
`control_addr` is set. Enables Tor's `Sandbox 1` for maximum
security.
The Tor transport requires an external Tor daemon. Named instances are
supported for multiple proxy endpoints.
The Tor transport requires an external Tor daemon. Named instances
are supported for multiple proxy endpoints.
### Implementation Roadmap
@@ -645,21 +593,11 @@ supported for multiple proxy endpoints.
### Statistics
The transport tracks per-instance statistics:
| Counter | Description |
| ------- | ----------- |
| `packets_sent` / `bytes_sent` | Successful sends |
| `packets_recv` / `bytes_recv` | Successful receives |
| `send_errors` / `recv_errors` | Send/receive failures |
| `connections_established` | Successful SOCKS5 connections |
| `connect_timeouts` | Connection timeout count |
| `connect_refused` | Connection refused count |
| `socks5_errors` | SOCKS5 protocol errors |
| `mtu_exceeded` | Packets rejected for MTU violation |
| `connections_accepted` | Accepted inbound connections via onion service |
| `connections_rejected` | Rejected inbound connections (limit exceeded) |
| `control_errors` | Tor control port errors |
The Tor transport exposes per-instance counters covering successful
send/receive, send/receive errors, connection establishment,
SOCKS5-level errors, MTU rejections, accepted/rejected inbound
connections, and Tor control-port errors. The full counter table
lives in [../reference/transports.md](../reference/transports.md).
## Discovery
@@ -731,8 +669,8 @@ Key properties:
> control whether discovered peers are connected automatically or require
> explicit configuration. TCP and Tor have no built-in discovery mechanism.
> Nostr relay discovery and STUN-assisted UDP hole punching are
> implemented behind the `nostr-discovery` cargo feature; see
> [fips-configuration.md](fips-configuration.md) for the
> implemented and toggled via configuration; see
> [../reference/configuration.md](../reference/configuration.md) for the
> `node.discovery.nostr.*` configuration tree.
## Transport Interface
@@ -817,8 +755,9 @@ on all forwarded datagrams.
### Transport Addresses
Transport addresses (`TransportAddr`) are opaque byte vectors. The transport
layer interprets them (e.g., UDP/TCP resolve "host:port" strings (IP fast path, DNS fallback with 60s cache for UDP)); all layers above
treat them as opaque handles passed back to the transport for sending.
layer interprets them — e.g. UDP and TCP resolve `host:port` strings (IP
fast path, DNS fallback with a 60s cache on UDP). All layers above treat
them as opaque handles passed back to the transport for sending.
### Transport State Machine
@@ -839,9 +778,9 @@ transitions through `Starting` to `Up` (operational). `stop()` moves to
| UDP/IP | **Implemented** | Primary transport, AsyncFd/recvmsg, SO_RXQ_OVFL kernel drop detection |
| TCP/IP | **Implemented** | FMP header-based framing, non-blocking connect, per-connection MSS MTU |
| Ethernet | **Implemented** | AF_PACKET SOCK_DGRAM, EtherType 0x2121, beacon discovery, Linux only |
| WiFi | Future direction | Infrastructure mode = Ethernet driver |
| WiFi | **Implemented** (via Ethernet transport, infrastructure mode) | mac80211 translates 802.11↔802.3; broadcast beacons unreliable through APs |
| Tor | **Implemented** | Outbound SOCKS5, inbound via onion service, .onion and clearnet addressing |
| BLE | Future direction | ATT_MTU negotiation, per-link MTU |
| BLE | **Implemented** (Linux/glibc only; experimental) | L2CAP CoC, ATT_MTU negotiation, per-link MTU; musl/macOS/Windows skip |
| Radio | Future direction | Constrained MTU (51–222 bytes) |
| Serial | Future direction | SLIP/COBS framing, point-to-point |
@@ -849,7 +788,7 @@ transitions through `Starting` to `Up` (operational). `stop()` moves to
### TCP-over-TCP Avoidance
Running TCP application traffic over a reliable transport (TCP, WebSocket)
Running TCP application traffic over a reliable transport (TCP, Tor)
creates a layering violation where retransmission and congestion control
operate at both levels. When the inner TCP detects loss (which may just be
transport-layer retransmission delay), it retransmits, creating more traffic
@@ -884,6 +823,15 @@ quality difference is significant. Link cost is not yet used in
## References
- [fips-intro.md](fips-intro.md) — Protocol overview and layer architecture
- [fips-mesh-layer.md](fips-mesh-layer.md) — FMP specification (the layer above)
- [fips-wire-formats.md](fips-wire-formats.md) — Transport framing details
- [fips-concepts.md](fips-concepts.md) — Protocol overview
- [fips-architecture.md](fips-architecture.md) — Layer architecture
- [fips-mesh-layer.md](fips-mesh-layer.md) — FMP specification (the
layer above)
- [fips-mtu.md](fips-mtu.md) — How transport-reported `link_mtu`
feeds the unified path-MTU model
- [../reference/wire-formats.md](../reference/wire-formats.md) —
Transport framing details
- [../reference/configuration.md](../reference/configuration.md) —
Per-transport configuration blocks
- [../reference/transports.md](../reference/transports.md) —
Per-transport statistics counter inventory
docs/design/fips-wire-formats.md
-886
View File
@@ -1,886 +0,0 @@
# FIPS Wire Formats
This document is the comprehensive wire format reference for all three
protocol layers. It covers transport framing, link-layer message formats,
and session-layer message formats, with an encapsulation walkthrough showing
how application data is wrapped through each layer.
## Encoding Rules
- All multi-byte integers are **little-endian** (LE)
- NodeAddr is **16 bytes** — truncated SHA-256 hash of public key
- Signatures are **64 bytes** — secp256k1 Schnorr
- Variable-length arrays use a **2-byte u16 LE count prefix** followed by
that many items
- Public keys are **33 bytes** — compressed secp256k1 (02/03 prefix + 32
bytes)
## Transport Framing
Datagram-oriented transports (UDP, raw Ethernet, radio) preserve natural
packet boundaries and require no additional framing. Stream-oriented
transports (TCP, WebSocket, Tor) must delineate FIPS packets within the
byte stream; the common prefix `payload_len` field provides this
framing directly. TCP and Tor share a common stream reader
(`tcp/stream.rs`) that implements this framing.
**Ethernet data frame header.** The Ethernet transport prepends a 3-byte
header before the FMP payload on data frames: a 1-byte frame type
(`0x00`) followed by a 2-byte little-endian payload length. The length
field allows the receiver to trim Ethernet minimum-frame padding that
would otherwise corrupt AEAD verification. Beacon frames (`0x01`) have
no length field (fixed 34-byte payload). These bytes are consumed by the
transport layer and are not visible to FMP. The effective MTU for FMP is
the interface MTU minus three bytes (typically 1497).
## Link-Layer Formats
All FMP packets begin with a **4-byte common prefix** that identifies the
protocol version, session lifecycle phase, per-packet flags, and payload
length.
### Common Prefix (4 bytes)
![Common Prefix](diagrams/common-prefix.svg)
| Field | Size | Description |
| ----- | ---- | ----------- |
| version | 4 bits (high) | Protocol version. Currently 0x0 |
| phase | 4 bits (low) | Session lifecycle phase (see table) |
| flags | 1 byte | Per-packet signal flags (zero during handshake) |
| payload_len | 2 bytes LE | Length of payload after phase-specific header, excluding AEAD tag |
### Phase Table
| Phase | Type | Description |
| ----- | ---- | ----------- |
| 0x0 | Established frame | Post-handshake encrypted traffic |
| 0x1 | Noise IK msg1 | Handshake initiation |
| 0x2 | Noise IK msg2 | Handshake response |
### Flags (Established Phase Only)
| Bit | Name | Description |
| --- | ---- | ----------- |
| 0 | K (key epoch) | Selects active key during rekeying |
| 1 | CE | Congestion Experienced echo |
| 2 | SP (spin bit) | RTT measurement |
| 3-7 | — | Reserved (must be zero) |
Flags must be zero in handshake packets (phase 0x1 and 0x2).
### Established Frame (phase 0x0)
All post-handshake traffic between authenticated peers. Contains one
encrypted link-layer message.
**Outer header** (16 bytes, used as AEAD AAD):
![Established frame outer header](diagrams/established-outer-header.svg)
| Field | Size | Description |
| ----- | ---- | ----------- |
| common prefix | 4 bytes | ver=0, phase=0, flags, payload_len |
| receiver_idx | 4 bytes LE | Session index for O(1) lookup |
| counter | 8 bytes LE | Monotonic nonce, used as AEAD nonce and for replay detection |
The entire 16-byte header is authenticated as Associated Data (AAD) in the
ChaCha20-Poly1305 AEAD construction.
**Encrypted inner header** (5 bytes, first bytes of plaintext):
![Established frame inner header](diagrams/established-inner-header.svg)
| Field | Size | Description |
| ----- | ---- | ----------- |
| timestamp | 4 bytes LE | Session-relative milliseconds (u32) |
| msg_type | 1 byte | Link-layer message type |
After decryption, the plaintext begins with the 4-byte timestamp followed by
the 1-byte message type and message-specific fields.
**Complete encrypted frame**:
![Complete encrypted frame](diagrams/established-complete-frame.svg)
### Message Type Table
| Type | Message | Description |
| ---- | ------- | ----------- |
| 0x00 | SessionDatagram | Encapsulated session-layer payload for forwarding |
| 0x01 | SenderReport | MMP sender-side metrics report (48 bytes) |
| 0x02 | ReceiverReport | MMP receiver-side metrics report (68 bytes) |
| 0x10 | TreeAnnounce | Spanning tree state announcement |
| 0x20 | FilterAnnounce | Bloom filter reachability update |
| 0x30 | LookupRequest | Coordinate discovery request |
| 0x31 | LookupResponse | Coordinate discovery response |
| 0x50 | Disconnect | Orderly link teardown |
| 0x51 | Heartbeat | Link liveness probe |
### Noise IK Message 1 (phase 0x1)
Handshake initiation from connecting party.
![Noise IK message 1](diagrams/noise-ik-msg1.svg)
Common prefix: ver=0, phase=0x1, flags=0, payload_len=110 (4 + 106).
| Field | Size | Description |
| ----- | ---- | ----------- |
| common prefix | 4 bytes | ver=0, phase=1, flags=0, payload_len |
| sender_idx | 4 bytes LE | Initiator's session index (becomes receiver's `receiver_idx`) |
| noise_msg1 | 106 bytes | Noise IK first message |
**Noise msg1 breakdown** (106 bytes):
| Offset | Field | Size | Description |
| ------ | ----- | ---- | ----------- |
| 0 | ephemeral_pubkey | 33 bytes | Initiator's ephemeral key (compressed secp256k1) |
| 33 | encrypted_static | 49 bytes | Initiator's static key (33) + AEAD tag (16) |
| 82 | encrypted_epoch | 24 bytes | Startup epoch (8) + AEAD tag (16) |
Noise pattern: `-> e, es, s, ss` with epoch payload
### Noise IK Message 2 (phase 0x2)
Handshake response from responder.
![Noise IK message 2](diagrams/noise-ik-msg2.svg)
Common prefix: ver=0, phase=0x2, flags=0, payload_len=65 (4 + 4 + 57).
| Field | Size | Description |
| ----- | ---- | ----------- |
| common prefix | 4 bytes | ver=0, phase=2, flags=0, payload_len |
| sender_idx | 4 bytes LE | Responder's session index |
| receiver_idx | 4 bytes LE | Echo of initiator's sender_idx from msg1 |
| noise_msg2 | 57 bytes | Noise IK second message |
**Noise msg2 breakdown** (57 bytes):
| Offset | Field | Size | Description |
| ------ | ----- | ---- | ----------- |
| 0 | ephemeral_pubkey | 33 bytes | Responder's ephemeral key (compressed secp256k1) |
| 33 | encrypted_epoch | 24 bytes | Startup epoch (8) + AEAD tag (16) |
Noise pattern: `<- e, ee, se` with epoch payload
After msg2, both parties derive identical symmetric session keys. The
encrypted epoch in msg1 and msg2 enables peer restart detection — if a
peer's epoch changes, the other side knows it restarted and must
re-establish the link.
### Index Semantics
Each party in a link session maintains two indices:
| Index | Chosen By | Used By | Purpose |
| ----- | --------- | ------- | ------- |
| our_index | Us | Them | They include this as `receiver_idx` in packets to us |
| their_index | Them | Us | We include this as `receiver_idx` in packets to them |
### Handshake Flow
![Handshake flow](diagrams/handshake-flow.svg)
## Link-Layer Message Types
These messages are carried as plaintext inside encrypted frames (phase 0x0).
After decryption of the AEAD ciphertext, the plaintext begins with a 4-byte
session-relative timestamp followed by the 1-byte message type and
message-specific fields.
### TreeAnnounce (0x10)
Spanning tree state announcement, exchanged between direct peers only.
![TreeAnnounce](diagrams/tree-announce.svg)
| Offset | Field | Size | Description |
| ------ | ----- | ---- | ----------- |
| 0 | msg_type | 1 byte | 0x10 |
| 1 | version | 1 byte | 0x01 (v1) |
| 2 | sequence | 8 bytes LE | Monotonic counter, increments on parent change |
| 10 | timestamp | 8 bytes LE | Unix seconds |
| 18 | parent | 16 bytes | NodeAddr of selected parent (self = root) |
| 34 | ancestry_count | 2 bytes LE | Number of AncestryEntry records |
| 36 | ancestry | 32 x n bytes | AncestryEntry array (self -> root) |
| 36 + 32n | signature | 64 bytes | Schnorr signature over entire message |
**AncestryEntry** (32 bytes):
![AncestryEntry](diagrams/ancestry-entry.svg)
| Offset | Field | Size | Description |
| ------ | ----- | ---- | ----------- |
| 0 | node_addr | 16 bytes | Node's routing identifier |
| 16 | sequence | 8 bytes LE | Node's sequence number |
| 24 | timestamp | 8 bytes LE | Node's Unix timestamp |
**Size**: `100 + (n x 32)` bytes, where n = `ancestry_count` (depth + 1,
includes self)
| Tree Depth | Payload | With Link Overhead |
| ---------- | ------- | ------------------ |
| 0 (root) | 132 bytes | 169 bytes |
| 3 | 228 bytes | 265 bytes |
| 5 | 292 bytes | 329 bytes |
| 10 | 452 bytes | 489 bytes |
### FilterAnnounce (0x20)
Bloom filter reachability update, exchanged between direct peers only.
![FilterAnnounce](diagrams/filter-announce.svg)
| Offset | Field | Size | Description |
| ------ | ----- | ---- | ----------- |
| 0 | msg_type | 1 byte | 0x20 |
| 1 | sequence | 8 bytes LE | Monotonic counter for freshness |
| 9 | hash_count | 1 byte | Number of hash functions (5 in v1) |
| 10 | size_class | 1 byte | Filter size: `512 << size_class` bytes |
| 11 | filter_bits | variable | Bloom filter bit array |
**Size class table**:
| size_class | Bytes | Bits | Status |
| ---------- | ----- | ---- | ------ |
| 0 | 512 | 4,096 | Reserved |
| 1 | 1,024 | 8,192 | **v1 (MUST use)** |
| 2 | 2,048 | 16,384 | Reserved |
| 3 | 4,096 | 32,768 | Reserved |
**v1 payload**: 1,035 bytes (11 header + 1,024 filter).
With link overhead: 1,072 bytes.
### LookupRequest (0x30)
Coordinate discovery request, routed through the spanning tree via
bloom-filter-guided forwarding. Each transit node forwards only to tree
peers (parent + children) whose bloom filter contains the target.
Request_id dedup in recent_requests handles edge cases from tree
restructuring.
![LookupRequest](diagrams/lookup-request.svg)
| Offset | Field | Size | Description |
| ------ | ----- | ---- | ----------- |
| 0 | msg_type | 1 byte | 0x30 |
| 1 | request_id | 8 bytes LE | Unique random identifier |
| 9 | target | 16 bytes | NodeAddr being sought |
| 25 | origin | 16 bytes | Requester's NodeAddr |
| 41 | ttl | 1 byte | Remaining hops (default 64) |
| 42 | min_mtu | 2 bytes LE | Minimum transport MTU the origin requires (0 = no requirement) |
| 44 | origin_coords_cnt | 2 bytes LE | Number of coordinate entries |
| 46 | origin_coords | 16 x n bytes | Requester's ancestry (NodeAddr only) |
**Size**: `46 + (n x 16)` bytes, where n = origin depth + 1
| Origin Depth | Payload |
| ------------ | ------- |
| 3 | 110 bytes |
| 5 | 142 bytes |
| 10 | 222 bytes |
### LookupResponse (0x31)
Coordinate discovery response, reverse-path routed back to the
requester via the transit nodes that forwarded the request.
![LookupResponse](diagrams/lookup-response.svg)
| Offset | Field | Size | Description |
| ------ | ----- | ---- | ----------- |
| 0 | msg_type | 1 byte | 0x31 |
| 1 | request_id | 8 bytes LE | Echoes the request's ID |
| 9 | target | 16 bytes | NodeAddr that was found |
| 25 | path_mtu | 2 bytes LE | Minimum MTU along response path (transit-annotated) |
| 27 | target_coords_cnt | 2 bytes LE | Number of coordinate entries |
| 29 | target_coords | 16 x n bytes | Target's ancestry (NodeAddr only) |
| 29 + 16n | proof | 64 bytes | Schnorr signature over `(request_id \|\| target \|\| target_coords)` |
**Size**: `93 + (n x 16)` bytes
| Target Depth | Payload |
| ------------ | ------- |
| 3 | 141 bytes |
| 5 | 173 bytes |
| 10 | 253 bytes |
The `path_mtu` field is initialized to `u16::MAX` by the target and each
transit hop applies `min(path_mtu, outgoing_link_mtu)`, giving the
originator an MTU estimate for the discovered path.
**Proof coverage**: Signs `(request_id || target || target_coords)` —
`path_mtu` is excluded from the proof because it is a transit annotation
modified at each hop. Coordinates are included because proof verification
at the source confirms the target actually holds the claimed tree position.
The source verifies the proof upon receipt.
### SessionDatagram (0x00)
Encapsulated session-layer payload for multi-hop forwarding.
![SessionDatagram](diagrams/session-datagram.svg)
| Offset | Field | Size | Description |
| ------ | ----- | ---- | ----------- |
| 0 | msg_type | 1 byte | 0x00 |
| 1 | ttl | 1 byte | Remaining hops, decremented each hop |
| 2 | path_mtu | 2 bytes LE | Path MTU, min'd at each forwarding hop |
| 4 | src_addr | 16 bytes | Source NodeAddr |
| 20 | dest_addr | 16 bytes | Destination NodeAddr |
| 36 | payload | variable | Session-layer message |
**Fixed header**: 36 bytes (`SESSION_DATAGRAM_HEADER_SIZE`)
The `path_mtu` field is initialized to `u16::MAX` by the sender and each
forwarding hop applies `min(path_mtu, outgoing_link_mtu)`, giving the
receiver an estimate of the minimum MTU along the path.
The payload is opaque to transit nodes — session-layer encrypted
independently of link encryption.
### Disconnect (0x50)
Orderly link teardown with reason code.
![Disconnect](diagrams/disconnect.svg)
| Offset | Field | Size | Description |
| ------ | ----- | ---- | ----------- |
| 0 | msg_type | 1 byte | 0x50 |
| 1 | reason | 1 byte | Disconnect reason code |
**Reason codes**:
| Code | Name | Description |
| ---- | ---- | ----------- |
| 0x00 | Shutdown | Normal operator-requested stop |
| 0x01 | Restart | Restarting, may reconnect soon |
| 0x02 | ProtocolError | Protocol error encountered |
| 0x03 | TransportFailure | Transport failure |
| 0x04 | ResourceExhaustion | Memory or connection limit |
| 0x05 | SecurityViolation | Authentication or policy violation |
| 0x06 | ConfigurationChange | Peer removed from configuration |
| 0x07 | Timeout | Heartbeat liveness timeout |
| 0xFF | Other | Unspecified reason |
### SenderReport (0x01)
Sent by the frame sender to provide interval-based transmission statistics.
![SenderReport](diagrams/sender-report.svg)
| Offset | Field | Size | Encoding |
| ------ | ----- | ---- | -------- |
| 0 | msg_type | 1 | `0x01` |
| 1 | reserved | 3 | Zero |
| 4 | interval_start_counter | 8 | u64 LE — first counter in this interval |
| 12 | interval_end_counter | 8 | u64 LE — last counter in this interval |
| 20 | interval_start_timestamp | 4 | u32 LE — timestamp at interval start |
| 24 | interval_end_timestamp | 4 | u32 LE — timestamp at interval end |
| 28 | interval_bytes_sent | 4 | u32 LE — payload bytes sent in interval |
| 32 | cumulative_packets_sent | 8 | u64 LE — total packets sent on this link |
| 40 | cumulative_bytes_sent | 8 | u64 LE — total bytes sent on this link |
**Total: 48 bytes.**
### ReceiverReport (0x02)
Sent by the frame receiver to provide loss, jitter, and timing feedback.
![ReceiverReport](diagrams/receiver-report.svg)
| Offset | Field | Size | Encoding |
| ------ | ----- | ---- | -------- |
| 0 | msg_type | 1 | `0x02` |
| 1 | reserved | 3 | Zero |
| 4 | highest_counter | 8 | u64 LE — highest counter value received |
| 12 | cumulative_packets_recv | 8 | u64 LE — total packets received |
| 20 | cumulative_bytes_recv | 8 | u64 LE — total bytes received |
| 28 | timestamp_echo | 4 | u32 LE — echoed sender timestamp for RTT |
| 32 | dwell_time | 2 | u16 LE — time between receive and echo (ms) |
| 34 | max_burst_loss | 2 | u16 LE — largest loss burst in interval |
| 36 | mean_burst_loss | 2 | u16 LE — mean burst length (u8.8 fixed-point) |
| 38 | reserved | 2 | Zero |
| 40 | jitter | 4 | u32 LE — interarrival jitter (microseconds) |
| 44 | ecn_ce_count | 4 | u32 LE — cumulative ECN-CE marked packets |
| 48 | owd_trend | 4 | i32 LE — one-way delay trend (µs/s, signed) |
| 52 | burst_loss_count | 4 | u32 LE — number of loss bursts in interval |
| 56 | cumulative_reorder_count | 4 | u32 LE — total reordered packets |
| 60 | interval_packets_recv | 4 | u32 LE — packets received in interval |
| 64 | interval_bytes_recv | 4 | u32 LE — bytes received in interval |
**Total: 68 bytes.**
## Session-Layer Message Formats
Session-layer messages are carried as the payload of a SessionDatagram (0x00).
All FSP messages begin with a **4-byte common prefix** that identifies the
protocol version, session lifecycle phase, per-packet flags, and payload length.
### FSP Common Prefix (4 bytes)
| Field | Size | Description |
| ----- | ---- | ----------- |
| version | 4 bits (high) | Protocol version. Currently 0x0 |
| phase | 4 bits (low) | Session lifecycle phase (see table) |
| flags | 1 byte | Per-packet signal flags (zero during handshake) |
| payload_len | 2 bytes LE | Length of payload after phase-specific header |
### FSP Phase Table
| Phase | Type | Description |
| ----- | ---- | ----------- |
| 0x0 | Established | Post-handshake encrypted traffic or plaintext error signals |
| 0x1 | Handshake msg1 | SessionSetup (Noise XK msg1) |
| 0x2 | Handshake msg2 | SessionAck (Noise XK msg2) |
| 0x3 | Handshake msg3 | SessionMsg3 (Noise XK msg3) |
### FSP Flags (Established Phase Only)
| Bit | Name | Description |
| --- | ---- | ----------- |
| 0 | CP (coords present) | Source and destination coordinates follow the header in cleartext |
| 1 | K (key epoch) | Selects active key during rekeying |
| 2 | U (unencrypted) | Payload is plaintext (error signals) |
| 3-7 | — | Reserved (must be zero) |
Flags must be zero in handshake packets (phase 0x1, 0x2, and 0x3).
### FSP Encrypted Message (phase 0x0, U flag clear)
Post-handshake encrypted data. The 12-byte cleartext header is used as AEAD
AAD. Coordinates may appear in cleartext between the header and ciphertext
when the CP flag is set.
**Cleartext header** (12 bytes, used as AEAD AAD):
| Field | Size | Description |
| ----- | ---- | ----------- |
| common prefix | 4 bytes | ver=0, phase=0, flags, payload_len |
| counter | 8 bytes LE | Monotonic nonce, used as AEAD nonce and for replay detection |
**Optional cleartext coordinates** (when CP flag is set):
| Field | Size | Description |
| ----- | ---- | ----------- |
| src_coords_count | 2 bytes LE | Number of source coordinate entries |
| src_coords | 16 x n bytes | Source's ancestry (NodeAddr, self -> root) |
| dest_coords_count | 2 bytes LE | Number of dest coordinate entries |
| dest_coords | 16 x m bytes | Destination's ancestry |
Transit nodes parse the CP flag and extract coordinates without decryption.
**Encrypted inner header** (6 bytes, first bytes of AEAD plaintext):
| Field | Size | Description |
| ----- | ---- | ----------- |
| timestamp | 4 bytes LE | Session-relative milliseconds (u32) |
| msg_type | 1 byte | Session-layer message type |
| inner_flags | 1 byte | Bit 0: SP (spin bit for RTT measurement) |
After the inner header, the remaining plaintext is the message-type-specific
body.
**Complete encrypted message**:
![FSP complete encrypted message](diagrams/fsp-complete-message.svg)
### FSP Session Message Types
| Type | Message | Description |
| ---- | ------- | ----------- |
| 0x10 | Data | Port-multiplexed service payload (see DataPacket below) |
| 0x11 | SenderReport | MMP sender-side metrics report |
| 0x12 | ReceiverReport | MMP receiver-side metrics report |
| 0x13 | PathMtuNotification | End-to-end path MTU echo |
| 0x14 | CoordsWarmup | Standalone coordinate cache warming |
| 0x20 | CoordsRequired | Error: transit node lacks destination coordinates |
| 0x21 | PathBroken | Error: greedy routing reached dead end |
| 0x22 | MtuExceeded | Error: forwarded packet exceeds next-hop MTU |
Message types 0x10-0x14 are carried inside the AEAD ciphertext (dispatched
by the `msg_type` field in the encrypted inner header). Types 0x20-0x22 are
plaintext error signals (U flag set, no encryption).
Session-layer SenderReport (0x11) and ReceiverReport (0x12) use the same
body format as their link-layer counterparts (0x01 and 0x02). The msg_type
byte in the body matches the link-layer value; dispatch to the correct layer
happens at the session level based on the FSP message type.
### SessionSetup (phase 0x1)
Establishes a session and warms transit coordinate caches. Contains the
first message of the Noise XK handshake (ephemeral key only — the
initiator's static identity is not revealed until msg3).
SessionSetup, SessionAck, and SessionMsg3 are identified by the **phase**
field in the FSP common prefix (0x1, 0x2, 0x3), not by a message type
byte. The `msg_type` field in the encrypted inner header applies only to
established-phase (0x0) messages.
Encoded with FSP prefix: ver=0, phase=0x1, flags=0, payload_len.
![SessionSetup](diagrams/session-setup.svg)
**Body** (after 4-byte FSP prefix):
| Offset | Field | Size | Description |
| ------ | ----- | ---- | ----------- |
| 0 | flags | 1 byte | Bit 0: REQUEST_ACK, Bit 1: BIDIRECTIONAL |
| 1 | src_coords_count | 2 bytes LE | Number of source coordinate entries |
| 3 | src_coords | 16 x n bytes | Source's ancestry (NodeAddr, self -> root) |
| ... | dest_coords_count | 2 bytes LE | Number of dest coordinate entries |
| ... | dest_coords | 16 x m bytes | Destination's ancestry |
| ... | handshake_len | 2 bytes LE | Noise payload length |
| ... | handshake_payload | variable | Noise XK msg1 (33 bytes — ephemeral key only) |
### SessionAck (phase 0x2)
Second message of the Noise XK handshake. The responder sends its
ephemeral key and encrypted epoch.
Encoded with FSP prefix: ver=0, phase=0x2, flags=0, payload_len.
![SessionAck](diagrams/session-ack.svg)
**Body** (after 4-byte FSP prefix):
| Offset | Field | Size | Description |
| ------ | ----- | ---- | ----------- |
| 0 | flags | 1 byte | Reserved |
| 1 | src_coords_count | 2 bytes LE | Number of acknowledger coordinate entries |
| 3 | src_coords | 16 x n bytes | Acknowledger's ancestry (for cache warming) |
| ... | dest_coords_count | 2 bytes LE | Number of initiator coordinate entries |
| ... | dest_coords | 16 x m bytes | Initiator's ancestry (for return-path cache warming) |
| ... | handshake_len | 2 bytes LE | Noise payload length |
| ... | handshake_payload | variable | Noise XK msg2 (57 bytes — ephemeral key + encrypted epoch) |
### SessionMsg3 (phase 0x3)
Third and final message of the Noise XK handshake. The initiator reveals
its encrypted static identity and epoch. After msg3, both parties derive
identical symmetric session keys and the session is established.
Encoded with FSP prefix: ver=0, phase=0x3, flags=0, payload_len.
![SessionMsg3](diagrams/session-msg3.svg)
**Body** (after 4-byte FSP prefix):
| Offset | Field | Size | Description |
| ------ | ----- | ---- | ----------- |
| 0 | flags | 1 byte | Reserved |
| 1 | handshake_len | 2 bytes LE | Noise payload length |
| 3 | handshake_payload | variable | Noise XK msg3 (73 bytes — encrypted static + encrypted epoch) |
**Noise XK msg3 breakdown** (73 bytes):
| Offset | Field | Size | Description |
| ------ | ----- | ---- | ----------- |
| 0 | encrypted_static | 49 bytes | Initiator's static key (33) + AEAD tag (16) |
| 49 | encrypted_epoch | 24 bytes | Startup epoch (8) + AEAD tag (16) |
SessionMsg3 does not carry coordinates — both endpoints already have each
other's coordinates from SessionSetup (msg1) and SessionAck (msg2).
### Data (0x10) — DataPacket Port Multiplexing
DataPacket is the primary application data carrier. The body after the
6-byte encrypted inner header contains a 4-byte port header followed by
the service payload:
| Offset | Field | Size | Description |
| ------ | ----- | ---- | ----------- |
| 0 | src_port | 2 bytes LE | Source service port |
| 2 | dst_port | 2 bytes LE | Destination service port |
| 4 | payload | variable | Service-specific payload |
The receiver dispatches by `dst_port` to the registered service handler.
**Port registry (three tiers):**
| Range | Purpose |
| ----- | ------- |
| 0–255 (0x00–0xFF) | Reserved, protocol use |
| 256–1023 (0x100–0x3FF) | Reserved, FIPS standard services |
| 1024–65535 (0x400–0xFFFF) | Application use |
**Initial assignment**: Port 256 (0x100) = IPv6 shim.
#### IPv6 Shim Payload Format (Port 256)
The IPv6 shim defines its own payload format with a leading format byte:
| Offset | Field | Size | Description |
| ------ | ----- | ---- | ----------- |
| 0 | format | 1 byte | Compression format (0x00 = mesh-internal compressed) |
| 1 | fields | variable | Format-specific residual fields |
**Format 0x00 — mesh-internal compressed (default):**
Strips source and destination IPv6 addresses (32 bytes) and payload length
(2 bytes) from each packet. Carries residual fields that cannot be derived
from session context:
| Offset | Field | Size | Description |
| ------ | ----- | ---- | ----------- |
| 0 | format | 1 byte | 0x00 |
| 1 | traffic_class | 1 byte | IPv6 Traffic Class (DSCP + ECN) |
| 2 | flow_label | 3 bytes | IPv6 Flow Label (20 bits, big-endian, zero-padded) |
| 5 | next_header | 1 byte | IPv6 Next Header (protocol identifier) |
| 6 | hop_limit | 1 byte | IPv6 Hop Limit |
| 7 | upper_payload | variable | Upper-layer payload (TCP, UDP, ICMPv6, etc.) |
The receiver reconstructs the full 40-byte IPv6 header from session context
(source and destination addresses derived from session npubs, version = 6,
payload length from outer packet length) plus the 6 bytes of residual fields,
then delivers the complete IPv6 packet to the TUN interface.
**Format 0x01+**: Reserved for future use (e.g., full-header gateway traffic).
**Compression savings**: 29 bytes per packet (34 bytes stripped, 7 bytes
format + residual added). Net overhead for IPv6 traffic: 77 bytes
(`FIPS_IPV6_OVERHEAD`), down from 110 bytes base DataPacket overhead.
### PathMtuNotification (0x13)
Sent by the destination to report the observed forward-path MTU.
![PathMtuNotification](diagrams/path-mtu-notification.svg)
| Offset | Field | Size | Encoding |
| ------ | ----- | ---- | -------- |
| 0 | path_mtu | 2 | u16 LE — minimum MTU observed along the forward path |
**Total body: 2 bytes** (plus FSP common prefix + encrypted header + AEAD tag).
### CoordsWarmup (0x14)
Standalone coordinate cache warming message. Sent when piggybacking coordinates
via the CP flag on a data packet would exceed the transport MTU, or as an
immediate response to CoordsRequired/PathBroken signals (rate-limited).
CoordsWarmup is an encrypted FSP message with the CP flag set and an empty
body. Transit nodes extract coordinates via the existing CP-flag parsing
path — no transit-side changes required.
**Wire format**:
```text
FSP header (12 bytes, AAD): ver=0, phase=0, flags=CP, counter, payload_len
Cleartext coords: src_coords + dst_coords (same encoding as CP flag)
AEAD ciphertext: inner_header(6) + Poly1305 tag(16) = 22 bytes
Total FSP payload: 12 + coords + 22
```
The cleartext coords section uses the same variable-length encoding as any
CP-flagged message: `src_coords_count(2) + src_coords(16×n) +
dest_coords_count(2) + dest_coords(16×m)`.
**Typical size** (depth-3 tree): 12 + (2+64+2+64) + 22 = **166 bytes** FSP
payload. With SessionDatagram + link overhead: 166 + 36 + 37 = **239 bytes**
on the wire.
### CoordsRequired (0x20)
Plaintext error signal — transit node lacks coordinates for destination.
Identified by the FSP phase field, not a separate message type byte
(same as SessionSetup/SessionAck/SessionMsg3).
Encoded with FSP prefix: ver=0, phase=0x0, U flag set, payload_len=34.
![CoordsRequired](diagrams/coords-required.svg)
**Payload** (after 4-byte FSP prefix):
| Offset | Field | Size | Description |
| ------ | ----- | ---- | ----------- |
| 0 | msg_type | 1 byte | 0x20 |
| 1 | flags | 1 byte | Reserved |
| 2 | dest_addr | 16 bytes | NodeAddr we couldn't route to |
| 18 | reporter | 16 bytes | NodeAddr of reporting router |
**Payload size**: 34 bytes. **Total on wire**: 38 bytes (4 prefix + 34 payload).
### PathBroken (0x21)
Plaintext error signal — greedy routing reached a dead end.
Encoded with FSP prefix: ver=0, phase=0x0, U flag set, payload_len.
![PathBroken](diagrams/path-broken.svg)
**Payload** (after 4-byte FSP prefix):
| Offset | Field | Size | Description |
| ------ | ----- | ---- | ----------- |
| 0 | msg_type | 1 byte | 0x21 |
| 1 | flags | 1 byte | Reserved |
| 2 | dest_addr | 16 bytes | Unreachable NodeAddr |
| 18 | reporter | 16 bytes | NodeAddr of reporting router |
| 34 | last_coords_count | 2 bytes LE | Number of stale coordinate entries |
| 36 | last_known_coords | 16 x n bytes | Stale coordinates that failed |
### MtuExceeded (0x22)
Plaintext error signal — forwarded packet exceeds the next-hop link MTU.
Sent by a transit router back to the source when a SessionDatagram cannot
be forwarded because its size exceeds the outgoing link's MTU.
Encoded with FSP prefix: ver=0, phase=0x0, U flag set, payload_len=36.
![MtuExceeded](diagrams/mtu-exceeded.svg)
**Payload** (after 4-byte FSP prefix):
| Offset | Field | Size | Description |
| ------ | ----- | ---- | ----------- |
| 0 | msg_type | 1 byte | 0x22 |
| 1 | flags | 1 byte | Reserved |
| 2 | dest_addr | 16 bytes | NodeAddr of the destination being forwarded to |
| 18 | reporter | 16 bytes | NodeAddr of the router that detected the MTU violation |
| 34 | mtu | 2 bytes LE | Bottleneck MTU at the reporting router |
**Payload size**: 36 bytes. **Total on wire**: 40 bytes (4 prefix + 36 payload).
The source uses the reported MTU to adjust its session-layer path MTU
estimate. MtuExceeded is the reactive complement to the proactive
`path_mtu` field in SessionDatagram and LookupResponse.
## Encapsulation Walkthrough
A complete picture of how application data is wrapped through each layer.
### Application Data -> Wire
Starting with an IPv6 application sending a 1024-byte TCP payload to a
destination (the original IPv6 packet at the TUN is 1064 bytes: 40-byte
header + 1024-byte payload):
```text
Layer 5: Application data
1024 bytes (TCP payload inside 1064-byte IPv6 packet)
Layer 4: IPv6 shim compression (port 256)
Strip IPv6 addresses (32) + payload length (2), keep residual fields
format (1) + residual (6) + upper payload (1024) = 1031 bytes
Layer 3: Session encryption (FSP)
FSP header (12) + AEAD(inner_hdr (6) + port_hdr (4) + shim (1031)) + tag (16)
= 12 + 1041 + 16 = 1069 bytes
Layer 2: SessionDatagram envelope (FMP routing)
msg_type (1) + ttl (1) + path_mtu (2) + src_addr (16) + dest_addr (16) + payload (1069)
= 1105 bytes
Layer 1: Link encryption (FMP per-hop)
outer header (16) + encrypted(inner_hdr (5) + datagram (1105)) + AEAD tag (16)
= 1142 bytes
Layer 0: Transport
UDP datagram containing 1142 bytes
```
Total overhead for IPv6 traffic: 1142 − 1064 = 78 bytes per packet. The
difference from the `FIPS_IPV6_OVERHEAD` constant (77 bytes) is the 1-byte
FMP `msg_type` counted in the link inner header rather than the
SessionDatagram body.
### Overhead Budget
| Layer | Overhead | Component |
| ----- | -------- | --------- |
| Link encryption | 37 bytes | 16 outer header (AAD) + 5 inner header (timestamp + msg_type) + 16 AEAD tag |
| SessionDatagram body | 35 bytes | 1 ttl + 2 path_mtu + 16 src + 16 dest (msg_type counted in inner header) |
| FSP header | 12 bytes | 4 prefix + 8 counter |
| FSP inner header | 6 bytes | 4 timestamp + 1 msg_type + 1 inner_flags (inside AEAD) |
| Session AEAD tag | 16 bytes | Poly1305 tag on session-encrypted payload |
| **Protocol envelope** | **106 bytes** | `FIPS_OVERHEAD` constant |
| Port header | 4 bytes | src_port + dst_port (DataPacket only) |
| **DataPacket total** | **110 bytes** | Base overhead for any port-multiplexed service |
| IPv6 compression | −33 bytes | 40-byte IPv6 header → 7-byte format + residual |
| **IPv6 data path total** | **77 bytes** | `FIPS_IPV6_OVERHEAD` constant |
### At Each Transit Node
```text
1. Receive UDP datagram
2. Parse common prefix -> version, phase, flags, payload_len
3. Phase 0x0 -> established frame
4. Look up (transport_id, receiver_idx) -> session
5. Check replay window (counter)
6. Decrypt with link keys (16-byte header as AAD) -> plaintext
7. Strip inner header -> timestamp, msg_type
8. msg_type 0x00 -> SessionDatagram
9. Read dest_addr -> routing decision
10. Decrement ttl, min path_mtu
11. Re-encrypt with next-hop link keys
12. Send via next-hop transport
```
Transit nodes see the SessionDatagram envelope (src_addr, dest_addr,
ttl, path_mtu) but cannot read the session-layer payload (encrypted with
endpoint session keys).
## Size Summary
### FMP Handshake Messages (Noise IK)
| Message | Raw Noise | Wire Frame |
| ------- | --------- | ---------- |
| IK msg1 (ephemeral + encrypted static + encrypted epoch) | 106 bytes | 114 bytes |
| IK msg2 (ephemeral + encrypted epoch) | 57 bytes | 69 bytes |
### FSP Handshake Messages (Noise XK)
| Message | Raw Noise | Notes |
| ------- | --------- | ----- |
| XK msg1 (ephemeral only) | 33 bytes | Carried in SessionSetup |
| XK msg2 (ephemeral + encrypted epoch) | 57 bytes | Carried in SessionAck |
| XK msg3 (encrypted static + encrypted epoch) | 73 bytes | Carried in SessionMsg3 |
### Link-Layer Messages (inside encrypted frame)
| Message | Size | Notes |
| ------- | ---- | ----- |
| TreeAnnounce | 100 + 32n bytes | n = depth + 1 |
| FilterAnnounce | 1,035 bytes | v1 (1KB filter) |
| LookupRequest | 303 + 16n bytes | n = origin depth + 1 |
| LookupResponse | 93 + 16n bytes | n = target depth + 1 |
| SessionDatagram | 36 + payload bytes | Fixed 36-byte header |
| Disconnect | 2 bytes | |
### Session-Layer Messages (inside SessionDatagram)
| Message | Typical Size | Notes |
| ------- | ------------ | ----- |
| SessionSetup | ~170 bytes | Depth-dependent (XK msg1 = 33 bytes) |
| SessionAck | ~190 bytes | Depth-dependent, carries both endpoints' coords (XK msg2 = 57 bytes) |
| SessionMsg3 | ~80 bytes | Fixed (XK msg3 = 73 bytes, no coords) |
| Data (minimal) | 12 + 6 + 4 + payload + 16 bytes | Steady state (port header included) |
| Data (with coords) | 12 + ~130 + 6 + 4 + payload + 16 bytes | Warmup/recovery (port header included) |
| SenderReport | 12 + 6 + 46 + 16 bytes | MMP metrics |
| ReceiverReport | 12 + 6 + 66 + 16 bytes | MMP metrics |
| PathMtuNotification | 12 + 6 + 2 + 16 bytes | MTU signal |
| CoordsWarmup | 12 + coords + 6 + 16 bytes | Standalone warmup (empty body) |
| CoordsRequired | 38 bytes | Fixed (prefix + msg_type + body) |
| PathBroken | 35 + 16n bytes | Includes stale coords |
| MtuExceeded | 40 bytes | Fixed (prefix + msg_type + body) |
### Complete Packet Sizes (link + session)
| Scenario | Wire Size | Notes |
| -------- | --------- | ----- |
| Encrypted frame minimum | 37 bytes | Empty body |
| SessionDatagram + Data (minimal) | 37 + 35 + 12 + 6 + 4 + payload + 16 | 110 + payload (any service) |
| SessionDatagram + IPv6 Data (minimal) | 110 + 7 + upper_payload − 34 | 77 + IPv6 payload (compressed) |
| SessionDatagram + Data (with coords) | 110 + coords + payload | Coords vary with tree depth |
| SessionDatagram + SessionSetup | ~275 bytes | Depth-3, both dirs |
| SessionDatagram + CoordsRequired | 37 + 36 + 38 = 111 bytes | Including link overhead |
## References
- [fips-mesh-layer.md](fips-mesh-layer.md) — FMP behavioral specification
- [fips-session-layer.md](fips-session-layer.md) — FSP behavioral specification
- [fips-transport-layer.md](fips-transport-layer.md) — Transport framing
- [fips-mesh-operation.md](fips-mesh-operation.md) — How messages work together
- [fips-ipv6-adapter.md](fips-ipv6-adapter.md) — MTU enforcement
docs/design/port-advertisement-and-nat-traversal.md
+594
View File
@@ -0,0 +1,594 @@
# Port Advertisement and NAT Traversal via Nostr
## Abstract
This document describes two related-but-independent mechanisms that an
application protocol can build on top of Nostr relays:
1. **Port advertisement.** A node publishes a parameterized replaceable
event describing the application protocol it speaks, the version, and
the endpoint(s) at which it can be reached. Other nodes discover the
advert by querying relays.
2. **NAT traversal.** When the advertised endpoint indicates that the
responder is behind NAT, the two peers exchange ephemeral
gift-wrapped offer/answer events through Nostr relays, run STUN
against a public server to learn their reflexive addresses, and
coordinate UDP hole punching so they can exchange application traffic
over a direct UDP path.
The two mechanisms compose naturally — an advert that includes a
`<protocol>:nat` endpoint signals "reach me by running the traversal
protocol" — but they are independently useful. An advert with only
public-IP endpoints needs no traversal. A pair of peers that already
know each other's pubkeys but want to coordinate a traversal can do so
without ever publishing a public advert.
The protocol described here is generic. Any application protocol can
adopt it by picking its own kind number, `d`-tag scope, and endpoint
schema. [FIPS](https://github.com/jmcorgan/fips) (the Free
Internetworking Peering System) is used throughout the document as an
example implementation; FIPS-specific values appear in clearly marked
example blocks and do not affect the generic protocol shape.
No WebRTC, DTLS, or ICE stack is required. The protocol operates at
the raw UDP level, using Nostr solely for ephemeral signaling and STUN
solely for reflexive address discovery.
---
## Terminology
- **Application protocol.** The protocol that runs on top of the
punched UDP channel after this document's procedures complete.
- **Initiator.** The peer that discovers the responder's advert and
begins the traversal exchange.
- **Responder.** The peer that publishes a service advertisement and
is willing to be dialled.
- **Reflexive address.** The public `IP:port` tuple that a STUN server
observes for a UDP socket — i.e., the NAT's external mapping for
that socket.
- **Punch socket.** The single UDP socket a peer uses for STUN, for
the offer/answer exchange's address fields, for the punch packets
themselves, and for the application traffic that follows. The same
socket must be used across all phases of one traversal attempt.
### Socket lifecycle
The protocol assumes **per-peer, per-attempt punch sockets**:
- Each outbound traversal attempt allocates a fresh UDP socket bound
to `0.0.0.0:0` (OS-assigned port).
- That socket is owned by exactly one remote peer and exactly one
traversal session.
- STUN, the offer/answer reflexive-address fields, the punch packets,
and the eventual adopted application transport all share that
socket for the lifetime of the attempt.
- If the attempt fails, the socket is discarded. A retry allocates a
new socket and obtains a fresh reflexive address.
- A long-lived application listener (for example, a fixed UDP port
shared across peers) must **not** be reused as the punch socket —
doing so couples NAT mappings and retry state across peers.
This rule is not optional: closing or rebinding the socket between
phases invalidates the NAT mapping that the rest of the protocol
depends on.
---
## Part 1: Service Advertisement
### Event shape
The advert is a NIP-01 parameterized replaceable event whose kind
falls in the application-defined replaceable range
`30000–39999`. The event carries:
- A `d` tag scoping the advert (so the same pubkey can publish
multiple distinct adverts under different scopes).
- A `protocol` tag carrying the application protocol's name, used as
a discovery filter for peers that don't already know the
responder's pubkey.
- A `version` tag carrying the application protocol version.
- An optional `expiration` tag (NIP-40) so a relay garbage-collects
the advert when the responder goes offline without explicitly
deleting it.
- An optional `relays` tag listing relays where the responder
subscribes for incoming signaling messages (used by Part 2).
- An optional `stun` tag listing STUN servers the responder
recommends.
- A `content` field carrying the application-specific payload —
typically the endpoint set, capability flags, and any encryption
keys the application layer needs. The content may be plaintext or
NIP-44-encrypted; encryption requires the consumer to already know
the responder's pubkey.
The replaceable semantics let the responder update the advert in
place under the same `d` tag. A NIP-09 deletion event removes the
advert when the responder permanently retires.
```json
{
"kind": <application-specific>,
"pubkey": "<responder_pubkey>",
"created_at": <unix_seconds>,
"tags": [
["d", "<application-defined-scope>"],
["protocol", "<application_protocol_name>"],
["version", "<protocol_version>"],
["relays", "wss://relay1.example.com", "wss://relay2.example.com"],
["stun", "stun.l.google.com:19302"],
["expiration", "<unix_seconds + ttl>"]
],
"content": "<application payload, optionally NIP-44 encrypted>",
"sig": "<signature>"
}
```
### Endpoint schema
The `content` field is application-defined. Its structure typically
includes a list of endpoints describing how the responder can be
reached. Endpoint entries should distinguish:
- **Direct public endpoints** (transport + address + port) where any
initiator can connect without traversal.
- **NAT-mapped endpoints** that signal "I can be reached by running
the traversal protocol against this transport on my pubkey."
- **Anonymity-network endpoints** (e.g. Tor onion services) where
the addressing scheme implies its own connection semantics.
#### FIPS example: kind 37195 advertisement
FIPS uses **kind `37195`** (the digits visually spell `FIPS` —
7=F, 1=I, 9=P, 5=S). The `d` tag is hardcoded to
`fips-overlay-v1`; the configurable `app` value populates the
separate `protocol` tag, scoping adverts within a relay set
without splitting them across multiple `d`-tag streams.
The advert content is a JSON document carrying a list of endpoint
entries, each shaped as `{transport, addr}`. The transport string
takes one of:
- `udp:host:port` — direct public UDP endpoint.
- `udp:nat` — NAT-mapped UDP endpoint; reach via Part 2 traversal.
- `tcp:host:port` — direct public TCP endpoint, for peers whose
networks filter outbound UDP. Public-only; there is no
`tcp:nat` analogue.
- `tor:<onion>:<port>` — Tor onion-service endpoint.
FIPS publishes the advert with `expiration` set to `now +
advert_ttl_secs` (default 1 hour) and refreshes it every
`advert_refresh_secs` (default 30 minutes).
### Public-IP discovery on advertisement
A responder behind a NAT or wildcard-bound to a non-routable address
needs to determine what external address to put in its advert. The
responder uses a fixed precedence:
1. An operator-supplied external address override (FIPS:
`transports.{udp,tcp}.external_addr`) wins.
2. A non-wildcard `local_addr` is used directly.
3. For a wildcard-bound UDP listener with an explicit "publish this"
flag (FIPS: `public: true`), the runtime queries STUN against
the configured servers and publishes the reflexive address.
4. For a wildcard-bound TCP listener, no STUN equivalent exists.
Implementations should refuse to silently advertise an unreachable
endpoint; FIPS emits a loud WARN and omits the endpoint.
This precedence keeps adverts honest: an endpoint that appears in
the published content is one the responder believes is reachable.
### Discovery (consumer side)
A consumer queries one or more relays for an advert it can act on.
Two filter shapes are typical:
By author, when the responder's pubkey is already known:
```json
["REQ", "<sub_id>", {
"kinds": [<advert_kind>],
"authors": ["<responder_pubkey>"],
"#d": ["<application-defined-scope>"]
}]
```
By application protocol, for "open discovery" of any peer running
the same application:
```json
["REQ", "<sub_id>", {
"kinds": [<advert_kind>],
"#protocol": ["<application_protocol_name>"]
}]
```
Adverts whose `protocol` tag does not match the consumer's expected
value, or whose `expiration` tag has elapsed, are rejected at
validation. Consumers cache adverts in memory keyed by author npub
and respect the embedded expiration.
#### FIPS example: discovery filters
The FIPS daemon issues both filter shapes: by-author for peers it
intends to dial directly, and by-`#protocol` when an operator has
opted into open discovery against the same application namespace.
Cached adverts persist until their `expiration` lapses; a periodic
prune drops expired entries.
---
## Part 2: NAT Traversal
The traversal protocol coordinates UDP hole punching between two
peers via gift-wrapped Nostr signaling. It is invoked when the
initiator decides to dial a NAT-mapped endpoint advertised by the
responder.
### Signaling event shape
Signaling messages are ephemeral kinds in the range `20000–29999`,
NIP-44-encrypted to the recipient, and NIP-59 gift-wrapped so the
outer event is signed by an ephemeral keypair rather than the
sender's long-term identity. The wrap carries a `p` tag pointing at
the recipient's pubkey and an NIP-40 `expiration` tag bounding how
long the relay should retain it.
#### FIPS example: signaling kind 21059
FIPS signaling uses **kind `21059`**. Wraps are addressed by `p`
tag and published to the responder's NIP-17 inbox relay list (kind
`10050`) when one is available, falling back to the local
`dm_relays` configuration otherwise. Each side publishes its own
inbox relay list on startup so dialers can discover it.
### Phase 1: Initiator STUN binding
Before constructing any signaling message, the initiator:
1. Allocates a fresh UDP punch socket bound to `0.0.0.0:0`.
2. Sends a STUN Binding Request (RFC 8489) to one of its locally
configured STUN servers.
3. Parses the Binding Response, extracts the
`XOR-MAPPED-ADDRESS` attribute, and records that as its
reflexive address. Other STUN attributes are ignored.
4. Records local-candidate addresses for the same socket port:
active private non-loopback interface addresses (RFC1918 IPv4,
IPv6 ULA) and probed local egress addresses.
The punch socket must remain open across all subsequent phases.
Closing or rebinding it discards the NAT mapping.
### Phase 2: Initiator sends offer
The initiator constructs an offer payload containing its reflexive
address, its local-candidate addresses, an opaque session
identifier, freshness timestamps, and any application-specific
parameters. The payload is NIP-44-encrypted to the responder's
pubkey, wrapped with NIP-59, and published to the responder's
signaling relays. The initiator also subscribes by `p` tag on
those relays to receive the answer.
```json
{
"type": "offer",
"sessionId": "<random_hex_32>",
"issuedAt": <unix_millis>,
"expiresAt": <unix_millis>,
"nonce": "<random_nonce>",
"senderNpub": "<initiator_npub>",
"recipientNpub": "<responder_npub>",
"reflexiveAddress": {"protocol":"udp","ip":"<ip>","port":<port>},
"localAddresses": [{"protocol":"udp","ip":"<ip>","port":<port>}],
"stunServer": "<host>:<port>",
"app_params": { ... }
}
```
- `sessionId` is a random identifier correlating offer and answer.
- `reflexiveAddress` is the address STUN observed in Phase 1.
- `localAddresses` enables a same-LAN fast path when both peers
happen to share a private subnet.
- `stunServer` is informational, recording which server the
initiator used.
- `issuedAt` / `expiresAt` bound the freshness window — the
responder rejects stale offers, since a NAT mapping that has not
been refreshed in tens of seconds may already be gone.
### Phase 3: Responder validates and answers
The responder maintains a standing `p`-tagged subscription on its
advertised signaling relays. On receiving an offer:
1. Decrypts the wrap and recovers the offer payload.
2. Validates freshness (rejects if outside the configured window;
see *Skew tolerance* below).
3. Rejects replays — if the `sessionId` is in a recently-seen
cache, drop the offer.
4. Allocates its own punch socket (`0.0.0.0:0`) and runs its own
STUN query.
5. Constructs an answer payload that echoes `sessionId`, carries
the responder's reflexive and local addresses, includes a
`PunchHint { startAtMs, intervalMs, durationMs }` telling both
sides when to begin probing and how aggressively, and is
wrapped, encrypted, and published the same way as the offer.
```json
{
"type": "answer",
"sessionId": "<same as offer>",
"issuedAt": <unix_millis>,
"expiresAt": <unix_millis>,
"nonce": "<random_nonce>",
"senderNpub": "<responder_npub>",
"recipientNpub": "<initiator_npub>",
"inReplyTo": "<offer_event_id>",
"accepted": true,
"reflexiveAddress": {"protocol":"udp","ip":"<ip>","port":<port>},
"localAddresses": [{"protocol":"udp","ip":"<ip>","port":<port>}],
"stunServer": "<host>:<port>",
"punch": {"startAtMs": <ms>, "intervalMs": <ms>, "durationMs": <ms>},
"offerReceivedAt": <unix_millis>,
"app_params": { ... }
}
```
If the responder has no usable addresses, it returns
`accepted: false` with an explanatory `reason` and no `punch`.
The optional `offerReceivedAt` field carries the responder's
wall-clock at the moment the offer arrived. The initiator can
combine its own `T1` (offer-publish time), `T2 = offerReceivedAt`,
`T3` (answer's `issuedAt`), and `T4` (answer-receive time) into the
NTP-style estimate `((T2 − T1) + (T3 − T4)) / 2`, giving a per-peer
clock-skew measurement that's useful for tuning freshness windows
and for telemetry.
**Immediately after publishing the answer**, the responder begins
Phase 4 punching without waiting for any acknowledgement that the
initiator received the answer. NAT mappings are decaying and time
is the binding constraint.
The responder must bind the inner JSON `senderNpub` /
`recipientNpub` fields to the actual Nostr pubkeys that delivered
the gift wrap, rather than treating those JSON fields as
independently trustworthy. The wrap pubkey is the authentication
ground-truth.
### Phase 4: Hole punching
Both peers now know each other's reflexive and local addresses.
Both begin sending UDP packets from their respective punch sockets:
1. Send punch packets every **`intervalMs`** (typically 200 ms)
across each planned target path:
- reflexive-to-reflexive
- private-subnet local-address paths (when subnet-compatible)
- mixed local/reflexive fallbacks
2. Each punch packet carries a fixed magic header so transit and
peer code can distinguish it from stray UDP traffic:
```text
Bytes 0–3: <PROBE_MAGIC> (application-defined u32)
Bytes 4–7: sequence number (u32, big-endian, starting at 0)
Bytes 8–23: first 16 bytes of SHA-256(sessionId)
```
3. On receiving a valid punch packet (magic matches, session-id
hash matches), the peer records the source address as the
confirmed peer address and replies with an acknowledgement
packet under a different magic value:
```text
Bytes 0–3: <ACK_MAGIC> (application-defined u32)
Bytes 4–7: echoed sequence number
Bytes 8–23: first 16 bytes of SHA-256(sessionId)
```
4. On receiving an acknowledgement, the peer considers the path
punched and transitions to Phase 5.
If both peers advertised compatible local-subnet candidates, the
local-address path will typically punch through faster than the
reflexive path. The first path to acknowledge wins.
### Phase 5: Application protocol takeover
Once the path has acknowledged in both directions:
- The application protocol takes over the punch socket.
- The signaling subscription can be closed.
- The application is responsible for sending keepalive traffic at
least every 15 seconds to refresh the NAT mapping. A flow that
goes idle longer risks losing its mapping and having to retraverse.
### Phase 6: Cleanup
After the attempt completes (success or failure):
1. Close the relay subscription used for signaling.
2. Optionally publish a NIP-09 deletion event referencing any
signaling events the peer published. Because the wraps were
ephemeral kinds with NIP-40 expiration tags, well-behaved relays
will discard them automatically without explicit deletion.
3. Discard the per-attempt punch socket if the attempt failed; a
retry must allocate a new socket and a fresh reflexive address.
If the responder is going offline permanently it should also
delete its kind-37195 (or equivalent) advert.
### Timeouts and retries
- If the initiator publishes an offer and receives no answer
within a configured window (e.g. 10 s from offer publish), the
attempt has failed. Causes: responder offline, advert stale,
responder relay unreachable.
- If the answer arrives but no valid punch acknowledgement is
observed within `durationMs` (typically 10 s), the attempt has
failed. Causes: symmetric NAT on either side, firewall
interference, stale reflexive addresses.
The initiator may retry with a fresh STUN query, a fresh punch
socket, and a new offer. Repeated failures against the same
responder should be suppressed by the application layer; see
*Application-specific failure handling* below.
---
## Security
### Authentication
Offer and answer payloads are NIP-44-encrypted to the recipient and
NIP-59 gift-wrapped, so only the intended recipient can decrypt.
Authentication of the sender comes from the inner-wrap signature
(the rumour signed by the sender's long-term identity inside the
NIP-59 seal), **not** from the outer wrap signature (which is the
ephemeral pubkey).
The inner JSON `senderNpub` / `recipientNpub` fields must be bound
to the actual signing pubkey of the inner rumour. Treating those
JSON fields as independently trustworthy is a vulnerability —
implementations must compare them against the unwrapped signature.
Once the UDP path is punched, the raw UDP channel has **no inherent
authentication or encryption**. The application layer is responsible
for establishing its own security on the punched channel — for
example, a Noise Protocol handshake keyed from the Nostr identity,
or an application-specific authenticated-encryption layer. FIPS
runs its FMP Noise IK handshake immediately after adoption; the
identity proven by the Noise handshake is the same Nostr pubkey
that signed the inner offer/answer rumour, so a man-in-the-middle on
the relay cannot impersonate the responder.
### Replay protection
The `sessionId` and `issuedAt` / `expiresAt` fields together
defeat replays at the signaling layer. The responder must keep a
bounded cache of recently-seen `sessionId` values and reject
duplicates within the freshness window.
### Skew tolerance
Strict freshness checks fail under modest clock skew between
peers. Implementations should accept offers and answers whose
timestamps are off by a small absolute amount (FIPS uses ±60 s),
and feed observed skew into a per-peer estimate for telemetry and
tuning. Outright rejection should be reserved for grossly stale or
future-dated messages.
### Metadata exposure
Even though signaling content is encrypted, the gift-wrap metadata
reveals that the initiator's ephemeral pubkey contacted the
responder's pubkey at a particular time, through a particular
relay. The advert itself is public and reveals the responder's
pubkey and the application protocol it speaks.
If metadata privacy is required, the advert content can be
encrypted (consumers must already know the responder's pubkey),
both peers can use ephemeral Nostr identities rather than their
long-term keys, and the operator can run a private relay.
### NAT mapping integrity
If too much wall-clock time elapses between STUN discovery and the
hole-punch attempt, the reflexive address goes stale. Both peers
should complete the entire signaling exchange within tens of
seconds of their respective STUN queries. Relay latency is the
primary risk factor. Implementations targeting flaky relays should
prefer relays known to deliver ephemeral events sub-second.
---
## Relay requirements
The protocol works best with relays that:
- Support ephemeral event kinds (`20000–29999`) and do not persist
them.
- Honor NIP-40 `expiration` tags and garbage-collect expired
events.
- Deliver events with low latency (sub-second WebSocket push).
- Support NIP-09 deletion requests.
Relays that do not support ephemeral kinds will store the
signaling events as regular events. The encrypted content remains
opaque, but persisted wraps are wasteful and expose metadata
unnecessarily. Operators deploying this protocol at scale should
prefer relays that handle ephemeral kinds correctly, or run their
own.
---
## Failure modes
| Failure | Symptom | Mitigation |
| --- | --- | --- |
| Symmetric NAT (one side) | Punch timeout | Retry with port-prediction heuristics; otherwise fall back to a relay or different transport |
| Symmetric NAT (both sides) | Punch timeout | Application-level relay required |
| Relay latency > 60 s | Stale reflexive address | Use low-latency relays; consider self-hosted relay |
| Relay does not support ephemeral kinds | Signaling events persist | Use NIP-40 expiration + NIP-09 deletion as fallback |
| Responder offline | No answer received | Initiator times out after configurable period |
| Stale advert (responder no longer up) | Offer reaches no listener | Application-level failure suppression (see below) |
| STUN server unreachable | No reflexive address | Fall back to alternate STUN server; fail if none reachable |
| Firewall blocks outbound UDP | STUN fails entirely | NAT-traversal does not apply; reachable peers are limited to those that publish a non-UDP transport (e.g. TCP) and accept inbound |
### Application-specific failure handling
Repeated traversal failures against the same responder are common
in practice — the responder may be offline, the advert may be
stale, or the responder may be on a network that doesn't admit
incoming UDP. A naive implementation that retries on every dial
attempt floods the relay layer and the operator's logs.
Implementations should layer per-peer suppression on top of the
basic retry. The shape of that suppression is application-specific.
#### FIPS example: failure suppression
FIPS layers the following suppression machinery on the basic retry
loop:
- **Per-npub WARN log rate-limit** (`warn_log_interval_secs`,
default 5 minutes). Subsequent failures inside the window log
at debug level instead.
- **Per-npub consecutive-failure counter and extended cooldown.**
After `failure_streak_threshold` (default 5) consecutive
failures, the per-peer retry deadline is pushed past
`extended_cooldown_secs` (default 30 minutes). Open-discovery
sweeps consult the cooldown so they don't immediately re-enqueue
the same peer.
- **Stale-advert eviction on streak transition.** When a peer
hits the failure-streak threshold, the daemon actively
re-fetches its advert from the configured advert relays. If the
advert has been removed or replaced, the cache entry is evicted
and the streak resets; if the advert is unchanged, the cooldown
applies.
- **Per-peer skew estimate.** The NTP-style skew computed from
`offerReceivedAt` is recorded so consistently-skewed peers don't
trip the freshness check on every attempt.
- **Bounded failure-state cache** (`failure_state_max_entries`,
default 4096) with LRU eviction so the suppression machinery
itself does not grow unbounded.
These knobs are documented in
[FIPS configuration reference](https://github.com/jmcorgan/fips/blob/master/docs/reference/configuration.md)
under `node.discovery.nostr`.
---
## References
- **RFC 8489** — Session Traversal Utilities for NAT (STUN)
- **RFC 8445** — Interactive Connectivity Establishment (ICE)
- **RFC 4787** — NAT Behavioral Requirements for Unicast UDP
- **NIP-01** — Basic Nostr protocol flow
- **NIP-09** — Event deletion request
- **NIP-17** — Inbox relay list (kind `10050`) for direct-message
routing
- **NIP-40** — Expiration timestamp
- **NIP-44** — Versioned encryption
- **NIP-59** — Gift wrap
- **NIP-78** — Application-specific data
docs/design/spanning-tree-dynamics.md
+55 -109
View File
@@ -1,14 +1,20 @@
# FIPS Spanning Tree Protocol Dynamics
A detailed study of the gossip-based spanning tree protocol, focusing on
operational behavior under various mesh conditions. This document complements
[fips-intro.md](fips-intro.md) with step-by-step walkthroughs of protocol
dynamics rather than message formats and data structures.
A detailed study of the gossip-based spanning tree protocol, focusing
on operational behavior under various mesh conditions. This document
complements [fips-concepts.md](fips-concepts.md) and
[fips-architecture.md](fips-architecture.md) with step-by-step
walkthroughs of protocol dynamics rather than message formats and
data structures.
For wire formats, see [fips-wire-formats.md](fips-wire-formats.md) (TreeAnnounce section).
For spanning tree algorithms and data structures, see
[fips-spanning-tree.md](fips-spanning-tree.md). For how the spanning tree fits
into mesh routing, see [fips-mesh-operation.md](fips-mesh-operation.md).
For wire formats, see
[../reference/wire-formats.md](../reference/wire-formats.md)
(TreeAnnounce section). For spanning tree algorithms and data
structures, see [fips-spanning-tree.md](fips-spanning-tree.md). For
how the spanning tree fits into mesh routing, see
[fips-mesh-operation.md](fips-mesh-operation.md). For the academic
foundations and references that underpin this document, see
[fips-prior-work.md](fips-prior-work.md).
## Contents
@@ -93,7 +99,7 @@ When a node starts with no peers, it bootstraps as a single-node network.
**T0: Node A starts.**
- Generates or loads keypair `(npub_A, nsec_A)`
- Computes `node_addr_A = SHA-256(npub_A)`
- Computes `node_addr_A = SHA-256(pubkey_A)[..16]` (128 bits)
- Initializes empty TreeState
- Sets `parent = self` (A is its own root), `sequence = 1`
- Records current timestamp
@@ -518,6 +524,11 @@ converge to the same "link failed" state, though B detects it up to
## 8. Parent Selection
Parent selection determines tree structure and routing efficiency.
The algorithm itself (effective-depth ranking, hold-down, hysteresis,
mandatory-switch bypass) is canonically documented in
[fips-spanning-tree.md](fips-spanning-tree.md); this section walks
through what re-selection looks like under specific dynamic
conditions and the rationale for the local-only cost metric.
### Cost-Based Selection with Effective Depth
@@ -537,10 +548,15 @@ purely by tree depth without link quality consideration.
2. **Compute effective depth for each candidate.** For every peer whose
announced root matches the smallest root, the algorithm calculates
`effective_depth = peer.depth + link_cost`, where `link_cost` comes from
`peer_costs` (MMP-derived) or defaults to 1.0 when metrics have not yet
converged. The best candidate is the peer with the lowest effective depth,
with ties broken by numerically smallest `NodeAddr`. If the best candidate
is already the current parent, no switch is needed.
`peer_costs` (MMP-derived). During cold start, when no peer has MMP data
yet (`peer_costs` is empty), unmeasured candidates default to 1.0; once
any peer has MMP data, unmeasured candidates are skipped so a freshly
connected peer cannot win on its default cost. Candidates whose ancestry
already contains the local node are also rejected, preventing an
alternating two-node loop. The best candidate is the peer with the
lowest effective depth, with ties broken by numerically smallest
`NodeAddr`. If the best candidate is already the current parent, no
switch is needed.
3. **Check for mandatory switches.** Two conditions bypass all stability
mechanisms and trigger an immediate parent change: the current parent is no
@@ -571,10 +587,10 @@ re-evaluation independent of TreeAnnounce traffic).
Where ETX (Expected Transmission Count, from De Couto et al., "A
High-Throughput Path Metric for Multi-Hop Wireless Routing", 2003) comes from
bidirectional MMP delivery ratios and SRTT (Smoothed Round-Trip Time) from MMP
timestamp-echo. When MMP
metrics have not yet converged, `link_cost` defaults to 1.0, preserving
depth-only behavior as a graceful fallback.
bidirectional MMP delivery ratios and SRTT (Smoothed Round-Trip Time) from
MMP timestamp-echo. During cold start, before any peer has MMP data, the
default cost of 1.0 is used and the algorithm reduces to depth-only
selection.
**What this means for tree structure**: The algorithm can prefer a deeper parent
with a better link over a shallower parent with a poor link, when the effective
@@ -942,28 +958,16 @@ costs to form efficient tree structures.
### Prior Art and FIPS Contributions
The protocol builds on established foundations and adds several new elements:
**Derived from prior work**:
- Spanning tree coordinate routing (Yggdrasil/Ironwood, building on Kleinberg
2007 and Cvetkovski/Crovella 2009)
- Deterministic root discovery via smallest identifier (Yggdrasil; echoes
IEEE 802.1D STP bridge ID selection)
- CRDT-based distributed state (Shapiro et al. 2011)
- Gossip dissemination (epidemic model; Kermarrec 2007)
- Heartbeat-based failure detection (SWIM; Das et al. 2002)
- ETX link metric (De Couto et al. 2003)
- Hysteresis and hold-down for route stability (OSPF, BGP, IS-IS)
**FIPS additions**:
- Cost-aware parent selection using local-only link metrics (effective depth =
tree depth + link cost), replacing Yggdrasil's depth-only selection
- Combined ETX + SRTT link cost formula with MMP-measured components
- Flap dampening with mandatory switch bypass
- Announcement suppression for transient state changes
- Tree-only bloom filter merge with split-horizon exclusion
The protocol builds on established foundations (Yggdrasil/Ironwood
tree-coordinate routing, IEEE 802.1D STP root election, CRDT-based
distributed state, SWIM-style failure detection, ETX, OSPF-style
hysteresis and hold-down) and adds several new elements (cost-aware
parent selection on local-only metrics, the combined ETX + SRTT cost
formula, flap dampening with mandatory-switch bypass, announcement
suppression, and tree-only bloom filter merge with split-horizon).
Both the prior-art map and the FIPS contributions list are
consolidated in
[fips-prior-work.md](fips-prior-work.md#fips-contributions).
---
@@ -971,75 +975,17 @@ The protocol builds on established foundations and adds several new elements:
### FIPS Internal Documentation
- [fips-spanning-tree.md](fips-spanning-tree.md) — Spanning tree algorithms and data structures
- [fips-mesh-operation.md](fips-mesh-operation.md) — How the spanning tree fits into mesh routing
- [fips-wire-formats.md](fips-wire-formats.md) — TreeAnnounce wire format
- [fips-spanning-tree.md](fips-spanning-tree.md) — Spanning tree
algorithms and data structures
- [fips-mesh-operation.md](fips-mesh-operation.md) — How the spanning
tree fits into mesh routing
- [../reference/wire-formats.md](../reference/wire-formats.md) —
TreeAnnounce wire format
### Yggdrasil Documentation
### Prior Art and Academic Foundations
- [Yggdrasil v0.5 Release Notes](https://yggdrasil-network.github.io/2023/10/22/upcoming-v05-release.html)
- [Ironwood Routing Library](https://github.com/Arceliar/ironwood)
- [The World Tree (Yggdrasil Blog)](https://yggdrasil-network.github.io/2018/07/17/world-tree.html)
- [Yggdrasil Implementation Overview](https://yggdrasil-network.github.io/implementation.html)
### Academic Foundations
#### Virtual Coordinate Routing
- Rao, A., Ratnasamy, S., Papadimitriou, C., Shenker, S., Stoica, I.
["Geographic Routing without Location Information"](https://people.eecs.berkeley.edu/~sylvia/papers/p327-rao.pdf).
MobiCom 2003. *Established virtual coordinate routing using network topology.*
#### Greedy Embedding Theory
- Kleinberg, R.
["Geographic Routing Using Hyperbolic Space"](https://www.semanticscholar.org/paper/Geographic-Routing-Using-Hyperbolic-Space-Kleinberg/f506b2ddb142d2ec539400297ba53383d958abef).
IEEE INFOCOM 2007. *Proved every connected graph has a greedy embedding in
hyperbolic space; showed spanning trees enable coordinate assignment.*
- Cvetkovski, A., Crovella, M.
["Hyperbolic Embedding and Routing for Dynamic Graphs"](https://www.cs.bu.edu/faculty/crovella/paper-archive/infocom09-hyperbolic.pdf).
IEEE INFOCOM 2009. *Dynamic embedding for nodes joining/leaving; introduced
Gravity-Pressure routing for failure recovery.*
- Crovella, M. et al.
["On the Choice of a Spanning Tree for Greedy Embedding"](https://www.cs.bu.edu/faculty/crovella/paper-archive/networking-science13.pdf).
Networking Science 2013. *Analysis of how tree structure affects routing stretch.*
- Bläsius, T. et al.
["Hyperbolic Embeddings for Near-Optimal Greedy Routing"](https://dl.acm.org/doi/10.1145/3381751).
ACM Journal of Experimental Algorithmics 2020. *Achieved 100% success ratio
with 6% stretch on Internet graph.*
#### Link Metrics
- De Couto, D., Aguayo, D., Bicket, J., Morris, R.
"A High-Throughput Path Metric for Multi-Hop Wireless Routing".
MobiCom 2003. *Introduced ETX (Expected Transmission Count) as a link
quality metric for wireless mesh networks.*
#### Routing Protocol Stability
- IEEE 802.1D. "IEEE Standard for Local and Metropolitan Area
Networks: Media Access Control (MAC) Bridges". *Spanning Tree
Protocol (STP) — root election via bridge ID, BPDU exchange.*
- Moy, J. [RFC 2328](https://datatracker.ietf.org/doc/html/rfc2328):
"OSPF Version 2". 1998. *Link-state routing with cumulative path
costs and SPF computation. FIPS's local-only cost approach is
contrasted with OSPF's cumulative model in §8.*
#### Distributed Systems Primitives
- Shapiro, M., Preguiça, N., Baquero, C., Zawirski, M.
"Conflict-free Replicated Data Types". SSS 2011.
*Formal definition of CRDTs enabling coordination-free consistency.*
- Das, A., Gupta, I., Motivala, A.
["SWIM: Scalable Weakly-consistent Infection-style Process Group Membership"](https://www.cs.cornell.edu/projects/Quicksilver/public_pdfs/SWIM.pdf).
IPDPS 2002. *O(1) failure detection, O(log N) dissemination via gossip.*
- Kermarrec, A-M.
["Gossiping in Distributed Systems"](https://www.distributed-systems.net/my-data/papers/2007.osr.pdf).
ACM SIGOPS Operating Systems Review 2007. *Framework for gossip-based
protocols achieving O(log N) propagation.*
The Yggdrasil documentation and the academic-foundations bibliography
(virtual coordinate routing, greedy embedding theory, link metrics,
routing-protocol stability, and distributed systems primitives) are
collected in
[fips-prior-work.md](fips-prior-work.md#spanning-tree-dynamics-foundations).