From d46dc874ef4fb31f005ec8c1ad8b5e5d49783844 Mon Sep 17 00:00:00 2001 From: Johnathan Corgan Date: Tue, 17 Feb 2026 04:50:04 +0000 Subject: [PATCH] Restructure design docs around protocol layers MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Reorganize FIPS design documentation from implementation-centric structure (routing, gossip protocol, wire protocol, transports) to protocol-layer organization with clear service boundaries. New documents (8): - fips-transport-layer.md — transport layer spec - fips-link-layer.md — FLP spec (peer auth, link encryption, forwarding) - fips-session-layer.md — FSP spec (end-to-end encryption, sessions) - fips-ipv6-adapter.md — IPv6 adaptation (TUN, DNS, MTU enforcement) - fips-mesh-operation.md — routing, discovery, error recovery - fips-wire-formats.md — consolidated wire format reference - fips-spanning-tree.md — tree algorithm reference - fips-bloom-filters.md — bloom filter math reference Rewritten (2): - fips-intro.md — breadth-first intro with layer model diagrams - fips-software-architecture.md — slimmed to stable decisions Updated (3): - spanning-tree-dynamics.md — removed stale root refresh, aligned terminology - fips-configuration.md — fixed priority type (u16 → u8) - fips-state-machines.md — synced code examples with codebase Deleted (6): fips-transports.md, fips-wire-protocol.md, fips-gossip-protocol.md, fips-session-protocol.md, fips-routing.md, fips-tun-driver.md (content absorbed into new structure) --- docs/design/README.md | 100 +- docs/design/fips-bloom-filters.md | 249 +++++ docs/design/fips-configuration.md | 2 +- docs/design/fips-gossip-protocol.md | 794 ------------- docs/design/fips-intro.md | 708 ++++++------ docs/design/fips-ipv6-adapter.md | 325 ++++++ docs/design/fips-link-layer.md | 379 +++++++ docs/design/fips-mesh-operation.md | 570 ++++++++++ docs/design/fips-routing.md | 903 --------------- docs/design/fips-session-layer.md | 402 +++++++ docs/design/fips-session-protocol.md | 848 -------------- docs/design/fips-software-architecture.md | 1240 ++++----------------- docs/design/fips-spanning-tree.md | 258 +++++ docs/design/fips-state-machines.md | 305 +++-- docs/design/fips-transport-layer.md | 382 +++++++ docs/design/fips-transports.md | 190 ---- docs/design/fips-tun-driver.md | 236 ---- docs/design/fips-wire-formats.md | 534 +++++++++ docs/design/fips-wire-protocol.md | 1139 ------------------- docs/design/spanning-tree-dynamics.md | 120 +- 20 files changed, 3925 insertions(+), 5759 deletions(-) create mode 100644 docs/design/fips-bloom-filters.md delete mode 100644 docs/design/fips-gossip-protocol.md create mode 100644 docs/design/fips-ipv6-adapter.md create mode 100644 docs/design/fips-link-layer.md create mode 100644 docs/design/fips-mesh-operation.md delete mode 100644 docs/design/fips-routing.md create mode 100644 docs/design/fips-session-layer.md delete mode 100644 docs/design/fips-session-protocol.md create mode 100644 docs/design/fips-spanning-tree.md create mode 100644 docs/design/fips-transport-layer.md delete mode 100644 docs/design/fips-transports.md delete mode 100644 docs/design/fips-tun-driver.md create mode 100644 docs/design/fips-wire-formats.md delete mode 100644 docs/design/fips-wire-protocol.md diff --git a/docs/design/README.md b/docs/design/README.md index 8c09960..d3a3113 100644 --- a/docs/design/README.md +++ b/docs/design/README.md @@ -1,60 +1,76 @@ # FIPS Design Documents -Protocol design specifications and analysis for the Federated Interoperable Peering System. +Protocol design specifications for the Federated Interoperable Peering +System — a self-organizing encrypted mesh network built on Nostr identities. -## Suggested Reading Order +## Reading Order -Start with the high-level architecture, then work through session flow, routing -concepts, and finally the wire-level protocol details. +Start with the introduction, then follow the protocol stack from bottom to +top. After the stack, the mesh operation document explains how all the +pieces work together. Supporting references provide deeper dives into +specific topics. -### 1. Introduction and Overview +### Protocol Stack -| Document | Description | -|------------------------------------------|-------------------------------------------------------------| -| [fips-intro.md](fips-intro.md) | Protocol introduction: goals, concepts, architecture | +| Document | Description | +| -------- | ----------- | +| [fips-intro.md](fips-intro.md) | Protocol introduction: goals, architecture, layer model | +| [fips-transport-layer.md](fips-transport-layer.md) | Transport layer: datagram delivery over arbitrary media | +| [fips-link-layer.md](fips-link-layer.md) | FIPS Link Protocol (FLP): peer authentication, link encryption, forwarding | +| [fips-session-layer.md](fips-session-layer.md) | FIPS Session Protocol (FSP): end-to-end encryption, sessions | +| [fips-ipv6-adapter.md](fips-ipv6-adapter.md) | IPv6 adaptation: TUN interface, DNS, MTU enforcement | -### 2. Protocol Flow (How Traffic Works) +### Mesh Behavior -| Document | Description | -|------------------------------------------------------|-----------------------------------------------------------| -| [fips-session-protocol.md](fips-session-protocol.md) | End-to-end session flow, Noise IK encryption, terminology | +| Document | Description | +| -------- | ----------- | +| [fips-mesh-operation.md](fips-mesh-operation.md) | How the mesh operates: routing, discovery, error recovery | +| [fips-wire-formats.md](fips-wire-formats.md) | Wire format reference for all message types | -### 3. Routing (How Packets Find Their Way) +### Supporting References -| Document | Description | -|--------------------------------------------------------|-------------------------------------------------------------| -| [fips-routing.md](fips-routing.md) | Routing concepts: bloom filters, discovery, greedy routing | -| [spanning-tree-dynamics.md](spanning-tree-dynamics.md) | Tree protocol behavior: convergence, partitions, recovery | -| [fips-gossip-protocol.md](fips-gossip-protocol.md) | Wire formats: TreeAnnounce, FilterAnnounce, Lookup messages | +| Document | Description | +| -------- | ----------- | +| [fips-spanning-tree.md](fips-spanning-tree.md) | Spanning tree algorithms: root election, parent selection, coordinates | +| [fips-bloom-filters.md](fips-bloom-filters.md) | Bloom filter math: FPR analysis, size classes, split-horizon | -### 4. Link Layer (How Peer Connections Work) +### Implementation -| Document | Description | -|------------------------------------------------|---------------------------------------------------------------| -| [fips-wire-protocol.md](fips-wire-protocol.md) | Transport layer: Noise IK, session indices, roaming, security | -| [fips-transports.md](fips-transports.md) | Transport-specific: UDP, Ethernet, Tor, radio characteristics | +| Document | Description | +| -------- | ----------- | +| [fips-software-architecture.md](fips-software-architecture.md) | Stable architectural decisions guiding the codebase | +| [fips-state-machines.md](fips-state-machines.md) | Phase-based state machine pattern (Rust enum-of-structs) | +| [fips-configuration.md](fips-configuration.md) | YAML configuration reference | -## Implementation +### Supplemental -| Document | Description | -|------------------------------------------------------------------|------------------------------------------------------------------| -| [fips-software-architecture.md](fips-software-architecture.md) | Software architecture: entities, state machines, configuration | -| [fips-tun-driver.md](fips-tun-driver.md) | TUN interface driver: reader/writer threads, ICMPv6, packet flow | -| [fips-state-machines.md](fips-state-machines.md) | Phase-based state machine pattern: peer lifecycle, transitions | +| Document | Description | +| -------- | ----------- | +| [spanning-tree-dynamics.md](spanning-tree-dynamics.md) | Spanning tree walkthroughs: convergence scenarios, worked examples | -## Document Cross-References +## Document Relationships ```text - fips-intro.md - │ - ┌────────────┴────────────┐ - ▼ ▼ - fips-session-protocol.md fips-software-architecture.md - │ │ - ┌─────────┴─────────┐ ▼ - ▼ ▼ fips-transports.md -fips-routing.md fips-wire-protocol.md - │ │ - ▼ ▼ -spanning-tree-dynamics.md ←→ fips-gossip-protocol.md + fips-intro.md + │ + ┌──────────────┼──────────────┐ + ▼ ▼ ▼ + fips-transport-layer fips-mesh- fips-software- + │ operation architecture + ▼ │ │ + fips-link-layer ◄────────┤ ▼ + │ │ fips-state-machines + ▼ │ + fips-session-layer ├──► fips-spanning-tree + │ │ │ + ▼ └──► fips-bloom-filters + fips-ipv6-adapter + fips-wire-formats + (referenced by all layer docs) + + fips-configuration + (standalone reference) + + spanning-tree-dynamics + (pedagogical companion to fips-spanning-tree) ``` diff --git a/docs/design/fips-bloom-filters.md b/docs/design/fips-bloom-filters.md new file mode 100644 index 0000000..391af87 --- /dev/null +++ b/docs/design/fips-bloom-filters.md @@ -0,0 +1,249 @@ +# FIPS Bloom Filters + +This document describes the bloom filter data structures, parameters, and +mathematical properties used by FIPS for reachability-based candidate +selection. It is a supporting reference — for how bloom filters fit into +the overall routing system, see +[fips-mesh-operation.md](fips-mesh-operation.md). + +## Purpose + +Each node maintains bloom filters summarizing which destinations are +reachable through each of its peers. Bloom filters provide **candidate +selection**: they narrow which peers are worth considering when forwarding +a packet to a given destination. The actual forwarding decision is made by +tree coordinate distance ranking. + +Bloom filters answer a single question: "can peer P possibly reach +destination D?" The answer is either "no" (definitive) or "maybe" +(probabilistic — false positives are possible, false negatives are not). + +## Filter Parameters + +| Parameter | Value | Rationale | +| --------- | ----- | --------- | +| Size | 1 KB (8,192 bits) | Balance between accuracy and bandwidth | +| Hash functions (k) | 5 | Compromise between optimal k=7 for 800 entries and accommodating up to ~1,600 entries | +| Size class | 1 (v1 mandated) | `512 << 1 = 1024` bytes | + +### Why k = 5 + +The optimal number of hash functions depends on the expected occupancy: + +- At 800 entries (typical for moderate-degree nodes), optimal k ≈ 7 +- At 1,600 entries (hub nodes), optimal k ≈ 4 + +k = 5 is a practical compromise that provides reasonable false positive +rates across the expected range of occupancy. + +## False Positive Rate (FPR) Analysis + +The false positive rate for a bloom filter with m bits, k hash functions, +and n entries is approximately: + +```text +FPR ≈ (1 - e^(-kn/m))^k +``` + +With m = 8,192 bits and k = 5: + +| Node Degree | Expected Entries | FPR | Impact | +| ----------- | ---------------- | --- | ------ | +| 5 (IoT) | 100–200 | ~0.02% | Negligible | +| 8 (typical) | 250–400 | ~0.3% | Negligible | +| 12 (well-connected) | 500–800 | ~2.4% | Minor — occasional unnecessary discovery | +| 20+ (hub) | 1,200–1,800 | 7.5–15% | Elevated — more false positive candidates | + +At moderate network sizes, filters are highly accurate. At larger scales +(~1M nodes), hub nodes with many peers see elevated FPR. False positives +cause unnecessary candidate evaluation (and potentially unnecessary +discovery attempts) but do not affect routing correctness — the tree +distance calculation makes the actual forwarding decision. + +### Saturation Behavior + +A bloom filter with 8,192 bits and k = 5 saturates (FPR approaches 100%) +around 3,000–4,000 entries. Beyond this, every query returns "maybe" and the +filter provides no candidate selection value. Hub nodes approaching this +threshold effectively fall back to tree-distance-only routing. + +## Per-Peer Filter Model + +Each node maintains a separate outbound filter for each peer. The filter +for peer Q answers: "which destinations are reachable through me (but not +through Q itself)?" + +### Filter Computation + +The outbound filter for peer Q is computed by merging: + +1. **This node's own identity** (node_addr) +2. **Leaf-only dependents** (if any — future direction) +3. **All other peers' inbound filters except Q's** (split-horizon exclusion) + +### Split-Horizon Exclusion + +The exclusion of Q's own inbound filter prevents echo loops. Without it, +a node would advertise back to Q the destinations it learned from Q, +creating a routing loop where Q thinks it can reach a destination through +this node, and this node thinks it can reach the same destination through Q. + +Split-horizon is computed per-peer: the outbound filter for peer Q merges +all inbound filters except Q's. For a node with peers A, B, C: + +| Outbound to | Includes entries from | +| ----------- | -------------------- | +| A | Self + B's filter + C's filter | +| B | Self + A's filter + C's filter | +| C | Self + A's filter + B's filter | + +## Filter Propagation + +Filters propagate via **FilterAnnounce** messages exchanged between direct +peers. Each FilterAnnounce replaces the previous filter for that peer — +there is no incremental update. + +### Update Triggers + +Filter updates are event-driven, not periodic: + +- Peer connects (new filter includes the new peer's reachability) +- Peer disconnects (filter must exclude the departed peer's entries) +- A peer's inbound filter changes (outbound filters to other peers must + be recomputed) +- Local state changes (new identity, leaf-only dependent changes) + +### Rate Limiting + +Updates are rate-limited at 500ms minimum interval per peer to prevent +storms during topology changes. Multiple pending changes within the +cooldown period are coalesced into a single announcement. + +### Propagation Scope + +Filters propagate unboundedly — there is no TTL on filter propagation. At +steady state, every reachable destination appears in at least one peer's +filter. New nodes added to the network propagate through the entire mesh +within O(D × 500ms) where D is the network diameter. + +## Filter Expiration + +Bloom filters cannot remove individual entries (this is a fundamental +property of the data structure). Entries are expired through: + +- **Peer disconnect**: The entire inbound filter for the departed peer is + removed, and outbound filters are recomputed +- **Filter replacement**: Each FilterAnnounce completely replaces the + previous filter for that peer +- **Implicit timeout**: If no FilterAnnounce is received from a peer within + a threshold period, the peer's filter may be considered stale (tied to + link liveness detection) + +## Size Classes and Folding + +### Size Class Table + +| size_class | Bytes | Bits | Status | +| ---------- | ----- | ---- | ------ | +| 0 | 512 | 4,096 | Reserved | +| 1 | 1,024 | 8,192 | **v1 (MUST use)** | +| 2 | 2,048 | 16,384 | Reserved | +| 3 | 4,096 | 32,768 | Reserved | + +v1 nodes MUST use size_class = 1 and MUST reject FilterAnnounce messages +with any other size_class. + +### Folding (Forward Compatibility) + +Larger filters can be **folded** to smaller sizes by OR-ing the two halves +together. A 2 KB filter folds to 1 KB by OR-ing the upper and lower +halves. This preserves the "maybe present" property (no false negatives +introduced) but increases the false positive rate. + +Folding enables future protocol versions where hub nodes maintain larger +filters (lower FPR) while constrained nodes fold down to the mandated size. +A node receiving a larger filter folds it locally before use if needed. + +The hash function design supports folding: membership tests at a smaller +size use `hash(item, i) % smaller_bit_count`, which maps to the same bit +positions that folding produces. + +## Membership Test + +To test whether a node_addr is in a bloom filter: + +```text +for i in 0..hash_count: + bit_index = hash(node_addr, i) % filter_bits + if not bits[bit_index]: + return false // Definitely not present +return true // Maybe present (possible false positive) +``` + +Where `filter_bits = 8 × (512 << size_class)` — 8,192 for v1. + +## Wire Format + +FilterAnnounce messages are carried inside encrypted link-layer frames: + +| Offset | Field | Size | Description | +| ------ | ----- | ---- | ----------- | +| 0 | msg_type | 1 byte | 0x20 | +| 1 | sequence | 8 bytes LE | Monotonic counter for freshness | +| 9 | hash_count | 1 byte | Number of hash functions (5 in v1) | +| 10 | size_class | 1 byte | Filter size: `512 << size_class` bytes | +| 11 | filter_bits | 1,024 bytes | Bloom filter bit array (v1) | + +**v1 total**: 1,035 bytes payload, 1,064 bytes with link encryption +overhead. + +See [fips-wire-formats.md](fips-wire-formats.md) for the complete wire +format reference. + +## Scale Considerations + +### Small Networks (< 1,000 nodes) + +Filters are very accurate (FPR < 1% for typical node degrees). Candidate +selection effectively identifies the correct forwarding peer on the first +try for most destinations. + +### Medium Networks (1,000–100,000 nodes) + +Filters remain accurate for typical nodes. Hub nodes (20+ peers) may see +elevated FPR but the tree distance ranking correctly selects the best +candidate regardless. + +### Large Networks (> 100,000 nodes) + +Hub nodes approach filter saturation. The filter still provides value +(some peers can be definitively excluded) but the candidate set grows +larger. The tree distance calculation becomes the primary routing +discriminator. + +Future mitigation: hub nodes could use larger filters (size_class 2 or 3) +while constrained nodes fold to size_class 1. This requires protocol +negotiation not present in v1. + +## Implementation Status + +| Feature | Status | +| ------- | ------ | +| 1 KB bloom filter (size_class 1) | **Implemented** | +| 5 hash functions | **Implemented** | +| Split-horizon filter computation | **Implemented** | +| Per-peer filter maintenance | **Implemented** | +| Event-driven updates | **Implemented** | +| 500ms rate limiting | **Implemented** | +| FilterAnnounce gossip | **Implemented** | +| Size class negotiation | Future direction | +| Folding support | Future direction | +| Adaptive filter sizing | Future direction | + +## References + +- [fips-mesh-operation.md](fips-mesh-operation.md) — How bloom filters fit + into routing +- [fips-wire-formats.md](fips-wire-formats.md) — FilterAnnounce wire format +- [fips-spanning-tree.md](fips-spanning-tree.md) — The coordinate system + that bloom filter candidates are ranked by diff --git a/docs/design/fips-configuration.md b/docs/design/fips-configuration.md index dd8f99d..91e8ae0 100644 --- a/docs/design/fips-configuration.md +++ b/docs/design/fips-configuration.md @@ -207,7 +207,7 @@ Static peer list. Each entry defines a peer to connect to. | `peers[].alias` | string | *(none)* | Human-readable name for logging | | `peers[].addresses[].transport` | string | *(required)* | Transport type (`udp`) | | `peers[].addresses[].addr` | string | *(required)* | Transport address (e.g., `"10.0.0.2:4000"`) | -| `peers[].addresses[].priority` | u16 | `100` | Address priority (lower = preferred) | +| `peers[].addresses[].priority` | u8 | `100` | Address priority (lower = preferred) | | `peers[].connect_policy` | string | `"auto_connect"` | Connection policy: `auto_connect`, `on_demand`, or `manual` | ## Minimal Example diff --git a/docs/design/fips-gossip-protocol.md b/docs/design/fips-gossip-protocol.md deleted file mode 100644 index 9909a5e..0000000 --- a/docs/design/fips-gossip-protocol.md +++ /dev/null @@ -1,794 +0,0 @@ -# FIPS Gossip Protocol - -This document specifies the wire formats and exchange rules for FIPS gossip -messages: TreeAnnounce, FilterAnnounce, and the discovery protocol -(LookupRequest/LookupResponse). - -For conceptual background on how these protocols work: - -- Spanning tree dynamics: [spanning-tree-dynamics.md](spanning-tree-dynamics.md) -- Routing design and bloom filter concepts: [fips-routing.md](fips-routing.md) - ---- - -## 1. Message Type Summary - -All gossip messages are link-layer messages, encrypted with per-peer Noise IK -session keys. They travel one hop (peer-to-peer), though their effects may -propagate further through subsequent gossip. - -| Type | Purpose | Direction | Trigger | -|------|---------|-----------|---------| -| TreeAnnounce | Spanning tree state | Bidirectional | Peer connect, parent change, periodic | -| FilterAnnounce | Bloom filter reachability | Bidirectional | Peer connect, filter change | -| LookupRequest | Coordinate discovery | Flooded | Route cache miss | -| LookupResponse | Return coordinates | Routed back | LookupRequest reaches target | - ---- - -## 2. TreeAnnounce - -TreeAnnounce messages propagate spanning tree state between peers. Each node -announces its parent selection and ancestry, enabling peers to compute tree -coordinates and distances. - -### 2.1 Wire Format - -```text -TreeAnnounce (v1) { - version: u8, // Protocol version (0x01 for v1) - sequence: u64, // Monotonic, increments on parent change - timestamp: u64, // Unix timestamp (seconds) - parent: NodeAddr, // 16 bytes, truncated SHA-256(pubkey) of selected parent - ancestry_count: u16, // Number of ancestry entries - ancestry: [AncestryEntry], // Path from self to root - signature: Signature, // 64 bytes, outer signature over entire message -} - -AncestryEntry (v1) { - node_addr: NodeAddr, // 16 bytes - sequence: u64, // That node's sequence number - timestamp: u64, // That node's timestamp -} -``` - -Note: v1 ancestry entries are 32 bytes each (no per-entry signature). See §2.7 Trust Model. - -### 2.2 Field Semantics - -**version**: Protocol version number. v1 = 0x01. Receivers MUST reject messages -with unrecognized version numbers to ensure forward compatibility. - -**sequence**: Incremented each time the node changes its parent declaration. -Higher sequence numbers supersede lower ones for conflict resolution. - -**timestamp**: Used for distributed consistency. A declaration is considered -stale if `now - timestamp > ROOT_TIMEOUT` (default 60 minutes for root). - -**parent**: The node_addr of the selected parent. If `parent == self.node_addr`, -the node is declaring itself as root. - -**ancestry**: The chain from this node up to the root. The first entry is this -node's own declaration, followed by parent, grandparent, etc. In v1, entries -carry only routing metadata (node_addr, sequence, timestamp) without per-entry -signatures. See §2.7 for the trust model. - -### 2.3 Size Estimate - -| Component | Size | -|-----------|------| -| version | 1 byte | -| sequence | 8 bytes | -| timestamp | 8 bytes | -| parent | 16 bytes | -| ancestry_count | 2 bytes | -| signature | 64 bytes | -| Per ancestry entry (v1) | 16 + 8 + 8 = 32 bytes | - -For tree depth D (ancestry_count = D + 1): `100 + (D + 1) × 32` bytes payload. - -| Tree Depth | Payload Size | With Link Overhead | -|------------|--------------|--------------------| -| 0 (root) | 132 bytes | 161 bytes | -| 3 | 228 bytes | 257 bytes | -| 5 | 292 bytes | 321 bytes | -| 10 | 452 bytes | 481 bytes | - -Note: v1 ancestry entries omit per-entry signatures (32 bytes vs 96 bytes in -the original design). See §2.7 for the rationale. - -### 2.4 Exchange Rules - -**On peer connection:** - -1. After Noise IK handshake completes, both peers send TreeAnnounce -2. Each peer processes the received announcement (see §2.5) -3. If processing triggers a parent change, send updated TreeAnnounce to all peers - -**On parent change:** - -1. Increment sequence number -2. Update timestamp -3. Sign new declaration -4. Send TreeAnnounce to all peers - -**Periodic refresh:** - -1. Root refreshes every 30 minutes (prevents stale root detection) -2. Non-root nodes forward root refresh when received -3. Nodes may refresh their own declaration periodically (implementation choice) - -### 2.5 Processing Rules - -When receiving TreeAnnounce from peer P: - -```text -1. Decode message; reject if version != 0x01 -2. Verify P's declaration signature using P's known public key (from Noise IK) -3. Verify that declaration node_addr matches sender's identity -4. Check sequence freshness: - - If sequence <= stored sequence for P: discard (stale) -5. Update peer P's tree state (declaration + ancestry) -6. Re-evaluate parent selection: - - Find smallest root visible across all peers - - Among peers reaching smallest root, prefer shallowest depth - - Apply stability threshold to prevent flapping (depth improvement ≥ 1) - - If parent changed: increment own sequence, sign, recompute coords, announce to all -``` - -Note: In v1, only the sender's declaration signature is verified (step 2). -Ancestry entries beyond the direct peer are accepted on trust. See §2.7. - -### 2.6 Rate Limiting - -To prevent announcement storms during reconvergence: - -- Minimum interval between announcements to same peer: 500ms -- If change occurs during cooldown: mark pending, send after cooldown -- Coalesce multiple pending changes into single announcement - -### 2.7 Trust Model (v1) - -**v1 uses transitive trust**: each node verifies only its direct peer's -declaration signature. The peer's public key is known from the Noise IK -handshake, so verification is straightforward. Ancestry entries from nodes -beyond the direct peer are accepted on trust from the authenticated sender. - -**Why transitive trust?** NodeAddr values are truncated SHA-256 hashes of -public keys — this mapping is intentionally one-way. To verify an ancestry -entry's signature, a node would need the entry's public key, but FIPS does not -distribute node_addr→pubkey mappings by design. Exposing these mappings would -enable traffic analysis, undermining a core privacy property. - -**Limitation**: An adversarial interior node could fabricate ancestry chains, -potentially attracting traffic to itself (sinkhole attack) or manipulating tree -topology. This risk is mitigated by: - -- **Authenticated peers have reputation cost**: Misbehaving nodes can be - disconnected and blocked by their direct peers. -- **Multi-path observation**: Nodes receiving conflicting tree state from - multiple peers can detect inconsistencies (future enhancement). - -**Versioning**: The wire format includes a version byte (v1 = 0x01) to enable -future protocol evolution. A future version could introduce stronger ancestry -verification (e.g., zero-knowledge proofs of key ownership) without breaking -backward compatibility. Nodes MUST reject TreeAnnounce messages with -unrecognized version numbers. - ---- - -## 3. FilterAnnounce - -FilterAnnounce messages propagate Bloom filter reachability information. Each -node's filter indicates which destinations are reachable through it. - -### 3.1 Wire Format - -```text -FilterAnnounce { - sequence: u64, // For freshness/deduplication - filter: BloomFilter, // Variable size based on size_class -} - -BloomFilter { - hash_count: u8, // Number of hash functions (5 for v1) - size_class: u8, // Filter size: bytes = 512 << size_class - bits: [u8; 512 << size_class], // Bit array (1024 bytes for v1) -} -``` - -### 3.2 Size Classes - -Filter sizes are powers of 2 to enable folding (shrinking by ORing halves): - -| size_class | Bits | Bytes | Status | -|------------|--------|-------|---------------------| -| 0 | 4,096 | 512 | Reserved (future) | -| 1 | 8,192 | 1,024 | **v1 default** | -| 2 | 16,384 | 2,048 | Reserved (future) | -| 3 | 32,768 | 4,096 | Reserved (future) | - -**v1 protocol**: All nodes MUST use size_class=1 (1 KB filters). Nodes MUST -reject FilterAnnounce with size_class ≠ 1. - -**Future versions**: Nodes may negotiate larger filters via capability exchange. -Receivers can fold larger filters down to their preferred size. - -### 3.3 Field Semantics - -**sequence**: Monotonic counter for this node's filter. Allows receivers to -detect stale or duplicate announcements. - -**hash_count**: Number of hash functions used. v1 uses k=5, which is optimal -for 800-1,600 entries in a 1 KB filter. - -**size_class**: Indicates filter size as `512 << size_class` bytes. Allows -forward-compatible extension to larger filters. - -**bits**: The Bloom filter bit array. To test membership: - -```text -for i in 0..hash_count: - bit_index = hash(node_addr, i) % (8 * bits.len()) - if !bits[bit_index]: return false -return true // "maybe present" -``` - -### 3.3 Filter Contents - -A node's outgoing filter to peer Q contains: - -1. This node's own node_addr -2. Node_ids of leaf-only dependents (nodes using this node as sole peer) -3. Entries merged from filters received from all other peers (not Q) - -This split-horizon merge (excluding the destination peer's own filter from -the computation) prevents a node's entries from being echoed back to it, -providing loop prevention. Filters propagate transitively through the -network without any hop limit. - -### 3.4 Exchange Rules - -**On peer connection:** - -1. After TreeAnnounce exchange, send FilterAnnounce -2. Filter contains current reachability view - -**On filter change:** - -Triggering events: - -- Peer connects or disconnects -- Received filter changes outgoing filter -- Local state change (new leaf dependent, become gateway) - -Rate limiting: - -- Minimum interval between filter announcements: 500ms -- Debounce rapid changes into single announcement - -**Processing received filter:** - -```text -1. Store: peer_filters[P] = received.filter -2. Recompute outgoing filters for all other peers: - - For each peer Q (Q != P): - outgoing[Q] = merge(self_filter, peer_filters[all peers except Q]) - - If outgoing[Q] changed, send FilterAnnounce to Q -``` - -### 3.5 Filter Expiration - -Bloom filters cannot remove individual entries. Expiration handled by: - -- **Peer disconnect**: Remove that peer's filter entirely, recompute -- **Filter replacement**: Each FilterAnnounce replaces the previous one -- **Implicit timeout**: If no updates from peer within threshold, consider stale - ---- - -## 4. LookupRequest - -LookupRequest initiates coordinate discovery for destinations not covered by -local Bloom filters. - -### 4.1 Wire Format - -```text -LookupRequest { - request_id: u64, // Unique identifier for this request - target: NodeAddr, // 16 bytes, who we're looking for - origin: NodeAddr, // 16 bytes, who's asking - origin_coords: Vec, // Origin's ancestry (for return path) - ttl: u8, // Remaining propagation hops - visited: CompactBloomFilter,// ~256 bytes, prevents loops -} - -CompactBloomFilter { - bits: [u8; 256], // Smaller filter for visited set - hash_count: u8, -} -``` - -### 4.2 Field Semantics - -**request_id**: Randomly generated, used to match responses and detect -duplicates. - -**target**: The node_addr being searched for. - -**origin**: The node_addr of the original requester. Used for response routing. - -**origin_coords**: The requester's current tree coordinates. Used by the target -for the first hop of response routing (via `find_next_hop`). - -**ttl**: Propagation limit. Prevents unbounded flooding. - -**visited**: Compact Bloom filter tracking nodes that have seen this request. -Prevents loops (revisiting a node on the same path). Note: does NOT prevent -convergent duplicates arriving via different paths — `request_id` dedup -(section 4.4) is required for that. - -### 4.3 Propagation Rules - -When receiving LookupRequest: - -```text -1. Check request_id against recent-request cache - if present, drop (duplicate - via convergent path). This is REQUIRED, not optional — the visited filter - alone does not prevent duplicates arriving via different paths. -2. Add request_id to recent-request cache -3. Check visited filter - if self likely present, drop (already processed) -4. Add self to visited filter -5. Decrement TTL - -6. Check if target is local: - - If target == self.node_addr: generate LookupResponse - - If target in local peer_filters: may respond on behalf (optional) - -7. If TTL > 0 and not found locally: - - Forward to peers not in visited filter - - Optionally prioritize peers whose filter indicates target "maybe" present -``` - -The recent-request cache need only retain entries for a few seconds (long -enough for the flood to complete across the TTL scope) and is bounded by the -rate limit on incoming requests. - -### 4.4 Rate Limiting - -- Limit requests per origin per time window -- Limit total outstanding requests - ---- - -## 5. LookupResponse - -LookupResponse returns the target's coordinates to the requester. - -### 5.1 Wire Format - -```text -LookupResponse { - request_id: u64, // Echoes LookupRequest.request_id - target: NodeAddr, // 16 bytes, confirms who was found - target_coords: Vec, // Target's ancestry (the key payload) - proof: Signature, // 64 bytes, target signs to prove existence -} -``` - -### 5.2 Field Semantics - -**request_id**: Matches the original request, allowing requester to correlate. - -**target**: Confirms the identity found. - -**target_coords**: The target's current tree coordinates. This is the primary -payload — cached by the originator to enable routing to the target. - -**proof**: Target's signature over `(request_id || target)`. Prevents -malicious nodes from claiming reachability and blackholing traffic. -Coordinates are excluded from the proof to avoid invalidation during -tree reconvergence (see [fips-routing.md](fips-routing.md) §2.4). - -### 5.3 Routing - -LookupResponse uses a two-phase routing mechanism: - -```text -1. Response created at target (or node with target in filter) -2. First hop: target routes toward origin via find_next_hop - (standard bloom filter → tree routing path) -3. Subsequent hops: reverse-path forwarding via recent_requests - (each transit node recorded which peer sent the request) -4. Origin receives response, caches target_coords in RouteCache -``` - -The first hop from the target uses `find_next_hop(origin)` because -the target was not a transit node for the request (it was the -destination). All transit nodes that forwarded the request stored a -`(request_id → from_peer)` entry in `recent_requests`, enabling -reverse-path forwarding for the response. - -### 5.4 Security - -The proof signature is critical: - -- Without it, any node could claim to be (or know) any target -- Requester verifies signature against target's known public key -- Invalid signatures cause response to be dropped - ---- - -## 6. Message Type Codes - -Within the link-layer message framing: - -| Type Code | Message | -|-----------|---------| -| 0x10 | TreeAnnounce | -| 0x20 | FilterAnnounce | -| 0x30 | LookupRequest | -| 0x31 | LookupResponse | - -These are carried inside the encrypted link-layer payload after Noise IK -handshake completion. See [fips-wire-protocol.md](fips-wire-protocol.md) §2.6 -for the full link message type table. - ---- - -## 7. Timing Parameters - -| Parameter | Default | Notes | -|-----------|---------|-------| -| ROOT_REFRESH_INTERVAL | 30 min | Root regenerates timestamp | -| ROOT_TIMEOUT | 60 min | Root declaration considered stale | -| TREE_ENTRY_TTL | 5-10 min | Individual entry expiration | -| ANNOUNCE_MIN_INTERVAL | 500 ms | Rate limit for announcements | -| LOOKUP_TTL | 64 | Discovery request propagation limit | -| LOOKUP_TIMEOUT | 10 sec | Time to wait for response | - ---- - -## 8. Encoding - -All multi-byte integers are little-endian. NodeAddr is 16 bytes (truncated SHA-256 hash). -Signatures are 64 bytes (secp256k1 Schnorr). - -Variable-length fields (ancestry, coordinates) are prefixed with a 2-byte -length count indicating number of entries. - -```text -Vec encoding: - count: u16 (little-endian) - items: T[count] -``` - ---- - -## Appendix A: Detailed Packet Layouts - -All gossip messages are link-layer messages carried inside encrypted frames -(discriminator 0x00). The layouts below show the plaintext structure after -link-layer decryption. - -### A.1 TreeAnnounce (0x10) - -Propagates spanning tree state between directly connected peers. - -```text -┌─────────────────────────────────────────────────────────────────────────────┐ -│ FULL PACKET (Link Layer + TreeAnnounce) │ -├─────────────────────────────────────────────────────────────────────────────┤ -│ │ -│ ┌───────────────────────────────────────────────────────────────────────┐ │ -│ │ LINK LAYER FRAME (encrypted) │ │ -│ ├───────────┬──────────────┬────────────┬───────────────────────────────┤ │ -│ │ 0x00 │ receiver_idx │ counter │ ciphertext + tag │ │ -│ │ 1 byte │ 4 bytes LE │ 8 bytes LE │ N + 16 bytes │ │ -│ └───────────┴──────────────┴────────────┴───────────────────────────────┘ │ -│ │ │ -│ ┌───────────────────────┘ │ -│ │ Decrypt │ -│ ▼ │ -│ ┌───────────────────────────────────────────────────────────────────────┐ │ -│ │ TREE ANNOUNCE v1 (plaintext) │ │ -│ ├────────┬──────────────────┬───────────┬───────────────────────────────┤ │ -│ │ Offset │ Field │ Size │ Description │ │ -│ ├────────┼──────────────────┼───────────┼───────────────────────────────┤ │ -│ │ 0 │ msg_type │ 1 byte │ 0x10 │ │ -│ │ 1 │ version │ 1 byte │ 0x01 (v1) │ │ -│ │ 2 │ sequence │ 8 bytes │ u64 LE, monotonic counter │ │ -│ │ 10 │ timestamp │ 8 bytes │ u64 LE, Unix seconds │ │ -│ │ 18 │ parent │ 16 bytes │ NodeAddr of selected parent │ │ -│ │ 34 │ ancestry_count │ 2 bytes │ u16 LE, number of entries │ │ -│ │ 36 │ ancestry[0..n] │ 32 × n │ AncestryEntry array │ │ -│ │ ... │ signature │ 64 bytes │ Schnorr sig over all above │ │ -│ └────────┴──────────────────┴───────────┴───────────────────────────────┘ │ -│ │ -│ ┌───────────────────────────────────────────────────────────────────────┐ │ -│ │ ANCESTRY ENTRY v1 (32 bytes each) │ │ -│ ├────────┬──────────────────┬───────────┬───────────────────────────────┤ │ -│ │ Offset │ Field │ Size │ Description │ │ -│ ├────────┼──────────────────┼───────────┼───────────────────────────────┤ │ -│ │ 0 │ node_addr │ 16 bytes │ Truncated SHA-256(pubkey) │ │ -│ │ 16 │ sequence │ 8 bytes │ u64 LE, node's seq number │ │ -│ │ 24 │ timestamp │ 8 bytes │ u64 LE, node's timestamp │ │ -│ └────────┴──────────────────┴───────────┴───────────────────────────────┘ │ -│ │ -│ Note: v1 entries omit per-entry signatures. Only the sender's outer │ -│ signature is verified (transitive trust model, see §2.7). │ -│ │ -└─────────────────────────────────────────────────────────────────────────────┘ -``` - -**Size calculation**: `1 + 1 + 8 + 8 + 16 + 2 + (depth × 32) + 64 = 100 + (depth × 32)` bytes - -| Tree Depth | Payload Size | With Link Overhead | -|------------|--------------|--------------------| -| 0 (root) | 132 bytes | 161 bytes | -| 3 | 228 bytes | 257 bytes | -| 5 | 292 bytes | 321 bytes | -| 10 | 452 bytes | 481 bytes | - -**Concrete example** (node D at depth 3, ancestry = [D, P1, P2, Root]): - -```text -PLAINTEXT BYTES (hex layout): -10 ← msg_type = TreeAnnounce -01 ← version = 1 -05 00 00 00 00 00 00 00 ← sequence = 5 -C3 B2 A1 67 00 00 00 00 ← timestamp (Unix seconds) -[16 bytes P1's node_addr] ← parent -04 00 ← ancestry_count = 4 - -ANCESTRY[0] - Self (D): - [16 bytes D's node_addr] - 05 00 00 00 00 00 00 00 ← D's sequence - C3 B2 A1 67 00 00 00 00 ← D's timestamp - -ANCESTRY[1] - Parent (P1): - [16 bytes P1's node_addr] - 0A 00 00 00 00 00 00 00 ← P1's sequence - 00 B0 A1 67 00 00 00 00 ← P1's timestamp - -ANCESTRY[2] - Grandparent (P2): - [16 bytes P2's node_addr] - 03 00 00 00 00 00 00 00 ← P2's sequence - 00 A0 A1 67 00 00 00 00 ← P2's timestamp - -ANCESTRY[3] - Root: - [16 bytes Root's node_addr] - 01 00 00 00 00 00 00 00 ← Root's sequence - 00 90 A1 67 00 00 00 00 ← Root's timestamp - -[64 bytes D's outer signature] ← signs entire message - -Total payload: 1 + 1 + 8 + 8 + 16 + 2 + (4 × 32) + 64 = 228 bytes -``` - -### A.2 FilterAnnounce (0x20) - -Propagates Bloom filter reachability information. - -```text -┌─────────────────────────────────────────────────────────────────────────────┐ -│ FILTER ANNOUNCE (0x20) │ -├─────────────────────────────────────────────────────────────────────────────┤ -│ │ -│ ┌───────────────────────────────────────────────────────────────────────┐ │ -│ │ WIRE FORMAT │ │ -│ ├────────┬──────────────────┬───────────┬───────────────────────────────┤ │ -│ │ Offset │ Field │ Size │ Description │ │ -│ ├────────┼──────────────────┼───────────┼───────────────────────────────┤ │ -│ │ 0 │ msg_type │ 1 byte │ 0x20 │ │ -│ │ 1 │ sequence │ 8 bytes │ u64 LE, monotonic counter │ │ -│ │ 9 │ hash_count │ 1 byte │ Number of hash functions (5) │ │ -│ │ 10 │ size_class │ 1 byte │ Filter size: 512 << class │ │ -│ │ 11 │ filter_bits │ variable │ 512 << size_class bytes │ │ -│ └────────┴──────────────────┴───────────┴───────────────────────────────┘ │ -│ │ -│ Size classes (powers of 2 for foldability): │ -│ 0 = 512 bytes (4,096 bits) - Reserved for future │ -│ 1 = 1,024 bytes (8,192 bits) - v1 default │ -│ 2 = 2,048 bytes (16,384 bits) - Reserved for future │ -│ 3 = 4,096 bytes (32,768 bits) - Reserved for future │ -│ │ -│ v1 total payload: 1 + 8 + 1 + 1 + 1024 = 1035 bytes │ -│ With link overhead: 1064 bytes │ -│ │ -├─────────────────────────────────────────────────────────────────────────────┤ -│ BLOOM FILTER STRUCTURE │ -├─────────────────────────────────────────────────────────────────────────────┤ -│ │ -│ filter_bits[1024] (v1, size_class=1): │ -│ ┌─────────────────────────────────────────────────────────────────────┐ │ -│ │ Byte 0 │ Byte 1 │ ... │ Byte 1023 │ │ -│ │ bits 0-7 │ bits 8-15 │ │ bits 8184-8191 │ │ -│ └─────────────────────────────────────────────────────────────────────┘ │ -│ │ -│ To test membership of node_addr: │ -│ filter_bits = 8 * (512 << size_class) // 8192 for v1 │ -│ for i in 0..hash_count: │ -│ bit_index = hash(node_addr, i) % filter_bits │ -│ if !bits[bit_index]: return false │ -│ return true // "maybe present" │ -│ │ -│ Folding (for future heterogeneous sizes): │ -│ To shrink a filter by half, OR its two halves: │ -│ small[i] = large[i] | large[i + small.len()] │ -│ This increases FPR but preserves correctness (no false negatives). │ -│ │ -└─────────────────────────────────────────────────────────────────────────────┘ -``` - -**Concrete example** (v1 with size_class=1): - -```text -PLAINTEXT BYTES: -20 ← msg_type = FilterAnnounce -2A 00 00 00 00 00 00 00 ← sequence = 42 -05 ← hash_count = 5 -01 ← size_class = 1 (1 KB filter) -[1024 bytes of filter bits] ← Bloom filter - -Total: 1035 bytes -``` - -### A.3 LookupRequest (0x30) - -Discovers tree coordinates for distant destinations. - -```text -┌─────────────────────────────────────────────────────────────────────────────┐ -│ LOOKUP REQUEST (0x30) │ -├─────────────────────────────────────────────────────────────────────────────┤ -│ │ -│ ┌───────────────────────────────────────────────────────────────────────┐ │ -│ │ WIRE FORMAT │ │ -│ ├────────┬──────────────────┬───────────┬───────────────────────────────┤ │ -│ │ Offset │ Field │ Size │ Description │ │ -│ ├────────┼──────────────────┼───────────┼───────────────────────────────┤ │ -│ │ 0 │ msg_type │ 1 byte │ 0x30 │ │ -│ │ 1 │ request_id │ 8 bytes │ u64 LE, unique identifier │ │ -│ │ 9 │ target │ 16 bytes │ NodeAddr being searched for │ │ -│ │ 25 │ origin │ 16 bytes │ NodeAddr of requester │ │ -│ │ 41 │ ttl │ 1 byte │ Remaining propagation hops │ │ -│ │ 42 │ origin_coords_cnt│ 2 bytes │ u16 LE │ │ -│ │ 44 │ origin_coords │ 16 × n │ Requester's ancestry │ │ -│ │ ... │ visited_hash_cnt │ 1 byte │ Hash functions for visited │ │ -│ │ ... │ visited_bits │ 256 bytes │ Compact bloom of visited nodes│ │ -│ └────────┴──────────────────┴───────────┴───────────────────────────────┘ │ -│ │ -└─────────────────────────────────────────────────────────────────────────────┘ -``` - -**Size calculation**: `1 + 8 + 16 + 16 + 1 + 2 + (depth × 16) + 1 + 256` bytes - -| Origin Depth | Payload Size | -|--------------|--------------| -| 3 | 349 bytes | -| 5 | 381 bytes | -| 10 | 461 bytes | - -**Concrete example** (origin at depth 4): - -```text -PLAINTEXT BYTES: -30 ← msg_type = LookupRequest -[8 bytes request_id] ← random unique ID -[16 bytes target node_addr] ← who we're looking for -[16 bytes origin node_addr] ← who's asking -40 ← ttl = 64 -04 00 ← origin_coords_count = 4 -[16 bytes] × 4 ← origin's ancestry (64 bytes) -05 ← visited hash_count = 5 -[256 bytes visited bloom] ← nodes that have seen this request - -Total: 1 + 8 + 16 + 16 + 1 + 2 + 64 + 1 + 256 = 365 bytes -``` - -### A.4 LookupResponse (0x31) - -Returns target's coordinates to the requester. - -```text -┌─────────────────────────────────────────────────────────────────────────────┐ -│ LOOKUP RESPONSE (0x31) │ -├─────────────────────────────────────────────────────────────────────────────┤ -│ │ -│ ┌───────────────────────────────────────────────────────────────────────┐ │ -│ │ WIRE FORMAT │ │ -│ ├────────┬──────────────────┬───────────┬───────────────────────────────┤ │ -│ │ Offset │ Field │ Size │ Description │ │ -│ ├────────┼──────────────────┼───────────┼───────────────────────────────┤ │ -│ │ 0 │ msg_type │ 1 byte │ 0x31 │ │ -│ │ 1 │ request_id │ 8 bytes │ u64 LE, echoes request │ │ -│ │ 9 │ target │ 16 bytes │ NodeAddr that was found │ │ -│ │ 25 │ target_coords_cnt│ 2 bytes │ u16 LE │ │ -│ │ 27 │ target_coords │ 16 × n │ Target's ancestry to root │ │ -│ │ ... │ proof │ 64 bytes │ Target's signature │ │ -│ └────────┴──────────────────┴───────────┴───────────────────────────────┘ │ -│ │ -│ Proof signature covers: (request_id || target) │ -│ Coords excluded to survive tree reconvergence during lookup RTT. │ -│ │ -└─────────────────────────────────────────────────────────────────────────────┘ -``` - -**Size calculation**: `1 + 8 + 16 + 2 + (depth × 16) + 64` bytes - -| Target Depth | Payload Size | -|--------------|--------------| -| 3 | 139 bytes | -| 5 | 171 bytes | -| 10 | 251 bytes | - -**Concrete example** (target at depth 5): - -```text -PLAINTEXT BYTES: -31 ← msg_type = LookupResponse -[8 bytes request_id] ← echoed from request -[16 bytes target node_addr] ← confirms who was found -05 00 ← target_coords_count = 5 -[16 bytes] × 5 ← target's ancestry (80 bytes) -[64 bytes proof signature] ← target signs to prove existence - -Total: 1 + 8 + 16 + 2 + 80 + 64 = 171 bytes -``` - -### A.5 Message Flow Example - -Complete lookup flow showing packet nesting: - -```text -Source S wants to reach distant destination D (not in local filters) - -1. S creates LookupRequest, sends to peer P1: - - UDP DATAGRAM - ┌──────────────────────────────────────────────────────────────┐ - │ LINK FRAME (S→P1 encrypted) │ - │ ┌──────┬────────────┬─────────┬─────────────────────────────┐│ - │ │ 0x00 │ P1_recv_idx│ counter │ ciphertext + tag ││ - │ └──────┴────────────┴─────────┴─────────────────────────────┘│ - │ │ │ - │ ┌───────────┘ │ - │ ▼ │ - │ ┌──────┬───────────────────────────────────┐ │ - │ │ 0x30 │ LookupRequest payload │ │ - │ │ │ (target=D, origin=S, ttl=64, ...) │ │ - │ └──────┴───────────────────────────────────┘ │ - └──────────────────────────────────────────────────────────────┘ - -2. Request propagates through network, reaches D - -3. D creates LookupResponse, routes back via find_next_hop + reverse-path: - - UDP DATAGRAM - ┌──────────────────────────────────────────────────────────────┐ - │ LINK FRAME (D→Pn encrypted) │ - │ ┌──────┬────────────┬─────────┬─────────────────────────────┐│ - │ │ 0x00 │ Pn_recv_idx│ counter │ ciphertext + tag ││ - │ └──────┴────────────┴─────────┴─────────────────────────────┘│ - │ │ │ - │ ┌───────────┘ │ - │ ▼ │ - │ ┌──────┬───────────────────────────────────┐ │ - │ │ 0x31 │ LookupResponse payload │ │ - │ │ │ (target=D, coords=[D,P1,P2,Root]) │ │ - │ └──────┴───────────────────────────────────┘ │ - └──────────────────────────────────────────────────────────────┘ - -4. S receives response, caches D's coordinates, can now route directly -``` - ---- - -## References - -- [spanning-tree-dynamics.md](spanning-tree-dynamics.md) - Tree protocol behavior -- [fips-routing.md](fips-routing.md) - Routing concepts and algorithms -- [fips-wire-protocol.md](fips-wire-protocol.md) - Link-layer framing -- [fips-session-protocol.md](fips-session-protocol.md) - End-to-end sessions diff --git a/docs/design/fips-intro.md b/docs/design/fips-intro.md index 37c5a70..bb7e87b 100644 --- a/docs/design/fips-intro.md +++ b/docs/design/fips-intro.md @@ -3,9 +3,9 @@ ## What is FIPS? FIPS is a self-organizing mesh network that can operate over any transport -medium—radio, serial links, Tor, local networks, or the existing internet as -an overlay. The long-term goal is infrastructure that can function alongside -or ultimately replace dependence on the Internet. +medium — radio, serial links, Tor, local networks, or the existing internet as +an overlay. The long-term goal is infrastructure that can function alongside or +ultimately replace dependence on the Internet. Nodes in the mesh route traffic for each other using Nostr identities (npubs) as network addresses. Applications can access the mesh through a native FIPS @@ -15,7 +15,7 @@ as an IPv6 endpoint for compatibility with existing IP-based applications. ## Why FIPS? **Infrastructure independence**: The internet depends on centralized -infrastructure—ISPs, backbone providers, DNS, certificate authorities. FIPS +infrastructure — ISPs, backbone providers, DNS, certificate authorities. FIPS works over any transport that can carry packets: a LoRa radio link between mountain towns, a serial cable between air-gapped systems, onion-routed connections through Tor, or the existing internet as an overlay. When the @@ -38,80 +38,185 @@ and security credentials without coordination with any central authority. The identity system uses Nostr keypairs (secp256k1), so existing npub/nsec pairs work directly. -## How It Works (Overview) +## A Self-Organizing Mesh -Each FIPS node selects which transports to use (Ethernet, radio links, internet -overlay, etc.) and which peer nodes to connect to. From these local peering -decisions, the distributed spanning tree and bloom filter propagation algorithms -self-organize reachability and path information throughout the mesh. +Traditional networks are built top-down. A central authority assigns addresses, +configures routing tables, provisions hardware, and manages the topology. If the +authority disappears or the infrastructure fails, the network fails with it. +Nodes cannot reach each other without infrastructure mediating the connection. -Nodes form a spanning tree rooted at a deterministically-elected node. Each -node knows its path to the root, enabling shortest-path routing to be -calculated between any two nodes (not necessarily through the root). Bloom filters propagate reachability information so nodes can find -each other. Traffic flows via greedy routing toward destinations, encrypted -end-to-end. +FIPS inverts this model. There is no central authority, no address assignment +service, no routing table pushed from above. Each node generates its own +identity from a cryptographic keypair. Each node independently decides which +peers to connect to and which transports to use. From these local decisions +alone, the network self-organizes: -Nodes with multiple transports automatically bridge between networks—the same -routing logic works regardless of the underlying transport mix. +- A **spanning tree** forms through distributed parent selection, giving every + node a coordinate in the network without any node knowing the full topology +- **Bloom filters** propagate through gossip, so each node learns which peers + can reach which destinations — again without global knowledge +- **Routing decisions** are made locally at each hop, using only the node's + immediate peers and cached coordinate information + +The result is a network that builds itself from the bottom up, heals around +failures automatically, and scales without central coordination. Adding a node +is as simple as connecting to one existing peer — the network integrates the +new node through its normal gossip protocols. ## Design Goals -1. **Nostr-native identity** - Use Nostr keypairs as node identities -2. **Transport agnostic** - Support IP, wireless, serial, onion, and other link types -3. **Self-organizing** - Automatic topology discovery and route optimization -4. **Privacy preserving** - Minimize metadata leakage across untrusted links -5. **Resilient** - Self-healing with graceful degradation -6. **Reuse Nostr primitives** - Leverage secp256k1, Schnorr signatures, and SHA-256 +1. **Nostr-native identity** — Use Nostr keypairs as node identities +2. **Transport agnostic** — Support IP, wireless, serial, onion, and other + link types +3. **Self-organizing** — Automatic topology discovery and route optimization +4. **Privacy preserving** — Minimize metadata leakage across untrusted links +5. **Resilient** — Self-healing with graceful degradation +6. **Reuse Nostr primitives** — Leverage secp256k1, Schnorr signatures, and + SHA-256 + +--- + +## Protocol Architecture + +FIPS is organized in three protocol layers, each with distinct responsibilities +and clean service boundaries. Understanding these layers is key to understanding +how FIPS works. + +```text +┌─────────────────────────────────────────────────────────────┐ +│ Applications │ +│ (native FIPS API / IPv6 adapter) │ +├─────────────────────────────────────────────────────────────┤ +│ │ +│ FIPS Session Protocol (FSP) │ +│ End-to-end authenticated encryption between endpoints │ +│ Session lifecycle, coordinate caching, replay protection │ +│ │ +├─────────────────────────────────────────────────────────────┤ +│ │ +│ FIPS Link Protocol (FLP) │ +│ Hop-by-hop link encryption, peer authentication │ +│ Spanning tree, bloom filters, routing, forwarding │ +│ │ +├─────────────────────────────────────────────────────────────┤ +│ │ +│ Transport Layer │ +│ Datagram delivery over arbitrary media │ +│ UDP, Ethernet, LoRa, Tor, serial, ... │ +│ │ +└─────────────────────────────────────────────────────────────┘ +``` + +### Mapping to Traditional Networking + +Readers familiar with the OSI model or TCP/IP networking may find it helpful to +see how FIPS concepts relate to traditional layers: + +```text + Traditional FIPS Key Difference +───────────────────────────────────────────────────────────────────── + Application Applications Same role — user-facing + (IPv6 adapter) software +───────────────────────────────────────────────────────────────────── + Transport (TCP/UDP) (not present) FIPS provides datagrams, + not reliable streams +───────────────────────────────────────────────────────────────────── + Session FSP End-to-end encryption + and session management +───────────────────────────────────────────────────────────────────── + Network (IP) FLP Routing, forwarding, + address resolution — + but self-organizing +───────────────────────────────────────────────────────────────────── + Link (Ethernet/WiFi) FLP (link encryption) Peer authentication + and hop-by-hop crypto +───────────────────────────────────────────────────────────────────── + Physical (PHY) Transport layer Abstracted — FIPS + (UDP, radio, serial) treats all media the + same way +───────────────────────────────────────────────────────────────────── +``` + +Note that FLP spans what would traditionally be separate link and network +layers. This is intentional — in a self-organizing mesh, the same layer that +authenticates peers also makes routing decisions, because routing depends on +authenticated peer state (spanning tree positions, bloom filters). + +### Layer Responsibilities + +**Transport layer**: Delivers datagrams between endpoints over a specific +medium. Each transport type (UDP socket, Ethernet interface, radio modem) +implements the same abstract interface: send and receive datagrams, report MTU. +The transport layer knows nothing about FIPS identities, routing, or encryption. +It provides raw datagram delivery to FLP above. + +See [fips-transport-layer.md](fips-transport-layer.md) for the transport layer +specification. + +**FIPS Link Protocol (FLP)**: Manages peer connections, authenticates peers via +Noise IK handshakes, and encrypts all traffic on each link. FLP is where the +mesh organizes itself — nodes exchange spanning tree announcements and bloom +filters with their direct peers, and FLP makes forwarding decisions for transit +traffic. FLP provides authenticated, encrypted forwarding to FSP above. + +See [fips-link-layer.md](fips-link-layer.md) for the FLP specification and +[fips-mesh-operation.md](fips-mesh-operation.md) for how FLP's routing and +self-organization work in practice. + +**FIPS Session Protocol (FSP)**: Provides end-to-end authenticated encryption +between any two nodes, regardless of how many intermediate hops separate them. +FSP manages session lifecycle (setup, data transfer, teardown), caches +destination coordinates for efficient routing, and handles the warmup strategy +that keeps transit node caches populated. FSP provides a datagram service to +applications above. + +See [fips-session-layer.md](fips-session-layer.md) for the FSP specification. + +**IPv6 adaptation layer**: Sits above FSP and adapts the FIPS datagram service +for unmodified IPv6 applications. Provides DNS resolution (npub → fd::/8 +address), identity cache management, MTU enforcement, and a TUN interface. +This is the primary way existing applications use the FIPS mesh. + +See [fips-ipv6-adapter.md](fips-ipv6-adapter.md) for the IPv6 adapter. + +--- ## Architecture Overview ![Architecture Overview](fips-architecture-overview.svg) -Each link uses a different transport, but the end-to-end session encryption -is independent of the transport mix. Intermediate nodes decrypt the link -layer to make routing decisions, then re-encrypt for the next hop. They -cannot read the session-layer payload. +Each link uses a different transport, but the end-to-end session encryption is +independent of the transport mix. Intermediate nodes decrypt the link layer to +make routing decisions, then re-encrypt for the next hop. They cannot read the +session-layer payload. -Applications can use the native FIPS datagram service directly, or access the -mesh through an IPv6 adaptation layer (TUN device) for compatibility with -existing IP-based applications. The router handles discovery, routing, and -encryption transparently in either case. +```text + Application ──────────── End-to-end FSP session ──────────── Application + │ │ + ▼ ▼ + ┌─────────┐ FLP link ┌─────────┐ FLP link ┌─────────┐ + │ Node A │◄──────────────►│ Node B │◄──────────────►│ Node C │ + └────┬────┘ (Noise IK) └────┬────┘ (Noise IK) └────┬────┘ + │ │ │ + UDP/IP Ethernet LoRa + transport transport transport +``` -See [fips-transports.md](fips-transports.md) for transport options and characteristics. +Each FLP link operates over its own transport type, with independent link-layer +encryption. The FSP session spans the entire path, providing end-to-end +confidentiality that is independent of the transport mix along the route. ![Node Architecture](fips-node-architecture.svg) -Internally, each node is organized in three layers. At the top, two application -interfaces provide access to the mesh: a native datagram API addressed by npub, -and an IPv6 TUN adapter that maps npubs to `fd::/8` addresses so unmodified -IP applications can use the network transparently. The router core in the middle -implements spanning tree maintenance, bloom filter exchange, greedy routing, -session management, and Noise IK encryption. At the bottom, transport plugins -handle the physical diversity — each plugin implements the same interface, so the -router treats UDP, Ethernet, LoRa, Tor, and serial links identically. Adding a -new transport requires no changes to the routing or session layers. - -## Prior Work - -FIPS builds on proven designs rather than inventing new cryptography or routing -algorithms. - -**Routing**: The spanning tree coordinates, bloom filter discovery, and greedy -routing algorithms are adapted from [Yggdrasil v0.5](https://yggdrasil-network.github.io/2023/10/22/upcoming-v05-release.html) -and its [Ironwood](https://github.com/Arceliar/ironwood) routing library. FIPS -adapts these for multi-transport operation and Nostr identity integration. - -**Encryption**: Link and session encryption use the [Noise Protocol -Framework](https://noiseprotocol.org/), the same foundation used by WireGuard, -Lightning Network, and other production systems. FIPS uses the IK pattern for -link authentication and end-to-end sessions. - -**Cryptographic primitives**: FIPS reuses Nostr's cryptographic stack—secp256k1 -for keys, Schnorr signatures, SHA-256 for hashing, and ChaCha20-Poly1305 for -authenticated encryption. No novel cryptography. - -**Session management**: The index-based session dispatch follows WireGuard's -approach, enabling O(1) packet routing without relying on source addresses. +Internally, each node is organized in the three protocol layers. At the top, +two application interfaces provide access to the mesh: a native datagram API +addressed by npub, and an IPv6 TUN adapter that maps npubs to `fd::/8` +addresses so unmodified IP applications can use the network transparently. The +FSP and FLP layers in the middle implement session management, routing, and +encryption. At the bottom, transport plugins handle the physical diversity — +each plugin implements the same interface, so the router treats UDP, Ethernet, +LoRa, Tor, and serial links identically. Adding a new transport requires no +changes to the routing or session layers. --- @@ -125,7 +230,7 @@ The FIPS address (synonymous with the pubkey) is the primary means for application-layer software to identify communication endpoints. The bech32-encoded npub can be used interchangeably for user interface purposes. The FIPS datagram service is exposed to the application layer either via a -native API to the FIPS node software, or through an IPv6 shim driver that +native API to the FIPS node software, or through an IPv6 adaptation layer that converts the node identity into an IPv6 address and provides DNS resolution from npub to this address for traditional software. @@ -133,70 +238,70 @@ from npub to this address for traditional software. ![Identity Derivation](fips-identity-derivation.svg) -The pubkey is the node's cryptographic identity, used in Noise IK handshakes for -both link and session encryption. It is never exposed beyond the endpoints of an -encrypted channel. The node_addr, a one-way SHA-256 hash truncated to 16 bytes, -serves as the routing identifier in packet headers and bloom filters. Intermediate -routers see only node_addrs — they can forward traffic without learning the Nostr -identities of the endpoints. An observer can verify "does this node_addr belong -to pubkey X?" but cannot enumerate which pubkeys are communicating by inspecting -traffic. The IPv6 address prepends `fd` to the first 15 bytes of the node_addr, -providing an overlay address for unmodified IP applications via the TUN interface. +The pubkey is the node's cryptographic identity, used in Noise IK handshakes +for both link and session encryption. It is never exposed beyond the endpoints +of an encrypted channel. The node_addr, a one-way SHA-256 hash truncated to +16 bytes, serves as the routing identifier in packet headers and bloom filters. +Intermediate routers see only node_addrs — they can forward traffic without +learning the Nostr identities of the endpoints. An observer can verify "does +this node_addr belong to pubkey X?" but cannot enumerate which pubkeys are +communicating by inspecting traffic. The IPv6 address prepends `fd` to the +first 15 bytes of the node_addr, providing an overlay address for unmodified +IP applications via the TUN interface. ### Address Format -When using the IPv6 protocol adapter, FIPS addresses use the IPv6 Unique Local +When using the IPv6 adaptation layer, FIPS addresses use the IPv6 Unique Local Address (ULA) prefix `fd00::/8`, providing 120 bits from the node_addr hash. -These are overlay identifiers—they appear in the TUN interface for application +These are overlay identifiers — they appear in the TUN interface for application compatibility but are not routable on the underlying transport. The fd prefix ensures no collision with addresses that may be in use on the transport network. -FIPS provides a local DNS service that maps npub bech32 names to IPv6 addresses -for this purpose. ### Identity Verification The Noise Protocol Framework is used to mutually authenticate both peer-to-peer -link connections and end-to-end session traffic, proving each party controls -the private key for their claimed identity. +link connections (at FLP) and end-to-end session traffic (at FSP), proving each +party controls the private key for their claimed identity. -See [fips-wire-protocol.md](fips-wire-protocol.md) for the Noise IK handshake -and [fips-session-protocol.md](fips-session-protocol.md) for end-to-end -session establishment. +See [fips-link-layer.md](fips-link-layer.md) for peer authentication and +[fips-session-layer.md](fips-session-layer.md) for end-to-end session +establishment. ### Terminology: Addresses and Identifiers FIPS uses several related but distinct identifiers at different protocol layers: -| Term | Layer | Visible To | Description | -|----------------------------|---------------------|----------------|----------------------------------------------------------------------| -| **FIPS address / pubkey** | Application/Session | Endpoints only | 32-byte secp256k1 public key - the endpoint identity | -| **npub** | (encoding) | Human readers | Bech32 encoding of pubkey for display/config | -| **node_addr** | Routing | Routing nodes | SHA-256(pubkey) truncated to 128 bits - cannot be reversed to pubkey | -| **link_addr** | Transport | Direct peers | IP:port, MAC, .onion - transport-specific | -| **IPv6 address** | IPv6 shim | Applications | fd::/8 derived from node_addr - optional compatibility | +| Term | Layer | Visible To | Description | +| ---- | ----- | ---------- | ----------- | +| **FIPS address / pubkey** | Application/FSP | Endpoints only | 32-byte secp256k1 public key — the endpoint identity | +| **npub** | (encoding) | Human readers | Bech32 encoding of pubkey for display/config | +| **node_addr** | FLP (routing) | Routing nodes | SHA-256(pubkey) truncated to 128 bits — cannot be reversed to pubkey | +| **link_addr** | Transport | Direct peers | IP:port, MAC, .onion — transport-specific | +| **IPv6 address** | IPv6 adapter | Applications | fd::/8 derived from node_addr — optional compatibility | -**Privacy property**: The pubkey (FIPS address / Nostr identity) is never exposed to -intermediate routing nodes. They see only the node_addr, a one-way hash. An observer -can verify "does this node_addr belong to pubkey X?" but cannot derive the pubkey from -traffic. +**Privacy property**: The pubkey (FIPS address / Nostr identity) is never +exposed to intermediate routing nodes. They see only the node_addr, a one-way +hash. An observer can verify "does this node_addr belong to pubkey X?" but +cannot derive the pubkey from traffic. --- ## Two-Layer Encryption -FIPS uses independent encryption at two layers: +FIPS uses independent encryption at two protocol layers: -| Layer | Scope | Pattern | Purpose | -|-------------|-------------|----------|------------------------------------------------| -| **Link** | Hop-by-hop | Noise IK | Encrypt all traffic on each link | -| **Session** | End-to-end | Noise IK | Encrypt application payload between endpoints | +| Layer | Scope | Pattern | Purpose | +| ----- | ----- | ------- | ------- | +| **FLP (Link)** | Hop-by-hop | Noise IK | Encrypt all traffic on each peer link | +| **FSP (Session)** | End-to-end | Noise IK | Encrypt application payload between endpoints | ### Link Layer (Hop-by-Hop) -When two nodes establish a direct connection, they perform a Noise IK handshake. -This authenticates both parties and establishes symmetric keys for encrypting -all traffic on that link. Every packet between direct peers is encrypted—gossip -messages, routing queries, and forwarded session datagrams alike. +When two nodes establish a direct connection, they perform a Noise IK +handshake. This authenticates both parties and establishes symmetric keys for +encrypting all traffic on that link. Every packet between direct peers is +encrypted — gossip messages, routing queries, and forwarded session datagrams +alike. The IK pattern is used because outbound connections know the peer's npub from configuration, while inbound connections learn the initiator's identity from @@ -207,188 +312,130 @@ the first handshake message. FIPS establishes end-to-end encrypted sessions between any two communicating nodes using Noise IK, regardless of whether they are direct peers or separated by intermediate routers. The initiator knows the destination's npub; the -responder learns the initiator's identity from the handshake—the same asymmetry -as link-layer connections. +responder learns the initiator's identity from the handshake — the same +asymmetry as link-layer connections. Both layers always apply. For adjacent peers, application traffic is encrypted twice: once by the session layer (end-to-end) and once by the link layer (hop-by-hop). This uniform model means: - No special case for "local peer" vs "remote destination" -- Topology changes (a direct peer becomes reachable only through intermediaries) - don't affect sessions +- Topology changes (a direct peer becomes reachable only through + intermediaries) don't affect sessions - The link layer remains purely a transport concern A packet from A to adjacent peer B: -1. A encrypts payload with A↔B session key -2. A wraps in SessionDatagram, encrypts with A↔B link key, sends to B +1. A encrypts payload with A↔B session key (FSP) +2. A wraps in SessionDatagram, encrypts with A↔B link key (FLP), sends to B 3. B decrypts link layer, then decrypts session layer to get payload A packet from A to D through intermediate node B: -1. A encrypts payload with A↔D session key -2. A wraps in SessionDatagram, encrypts with A↔B link key, sends to B -3. B decrypts link layer, sees destination, re-encrypts with B↔D link key +1. A encrypts payload with A↔D session key (FSP) +2. A wraps in SessionDatagram, encrypts with A↔B link key (FLP), sends to B +3. B decrypts link layer, reads destination, re-encrypts with B↔D link key 4. D decrypts link layer, then decrypts session layer to get payload -Intermediate nodes can route based on destination address but cannot read +Intermediate nodes can route based on destination node_addr but cannot read session-layer payloads. -FIPS session setup also warms up route caches along the path between the -endpoints, so that when application traffic flows the network is already ready. - -See [fips-wire-protocol.md](fips-wire-protocol.md) for link encryption and -[fips-session-protocol.md](fips-session-protocol.md) for session encryption. +See [fips-link-layer.md](fips-link-layer.md) for link encryption and +[fips-session-layer.md](fips-session-layer.md) for session encryption. --- -## Spanning Tree Protocol +## Routing and Mesh Operation + +FIPS routing is entirely distributed — each node makes forwarding decisions +using only local information. There are no routing tables pushed from above, no +link-state floods, and no distance-vector exchanges. Instead, two complementary +mechanisms provide the information each node needs. + +### Spanning Tree: The Coordinate System ![Mesh Topology](fips-mesh-topology.svg) -The spanning tree is a subset of the full mesh network that connects all nodes, -forming a tree structure rooted at a deterministically-elected node. Each node -selects a single parent, and the resulting tree serves as the routing backbone. -This enables routing without global routing tables. +Nodes self-organize into a spanning tree rooted at a deterministically-elected +node (the one with the smallest node_addr). Each node selects a single parent +from among its direct peers, and the resulting tree gives every node a +**coordinate** — its path from itself to the root. -### Why a Spanning Tree? +These coordinates enable distance calculations between any two nodes: the +distance is the number of hops from each node to their lowest common ancestor +in the tree. This provides a metric for routing decisions without any node +needing to know the full network topology. -A spanning tree has useful properties: +The tree maintains itself through gossip — nodes exchange TreeAnnounce messages +with their peers, propagating parent selections and ancestry chains. Changes +cascade through the tree proportional to depth, not network size. If the +network partitions, each segment elects its own root and reconverges +automatically when segments rejoin. -- **Unique paths**: Exactly one path exists between any two nodes -- **Minimal state**: Nodes only track their parent and immediate peers -- **Coordinates**: A node's position in the tree enables distance calculations +See [fips-spanning-tree.md](fips-spanning-tree.md) for the tree algorithms and +[spanning-tree-dynamics.md](spanning-tree-dynamics.md) for detailed convergence +walkthroughs. -The tree provides structure for routing while bloom filters provide reachability -information. Together they enable efficient packet delivery without requiring -nodes to know the full network topology. +### Bloom Filters: Candidate Selection -### Tree Coordinates +Each node maintains bloom filters summarizing which destinations are reachable +through each of its peers. Bloom filters propagate via gossip, with each node +computing outbound filters by merging the filters received from its other +peers. At steady state, filters represent the entire reachable network. -A node's coordinate is its path from itself to the root. The distance between -two nodes is the sum of hops from each to their lowest common ancestor (LCA). -This distance metric enables greedy routing: forward packets to the peer that -minimizes distance to the destination. +Bloom filters answer a single question: "can peer P possibly reach destination +D?" The answer is either "no" (definitive) or "maybe" (probabilistic — false +positives are possible). This is **candidate selection**, not routing — bloom +filters identify which peers are worth considering, but the actual forwarding +decision requires tree coordinates to rank those candidates by distance. -### Root Election +See [fips-bloom-filters.md](fips-bloom-filters.md) for filter parameters and +mathematical properties. -The root is the node with the lexicographically smallest node_addr among all -reachable nodes. This election is deterministic and requires no coordination— -each node independently examines its view of the network and reaches the same -conclusion. +### Routing Decisions -The root provides a coordinate reference point but does not participate in -routing unless the paths from source and destination to the root share no -common ancestors (i.e., the root is their lowest common ancestor). +At each hop, FLP makes a local forwarding decision using the following priority +chain: -### Parent Selection +1. **Local delivery** — the destination is this node +2. **Direct peer** — the destination is an authenticated neighbor +3. **Bloom-guided candidate selection** — bloom filters identify peers that can + reach the destination; tree coordinates rank them by distance +4. **Greedy tree routing** — fallback when bloom filters haven't converged; + forward to the peer that minimizes tree distance to the destination +5. **No route** — destination unreachable; send error signal to source -Each node selects a parent that provides the best path to root, considering: +All multi-hop routing depends on knowing the destination's tree coordinates. +These are cached at each node after being learned through discovery +(LookupRequest/LookupResponse) or session establishment (SessionSetup). The +coordinate cache is the critical piece that enables efficient forwarding. -- Reachability (the parent must have a path to root) -- Link quality (latency, packet loss, bandwidth) -- Stability (hysteresis prevents flapping on minor changes) +### Coordinate Caching and Discovery -The parent must be a direct peer—nodes cannot select non-peers as parents. +When a node first needs to reach an unknown destination, it sends a +LookupRequest that floods through the network guided by bloom filters. The +destination responds with its coordinates, which the source caches. Subsequent +traffic routes efficiently using the cached coordinates. -### Tree Gossip +Session establishment (SessionSetup) also carries coordinates, warming transit +node caches along the path so that data packets can be forwarded without +individual discovery at each hop. -Nodes exchange TreeAnnounce messages containing their parent selection and -ancestry chain (path to root). When a node changes its parent, it announces the -change; peers propagate relevant updates. The tree converges through this -gossip without centralized coordination. +### Error Recovery -Changes propagate only as far as they need to—distantly connected nodes are -unaffected by local path changes and don't receive updates for them. +When routing fails — because cached coordinates are stale or a path has +broken — transit nodes signal the source: -### Partition Handling +- **CoordsRequired**: A transit node lacks the destination's coordinates. + The source re-initiates discovery and resets its coordinate warmup strategy. +- **PathBroken**: Greedy routing reached a dead end. The source re-discovers + the destination's current coordinates. -If the network partitions, each isolated segment elects its own root (the -smallest node_addr within that segment). When partitions merge, nodes in the -segment with the larger root discover the globally smaller root and re-parent. -The tree reconverges automatically. +Both signals trigger active recovery, and are rate-limited to prevent storms +during topology changes. -See [fips-routing.md](fips-routing.md) for routing concepts, -[fips-gossip-protocol.md](fips-gossip-protocol.md) for message formats, and -[spanning-tree-dynamics.md](spanning-tree-dynamics.md) for convergence behavior. - ---- - -## Bloom Filter Routing - -Tree coordinates enable routing once you know a destination's position. Bloom -filters enable finding that position in the first place. - -A bloom filter is a space-efficient probabilistic data structure that can test -whether an element is a member of a set. It may produce false positives (saying -an element is present when it isn't) but never false negatives. This makes it -ideal for routing: a node can quickly check if a destination might be reachable -through a given peer, with occasional false positives handled by backtracking. - -### How It Works - -Each node maintains bloom filters summarizing which node_addrs are reachable -through each of its peers. These filters propagate through the tree: a node -aggregates filters from its children and announces the combined filter to its -parent (and vice versa). - -When a node needs to reach an unknown destination: - -1. Check local bloom filters—which peers might be able to reach this node_addr? -2. Send a LookupRequest to peers whose filters indicate "maybe" -3. The request propagates through the tree toward matching subtrees -4. The destination responds with a LookupResponse containing its coordinates -5. The sender caches the coordinates and routes directly via greedy forwarding - -Bloom filters have false positives (a filter may indicate "maybe" when the node -isn't actually reachable through that path) but no false negatives. Extra -queries are harmless; missing a reachable node is not. - -### Filter Propagation - -Filters propagate in the opposite direction from tree announcements: - -- Tree state propagates upward (toward root) via ancestry chains -- Bloom filters propagate downward (toward leaves) via subtree aggregation - -A node's filter contains all node_addrs reachable through its subtree. The root's -filter contains everyone; leaf nodes have empty outbound filters. - -See [fips-routing.md](fips-routing.md) for bloom filter design and -[fips-gossip-protocol.md](fips-gossip-protocol.md) for FilterAnnounce format. - ---- - -## Greedy Routing - -Once a destination's tree coordinates are known, packets are forwarded using -greedy routing: at each hop, forward to the peer that minimizes tree distance -to the destination. - -### The Algorithm - -1. If I am the destination, deliver the packet locally -2. Calculate my tree distance to the destination -3. For each peer, calculate their tree distance to the destination -4. Forward to the peer with the smallest distance (must be less than mine) -5. If no peer is closer, routing has failed (local minimum) - -### Path-Broken Recovery - -Greedy routing can fail if the destination has moved or the cached coordinates -are stale. When this happens, the node that cannot make progress sends a -PathBroken notification back to the source. The source then initiates a fresh -bloom filter lookup to find the destination's current coordinates. - -### Session Establishment - -For efficiency, FIPS establishes routing sessions that cache coordinate -information at intermediate routers. The first packet (SessionSetup) carries -full coordinates; subsequent packets use cached state for minimal overhead. - -See [fips-routing.md](fips-routing.md) for the complete routing design and -[fips-session-protocol.md](fips-session-protocol.md) for session establishment. +See [fips-mesh-operation.md](fips-mesh-operation.md) for the complete routing +and mesh behavior description. --- @@ -397,84 +444,33 @@ See [fips-routing.md](fips-routing.md) for the complete routing design and FIPS is transport-agnostic. The protocol operates identically whether peers connect over UDP, Ethernet, LoRa radio, serial cables, or Tor hidden services. -### Transports and Links - A **transport** is a physical or logical interface: a UDP socket, an Ethernet -NIC, a Tor client, a radio modem. A **link** is a connection instance to a -specific peer over a transport. - -### Multi-Transport Bridging +NIC, a Tor client, a radio modem. A **link** is a peer connection established +over a transport. Transport addresses (IP:port, MAC address, .onion) are opaque +to all layers above FLP — they are used only to deliver datagrams and are +discarded once FLP has authenticated the peer. A node with multiple transports automatically bridges between networks. Peers from all transports feed into a single spanning tree; the router selects the best path regardless of transport type. If one transport fails, traffic automatically routes through alternatives. -### Transport Types - | Category | Examples | Characteristics | -|----------|----------|-----------------| -| Overlay | UDP/IP, TCP/TLS, QUIC, WebSocket | Internet connectivity, NAT considerations | -| Shared medium | Ethernet, WiFi, Bluetooth, LoRa | Broadcast/multicast discovery | -| Point-to-point | Serial, dialup | No discovery needed, static config | +| -------- | -------- | --------------- | +| Overlay | UDP/IP, TCP/TLS, WebSocket | Internet connectivity, NAT traversal | +| Shared medium | Ethernet, WiFi, Bluetooth, LoRa | Local discovery, broadcast | +| Point-to-point | Serial, dialup | Static config, no discovery | | Anonymity | Tor, I2P | High latency, strong privacy | -UDP over IP is expected to be the most common transport for internet-connected -nodes. Radio transports enable connectivity where internet infrastructure is -unavailable. +> **Implementation status**: UDP/IP is the only implemented transport. All +> others are future directions. -See [fips-transports.md](fips-transports.md) for transport characteristics. +See [fips-transport-layer.md](fips-transport-layer.md) for transport layer +design. --- -## Protocol Messages - -FIPS uses a discriminator-based wire format for efficient message dispatch. - -### Link Layer Messages - -Exchanged between directly connected peers, encrypted with link session keys: - -| Type | Name | Purpose | -|------|----------------|------------------------------------------------------| -| 0x10 | TreeAnnounce | Spanning tree state (parent, ancestry) | -| 0x20 | FilterAnnounce | Bloom filter reachability update | -| 0x30 | LookupRequest | Query for node's tree coordinates | -| 0x31 | LookupResponse | Response with coordinates and proof | -| 0x40 | SessionDatagram| Carries session-layer payloads through the mesh | -| 0x50 | Disconnect | Orderly disconnect notification | - -### SessionDatagram Payload Messages - -Carried inside SessionDatagram (0x40), which provides src_addr, dest_addr, and -hop_limit for multi-hop forwarding. - -**Session-layer messages** (end-to-end encrypted between source and destination): - -| Type | Name | Purpose | -|------|----------------|--------------------------------------------| -| 0x00 | SessionSetup | Establish routing session with coordinates | -| 0x01 | SessionAck | Acknowledge session establishment | -| 0x10 | DataPacket | Encrypted application data (IPv6 payload) | - -**Link-layer error signals** (plaintext, generated by transit routers): - -| Type | Name | Purpose | -|------|----------------|--------------------------------------------| -| 0x20 | CoordsRequired | Router cache miss—need fresh coordinates | -| 0x21 | PathBroken | Greedy routing failed—need re-lookup | - -Error signals are sent back to the original source using the `src_addr` from -the SessionDatagram that triggered the error. They are plaintext because the -transit router has no end-to-end session with the source; link-layer encryption -protects them hop-by-hop. - -See [fips-wire-protocol.md](fips-wire-protocol.md) for wire format details and -[fips-gossip-protocol.md](fips-gossip-protocol.md) for gossip message formats. - ---- - -## Security Considerations +## Security ### Threat Model @@ -493,79 +489,105 @@ sees only encrypted packets. **End-to-end encryption**: Session-layer Noise IK encrypts payloads between endpoints. Intermediate routers cannot read application data. -**Signature verification**: All protocol messages (TreeAnnounce, LookupResponse) -are signed. The full ancestry chain in TreeAnnounce includes signatures from -each node, preventing forged tree positions. +**Signature verification**: TreeAnnounce messages carry signed parent +declarations. Direct peers verify signatures using keys established during +the Noise IK handshake; ancestry beyond direct peers uses transitive trust +in the v1 protocol. -**Replay protection**: Sequence numbers and timestamps on announcements. -Counter-based nonces with sliding window for encrypted packets. +**Replay protection**: Counter-based nonces with sliding window for encrypted +packets at both the link and session layers. Sequence numbers on protocol +announcements prevent replay of stale state. ### Sybil Resistance Creating many identities is cheap, but exploiting them is constrained: -- **Discretionary peering**: Node operators choose who to peer with. An attacker - with many identities still needs real nodes to accept their connections. +- **Discretionary peering**: Node operators choose who to peer with. An + attacker with many identities still needs real nodes to accept their + connections. - **Tree coordinate verification**: Nodes cannot claim arbitrary tree positions - without valid signed ancestry chains from real nodes + without valid signed ancestry chains from real nodes. - **Rate limiting**: Handshake rate limiting constrains how fast attackers can - establish connections + establish connections. ### Metadata Exposure Each entity in the network sees different information: | Entity | Can See | -|--------|---------| +| ------ | ------- | | Transport observer | Encrypted packets, timing, packet sizes | | Direct peer | Your npub (identity), traffic volume, timing | | Intermediate router | Source and destination node_addrs, packet size | | Destination | Your npub (identity), payload content | -Intermediate routers see node_addrs, not npubs. Since node_addrs are derived from -pubkeys via one-way SHA-256 hash, routers cannot determine the actual identities -of the endpoints they route for. +Intermediate routers see node_addrs, not npubs. Since node_addrs are derived +from pubkeys via one-way SHA-256 hash, routers cannot determine the actual +identities of the endpoints they route for. The session layer hides payload content from intermediate routers. The link layer hides everything from passive observers on the underlying transport. --- -## Conclusion +## Prior Work -FIPS combines these elements into a cohesive system that achieves its design -goals: +FIPS builds on proven designs rather than inventing new cryptography or routing +algorithms. -- **Self-sovereign identity** through Nostr keypairs, with node_addrs providing - routing-level privacy -- **Transport agnosticism** via the transport abstraction layer, enabling the - same routing logic across UDP, Ethernet/WiFi, Tor, and other link types -- **Self-organization** through distributed spanning tree formation and bloom - filter propagation, requiring no central coordination -- **Privacy preservation** with two-layer encryption that hides payloads from - intermediate routers and hides everything from transport observers -- **Resilience** through automatic partition detection, re-election, and tree - reconvergence when the network topology changes +**Routing**: The spanning tree coordinates, bloom filter candidate selection, +and greedy routing algorithms are adapted from +[Yggdrasil v0.5](https://yggdrasil-network.github.io/2023/10/22/upcoming-v05-release.html) +and its [Ironwood](https://github.com/Arceliar/ironwood) routing library. FIPS +adapts these for multi-transport operation and Nostr identity integration. -The result is a mesh network where nodes can find and communicate with each -other securely, regardless of the underlying transport infrastructure, while -maintaining control over their own identities and peering relationships. +**Encryption**: Link and session encryption use the +[Noise Protocol Framework](https://noiseprotocol.org/), the same foundation +used by WireGuard, Lightning Network, and other production systems. FIPS uses +the IK pattern for both link authentication and end-to-end sessions. + +**Cryptographic primitives**: FIPS reuses Nostr's cryptographic stack — +secp256k1 for keys, Schnorr signatures, SHA-256 for hashing, and +ChaCha20-Poly1305 for authenticated encryption. No novel cryptography. + +**Session management**: The index-based session dispatch follows WireGuard's +approach, enabling O(1) packet routing without relying on source addresses. --- -## References +## Further Reading -### FIPS Design Documents +### Protocol Layers | Document | Description | -|----------|-------------| -| [fips-session-protocol.md](fips-session-protocol.md) | End-to-end session flow, Noise IK encryption | -| [fips-wire-protocol.md](fips-wire-protocol.md) | Link-layer transport, Noise IK handshake | -| [fips-gossip-protocol.md](fips-gossip-protocol.md) | TreeAnnounce, FilterAnnounce, Lookup formats | -| [fips-routing.md](fips-routing.md) | Bloom filters, discovery, greedy routing | -| [spanning-tree-dynamics.md](spanning-tree-dynamics.md) | Tree protocol dynamics and convergence | -| [fips-transports.md](fips-transports.md) | Transport protocol characteristics | -| [fips-software-architecture.md](fips-software-architecture.md) | Software architecture, configuration | +| -------- | ----------- | +| [fips-transport-layer.md](fips-transport-layer.md) | Transport layer: abstraction, types, services provided to FLP | +| [fips-link-layer.md](fips-link-layer.md) | FLP: peer authentication, link encryption, forwarding | +| [fips-session-layer.md](fips-session-layer.md) | FSP: end-to-end encryption, session lifecycle | +| [fips-ipv6-adapter.md](fips-ipv6-adapter.md) | IPv6 adaptation: DNS, TUN interface, MTU enforcement | + +### Mesh Behavior and Wire Formats + +| Document | Description | +| -------- | ----------- | +| [fips-mesh-operation.md](fips-mesh-operation.md) | How the mesh operates: routing, discovery, error recovery | +| [fips-wire-formats.md](fips-wire-formats.md) | Complete wire format reference for all protocol layers | + +### Supporting References + +| Document | Description | +| -------- | ----------- | +| [fips-spanning-tree.md](fips-spanning-tree.md) | Spanning tree algorithms and data structures | +| [fips-bloom-filters.md](fips-bloom-filters.md) | Bloom filter parameters, math, and computation | +| [spanning-tree-dynamics.md](spanning-tree-dynamics.md) | Scenario walkthroughs: convergence, partitions, recovery | + +### Implementation + +| Document | Description | +| -------- | ----------- | +| [fips-software-architecture.md](fips-software-architecture.md) | Stable architectural decisions guiding the codebase | +| [fips-state-machines.md](fips-state-machines.md) | Phase-based state machine pattern (Rust) | +| [fips-configuration.md](fips-configuration.md) | YAML configuration reference | ### External References diff --git a/docs/design/fips-ipv6-adapter.md b/docs/design/fips-ipv6-adapter.md new file mode 100644 index 0000000..0dd5357 --- /dev/null +++ b/docs/design/fips-ipv6-adapter.md @@ -0,0 +1,325 @@ +# FIPS IPv6 Adapter + +The IPv6 adapter sits above the FIPS Session Protocol (FSP) and adapts the +FIPS datagram service for unmodified IPv6 applications. It presents each FIPS +node as an IPv6 endpoint, so standard socket applications (SSH, HTTP, SCP) +can communicate over the mesh without modification. + +## Role + +The adapter bridges two worlds: IPv6 applications that address destinations by +IP address, and the FIPS mesh that addresses destinations by public key +(npub). The adapter handles the translation: DNS resolution from npub to +`fd::/8` address, identity cache management so FIPS can route IPv6 packets, +MTU enforcement so packets fit through the mesh, and the TUN interface that +connects to the kernel's IPv6 stack. + +Applications that are FIPS-aware can bypass the adapter entirely and use the +native FIPS datagram API, addressing destinations directly by npub. + +## DNS Integration + +### The Problem + +IPv6 addresses in the `fd::/8` range are derived from public keys via a +one-way hash (SHA-256). Given only an IPv6 address, the public key cannot be +recovered — and without the public key, FIPS cannot compute the node_addr +needed for routing. + +The identity cache must be populated *before* packets arrive at the TUN +interface, or they cannot be routed. + +### DNS as Entry Point + +DNS resolution serves as the "routing intent" signal. When an application +resolves `npub1xxx...xxx.fips`, the FIPS DNS service: + +1. Extracts the npub from the `.fips` domain name +2. Derives the `fd::/8` IPv6 address from the public key +3. Primes the identity cache with the mapping + (IPv6 address prefix ↔ NodeAddr ↔ PublicKey) +4. Returns the IPv6 address to the application + +When the application subsequently sends packets to that address, the identity +cache already contains the mapping needed for routing. + +### DNS Name Format + +```text +npub1xxxxxx...xxxxx.fips +``` + +The FIPS DNS server recognizes names ending in `.fips` and extracts the npub +for address derivation. + +### Traffic Without Prior DNS Lookup + +A packet may arrive at the TUN for an `fd::/8` destination without a prior DNS +lookup — cached address, manual configuration, etc. Since the address derivation +is one-way, the npub cannot be recovered from the address alone. + +FIPS returns ICMPv6 Destination Unreachable (Code 0: No route to destination) +for packets to unknown addresses. The identity cache must be populated before +traffic can be routed. + +Known cache population mechanisms: + +- **DNS lookup**: The primary path +- **Inbound traffic**: Authenticated sessions from other nodes populate the + cache with their identity information + +## IPv6 Address Derivation + +FIPS addresses use the IPv6 Unique Local Address (ULA) prefix `fd00::/8`: + +```text +Public Key (32 bytes) + │ + ▼ +SHA-256 → node_addr (16 bytes, truncated) + │ + ▼ +fd + node_addr[0..15] → IPv6 address (16 bytes) +``` + +The `fd` prefix ensures no collision with addresses in use on the underlying +transport network. These are overlay identifiers — they appear in the TUN +interface for application compatibility but are not routable on the underlying +transport. + +## Identity Cache + +The identity cache maps the FIPS address prefix (15 bytes — the IPv6 address +minus the `fd` prefix) to `(NodeAddr, PublicKey)`. This cache is needed only +when using the IPv6 adapter; the native FIPS API provides the public key +directly. + +### Eviction Policy + +The mapping is deterministic (derived from the public key) and never becomes +stale. The cache uses **LRU-only eviction** bounded by a configurable size +(default 10K entries). There is no TTL — entries are evicted only when the +cache is full and space is needed for a new entry. + +### Relationship to DNS TTL + +The identity cache timeout must be longer than the DNS TTL to ensure that while +an application believes its DNS resolution is valid, the corresponding routing +entry remains present. The DNS TTL (default 300s) governs when applications +re-query; the identity cache (LRU, no TTL) is always available as long as the +entry hasn't been evicted by memory pressure. + +## MTU Enforcement + +FIPS encapsulation adds overhead to every packet. The adapter must ensure +that IPv6 packets from applications fit within the FIPS encapsulation budget +after all layers of wrapping. + +### Encapsulation Overhead + +| Layer | Overhead | Purpose | +| ----- | -------- | ------- | +| Link encryption | 29 bytes | discriminator + receiver_idx + counter + AEAD tag | +| SessionDatagram envelope | 34 bytes | type + src_addr + dest_addr + hop_limit | +| DataPacket header | 12 bytes | type + flags + counter + payload_len | +| Session encryption | 16 bytes | ChaCha20-Poly1305 AEAD tag | +| **Minimal total** | **91 bytes** | | +| Coordinates (if present) | ~44 bytes | Depth-dependent, first few packets only | +| **Worst case total** | **135 bytes** | With COORDS_PRESENT for depth-3 paths | + +The `FIPS_OVERHEAD` constant (135 bytes) is used for conservative MTU +calculations. + +### Effective IPv6 MTU + +The effective IPv6 MTU visible to applications is: + +```text +effective_ipv6_mtu = transport_mtu - FIPS_OVERHEAD +``` + +For typical deployments: + +| Transport MTU | Effective IPv6 MTU | Notes | +| ------------- | ------------------ | ----- | +| 1472 (UDP/Ethernet) | 1337 | Standard deployment | +| 1280 (UDP minimum) | 1145 | Below IPv6 minimum | + +IPv6 mandates that every link support at least 1280 bytes. The minimum +transport path MTU for the IPv6 adapter is therefore: + +```text +1280 + 135 = 1415 bytes +``` + +Transports with smaller MTUs (LoRa at ~250 bytes, serial at 256 bytes) cannot +support the IPv6 adapter — applications on those transports must use the +native FIPS datagram API. + +### ICMP Packet Too Big + +When an outbound packet at the TUN exceeds the effective IPv6 MTU, the adapter +generates an ICMPv6 Packet Too Big message and delivers it back to the +application via the TUN. This triggers the kernel's Path MTU Discovery (PMTUD) +mechanism, which adjusts TCP segment sizes for subsequent transmissions. + +ICMP Packet Too Big generation is rate-limited per source address (100ms +interval) to prevent storms from applications sending many oversized packets. + +The ICMP response is delivered locally (back through the TUN to the kernel) — +no network traversal is needed, so delivery is reliable. + +### TCP MSS Clamping + +The adapter intercepts TCP SYN and SYN-ACK packets at the TUN interface and +clamps the Maximum Segment Size (MSS) option: + +```text +clamped_mss = effective_ipv6_mtu - 40 (IPv6 header) - 20 (TCP header) +``` + +This prevents TCP connections from negotiating segment sizes that would exceed +the FIPS path MTU. Clamping is applied in two places: + +- **TUN reader** (outbound): Clamps MSS on outbound SYN packets +- **TUN writer** (inbound): Clamps MSS on inbound SYN-ACK packets + +Together, these ensure both directions of a TCP connection use appropriately +sized segments from the start, avoiding the initial oversized packet loss +that would occur with ICMP Packet Too Big alone. + +### ICMP Rate Limiting + +ICMPv6 error generation is rate-limited per source address using a token bucket +(100ms interval). This matches the standard ICMP rate limiting approach and +prevents amplification when an application sends a burst of oversized packets. + +## TUN Interface + +The TUN device (`fips0`) is the mechanism that connects the adapter to the +kernel's IPv6 stack. It is an implementation detail of the adapter, not its +defining feature. + +### Architecture + +```text +Applications (sockets using fd::/8 addresses) + │ + ▼ +Kernel IPv6 Stack (routing: fd::/8 → fips0) + │ + ▼ +TUN Device (fips0) + ├── Reader Thread (blocking I/O → packet processing) + └── Writer Thread (mpsc queue → TUN writes) +``` + +### Reader Thread + +The TUN reader receives raw IPv6 packets from applications and processes them: + +1. Validate IPv6 header +2. Extract destination `fd::/8` address +3. Look up identity cache — miss returns ICMPv6 Destination Unreachable +4. Retrieve NodeAddr and PublicKey from cache +5. Look up or establish FSP session +6. Encrypt payload with session keys +7. Route through FLP toward destination + +### Writer Thread + +A single writer thread services an mpsc queue of outbound packets: + +- Inbound mesh traffic (decrypted session payloads destined for local + applications) +- ICMPv6 error responses (Packet Too Big, Destination Unreachable) +- TCP MSS-clamped SYN-ACK packets + +The queue-based design eliminates contention on TUN writes and cleanly +separates concerns. New packet sources can be added by cloning the sender +handle. + +### Local Address Guarantee + +The Linux kernel routing table processes rules in priority order: + +1. **Local table**: Intercepts traffic to addresses assigned to this machine +2. **Main table**: Routes `fd::/8` to the TUN device + +This means every packet arriving at the TUN reader is guaranteed to be for a +*remote* FIPS destination. No "is this for me?" check is needed on the read +path. + +### Configuration + +```yaml +tun: + enabled: true + name: fips0 + mtu: 1280 +``` + +### Privileges + +TUN device creation requires `CAP_NET_ADMIN`. Options: + +- Run as root +- Set capability: `sudo setcap cap_net_admin+ep ./target/debug/fips` +- Pre-created persistent TUN device + +## Implementation Status + +| Feature | Status | +| ------- | ------ | +| TUN device creation and configuration | **Implemented** | +| IPv6 address assignment (netlink) | **Implemented** | +| TUN reader/writer threads | **Implemented** | +| ICMPv6 Destination Unreachable | **Implemented** | +| ICMPv6 Packet Too Big | **Implemented** | +| ICMP rate limiting (per-source) | **Implemented** | +| TCP MSS clamping (SYN + SYN-ACK) | **Implemented** | +| DNS service (.fips domain) | Planned | +| Per-destination route MTU (netlink) | Planned | +| Transit MTU error signal | Planned | +| Path MTU discovery (envelope field) | Future direction | +| Endpoint fragmentation/reassembly | Future direction | + +## Design Considerations + +### Path MTU Discovery (Planned) + +Two complementary mechanisms are planned for full PMTUD: + +1. **Proactive**: A `path_mtu` field (2 bytes) in the SessionDatagram envelope. + The source sets it to its outbound link MTU minus overhead; each transit + node applies `min(current, own_outbound_mtu - overhead)`. The destination + receives the forward-path minimum. A session-layer echo (2 bytes inside + encryption) returns the value to the source. + +2. **Reactive**: When a transit node cannot forward a packet (MTU exceeded), it + sends an error signal back to the source. This handles the in-flight gap + between a path MTU decrease and the source learning via the echo. + +Both are needed: proactive handles steady state; reactive handles the transient +window when oversized packets hit a new bottleneck before the source adapts. + +### No Fragmentation + +FIPS remains a pure datagram service with no fragmentation at transit nodes. +Session-layer encryption is end-to-end — the AEAD tag authenticates the entire +plaintext. Fragmenting encrypted datagrams would require either exposing +plaintext structure to transit nodes (unacceptable) or reassembly before +decryption (opens attack surface). + +Endpoint-only fragmentation (fragment before session encryption, reassemble +after decryption) is a future direction that avoids these objections. Each +fragment would be independently encrypted and look like a normal DataPacket +to transit nodes. + +## References + +- [fips-intro.md](fips-intro.md) — Protocol overview and architecture +- [fips-session-layer.md](fips-session-layer.md) — FSP (below the adapter) +- [fips-wire-formats.md](fips-wire-formats.md) — DataPacket and SessionDatagram + formats +- [fips-configuration.md](fips-configuration.md) — TUN configuration parameters diff --git a/docs/design/fips-link-layer.md b/docs/design/fips-link-layer.md new file mode 100644 index 0000000..eb2f2aa --- /dev/null +++ b/docs/design/fips-link-layer.md @@ -0,0 +1,379 @@ +# FIPS Link Protocol (FLP) + +The FIPS Link Protocol is the middle layer of the FIPS protocol stack. It sits +between the transport layer below and the FIPS Session Protocol (FSP) above. +FLP is where anonymous transport addresses become authenticated peers, where +the mesh self-organizes, and where forwarding decisions are made. + +## Role + +FLP manages direct peer connections over transports. When a transport delivers +a datagram from an unknown address, FLP authenticates the sender through a +Noise IK handshake, establishing a cryptographic link. Once authenticated, the +link carries all inter-peer communication: spanning tree gossip, bloom filter +updates, coordinate discovery, and forwarded session datagrams — all encrypted +per-hop. + +FLP is the boundary between opaque transport addresses and identified peers. +Below FLP, everything is transport-specific addresses (IP:port, MAC, .onion). +Above FLP, everything is peers identified by public keys and routable by +node_addr. The transport layer never sees FIPS-level structure; FSP never sees +transport addresses or routing details. + +## Services Provided to FSP + +From the session layer's perspective, FLP is a black box providing three +services. FSP knows nothing about transports, links, peers, spanning trees, +coordinates, bloom filters, hop counts, or network topology. + +### Datagram Forwarding + +FLP accepts a datagram addressed by source and destination node_addr and +delivers it best-effort toward the destination. The datagram travels hop by +hop — at each node, FLP decrypts the link layer, reads the destination +node_addr, makes a local forwarding decision, and re-encrypts onto the +next-hop link. + +FSP provides: source node_addr, destination node_addr, hop limit, and an +opaque payload (the session-layer encrypted message). + +FLP provides: best-effort delivery. No acknowledgment, no retransmission, no +ordering guarantee. Datagrams may be dropped, duplicated, or delivered out of +order. + +### Error Signaling + +When forwarding fails, FLP signals the source endpoint asynchronously: + +- **CoordsRequired**: A transit node lacks the destination's tree coordinates + and cannot make a forwarding decision. The source should re-initiate + discovery and reset its coordinate warmup strategy. +- **PathBroken**: Greedy routing reached a dead end — no peer is closer to the + destination than the current node. The source should re-discover the + destination's current coordinates. + +Both signals travel inside the SessionDatagram envelope (using the existing +src/dest/hop_limit addressing) but are generated by transit nodes and are not +end-to-end encrypted. They are rate-limited at 100ms per destination to prevent +storms during topology changes. + +### Local Delivery + +When a datagram arrives with a destination node_addr matching the local node, +FLP delivers it up to FSP for session-layer processing. + +## Services Required from Transport Layer + +FLP requires the following from each transport: + +### Datagram Delivery + +Send and receive raw datagrams to/from transport addresses. The transport +handles all medium-specific details. FLP sees only "send bytes to address" and +"bytes arrived from address." + +### MTU Reporting + +The maximum datagram size for a given link. FLP needs this to determine how +much payload fits in a single packet after link encryption overhead (29 bytes +for the encrypted frame wrapper). + +### Connection Lifecycle + +For connection-oriented transports, the transport must establish the underlying +connection before FLP can begin the Noise IK handshake. For connectionless +transports, datagrams can flow immediately. + +### Endpoint Discovery (Optional) + +When a transport discovers a FIPS-capable endpoint (via beacon, query, or +other transport-specific mechanism), it notifies FLP so that link setup can +be initiated. Transports without discovery support provide peer addresses +through configuration. + +## Peer Authentication + +### Noise IK Handshake + +Every peer connection begins with a Noise IK handshake that mutually +authenticates both parties and establishes symmetric keys for link encryption. + +The IK pattern is chosen because: +- The **initiator** knows the responder's static public key from configuration + or discovery, and sends their own static key encrypted in the first message +- The **responder** learns the initiator's identity from the first message, + then responds with their own ephemeral key + +After the two-message handshake completes, both parties share symmetric +session keys derived from four DH operations (es, ss, ee, se). The handshake +provides mutual authentication, forward secrecy, and identity hiding for the +initiator. + +### Identity Binding + +The Noise handshake binds the link to the peer's cryptographic identity. After +handshake completion: + +- The peer's public key (FIPS identity) is confirmed +- The node_addr is computed from the public key (SHA-256, truncated to 128 bits) +- All subsequent traffic on the link is authenticated by the Noise session — + successful decryption proves the sender is the authenticated peer +- The link is registered in the dispatch table for O(1) packet routing + +### Reconnection + +When a Noise IK msg1 arrives from a peer that already has an authenticated +link, FLP accepts the new handshake alongside the existing session. If the new +handshake completes successfully, it replaces the old session. This handles +legitimate reconnection (network change, process restart, NAT rebinding) +without disrupting ongoing traffic until the new session is confirmed. + +## Link Encryption + +All traffic between authenticated peers is encrypted. Every packet on a link +— gossip messages, routing queries, forwarded session datagrams, disconnect +notifications — passes through Noise's ChaCha20-Poly1305 AEAD. + +### Encrypted Frame Structure + +Post-handshake packets are wrapped in an encrypted frame consisting of: +- A discriminator byte identifying the frame type +- A receiver index for O(1) session lookup +- An explicit counter used as the AEAD nonce +- The ciphertext with a Poly1305 authentication tag + +The plaintext inside the encrypted frame begins with a message type byte +followed by the message-specific payload. + +See [fips-wire-formats.md](fips-wire-formats.md) for the complete wire format +specification. + +### What Encryption Provides + +- **Confidentiality**: An observer on the underlying transport sees only + encrypted packets, packet timing, and packet sizes +- **Integrity**: Any modification to a packet is detected by the AEAD tag +- **Authentication**: Only the authenticated peer can produce valid ciphertext + for this link's session keys + +### What Encryption Does Not Provide + +- **End-to-end confidentiality**: Link encryption protects traffic on a single + hop. Each transit node decrypts, reads the routing envelope, and re-encrypts + for the next hop. Session-layer encryption (FSP) provides end-to-end + confidentiality. +- **Traffic analysis protection**: Packet timing, sizes, and volume are visible + to transport-layer observers. + +## Index-Based Session Dispatch + +Incoming packets are dispatched to the correct Noise session using a +receiver index — a random 32-bit value chosen by the receiver during the +handshake. This enables O(1) lookup without relying on source addresses. + +Each party in a link has two indices: +- **our_index**: Chosen by us, included by the peer in packets sent to us +- **their_index**: Chosen by them, included by us in packets sent to them + +The tuple `(transport_id, receiver_idx)` uniquely identifies a session. + +### Index Properties + +- **Random**: Cryptographically random to prevent guessing +- **Unique per transport**: No two active sessions on the same transport share + an index +- **Scoped to transport**: The same index value may appear on different + transports +- **Rotated on rekey**: New indices allocated on rekey to prevent cross-session + correlation by passive observers + +### Dispatch Flow + +1. Read discriminator byte to determine packet type +2. For encrypted frames (0x00): look up `(transport_id, receiver_idx)` in the + session table — O(1) hash lookup. Unknown indices are rejected before any + cryptographic operation. +3. For handshake msg2 (0x02): look up by our sender index to match to a + pending outbound handshake +4. For handshake msg1 (0x01): rate-limited processing, creates new state + +This approach follows WireGuard's design: source address is informational, +not authoritative. Only successful cryptographic verification establishes +authenticity. + +## Roaming + +When an encrypted packet successfully decrypts, the sender is the +authenticated peer regardless of what transport address the packet arrived +from. FLP updates the peer's current address to the packet's source address, +and subsequent outbound packets use the updated address. + +This allows peers to change transport addresses (IP:port for UDP, connection +handle for TCP) without session interruption. The mechanism is: + +1. Packet arrives from a different address than expected +2. Receiver index lookup finds the session +3. AEAD decryption succeeds — the sender is cryptographically authenticated +4. Peer's address is updated to the new source address + +Roaming is most useful for UDP, where source addresses can change due to NAT +rebinding or network changes. For connection-oriented transports, "roaming" +manifests as reconnection rather than mid-session address change. + +## Replay Protection + +Each link session maintains per-direction counters: + +- **Send counter**: Monotonically increasing, used as the AEAD nonce for each + outbound packet +- **Receive window**: A sliding bitmap (2048 entries) tracking which counters + have been seen + +The receive window handles the realities of unreliable transports: packets may +arrive out of order, be duplicated, or be lost. The window accepts any counter +not yet seen and within 2048 of the highest counter received. Counters older +than the window are rejected. + +The replay check is performed before decryption to prevent CPU exhaustion from +replayed packets that would pass the index lookup but fail decryption. + +## Rate Limiting + +Handshake initiation (msg1) is the primary attack surface for unauthenticated +traffic. Each msg1 requires Noise DH operations (~200µs on modern CPUs), +state allocation, and response generation. + +FLP uses a global token bucket rate limiter: +- **Burst capacity**: Handles legitimate connection storms (e.g., node restart + with many configured peers) +- **Sustained rate**: Limits steady-state new connections per second +- **Global scope**: Rate limiting is global, not per-source, because UDP + source addresses are trivially spoofable + +Additional protections: +- **Connection limit**: Maximum number of pending inbound handshakes, capping + memory usage +- **Handshake timeout**: Stale pending handshakes are cleaned up after a + configurable timeout +- **Allowlist/blocklist**: Optional peer filtering before handshake processing + +## Disconnect + +FLP supports orderly link teardown via a Disconnect message carrying a reason +code (shutdown, restart, protocol error, transport failure, resource +exhaustion, security violation, configuration change, timeout). + +On receiving Disconnect, FLP immediately cleans up state: removes the peer +from the peer table, frees the session index, removes the link, and cleans up +address mappings. If the departed peer was the tree parent, FLP triggers parent +reselection. + +Disconnect is best-effort — if the transport is broken, the message won't +arrive. Timeout-based detection remains the fallback for detecting failed +links. + +On node shutdown, Disconnect is sent to all active peers before transports are +stopped. + +## Liveness Detection + +FLP detects link liveness through timeout-based mechanisms. There are no +dedicated keepalive or ping/pong messages. Liveness is inferred from: + +- **Data traffic**: Any successfully decrypted packet confirms the peer is + alive +- **Gossip messages**: TreeAnnounce and FilterAnnounce messages sent + periodically as part of normal mesh operation serve as implicit heartbeats + +When no traffic is received for the configured timeout period, FLP marks the +link as stale. If the stale state persists, the link is torn down. + +The two-state liveness model: +- **Connected → Stale**: No traffic received for threshold duration +- **Stale → Connected**: Valid traffic received (peer is alive again) +- **Stale → Disconnected**: Timeout exceeded, link torn down + +## Link Message Types + +FLP defines six message types carried inside encrypted frames: + +| Type | Name | Purpose | +| ---- | ---- | ------- | +| 0x10 | TreeAnnounce | Spanning tree state announcements between peers | +| 0x20 | FilterAnnounce | Bloom filter reachability updates | +| 0x30 | LookupRequest | Coordinate discovery — flood toward destination | +| 0x31 | LookupResponse | Coordinate discovery — response with coordinates | +| 0x40 | SessionDatagram | Encapsulated session-layer payload for forwarding | +| 0x50 | Disconnect | Orderly link teardown with reason code | + +Additionally, handshake messages (0x01 msg1, 0x02 msg2) are sent unencrypted +before the link session is established. + +TreeAnnounce and FilterAnnounce are exchanged between direct peers only — they +are not forwarded. LookupRequest and LookupResponse are forwarded through the +mesh (flooded with deduplication). SessionDatagram is forwarded hop-by-hop +toward the destination. Disconnect is peer-to-peer. + +See [fips-mesh-operation.md](fips-mesh-operation.md) for how these messages +work together to build and maintain the mesh, and +[fips-wire-formats.md](fips-wire-formats.md) for byte-level message layouts. + +## Security Properties + +### Threat Resistance + +| Threat | Mitigation | +| ------ | ---------- | +| Connection exhaustion | Token bucket rate limit + connection count limit | +| CPU exhaustion (msg1 flood) | Rate limit before crypto operations | +| Replay attacks | Counter-based nonces with sliding window | +| State confusion | Strict handshake state machine validation | +| Spoofed encrypted packets | Index lookup + AEAD verification | +| Spoofed msg2 | Index lookup + Noise ephemeral key binding | +| Address spoofing | Cryptographic authority, not address-based | +| Session correlation | Index rotation on rekey | + +### Unauthenticated Attack Surface + +Only handshake msg1 can be sent by unauthenticated parties. Encrypted frames +require a known session index, and msg2 requires a response to a specific +ephemeral key. The msg1 attack surface is protected by rate limiting, +connection limits, and handshake timeouts. + +### Authenticated Peer Misbehavior + +Authentication establishes identity but does not grant trust. An authenticated +peer can send malformed packets (which fail AEAD and are dropped) or +high-frequency traffic (rate-limited by higher layers). False tree coordinate +claims are constrained by signature verification on TreeAnnounce messages. + +### Silent Drop Policy + +Invalid packets are silently dropped without error responses. This prevents +information leakage about internal state and avoids amplification attacks where +an attacker sends invalid packets to elicit responses. + +## Implementation Status + +| Feature | Status | +| ------- | ------ | +| Noise IK handshake | **Implemented** | +| Link encryption (ChaCha20-Poly1305) | **Implemented** | +| Index-based session dispatch | **Implemented** | +| Replay protection (sliding window) | **Implemented** | +| Roaming (address-follows-crypto) | **Implemented** | +| Rate limiting (token bucket) | **Implemented** | +| Disconnect with reason codes | **Implemented** | +| Liveness detection (timeout-based) | **Implemented** | +| Reconnection handling | **Implemented** | +| Rekey with index rotation | Planned | +| Allowlist/blocklist | Planned | + +## References + +- [fips-intro.md](fips-intro.md) — Protocol overview and architecture +- [fips-transport-layer.md](fips-transport-layer.md) — Transport layer (below FLP) +- [fips-session-layer.md](fips-session-layer.md) — FSP (above FLP) +- [fips-mesh-operation.md](fips-mesh-operation.md) — How FLP's routing and + self-organization work in practice +- [fips-wire-formats.md](fips-wire-formats.md) — Byte-level wire format reference diff --git a/docs/design/fips-mesh-operation.md b/docs/design/fips-mesh-operation.md new file mode 100644 index 0000000..f42aeac --- /dev/null +++ b/docs/design/fips-mesh-operation.md @@ -0,0 +1,570 @@ +# FIPS Mesh Operation + +This document describes how the FIPS mesh operates at the link layer — how +spanning tree, bloom filters, routing decisions, discovery, and error recovery +work together as a coherent system. It treats spanning tree and bloom filters +as black boxes (what they provide to routing) and focuses on how the pieces +interact. + +For spanning tree algorithms and data structures, see +[fips-spanning-tree.md](fips-spanning-tree.md). For bloom filter parameters +and mathematics, see [fips-bloom-filters.md](fips-bloom-filters.md). + +## Overview + +FIPS mesh operation is entirely distributed. Each node makes forwarding +decisions using only local information: its direct peers, their spanning tree +positions, and their bloom filters. There are no routing tables pushed from +above, no link-state floods, and no distance-vector exchanges. + +Two complementary mechanisms provide the information each node needs: + +- **Spanning tree** gives every node a coordinate in the network — its + ancestry path from itself to the root. These coordinates enable distance + calculations between any two nodes without global topology knowledge. +- **Bloom filters** summarize which destinations are reachable through each + peer. They provide candidate selection — narrowing which peers are worth + considering for forwarding a given destination. + +Together, they enable a routing decision process that is local, efficient, +and self-healing. + +## Spanning Tree Formation and Maintenance + +### What the Spanning Tree Provides + +The spanning tree gives each node a **coordinate**: its ancestry path from +itself to the root, expressed as a sequence of node_addrs. These coordinates +enable: + +- **Distance calculation**: The tree distance between two nodes is the number + of hops from each to their lowest common ancestor (LCA). This provides a + routing metric without any node knowing the full topology. +- **Greedy routing**: At each hop, forward to the peer that minimizes tree + distance to the destination. The strictly-decreasing distance invariant + guarantees loop-free forwarding. + +### How the Tree Forms + +Nodes self-organize into a spanning tree through distributed parent selection: + +1. **Root election**: The node with the smallest node_addr becomes the root. + No election protocol — this is a consequence of each node independently + preferring lower-addressed roots. +2. **Parent selection**: Each node selects a single parent from among its + direct peers based on which offers the best path to root (considering + depth improvement threshold). +3. **Coordinate computation**: Once a node has a parent, its coordinate is + computed from its ancestry path. + +### How the Tree Maintains Itself + +Nodes exchange **TreeAnnounce** messages with their direct peers (not +forwarded — peer-to-peer only). Each TreeAnnounce carries the sender's +current ancestry chain and a sequence number. + +Changes cascade through the tree: + +- A node that changes its parent recomputes its coordinates and announces to + all peers +- Each receiving peer evaluates whether the change affects its own parent + selection +- Only nodes that actually change their coordinates (root or depth changed) + propagate further + +TreeAnnounce propagation is rate-limited at 500ms minimum interval per peer. +A tree of depth D reconverges in roughly D×0.5s to D×1.0s. + +### Partition Handling + +If the network partitions, each segment independently elects its own root +(the smallest node_addr in the segment) and reconverges. When segments +rejoin, nodes discover the globally-smallest root through TreeAnnounce +exchange and reconverge to a single tree. + +See [fips-spanning-tree.md](fips-spanning-tree.md) for algorithm details +and [spanning-tree-dynamics.md](spanning-tree-dynamics.md) for convergence +walkthroughs. + +## Bloom Filter Gossip and Propagation + +### What Bloom Filters Provide + +Each node maintains a bloom filter per peer, answering: "can peer P possibly +reach destination D?" The answer is either "no" (definitive) or "maybe" +(probabilistic — false positives are possible). + +This is **candidate selection**, not routing. Bloom filters identify which +peers are worth considering for a destination, but the actual forwarding +decision uses tree coordinate distance to rank those candidates. + +### How Filters Propagate + +Nodes exchange **FilterAnnounce** messages with direct peers. Each +FilterAnnounce replaces the previous filter for that peer — there is no +incremental update. + +Filter computation uses **split-horizon exclusion**: the outbound filter +for peer Q is computed by merging the local node's own identity, its +leaf-only dependents (if any), and the filters received from all other +peers *except* Q. This prevents echo loops where a node advertises back +to Q the destinations it learned from Q. + +Filters propagate unboundedly (no TTL). At steady state, every reachable +destination appears in at least one peer's filter. + +### Update Triggers + +Filter updates are event-driven, not periodic: + +- Peer connects or disconnects +- A peer's incoming filter changes +- Local state changes (new identity, leaf-only dependent changes) + +Updates are rate-limited at 500ms to prevent storms during topology changes. + +### Scale Properties + +At moderate network sizes, bloom filters are highly accurate. At larger +scales (~1M nodes), hub nodes with many peers may see elevated false positive +rates (7–15% for nodes with 20+ peers). False positives cause unnecessary +discovery attempts but do not affect routing correctness — the tree distance +calculation makes the actual forwarding decision. + +See [fips-bloom-filters.md](fips-bloom-filters.md) for filter parameters, +FPR calculations, and size class folding. + +## Routing Decision Process + +At each hop, FLP makes a local forwarding decision using the `find_next_hop()` +priority chain. This is the core routing algorithm. + +### Priority Chain + +1. **Local delivery** — The destination node_addr matches the local node. + Deliver to FSP above. + +2. **Direct peer** — The destination is an authenticated neighbor. Forward + directly. No coordinates or bloom filters needed. + +3. **Bloom-guided candidate selection** — One or more peers' bloom filters + contain the destination. Select the best candidate by composite key: + `(link_cost, tree_distance, node_addr)`. This requires the destination's + tree coordinates to be in the local coordinate cache. + +4. **Greedy tree routing** — Fallback when bloom filters haven't converged + for this destination. Forward to the peer that minimizes tree distance. + Also requires destination coordinates. + +5. **No route** — Destination unreachable. Generate an error signal + (CoordsRequired or PathBroken) back to the source. + +### The Coordinate Requirement + +All multi-hop routing (steps 3–4) requires the destination's tree coordinates +to be in the local coordinate cache. Without coordinates, `find_next_hop()` +returns None immediately — bloom filters are never even consulted. + +This creates two simultaneous convergence requirements for multi-hop routing: + +1. **Bloom convergence**: Filters must propagate so peers advertise + reachability +2. **Coordinate availability**: Destination coordinates must be cached at + every transit node on the path + +Both must be satisfied simultaneously. Bloom convergence without coordinates +causes a coordinate cache miss. Coordinates without bloom convergence falls +through to greedy tree routing (functional but suboptimal). + +### Candidate Ranking + +When bloom filters identify multiple candidate peers, they are ranked by a +composite key: + +1. **link_cost** — Per-link quality metric (currently constant; designed for + future ETX or similar metrics) +2. **tree_distance** — Coordinate-based distance to destination through this + peer +3. **node_addr** — Deterministic tie-breaker + +A peer with a bloom filter hit but no entry in the peer ancestry table +(missing TreeAnnounce) defaults to maximum distance and is effectively +invisible to routing. + +### Loop Prevention + +The routing decision enforces strict progress: a packet is only forwarded +to a peer that is strictly closer (by tree distance) to the destination than +the current node. This self-distance check prevents routing loops even with +stale coordinates, because each transit node evaluates using its own +freshly-computed coordinates. + +If no peer is closer than the current node (a local minimum in the tree +distance metric), `find_next_hop()` returns None and the caller generates a +PathBroken error. + +## Coordinate Caching + +The coordinate cache maps `NodeAddr → TreeCoordinate` and is the critical +data structure for multi-hop routing. Without it, forwarding decisions cannot +be made. + +### Unified Cache + +The coordinate cache is a single unified cache. All sources — SessionSetup +transit, COORDS_PRESENT DataPackets, LookupResponse — write to the same +cache. + +### Population Sources + +| Source | When | What | +| ------ | ---- | ---- | +| SessionSetup transit | Session establishment | Both src and dest coordinates | +| SessionAck transit | Session establishment | Responder's coordinates | +| DataPacket with COORDS_PRESENT | Warmup or recovery | Both src and dest coordinates | +| LookupResponse | Discovery | Target's coordinates | + +### Eviction + +- **TTL-based**: Entries expire after 300s (configurable) +- **Refresh on use**: Active routing refreshes the TTL, keeping hot entries + alive +- **LRU**: When full, least recently used entries are evicted first +- **Flush on parent change**: When the local node's tree parent changes, the + entire cache is flushed. Parent changes mean the node's own coordinates + have changed, making relative distance calculations with cached coordinates + potentially invalid. Flushing is preferred over stale routing: the cost of + re-discovery is lower than routing packets to dead ends. + +### Cache and Session Timer Ordering + +Timer values are ordered so that idle sessions tear down before transit +caches expire: + +| Timer | Default | Purpose | +| ----- | ------- | ------- | +| Session idle | 90s | Session teardown | +| Coordinate cache TTL | 300s | Coordinate expiration | + +When traffic stops, the session tears down at 90s. When traffic resumes, a +fresh SessionSetup re-warms transit caches (still within their 300s TTL). + +## Discovery Protocol + +Discovery resolves a destination's tree coordinates so that multi-hop routing +can proceed. + +### When Discovery Is Needed + +- First contact with a destination (no cached coordinates) +- After receiving CoordsRequired (transit node lost coordinates) +- After receiving PathBroken (coordinates may be stale) + +### LookupRequest + +The source creates a LookupRequest containing: + +- **request_id**: Unique identifier for deduplication +- **target**: The node_addr being sought +- **origin**: The requester's node_addr +- **origin_coords**: The requester's current tree coordinates (so the + response can route back) +- **TTL**: Bounds the flood radius + +The request floods through the mesh: each node decrements TTL, adds itself +to a visited filter (preventing loops on a single path), and forwards to all +peers not in the visited filter. Bloom filters may help direct the flood +toward likely candidates. + +**Deduplication**: Nodes maintain a short-lived request_id dedup cache to +drop convergent duplicates (the same request arriving via different paths). +This is a protocol requirement, not an optimization. + +### LookupResponse + +When the request reaches the target (or a node that has the target as a +direct peer), a LookupResponse is created containing: + +- **request_id**: Echoed from the request +- **target**: The target's node_addr +- **target_coords**: The target's current tree coordinates +- **proof**: Signature covering `(request_id || target)` — authenticates + that the response is genuine + +The response routes back to the requester using greedy tree routing toward +the origin_coords from the request. + +**Security**: Coordinates are intentionally excluded from the signed proof. +Binding coordinates would invalidate signatures whenever the spanning tree +reconverges. Coordinate tampering by transit nodes causes only routing +inefficiency, not a security breach (data integrity is protected by +session-layer encryption). + +### Discovery Outcome + +On receiving a LookupResponse, the source caches the target's coordinates. +Subsequent routing to that destination can proceed via the normal +`find_next_hop()` priority chain. + +If discovery times out (no response), queued packets receive ICMPv6 +Destination Unreachable. + +## SessionSetup Self-Bootstrapping + +SessionSetup is the mechanism that warms transit node coordinate caches +along a path, enabling subsequent DataPackets to route efficiently. + +### How It Works + +SessionSetup carries plaintext coordinates (outside the Noise handshake +payload, visible to transit nodes): + +- **src_coords**: Source's current tree coordinates +- **dest_coords**: Destination's tree coordinates (learned from discovery) + +As the SessionSetup transits each intermediate node: + +1. The transit node extracts both coordinate sets +2. Caches `src_addr → src_coords` and `dest_addr → dest_coords` in its + coordinate cache +3. Forwards the message using the cached destination coordinates + +SessionAck returns along the reverse path, carrying the responder's +coordinates and warming caches in the other direction. + +### Result + +After the handshake completes, the entire forward and reverse paths have +cached coordinates for both endpoints. Subsequent DataPackets use minimal +headers (no coordinates) and route efficiently through the warmed caches. + +## COORDS_PRESENT Warmup + +The COORDS_PRESENT flag on DataPackets provides a secondary cache-warming +mechanism that complements SessionSetup. See +[fips-session-layer.md](fips-session-layer.md) for the warmup-then-reactive +strategy. + +Transit nodes process DataPackets with COORDS_PRESENT by extracting and +caching both source and destination coordinates — the same operation +performed for SessionSetup coordinates. + +## Error Recovery + +When routing fails, transit nodes signal the source endpoint so it can take +corrective action. + +### CoordsRequired + +**Trigger**: A transit node receives a SessionDatagram but has no cached +coordinates for the destination. It cannot make a forwarding decision. + +**Transit node action**: +1. Create a new SessionDatagram addressed back to the original source, + carrying a CoordsRequired payload identifying the unreachable destination +2. Route the error via `find_next_hop(src_addr)` +3. If the source is also unreachable, drop silently (no cascading errors) + +**Source recovery**: +1. Initiate discovery (LookupRequest flood) for the destination +2. Reset COORDS_PRESENT warmup counter — subsequent DataPackets include + coordinates +3. When discovery completes, warmup counter resets again (covers timing gap) + +The crypto session remains active throughout — only routing state is +refreshed. + +### PathBroken + +**Trigger**: A transit node has cached coordinates for the destination but +no peer is closer to the destination than itself (a local minimum in the +tree distance metric). The cached coordinates may be stale. + +**Transit node action**: Same as CoordsRequired — generate error back to +source. + +**Source recovery**: +1. Remove stale coordinates from cache +2. Initiate discovery for the destination +3. Reset COORDS_PRESENT warmup counter + +### Error Signal Rate Limiting + +Both error types are rate-limited at transit nodes: maximum one error per +destination per 100ms. This prevents storms during topology changes when many +packets to the same destination hit the same routing failure simultaneously. + +### Error Routing Limitation + +Error signals route back to the source using `find_next_hop(src_addr)`. For +steady-state DataPackets (after the COORDS_PRESENT warmup window), the +transit node may lack cached coordinates for the source. If so, the error is +silently dropped. + +This blind spot is partially addressed by COORDS_PRESENT warmup: transit +nodes receive source coordinates during the warmup phase. But after warmup +expires and transit caches for the source expire, errors may be lost. The +session idle timeout (90s) limits the window — if traffic stops long enough +for transit caches to fully expire, the session tears down and re-establishment +re-warms the path. + +## Cold Start → Warm Cache → Steady State + +### Cold Start + +A new node or a node reaching a new destination goes through the following +sequence: + +1. **DNS resolution** (IPv6 adapter only): Resolve `npub.fips` → populate + identity cache with NodeAddr + PublicKey +2. **Session initiation attempt**: Fails because no coordinates are cached + for the destination +3. **Discovery**: LookupRequest floods through the mesh; LookupResponse + returns the destination's coordinates +4. **Session establishment**: SessionSetup carries coordinates, warming + transit caches along the path +5. **Warmup**: First N DataPackets include COORDS_PRESENT, reinforcing + transit caches + +The first packet to a new destination always triggers this sequence. The +packet is queued (bounded) until the session is established. + +### Warm Cache + +After session establishment and warmup: + +- Transit nodes have cached coordinates for both endpoints +- Bloom filters have converged for the destination +- DataPackets use minimal headers (no coordinates) +- Routing decisions are fast: bloom candidate selection + distance ranking + +### Steady State + +In steady state, the mesh is mostly self-maintaining: + +- TreeAnnounce gossip keeps the spanning tree current +- FilterAnnounce gossip keeps bloom filters current +- Coordinate caches are refreshed by active routing traffic +- Occasional cache misses trigger COORDS_PRESENT or discovery, but these + are rare when traffic is flowing + +### Cache Expiry and Recovery + +When traffic to a destination stops: + +1. **Session idles out** (90s) — session torn down +2. **Coordinate caches expire** (300s) — transit nodes forget coordinates +3. **Bloom filters remain** — they have no TTL, so reachability information + persists + +When traffic resumes: + +1. Identity cache: usually still populated (LRU, no TTL) +2. Session: new establishment required (full handshake) +3. Coordinates: discovery may be needed if cache has expired +4. SessionSetup re-warms transit caches on the new path + +## Leaf-Only Operation *(future direction)* + +Leaf-only operation is a planned optimization for resource-constrained nodes +(sensors, battery-powered devices). Not currently implemented. + +### Concept + +A leaf-only node connects to a single upstream peer that handles all routing +on its behalf: + +- **No bloom filter storage or processing**: The upstream peer includes the + leaf's identity in its own outbound bloom filters +- **No spanning tree participation**: The leaf does not offer itself as a + potential parent to other nodes +- **Simplified routing**: All traffic tunnels through the upstream peer +- **Minimal resource usage**: Suitable for ESP32-class devices (~500KB RAM) + +### Upstream Peer Responsibilities + +The upstream peer: + +- Includes the leaf's identity in its outbound bloom filters +- Forwards all traffic addressed to the leaf +- Handles discovery responses on behalf of the leaf +- Maintains the link session with the leaf + +### What the Leaf Retains + +Even as a leaf-only node, it still: + +- Maintains its own Noise IK link session with the upstream peer +- Can establish end-to-end FSP sessions with arbitrary destinations +- Has its own identity (npub, node_addr) + +The optimization is purely at the routing/mesh layer — the leaf delegates +routing decisions but retains its own end-to-end encryption and identity. + +## Packet Type Summary + +| Message | Typical Size | When | Forwarded? | +| ------- | ------------ | ---- | ---------- | +| TreeAnnounce | Variable (depth-dependent) | Topology changes | No (peer-to-peer) | +| FilterAnnounce | ~1 KB | Topology changes | No (peer-to-peer) | +| LookupRequest | ~300 bytes | First contact, recovery | Yes (flood) | +| LookupResponse | ~400 bytes | Response to discovery | Yes (greedy routed) | +| SessionDatagram + SessionSetup | ~230–400 bytes | Session establishment | Yes (routed) | +| SessionDatagram + SessionAck | ~120 bytes | Session confirmation | Yes (routed) | +| SessionDatagram + DataPacket (minimal) | 38 bytes + payload | Bulk traffic | Yes (routed) | +| SessionDatagram + DataPacket (with coords) | ~170 bytes + payload | Warmup/recovery | Yes (routed) | +| SessionDatagram + CoordsRequired | 68 bytes | Cache miss error | Yes (routed) | +| SessionDatagram + PathBroken | 68+ bytes | Dead-end error | Yes (routed) | +| Disconnect | 2 bytes | Link teardown | No (peer-to-peer) | + +See [fips-wire-formats.md](fips-wire-formats.md) for byte-level layouts. + +## Privacy Considerations + +Source and destination node_addrs are visible to every transit node (required +for forwarding decisions and error signal routing). FIPS prioritizes +low-latency greedy routing with explicit error signaling over metadata +privacy. + +The node_addr is `SHA-256(pubkey)` truncated to 128 bits — a one-way hash. +Transit nodes learn which node_addr pairs are communicating but cannot +determine the actual Nostr identities (npubs) of the endpoints. An observer +can verify "does this node_addr belong to pubkey X?" but cannot enumerate +communicating identities from traffic alone. + +Onion routing was considered and rejected because it requires the sender to +know the full path upfront (incompatible with self-organizing routing) and +prevents per-hop error feedback (incompatible with CoordsRequired/PathBroken +recovery). + +## Implementation Status + +| Feature | Status | +| ------- | ------ | +| Spanning tree formation | **Implemented** | +| TreeAnnounce gossip | **Implemented** | +| Bloom filter computation (split-horizon) | **Implemented** | +| FilterAnnounce gossip | **Implemented** | +| find_next_hop() priority chain | **Implemented** | +| Coordinate cache (unified, TTL + refresh) | **Implemented** | +| Flush coord cache on parent change | **Implemented** | +| LookupRequest/LookupResponse discovery | **Implemented** | +| SessionSetup self-bootstrapping | **Implemented** | +| COORDS_PRESENT warmup-then-reactive | **Implemented** | +| CoordsRequired recovery | **Implemented** | +| PathBroken recovery | **Implemented** | +| Error signal rate limiting | **Implemented** | +| Leaf-only operation | Future direction | +| Link cost metrics (ETX) | Future direction | +| Discovery path accumulation | Future direction | + +## References + +- [fips-intro.md](fips-intro.md) — Protocol overview +- [fips-link-layer.md](fips-link-layer.md) — FLP specification +- [fips-spanning-tree.md](fips-spanning-tree.md) — Tree algorithms and data + structures +- [fips-bloom-filters.md](fips-bloom-filters.md) — Filter parameters and math +- [fips-wire-formats.md](fips-wire-formats.md) — Wire format reference +- [spanning-tree-dynamics.md](spanning-tree-dynamics.md) — Convergence + walkthroughs diff --git a/docs/design/fips-routing.md b/docs/design/fips-routing.md deleted file mode 100644 index ee6de29..0000000 --- a/docs/design/fips-routing.md +++ /dev/null @@ -1,903 +0,0 @@ -# FIPS Routing Design - -This document describes the routing architecture for FIPS, including Bloom -filter routing, greedy tree routing, discovery protocol, and routing session -establishment. - -For wire formats and exchange rules, see [fips-gossip-protocol.md](fips-gossip-protocol.md). -For spanning tree dynamics and convergence, see [spanning-tree-dynamics.md](spanning-tree-dynamics.md). - -## Overview - -FIPS uses a layered routing strategy where each mechanism handles different -situations. In steady state, bloom filter routing handles the vast majority -of forwarding decisions. - -### Next-Hop Selection (in priority order) - -1. **Local delivery** — destination is self -2. **Direct peer** — destination is an authenticated peer -3. **Bloom filter routing** — one or more peers' bloom filters contain the - destination; select the best candidate by `(link_cost, tree_distance, - node_addr)`. Since filters propagate unboundedly through the network, - every reachable destination eventually appears in at least one peer's - filter. This is the **primary routing path** for most traffic. -4. **Greedy tree routing** — fallback when bloom filters haven't yet - converged (transient condition during topology changes). Requires the - destination's tree coordinates to be in the local coordinate cache, - populated by a prior SessionSetup or LookupResponse. -5. **No route** — destination unreachable - -### Role of Each Mechanism - -- **Bloom filters**: Primary forwarding — tell each node which peer to - send through for a given destination. Propagate unboundedly via - split-horizon merge, so they cover the entire reachable network at - steady state. -- **Greedy tree routing**: Fallback forwarding during convergence windows - when bloom filters are incomplete. Also serves as tie-breaker among - bloom filter candidates (closest tree distance wins). -- **Discovery protocol**: Populates the coordinate cache to enable greedy - tree routing and to provide intermediate routers with coordinate data - for more efficient path selection. Not required for basic reachability - once bloom filters have converged. - -## Design Goals - -- Minimize per-packet overhead for data transfer -- Bounded state at each node (independent of network size) -- Efficient routing without global knowledge -- Graceful degradation for constrained devices -- Fast convergence on topology changes - -## Network Scale Assumptions - -| Scale | Nodes | Bloom Filter Role | -|-------|-------|-------------------| -| Small private network | 100-1,000 | Covers entire network with low FPR | -| Modest public network | ~1,000,000 | Covers entire network but FPR increases at hub nodes due to filter saturation | -| Internet-scale | Billions | Out of scope (requires different architecture) | - -The primary design target is networks up to ~1M nodes. Since bloom filters -propagate unboundedly (no TTL), they converge to represent the entire -reachable network. At large scale, the fixed 1KB filter size means higher -false positive rates at well-connected hub nodes, which may trigger -unnecessary discovery queries but does not affect correctness. - -## Node Participation Modes - -### Full Participant - -- Maintains Bloom filters for peer reachability -- Participates in spanning tree (can be selected as parent) -- Routes packets for other nodes -- Minimum viable device: ESP32-class (~500KB RAM) - -### Leaf-Only - -- Single peer handles all routing on its behalf -- No Bloom filter storage or processing -- Does not participate in spanning tree as potential parent -- Suitable for highly constrained devices (sensors, battery-powered nodes) - -Leaf-only nodes appear as a single entry in their peer's Bloom filter. All -traffic tunnels through that peer. - ---- - -## Part 1: Bloom Filter Design - -### Parameters - -| Parameter | Value | Rationale | -|----------------|-------------------|--------------------------------------------| -| Filter size | 1 KB (8,192 bits) | Sized for expected occupancy with margin | -| Hash functions | 5 | Optimal for 800-1,600 entries at this size | - -### Mathematical Foundation - -**False Positive Rate (FPR):** - -```text -FPR = (1 - e^(-kn/m))^k -``` - -Where m = bits, n = entries, k = hash functions. - -**Optimal hash count:** - -```text -k_opt = (m/n) × ln(2) ≈ 0.693 × (m/n) -``` - -For m=8,192 and expected n=800: k_opt ≈ 7. We use k=5 to accommodate higher -occupancy scenarios (up to ~1,600 entries) while maintaining acceptable FPR. - -**Required bits for target FPR:** - -```text -m = -1.44 × n × ln(p) -``` - -For 1% FPR: m ≈ 9.6n bits. For 5% FPR: m ≈ 6.2n bits. - -### Expected Filter Occupancy - -Filter occupancy depends on network topology and node degree. In practice, -filters reach a natural equilibrium determined by the network's structure — -merging peer filters transitively means each filter converges to represent -the node's reachable neighborhood. - -**Outgoing filter to peer Q contains:** - -- Self (1 entry) -- Entries from (d-1) other peers' filters (excluding Q), with overlap - -**Expected occupancy by node degree:** - -| Degree (d) | Expected Entries | Notes | -|------------|------------------|-------| -| 5 | 100-200 | Constrained/IoT | -| 8 | 250-400 | Typical node | -| 12 | 500-800 | Well-connected | -| 20+ | 1,200-1,800 | Hub node | - -### False Positive Rates (1 KB filter, k=5) - -| Entries | FPR | Scenario | -|---------|-----|----------| -| 200 | 0.02% | Low-degree node | -| 400 | 0.3% | Typical node | -| 800 | 2.4% | Well-connected | -| 1,200 | 7.5% | Hub node | -| 1,600 | 15% | Heavily loaded hub | - -Since filters propagate unboundedly, hub nodes with many peers will have -higher occupancy (more entries merged from more peers). FPR above 5% means -bloom filter routing may occasionally select a peer that can't actually -reach the destination (false positive), requiring fallback to greedy tree -routing or error recovery. Hub nodes may benefit from larger filters in -future protocol versions (see §1.6). - -### Size Classes (Forward Compatibility) - -Filter sizes are powers of 2 to enable **folding** — a technique for shrinking -filters by ORing halves: - -```rust -fn fold(filter: &[u8]) -> Vec { - let half = filter.len() / 2; - (0..half).map(|i| filter[i] | filter[i + half]).collect() -} -``` - -Folding preserves correctness (no false negatives) but increases FPR. - -| size_class | Bits | Bytes | Status | -|------------|------|-------|--------| -| 0 | 4,096 | 512 | Reserved (future) | -| 1 | 8,192 | 1,024 | **Current default** | -| 2 | 16,384 | 2,048 | Reserved (future) | -| 3 | 32,768 | 4,096 | Reserved (future) | - -**v1 protocol**: All nodes MUST use size_class=1. The field is present in the -wire format for forward compatibility. - -**Future versions**: Nodes may negotiate larger filters. Receivers fold down -to their preferred size if sender's filter is larger. This allows hub nodes -to maintain higher precision while constrained nodes use smaller filters. - -### Filter Contents - -Each node's filter contains Node IDs (and optionally gateway /64 prefixes) -that are reachable through that node. A Node ID is the SHA-256 hash of the -node's npub, truncated or used directly as the filter key. - -### Per-Peer Filters - -Each node maintains a Bloom filter for each peer direction: - -```rust -peer_filters: HashMap -``` - -The filter for peer P answers: "Which destinations are reachable through P?" - -### Update Mechanism: Event-Driven - -Filters are updated on events rather than periodic refresh: - -**Triggering events:** - -1. Peer connects — exchange current filters -2. Peer disconnects — remove their filter, recompute, notify other peers -3. Received filter changes outgoing filter — recompute, send updates -4. Local state change — new leaf dependent, become gateway, etc. - -Updates are rate-limited to prevent storms during reconvergence. See -[fips-gossip-protocol.md](fips-gossip-protocol.md) §3 for FilterAnnounce wire -format and exchange rules. - -### Filter Contents - -A node's outgoing filter to peer Q contains: - -1. This node's own Node ID -2. Node IDs of leaf-only dependents -3. Entries merged from filters received from all other peers (not Q) - -Filters propagate transitively through the network. Each node merges all -inbound peer filters (excluding the destination peer) into its outgoing -filter — this split-horizon approach prevents a node's own entries from -being echoed back to it, providing loop prevention. Propagation is -unbounded; filters naturally converge as the Bloom filter's fixed size -limits information density. - -### Expiration - -Bloom filters cannot remove individual entries. Expiration is handled via: - -- **Peer disconnect**: Remove that peer's filter entirely, recompute -- **Filter replacement**: Each FilterAnnounce replaces the previous one -- **Implicit timeout**: If no updates received from peer within threshold, - consider their filter stale - ---- - -## Part 2: Discovery Protocol - -### Purpose - -Discover the tree coordinates of a destination to enable greedy tree routing -and to populate coordinate caches at intermediate routers for more efficient -forwarding. In steady state, bloom filters handle reachability; discovery -provides the coordinate information that improves path selection quality. - -### When Used - -- During bloom filter convergence (destination not yet in any peer's filter) -- To populate coordinate caches for greedy tree routing (optimization) -- After cached route failure (coordinates may be stale) - -For wire formats, see [fips-gossip-protocol.md](fips-gossip-protocol.md) §4-5. - -### Discovery Flow - -```text -1. S wants to reach D, D not in any local filter -2. S checks route cache — miss -3. S creates LookupRequest with own coordinates, floods to peers -4. Request propagates (Bloom filters may help direct it) -5. Request reaches D (or node with D in filter) -6. D creates LookupResponse with its coordinates, signs it -7. Response routes back to S using S's coordinates (greedy) -8. S caches D's coordinates -9. S can now route to D using greedy tree routing -``` - -### Request Propagation - -**Flood with TTL and visited filter:** - -- Send to all peers not in `visited` filter -- Each hop decrements TTL, adds self to `visited` -- At TTL=0, stop propagating -- `visited` filter prevents loops (a packet revisiting a node on its own - path), but does NOT prevent convergent duplicates — the same request - arriving at a node via different paths with different visited filters. - See [Known Limitation: Flood - Convergence](#known-limitation-flood-convergence). - -**Bloom filter assistance (optional optimization):** - -If a node's peer filter indicates "maybe" for the target, prioritize that -direction. Reduces flood scope when target is partially in range. - -### Response Routing - -Response uses greedy tree routing based on `origin_coords` from the request. -Each router forwards toward the origin using tree distance. - -### Security - -The target signs the LookupResponse with a proof covering -`(request_id || target)`. Without this signature, a malicious node could -claim reachability for any target and blackhole traffic. The signature -proves the target authorized the route. - -Note: `target_coords` are intentionally excluded from the proof. Binding -coordinates would invalidate the signature whenever the spanning tree -reconverges (parent switch, root change), causing valid responses to be -rejected if tree topology shifts during the lookup RTT. Since coordinates -are ephemeral routing hints and data integrity is protected by the -session layer, coordinate tampering by a transit node only causes routing -inefficiency, not a security breach. - -### Caching - -Discovered coordinates are stored in the route cache (`RouteCache`): - -```rust -struct RouteCache { - entries: HashMap, - max_entries: usize, // default: 10,000 -} - -struct CachedCoords { - coords: TreeCoordinate, - discovered_at: Timestamp, - last_used: Timestamp, -} -``` - -- **Eviction**: LRU when cache full (no automatic TTL expiration) -- **Invalidation**: On route failure, evict and re-discover - -This is distinct from the `CoordCache` used for session-populated -coordinates (see Part 4 for the full dual-cache architecture). - ---- - -## Part 3: Tree Coordinates and Greedy Routing - -Tree coordinates and greedy routing serve two roles: - -1. **Fallback forwarding** during bloom filter convergence windows -2. **Tie-breaking** among bloom filter candidates — tree distance between - a candidate peer and the destination helps select the best path when - multiple peers advertise reachability - -### Tree Coordinates - -A node's coordinates are its ancestry path from self to root: - -```text -coords(N) = [N, Parent(N), Parent(Parent(N)), ..., Root] -``` - -Example: Node D at depth 4 has coordinates `[D, P1, P2, P3, Root]`. - -### Tree Distance - -Distance between two nodes is hops through their lowest common ancestor (LCA): - -```rust -fn tree_distance(a_coords: &[NodeAddr], b_coords: &[NodeAddr]) -> usize { - let lca_depth = longest_common_suffix_length(a_coords, b_coords); - let a_to_lca = a_coords.len() - lca_depth; - let b_to_lca = b_coords.len() - lca_depth; - a_to_lca + b_to_lca -} -``` - -Note: Coordinates are ordered self-to-root, so common ancestry is a suffix. - -### Greedy Routing Algorithm - -When used as a fallback (no bloom filter hits), greedy routing forwards to -the peer that minimizes tree distance to the destination. A self-distance -check ensures progress — the packet is only forwarded if the chosen peer is -strictly closer than the current node. - -```rust -fn greedy_next_hop(&self, dest_coords: &TreeCoordinate) -> Option { - if self.my_coords.root_id() != dest_coords.root_id() { - return None; // different tree - } - - let my_distance = self.my_coords.distance_to(dest_coords); - - // Find peer with minimum distance, tie-break by smallest node_addr - let best = self.peer_ancestry.iter() - .min_by(|(id_a, coords_a), (id_b, coords_b)| { - coords_a.distance_to(dest_coords) - .cmp(&coords_b.distance_to(dest_coords)) - .then_with(|| id_a.cmp(id_b)) - }); - - match best { - Some((peer_id, coords)) if coords.distance_to(dest_coords) < my_distance => { - Some(*peer_id) - } - _ => None, // no peer is closer (local minimum) - } -} -``` - -### Progress Guarantee - -When the coordinate cache is populated, greedy routing makes progress as -long as: - -1. Tree is connected -2. Destination's coordinates are accurate -3. A peer is closer to the destination than the current node - -If no peer is closer (local minimum), routing returns `None` and the caller -generates a PathBroken error. In a properly formed tree this should not -occur, but the self-distance check provides a safety net. - -### What Each Node Knows - -| Information | Source | -|-------------|--------| -| Own coordinates | Spanning tree protocol (ancestry to root) | -| Each peer's coordinates | Exchanged on peering | -| Destination coordinates | From packet header (established via session) | - -No global routing tables. Each node makes purely local decisions. - -### Privacy Considerations - -Intermediate routers can observe `src_addr` and `dest_addr` in the -SessionDatagram envelope of transiting packets. This enables traffic analysis -(who is communicating with whom) but not content inspection (the payload is -end-to-end encrypted with session keys). - -**Why source address is visible**: The `src_addr` field in the SessionDatagram -is required for transit routers to send error signals (CoordsRequired, -PathBroken) back to the sender. When a transit router R cannot forward a -SessionDatagram `{src: S, dest: D}`, it creates a new SessionDatagram -`{src: R, dest: S}` carrying the error signal, and routes it toward S using -`find_next_hop(S)`. This is a deliberate design choice: rather than silently -dropping unroutable packets and relying on application-layer timeouts to detect -failures, FIPS provides explicit feedback that allows rapid route recovery. -The tradeoff favors responsiveness over metadata privacy. - -**Partial mitigation**: FIPS addresses are derived from `SHA-256(pubkey)`, not the -npub itself. An observer learns that `fd12:3456:...` is communicating with -`fd78:9abc:...`, but cannot directly determine the Nostr identities without -additional information (e.g., DNS lookup correlation, prior knowledge of the -address-to-npub mapping). - -**Alternative considered**: Onion routing (like Tor) hides routing metadata from -intermediate nodes but requires the sender to know the full path upfront and -prevents per-hop error feedback. FIPS prioritizes low-latency greedy routing -with explicit error signaling over metadata privacy. - ---- - -## Part 4: Coordinate Cache Architecture - -> **Wire formats**: For session layer message wire formats (SessionSetup, -> SessionAck, DataPacket, CoordsRequired, PathBroken), see -> [fips-session-protocol.md](fips-session-protocol.md) §8. - -### Dual Cache Architecture - -FIPS uses two coordinate caches with different lifecycles: - -- **CoordCache** (session-populated): Stores coordinates learned from - SessionSetup and SessionAck coordinate fields during session establishment. - TTL-based expiration (300s). 50,000 entries max. Consulted first by - `find_next_hop()`. - -- **RouteCache** (discovery-populated): Stores coordinates learned from - LookupResponse during discovery protocol completion. LRU eviction only - (no automatic TTL). 10,000 entries max. Consulted as fallback when - CoordCache misses. - -Both caches store `TreeCoordinate` values (node_addr → coordinates). Neither -stores next-hop information — routing decisions are computed at lookup time by -`find_next_hop()` using the cached coordinates. - -### Cache Purpose - -The coordinate caches serve two functions: - -1. **Greedy routing fallback** — when bloom filters haven't converged, - cached coordinates enable tree-distance-based forwarding. -2. **Reduced packet overhead** — once coordinates are cached at intermediate - routers, data packets can use minimal DataPacket headers (4 bytes inside - a 34-byte SessionDatagram = 38 bytes total) rather than including full - coordinates (~170 bytes total). - -### Cache Lifecycle - -```text -┌─────────────────────────────────────────────────────────────────┐ -│ 1. Discovery: S queries for D's coordinates │ -│ 2. Setup: S sends SessionSetup, routers cache coordinates │ -│ 3. Data: Packets carry only addresses, routers use cache │ -│ 4. Refresh: Periodic or on-demand to prevent cache expiry │ -│ 5. Teardown: Implicit (cache expires) or explicit │ -└─────────────────────────────────────────────────────────────────┘ -``` - -### Session Setup Flow - -All messages are carried inside SessionDatagram envelopes that provide -`src_addr`, `dest_addr`, and `hop_limit` at the link layer. - -```text -S R1 R2 D -│ │ │ │ -│──SessionDatagram──────>│ │ │ -│ {src:S, dest:D, │──SessionDatagram─────>│ │ -│ payload:SessionSetup │ │──SessionDatagram─────>│ -│ (src_coords, │ │ │ -│ dest_coords)} │ cache: │ cache: │ -│ │ dest_addr→dest_coords│ dest_addr→dest_coords│ -│ │ src_addr→src_coords │ src_addr→src_coords │ -│ │ │ │ -│<──────────────────────────────────────────SessionDatagram{SessionAck}──│ -│ │ │ │ -│══SessionDatagram══════>│══════════════════════>│══════════════════════>│ -│ {src:S, dest:D, │ (use cached coords) │ (use cached coords) │ -│ payload:DataPacket} │ │ │ -``` - -### Router Behavior - -Transit routers process SessionDatagram envelopes. The envelope provides -`src_addr` and `dest_addr` for routing decisions and error signaling. - -```rust -impl Router { - /// Handle a SessionDatagram carrying a SessionSetup payload. - /// Cache coordinates from the setup message for both directions. - fn handle_session_setup(&mut self, dg: &SessionDatagram, setup: SessionSetup) { - // Cache coordinates for both directions - self.coord_cache.insert(dg.dest_addr, CacheEntry { - coords: setup.dest_coords.clone(), - expires: now() + CACHE_TTL, - }); - self.coord_cache.insert(dg.src_addr, CacheEntry { - coords: setup.src_coords.clone(), - expires: now() + CACHE_TTL, - }); - - // Forward toward destination using find_next_hop - if let Some(next) = self.find_next_hop(&dg.dest_addr) { - self.forward(next, dg); - } - } - - /// Handle a SessionDatagram carrying a DataPacket payload. - /// Addresses come from the SessionDatagram envelope (dg.src_addr, dg.dest_addr). - fn handle_data_packet(&mut self, dg: &SessionDatagram, packet: DataPacket) { - // If packet carries coordinates, cache them - if packet.flags & COORDS_PRESENT != 0 { - if let (Some(src_coords), Some(dest_coords)) = - (&packet.src_coords, &packet.dest_coords) - { - self.coord_cache.insert(dg.dest_addr, CacheEntry { - coords: dest_coords.clone(), - expires: now() + CACHE_TTL, - }); - self.coord_cache.insert(dg.src_addr, CacheEntry { - coords: src_coords.clone(), - expires: now() + CACHE_TTL, - }); - } - } - - // Route using find_next_hop (bloom filter → greedy tree → None) - match self.find_next_hop(&dg.dest_addr) { - Some(next) => self.forward(next, dg), - None => { - // Cannot route — send error back to source via src_addr - self.send_error_to_source(dg, CoordsRequired { - dest_addr: dg.dest_addr, - reporter: self.node_addr, - }); - } - } - } - - /// Send an error signal back to the source of a SessionDatagram. - /// Creates a new SessionDatagram addressed to dg.src_addr. - fn send_error_to_source(&self, dg: &SessionDatagram, error: impl LinkError) { - let error_dg = SessionDatagram { - src_addr: self.node_addr, // We are the reporter - dest_addr: dg.src_addr, // Route back to original source - hop_limit: 64, - payload: error.encode(), // CoordsRequired or PathBroken - }; - // Route the error — if we can't reach source either, drop silently - if let Some(next) = self.find_next_hop(&dg.src_addr) { - self.forward(next, &error_dg); - } - // If find_next_hop returns None for source: drop silently. - // No cascading errors. - } -} -``` - -### Cache Data Structures - -```rust -/// Session-populated coordinate cache (TTL-based). -struct CoordCache { - entries: HashMap, - max_entries: usize, // default: 50,000 - ttl_ms: u64, // default: 300,000 (5 minutes) -} - -struct CacheEntry { - coords: TreeCoordinate, - created_at: u64, // Unix milliseconds - last_used: u64, - expires_at: u64, -} - -/// Discovery-populated coordinate cache (LRU-based, no TTL). -struct RouteCache { - entries: HashMap, - max_entries: usize, // default: 10,000 -} - -struct CachedCoords { - coords: TreeCoordinate, - discovered_at: u64, - last_used: u64, -} -``` - -**CoordCache eviction**: Expired entries removed first, then LRU when cache -exceeds max_entries. Entries expire after TTL (300 seconds). Refreshed by -subsequent SessionSetup/SessionAck or DataPacket with COORDS_PRESENT. - -**RouteCache eviction**: Pure LRU when cache exceeds max_entries. No automatic -TTL expiration. Invalidated explicitly on route failure. - -### Cache Miss Recovery - -When a router's cache entry is evicted mid-session: - -```text -1. SessionDatagram{src:S, dest:D, payload:DataPacket} arrives, router can't route -2. Router R creates SessionDatagram{src:R, dest:S, payload:CoordsRequired{dest:D}} -3. Router routes the error back to S using find_next_hop(S) -4. S receives CoordsRequired, marks route to D as "cold" -5. S resends with COORDS_PRESENT flag set in DataPacket -6. Routers cache coordinates from DataPacket, forward normally -7. After N successful packets, S clears the flag -``` - -The crypto session remains active throughout—only routing state is refreshed. -From application perspective: one packet delayed, transparent recovery. - -If the router also cannot route to S (no bloom filter hit, no cached -coordinates for S), the error is dropped silently. No cascading errors are -generated. The source will eventually detect the loss via application-layer -timeout. - -### Sender State Machine - -```rust -impl Sender { - fn send(&mut self, dest: NodeAddr, data: &[u8]) { - if !self.session_established(dest) { - // Need to establish crypto session first - let dest_coords = self.discover_or_cached(dest)?; - self.send_session_setup(dest, &dest_coords); - self.await_session_ack(dest)?; - } - - // Check route state - let include_coords = self.route_state(dest) == RouteCold; - let data_packet = DataPacket::new(data, include_coords); - - // Wrap in SessionDatagram for forwarding - let dg = SessionDatagram { - src_addr: self.node_addr, - dest_addr: dest, - hop_limit: 64, - payload: data_packet.encode(), - }; - self.forward_datagram(dg); - } - - /// Handle CoordsRequired received inside a SessionDatagram addressed to us. - /// The SessionDatagram.src_addr identifies the reporting router (informational). - fn handle_coords_required(&mut self, err: CoordsRequired) { - // Route cache expired at intermediate router - // Crypto session still valid - just need to re-warm route - self.mark_route_cold(err.dest_addr); - // Next send() will include coordinates - } -} - -enum RouteState { - RouteWarm, // Send minimal headers - RouteCold, // Include coordinates until warm -} -``` - ---- - -## Part 5: Packet Type Summary - -All session-layer messages and error signals are carried inside a -SessionDatagram envelope (34 bytes: msg_type + src_addr + dest_addr + -hop_limit). Sizes below include the SessionDatagram header. - -| Type | Purpose | Size | When Used | -|------|---------|------|-----------| -| FilterAnnounce | Bloom filter propagation | ~1 KB | Topology changes | -| LookupRequest | Discover coordinates | ~300 bytes | First contact with distant node | -| LookupResponse | Return coordinates | ~400 bytes | Reply to discovery | -| SessionDatagram+SessionSetup | Warm caches + crypto init | ~230-400 bytes | Before data transfer | -| SessionDatagram+SessionAck | Confirm session + crypto | ~100-200 bytes | Session confirmation | -| SessionDatagram+DataPacket | Application data | 38 bytes + payload (minimal) | Bulk of traffic | -| SessionDatagram+DataPacket | With coordinates | ~170 bytes + payload | After CoordsRequired | -| SessionDatagram+CoordsRequired | Coords needed signal | 68 bytes | Cache miss recovery | -| SessionDatagram+PathBroken | Routing failed signal | 68+ bytes | Greedy routing local minimum | - -> **Note**: SessionSetup/SessionAck sizes vary based on coordinate depth and -> whether they carry crypto handshake payloads (combined establishment per -> [fips-session-protocol.md](fips-session-protocol.md) §3.4 and §5.1). - ---- - -## Part 6: Traffic Analysis - -### Steady State (Stable Network) - -- **Bloom filter traffic**: Near zero (event-driven, no changes) -- **Discovery traffic**: Rare (warm caches) -- **Session traffic**: Rare (established sessions) -- **Data traffic**: Minimal overhead (38-byte header: 34 envelope + 4 DataPacket) - -### Network Churn - -When nodes join/leave: - -- Bloom filter updates propagate through affected peers -- Affected sessions may need re-establishment -- Discovery queries for newly-joined nodes - -### Per-Node Resource Requirements - -| Resource | Full Participant | Leaf-Only | -|----------|------------------|-----------| -| Bloom filter storage | d × 1 KB (d = peer count) | None | -| CoordCache (session) | 50K entries, 300s TTL | None | -| RouteCache (discovery) | 10K entries, LRU | Minimal | -| Bandwidth (idle) | < 1 KB/sec | Near zero | - ---- - -## Known Limitations - -### Known Limitation: Flood Convergence - -The visited Bloom filter in LookupRequest prevents **loops** (a packet -revisiting a node on its own path) but does not prevent **convergent -duplicates** — the same request arriving at a node via different paths, each -carrying a different visited filter. - -Example: Source S floods to peers B and C. Both forward to node D. D receives -two copies — one with visited={S,B}, one with visited={S,C}. Neither copy's -visited filter contains D, so D processes both. This redundancy compounds at -every well-connected node in the flood path, and the destination may generate -multiple LookupResponses. - -**Required fix**: Nodes MUST maintain a short-lived cache of recently-seen -`request_id` values (retention: a few seconds, bounded by request rate limit). -On receiving a LookupRequest, check this cache first and drop duplicates before -consulting the visited filter. This is referenced in the gossip protocol spec -(section 4.4, rate limiting) but needs to be elevated to a protocol -requirement, not an optimization. - -### Known Limitation: Capacity-Blind Routing - -Both bloom filter candidate selection and greedy tree routing currently -select next hops without considering link quality. When multiple peers -can reach a destination, the selection is based on tree distance and -node address tie-breaking — purely topological metrics that ignore link -capacity, latency, and loss. - -This creates a problem when a topologically short but low-capacity link -exists alongside a longer but high-capacity path. The routing algorithm -will prefer the topologically closer peer, potentially saturating a slow -link while a higher-capacity path goes underutilized. - -**Proposed mitigation**: Each node locally measures the quality of its -direct peer links (RTT, bandwidth, loss) and incorporates this into a -`link_cost()` metric. The next-hop selection uses a composite ordering -of `(link_cost, tree_distance, node_addr)` — link quality takes priority -over topological distance. - -The `link_cost()` interface is implemented (currently returning a constant), -ready to be populated with real measurements using an established link -quality algorithm (ETX, Babel composite metric, etc.). - -This requires no protocol changes — link quality is measured locally, not -advertised. Self-reported cost claims are intentionally excluded from the -protocol to prevent adversarial traffic attraction (a node advertising -artificially low costs to become a transit point for surveillance or -disruption). Only locally-measured, first-hop metrics are used. - -## Enhancement Opportunity: Discovery Path Accumulation - -The LookupRequest flood naturally finds the lowest-latency path to the -destination — the first copy to arrive traveled the fastest route. Currently -this path information is discarded; only the destination's tree coordinates -survive in the LookupResponse. - -### Concept - -Each node forwarding a LookupRequest appends a signed path entry: - -```text -PathEntry { - node_addr: NodeAddr, // 16 bytes, forwarding node - signature: Signature, // 64 bytes, signs - // (request_id || position || node_addr) -} -``` - -The destination includes the accumulated path in the LookupResponse alongside -the existing `target_coords`. Per-hop signatures prevent path fabrication — a -malicious node cannot insert fake hops or claim a path it didn't traverse. - -### Potential Uses - -**Source peer bias**: The source examines the first hop of the discovered path -to learn which of its direct peers leads to the empirically fastest route to -the destination. This biases forwarding for that destination toward that peer, -complementing tree distance and local link quality measurements. The source can -verify this directly since the first hop is a direct peer. - -**Intermediate router hints**: Routers along the path can verify their own -position and adjacent hops (which should be direct peers). This gives them -empirical data about which peer directions lead toward specific destinations, -potentially informing their own forwarding decisions for future packets. - -**Coordinate cache seeding**: Intermediate routers on the discovered path learn -about both endpoints before data traffic begins, enabling pre-warming of -coordinate caches and reducing CoordsRequired errors on the first data packets. - -### Tradeoffs - -- **Size overhead**: 80 bytes per hop in the LookupRequest. An 8-hop path adds - 640 bytes. Acceptable for a one-time discovery message, not suitable for data - packets. -- **Staleness**: The path is a snapshot. Nodes may disconnect or links may - degrade after discovery. The path should be treated as a hint, not a - commitment. Greedy coordinate-based routing remains the primary forwarding - mechanism. -- **Latency vs capacity**: The fastest flood path is the lowest-latency path, - which is not necessarily the highest-capacity path. For bulk transfers, a - slower but higher-bandwidth path may be preferable. The path signal is most - useful for latency-sensitive traffic. - -### Interaction with Request Deduplication - -With strict `request_id` dedup at every node, the destination receives exactly -one request via the fastest path. If multiple candidate paths are desired (for -failover or load balancing), the destination could be exempted from dedup to -accept the first N arrivals, at the cost of generating multiple responses. - -## Open Questions - -1. **Coordinate compression**: Can tree coordinates be compressed for smaller - SessionSetup messages? (e.g., delta encoding, shorter node ID representation) - -2. **Multi-path routing**: How to handle multiple valid paths? Load balancing? - Failover? - -3. **Asymmetric paths**: S→D and D→S may traverse different routers. Is this - acceptable or should paths be symmetric? - -4. **Gateway /64 prefixes**: How do subnet prefixes interact with Bloom filters - and discovery? One filter entry per gateway regardless of devices behind it? - -5. **Cache sizing**: What's the right cache size for different node roles? - Core nodes vs. edge nodes? - -6. **Mobility**: When a node changes tree position (new parent), how quickly - do sessions recover? Should nodes announce position changes? - ---- - -## References - -- [fips-intro.md](fips-intro.md) — Overall FIPS architecture -- [fips-gossip-protocol.md](fips-gossip-protocol.md) — Wire formats for TreeAnnounce, FilterAnnounce, Lookup -- [fips-session-protocol.md](fips-session-protocol.md) — Traffic flow, crypto sessions, terminology -- [fips-wire-protocol.md](fips-wire-protocol.md) — Link-layer transport and Noise IK handshake -- [fips-transports.md](fips-transports.md) — Transport protocol characteristics -- [spanning-tree-dynamics.md](spanning-tree-dynamics.md) — Tree protocol dynamics and convergence diff --git a/docs/design/fips-session-layer.md b/docs/design/fips-session-layer.md new file mode 100644 index 0000000..8f48655 --- /dev/null +++ b/docs/design/fips-session-layer.md @@ -0,0 +1,402 @@ +# FIPS Session Protocol (FSP) + +The FIPS Session Protocol is the top protocol layer in the FIPS stack. It sits +above the FIPS Link Protocol (FLP) and below applications (native FIPS API or +IPv6 adapter). FSP provides end-to-end authenticated, encrypted datagram +delivery between any two FIPS nodes, regardless of how many intermediate hops +separate them. + +## Role + +FSP manages end-to-end communication sessions between FIPS nodes identified by +their public keys (npubs). Each session provides: + +- **End-to-end encryption**: Payload confidentiality independent of how many + intermediate nodes handle the traffic +- **Mutual authentication**: Both parties prove they control the private key + for their claimed identity +- **Replay protection**: Counter-based nonces with sliding window, tolerant of + UDP packet loss and reordering +- **Transport independence**: Sessions survive transport changes, route + changes, and address changes — they are bound to npub identities, not to + transport paths + +FSP is a datagram session protocol. It provides encrypted datagrams, not +reliable streams. There is no FIPS equivalent of TCP; if applications need +reliability, ordering, or flow control, they provide it themselves (typically +by running TCP over the FIPS IPv6 adapter). + +## Services Provided to Applications + +Applications access the FIPS mesh through two interfaces, both served by FSP: + +### Native FIPS API + +Applications address destinations directly by npub or public key. The FIPS +stack resolves the destination's node_addr, establishes or reuses a session, +encrypts the payload, and routes through FLP. No DNS involvement. + +### IPv6 Adapter + +Unmodified IPv6 applications use a TUN device with `fd::/8` routing. A local +DNS service maps npub → IPv6 address and primes the identity cache. Packets +arriving at the TUN are translated to FIPS datagrams and routed through FSP. + +See [fips-ipv6-adapter.md](fips-ipv6-adapter.md) for the IPv6 adaptation +layer. + +### What Applications Get + +- **Authenticated datagram delivery**: Each datagram is encrypted and + authenticated with session keys bound to both parties' npubs +- **Session transparency**: Sessions are established on demand and maintained + automatically. Applications send packets; FSP handles session setup, + encryption, and teardown. +- **Endpoint identity**: Applications address destinations by npub. The FIPS + address is the public key. + +### What Applications Do Not Get + +- **Reliability**: Datagrams may be lost, duplicated, or delivered out of + order. FSP provides no retransmission or ordering. +- **Path MTU discovery**: FSP does not signal MTU to applications. The IPv6 + adapter handles MTU enforcement via ICMP Packet Too Big and TCP MSS + clamping. +- **Congestion control**: FSP does not throttle traffic. Applications running + TCP over IPv6 get TCP's congestion control; native API applications must + manage their own sending rate. + +## Services Required from FLP + +FSP treats FLP as a black box providing three services. FSP knows nothing about +transports, transport addresses, links, peers, spanning trees, coordinates, +bloom filters, hop counts, or network topology. + +### SessionDatagram Forwarding + +FLP accepts a SessionDatagram (source node_addr, destination node_addr, hop +limit, payload) and delivers it best-effort toward the destination. Delivery +may traverse multiple hops, each with independent link encryption. + +### Error Signaling + +FLP signals routing failures asynchronously: + +- **CoordsRequired**: A transit node lacks the destination's tree coordinates. + FSP responds by re-initiating discovery and resetting the coordinate warmup + strategy. +- **PathBroken**: Greedy routing reached a dead end. FSP responds by + re-discovering the destination's current coordinates and resetting warmup. + +Both signals are generated by transit nodes (not the destination) and travel +back to the source inside a new SessionDatagram. They are plaintext (not +end-to-end encrypted) because transit nodes have no session with the source. + +### Local Delivery + +When a SessionDatagram arrives with a destination node_addr matching the local +node, FLP delivers it to FSP for session-layer processing. + +## Session Lifecycle + +### Session Establishment + +Sessions are established on demand when the first datagram needs to be sent to +a destination with no existing session. + +FSP uses Noise IK for session key agreement. The initiator knows the +destination's npub (from DNS lookup or native API); the responder learns the +initiator's identity from the handshake. This is the same asymmetry as +link-layer peer connections. + +The handshake is carried in SessionSetup and SessionAck messages: + +1. **Initiator** sends SessionSetup containing Noise IK msg1 and both + parties' tree coordinates +2. **Responder** processes msg1, learns initiator identity, sends SessionAck + containing Noise IK msg2 and its own coordinates +3. Both parties derive identical symmetric session keys + +Packets that trigger session establishment are queued (with bounded buffer) +and transmitted after the session is established. + +### Self-Bootstrapping + +SessionSetup is self-bootstrapping for routing. It carries the source's and +destination's tree coordinates in the clear (not inside the Noise payload). +As the message transits intermediate nodes, each node caches these coordinates, +warming the path for subsequent DataPackets that carry only addresses (no +coordinates). + +SessionAck carries the responder's coordinates back along the reverse path, +warming caches in the other direction. + +### Simultaneous Initiation + +When both nodes attempt to establish a session simultaneously ("crossing +hellos"), a deterministic tie-breaker resolves the conflict: + +- If `local_node_addr < remote_node_addr`: Continue as initiator, ignore + incoming setup +- If `local_node_addr > remote_node_addr`: Abort own initiation, switch to + responder role + +This ensures exactly one handshake completes. + +### Data Transfer + +Once established, sessions carry DataPacket messages containing encrypted +application data. Each DataPacket includes: + +- An explicit 8-byte counter for replay protection (used as the AEAD nonce) +- A flags byte (including COORDS_PRESENT for cache warming) +- The encrypted payload + +### Session Idle Timeout + +Sessions that see no traffic for a configurable duration (default 90s) are +torn down. When traffic resumes, a new session is established automatically. + +The idle timeout is deliberately shorter than the coordinate cache TTL (300s). +This ordering ensures that when traffic stops and the session tears down, the +transit node coordinate caches are still warm when a new session is established. +The fresh SessionSetup re-warms the caches, maintaining routing continuity. + +### Session Independence from Transport + +Sessions exist above the routing layer and are bound to npub identities, not +transport addresses or routing paths. A session survives: + +- Transport failover (UDP → Ethernet → back to UDP) +- Route changes (different intermediate hops) +- Transport address changes (IP address or port changes) +- Topology changes (direct peer becomes multi-hop or vice versa) + +## End-to-End Encryption + +### Noise IK Pattern + +FSP uses the same Noise IK pattern as FLP link encryption, but with +independent keys and sessions. The full Noise descriptor is +`Noise_IK_secp256k1_ChaChaPoly_SHA256`. + +The IK pattern: +- **msg1** (`→ e, es, s, ss`): Initiator sends ephemeral key, encrypts static + key to responder. Four DH operations establish session keys. +- **msg2** (`← e, ee, se`): Responder sends ephemeral key. Both parties now + share identical session keys. + +After the handshake, Noise produces two directional symmetric keys +(`send_key`, `recv_key`) used with ChaCha20-Poly1305 for all subsequent data. + +### Cryptographic Primitives + +| Component | Choice | Notes | +| --------- | ------ | ----- | +| Curve | secp256k1 | Nostr-native | +| DH | ECDH on secp256k1 | Standard EC Diffie-Hellman | +| Cipher | ChaCha20-Poly1305 | AEAD, same as NIP-44 | +| Hash | SHA-256 | Nostr-native | +| Key derivation | HKDF-SHA256 | Standard Noise KDF | + +These choices prioritize compatibility with the Nostr cryptographic stack. + +### secp256k1 Parity Normalization + +Nostr npubs encode x-only public keys (32 bytes, no y-coordinate parity). The +Noise IK pre-message mixes the responder's static key as a 33-byte compressed +key, and the default secp256k1 ECDH hash includes a parity-dependent version +byte. + +Both operations are normalized to be parity-independent: the pre-message hash +uses even parity (`0x02` prefix), and ECDH hashes only the x-coordinate of the +result point. This ensures handshakes succeed regardless of the responder's +actual key parity. + +### Privacy Note + +Noise IK does not provide initiator anonymity if the responder's static key is +compromised. An attacker who obtains the responder's nsec can decrypt the +initiator's identity from captured handshake messages. Noise XK would protect +initiator identity in this scenario but requires an additional round-trip (3 +handshake messages vs. 2). The privacy/latency tradeoff may be revisited with +deployment experience. + +### Data Packet Authentication + +FSP uses AEAD authentication only — no per-packet signatures. The Noise +handshake binds session keys to both parties' static keys, so only holders of +the corresponding nsecs can derive the session keys. This provides implicit +authentication for every packet, matching WireGuard and Lightning's approach. + +### Forward Secrecy + +Ephemeral keys in the Noise handshake provide forward secrecy. Compromise of +static keys (nsec) does not reveal past session keys, because session keys are +derived in part from ephemeral-ephemeral DH (`ee`), and ephemeral keys are +discarded after the handshake. + +## Replay Protection + +FSP uses explicit 8-byte counters on the wire for replay protection. Each side +maintains a monotonically increasing send counter, transmitted with every +DataPacket. The receiver maintains a sliding window (2048-entry bitmap) +tracking which counters have been seen. + +This design is critical for operation over unreliable transports. Under UDP +packet loss or reordering, implicit nonce counters (where the receiver +increments on each decrypt attempt) would desynchronize permanently — a failed +`decrypt()` increments the nonce, and the desync grows with each lost packet. +Explicit counters allow the receiver to decrypt any packet independently, +regardless of what packets were lost or reordered. + +The same `ReplayWindow` and `decrypt_with_replay_check()` implementation is +used at both the link and session layers. + +## COORDS_PRESENT Warmup Strategy + +Session establishment (SessionSetup/SessionAck) warms transit node coordinate +caches along the path. But coordinate caches have a finite TTL (default 300s), +and entries may be evicted under memory pressure. When a transit node's cache +entry expires, it cannot forward DataPackets (which carry only addresses, not +coordinates) and sends a CoordsRequired error. + +FSP uses a warmup-then-reactive strategy to keep transit caches populated: + +### Warmup Phase + +After session establishment, the first N DataPackets (configurable, default 5) +include both source and destination coordinates via the COORDS_PRESENT flag. +Transit nodes cache these coordinates as packets pass through, reinforcing the +path established by SessionSetup. + +### Steady State + +After the warmup count is reached, FSP clears the COORDS_PRESENT flag and +sends minimal DataPackets (4-byte header instead of ~136 bytes with +coordinates). Transit nodes serve from their coordinate caches. + +### Reactive Recovery + +When FSP receives a CoordsRequired signal: + +1. The warmup counter resets — subsequent DataPackets include coordinates again +2. A new LookupRequest may be initiated to rediscover the destination's + current coordinates +3. When the LookupResponse arrives for an established session, the warmup + counter resets again (handling the timing gap where warmup packets might + fire before transit caches are repopulated by discovery) + +When FSP receives a PathBroken signal: + +1. A LookupRequest is initiated to discover the destination's current + coordinates (which may have changed due to topology change) +2. The warmup counter resets + +Both signals are rate-limited at transit nodes (100ms per destination) to +prevent storms during topology changes. + +### Warmup State Machine + +```text + ┌──────────────┐ + │ WARMUP │ ◄── Send first N packets with coords + └──────┬───────┘ + │ N packets sent without CoordsRequired + ▼ + ┌──────────────┐ + │ MINIMAL │ ◄── Send packets without coords + └──────┬───────┘ + │ CoordsRequired or PathBroken received + ▼ + ┌──────────────┐ + │ WARMUP │ ◄── Counter reset, send coords again + └──────────────┘ +``` + +## Identity Cache + +The identity cache maps FIPS address prefix (15 bytes, the `fd::/8` IPv6 +address minus the `fd` prefix) to `(NodeAddr, PublicKey)`. This cache is +needed only when using the IPv6 adapter — the native FIPS API provides the +public key directly. + +The mapping is deterministic (derived from the public key via SHA-256) and +never becomes stale. The cache uses LRU-only eviction bounded by a +configurable size (default 10K entries). There is no TTL — entries are evicted +only when the cache is full and space is needed for a new entry. + +Cache population mechanisms: +- **DNS lookup**: The primary path. Resolving `npub1xxx...xxx.fips` derives + the IPv6 address and populates the identity cache. +- **Inbound traffic**: Authenticated sessions from other nodes populate the + cache with their identity information. + +## Coordinate Cache + +The coordinate cache maps `NodeAddr → TreeCoordinate` and is the critical +data structure that enables efficient multi-hop routing. Without cached +coordinates for a destination, FLP cannot make forwarding decisions and must +either fall back to bloom-filter-only routing or signal CoordsRequired. + +### Unified Cache + +The coordinate cache is a single unified cache (merged from previously +separate coord_cache and route_cache). All coordinate sources — SessionSetup +transit, COORDS_PRESENT DataPackets, LookupResponse — write to the same cache. + +### Eviction Policy + +- **TTL-based expiration**: Entries expire after a configurable duration + (default 300s) +- **Refresh on use**: Active routing through a cache entry resets its TTL, + keeping hot entries alive +- **LRU eviction**: When the cache is full, least recently used entries are + evicted first +- **Flush on parent change**: When the local node's tree parent changes, the + entire coordinate cache is flushed. Tree parent changes mean the node's own + coordinates have changed, making cached coordinates for other nodes + potentially stale for routing purposes. + +### Timer Ordering + +Cache and session timers are ordered so that idle sessions tear down before +transit caches expire: + +| Timer | Default | Purpose | +| ----- | ------- | ------- | +| Session idle timeout | 90s | Tear down unused sessions | +| Coordinate cache TTL | 300s | Expire stale coordinates | +| DNS TTL | 300s | Expire DNS resolutions | + +When traffic stops: the session tears down at 90s. When traffic resumes: DNS +re-resolves the identity, a fresh SessionSetup carries coordinates, and transit +node caches (still within their 300s TTL) are re-warmed. + +## Implementation Status + +| Feature | Status | +| ------- | ------ | +| Session establishment (Noise IK) | **Implemented** | +| End-to-end encryption (ChaCha20-Poly1305) | **Implemented** | +| Explicit counter replay protection | **Implemented** | +| COORDS_PRESENT warmup-then-reactive | **Implemented** | +| Identity cache (LRU-only) | **Implemented** | +| Coordinate cache (unified, TTL + refresh) | **Implemented** | +| Session idle timeout | **Implemented** | +| CoordsRequired handling | **Implemented** | +| PathBroken handling | **Implemented** | +| Simultaneous initiation tie-breaker | **Implemented** | +| Flush coord cache on parent change | **Implemented** | +| Rekey | Planned | +| Path MTU discovery | Planned | + +## References + +- [fips-intro.md](fips-intro.md) — Protocol overview and architecture +- [fips-link-layer.md](fips-link-layer.md) — FLP specification (below FSP) +- [fips-ipv6-adapter.md](fips-ipv6-adapter.md) — IPv6 adaptation layer (above FSP) +- [fips-mesh-operation.md](fips-mesh-operation.md) — Routing, discovery, and + error recovery +- [fips-wire-formats.md](fips-wire-formats.md) — Wire format reference for all + session message types diff --git a/docs/design/fips-session-protocol.md b/docs/design/fips-session-protocol.md deleted file mode 100644 index 2123b76..0000000 --- a/docs/design/fips-session-protocol.md +++ /dev/null @@ -1,848 +0,0 @@ -# FIPS Session Protocol Flow - -## Overview - -This document captures design considerations for FIPS protocol message flow, -including peer discovery, authentication, tree announcements, and data routing. - -### Access to the FIPS Datagram Service - -Applications can access FIPS datagram delivery through two interfaces: - -- **Native FIPS API**: Applications address destinations directly by npub or - public key. The FIPS stack resolves the destination's node_addr and routes - without any DNS involvement. This is the preferred interface for - FIPS-aware applications. - -- **IPv6 adapter (TUN interface)**: Unmodified IPv6 applications use a TUN - device with `fd::/8` routing. Because IPv6 addresses are one-way hashes of - public keys, a local DNS service maps npub → IPv6 address and primes the - identity cache so the TUN can route arriving packets. - -The remainder of this document details the sequence of actions initiated through -the IPv6 adapter path. The native FIPS API bypasses sections 1.1–1.4 (DNS -entry point, identity cache) entirely, entering the flow at route discovery. - ---- - -## 1. Application-Initiated Traffic Flow (IPv6 Adapter) - -> **Note**: This section applies to traditional IP-based applications using the -> TUN interface. Applications using the native FIPS API address destinations -> directly by npub or public key; routing proceeds from there without DNS. - -Traffic flow begins at the application layer with a DNS query, which triggers -a cascade of events through the FIPS stack. - -### 1.1 DNS as Entry Point - -An application wants to send IPv6 traffic to another FIPS node, identified by -an npub. The flow: - -1. **DNS Query**: Application queries the local FIPS DNS service for the npub - mapping to an IPv6 address - -2. **FIPS DNS service** performs two functions: - - **Address derivation**: Converts the npub to an identity and derives the - corresponding `fd::/8` IPv6 address - - **Cache priming**: Stores the identity mapping (IPv6 address ↔ npub ↔ node_addr) - in the local FIPS routing cache - -3. **DNS Response**: Returns the derived IPv6 address to the application - -4. **Packet Transmission**: Application sends IPv6 packet to the returned address, - which routes to the TUN interface via the `fd::/8` route - -5. **TUN Processing**: When the packet arrives at the TUN, FIPS already has the - cached mapping from the DNS lookup, enabling immediate routing decisions - -6. **Note**: This identity cache is only necessary when using the FIPS IPv6 shim - -### 1.2 Design Rationale - -Using DNS as the trigger point ensures the routing cache is populated *before* -packets arrive. This avoids: - -- Blocking packets while performing identity lookups -- Packet drops during cold-cache scenarios -- Complex async lookup machinery in the hot path - -The DNS server acts as a "routing intent" signal - if an application queries -for a destination, it likely intends to send traffic there. - -### 1.3 DNS Name Format - -NPUBs are represented as DNS names in the format: - -```text -npub1xxxxxx...xxxxx.fips -``` - -The FIPS DNS server recognizes names ending in `.fips` and extracts the npub -for address derivation. - -### 1.4 Identity Cache Lifetime - -The identity cache (IPv6 address ↔ npub ↔ node_addr) has the following lifetime -semantics: - -- **Configurable timeout**: Cache entries expire after a configured duration -- **Traffic refresh**: Timer resets to zero whenever traffic is sent to that - destination (LRU-style keep-alive) -- **TTL relationship**: Cache timeout MUST be longer than DNS TTL - -The TTL constraint ensures that while an application believes its DNS resolution -is valid (within TTL), the corresponding FIPS routing entry remains present. -Example: DNS TTL = 300s, Cache timeout = 600s. - -```text -DNS query → cache entry created (timeout = 600s) - ...traffic... → timeout reset to 600s - ...traffic... → timeout reset to 600s -DNS TTL expires (300s) → app may re-query, but cache still valid - ...no traffic for 600s... -Cache entry expires -``` - -### 1.5 Traffic Without Prior DNS Lookup - -A packet may arrive at the TUN for an `fd::/8` destination without a prior -DNS lookup (cached address, manual configuration, etc.). Since address -derivation is one-way (SHA-256), the npub cannot be recovered from the address, -and without the npub we cannot determine the node_addr needed for routing. - -FIPS returns ICMPv6 Destination Unreachable (Code 0: No route to destination) -for packets to unknown addresses. The identity cache must be populated before -traffic can be routed. - -Known cache population mechanisms: - -- DNS lookup (primary path, described above) -- Inbound traffic from authenticated peers - ---- - -## 2. TUN Reader Processing - -After DNS resolution, the application sends an IPv6 datagram to the destination -address. The kernel routes it to the TUN interface (via the `fd::/8` route), -where the FIPS TUN reader receives it. - -### 2.1 Packet Arrival - -```text -Application - │ - ▼ -IPv6 datagram (src=local_addr, dst=target_addr) - │ - ▼ -Kernel routing table: fd::/8 → fips0 - │ - ▼ -TUN reader receives raw IPv6 packet -``` - -### 2.2 TUN Reader Actions - -On receiving a packet, the TUN reader: - -1. **Validate IPv6 header**: Version = 6, payload length sane, etc. - -2. **Extract destination address**: The `fd::/8` address from the IPv6 header - -3. **Identity cache lookup**: Query cache for destination address - - **Miss**: Return ICMPv6 Destination Unreachable (see §1.5) - - **Hit**: Proceed with routing - -4. **Retrieve routing identity**: Cache hit provides: - - `npub`: The Nostr public key of the destination - - `node_addr`: SHA-256(pubkey), used for spanning tree routing - -5. **Session lookup**: Check for existing FIPS session with destination npub - - **Hit**: Use existing session for encryption/signing - - **Miss**: Initiate session establishment (see §3) - -6. **Route determination**: Using node_addr, determine the next hop peer: - - Check route cache for destination's spanning tree coordinates - - If cache miss, initiate route discovery (see §4.4) - - Select next hop via greedy routing toward destination coordinates - -7. **Packet forwarding**: Encapsulate and send via appropriate transport: - - Encrypt payload with session keys (end-to-end) - - Wrap in link-layer frame for next hop peer - - Encrypt with link keys and transmit via peer's transport - ---- - -## 3. FIPS Sessions - -A FIPS session represents a bidirectionally authenticated, encrypted channel -between two FIPS nodes. - -### 3.1 Session Properties - -Each session contains: - -- **Peer identity**: The remote node's npub and node_addr -- **Symmetric session keys**: Directional keys for encryption (send_key, recv_key) -- **Nonce counters**: Per-direction counters for replay protection - -Payloads within a session are: - -1. **Encrypted** with the session key (provides confidentiality) -2. **Authenticated** via AEAD tag (session keys bound to npub identities) - -Authentication derives from the Noise IK handshake binding session keys to -both parties' static keys. See §6 for cryptographic details. - -### 3.2 Session Establishment Trigger - -When the TUN reader has a packet for a destination with no existing session: - -```text -TUN reader - │ - ├─► Identity cache lookup → node_addr - │ - ├─► Session lookup (by npub) → MISS - │ - └─► Initiate session establishment -``` - -Packets that trigger session establishment are queued (with bounded buffer -management) and transmitted after the session is established. - -### 3.3 Session Independence from Transport - -FIPS sessions exist above the routing layer. A session between two npubs -survives: - -- Transport failover (UDP → WiFi → back to UDP) -- Route changes (different intermediate hops) -- Transport address changes on either end (WiFi → LTE → WiFi) - -The session is bound to **npub identities**, not transport addresses or routing -paths. This allows FIPS endpoints to roam over their transports as needed while -maintaining an established session. - -### 3.4 Session Establishment Flow - -FIPS uses Noise IK for session establishment. The initiator knows the -destination's npub; the responder learns the initiator's identity from the -handshake. This is the same asymmetry as link-layer connections. - -The handshake is carried inside SessionSetup/SessionAck messages (see §5.1), -which also establish routing session state at intermediate nodes. - -### 3.5 Simultaneous Session Initiation (Crossing Hellos) - -When both nodes attempt to establish a session simultaneously, a deterministic -tie-breaker resolves the conflict using node_addr ordering (consistent -with the link-layer cross-connection tie-breaker): - -- If local node_addr < remote node_addr: Continue as initiator, ignore incoming setup -- If local node_addr > remote node_addr: Abort own initiation, switch to responder role - -This ensures exactly one handshake completes with minimal wasted effort. - ---- - -## 4. FIPS Mesh Routing - -Below the session layer, all FIPS packets (session handshake messages, encrypted -payloads, control traffic) must be routed through the mesh to their destination. -See [fips-routing.md](fips-routing.md) for the full routing design. - -### 4.1 Routing Overview - -FIPS routing combines three mechanisms: - -1. **Bloom filters**: Fast reachability lookup for nearby destinations -2. **Discovery protocol**: Query-based lookup for distant destinations -3. **Greedy tree routing**: Coordinate-based forwarding using spanning tree position - -The routing layer maintains coordinate caches mapping `node_addr → -TreeCoordinate` (see [fips-routing.md](fips-routing.md) §4 for details). -Cache hits enable immediate routing via `find_next_hop()`; cache misses -trigger route discovery via bloom filter queries or LookupRequest flooding. - -### 4.2 Packet Handling During Discovery - -Packets are queued (with bounded buffer) while route discovery is in progress -and transmitted once coordinates are obtained. - -### 4.3 Coordinate Cache Lifetime - -Two caches store `node_addr → TreeCoordinate` mappings (see -[fips-routing.md](fips-routing.md) §4): - -- **CoordCache** (session-populated): TTL-based eviction (300s default), 50K entries -- **RouteCache** (discovery-populated): LRU eviction only, 10K entries - -Neither cache stores next-hop information — routing decisions are computed at -lookup time by `find_next_hop()` using cached coordinates. - ---- - -## 5. Route Cache Warming - -### 5.1 Initial Warming via Handshake - -The crypto session handshake (SessionSetup/SessionAck) warms route caches at -intermediate routers as it transits. Each message carries the sender's -coordinates; routers extract and cache `node_addr → TreeCoordinate` in the -CoordCache. After the handshake completes, data packets use minimal 38-byte -headers (34 envelope + 4 DataPacket) and routers forward via `find_next_hop()`. - -### 5.2 Cache Miss Recovery - -When an intermediate router's cache entry expires or is evicted, it cannot -forward data packets (which carry only addresses, not coordinates). The router -returns a CoordsRequired error to the sender. - -The crypto session remains valid—only the routing state is lost. Recovery uses -coords-on-demand: data packets include an optional coordinates field. When the -sender receives CoordsRequired: - -1. Sender marks the route as "cold" -2. Subsequent data packets include coordinates (flag bit set) -3. Routers along the path cache coordinates as packets transit -4. Once route is warm again, sender clears the flag and resumes minimal headers - -This avoids a full SessionSetup round-trip for what is purely a routing cache -refresh. - -### 5.3 DataPacket Coordinate Flag - -The DataPacket `flags` field includes a `COORDS_PRESENT` bit: - -| Bit | Meaning | -|-----|------------------------------------------------------| -| 0 | COORDS_PRESENT - coordinates follow the fixed header | - -When set, the packet includes `src_coords` and `dest_coords` after the standard -header fields. Routers process these coordinates the same way as SessionSetup: -cache both directions and forward using greedy routing. - -### 5.4 Sender State Machine - -```text - ┌──────────────┐ - │ WARM │ ◄── Normal: send minimal headers - └──────┬───────┘ - │ CoordsRequired received - ▼ - ┌──────────────┐ - │ COLD │ ◄── Send packets with coords - └──────┬───────┘ - │ N packets sent successfully - ▼ - ┌──────────────┐ - │ WARM │ - └──────────────┘ -``` - -The sender transitions back to WARM after sending a configurable number of -packets with coordinates (e.g., 3) without receiving CoordsRequired. This -provides confidence that caches along the path are populated. - ---- - -## 6. Session-Layer Encryption - -FIPS uses two independent Noise Protocol handshakes at different layers: - -| Layer | Scope | Pattern | Purpose | -|---------|-------------|----------|-------------------------------------------| -| Link | Hop-by-hop | Noise IK | Authenticate peers, encrypt link | -| Session | End-to-end | Noise IK | Authenticate endpoints, encrypt payload | - -Both use `Noise_IK_secp256k1_ChaChaPoly_SHA256` with the same cryptographic -primitives, but with separate keys and sessions. - -> **Privacy note**: Noise IK does not provide initiator anonymity if the -> responder's static key is compromised—an attacker who obtains the responder's -> nsec can decrypt the initiator's identity from captured handshake messages. -> Noise XK would protect initiator identity in this scenario, but requires an -> additional round-trip (3 handshake messages vs 2), increasing session setup -> from 3 packets to 4. Further deployment experience is needed to evaluate -> whether the privacy benefit justifies the latency cost. - -### 6.1 Why Two Layers? - -**Link encryption** protects against passive observers on each hop but allows -intermediate nodes to see routing information (destination address). - -**Session encryption** protects the actual payload end-to-end. Intermediate -nodes forward opaque ciphertext without being able to read the contents. - -### 6.2 Session Noise Handshake - -The session-layer Noise IK handshake is carried inside `SessionSetup`/`SessionAck` -messages, which themselves travel through the link-encrypted channel: - -```text -Initiator knows destination npub (from DNS lookup) - │ - ▼ -SessionSetup { coords, handshake_payload: Noise IK msg1 } - │ - ▼ (travels through link-encrypted hops) - │ -Responder processes msg1, learns initiator identity - │ - ▼ -SessionAck { coords, handshake_payload: Noise IK msg2 } - │ - ▼ -Session keys established (independent of link keys) -``` - -### 6.3 Cryptographic Primitives - -Both link and session layers use the same cryptographic stack: - -| Component | Choice | Notes | -|----------------|---------------------|----------------------------| -| Curve | secp256k1 | Nostr-native | -| DH | ECDH on secp256k1 | Standard EC Diffie-Hellman | -| Cipher | ChaCha20-Poly1305 | AEAD, same as NIP-44 | -| Hash | SHA-256 | Nostr-native | -| Key derivation | HKDF-SHA256 | Standard Noise KDF | - -These choices prioritize compatibility with existing Nostr infrastructure. -Secp256k1 and SHA-256 are already used for Nostr identities, and -ChaCha20-Poly1305 matches NIP-44 encryption. Lightning's BOLT 8 provides a -proven reference for adapting Noise Protocol to secp256k1. - -**secp256k1 parity normalization**: Nostr npubs encode x-only public keys -(32 bytes, no y-coordinate parity). The Noise IK pre-message mixes the -responder's static key as a 33-byte compressed key, and the default -secp256k1 ECDH hash includes a parity-dependent version byte. Since the -initiator may only have the x-only key, both operations are normalized to be -parity-independent: the pre-message hash uses even parity (`0x02` prefix), -and ECDH hashes only the x-coordinate of the result point. This ensures -handshakes succeed regardless of the responder's actual key parity. See -`secp256k1-parity-fix.md` for detailed analysis. - -### 6.4 Handshake Integration with SessionSetup - -The Noise handshake messages embed in SessionSetup/SessionAck, which are -carried inside a SessionDatagram envelope that provides addressing: - -```text -SessionDatagram { - // Addressing (used by routers for forwarding and error routing) - src_addr: NodeAddr, - dest_addr: NodeAddr, - hop_limit: u8, - - // Payload: SessionSetup - payload: SessionSetup { - // Routing portion (processed by routers for cache warming) - src_coords: Vec, - dest_coords: Vec, - - // Crypto portion (opaque to routers, processed by destination) - handshake_payload: Vec, // Noise IK message 1 - } -} - -SessionDatagram { - src_addr: NodeAddr, // Responder - dest_addr: NodeAddr, // Original sender - hop_limit: u8, - - payload: SessionAck { - // Routing portion - src_coords: Vec, // Responder's coordinates - - // Crypto portion - handshake_payload: Vec, // Noise IK message 2 - } -} -``` - -### 6.5 Session Keys - -After handshake completion, Noise produces two symmetric keys: - -- `send_key`: For encrypting outbound packets -- `recv_key`: For decrypting inbound packets - -These are used with ChaCha20-Poly1305 for all subsequent data packets. - -### 6.6 Nonce Management - -FIPS uses counter-based nonces for ChaCha20-Poly1305. Each side maintains a -64-bit send counter, incremented per packet. No coordination is needed since -keys are directional. The counter also enables replay detection by rejecting -packets with nonce ≤ last seen. - -### 6.7 Forward Secrecy - -The ephemeral keys (`e` in Noise notation) provide forward secrecy: - -- Compromise of static keys (npub/nsec) doesn't reveal past session keys -- Each session has unique ephemeral keys -- Session keys derived from ephemeral-ephemeral DH (`ee`) - -### 6.8 Reference: Lightning BOLT 8 - -Lightning's adaptation of Noise for secp256k1 (BOLT 8) provides a proven -reference implementation: - -- Uses Noise XK pattern (different from our IK) -- Same secp256k1 + ChaCha20-Poly1305 + SHA-256 stack -- Handles the secp256k1 ECDH correctly -- Open source implementations available in multiple languages - -FIPS can reference BOLT 8's cryptographic details while using the IK pattern. - -### 6.9 Data Packet Authentication - -**Decision**: Use AEAD authentication only (no per-packet signatures). - -The Noise handshake binds session keys to both parties' static keys. After -handshake completion: - -- Session keys are cryptographically tied to both npubs -- AEAD (ChaCha20-Poly1305) provides integrity and authenticity -- Only the holder of the session key can produce valid ciphertext -- Session keys can only be derived by holders of the corresponding nsecs - -Per-packet signatures would add: - -- 64 bytes overhead per packet -- Signing CPU cost (secp256k1 Schnorr) -- Verification CPU cost at receiver - -Since Noise already provides authentication through key binding, signatures -are redundant. This matches WireGuard and Lightning's approach. - ---- - -## 7. Peer Connection Establishment - -Before any session-layer traffic can flow, nodes must establish authenticated -link-layer connections with their peers using Noise IK. See -[fips-wire-protocol.md](fips-wire-protocol.md) for the complete wire protocol -specification including handshake flow, session lifecycle, index management, -roaming support, and transport-specific considerations. - -After successful Noise IK handshake: - -1. **Link encrypted**: All subsequent messages use AEAD encryption -2. **TreeAnnounce exchange**: Both peers send their current spanning tree state -3. **FilterAnnounce exchange**: Both peers send their bloom filters -4. **Peer is Active**: Can now participate in routing and forwarding - -The first TreeAnnounce from a new peer may trigger parent reselection if that -peer offers a better path to root. See [fips-gossip-protocol.md](fips-gossip-protocol.md) -for TreeAnnounce and FilterAnnounce wire formats. - ---- - -## 8. Session Layer Wire Format - -Session layer messages are carried inside `SessionDatagram` (type 0x40) at the -link layer. The session datagram provides source and destination addressing for -multi-hop forwarding: it is encrypted hop-by-hop with link keys, and the inner -payload is either encrypted end-to-end with session keys (for data traffic) or -plaintext (for link-layer error signals from transit routers). - -### 8.0 SessionDatagram Envelope - -The SessionDatagram is a link-layer message (type 0x40) that carries all -routable traffic through the mesh. It provides the source and destination -addresses that transit routers use for forwarding decisions. - -```text -┌─────────────────────────────────────────────────────────────────────────────┐ -│ SESSION DATAGRAM (0x40) │ -├────────┬──────────────────┬───────────┬─────────────────────────────────────┤ -│ Offset │ Field │ Size │ Description │ -├────────┼──────────────────┼───────────┼─────────────────────────────────────┤ -│ 0 │ msg_type │ 1 byte │ 0x40 (link-layer SessionDatagram) │ -│ 1 │ src_addr │ 16 bytes │ Source node_addr │ -│ 17 │ dest_addr │ 16 bytes │ Destination node_addr │ -│ 33 │ hop_limit │ 1 byte │ Decremented each hop │ -│ 34 │ payload │ variable │ Session-layer message (see §8.1) │ -└────────┴──────────────────┴───────────┴─────────────────────────────────────┘ - -Fixed header: 34 bytes -``` - -**src_addr**: The originator of the datagram. For data traffic, this is the -source endpoint. For error signals (CoordsRequired, PathBroken), this is the -transit router that generated the error. Transit routers use `src_addr` to -route error signals back to the packet's originator. - -**dest_addr**: The intended recipient. Transit routers use this for next-hop -selection via `find_next_hop()`. - -**hop_limit**: Decremented at each forwarding hop. When it reaches zero, the -datagram is dropped silently. This prevents infinite forwarding loops. Initial -value is configurable (default 64). - -### 8.1 Payload Message Types - -The SessionDatagram payload begins with a message type byte, followed by the -message-specific content. - -**Session-layer messages** (end-to-end encrypted with session keys): - -| Type Code | Message | Direction | Purpose | -|-----------|----------------|-----------|-----------------------------------| -| 0x00 | SessionSetup | S → D | Establish session + warm caches | -| 0x01 | SessionAck | D → S | Confirm session establishment | -| 0x10 | DataPacket | Both | Application data | - -**Link-layer error signals** (plaintext, generated by transit routers): - -| Type Code | Message | Direction | Purpose | -|-----------|----------------|-----------|-----------------------------------| -| 0x20 | CoordsRequired | R → S | Router cache miss | -| 0x21 | PathBroken | R → S | Greedy routing failed | - -Error signals are generated by transit router R and sent back to the source S -inside a *new* SessionDatagram with `src_addr=R, dest_addr=S`. They are -plaintext (not end-to-end encrypted) because the transit router has no session -with the source. Link-layer encryption protects them hop-by-hop. - -> **Address terminology**: The `src_addr` and `dest_addr` fields in the -> SessionDatagram header are node_addrs (16-byte truncated SHA-256 hashes of -> pubkeys). These are visible to intermediate routers for routing decisions. The -> actual FIPS addresses (pubkeys/npubs) are exchanged only during the Noise IK -> handshake and never appear in packet headers—routers cannot determine endpoint -> identities from the node_addrs they see. - -### 8.2 SessionSetup (0x00) - -Establishes a crypto session and warms router coordinate caches along the path. -The `src_addr` and `dest_addr` are carried in the enclosing SessionDatagram -envelope (§8.0); the SessionSetup payload carries only coordinates and the -Noise handshake. - -```text -┌─────────────────────────────────────────────────────────────────────────────┐ -│ SESSION SETUP PAYLOAD (inside SessionDatagram) │ -├────────┬──────────────────┬───────────┬─────────────────────────────────────┤ -│ Offset │ Field │ Size │ Description │ -├────────┼──────────────────┼───────────┼─────────────────────────────────────┤ -│ 0 │ msg_type │ 1 byte │ 0x00 │ -│ 1 │ flags │ 1 byte │ Bit 0: REQUEST_ACK │ -│ │ │ │ Bit 1: BIDIRECTIONAL │ -│ 2 │ src_coords_count │ 2 bytes │ u16 LE, number of src coord entries │ -│ 4 │ src_coords │ 16 × n │ NodeAddr array (self → root) │ -│ ... │ dest_coords_count│ 2 bytes │ u16 LE, number of dest coord entries│ -│ ... │ dest_coords │ 16 × m │ NodeAddr array (dest → root) │ -│ ... │ handshake_len │ 2 bytes │ u16 LE, Noise payload length │ -│ ... │ handshake_payload│ variable │ Noise IK msg1 (82 bytes typical) │ -└────────┴──────────────────┴───────────┴─────────────────────────────────────┘ -``` - -**Example** (depth 3 source, depth 4 destination, with Noise handshake): - -```text -SessionDatagram { src: S, dest: D, hop_limit: 64 } + -┌──────┬───────┬───────┬─────────────┬───────┬───────────────┬───────┬───────┐ -│ 0x00 │ 0x01 │ 0x03 │ src_coords │ 0x04 │ dest_coords │ 0x52 │ noise │ -│ type │ flags │ count │ 3 × 16 bytes│ count │ 4 × 16 bytes │ len=82│ msg1 │ -└──────┴───────┴───────┴─────────────┴───────┴───────────────┴───────┴───────┘ - -SessionDatagram header: 34 bytes -SessionSetup payload: 1 + 1 + 2 + 48 + 2 + 64 + 2 + 82 = 202 bytes -Total: 236 bytes -``` - -### 8.3 SessionAck (0x01) - -Confirms session establishment and completes the Noise handshake. Addressing -is in the enclosing SessionDatagram envelope. - -```text -┌─────────────────────────────────────────────────────────────────────────────┐ -│ SESSION ACK PAYLOAD (inside SessionDatagram) │ -├────────┬──────────────────┬───────────┬─────────────────────────────────────┤ -│ Offset │ Field │ Size │ Description │ -├────────┼──────────────────┼───────────┼─────────────────────────────────────┤ -│ 0 │ msg_type │ 1 byte │ 0x01 │ -│ 1 │ flags │ 1 byte │ Reserved │ -│ 2 │ src_coords_count │ 2 bytes │ u16 LE │ -│ 4 │ src_coords │ 16 × n │ Acknowledger's coords (for caching) │ -│ ... │ handshake_len │ 2 bytes │ u16 LE, Noise payload length │ -│ ... │ handshake_payload│ variable │ Noise IK msg2 (33 bytes typical) │ -└────────┴──────────────────┴───────────┴─────────────────────────────────────┘ -``` - -### 8.4 DataPacket (0x10) - -Carries encrypted application data (typically IPv6 payloads). Addressing and -hop limit are in the enclosing SessionDatagram envelope. - -```text -┌─────────────────────────────────────────────────────────────────────────────┐ -│ DATA PACKET (Minimal Header, inside SessionDatagram) │ -├────────┬──────────────────┬───────────┬─────────────────────────────────────┤ -│ Offset │ Field │ Size │ Description │ -├────────┼──────────────────┼───────────┼─────────────────────────────────────┤ -│ 0 │ msg_type │ 1 byte │ 0x10 │ -│ 1 │ flags │ 1 byte │ Bit 0: COORDS_PRESENT │ -│ 2 │ payload_length │ 2 bytes │ u16 LE │ -│ 4 │ payload │ variable │ Encrypted application data │ -└────────┴──────────────────┴───────────┴─────────────────────────────────────┘ - -Minimal header: 4 bytes (+ 34 SessionDatagram header = 38 bytes total) -``` - -When `COORDS_PRESENT` flag is set (route warming after CoordsRequired): - -```text -┌─────────────────────────────────────────────────────────────────────────────┐ -│ DATA PACKET (With Coordinates, inside SessionDatagram) │ -├────────┬──────────────────┬───────────┬─────────────────────────────────────┤ -│ Offset │ Field │ Size │ Description │ -├────────┼──────────────────┼───────────┼─────────────────────────────────────┤ -│ 0 │ msg_type │ 1 byte │ 0x10 │ -│ 1 │ flags │ 1 byte │ 0x01 (COORDS_PRESENT) │ -│ 2 │ payload_length │ 2 bytes │ u16 LE │ -│ 4 │ src_coords_count │ 2 bytes │ u16 LE │ -│ 6 │ src_coords │ 16 × n │ Source coordinates │ -│ ... │ dest_coords_count│ 2 bytes │ u16 LE │ -│ ... │ dest_coords │ 16 × m │ Destination coordinates │ -│ ... │ payload │ variable │ Encrypted application data │ -└────────┴──────────────────┴───────────┴─────────────────────────────────────┘ - -With depth-4 coords both directions: - DataPacket: 4 + 2 + 64 + 2 + 64 = 136 bytes header - + SessionDatagram: 34 bytes - Total: 170 bytes header -``` - -### 8.5 CoordsRequired (0x20) - -Sent by an intermediate router when it cannot forward a SessionDatagram due to -coordinate cache miss for the destination. This is a **link-layer error signal**: -the transit router generates a new SessionDatagram addressed back to the -original source, with the CoordsRequired payload in plaintext (not end-to-end -encrypted, since the transit router has no session with the source). - -**Error routing**: Router R receives a SessionDatagram `{src: S, dest: D}` but -cannot route to D. R creates a new SessionDatagram `{src: R, dest: S}` carrying -a CoordsRequired payload, and calls `find_next_hop(S)` to route it back to S. -If R also cannot route to S, the error is dropped silently (no cascading errors). - -**CoordsRequired payload** (inside SessionDatagram): - -```text -┌─────────────────────────────────────────────────────────────────────────────┐ -│ COORDS REQUIRED PAYLOAD │ -├────────┬──────────────────┬───────────┬─────────────────────────────────────┤ -│ Offset │ Field │ Size │ Description │ -├────────┼──────────────────┼───────────┼─────────────────────────────────────┤ -│ 0 │ msg_type │ 1 byte │ 0x20 │ -│ 1 │ flags │ 1 byte │ Reserved │ -│ 2 │ dest_addr │ 16 bytes │ The node_addr we couldn't route to │ -│ 18 │ reporter │ 16 bytes │ NodeAddr of reporting router │ -└────────┴──────────────────┴───────────┴─────────────────────────────────────┘ - -Payload: 34 bytes -Wrapped in SessionDatagram: 34 + 34 = 68 bytes total -``` - -### 8.6 PathBroken (0x21) - -Sent when greedy routing fails (no peer is closer to destination). Like -CoordsRequired, this is a **link-layer error signal** carried inside a new -SessionDatagram addressed back to the original source. - -**PathBroken payload** (inside SessionDatagram): - -```text -┌─────────────────────────────────────────────────────────────────────────────┐ -│ PATH BROKEN PAYLOAD │ -├────────┬──────────────────┬───────────┬─────────────────────────────────────┤ -│ Offset │ Field │ Size │ Description │ -├────────┼──────────────────┼───────────┼─────────────────────────────────────┤ -│ 0 │ msg_type │ 1 byte │ 0x21 │ -│ 1 │ flags │ 1 byte │ Reserved │ -│ 2 │ dest_addr │ 16 bytes │ The unreachable node_addr │ -│ 18 │ reporter │ 16 bytes │ NodeAddr of reporting router │ -│ 34 │ last_coords_count│ 2 bytes │ u16 LE │ -│ 36 │ last_known_coords│ 16 × n │ Stale coords that failed │ -└────────┴──────────────────┴───────────┴─────────────────────────────────────┘ -``` - -### 8.7 Full Packet Layout Example - -A DataPacket from source S to destination D, transiting router R: - -```text -┌─────────────────────────────────────────────────────────────────────────────┐ -│ UDP DATAGRAM │ -├─────────────────────────────────────────────────────────────────────────────┤ -│ │ -│ ┌───────────────────────────────────────────────────────────────────────┐ │ -│ │ FIPS LINK LAYER (S→R encrypted) │ │ -│ ├───────────┬──────────────┬────────────┬───────────────────────────────┤ │ -│ │ 0x00 │ R's recv_idx │ counter │ ciphertext + tag │ │ -│ │ 1 byte │ 4 bytes LE │ 8 bytes LE │ N + 16 bytes │ │ -│ └───────────┴──────────────┴────────────┴───────────────────────────────┘ │ -│ │ │ -│ ┌───────────────────────┘ │ -│ │ Decrypt with S↔R link keys │ -│ ▼ │ -│ ┌───────────────────────────────────────────────────────────────────────┐ │ -│ │ SESSION DATAGRAM (0x40) — plaintext for R │ │ -│ ├───────────┬──────────────┬──────────────┬──────────┬──────────────────┤ │ -│ │ 0x40 │ src_addr │ dest_addr │ hop_limit│ payload │ │ -│ │ msg_type │ S (16 bytes) │ D (16 bytes) │ 64 │ (session msg) │ │ -│ └───────────┴──────────────┴──────────────┴──────────┴──────────────────┘ │ -│ │ │ -│ │ R reads src_addr + dest_addr for routing │ -│ ▼ │ -│ ┌───────────────────────────────────────────────────────────────────────┐ │ -│ │ SESSION LAYER PAYLOAD (S↔D encrypted) │ │ -│ ├───────────┬───────┬──────────┬──────────┬─────────────────────────────┤ │ -│ │ 0x10 │ flags │ reserved │ pay_len │ encrypted application data │ │ -│ │ DataPacket│ 0x00 │ 0x00 │ 1400 │ (S↔D session keys + tag) │ │ -│ └───────────┴───────┴──────────┴──────────┴─────────────────────────────┘ │ -│ │ -└─────────────────────────────────────────────────────────────────────────────┘ - -Router R can see: src_addr, dest_addr, hop_limit (for routing/error signaling) -Router R cannot see: payload contents (encrypted with S↔D session keys) -``` - -### 8.7.1 Error Routing Example - -When router R cannot forward a SessionDatagram from S to D: - -```text -1. R receives SessionDatagram { src: S, dest: D, payload: DataPacket{...} } -2. R has no cached coordinates for D and no bloom filter hit -3. R creates NEW SessionDatagram: - { src: R, dest: S, hop_limit: 64, - payload: CoordsRequired { dest: D, reporter: R } } -4. R calls find_next_hop(S) to route the error back -5. If R can also not route to S: drop silently (no cascading errors) -``` - -This avoids the need for transit routers to have end-to-end sessions with -the source. The `src_addr` field in the original SessionDatagram tells the -transit router where to send the error. - -### 8.8 Encoding Rules - -- All multi-byte integers are **little-endian** -- NodeAddr is 16 bytes (truncated SHA-256 hash of npub) -- IPv6 addresses are 16 bytes (network byte order) -- Variable-length coordinate arrays use 2-byte u16 count prefix - -```text -Vec encoding: - count: u16 (little-endian) - items: [u8; 16] × count -``` diff --git a/docs/design/fips-software-architecture.md b/docs/design/fips-software-architecture.md index f043db2..17ed514 100644 --- a/docs/design/fips-software-architecture.md +++ b/docs/design/fips-software-architecture.md @@ -1,1101 +1,265 @@ # FIPS Software Architecture -**Status**: Design Draft +This document describes the stable architectural decisions that guide the +FIPS codebase — the "why" behind the code's shape. It covers design +principles and patterns that are expected to remain stable as the +implementation evolves. For protocol behavior and wire formats, see the +protocol layer documents. -This document describes the software architecture for a FIPS node implementation, -covering core entities, state machines, transport abstractions, and configuration. +## Ownership and Entity Hierarchy ---- +A FIPS node owns transports, which produce links, which authenticate into +peers: -## Overview - -FIPS is a Layer 3 mesh routing protocol that provides IPv6 connectivity over -heterogeneous link types. A FIPS node exposes a TUN interface to local applications, -routes packets via a spanning tree topology, and uses Bloom filters for efficient -reachability lookup. - -The architecture is event-driven with multiple focused state machines rather than -a single monolithic event handler. Control protocols are designed for eventual -consistency, tolerating packet loss without requiring acknowledgment/retry machinery. - ---- - -## Core Entities - -### Node - -The top-level entity representing a running FIPS instance. - -``` +```text Node -├── identity: Identity // cryptographic identity (keypair + derived IDs) -├── config: Config // loaded configuration -├── tun_state: TunState // TUN device lifecycle state -├── tree_state: TreeState // local view of spanning tree -├── coord_cache: CoordCache // address → coordinates for routing -├── transports: HashMap -├── links: HashMap -└── peers: HashMap +├── Transports (HashMap) +│ └── Each transport instance manages one communication medium +├── Links (HashMap) +│ └── Each link is a connection to a remote endpoint over a transport +├── Peers (HashMap) +│ └── Each peer is an authenticated remote FIPS node +├── TreeState — local view of the spanning tree +├── CoordCache — destination coordinates for routing +├── Sessions — end-to-end FSP sessions (HashMap by NodeAddr) +└── Identity — this node's cryptographic identity ``` -### Identity +**Key ownership rules**: -Cryptographic identity using Nostr keys (secp256k1). +- A transport exists for the lifetime of the node (configured at startup) +- A link is created when connecting to a remote endpoint and destroyed when + the connection terminates +- A peer is created when a link successfully authenticates (Noise IK + handshake) and destroyed when the link goes down +- Links and peers have a one-to-one mapping with coupled lifecycles — + peer teardown implies link teardown -``` -Identity -├── keypair: Keypair // secp256k1 keypair (secret + public) -├── node_addr: NodeAddr // SHA-256(pubkey) truncated, 16 bytes -└── address: FipsAddress // IPv6 ULA derived from node_addr (fd::/8) +## Event-Driven Execution Model + +The node uses an async select loop as its main event loop, multiplexing +events from all sources into a single processing stream: + +- **Transport events**: Inbound datagrams from all transports arrive via a + shared mpsc channel +- **Timer events**: Periodic and one-shot timers for keepalive, stale peer + detection, cache expiry, handshake timeouts +- **TUN events**: Outbound IPv6 packets from local applications +- **Control events**: Identity registrations from DNS, shutdown signals + +Within the select loop, events are dispatched to focused handler functions +organized by concern (handshake processing, gossip handling, forwarding, +session management, timeout handling). Each handler operates on the node's +state directly — there is no separate message-passing between internal +components. + +**Why a single select loop**: FIPS protocol operations frequently need to +read and modify multiple pieces of state (e.g., forwarding a packet reads +the coordinate cache, peer ancestry, and bloom filters simultaneously). +A single-threaded event loop avoids the complexity of locking and provides +deterministic ordering of state changes. + +**Exceptions**: The TUN reader and writer run in separate blocking threads +because TUN I/O is blocking (kernel file descriptor). They communicate with +the main event loop via channels. + +## Phase-Based State Machine Pattern + +FIPS entities use a Rust enum-of-structs pattern for state machines where +each phase carries only the data relevant to that phase: + +```rust +enum PeerSlot { + Connecting(PeerConnection), // handshake in progress + Active(ActivePeer), // authenticated, participating +} ``` -Accessor methods provide `pubkey()` (x-only), `pubkey_full()`, `npub()` (bech32), -`node_addr()`, and `address()`. The keypair is also exposed for Noise handshakes. +Each variant holds a different struct with phase-appropriate fields. The +`PeerConnection` struct carries handshake state; `ActivePeer` carries tree +position, bloom filters, and link statistics. Transitioning between phases +consumes the old struct and produces the new one, making it impossible to +access handshake state after authentication is complete. -`NodeAddr` is the routing identifier, derived deterministically from `npub`. -Transport addresses and FIPS identity are fully decoupled. +This pattern enforces at the type level that code handling an authenticated +peer cannot accidentally reference handshake state, and vice versa. -### Protocol Layer Visibility +See [fips-state-machines.md](fips-state-machines.md) for a detailed +treatment of this pattern. -| Observer | Link Addr | Node Addr | FIPS Addr (pubkey) | Payload | -|-------------------------------|---------------|--------------|--------------------| --------| -| Transport (IP router, switch) | Yes | No | No | No | -| FIPS routing node | Last hop only | Yes (header) | No | No | -| Destination endpoint | Yes | Yes | Yes | Yes | +## Two-Layer Encryption Rationale -**Key insight**: Three independent encryption layers ensure: +FIPS uses independent Noise IK encryption at two layers: -- Passive transport observers see only encrypted blobs -- FIPS routing nodes see node_addrs but not pubkeys (FIPS addresses) or payload -- Only endpoints know each other's FIPS addresses (pubkeys) and can decrypt payload +| Layer | Scope | What It Protects | +| ----- | ----- | ---------------- | +| FLP (link) | Hop-by-hop | All traffic on each peer link | +| FSP (session) | End-to-end | Application payload between endpoints | -### Transport +**Why two layers instead of one**: -A physical or logical interface over which links can be established. Transports -are statically configured and exist for the lifetime of the node after startup. +- **Link encryption** protects all traffic from passive observers on the + underlying transport — including routing metadata (TreeAnnounce, bloom + filters, discovery messages) that would otherwise be visible +- **Session encryption** protects application payloads from intermediate + routing nodes, which must decrypt link encryption to read routing headers +- Both layers always apply. For adjacent peers, traffic is encrypted twice. + This eliminates special cases ("local peer" vs. "remote destination") and + means topology changes (a direct peer becomes multi-hop) don't affect + sessions. -``` -Transport (trait) -├── transport_id: TransportId -├── transport_type: TransportType -├── config: TransportConfig -├── state: TransportState -├── mtu: u16 -│ -├── start() -> Result<()> -├── stop() -> Result<()> -├── send(addr: &TransportAddr, data: &[u8]) -> Result<()> -├── recv() -> Result<(TransportAddr, Vec)> -└── discover() -> Result> -``` +**Why the same pattern (Noise IK) at both layers**: Both layers need mutual +authentication with identity hiding for the initiator. Reusing the same +cryptographic stack (secp256k1, ChaCha20-Poly1305, SHA-256) simplifies the +implementation and reduces the number of cryptographic dependencies. -**Transport metadata (static per type):** +## Identity Model -``` -TransportType -├── name: &'static str // "udp", "ethernet", "wifi", "tor" -├── connection_oriented: bool // requires link establishment? -└── reliable: bool // delivery guaranteed by transport? -``` +FIPS uses three related but distinct identifiers at different layers: -Transports handle framing, fragmentation, and any transport-layer encryption -internally. The FIPS routing layer sees only FIPS packets. - -### Link - -A communication channel to a specific remote endpoint over a transport. Links are -created on demand when connecting to a peer and torn down when the peer connection -terminates. Link lifecycle is driven by the peer lifecycle. - -``` -Link -├── link_id: LinkId -├── transport_id: TransportId -├── remote_addr: TransportAddr // opaque, transport-specific -├── direction: Inbound | Outbound -├── state: LinkState // trivial for connectionless, real for Tor -├── base_rtt: Duration // hint from transport type -└── io: ... // connection handles for connection-oriented -``` - -For connectionless transports (UDP, Ethernet, WiFi), links are lightweight—just -`(transport_id, remote_addr)` with implicit "established" state (no connection -setup required). - -For connection-oriented transports (Tor), links track real connection state and -hold I/O handles. The link must complete transport-layer connection setup before -FIPS session establishment can proceed. - -**Link statistics (measured):** - -``` -LinkStats -├── packets_sent: u64 -├── packets_recv: u64 -├── bytes_sent: u64 -├── bytes_recv: u64 -├── last_recv: Timestamp -├── rtt_estimate: Duration // measured from probes -├── loss_rate: f32 // observed (meaningful for unreliable) -└── throughput_estimate: u64 // bytes/sec observed -``` - -### Peer Lifecycle (Two-Phase Model) - -Peers use a two-phase lifecycle managed by a `PeerSlot` enum: - -``` -PeerSlot -├── Connecting(PeerConnection) // handshake in progress -└── Active(ActivePeer) // authenticated, participating -``` - -**PeerConnection** — represents a peer during Noise IK handshake: - -``` -PeerConnection -├── node_addr: NodeAddr // routing identity (if known) -├── link_id: LinkId // which link reaches this peer -├── state: HandshakeState // Initiating | ReceivedMsg1 | AwaitingMsg2 -├── handshake: NoiseHandshake // Noise IK state machine -└── created_at: Instant // for timeout enforcement -``` - -**ActivePeer** — an authenticated remote FIPS node: - -``` -ActivePeer -├── identity: PeerIdentity // cryptographic identity (verified via handshake) -├── node_addr: NodeAddr // routing identity -├── link_id: LinkId // which link reaches this peer -├── state: ConnectivityState // Active | Stale | Disconnecting -│ -│ // Spanning tree -├── declaration: Option // their latest (None until received) -├── ancestry: Option // their path to root (None until received) -│ -│ // Bloom filter (inbound—what's reachable through them) -├── inbound_filter: Option // None until first received -├── filter_sequence: u64 -├── filter_received_at: u64 // Unix milliseconds -├── pending_filter_update: bool // we owe them an update -│ -│ // Statistics -└── link_stats: LinkStats -``` - -**Peer/Link Lifecycle:** - -Links and peers have a one-to-one mapping with coupled lifecycles: - -1. **Outbound connection**: Desire to connect to a peer triggers link creation - over the appropriate transport. For connection-oriented transports (Tor), the - link goes through connection setup; for connectionless (UDP), it immediately - becomes established. Once the link is ready, FIPS authentication proceeds. - -2. **Inbound connection**: Incoming data on a transport creates a link, then - authentication creates the peer. - -3. **Peer references link**: An authenticated peer always references exactly one - active link. - -4. **Termination**: When a peer connection terminates, the associated link is - torn down. For connectionless transports this is trivial cleanup; for - connection-oriented transports this closes the underlying connection. - -If the same remote node is reachable via multiple transports, that would be -multiple Peer entries (though for initial implementation, we assume single -transport per peer). - ---- - -## Spanning Tree State - -### Per-Node State - -``` -TreeState -├── my_declaration: ParentDeclaration -│ ├── node_addr: NodeAddr -│ ├── parent_id: NodeAddr // self if root candidate -│ ├── sequence: u64 // monotonic -│ └── signature: Signature -├── my_coords: Vec // [self, parent, grandparent, ..., root] -└── root: NodeAddr // elected root (smallest reachable node_addr) -``` - -### Per-Peer State - -From each peer, we receive and store: - -- Their `ParentDeclaration` -- Their `ancestry` (path from peer to root) - -This provides their tree coordinates for routing decisions. - -### Bounded State - -Each node's TreeState contains O(P × D) entries, not O(N): - -- P = direct peer count -- D = tree depth - -A node knows only: - -1. Its own parent declaration -2. Direct peers' parent declarations -3. Ancestry chains from each peer to root - -Nodes do NOT know about other subtrees—only paths toward root. - ---- - -## Bloom Filter State - -### Per-Node State - -``` -BloomState -├── own_node_addr: NodeAddr // always included in outgoing filters -├── leaf_dependents: HashSet // leaf-only nodes we speak for -├── is_leaf_only: bool // if true, no filter processing -├── update_debounce_ms: u64 // min interval between updates (ms) -├── last_update_sent: HashMap // per-peer last-sent timestamp -├── pending_updates: HashSet // peers needing filter update -├── sequence: u64 // monotonic outgoing sequence number -└── last_sent_filters: HashMap // for change detection -``` - -### Per-Peer State - -Stored on ActivePeer: - -- `inbound_filter: Option`: what they advertise to us (None until first received) -- `filter_sequence: u64`: freshness/dedup -- `filter_received_at: u64`: when received (Unix milliseconds), for staleness detection -- `pending_filter_update: bool`: whether we owe them an update - -### Computed (On-Demand) - -Outgoing filter to peer Q is computed, not stored: - -``` -outbound_filter(Q) = - own_node_addr - ∪ leaf_dependents - ∪ { entries from peer[P].inbound_filter for all P ≠ Q } -``` - -Recomputation is cheap (1KB filter, 5 hashes) so on-demand is preferred over -cache invalidation complexity. - ---- - -## State Machines - -### Transport Lifecycle - -``` - Configured ──► Starting ──► Up ──► Down - │ │ │ - v v │ - Failed ◄────────────┘ -``` - -- `Configured`: in config, not started -- `Starting`: initialization in progress (instant for UDP, slow for Tor) -- `Up`: ready for links -- `Down`: was up, now unavailable -- `Failed`: couldn't start - -**Events:** - -- `Start` (from config policy or API) -- `Started` / `StartFailed` -- `Shutdown` -- `TransportError` - -**Cascading:** Transport down → all links over it disconnect → all peers on -those links disconnect. - -### Link Lifecycle - -**Connectionless transports (UDP, Ethernet, WiFi):** - -Links are always implicitly "active"—no state machine needed. Link exists when -we have `(transport_id, remote_addr)`. - -**Connection-oriented transports (Tor):** - -``` -Outbound: - (connect requested) ──► Connecting ──► Connected ──► Disconnected - │ ▲ - v │ - Failed ─────────────────────────┘ - -Inbound: - (transport accepts) ──► Connected ──► Disconnected -``` - -- `Connecting`: establishing connection (circuit for Tor) -- `Connected`: ready for FIPS traffic -- `Disconnected`: was connected, now gone -- `Failed`: connection attempt failed - -### Peer Lifecycle - -The peer lifecycle uses Noise IK for authentication. Noise IK is a 2-message -handshake where the initiator knows the responder's static key. See -[fips-wire-protocol.md](fips-wire-protocol.md) §2 for wire format details. - -``` - ┌─────────────────────────────────────────┐ - │ Disconnected │ - └─────────────────────────────────────────┘ - │ │ - [outbound] │ │ [inbound msg1] - ▼ ▼ - ┌──────────────────┐ ┌──────────────────┐ - │ Connecting │ │ ReceivedMsg1 │ - │(conn-oriented) │ │ (send msg2) │ - └──────────────────┘ └──────────────────┘ - │ │ - [link ready] │ │ [recv encrypted] - [send msg1] │ │ [verify] - ▼ │ - ┌──────────────────┐ │ - │ AwaitingMsg2 │ │ - │ (sent msg1) │ │ - └──────────────────┘ │ - │ │ - [recv msg2] │ │ - [verify] │ │ - ▼ ▼ - ┌─────────────────────────────────────────┐ - │ Active │ - │ (tree gossip, filter exchange) │ - └─────────────────────────────────────────┘ - │ - │ [link down / timeout] - ▼ - ┌─────────────────────────────────────────┐ - │ Disconnected │ - │ (retry if static peer) │ - └─────────────────────────────────────────┘ -``` - -**State descriptions:** - -- `Disconnected`: No active connection; for static peers, retry with backoff -- `Connecting`: Link establishment in progress (connection-oriented transports only) -- `ReceivedMsg1`: Inbound; received Noise IK msg1, sent msg2, awaiting first encrypted frame -- `AwaitingMsg2`: Outbound; sent Noise IK msg1, waiting for msg2 -- `Active`: Authenticated; participating in tree gossip and filter exchange - -**Crossing connection handling:** - -When in `AwaitingMsg2` and we receive a msg1 from the same peer (both sides -initiated simultaneously): - -- If local node_addr < remote node_addr: Ignore incoming msg1, remain initiator -- If local node_addr > remote node_addr: Switch to responder role, send msg2, - transition to `ReceivedMsg1` - -**Events:** - -``` -PeerEvent -├── Discovered { transport_id, transport_addr, pubkey_hint: Option } -├── LinkConnected -├── LinkFailed { reason } -├── Msg1Received { noise_payload } -├── Msg2Received { noise_payload } -├── HandshakeComplete { npub, node_addr } -├── HandshakeFailed { reason } -├── TreeAnnounceReceived { declaration, ancestry } -├── FilterAnnounceReceived { filter, sequence, ttl } -├── Timeout { kind: TimeoutKind } -├── PacketReceived { ... } -└── LinkDisconnected { reason } -``` - ---- - -## Reference Transport Types - -### UDP/IP - -``` -UdpTransport -├── bind_addr: SocketAddr // e.g., 0.0.0.0:4000 -├── socket: UdpSocket -└── state: TransportState - -TransportAddr = SocketAddr // IP:port -``` - -| Property | Value | -|----------|-------| -| Connection-oriented | No | -| Reliable | No | -| MTU | 1280-1472 | -| Latency | Low (1-500ms) | -| Scope | Internet | -| Discovery | DNS-SD, Nostr, static config | -| Privileges | None | -| NAT | Hole punching possible | - -### Ethernet - -``` -EthernetTransport -├── interface: String // "eth0" -├── socket: RawSocket // AF_PACKET -├── local_mac: MacAddr -├── ethertype: u16 // FIPS ethertype -└── state: TransportState - -TransportAddr = MacAddr // 6 bytes -``` - -| Property | Value | -|----------|-------| -| Connection-oriented | No | -| Reliable | No | -| MTU | 1500 | -| Latency | <1ms | -| Scope | Local segment | -| Discovery | Multicast | -| Privileges | CAP_NET_RAW | - -### WiFi - -``` -WifiTransport -├── interface: String // "wlan0" -├── socket: RawSocket -├── local_mac: MacAddr -├── mode: Infrastructure | AdHoc | Direct -└── state: TransportState - -TransportAddr = MacAddr // same as Ethernet -``` - -| Property | Value | -|----------|-------| -| Connection-oriented | No | -| Reliable | No | -| MTU | 1500 | -| Latency | 1-10ms | -| Scope | Local segment (or Direct group) | -| Discovery | Multicast, P2P service discovery | -| Privileges | CAP_NET_RAW | - -Infrastructure and Ad-hoc modes behave like Ethernet. WiFi Direct has its own -service discovery mechanism. - -### Tor Onion - -``` -TorTransport -├── tor_client: TorClient // arti or external daemon -├── onion_service: Option -├── local_onion_addr: Option -└── state: TransportState - -TransportAddr = OnionAddr // "abc...xyz.onion:port" -``` - -| Property | Value | -|----------|-------| -| Connection-oriented | Yes | -| Reliable | Yes (stream) | -| MTU | Stream (framed) | -| Latency | 500ms-5s | -| Scope | Internet (anonymous) | -| Discovery | Nostr, static config | -| Privileges | None | -| Transport startup | Slow (30s-2min for Tor bootstrap) | - -Tor links require framing (length-prefix) over the stream. The .onion address -is independent of FIPS npub—identity verified via FIPS auth after connecting. - -### Transport Comparison - -| Aspect | UDP | Ethernet | WiFi | Tor | -|--------|-----|----------|------|-----| -| Connection | No | No | No | Yes | -| Link state machine | Trivial | Trivial | Trivial | Real | -| Address type | IP:port | MAC | MAC | .onion:port | -| Startup time | Instant | Instant | Instant | 30s-2min | -| Base RTT hint | 50ms | 1ms | 5ms | 2s | -| Framing | Datagram | Datagram | Datagram | Length-prefix | - ---- - -## Event-Driven Architecture - -The system uses multiple focused state machines rather than one giant event -handler: - -1. **Transport state machines** — one per transport instance -2. **Link state machines** — one per link (meaningful for connection-oriented) -3. **Peer state machines** — one per peer -4. **Spanning tree module** — reacts to peer events, emits announcements -5. **Bloom filter module** — reacts to peer/filter events, emits updates - -**Event flow:** - -``` -Transport +```text +keypair (secp256k1) │ - ├──► DiscoveredPeer { transport_id, addr, pubkey_hint } - ├──► InboundConnection { transport_id, addr, io } (connection-oriented) - └──► PacketReceived { transport_id, addr, data } - │ - v - Link - │ - ├──► LinkConnected { link_id } - ├──► LinkDisconnected { link_id, reason } - └──► FipsPacketReceived { link_id, packet } - │ - v - Peer - │ - ├──► AuthSuccess { peer_id } - ├──► TreeAnnounceReceived { peer_id, decl, ancestry } - └──► FilterAnnounceReceived { peer_id, filter, seq, ttl } - │ - v - TreeState / BloomState + ├── pubkey (32 bytes) — the endpoint identity, used in Noise handshakes + │ + ├── node_addr = SHA-256(pubkey)[0..16] — routing identifier, visible to + │ transit nodes, cannot be reversed to pubkey + │ + └── IPv6 address = fd + node_addr[0..15] — overlay address for IPv6 + applications ``` -Timers drive keepalives, timeouts, and periodic refresh (debounced announcements). +**Privacy property**: Transit nodes see only node_addrs in packet headers. +They can forward traffic without knowing the Nostr identities of the +endpoints. An observer can verify "does this node_addr belong to pubkey X?" +but cannot enumerate communicating identities from traffic alone. ---- +**Self-sovereign**: Nodes generate their own identities without coordination. +The identity system uses Nostr keypairs (secp256k1), so existing npub/nsec +pairs work directly. ## Protocol Self-Healing Design -Control protocols tolerate packet loss without ack/retry machinery: +FIPS control protocols are designed for eventual consistency, tolerating +packet loss without acknowledgment/retry machinery: -### TreeAnnounce +| Protocol | Self-Healing Property | +| -------- | --------------------- | +| TreeAnnounce | Full state with monotonic sequence; lost announcement recovered on next send | +| FilterAnnounce | Full filter replacement with sequence; stale filter recovered on next update | +| LookupRequest | Timeout-based retry at application level | +| SessionSetup | Timeout-based retry; lost setup triggers re-establishment on first data failure | +| CoordsRequired/PathBroken | Rate-limited, best-effort; lost error recovered by session idle timeout | -- Monotonic sequence numbers (receiver keeps highest, ignores stale/dup) -- Full state (declaration + ancestry), not deltas -- Periodic refresh ensures convergence -- Lost announcement? Next one carries same or newer state +**Why no ack/retry**: FIPS operates over unreliable transports (primarily +UDP). Adding reliability to control messages would require per-message state, +retransmission timers, and acknowledgment tracking — complexity that gossip +protocols avoid by sending full state periodically. A lost TreeAnnounce is +simply replaced by the next one, which carries the same or newer state. -### FilterAnnounce +## Bounded State Principle -- Full filter replacement with sequence number -- Periodic refresh -- Debounced on rapid changes -- Lost announcement? Peer has stale filter until next update +FIPS nodes maintain state proportional to O(P × D), where P is the number +of direct peers and D is the tree depth — not O(N) where N is the network +size. -### LookupRequest/LookupResponse +What each node stores: -- Request-response pattern -- Lost request/response → sender times out, retries at application level +| State | Size | Scope | +| ----- | ---- | ----- | +| Peer ancestry (TreeAnnounce) | P × D entries | Direct peers only | +| Bloom filters | P × 1 KB | One per peer | +| Coordinate cache | Configurable (50K default) | Destinations actively routed | +| Identity cache | Configurable (10K default) | IPv6 adapter only | +| Sessions | Configurable (10K default) | Active end-to-end sessions | -### SessionSetup/SessionAck +A node does not know about nodes in distant parts of the network. It knows +its direct peers, their tree positions, and the destinations it has recently +routed traffic to. This scales naturally: adding nodes to the network does +not increase the per-node state of existing nodes (except for a slight +increase in bloom filter occupancy). -- Same as lookup—sender retries on timeout -- Lost setup → first data packet fails → triggers re-establishment +## Transport Opacity -This gossip-style eventual consistency is simpler and avoids the complexity of -per-message reliability over unreliable links. +Transport addresses are opaque byte vectors above FLP. The transport layer +interprets them (e.g., UDP parses "ip:port" strings); all layers above treat +them as handles passed back to the transport for sending. ---- +**Architectural boundary**: Adding a new transport type (e.g., BLE) requires +implementing the transport trait and potentially a new `TransportHandle` +variant. No changes to FLP, FSP, or any routing logic. The transport trait +defines the interface: -## Leaf-Only Operation +- `send(addr, data)` — send a datagram +- `mtu()` — maximum datagram size +- `start()` / `stop()` — lifecycle +- `discover()` — optional endpoint discovery -> **Implementation status**: Leaf-only mode is partially implemented. The -> `node.leaf_only` config flag and `BloomState::leaf_only()` (bloom filter -> suppression) exist. The following are **not yet implemented**: routing tunnel -> to upstream peer, tree announcement suppression, simplified peer structure, -> `leaf_dependents` tracking on the upstream side, and LookupRequest proxying. -> A leaf-only node currently runs the full Node code path with bloom filters -> disabled. +Inbound datagrams are pushed via a shared channel, aggregating all transports +into a single event stream for the main loop. -Leaf-only mode enables constrained devices (sensors, battery-powered nodes, mobile -devices) to participate in FIPS without the overhead of full mesh routing. +## Cache Architecture -### Architectural Subset +### Unified Coordinate Cache -A leaf-only node uses a minimal subset of the full architecture: +The coordinate cache maps `NodeAddr → TreeCoordinate`. It was originally two +separate caches (session-populated and discovery-populated) but was merged +into a single cache because both stored the same type of data and the +distinction was conceptual, not functional. -``` -LeafOnlyNode -├── identity: Identity // full (npub, nsec, node_addr, address) -├── config: Config // simplified -├── tun: TunInterface // full (provides IPv6 to local apps) -├── transport: Transport // one -├── link: Link // one -└── upstream_peer: Peer // one (simplified) -``` +Key properties: -### What's Required +- **TTL-based expiration** (300s default) with **refresh on use** — active + routing resets the TTL, keeping hot entries alive +- **LRU eviction** when full — least recently used entries are evicted first +- **Flush on parent change** — when the local node's tree parent changes, + the entire cache is flushed because the node's own coordinates have + changed, making cached distance calculations potentially invalid -| Component | Usage | -|---------------------|------------------------------------------| -| Identity | Full—same cryptographic identity model | -| TUN interface | Full—provides IPv6 to local applications | -| Transport | One instance (to reach upstream peer) | -| Link | One instance (to upstream peer) | -| Peer | One instance (upstream), with auth only | -| Peer authentication | Full—must prove identity to upstream | -| Packet send/receive | Full | +### Identity Cache (LRU-Only) -### What's Not Required +The identity cache maps FIPS address prefix → (NodeAddr, PublicKey). The +mapping is deterministic (derived from public key) and never becomes stale, +so there is no TTL — only LRU eviction bounded by a configurable size. -| Component | Reason | -|--------------------|--------------------------------------------------| -| TreeState | No tree participation; upstream handles routing | -| ParentDeclaration | Doesn't announce position to network | -| Bloom filters | Upstream peer handles reachability | -| Filter computation | N/A | -| CoordCache | Doesn't route for others | -| Discovery protocol | Upstream peer handles lookups | -| Multiple peers | Single upstream by design | -| Session caching | Tunnels everything to upstream | -| Transit routing | Never forwards for others | +This cache is needed only by the IPv6 adapter. The native FIPS API provides +the public key directly. -### Simplified Peer Structure +### Timer Ordering -The upstream peer entry for a leaf-only node: - -``` -UpstreamPeer (leaf-only) -├── identity: PeerIdentity // cryptographic identity -├── node_addr: NodeAddr -├── link_id: LinkId -├── state: ConnectivityState // auth lifecycle only -└── link_stats: LinkStats // for keepalive/timeout - -// NOT present: -// - declaration, ancestry (no tree participation) -// - inbound_filter, filter_* (no Bloom filters) -``` - -### Routing Behavior - -``` -Outbound (local app → network): - TUN → upstream peer (unconditionally) - -Inbound (network → local app): - upstream peer → TUN (if dest == self) - upstream peer → DROP (if dest ≠ self, never transit) -``` - -The leaf-only node doesn't make routing decisions—it tunnels everything to/from -its upstream peer. - -### Upstream Peer Responsibilities - -The upstream peer (a full participant) handles: - -- Including leaf-only node in its Bloom filter -- Responding to LookupRequests for the leaf-only node -- Forwarding packets to/from the leaf-only node -- The leaf-only node appears as an entry in the upstream's `leaf_dependents` set - -### State Machine (Simplified) - -Only peer lifecycle matters: - -``` - Configured ──► Connecting ──► Authenticating ──► Active ──► Disconnected - │ │ │ - v v │ - [fail] ─────────────────────────────────────────┘ -``` - -No TreeAnnounce or FilterAnnounce processing in the Active state. - -### Configuration (Leaf-Only Subset) - -``` -# Required -node.identity.nsec # or auto-generated -node.leaf_only = true -node.tun.device -node.tun.mtu - -# One transport -transport.udp.enabled = true # or ethernet, wifi, tor -transport.udp.bind_addr - -# One peer (the upstream) -peers[0].npub # required: upstream identity -peers[0].addresses[0].type # transport type -peers[0].addresses[0].addr # how to reach them -peers[0].connect_policy = auto_connect - -# Timeouts -peer.auth.timeout -peer.keepalive.interval -peer.keepalive.timeout -peer.reconnect.* -``` - -Parameters NOT relevant to leaf-only operation: - -``` -tree.* # no tree participation -filter.* # no Bloom filters -discovery.* # upstream handles -session.* # no session caching -transport.*.discovery.* # single configured peer -transport.*.auto_connect # single configured peer -``` - -### Resource Comparison - -| Resource | Full Participant | Leaf-Only | -|---------------------|-----------------------|-----------| -| RAM (Bloom filters) | d × 4KB (d = peers) | 0 | -| RAM (CoordCache) | 50K entries, 300s TTL | 0 | -| RAM (RouteCache) | 10K entries, LRU | 0 | -| RAM (tree state) | O(P × D) entries | 0 | -| Bandwidth (idle) | < 1 KB/sec | Near zero | -| CPU (filter ops) | Moderate | None | -| Peers | Multiple | One | - -### Use Cases - -- **IoT sensors**: Send telemetry, receive commands -- **Mobile devices**: Battery/bandwidth constraints -- **Privacy-conscious**: Don't see others' traffic -- **Monitoring nodes**: Observe network, don't route -- **Embedded systems**: Limited RAM/CPU - ---- - -## Node Startup Sequence - -The startup sequence initializes components in dependency order: +Cache and session timers are ordered to ensure correct lifecycle behavior: ```text -1. Load configuration - ├── Parse config files (system, user, local) - ├── Validate transport and peer configurations - └── Merge with defaults - -2. Initialize identity - ├── Load nsec from config (or generate if absent) - ├── Derive npub, node_addr, and FIPS address - └── Log identity information - -3. Initialize transports - ├── Create transport instances from config - └── Transports in Configured state - -4. Start transports (begin listening) - ├── Bind sockets, open interfaces - ├── Transports transition to Up state - └── Ready to accept inbound connections - -5. Connect to static peers - ├── For each configured peer with AutoConnect policy: - │ ├── Create link via appropriate transport - │ ├── Send AuthInit to initiate authentication - │ └── On success: peer joins tree gossip - └── Failed connections enter retry with backoff - -6. Node operational - ├── Participating in spanning tree (even with 0 peers) - ├── Processing inbound connections - └── Retrying unreachable static peers in background +Session idle timeout (90s) < Coordinate cache TTL (300s) ≤ DNS TTL (300s) ``` -**Notes:** +When traffic stops, the session tears down first (90s). When traffic +resumes, a fresh SessionSetup re-warms transit caches that are still within +their TTL (300s). This ordering prevents the case where a session outlives +its transit cache entries, which would cause routing failures. -- Transports start listening (step 4) before outbound connections (step 5) to - accept inbound connections from peers who have us configured -- The node is "operational" as soon as any peer authenticates successfully -- Static peer connection attempts continue in background with retry policy -- With 0 authenticated peers, the node considers itself a potential root +## Receive Path Design -### Static Peer Retry Policy +Transports use a channel-push model rather than a poll/receive method. Each +transport takes a sender handle (`PacketTx`) at construction and spawns an +internal receive loop that pushes inbound datagrams onto the channel. The +node's main select loop reads from the corresponding receiver. -When a static peer is unreachable or authentication fails: +**Why push, not poll**: Async Rust cannot express async methods on trait +objects (the `Transport` trait is synchronous). The channel-push model works +around this limitation: the concrete transport implementation (e.g., +`UdpTransport`) spawns its own async receive task and pushes to a channel, +while the trait surface remains synchronous for `send()`, `mtu()`, etc. -| Parameter | Default | Description | -|-----------|---------|-------------| -| Initial delay | 1s | First retry delay | -| Max delay | 300s | Cap on exponential backoff | -| Backoff factor | 2.0 | Multiplier per attempt | -| Jitter | ±25% | Randomization to avoid thundering herd | -| Max attempts | unlimited | Static peers retry indefinitely | - -The retry timer resets to initial delay after a successful connection that -later disconnects. - -### Inbound Connection Acceptance - -For initial implementation, all inbound connections that successfully -authenticate are accepted. Future versions may add: - -- Peer allowlists/blocklists -- Connection limits per transport -- Rate limiting on authentication attempts -- Reputation-based acceptance - ---- - -## Configuration - -### Peer Configuration - -Peers are configured at the node level, separately from transports. For initial -implementation, only static peers with `AutoConnect` policy are supported; -discovery-based and on-demand peering are future enhancements. - -```text -PeerConfig -├── npub: PublicKey // required: who is this -├── alias: Option // human-readable label -├── addresses: Vec // how to reach them -└── connect_policy: ConnectPolicy // AutoConnect for initial impl - -PeerAddress -├── transport_type: TransportType // "udp", "ethernet", "tor", etc. -├── addr: String // transport-specific, parsed by driver -└── priority: u8 // preference order (lower = preferred) - -ConnectPolicy -├── AutoConnect // connect on startup (initial impl) -├── OnDemand // connect when traffic needs routing (future) -└── Manual // wait for explicit API call (future) -``` - -**Example configuration (YAML):** - -```yaml -node: - peers: - - npub: "npub1abc..." - alias: "gateway" - addresses: - - transport: udp - addr: "192.168.1.1:4000" - priority: 1 - - transport: tor - addr: "xyz...abc.onion:4000" - priority: 2 - connect_policy: auto_connect -``` - -### Transport Configuration - -``` -TransportConfig -├── transport_type: TransportType -├── driver_config: DriverConfig // type-specific -├── start_policy: StartPolicy -├── discovery_enabled: bool -└── auto_connect: bool // auto-connect to discovered peers -``` - -### Discovery - -Discovery is per-transport: - -- Transports emit `DiscoveredPeer { transport_id, addr, pubkey_hint }` events -- Node matches against known peer configs or creates "unknown peer" entries -- Policy (`auto_connect`, per-peer `connect_policy`) determines action - ---- - -## Configuration Reference (sysctl-style) - -### Node Identity - -| Parameter | Type | Default | Description | -|-----------|------|---------|-------------| -| `node.identity.nsec` | string | (generated) | Secret key (nsec1... or hex) | - -### TUN Interface - -| Parameter | Type | Default | Description | -|-----------|------|---------|-------------| -| `node.tun.device` | string | "fips0" | TUN device name | -| `node.tun.mtu` | u16 | 1280 | TUN interface MTU | - -### Resource Limits - -| Parameter | Type | Default | Description | -|-----------|------|---------|-------------| -| `limits.max_peers` | u32 | 128 | Maximum concurrent authenticated peers | -| `limits.max_transports` | u8 | 8 | Maximum configured transports | -| `limits.max_pending_auth` | u32 | 32 | Maximum connections awaiting authentication | -| `limits.max_pending_lookups` | u32 | 1000 | Maximum in-flight discovery lookups | -| `limits.memory_budget` | u64 | 0 | Soft memory limit in bytes (0 = unlimited) | - -Limits are enforced at connection time. Exceeding `max_pending_auth` rejects new -inbound connections; exceeding `max_peers` prevents new outbound connections. -The `memory_budget` is advisory—implementations should shed load when approaching -the limit but need not enforce it strictly. - -### Spanning Tree - -| Parameter | Type | Default | Description | -|-----------|------|---------|-------------| -| `tree.announce.interval` | duration | 30s | Periodic TreeAnnounce refresh | -| `tree.announce.on_change` | bool | true | Immediate announce on parent change | -| `tree.parent.hold_time` | duration | 10s | Min time before switching parent | -| `tree.parent.hysteresis` | f32 | 0.1 | Cost improvement threshold to switch | - -### Bloom Filters - -| Parameter | Type | Default | Description | -|-----------|------|---------|-------------| -| `filter.size_class` | u8 | 1 | Size class: 0=512B, 1=1KB, 2=2KB, 3=4KB | -| `filter.hash_count` | u8 | 5 | Number of hash functions | -| `filter.scope` | u8 | 2 | TTL for filter propagation (K) | -| `filter.refresh.interval` | duration | 60s | Periodic FilterAnnounce refresh | -| `filter.update.debounce` | duration | 500ms | Min interval between updates | -| `filter.stale.threshold` | duration | 300s | Consider peer's filter stale | - -v1 protocol requires `size_class=1` (1 KB filters). The size_class field is -present in the wire format for forward compatibility with larger filters. - -### Discovery Protocol - -| Parameter | Type | Default | Description | -|-----------|------|---------|-------------| -| `discovery.lookup.ttl` | u8 | 64 | Initial TTL for LookupRequest | -| `discovery.lookup.timeout` | duration | 10s | Timeout waiting for response | -| `discovery.lookup.retry_count` | u8 | 3 | Retries before giving up | -| `discovery.cache.max_entries` | u32 | 10000 | Route cache size | - -The discovery route cache (`RouteCache`) stores coordinates learned from -LookupResponses for destinations this node wants to reach. It uses LRU eviction -only (no automatic TTL expiration). This is the primary cache for endpoint nodes. - -### Routing Session Management - -> **Terminology note**: These parameters configure *routing sessions*—hop-by-hop -> cached state at intermediate routers. For *crypto session* (end-to-end -> encryption) parameters, see the Crypto Session section below. See -> [fips-session-protocol.md](fips-session-protocol.md) §3 for crypto sessions -> and §5 for route cache warming. - -| Parameter | Type | Default | Description | -|-----------|------|---------|-------------| -| `session.setup.timeout` | duration | 5s | SessionSetup ack timeout | -| `session.cache.max_entries` | u32 | 50000 | Coord cache size (per router) | -| `session.cache.ttl` | duration | 300s | Cached coordinates expiry | -| `session.refresh.interval` | duration | 240s | Proactive session refresh | - -The session coordinate cache (`CoordCache`) stores coordinates learned from -SessionSetup/SessionAck packets passing through this node as a transit router. -Larger than the discovery route cache since routers see traffic for many -destinations. Uses TTL-based expiration (300s) with LRU fallback. - -### Crypto Session Management - -> **Note**: Crypto sessions provide end-to-end authenticated encryption using -> Noise IK. See [fips-session-protocol.md](fips-session-protocol.md) §6 for details. - -| Parameter | Type | Default | Description | -|-----------|------|---------|-------------| -| `crypto.session.max_entries` | u32 | 10000 | Max concurrent crypto sessions | -| `crypto.session.idle_timeout` | duration | 3600s | Expire idle sessions | -| `crypto.session.rekey_interval` | duration | 86400s | Rekey after this interval | -| `crypto.session.rekey_bytes` | u64 | 0 | Rekey after N bytes (0 = disabled) | - -Crypto sessions are keyed by remote npub and survive transport changes. The -handshake is carried within SessionSetup/SessionAck messages (combined -establishment). - -### Peer Defaults - -| Parameter | Type | Default | Description | -|-----------|------|---------|-------------| -| `peer.auth.timeout` | duration | 10s | Auth handshake timeout | -| `peer.auth.timeout_tor` | duration | 60s | Auth timeout for Tor links | -| `peer.keepalive.interval` | duration | 30s | Keepalive probe interval | -| `peer.keepalive.timeout` | duration | 90s | Declare peer dead after silence | -| `peer.reconnect.policy` | enum | backoff | none, immediate, backoff | -| `peer.reconnect.delay_initial` | duration | 1s | Initial reconnect delay | -| `peer.reconnect.delay_max` | duration | 300s | Maximum reconnect delay | -| `peer.reconnect.max_attempts` | u32 | 0 | 0 = unlimited | - -### Transport: UDP - -| Parameter | Type | Default | Description | -|-----------|------|---------|-------------| -| `transport.udp.enabled` | bool | true | Enable UDP transport | -| `transport.udp.bind_addr` | string | "0.0.0.0:4000" | Bind address | -| `transport.udp.discovery.enabled` | bool | true | Enable discovery | -| `transport.udp.discovery.dns_sd` | bool | false | Use DNS-SD discovery | -| `transport.udp.discovery.nostr_relays` | list | [] | Relays for peer discovery | -| `transport.udp.auto_connect` | bool | true | Connect to discovered peers | -| `transport.udp.base_rtt` | duration | 50ms | RTT hint for timeouts | - -### Transport: Ethernet - -| Parameter | Type | Default | Description | -|-----------|------|---------|-------------| -| `transport.ethernet.enabled` | bool | false | Enable Ethernet transport | -| `transport.ethernet.interface` | string | "eth0" | Interface name | -| `transport.ethernet.ethertype` | u16 | 0x88b5 | FIPS EtherType | -| `transport.ethernet.discovery.enabled` | bool | true | Enable multicast discovery | -| `transport.ethernet.discovery.multicast_addr` | string | "33:33:00:00:ff:05" | Discovery multicast MAC | -| `transport.ethernet.auto_connect` | bool | true | Connect to discovered peers | -| `transport.ethernet.base_rtt` | duration | 1ms | RTT hint for timeouts | - -### Transport: WiFi - -| Parameter | Type | Default | Description | -|-----------|------|---------|-------------| -| `transport.wifi.enabled` | bool | false | Enable WiFi transport | -| `transport.wifi.interface` | string | "wlan0" | Interface name | -| `transport.wifi.mode` | enum | infrastructure | infrastructure, adhoc, direct | -| `transport.wifi.discovery.enabled` | bool | true | Enable discovery | -| `transport.wifi.auto_connect` | bool | true | Connect to discovered peers | -| `transport.wifi.base_rtt` | duration | 5ms | RTT hint for timeouts | - -### Transport: Tor - -| Parameter | Type | Default | Description | -|-----------|------|---------|-------------| -| `transport.tor.enabled` | bool | false | Enable Tor transport | -| `transport.tor.mode` | enum | embedded | embedded, external | -| `transport.tor.control_port` | string | "127.0.0.1:9051" | External daemon control | -| `transport.tor.onion_service.enabled` | bool | true | Publish onion service | -| `transport.tor.onion_service.port` | u16 | 4000 | Onion service port | -| `transport.tor.discovery.enabled` | bool | true | Enable discovery | -| `transport.tor.discovery.nostr_relays` | list | [] | Relays for peer discovery | -| `transport.tor.auto_connect` | bool | false | Connect to discovered peers | -| `transport.tor.base_rtt` | duration | 2s | RTT hint for timeouts | -| `transport.tor.startup_timeout` | duration | 180s | Tor bootstrap timeout | - -### Adaptive Timeouts - -| Parameter | Type | Default | Description | -|-----------|------|---------|-------------| -| `timeout.adaptive.enabled` | bool | true | Use measured RTT for timeouts | -| `timeout.adaptive.rtt_multiplier` | f32 | 3.0 | Timeout = RTT × multiplier | -| `timeout.adaptive.min` | duration | 100ms | Minimum timeout | -| `timeout.adaptive.max` | duration | 60s | Maximum timeout | - -### Leaf-Only Mode - -| Parameter | Type | Default | Description | -|-----------|------|---------|-------------| -| `node.leaf_only` | bool | false | Operate as leaf-only node | - ---- +The `TransportHandle` enum provides async dispatch for methods that need it +(like `send_async()`) without requiring dyn dispatch. ## References -- [fips-intro.md](fips-intro.md) — Overall FIPS protocol design -- [fips-session-protocol.md](fips-session-protocol.md) — Traffic flow, session terminology, crypto sessions -- [fips-transports.md](fips-transports.md) — Transport protocol characteristics -- [fips-routing.md](fips-routing.md) — Routing, Bloom filters, discovery -- [spanning-tree-dynamics.md](spanning-tree-dynamics.md) — Tree protocol dynamics +- [fips-intro.md](fips-intro.md) — Protocol overview +- [fips-link-layer.md](fips-link-layer.md) — FLP specification +- [fips-session-layer.md](fips-session-layer.md) — FSP specification +- [fips-state-machines.md](fips-state-machines.md) — Phase-based state + machine pattern +- [fips-configuration.md](fips-configuration.md) — YAML configuration + reference diff --git a/docs/design/fips-spanning-tree.md b/docs/design/fips-spanning-tree.md new file mode 100644 index 0000000..ecc0eb5 --- /dev/null +++ b/docs/design/fips-spanning-tree.md @@ -0,0 +1,258 @@ +# FIPS Spanning Tree + +This document describes the spanning tree algorithms and data structures +used by FIPS for coordinate-based routing. It is a supporting reference +for readers who want to understand the tree internals — for how the +spanning tree fits into the overall mesh operation, see +[fips-mesh-operation.md](fips-mesh-operation.md). + +## Purpose + +The spanning tree gives every node in the mesh a **coordinate** — its +ancestry path from itself to the root. These coordinates enable: + +- Distance calculation between any two nodes without global topology + knowledge +- Greedy routing where each hop reduces distance to the destination +- Loop-free forwarding guaranteed by strictly-decreasing distance + +## Root Election + +The root is the node with the **lexicographically smallest node_addr** among +all reachable nodes. There is no election protocol, no voting, no +negotiation. Each node independently evaluates the TreeAnnounce messages +from its peers and selects the minimum root. + +When a node first joins the network with no peers, it is its own root. As +it connects to peers and receives their TreeAnnounce messages, it discovers +smaller node_addrs and converges to the global root. + +If the network partitions, each segment independently elects its own root +(the smallest node_addr in that segment). When segments rejoin, all nodes +discover the globally-smallest root through TreeAnnounce exchange and +reconverge to a single tree. + +## Parent Selection + +Each node selects a single parent from among its direct peers. Parent +selection follows these rules: + +### Selection Criteria + +1. **Find the smallest root** visible across all peers' TreeAnnounce messages +2. Among peers that can reach that root, prefer the one offering the + **shallowest depth** (shortest path to root) +3. Apply the **depth improvement threshold**: switching parents requires the + proposed parent to offer a path at least 1 hop shallower than the current + parent (when the root is the same) + +### Immediate Switch Triggers + +Three conditions bypass the depth threshold and trigger immediate parent +reselection: + +1. **Parent loss**: Current parent is no longer in the peer set (link + broken, peer disconnected) +2. **Better root**: A peer advertises a smaller root than the current + tree's root — always switch regardless of depth +3. **Depth improvement**: Same root, but the proposed parent offers depth + at least `PARENT_SWITCH_THRESHOLD` (1 hop) better than the current parent + +### After Parent Change + +When a node changes its parent: + +1. Increment its own sequence number +2. Recompute its coordinates from the new ancestry path +3. Sign a new TreeAnnounce declaration +4. Announce to all peers +5. Flush the coordinate cache (cached coordinates are relative to the old + position and may be invalid for routing) + +## Coordinate Computation + +A node's coordinate is its full ancestry path from itself to the root: + +```text +coords(N) = [N, Parent(N), Parent(Parent(N)), ..., Root] +``` + +Coordinates are ordered self-to-root. For a node D at depth 4: + +```text +coords(D) = [D, P1, P2, P3, Root] +``` + +The root's coordinate is simply `[Root]` (depth 0). + +## Tree Distance + +Tree distance between two nodes is the number of hops through their lowest +common ancestor (LCA). Because coordinates are ordered self-to-root, common +ancestry appears as a common suffix. + +```text +tree_distance(a, b): + lca_depth = longest_common_suffix_length(a.coords, b.coords) + a_to_lca = len(a.coords) - lca_depth + b_to_lca = len(b.coords) - lca_depth + return a_to_lca + b_to_lca +``` + +Example: If A has coordinates `[A, X, Y, Root]` and B has coordinates +`[B, Z, Y, Root]`, the common suffix is `[Y, Root]` (length 2). Distance = +(3 - 2) + (3 - 2) = 2 hops. + +The self-distance check in greedy routing uses this calculation: a packet is +forwarded to a peer only if the peer is strictly closer to the destination +than the current node. + +## TreeAnnounce Processing + +When a node receives a TreeAnnounce from peer P: + +1. **Validate version**: Reject if version ≠ 0x01 +2. **Verify signature**: Check P's declaration signature using P's known + public key (established during Noise IK handshake) +3. **Verify identity**: Confirm the declaration's node_addr matches the + sender's known identity +4. **Check freshness**: If `sequence ≤ stored sequence for P`, discard + (stale or duplicate) +5. **Update peer state**: Store P's tree declaration and ancestry +6. **Evaluate parent selection**: Re-run parent selection with the updated + peer state + +### Propagation Rules + +A node re-announces (propagates) only when its own state changes: + +- **Root changed**: Always propagate — this is a significant topology event +- **Depth changed**: Always propagate — affects routing distance calculations +- **Sequence-only refresh**: Does NOT propagate beyond depth 1 — peers that + receive a sequence-only update do not re-announce, because their own root + and depth have not changed + +This means TreeAnnounce cascades through the tree proportional to depth, +not network size. A change at depth D affects at most D nodes along the +branch, and each only re-announces to its peers. + +### Rate Limiting + +- **Minimum interval**: 500ms between announcements to the same peer +- **Coalescing**: If changes occur during cooldown, they are coalesced and + sent as a single announcement after the cooldown expires +- **Convergence time**: A tree of depth D reconverges in roughly D × 0.5s + to D × 1.0s + +### Transitive Trust (v1) + +In the v1 protocol, only the sender's outer signature on the TreeAnnounce +is verified. Ancestry entries beyond the direct peer (the sender's parent, +grandparent, etc.) are accepted on transitive trust through the +authenticated sender. The sender is a known, authenticated peer — if it +claims a particular ancestry, v1 trusts that claim. + +Future protocol versions may add per-entry signatures in the ancestry chain +for stronger verification. + +## Sequence Numbers and Timestamps + +### Sequence Number + +- Type: u64, monotonically increasing +- Incremented on each parent change +- Used for freshness: incoming TreeAnnounce with sequence ≤ stored sequence + for that peer is discarded +- Higher sequence numbers always supersede lower ones + +### Timestamp + +- Type: u64, Unix seconds +- Used for stale detection, not versioning +- A root declaration is considered stale after `ROOT_TIMEOUT` (60 minutes) + without refresh + +## Reconvergence + +### Single Node Failure + +When a node fails (link timeout or disconnect): + +1. Nodes that had the failed node as their parent lose their parent +2. Parent loss triggers immediate reselection from remaining peers +3. Each affected node recomputes coordinates and announces +4. Changes cascade down the subtree proportional to depth + +### Partition + +When the network partitions: + +1. Nodes in each segment lose peers across the partition boundary +2. If the root was in the other segment, affected nodes elect a new segment + root (smallest node_addr in their segment) +3. Each segment reconverges independently + +### Partition Merge + +When two partitions rejoin: + +1. Nodes at the boundary exchange TreeAnnounce messages with new peers +2. Both segments discover each other's root +3. The globally-smaller root wins; the other segment's nodes switch parents +4. Coordinate caches are flushed at switching nodes (stale cross-partition + coordinates) +5. Bloom filters update within ~500ms per hop, restoring reachability + information + +## Bounded State + +Each node's spanning tree state is O(P × D), where P is the number of +direct peers and D is the tree depth. This is NOT O(N) where N is the +network size. + +What a node stores: +- Its own declaration (coordinates, sequence, timestamp, signature) +- Each peer's declaration and ancestry chain (P entries, each with D + ancestry entries) + +What a node does NOT know: +- Other subtrees branching off its ancestors +- Siblings of ancestors +- Nodes in distant parts of the network + +Example: In a 1000-node network with depth 10 and 5 peers, a node stores +~50 ancestry entries — not 1000 routing table entries. + +## Timing Parameters + +| Parameter | Default | Description | +| --------- | ------- | ----------- | +| PARENT_SWITCH_THRESHOLD | 1 hop | Minimum depth improvement for same-root switch | +| ANNOUNCE_MIN_INTERVAL | 500ms | Minimum between announcements to same peer | +| ROOT_TIMEOUT | 60 min | Root declaration considered stale | +| TREE_ENTRY_TTL | 5–10 min | Individual entry expiration | + +## Implementation Status + +| Feature | Status | +| ------- | ------ | +| Root election (smallest node_addr) | **Implemented** | +| Parent selection with depth threshold | **Implemented** | +| Coordinate computation | **Implemented** | +| TreeAnnounce gossip | **Implemented** | +| Signature verification (outer) | **Implemented** | +| Sequence-based freshness | **Implemented** | +| Rate limiting (500ms per peer) | **Implemented** | +| Coord cache flush on parent change | **Implemented** | +| Root timeout enforcement | Planned | +| Tree entry TTL enforcement | Planned | +| Hold-down timer after parent change | Planned | +| Per-ancestry-entry signatures | Future direction | + +## References + +- [fips-mesh-operation.md](fips-mesh-operation.md) — How the spanning tree + fits into mesh routing +- [fips-wire-formats.md](fips-wire-formats.md) — TreeAnnounce wire format +- [spanning-tree-dynamics.md](spanning-tree-dynamics.md) — Convergence + scenario walkthroughs diff --git a/docs/design/fips-state-machines.md b/docs/design/fips-state-machines.md index 0037526..9c52324 100644 --- a/docs/design/fips-state-machines.md +++ b/docs/design/fips-state-machines.md @@ -49,33 +49,35 @@ impl Peer { ```rust /// What the Node stores per peer slot enum PeerSlot { - Connecting(PeerConnection), - Active(ActivePeer), + Connecting(Box), + Active(Box), } /// Handles authentication handshake only struct PeerConnection { - identity: PeerIdentity, link_id: LinkId, - direction: Direction, + direction: LinkDirection, // Handshake-specific state - ephemeral_keypair: Keypair, - remote_ephemeral: Option, - handshake_hash: [u8; 32], - attempts: u32, - last_sent: Instant, + handshake_state: HandshakeState, + expected_identity: Option, + noise_handshake: Option, + noise_session: Option, + // Timing + started_at: u64, + last_activity: u64, } /// Fully authenticated peer struct ActivePeer { identity: PeerIdentity, link_id: LinkId, - session: SessionKeys, + connectivity: ConnectivityState, + noise_session: Option, // Routing state declaration: Option, - coords: Option, + ancestry: Option, inbound_filter: Option, - last_seen: Instant, + last_seen: u64, } ``` @@ -83,142 +85,103 @@ struct ActivePeer { - Each struct only contains fields relevant to that phase - Methods can't be called in wrong state (compile-time safety) -- Ephemeral keys automatically dropped when `PeerConnection` → `ActivePeer` +- Noise handshake state automatically dropped when `PeerConnection` → `ActivePeer` - Each phase struct is smaller, simpler, independently testable ## Transition Pattern -Phase transitions return the new phase, consuming the old: +In FIPS, connections and peers are stored in separate maps rather than a +unified `PeerSlot` map (though `PeerSlot` exists for cases where either phase +is needed). PeerConnection uses `&mut self` methods for handshake steps, and +the Node orchestrates promotion when the handshake completes: ```rust impl PeerConnection { - /// Handle incoming packet during handshake - fn handle_packet(self, packet: &[u8]) -> ConnectionResult { - // Parse and validate... - match self.state { - HandshakeState::WaitingForResponse => { - // Verify response, derive session keys - let session = self.derive_session_keys(&response); - ConnectionResult::Authenticated { - peer: ActivePeer { - identity: self.identity, - link_id: self.link_id, - session, - declaration: None, - coords: None, - inbound_filter: None, - last_seen: Instant::now(), - }, - } - } - // ... - } - } + /// Start handshake as initiator, returns msg1 to send. + fn start_handshake(&mut self, our_keypair: Keypair, current_time_ms: u64) + -> Result, NoiseError>; + + /// Process incoming msg1 (responder), returns msg2 to send. + fn receive_handshake_init(&mut self, our_keypair: Keypair, message: &[u8], + current_time_ms: u64) -> Result, NoiseError>; + + /// Complete handshake by processing msg2 (initiator). + fn complete_handshake(&mut self, message: &[u8], current_time_ms: u64) + -> Result<(), NoiseError>; + + /// Take the completed NoiseSession for use in ActivePeer. + fn take_session(&mut self) -> Option; } -enum ConnectionResult { - /// Stay in connecting phase, send this response - Continue(Vec), - /// Auth complete, here's the active peer - Authenticated { peer: ActivePeer }, - /// Auth failed - Failed(String), +/// Result of promoting a connection to active peer. +enum PromotionResult { + /// New peer created successfully. + Promoted(NodeAddr), + /// Cross-connection detected, this connection lost tie-breaker. + CrossConnectionLost { winner_link_id: LinkId }, + /// Cross-connection detected, this connection won, old one replaced. + CrossConnectionWon { loser_link_id: LinkId, node_addr: NodeAddr }, } ``` -The Node's event loop handles the transition: +The Node promotes a completed connection by moving data between maps: ```rust -fn handle_packet(&mut self, link_id: LinkId, packet: &[u8]) { - let slot = self.peers.get_mut(&link_id); +// In the Node's handshake completion handler: +// 1. Remove from connections map +let conn = self.connections.remove(&link_id).unwrap(); - match slot { - PeerSlot::Connecting(conn) => { - // Note: take() to move ownership for transition - let conn = std::mem::take(conn); - match conn.handle_packet(packet) { - ConnectionResult::Continue(response) => { - *slot = PeerSlot::Connecting(conn); - self.send(link_id, response); - } - ConnectionResult::Authenticated { peer } => { - *slot = PeerSlot::Active(peer); - self.on_peer_active(link_id); - } - ConnectionResult::Failed(reason) => { - self.peers.remove(&link_id); - warn!(%reason, "Peer auth failed"); - } - } - } - PeerSlot::Active(peer) => { - let actions = peer.handle_packet(packet); - self.execute(actions); - } - } -} +// 2. Extract session and identity from completed connection +let session = conn.take_session().unwrap(); +let identity = conn.expected_identity().unwrap().clone(); + +// 3. Create active peer with the session +let peer = ActivePeer::with_session( + identity, link_id, current_time_ms, + session, our_index, their_index, + transport_id, source_addr, conn.link_stats().clone(), +); + +// 4. Insert into peers map (checking for cross-connections) +self.peers.insert(*peer.node_addr(), peer); ``` ## Timeout Handling -Each phase struct tracks its own timing. The Node's event loop periodically -scans for timeouts: +Each phase struct tracks its own timing using Unix milliseconds (`u64`). +PeerConnection provides a simple boolean timeout check: ```rust impl PeerConnection { - fn check_timeout(&mut self, now: Instant) -> TimeoutResult { - if now.duration_since(self.last_sent) < HANDSHAKE_TIMEOUT { - return TimeoutResult::Ok; - } - - self.attempts += 1; - if self.attempts > MAX_HANDSHAKE_ATTEMPTS { - return TimeoutResult::GiveUp; - } - - self.last_sent = now; - TimeoutResult::Retry(self.build_retry_packet()) + /// Check if the connection has timed out. + fn is_timed_out(&self, current_time_ms: u64, timeout_ms: u64) -> bool { + self.idle_time(current_time_ms) > timeout_ms } -} -enum TimeoutResult { - Ok, - Retry(Vec), - GiveUp, + /// Time since last activity. + fn idle_time(&self, current_time_ms: u64) -> u64 { + current_time_ms.saturating_sub(self.last_activity) + } } ``` -Node event loop (simple periodic scan): +The Node's event loop periodically scans the connections map for timeouts: ```rust -loop { - select! { - packet = packet_rx.recv() => { /* dispatch */ } +// In the Node's periodic maintenance: +let now_ms = current_time_ms(); +let mut timed_out = vec![]; - _ = interval.tick() => { - let now = Instant::now(); - let mut to_remove = vec![]; - - for (id, slot) in &mut self.peers { - if let PeerSlot::Connecting(conn) = slot { - match conn.check_timeout(now) { - TimeoutResult::Retry(packet) => { - self.send(conn.link_id, packet); - } - TimeoutResult::GiveUp => { - to_remove.push(*id); - } - TimeoutResult::Ok => {} - } - } - } - - for id in to_remove { - self.peers.remove(&id); - } - } +for (link_id, conn) in &self.connections { + if conn.is_timed_out(now_ms, HANDSHAKE_TIMEOUT_MS) { + timed_out.push(*link_id); } } + +for link_id in timed_out { + self.connections.remove(&link_id); + // Clean up link, free index, etc. +} ``` ## Application in FIPS @@ -239,75 +202,82 @@ PeerSlot::Active(ActivePeer) **PeerConnection** contains: -- Noise IK handshake state (ephemeral keys, handshake hash) -- Expected identity (for outbound) or discovered identity (for inbound) -- Direction (Inbound vs Outbound) +- Noise IK handshake state (`noise_handshake`, `handshake_state`) +- Expected identity (known for outbound, learned for inbound) +- Direction (`LinkDirection::Inbound` vs `LinkDirection::Outbound`) +- Session index tracking (`our_index`, `their_index`) +- Timing (`started_at`, `last_activity` in Unix milliseconds) **ActivePeer** contains: -- NoiseSession (symmetric keys for encrypt/decrypt) -- Tree position (declaration, coordinates) -- Bloom filter (what's reachable through this peer) -- Statistics (last_seen, link_stats) +- `NoiseSession` (symmetric keys for encrypt/decrypt) +- `ConnectivityState` (Connected, Stale, Reconnecting, Disconnected) +- Tree position (`declaration`, `ancestry`) +- Bloom filter (`inbound_filter`, with sequence tracking) +- Session indices and transport address (for wire protocol dispatch) +- Statistics (`last_seen`, `link_stats`, `authenticated_at`) ### Link Lifecycle (Connection-Oriented Transports) -For transports like Tor that require connection setup: +Links use a simple state enum rather than phase-based structs, since link +state transitions are straightforward: ```rust -enum LinkSlot { - Connecting(LinkConnection), - Established(EstablishedLink), +enum LinkState { + Connecting, // Connection in progress (connection-oriented only) + Connected, // Ready for traffic + Disconnected, // Was connected, now gone + Failed, // Connection attempt failed } -struct LinkConnection { +struct Link { + link_id: LinkId, transport_id: TransportId, remote_addr: TransportAddr, - connect_started: Instant, - // Tor circuit build state, etc. -} - -struct EstablishedLink { - transport_id: TransportId, - remote_addr: TransportAddr, - // I/O handles - writer: TorWriter, - // Stats - established_at: Instant, + direction: LinkDirection, + state: LinkState, + stats: LinkStats, + // ... } ``` -For connectionless transports (UDP), links are immediately "established" - -no `LinkConnection` phase needed. +For connectionless transports (UDP), links are immediately `Connected` - +no `Connecting` phase needed. A phase-based approach (separate structs for +connecting vs established) would be valuable for transports with complex +connection setup like Tor circuit building. ### Node Lifecycle ```rust -enum NodePhase { - Created(CreatedNode), - Starting(StartingNode), - Running(RunningNode), - Stopping(StoppingNode), +enum NodeState { + Created, + Starting, + Running, + Stopping, + Stopped, } ``` -Currently the Node uses a simpler `NodeState` enum because startup/shutdown -are brief and don't need complex per-phase logic. Phase-based approach would -be useful if startup involved multi-step async operations with retries. +The Node uses a simple state enum because startup/shutdown are brief and +don't need complex per-phase logic. A phase-based approach (separate structs +per phase) would be useful if startup involved multi-step async operations +with retries. ### Transport Lifecycle ```rust -enum TransportPhase { - Configured(ConfiguredTransport), - Starting(StartingTransport), - Up(UpTransport), - Failed(FailedTransport), +enum TransportState { + Configured, + Starting, + Up, + Down, + Failed, } ``` -Again, currently simpler because transport startup is straightforward. -Would be valuable for transports with complex initialization (Tor bootstrap). +Like `NodeState`, this uses a simple state enum because transport startup is +straightforward. A phase-based approach would be valuable for transports with +complex initialization (e.g., Tor bootstrap with multi-step circuit building). ## When to Use This Pattern @@ -326,16 +296,22 @@ Would be valuable for transports with complex initialization (Tor bootstrap). ## Lookup Tables -When using `PeerSlot` enum, need reverse lookups for packet dispatch: +Rather than a unified `PeerSlot` map, the Node stores connections and peers +in separate maps optimized for their phase: ```rust struct Node { - // Primary storage - peers: HashMap, + // Handshake phase: indexed by LinkId (identity not yet known) + connections: HashMap, - // Reverse lookup: (transport, remote_addr) → NodeAddr - // Needed because ReceivedPacket has addr, not NodeAddr - addr_to_peer: HashMap<(TransportId, TransportAddr), NodeAddr>, + // Active phase: indexed by NodeAddr (verified identity) + peers: HashMap, + + // Reverse lookup: (transport_id, remote_addr) → LinkId + addr_to_link: HashMap, + + // Wire protocol dispatch: (transport_id, our_index) → NodeAddr + peers_by_index: HashMap<(TransportId, u32), NodeAddr>, } ``` @@ -344,7 +320,8 @@ For inbound connections from unknown addresses: 1. Receive Noise IK msg1 → decrypt to extract sender's static key (identity) 2. Create new PeerConnection with discovered identity 3. Add to `connections` (by LinkId) and `addr_to_link` -4. After handshake completes, promote to ActivePeer (indexed by NodeAddr) +4. After handshake completes, promote to `peers` (indexed by NodeAddr) and + `peers_by_index` (for session index dispatch) ## Summary @@ -353,7 +330,7 @@ The phase-based state machine pattern provides: 1. **Type safety** - Can't call auth methods on active peer 2. **Memory efficiency** - Phase-specific data dropped on transition 3. **Clarity** - Each struct is focused and comprehensible -4. **Security** - Ephemeral keys don't linger after auth +4. **Security** - Handshake state (ephemeral keys) dropped after auth 5. **Testability** - Each phase testable in isolation The cost is slightly more complex transition handling in the event loop, diff --git a/docs/design/fips-transport-layer.md b/docs/design/fips-transport-layer.md new file mode 100644 index 0000000..f1d216a --- /dev/null +++ b/docs/design/fips-transport-layer.md @@ -0,0 +1,382 @@ +# FIPS Transport Layer + +The transport layer is the bottom of the FIPS protocol stack. It delivers +datagrams between transport-specific endpoints over arbitrary physical or +logical media. Everything above — peer authentication, routing, encryption, +session management — is built on the services the transport layer provides. + +## Role + +A **transport** is a driver for a particular communication medium: a UDP +socket, an Ethernet interface, a LoRa radio, a serial line, a Tor circuit. +The transport layer's job is simple: accept a datagram and a transport +address, deliver the datagram to that address, and push inbound datagrams up +to the FIPS Link Protocol (FLP) above. + +The transport layer deals exclusively in **transport addresses** — IP:port +tuples, MAC addresses, LoRa device addresses, .onion identifiers. These are +opaque to every layer above FLP. The mapping from transport address to FIPS +identity happens at the link layer after the Noise IK handshake completes. +The word "peer" belongs to the link layer and above; the transport layer +knows only about remote endpoints identified by transport addresses. + +A single transport instance can serve multiple remote endpoints +simultaneously — a UDP socket exchanges datagrams with many remote +addresses, an Ethernet interface communicates with many MAC addresses on the +same segment. Each endpoint may become a separate FLP link, but the +transport layer itself maintains no per-endpoint state. + +## Services Provided to FLP + +The transport layer provides four services to the FIPS Link Protocol above: + +### Datagram Delivery + +Send and receive datagrams to/from transport addresses. The transport +handles all medium-specific details: socket management, framing for stream +transports, radio configuration. FLP sees only "send bytes to address" and +"bytes arrived from address." + +Inbound datagrams are pushed to FLP through a channel. The transport spawns +a receive task that pushes arriving datagrams (along with the source +transport address and transport identifier) onto a bounded channel. FLP +reads from this channel and dispatches based on the source address and +packet content. + +### MTU Reporting + +Report the maximum datagram size for a given link. FLP needs this to +determine how much payload can fit in a single packet after link-layer +encryption overhead. + +MTU is fundamentally a per-link property. A transport with a fixed MTU +(Ethernet: 1500, UDP configured at 1472) returns the same value for every +link — this is the degenerate case. Transports that negotiate MTU +per-connection (e.g., BLE ATT_MTU) report the negotiated value for each +link individually. + +> **Implementation note**: The current transport trait exposes MTU as a +> transport-wide method (`fn mtu(&self) -> u16`). This works for UDP, where +> MTU is a static configuration value. Supporting per-link MTU for future +> transports will require extending this interface. + +### Connection Lifecycle + +For connection-oriented transports, manage the underlying connection: TCP +handshake, Tor circuit establishment, Bluetooth pairing. FLP cannot begin +the Noise IK handshake until the transport-layer connection is established. + +Connectionless transports (UDP, raw Ethernet) skip this — datagrams can flow +immediately to any reachable address. + +### Discovery (Optional) + +Notify FLP when FIPS-capable endpoints are discovered on the local medium. +This is an optional capability — transports that don't support it simply +don't provide discovery events. + +See [Discovery](#discovery) below for details. + +## Transport Properties + +Transports vary widely in their characteristics. FIPS operates over all of +them because the transport interface abstracts these differences behind a +uniform datagram service. + +### Transport Categories + +**Overlay transports** tunnel FIPS over an existing network layer, typically +for internet connectivity: + +| Transport | Addressing | MTU | Reliability | Notes | +| --------- | ---------- | --- | ----------- | ----- | +| UDP/IP | IP:port | 1280–1472 | Unreliable | Primary internet transport | +| TCP/IP | IP:port | Stream | Reliable | Requires length-prefix framing | +| WebSocket | URL | Stream | Reliable | Browser-compatible | +| Tor | .onion | Stream | Reliable | High latency, strong anonymity | +| I2P | Destination | ~32K | Unreliable | Datagram mode | + +**Shared medium transports** operate over broadcast- or multicast-capable +media: + +| Transport | Addressing | MTU | Reliability | Notes | +| --------- | ---------- | --- | ----------- | ----- | +| Ethernet | MAC | 1500 | Unreliable | Raw AF_PACKET frames | +| WiFi | MAC | 1500 | Unreliable | Infrastructure mode = Ethernet | +| Bluetooth | BD_ADDR | 672–64K | Reliable | L2CAP | +| BLE | BD_ADDR | 23–517 | Reliable | Negotiated ATT_MTU | +| LoRa | Device addr | 51–222 | Unreliable | Low bandwidth, long range | + +**Point-to-point transports** connect exactly two endpoints: + +| Transport | Addressing | MTU | Reliability | Notes | +| --------- | ---------- | --- | ----------- | ----- | +| Serial | None (P2P) | 256–1500 | Reliable | SLIP/COBS framing | +| Dialup | None (P2P) | 1500 | Reliable | PPP framing | + +### Properties That Matter to FLP + +**MTU**: Determines how much data FLP can pack into a single datagram after +accounting for link encryption overhead. Heterogeneous MTUs across the mesh +are normal — the IPv6 minimum (1280 bytes) is the safe baseline for FIPS +packet sizing. + +**Reliability**: Whether the transport guarantees delivery. FIPS prefers +unreliable transports because running TCP application traffic over a reliable +transport creates TCP-over-TCP, where retransmission and congestion control +at both layers interact adversely. FIPS tolerates packet loss, reordering, +and duplication at the routing layer. + +**Connection model**: Connectionless transports (UDP, raw Ethernet) allow +immediate datagram exchange. Connection-oriented transports (TCP, Tor, BLE) +require connection setup before FLP can begin the Noise IK handshake, +adding startup latency. + +**Stream vs. datagram**: Datagram transports have natural packet boundaries. +Stream transports (TCP, WebSocket, Tor) require length-prefix framing to +delineate FIPS packets within the byte stream. + +**Addressing opacity**: Transport addresses are opaque byte vectors. FLP +doesn't interpret them — it just passes them back to the transport when +sending. This means adding a new transport type with a novel address format +requires no changes to FLP or FSP. + +## Connection Model + +### Connectionless Transports + +Datagrams can be sent to any reachable address without prior setup. Links +are lightweight — a transport address is sufficient to begin communication. + +| Transport | Notes | +| --------- | ----- | +| UDP/IP | Stateless datagrams; NAT state is implicit | +| Ethernet | Send to MAC address directly | +| LoRa | Raw packets to device address | +| I2P | Datagram mode | + +### Connection-Oriented Transports + +Explicit connection setup is required before FIPS traffic can flow. The link +must complete transport-layer connection before FLP authentication can +proceed. + +| Transport | Connection Setup | +| --------- | ---------------- | +| TCP/IP | TCP three-way handshake | +| WebSocket | HTTP upgrade + TCP | +| Tor | Circuit establishment (500ms–5s) | +| Bluetooth | L2CAP connection | +| BLE | L2CAP CoC or GATT connection | +| Serial | Physical connection (static) | + +### Implications + +**Link lifecycle**: Connectionless transports use a trivial link model. +Connection-oriented transports need a real state machine: Connecting → +Connected → Disconnected. Failure can occur during connection setup, adding +error handling paths that connectionless transports don't have. + +**Startup latency**: Connection-oriented transports add delay before a peer +becomes usable. This ranges from milliseconds (TCP) to seconds (Tor +circuit). Peer timeout configuration must account for transport-specific +setup times. + +**Framing**: Stream transports must delimit FIPS packets within the byte +stream using length-prefix framing. Datagram transports preserve packet +boundaries naturally. + +## UDP/IP: The Primary Internet Transport + +For internet-connected nodes, UDP/IP is the recommended transport: + +- **No TCP-over-TCP**: UDP's unreliable delivery avoids the adverse + interaction between application-layer TCP retransmission and transport-layer + TCP retransmission +- **NAT traversal**: UDP hole punching enables peer connections through NAT + without relay infrastructure +- **Low overhead**: 8-byte UDP header, no connection state +- **Matches FIPS model**: FIPS is datagram-oriented; UDP preserves this + naturally without framing + +Raw IP with a custom protocol number would be simpler but is blocked by most +NAT devices and firewalls, limiting deployment to networks without NAT. + +## Discovery + +Discovery determines that a FIPS-capable endpoint is reachable at a given +transport address. It is distinct from raw transport-level endpoint +detection — a new TCP connection or UDP packet from an unknown source is not +discovery; a FIPS-specific announcement or response is. + +Discovery is an optional transport capability. Transports that don't support +it (configured UDP endpoints, TCP) simply don't provide discovery events. +FLP handles both cases uniformly: with discovery, it waits for events then +initiates link setup; without discovery, it initiates link setup directly to +configured addresses. + +### Local/Medium Discovery *(future direction)* + +For transports where endpoints share a physical or link-layer medium — LAN +broadcast, LoRa, BLE — discovery uses beacon and query mechanisms: + +- **Beacon**: A node periodically broadcasts its FIPS presence on the shared + medium. Content is a FIPS-defined discovery frame carrying enough + information to initiate a link. Non-FIPS endpoints ignore the frame. +- **Query**: A node broadcasts a one-shot solicitation. FIPS-capable nodes + respond. Responses arrive on the same channel as beacon events. + +Both produce the same result: "FIPS endpoint available at transport address +X." FLP does not need to distinguish beacons from query responses. + +| Transport | Discovery | Notes | +| --------- | --------- | ----- | +| UDP (LAN) | Broadcast/multicast | On local network segment | +| Ethernet | Broadcast | Custom EtherType, ff:ff:ff:ff:ff:ff | +| LoRa | Beacon | Shared RF channel, natural fit | +| BLE | Advertising | GATT service UUID | + +### Nostr Relay Discovery *(future direction)* + +For internet-reachable transports, a node publishes a signed Nostr event +containing its FIPS discovery information — public key and reachable +transport endpoints (UDP IP:port, TCP IP:port, .onion address). Other FIPS +nodes subscribing on the same relays learn about available peers. + +Nostr relay discovery is not a transport — it is a discovery service that +feeds addresses to other transports. A node discovers via Nostr that a peer +is reachable at UDP 1.2.3.4:9735, then establishes the link over the UDP +transport. + +Key properties: +- Identity is built in — Nostr events are signed, so discovery information + is authenticated +- Relay selection acts as scoping — which relays a node publishes to and + subscribes on determines its discovery neighborhood +- Can only advertise IP-reachable endpoints (not LoRa, BLE, serial) +- Higher latency than local discovery (relay propagation delays) + +### Current State + +> **Implemented**: Peer addresses come from YAML configuration. The +> transport trait's `discover()` method exists but returns an empty list for +> UDP. Transport-level discovery (beacon/query, Nostr relay) is not yet +> implemented. + +## Transport Interface + +The transport interface defines what every transport driver must provide. + +### Trait Surface + +```text +transport_id() → TransportId Unique identifier for this transport instance +transport_type() → &TransportType Static metadata (name, connection-oriented, reliable) +state() → TransportState Current lifecycle state +mtu() → u16 Maximum datagram size +start() → lifecycle Bring transport up (bind socket, open device) +stop() → lifecycle Bring transport down +send(addr, data) → delivery Send datagram to transport address +discover() → Vec Report discovered FIPS endpoints (optional) +``` + +### Receive Path + +Rather than a synchronous receive method, transports use a channel-push +model. Each transport takes a sender handle at construction and spawns an +internal receive loop that pushes inbound datagrams onto the channel. The +node's main event loop reads from the corresponding receiver, which +aggregates datagrams from all active transports into a single stream. + +Each inbound datagram carries: +- **transport_id** — which transport it arrived on +- **remote_addr** — the transport address of the sender +- **data** — the raw datagram bytes +- **timestamp** — arrival time + +### Transport Metadata + +Transport types carry static metadata that FLP can query: + +```text +TransportType { + name "udp", "ethernet", "tor", etc. + connection_oriented bool + reliable bool +} +``` + +Predefined types exist for UDP, TCP, Ethernet, WiFi, Tor, and Serial. + +### Transport Addresses + +Transport addresses (`TransportAddr`) are opaque byte vectors. The transport +layer interprets them (e.g., UDP parses "ip:port" strings); all layers above +treat them as opaque handles passed back to the transport for sending. + +### Transport State Machine + +```text +Configured → Starting → Up → Down + ↓ + Failed +``` + +Transports begin in `Configured` state with all parameters set. `start()` +transitions through `Starting` to `Up` (operational). `stop()` moves to +`Down`. Transport failures move to `Failed`. + +## Implementation Status + +| Transport | Status | Notes | +| --------- | ------ | ----- | +| UDP/IP | **Implemented** | Primary transport, async send/receive, configurable MTU | +| TCP/IP | Future direction | Requires stream framing, TCP-over-TCP concern | +| Ethernet | Future direction | AF_PACKET raw frames, EtherType TBD | +| WiFi | Future direction | Infrastructure mode = Ethernet driver | +| Tor | Future direction | High latency, .onion addressing | +| BLE | Future direction | ATT_MTU negotiation, per-link MTU | +| LoRa | Future direction | Constrained MTU (51–222 bytes) | +| Serial | Future direction | SLIP/COBS framing, point-to-point | + +## Design Considerations + +### TCP-over-TCP Avoidance + +Running TCP application traffic over a reliable transport (TCP, WebSocket) +creates a layering violation where retransmission and congestion control +operate at both levels. When the inner TCP detects loss (which may just be +transport-layer retransmission delay), it retransmits, creating more traffic +for the outer TCP, which may itself be retransmitting. This amplification +loop degrades performance severely under any packet loss. + +FIPS prefers unreliable transports for this reason. When a reliable transport +must be used (e.g., Tor), applications should be aware of the performance +implications. + +### Multi-Transport Operation + +A node can run multiple transports simultaneously. Peers from all transports +feed into a single spanning tree and routing table. If one transport fails, +traffic automatically routes through alternatives. A node with both UDP and +Ethernet transports bridges between internet-connected and local-only +networks transparently. + +Multiple links to the same peer over different transports are possible. FLP +manages these independently — each link has its own Noise session, its own +MTU, and its own liveness tracking. + +### Transport Quality and Path Selection + +Transport characteristics (latency, bandwidth, reliability) affect path +quality but are not currently factored into routing decisions. The spanning +tree parent selection uses a depth improvement threshold but does not +consider transport quality. This is a potential area for future optimization. + +## References + +- [fips-intro.md](fips-intro.md) — Protocol overview and layer architecture +- [fips-link-layer.md](fips-link-layer.md) — FLP specification (the layer above) +- [fips-wire-formats.md](fips-wire-formats.md) — Transport framing details +- [fips-software-architecture.md](fips-software-architecture.md) — Transport + trait implementation details diff --git a/docs/design/fips-transports.md b/docs/design/fips-transports.md deleted file mode 100644 index 008f8dd..0000000 --- a/docs/design/fips-transports.md +++ /dev/null @@ -1,190 +0,0 @@ -# FIPS Transport Protocols - -FIPS nodes peer with each other over a variety of transport types. This document -explores the requirements and characteristics of different transport protocols -that FIPS can operate over. - -## Terminology - -- **Transport**: A physical or logical interface over which FIPS communicates - (e.g., a UDP socket, Ethernet NIC, or Tor client) -- **Link**: A connection instance to a specific peer over a transport - -This document describes transport-level characteristics. See -[fips-software-architecture.md](fips-software-architecture.md) for the Transport trait definition. - -## Design Principles - -FIPS is a Layer 3 (network) protocol. It exposes an IPv6 interface to local -applications, with an address deterministically derived from the node's npub. -This means existing UDP, TCP, and other IP-based applications work unmodified -over FIPS. - -However, this IPv6-over-transport architecture requires care to avoid classic -encapsulation pitfalls. In particular, running TCP over a reliable transport -(like TCP/IP overlay) creates "TCP-over-TCP" where retransmission and congestion -control mechanisms at both layers interact adversely. FIPS prefers unreliable -transports for this reason. - -FIPS treats underlying connectivity as abstract transports, regardless of -whether those transports are: - -- True L2 protocols (Ethernet, Bluetooth) -- L4-over-L3 tunnels (UDP/IP) used as transport substrate for NAT traversal -- Application-layer overlays (Tor, I2P) - -Each transport driver presents a uniform interface to the FIPS routing layer: -send/receive datagrams to/from a transport-layer peer address. - -## Transport Characteristics - -### Overlay Transports (L3/L4 Substrate) - -These transports tunnel FIPS over an existing network layer, typically for -internet connectivity or anonymity. Overlay transports are expected to be the -majority in early deployments, but all depend on existing IP/Internet -infrastructure that FIPS is ultimately designed to replace. - -| Transport | Encapsulation | Addressing | MTU | Latency | Reliability | Bandwidth | Discovery | -|-----------|---------------|------------|-----|---------|-------------|-----------|-----------| -| UDP/IP | UDP datagram | IP:port | 1280-1472 | 1-500ms | Unreliable | High | DNS-SD, Nostr | -| TCP/IP | Framed stream | IP:port | Stream | 10-500ms | Reliable | High | DNS-SD, Nostr | -| WebSocket | WS frames | URL | Stream | 10-500ms | Reliable | High | Nostr | -| Tor | TCP stream | .onion | Stream | 500ms-5s | Reliable | Low-Med | Static, Nostr | -| I2P | I2P datagram | Destination | ~32K | 1-10s | Unreliable | Low | I2P directory | - -### Shared Medium Transports - -These transports operate over broadcast or multicast-capable media where multiple -endpoints share the same physical or logical channel. - -| Transport | Encapsulation | Addressing | MTU | Latency | Reliability | Bandwidth | Discovery | -|-----------|---------------|------------|-----|---------|-------------|-----------|-----------| -| Ethernet | EtherType frame | MAC | 1500 | <1ms | Unreliable | High | Multicast | -| WiFi Direct | 802.11 frame | MAC | 1500 | 1-10ms | Unreliable | High | Service discovery | -| DOCSIS (Cable) | DOCSIS frame | MAC | 1500 | 10-50ms | Unreliable | 1M-1G | N/A (uses IP) | -| Bluetooth Classic | L2CAP | BD_ADDR | 672-64K | 10-100ms | Reliable | 2-3 Mbps | Inquiry + SDP | -| BLE | L2CAP CoC/GATT | BD_ADDR | 23-517 | 10-30ms | Reliable | 125K-2M | GATT advertising | -| Zigbee | 802.15.4 frame | 16/64-bit | ~100 | 15-30ms | Reliable | 250 kbps | Network scan | -| LoRa | Raw packet | Device addr | 51-222 | 100ms-10s | Unreliable | 0.3-50 kbps | Beacons | - -### Point-to-Point Transports - -These transports connect exactly two endpoints with no shared medium or -addressing. - -| Transport | Encapsulation | Addressing | MTU | Latency | Reliability | Bandwidth | Discovery | -|-----------|---------------|------------|----------|-----------|-------------|-----------|------------| -| Serial | SLIP/COBS frame | None (P2P) | 256-1500 | 1-100ms | Reliable | 9.6K-1M | Configured | -| Dialup | PPP frame | None (P2P) | 1500 | 100-200ms | Reliable | 33.6-56K | Configured | - -### Notes - -**MTU**: Minimum/maximum or typical range. FIPS must handle heterogeneous MTUs -across the mesh; the IPv6 minimum (1280) is a safe baseline for the FIPS -packet format. - -**Latency**: Typical range from best-case to worst-case. Affects spanning tree -convergence and keepalive timing. - -**Reliability**: Whether the link provides delivery guarantees. Unreliable -links may drop, reorder, or duplicate packets. FIPS must tolerate this at -the routing layer. - -**Bandwidth**: Order of magnitude. Affects flow control and congestion -decisions, but FIPS routing itself is low-bandwidth (control plane only). - -## Connection Model - -Transports fall into two categories based on whether they require connection -establishment before data can be exchanged: - -### Connectionless Transports - -These transports can send datagrams to a peer address without prior setup. -Links are lightweight—just a `(transport_id, remote_addr)` tuple with implicit -"established" state. - -| Transport | Notes | -|-----------|-------| -| UDP/IP | Stateless datagrams; NAT state is implicit | -| Ethernet | Send to MAC address directly | -| WiFi | Same as Ethernet (802.11 frame) | -| DOCSIS | Cable modem layer; uses IP in practice | -| LoRa | Raw packets to device address | -| I2P | Datagram mode (not streaming) | - -### Connection-Oriented Transports - -These transports require explicit connection setup before FIPS traffic can flow. -Links track real connection state and hold I/O handles. The link must complete -transport-layer connection before FIPS authentication can proceed. - -| Transport | Connection Setup | -|-----------|------------------| -| TCP/IP | TCP handshake | -| WebSocket | HTTP upgrade + TCP | -| Tor | Circuit establishment (slow: 500ms-5s) | -| Bluetooth Classic | L2CAP connection | -| BLE | L2CAP CoC or GATT connection | -| Zigbee | Network join + binding | -| Serial | Physical connection (static) | -| Dialup | PPP negotiation | - -### Implications for FIPS - -**Link lifecycle**: Connectionless transports use a trivial link model (no state -machine). Connection-oriented transports require a real state machine: -`Connecting → Connected → Disconnected`. See -[fips-software-architecture.md](fips-software-architecture.md) for link lifecycle details. - -**Startup latency**: Connection-oriented transports add latency before a peer -becomes usable. Tor is particularly slow (circuit setup). This affects peer -timeout configuration. - -**Failure modes**: Connectionless links "fail" only when the transport itself -is down or the peer stops responding. Connection-oriented links can fail during -connection setup, adding more error handling paths. - -**Framing**: Connection-oriented stream transports (TCP, WebSocket, Tor) require -length-prefix framing to delineate FIPS packets. Datagram transports have -natural packet boundaries. - -## UDP/IP as Primary Internet Transport - -For internet-connected nodes, UDP/IP is the recommended transport: - -- **NAT traversal**: UDP hole punching enables peer connections through NAT -- **Firewall compatibility**: UDP outbound rarely blocked; stateful firewalls - pass return traffic -- **No connection state**: Matches FIPS datagram model -- **Low overhead**: 8-byte UDP header is negligible -- **Avoids TCP-over-TCP**: As noted in Design Principles, unreliable transports - avoid adverse interactions with application-layer TCP - -Raw IP with a custom protocol number would be cleaner but is blocked by most -NAT devices and firewalls, limiting deployment to networks without NAT. - -## Transport Driver Interface - -> **Note**: The definitive Transport trait is defined in -> [fips-software-architecture.md](fips-software-architecture.md). This section provides a -> simplified conceptual view. - -Each transport driver provides: - -- `send(addr, data)` — Send a FIPS packet to a transport-layer address -- `recv()` — Receive a FIPS packet from any peer -- `mtu()` — Maximum FIPS packet size for this transport -- `discover()` — Find potential peers (transport-specific mechanism) - -Transport drivers handle any necessary framing, fragmentation, or encryption -at the transport layer. The FIPS routing layer sees only FIPS packets. - -## Topics for Further Design - -- Framing protocols for stream-based transports (TCP, WebSocket) -- Transport-layer encryption requirements vs FIPS-layer encryption -- Congestion control and flow control per transport type -- Multi-path: using multiple transports to same peer -- Transport quality metrics for parent selection diff --git a/docs/design/fips-tun-driver.md b/docs/design/fips-tun-driver.md deleted file mode 100644 index 4646f27..0000000 --- a/docs/design/fips-tun-driver.md +++ /dev/null @@ -1,236 +0,0 @@ -# FIPS TUN Driver Design - -This document describes the design and implementation of the TUN interface -driver that connects FIPS to the local system's network stack. - -## Overview - -The TUN driver provides the interface between local applications and the FIPS -mesh network. It presents a virtual network interface (`fips0`) with the node's -FIPS address, allowing standard socket applications to communicate over the -mesh transparently. - -## Architecture - -```text -┌─────────────────────────────────────────────────────────────────┐ -│ Local System │ -│ ┌──────────────┐ │ -│ │ Applications │ (sockets using fd00::/8 addresses) │ -│ └──────┬───────┘ │ -│ │ │ -│ ┌──────▼───────┐ │ -│ │ Kernel │ routing: fd00::/8 → fips0 │ -│ │ IPv6 Stack │ local table intercepts traffic to self │ -│ └──────┬───────┘ │ -└─────────┼───────────────────────────────────────────────────────┘ - │ raw IPv6 packets -┌─────────▼───────────────────────────────────────────────────────┐ -│ TUN Device (fips0) │ -│ ┌─────────────────────────────────────────────────────────────┐│ -│ │ File Descriptor ││ -│ │ (duplicated for reader/writer) ││ -│ └──────────┬─────────────────────────────┬────────────────────┘│ -│ │ │ │ -│ ┌──────▼──────┐ ┌──────▼──────┐ │ -│ │ Reader │ │ Writer │ │ -│ │ Thread │ │ Thread │ │ -│ │ (blocking) │ │ (blocking) │ │ -│ └──────┬──────┘ └──────▲──────┘ │ -│ │ │ │ -└─────────────┼─────────────────────────────┼──────────────────────┘ - │ │ - ▼ │ - ┌─────────────┐ ┌───────┴───────┐ - │ Packet │ │ TX Queue │ - │ Processing │──────────────▶ (mpsc) │ - │ (routing) │ │ │ - └─────────────┘ └───────▲───────┘ - │ - (future: transports) -``` - -## Components - -### TunDevice (`src/tun.rs`) - -The main TUN device wrapper that handles creation, configuration, and lifecycle. - -**Responsibilities:** - -- Create TUN interface via `tun` crate -- Configure IPv6 address via netlink (`rtnetlink`) -- Set MTU and bring interface up -- Provide read access for incoming packets -- Create writer handle via fd duplication - -**Lifecycle:** - -1. **Startup**: Delete existing interface if present, create new -2. **Active**: Reader and writer threads operate independently -3. **Shutdown**: Delete interface via netlink, threads exit on I/O error - -### TunWriter (`src/tun.rs`) - -Services a queue of outbound packets and writes them to the TUN device. - -**Design rationale:** - -Multiple sources will need to write to TUN: - -- ICMPv6 error responses (from packet processing) -- Inbound mesh traffic from peers (future: transports) -- Locally-generated control traffic (future) - -A single writer thread with an mpsc queue provides: - -- No contention on TUN writes -- Clean separation of concerns -- Easy addition of new packet sources via `TunTx::clone()` - -### ICMPv6 Module (`src/icmp.rs`) - -Generates RFC 4443 compliant ICMPv6 error messages. - -**Currently implemented:** - -- Type 1 Code 0: Destination Unreachable - No route - -**Validation (when NOT to send errors):** - -- Original packet was an ICMPv6 error (types 0-127) -- Source address is multicast (0xff prefix) -- Source address is unspecified (::) - -**Response format:** - -- Total size ≤ 1280 bytes (IPv6 minimum MTU) -- Includes as much of original packet as fits -- Proper checksum with pseudo-header - -## Packet Flow - -### Outbound (local → mesh) - -1. Application sends to `fd00::/8` address -2. Kernel routes to `fips0` (requires manual route addition) -3. TUN reader receives raw IPv6 packet -4. Packet processing determines next hop -5. If routable: forward to transport (future) -6. If not routable: send ICMPv6 Destination Unreachable via TX queue - -### Inbound (mesh → local) - -1. Transport receives packet from peer (future) -2. Check destination address -3. If destination is self: write to TUN via TX queue -4. If destination is other: forward to next hop (transit) - -### Local Address Guarantee - -Packets arriving at the TUN reader are guaranteed NOT to be destined for -local addresses. The Linux kernel routing order ensures this: - -1. **Local routing table** - intercepts traffic to addresses on this machine -2. **Main routing table** - routes `fd00::/8` to fips0 - -This means every packet in the TUN reader requires a routing decision. -No "is this for me?" check needed on the read path. - -## Configuration - -From `fips.yaml`: - -```yaml -tun: - enabled: true - name: fips0 - mtu: 1400 -``` - -**Parameters:** - -| Field | Default | Description | -|-------|---------|-------------| -| `enabled` | `true` | Enable TUN interface | -| `name` | `fips0` | Interface name | -| `mtu` | `1400` | Maximum transmission unit | - -## Privileges - -TUN device creation requires `CAP_NET_ADMIN`. Options: - -1. **Run as root**: `sudo ./fips` -2. **Set capability**: `sudo setcap cap_net_admin+ep ./target/debug/fips` -3. **Pre-created device**: Admin creates persistent TUN, FIPS just opens it - -## Route Configuration - -The kernel route must be added manually (not done by FIPS): - -```bash -sudo ip -6 route add fd00::/8 dev fips0 -``` - -This routes all FIPS addresses through the TUN interface. - -## Implementation Status - -### Completed - -- [x] TUN device creation and configuration -- [x] IPv6 address assignment via netlink -- [x] Interface lifecycle (startup cleanup, graceful shutdown) -- [x] Reader thread with blocking I/O -- [x] Writer thread with mpsc queue -- [x] fd duplication for independent read/write -- [x] ICMPv6 Destination Unreachable (Type 1 Code 0) -- [x] Packet validation for ICMPv6 error generation - -### Planned - -- [ ] ICMPv6 Echo Reply (respond to ping) -- [ ] ICMPv6 Packet Too Big (PMTUD support) -- [ ] ICMPv6 Time Exceeded (hop limit) -- [ ] Rate limiting for ICMPv6 errors -- [ ] Integration with routing/forwarding logic -- [ ] Transit packet handling (decrement hop limit, forward) -- [ ] Automatic route management (add/remove fd00::/8 route) - -## Testing - -### Manual Testing - -```bash -# Terminal 1: Run FIPS with debug logging -sudo RUST_LOG=debug ./target/debug/fips - -# Terminal 2: Add route and test -sudo ip -6 route add fd00::/8 dev fips0 -ping6 -c 1 fd00::1 - -# Expected: "Destination unreachable: No route" (not timeout) -``` - -### Verifying Local Routing - -```bash -# Check that local address goes via loopback, not TUN -ip -6 route get -# Should show: local ... dev lo -``` - -## Dependencies - -| Crate | Purpose | -|-------|---------| -| `tun` | TUN device creation | -| `rtnetlink` | Netlink interface configuration | -| `libc` | fd duplication (`dup`) | -| `futures` | Async netlink operations | - -## References - -- RFC 4443: ICMPv6 for IPv6 -- RFC 4291: IPv6 Addressing Architecture -- Linux TUN/TAP documentation diff --git a/docs/design/fips-wire-formats.md b/docs/design/fips-wire-formats.md new file mode 100644 index 0000000..eaa3fd4 --- /dev/null +++ b/docs/design/fips-wire-formats.md @@ -0,0 +1,534 @@ +# FIPS Wire Formats + +This document is the comprehensive wire format reference for all three +protocol layers. It covers transport framing, link-layer message formats, +and session-layer message formats, with an encapsulation walkthrough showing +how application data is wrapped through each layer. + +## Encoding Rules + +- All multi-byte integers are **little-endian** (LE) +- NodeAddr is **16 bytes** — truncated SHA-256 hash of public key +- Signatures are **64 bytes** — secp256k1 Schnorr +- Variable-length arrays use a **2-byte u16 LE count prefix** followed by + that many items +- Public keys are **33 bytes** — compressed secp256k1 (02/03 prefix + 32 + bytes) + +## Transport Framing + +### UDP + +FIPS packets are carried directly in UDP datagrams. No additional framing is +needed — each UDP datagram contains exactly one FIPS link-layer packet. + +### Stream Transports *(future direction)* + +TCP, WebSocket, and Tor transports require length-prefix framing because +they provide a byte stream, not datagrams: + +```text +┌────────────┬───────────────────────────────────┐ +│ Length │ FIPS Packet │ +│ 2 bytes LE │ Variable │ +└────────────┴───────────────────────────────────┘ +``` + +## Link-Layer Formats + +All link-layer packets begin with a **discriminator byte** that determines +the payload format. + +### Discriminator Table + +| Byte | Type | Description | +| ---- | ---- | ----------- | +| 0x00 | Encrypted frame | Post-handshake encrypted traffic | +| 0x01 | Noise IK msg1 | Handshake initiation | +| 0x02 | Noise IK msg2 | Handshake response | + +### Encrypted Frame (0x00) + +All post-handshake traffic between authenticated peers. Contains one +encrypted link-layer message. + +```text +┌────────┬──────────────┬──────────┬───────────────────────────┐ +│ 0x00 │ receiver_idx │ counter │ ciphertext + AEAD tag │ +│ 1 byte │ 4 bytes LE │ 8 bytes LE│ N + 16 bytes │ +└────────┴──────────────┴──────────┴───────────────────────────┘ + +Total overhead: 29 bytes (1 + 4 + 8 + 16) +Minimum frame: 30 bytes (1-byte plaintext) +``` + +| Field | Size | Description | +| ----- | ---- | ----------- | +| discriminator | 1 byte | 0x00 | +| receiver_idx | 4 bytes LE | Session index for O(1) lookup | +| counter | 8 bytes LE | Monotonic nonce, used as AEAD nonce and for replay detection | +| ciphertext | N bytes | ChaCha20 encrypted payload | +| tag | 16 bytes | Poly1305 authentication tag | + +The **plaintext** inside the encrypted frame begins with a message type byte: + +| Type | Message | +| ---- | ------- | +| 0x10 | TreeAnnounce | +| 0x20 | FilterAnnounce | +| 0x30 | LookupRequest | +| 0x31 | LookupResponse | +| 0x40 | SessionDatagram | +| 0x50 | Disconnect | + +### Noise IK Message 1 (0x01) + +Handshake initiation from connecting party. + +```text +┌────────┬─────────────┬─────────────────────────────────────────┐ +│ 0x01 │ sender_idx │ Noise IK message 1 │ +│ 1 byte │ 4 bytes LE │ 82 bytes │ +└────────┴─────────────┴─────────────────────────────────────────┘ + +Total: 87 bytes +``` + +| Field | Size | Description | +| ----- | ---- | ----------- | +| discriminator | 1 byte | 0x01 | +| sender_idx | 4 bytes LE | Initiator's session index (becomes receiver's `receiver_idx`) | +| noise_msg1 | 82 bytes | Noise IK first message | + +**Noise msg1 breakdown** (82 bytes): + +| Offset | Field | Size | Description | +| ------ | ----- | ---- | ----------- | +| 0 | ephemeral_pubkey | 33 bytes | Initiator's ephemeral key (compressed secp256k1) | +| 33 | encrypted_static | 33 bytes | Initiator's static key (encrypted with es key) | +| 66 | tag | 16 bytes | AEAD tag for encrypted_static | + +Noise pattern: `→ e, es, s, ss` + +### Noise IK Message 2 (0x02) + +Handshake response from responder. + +```text +┌────────┬─────────────┬──────────────┬──────────────────────────┐ +│ 0x02 │ sender_idx │ receiver_idx │ Noise IK message 2 │ +│ 1 byte │ 4 bytes LE │ 4 bytes LE │ 33 bytes │ +└────────┴─────────────┴──────────────┴──────────────────────────┘ + +Total: 42 bytes +``` + +| Field | Size | Description | +| ----- | ---- | ----------- | +| discriminator | 1 byte | 0x02 | +| sender_idx | 4 bytes LE | Responder's session index | +| receiver_idx | 4 bytes LE | Echo of initiator's sender_idx from msg1 | +| noise_msg2 | 33 bytes | Noise IK second message | + +**Noise msg2 breakdown** (33 bytes): + +| Offset | Field | Size | Description | +| ------ | ----- | ---- | ----------- | +| 0 | ephemeral_pubkey | 33 bytes | Responder's ephemeral key (compressed secp256k1) | + +Noise pattern: `← e, ee, se` + +After msg2, both parties derive identical symmetric session keys. + +### Index Semantics + +Each party in a link session maintains two indices: + +| Index | Chosen By | Used By | Purpose | +| ----- | --------- | ------- | ------- | +| our_index | Us | Them | They include this as `receiver_idx` in packets to us | +| their_index | Them | Us | We include this as `receiver_idx` in packets to them | + +### Handshake Flow + +```text +Initiator Responder +───────── ───────── +generates sender_idx +generates ephemeral keypair + + 0x01 | sender_idx | noise_msg1 + ────────────────────────────────► + + validates msg1 + learns initiator's static key + generates sender_idx + generates ephemeral keypair + + 0x02 | sender_idx | receiver_idx | noise_msg2 + ◄──────────────────────────────── + +validates msg2 +derives session keys + +═══════════════ HANDSHAKE COMPLETE ═══════════════ + +First encrypted frame: + 0x00 | receiver_idx | counter=0 | ciphertext+tag + ────────────────────────────────► +``` + +## Link-Layer Message Types + +These messages are carried as plaintext inside encrypted frames (0x00). + +### TreeAnnounce (0x10) + +Spanning tree state announcement, exchanged between direct peers only. + +| Offset | Field | Size | Description | +| ------ | ----- | ---- | ----------- | +| 0 | msg_type | 1 byte | 0x10 | +| 1 | version | 1 byte | 0x01 (v1) | +| 2 | sequence | 8 bytes LE | Monotonic counter, increments on parent change | +| 10 | timestamp | 8 bytes LE | Unix seconds | +| 18 | parent | 16 bytes | NodeAddr of selected parent (self = root) | +| 34 | ancestry_count | 2 bytes LE | Number of AncestryEntry records | +| 36 | ancestry | 32 × n bytes | AncestryEntry array (self → root) | +| 36 + 32n | signature | 64 bytes | Schnorr signature over entire message | + +**AncestryEntry** (32 bytes): + +| Offset | Field | Size | Description | +| ------ | ----- | ---- | ----------- | +| 0 | node_addr | 16 bytes | Node's routing identifier | +| 16 | sequence | 8 bytes LE | Node's sequence number | +| 24 | timestamp | 8 bytes LE | Node's Unix timestamp | + +**Size**: `100 + (n × 32)` bytes, where n = `ancestry_count` (depth + 1, +includes self) + +| Tree Depth | Payload | With Link Overhead | +| ---------- | ------- | ------------------ | +| 0 (root) | 132 bytes | 161 bytes | +| 3 | 228 bytes | 257 bytes | +| 5 | 292 bytes | 321 bytes | +| 10 | 452 bytes | 481 bytes | + +### FilterAnnounce (0x20) + +Bloom filter reachability update, exchanged between direct peers only. + +| Offset | Field | Size | Description | +| ------ | ----- | ---- | ----------- | +| 0 | msg_type | 1 byte | 0x20 | +| 1 | sequence | 8 bytes LE | Monotonic counter for freshness | +| 9 | hash_count | 1 byte | Number of hash functions (5 in v1) | +| 10 | size_class | 1 byte | Filter size: `512 << size_class` bytes | +| 11 | filter_bits | variable | Bloom filter bit array | + +**Size class table**: + +| size_class | Bytes | Bits | Status | +| ---------- | ----- | ---- | ------ | +| 0 | 512 | 4,096 | Reserved | +| 1 | 1,024 | 8,192 | **v1 (MUST use)** | +| 2 | 2,048 | 16,384 | Reserved | +| 3 | 4,096 | 32,768 | Reserved | + +**v1 payload**: 1,035 bytes (11 header + 1,024 filter). +With link overhead: 1,064 bytes. + +### LookupRequest (0x30) + +Coordinate discovery request, flooded through the mesh. + +| Offset | Field | Size | Description | +| ------ | ----- | ---- | ----------- | +| 0 | msg_type | 1 byte | 0x30 | +| 1 | request_id | 8 bytes LE | Unique random identifier | +| 9 | target | 16 bytes | NodeAddr being sought | +| 25 | origin | 16 bytes | Requester's NodeAddr | +| 41 | ttl | 1 byte | Remaining hops (default 64) | +| 42 | origin_coords_cnt | 2 bytes LE | Number of coordinate entries | +| 44 | origin_coords | 16 × n bytes | Requester's ancestry (NodeAddr only) | +| 44 + 16n | visited_hash_cnt | 1 byte | Hash count for visited filter | +| 45 + 16n | visited_bits | 256 bytes | Compact bloom of visited nodes | + +**Size**: `301 + (n × 16)` bytes, where n = origin depth + 1 + +| Origin Depth | Payload | +| ------------ | ------- | +| 3 | 349 bytes | +| 5 | 381 bytes | +| 10 | 461 bytes | + +### LookupResponse (0x31) + +Coordinate discovery response, greedy-routed back to requester. + +| Offset | Field | Size | Description | +| ------ | ----- | ---- | ----------- | +| 0 | msg_type | 1 byte | 0x31 | +| 1 | request_id | 8 bytes LE | Echoes the request's ID | +| 9 | target | 16 bytes | NodeAddr that was found | +| 25 | target_coords_cnt | 2 bytes LE | Number of coordinate entries | +| 27 | target_coords | 16 × n bytes | Target's ancestry (NodeAddr only) | +| 27 + 16n | proof | 64 bytes | Schnorr signature over `(request_id \|\| target)` | + +**Size**: `91 + (n × 16)` bytes + +| Target Depth | Payload | +| ------------ | ------- | +| 3 | 139 bytes | +| 5 | 171 bytes | +| 10 | 251 bytes | + +**Proof coverage**: Signs `(request_id || target)` only — coordinates are +excluded so the proof survives tree reconvergence during the lookup +round-trip. + +### SessionDatagram (0x40) + +Encapsulated session-layer payload for multi-hop forwarding. + +| Offset | Field | Size | Description | +| ------ | ----- | ---- | ----------- | +| 0 | msg_type | 1 byte | 0x40 | +| 1 | src_addr | 16 bytes | Source NodeAddr | +| 17 | dest_addr | 16 bytes | Destination NodeAddr | +| 33 | hop_limit | 1 byte | Decremented each hop | +| 34 | payload | variable | Session-layer message | + +**Fixed header**: 34 bytes (`SESSION_DATAGRAM_HEADER_SIZE`) + +The payload is opaque to transit nodes — session-layer encrypted +independently of link encryption. + +### Disconnect (0x50) + +Orderly link teardown with reason code. + +| Offset | Field | Size | Description | +| ------ | ----- | ---- | ----------- | +| 0 | msg_type | 1 byte | 0x50 | +| 1 | reason | 1 byte | Disconnect reason code | + +**Reason codes**: + +| Code | Name | Description | +| ---- | ---- | ----------- | +| 0x00 | Shutdown | Normal operator-requested stop | +| 0x01 | Restart | Restarting, may reconnect soon | +| 0x02 | ProtocolError | Protocol error encountered | +| 0x03 | TransportFailure | Transport failure | +| 0x04 | ResourceExhaustion | Memory or connection limit | +| 0x05 | SecurityViolation | Authentication or policy violation | +| 0x06 | ConfigurationChange | Peer removed from configuration | +| 0x07 | Timeout | Keepalive or stale detection timeout | +| 0xFF | Other | Unspecified reason | + +## Session-Layer Message Types + +These messages are carried as the payload of a SessionDatagram (0x40). + +### SessionSetup (0x00) + +Establishes a session and warms transit coordinate caches. + +| Offset | Field | Size | Description | +| ------ | ----- | ---- | ----------- | +| 0 | msg_type | 1 byte | 0x00 | +| 1 | flags | 1 byte | Bit 0: REQUEST_ACK, Bit 1: BIDIRECTIONAL | +| 2 | src_coords_count | 2 bytes LE | Number of source coordinate entries | +| 4 | src_coords | 16 × n bytes | Source's ancestry (NodeAddr, self → root) | +| ... | dest_coords_count | 2 bytes LE | Number of dest coordinate entries | +| ... | dest_coords | 16 × m bytes | Destination's ancestry | +| ... | handshake_len | 2 bytes LE | Noise payload length | +| ... | handshake_payload | variable | Noise IK msg1 (82 bytes typical) | + +**Example** (depth-3 source, depth-4 destination): + +```text +SessionDatagram header: 34 bytes +SessionSetup payload: 1 + 1 + 2 + 48 + 2 + 64 + 2 + 82 = 202 bytes +Total: 236 bytes +``` + +### SessionAck (0x01) + +Confirms session establishment, completes the Noise handshake. + +| Offset | Field | Size | Description | +| ------ | ----- | ---- | ----------- | +| 0 | msg_type | 1 byte | 0x01 | +| 1 | flags | 1 byte | Reserved | +| 2 | src_coords_count | 2 bytes LE | Number of coordinate entries | +| 4 | src_coords | 16 × n bytes | Acknowledger's ancestry (for cache warming) | +| ... | handshake_len | 2 bytes LE | Noise payload length | +| ... | handshake_payload | variable | Noise IK msg2 (33 bytes typical) | + +### DataPacket (0x10) + +Encrypted application data with explicit replay protection counter. + +**Minimal header** (COORDS_PRESENT = 0): + +| Offset | Field | Size | Description | +| ------ | ----- | ---- | ----------- | +| 0 | msg_type | 1 byte | 0x10 | +| 1 | flags | 1 byte | Bit 0: COORDS_PRESENT | +| 2 | counter | 8 bytes LE | Session encryption counter / replay nonce | +| 10 | payload_length | 2 bytes LE | Length of encrypted payload | +| 12 | payload | variable | Encrypted application data + 16-byte AEAD tag | + +**Header size**: 12 bytes (`DATA_HEADER_SIZE`) + +**With coordinates** (COORDS_PRESENT = 1): + +| Offset | Field | Size | Description | +| ------ | ----- | ---- | ----------- | +| 0 | msg_type | 1 byte | 0x10 | +| 1 | flags | 1 byte | 0x01 (COORDS_PRESENT) | +| 2 | counter | 8 bytes LE | Session encryption counter | +| 10 | payload_length | 2 bytes LE | Length of encrypted payload | +| 12 | src_coords_count | 2 bytes LE | Source coordinate entries | +| 14 | src_coords | 16 × n bytes | Source's ancestry | +| ... | dest_coords_count | 2 bytes LE | Dest coordinate entries | +| ... | dest_coords | 16 × m bytes | Destination's ancestry | +| ... | payload | variable | Encrypted application data | + +### CoordsRequired (0x20) + +Link-layer error signal — transit node lacks coordinates for destination. +Plaintext (not end-to-end encrypted), generated by transit nodes. + +| Offset | Field | Size | Description | +| ------ | ----- | ---- | ----------- | +| 0 | msg_type | 1 byte | 0x20 | +| 1 | flags | 1 byte | Reserved | +| 2 | dest_addr | 16 bytes | NodeAddr we couldn't route to | +| 18 | reporter | 16 bytes | NodeAddr of reporting router | + +**Payload**: 34 bytes. Wrapped in SessionDatagram: 68 bytes total. + +### PathBroken (0x21) + +Link-layer error signal — greedy routing reached a dead end. Plaintext, +generated by transit nodes. + +| Offset | Field | Size | Description | +| ------ | ----- | ---- | ----------- | +| 0 | msg_type | 1 byte | 0x21 | +| 1 | flags | 1 byte | Reserved | +| 2 | dest_addr | 16 bytes | Unreachable NodeAddr | +| 18 | reporter | 16 bytes | NodeAddr of reporting router | +| 34 | last_coords_count | 2 bytes LE | Number of stale coordinate entries | +| 36 | last_known_coords | 16 × n bytes | Stale coordinates that failed | + +## Encapsulation Walkthrough + +A complete picture of how application data is wrapped through each layer. + +### Application Data → Wire + +Starting with an application sending a 1024-byte payload to a destination: + +```text +Layer 4: Application data + 1024 bytes + +Layer 3: Session encryption (FSP) + DataPacket header (12 bytes) + encrypted payload (1024) + AEAD tag (16) + = 1052 bytes + +Layer 2: SessionDatagram envelope (FLP routing) + msg_type (1) + src_addr (16) + dest_addr (16) + hop_limit (1) + payload (1052) + = 1086 bytes + +Layer 1: Link encryption (FLP per-hop) + discriminator (1) + receiver_idx (4) + counter (8) + ciphertext (1086) + tag (16) + = 1115 bytes + +Layer 0: Transport + UDP datagram containing 1115 bytes +``` + +### Overhead Budget + +| Layer | Overhead | Component | +| ----- | -------- | --------- | +| Link encryption | 29 bytes | 1 discriminator + 4 index + 8 counter + 16 AEAD tag | +| SessionDatagram | 34 bytes | 1 type + 16 src + 16 dest + 1 hop_limit | +| DataPacket header | 12 bytes | 1 type + 1 flags + 8 counter + 2 length | +| Session AEAD tag | 16 bytes | Poly1305 tag on session-encrypted payload | +| **Minimal total** | **91 bytes** | | +| Coordinates (if present) | ~44 bytes | Varies with tree depth | +| **Worst case** | **135 bytes** | `FIPS_OVERHEAD` constant | + +### At Each Transit Node + +```text +1. Receive UDP datagram +2. Read discriminator (0x00) → encrypted frame +3. Look up (transport_id, receiver_idx) → session +4. Check replay window (counter) +5. Decrypt with link keys → plaintext link message +6. Read msg_type (0x40) → SessionDatagram +7. Read dest_addr → routing decision +8. Decrement hop_limit +9. Re-encrypt with next-hop link keys +10. Send via next-hop transport +``` + +Transit nodes see the SessionDatagram envelope (src_addr, dest_addr, +hop_limit) but cannot read the session-layer payload (encrypted with +endpoint session keys). + +## Size Summary + +### Handshake Messages + +| Message | Size | +| ------- | ---- | +| Noise IK msg1 | 87 bytes | +| Noise IK msg2 | 42 bytes | + +### Link-Layer Messages (inside encrypted frame) + +| Message | Size | Notes | +| ------- | ---- | ----- | +| TreeAnnounce | 100 + 32n bytes | n = depth + 1 | +| FilterAnnounce | 1,035 bytes | v1 (1KB filter) | +| LookupRequest | 301 + 16n bytes | n = origin depth + 1 | +| LookupResponse | 91 + 16n bytes | n = target depth + 1 | +| Disconnect | 2 bytes | | + +### Session-Layer Messages (inside SessionDatagram) + +| Message | Typical Size | Notes | +| ------- | ------------ | ----- | +| SessionSetup | ~200 bytes | Depth-dependent | +| SessionAck | ~80 bytes | Depth-dependent | +| DataPacket (minimal) | 12 + payload bytes | Steady state | +| DataPacket (with coords) | 12 + ~130 + payload bytes | Warmup/recovery | +| CoordsRequired | 34 bytes | Fixed | +| PathBroken | 36 + 16n bytes | Includes stale coords | + +### Complete Packet Sizes (link + session) + +| Scenario | Wire Size | Notes | +| -------- | --------- | ----- | +| Encrypted frame minimum | 30 bytes | 1-byte plaintext | +| SessionDatagram + DataPacket (minimal) | 29 + 34 + 12 + payload + 16 | 91 + payload | +| SessionDatagram + DataPacket (with coords) | ~135 + payload | Worst case | +| SessionDatagram + SessionSetup | ~265 bytes | Depth-3, both dirs | +| SessionDatagram + CoordsRequired | 29 + 34 + 34 = 97 bytes | Including link overhead | + +## References + +- [fips-link-layer.md](fips-link-layer.md) — FLP behavioral specification +- [fips-session-layer.md](fips-session-layer.md) — FSP behavioral specification +- [fips-transport-layer.md](fips-transport-layer.md) — Transport framing +- [fips-mesh-operation.md](fips-mesh-operation.md) — How messages work together +- [fips-ipv6-adapter.md](fips-ipv6-adapter.md) — MTU enforcement diff --git a/docs/design/fips-wire-protocol.md b/docs/design/fips-wire-protocol.md deleted file mode 100644 index 99afc01..0000000 --- a/docs/design/fips-wire-protocol.md +++ /dev/null @@ -1,1139 +0,0 @@ -# FIPS Wire Protocol and Transport Layer Management - -This document describes the FIPS wire protocol message flow at the transport level: -how peers establish and maintain cryptographic sessions with each other. - -The transport layer provides the link-level communications path for a node to each -of its outbound and inbound peers, delivering an authenticated, encrypted, and -roaming-friendly peer-to-peer mesh connection over which both link-layer routing -control messages and end-to-end FIPS session layer data flow. - -Topics include: - -- Wire format with session indices for O(1) packet dispatch -- Handshake and session lifecycle -- Transport-layer roaming via index-based session lookup -- Security properties: rate limiting, replay protection, state machine strictness -- Adaptation to different transport types (UDP, TCP, Tor, Ethernet, radio) - ---- - -## 1. Design Goals - -### 1.1 Primary Goals - -1. **Cryptographic authority**: A packet that properly decrypts is authentic, - regardless of source address -2. **Roaming support**: Peers can change transport addresses (IP:port, etc.) - without session interruption -3. **Efficient dispatch**: O(1) lookup for authenticated traffic, no trial - decryption across multiple sessions -4. **DoS resistance**: Minimize resources consumed by unauthenticated traffic -5. **State machine correctness**: Strict validation prevents confusion attacks - -### 1.2 WireGuard Influence - -This design follows WireGuard's principle: source address is informational, not -authoritative. Only successful cryptographic verification establishes authenticity. -When a valid packet arrives from a different address than expected, the peer's -address is updated rather than the packet being rejected. - -> **Terminology note**: "Source address" in this document refers to the transport-layer -> address (link_addr)—e.g., IP:port for UDP, MAC for Ethernet, .onion for Tor. This is -> distinct from node_addr (the routing identifier) and FIPS address/pubkey (the endpoint -> identity). See [fips-intro.md](fips-intro.md) §Identity System for the full terminology. - ---- - -## 2. Wire Format - -All FIPS link-layer packets use the following format: - -```text -┌─────────────┬────────────────────────────────────────────────┐ -│ Discriminator│ Type-Specific Payload │ -│ 1 byte │ Variable │ -└─────────────┴────────────────────────────────────────────────┘ -``` - -The discriminator byte determines the payload format: - -| Byte | Type | Payload Format | -|------|------|----------------| -| 0x00 | Encrypted frame | `[receiver_idx:4][counter:8][ciphertext+tag:N+16]` | -| 0x01 | Noise IK msg1 | `[sender_idx:4][noise_msg1:82]` | -| 0x02 | Noise IK msg2 | `[sender_idx:4][receiver_idx:4][noise_msg2:33]` | - -### 2.1 Encrypted Frame (0x00) - -Post-handshake encrypted packets: - -```text -┌────────┬──────────────┬──────────┬───────────────────────────┐ -│ 0x00 │ receiver_idx │ counter │ ciphertext + AEAD tag │ -│ 1 byte │ 4 bytes LE │ 8 bytes LE│ N + 16 bytes │ -└────────┴──────────────┴──────────┴───────────────────────────┘ - -Total overhead: 29 bytes (1 + 4 + 8 + 16) -``` - -- **receiver_idx**: Session index assigned by the receiver during handshake. - Enables O(1) session lookup without relying on source address. -- **counter**: Monotonically increasing per-session, per-direction counter. - Used as AEAD nonce and for replay detection. -- **ciphertext**: ChaCha20-Poly1305 encrypted payload. -- **tag**: 16-byte Poly1305 authentication tag. - -The plaintext inside the encrypted frame begins with a message type byte, -followed by the message-specific payload (see fips-intro.md for message -types 0x10-0x4F). - -### 2.2 Noise IK Message 1 (0x01) - -Handshake initiation from the connecting party: - -```text -┌────────┬─────────────┬─────────────────────────────────────────┐ -│ 0x01 │ sender_idx │ Noise IK message 1 │ -│ 1 byte │ 4 bytes LE │ 82 bytes │ -└────────┴─────────────┴─────────────────────────────────────────┘ - -Total: 87 bytes -``` - -- **sender_idx**: Index chosen by the initiator. This becomes the responder's - `receiver_idx` when sending packets TO the initiator. -- **Noise msg1**: Standard Noise IK first message (ephemeral pubkey 33 bytes + - encrypted static pubkey 33 + 16 bytes). - -### 2.3 Noise IK Message 2 (0x02) - -Handshake response from the responder: - -```text -┌────────┬─────────────┬──────────────┬──────────────────────────┐ -│ 0x02 │ sender_idx │ receiver_idx │ Noise IK message 2 │ -│ 1 byte │ 4 bytes LE │ 4 bytes LE │ 33 bytes │ -└────────┴─────────────┴──────────────┴──────────────────────────┘ - -Total: 42 bytes -``` - -- **sender_idx**: Index chosen by the responder. This becomes the initiator's - `receiver_idx` when sending packets TO the responder. -- **receiver_idx**: Echo of the initiator's `sender_idx` from msg1. Enables the - initiator to match the response to their pending handshake. -- **Noise msg2**: Standard Noise IK second message (ephemeral pubkey 33 bytes). - -### 2.4 Index Semantics - -Each party in a session has two indices: - -| Index | Chosen By | Used By | Purpose | -|-------|-----------|---------|---------| -| our_index | Us | Them | They include this in packets TO us | -| their_index | Them | Us | We include this in packets TO them | - -After handshake completion: - -- Initiator's `our_index` = initiator's `sender_idx` from msg1 -- Responder's `our_index` = responder's `sender_idx` from msg2 -- Each party's `their_index` = the other party's `sender_idx` - -### 2.5 Index Properties - -Indices MUST be: - -1. **Random**: Unpredictable to prevent guessing attacks. Use cryptographically - secure random generation. -2. **Unique per transport**: No two active sessions on the same transport may - share the same `our_index`. -3. **Scoped to transport**: The tuple `(transport_id, receiver_idx)` identifies - a session. The same index value may appear on different transports. - -Indices SHOULD be: - -4. **Rotated on rekey**: When a session rekeys, allocate new indices to prevent - cross-session correlation. - -### 2.6 Link Control Messages - -Link control messages are sent inside encrypted frames (discriminator 0x00) and -use the 0x50–0x5F message type range. The first defined control message is -Disconnect (0x50). - -#### Disconnect (0x50) - -Orderly disconnect notification sent before closing a peer link: - -```text -ENCRYPTED FRAME (discriminator 0x00): - [receiver_idx][counter][ENCRYPTED_PAYLOAD + tag] - -DECRYPTED PLAINTEXT: -┌──────────┬──────────┐ -│ 0x50 │ reason │ -│ 1 byte │ 1 byte │ -└──────────┴──────────┘ - -Total plaintext: 2 bytes -``` - -**Reason codes:** - -| Code | Name | Description | -|------|---------------------|------------------------------------------| -| 0x00 | Shutdown | Normal operator-requested stop | -| 0x01 | Restart | Restarting, may reconnect soon | -| 0x02 | ProtocolError | Protocol error encountered | -| 0x03 | TransportFailure | Transport failure | -| 0x04 | ResourceExhaustion | Memory or connection limit | -| 0x05 | SecurityViolation | Authentication or policy violation | -| 0x06 | ConfigurationChange | Peer removed from configuration | -| 0x07 | Timeout | Keepalive or stale detection timeout | -| 0xFF | Other | Unspecified reason | - -**Semantics:** - -- **Best-effort delivery**: If the transport is broken, the message won't arrive. - Timeout-based detection (stale peer, keepalive failure) remains the fallback. -- **Receiver action**: Immediately remove the peer from the peer table, free the - session index, remove the link, and clean up address mappings. If the departed - peer was a tree parent, trigger parent reselection. -- **Shutdown sequence**: On node shutdown, Disconnect is sent to all active peers - *before* transports are stopped. - ---- - -## 3. Packet Dispatch - -### 3.1 Overview - -Packet dispatch follows a two-phase approach: - -1. **Parse discriminator**: Determine packet type (O(1)) -2. **Route by type**: - - Encrypted (0x00): Index-based lookup, cryptographic verification - - Handshake msg2 (0x02): Index-based lookup for pending outbound - - Handshake msg1 (0x01): Rate-limited processing, create new state - -### 3.2 Data Structures - -``` -Node: - // === Authenticated sessions === - // Primary dispatch: our_index → NodeAddr - peers_by_index: HashMap<(TransportId, u32), NodeAddr> - - // Peer data by identity - peers: HashMap - - // === Pending handshakes === - // Outbound: our sender_idx → connection state - pending_outbound: HashMap<(TransportId, u32), PeerConnection> - - // Inbound: source address → connection state (before we know identity) - pending_inbound_by_addr: HashMap<(TransportId, TransportAddr), PeerConnection> - - // === Resource management === - index_allocator: IndexAllocator - msg1_rate_limiter: TokenBucket -``` - -### 3.3 Encrypted Frame Dispatch (0x00) - -``` -receive_encrypted(transport_id, source_addr, data): - // Parse header (fail fast on malformed) - if data.len() < 29: // 1 + 4 + 8 + 16 minimum - drop("too short") - - receiver_idx = u32_le(data[1..5]) - counter = u64_le(data[5..13]) - ciphertext = data[13..] - - // O(1) session lookup by index - node_addr = peers_by_index.get((transport_id, receiver_idx)) - if node_addr is None: - drop("unknown index") // No crypto, minimal CPU cost - - peer = peers.get(node_addr) - - // Replay check BEFORE decryption (cheap) - if not peer.replay_window.check(counter): - drop("replay or too old") - - // Decrypt (expensive, but only for valid-looking packets) - plaintext = peer.session.decrypt(counter, ciphertext) - if plaintext is Err: - drop("decrypt failed") // Corrupted or wrong key - - // === PACKET IS AUTHENTIC === - - // Accept counter into replay window - peer.replay_window.accept(counter) - - // Update address (ROAMING) - peer.current_addr = source_addr - - // Update statistics - peer.stats.record_recv(data.len()) - - // Dispatch to message handler - dispatch_link_message(node_addr, plaintext) -``` - -**Key properties**: - -- Unknown index rejected before any crypto (O(1) map lookup) -- Replay check before decryption (fast bitfield check) -- Source address updated on successful decrypt (roaming) -- Single decryption attempt per packet (no trial decryption) - -### 3.4 Handshake Message 2 Dispatch (0x02) - -``` -receive_msg2(transport_id, source_addr, data): - // Parse header - if data.len() != 42: // 1 + 4 + 4 + 33 - drop("wrong size") - - their_sender_idx = u32_le(data[1..5]) - our_receiver_idx = u32_le(data[5..9]) - noise_msg2 = data[9..42] - - // Lookup OUR pending handshake by our sender_idx - key = (transport_id, our_receiver_idx) - conn = pending_outbound.get(key) - if conn is None: - drop("no pending handshake") // We didn't initiate this - - if conn.state != SentMsg1: - drop("unexpected state") // State machine violation - - // Process Noise msg2 (crypto cost paid here) - result = conn.noise.read_msg2(noise_msg2) - if result is Err: - conn.state = Failed - drop("handshake failed") - - // Handshake complete - conn.their_index = their_sender_idx - conn.source_addr = source_addr // Update address - - // Promote to authenticated peer - promote_connection(key) -``` - -**Key properties**: - -- Lookup by OUR index (which we chose), not source address -- State machine enforced: msg2 only valid in SentMsg1 state -- Cannot be spoofed: requires responding to our ephemeral key - -### 3.5 Handshake Message 1 Dispatch (0x01) - -This is the primary attack surface for unauthenticated traffic. - -``` -receive_msg1(transport_id, source_addr, data): - // === RATE LIMITING (before any processing) === - if not msg1_rate_limiter.try_acquire(): - drop("rate limited") - - // === CONNECTION LIMITS === - if pending_inbound_by_addr.len() >= MAX_PENDING_INBOUND: - drop("too many pending") - - // Parse header - if data.len() != 87: // 1 + 4 + 82 - drop("wrong size") - - their_sender_idx = u32_le(data[1..5]) - noise_msg1 = data[5..87] - - // Check for existing connection from this address - addr_key = (transport_id, source_addr) - if pending_inbound_by_addr.contains(addr_key): - // Could be retry or attack; existing state handles it - drop("duplicate") - - // === CRYPTO COST PAID HERE === - result = NoiseHandshake::process_msg1(our_identity, noise_msg1) - if result is Err: - drop("invalid msg1") - - (peer_identity, handshake, msg2_payload) = result - - // === IDENTITY CHECKS === - - // Check if this is a known peer reconnecting - if peers.contains(peer_identity.node_addr): - // Existing peer from new address - handle reconnection - handle_peer_reconnection(peer_identity, source_addr, ...) - return - - // Optional: check allowlist/blocklist - if not should_accept_peer(peer_identity): - drop("not allowed") - - // === CREATE STATE === - our_index = index_allocator.allocate(transport_id) - - conn = PeerConnection { - direction: Inbound, - transport_id, - our_index, - their_index: their_sender_idx, - state: ReceivedMsg1, - noise: handshake, - discovered_identity: peer_identity, - source_addr, - created_at: now(), - } - - pending_inbound_by_addr.insert(addr_key, conn) - - // === SEND RESPONSE === - // [0x02][our_index:4][their_index:4][noise_msg2:33] - msg2 = [0x02] - ++ our_index.to_le_bytes() - ++ their_sender_idx.to_le_bytes() - ++ msg2_payload - - send_to_transport(transport_id, source_addr, msg2) -``` - -**Key properties**: - -- Rate limiting BEFORE any parsing or crypto -- Connection limit caps memory usage -- Crypto cost (DH operations) only paid after rate limit passes -- Duplicate detection prevents state accumulation from retries -- Identity learned from msg1, checked against allowlist - -### 3.6 Dispatch Summary - -| Packet Type | Lookup Key | Crypto Before Dispatch? | Can Create State? | -|-------------|------------|------------------------|-------------------| -| Encrypted (0x00) | `(transport_id, receiver_idx)` | Yes (AEAD decrypt) | No | -| Msg2 (0x02) | `(transport_id, our_sender_idx)` | Yes (Noise) | No (existing state) | -| Msg1 (0x01) | `(transport_id, source_addr)` | Yes (Noise) | Yes (rate limited) | - ---- - -## 4. Roaming - -### 4.1 Definition - -Roaming allows a peer to change their transport-layer address (IP:port for UDP, -connection handle for TCP, etc.) while maintaining their authenticated session. - -### 4.2 Mechanism - -When an encrypted packet (0x00) successfully decrypts: - -1. The packet is authentic (AEAD tag verified with session keys) -2. Session keys are bound to peer identity via Noise handshake -3. Therefore, the sender is the authenticated peer, regardless of source address -4. Update `peer.current_addr` to the packet's source address - -``` -// After successful decryption -peer.current_addr = source_addr -``` - -Subsequent outbound packets to this peer use the updated address. - -### 4.3 Transport Applicability - -| Transport | Roaming Applicable? | Notes | -|-----------|--------------------|----| -| UDP | Yes | Source IP:port can change freely | -| TCP | Limited | Reconnection, not mid-session change | -| Tor | Limited | Circuit changes, onion address stable | -| Ethernet | Rare | MAC address typically stable | -| Radio | Yes | Node may move between base stations | - -For connection-oriented transports (TCP, Tor), "roaming" manifests as -reconnection rather than mid-session address change. The index-based lookup -still applies: a new connection that produces a valid encrypted packet with -a known `receiver_idx` is accepted as the peer returning. - -### 4.4 Security Consideration - -Roaming enables an attacker who compromises session keys to redirect traffic. -However, session key compromise already allows full impersonation, so roaming -doesn't add attack surface. The session keys are the authority, not the address. - ---- - -## 5. Replay Protection - -### 5.1 Counter-Based Nonces - -Each session maintains per-direction counters: - -- **send_counter**: Incremented for each packet sent, used as AEAD nonce -- **recv_window**: Sliding window tracking received counters - -### 5.2 Sliding Window - -The receive window allows for UDP packet reordering while detecting replays: - -``` -ReplayWindow: - top: u64 // Highest counter seen - bitmap: [u64; 32] // 2048-bit bitmap for window below top - -check(counter) -> bool: - if counter > top: - return true // New high, definitely not replay - if counter + WINDOW_SIZE < top: - return false // Too old, outside window - - bit = (top - counter) as usize - return not bitmap.test(bit) // True if not seen - -accept(counter): - if counter > top: - // Advance window - shift = min(counter - top, WINDOW_SIZE) - bitmap.shift_left(shift) - bitmap.set(0) // Mark new counter as seen - top = counter - else: - bit = (top - counter) as usize - bitmap.set(bit) -``` - -### 5.3 Window Size - -A 2048-packet window (matching WireGuard) handles: - -- Typical UDP reordering (tens of packets) -- Burst loss followed by retransmission -- Multi-path scenarios where packets take different routes - -Packets older than the window are rejected. This bounds the memory for replay -state to O(1) per session regardless of session duration. - ---- - -## 6. Rate Limiting - -### 6.1 Purpose - -Rate limiting protects against CPU exhaustion from msg1 processing. Each msg1 -requires: - -- Noise DH operations (~200μs on modern CPU) -- State allocation -- Response generation - -An attacker flooding msg1 from spoofed addresses can exhaust CPU without the -rate limit. - -### 6.2 Token Bucket Algorithm - -``` -TokenBucket: - tokens: u32 - max_tokens: u32 - refill_rate: u32 // Tokens per second - last_refill: Instant - -try_acquire() -> bool: - refill() - if tokens > 0: - tokens -= 1 - return true - return false - -refill(): - elapsed = now() - last_refill - new_tokens = elapsed.as_secs() * refill_rate - tokens = min(tokens + new_tokens, max_tokens) - last_refill = now() -``` - -### 6.3 Recommended Parameters - -| Parameter | Value | Rationale | -|-----------|-------|-----------| -| `max_tokens` | 100 | Burst capacity for legitimate connection storms | -| `refill_rate` | 10/sec | Sustained rate of new connections | -| `MAX_PENDING_INBOUND` | 1000 | Memory bound on pending handshakes | -| `HANDSHAKE_TIMEOUT` | 30 sec | Cleanup interval for stale handshakes | - -These values should be configurable to accommodate different deployment -scenarios (high-traffic relays vs. leaf nodes). - -### 6.4 Per-Source vs. Global - -Rate limiting is **global** (not per-source) because: - -- UDP source addresses are trivially spoofable -- Per-source limits don't protect against distributed attacks -- Global limit bounds total CPU regardless of attack distribution - -The tradeoff is that a flooding attack can deny service to legitimate new -connections. Mitigations include: - -- Higher limits for nodes expecting many connections -- Priority for configured/known peer addresses -- Optional proof-of-work extension (future) - ---- - -## 7. State Machine Strictness - -### 7.1 Valid State Transitions - -``` -PeerConnection states: - Initial → SentMsg1 (outbound: we sent msg1) - Initial → ReceivedMsg1 (inbound: we received msg1, sent msg2) - SentMsg1 → Complete (received valid msg2) - ReceivedMsg1 → Complete (received valid encrypted packet) - * → Failed (any error) - -ActivePeer states: - Connected → Stale (no traffic for threshold) - Stale → Connected (valid traffic received) - * → Disconnected (explicit close or timeout) -``` - -### 7.2 Strict Validation - -Each received packet is validated against expected state: - -| Current State | Received | Valid? | Action | -|---------------|----------|--------|--------| -| No state | 0x00 (encrypted) | No | Drop (unknown index) | -| No state | 0x01 (msg1) | Yes | Create PeerConnection (rate limited) | -| No state | 0x02 (msg2) | No | Drop (no pending handshake) | -| SentMsg1 | 0x00 | No | Drop (not authenticated yet) | -| SentMsg1 | 0x01 | No | Drop (we're initiator, not responder) | -| SentMsg1 | 0x02 | Yes | Complete handshake | -| ReceivedMsg1 | 0x00 | Yes | First authenticated packet, promote | -| ReceivedMsg1 | 0x01 | No | Drop (duplicate initiation) | -| ReceivedMsg1 | 0x02 | No | Drop (we're responder, not initiator) | -| Authenticated | 0x00 | Yes | Normal encrypted traffic | -| Authenticated | 0x01 | See 7.3 | Peer reconnection | -| Authenticated | 0x02 | No | Drop (handshake already complete) | - -### 7.3 Reconnection Handling - -When msg1 arrives for an already-authenticated peer (identified by npub in the -decrypted static key), the new handshake is accepted alongside the existing -session. If the new handshake completes successfully within a timeout, it -replaces the old session; otherwise it is discarded. - -This approach handles: - -- Legitimate reconnection (network changed, process restarted) -- NAT rebinding (source port changed) -- Cross-connection resolution (both sides initiated simultaneously) - ---- - -## 8. Index Management - -### 8.1 Allocation - -``` -IndexAllocator: - allocated: HashSet<(TransportId, u32)> - rng: CryptoRng - -allocate(transport_id) -> u32: - loop: - idx = rng.random_u32() - key = (transport_id, idx) - if not allocated.contains(key): - allocated.insert(key) - return idx - -release(transport_id, idx): - allocated.remove((transport_id, idx)) -``` - -### 8.2 Rekey Index Rotation - -When a session rekeys, new indices are allocated: - -``` -rekey(node_addr): - peer = peers.get(node_addr) - old_index = peer.our_index - new_index = index_allocator.allocate(peer.transport_id) - - // Update index mapping - peers_by_index.remove((peer.transport_id, old_index)) - peers_by_index.insert((peer.transport_id, new_index), node_addr) - - // Release old index - index_allocator.release(peer.transport_id, old_index) - - // Update peer - peer.our_index = new_index - peer.session.rekey() - peer.replay_window.reset() - - // Exchange new indices via encrypted rekey message - send_rekey_notification(peer) -``` - -Index rotation prevents correlation of sessions across rekey events by a -passive observer who can see the cleartext `receiver_idx`. - -### 8.3 Index Exhaustion - -With 32-bit indices and random allocation, birthday collision becomes likely -around 2^16 = 65536 active sessions per transport. For most deployments this -is far beyond expected peer counts. If index exhaustion becomes a concern: - -- Use 64-bit indices (adds 4 bytes to all packets) -- Implement index recycling with reuse delay -- Partition index space by transport or peer class - ---- - -## 9. Transport-Specific Considerations - -### 9.1 UDP - -UDP transport is expected to be the majority of deployments in the initial -stages of development. - -**Address semantics**: `TransportAddr` is `SocketAddr` (IP:port string). - -**Roaming**: Fully supported. Source address updated on valid decrypt. - -**Connection model**: Connectionless. No connection state at transport layer. -"Links" are virtual tuples of `(transport_id, remote_addr)`. - -**NAT considerations**: Source port may change due to NAT rebinding. Index-based -lookup handles this automatically. Hole punching for NAT traversal is a separate -concern (not covered here). - -### 9.2 TCP - -**Address semantics**: `TransportAddr` is the connection handle or remote -`SocketAddr` at connection time. - -**Roaming**: Manifests as reconnection. When TCP connection breaks, peer may -reconnect from different address. The new connection's first packet should be -msg1 (new handshake) which will be recognized as an existing peer reconnecting. - -**Connection model**: Connection-oriented. The transport maintains TCP -connection state. A "link" corresponds to a TCP connection. - -**Framing**: TCP is stream-oriented. Requires length-prefix framing: - -``` -┌────────────┬───────────────────────────────────────────────┐ -│ Length │ FIPS Packet (as specified in §2) │ -│ 2 bytes BE │ Variable │ -└────────────┴───────────────────────────────────────────────┘ -``` - -### 9.3 Tor - -**Address semantics**: `TransportAddr` is onion address + port, or circuit ID. - -**Roaming**: Limited. Onion address is stable but circuits may change. The -index-based lookup handles circuit changes transparently. - -**Connection model**: Connection-oriented (Tor circuits). Similar to TCP for -framing and connection state. - -**Privacy note**: Tor already provides transport encryption. Link-layer Noise -encryption is still applied for defense-in-depth and to maintain consistent -security model across transports. - -### 9.4 Ethernet / WiFi - -**Address semantics**: `TransportAddr` is MAC address. - -**Roaming**: MAC addresses are typically stable. However, some devices randomize -MACs for privacy. Index-based lookup handles MAC changes. - -**Connection model**: Connectionless (like UDP). Frames are independent. - -**Broadcast**: Ethernet supports broadcast/multicast for discovery. This is -outside the scope of packet dispatch. - -### 9.5 Radio (LoRa, etc.) - -**Address semantics**: Transport-specific identifier (device ID, call sign, etc.). - -**Roaming**: A node may be reachable through different base stations. Index-based -lookup handles this. - -**MTU**: Radio often has small MTU (LoRa: ~250 bytes). Wire format overhead -(29 bytes for encrypted) is significant. Consider: - -- Header compression for repeated fields -- Fragment/reassemble at transport layer -- Accept higher overhead as cost of security - ---- - -## 10. Security Analysis - -### 10.1 Attack Resistance Summary - -| Attack | Mitigation | Section | -|--------|------------|---------| -| Connection exhaustion | Rate limit + connection limit | §6 | -| CPU exhaustion (msg1) | Rate limit before crypto | §6 | -| Replay | Counter + sliding window | §5 | -| State confusion | Strict state machine | §7 | -| Spoofed encrypted | Index lookup + AEAD | §3.3 | -| Spoofed msg2 | Index lookup + Noise binding | §3.4 | -| Address spoofing | Crypto authority, not address | §4 | -| Session correlation | Index rotation on rekey | §8.2 | - -### 10.2 Unauthenticated Attack Surface - -Only msg1 (0x01) can be sent by unauthenticated parties. All other packet types -require either: - -- Known session index (encrypted frames) -- Response to our ephemeral key (msg2) - -Msg1 processing is protected by: - -- Global rate limit -- Connection count limit -- Handshake timeout cleanup -- Optional peer allowlist - -### 10.3 Authenticated Peer Misbehavior - -An authenticated peer can: - -- Send malformed encrypted packets (fail AEAD, no effect) -- Send high-frequency traffic (rate limit at higher layer) -- Claim false tree coordinates (validated by signature) - -The authentication layer establishes identity but doesn't grant trust. Higher -protocol layers apply additional policy. - -### 10.4 Implementation Notes - -1. **Constant-time comparison**: Use constant-time comparison for indices and - counters to prevent timing side channels. - -2. **Memory clearing**: Clear session keys and handshake state from memory - after use to limit exposure window. - -3. **Entropy**: Use cryptographically secure RNG for index allocation and - ephemeral key generation. - -4. **Error messages**: Avoid detailed error responses that could leak state - information. Silent drop is preferred for invalid packets. - ---- - -## 11. References - -### Internal Documents - -- [fips-intro.md](fips-intro.md) - Overall protocol design -- [fips-session-protocol.md](fips-session-protocol.md) - Session establishment flow -- [fips-software-architecture.md](fips-software-architecture.md) - Software architecture - -### External References - -- [WireGuard Protocol](https://www.wireguard.com/protocol/) - Index-based - dispatch inspiration -- [Noise Protocol Framework](https://noiseprotocol.org/) - IK pattern -- [RFC 6479](https://tools.ietf.org/html/rfc6479) - IPsec anti-replay window - ---- - -## Appendix A: Detailed Packet Layouts - -### A.1 Encrypted Frame (0x00) - -Post-handshake data packets between authenticated peers. - -```text -┌─────────────────────────────────────────────────────────────────────────────┐ -│ ENCRYPTED FRAME (0x00) │ -├─────────────────────────────────────────────────────────────────────────────┤ -│ │ -│ ┌───────────────────────────────────────────────────────────────────────┐ │ -│ │ WIRE FORMAT │ │ -│ ├────────┬──────────────────┬───────────┬───────────────────────────────┤ │ -│ │ Offset │ Field │ Size │ Description │ │ -│ ├────────┼──────────────────┼───────────┼───────────────────────────────┤ │ -│ │ 0 │ discriminator │ 1 byte │ 0x00 │ │ -│ │ 1 │ receiver_idx │ 4 bytes │ u32 LE, receiver's session idx│ │ -│ │ 5 │ counter │ 8 bytes │ u64 LE, monotonic nonce │ │ -│ │ 13 │ ciphertext │ N bytes │ ChaCha20 encrypted payload │ │ -│ │ 13+N │ tag │ 16 bytes │ Poly1305 auth tag │ │ -│ └────────┴──────────────────┴───────────┴───────────────────────────────┘ │ -│ │ -│ Total overhead: 29 bytes (1 + 4 + 8 + 16) │ -│ │ -├─────────────────────────────────────────────────────────────────────────────┤ -│ PLAINTEXT STRUCTURE │ -│ (after decryption) │ -├─────────────────────────────────────────────────────────────────────────────┤ -│ │ -│ ┌────────┬──────────────────┬───────────┬───────────────────────────────┐ │ -│ │ Offset │ Field │ Size │ Description │ │ -│ ├────────┼──────────────────┼───────────┼───────────────────────────────┤ │ -│ │ 0 │ msg_type │ 1 byte │ Link message type (see below) │ │ -│ │ 1 │ payload │ variable │ Message-specific payload │ │ -│ └────────┴──────────────────┴───────────┴───────────────────────────────┘ │ -│ │ -│ Link message types (inside encrypted frame): │ -│ 0x10 = TreeAnnounce 0x30 = LookupRequest │ -│ 0x20 = FilterAnnounce 0x31 = LookupResponse │ -│ 0x40 = SessionDatagram 0x50 = Disconnect │ -│ │ -│ SessionDatagram (0x40) carries session-layer payloads: │ -│ 0x00 = SessionSetup 0x10 = DataPacket │ -│ 0x01 = SessionAck 0x20 = CoordsRequired │ -│ 0x21 = PathBroken │ -│ │ -└─────────────────────────────────────────────────────────────────────────────┘ -``` - -**Concrete example** (TreeAnnounce inside encrypted frame): - -```text -WIRE BYTES (hex): -00 ← discriminator -78 56 34 12 ← receiver_idx = 0x12345678 (LE) -2A 00 00 00 00 00 00 00 ← counter = 42 (LE) -[N bytes ciphertext] ← encrypted link message -[16 bytes tag] ← Poly1305 authentication tag - -DECRYPTED PLAINTEXT: -10 ← msg_type = TreeAnnounce -[TreeAnnounce payload] ← see fips-gossip-protocol.md -``` - -### A.2 Noise IK Message 1 (0x01) - -Handshake initiation from connecting party (initiator → responder). - -```text -┌─────────────────────────────────────────────────────────────────────────────┐ -│ NOISE IK MESSAGE 1 (0x01) │ -├─────────────────────────────────────────────────────────────────────────────┤ -│ │ -│ ┌───────────────────────────────────────────────────────────────────────┐ │ -│ │ WIRE FORMAT │ │ -│ ├────────┬──────────────────┬───────────┬───────────────────────────────┤ │ -│ │ Offset │ Field │ Size │ Description │ │ -│ ├────────┼──────────────────┼───────────┼───────────────────────────────┤ │ -│ │ 0 │ discriminator │ 1 byte │ 0x01 │ │ -│ │ 1 │ sender_idx │ 4 bytes │ u32 LE, initiator's session idx│ │ -│ │ 5 │ noise_msg1 │ 82 bytes │ Noise IK first message │ │ -│ └────────┴──────────────────┴───────────┴───────────────────────────────┘ │ -│ │ -│ Total: 87 bytes │ -│ │ -├─────────────────────────────────────────────────────────────────────────────┤ -│ NOISE MSG1 BREAKDOWN │ -├─────────────────────────────────────────────────────────────────────────────┤ -│ │ -│ ┌────────┬──────────────────┬───────────┬───────────────────────────────┐ │ -│ │ Offset │ Field │ Size │ Description │ │ -│ ├────────┼──────────────────┼───────────┼───────────────────────────────┤ │ -│ │ 0 │ ephemeral_pubkey │ 33 bytes │ Initiator's ephemeral pubkey │ │ -│ │ │ │ │ (compressed secp256k1) │ │ -│ │ 33 │ encrypted_static │ 33 bytes │ Initiator's static pubkey │ │ -│ │ │ │ │ (encrypted with es key) │ │ -│ │ 66 │ tag │ 16 bytes │ AEAD tag for encrypted_static │ │ -│ └────────┴──────────────────┴───────────┴───────────────────────────────┘ │ -│ │ -│ Noise pattern: -> e, es, s, ss │ -│ - e: ephemeral pubkey sent in clear │ -│ - es: DH(ephemeral, responder_static) → mix into key │ -│ - s: static pubkey encrypted with current key │ -│ - ss: DH(static, responder_static) → mix into key │ -│ │ -└─────────────────────────────────────────────────────────────────────────────┘ -``` - -**Concrete example**: - -```text -WIRE BYTES (hex): -01 ← discriminator -78 56 34 12 ← sender_idx = 0x12345678 (LE) -02 [32 bytes] ← ephemeral pubkey (compressed, 02/03 prefix) -[33 bytes] ← encrypted static pubkey -[16 bytes] ← AEAD tag - -Total: 87 bytes -``` - -### A.3 Noise IK Message 2 (0x02) - -Handshake response from responder (responder → initiator). - -```text -┌─────────────────────────────────────────────────────────────────────────────┐ -│ NOISE IK MESSAGE 2 (0x02) │ -├─────────────────────────────────────────────────────────────────────────────┤ -│ │ -│ ┌───────────────────────────────────────────────────────────────────────┐ │ -│ │ WIRE FORMAT │ │ -│ ├────────┬──────────────────┬───────────┬───────────────────────────────┤ │ -│ │ Offset │ Field │ Size │ Description │ │ -│ ├────────┼──────────────────┼───────────┼───────────────────────────────┤ │ -│ │ 0 │ discriminator │ 1 byte │ 0x02 │ │ -│ │ 1 │ sender_idx │ 4 bytes │ u32 LE, responder's session idx│ │ -│ │ 5 │ receiver_idx │ 4 bytes │ u32 LE, echo of initiator's idx│ │ -│ │ 9 │ noise_msg2 │ 33 bytes │ Noise IK second message │ │ -│ └────────┴──────────────────┴───────────┴───────────────────────────────┘ │ -│ │ -│ Total: 42 bytes │ -│ │ -├─────────────────────────────────────────────────────────────────────────────┤ -│ NOISE MSG2 BREAKDOWN │ -├─────────────────────────────────────────────────────────────────────────────┤ -│ │ -│ ┌────────┬──────────────────┬───────────┬───────────────────────────────┐ │ -│ │ Offset │ Field │ Size │ Description │ │ -│ ├────────┼──────────────────┼───────────┼───────────────────────────────┤ │ -│ │ 0 │ ephemeral_pubkey │ 33 bytes │ Responder's ephemeral pubkey │ │ -│ │ │ │ │ (compressed secp256k1) │ │ -│ └────────┴──────────────────┴───────────┴───────────────────────────────┘ │ -│ │ -│ Noise pattern: <- e, ee, se │ -│ - e: ephemeral pubkey sent in clear │ -│ - ee: DH(responder_ephemeral, initiator_ephemeral) → mix into key │ -│ - se: DH(responder_ephemeral, initiator_static) → mix into key │ -│ │ -│ After msg2, both parties derive identical session keys. │ -│ │ -└─────────────────────────────────────────────────────────────────────────────┘ -``` - -**Concrete example**: - -```text -WIRE BYTES (hex): -02 ← discriminator -01 EF CD AB ← sender_idx = 0xABCDEF01 (LE) -78 56 34 12 ← receiver_idx = 0x12345678 (LE, echoed) -03 [32 bytes] ← ephemeral pubkey (compressed, 02/03 prefix) - -Total: 42 bytes -``` - -### A.4 Complete Handshake Flow - -```text -Initiator (A) Responder (B) -───────────── ───────────── -generates sender_idx = 0x12345678 -generates ephemeral keypair - - ┌──────────────────────────────────────────────────────┐ - │ 0x01 | 0x12345678 | [82 bytes noise_msg1] │ - └──────────────────────────────────────────────────────┘ - ──────────────────────────────► - - validates msg1 - learns A's static pubkey - generates sender_idx = 0xABCDEF01 - generates ephemeral keypair - - ┌──────────────────────────────────────────────────────┐ - │ 0x02 | 0xABCDEF01 | 0x12345678 | [33 bytes noise_msg2]│ - └──────────────────────────────────────────────────────┘ - ◄────────────────────────────── - -validates msg2 -derives session keys - -═══════════════════════ HANDSHAKE COMPLETE ═══════════════════════ - -A's view: B's view: - our_index = 0x12345678 our_index = 0xABCDEF01 - their_index = 0xABCDEF01 their_index = 0x12345678 - -A sends to B: B sends to A: - receiver_idx = 0xABCDEF01 receiver_idx = 0x12345678 - - ┌──────────────────────────────────────────────────────┐ - │ 0x00 | 0xABCDEF01 | counter=0 | [ciphertext+tag] │ - └──────────────────────────────────────────────────────┘ - ──────────────────────────────► -``` - ---- - -## Appendix B: Message Size Summary - -| Packet Type | Size | Overhead | -|-------------|------|----------| -| Noise IK msg1 | 87 bytes | - | -| Noise IK msg2 | 42 bytes | - | -| Encrypted frame | N + 29 bytes | 29 bytes | -| Minimum encrypted | 30 bytes | (1 byte payload) | - -For comparison: - -- IPv6 header: 40 bytes -- WireGuard data: N + 32 bytes (type 4, idx 4, counter 8, tag 16) -- FIPS: slightly more compact due to 1-byte discriminator vs 4-byte type - ---- - -## Appendix B: Example Packet Traces - -### B.1 Outbound Connection - -``` -Node A (initiator) → Node B (responder) - -A generates: sender_idx = 0x12345678 -A sends msg1: - [01] [78 56 34 12] [82 bytes noise_msg1] - -B receives, processes msg1, generates: sender_idx = 0xABCDEF01 -B sends msg2: - [02] [01 EF CD AB] [78 56 34 12] [33 bytes noise_msg2] - -A receives msg2, handshake complete. -A's our_index = 0x12345678, their_index = 0xABCDEF01 -B's our_index = 0xABCDEF01, their_index = 0x12345678 - -A sends encrypted: - [00] [01 EF CD AB] [00 00 00 00 00 00 00 00] [ciphertext+tag] - ^ B's our_index (A's their_index) - -B receives, looks up 0xABCDEF01 → finds session with A -B decrypts, updates A's address if changed -``` - -### B.2 Roaming Scenario - -``` -Initial: A connected from 10.0.0.1:4000, established session - -A's network changes to 10.0.0.2:5000 - -A sends encrypted from new address: - src=10.0.0.2:5000 - [00] [01 EF CD AB] [01 00 00 00 00 00 00 00] [ciphertext+tag] - -B receives: - 1. Lookup index 0xABCDEF01 → finds A's session - 2. Decrypt succeeds - 3. Update A's address: 10.0.0.1:4000 → 10.0.0.2:5000 - -B's subsequent packets to A now go to 10.0.0.2:5000 -``` diff --git a/docs/design/spanning-tree-dynamics.md b/docs/design/spanning-tree-dynamics.md index 7014fd0..c8608ed 100644 --- a/docs/design/spanning-tree-dynamics.md +++ b/docs/design/spanning-tree-dynamics.md @@ -5,7 +5,10 @@ operational behavior under various mesh conditions. This document complements [fips-intro.md](fips-intro.md) with step-by-step walkthroughs of protocol dynamics rather than message formats and data structures. -For wire formats, see [fips-gossip-protocol.md](fips-gossip-protocol.md) §2 (TreeAnnounce). +For wire formats, see [fips-wire-formats.md](fips-wire-formats.md) (TreeAnnounce section). +For spanning tree algorithms and data structures, see +[fips-spanning-tree.md](fips-spanning-tree.md). For how the spanning tree fits +into mesh routing, see [fips-mesh-operation.md](fips-mesh-operation.md). The protocol is based on Yggdrasil v0.5's CRDT gossip design. @@ -69,13 +72,6 @@ The root is deterministic: the node with the lexicographically smallest node_add among all reachable nodes. No explicit election protocol exists—each node independently derives the same answer from its local TreeState. -### Announcement Timing (from Yggdrasil v0.5) - -- Root timestamp refresh: every 30 minutes -- Root timeout: 60 minutes without refresh -- Peer keepalive: triggered by data activity, not periodic -- TTL on tree entries: implementation-specific, typically minutes - --- ## 2. Single Node Startup @@ -109,9 +105,7 @@ At this point, node A is a fully functional single-node FIPS network. It can: While isolated, A's state only changes on: -1. **Periodic refresh**: A regenerates its announcement with incremented sequence - and fresh timestamp (maintains liveness for future peers) -2. **Peer connection**: A new peer triggers gossip exchange (covered in Section 3) +1. **Peer connection**: A new peer triggers gossip exchange (covered in Section 3) --- @@ -220,11 +214,12 @@ The propagation time is O(tree depth), not O(network size). In the example: - B's coordinates are known immediately (B computes from D's ancestry) - B's reachability propagates via bloom filter: D → A (1 hop to root) -- Any node wanting to reach B does a bloom filter lookup +- Any node wanting to reach B does a bloom filter lookup for candidate selection - Total: 1-2 gossip rounds for B to be locatable Note: Nodes A, C, E never add B to their TreeState. They can still route to B -by using bloom filter lookup to get B's coordinates, then greedy forwarding. +by using bloom filter lookup for candidate selection to get B's coordinates, +then coordinate-based greedy routing. --- @@ -291,8 +286,8 @@ T7: Converged state - Any two nodes can compute accurate distance via their coordinates Nodes do *not* have global knowledge—a leaf node knows nothing about distant -subtrees. But any node can locate any other node via bloom filter lookup and -then route using coordinates. +subtrees. But any node can locate any other node via Bloom-guided candidate +selection and then route using tree coordinate distance. **Convergence time**: Bounded by tree depth × gossip interval. For a tree of depth D with gossip interval G: @@ -312,7 +307,7 @@ During convergence, the network may temporarily have: - **Multiple roots**: Different partitions with different root beliefs - **Inconsistent coordinates**: Nodes computing distances from stale state -- **Routing failures**: Greedy routing may fail until coordinates stabilize +- **Routing failures**: Coordinate-based greedy routing may fail until coordinates stabilize These are transient. The protocol guarantees eventual convergence, not instant consistency. @@ -424,18 +419,15 @@ Nodes detect they're partitioned when: 1. **Parent unreachable**: Direct link to parent fails 2. **Root unreachable**: No peer has path to current root -3. **Stale root timestamp**: Root's announcement exceeds timeout (60 min) -**Detection via gossip staleness**: +**Detection via gossip staleness** (not currently implemented — see +[Known Limitations](#known-limitations-v1-implementation)): -``` -For each entry in TreeState: - if now - entry.timestamp > TTL: - expire(entry) - -If root entry expires: - re-evaluate root from remaining entries -``` +In principle, nodes would also detect partitions through root entry staleness: +if no fresh root announcements arrive within a timeout, the root is presumed +departed. This requires tracking root entry timestamps and enforcing expiration, +which is not yet implemented. Currently, only direct parent loss (case 1) and +absence of any peer with a path to root (case 2) trigger partition detection. ### Independent Operation @@ -551,14 +543,14 @@ link_failed(peer): - Short timeout: Quick failure detection, but transient issues cause flapping - Long timeout: Stable under jitter, but slow to respond to real failures -**Typical values**: +**Typical values** (see [fips-spanning-tree.md](fips-spanning-tree.md) for +current FIPS-specific parameters): ``` peer_timeout: 10-30 seconds keepalive_interval: peer_timeout / 3 -gossip_interval: 1-5 seconds (or on-change) -tree_entry_ttl: 5-10 minutes -root_timeout: 60 minutes +gossip_interval: on topology change (no periodic refresh) +tree_entry_ttl: not currently enforced (see Known Limitations) ``` ### Asymmetric Failures @@ -637,9 +629,9 @@ the current parent (under the same root) to trigger a switch. Root changes always trigger a switch regardless of depth. **What this means for tree structure**: The v1 algorithm produces minimum-depth -trees, which minimizes coordinate path length and hop count for greedy routing. -However, it does not account for link quality—a high-latency or lossy link at -depth 1 is preferred over a fast link at depth 2. +trees, which minimizes coordinate path length and hop count for coordinate-based +greedy routing. However, it does not account for link quality—a high-latency or +lossy link at depth 1 is preferred over a fast link at depth 2. ### v2 Planned: Cost Metrics @@ -699,32 +691,33 @@ Once converged, what does the network look like and how does it behave? **Quiescent gossip**: -- Announcements only on periodic refresh (every few minutes) -- Delta encoding minimizes redundant information +- TreeAnnounce messages sent only on topology changes, not periodically +- No periodic root refresh — the tree is maintained purely by change-driven gossip +- In a stable network, gossip traffic drops to zero - Bandwidth usage proportional to tree depth, not network size **Consistent coordinates**: - Every node knows its full path to root - Distance calculations are accurate -- Greedy routing succeeds +- Coordinate-based greedy routing succeeds ### Steady State Gossip Pattern ``` Normal operation (no topology changes): -Root A: Refreshes timestamp every 30 minutes - └── Gossips refresh to children +Root A: No periodic announcements — announces only if topology changes + └── Root does not refresh its timestamp periodically -Each node: Forwards root's refresh when received - └── Only sends if peer's view is stale +Each node: Sends TreeAnnounce only when its own state changes + └── Parent change, new peer, or peer departure -Typical gossip per node: -├── Receive refresh from parent (periodic) -├── Forward to children if needed -├── Send own refresh periodically (separate from root's) -└── No gossip if nothing changed and peer is up-to-date +Typical gossip per node in steady state: +├── No periodic sends — tree gossip is entirely change-driven +├── Announce on parent selection change +├── Announce on peer link up/down +└── Zero gossip traffic when topology is stable ``` ### Expected Steady State Properties @@ -732,15 +725,14 @@ Typical gossip per node: **Gossip volume**: ``` -Per link, per refresh cycle: -├── Root timestamp update: ~100 bytes -├── Own declaration (if changed): ~100 bytes +Per topology change event: +├── Own declaration update: ~100 bytes ├── Delta of changed ancestors: varies └── Total: O(100 bytes) to O(depth * 100 bytes) -For 1000-node network with depth ~10: -├── Each node sends O(1 KB) per refresh cycle -├── With 30-minute refresh: ~0.5 bytes/second per link +In steady state (no topology changes): +├── Zero gossip traffic — no periodic refreshes +├── Traffic resumes only when links change or nodes join/depart └── Negligible compared to application traffic ``` @@ -774,10 +766,10 @@ In steady state with infrequent updates: Indicators the network has converged: -1. **Root stability**: Same root for multiple refresh cycles +1. **Root stability**: Same root over extended period 2. **Parent stability**: No parent changes in recent interval -3. **Sequence number stability**: Sequence numbers increment slowly (refresh only) -4. **Routing success**: Greedy routing doesn't hit local minima +3. **Sequence number stability**: Sequence numbers increment only on topology changes +4. **Routing success**: Coordinate-based greedy routing doesn't hit local minima Warning signs of instability: @@ -832,10 +824,10 @@ T3: Converged tree (assuming equal link costs): **Steady state**: -- A is root, refreshes every 30 min +- A is root - B, C, D are direct children of A - E is child of C (one hop to A through C) -- Gossip: Each refresh cycle propagates through 2 levels +- No periodic gossip — TreeAnnounce only on topology changes **Link failure scenario**: @@ -1035,9 +1027,8 @@ work. ### Known Limitation: Root Timeout Not Enforced -The design specifies a 60-minute root timeout (§1 timing parameters, §6 -partition detection) after which nodes should treat the root as departed and -re-elect. The current implementation does not track root entry timestamps or +The design specifies a 60-minute root timeout (§6 partition detection) after +which nodes should treat the root as departed and re-elect. The current implementation does not track root entry timestamps or perform staleness checks. **Impact**: If the root node disappears permanently without a graceful @@ -1137,8 +1128,9 @@ through: 7. **Stability thresholds** - Hysteresis prevents flapping on similar-cost paths Each node maintains only its own ancestry and direct peer information—not global -topology. Reachability to arbitrary destinations is provided by bloom filters -propagating up the tree, with coordinate discovery via lookup protocol. +topology. Reachability to arbitrary destinations is provided by Bloom-guided +candidate selection (bloom filters propagating up the tree), with coordinate +discovery via lookup protocol and coordinate-based greedy routing for forwarding. The protocol handles partitions gracefully (independent operation), heals automatically when connectivity returns, and adapts to heterogeneous link @@ -1148,6 +1140,12 @@ costs to form efficient tree structures. ## References +### FIPS Internal Documentation + +- [fips-spanning-tree.md](fips-spanning-tree.md) — Spanning tree algorithms and data structures +- [fips-mesh-operation.md](fips-mesh-operation.md) — How the spanning tree fits into mesh routing +- [fips-wire-formats.md](fips-wire-formats.md) — TreeAnnounce wire format + ### Yggdrasil Documentation - [Yggdrasil v0.5 Release Notes](https://yggdrasil-network.github.io/2023/10/22/upcoming-v05-release.html)