diff --git a/docs/design/README.md b/docs/design/README.md index 38da2f2..431455a 100644 --- a/docs/design/README.md +++ b/docs/design/README.md @@ -6,4 +6,5 @@ Protocol design specifications and analysis. |--------------------------------------------------------|-------------------------------------------------| | [fips-design.md](fips-design.md) | Full protocol design specification | | [fips-links.md](fips-links.md) | Link protocol requirements and characteristics | +| [fips-routing.md](fips-routing.md) | Routing, Bloom filters, discovery, sessions | | [spanning-tree-dynamics.md](spanning-tree-dynamics.md) | Spanning tree protocol dynamics study | diff --git a/docs/design/fips-routing.md b/docs/design/fips-routing.md new file mode 100644 index 0000000..d8ee1e6 --- /dev/null +++ b/docs/design/fips-routing.md @@ -0,0 +1,608 @@ +# FIPS Routing Design + +**Status**: Work in Progress + +This document describes the routing architecture for FIPS, including Bloom +filter reachability, discovery protocol, greedy tree routing, and session +establishment. + +## Overview + +FIPS routing combines three mechanisms: + +1. **Bloom filters**: Fast reachability lookup for nearby destinations (within + K-hop scope) +2. **Discovery protocol**: Query-based lookup for distant destinations +3. **Greedy tree routing**: Coordinate-based forwarding using spanning tree + position + +The design separates discovery (finding where a destination is) from routing +(getting packets there). Bloom filters and discovery handle the former; tree +coordinates handle the latter. + +## Design Goals + +- Minimize per-packet overhead for data transfer +- Bounded state at each node (independent of network size) +- Efficient routing without global knowledge +- Graceful degradation for constrained devices +- Fast convergence on topology changes + +## Network Scale Assumptions + +| Scale | Nodes | Bloom Filter Role | +|-------|-------|-------------------| +| Small private network | 100-1,000 | Covers entire network | +| Modest public network | ~1,000,000 | Covers K-hop neighborhood | +| Internet-scale | Billions | Out of scope (requires different architecture) | + +The primary design target is networks up to ~1M nodes. + +## Node Participation Modes + +### Full Participant + +- Maintains Bloom filters for peer reachability +- Participates in spanning tree (can be selected as parent) +- Routes packets for other nodes +- Minimum viable device: ESP32-class (~500KB RAM) + +### Leaf-Only + +- Single peer handles all routing on its behalf +- No Bloom filter storage or processing +- Does not participate in spanning tree as potential parent +- Suitable for highly constrained devices (sensors, battery-powered nodes) + +Leaf-only nodes appear as a single entry in their peer's Bloom filter. All +traffic tunnels through that peer. + +--- + +## Part 1: Bloom Filter Design + +### Parameters + +| Parameter | Value | Rationale | +|-----------|-------|-----------| +| Filter size | 4 KB (32,768 bits) | Balances accuracy vs. memory | +| Hash functions | 7 | Near-optimal for expected fill ratio | +| Scope (K) | 2 | Effective ~4-hop reach with TTL propagation | + +### False Positive Rates + +| Nodes in Filter | FPR | +|-----------------|-----| +| 1,000 | ~0.05% | +| 2,000 | ~0.5% | +| 5,000 | ~1.3% | +| 10,000 | ~8% | + +With K=2 and average degree d=8, expected nodes in scope ≈ d^(2K) ≈ 4,096. + +### Filter Contents + +Each node's filter contains Node IDs (and optionally gateway /64 prefixes) +that are reachable through that node. A Node ID is the SHA-256 hash of the +node's npub, truncated or used directly as the filter key. + +### Per-Peer Filters + +Each node maintains a Bloom filter for each peer direction: + +```rust +peer_filters: HashMap +``` + +The filter for peer P answers: "Which destinations are reachable through P?" + +### Update Mechanism: Event-Driven + +Filters are updated on events rather than periodic refresh: + +**Triggering events:** + +1. Peer connects — exchange current filters +2. Peer disconnects — remove their filter, recompute, notify other peers +3. Received filter changes outgoing filter — recompute, send updates +4. Local state change — new leaf dependent, become gateway, etc. + +**Rate limiting:** + +To prevent update storms, rate-limit or debounce updates: + +```rust +const MIN_UPDATE_INTERVAL: Duration = Duration::from_millis(500); + +fn maybe_send_update(&mut self, peer: PeerId) { + if self.last_send_time[peer].elapsed() < MIN_UPDATE_INTERVAL { + self.pending_update[peer] = true; + return; + } + // ... send update +} +``` + +### Filter Announcement Message + +```rust +struct FilterAnnounce { + filter: BloomFilter, // 4 KB + ttl: u8, // Remaining propagation hops + sequence: u64, // Freshness / deduplication +} +``` + +### Propagation Rules + +When node N sends a FilterAnnounce to peer Q: + +1. Include N's own Node ID +2. Include N's leaf-only dependents +3. Include entries from filters received from other peers (not Q) with TTL > 0 + +When node N receives FilterAnnounce from peer P: + +1. Store: `peer_filters[P] = received.filter` +2. If `received.ttl > 0`: include P's filter contents in N's next announcement + to other peers, with TTL decremented + +### K-Hop Scope Emergence + +With TTL starting at K=2: + +- Entries propagate ~2K hops before stopping +- Each node's filter contains destinations within ~4-hop effective range +- Bounded by O(d^2K) entries regardless of total network size + +### Expiration + +Bloom filters cannot remove individual entries. Expiration is handled via: + +- **Peer disconnect**: Remove that peer's filter entirely, recompute +- **Filter replacement**: Each FilterAnnounce replaces the previous one +- **Implicit timeout**: If no updates received from peer within threshold, + consider their filter stale + +--- + +## Part 2: Discovery Protocol + +### Purpose + +Discover the tree coordinates of distant destinations not covered by local +Bloom filters. + +### When Used + +- Destination not found in any peer's Bloom filter +- Route cache miss +- After cached route failure + +### Message Formats + +```rust +struct LookupRequest { + request_id: u64, + target: NodeId, // Who we're looking for + origin: NodeId, // Who's asking + origin_coords: Vec, // Origin's ancestry (for return path) + ttl: u8, // Propagation limit + visited: BloomFilter, // Prevent loops (compact, ~256 bytes) +} + +struct LookupResponse { + request_id: u64, + target: NodeId, + target_coords: Vec, // Target's ancestry — the key payload + proof: Signature, // Target signs to prove existence +} +``` + +### Discovery Flow + +```text +1. S wants to reach D, D not in any local filter +2. S checks route cache — miss +3. S creates LookupRequest with own coordinates, floods to peers +4. Request propagates (Bloom filters may help direct it) +5. Request reaches D (or node with D in filter) +6. D creates LookupResponse with its coordinates, signs it +7. Response routes back to S using S's coordinates (greedy) +8. S caches D's coordinates +9. S can now route to D using greedy tree routing +``` + +### Request Propagation + +**Flood with TTL and visited filter:** + +- Send to all peers not in `visited` filter +- Each hop decrements TTL, adds self to `visited` +- At TTL=0, stop propagating +- `visited` filter prevents redundant processing + +**Bloom filter assistance (optional optimization):** + +If a node's peer filter indicates "maybe" for the target, prioritize that +direction. Reduces flood scope when target is partially in range. + +### Response Routing + +Response uses greedy tree routing based on `origin_coords` from the request. +Each router forwards toward the origin using tree distance. + +### Security + +**Target signs response:** + +```rust +struct LookupResponse { + // ... + proof: Signature, // Sign(request_id || target || target_coords) +} +``` + +Without this, a malicious node could claim reachability for any target and +blackhole traffic. The signature proves the target authorized the route. + +### Caching + +Discovered coordinates are cached: + +```rust +struct RouteCache { + entries: HashMap, +} + +struct CachedCoords { + coords: Vec, + discovered_at: Timestamp, + last_used: Timestamp, +} +``` + +- **Eviction**: LRU when cache full +- **Expiration**: TTL-based (coordinates may go stale if target moves in tree) +- **Invalidation**: On route failure, evict and re-discover + +--- + +## Part 3: Tree Coordinates and Greedy Routing + +### Tree Coordinates + +A node's coordinates are its ancestry path from self to root: + +```text +coords(N) = [N, Parent(N), Parent(Parent(N)), ..., Root] +``` + +Example: Node D at depth 4 has coordinates `[D, P1, P2, P3, Root]`. + +### Tree Distance + +Distance between two nodes is hops through their lowest common ancestor (LCA): + +```rust +fn tree_distance(a_coords: &[NodeId], b_coords: &[NodeId]) -> usize { + let lca_depth = longest_common_suffix_length(a_coords, b_coords); + let a_to_lca = a_coords.len() - lca_depth; + let b_to_lca = b_coords.len() - lca_depth; + a_to_lca + b_to_lca +} +``` + +Note: Coordinates are ordered self-to-root, so common ancestry is a suffix. + +### Greedy Routing Algorithm + +```rust +fn greedy_next_hop(&self, dest_coords: &[NodeId]) -> PeerId { + // Check if we are the destination + if dest_coords[0] == self.node_id { + return LOCAL_DELIVERY; + } + + // Check if destination is a direct peer + for peer in &self.peers { + if peer.node_id == dest_coords[0] { + return peer.id; + } + } + + // Forward to peer closest to destination + self.peers + .iter() + .min_by_key(|p| tree_distance(&p.coords, dest_coords)) + .map(|p| p.id) + .expect("no peers") +} +``` + +### Guaranteed Progress + +Greedy routing makes progress as long as: + +1. Tree is connected +2. Destination's coordinates are accurate +3. Current node is not the destination + +Unlike DHT routing, greedy tree routing cannot get stuck in local minima if +the tree is properly formed. + +### What Each Node Knows + +| Information | Source | +|-------------|--------| +| Own coordinates | Spanning tree protocol (ancestry to root) | +| Each peer's coordinates | Exchanged on peering | +| Destination coordinates | From packet header (established via session) | + +No global routing tables. Each node makes purely local decisions. + +--- + +## Part 4: Session Establishment + +### Session Purpose + +Establish cached coordinate state along a path so that subsequent data packets +can omit coordinates, minimizing per-packet overhead. + +### Session Lifecycle + +```text +┌─────────────────────────────────────────────────────────────────┐ +│ 1. Discovery: S queries for D's coordinates │ +│ 2. Setup: S sends SessionSetup, routers cache coordinates │ +│ 3. Data: Packets carry only addresses, routers use cache │ +│ 4. Refresh: Periodic or on-demand to prevent cache expiry │ +│ 5. Teardown: Implicit (cache expires) or explicit │ +└─────────────────────────────────────────────────────────────────┘ +``` + +### Session Message Formats + +```rust +/// Establishes cached state along path +struct SessionSetup { + src_addr: Ipv6Addr, + dest_addr: Ipv6Addr, + src_coords: Vec, // For return path caching + dest_coords: Vec, // For forward path routing + flags: SessionFlags, +} + +struct SessionFlags { + request_ack: bool, // Ask destination to confirm + bidirectional: bool, // Set up both directions +} + +/// Confirms session establishment +struct SessionAck { + src_addr: Ipv6Addr, + dest_addr: Ipv6Addr, + src_coords: Vec, // Acknowledger's coords (for return caching) +} + +/// Minimal data packet +struct DataPacket { + flags: u8, + hop_limit: u8, + payload_length: u16, + src_addr: Ipv6Addr, // 16 bytes + dest_addr: Ipv6Addr, // 16 bytes + payload: Vec, +} + +/// Error when router cannot route (cache miss) +struct CoordsRequired { + dest_addr: Ipv6Addr, + reporter: NodeId, // Which router had the miss +} +``` + +### Data Packet Overhead + +| Field | Size | +|-------|------| +| flags | 1 byte | +| hop_limit | 1 byte | +| payload_length | 2 bytes | +| src_addr | 16 bytes | +| dest_addr | 16 bytes | +| **Total header** | **36 bytes** | + +Comparable to IPv6 (40 bytes). No coordinates in data packets. + +### Session Setup Flow + +```text +S R1 R2 D +│ │ │ │ +│──SessionSetup─────────>│ │ │ +│ (src_coords, │──SessionSetup────────>│ │ +│ dest_coords) │ │──SessionSetup────────>│ +│ │ │ │ +│ │ cache: │ cache: │ +│ │ dest_addr→dest_coords│ dest_addr→dest_coords│ +│ │ src_addr→src_coords │ src_addr→src_coords │ +│ │ │ │ +│<─────────────────────────────────────────────────────────SessionAck───│ +│ │ │ │ +│══DataPacket═══════════>│══════════════════════>│══════════════════════>│ +│ (addresses only) │ (use cached coords) │ (use cached coords) │ +``` + +### Router Behavior + +```rust +impl Router { + fn handle_session_setup(&mut self, setup: SessionSetup, from: PeerId) { + // Cache coordinates for both directions + self.coord_cache.insert(setup.dest_addr, CacheEntry { + coords: setup.dest_coords.clone(), + expires: now() + CACHE_TTL, + }); + self.coord_cache.insert(setup.src_addr, CacheEntry { + coords: setup.src_coords.clone(), + expires: now() + CACHE_TTL, + }); + + // Forward toward destination + let next = self.greedy_next_hop(&setup.dest_coords); + self.forward(next, setup); + } + + fn handle_data_packet(&mut self, packet: DataPacket, from: PeerId) { + match self.coord_cache.get(&packet.dest_addr) { + Some(entry) => { + entry.last_used = now(); + let next = self.greedy_next_hop(&entry.coords); + self.forward(next, packet); + } + None => { + // Cache miss — request coordinates + self.send_error(from, CoordsRequired { + dest_addr: packet.dest_addr, + reporter: self.node_id, + }); + } + } + } +} +``` + +### Cache Management + +```rust +struct CoordCache { + entries: HashMap, + max_entries: usize, +} + +struct CacheEntry { + coords: Vec, + created: Timestamp, + last_used: Timestamp, + expires: Timestamp, +} +``` + +**Eviction policy**: LRU (least recently used) when cache exceeds max_entries. + +**Expiration**: Entries expire after TTL (e.g., 300 seconds). Can be refreshed +by: + +- Subsequent SessionSetup +- SessionRefresh message (lightweight, just touches expiry) +- Data packet transit (optional: refresh on use) + +### Handling Cache Eviction + +When a router's cache entry is evicted mid-session: + +```text +1. Data packet arrives, cache miss +2. Router sends CoordsRequired to packet source +3. Source receives error, re-sends SessionSetup (or packet with coords) +4. Path warms again, data flow resumes +``` + +From application perspective: brief latency spike, transparent recovery. + +### Sender Behavior + +```rust +impl Sender { + fn send(&mut self, dest: Ipv6Addr, data: &[u8]) { + if !self.session_established(dest) { + // Need to establish session first + let dest_coords = self.discover_or_cached(dest)?; + self.send_session_setup(dest, &dest_coords); + self.await_session_ack(dest)?; + } + + self.send_data_packet(dest, data); + } + + fn handle_coords_required(&mut self, err: CoordsRequired) { + // Path went cold, re-establish + self.mark_session_cold(err.dest_addr); + self.send_session_setup(err.dest_addr, &self.cached_coords(err.dest_addr)); + } +} +``` + +--- + +## Part 5: Packet Type Summary + +| Type | Purpose | Size | When Used | +|------|---------|------|-----------| +| FilterAnnounce | Bloom filter propagation | ~4.1 KB | Topology changes | +| LookupRequest | Discover coordinates | ~300 bytes | First contact with distant node | +| LookupResponse | Return coordinates | ~400 bytes | Reply to discovery | +| SessionSetup | Warm router caches | ~400-600 bytes | Before data transfer | +| SessionAck | Confirm session | ~300 bytes | Optional confirmation | +| DataPacket | Application data | 36 bytes + payload | Bulk of traffic | +| CoordsRequired | Request retransmit | ~50 bytes | Cache miss recovery | + +--- + +## Part 6: Traffic Analysis + +### Steady State (Stable Network) + +- **Bloom filter traffic**: Near zero (event-driven, no changes) +- **Discovery traffic**: Rare (warm caches) +- **Session traffic**: Rare (established sessions) +- **Data traffic**: Minimal overhead (36-byte header) + +### Network Churn + +When nodes join/leave: + +- Bloom filter updates propagate (bounded by K-hop scope) +- Affected sessions may need re-establishment +- Discovery queries for newly-joined nodes + +### Per-Node Resource Requirements + +| Resource | Full Participant | Leaf-Only | +|----------|------------------|-----------| +| Bloom filter storage | d × 4 KB (d = peer count) | None | +| Coordinate cache | 10K-100K entries | None | +| Route cache | 1K-10K entries | Minimal | +| Bandwidth (idle) | < 1 KB/sec | Near zero | + +--- + +## Open Questions + +1. **Coordinate compression**: Can tree coordinates be compressed for smaller + SessionSetup messages? (e.g., delta encoding, shorter node ID representation) + +2. **Multi-path routing**: How to handle multiple valid paths? Load balancing? + Failover? + +3. **Asymmetric paths**: S→D and D→S may traverse different routers. Is this + acceptable or should paths be symmetric? + +4. **Gateway /64 prefixes**: How do subnet prefixes interact with Bloom filters + and discovery? One filter entry per gateway regardless of devices behind it? + +5. **Cache sizing**: What's the right cache size for different node roles? + Core nodes vs. edge nodes? + +6. **Mobility**: When a node changes tree position (new parent), how quickly + do sessions recover? Should nodes announce position changes? + +--- + +## References + +- [fips-design.md](fips-design.md) — Overall FIPS architecture +- [fips-links.md](fips-links.md) — Link layer requirements +- [spanning-tree-dynamics.md](spanning-tree-dynamics.md) — Tree protocol details