fips/docs/design/fips-routing.md

FIPS Routing Design

Status: Work in Progress

This document describes the routing architecture for FIPS, including Bloom filter reachability, discovery protocol, greedy tree routing, and session establishment.

Overview

FIPS routing combines three mechanisms:

1. Bloom filters: Fast reachability lookup for nearby destinations (within K-hop scope)
2. Discovery protocol: Query-based lookup for distant destinations
3. Greedy tree routing: Coordinate-based forwarding using spanning tree position

The design separates discovery (finding where a destination is) from routing (getting packets there). Bloom filters and discovery handle the former; tree coordinates handle the latter.

Design Goals

• Minimize per-packet overhead for data transfer
• Bounded state at each node (independent of network size)
• Efficient routing without global knowledge
• Graceful degradation for constrained devices
• Fast convergence on topology changes

Network Scale Assumptions

Scale	Nodes	Bloom Filter Role
Small private network	100-1,000	Covers entire network
Modest public network	~1,000,000	Covers K-hop neighborhood
Internet-scale	Billions	Out of scope (requires different architecture)

The primary design target is networks up to ~1M nodes.

Node Participation Modes

Full Participant

• Maintains Bloom filters for peer reachability
• Participates in spanning tree (can be selected as parent)
• Routes packets for other nodes
• Minimum viable device: ESP32-class (~500KB RAM)

Leaf-Only

• Single peer handles all routing on its behalf
• No Bloom filter storage or processing
• Does not participate in spanning tree as potential parent
• Suitable for highly constrained devices (sensors, battery-powered nodes)

Leaf-only nodes appear as a single entry in their peer's Bloom filter. All traffic tunnels through that peer.

---

Part 1: Bloom Filter Design

Parameters

Parameter	Value	Rationale
Filter size	4 KB (32,768 bits)	Balances accuracy vs. memory
Hash functions	7	Near-optimal for expected fill ratio
Scope (K)	2	Effective ~4-hop reach with TTL propagation

False Positive Rates

Nodes in Filter	FPR
1,000	~0.05%
2,000	~0.5%
5,000	~1.3%
10,000	~8%

With K=2 and average degree d=8, expected nodes in scope ≈ d^(2K) ≈ 4,096.

Filter Contents

Each node's filter contains Node IDs (and optionally gateway /64 prefixes) that are reachable through that node. A Node ID is the SHA-256 hash of the node's npub, truncated or used directly as the filter key.

Per-Peer Filters

Each node maintains a Bloom filter for each peer direction:

peer_filters: HashMap<PeerId, BloomFilter>

The filter for peer P answers: "Which destinations are reachable through P?"

Update Mechanism: Event-Driven

Filters are updated on events rather than periodic refresh:

Triggering events:

1. Peer connects — exchange current filters
2. Peer disconnects — remove their filter, recompute, notify other peers
3. Received filter changes outgoing filter — recompute, send updates
4. Local state change — new leaf dependent, become gateway, etc.

Rate limiting:

To prevent update storms, rate-limit or debounce updates:

const MIN_UPDATE_INTERVAL: Duration = Duration::from_millis(500);

fn maybe_send_update(&mut self, peer: PeerId) {
    if self.last_send_time[peer].elapsed() < MIN_UPDATE_INTERVAL {
        self.pending_update[peer] = true;
        return;
    }
    // ... send update
}

Filter Announcement Message

struct FilterAnnounce {
    filter: BloomFilter,  // 4 KB
    ttl: u8,              // Remaining propagation hops
    sequence: u64,        // Freshness / deduplication
}

Propagation Rules

When node N sends a FilterAnnounce to peer Q:

1. Include N's own Node ID
2. Include N's leaf-only dependents
3. Include entries from filters received from other peers (not Q) with TTL > 0

When node N receives FilterAnnounce from peer P:

1. Store: peer_filters[P] = received.filter
2. If received.ttl > 0: include P's filter contents in N's next announcement to other peers, with TTL decremented

K-Hop Scope Emergence

With TTL starting at K=2:

• Entries propagate ~2K hops before stopping
• Each node's filter contains destinations within ~4-hop effective range
• Bounded by O(d^2K) entries regardless of total network size

Expiration

Bloom filters cannot remove individual entries. Expiration is handled via:

• Peer disconnect: Remove that peer's filter entirely, recompute
• Filter replacement: Each FilterAnnounce replaces the previous one
• Implicit timeout: If no updates received from peer within threshold, consider their filter stale

---

Part 2: Discovery Protocol

Purpose

Discover the tree coordinates of distant destinations not covered by local Bloom filters.

When Used

• Destination not found in any peer's Bloom filter
• Route cache miss
• After cached route failure

Message Formats

struct LookupRequest {
    request_id: u64,
    target: NodeId,              // Who we're looking for
    origin: NodeId,              // Who's asking
    origin_coords: Vec<NodeId>,  // Origin's ancestry (for return path)
    ttl: u8,                     // Propagation limit
    visited: BloomFilter,        // Prevent loops (compact, ~256 bytes)
}

struct LookupResponse {
    request_id: u64,
    target: NodeId,
    target_coords: Vec<NodeId>,  // Target's ancestry — the key payload
    proof: Signature,            // Target signs to prove existence
}

Discovery Flow

1. S wants to reach D, D not in any local filter
2. S checks route cache — miss
3. S creates LookupRequest with own coordinates, floods to peers
4. Request propagates (Bloom filters may help direct it)
5. Request reaches D (or node with D in filter)
6. D creates LookupResponse with its coordinates, signs it
7. Response routes back to S using S's coordinates (greedy)
8. S caches D's coordinates
9. S can now route to D using greedy tree routing

Request Propagation

Flood with TTL and visited filter:

• Send to all peers not in visited filter
• Each hop decrements TTL, adds self to visited
• At TTL=0, stop propagating
• visited filter prevents redundant processing

Bloom filter assistance (optional optimization):

If a node's peer filter indicates "maybe" for the target, prioritize that direction. Reduces flood scope when target is partially in range.

Response Routing

Response uses greedy tree routing based on origin_coords from the request. Each router forwards toward the origin using tree distance.

Security

Target signs response:

struct LookupResponse {
    // ...
    proof: Signature,  // Sign(request_id || target || target_coords)
}

Without this, a malicious node could claim reachability for any target and blackhole traffic. The signature proves the target authorized the route.

Caching

Discovered coordinates are cached:

struct RouteCache {
    entries: HashMap<NodeId, CachedCoords>,
}

struct CachedCoords {
    coords: Vec<NodeId>,
    discovered_at: Timestamp,
    last_used: Timestamp,
}

• Eviction: LRU when cache full
• Expiration: TTL-based (coordinates may go stale if target moves in tree)
• Invalidation: On route failure, evict and re-discover

---

Part 3: Tree Coordinates and Greedy Routing

Tree Coordinates

A node's coordinates are its ancestry path from self to root:

coords(N) = [N, Parent(N), Parent(Parent(N)), ..., Root]

Example: Node D at depth 4 has coordinates [D, P1, P2, P3, Root].

Tree Distance

Distance between two nodes is hops through their lowest common ancestor (LCA):

fn tree_distance(a_coords: &[NodeId], b_coords: &[NodeId]) -> usize {
    let lca_depth = longest_common_suffix_length(a_coords, b_coords);
    let a_to_lca = a_coords.len() - lca_depth;
    let b_to_lca = b_coords.len() - lca_depth;
    a_to_lca + b_to_lca
}

Note: Coordinates are ordered self-to-root, so common ancestry is a suffix.

Greedy Routing Algorithm

fn greedy_next_hop(&self, dest_coords: &[NodeId]) -> PeerId {
    // Check if we are the destination
    if dest_coords[0] == self.node_id {
        return LOCAL_DELIVERY;
    }

    // Check if destination is a direct peer
    for peer in &self.peers {
        if peer.node_id == dest_coords[0] {
            return peer.id;
        }
    }

    // Forward to peer closest to destination
    self.peers
        .iter()
        .min_by_key(|p| tree_distance(&p.coords, dest_coords))
        .map(|p| p.id)
        .expect("no peers")
}

Guaranteed Progress

Greedy routing makes progress as long as:

1. Tree is connected
2. Destination's coordinates are accurate
3. Current node is not the destination

Unlike DHT routing, greedy tree routing cannot get stuck in local minima if the tree is properly formed.

What Each Node Knows

Information	Source
Own coordinates	Spanning tree protocol (ancestry to root)
Each peer's coordinates	Exchanged on peering
Destination coordinates	From packet header (established via session)

No global routing tables. Each node makes purely local decisions.

---

Part 4: Session Establishment

Session Purpose

Establish cached coordinate state along a path so that subsequent data packets can omit coordinates, minimizing per-packet overhead.

Session Lifecycle

┌─────────────────────────────────────────────────────────────────┐
│  1. Discovery: S queries for D's coordinates                    │
│  2. Setup: S sends SessionSetup, routers cache coordinates      │
│  3. Data: Packets carry only addresses, routers use cache       │
│  4. Refresh: Periodic or on-demand to prevent cache expiry      │
│  5. Teardown: Implicit (cache expires) or explicit              │
└─────────────────────────────────────────────────────────────────┘

Session Message Formats

/// Establishes cached state along path
struct SessionSetup {
    src_addr: Ipv6Addr,
    dest_addr: Ipv6Addr,
    src_coords: Vec<NodeId>,   // For return path caching
    dest_coords: Vec<NodeId>,  // For forward path routing
    flags: SessionFlags,
}

struct SessionFlags {
    request_ack: bool,         // Ask destination to confirm
    bidirectional: bool,       // Set up both directions
}

/// Confirms session establishment
struct SessionAck {
    src_addr: Ipv6Addr,
    dest_addr: Ipv6Addr,
    src_coords: Vec<NodeId>,   // Acknowledger's coords (for return caching)
}

/// Minimal data packet
struct DataPacket {
    flags: u8,
    hop_limit: u8,
    payload_length: u16,
    src_addr: Ipv6Addr,        // 16 bytes
    dest_addr: Ipv6Addr,       // 16 bytes
    payload: Vec<u8>,
}

/// Error when router cannot route (cache miss)
struct CoordsRequired {
    dest_addr: Ipv6Addr,
    reporter: NodeId,          // Which router had the miss
}

Data Packet Overhead

Field	Size
flags	1 byte
hop_limit	1 byte
payload_length	2 bytes
src_addr	16 bytes
dest_addr	16 bytes
Total header	36 bytes

Comparable to IPv6 (40 bytes). No coordinates in data packets.

Session Setup Flow

S                       R1                      R2                      D
│                        │                       │                       │
│──SessionSetup─────────>│                       │                       │
│  (src_coords,          │──SessionSetup────────>│                       │
│   dest_coords)         │                       │──SessionSetup────────>│
│                        │                       │                       │
│                        │  cache:               │  cache:               │
│                        │  dest_addr→dest_coords│  dest_addr→dest_coords│
│                        │  src_addr→src_coords  │  src_addr→src_coords  │
│                        │                       │                       │
│<─────────────────────────────────────────────────────────SessionAck───│
│                        │                       │                       │
│══DataPacket═══════════>│══════════════════════>│══════════════════════>│
│  (addresses only)      │  (use cached coords)  │  (use cached coords)  │

Router Behavior

impl Router {
    fn handle_session_setup(&mut self, setup: SessionSetup, from: PeerId) {
        // Cache coordinates for both directions
        self.coord_cache.insert(setup.dest_addr, CacheEntry {
            coords: setup.dest_coords.clone(),
            expires: now() + CACHE_TTL,
        });
        self.coord_cache.insert(setup.src_addr, CacheEntry {
            coords: setup.src_coords.clone(),
            expires: now() + CACHE_TTL,
        });

        // Forward toward destination
        let next = self.greedy_next_hop(&setup.dest_coords);
        self.forward(next, setup);
    }

    fn handle_data_packet(&mut self, packet: DataPacket, from: PeerId) {
        match self.coord_cache.get(&packet.dest_addr) {
            Some(entry) => {
                entry.last_used = now();
                let next = self.greedy_next_hop(&entry.coords);
                self.forward(next, packet);
            }
            None => {
                // Cache miss — request coordinates
                self.send_error(from, CoordsRequired {
                    dest_addr: packet.dest_addr,
                    reporter: self.node_id,
                });
            }
        }
    }
}

Cache Management

struct CoordCache {
    entries: HashMap<Ipv6Addr, CacheEntry>,
    max_entries: usize,
}

struct CacheEntry {
    coords: Vec<NodeId>,
    created: Timestamp,
    last_used: Timestamp,
    expires: Timestamp,
}

Eviction policy: LRU (least recently used) when cache exceeds max_entries.

Expiration: Entries expire after TTL (e.g., 300 seconds). Can be refreshed by:

• Subsequent SessionSetup
• SessionRefresh message (lightweight, just touches expiry)
• Data packet transit (optional: refresh on use)

Handling Cache Eviction

When a router's cache entry is evicted mid-session:

1. Data packet arrives, cache miss
2. Router sends CoordsRequired to packet source
3. Source receives error, re-sends SessionSetup (or packet with coords)
4. Path warms again, data flow resumes

From application perspective: brief latency spike, transparent recovery.

Sender Behavior

impl Sender {
    fn send(&mut self, dest: Ipv6Addr, data: &[u8]) {
        if !self.session_established(dest) {
            // Need to establish session first
            let dest_coords = self.discover_or_cached(dest)?;
            self.send_session_setup(dest, &dest_coords);
            self.await_session_ack(dest)?;
        }

        self.send_data_packet(dest, data);
    }

    fn handle_coords_required(&mut self, err: CoordsRequired) {
        // Path went cold, re-establish
        self.mark_session_cold(err.dest_addr);
        self.send_session_setup(err.dest_addr, &self.cached_coords(err.dest_addr));
    }
}

---

Part 5: Packet Type Summary

Type	Purpose	Size	When Used
FilterAnnounce	Bloom filter propagation	~4.1 KB	Topology changes
LookupRequest	Discover coordinates	~300 bytes	First contact with distant node
LookupResponse	Return coordinates	~400 bytes	Reply to discovery
SessionSetup	Warm router caches	~400-600 bytes	Before data transfer
SessionAck	Confirm session	~300 bytes	Optional confirmation
DataPacket	Application data	36 bytes + payload	Bulk of traffic
CoordsRequired	Request retransmit	~50 bytes	Cache miss recovery

---

Part 6: Traffic Analysis

Steady State (Stable Network)

• Bloom filter traffic: Near zero (event-driven, no changes)
• Discovery traffic: Rare (warm caches)
• Session traffic: Rare (established sessions)
• Data traffic: Minimal overhead (36-byte header)

Network Churn

When nodes join/leave:

• Bloom filter updates propagate (bounded by K-hop scope)
• Affected sessions may need re-establishment
• Discovery queries for newly-joined nodes

Per-Node Resource Requirements

Resource	Full Participant	Leaf-Only
Bloom filter storage	d × 4 KB (d = peer count)	None
Coordinate cache	10K-100K entries	None
Route cache	1K-10K entries	Minimal
Bandwidth (idle)	< 1 KB/sec	Near zero

---

Open Questions

1. Coordinate compression: Can tree coordinates be compressed for smaller SessionSetup messages? (e.g., delta encoding, shorter node ID representation)

2. Multi-path routing: How to handle multiple valid paths? Load balancing? Failover?

3. Asymmetric paths: S→D and D→S may traverse different routers. Is this acceptable or should paths be symmetric?

4. Gateway /64 prefixes: How do subnet prefixes interact with Bloom filters and discovery? One filter entry per gateway regardless of devices behind it?

5. Cache sizing: What's the right cache size for different node roles? Core nodes vs. edge nodes?

6. Mobility: When a node changes tree position (new parent), how quickly do sessions recover? Should nodes announce position changes?

---

References

• fips-design.md — Overall FIPS architecture
• fips-links.md — Link layer requirements
• spanning-tree-dynamics.md — Tree protocol details