laantungir/fips
1
0
Fork 0
mirror of https://github.com/jmcorgan/fips.git synced 2026-07-30 19:46:15 +00:00
Code Issues Packages Projects Releases Wiki Activity

Restructure design docs around protocol layers

Browse Source 
Reorganize FIPS design documentation from implementation-centric
structure (routing, gossip protocol, wire protocol, transports) to
protocol-layer organization with clear service boundaries.

New documents (8):
- fips-transport-layer.md — transport layer spec
- fips-link-layer.md — FLP spec (peer auth, link encryption, forwarding)
- fips-session-layer.md — FSP spec (end-to-end encryption, sessions)
- fips-ipv6-adapter.md — IPv6 adaptation (TUN, DNS, MTU enforcement)
- fips-mesh-operation.md — routing, discovery, error recovery
- fips-wire-formats.md — consolidated wire format reference
- fips-spanning-tree.md — tree algorithm reference
- fips-bloom-filters.md — bloom filter math reference

Rewritten (2):
- fips-intro.md — breadth-first intro with layer model diagrams
- fips-software-architecture.md — slimmed to stable decisions

Updated (3):
- spanning-tree-dynamics.md — removed stale root refresh, aligned terminology
- fips-configuration.md — fixed priority type (u16 → u8)
- fips-state-machines.md — synced code examples with codebase

Deleted (6): fips-transports.md, fips-wire-protocol.md,
fips-gossip-protocol.md, fips-session-protocol.md, fips-routing.md,
fips-tun-driver.md (content absorbed into new structure)
This commit is contained in:
Johnathan Corgan
2026-02-17 04:50:04 +00:00
parent d3477a7c0f
commit d46dc874ef
20 changed files with 3925 additions and 5759 deletions
Download Patch File Download Diff File Expand all files Collapse all files
docs/design/README.md
+51 -35
View File
@@ -1,60 +1,76 @@
# FIPS Design Documents
Protocol design specifications and analysis for the Federated Interoperable Peering System.
Protocol design specifications for the Federated Interoperable Peering
System — a self-organizing encrypted mesh network built on Nostr identities.
## Suggested Reading Order
## Reading Order
Start with the high-level architecture, then work through session flow, routing
concepts, and finally the wire-level protocol details.
Start with the introduction, then follow the protocol stack from bottom to
top. After the stack, the mesh operation document explains how all the
pieces work together. Supporting references provide deeper dives into
specific topics.
### 1. Introduction and Overview
### Protocol Stack
| Document | Description |
|------------------------------------------|-------------------------------------------------------------|
| [fips-intro.md](fips-intro.md) | Protocol introduction: goals, concepts, architecture |
| -------- | ----------- |
| [fips-intro.md](fips-intro.md) | Protocol introduction: goals, architecture, layer model |
| [fips-transport-layer.md](fips-transport-layer.md) | Transport layer: datagram delivery over arbitrary media |
| [fips-link-layer.md](fips-link-layer.md) | FIPS Link Protocol (FLP): peer authentication, link encryption, forwarding |
| [fips-session-layer.md](fips-session-layer.md) | FIPS Session Protocol (FSP): end-to-end encryption, sessions |
| [fips-ipv6-adapter.md](fips-ipv6-adapter.md) | IPv6 adaptation: TUN interface, DNS, MTU enforcement |
### 2. Protocol Flow (How Traffic Works)
### Mesh Behavior
| Document | Description |
|------------------------------------------------------|-----------------------------------------------------------|
| [fips-session-protocol.md](fips-session-protocol.md) | End-to-end session flow, Noise IK encryption, terminology |
| -------- | ----------- |
| [fips-mesh-operation.md](fips-mesh-operation.md) | How the mesh operates: routing, discovery, error recovery |
| [fips-wire-formats.md](fips-wire-formats.md) | Wire format reference for all message types |
### 3. Routing (How Packets Find Their Way)
### Supporting References
| Document | Description |
|--------------------------------------------------------|-------------------------------------------------------------|
| [fips-routing.md](fips-routing.md) | Routing concepts: bloom filters, discovery, greedy routing |
| [spanning-tree-dynamics.md](spanning-tree-dynamics.md) | Tree protocol behavior: convergence, partitions, recovery |
| [fips-gossip-protocol.md](fips-gossip-protocol.md) | Wire formats: TreeAnnounce, FilterAnnounce, Lookup messages |
| -------- | ----------- |
| [fips-spanning-tree.md](fips-spanning-tree.md) | Spanning tree algorithms: root election, parent selection, coordinates |
| [fips-bloom-filters.md](fips-bloom-filters.md) | Bloom filter math: FPR analysis, size classes, split-horizon |
### 4. Link Layer (How Peer Connections Work)
### Implementation
| Document | Description |
|------------------------------------------------|---------------------------------------------------------------|
| [fips-wire-protocol.md](fips-wire-protocol.md) | Transport layer: Noise IK, session indices, roaming, security |
| [fips-transports.md](fips-transports.md) | Transport-specific: UDP, Ethernet, Tor, radio characteristics |
| -------- | ----------- |
| [fips-software-architecture.md](fips-software-architecture.md) | Stable architectural decisions guiding the codebase |
| [fips-state-machines.md](fips-state-machines.md) | Phase-based state machine pattern (Rust enum-of-structs) |
| [fips-configuration.md](fips-configuration.md) | YAML configuration reference |
## Implementation
### Supplemental
| Document | Description |
|------------------------------------------------------------------|------------------------------------------------------------------|
| [fips-software-architecture.md](fips-software-architecture.md) | Software architecture: entities, state machines, configuration |
| [fips-tun-driver.md](fips-tun-driver.md) | TUN interface driver: reader/writer threads, ICMPv6, packet flow |
| [fips-state-machines.md](fips-state-machines.md) | Phase-based state machine pattern: peer lifecycle, transitions |
| -------- | ----------- |
| [spanning-tree-dynamics.md](spanning-tree-dynamics.md) | Spanning tree walkthroughs: convergence scenarios, worked examples |
## Document Cross-References
## Document Relationships
```text
fips-intro.md
│
┌────────────┴────────────┐
▼ ▼
fips-session-protocol.md fips-software-architecture.md
│ │
┌─────────┴─────────┐ ▼
▼ ▼ fips-transports.md
fips-routing.md fips-wire-protocol.md
│ │
▼ ▼
spanning-tree-dynamics.md ←→ fips-gossip-protocol.md
┌──────────────┼──────────────┐
▼ ▼ ▼
fips-transport-layer fips-mesh- fips-software-
│ operation architecture
▼ │ │
fips-link-layer ◄────────┤ ▼
│ │ fips-state-machines
▼ │
fips-session-layer ├──► fips-spanning-tree
│ │ │
▼ └──► fips-bloom-filters
fips-ipv6-adapter
fips-wire-formats
(referenced by all layer docs)
fips-configuration
(standalone reference)
spanning-tree-dynamics
(pedagogical companion to fips-spanning-tree)
```
docs/design/fips-bloom-filters.md
+249
View File
@@ -0,0 +1,249 @@
# FIPS Bloom Filters
This document describes the bloom filter data structures, parameters, and
mathematical properties used by FIPS for reachability-based candidate
selection. It is a supporting reference — for how bloom filters fit into
the overall routing system, see
[fips-mesh-operation.md](fips-mesh-operation.md).
## Purpose
Each node maintains bloom filters summarizing which destinations are
reachable through each of its peers. Bloom filters provide **candidate
selection**: they narrow which peers are worth considering when forwarding
a packet to a given destination. The actual forwarding decision is made by
tree coordinate distance ranking.
Bloom filters answer a single question: "can peer P possibly reach
destination D?" The answer is either "no" (definitive) or "maybe"
(probabilistic — false positives are possible, false negatives are not).
## Filter Parameters
| Parameter | Value | Rationale |
| --------- | ----- | --------- |
| Size | 1 KB (8,192 bits) | Balance between accuracy and bandwidth |
| Hash functions (k) | 5 | Compromise between optimal k=7 for 800 entries and accommodating up to ~1,600 entries |
| Size class | 1 (v1 mandated) | `512 << 1 = 1024` bytes |
### Why k = 5
The optimal number of hash functions depends on the expected occupancy:
- At 800 entries (typical for moderate-degree nodes), optimal k ≈ 7
- At 1,600 entries (hub nodes), optimal k ≈ 4
k = 5 is a practical compromise that provides reasonable false positive
rates across the expected range of occupancy.
## False Positive Rate (FPR) Analysis
The false positive rate for a bloom filter with m bits, k hash functions,
and n entries is approximately:
```text
FPR ≈ (1 - e^(-kn/m))^k
```
With m = 8,192 bits and k = 5:
| Node Degree | Expected Entries | FPR | Impact |
| ----------- | ---------------- | --- | ------ |
| 5 (IoT) | 100–200 | ~0.02% | Negligible |
| 8 (typical) | 250–400 | ~0.3% | Negligible |
| 12 (well-connected) | 500–800 | ~2.4% | Minor — occasional unnecessary discovery |
| 20+ (hub) | 1,200–1,800 | 7.5–15% | Elevated — more false positive candidates |
At moderate network sizes, filters are highly accurate. At larger scales
(~1M nodes), hub nodes with many peers see elevated FPR. False positives
cause unnecessary candidate evaluation (and potentially unnecessary
discovery attempts) but do not affect routing correctness — the tree
distance calculation makes the actual forwarding decision.
### Saturation Behavior
A bloom filter with 8,192 bits and k = 5 saturates (FPR approaches 100%)
around 3,000–4,000 entries. Beyond this, every query returns "maybe" and the
filter provides no candidate selection value. Hub nodes approaching this
threshold effectively fall back to tree-distance-only routing.
## Per-Peer Filter Model
Each node maintains a separate outbound filter for each peer. The filter
for peer Q answers: "which destinations are reachable through me (but not
through Q itself)?"
### Filter Computation
The outbound filter for peer Q is computed by merging:
1. **This node's own identity** (node_addr)
2. **Leaf-only dependents** (if any — future direction)
3. **All other peers' inbound filters except Q's** (split-horizon exclusion)
### Split-Horizon Exclusion
The exclusion of Q's own inbound filter prevents echo loops. Without it,
a node would advertise back to Q the destinations it learned from Q,
creating a routing loop where Q thinks it can reach a destination through
this node, and this node thinks it can reach the same destination through Q.
Split-horizon is computed per-peer: the outbound filter for peer Q merges
all inbound filters except Q's. For a node with peers A, B, C:
| Outbound to | Includes entries from |
| ----------- | -------------------- |
| A | Self + B's filter + C's filter |
| B | Self + A's filter + C's filter |
| C | Self + A's filter + B's filter |
## Filter Propagation
Filters propagate via **FilterAnnounce** messages exchanged between direct
peers. Each FilterAnnounce replaces the previous filter for that peer —
there is no incremental update.
### Update Triggers
Filter updates are event-driven, not periodic:
- Peer connects (new filter includes the new peer's reachability)
- Peer disconnects (filter must exclude the departed peer's entries)
- A peer's inbound filter changes (outbound filters to other peers must
be recomputed)
- Local state changes (new identity, leaf-only dependent changes)
### Rate Limiting
Updates are rate-limited at 500ms minimum interval per peer to prevent
storms during topology changes. Multiple pending changes within the
cooldown period are coalesced into a single announcement.
### Propagation Scope
Filters propagate unboundedly — there is no TTL on filter propagation. At
steady state, every reachable destination appears in at least one peer's
filter. New nodes added to the network propagate through the entire mesh
within O(D × 500ms) where D is the network diameter.
## Filter Expiration
Bloom filters cannot remove individual entries (this is a fundamental
property of the data structure). Entries are expired through:
- **Peer disconnect**: The entire inbound filter for the departed peer is
removed, and outbound filters are recomputed
- **Filter replacement**: Each FilterAnnounce completely replaces the
previous filter for that peer
- **Implicit timeout**: If no FilterAnnounce is received from a peer within
a threshold period, the peer's filter may be considered stale (tied to
link liveness detection)
## Size Classes and Folding
### Size Class Table
| size_class | Bytes | Bits | Status |
| ---------- | ----- | ---- | ------ |
| 0 | 512 | 4,096 | Reserved |
| 1 | 1,024 | 8,192 | **v1 (MUST use)** |
| 2 | 2,048 | 16,384 | Reserved |
| 3 | 4,096 | 32,768 | Reserved |
v1 nodes MUST use size_class = 1 and MUST reject FilterAnnounce messages
with any other size_class.
### Folding (Forward Compatibility)
Larger filters can be **folded** to smaller sizes by OR-ing the two halves
together. A 2 KB filter folds to 1 KB by OR-ing the upper and lower
halves. This preserves the "maybe present" property (no false negatives
introduced) but increases the false positive rate.
Folding enables future protocol versions where hub nodes maintain larger
filters (lower FPR) while constrained nodes fold down to the mandated size.
A node receiving a larger filter folds it locally before use if needed.
The hash function design supports folding: membership tests at a smaller
size use `hash(item, i) % smaller_bit_count`, which maps to the same bit
positions that folding produces.
## Membership Test
To test whether a node_addr is in a bloom filter:
```text
for i in 0..hash_count:
bit_index = hash(node_addr, i) % filter_bits
if not bits[bit_index]:
return false // Definitely not present
return true // Maybe present (possible false positive)
```
Where `filter_bits = 8 × (512 << size_class)` — 8,192 for v1.
## Wire Format
FilterAnnounce messages are carried inside encrypted link-layer frames:
| Offset | Field | Size | Description |
| ------ | ----- | ---- | ----------- |
| 0 | msg_type | 1 byte | 0x20 |
| 1 | sequence | 8 bytes LE | Monotonic counter for freshness |
| 9 | hash_count | 1 byte | Number of hash functions (5 in v1) |
| 10 | size_class | 1 byte | Filter size: `512 << size_class` bytes |
| 11 | filter_bits | 1,024 bytes | Bloom filter bit array (v1) |
**v1 total**: 1,035 bytes payload, 1,064 bytes with link encryption
overhead.
See [fips-wire-formats.md](fips-wire-formats.md) for the complete wire
format reference.
## Scale Considerations
### Small Networks (< 1,000 nodes)
Filters are very accurate (FPR < 1% for typical node degrees). Candidate
selection effectively identifies the correct forwarding peer on the first
try for most destinations.
### Medium Networks (1,000–100,000 nodes)
Filters remain accurate for typical nodes. Hub nodes (20+ peers) may see
elevated FPR but the tree distance ranking correctly selects the best
candidate regardless.
### Large Networks (> 100,000 nodes)
Hub nodes approach filter saturation. The filter still provides value
(some peers can be definitively excluded) but the candidate set grows
larger. The tree distance calculation becomes the primary routing
discriminator.
Future mitigation: hub nodes could use larger filters (size_class 2 or 3)
while constrained nodes fold to size_class 1. This requires protocol
negotiation not present in v1.
## Implementation Status
| Feature | Status |
| ------- | ------ |
| 1 KB bloom filter (size_class 1) | **Implemented** |
| 5 hash functions | **Implemented** |
| Split-horizon filter computation | **Implemented** |
| Per-peer filter maintenance | **Implemented** |
| Event-driven updates | **Implemented** |
| 500ms rate limiting | **Implemented** |
| FilterAnnounce gossip | **Implemented** |
| Size class negotiation | Future direction |
| Folding support | Future direction |
| Adaptive filter sizing | Future direction |
## References
- [fips-mesh-operation.md](fips-mesh-operation.md) — How bloom filters fit
into routing
- [fips-wire-formats.md](fips-wire-formats.md) — FilterAnnounce wire format
- [fips-spanning-tree.md](fips-spanning-tree.md) — The coordinate system
that bloom filter candidates are ranked by
docs/design/fips-configuration.md
+1 -1
View File
@@ -207,7 +207,7 @@ Static peer list. Each entry defines a peer to connect to.
| `peers[].alias` | string | *(none)* | Human-readable name for logging |
| `peers[].addresses[].transport` | string | *(required)* | Transport type (`udp`) |
| `peers[].addresses[].addr` | string | *(required)* | Transport address (e.g., `"10.0.0.2:4000"`) |
| `peers[].addresses[].priority` | u16 | `100` | Address priority (lower = preferred) |
| `peers[].addresses[].priority` | u8 | `100` | Address priority (lower = preferred) |
| `peers[].connect_policy` | string | `"auto_connect"` | Connection policy: `auto_connect`, `on_demand`, or `manual` |
## Minimal Example
docs/design/fips-gossip-protocol.md
-794
View File
@@ -1,794 +0,0 @@
# FIPS Gossip Protocol
This document specifies the wire formats and exchange rules for FIPS gossip
messages: TreeAnnounce, FilterAnnounce, and the discovery protocol
(LookupRequest/LookupResponse).
For conceptual background on how these protocols work:
- Spanning tree dynamics: [spanning-tree-dynamics.md](spanning-tree-dynamics.md)
- Routing design and bloom filter concepts: [fips-routing.md](fips-routing.md)
---
## 1. Message Type Summary
All gossip messages are link-layer messages, encrypted with per-peer Noise IK
session keys. They travel one hop (peer-to-peer), though their effects may
propagate further through subsequent gossip.
| Type | Purpose | Direction | Trigger |
|------|---------|-----------|---------|
| TreeAnnounce | Spanning tree state | Bidirectional | Peer connect, parent change, periodic |
| FilterAnnounce | Bloom filter reachability | Bidirectional | Peer connect, filter change |
| LookupRequest | Coordinate discovery | Flooded | Route cache miss |
| LookupResponse | Return coordinates | Routed back | LookupRequest reaches target |
---
## 2. TreeAnnounce
TreeAnnounce messages propagate spanning tree state between peers. Each node
announces its parent selection and ancestry, enabling peers to compute tree
coordinates and distances.
### 2.1 Wire Format
```text
TreeAnnounce (v1) {
version: u8, // Protocol version (0x01 for v1)
sequence: u64, // Monotonic, increments on parent change
timestamp: u64, // Unix timestamp (seconds)
parent: NodeAddr, // 16 bytes, truncated SHA-256(pubkey) of selected parent
ancestry_count: u16, // Number of ancestry entries
ancestry: [AncestryEntry], // Path from self to root
signature: Signature, // 64 bytes, outer signature over entire message
}
AncestryEntry (v1) {
node_addr: NodeAddr, // 16 bytes
sequence: u64, // That node's sequence number
timestamp: u64, // That node's timestamp
}
```
Note: v1 ancestry entries are 32 bytes each (no per-entry signature). See §2.7 Trust Model.
### 2.2 Field Semantics
**version**: Protocol version number. v1 = 0x01. Receivers MUST reject messages
with unrecognized version numbers to ensure forward compatibility.
**sequence**: Incremented each time the node changes its parent declaration.
Higher sequence numbers supersede lower ones for conflict resolution.
**timestamp**: Used for distributed consistency. A declaration is considered
stale if `now - timestamp > ROOT_TIMEOUT` (default 60 minutes for root).
**parent**: The node_addr of the selected parent. If `parent == self.node_addr`,
the node is declaring itself as root.
**ancestry**: The chain from this node up to the root. The first entry is this
node's own declaration, followed by parent, grandparent, etc. In v1, entries
carry only routing metadata (node_addr, sequence, timestamp) without per-entry
signatures. See §2.7 for the trust model.
### 2.3 Size Estimate
| Component | Size |
|-----------|------|
| version | 1 byte |
| sequence | 8 bytes |
| timestamp | 8 bytes |
| parent | 16 bytes |
| ancestry_count | 2 bytes |
| signature | 64 bytes |
| Per ancestry entry (v1) | 16 + 8 + 8 = 32 bytes |
For tree depth D (ancestry_count = D + 1): `100 + (D + 1) × 32` bytes payload.
| Tree Depth | Payload Size | With Link Overhead |
|------------|--------------|--------------------|
| 0 (root) | 132 bytes | 161 bytes |
| 3 | 228 bytes | 257 bytes |
| 5 | 292 bytes | 321 bytes |
| 10 | 452 bytes | 481 bytes |
Note: v1 ancestry entries omit per-entry signatures (32 bytes vs 96 bytes in
the original design). See §2.7 for the rationale.
### 2.4 Exchange Rules
**On peer connection:**
1. After Noise IK handshake completes, both peers send TreeAnnounce
2. Each peer processes the received announcement (see §2.5)
3. If processing triggers a parent change, send updated TreeAnnounce to all peers
**On parent change:**
1. Increment sequence number
2. Update timestamp
3. Sign new declaration
4. Send TreeAnnounce to all peers
**Periodic refresh:**
1. Root refreshes every 30 minutes (prevents stale root detection)
2. Non-root nodes forward root refresh when received
3. Nodes may refresh their own declaration periodically (implementation choice)
### 2.5 Processing Rules
When receiving TreeAnnounce from peer P:
```text
1. Decode message; reject if version != 0x01
2. Verify P's declaration signature using P's known public key (from Noise IK)
3. Verify that declaration node_addr matches sender's identity
4. Check sequence freshness:
- If sequence <= stored sequence for P: discard (stale)
5. Update peer P's tree state (declaration + ancestry)
6. Re-evaluate parent selection:
- Find smallest root visible across all peers
- Among peers reaching smallest root, prefer shallowest depth
- Apply stability threshold to prevent flapping (depth improvement ≥ 1)
- If parent changed: increment own sequence, sign, recompute coords, announce to all
```
Note: In v1, only the sender's declaration signature is verified (step 2).
Ancestry entries beyond the direct peer are accepted on trust. See §2.7.
### 2.6 Rate Limiting
To prevent announcement storms during reconvergence:
- Minimum interval between announcements to same peer: 500ms
- If change occurs during cooldown: mark pending, send after cooldown
- Coalesce multiple pending changes into single announcement
### 2.7 Trust Model (v1)
**v1 uses transitive trust**: each node verifies only its direct peer's
declaration signature. The peer's public key is known from the Noise IK
handshake, so verification is straightforward. Ancestry entries from nodes
beyond the direct peer are accepted on trust from the authenticated sender.
**Why transitive trust?** NodeAddr values are truncated SHA-256 hashes of
public keys — this mapping is intentionally one-way. To verify an ancestry
entry's signature, a node would need the entry's public key, but FIPS does not
distribute node_addr→pubkey mappings by design. Exposing these mappings would
enable traffic analysis, undermining a core privacy property.
**Limitation**: An adversarial interior node could fabricate ancestry chains,
potentially attracting traffic to itself (sinkhole attack) or manipulating tree
topology. This risk is mitigated by:
- **Authenticated peers have reputation cost**: Misbehaving nodes can be
disconnected and blocked by their direct peers.
- **Multi-path observation**: Nodes receiving conflicting tree state from
multiple peers can detect inconsistencies (future enhancement).
**Versioning**: The wire format includes a version byte (v1 = 0x01) to enable
future protocol evolution. A future version could introduce stronger ancestry
verification (e.g., zero-knowledge proofs of key ownership) without breaking
backward compatibility. Nodes MUST reject TreeAnnounce messages with
unrecognized version numbers.
---
## 3. FilterAnnounce
FilterAnnounce messages propagate Bloom filter reachability information. Each
node's filter indicates which destinations are reachable through it.
### 3.1 Wire Format
```text
FilterAnnounce {
sequence: u64, // For freshness/deduplication
filter: BloomFilter, // Variable size based on size_class
}
BloomFilter {
hash_count: u8, // Number of hash functions (5 for v1)
size_class: u8, // Filter size: bytes = 512 << size_class
bits: [u8; 512 << size_class], // Bit array (1024 bytes for v1)
}
```
### 3.2 Size Classes
Filter sizes are powers of 2 to enable folding (shrinking by ORing halves):
| size_class | Bits | Bytes | Status |
|------------|--------|-------|---------------------|
| 0 | 4,096 | 512 | Reserved (future) |
| 1 | 8,192 | 1,024 | **v1 default** |
| 2 | 16,384 | 2,048 | Reserved (future) |
| 3 | 32,768 | 4,096 | Reserved (future) |
**v1 protocol**: All nodes MUST use size_class=1 (1 KB filters). Nodes MUST
reject FilterAnnounce with size_class ≠ 1.
**Future versions**: Nodes may negotiate larger filters via capability exchange.
Receivers can fold larger filters down to their preferred size.
### 3.3 Field Semantics
**sequence**: Monotonic counter for this node's filter. Allows receivers to
detect stale or duplicate announcements.
**hash_count**: Number of hash functions used. v1 uses k=5, which is optimal
for 800-1,600 entries in a 1 KB filter.
**size_class**: Indicates filter size as `512 << size_class` bytes. Allows
forward-compatible extension to larger filters.
**bits**: The Bloom filter bit array. To test membership:
```text
for i in 0..hash_count:
bit_index = hash(node_addr, i) % (8 * bits.len())
if !bits[bit_index]: return false
return true // "maybe present"
```
### 3.3 Filter Contents
A node's outgoing filter to peer Q contains:
1. This node's own node_addr
2. Node_ids of leaf-only dependents (nodes using this node as sole peer)
3. Entries merged from filters received from all other peers (not Q)
This split-horizon merge (excluding the destination peer's own filter from
the computation) prevents a node's entries from being echoed back to it,
providing loop prevention. Filters propagate transitively through the
network without any hop limit.
### 3.4 Exchange Rules
**On peer connection:**
1. After TreeAnnounce exchange, send FilterAnnounce
2. Filter contains current reachability view
**On filter change:**
Triggering events:
- Peer connects or disconnects
- Received filter changes outgoing filter
- Local state change (new leaf dependent, become gateway)
Rate limiting:
- Minimum interval between filter announcements: 500ms
- Debounce rapid changes into single announcement
**Processing received filter:**
```text
1. Store: peer_filters[P] = received.filter
2. Recompute outgoing filters for all other peers:
- For each peer Q (Q != P):
outgoing[Q] = merge(self_filter, peer_filters[all peers except Q])
- If outgoing[Q] changed, send FilterAnnounce to Q
```
### 3.5 Filter Expiration
Bloom filters cannot remove individual entries. Expiration handled by:
- **Peer disconnect**: Remove that peer's filter entirely, recompute
- **Filter replacement**: Each FilterAnnounce replaces the previous one
- **Implicit timeout**: If no updates from peer within threshold, consider stale
---
## 4. LookupRequest
LookupRequest initiates coordinate discovery for destinations not covered by
local Bloom filters.
### 4.1 Wire Format
```text
LookupRequest {
request_id: u64, // Unique identifier for this request
target: NodeAddr, // 16 bytes, who we're looking for
origin: NodeAddr, // 16 bytes, who's asking
origin_coords: Vec<NodeAddr>, // Origin's ancestry (for return path)
ttl: u8, // Remaining propagation hops
visited: CompactBloomFilter,// ~256 bytes, prevents loops
}
CompactBloomFilter {
bits: [u8; 256], // Smaller filter for visited set
hash_count: u8,
}
```
### 4.2 Field Semantics
**request_id**: Randomly generated, used to match responses and detect
duplicates.
**target**: The node_addr being searched for.
**origin**: The node_addr of the original requester. Used for response routing.
**origin_coords**: The requester's current tree coordinates. Used by the target
for the first hop of response routing (via `find_next_hop`).
**ttl**: Propagation limit. Prevents unbounded flooding.
**visited**: Compact Bloom filter tracking nodes that have seen this request.
Prevents loops (revisiting a node on the same path). Note: does NOT prevent
convergent duplicates arriving via different paths — `request_id` dedup
(section 4.4) is required for that.
### 4.3 Propagation Rules
When receiving LookupRequest:
```text
1. Check request_id against recent-request cache - if present, drop (duplicate
via convergent path). This is REQUIRED, not optional — the visited filter
alone does not prevent duplicates arriving via different paths.
2. Add request_id to recent-request cache
3. Check visited filter - if self likely present, drop (already processed)
4. Add self to visited filter
5. Decrement TTL
6. Check if target is local:
- If target == self.node_addr: generate LookupResponse
- If target in local peer_filters: may respond on behalf (optional)
7. If TTL > 0 and not found locally:
- Forward to peers not in visited filter
- Optionally prioritize peers whose filter indicates target "maybe" present
```
The recent-request cache need only retain entries for a few seconds (long
enough for the flood to complete across the TTL scope) and is bounded by the
rate limit on incoming requests.
### 4.4 Rate Limiting
- Limit requests per origin per time window
- Limit total outstanding requests
---
## 5. LookupResponse
LookupResponse returns the target's coordinates to the requester.
### 5.1 Wire Format
```text
LookupResponse {
request_id: u64, // Echoes LookupRequest.request_id
target: NodeAddr, // 16 bytes, confirms who was found
target_coords: Vec<NodeAddr>, // Target's ancestry (the key payload)
proof: Signature, // 64 bytes, target signs to prove existence
}
```
### 5.2 Field Semantics
**request_id**: Matches the original request, allowing requester to correlate.
**target**: Confirms the identity found.
**target_coords**: The target's current tree coordinates. This is the primary
payload — cached by the originator to enable routing to the target.
**proof**: Target's signature over `(request_id || target)`. Prevents
malicious nodes from claiming reachability and blackholing traffic.
Coordinates are excluded from the proof to avoid invalidation during
tree reconvergence (see [fips-routing.md](fips-routing.md) §2.4).
### 5.3 Routing
LookupResponse uses a two-phase routing mechanism:
```text
1. Response created at target (or node with target in filter)
2. First hop: target routes toward origin via find_next_hop
(standard bloom filter → tree routing path)
3. Subsequent hops: reverse-path forwarding via recent_requests
(each transit node recorded which peer sent the request)
4. Origin receives response, caches target_coords in RouteCache
```
The first hop from the target uses `find_next_hop(origin)` because
the target was not a transit node for the request (it was the
destination). All transit nodes that forwarded the request stored a
`(request_id → from_peer)` entry in `recent_requests`, enabling
reverse-path forwarding for the response.
### 5.4 Security
The proof signature is critical:
- Without it, any node could claim to be (or know) any target
- Requester verifies signature against target's known public key
- Invalid signatures cause response to be dropped
---
## 6. Message Type Codes
Within the link-layer message framing:
| Type Code | Message |
|-----------|---------|
| 0x10 | TreeAnnounce |
| 0x20 | FilterAnnounce |
| 0x30 | LookupRequest |
| 0x31 | LookupResponse |
These are carried inside the encrypted link-layer payload after Noise IK
handshake completion. See [fips-wire-protocol.md](fips-wire-protocol.md) §2.6
for the full link message type table.
---
## 7. Timing Parameters
| Parameter | Default | Notes |
|-----------|---------|-------|
| ROOT_REFRESH_INTERVAL | 30 min | Root regenerates timestamp |
| ROOT_TIMEOUT | 60 min | Root declaration considered stale |
| TREE_ENTRY_TTL | 5-10 min | Individual entry expiration |
| ANNOUNCE_MIN_INTERVAL | 500 ms | Rate limit for announcements |
| LOOKUP_TTL | 64 | Discovery request propagation limit |
| LOOKUP_TIMEOUT | 10 sec | Time to wait for response |
---
## 8. Encoding
All multi-byte integers are little-endian. NodeAddr is 16 bytes (truncated SHA-256 hash).
Signatures are 64 bytes (secp256k1 Schnorr).
Variable-length fields (ancestry, coordinates) are prefixed with a 2-byte
length count indicating number of entries.
```text
Vec<T> encoding:
count: u16 (little-endian)
items: T[count]
```
---
## Appendix A: Detailed Packet Layouts
All gossip messages are link-layer messages carried inside encrypted frames
(discriminator 0x00). The layouts below show the plaintext structure after
link-layer decryption.
### A.1 TreeAnnounce (0x10)
Propagates spanning tree state between directly connected peers.
```text
┌─────────────────────────────────────────────────────────────────────────────┐
│ FULL PACKET (Link Layer + TreeAnnounce) │
├─────────────────────────────────────────────────────────────────────────────┤
│ │
│ ┌───────────────────────────────────────────────────────────────────────┐ │
│ │ LINK LAYER FRAME (encrypted) │ │
│ ├───────────┬──────────────┬────────────┬───────────────────────────────┤ │
│ │ 0x00 │ receiver_idx │ counter │ ciphertext + tag │ │
│ │ 1 byte │ 4 bytes LE │ 8 bytes LE │ N + 16 bytes │ │
│ └───────────┴──────────────┴────────────┴───────────────────────────────┘ │
│ │ │
│ ┌───────────────────────┘ │
│ │ Decrypt │
│ ▼ │
│ ┌───────────────────────────────────────────────────────────────────────┐ │
│ │ TREE ANNOUNCE v1 (plaintext) │ │
│ ├────────┬──────────────────┬───────────┬───────────────────────────────┤ │
│ │ Offset │ Field │ Size │ Description │ │
│ ├────────┼──────────────────┼───────────┼───────────────────────────────┤ │
│ │ 0 │ msg_type │ 1 byte │ 0x10 │ │
│ │ 1 │ version │ 1 byte │ 0x01 (v1) │ │
│ │ 2 │ sequence │ 8 bytes │ u64 LE, monotonic counter │ │
│ │ 10 │ timestamp │ 8 bytes │ u64 LE, Unix seconds │ │
│ │ 18 │ parent │ 16 bytes │ NodeAddr of selected parent │ │
│ │ 34 │ ancestry_count │ 2 bytes │ u16 LE, number of entries │ │
│ │ 36 │ ancestry[0..n] │ 32 × n │ AncestryEntry array │ │
│ │ ... │ signature │ 64 bytes │ Schnorr sig over all above │ │
│ └────────┴──────────────────┴───────────┴───────────────────────────────┘ │
│ │
│ ┌───────────────────────────────────────────────────────────────────────┐ │
│ │ ANCESTRY ENTRY v1 (32 bytes each) │ │
│ ├────────┬──────────────────┬───────────┬───────────────────────────────┤ │
│ │ Offset │ Field │ Size │ Description │ │
│ ├────────┼──────────────────┼───────────┼───────────────────────────────┤ │
│ │ 0 │ node_addr │ 16 bytes │ Truncated SHA-256(pubkey) │ │
│ │ 16 │ sequence │ 8 bytes │ u64 LE, node's seq number │ │
│ │ 24 │ timestamp │ 8 bytes │ u64 LE, node's timestamp │ │
│ └────────┴──────────────────┴───────────┴───────────────────────────────┘ │
│ │
│ Note: v1 entries omit per-entry signatures. Only the sender's outer │
│ signature is verified (transitive trust model, see §2.7). │
│ │
└─────────────────────────────────────────────────────────────────────────────┘
```
**Size calculation**: `1 + 1 + 8 + 8 + 16 + 2 + (depth × 32) + 64 = 100 + (depth × 32)` bytes
| Tree Depth | Payload Size | With Link Overhead |
|------------|--------------|--------------------|
| 0 (root) | 132 bytes | 161 bytes |
| 3 | 228 bytes | 257 bytes |
| 5 | 292 bytes | 321 bytes |
| 10 | 452 bytes | 481 bytes |
**Concrete example** (node D at depth 3, ancestry = [D, P1, P2, Root]):
```text
PLAINTEXT BYTES (hex layout):
10 ← msg_type = TreeAnnounce
01 ← version = 1
05 00 00 00 00 00 00 00 ← sequence = 5
C3 B2 A1 67 00 00 00 00 ← timestamp (Unix seconds)
[16 bytes P1's node_addr] ← parent
04 00 ← ancestry_count = 4
ANCESTRY[0] - Self (D):
[16 bytes D's node_addr]
05 00 00 00 00 00 00 00 ← D's sequence
C3 B2 A1 67 00 00 00 00 ← D's timestamp
ANCESTRY[1] - Parent (P1):
[16 bytes P1's node_addr]
0A 00 00 00 00 00 00 00 ← P1's sequence
00 B0 A1 67 00 00 00 00 ← P1's timestamp
ANCESTRY[2] - Grandparent (P2):
[16 bytes P2's node_addr]
03 00 00 00 00 00 00 00 ← P2's sequence
00 A0 A1 67 00 00 00 00 ← P2's timestamp
ANCESTRY[3] - Root:
[16 bytes Root's node_addr]
01 00 00 00 00 00 00 00 ← Root's sequence
00 90 A1 67 00 00 00 00 ← Root's timestamp
[64 bytes D's outer signature] ← signs entire message
Total payload: 1 + 1 + 8 + 8 + 16 + 2 + (4 × 32) + 64 = 228 bytes
```
### A.2 FilterAnnounce (0x20)
Propagates Bloom filter reachability information.
```text
┌─────────────────────────────────────────────────────────────────────────────┐
│ FILTER ANNOUNCE (0x20) │
├─────────────────────────────────────────────────────────────────────────────┤
│ │
│ ┌───────────────────────────────────────────────────────────────────────┐ │
│ │ WIRE FORMAT │ │
│ ├────────┬──────────────────┬───────────┬───────────────────────────────┤ │
│ │ Offset │ Field │ Size │ Description │ │
│ ├────────┼──────────────────┼───────────┼───────────────────────────────┤ │
│ │ 0 │ msg_type │ 1 byte │ 0x20 │ │
│ │ 1 │ sequence │ 8 bytes │ u64 LE, monotonic counter │ │
│ │ 9 │ hash_count │ 1 byte │ Number of hash functions (5) │ │
│ │ 10 │ size_class │ 1 byte │ Filter size: 512 << class │ │
│ │ 11 │ filter_bits │ variable │ 512 << size_class bytes │ │
│ └────────┴──────────────────┴───────────┴───────────────────────────────┘ │
│ │
│ Size classes (powers of 2 for foldability): │
│ 0 = 512 bytes (4,096 bits) - Reserved for future │
│ 1 = 1,024 bytes (8,192 bits) - v1 default │
│ 2 = 2,048 bytes (16,384 bits) - Reserved for future │
│ 3 = 4,096 bytes (32,768 bits) - Reserved for future │
│ │
│ v1 total payload: 1 + 8 + 1 + 1 + 1024 = 1035 bytes │
│ With link overhead: 1064 bytes │
│ │
├─────────────────────────────────────────────────────────────────────────────┤
│ BLOOM FILTER STRUCTURE │
├─────────────────────────────────────────────────────────────────────────────┤
│ │
│ filter_bits[1024] (v1, size_class=1): │
│ ┌─────────────────────────────────────────────────────────────────────┐ │
│ │ Byte 0 │ Byte 1 │ ... │ Byte 1023 │ │
│ │ bits 0-7 │ bits 8-15 │ │ bits 8184-8191 │ │
│ └─────────────────────────────────────────────────────────────────────┘ │
│ │
│ To test membership of node_addr: │
│ filter_bits = 8 * (512 << size_class) // 8192 for v1 │
│ for i in 0..hash_count: │
│ bit_index = hash(node_addr, i) % filter_bits │
│ if !bits[bit_index]: return false │
│ return true // "maybe present" │
│ │
│ Folding (for future heterogeneous sizes): │
│ To shrink a filter by half, OR its two halves: │
│ small[i] = large[i] | large[i + small.len()] │
│ This increases FPR but preserves correctness (no false negatives). │
│ │
└─────────────────────────────────────────────────────────────────────────────┘
```
**Concrete example** (v1 with size_class=1):
```text
PLAINTEXT BYTES:
20 ← msg_type = FilterAnnounce
2A 00 00 00 00 00 00 00 ← sequence = 42
05 ← hash_count = 5
01 ← size_class = 1 (1 KB filter)
[1024 bytes of filter bits] ← Bloom filter
Total: 1035 bytes
```
### A.3 LookupRequest (0x30)
Discovers tree coordinates for distant destinations.
```text
┌─────────────────────────────────────────────────────────────────────────────┐
│ LOOKUP REQUEST (0x30) │
├─────────────────────────────────────────────────────────────────────────────┤
│ │
│ ┌───────────────────────────────────────────────────────────────────────┐ │
│ │ WIRE FORMAT │ │
│ ├────────┬──────────────────┬───────────┬───────────────────────────────┤ │
│ │ Offset │ Field │ Size │ Description │ │
│ ├────────┼──────────────────┼───────────┼───────────────────────────────┤ │
│ │ 0 │ msg_type │ 1 byte │ 0x30 │ │
│ │ 1 │ request_id │ 8 bytes │ u64 LE, unique identifier │ │
│ │ 9 │ target │ 16 bytes │ NodeAddr being searched for │ │
│ │ 25 │ origin │ 16 bytes │ NodeAddr of requester │ │
│ │ 41 │ ttl │ 1 byte │ Remaining propagation hops │ │
│ │ 42 │ origin_coords_cnt│ 2 bytes │ u16 LE │ │
│ │ 44 │ origin_coords │ 16 × n │ Requester's ancestry │ │
│ │ ... │ visited_hash_cnt │ 1 byte │ Hash functions for visited │ │
│ │ ... │ visited_bits │ 256 bytes │ Compact bloom of visited nodes│ │
│ └────────┴──────────────────┴───────────┴───────────────────────────────┘ │
│ │
└─────────────────────────────────────────────────────────────────────────────┘
```
**Size calculation**: `1 + 8 + 16 + 16 + 1 + 2 + (depth × 16) + 1 + 256` bytes
| Origin Depth | Payload Size |
|--------------|--------------|
| 3 | 349 bytes |
| 5 | 381 bytes |
| 10 | 461 bytes |
**Concrete example** (origin at depth 4):
```text
PLAINTEXT BYTES:
30 ← msg_type = LookupRequest
[8 bytes request_id] ← random unique ID
[16 bytes target node_addr] ← who we're looking for
[16 bytes origin node_addr] ← who's asking
40 ← ttl = 64
04 00 ← origin_coords_count = 4
[16 bytes] × 4 ← origin's ancestry (64 bytes)
05 ← visited hash_count = 5
[256 bytes visited bloom] ← nodes that have seen this request
Total: 1 + 8 + 16 + 16 + 1 + 2 + 64 + 1 + 256 = 365 bytes
```
### A.4 LookupResponse (0x31)
Returns target's coordinates to the requester.
```text
┌─────────────────────────────────────────────────────────────────────────────┐
│ LOOKUP RESPONSE (0x31) │
├─────────────────────────────────────────────────────────────────────────────┤
│ │
│ ┌───────────────────────────────────────────────────────────────────────┐ │
│ │ WIRE FORMAT │ │
│ ├────────┬──────────────────┬───────────┬───────────────────────────────┤ │
│ │ Offset │ Field │ Size │ Description │ │
│ ├────────┼──────────────────┼───────────┼───────────────────────────────┤ │
│ │ 0 │ msg_type │ 1 byte │ 0x31 │ │
│ │ 1 │ request_id │ 8 bytes │ u64 LE, echoes request │ │
│ │ 9 │ target │ 16 bytes │ NodeAddr that was found │ │
│ │ 25 │ target_coords_cnt│ 2 bytes │ u16 LE │ │
│ │ 27 │ target_coords │ 16 × n │ Target's ancestry to root │ │
│ │ ... │ proof │ 64 bytes │ Target's signature │ │
│ └────────┴──────────────────┴───────────┴───────────────────────────────┘ │
│ │
│ Proof signature covers: (request_id || target) │
│ Coords excluded to survive tree reconvergence during lookup RTT. │
│ │
└─────────────────────────────────────────────────────────────────────────────┘
```
**Size calculation**: `1 + 8 + 16 + 2 + (depth × 16) + 64` bytes
| Target Depth | Payload Size |
|--------------|--------------|
| 3 | 139 bytes |
| 5 | 171 bytes |
| 10 | 251 bytes |
**Concrete example** (target at depth 5):
```text
PLAINTEXT BYTES:
31 ← msg_type = LookupResponse
[8 bytes request_id] ← echoed from request
[16 bytes target node_addr] ← confirms who was found
05 00 ← target_coords_count = 5
[16 bytes] × 5 ← target's ancestry (80 bytes)
[64 bytes proof signature] ← target signs to prove existence
Total: 1 + 8 + 16 + 2 + 80 + 64 = 171 bytes
```
### A.5 Message Flow Example
Complete lookup flow showing packet nesting:
```text
Source S wants to reach distant destination D (not in local filters)
1. S creates LookupRequest, sends to peer P1:
UDP DATAGRAM
┌──────────────────────────────────────────────────────────────┐
│ LINK FRAME (S→P1 encrypted) │
│ ┌──────┬────────────┬─────────┬─────────────────────────────┐│
│ │ 0x00 │ P1_recv_idx│ counter │ ciphertext + tag ││
│ └──────┴────────────┴─────────┴─────────────────────────────┘│
│ │ │
│ ┌───────────┘ │
│ ▼ │
│ ┌──────┬───────────────────────────────────┐ │
│ │ 0x30 │ LookupRequest payload │ │
│ │ │ (target=D, origin=S, ttl=64, ...) │ │
│ └──────┴───────────────────────────────────┘ │
└──────────────────────────────────────────────────────────────┘
2. Request propagates through network, reaches D
3. D creates LookupResponse, routes back via find_next_hop + reverse-path:
UDP DATAGRAM
┌──────────────────────────────────────────────────────────────┐
│ LINK FRAME (D→Pn encrypted) │
│ ┌──────┬────────────┬─────────┬─────────────────────────────┐│
│ │ 0x00 │ Pn_recv_idx│ counter │ ciphertext + tag ││
│ └──────┴────────────┴─────────┴─────────────────────────────┘│
│ │ │
│ ┌───────────┘ │
│ ▼ │
│ ┌──────┬───────────────────────────────────┐ │
│ │ 0x31 │ LookupResponse payload │ │
│ │ │ (target=D, coords=[D,P1,P2,Root]) │ │
│ └──────┴───────────────────────────────────┘ │
└──────────────────────────────────────────────────────────────┘
4. S receives response, caches D's coordinates, can now route directly
```
---
## References
- [spanning-tree-dynamics.md](spanning-tree-dynamics.md) - Tree protocol behavior
- [fips-routing.md](fips-routing.md) - Routing concepts and algorithms
- [fips-wire-protocol.md](fips-wire-protocol.md) - Link-layer framing
- [fips-session-protocol.md](fips-session-protocol.md) - End-to-end sessions
docs/design/fips-intro.md
+359 -337
View File
@@ -4,8 +4,8 @@
FIPS is a self-organizing mesh network that can operate over any transport
medium — radio, serial links, Tor, local networks, or the existing internet as
an overlay. The long-term goal is infrastructure that can function alongside
or ultimately replace dependence on the Internet.
an overlay. The long-term goal is infrastructure that can function alongside or
ultimately replace dependence on the Internet.
Nodes in the mesh route traffic for each other using Nostr identities (npubs)
as network addresses. Applications can access the mesh through a native FIPS
@@ -38,80 +38,185 @@ and security credentials without coordination with any central authority. The
identity system uses Nostr keypairs (secp256k1), so existing npub/nsec pairs
work directly.
## How It Works (Overview)
## A Self-Organizing Mesh
Each FIPS node selects which transports to use (Ethernet, radio links, internet
overlay, etc.) and which peer nodes to connect to. From these local peering
decisions, the distributed spanning tree and bloom filter propagation algorithms
self-organize reachability and path information throughout the mesh.
Traditional networks are built top-down. A central authority assigns addresses,
configures routing tables, provisions hardware, and manages the topology. If the
authority disappears or the infrastructure fails, the network fails with it.
Nodes cannot reach each other without infrastructure mediating the connection.
Nodes form a spanning tree rooted at a deterministically-elected node. Each
node knows its path to the root, enabling shortest-path routing to be
calculated between any two nodes (not necessarily through the root). Bloom filters propagate reachability information so nodes can find
each other. Traffic flows via greedy routing toward destinations, encrypted
end-to-end.
FIPS inverts this model. There is no central authority, no address assignment
service, no routing table pushed from above. Each node generates its own
identity from a cryptographic keypair. Each node independently decides which
peers to connect to and which transports to use. From these local decisions
alone, the network self-organizes:
Nodes with multiple transports automatically bridge between networks—the same
routing logic works regardless of the underlying transport mix.
- A **spanning tree** forms through distributed parent selection, giving every
node a coordinate in the network without any node knowing the full topology
- **Bloom filters** propagate through gossip, so each node learns which peers
can reach which destinations — again without global knowledge
- **Routing decisions** are made locally at each hop, using only the node's
immediate peers and cached coordinate information
The result is a network that builds itself from the bottom up, heals around
failures automatically, and scales without central coordination. Adding a node
is as simple as connecting to one existing peer — the network integrates the
new node through its normal gossip protocols.
## Design Goals
1. **Nostr-native identity** - Use Nostr keypairs as node identities
2. **Transport agnostic** - Support IP, wireless, serial, onion, and other link types
3. **Self-organizing** - Automatic topology discovery and route optimization
4. **Privacy preserving** - Minimize metadata leakage across untrusted links
5. **Resilient** - Self-healing with graceful degradation
6. **Reuse Nostr primitives** - Leverage secp256k1, Schnorr signatures, and SHA-256
1. **Nostr-native identity** — Use Nostr keypairs as node identities
2. **Transport agnostic** — Support IP, wireless, serial, onion, and other
link types
3. **Self-organizing** — Automatic topology discovery and route optimization
4. **Privacy preserving** — Minimize metadata leakage across untrusted links
5. **Resilient** — Self-healing with graceful degradation
6. **Reuse Nostr primitives** — Leverage secp256k1, Schnorr signatures, and
SHA-256
---
## Protocol Architecture
FIPS is organized in three protocol layers, each with distinct responsibilities
and clean service boundaries. Understanding these layers is key to understanding
how FIPS works.
```text
┌─────────────────────────────────────────────────────────────┐
│ Applications │
│ (native FIPS API / IPv6 adapter) │
├─────────────────────────────────────────────────────────────┤
│ │
│ FIPS Session Protocol (FSP) │
│ End-to-end authenticated encryption between endpoints │
│ Session lifecycle, coordinate caching, replay protection │
│ │
├─────────────────────────────────────────────────────────────┤
│ │
│ FIPS Link Protocol (FLP) │
│ Hop-by-hop link encryption, peer authentication │
│ Spanning tree, bloom filters, routing, forwarding │
│ │
├─────────────────────────────────────────────────────────────┤
│ │
│ Transport Layer │
│ Datagram delivery over arbitrary media │
│ UDP, Ethernet, LoRa, Tor, serial, ... │
│ │
└─────────────────────────────────────────────────────────────┘
```
### Mapping to Traditional Networking
Readers familiar with the OSI model or TCP/IP networking may find it helpful to
see how FIPS concepts relate to traditional layers:
```text
Traditional FIPS Key Difference
─────────────────────────────────────────────────────────────────────
Application Applications Same role — user-facing
(IPv6 adapter) software
─────────────────────────────────────────────────────────────────────
Transport (TCP/UDP) (not present) FIPS provides datagrams,
not reliable streams
─────────────────────────────────────────────────────────────────────
Session FSP End-to-end encryption
and session management
─────────────────────────────────────────────────────────────────────
Network (IP) FLP Routing, forwarding,
address resolution —
but self-organizing
─────────────────────────────────────────────────────────────────────
Link (Ethernet/WiFi) FLP (link encryption) Peer authentication
and hop-by-hop crypto
─────────────────────────────────────────────────────────────────────
Physical (PHY) Transport layer Abstracted — FIPS
(UDP, radio, serial) treats all media the
same way
─────────────────────────────────────────────────────────────────────
```
Note that FLP spans what would traditionally be separate link and network
layers. This is intentional — in a self-organizing mesh, the same layer that
authenticates peers also makes routing decisions, because routing depends on
authenticated peer state (spanning tree positions, bloom filters).
### Layer Responsibilities
**Transport layer**: Delivers datagrams between endpoints over a specific
medium. Each transport type (UDP socket, Ethernet interface, radio modem)
implements the same abstract interface: send and receive datagrams, report MTU.
The transport layer knows nothing about FIPS identities, routing, or encryption.
It provides raw datagram delivery to FLP above.
See [fips-transport-layer.md](fips-transport-layer.md) for the transport layer
specification.
**FIPS Link Protocol (FLP)**: Manages peer connections, authenticates peers via
Noise IK handshakes, and encrypts all traffic on each link. FLP is where the
mesh organizes itself — nodes exchange spanning tree announcements and bloom
filters with their direct peers, and FLP makes forwarding decisions for transit
traffic. FLP provides authenticated, encrypted forwarding to FSP above.
See [fips-link-layer.md](fips-link-layer.md) for the FLP specification and
[fips-mesh-operation.md](fips-mesh-operation.md) for how FLP's routing and
self-organization work in practice.
**FIPS Session Protocol (FSP)**: Provides end-to-end authenticated encryption
between any two nodes, regardless of how many intermediate hops separate them.
FSP manages session lifecycle (setup, data transfer, teardown), caches
destination coordinates for efficient routing, and handles the warmup strategy
that keeps transit node caches populated. FSP provides a datagram service to
applications above.
See [fips-session-layer.md](fips-session-layer.md) for the FSP specification.
**IPv6 adaptation layer**: Sits above FSP and adapts the FIPS datagram service
for unmodified IPv6 applications. Provides DNS resolution (npub → fd::/8
address), identity cache management, MTU enforcement, and a TUN interface.
This is the primary way existing applications use the FIPS mesh.
See [fips-ipv6-adapter.md](fips-ipv6-adapter.md) for the IPv6 adapter.
---
## Architecture Overview
![Architecture Overview](fips-architecture-overview.svg)
Each link uses a different transport, but the end-to-end session encryption
is independent of the transport mix. Intermediate nodes decrypt the link
layer to make routing decisions, then re-encrypt for the next hop. They
cannot read the session-layer payload.
Each link uses a different transport, but the end-to-end session encryption is
independent of the transport mix. Intermediate nodes decrypt the link layer to
make routing decisions, then re-encrypt for the next hop. They cannot read the
session-layer payload.
Applications can use the native FIPS datagram service directly, or access the
mesh through an IPv6 adaptation layer (TUN device) for compatibility with
existing IP-based applications. The router handles discovery, routing, and
encryption transparently in either case.
```text
Application ──────────── End-to-end FSP session ──────────── Application
│ │
▼ ▼
┌─────────┐ FLP link ┌─────────┐ FLP link ┌─────────┐
│ Node A │◄──────────────►│ Node B │◄──────────────►│ Node C │
└────┬────┘ (Noise IK) └────┬────┘ (Noise IK) └────┬────┘
│ │ │
UDP/IP Ethernet LoRa
transport transport transport
```
See [fips-transports.md](fips-transports.md) for transport options and characteristics.
Each FLP link operates over its own transport type, with independent link-layer
encryption. The FSP session spans the entire path, providing end-to-end
confidentiality that is independent of the transport mix along the route.
![Node Architecture](fips-node-architecture.svg)
Internally, each node is organized in three layers. At the top, two application
interfaces provide access to the mesh: a native datagram API addressed by npub,
and an IPv6 TUN adapter that maps npubs to `fd::/8` addresses so unmodified
IP applications can use the network transparently. The router core in the middle
implements spanning tree maintenance, bloom filter exchange, greedy routing,
session management, and Noise IK encryption. At the bottom, transport plugins
handle the physical diversity — each plugin implements the same interface, so the
router treats UDP, Ethernet, LoRa, Tor, and serial links identically. Adding a
new transport requires no changes to the routing or session layers.
## Prior Work
FIPS builds on proven designs rather than inventing new cryptography or routing
algorithms.
**Routing**: The spanning tree coordinates, bloom filter discovery, and greedy
routing algorithms are adapted from [Yggdrasil v0.5](https://yggdrasil-network.github.io/2023/10/22/upcoming-v05-release.html)
and its [Ironwood](https://github.com/Arceliar/ironwood) routing library. FIPS
adapts these for multi-transport operation and Nostr identity integration.
**Encryption**: Link and session encryption use the [Noise Protocol
Framework](https://noiseprotocol.org/), the same foundation used by WireGuard,
Lightning Network, and other production systems. FIPS uses the IK pattern for
link authentication and end-to-end sessions.
**Cryptographic primitives**: FIPS reuses Nostr's cryptographic stack—secp256k1
for keys, Schnorr signatures, SHA-256 for hashing, and ChaCha20-Poly1305 for
authenticated encryption. No novel cryptography.
**Session management**: The index-based session dispatch follows WireGuard's
approach, enabling O(1) packet routing without relying on source addresses.
Internally, each node is organized in the three protocol layers. At the top,
two application interfaces provide access to the mesh: a native datagram API
addressed by npub, and an IPv6 TUN adapter that maps npubs to `fd::/8`
addresses so unmodified IP applications can use the network transparently. The
FSP and FLP layers in the middle implement session management, routing, and
encryption. At the bottom, transport plugins handle the physical diversity —
each plugin implements the same interface, so the router treats UDP, Ethernet,
LoRa, Tor, and serial links identically. Adding a new transport requires no
changes to the routing or session layers.
---
@@ -125,7 +230,7 @@ The FIPS address (synonymous with the pubkey) is the primary means for
application-layer software to identify communication endpoints. The
bech32-encoded npub can be used interchangeably for user interface purposes.
The FIPS datagram service is exposed to the application layer either via a
native API to the FIPS node software, or through an IPv6 shim driver that
native API to the FIPS node software, or through an IPv6 adaptation layer that
converts the node identity into an IPv6 address and provides DNS resolution
from npub to this address for traditional software.
@@ -133,70 +238,70 @@ from npub to this address for traditional software.
![Identity Derivation](fips-identity-derivation.svg)
The pubkey is the node's cryptographic identity, used in Noise IK handshakes for
both link and session encryption. It is never exposed beyond the endpoints of an
encrypted channel. The node_addr, a one-way SHA-256 hash truncated to 16 bytes,
serves as the routing identifier in packet headers and bloom filters. Intermediate
routers see only node_addrs — they can forward traffic without learning the Nostr
identities of the endpoints. An observer can verify "does this node_addr belong
to pubkey X?" but cannot enumerate which pubkeys are communicating by inspecting
traffic. The IPv6 address prepends `fd` to the first 15 bytes of the node_addr,
providing an overlay address for unmodified IP applications via the TUN interface.
The pubkey is the node's cryptographic identity, used in Noise IK handshakes
for both link and session encryption. It is never exposed beyond the endpoints
of an encrypted channel. The node_addr, a one-way SHA-256 hash truncated to
16 bytes, serves as the routing identifier in packet headers and bloom filters.
Intermediate routers see only node_addrs — they can forward traffic without
learning the Nostr identities of the endpoints. An observer can verify "does
this node_addr belong to pubkey X?" but cannot enumerate which pubkeys are
communicating by inspecting traffic. The IPv6 address prepends `fd` to the
first 15 bytes of the node_addr, providing an overlay address for unmodified
IP applications via the TUN interface.
### Address Format
When using the IPv6 protocol adapter, FIPS addresses use the IPv6 Unique Local
When using the IPv6 adaptation layer, FIPS addresses use the IPv6 Unique Local
Address (ULA) prefix `fd00::/8`, providing 120 bits from the node_addr hash.
These are overlay identifiers — they appear in the TUN interface for application
compatibility but are not routable on the underlying transport. The fd prefix
ensures no collision with addresses that may be in use on the transport network.
FIPS provides a local DNS service that maps npub bech32 names to IPv6 addresses
for this purpose.
### Identity Verification
The Noise Protocol Framework is used to mutually authenticate both peer-to-peer
link connections and end-to-end session traffic, proving each party controls
the private key for their claimed identity.
link connections (at FLP) and end-to-end session traffic (at FSP), proving each
party controls the private key for their claimed identity.
See [fips-wire-protocol.md](fips-wire-protocol.md) for the Noise IK handshake
and [fips-session-protocol.md](fips-session-protocol.md) for end-to-end
session establishment.
See [fips-link-layer.md](fips-link-layer.md) for peer authentication and
[fips-session-layer.md](fips-session-layer.md) for end-to-end session
establishment.
### Terminology: Addresses and Identifiers
FIPS uses several related but distinct identifiers at different protocol layers:
| Term | Layer | Visible To | Description |
|----------------------------|---------------------|----------------|----------------------------------------------------------------------|
| **FIPS address / pubkey** | Application/Session | Endpoints only | 32-byte secp256k1 public key - the endpoint identity |
| ---- | ----- | ---------- | ----------- |
| **FIPS address / pubkey** | Application/FSP | Endpoints only | 32-byte secp256k1 public key — the endpoint identity |
| **npub** | (encoding) | Human readers | Bech32 encoding of pubkey for display/config |
| **node_addr** | Routing | Routing nodes | SHA-256(pubkey) truncated to 128 bits - cannot be reversed to pubkey |
| **link_addr** | Transport | Direct peers | IP:port, MAC, .onion - transport-specific |
| **IPv6 address** | IPv6 shim | Applications | fd::/8 derived from node_addr - optional compatibility |
| **node_addr** | FLP (routing) | Routing nodes | SHA-256(pubkey) truncated to 128 bits — cannot be reversed to pubkey |
| **link_addr** | Transport | Direct peers | IP:port, MAC, .onion — transport-specific |
| **IPv6 address** | IPv6 adapter | Applications | fd::/8 derived from node_addr — optional compatibility |
**Privacy property**: The pubkey (FIPS address / Nostr identity) is never exposed to
intermediate routing nodes. They see only the node_addr, a one-way hash. An observer
can verify "does this node_addr belong to pubkey X?" but cannot derive the pubkey from
traffic.
**Privacy property**: The pubkey (FIPS address / Nostr identity) is never
exposed to intermediate routing nodes. They see only the node_addr, a one-way
hash. An observer can verify "does this node_addr belong to pubkey X?" but
cannot derive the pubkey from traffic.
---
## Two-Layer Encryption
FIPS uses independent encryption at two layers:
FIPS uses independent encryption at two protocol layers:
| Layer | Scope | Pattern | Purpose |
|-------------|-------------|----------|------------------------------------------------|
| **Link** | Hop-by-hop | Noise IK | Encrypt all traffic on each link |
| **Session** | End-to-end | Noise IK | Encrypt application payload between endpoints |
| ----- | ----- | ------- | ------- |
| **FLP (Link)** | Hop-by-hop | Noise IK | Encrypt all traffic on each peer link |
| **FSP (Session)** | End-to-end | Noise IK | Encrypt application payload between endpoints |
### Link Layer (Hop-by-Hop)
When two nodes establish a direct connection, they perform a Noise IK handshake.
This authenticates both parties and establishes symmetric keys for encrypting
all traffic on that link. Every packet between direct peers is encrypted—gossip
messages, routing queries, and forwarded session datagrams alike.
When two nodes establish a direct connection, they perform a Noise IK
handshake. This authenticates both parties and establishes symmetric keys for
encrypting all traffic on that link. Every packet between direct peers is
encrypted — gossip messages, routing queries, and forwarded session datagrams
alike.
The IK pattern is used because outbound connections know the peer's npub from
configuration, while inbound connections learn the initiator's identity from
@@ -207,188 +312,130 @@ the first handshake message.
FIPS establishes end-to-end encrypted sessions between any two communicating
nodes using Noise IK, regardless of whether they are direct peers or separated
by intermediate routers. The initiator knows the destination's npub; the
responder learns the initiator's identity from the handshake—the same asymmetry
as link-layer connections.
responder learns the initiator's identity from the handshake — the same
asymmetry as link-layer connections.
Both layers always apply. For adjacent peers, application traffic is encrypted
twice: once by the session layer (end-to-end) and once by the link layer
(hop-by-hop). This uniform model means:
- No special case for "local peer" vs "remote destination"
- Topology changes (a direct peer becomes reachable only through intermediaries)
don't affect sessions
- Topology changes (a direct peer becomes reachable only through
intermediaries) don't affect sessions
- The link layer remains purely a transport concern
A packet from A to adjacent peer B:
1. A encrypts payload with A↔B session key
2. A wraps in SessionDatagram, encrypts with A↔B link key, sends to B
1. A encrypts payload with A↔B session key (FSP)
2. A wraps in SessionDatagram, encrypts with A↔B link key (FLP), sends to B
3. B decrypts link layer, then decrypts session layer to get payload
A packet from A to D through intermediate node B:
1. A encrypts payload with A↔D session key
2. A wraps in SessionDatagram, encrypts with A↔B link key, sends to B
3. B decrypts link layer, sees destination, re-encrypts with B↔D link key
1. A encrypts payload with A↔D session key (FSP)
2. A wraps in SessionDatagram, encrypts with A↔B link key (FLP), sends to B
3. B decrypts link layer, reads destination, re-encrypts with B↔D link key
4. D decrypts link layer, then decrypts session layer to get payload
Intermediate nodes can route based on destination address but cannot read
Intermediate nodes can route based on destination node_addr but cannot read
session-layer payloads.
FIPS session setup also warms up route caches along the path between the
endpoints, so that when application traffic flows the network is already ready.
See [fips-wire-protocol.md](fips-wire-protocol.md) for link encryption and
[fips-session-protocol.md](fips-session-protocol.md) for session encryption.
See [fips-link-layer.md](fips-link-layer.md) for link encryption and
[fips-session-layer.md](fips-session-layer.md) for session encryption.
---
## Spanning Tree Protocol
## Routing and Mesh Operation
FIPS routing is entirely distributed — each node makes forwarding decisions
using only local information. There are no routing tables pushed from above, no
link-state floods, and no distance-vector exchanges. Instead, two complementary
mechanisms provide the information each node needs.
### Spanning Tree: The Coordinate System
![Mesh Topology](fips-mesh-topology.svg)
The spanning tree is a subset of the full mesh network that connects all nodes,
forming a tree structure rooted at a deterministically-elected node. Each node
selects a single parent, and the resulting tree serves as the routing backbone.
This enables routing without global routing tables.
Nodes self-organize into a spanning tree rooted at a deterministically-elected
node (the one with the smallest node_addr). Each node selects a single parent
from among its direct peers, and the resulting tree gives every node a
**coordinate** — its path from itself to the root.
### Why a Spanning Tree?
These coordinates enable distance calculations between any two nodes: the
distance is the number of hops from each node to their lowest common ancestor
in the tree. This provides a metric for routing decisions without any node
needing to know the full network topology.
A spanning tree has useful properties:
The tree maintains itself through gossip — nodes exchange TreeAnnounce messages
with their peers, propagating parent selections and ancestry chains. Changes
cascade through the tree proportional to depth, not network size. If the
network partitions, each segment elects its own root and reconverges
automatically when segments rejoin.
- **Unique paths**: Exactly one path exists between any two nodes
- **Minimal state**: Nodes only track their parent and immediate peers
- **Coordinates**: A node's position in the tree enables distance calculations
See [fips-spanning-tree.md](fips-spanning-tree.md) for the tree algorithms and
[spanning-tree-dynamics.md](spanning-tree-dynamics.md) for detailed convergence
walkthroughs.
The tree provides structure for routing while bloom filters provide reachability
information. Together they enable efficient packet delivery without requiring
nodes to know the full network topology.
### Bloom Filters: Candidate Selection
### Tree Coordinates
Each node maintains bloom filters summarizing which destinations are reachable
through each of its peers. Bloom filters propagate via gossip, with each node
computing outbound filters by merging the filters received from its other
peers. At steady state, filters represent the entire reachable network.
A node's coordinate is its path from itself to the root. The distance between
two nodes is the sum of hops from each to their lowest common ancestor (LCA).
This distance metric enables greedy routing: forward packets to the peer that
minimizes distance to the destination.
Bloom filters answer a single question: "can peer P possibly reach destination
D?" The answer is either "no" (definitive) or "maybe" (probabilistic — false
positives are possible). This is **candidate selection**, not routing — bloom
filters identify which peers are worth considering, but the actual forwarding
decision requires tree coordinates to rank those candidates by distance.
### Root Election
See [fips-bloom-filters.md](fips-bloom-filters.md) for filter parameters and
mathematical properties.
The root is the node with the lexicographically smallest node_addr among all
reachable nodes. This election is deterministic and requires no coordination—
each node independently examines its view of the network and reaches the same
conclusion.
### Routing Decisions
The root provides a coordinate reference point but does not participate in
routing unless the paths from source and destination to the root share no
common ancestors (i.e., the root is their lowest common ancestor).
At each hop, FLP makes a local forwarding decision using the following priority
chain:
### Parent Selection
1. **Local delivery** — the destination is this node
2. **Direct peer** — the destination is an authenticated neighbor
3. **Bloom-guided candidate selection** — bloom filters identify peers that can
reach the destination; tree coordinates rank them by distance
4. **Greedy tree routing** — fallback when bloom filters haven't converged;
forward to the peer that minimizes tree distance to the destination
5. **No route** — destination unreachable; send error signal to source
Each node selects a parent that provides the best path to root, considering:
All multi-hop routing depends on knowing the destination's tree coordinates.
These are cached at each node after being learned through discovery
(LookupRequest/LookupResponse) or session establishment (SessionSetup). The
coordinate cache is the critical piece that enables efficient forwarding.
- Reachability (the parent must have a path to root)
- Link quality (latency, packet loss, bandwidth)
- Stability (hysteresis prevents flapping on minor changes)
### Coordinate Caching and Discovery
The parent must be a direct peer—nodes cannot select non-peers as parents.
When a node first needs to reach an unknown destination, it sends a
LookupRequest that floods through the network guided by bloom filters. The
destination responds with its coordinates, which the source caches. Subsequent
traffic routes efficiently using the cached coordinates.
### Tree Gossip
Session establishment (SessionSetup) also carries coordinates, warming transit
node caches along the path so that data packets can be forwarded without
individual discovery at each hop.
Nodes exchange TreeAnnounce messages containing their parent selection and
ancestry chain (path to root). When a node changes its parent, it announces the
change; peers propagate relevant updates. The tree converges through this
gossip without centralized coordination.
### Error Recovery
Changes propagate only as far as they need to—distantly connected nodes are
unaffected by local path changes and don't receive updates for them.
When routing fails — because cached coordinates are stale or a path has
broken — transit nodes signal the source:
### Partition Handling
- **CoordsRequired**: A transit node lacks the destination's coordinates.
The source re-initiates discovery and resets its coordinate warmup strategy.
- **PathBroken**: Greedy routing reached a dead end. The source re-discovers
the destination's current coordinates.
If the network partitions, each isolated segment elects its own root (the
smallest node_addr within that segment). When partitions merge, nodes in the
segment with the larger root discover the globally smaller root and re-parent.
The tree reconverges automatically.
Both signals trigger active recovery, and are rate-limited to prevent storms
during topology changes.
See [fips-routing.md](fips-routing.md) for routing concepts,
[fips-gossip-protocol.md](fips-gossip-protocol.md) for message formats, and
[spanning-tree-dynamics.md](spanning-tree-dynamics.md) for convergence behavior.
---
## Bloom Filter Routing
Tree coordinates enable routing once you know a destination's position. Bloom
filters enable finding that position in the first place.
A bloom filter is a space-efficient probabilistic data structure that can test
whether an element is a member of a set. It may produce false positives (saying
an element is present when it isn't) but never false negatives. This makes it
ideal for routing: a node can quickly check if a destination might be reachable
through a given peer, with occasional false positives handled by backtracking.
### How It Works
Each node maintains bloom filters summarizing which node_addrs are reachable
through each of its peers. These filters propagate through the tree: a node
aggregates filters from its children and announces the combined filter to its
parent (and vice versa).
When a node needs to reach an unknown destination:
1. Check local bloom filters—which peers might be able to reach this node_addr?
2. Send a LookupRequest to peers whose filters indicate "maybe"
3. The request propagates through the tree toward matching subtrees
4. The destination responds with a LookupResponse containing its coordinates
5. The sender caches the coordinates and routes directly via greedy forwarding
Bloom filters have false positives (a filter may indicate "maybe" when the node
isn't actually reachable through that path) but no false negatives. Extra
queries are harmless; missing a reachable node is not.
### Filter Propagation
Filters propagate in the opposite direction from tree announcements:
- Tree state propagates upward (toward root) via ancestry chains
- Bloom filters propagate downward (toward leaves) via subtree aggregation
A node's filter contains all node_addrs reachable through its subtree. The root's
filter contains everyone; leaf nodes have empty outbound filters.
See [fips-routing.md](fips-routing.md) for bloom filter design and
[fips-gossip-protocol.md](fips-gossip-protocol.md) for FilterAnnounce format.
---
## Greedy Routing
Once a destination's tree coordinates are known, packets are forwarded using
greedy routing: at each hop, forward to the peer that minimizes tree distance
to the destination.
### The Algorithm
1. If I am the destination, deliver the packet locally
2. Calculate my tree distance to the destination
3. For each peer, calculate their tree distance to the destination
4. Forward to the peer with the smallest distance (must be less than mine)
5. If no peer is closer, routing has failed (local minimum)
### Path-Broken Recovery
Greedy routing can fail if the destination has moved or the cached coordinates
are stale. When this happens, the node that cannot make progress sends a
PathBroken notification back to the source. The source then initiates a fresh
bloom filter lookup to find the destination's current coordinates.
### Session Establishment
For efficiency, FIPS establishes routing sessions that cache coordinate
information at intermediate routers. The first packet (SessionSetup) carries
full coordinates; subsequent packets use cached state for minimal overhead.
See [fips-routing.md](fips-routing.md) for the complete routing design and
[fips-session-protocol.md](fips-session-protocol.md) for session establishment.
See [fips-mesh-operation.md](fips-mesh-operation.md) for the complete routing
and mesh behavior description.
---
@@ -397,84 +444,33 @@ See [fips-routing.md](fips-routing.md) for the complete routing design and
FIPS is transport-agnostic. The protocol operates identically whether peers
connect over UDP, Ethernet, LoRa radio, serial cables, or Tor hidden services.
### Transports and Links
A **transport** is a physical or logical interface: a UDP socket, an Ethernet
NIC, a Tor client, a radio modem. A **link** is a connection instance to a
specific peer over a transport.
### Multi-Transport Bridging
NIC, a Tor client, a radio modem. A **link** is a peer connection established
over a transport. Transport addresses (IP:port, MAC address, .onion) are opaque
to all layers above FLP — they are used only to deliver datagrams and are
discarded once FLP has authenticated the peer.
A node with multiple transports automatically bridges between networks. Peers
from all transports feed into a single spanning tree; the router selects the
best path regardless of transport type. If one transport fails, traffic
automatically routes through alternatives.
### Transport Types
| Category | Examples | Characteristics |
|----------|----------|-----------------|
| Overlay | UDP/IP, TCP/TLS, QUIC, WebSocket | Internet connectivity, NAT considerations |
| Shared medium | Ethernet, WiFi, Bluetooth, LoRa | Broadcast/multicast discovery |
| Point-to-point | Serial, dialup | No discovery needed, static config |
| -------- | -------- | --------------- |
| Overlay | UDP/IP, TCP/TLS, WebSocket | Internet connectivity, NAT traversal |
| Shared medium | Ethernet, WiFi, Bluetooth, LoRa | Local discovery, broadcast |
| Point-to-point | Serial, dialup | Static config, no discovery |
| Anonymity | Tor, I2P | High latency, strong privacy |
UDP over IP is expected to be the most common transport for internet-connected
nodes. Radio transports enable connectivity where internet infrastructure is
unavailable.
> **Implementation status**: UDP/IP is the only implemented transport. All
> others are future directions.
See [fips-transports.md](fips-transports.md) for transport characteristics.
See [fips-transport-layer.md](fips-transport-layer.md) for transport layer
design.
---
## Protocol Messages
FIPS uses a discriminator-based wire format for efficient message dispatch.
### Link Layer Messages
Exchanged between directly connected peers, encrypted with link session keys:
| Type | Name | Purpose |
|------|----------------|------------------------------------------------------|
| 0x10 | TreeAnnounce | Spanning tree state (parent, ancestry) |
| 0x20 | FilterAnnounce | Bloom filter reachability update |
| 0x30 | LookupRequest | Query for node's tree coordinates |
| 0x31 | LookupResponse | Response with coordinates and proof |
| 0x40 | SessionDatagram| Carries session-layer payloads through the mesh |
| 0x50 | Disconnect | Orderly disconnect notification |
### SessionDatagram Payload Messages
Carried inside SessionDatagram (0x40), which provides src_addr, dest_addr, and
hop_limit for multi-hop forwarding.
**Session-layer messages** (end-to-end encrypted between source and destination):
| Type | Name | Purpose |
|------|----------------|--------------------------------------------|
| 0x00 | SessionSetup | Establish routing session with coordinates |
| 0x01 | SessionAck | Acknowledge session establishment |
| 0x10 | DataPacket | Encrypted application data (IPv6 payload) |
**Link-layer error signals** (plaintext, generated by transit routers):
| Type | Name | Purpose |
|------|----------------|--------------------------------------------|
| 0x20 | CoordsRequired | Router cache miss—need fresh coordinates |
| 0x21 | PathBroken | Greedy routing failed—need re-lookup |
Error signals are sent back to the original source using the `src_addr` from
the SessionDatagram that triggered the error. They are plaintext because the
transit router has no end-to-end session with the source; link-layer encryption
protects them hop-by-hop.
See [fips-wire-protocol.md](fips-wire-protocol.md) for wire format details and
[fips-gossip-protocol.md](fips-gossip-protocol.md) for gossip message formats.
---
## Security Considerations
## Security
### Threat Model
@@ -493,79 +489,105 @@ sees only encrypted packets.
**End-to-end encryption**: Session-layer Noise IK encrypts payloads between
endpoints. Intermediate routers cannot read application data.
**Signature verification**: All protocol messages (TreeAnnounce, LookupResponse)
are signed. The full ancestry chain in TreeAnnounce includes signatures from
each node, preventing forged tree positions.
**Signature verification**: TreeAnnounce messages carry signed parent
declarations. Direct peers verify signatures using keys established during
the Noise IK handshake; ancestry beyond direct peers uses transitive trust
in the v1 protocol.
**Replay protection**: Sequence numbers and timestamps on announcements.
Counter-based nonces with sliding window for encrypted packets.
**Replay protection**: Counter-based nonces with sliding window for encrypted
packets at both the link and session layers. Sequence numbers on protocol
announcements prevent replay of stale state.
### Sybil Resistance
Creating many identities is cheap, but exploiting them is constrained:
- **Discretionary peering**: Node operators choose who to peer with. An attacker
with many identities still needs real nodes to accept their connections.
- **Discretionary peering**: Node operators choose who to peer with. An
attacker with many identities still needs real nodes to accept their
connections.
- **Tree coordinate verification**: Nodes cannot claim arbitrary tree positions
without valid signed ancestry chains from real nodes
without valid signed ancestry chains from real nodes.
- **Rate limiting**: Handshake rate limiting constrains how fast attackers can
establish connections
establish connections.
### Metadata Exposure
Each entity in the network sees different information:
| Entity | Can See |
|--------|---------|
| ------ | ------- |
| Transport observer | Encrypted packets, timing, packet sizes |
| Direct peer | Your npub (identity), traffic volume, timing |
| Intermediate router | Source and destination node_addrs, packet size |
| Destination | Your npub (identity), payload content |
Intermediate routers see node_addrs, not npubs. Since node_addrs are derived from
pubkeys via one-way SHA-256 hash, routers cannot determine the actual identities
of the endpoints they route for.
Intermediate routers see node_addrs, not npubs. Since node_addrs are derived
from pubkeys via one-way SHA-256 hash, routers cannot determine the actual
identities of the endpoints they route for.
The session layer hides payload content from intermediate routers. The link
layer hides everything from passive observers on the underlying transport.
---
## Conclusion
## Prior Work
FIPS combines these elements into a cohesive system that achieves its design
goals:
FIPS builds on proven designs rather than inventing new cryptography or routing
algorithms.
- **Self-sovereign identity** through Nostr keypairs, with node_addrs providing
routing-level privacy
- **Transport agnosticism** via the transport abstraction layer, enabling the
same routing logic across UDP, Ethernet/WiFi, Tor, and other link types
- **Self-organization** through distributed spanning tree formation and bloom
filter propagation, requiring no central coordination
- **Privacy preservation** with two-layer encryption that hides payloads from
intermediate routers and hides everything from transport observers
- **Resilience** through automatic partition detection, re-election, and tree
reconvergence when the network topology changes
**Routing**: The spanning tree coordinates, bloom filter candidate selection,
and greedy routing algorithms are adapted from
[Yggdrasil v0.5](https://yggdrasil-network.github.io/2023/10/22/upcoming-v05-release.html)
and its [Ironwood](https://github.com/Arceliar/ironwood) routing library. FIPS
adapts these for multi-transport operation and Nostr identity integration.
The result is a mesh network where nodes can find and communicate with each
other securely, regardless of the underlying transport infrastructure, while
maintaining control over their own identities and peering relationships.
**Encryption**: Link and session encryption use the
[Noise Protocol Framework](https://noiseprotocol.org/), the same foundation
used by WireGuard, Lightning Network, and other production systems. FIPS uses
the IK pattern for both link authentication and end-to-end sessions.
**Cryptographic primitives**: FIPS reuses Nostr's cryptographic stack —
secp256k1 for keys, Schnorr signatures, SHA-256 for hashing, and
ChaCha20-Poly1305 for authenticated encryption. No novel cryptography.
**Session management**: The index-based session dispatch follows WireGuard's
approach, enabling O(1) packet routing without relying on source addresses.
---
## References
## Further Reading
### FIPS Design Documents
### Protocol Layers
| Document | Description |
|----------|-------------|
| [fips-session-protocol.md](fips-session-protocol.md) | End-to-end session flow, Noise IK encryption |
| [fips-wire-protocol.md](fips-wire-protocol.md) | Link-layer transport, Noise IK handshake |
| [fips-gossip-protocol.md](fips-gossip-protocol.md) | TreeAnnounce, FilterAnnounce, Lookup formats |
| [fips-routing.md](fips-routing.md) | Bloom filters, discovery, greedy routing |
| [spanning-tree-dynamics.md](spanning-tree-dynamics.md) | Tree protocol dynamics and convergence |
| [fips-transports.md](fips-transports.md) | Transport protocol characteristics |
| [fips-software-architecture.md](fips-software-architecture.md) | Software architecture, configuration |
| -------- | ----------- |
| [fips-transport-layer.md](fips-transport-layer.md) | Transport layer: abstraction, types, services provided to FLP |
| [fips-link-layer.md](fips-link-layer.md) | FLP: peer authentication, link encryption, forwarding |
| [fips-session-layer.md](fips-session-layer.md) | FSP: end-to-end encryption, session lifecycle |
| [fips-ipv6-adapter.md](fips-ipv6-adapter.md) | IPv6 adaptation: DNS, TUN interface, MTU enforcement |
### Mesh Behavior and Wire Formats
| Document | Description |
| -------- | ----------- |
| [fips-mesh-operation.md](fips-mesh-operation.md) | How the mesh operates: routing, discovery, error recovery |
| [fips-wire-formats.md](fips-wire-formats.md) | Complete wire format reference for all protocol layers |
### Supporting References
| Document | Description |
| -------- | ----------- |
| [fips-spanning-tree.md](fips-spanning-tree.md) | Spanning tree algorithms and data structures |
| [fips-bloom-filters.md](fips-bloom-filters.md) | Bloom filter parameters, math, and computation |
| [spanning-tree-dynamics.md](spanning-tree-dynamics.md) | Scenario walkthroughs: convergence, partitions, recovery |
### Implementation
| Document | Description |
| -------- | ----------- |
| [fips-software-architecture.md](fips-software-architecture.md) | Stable architectural decisions guiding the codebase |
| [fips-state-machines.md](fips-state-machines.md) | Phase-based state machine pattern (Rust) |
| [fips-configuration.md](fips-configuration.md) | YAML configuration reference |
### External References
docs/design/fips-ipv6-adapter.md
+325
View File
@@ -0,0 +1,325 @@
# FIPS IPv6 Adapter
The IPv6 adapter sits above the FIPS Session Protocol (FSP) and adapts the
FIPS datagram service for unmodified IPv6 applications. It presents each FIPS
node as an IPv6 endpoint, so standard socket applications (SSH, HTTP, SCP)
can communicate over the mesh without modification.
## Role
The adapter bridges two worlds: IPv6 applications that address destinations by
IP address, and the FIPS mesh that addresses destinations by public key
(npub). The adapter handles the translation: DNS resolution from npub to
`fd::/8` address, identity cache management so FIPS can route IPv6 packets,
MTU enforcement so packets fit through the mesh, and the TUN interface that
connects to the kernel's IPv6 stack.
Applications that are FIPS-aware can bypass the adapter entirely and use the
native FIPS datagram API, addressing destinations directly by npub.
## DNS Integration
### The Problem
IPv6 addresses in the `fd::/8` range are derived from public keys via a
one-way hash (SHA-256). Given only an IPv6 address, the public key cannot be
recovered — and without the public key, FIPS cannot compute the node_addr
needed for routing.
The identity cache must be populated *before* packets arrive at the TUN
interface, or they cannot be routed.
### DNS as Entry Point
DNS resolution serves as the "routing intent" signal. When an application
resolves `npub1xxx...xxx.fips`, the FIPS DNS service:
1. Extracts the npub from the `.fips` domain name
2. Derives the `fd::/8` IPv6 address from the public key
3. Primes the identity cache with the mapping
(IPv6 address prefix ↔ NodeAddr ↔ PublicKey)
4. Returns the IPv6 address to the application
When the application subsequently sends packets to that address, the identity
cache already contains the mapping needed for routing.
### DNS Name Format
```text
npub1xxxxxx...xxxxx.fips
```
The FIPS DNS server recognizes names ending in `.fips` and extracts the npub
for address derivation.
### Traffic Without Prior DNS Lookup
A packet may arrive at the TUN for an `fd::/8` destination without a prior DNS
lookup — cached address, manual configuration, etc. Since the address derivation
is one-way, the npub cannot be recovered from the address alone.
FIPS returns ICMPv6 Destination Unreachable (Code 0: No route to destination)
for packets to unknown addresses. The identity cache must be populated before
traffic can be routed.
Known cache population mechanisms:
- **DNS lookup**: The primary path
- **Inbound traffic**: Authenticated sessions from other nodes populate the
cache with their identity information
## IPv6 Address Derivation
FIPS addresses use the IPv6 Unique Local Address (ULA) prefix `fd00::/8`:
```text
Public Key (32 bytes)
│
▼
SHA-256 → node_addr (16 bytes, truncated)
│
▼
fd + node_addr[0..15] → IPv6 address (16 bytes)
```
The `fd` prefix ensures no collision with addresses in use on the underlying
transport network. These are overlay identifiers — they appear in the TUN
interface for application compatibility but are not routable on the underlying
transport.
## Identity Cache
The identity cache maps the FIPS address prefix (15 bytes — the IPv6 address
minus the `fd` prefix) to `(NodeAddr, PublicKey)`. This cache is needed only
when using the IPv6 adapter; the native FIPS API provides the public key
directly.
### Eviction Policy
The mapping is deterministic (derived from the public key) and never becomes
stale. The cache uses **LRU-only eviction** bounded by a configurable size
(default 10K entries). There is no TTL — entries are evicted only when the
cache is full and space is needed for a new entry.
### Relationship to DNS TTL
The identity cache timeout must be longer than the DNS TTL to ensure that while
an application believes its DNS resolution is valid, the corresponding routing
entry remains present. The DNS TTL (default 300s) governs when applications
re-query; the identity cache (LRU, no TTL) is always available as long as the
entry hasn't been evicted by memory pressure.
## MTU Enforcement
FIPS encapsulation adds overhead to every packet. The adapter must ensure
that IPv6 packets from applications fit within the FIPS encapsulation budget
after all layers of wrapping.
### Encapsulation Overhead
| Layer | Overhead | Purpose |
| ----- | -------- | ------- |
| Link encryption | 29 bytes | discriminator + receiver_idx + counter + AEAD tag |
| SessionDatagram envelope | 34 bytes | type + src_addr + dest_addr + hop_limit |
| DataPacket header | 12 bytes | type + flags + counter + payload_len |
| Session encryption | 16 bytes | ChaCha20-Poly1305 AEAD tag |
| **Minimal total** | **91 bytes** | |
| Coordinates (if present) | ~44 bytes | Depth-dependent, first few packets only |
| **Worst case total** | **135 bytes** | With COORDS_PRESENT for depth-3 paths |
The `FIPS_OVERHEAD` constant (135 bytes) is used for conservative MTU
calculations.
### Effective IPv6 MTU
The effective IPv6 MTU visible to applications is:
```text
effective_ipv6_mtu = transport_mtu - FIPS_OVERHEAD
```
For typical deployments:
| Transport MTU | Effective IPv6 MTU | Notes |
| ------------- | ------------------ | ----- |
| 1472 (UDP/Ethernet) | 1337 | Standard deployment |
| 1280 (UDP minimum) | 1145 | Below IPv6 minimum |
IPv6 mandates that every link support at least 1280 bytes. The minimum
transport path MTU for the IPv6 adapter is therefore:
```text
1280 + 135 = 1415 bytes
```
Transports with smaller MTUs (LoRa at ~250 bytes, serial at 256 bytes) cannot
support the IPv6 adapter — applications on those transports must use the
native FIPS datagram API.
### ICMP Packet Too Big
When an outbound packet at the TUN exceeds the effective IPv6 MTU, the adapter
generates an ICMPv6 Packet Too Big message and delivers it back to the
application via the TUN. This triggers the kernel's Path MTU Discovery (PMTUD)
mechanism, which adjusts TCP segment sizes for subsequent transmissions.
ICMP Packet Too Big generation is rate-limited per source address (100ms
interval) to prevent storms from applications sending many oversized packets.
The ICMP response is delivered locally (back through the TUN to the kernel) —
no network traversal is needed, so delivery is reliable.
### TCP MSS Clamping
The adapter intercepts TCP SYN and SYN-ACK packets at the TUN interface and
clamps the Maximum Segment Size (MSS) option:
```text
clamped_mss = effective_ipv6_mtu - 40 (IPv6 header) - 20 (TCP header)
```
This prevents TCP connections from negotiating segment sizes that would exceed
the FIPS path MTU. Clamping is applied in two places:
- **TUN reader** (outbound): Clamps MSS on outbound SYN packets
- **TUN writer** (inbound): Clamps MSS on inbound SYN-ACK packets
Together, these ensure both directions of a TCP connection use appropriately
sized segments from the start, avoiding the initial oversized packet loss
that would occur with ICMP Packet Too Big alone.
### ICMP Rate Limiting
ICMPv6 error generation is rate-limited per source address using a token bucket
(100ms interval). This matches the standard ICMP rate limiting approach and
prevents amplification when an application sends a burst of oversized packets.
## TUN Interface
The TUN device (`fips0`) is the mechanism that connects the adapter to the
kernel's IPv6 stack. It is an implementation detail of the adapter, not its
defining feature.
### Architecture
```text
Applications (sockets using fd::/8 addresses)
│
▼
Kernel IPv6 Stack (routing: fd::/8 → fips0)
│
▼
TUN Device (fips0)
├── Reader Thread (blocking I/O → packet processing)
└── Writer Thread (mpsc queue → TUN writes)
```
### Reader Thread
The TUN reader receives raw IPv6 packets from applications and processes them:
1. Validate IPv6 header
2. Extract destination `fd::/8` address
3. Look up identity cache — miss returns ICMPv6 Destination Unreachable
4. Retrieve NodeAddr and PublicKey from cache
5. Look up or establish FSP session
6. Encrypt payload with session keys
7. Route through FLP toward destination
### Writer Thread
A single writer thread services an mpsc queue of outbound packets:
- Inbound mesh traffic (decrypted session payloads destined for local
applications)
- ICMPv6 error responses (Packet Too Big, Destination Unreachable)
- TCP MSS-clamped SYN-ACK packets
The queue-based design eliminates contention on TUN writes and cleanly
separates concerns. New packet sources can be added by cloning the sender
handle.
### Local Address Guarantee
The Linux kernel routing table processes rules in priority order:
1. **Local table**: Intercepts traffic to addresses assigned to this machine
2. **Main table**: Routes `fd::/8` to the TUN device
This means every packet arriving at the TUN reader is guaranteed to be for a
*remote* FIPS destination. No "is this for me?" check is needed on the read
path.
### Configuration
```yaml
tun:
enabled: true
name: fips0
mtu: 1280
```
### Privileges
TUN device creation requires `CAP_NET_ADMIN`. Options:
- Run as root
- Set capability: `sudo setcap cap_net_admin+ep ./target/debug/fips`
- Pre-created persistent TUN device
## Implementation Status
| Feature | Status |
| ------- | ------ |
| TUN device creation and configuration | **Implemented** |
| IPv6 address assignment (netlink) | **Implemented** |
| TUN reader/writer threads | **Implemented** |
| ICMPv6 Destination Unreachable | **Implemented** |
| ICMPv6 Packet Too Big | **Implemented** |
| ICMP rate limiting (per-source) | **Implemented** |
| TCP MSS clamping (SYN + SYN-ACK) | **Implemented** |
| DNS service (.fips domain) | Planned |
| Per-destination route MTU (netlink) | Planned |
| Transit MTU error signal | Planned |
| Path MTU discovery (envelope field) | Future direction |
| Endpoint fragmentation/reassembly | Future direction |
## Design Considerations
### Path MTU Discovery (Planned)
Two complementary mechanisms are planned for full PMTUD:
1. **Proactive**: A `path_mtu` field (2 bytes) in the SessionDatagram envelope.
The source sets it to its outbound link MTU minus overhead; each transit
node applies `min(current, own_outbound_mtu - overhead)`. The destination
receives the forward-path minimum. A session-layer echo (2 bytes inside
encryption) returns the value to the source.
2. **Reactive**: When a transit node cannot forward a packet (MTU exceeded), it
sends an error signal back to the source. This handles the in-flight gap
between a path MTU decrease and the source learning via the echo.
Both are needed: proactive handles steady state; reactive handles the transient
window when oversized packets hit a new bottleneck before the source adapts.
### No Fragmentation
FIPS remains a pure datagram service with no fragmentation at transit nodes.
Session-layer encryption is end-to-end — the AEAD tag authenticates the entire
plaintext. Fragmenting encrypted datagrams would require either exposing
plaintext structure to transit nodes (unacceptable) or reassembly before
decryption (opens attack surface).
Endpoint-only fragmentation (fragment before session encryption, reassemble
after decryption) is a future direction that avoids these objections. Each
fragment would be independently encrypted and look like a normal DataPacket
to transit nodes.
## References
- [fips-intro.md](fips-intro.md) — Protocol overview and architecture
- [fips-session-layer.md](fips-session-layer.md) — FSP (below the adapter)
- [fips-wire-formats.md](fips-wire-formats.md) — DataPacket and SessionDatagram
formats
- [fips-configuration.md](fips-configuration.md) — TUN configuration parameters
docs/design/fips-link-layer.md
+379
View File
@@ -0,0 +1,379 @@
# FIPS Link Protocol (FLP)
The FIPS Link Protocol is the middle layer of the FIPS protocol stack. It sits
between the transport layer below and the FIPS Session Protocol (FSP) above.
FLP is where anonymous transport addresses become authenticated peers, where
the mesh self-organizes, and where forwarding decisions are made.
## Role
FLP manages direct peer connections over transports. When a transport delivers
a datagram from an unknown address, FLP authenticates the sender through a
Noise IK handshake, establishing a cryptographic link. Once authenticated, the
link carries all inter-peer communication: spanning tree gossip, bloom filter
updates, coordinate discovery, and forwarded session datagrams — all encrypted
per-hop.
FLP is the boundary between opaque transport addresses and identified peers.
Below FLP, everything is transport-specific addresses (IP:port, MAC, .onion).
Above FLP, everything is peers identified by public keys and routable by
node_addr. The transport layer never sees FIPS-level structure; FSP never sees
transport addresses or routing details.
## Services Provided to FSP
From the session layer's perspective, FLP is a black box providing three
services. FSP knows nothing about transports, links, peers, spanning trees,
coordinates, bloom filters, hop counts, or network topology.
### Datagram Forwarding
FLP accepts a datagram addressed by source and destination node_addr and
delivers it best-effort toward the destination. The datagram travels hop by
hop — at each node, FLP decrypts the link layer, reads the destination
node_addr, makes a local forwarding decision, and re-encrypts onto the
next-hop link.
FSP provides: source node_addr, destination node_addr, hop limit, and an
opaque payload (the session-layer encrypted message).
FLP provides: best-effort delivery. No acknowledgment, no retransmission, no
ordering guarantee. Datagrams may be dropped, duplicated, or delivered out of
order.
### Error Signaling
When forwarding fails, FLP signals the source endpoint asynchronously:
- **CoordsRequired**: A transit node lacks the destination's tree coordinates
and cannot make a forwarding decision. The source should re-initiate
discovery and reset its coordinate warmup strategy.
- **PathBroken**: Greedy routing reached a dead end — no peer is closer to the
destination than the current node. The source should re-discover the
destination's current coordinates.
Both signals travel inside the SessionDatagram envelope (using the existing
src/dest/hop_limit addressing) but are generated by transit nodes and are not
end-to-end encrypted. They are rate-limited at 100ms per destination to prevent
storms during topology changes.
### Local Delivery
When a datagram arrives with a destination node_addr matching the local node,
FLP delivers it up to FSP for session-layer processing.
## Services Required from Transport Layer
FLP requires the following from each transport:
### Datagram Delivery
Send and receive raw datagrams to/from transport addresses. The transport
handles all medium-specific details. FLP sees only "send bytes to address" and
"bytes arrived from address."
### MTU Reporting
The maximum datagram size for a given link. FLP needs this to determine how
much payload fits in a single packet after link encryption overhead (29 bytes
for the encrypted frame wrapper).
### Connection Lifecycle
For connection-oriented transports, the transport must establish the underlying
connection before FLP can begin the Noise IK handshake. For connectionless
transports, datagrams can flow immediately.
### Endpoint Discovery (Optional)
When a transport discovers a FIPS-capable endpoint (via beacon, query, or
other transport-specific mechanism), it notifies FLP so that link setup can
be initiated. Transports without discovery support provide peer addresses
through configuration.
## Peer Authentication
### Noise IK Handshake
Every peer connection begins with a Noise IK handshake that mutually
authenticates both parties and establishes symmetric keys for link encryption.
The IK pattern is chosen because:
- The **initiator** knows the responder's static public key from configuration
or discovery, and sends their own static key encrypted in the first message
- The **responder** learns the initiator's identity from the first message,
then responds with their own ephemeral key
After the two-message handshake completes, both parties share symmetric
session keys derived from four DH operations (es, ss, ee, se). The handshake
provides mutual authentication, forward secrecy, and identity hiding for the
initiator.
### Identity Binding
The Noise handshake binds the link to the peer's cryptographic identity. After
handshake completion:
- The peer's public key (FIPS identity) is confirmed
- The node_addr is computed from the public key (SHA-256, truncated to 128 bits)
- All subsequent traffic on the link is authenticated by the Noise session —
successful decryption proves the sender is the authenticated peer
- The link is registered in the dispatch table for O(1) packet routing
### Reconnection
When a Noise IK msg1 arrives from a peer that already has an authenticated
link, FLP accepts the new handshake alongside the existing session. If the new
handshake completes successfully, it replaces the old session. This handles
legitimate reconnection (network change, process restart, NAT rebinding)
without disrupting ongoing traffic until the new session is confirmed.
## Link Encryption
All traffic between authenticated peers is encrypted. Every packet on a link
— gossip messages, routing queries, forwarded session datagrams, disconnect
notifications — passes through Noise's ChaCha20-Poly1305 AEAD.
### Encrypted Frame Structure
Post-handshake packets are wrapped in an encrypted frame consisting of:
- A discriminator byte identifying the frame type
- A receiver index for O(1) session lookup
- An explicit counter used as the AEAD nonce
- The ciphertext with a Poly1305 authentication tag
The plaintext inside the encrypted frame begins with a message type byte
followed by the message-specific payload.
See [fips-wire-formats.md](fips-wire-formats.md) for the complete wire format
specification.
### What Encryption Provides
- **Confidentiality**: An observer on the underlying transport sees only
encrypted packets, packet timing, and packet sizes
- **Integrity**: Any modification to a packet is detected by the AEAD tag
- **Authentication**: Only the authenticated peer can produce valid ciphertext
for this link's session keys
### What Encryption Does Not Provide
- **End-to-end confidentiality**: Link encryption protects traffic on a single
hop. Each transit node decrypts, reads the routing envelope, and re-encrypts
for the next hop. Session-layer encryption (FSP) provides end-to-end
confidentiality.
- **Traffic analysis protection**: Packet timing, sizes, and volume are visible
to transport-layer observers.
## Index-Based Session Dispatch
Incoming packets are dispatched to the correct Noise session using a
receiver index — a random 32-bit value chosen by the receiver during the
handshake. This enables O(1) lookup without relying on source addresses.
Each party in a link has two indices:
- **our_index**: Chosen by us, included by the peer in packets sent to us
- **their_index**: Chosen by them, included by us in packets sent to them
The tuple `(transport_id, receiver_idx)` uniquely identifies a session.
### Index Properties
- **Random**: Cryptographically random to prevent guessing
- **Unique per transport**: No two active sessions on the same transport share
an index
- **Scoped to transport**: The same index value may appear on different
transports
- **Rotated on rekey**: New indices allocated on rekey to prevent cross-session
correlation by passive observers
### Dispatch Flow
1. Read discriminator byte to determine packet type
2. For encrypted frames (0x00): look up `(transport_id, receiver_idx)` in the
session table — O(1) hash lookup. Unknown indices are rejected before any
cryptographic operation.
3. For handshake msg2 (0x02): look up by our sender index to match to a
pending outbound handshake
4. For handshake msg1 (0x01): rate-limited processing, creates new state
This approach follows WireGuard's design: source address is informational,
not authoritative. Only successful cryptographic verification establishes
authenticity.
## Roaming
When an encrypted packet successfully decrypts, the sender is the
authenticated peer regardless of what transport address the packet arrived
from. FLP updates the peer's current address to the packet's source address,
and subsequent outbound packets use the updated address.
This allows peers to change transport addresses (IP:port for UDP, connection
handle for TCP) without session interruption. The mechanism is:
1. Packet arrives from a different address than expected
2. Receiver index lookup finds the session
3. AEAD decryption succeeds — the sender is cryptographically authenticated
4. Peer's address is updated to the new source address
Roaming is most useful for UDP, where source addresses can change due to NAT
rebinding or network changes. For connection-oriented transports, "roaming"
manifests as reconnection rather than mid-session address change.
## Replay Protection
Each link session maintains per-direction counters:
- **Send counter**: Monotonically increasing, used as the AEAD nonce for each
outbound packet
- **Receive window**: A sliding bitmap (2048 entries) tracking which counters
have been seen
The receive window handles the realities of unreliable transports: packets may
arrive out of order, be duplicated, or be lost. The window accepts any counter
not yet seen and within 2048 of the highest counter received. Counters older
than the window are rejected.
The replay check is performed before decryption to prevent CPU exhaustion from
replayed packets that would pass the index lookup but fail decryption.
## Rate Limiting
Handshake initiation (msg1) is the primary attack surface for unauthenticated
traffic. Each msg1 requires Noise DH operations (~200µs on modern CPUs),
state allocation, and response generation.
FLP uses a global token bucket rate limiter:
- **Burst capacity**: Handles legitimate connection storms (e.g., node restart
with many configured peers)
- **Sustained rate**: Limits steady-state new connections per second
- **Global scope**: Rate limiting is global, not per-source, because UDP
source addresses are trivially spoofable
Additional protections:
- **Connection limit**: Maximum number of pending inbound handshakes, capping
memory usage
- **Handshake timeout**: Stale pending handshakes are cleaned up after a
configurable timeout
- **Allowlist/blocklist**: Optional peer filtering before handshake processing
## Disconnect
FLP supports orderly link teardown via a Disconnect message carrying a reason
code (shutdown, restart, protocol error, transport failure, resource
exhaustion, security violation, configuration change, timeout).
On receiving Disconnect, FLP immediately cleans up state: removes the peer
from the peer table, frees the session index, removes the link, and cleans up
address mappings. If the departed peer was the tree parent, FLP triggers parent
reselection.
Disconnect is best-effort — if the transport is broken, the message won't
arrive. Timeout-based detection remains the fallback for detecting failed
links.
On node shutdown, Disconnect is sent to all active peers before transports are
stopped.
## Liveness Detection
FLP detects link liveness through timeout-based mechanisms. There are no
dedicated keepalive or ping/pong messages. Liveness is inferred from:
- **Data traffic**: Any successfully decrypted packet confirms the peer is
alive
- **Gossip messages**: TreeAnnounce and FilterAnnounce messages sent
periodically as part of normal mesh operation serve as implicit heartbeats
When no traffic is received for the configured timeout period, FLP marks the
link as stale. If the stale state persists, the link is torn down.
The two-state liveness model:
- **Connected → Stale**: No traffic received for threshold duration
- **Stale → Connected**: Valid traffic received (peer is alive again)
- **Stale → Disconnected**: Timeout exceeded, link torn down
## Link Message Types
FLP defines six message types carried inside encrypted frames:
| Type | Name | Purpose |
| ---- | ---- | ------- |
| 0x10 | TreeAnnounce | Spanning tree state announcements between peers |
| 0x20 | FilterAnnounce | Bloom filter reachability updates |
| 0x30 | LookupRequest | Coordinate discovery — flood toward destination |
| 0x31 | LookupResponse | Coordinate discovery — response with coordinates |
| 0x40 | SessionDatagram | Encapsulated session-layer payload for forwarding |
| 0x50 | Disconnect | Orderly link teardown with reason code |
Additionally, handshake messages (0x01 msg1, 0x02 msg2) are sent unencrypted
before the link session is established.
TreeAnnounce and FilterAnnounce are exchanged between direct peers only — they
are not forwarded. LookupRequest and LookupResponse are forwarded through the
mesh (flooded with deduplication). SessionDatagram is forwarded hop-by-hop
toward the destination. Disconnect is peer-to-peer.
See [fips-mesh-operation.md](fips-mesh-operation.md) for how these messages
work together to build and maintain the mesh, and
[fips-wire-formats.md](fips-wire-formats.md) for byte-level message layouts.
## Security Properties
### Threat Resistance
| Threat | Mitigation |
| ------ | ---------- |
| Connection exhaustion | Token bucket rate limit + connection count limit |
| CPU exhaustion (msg1 flood) | Rate limit before crypto operations |
| Replay attacks | Counter-based nonces with sliding window |
| State confusion | Strict handshake state machine validation |
| Spoofed encrypted packets | Index lookup + AEAD verification |
| Spoofed msg2 | Index lookup + Noise ephemeral key binding |
| Address spoofing | Cryptographic authority, not address-based |
| Session correlation | Index rotation on rekey |
### Unauthenticated Attack Surface
Only handshake msg1 can be sent by unauthenticated parties. Encrypted frames
require a known session index, and msg2 requires a response to a specific
ephemeral key. The msg1 attack surface is protected by rate limiting,
connection limits, and handshake timeouts.
### Authenticated Peer Misbehavior
Authentication establishes identity but does not grant trust. An authenticated
peer can send malformed packets (which fail AEAD and are dropped) or
high-frequency traffic (rate-limited by higher layers). False tree coordinate
claims are constrained by signature verification on TreeAnnounce messages.
### Silent Drop Policy
Invalid packets are silently dropped without error responses. This prevents
information leakage about internal state and avoids amplification attacks where
an attacker sends invalid packets to elicit responses.
## Implementation Status
| Feature | Status |
| ------- | ------ |
| Noise IK handshake | **Implemented** |
| Link encryption (ChaCha20-Poly1305) | **Implemented** |
| Index-based session dispatch | **Implemented** |
| Replay protection (sliding window) | **Implemented** |
| Roaming (address-follows-crypto) | **Implemented** |
| Rate limiting (token bucket) | **Implemented** |
| Disconnect with reason codes | **Implemented** |
| Liveness detection (timeout-based) | **Implemented** |
| Reconnection handling | **Implemented** |
| Rekey with index rotation | Planned |
| Allowlist/blocklist | Planned |
## References
- [fips-intro.md](fips-intro.md) — Protocol overview and architecture
- [fips-transport-layer.md](fips-transport-layer.md) — Transport layer (below FLP)
- [fips-session-layer.md](fips-session-layer.md) — FSP (above FLP)
- [fips-mesh-operation.md](fips-mesh-operation.md) — How FLP's routing and
self-organization work in practice
- [fips-wire-formats.md](fips-wire-formats.md) — Byte-level wire format reference
docs/design/fips-mesh-operation.md
+570
View File
@@ -0,0 +1,570 @@
# FIPS Mesh Operation
This document describes how the FIPS mesh operates at the link layer — how
spanning tree, bloom filters, routing decisions, discovery, and error recovery
work together as a coherent system. It treats spanning tree and bloom filters
as black boxes (what they provide to routing) and focuses on how the pieces
interact.
For spanning tree algorithms and data structures, see
[fips-spanning-tree.md](fips-spanning-tree.md). For bloom filter parameters
and mathematics, see [fips-bloom-filters.md](fips-bloom-filters.md).
## Overview
FIPS mesh operation is entirely distributed. Each node makes forwarding
decisions using only local information: its direct peers, their spanning tree
positions, and their bloom filters. There are no routing tables pushed from
above, no link-state floods, and no distance-vector exchanges.
Two complementary mechanisms provide the information each node needs:
- **Spanning tree** gives every node a coordinate in the network — its
ancestry path from itself to the root. These coordinates enable distance
calculations between any two nodes without global topology knowledge.
- **Bloom filters** summarize which destinations are reachable through each
peer. They provide candidate selection — narrowing which peers are worth
considering for forwarding a given destination.
Together, they enable a routing decision process that is local, efficient,
and self-healing.
## Spanning Tree Formation and Maintenance
### What the Spanning Tree Provides
The spanning tree gives each node a **coordinate**: its ancestry path from
itself to the root, expressed as a sequence of node_addrs. These coordinates
enable:
- **Distance calculation**: The tree distance between two nodes is the number
of hops from each to their lowest common ancestor (LCA). This provides a
routing metric without any node knowing the full topology.
- **Greedy routing**: At each hop, forward to the peer that minimizes tree
distance to the destination. The strictly-decreasing distance invariant
guarantees loop-free forwarding.
### How the Tree Forms
Nodes self-organize into a spanning tree through distributed parent selection:
1. **Root election**: The node with the smallest node_addr becomes the root.
No election protocol — this is a consequence of each node independently
preferring lower-addressed roots.
2. **Parent selection**: Each node selects a single parent from among its
direct peers based on which offers the best path to root (considering
depth improvement threshold).
3. **Coordinate computation**: Once a node has a parent, its coordinate is
computed from its ancestry path.
### How the Tree Maintains Itself
Nodes exchange **TreeAnnounce** messages with their direct peers (not
forwarded — peer-to-peer only). Each TreeAnnounce carries the sender's
current ancestry chain and a sequence number.
Changes cascade through the tree:
- A node that changes its parent recomputes its coordinates and announces to
all peers
- Each receiving peer evaluates whether the change affects its own parent
selection
- Only nodes that actually change their coordinates (root or depth changed)
propagate further
TreeAnnounce propagation is rate-limited at 500ms minimum interval per peer.
A tree of depth D reconverges in roughly D×0.5s to D×1.0s.
### Partition Handling
If the network partitions, each segment independently elects its own root
(the smallest node_addr in the segment) and reconverges. When segments
rejoin, nodes discover the globally-smallest root through TreeAnnounce
exchange and reconverge to a single tree.
See [fips-spanning-tree.md](fips-spanning-tree.md) for algorithm details
and [spanning-tree-dynamics.md](spanning-tree-dynamics.md) for convergence
walkthroughs.
## Bloom Filter Gossip and Propagation
### What Bloom Filters Provide
Each node maintains a bloom filter per peer, answering: "can peer P possibly
reach destination D?" The answer is either "no" (definitive) or "maybe"
(probabilistic — false positives are possible).
This is **candidate selection**, not routing. Bloom filters identify which
peers are worth considering for a destination, but the actual forwarding
decision uses tree coordinate distance to rank those candidates.
### How Filters Propagate
Nodes exchange **FilterAnnounce** messages with direct peers. Each
FilterAnnounce replaces the previous filter for that peer — there is no
incremental update.
Filter computation uses **split-horizon exclusion**: the outbound filter
for peer Q is computed by merging the local node's own identity, its
leaf-only dependents (if any), and the filters received from all other
peers *except* Q. This prevents echo loops where a node advertises back
to Q the destinations it learned from Q.
Filters propagate unboundedly (no TTL). At steady state, every reachable
destination appears in at least one peer's filter.
### Update Triggers
Filter updates are event-driven, not periodic:
- Peer connects or disconnects
- A peer's incoming filter changes
- Local state changes (new identity, leaf-only dependent changes)
Updates are rate-limited at 500ms to prevent storms during topology changes.
### Scale Properties
At moderate network sizes, bloom filters are highly accurate. At larger
scales (~1M nodes), hub nodes with many peers may see elevated false positive
rates (7–15% for nodes with 20+ peers). False positives cause unnecessary
discovery attempts but do not affect routing correctness — the tree distance
calculation makes the actual forwarding decision.
See [fips-bloom-filters.md](fips-bloom-filters.md) for filter parameters,
FPR calculations, and size class folding.
## Routing Decision Process
At each hop, FLP makes a local forwarding decision using the `find_next_hop()`
priority chain. This is the core routing algorithm.
### Priority Chain
1. **Local delivery** — The destination node_addr matches the local node.
Deliver to FSP above.
2. **Direct peer** — The destination is an authenticated neighbor. Forward
directly. No coordinates or bloom filters needed.
3. **Bloom-guided candidate selection** — One or more peers' bloom filters
contain the destination. Select the best candidate by composite key:
`(link_cost, tree_distance, node_addr)`. This requires the destination's
tree coordinates to be in the local coordinate cache.
4. **Greedy tree routing** — Fallback when bloom filters haven't converged
for this destination. Forward to the peer that minimizes tree distance.
Also requires destination coordinates.
5. **No route** — Destination unreachable. Generate an error signal
(CoordsRequired or PathBroken) back to the source.
### The Coordinate Requirement
All multi-hop routing (steps 3–4) requires the destination's tree coordinates
to be in the local coordinate cache. Without coordinates, `find_next_hop()`
returns None immediately — bloom filters are never even consulted.
This creates two simultaneous convergence requirements for multi-hop routing:
1. **Bloom convergence**: Filters must propagate so peers advertise
reachability
2. **Coordinate availability**: Destination coordinates must be cached at
every transit node on the path
Both must be satisfied simultaneously. Bloom convergence without coordinates
causes a coordinate cache miss. Coordinates without bloom convergence falls
through to greedy tree routing (functional but suboptimal).
### Candidate Ranking
When bloom filters identify multiple candidate peers, they are ranked by a
composite key:
1. **link_cost** — Per-link quality metric (currently constant; designed for
future ETX or similar metrics)
2. **tree_distance** — Coordinate-based distance to destination through this
peer
3. **node_addr** — Deterministic tie-breaker
A peer with a bloom filter hit but no entry in the peer ancestry table
(missing TreeAnnounce) defaults to maximum distance and is effectively
invisible to routing.
### Loop Prevention
The routing decision enforces strict progress: a packet is only forwarded
to a peer that is strictly closer (by tree distance) to the destination than
the current node. This self-distance check prevents routing loops even with
stale coordinates, because each transit node evaluates using its own
freshly-computed coordinates.
If no peer is closer than the current node (a local minimum in the tree
distance metric), `find_next_hop()` returns None and the caller generates a
PathBroken error.
## Coordinate Caching
The coordinate cache maps `NodeAddr → TreeCoordinate` and is the critical
data structure for multi-hop routing. Without it, forwarding decisions cannot
be made.
### Unified Cache
The coordinate cache is a single unified cache. All sources — SessionSetup
transit, COORDS_PRESENT DataPackets, LookupResponse — write to the same
cache.
### Population Sources
| Source | When | What |
| ------ | ---- | ---- |
| SessionSetup transit | Session establishment | Both src and dest coordinates |
| SessionAck transit | Session establishment | Responder's coordinates |
| DataPacket with COORDS_PRESENT | Warmup or recovery | Both src and dest coordinates |
| LookupResponse | Discovery | Target's coordinates |
### Eviction
- **TTL-based**: Entries expire after 300s (configurable)
- **Refresh on use**: Active routing refreshes the TTL, keeping hot entries
alive
- **LRU**: When full, least recently used entries are evicted first
- **Flush on parent change**: When the local node's tree parent changes, the
entire cache is flushed. Parent changes mean the node's own coordinates
have changed, making relative distance calculations with cached coordinates
potentially invalid. Flushing is preferred over stale routing: the cost of
re-discovery is lower than routing packets to dead ends.
### Cache and Session Timer Ordering
Timer values are ordered so that idle sessions tear down before transit
caches expire:
| Timer | Default | Purpose |
| ----- | ------- | ------- |
| Session idle | 90s | Session teardown |
| Coordinate cache TTL | 300s | Coordinate expiration |
When traffic stops, the session tears down at 90s. When traffic resumes, a
fresh SessionSetup re-warms transit caches (still within their 300s TTL).
## Discovery Protocol
Discovery resolves a destination's tree coordinates so that multi-hop routing
can proceed.
### When Discovery Is Needed
- First contact with a destination (no cached coordinates)
- After receiving CoordsRequired (transit node lost coordinates)
- After receiving PathBroken (coordinates may be stale)
### LookupRequest
The source creates a LookupRequest containing:
- **request_id**: Unique identifier for deduplication
- **target**: The node_addr being sought
- **origin**: The requester's node_addr
- **origin_coords**: The requester's current tree coordinates (so the
response can route back)
- **TTL**: Bounds the flood radius
The request floods through the mesh: each node decrements TTL, adds itself
to a visited filter (preventing loops on a single path), and forwards to all
peers not in the visited filter. Bloom filters may help direct the flood
toward likely candidates.
**Deduplication**: Nodes maintain a short-lived request_id dedup cache to
drop convergent duplicates (the same request arriving via different paths).
This is a protocol requirement, not an optimization.
### LookupResponse
When the request reaches the target (or a node that has the target as a
direct peer), a LookupResponse is created containing:
- **request_id**: Echoed from the request
- **target**: The target's node_addr
- **target_coords**: The target's current tree coordinates
- **proof**: Signature covering `(request_id || target)` — authenticates
that the response is genuine
The response routes back to the requester using greedy tree routing toward
the origin_coords from the request.
**Security**: Coordinates are intentionally excluded from the signed proof.
Binding coordinates would invalidate signatures whenever the spanning tree
reconverges. Coordinate tampering by transit nodes causes only routing
inefficiency, not a security breach (data integrity is protected by
session-layer encryption).
### Discovery Outcome
On receiving a LookupResponse, the source caches the target's coordinates.
Subsequent routing to that destination can proceed via the normal
`find_next_hop()` priority chain.
If discovery times out (no response), queued packets receive ICMPv6
Destination Unreachable.
## SessionSetup Self-Bootstrapping
SessionSetup is the mechanism that warms transit node coordinate caches
along a path, enabling subsequent DataPackets to route efficiently.
### How It Works
SessionSetup carries plaintext coordinates (outside the Noise handshake
payload, visible to transit nodes):
- **src_coords**: Source's current tree coordinates
- **dest_coords**: Destination's tree coordinates (learned from discovery)
As the SessionSetup transits each intermediate node:
1. The transit node extracts both coordinate sets
2. Caches `src_addr → src_coords` and `dest_addr → dest_coords` in its
coordinate cache
3. Forwards the message using the cached destination coordinates
SessionAck returns along the reverse path, carrying the responder's
coordinates and warming caches in the other direction.
### Result
After the handshake completes, the entire forward and reverse paths have
cached coordinates for both endpoints. Subsequent DataPackets use minimal
headers (no coordinates) and route efficiently through the warmed caches.
## COORDS_PRESENT Warmup
The COORDS_PRESENT flag on DataPackets provides a secondary cache-warming
mechanism that complements SessionSetup. See
[fips-session-layer.md](fips-session-layer.md) for the warmup-then-reactive
strategy.
Transit nodes process DataPackets with COORDS_PRESENT by extracting and
caching both source and destination coordinates — the same operation
performed for SessionSetup coordinates.
## Error Recovery
When routing fails, transit nodes signal the source endpoint so it can take
corrective action.
### CoordsRequired
**Trigger**: A transit node receives a SessionDatagram but has no cached
coordinates for the destination. It cannot make a forwarding decision.
**Transit node action**:
1. Create a new SessionDatagram addressed back to the original source,
carrying a CoordsRequired payload identifying the unreachable destination
2. Route the error via `find_next_hop(src_addr)`
3. If the source is also unreachable, drop silently (no cascading errors)
**Source recovery**:
1. Initiate discovery (LookupRequest flood) for the destination
2. Reset COORDS_PRESENT warmup counter — subsequent DataPackets include
coordinates
3. When discovery completes, warmup counter resets again (covers timing gap)
The crypto session remains active throughout — only routing state is
refreshed.
### PathBroken
**Trigger**: A transit node has cached coordinates for the destination but
no peer is closer to the destination than itself (a local minimum in the
tree distance metric). The cached coordinates may be stale.
**Transit node action**: Same as CoordsRequired — generate error back to
source.
**Source recovery**:
1. Remove stale coordinates from cache
2. Initiate discovery for the destination
3. Reset COORDS_PRESENT warmup counter
### Error Signal Rate Limiting
Both error types are rate-limited at transit nodes: maximum one error per
destination per 100ms. This prevents storms during topology changes when many
packets to the same destination hit the same routing failure simultaneously.
### Error Routing Limitation
Error signals route back to the source using `find_next_hop(src_addr)`. For
steady-state DataPackets (after the COORDS_PRESENT warmup window), the
transit node may lack cached coordinates for the source. If so, the error is
silently dropped.
This blind spot is partially addressed by COORDS_PRESENT warmup: transit
nodes receive source coordinates during the warmup phase. But after warmup
expires and transit caches for the source expire, errors may be lost. The
session idle timeout (90s) limits the window — if traffic stops long enough
for transit caches to fully expire, the session tears down and re-establishment
re-warms the path.
## Cold Start → Warm Cache → Steady State
### Cold Start
A new node or a node reaching a new destination goes through the following
sequence:
1. **DNS resolution** (IPv6 adapter only): Resolve `npub.fips` → populate
identity cache with NodeAddr + PublicKey
2. **Session initiation attempt**: Fails because no coordinates are cached
for the destination
3. **Discovery**: LookupRequest floods through the mesh; LookupResponse
returns the destination's coordinates
4. **Session establishment**: SessionSetup carries coordinates, warming
transit caches along the path
5. **Warmup**: First N DataPackets include COORDS_PRESENT, reinforcing
transit caches
The first packet to a new destination always triggers this sequence. The
packet is queued (bounded) until the session is established.
### Warm Cache
After session establishment and warmup:
- Transit nodes have cached coordinates for both endpoints
- Bloom filters have converged for the destination
- DataPackets use minimal headers (no coordinates)
- Routing decisions are fast: bloom candidate selection + distance ranking
### Steady State
In steady state, the mesh is mostly self-maintaining:
- TreeAnnounce gossip keeps the spanning tree current
- FilterAnnounce gossip keeps bloom filters current
- Coordinate caches are refreshed by active routing traffic
- Occasional cache misses trigger COORDS_PRESENT or discovery, but these
are rare when traffic is flowing
### Cache Expiry and Recovery
When traffic to a destination stops:
1. **Session idles out** (90s) — session torn down
2. **Coordinate caches expire** (300s) — transit nodes forget coordinates
3. **Bloom filters remain** — they have no TTL, so reachability information
persists
When traffic resumes:
1. Identity cache: usually still populated (LRU, no TTL)
2. Session: new establishment required (full handshake)
3. Coordinates: discovery may be needed if cache has expired
4. SessionSetup re-warms transit caches on the new path
## Leaf-Only Operation *(future direction)*
Leaf-only operation is a planned optimization for resource-constrained nodes
(sensors, battery-powered devices). Not currently implemented.
### Concept
A leaf-only node connects to a single upstream peer that handles all routing
on its behalf:
- **No bloom filter storage or processing**: The upstream peer includes the
leaf's identity in its own outbound bloom filters
- **No spanning tree participation**: The leaf does not offer itself as a
potential parent to other nodes
- **Simplified routing**: All traffic tunnels through the upstream peer
- **Minimal resource usage**: Suitable for ESP32-class devices (~500KB RAM)
### Upstream Peer Responsibilities
The upstream peer:
- Includes the leaf's identity in its outbound bloom filters
- Forwards all traffic addressed to the leaf
- Handles discovery responses on behalf of the leaf
- Maintains the link session with the leaf
### What the Leaf Retains
Even as a leaf-only node, it still:
- Maintains its own Noise IK link session with the upstream peer
- Can establish end-to-end FSP sessions with arbitrary destinations
- Has its own identity (npub, node_addr)
The optimization is purely at the routing/mesh layer — the leaf delegates
routing decisions but retains its own end-to-end encryption and identity.
## Packet Type Summary
| Message | Typical Size | When | Forwarded? |
| ------- | ------------ | ---- | ---------- |
| TreeAnnounce | Variable (depth-dependent) | Topology changes | No (peer-to-peer) |
| FilterAnnounce | ~1 KB | Topology changes | No (peer-to-peer) |
| LookupRequest | ~300 bytes | First contact, recovery | Yes (flood) |
| LookupResponse | ~400 bytes | Response to discovery | Yes (greedy routed) |
| SessionDatagram + SessionSetup | ~230–400 bytes | Session establishment | Yes (routed) |
| SessionDatagram + SessionAck | ~120 bytes | Session confirmation | Yes (routed) |
| SessionDatagram + DataPacket (minimal) | 38 bytes + payload | Bulk traffic | Yes (routed) |
| SessionDatagram + DataPacket (with coords) | ~170 bytes + payload | Warmup/recovery | Yes (routed) |
| SessionDatagram + CoordsRequired | 68 bytes | Cache miss error | Yes (routed) |
| SessionDatagram + PathBroken | 68+ bytes | Dead-end error | Yes (routed) |
| Disconnect | 2 bytes | Link teardown | No (peer-to-peer) |
See [fips-wire-formats.md](fips-wire-formats.md) for byte-level layouts.
## Privacy Considerations
Source and destination node_addrs are visible to every transit node (required
for forwarding decisions and error signal routing). FIPS prioritizes
low-latency greedy routing with explicit error signaling over metadata
privacy.
The node_addr is `SHA-256(pubkey)` truncated to 128 bits — a one-way hash.
Transit nodes learn which node_addr pairs are communicating but cannot
determine the actual Nostr identities (npubs) of the endpoints. An observer
can verify "does this node_addr belong to pubkey X?" but cannot enumerate
communicating identities from traffic alone.
Onion routing was considered and rejected because it requires the sender to
know the full path upfront (incompatible with self-organizing routing) and
prevents per-hop error feedback (incompatible with CoordsRequired/PathBroken
recovery).
## Implementation Status
| Feature | Status |
| ------- | ------ |
| Spanning tree formation | **Implemented** |
| TreeAnnounce gossip | **Implemented** |
| Bloom filter computation (split-horizon) | **Implemented** |
| FilterAnnounce gossip | **Implemented** |
| find_next_hop() priority chain | **Implemented** |
| Coordinate cache (unified, TTL + refresh) | **Implemented** |
| Flush coord cache on parent change | **Implemented** |
| LookupRequest/LookupResponse discovery | **Implemented** |
| SessionSetup self-bootstrapping | **Implemented** |
| COORDS_PRESENT warmup-then-reactive | **Implemented** |
| CoordsRequired recovery | **Implemented** |
| PathBroken recovery | **Implemented** |
| Error signal rate limiting | **Implemented** |
| Leaf-only operation | Future direction |
| Link cost metrics (ETX) | Future direction |
| Discovery path accumulation | Future direction |
## References
- [fips-intro.md](fips-intro.md) — Protocol overview
- [fips-link-layer.md](fips-link-layer.md) — FLP specification
- [fips-spanning-tree.md](fips-spanning-tree.md) — Tree algorithms and data
structures
- [fips-bloom-filters.md](fips-bloom-filters.md) — Filter parameters and math
- [fips-wire-formats.md](fips-wire-formats.md) — Wire format reference
- [spanning-tree-dynamics.md](spanning-tree-dynamics.md) — Convergence
walkthroughs
docs/design/fips-routing.md
-903
View File
@@ -1,903 +0,0 @@
# FIPS Routing Design
This document describes the routing architecture for FIPS, including Bloom
filter routing, greedy tree routing, discovery protocol, and routing session
establishment.
For wire formats and exchange rules, see [fips-gossip-protocol.md](fips-gossip-protocol.md).
For spanning tree dynamics and convergence, see [spanning-tree-dynamics.md](spanning-tree-dynamics.md).
## Overview
FIPS uses a layered routing strategy where each mechanism handles different
situations. In steady state, bloom filter routing handles the vast majority
of forwarding decisions.
### Next-Hop Selection (in priority order)
1. **Local delivery** — destination is self
2. **Direct peer** — destination is an authenticated peer
3. **Bloom filter routing** — one or more peers' bloom filters contain the
destination; select the best candidate by `(link_cost, tree_distance,
node_addr)`. Since filters propagate unboundedly through the network,
every reachable destination eventually appears in at least one peer's
filter. This is the **primary routing path** for most traffic.
4. **Greedy tree routing** — fallback when bloom filters haven't yet
converged (transient condition during topology changes). Requires the
destination's tree coordinates to be in the local coordinate cache,
populated by a prior SessionSetup or LookupResponse.
5. **No route** — destination unreachable
### Role of Each Mechanism
- **Bloom filters**: Primary forwarding — tell each node which peer to
send through for a given destination. Propagate unboundedly via
split-horizon merge, so they cover the entire reachable network at
steady state.
- **Greedy tree routing**: Fallback forwarding during convergence windows
when bloom filters are incomplete. Also serves as tie-breaker among
bloom filter candidates (closest tree distance wins).
- **Discovery protocol**: Populates the coordinate cache to enable greedy
tree routing and to provide intermediate routers with coordinate data
for more efficient path selection. Not required for basic reachability
once bloom filters have converged.
## Design Goals
- Minimize per-packet overhead for data transfer
- Bounded state at each node (independent of network size)
- Efficient routing without global knowledge
- Graceful degradation for constrained devices
- Fast convergence on topology changes
## Network Scale Assumptions
| Scale | Nodes | Bloom Filter Role |
|-------|-------|-------------------|
| Small private network | 100-1,000 | Covers entire network with low FPR |
| Modest public network | ~1,000,000 | Covers entire network but FPR increases at hub nodes due to filter saturation |
| Internet-scale | Billions | Out of scope (requires different architecture) |
The primary design target is networks up to ~1M nodes. Since bloom filters
propagate unboundedly (no TTL), they converge to represent the entire
reachable network. At large scale, the fixed 1KB filter size means higher
false positive rates at well-connected hub nodes, which may trigger
unnecessary discovery queries but does not affect correctness.
## Node Participation Modes
### Full Participant
- Maintains Bloom filters for peer reachability
- Participates in spanning tree (can be selected as parent)
- Routes packets for other nodes
- Minimum viable device: ESP32-class (~500KB RAM)
### Leaf-Only
- Single peer handles all routing on its behalf
- No Bloom filter storage or processing
- Does not participate in spanning tree as potential parent
- Suitable for highly constrained devices (sensors, battery-powered nodes)
Leaf-only nodes appear as a single entry in their peer's Bloom filter. All
traffic tunnels through that peer.
---
## Part 1: Bloom Filter Design
### Parameters
| Parameter | Value | Rationale |
|----------------|-------------------|--------------------------------------------|
| Filter size | 1 KB (8,192 bits) | Sized for expected occupancy with margin |
| Hash functions | 5 | Optimal for 800-1,600 entries at this size |
### Mathematical Foundation
**False Positive Rate (FPR):**
```text
FPR = (1 - e^(-kn/m))^k
```
Where m = bits, n = entries, k = hash functions.
**Optimal hash count:**
```text
k_opt = (m/n) × ln(2) ≈ 0.693 × (m/n)
```
For m=8,192 and expected n=800: k_opt ≈ 7. We use k=5 to accommodate higher
occupancy scenarios (up to ~1,600 entries) while maintaining acceptable FPR.
**Required bits for target FPR:**
```text
m = -1.44 × n × ln(p)
```
For 1% FPR: m ≈ 9.6n bits. For 5% FPR: m ≈ 6.2n bits.
### Expected Filter Occupancy
Filter occupancy depends on network topology and node degree. In practice,
filters reach a natural equilibrium determined by the network's structure —
merging peer filters transitively means each filter converges to represent
the node's reachable neighborhood.
**Outgoing filter to peer Q contains:**
- Self (1 entry)
- Entries from (d-1) other peers' filters (excluding Q), with overlap
**Expected occupancy by node degree:**
| Degree (d) | Expected Entries | Notes |
|------------|------------------|-------|
| 5 | 100-200 | Constrained/IoT |
| 8 | 250-400 | Typical node |
| 12 | 500-800 | Well-connected |
| 20+ | 1,200-1,800 | Hub node |
### False Positive Rates (1 KB filter, k=5)
| Entries | FPR | Scenario |
|---------|-----|----------|
| 200 | 0.02% | Low-degree node |
| 400 | 0.3% | Typical node |
| 800 | 2.4% | Well-connected |
| 1,200 | 7.5% | Hub node |
| 1,600 | 15% | Heavily loaded hub |
Since filters propagate unboundedly, hub nodes with many peers will have
higher occupancy (more entries merged from more peers). FPR above 5% means
bloom filter routing may occasionally select a peer that can't actually
reach the destination (false positive), requiring fallback to greedy tree
routing or error recovery. Hub nodes may benefit from larger filters in
future protocol versions (see §1.6).
### Size Classes (Forward Compatibility)
Filter sizes are powers of 2 to enable **folding** — a technique for shrinking
filters by ORing halves:
```rust
fn fold(filter: &[u8]) -> Vec<u8> {
let half = filter.len() / 2;
(0..half).map(|i| filter[i] | filter[i + half]).collect()
}
```
Folding preserves correctness (no false negatives) but increases FPR.
| size_class | Bits | Bytes | Status |
|------------|------|-------|--------|
| 0 | 4,096 | 512 | Reserved (future) |
| 1 | 8,192 | 1,024 | **Current default** |
| 2 | 16,384 | 2,048 | Reserved (future) |
| 3 | 32,768 | 4,096 | Reserved (future) |
**v1 protocol**: All nodes MUST use size_class=1. The field is present in the
wire format for forward compatibility.
**Future versions**: Nodes may negotiate larger filters. Receivers fold down
to their preferred size if sender's filter is larger. This allows hub nodes
to maintain higher precision while constrained nodes use smaller filters.
### Filter Contents
Each node's filter contains Node IDs (and optionally gateway /64 prefixes)
that are reachable through that node. A Node ID is the SHA-256 hash of the
node's npub, truncated or used directly as the filter key.
### Per-Peer Filters
Each node maintains a Bloom filter for each peer direction:
```rust
peer_filters: HashMap<NodeAddr, BloomFilter>
```
The filter for peer P answers: "Which destinations are reachable through P?"
### Update Mechanism: Event-Driven
Filters are updated on events rather than periodic refresh:
**Triggering events:**
1. Peer connects — exchange current filters
2. Peer disconnects — remove their filter, recompute, notify other peers
3. Received filter changes outgoing filter — recompute, send updates
4. Local state change — new leaf dependent, become gateway, etc.
Updates are rate-limited to prevent storms during reconvergence. See
[fips-gossip-protocol.md](fips-gossip-protocol.md) §3 for FilterAnnounce wire
format and exchange rules.
### Filter Contents
A node's outgoing filter to peer Q contains:
1. This node's own Node ID
2. Node IDs of leaf-only dependents
3. Entries merged from filters received from all other peers (not Q)
Filters propagate transitively through the network. Each node merges all
inbound peer filters (excluding the destination peer) into its outgoing
filter — this split-horizon approach prevents a node's own entries from
being echoed back to it, providing loop prevention. Propagation is
unbounded; filters naturally converge as the Bloom filter's fixed size
limits information density.
### Expiration
Bloom filters cannot remove individual entries. Expiration is handled via:
- **Peer disconnect**: Remove that peer's filter entirely, recompute
- **Filter replacement**: Each FilterAnnounce replaces the previous one
- **Implicit timeout**: If no updates received from peer within threshold,
consider their filter stale
---
## Part 2: Discovery Protocol
### Purpose
Discover the tree coordinates of a destination to enable greedy tree routing
and to populate coordinate caches at intermediate routers for more efficient
forwarding. In steady state, bloom filters handle reachability; discovery
provides the coordinate information that improves path selection quality.
### When Used
- During bloom filter convergence (destination not yet in any peer's filter)
- To populate coordinate caches for greedy tree routing (optimization)
- After cached route failure (coordinates may be stale)
For wire formats, see [fips-gossip-protocol.md](fips-gossip-protocol.md) §4-5.
### Discovery Flow
```text
1. S wants to reach D, D not in any local filter
2. S checks route cache — miss
3. S creates LookupRequest with own coordinates, floods to peers
4. Request propagates (Bloom filters may help direct it)
5. Request reaches D (or node with D in filter)
6. D creates LookupResponse with its coordinates, signs it
7. Response routes back to S using S's coordinates (greedy)
8. S caches D's coordinates
9. S can now route to D using greedy tree routing
```
### Request Propagation
**Flood with TTL and visited filter:**
- Send to all peers not in `visited` filter
- Each hop decrements TTL, adds self to `visited`
- At TTL=0, stop propagating
- `visited` filter prevents loops (a packet revisiting a node on its own
path), but does NOT prevent convergent duplicates — the same request
arriving at a node via different paths with different visited filters.
See [Known Limitation: Flood
Convergence](#known-limitation-flood-convergence).
**Bloom filter assistance (optional optimization):**
If a node's peer filter indicates "maybe" for the target, prioritize that
direction. Reduces flood scope when target is partially in range.
### Response Routing
Response uses greedy tree routing based on `origin_coords` from the request.
Each router forwards toward the origin using tree distance.
### Security
The target signs the LookupResponse with a proof covering
`(request_id || target)`. Without this signature, a malicious node could
claim reachability for any target and blackhole traffic. The signature
proves the target authorized the route.
Note: `target_coords` are intentionally excluded from the proof. Binding
coordinates would invalidate the signature whenever the spanning tree
reconverges (parent switch, root change), causing valid responses to be
rejected if tree topology shifts during the lookup RTT. Since coordinates
are ephemeral routing hints and data integrity is protected by the
session layer, coordinate tampering by a transit node only causes routing
inefficiency, not a security breach.
### Caching
Discovered coordinates are stored in the route cache (`RouteCache`):
```rust
struct RouteCache {
entries: HashMap<NodeAddr, CachedCoords>,
max_entries: usize, // default: 10,000
}
struct CachedCoords {
coords: TreeCoordinate,
discovered_at: Timestamp,
last_used: Timestamp,
}
```
- **Eviction**: LRU when cache full (no automatic TTL expiration)
- **Invalidation**: On route failure, evict and re-discover
This is distinct from the `CoordCache` used for session-populated
coordinates (see Part 4 for the full dual-cache architecture).
---
## Part 3: Tree Coordinates and Greedy Routing
Tree coordinates and greedy routing serve two roles:
1. **Fallback forwarding** during bloom filter convergence windows
2. **Tie-breaking** among bloom filter candidates — tree distance between
a candidate peer and the destination helps select the best path when
multiple peers advertise reachability
### Tree Coordinates
A node's coordinates are its ancestry path from self to root:
```text
coords(N) = [N, Parent(N), Parent(Parent(N)), ..., Root]
```
Example: Node D at depth 4 has coordinates `[D, P1, P2, P3, Root]`.
### Tree Distance
Distance between two nodes is hops through their lowest common ancestor (LCA):
```rust
fn tree_distance(a_coords: &[NodeAddr], b_coords: &[NodeAddr]) -> usize {
let lca_depth = longest_common_suffix_length(a_coords, b_coords);
let a_to_lca = a_coords.len() - lca_depth;
let b_to_lca = b_coords.len() - lca_depth;
a_to_lca + b_to_lca
}
```
Note: Coordinates are ordered self-to-root, so common ancestry is a suffix.
### Greedy Routing Algorithm
When used as a fallback (no bloom filter hits), greedy routing forwards to
the peer that minimizes tree distance to the destination. A self-distance
check ensures progress — the packet is only forwarded if the chosen peer is
strictly closer than the current node.
```rust
fn greedy_next_hop(&self, dest_coords: &TreeCoordinate) -> Option<NodeAddr> {
if self.my_coords.root_id() != dest_coords.root_id() {
return None; // different tree
}
let my_distance = self.my_coords.distance_to(dest_coords);
// Find peer with minimum distance, tie-break by smallest node_addr
let best = self.peer_ancestry.iter()
.min_by(|(id_a, coords_a), (id_b, coords_b)| {
coords_a.distance_to(dest_coords)
.cmp(&coords_b.distance_to(dest_coords))
.then_with(|| id_a.cmp(id_b))
});
match best {
Some((peer_id, coords)) if coords.distance_to(dest_coords) < my_distance => {
Some(*peer_id)
}
_ => None, // no peer is closer (local minimum)
}
}
```
### Progress Guarantee
When the coordinate cache is populated, greedy routing makes progress as
long as:
1. Tree is connected
2. Destination's coordinates are accurate
3. A peer is closer to the destination than the current node
If no peer is closer (local minimum), routing returns `None` and the caller
generates a PathBroken error. In a properly formed tree this should not
occur, but the self-distance check provides a safety net.
### What Each Node Knows
| Information | Source |
|-------------|--------|
| Own coordinates | Spanning tree protocol (ancestry to root) |
| Each peer's coordinates | Exchanged on peering |
| Destination coordinates | From packet header (established via session) |
No global routing tables. Each node makes purely local decisions.
### Privacy Considerations
Intermediate routers can observe `src_addr` and `dest_addr` in the
SessionDatagram envelope of transiting packets. This enables traffic analysis
(who is communicating with whom) but not content inspection (the payload is
end-to-end encrypted with session keys).
**Why source address is visible**: The `src_addr` field in the SessionDatagram
is required for transit routers to send error signals (CoordsRequired,
PathBroken) back to the sender. When a transit router R cannot forward a
SessionDatagram `{src: S, dest: D}`, it creates a new SessionDatagram
`{src: R, dest: S}` carrying the error signal, and routes it toward S using
`find_next_hop(S)`. This is a deliberate design choice: rather than silently
dropping unroutable packets and relying on application-layer timeouts to detect
failures, FIPS provides explicit feedback that allows rapid route recovery.
The tradeoff favors responsiveness over metadata privacy.
**Partial mitigation**: FIPS addresses are derived from `SHA-256(pubkey)`, not the
npub itself. An observer learns that `fd12:3456:...` is communicating with
`fd78:9abc:...`, but cannot directly determine the Nostr identities without
additional information (e.g., DNS lookup correlation, prior knowledge of the
address-to-npub mapping).
**Alternative considered**: Onion routing (like Tor) hides routing metadata from
intermediate nodes but requires the sender to know the full path upfront and
prevents per-hop error feedback. FIPS prioritizes low-latency greedy routing
with explicit error signaling over metadata privacy.
---
## Part 4: Coordinate Cache Architecture
> **Wire formats**: For session layer message wire formats (SessionSetup,
> SessionAck, DataPacket, CoordsRequired, PathBroken), see
> [fips-session-protocol.md](fips-session-protocol.md) §8.
### Dual Cache Architecture
FIPS uses two coordinate caches with different lifecycles:
- **CoordCache** (session-populated): Stores coordinates learned from
SessionSetup and SessionAck coordinate fields during session establishment.
TTL-based expiration (300s). 50,000 entries max. Consulted first by
`find_next_hop()`.
- **RouteCache** (discovery-populated): Stores coordinates learned from
LookupResponse during discovery protocol completion. LRU eviction only
(no automatic TTL). 10,000 entries max. Consulted as fallback when
CoordCache misses.
Both caches store `TreeCoordinate` values (node_addr → coordinates). Neither
stores next-hop information — routing decisions are computed at lookup time by
`find_next_hop()` using the cached coordinates.
### Cache Purpose
The coordinate caches serve two functions:
1. **Greedy routing fallback** — when bloom filters haven't converged,
cached coordinates enable tree-distance-based forwarding.
2. **Reduced packet overhead** — once coordinates are cached at intermediate
routers, data packets can use minimal DataPacket headers (4 bytes inside
a 34-byte SessionDatagram = 38 bytes total) rather than including full
coordinates (~170 bytes total).
### Cache Lifecycle
```text
┌─────────────────────────────────────────────────────────────────┐
│ 1. Discovery: S queries for D's coordinates │
│ 2. Setup: S sends SessionSetup, routers cache coordinates │
│ 3. Data: Packets carry only addresses, routers use cache │
│ 4. Refresh: Periodic or on-demand to prevent cache expiry │
│ 5. Teardown: Implicit (cache expires) or explicit │
└─────────────────────────────────────────────────────────────────┘
```
### Session Setup Flow
All messages are carried inside SessionDatagram envelopes that provide
`src_addr`, `dest_addr`, and `hop_limit` at the link layer.
```text
S R1 R2 D
│ │ │ │
│──SessionDatagram──────>│ │ │
│ {src:S, dest:D, │──SessionDatagram─────>│ │
│ payload:SessionSetup │ │──SessionDatagram─────>│
│ (src_coords, │ │ │
│ dest_coords)} │ cache: │ cache: │
│ │ dest_addr→dest_coords│ dest_addr→dest_coords│
│ │ src_addr→src_coords │ src_addr→src_coords │
│ │ │ │
│<──────────────────────────────────────────SessionDatagram{SessionAck}──│
│ │ │ │
│══SessionDatagram══════>│══════════════════════>│══════════════════════>│
│ {src:S, dest:D, │ (use cached coords) │ (use cached coords) │
│ payload:DataPacket} │ │ │
```
### Router Behavior
Transit routers process SessionDatagram envelopes. The envelope provides
`src_addr` and `dest_addr` for routing decisions and error signaling.
```rust
impl Router {
/// Handle a SessionDatagram carrying a SessionSetup payload.
/// Cache coordinates from the setup message for both directions.
fn handle_session_setup(&mut self, dg: &SessionDatagram, setup: SessionSetup) {
// Cache coordinates for both directions
self.coord_cache.insert(dg.dest_addr, CacheEntry {
coords: setup.dest_coords.clone(),
expires: now() + CACHE_TTL,
});
self.coord_cache.insert(dg.src_addr, CacheEntry {
coords: setup.src_coords.clone(),
expires: now() + CACHE_TTL,
});
// Forward toward destination using find_next_hop
if let Some(next) = self.find_next_hop(&dg.dest_addr) {
self.forward(next, dg);
}
}
/// Handle a SessionDatagram carrying a DataPacket payload.
/// Addresses come from the SessionDatagram envelope (dg.src_addr, dg.dest_addr).
fn handle_data_packet(&mut self, dg: &SessionDatagram, packet: DataPacket) {
// If packet carries coordinates, cache them
if packet.flags & COORDS_PRESENT != 0 {
if let (Some(src_coords), Some(dest_coords)) =
(&packet.src_coords, &packet.dest_coords)
{
self.coord_cache.insert(dg.dest_addr, CacheEntry {
coords: dest_coords.clone(),
expires: now() + CACHE_TTL,
});
self.coord_cache.insert(dg.src_addr, CacheEntry {
coords: src_coords.clone(),
expires: now() + CACHE_TTL,
});
}
}
// Route using find_next_hop (bloom filter → greedy tree → None)
match self.find_next_hop(&dg.dest_addr) {
Some(next) => self.forward(next, dg),
None => {
// Cannot route — send error back to source via src_addr
self.send_error_to_source(dg, CoordsRequired {
dest_addr: dg.dest_addr,
reporter: self.node_addr,
});
}
}
}
/// Send an error signal back to the source of a SessionDatagram.
/// Creates a new SessionDatagram addressed to dg.src_addr.
fn send_error_to_source(&self, dg: &SessionDatagram, error: impl LinkError) {
let error_dg = SessionDatagram {
src_addr: self.node_addr, // We are the reporter
dest_addr: dg.src_addr, // Route back to original source
hop_limit: 64,
payload: error.encode(), // CoordsRequired or PathBroken
};
// Route the error — if we can't reach source either, drop silently
if let Some(next) = self.find_next_hop(&dg.src_addr) {
self.forward(next, &error_dg);
}
// If find_next_hop returns None for source: drop silently.
// No cascading errors.
}
}
```
### Cache Data Structures
```rust
/// Session-populated coordinate cache (TTL-based).
struct CoordCache {
entries: HashMap<NodeAddr, CacheEntry>,
max_entries: usize, // default: 50,000
ttl_ms: u64, // default: 300,000 (5 minutes)
}
struct CacheEntry {
coords: TreeCoordinate,
created_at: u64, // Unix milliseconds
last_used: u64,
expires_at: u64,
}
/// Discovery-populated coordinate cache (LRU-based, no TTL).
struct RouteCache {
entries: HashMap<NodeAddr, CachedCoords>,
max_entries: usize, // default: 10,000
}
struct CachedCoords {
coords: TreeCoordinate,
discovered_at: u64,
last_used: u64,
}
```
**CoordCache eviction**: Expired entries removed first, then LRU when cache
exceeds max_entries. Entries expire after TTL (300 seconds). Refreshed by
subsequent SessionSetup/SessionAck or DataPacket with COORDS_PRESENT.
**RouteCache eviction**: Pure LRU when cache exceeds max_entries. No automatic
TTL expiration. Invalidated explicitly on route failure.
### Cache Miss Recovery
When a router's cache entry is evicted mid-session:
```text
1. SessionDatagram{src:S, dest:D, payload:DataPacket} arrives, router can't route
2. Router R creates SessionDatagram{src:R, dest:S, payload:CoordsRequired{dest:D}}
3. Router routes the error back to S using find_next_hop(S)
4. S receives CoordsRequired, marks route to D as "cold"
5. S resends with COORDS_PRESENT flag set in DataPacket
6. Routers cache coordinates from DataPacket, forward normally
7. After N successful packets, S clears the flag
```
The crypto session remains active throughout—only routing state is refreshed.
From application perspective: one packet delayed, transparent recovery.
If the router also cannot route to S (no bloom filter hit, no cached
coordinates for S), the error is dropped silently. No cascading errors are
generated. The source will eventually detect the loss via application-layer
timeout.
### Sender State Machine
```rust
impl Sender {
fn send(&mut self, dest: NodeAddr, data: &[u8]) {
if !self.session_established(dest) {
// Need to establish crypto session first
let dest_coords = self.discover_or_cached(dest)?;
self.send_session_setup(dest, &dest_coords);
self.await_session_ack(dest)?;
}
// Check route state
let include_coords = self.route_state(dest) == RouteCold;
let data_packet = DataPacket::new(data, include_coords);
// Wrap in SessionDatagram for forwarding
let dg = SessionDatagram {
src_addr: self.node_addr,
dest_addr: dest,
hop_limit: 64,
payload: data_packet.encode(),
};
self.forward_datagram(dg);
}
/// Handle CoordsRequired received inside a SessionDatagram addressed to us.
/// The SessionDatagram.src_addr identifies the reporting router (informational).
fn handle_coords_required(&mut self, err: CoordsRequired) {
// Route cache expired at intermediate router
// Crypto session still valid - just need to re-warm route
self.mark_route_cold(err.dest_addr);
// Next send() will include coordinates
}
}
enum RouteState {
RouteWarm, // Send minimal headers
RouteCold, // Include coordinates until warm
}
```
---
## Part 5: Packet Type Summary
All session-layer messages and error signals are carried inside a
SessionDatagram envelope (34 bytes: msg_type + src_addr + dest_addr +
hop_limit). Sizes below include the SessionDatagram header.
| Type | Purpose | Size | When Used |
|------|---------|------|-----------|
| FilterAnnounce | Bloom filter propagation | ~1 KB | Topology changes |
| LookupRequest | Discover coordinates | ~300 bytes | First contact with distant node |
| LookupResponse | Return coordinates | ~400 bytes | Reply to discovery |
| SessionDatagram+SessionSetup | Warm caches + crypto init | ~230-400 bytes | Before data transfer |
| SessionDatagram+SessionAck | Confirm session + crypto | ~100-200 bytes | Session confirmation |
| SessionDatagram+DataPacket | Application data | 38 bytes + payload (minimal) | Bulk of traffic |
| SessionDatagram+DataPacket | With coordinates | ~170 bytes + payload | After CoordsRequired |
| SessionDatagram+CoordsRequired | Coords needed signal | 68 bytes | Cache miss recovery |
| SessionDatagram+PathBroken | Routing failed signal | 68+ bytes | Greedy routing local minimum |
> **Note**: SessionSetup/SessionAck sizes vary based on coordinate depth and
> whether they carry crypto handshake payloads (combined establishment per
> [fips-session-protocol.md](fips-session-protocol.md) §3.4 and §5.1).
---
## Part 6: Traffic Analysis
### Steady State (Stable Network)
- **Bloom filter traffic**: Near zero (event-driven, no changes)
- **Discovery traffic**: Rare (warm caches)
- **Session traffic**: Rare (established sessions)
- **Data traffic**: Minimal overhead (38-byte header: 34 envelope + 4 DataPacket)
### Network Churn
When nodes join/leave:
- Bloom filter updates propagate through affected peers
- Affected sessions may need re-establishment
- Discovery queries for newly-joined nodes
### Per-Node Resource Requirements
| Resource | Full Participant | Leaf-Only |
|----------|------------------|-----------|
| Bloom filter storage | d × 1 KB (d = peer count) | None |
| CoordCache (session) | 50K entries, 300s TTL | None |
| RouteCache (discovery) | 10K entries, LRU | Minimal |
| Bandwidth (idle) | < 1 KB/sec | Near zero |
---
## Known Limitations
### Known Limitation: Flood Convergence
The visited Bloom filter in LookupRequest prevents **loops** (a packet
revisiting a node on its own path) but does not prevent **convergent
duplicates** — the same request arriving at a node via different paths, each
carrying a different visited filter.
Example: Source S floods to peers B and C. Both forward to node D. D receives
two copies — one with visited={S,B}, one with visited={S,C}. Neither copy's
visited filter contains D, so D processes both. This redundancy compounds at
every well-connected node in the flood path, and the destination may generate
multiple LookupResponses.
**Required fix**: Nodes MUST maintain a short-lived cache of recently-seen
`request_id` values (retention: a few seconds, bounded by request rate limit).
On receiving a LookupRequest, check this cache first and drop duplicates before
consulting the visited filter. This is referenced in the gossip protocol spec
(section 4.4, rate limiting) but needs to be elevated to a protocol
requirement, not an optimization.
### Known Limitation: Capacity-Blind Routing
Both bloom filter candidate selection and greedy tree routing currently
select next hops without considering link quality. When multiple peers
can reach a destination, the selection is based on tree distance and
node address tie-breaking — purely topological metrics that ignore link
capacity, latency, and loss.
This creates a problem when a topologically short but low-capacity link
exists alongside a longer but high-capacity path. The routing algorithm
will prefer the topologically closer peer, potentially saturating a slow
link while a higher-capacity path goes underutilized.
**Proposed mitigation**: Each node locally measures the quality of its
direct peer links (RTT, bandwidth, loss) and incorporates this into a
`link_cost()` metric. The next-hop selection uses a composite ordering
of `(link_cost, tree_distance, node_addr)` — link quality takes priority
over topological distance.
The `link_cost()` interface is implemented (currently returning a constant),
ready to be populated with real measurements using an established link
quality algorithm (ETX, Babel composite metric, etc.).
This requires no protocol changes — link quality is measured locally, not
advertised. Self-reported cost claims are intentionally excluded from the
protocol to prevent adversarial traffic attraction (a node advertising
artificially low costs to become a transit point for surveillance or
disruption). Only locally-measured, first-hop metrics are used.
## Enhancement Opportunity: Discovery Path Accumulation
The LookupRequest flood naturally finds the lowest-latency path to the
destination — the first copy to arrive traveled the fastest route. Currently
this path information is discarded; only the destination's tree coordinates
survive in the LookupResponse.
### Concept
Each node forwarding a LookupRequest appends a signed path entry:
```text
PathEntry {
node_addr: NodeAddr, // 16 bytes, forwarding node
signature: Signature, // 64 bytes, signs
// (request_id || position || node_addr)
}
```
The destination includes the accumulated path in the LookupResponse alongside
the existing `target_coords`. Per-hop signatures prevent path fabrication — a
malicious node cannot insert fake hops or claim a path it didn't traverse.
### Potential Uses
**Source peer bias**: The source examines the first hop of the discovered path
to learn which of its direct peers leads to the empirically fastest route to
the destination. This biases forwarding for that destination toward that peer,
complementing tree distance and local link quality measurements. The source can
verify this directly since the first hop is a direct peer.
**Intermediate router hints**: Routers along the path can verify their own
position and adjacent hops (which should be direct peers). This gives them
empirical data about which peer directions lead toward specific destinations,
potentially informing their own forwarding decisions for future packets.
**Coordinate cache seeding**: Intermediate routers on the discovered path learn
about both endpoints before data traffic begins, enabling pre-warming of
coordinate caches and reducing CoordsRequired errors on the first data packets.
### Tradeoffs
- **Size overhead**: 80 bytes per hop in the LookupRequest. An 8-hop path adds
640 bytes. Acceptable for a one-time discovery message, not suitable for data
packets.
- **Staleness**: The path is a snapshot. Nodes may disconnect or links may
degrade after discovery. The path should be treated as a hint, not a
commitment. Greedy coordinate-based routing remains the primary forwarding
mechanism.
- **Latency vs capacity**: The fastest flood path is the lowest-latency path,
which is not necessarily the highest-capacity path. For bulk transfers, a
slower but higher-bandwidth path may be preferable. The path signal is most
useful for latency-sensitive traffic.
### Interaction with Request Deduplication
With strict `request_id` dedup at every node, the destination receives exactly
one request via the fastest path. If multiple candidate paths are desired (for
failover or load balancing), the destination could be exempted from dedup to
accept the first N arrivals, at the cost of generating multiple responses.
## Open Questions
1. **Coordinate compression**: Can tree coordinates be compressed for smaller
SessionSetup messages? (e.g., delta encoding, shorter node ID representation)
2. **Multi-path routing**: How to handle multiple valid paths? Load balancing?
Failover?
3. **Asymmetric paths**: S→D and D→S may traverse different routers. Is this
acceptable or should paths be symmetric?
4. **Gateway /64 prefixes**: How do subnet prefixes interact with Bloom filters
and discovery? One filter entry per gateway regardless of devices behind it?
5. **Cache sizing**: What's the right cache size for different node roles?
Core nodes vs. edge nodes?
6. **Mobility**: When a node changes tree position (new parent), how quickly
do sessions recover? Should nodes announce position changes?
---
## References
- [fips-intro.md](fips-intro.md) — Overall FIPS architecture
- [fips-gossip-protocol.md](fips-gossip-protocol.md) — Wire formats for TreeAnnounce, FilterAnnounce, Lookup
- [fips-session-protocol.md](fips-session-protocol.md) — Traffic flow, crypto sessions, terminology
- [fips-wire-protocol.md](fips-wire-protocol.md) — Link-layer transport and Noise IK handshake
- [fips-transports.md](fips-transports.md) — Transport protocol characteristics
- [spanning-tree-dynamics.md](spanning-tree-dynamics.md) — Tree protocol dynamics and convergence
docs/design/fips-session-layer.md
+402
View File
@@ -0,0 +1,402 @@
# FIPS Session Protocol (FSP)
The FIPS Session Protocol is the top protocol layer in the FIPS stack. It sits
above the FIPS Link Protocol (FLP) and below applications (native FIPS API or
IPv6 adapter). FSP provides end-to-end authenticated, encrypted datagram
delivery between any two FIPS nodes, regardless of how many intermediate hops
separate them.
## Role
FSP manages end-to-end communication sessions between FIPS nodes identified by
their public keys (npubs). Each session provides:
- **End-to-end encryption**: Payload confidentiality independent of how many
intermediate nodes handle the traffic
- **Mutual authentication**: Both parties prove they control the private key
for their claimed identity
- **Replay protection**: Counter-based nonces with sliding window, tolerant of
UDP packet loss and reordering
- **Transport independence**: Sessions survive transport changes, route
changes, and address changes — they are bound to npub identities, not to
transport paths
FSP is a datagram session protocol. It provides encrypted datagrams, not
reliable streams. There is no FIPS equivalent of TCP; if applications need
reliability, ordering, or flow control, they provide it themselves (typically
by running TCP over the FIPS IPv6 adapter).
## Services Provided to Applications
Applications access the FIPS mesh through two interfaces, both served by FSP:
### Native FIPS API
Applications address destinations directly by npub or public key. The FIPS
stack resolves the destination's node_addr, establishes or reuses a session,
encrypts the payload, and routes through FLP. No DNS involvement.
### IPv6 Adapter
Unmodified IPv6 applications use a TUN device with `fd::/8` routing. A local
DNS service maps npub → IPv6 address and primes the identity cache. Packets
arriving at the TUN are translated to FIPS datagrams and routed through FSP.
See [fips-ipv6-adapter.md](fips-ipv6-adapter.md) for the IPv6 adaptation
layer.
### What Applications Get
- **Authenticated datagram delivery**: Each datagram is encrypted and
authenticated with session keys bound to both parties' npubs
- **Session transparency**: Sessions are established on demand and maintained
automatically. Applications send packets; FSP handles session setup,
encryption, and teardown.
- **Endpoint identity**: Applications address destinations by npub. The FIPS
address is the public key.
### What Applications Do Not Get
- **Reliability**: Datagrams may be lost, duplicated, or delivered out of
order. FSP provides no retransmission or ordering.
- **Path MTU discovery**: FSP does not signal MTU to applications. The IPv6
adapter handles MTU enforcement via ICMP Packet Too Big and TCP MSS
clamping.
- **Congestion control**: FSP does not throttle traffic. Applications running
TCP over IPv6 get TCP's congestion control; native API applications must
manage their own sending rate.
## Services Required from FLP
FSP treats FLP as a black box providing three services. FSP knows nothing about
transports, transport addresses, links, peers, spanning trees, coordinates,
bloom filters, hop counts, or network topology.
### SessionDatagram Forwarding
FLP accepts a SessionDatagram (source node_addr, destination node_addr, hop
limit, payload) and delivers it best-effort toward the destination. Delivery
may traverse multiple hops, each with independent link encryption.
### Error Signaling
FLP signals routing failures asynchronously:
- **CoordsRequired**: A transit node lacks the destination's tree coordinates.
FSP responds by re-initiating discovery and resetting the coordinate warmup
strategy.
- **PathBroken**: Greedy routing reached a dead end. FSP responds by
re-discovering the destination's current coordinates and resetting warmup.
Both signals are generated by transit nodes (not the destination) and travel
back to the source inside a new SessionDatagram. They are plaintext (not
end-to-end encrypted) because transit nodes have no session with the source.
### Local Delivery
When a SessionDatagram arrives with a destination node_addr matching the local
node, FLP delivers it to FSP for session-layer processing.
## Session Lifecycle
### Session Establishment
Sessions are established on demand when the first datagram needs to be sent to
a destination with no existing session.
FSP uses Noise IK for session key agreement. The initiator knows the
destination's npub (from DNS lookup or native API); the responder learns the
initiator's identity from the handshake. This is the same asymmetry as
link-layer peer connections.
The handshake is carried in SessionSetup and SessionAck messages:
1. **Initiator** sends SessionSetup containing Noise IK msg1 and both
parties' tree coordinates
2. **Responder** processes msg1, learns initiator identity, sends SessionAck
containing Noise IK msg2 and its own coordinates
3. Both parties derive identical symmetric session keys
Packets that trigger session establishment are queued (with bounded buffer)
and transmitted after the session is established.
### Self-Bootstrapping
SessionSetup is self-bootstrapping for routing. It carries the source's and
destination's tree coordinates in the clear (not inside the Noise payload).
As the message transits intermediate nodes, each node caches these coordinates,
warming the path for subsequent DataPackets that carry only addresses (no
coordinates).
SessionAck carries the responder's coordinates back along the reverse path,
warming caches in the other direction.
### Simultaneous Initiation
When both nodes attempt to establish a session simultaneously ("crossing
hellos"), a deterministic tie-breaker resolves the conflict:
- If `local_node_addr < remote_node_addr`: Continue as initiator, ignore
incoming setup
- If `local_node_addr > remote_node_addr`: Abort own initiation, switch to
responder role
This ensures exactly one handshake completes.
### Data Transfer
Once established, sessions carry DataPacket messages containing encrypted
application data. Each DataPacket includes:
- An explicit 8-byte counter for replay protection (used as the AEAD nonce)
- A flags byte (including COORDS_PRESENT for cache warming)
- The encrypted payload
### Session Idle Timeout
Sessions that see no traffic for a configurable duration (default 90s) are
torn down. When traffic resumes, a new session is established automatically.
The idle timeout is deliberately shorter than the coordinate cache TTL (300s).
This ordering ensures that when traffic stops and the session tears down, the
transit node coordinate caches are still warm when a new session is established.
The fresh SessionSetup re-warms the caches, maintaining routing continuity.
### Session Independence from Transport
Sessions exist above the routing layer and are bound to npub identities, not
transport addresses or routing paths. A session survives:
- Transport failover (UDP → Ethernet → back to UDP)
- Route changes (different intermediate hops)
- Transport address changes (IP address or port changes)
- Topology changes (direct peer becomes multi-hop or vice versa)
## End-to-End Encryption
### Noise IK Pattern
FSP uses the same Noise IK pattern as FLP link encryption, but with
independent keys and sessions. The full Noise descriptor is
`Noise_IK_secp256k1_ChaChaPoly_SHA256`.
The IK pattern:
- **msg1** (`→ e, es, s, ss`): Initiator sends ephemeral key, encrypts static
key to responder. Four DH operations establish session keys.
- **msg2** (`← e, ee, se`): Responder sends ephemeral key. Both parties now
share identical session keys.
After the handshake, Noise produces two directional symmetric keys
(`send_key`, `recv_key`) used with ChaCha20-Poly1305 for all subsequent data.
### Cryptographic Primitives
| Component | Choice | Notes |
| --------- | ------ | ----- |
| Curve | secp256k1 | Nostr-native |
| DH | ECDH on secp256k1 | Standard EC Diffie-Hellman |
| Cipher | ChaCha20-Poly1305 | AEAD, same as NIP-44 |
| Hash | SHA-256 | Nostr-native |
| Key derivation | HKDF-SHA256 | Standard Noise KDF |
These choices prioritize compatibility with the Nostr cryptographic stack.
### secp256k1 Parity Normalization
Nostr npubs encode x-only public keys (32 bytes, no y-coordinate parity). The
Noise IK pre-message mixes the responder's static key as a 33-byte compressed
key, and the default secp256k1 ECDH hash includes a parity-dependent version
byte.
Both operations are normalized to be parity-independent: the pre-message hash
uses even parity (`0x02` prefix), and ECDH hashes only the x-coordinate of the
result point. This ensures handshakes succeed regardless of the responder's
actual key parity.
### Privacy Note
Noise IK does not provide initiator anonymity if the responder's static key is
compromised. An attacker who obtains the responder's nsec can decrypt the
initiator's identity from captured handshake messages. Noise XK would protect
initiator identity in this scenario but requires an additional round-trip (3
handshake messages vs. 2). The privacy/latency tradeoff may be revisited with
deployment experience.
### Data Packet Authentication
FSP uses AEAD authentication only — no per-packet signatures. The Noise
handshake binds session keys to both parties' static keys, so only holders of
the corresponding nsecs can derive the session keys. This provides implicit
authentication for every packet, matching WireGuard and Lightning's approach.
### Forward Secrecy
Ephemeral keys in the Noise handshake provide forward secrecy. Compromise of
static keys (nsec) does not reveal past session keys, because session keys are
derived in part from ephemeral-ephemeral DH (`ee`), and ephemeral keys are
discarded after the handshake.
## Replay Protection
FSP uses explicit 8-byte counters on the wire for replay protection. Each side
maintains a monotonically increasing send counter, transmitted with every
DataPacket. The receiver maintains a sliding window (2048-entry bitmap)
tracking which counters have been seen.
This design is critical for operation over unreliable transports. Under UDP
packet loss or reordering, implicit nonce counters (where the receiver
increments on each decrypt attempt) would desynchronize permanently — a failed
`decrypt()` increments the nonce, and the desync grows with each lost packet.
Explicit counters allow the receiver to decrypt any packet independently,
regardless of what packets were lost or reordered.
The same `ReplayWindow` and `decrypt_with_replay_check()` implementation is
used at both the link and session layers.
## COORDS_PRESENT Warmup Strategy
Session establishment (SessionSetup/SessionAck) warms transit node coordinate
caches along the path. But coordinate caches have a finite TTL (default 300s),
and entries may be evicted under memory pressure. When a transit node's cache
entry expires, it cannot forward DataPackets (which carry only addresses, not
coordinates) and sends a CoordsRequired error.
FSP uses a warmup-then-reactive strategy to keep transit caches populated:
### Warmup Phase
After session establishment, the first N DataPackets (configurable, default 5)
include both source and destination coordinates via the COORDS_PRESENT flag.
Transit nodes cache these coordinates as packets pass through, reinforcing the
path established by SessionSetup.
### Steady State
After the warmup count is reached, FSP clears the COORDS_PRESENT flag and
sends minimal DataPackets (4-byte header instead of ~136 bytes with
coordinates). Transit nodes serve from their coordinate caches.
### Reactive Recovery
When FSP receives a CoordsRequired signal:
1. The warmup counter resets — subsequent DataPackets include coordinates again
2. A new LookupRequest may be initiated to rediscover the destination's
current coordinates
3. When the LookupResponse arrives for an established session, the warmup
counter resets again (handling the timing gap where warmup packets might
fire before transit caches are repopulated by discovery)
When FSP receives a PathBroken signal:
1. A LookupRequest is initiated to discover the destination's current
coordinates (which may have changed due to topology change)
2. The warmup counter resets
Both signals are rate-limited at transit nodes (100ms per destination) to
prevent storms during topology changes.
### Warmup State Machine
```text
┌──────────────┐
│ WARMUP │ ◄── Send first N packets with coords
└──────┬───────┘
│ N packets sent without CoordsRequired
▼
┌──────────────┐
│ MINIMAL │ ◄── Send packets without coords
└──────┬───────┘
│ CoordsRequired or PathBroken received
▼
┌──────────────┐
│ WARMUP │ ◄── Counter reset, send coords again
└──────────────┘
```
## Identity Cache
The identity cache maps FIPS address prefix (15 bytes, the `fd::/8` IPv6
address minus the `fd` prefix) to `(NodeAddr, PublicKey)`. This cache is
needed only when using the IPv6 adapter — the native FIPS API provides the
public key directly.
The mapping is deterministic (derived from the public key via SHA-256) and
never becomes stale. The cache uses LRU-only eviction bounded by a
configurable size (default 10K entries). There is no TTL — entries are evicted
only when the cache is full and space is needed for a new entry.
Cache population mechanisms:
- **DNS lookup**: The primary path. Resolving `npub1xxx...xxx.fips` derives
the IPv6 address and populates the identity cache.
- **Inbound traffic**: Authenticated sessions from other nodes populate the
cache with their identity information.
## Coordinate Cache
The coordinate cache maps `NodeAddr → TreeCoordinate` and is the critical
data structure that enables efficient multi-hop routing. Without cached
coordinates for a destination, FLP cannot make forwarding decisions and must
either fall back to bloom-filter-only routing or signal CoordsRequired.
### Unified Cache
The coordinate cache is a single unified cache (merged from previously
separate coord_cache and route_cache). All coordinate sources — SessionSetup
transit, COORDS_PRESENT DataPackets, LookupResponse — write to the same cache.
### Eviction Policy
- **TTL-based expiration**: Entries expire after a configurable duration
(default 300s)
- **Refresh on use**: Active routing through a cache entry resets its TTL,
keeping hot entries alive
- **LRU eviction**: When the cache is full, least recently used entries are
evicted first
- **Flush on parent change**: When the local node's tree parent changes, the
entire coordinate cache is flushed. Tree parent changes mean the node's own
coordinates have changed, making cached coordinates for other nodes
potentially stale for routing purposes.
### Timer Ordering
Cache and session timers are ordered so that idle sessions tear down before
transit caches expire:
| Timer | Default | Purpose |
| ----- | ------- | ------- |
| Session idle timeout | 90s | Tear down unused sessions |
| Coordinate cache TTL | 300s | Expire stale coordinates |
| DNS TTL | 300s | Expire DNS resolutions |
When traffic stops: the session tears down at 90s. When traffic resumes: DNS
re-resolves the identity, a fresh SessionSetup carries coordinates, and transit
node caches (still within their 300s TTL) are re-warmed.
## Implementation Status
| Feature | Status |
| ------- | ------ |
| Session establishment (Noise IK) | **Implemented** |
| End-to-end encryption (ChaCha20-Poly1305) | **Implemented** |
| Explicit counter replay protection | **Implemented** |
| COORDS_PRESENT warmup-then-reactive | **Implemented** |
| Identity cache (LRU-only) | **Implemented** |
| Coordinate cache (unified, TTL + refresh) | **Implemented** |
| Session idle timeout | **Implemented** |
| CoordsRequired handling | **Implemented** |
| PathBroken handling | **Implemented** |
| Simultaneous initiation tie-breaker | **Implemented** |
| Flush coord cache on parent change | **Implemented** |
| Rekey | Planned |
| Path MTU discovery | Planned |
## References
- [fips-intro.md](fips-intro.md) — Protocol overview and architecture
- [fips-link-layer.md](fips-link-layer.md) — FLP specification (below FSP)
- [fips-ipv6-adapter.md](fips-ipv6-adapter.md) — IPv6 adaptation layer (above FSP)
- [fips-mesh-operation.md](fips-mesh-operation.md) — Routing, discovery, and
error recovery
- [fips-wire-formats.md](fips-wire-formats.md) — Wire format reference for all
session message types
docs/design/fips-session-protocol.md
-848
View File
@@ -1,848 +0,0 @@
# FIPS Session Protocol Flow
## Overview
This document captures design considerations for FIPS protocol message flow,
including peer discovery, authentication, tree announcements, and data routing.
### Access to the FIPS Datagram Service
Applications can access FIPS datagram delivery through two interfaces:
- **Native FIPS API**: Applications address destinations directly by npub or
public key. The FIPS stack resolves the destination's node_addr and routes
without any DNS involvement. This is the preferred interface for
FIPS-aware applications.
- **IPv6 adapter (TUN interface)**: Unmodified IPv6 applications use a TUN
device with `fd::/8` routing. Because IPv6 addresses are one-way hashes of
public keys, a local DNS service maps npub → IPv6 address and primes the
identity cache so the TUN can route arriving packets.
The remainder of this document details the sequence of actions initiated through
the IPv6 adapter path. The native FIPS API bypasses sections 1.1–1.4 (DNS
entry point, identity cache) entirely, entering the flow at route discovery.
---
## 1. Application-Initiated Traffic Flow (IPv6 Adapter)
> **Note**: This section applies to traditional IP-based applications using the
> TUN interface. Applications using the native FIPS API address destinations
> directly by npub or public key; routing proceeds from there without DNS.
Traffic flow begins at the application layer with a DNS query, which triggers
a cascade of events through the FIPS stack.
### 1.1 DNS as Entry Point
An application wants to send IPv6 traffic to another FIPS node, identified by
an npub. The flow:
1. **DNS Query**: Application queries the local FIPS DNS service for the npub
mapping to an IPv6 address
2. **FIPS DNS service** performs two functions:
- **Address derivation**: Converts the npub to an identity and derives the
corresponding `fd::/8` IPv6 address
- **Cache priming**: Stores the identity mapping (IPv6 address ↔ npub ↔ node_addr)
in the local FIPS routing cache
3. **DNS Response**: Returns the derived IPv6 address to the application
4. **Packet Transmission**: Application sends IPv6 packet to the returned address,
which routes to the TUN interface via the `fd::/8` route
5. **TUN Processing**: When the packet arrives at the TUN, FIPS already has the
cached mapping from the DNS lookup, enabling immediate routing decisions
6. **Note**: This identity cache is only necessary when using the FIPS IPv6 shim
### 1.2 Design Rationale
Using DNS as the trigger point ensures the routing cache is populated *before*
packets arrive. This avoids:
- Blocking packets while performing identity lookups
- Packet drops during cold-cache scenarios
- Complex async lookup machinery in the hot path
The DNS server acts as a "routing intent" signal - if an application queries
for a destination, it likely intends to send traffic there.
### 1.3 DNS Name Format
NPUBs are represented as DNS names in the format:
```text
npub1xxxxxx...xxxxx.fips
```
The FIPS DNS server recognizes names ending in `.fips` and extracts the npub
for address derivation.
### 1.4 Identity Cache Lifetime
The identity cache (IPv6 address ↔ npub ↔ node_addr) has the following lifetime
semantics:
- **Configurable timeout**: Cache entries expire after a configured duration
- **Traffic refresh**: Timer resets to zero whenever traffic is sent to that
destination (LRU-style keep-alive)
- **TTL relationship**: Cache timeout MUST be longer than DNS TTL
The TTL constraint ensures that while an application believes its DNS resolution
is valid (within TTL), the corresponding FIPS routing entry remains present.
Example: DNS TTL = 300s, Cache timeout = 600s.
```text
DNS query → cache entry created (timeout = 600s)
...traffic... → timeout reset to 600s
...traffic... → timeout reset to 600s
DNS TTL expires (300s) → app may re-query, but cache still valid
...no traffic for 600s...
Cache entry expires
```
### 1.5 Traffic Without Prior DNS Lookup
A packet may arrive at the TUN for an `fd::/8` destination without a prior
DNS lookup (cached address, manual configuration, etc.). Since address
derivation is one-way (SHA-256), the npub cannot be recovered from the address,
and without the npub we cannot determine the node_addr needed for routing.
FIPS returns ICMPv6 Destination Unreachable (Code 0: No route to destination)
for packets to unknown addresses. The identity cache must be populated before
traffic can be routed.
Known cache population mechanisms:
- DNS lookup (primary path, described above)
- Inbound traffic from authenticated peers
---
## 2. TUN Reader Processing
After DNS resolution, the application sends an IPv6 datagram to the destination
address. The kernel routes it to the TUN interface (via the `fd::/8` route),
where the FIPS TUN reader receives it.
### 2.1 Packet Arrival
```text
Application
│
▼
IPv6 datagram (src=local_addr, dst=target_addr)
│
▼
Kernel routing table: fd::/8 → fips0
│
▼
TUN reader receives raw IPv6 packet
```
### 2.2 TUN Reader Actions
On receiving a packet, the TUN reader:
1. **Validate IPv6 header**: Version = 6, payload length sane, etc.
2. **Extract destination address**: The `fd::/8` address from the IPv6 header
3. **Identity cache lookup**: Query cache for destination address
- **Miss**: Return ICMPv6 Destination Unreachable (see §1.5)
- **Hit**: Proceed with routing
4. **Retrieve routing identity**: Cache hit provides:
- `npub`: The Nostr public key of the destination
- `node_addr`: SHA-256(pubkey), used for spanning tree routing
5. **Session lookup**: Check for existing FIPS session with destination npub
- **Hit**: Use existing session for encryption/signing
- **Miss**: Initiate session establishment (see §3)
6. **Route determination**: Using node_addr, determine the next hop peer:
- Check route cache for destination's spanning tree coordinates
- If cache miss, initiate route discovery (see §4.4)
- Select next hop via greedy routing toward destination coordinates
7. **Packet forwarding**: Encapsulate and send via appropriate transport:
- Encrypt payload with session keys (end-to-end)
- Wrap in link-layer frame for next hop peer
- Encrypt with link keys and transmit via peer's transport
---
## 3. FIPS Sessions
A FIPS session represents a bidirectionally authenticated, encrypted channel
between two FIPS nodes.
### 3.1 Session Properties
Each session contains:
- **Peer identity**: The remote node's npub and node_addr
- **Symmetric session keys**: Directional keys for encryption (send_key, recv_key)
- **Nonce counters**: Per-direction counters for replay protection
Payloads within a session are:
1. **Encrypted** with the session key (provides confidentiality)
2. **Authenticated** via AEAD tag (session keys bound to npub identities)
Authentication derives from the Noise IK handshake binding session keys to
both parties' static keys. See §6 for cryptographic details.
### 3.2 Session Establishment Trigger
When the TUN reader has a packet for a destination with no existing session:
```text
TUN reader
│
├─► Identity cache lookup → node_addr
│
├─► Session lookup (by npub) → MISS
│
└─► Initiate session establishment
```
Packets that trigger session establishment are queued (with bounded buffer
management) and transmitted after the session is established.
### 3.3 Session Independence from Transport
FIPS sessions exist above the routing layer. A session between two npubs
survives:
- Transport failover (UDP → WiFi → back to UDP)
- Route changes (different intermediate hops)
- Transport address changes on either end (WiFi → LTE → WiFi)
The session is bound to **npub identities**, not transport addresses or routing
paths. This allows FIPS endpoints to roam over their transports as needed while
maintaining an established session.
### 3.4 Session Establishment Flow
FIPS uses Noise IK for session establishment. The initiator knows the
destination's npub; the responder learns the initiator's identity from the
handshake. This is the same asymmetry as link-layer connections.
The handshake is carried inside SessionSetup/SessionAck messages (see §5.1),
which also establish routing session state at intermediate nodes.
### 3.5 Simultaneous Session Initiation (Crossing Hellos)
When both nodes attempt to establish a session simultaneously, a deterministic
tie-breaker resolves the conflict using node_addr ordering (consistent
with the link-layer cross-connection tie-breaker):
- If local node_addr < remote node_addr: Continue as initiator, ignore incoming setup
- If local node_addr > remote node_addr: Abort own initiation, switch to responder role
This ensures exactly one handshake completes with minimal wasted effort.
---
## 4. FIPS Mesh Routing
Below the session layer, all FIPS packets (session handshake messages, encrypted
payloads, control traffic) must be routed through the mesh to their destination.
See [fips-routing.md](fips-routing.md) for the full routing design.
### 4.1 Routing Overview
FIPS routing combines three mechanisms:
1. **Bloom filters**: Fast reachability lookup for nearby destinations
2. **Discovery protocol**: Query-based lookup for distant destinations
3. **Greedy tree routing**: Coordinate-based forwarding using spanning tree position
The routing layer maintains coordinate caches mapping `node_addr →
TreeCoordinate` (see [fips-routing.md](fips-routing.md) §4 for details).
Cache hits enable immediate routing via `find_next_hop()`; cache misses
trigger route discovery via bloom filter queries or LookupRequest flooding.
### 4.2 Packet Handling During Discovery
Packets are queued (with bounded buffer) while route discovery is in progress
and transmitted once coordinates are obtained.
### 4.3 Coordinate Cache Lifetime
Two caches store `node_addr → TreeCoordinate` mappings (see
[fips-routing.md](fips-routing.md) §4):
- **CoordCache** (session-populated): TTL-based eviction (300s default), 50K entries
- **RouteCache** (discovery-populated): LRU eviction only, 10K entries
Neither cache stores next-hop information — routing decisions are computed at
lookup time by `find_next_hop()` using cached coordinates.
---
## 5. Route Cache Warming
### 5.1 Initial Warming via Handshake
The crypto session handshake (SessionSetup/SessionAck) warms route caches at
intermediate routers as it transits. Each message carries the sender's
coordinates; routers extract and cache `node_addr → TreeCoordinate` in the
CoordCache. After the handshake completes, data packets use minimal 38-byte
headers (34 envelope + 4 DataPacket) and routers forward via `find_next_hop()`.
### 5.2 Cache Miss Recovery
When an intermediate router's cache entry expires or is evicted, it cannot
forward data packets (which carry only addresses, not coordinates). The router
returns a CoordsRequired error to the sender.
The crypto session remains valid—only the routing state is lost. Recovery uses
coords-on-demand: data packets include an optional coordinates field. When the
sender receives CoordsRequired:
1. Sender marks the route as "cold"
2. Subsequent data packets include coordinates (flag bit set)
3. Routers along the path cache coordinates as packets transit
4. Once route is warm again, sender clears the flag and resumes minimal headers
This avoids a full SessionSetup round-trip for what is purely a routing cache
refresh.
### 5.3 DataPacket Coordinate Flag
The DataPacket `flags` field includes a `COORDS_PRESENT` bit:
| Bit | Meaning |
|-----|------------------------------------------------------|
| 0 | COORDS_PRESENT - coordinates follow the fixed header |
When set, the packet includes `src_coords` and `dest_coords` after the standard
header fields. Routers process these coordinates the same way as SessionSetup:
cache both directions and forward using greedy routing.
### 5.4 Sender State Machine
```text
┌──────────────┐
│ WARM │ ◄── Normal: send minimal headers
└──────┬───────┘
│ CoordsRequired received
▼
┌──────────────┐
│ COLD │ ◄── Send packets with coords
└──────┬───────┘
│ N packets sent successfully
▼
┌──────────────┐
│ WARM │
└──────────────┘
```
The sender transitions back to WARM after sending a configurable number of
packets with coordinates (e.g., 3) without receiving CoordsRequired. This
provides confidence that caches along the path are populated.
---
## 6. Session-Layer Encryption
FIPS uses two independent Noise Protocol handshakes at different layers:
| Layer | Scope | Pattern | Purpose |
|---------|-------------|----------|-------------------------------------------|
| Link | Hop-by-hop | Noise IK | Authenticate peers, encrypt link |
| Session | End-to-end | Noise IK | Authenticate endpoints, encrypt payload |
Both use `Noise_IK_secp256k1_ChaChaPoly_SHA256` with the same cryptographic
primitives, but with separate keys and sessions.
> **Privacy note**: Noise IK does not provide initiator anonymity if the
> responder's static key is compromised—an attacker who obtains the responder's
> nsec can decrypt the initiator's identity from captured handshake messages.
> Noise XK would protect initiator identity in this scenario, but requires an
> additional round-trip (3 handshake messages vs 2), increasing session setup
> from 3 packets to 4. Further deployment experience is needed to evaluate
> whether the privacy benefit justifies the latency cost.
### 6.1 Why Two Layers?
**Link encryption** protects against passive observers on each hop but allows
intermediate nodes to see routing information (destination address).
**Session encryption** protects the actual payload end-to-end. Intermediate
nodes forward opaque ciphertext without being able to read the contents.
### 6.2 Session Noise Handshake
The session-layer Noise IK handshake is carried inside `SessionSetup`/`SessionAck`
messages, which themselves travel through the link-encrypted channel:
```text
Initiator knows destination npub (from DNS lookup)
│
▼
SessionSetup { coords, handshake_payload: Noise IK msg1 }
│
▼ (travels through link-encrypted hops)
│
Responder processes msg1, learns initiator identity
│
▼
SessionAck { coords, handshake_payload: Noise IK msg2 }
│
▼
Session keys established (independent of link keys)
```
### 6.3 Cryptographic Primitives
Both link and session layers use the same cryptographic stack:
| Component | Choice | Notes |
|----------------|---------------------|----------------------------|
| Curve | secp256k1 | Nostr-native |
| DH | ECDH on secp256k1 | Standard EC Diffie-Hellman |
| Cipher | ChaCha20-Poly1305 | AEAD, same as NIP-44 |
| Hash | SHA-256 | Nostr-native |
| Key derivation | HKDF-SHA256 | Standard Noise KDF |
These choices prioritize compatibility with existing Nostr infrastructure.
Secp256k1 and SHA-256 are already used for Nostr identities, and
ChaCha20-Poly1305 matches NIP-44 encryption. Lightning's BOLT 8 provides a
proven reference for adapting Noise Protocol to secp256k1.
**secp256k1 parity normalization**: Nostr npubs encode x-only public keys
(32 bytes, no y-coordinate parity). The Noise IK pre-message mixes the
responder's static key as a 33-byte compressed key, and the default
secp256k1 ECDH hash includes a parity-dependent version byte. Since the
initiator may only have the x-only key, both operations are normalized to be
parity-independent: the pre-message hash uses even parity (`0x02` prefix),
and ECDH hashes only the x-coordinate of the result point. This ensures
handshakes succeed regardless of the responder's actual key parity. See
`secp256k1-parity-fix.md` for detailed analysis.
### 6.4 Handshake Integration with SessionSetup
The Noise handshake messages embed in SessionSetup/SessionAck, which are
carried inside a SessionDatagram envelope that provides addressing:
```text
SessionDatagram {
// Addressing (used by routers for forwarding and error routing)
src_addr: NodeAddr,
dest_addr: NodeAddr,
hop_limit: u8,
// Payload: SessionSetup
payload: SessionSetup {
// Routing portion (processed by routers for cache warming)
src_coords: Vec<NodeAddr>,
dest_coords: Vec<NodeAddr>,
// Crypto portion (opaque to routers, processed by destination)
handshake_payload: Vec<u8>, // Noise IK message 1
}
}
SessionDatagram {
src_addr: NodeAddr, // Responder
dest_addr: NodeAddr, // Original sender
hop_limit: u8,
payload: SessionAck {
// Routing portion
src_coords: Vec<NodeAddr>, // Responder's coordinates
// Crypto portion
handshake_payload: Vec<u8>, // Noise IK message 2
}
}
```
### 6.5 Session Keys
After handshake completion, Noise produces two symmetric keys:
- `send_key`: For encrypting outbound packets
- `recv_key`: For decrypting inbound packets
These are used with ChaCha20-Poly1305 for all subsequent data packets.
### 6.6 Nonce Management
FIPS uses counter-based nonces for ChaCha20-Poly1305. Each side maintains a
64-bit send counter, incremented per packet. No coordination is needed since
keys are directional. The counter also enables replay detection by rejecting
packets with nonce ≤ last seen.
### 6.7 Forward Secrecy
The ephemeral keys (`e` in Noise notation) provide forward secrecy:
- Compromise of static keys (npub/nsec) doesn't reveal past session keys
- Each session has unique ephemeral keys
- Session keys derived from ephemeral-ephemeral DH (`ee`)
### 6.8 Reference: Lightning BOLT 8
Lightning's adaptation of Noise for secp256k1 (BOLT 8) provides a proven
reference implementation:
- Uses Noise XK pattern (different from our IK)
- Same secp256k1 + ChaCha20-Poly1305 + SHA-256 stack
- Handles the secp256k1 ECDH correctly
- Open source implementations available in multiple languages
FIPS can reference BOLT 8's cryptographic details while using the IK pattern.
### 6.9 Data Packet Authentication
**Decision**: Use AEAD authentication only (no per-packet signatures).
The Noise handshake binds session keys to both parties' static keys. After
handshake completion:
- Session keys are cryptographically tied to both npubs
- AEAD (ChaCha20-Poly1305) provides integrity and authenticity
- Only the holder of the session key can produce valid ciphertext
- Session keys can only be derived by holders of the corresponding nsecs
Per-packet signatures would add:
- 64 bytes overhead per packet
- Signing CPU cost (secp256k1 Schnorr)
- Verification CPU cost at receiver
Since Noise already provides authentication through key binding, signatures
are redundant. This matches WireGuard and Lightning's approach.
---
## 7. Peer Connection Establishment
Before any session-layer traffic can flow, nodes must establish authenticated
link-layer connections with their peers using Noise IK. See
[fips-wire-protocol.md](fips-wire-protocol.md) for the complete wire protocol
specification including handshake flow, session lifecycle, index management,
roaming support, and transport-specific considerations.
After successful Noise IK handshake:
1. **Link encrypted**: All subsequent messages use AEAD encryption
2. **TreeAnnounce exchange**: Both peers send their current spanning tree state
3. **FilterAnnounce exchange**: Both peers send their bloom filters
4. **Peer is Active**: Can now participate in routing and forwarding
The first TreeAnnounce from a new peer may trigger parent reselection if that
peer offers a better path to root. See [fips-gossip-protocol.md](fips-gossip-protocol.md)
for TreeAnnounce and FilterAnnounce wire formats.
---
## 8. Session Layer Wire Format
Session layer messages are carried inside `SessionDatagram` (type 0x40) at the
link layer. The session datagram provides source and destination addressing for
multi-hop forwarding: it is encrypted hop-by-hop with link keys, and the inner
payload is either encrypted end-to-end with session keys (for data traffic) or
plaintext (for link-layer error signals from transit routers).
### 8.0 SessionDatagram Envelope
The SessionDatagram is a link-layer message (type 0x40) that carries all
routable traffic through the mesh. It provides the source and destination
addresses that transit routers use for forwarding decisions.
```text
┌─────────────────────────────────────────────────────────────────────────────┐
│ SESSION DATAGRAM (0x40) │
├────────┬──────────────────┬───────────┬─────────────────────────────────────┤
│ Offset │ Field │ Size │ Description │
├────────┼──────────────────┼───────────┼─────────────────────────────────────┤
│ 0 │ msg_type │ 1 byte │ 0x40 (link-layer SessionDatagram) │
│ 1 │ src_addr │ 16 bytes │ Source node_addr │
│ 17 │ dest_addr │ 16 bytes │ Destination node_addr │
│ 33 │ hop_limit │ 1 byte │ Decremented each hop │
│ 34 │ payload │ variable │ Session-layer message (see §8.1) │
└────────┴──────────────────┴───────────┴─────────────────────────────────────┘
Fixed header: 34 bytes
```
**src_addr**: The originator of the datagram. For data traffic, this is the
source endpoint. For error signals (CoordsRequired, PathBroken), this is the
transit router that generated the error. Transit routers use `src_addr` to
route error signals back to the packet's originator.
**dest_addr**: The intended recipient. Transit routers use this for next-hop
selection via `find_next_hop()`.
**hop_limit**: Decremented at each forwarding hop. When it reaches zero, the
datagram is dropped silently. This prevents infinite forwarding loops. Initial
value is configurable (default 64).
### 8.1 Payload Message Types
The SessionDatagram payload begins with a message type byte, followed by the
message-specific content.
**Session-layer messages** (end-to-end encrypted with session keys):
| Type Code | Message | Direction | Purpose |
|-----------|----------------|-----------|-----------------------------------|
| 0x00 | SessionSetup | S → D | Establish session + warm caches |
| 0x01 | SessionAck | D → S | Confirm session establishment |
| 0x10 | DataPacket | Both | Application data |
**Link-layer error signals** (plaintext, generated by transit routers):
| Type Code | Message | Direction | Purpose |
|-----------|----------------|-----------|-----------------------------------|
| 0x20 | CoordsRequired | R → S | Router cache miss |
| 0x21 | PathBroken | R → S | Greedy routing failed |
Error signals are generated by transit router R and sent back to the source S
inside a *new* SessionDatagram with `src_addr=R, dest_addr=S`. They are
plaintext (not end-to-end encrypted) because the transit router has no session
with the source. Link-layer encryption protects them hop-by-hop.
> **Address terminology**: The `src_addr` and `dest_addr` fields in the
> SessionDatagram header are node_addrs (16-byte truncated SHA-256 hashes of
> pubkeys). These are visible to intermediate routers for routing decisions. The
> actual FIPS addresses (pubkeys/npubs) are exchanged only during the Noise IK
> handshake and never appear in packet headers—routers cannot determine endpoint
> identities from the node_addrs they see.
### 8.2 SessionSetup (0x00)
Establishes a crypto session and warms router coordinate caches along the path.
The `src_addr` and `dest_addr` are carried in the enclosing SessionDatagram
envelope (§8.0); the SessionSetup payload carries only coordinates and the
Noise handshake.
```text
┌─────────────────────────────────────────────────────────────────────────────┐
│ SESSION SETUP PAYLOAD (inside SessionDatagram) │
├────────┬──────────────────┬───────────┬─────────────────────────────────────┤
│ Offset │ Field │ Size │ Description │
├────────┼──────────────────┼───────────┼─────────────────────────────────────┤
│ 0 │ msg_type │ 1 byte │ 0x00 │
│ 1 │ flags │ 1 byte │ Bit 0: REQUEST_ACK │
│ │ │ │ Bit 1: BIDIRECTIONAL │
│ 2 │ src_coords_count │ 2 bytes │ u16 LE, number of src coord entries │
│ 4 │ src_coords │ 16 × n │ NodeAddr array (self → root) │
│ ... │ dest_coords_count│ 2 bytes │ u16 LE, number of dest coord entries│
│ ... │ dest_coords │ 16 × m │ NodeAddr array (dest → root) │
│ ... │ handshake_len │ 2 bytes │ u16 LE, Noise payload length │
│ ... │ handshake_payload│ variable │ Noise IK msg1 (82 bytes typical) │
└────────┴──────────────────┴───────────┴─────────────────────────────────────┘
```
**Example** (depth 3 source, depth 4 destination, with Noise handshake):
```text
SessionDatagram { src: S, dest: D, hop_limit: 64 } +
┌──────┬───────┬───────┬─────────────┬───────┬───────────────┬───────┬───────┐
│ 0x00 │ 0x01 │ 0x03 │ src_coords │ 0x04 │ dest_coords │ 0x52 │ noise │
│ type │ flags │ count │ 3 × 16 bytes│ count │ 4 × 16 bytes │ len=82│ msg1 │
└──────┴───────┴───────┴─────────────┴───────┴───────────────┴───────┴───────┘
SessionDatagram header: 34 bytes
SessionSetup payload: 1 + 1 + 2 + 48 + 2 + 64 + 2 + 82 = 202 bytes
Total: 236 bytes
```
### 8.3 SessionAck (0x01)
Confirms session establishment and completes the Noise handshake. Addressing
is in the enclosing SessionDatagram envelope.
```text
┌─────────────────────────────────────────────────────────────────────────────┐
│ SESSION ACK PAYLOAD (inside SessionDatagram) │
├────────┬──────────────────┬───────────┬─────────────────────────────────────┤
│ Offset │ Field │ Size │ Description │
├────────┼──────────────────┼───────────┼─────────────────────────────────────┤
│ 0 │ msg_type │ 1 byte │ 0x01 │
│ 1 │ flags │ 1 byte │ Reserved │
│ 2 │ src_coords_count │ 2 bytes │ u16 LE │
│ 4 │ src_coords │ 16 × n │ Acknowledger's coords (for caching) │
│ ... │ handshake_len │ 2 bytes │ u16 LE, Noise payload length │
│ ... │ handshake_payload│ variable │ Noise IK msg2 (33 bytes typical) │
└────────┴──────────────────┴───────────┴─────────────────────────────────────┘
```
### 8.4 DataPacket (0x10)
Carries encrypted application data (typically IPv6 payloads). Addressing and
hop limit are in the enclosing SessionDatagram envelope.
```text
┌─────────────────────────────────────────────────────────────────────────────┐
│ DATA PACKET (Minimal Header, inside SessionDatagram) │
├────────┬──────────────────┬───────────┬─────────────────────────────────────┤
│ Offset │ Field │ Size │ Description │
├────────┼──────────────────┼───────────┼─────────────────────────────────────┤
│ 0 │ msg_type │ 1 byte │ 0x10 │
│ 1 │ flags │ 1 byte │ Bit 0: COORDS_PRESENT │
│ 2 │ payload_length │ 2 bytes │ u16 LE │
│ 4 │ payload │ variable │ Encrypted application data │
└────────┴──────────────────┴───────────┴─────────────────────────────────────┘
Minimal header: 4 bytes (+ 34 SessionDatagram header = 38 bytes total)
```
When `COORDS_PRESENT` flag is set (route warming after CoordsRequired):
```text
┌─────────────────────────────────────────────────────────────────────────────┐
│ DATA PACKET (With Coordinates, inside SessionDatagram) │
├────────┬──────────────────┬───────────┬─────────────────────────────────────┤
│ Offset │ Field │ Size │ Description │
├────────┼──────────────────┼───────────┼─────────────────────────────────────┤
│ 0 │ msg_type │ 1 byte │ 0x10 │
│ 1 │ flags │ 1 byte │ 0x01 (COORDS_PRESENT) │
│ 2 │ payload_length │ 2 bytes │ u16 LE │
│ 4 │ src_coords_count │ 2 bytes │ u16 LE │
│ 6 │ src_coords │ 16 × n │ Source coordinates │
│ ... │ dest_coords_count│ 2 bytes │ u16 LE │
│ ... │ dest_coords │ 16 × m │ Destination coordinates │
│ ... │ payload │ variable │ Encrypted application data │
└────────┴──────────────────┴───────────┴─────────────────────────────────────┘
With depth-4 coords both directions:
DataPacket: 4 + 2 + 64 + 2 + 64 = 136 bytes header
+ SessionDatagram: 34 bytes
Total: 170 bytes header
```
### 8.5 CoordsRequired (0x20)
Sent by an intermediate router when it cannot forward a SessionDatagram due to
coordinate cache miss for the destination. This is a **link-layer error signal**:
the transit router generates a new SessionDatagram addressed back to the
original source, with the CoordsRequired payload in plaintext (not end-to-end
encrypted, since the transit router has no session with the source).
**Error routing**: Router R receives a SessionDatagram `{src: S, dest: D}` but
cannot route to D. R creates a new SessionDatagram `{src: R, dest: S}` carrying
a CoordsRequired payload, and calls `find_next_hop(S)` to route it back to S.
If R also cannot route to S, the error is dropped silently (no cascading errors).
**CoordsRequired payload** (inside SessionDatagram):
```text
┌─────────────────────────────────────────────────────────────────────────────┐
│ COORDS REQUIRED PAYLOAD │
├────────┬──────────────────┬───────────┬─────────────────────────────────────┤
│ Offset │ Field │ Size │ Description │
├────────┼──────────────────┼───────────┼─────────────────────────────────────┤
│ 0 │ msg_type │ 1 byte │ 0x20 │
│ 1 │ flags │ 1 byte │ Reserved │
│ 2 │ dest_addr │ 16 bytes │ The node_addr we couldn't route to │
│ 18 │ reporter │ 16 bytes │ NodeAddr of reporting router │
└────────┴──────────────────┴───────────┴─────────────────────────────────────┘
Payload: 34 bytes
Wrapped in SessionDatagram: 34 + 34 = 68 bytes total
```
### 8.6 PathBroken (0x21)
Sent when greedy routing fails (no peer is closer to destination). Like
CoordsRequired, this is a **link-layer error signal** carried inside a new
SessionDatagram addressed back to the original source.
**PathBroken payload** (inside SessionDatagram):
```text
┌─────────────────────────────────────────────────────────────────────────────┐
│ PATH BROKEN PAYLOAD │
├────────┬──────────────────┬───────────┬─────────────────────────────────────┤
│ Offset │ Field │ Size │ Description │
├────────┼──────────────────┼───────────┼─────────────────────────────────────┤
│ 0 │ msg_type │ 1 byte │ 0x21 │
│ 1 │ flags │ 1 byte │ Reserved │
│ 2 │ dest_addr │ 16 bytes │ The unreachable node_addr │
│ 18 │ reporter │ 16 bytes │ NodeAddr of reporting router │
│ 34 │ last_coords_count│ 2 bytes │ u16 LE │
│ 36 │ last_known_coords│ 16 × n │ Stale coords that failed │
└────────┴──────────────────┴───────────┴─────────────────────────────────────┘
```
### 8.7 Full Packet Layout Example
A DataPacket from source S to destination D, transiting router R:
```text
┌─────────────────────────────────────────────────────────────────────────────┐
│ UDP DATAGRAM │
├─────────────────────────────────────────────────────────────────────────────┤
│ │
│ ┌───────────────────────────────────────────────────────────────────────┐ │
│ │ FIPS LINK LAYER (S→R encrypted) │ │
│ ├───────────┬──────────────┬────────────┬───────────────────────────────┤ │
│ │ 0x00 │ R's recv_idx │ counter │ ciphertext + tag │ │
│ │ 1 byte │ 4 bytes LE │ 8 bytes LE │ N + 16 bytes │ │
│ └───────────┴──────────────┴────────────┴───────────────────────────────┘ │
│ │ │
│ ┌───────────────────────┘ │
│ │ Decrypt with S↔R link keys │
│ ▼ │
│ ┌───────────────────────────────────────────────────────────────────────┐ │
│ │ SESSION DATAGRAM (0x40) — plaintext for R │ │
│ ├───────────┬──────────────┬──────────────┬──────────┬──────────────────┤ │
│ │ 0x40 │ src_addr │ dest_addr │ hop_limit│ payload │ │
│ │ msg_type │ S (16 bytes) │ D (16 bytes) │ 64 │ (session msg) │ │
│ └───────────┴──────────────┴──────────────┴──────────┴──────────────────┘ │
│ │ │
│ │ R reads src_addr + dest_addr for routing │
│ ▼ │
│ ┌───────────────────────────────────────────────────────────────────────┐ │
│ │ SESSION LAYER PAYLOAD (S↔D encrypted) │ │
│ ├───────────┬───────┬──────────┬──────────┬─────────────────────────────┤ │
│ │ 0x10 │ flags │ reserved │ pay_len │ encrypted application data │ │
│ │ DataPacket│ 0x00 │ 0x00 │ 1400 │ (S↔D session keys + tag) │ │
│ └───────────┴───────┴──────────┴──────────┴─────────────────────────────┘ │
│ │
└─────────────────────────────────────────────────────────────────────────────┘
Router R can see: src_addr, dest_addr, hop_limit (for routing/error signaling)
Router R cannot see: payload contents (encrypted with S↔D session keys)
```
### 8.7.1 Error Routing Example
When router R cannot forward a SessionDatagram from S to D:
```text
1. R receives SessionDatagram { src: S, dest: D, payload: DataPacket{...} }
2. R has no cached coordinates for D and no bloom filter hit
3. R creates NEW SessionDatagram:
{ src: R, dest: S, hop_limit: 64,
payload: CoordsRequired { dest: D, reporter: R } }
4. R calls find_next_hop(S) to route the error back
5. If R can also not route to S: drop silently (no cascading errors)
```
This avoids the need for transit routers to have end-to-end sessions with
the source. The `src_addr` field in the original SessionDatagram tells the
transit router where to send the error.
### 8.8 Encoding Rules
- All multi-byte integers are **little-endian**
- NodeAddr is 16 bytes (truncated SHA-256 hash of npub)
- IPv6 addresses are 16 bytes (network byte order)
- Variable-length coordinate arrays use 2-byte u16 count prefix
```text
Vec<NodeAddr> encoding:
count: u16 (little-endian)
items: [u8; 16] × count
```
docs/design/fips-software-architecture.md
+204 -1040
View File
File diff suppressed because it is too large Load Diff
docs/design/fips-spanning-tree.md
+258
View File
@@ -0,0 +1,258 @@
# FIPS Spanning Tree
This document describes the spanning tree algorithms and data structures
used by FIPS for coordinate-based routing. It is a supporting reference
for readers who want to understand the tree internals — for how the
spanning tree fits into the overall mesh operation, see
[fips-mesh-operation.md](fips-mesh-operation.md).
## Purpose
The spanning tree gives every node in the mesh a **coordinate** — its
ancestry path from itself to the root. These coordinates enable:
- Distance calculation between any two nodes without global topology
knowledge
- Greedy routing where each hop reduces distance to the destination
- Loop-free forwarding guaranteed by strictly-decreasing distance
## Root Election
The root is the node with the **lexicographically smallest node_addr** among
all reachable nodes. There is no election protocol, no voting, no
negotiation. Each node independently evaluates the TreeAnnounce messages
from its peers and selects the minimum root.
When a node first joins the network with no peers, it is its own root. As
it connects to peers and receives their TreeAnnounce messages, it discovers
smaller node_addrs and converges to the global root.
If the network partitions, each segment independently elects its own root
(the smallest node_addr in that segment). When segments rejoin, all nodes
discover the globally-smallest root through TreeAnnounce exchange and
reconverge to a single tree.
## Parent Selection
Each node selects a single parent from among its direct peers. Parent
selection follows these rules:
### Selection Criteria
1. **Find the smallest root** visible across all peers' TreeAnnounce messages
2. Among peers that can reach that root, prefer the one offering the
**shallowest depth** (shortest path to root)
3. Apply the **depth improvement threshold**: switching parents requires the
proposed parent to offer a path at least 1 hop shallower than the current
parent (when the root is the same)
### Immediate Switch Triggers
Three conditions bypass the depth threshold and trigger immediate parent
reselection:
1. **Parent loss**: Current parent is no longer in the peer set (link
broken, peer disconnected)
2. **Better root**: A peer advertises a smaller root than the current
tree's root — always switch regardless of depth
3. **Depth improvement**: Same root, but the proposed parent offers depth
at least `PARENT_SWITCH_THRESHOLD` (1 hop) better than the current parent
### After Parent Change
When a node changes its parent:
1. Increment its own sequence number
2. Recompute its coordinates from the new ancestry path
3. Sign a new TreeAnnounce declaration
4. Announce to all peers
5. Flush the coordinate cache (cached coordinates are relative to the old
position and may be invalid for routing)
## Coordinate Computation
A node's coordinate is its full ancestry path from itself to the root:
```text
coords(N) = [N, Parent(N), Parent(Parent(N)), ..., Root]
```
Coordinates are ordered self-to-root. For a node D at depth 4:
```text
coords(D) = [D, P1, P2, P3, Root]
```
The root's coordinate is simply `[Root]` (depth 0).
## Tree Distance
Tree distance between two nodes is the number of hops through their lowest
common ancestor (LCA). Because coordinates are ordered self-to-root, common
ancestry appears as a common suffix.
```text
tree_distance(a, b):
lca_depth = longest_common_suffix_length(a.coords, b.coords)
a_to_lca = len(a.coords) - lca_depth
b_to_lca = len(b.coords) - lca_depth
return a_to_lca + b_to_lca
```
Example: If A has coordinates `[A, X, Y, Root]` and B has coordinates
`[B, Z, Y, Root]`, the common suffix is `[Y, Root]` (length 2). Distance =
(3 - 2) + (3 - 2) = 2 hops.
The self-distance check in greedy routing uses this calculation: a packet is
forwarded to a peer only if the peer is strictly closer to the destination
than the current node.
## TreeAnnounce Processing
When a node receives a TreeAnnounce from peer P:
1. **Validate version**: Reject if version ≠ 0x01
2. **Verify signature**: Check P's declaration signature using P's known
public key (established during Noise IK handshake)
3. **Verify identity**: Confirm the declaration's node_addr matches the
sender's known identity
4. **Check freshness**: If `sequence ≤ stored sequence for P`, discard
(stale or duplicate)
5. **Update peer state**: Store P's tree declaration and ancestry
6. **Evaluate parent selection**: Re-run parent selection with the updated
peer state
### Propagation Rules
A node re-announces (propagates) only when its own state changes:
- **Root changed**: Always propagate — this is a significant topology event
- **Depth changed**: Always propagate — affects routing distance calculations
- **Sequence-only refresh**: Does NOT propagate beyond depth 1 — peers that
receive a sequence-only update do not re-announce, because their own root
and depth have not changed
This means TreeAnnounce cascades through the tree proportional to depth,
not network size. A change at depth D affects at most D nodes along the
branch, and each only re-announces to its peers.
### Rate Limiting
- **Minimum interval**: 500ms between announcements to the same peer
- **Coalescing**: If changes occur during cooldown, they are coalesced and
sent as a single announcement after the cooldown expires
- **Convergence time**: A tree of depth D reconverges in roughly D × 0.5s
to D × 1.0s
### Transitive Trust (v1)
In the v1 protocol, only the sender's outer signature on the TreeAnnounce
is verified. Ancestry entries beyond the direct peer (the sender's parent,
grandparent, etc.) are accepted on transitive trust through the
authenticated sender. The sender is a known, authenticated peer — if it
claims a particular ancestry, v1 trusts that claim.
Future protocol versions may add per-entry signatures in the ancestry chain
for stronger verification.
## Sequence Numbers and Timestamps
### Sequence Number
- Type: u64, monotonically increasing
- Incremented on each parent change
- Used for freshness: incoming TreeAnnounce with sequence ≤ stored sequence
for that peer is discarded
- Higher sequence numbers always supersede lower ones
### Timestamp
- Type: u64, Unix seconds
- Used for stale detection, not versioning
- A root declaration is considered stale after `ROOT_TIMEOUT` (60 minutes)
without refresh
## Reconvergence
### Single Node Failure
When a node fails (link timeout or disconnect):
1. Nodes that had the failed node as their parent lose their parent
2. Parent loss triggers immediate reselection from remaining peers
3. Each affected node recomputes coordinates and announces
4. Changes cascade down the subtree proportional to depth
### Partition
When the network partitions:
1. Nodes in each segment lose peers across the partition boundary
2. If the root was in the other segment, affected nodes elect a new segment
root (smallest node_addr in their segment)
3. Each segment reconverges independently
### Partition Merge
When two partitions rejoin:
1. Nodes at the boundary exchange TreeAnnounce messages with new peers
2. Both segments discover each other's root
3. The globally-smaller root wins; the other segment's nodes switch parents
4. Coordinate caches are flushed at switching nodes (stale cross-partition
coordinates)
5. Bloom filters update within ~500ms per hop, restoring reachability
information
## Bounded State
Each node's spanning tree state is O(P × D), where P is the number of
direct peers and D is the tree depth. This is NOT O(N) where N is the
network size.
What a node stores:
- Its own declaration (coordinates, sequence, timestamp, signature)
- Each peer's declaration and ancestry chain (P entries, each with D
ancestry entries)
What a node does NOT know:
- Other subtrees branching off its ancestors
- Siblings of ancestors
- Nodes in distant parts of the network
Example: In a 1000-node network with depth 10 and 5 peers, a node stores
~50 ancestry entries — not 1000 routing table entries.
## Timing Parameters
| Parameter | Default | Description |
| --------- | ------- | ----------- |
| PARENT_SWITCH_THRESHOLD | 1 hop | Minimum depth improvement for same-root switch |
| ANNOUNCE_MIN_INTERVAL | 500ms | Minimum between announcements to same peer |
| ROOT_TIMEOUT | 60 min | Root declaration considered stale |
| TREE_ENTRY_TTL | 5–10 min | Individual entry expiration |
## Implementation Status
| Feature | Status |
| ------- | ------ |
| Root election (smallest node_addr) | **Implemented** |
| Parent selection with depth threshold | **Implemented** |
| Coordinate computation | **Implemented** |
| TreeAnnounce gossip | **Implemented** |
| Signature verification (outer) | **Implemented** |
| Sequence-based freshness | **Implemented** |
| Rate limiting (500ms per peer) | **Implemented** |
| Coord cache flush on parent change | **Implemented** |
| Root timeout enforcement | Planned |
| Tree entry TTL enforcement | Planned |
| Hold-down timer after parent change | Planned |
| Per-ancestry-entry signatures | Future direction |
## References
- [fips-mesh-operation.md](fips-mesh-operation.md) — How the spanning tree
fits into mesh routing
- [fips-wire-formats.md](fips-wire-formats.md) — TreeAnnounce wire format
- [spanning-tree-dynamics.md](spanning-tree-dynamics.md) — Convergence
scenario walkthroughs
docs/design/fips-state-machines.md
+138 -161
View File
@@ -49,33 +49,35 @@ impl Peer {
```rust
/// What the Node stores per peer slot
enum PeerSlot {
Connecting(PeerConnection),
Active(ActivePeer),
Connecting(Box<PeerConnection>),
Active(Box<ActivePeer>),
}
/// Handles authentication handshake only
struct PeerConnection {
identity: PeerIdentity,
link_id: LinkId,
direction: Direction,
direction: LinkDirection,
// Handshake-specific state
ephemeral_keypair: Keypair,
remote_ephemeral: Option<PublicKey>,
handshake_hash: [u8; 32],
attempts: u32,
last_sent: Instant,
handshake_state: HandshakeState,
expected_identity: Option<PeerIdentity>,
noise_handshake: Option<noise::HandshakeState>,
noise_session: Option<NoiseSession>,
// Timing
started_at: u64,
last_activity: u64,
}
/// Fully authenticated peer
struct ActivePeer {
identity: PeerIdentity,
link_id: LinkId,
session: SessionKeys,
connectivity: ConnectivityState,
noise_session: Option<NoiseSession>,
// Routing state
declaration: Option<ParentDeclaration>,
coords: Option<TreeCoordinate>,
ancestry: Option<TreeCoordinate>,
inbound_filter: Option<BloomFilter>,
last_seen: Instant,
last_seen: u64,
}
```
@@ -83,141 +85,102 @@ struct ActivePeer {
- Each struct only contains fields relevant to that phase
- Methods can't be called in wrong state (compile-time safety)
- Ephemeral keys automatically dropped when `PeerConnection` → `ActivePeer`
- Noise handshake state automatically dropped when `PeerConnection` → `ActivePeer`
- Each phase struct is smaller, simpler, independently testable
## Transition Pattern
Phase transitions return the new phase, consuming the old:
In FIPS, connections and peers are stored in separate maps rather than a
unified `PeerSlot` map (though `PeerSlot` exists for cases where either phase
is needed). PeerConnection uses `&mut self` methods for handshake steps, and
the Node orchestrates promotion when the handshake completes:
```rust
impl PeerConnection {
/// Handle incoming packet during handshake
fn handle_packet(self, packet: &[u8]) -> ConnectionResult {
// Parse and validate...
match self.state {
HandshakeState::WaitingForResponse => {
// Verify response, derive session keys
let session = self.derive_session_keys(&response);
ConnectionResult::Authenticated {
peer: ActivePeer {
identity: self.identity,
link_id: self.link_id,
session,
declaration: None,
coords: None,
inbound_filter: None,
last_seen: Instant::now(),
},
}
}
// ...
}
}
/// Start handshake as initiator, returns msg1 to send.
fn start_handshake(&mut self, our_keypair: Keypair, current_time_ms: u64)
-> Result<Vec<u8>, NoiseError>;
/// Process incoming msg1 (responder), returns msg2 to send.
fn receive_handshake_init(&mut self, our_keypair: Keypair, message: &[u8],
current_time_ms: u64) -> Result<Vec<u8>, NoiseError>;
/// Complete handshake by processing msg2 (initiator).
fn complete_handshake(&mut self, message: &[u8], current_time_ms: u64)
-> Result<(), NoiseError>;
/// Take the completed NoiseSession for use in ActivePeer.
fn take_session(&mut self) -> Option<NoiseSession>;
}
enum ConnectionResult {
/// Stay in connecting phase, send this response
Continue(Vec<u8>),
/// Auth complete, here's the active peer
Authenticated { peer: ActivePeer },
/// Auth failed
Failed(String),
/// Result of promoting a connection to active peer.
enum PromotionResult {
/// New peer created successfully.
Promoted(NodeAddr),
/// Cross-connection detected, this connection lost tie-breaker.
CrossConnectionLost { winner_link_id: LinkId },
/// Cross-connection detected, this connection won, old one replaced.
CrossConnectionWon { loser_link_id: LinkId, node_addr: NodeAddr },
}
```
The Node's event loop handles the transition:
The Node promotes a completed connection by moving data between maps:
```rust
fn handle_packet(&mut self, link_id: LinkId, packet: &[u8]) {
let slot = self.peers.get_mut(&link_id);
// In the Node's handshake completion handler:
// 1. Remove from connections map
let conn = self.connections.remove(&link_id).unwrap();
match slot {
PeerSlot::Connecting(conn) => {
// Note: take() to move ownership for transition
let conn = std::mem::take(conn);
match conn.handle_packet(packet) {
ConnectionResult::Continue(response) => {
*slot = PeerSlot::Connecting(conn);
self.send(link_id, response);
}
ConnectionResult::Authenticated { peer } => {
*slot = PeerSlot::Active(peer);
self.on_peer_active(link_id);
}
ConnectionResult::Failed(reason) => {
self.peers.remove(&link_id);
warn!(%reason, "Peer auth failed");
}
}
}
PeerSlot::Active(peer) => {
let actions = peer.handle_packet(packet);
self.execute(actions);
}
}
}
// 2. Extract session and identity from completed connection
let session = conn.take_session().unwrap();
let identity = conn.expected_identity().unwrap().clone();
// 3. Create active peer with the session
let peer = ActivePeer::with_session(
identity, link_id, current_time_ms,
session, our_index, their_index,
transport_id, source_addr, conn.link_stats().clone(),
);
// 4. Insert into peers map (checking for cross-connections)
self.peers.insert(*peer.node_addr(), peer);
```
## Timeout Handling
Each phase struct tracks its own timing. The Node's event loop periodically
scans for timeouts:
Each phase struct tracks its own timing using Unix milliseconds (`u64`).
PeerConnection provides a simple boolean timeout check:
```rust
impl PeerConnection {
fn check_timeout(&mut self, now: Instant) -> TimeoutResult {
if now.duration_since(self.last_sent) < HANDSHAKE_TIMEOUT {
return TimeoutResult::Ok;
/// Check if the connection has timed out.
fn is_timed_out(&self, current_time_ms: u64, timeout_ms: u64) -> bool {
self.idle_time(current_time_ms) > timeout_ms
}
self.attempts += 1;
if self.attempts > MAX_HANDSHAKE_ATTEMPTS {
return TimeoutResult::GiveUp;
/// Time since last activity.
fn idle_time(&self, current_time_ms: u64) -> u64 {
current_time_ms.saturating_sub(self.last_activity)
}
self.last_sent = now;
TimeoutResult::Retry(self.build_retry_packet())
}
}
enum TimeoutResult {
Ok,
Retry(Vec<u8>),
GiveUp,
}
```
Node event loop (simple periodic scan):
The Node's event loop periodically scans the connections map for timeouts:
```rust
loop {
select! {
packet = packet_rx.recv() => { /* dispatch */ }
// In the Node's periodic maintenance:
let now_ms = current_time_ms();
let mut timed_out = vec![];
_ = interval.tick() => {
let now = Instant::now();
let mut to_remove = vec![];
for (id, slot) in &mut self.peers {
if let PeerSlot::Connecting(conn) = slot {
match conn.check_timeout(now) {
TimeoutResult::Retry(packet) => {
self.send(conn.link_id, packet);
}
TimeoutResult::GiveUp => {
to_remove.push(*id);
}
TimeoutResult::Ok => {}
}
for (link_id, conn) in &self.connections {
if conn.is_timed_out(now_ms, HANDSHAKE_TIMEOUT_MS) {
timed_out.push(*link_id);
}
}
for id in to_remove {
self.peers.remove(&id);
}
}
}
for link_id in timed_out {
self.connections.remove(&link_id);
// Clean up link, free index, etc.
}
```
@@ -239,75 +202,82 @@ PeerSlot::Active(ActivePeer)
**PeerConnection** contains:
- Noise IK handshake state (ephemeral keys, handshake hash)
- Expected identity (for outbound) or discovered identity (for inbound)
- Direction (Inbound vs Outbound)
- Noise IK handshake state (`noise_handshake`, `handshake_state`)
- Expected identity (known for outbound, learned for inbound)
- Direction (`LinkDirection::Inbound` vs `LinkDirection::Outbound`)
- Session index tracking (`our_index`, `their_index`)
- Timing (`started_at`, `last_activity` in Unix milliseconds)
**ActivePeer** contains:
- NoiseSession (symmetric keys for encrypt/decrypt)
- Tree position (declaration, coordinates)
- Bloom filter (what's reachable through this peer)
- Statistics (last_seen, link_stats)
- `NoiseSession` (symmetric keys for encrypt/decrypt)
- `ConnectivityState` (Connected, Stale, Reconnecting, Disconnected)
- Tree position (`declaration`, `ancestry`)
- Bloom filter (`inbound_filter`, with sequence tracking)
- Session indices and transport address (for wire protocol dispatch)
- Statistics (`last_seen`, `link_stats`, `authenticated_at`)
### Link Lifecycle (Connection-Oriented Transports)
For transports like Tor that require connection setup:
Links use a simple state enum rather than phase-based structs, since link
state transitions are straightforward:
```rust
enum LinkSlot {
Connecting(LinkConnection),
Established(EstablishedLink),
enum LinkState {
Connecting, // Connection in progress (connection-oriented only)
Connected, // Ready for traffic
Disconnected, // Was connected, now gone
Failed, // Connection attempt failed
}
struct LinkConnection {
struct Link {
link_id: LinkId,
transport_id: TransportId,
remote_addr: TransportAddr,
connect_started: Instant,
// Tor circuit build state, etc.
}
struct EstablishedLink {
transport_id: TransportId,
remote_addr: TransportAddr,
// I/O handles
writer: TorWriter,
// Stats
established_at: Instant,
direction: LinkDirection,
state: LinkState,
stats: LinkStats,
// ...
}
```
For connectionless transports (UDP), links are immediately "established" -
no `LinkConnection` phase needed.
For connectionless transports (UDP), links are immediately `Connected` -
no `Connecting` phase needed. A phase-based approach (separate structs for
connecting vs established) would be valuable for transports with complex
connection setup like Tor circuit building.
### Node Lifecycle
```rust
enum NodePhase {
Created(CreatedNode),
Starting(StartingNode),
Running(RunningNode),
Stopping(StoppingNode),
enum NodeState {
Created,
Starting,
Running,
Stopping,
Stopped,
}
```
Currently the Node uses a simpler `NodeState` enum because startup/shutdown
are brief and don't need complex per-phase logic. Phase-based approach would
be useful if startup involved multi-step async operations with retries.
The Node uses a simple state enum because startup/shutdown are brief and
don't need complex per-phase logic. A phase-based approach (separate structs
per phase) would be useful if startup involved multi-step async operations
with retries.
### Transport Lifecycle
```rust
enum TransportPhase {
Configured(ConfiguredTransport),
Starting(StartingTransport),
Up(UpTransport),
Failed(FailedTransport),
enum TransportState {
Configured,
Starting,
Up,
Down,
Failed,
}
```
Again, currently simpler because transport startup is straightforward.
Would be valuable for transports with complex initialization (Tor bootstrap).
Like `NodeState`, this uses a simple state enum because transport startup is
straightforward. A phase-based approach would be valuable for transports with
complex initialization (e.g., Tor bootstrap with multi-step circuit building).
## When to Use This Pattern
@@ -326,16 +296,22 @@ Would be valuable for transports with complex initialization (Tor bootstrap).
## Lookup Tables
When using `PeerSlot` enum, need reverse lookups for packet dispatch:
Rather than a unified `PeerSlot` map, the Node stores connections and peers
in separate maps optimized for their phase:
```rust
struct Node {
// Primary storage
peers: HashMap<NodeAddr, PeerSlot>,
// Handshake phase: indexed by LinkId (identity not yet known)
connections: HashMap<LinkId, PeerConnection>,
// Reverse lookup: (transport, remote_addr) → NodeAddr
// Needed because ReceivedPacket has addr, not NodeAddr
addr_to_peer: HashMap<(TransportId, TransportAddr), NodeAddr>,
// Active phase: indexed by NodeAddr (verified identity)
peers: HashMap<NodeAddr, ActivePeer>,
// Reverse lookup: (transport_id, remote_addr) → LinkId
addr_to_link: HashMap<AddrKey, LinkId>,
// Wire protocol dispatch: (transport_id, our_index) → NodeAddr
peers_by_index: HashMap<(TransportId, u32), NodeAddr>,
}
```
@@ -344,7 +320,8 @@ For inbound connections from unknown addresses:
1. Receive Noise IK msg1 → decrypt to extract sender's static key (identity)
2. Create new PeerConnection with discovered identity
3. Add to `connections` (by LinkId) and `addr_to_link`
4. After handshake completes, promote to ActivePeer (indexed by NodeAddr)
4. After handshake completes, promote to `peers` (indexed by NodeAddr) and
`peers_by_index` (for session index dispatch)
## Summary
@@ -353,7 +330,7 @@ The phase-based state machine pattern provides:
1. **Type safety** - Can't call auth methods on active peer
2. **Memory efficiency** - Phase-specific data dropped on transition
3. **Clarity** - Each struct is focused and comprehensible
4. **Security** - Ephemeral keys don't linger after auth
4. **Security** - Handshake state (ephemeral keys) dropped after auth
5. **Testability** - Each phase testable in isolation
The cost is slightly more complex transition handling in the event loop,
docs/design/fips-transport-layer.md
+382
View File
@@ -0,0 +1,382 @@
# FIPS Transport Layer
The transport layer is the bottom of the FIPS protocol stack. It delivers
datagrams between transport-specific endpoints over arbitrary physical or
logical media. Everything above — peer authentication, routing, encryption,
session management — is built on the services the transport layer provides.
## Role
A **transport** is a driver for a particular communication medium: a UDP
socket, an Ethernet interface, a LoRa radio, a serial line, a Tor circuit.
The transport layer's job is simple: accept a datagram and a transport
address, deliver the datagram to that address, and push inbound datagrams up
to the FIPS Link Protocol (FLP) above.
The transport layer deals exclusively in **transport addresses** — IP:port
tuples, MAC addresses, LoRa device addresses, .onion identifiers. These are
opaque to every layer above FLP. The mapping from transport address to FIPS
identity happens at the link layer after the Noise IK handshake completes.
The word "peer" belongs to the link layer and above; the transport layer
knows only about remote endpoints identified by transport addresses.
A single transport instance can serve multiple remote endpoints
simultaneously — a UDP socket exchanges datagrams with many remote
addresses, an Ethernet interface communicates with many MAC addresses on the
same segment. Each endpoint may become a separate FLP link, but the
transport layer itself maintains no per-endpoint state.
## Services Provided to FLP
The transport layer provides four services to the FIPS Link Protocol above:
### Datagram Delivery
Send and receive datagrams to/from transport addresses. The transport
handles all medium-specific details: socket management, framing for stream
transports, radio configuration. FLP sees only "send bytes to address" and
"bytes arrived from address."
Inbound datagrams are pushed to FLP through a channel. The transport spawns
a receive task that pushes arriving datagrams (along with the source
transport address and transport identifier) onto a bounded channel. FLP
reads from this channel and dispatches based on the source address and
packet content.
### MTU Reporting
Report the maximum datagram size for a given link. FLP needs this to
determine how much payload can fit in a single packet after link-layer
encryption overhead.
MTU is fundamentally a per-link property. A transport with a fixed MTU
(Ethernet: 1500, UDP configured at 1472) returns the same value for every
link — this is the degenerate case. Transports that negotiate MTU
per-connection (e.g., BLE ATT_MTU) report the negotiated value for each
link individually.
> **Implementation note**: The current transport trait exposes MTU as a
> transport-wide method (`fn mtu(&self) -> u16`). This works for UDP, where
> MTU is a static configuration value. Supporting per-link MTU for future
> transports will require extending this interface.
### Connection Lifecycle
For connection-oriented transports, manage the underlying connection: TCP
handshake, Tor circuit establishment, Bluetooth pairing. FLP cannot begin
the Noise IK handshake until the transport-layer connection is established.
Connectionless transports (UDP, raw Ethernet) skip this — datagrams can flow
immediately to any reachable address.
### Discovery (Optional)
Notify FLP when FIPS-capable endpoints are discovered on the local medium.
This is an optional capability — transports that don't support it simply
don't provide discovery events.
See [Discovery](#discovery) below for details.
## Transport Properties
Transports vary widely in their characteristics. FIPS operates over all of
them because the transport interface abstracts these differences behind a
uniform datagram service.
### Transport Categories
**Overlay transports** tunnel FIPS over an existing network layer, typically
for internet connectivity:
| Transport | Addressing | MTU | Reliability | Notes |
| --------- | ---------- | --- | ----------- | ----- |
| UDP/IP | IP:port | 1280–1472 | Unreliable | Primary internet transport |
| TCP/IP | IP:port | Stream | Reliable | Requires length-prefix framing |
| WebSocket | URL | Stream | Reliable | Browser-compatible |
| Tor | .onion | Stream | Reliable | High latency, strong anonymity |
| I2P | Destination | ~32K | Unreliable | Datagram mode |
**Shared medium transports** operate over broadcast- or multicast-capable
media:
| Transport | Addressing | MTU | Reliability | Notes |
| --------- | ---------- | --- | ----------- | ----- |
| Ethernet | MAC | 1500 | Unreliable | Raw AF_PACKET frames |
| WiFi | MAC | 1500 | Unreliable | Infrastructure mode = Ethernet |
| Bluetooth | BD_ADDR | 672–64K | Reliable | L2CAP |
| BLE | BD_ADDR | 23–517 | Reliable | Negotiated ATT_MTU |
| LoRa | Device addr | 51–222 | Unreliable | Low bandwidth, long range |
**Point-to-point transports** connect exactly two endpoints:
| Transport | Addressing | MTU | Reliability | Notes |
| --------- | ---------- | --- | ----------- | ----- |
| Serial | None (P2P) | 256–1500 | Reliable | SLIP/COBS framing |
| Dialup | None (P2P) | 1500 | Reliable | PPP framing |
### Properties That Matter to FLP
**MTU**: Determines how much data FLP can pack into a single datagram after
accounting for link encryption overhead. Heterogeneous MTUs across the mesh
are normal — the IPv6 minimum (1280 bytes) is the safe baseline for FIPS
packet sizing.
**Reliability**: Whether the transport guarantees delivery. FIPS prefers
unreliable transports because running TCP application traffic over a reliable
transport creates TCP-over-TCP, where retransmission and congestion control
at both layers interact adversely. FIPS tolerates packet loss, reordering,
and duplication at the routing layer.
**Connection model**: Connectionless transports (UDP, raw Ethernet) allow
immediate datagram exchange. Connection-oriented transports (TCP, Tor, BLE)
require connection setup before FLP can begin the Noise IK handshake,
adding startup latency.
**Stream vs. datagram**: Datagram transports have natural packet boundaries.
Stream transports (TCP, WebSocket, Tor) require length-prefix framing to
delineate FIPS packets within the byte stream.
**Addressing opacity**: Transport addresses are opaque byte vectors. FLP
doesn't interpret them — it just passes them back to the transport when
sending. This means adding a new transport type with a novel address format
requires no changes to FLP or FSP.
## Connection Model
### Connectionless Transports
Datagrams can be sent to any reachable address without prior setup. Links
are lightweight — a transport address is sufficient to begin communication.
| Transport | Notes |
| --------- | ----- |
| UDP/IP | Stateless datagrams; NAT state is implicit |
| Ethernet | Send to MAC address directly |
| LoRa | Raw packets to device address |
| I2P | Datagram mode |
### Connection-Oriented Transports
Explicit connection setup is required before FIPS traffic can flow. The link
must complete transport-layer connection before FLP authentication can
proceed.
| Transport | Connection Setup |
| --------- | ---------------- |
| TCP/IP | TCP three-way handshake |
| WebSocket | HTTP upgrade + TCP |
| Tor | Circuit establishment (500ms–5s) |
| Bluetooth | L2CAP connection |
| BLE | L2CAP CoC or GATT connection |
| Serial | Physical connection (static) |
### Implications
**Link lifecycle**: Connectionless transports use a trivial link model.
Connection-oriented transports need a real state machine: Connecting →
Connected → Disconnected. Failure can occur during connection setup, adding
error handling paths that connectionless transports don't have.
**Startup latency**: Connection-oriented transports add delay before a peer
becomes usable. This ranges from milliseconds (TCP) to seconds (Tor
circuit). Peer timeout configuration must account for transport-specific
setup times.
**Framing**: Stream transports must delimit FIPS packets within the byte
stream using length-prefix framing. Datagram transports preserve packet
boundaries naturally.
## UDP/IP: The Primary Internet Transport
For internet-connected nodes, UDP/IP is the recommended transport:
- **No TCP-over-TCP**: UDP's unreliable delivery avoids the adverse
interaction between application-layer TCP retransmission and transport-layer
TCP retransmission
- **NAT traversal**: UDP hole punching enables peer connections through NAT
without relay infrastructure
- **Low overhead**: 8-byte UDP header, no connection state
- **Matches FIPS model**: FIPS is datagram-oriented; UDP preserves this
naturally without framing
Raw IP with a custom protocol number would be simpler but is blocked by most
NAT devices and firewalls, limiting deployment to networks without NAT.
## Discovery
Discovery determines that a FIPS-capable endpoint is reachable at a given
transport address. It is distinct from raw transport-level endpoint
detection — a new TCP connection or UDP packet from an unknown source is not
discovery; a FIPS-specific announcement or response is.
Discovery is an optional transport capability. Transports that don't support
it (configured UDP endpoints, TCP) simply don't provide discovery events.
FLP handles both cases uniformly: with discovery, it waits for events then
initiates link setup; without discovery, it initiates link setup directly to
configured addresses.
### Local/Medium Discovery *(future direction)*
For transports where endpoints share a physical or link-layer medium — LAN
broadcast, LoRa, BLE — discovery uses beacon and query mechanisms:
- **Beacon**: A node periodically broadcasts its FIPS presence on the shared
medium. Content is a FIPS-defined discovery frame carrying enough
information to initiate a link. Non-FIPS endpoints ignore the frame.
- **Query**: A node broadcasts a one-shot solicitation. FIPS-capable nodes
respond. Responses arrive on the same channel as beacon events.
Both produce the same result: "FIPS endpoint available at transport address
X." FLP does not need to distinguish beacons from query responses.
| Transport | Discovery | Notes |
| --------- | --------- | ----- |
| UDP (LAN) | Broadcast/multicast | On local network segment |
| Ethernet | Broadcast | Custom EtherType, ff:ff:ff:ff:ff:ff |
| LoRa | Beacon | Shared RF channel, natural fit |
| BLE | Advertising | GATT service UUID |
### Nostr Relay Discovery *(future direction)*
For internet-reachable transports, a node publishes a signed Nostr event
containing its FIPS discovery information — public key and reachable
transport endpoints (UDP IP:port, TCP IP:port, .onion address). Other FIPS
nodes subscribing on the same relays learn about available peers.
Nostr relay discovery is not a transport — it is a discovery service that
feeds addresses to other transports. A node discovers via Nostr that a peer
is reachable at UDP 1.2.3.4:9735, then establishes the link over the UDP
transport.
Key properties:
- Identity is built in — Nostr events are signed, so discovery information
is authenticated
- Relay selection acts as scoping — which relays a node publishes to and
subscribes on determines its discovery neighborhood
- Can only advertise IP-reachable endpoints (not LoRa, BLE, serial)
- Higher latency than local discovery (relay propagation delays)
### Current State
> **Implemented**: Peer addresses come from YAML configuration. The
> transport trait's `discover()` method exists but returns an empty list for
> UDP. Transport-level discovery (beacon/query, Nostr relay) is not yet
> implemented.
## Transport Interface
The transport interface defines what every transport driver must provide.
### Trait Surface
```text
transport_id() → TransportId Unique identifier for this transport instance
transport_type() → &TransportType Static metadata (name, connection-oriented, reliable)
state() → TransportState Current lifecycle state
mtu() → u16 Maximum datagram size
start() → lifecycle Bring transport up (bind socket, open device)
stop() → lifecycle Bring transport down
send(addr, data) → delivery Send datagram to transport address
discover() → Vec<DiscoveredPeer> Report discovered FIPS endpoints (optional)
```
### Receive Path
Rather than a synchronous receive method, transports use a channel-push
model. Each transport takes a sender handle at construction and spawns an
internal receive loop that pushes inbound datagrams onto the channel. The
node's main event loop reads from the corresponding receiver, which
aggregates datagrams from all active transports into a single stream.
Each inbound datagram carries:
- **transport_id** — which transport it arrived on
- **remote_addr** — the transport address of the sender
- **data** — the raw datagram bytes
- **timestamp** — arrival time
### Transport Metadata
Transport types carry static metadata that FLP can query:
```text
TransportType {
name "udp", "ethernet", "tor", etc.
connection_oriented bool
reliable bool
}
```
Predefined types exist for UDP, TCP, Ethernet, WiFi, Tor, and Serial.
### Transport Addresses
Transport addresses (`TransportAddr`) are opaque byte vectors. The transport
layer interprets them (e.g., UDP parses "ip:port" strings); all layers above
treat them as opaque handles passed back to the transport for sending.
### Transport State Machine
```text
Configured → Starting → Up → Down
↓
Failed
```
Transports begin in `Configured` state with all parameters set. `start()`
transitions through `Starting` to `Up` (operational). `stop()` moves to
`Down`. Transport failures move to `Failed`.
## Implementation Status
| Transport | Status | Notes |
| --------- | ------ | ----- |
| UDP/IP | **Implemented** | Primary transport, async send/receive, configurable MTU |
| TCP/IP | Future direction | Requires stream framing, TCP-over-TCP concern |
| Ethernet | Future direction | AF_PACKET raw frames, EtherType TBD |
| WiFi | Future direction | Infrastructure mode = Ethernet driver |
| Tor | Future direction | High latency, .onion addressing |
| BLE | Future direction | ATT_MTU negotiation, per-link MTU |
| LoRa | Future direction | Constrained MTU (51–222 bytes) |
| Serial | Future direction | SLIP/COBS framing, point-to-point |
## Design Considerations
### TCP-over-TCP Avoidance
Running TCP application traffic over a reliable transport (TCP, WebSocket)
creates a layering violation where retransmission and congestion control
operate at both levels. When the inner TCP detects loss (which may just be
transport-layer retransmission delay), it retransmits, creating more traffic
for the outer TCP, which may itself be retransmitting. This amplification
loop degrades performance severely under any packet loss.
FIPS prefers unreliable transports for this reason. When a reliable transport
must be used (e.g., Tor), applications should be aware of the performance
implications.
### Multi-Transport Operation
A node can run multiple transports simultaneously. Peers from all transports
feed into a single spanning tree and routing table. If one transport fails,
traffic automatically routes through alternatives. A node with both UDP and
Ethernet transports bridges between internet-connected and local-only
networks transparently.
Multiple links to the same peer over different transports are possible. FLP
manages these independently — each link has its own Noise session, its own
MTU, and its own liveness tracking.
### Transport Quality and Path Selection
Transport characteristics (latency, bandwidth, reliability) affect path
quality but are not currently factored into routing decisions. The spanning
tree parent selection uses a depth improvement threshold but does not
consider transport quality. This is a potential area for future optimization.
## References
- [fips-intro.md](fips-intro.md) — Protocol overview and layer architecture
- [fips-link-layer.md](fips-link-layer.md) — FLP specification (the layer above)
- [fips-wire-formats.md](fips-wire-formats.md) — Transport framing details
- [fips-software-architecture.md](fips-software-architecture.md) — Transport
trait implementation details
docs/design/fips-transports.md
-190
View File
@@ -1,190 +0,0 @@
# FIPS Transport Protocols
FIPS nodes peer with each other over a variety of transport types. This document
explores the requirements and characteristics of different transport protocols
that FIPS can operate over.
## Terminology
- **Transport**: A physical or logical interface over which FIPS communicates
(e.g., a UDP socket, Ethernet NIC, or Tor client)
- **Link**: A connection instance to a specific peer over a transport
This document describes transport-level characteristics. See
[fips-software-architecture.md](fips-software-architecture.md) for the Transport trait definition.
## Design Principles
FIPS is a Layer 3 (network) protocol. It exposes an IPv6 interface to local
applications, with an address deterministically derived from the node's npub.
This means existing UDP, TCP, and other IP-based applications work unmodified
over FIPS.
However, this IPv6-over-transport architecture requires care to avoid classic
encapsulation pitfalls. In particular, running TCP over a reliable transport
(like TCP/IP overlay) creates "TCP-over-TCP" where retransmission and congestion
control mechanisms at both layers interact adversely. FIPS prefers unreliable
transports for this reason.
FIPS treats underlying connectivity as abstract transports, regardless of
whether those transports are:
- True L2 protocols (Ethernet, Bluetooth)
- L4-over-L3 tunnels (UDP/IP) used as transport substrate for NAT traversal
- Application-layer overlays (Tor, I2P)
Each transport driver presents a uniform interface to the FIPS routing layer:
send/receive datagrams to/from a transport-layer peer address.
## Transport Characteristics
### Overlay Transports (L3/L4 Substrate)
These transports tunnel FIPS over an existing network layer, typically for
internet connectivity or anonymity. Overlay transports are expected to be the
majority in early deployments, but all depend on existing IP/Internet
infrastructure that FIPS is ultimately designed to replace.
| Transport | Encapsulation | Addressing | MTU | Latency | Reliability | Bandwidth | Discovery |
|-----------|---------------|------------|-----|---------|-------------|-----------|-----------|
| UDP/IP | UDP datagram | IP:port | 1280-1472 | 1-500ms | Unreliable | High | DNS-SD, Nostr |
| TCP/IP | Framed stream | IP:port | Stream | 10-500ms | Reliable | High | DNS-SD, Nostr |
| WebSocket | WS frames | URL | Stream | 10-500ms | Reliable | High | Nostr |
| Tor | TCP stream | .onion | Stream | 500ms-5s | Reliable | Low-Med | Static, Nostr |
| I2P | I2P datagram | Destination | ~32K | 1-10s | Unreliable | Low | I2P directory |
### Shared Medium Transports
These transports operate over broadcast or multicast-capable media where multiple
endpoints share the same physical or logical channel.
| Transport | Encapsulation | Addressing | MTU | Latency | Reliability | Bandwidth | Discovery |
|-----------|---------------|------------|-----|---------|-------------|-----------|-----------|
| Ethernet | EtherType frame | MAC | 1500 | <1ms | Unreliable | High | Multicast |
| WiFi Direct | 802.11 frame | MAC | 1500 | 1-10ms | Unreliable | High | Service discovery |
| DOCSIS (Cable) | DOCSIS frame | MAC | 1500 | 10-50ms | Unreliable | 1M-1G | N/A (uses IP) |
| Bluetooth Classic | L2CAP | BD_ADDR | 672-64K | 10-100ms | Reliable | 2-3 Mbps | Inquiry + SDP |
| BLE | L2CAP CoC/GATT | BD_ADDR | 23-517 | 10-30ms | Reliable | 125K-2M | GATT advertising |
| Zigbee | 802.15.4 frame | 16/64-bit | ~100 | 15-30ms | Reliable | 250 kbps | Network scan |
| LoRa | Raw packet | Device addr | 51-222 | 100ms-10s | Unreliable | 0.3-50 kbps | Beacons |
### Point-to-Point Transports
These transports connect exactly two endpoints with no shared medium or
addressing.
| Transport | Encapsulation | Addressing | MTU | Latency | Reliability | Bandwidth | Discovery |
|-----------|---------------|------------|----------|-----------|-------------|-----------|------------|
| Serial | SLIP/COBS frame | None (P2P) | 256-1500 | 1-100ms | Reliable | 9.6K-1M | Configured |
| Dialup | PPP frame | None (P2P) | 1500 | 100-200ms | Reliable | 33.6-56K | Configured |
### Notes
**MTU**: Minimum/maximum or typical range. FIPS must handle heterogeneous MTUs
across the mesh; the IPv6 minimum (1280) is a safe baseline for the FIPS
packet format.
**Latency**: Typical range from best-case to worst-case. Affects spanning tree
convergence and keepalive timing.
**Reliability**: Whether the link provides delivery guarantees. Unreliable
links may drop, reorder, or duplicate packets. FIPS must tolerate this at
the routing layer.
**Bandwidth**: Order of magnitude. Affects flow control and congestion
decisions, but FIPS routing itself is low-bandwidth (control plane only).
## Connection Model
Transports fall into two categories based on whether they require connection
establishment before data can be exchanged:
### Connectionless Transports
These transports can send datagrams to a peer address without prior setup.
Links are lightweight—just a `(transport_id, remote_addr)` tuple with implicit
"established" state.
| Transport | Notes |
|-----------|-------|
| UDP/IP | Stateless datagrams; NAT state is implicit |
| Ethernet | Send to MAC address directly |
| WiFi | Same as Ethernet (802.11 frame) |
| DOCSIS | Cable modem layer; uses IP in practice |
| LoRa | Raw packets to device address |
| I2P | Datagram mode (not streaming) |
### Connection-Oriented Transports
These transports require explicit connection setup before FIPS traffic can flow.
Links track real connection state and hold I/O handles. The link must complete
transport-layer connection before FIPS authentication can proceed.
| Transport | Connection Setup |
|-----------|------------------|
| TCP/IP | TCP handshake |
| WebSocket | HTTP upgrade + TCP |
| Tor | Circuit establishment (slow: 500ms-5s) |
| Bluetooth Classic | L2CAP connection |
| BLE | L2CAP CoC or GATT connection |
| Zigbee | Network join + binding |
| Serial | Physical connection (static) |
| Dialup | PPP negotiation |
### Implications for FIPS
**Link lifecycle**: Connectionless transports use a trivial link model (no state
machine). Connection-oriented transports require a real state machine:
`Connecting → Connected → Disconnected`. See
[fips-software-architecture.md](fips-software-architecture.md) for link lifecycle details.
**Startup latency**: Connection-oriented transports add latency before a peer
becomes usable. Tor is particularly slow (circuit setup). This affects peer
timeout configuration.
**Failure modes**: Connectionless links "fail" only when the transport itself
is down or the peer stops responding. Connection-oriented links can fail during
connection setup, adding more error handling paths.
**Framing**: Connection-oriented stream transports (TCP, WebSocket, Tor) require
length-prefix framing to delineate FIPS packets. Datagram transports have
natural packet boundaries.
## UDP/IP as Primary Internet Transport
For internet-connected nodes, UDP/IP is the recommended transport:
- **NAT traversal**: UDP hole punching enables peer connections through NAT
- **Firewall compatibility**: UDP outbound rarely blocked; stateful firewalls
pass return traffic
- **No connection state**: Matches FIPS datagram model
- **Low overhead**: 8-byte UDP header is negligible
- **Avoids TCP-over-TCP**: As noted in Design Principles, unreliable transports
avoid adverse interactions with application-layer TCP
Raw IP with a custom protocol number would be cleaner but is blocked by most
NAT devices and firewalls, limiting deployment to networks without NAT.
## Transport Driver Interface
> **Note**: The definitive Transport trait is defined in
> [fips-software-architecture.md](fips-software-architecture.md). This section provides a
> simplified conceptual view.
Each transport driver provides:
- `send(addr, data)` — Send a FIPS packet to a transport-layer address
- `recv()` — Receive a FIPS packet from any peer
- `mtu()` — Maximum FIPS packet size for this transport
- `discover()` — Find potential peers (transport-specific mechanism)
Transport drivers handle any necessary framing, fragmentation, or encryption
at the transport layer. The FIPS routing layer sees only FIPS packets.
## Topics for Further Design
- Framing protocols for stream-based transports (TCP, WebSocket)
- Transport-layer encryption requirements vs FIPS-layer encryption
- Congestion control and flow control per transport type
- Multi-path: using multiple transports to same peer
- Transport quality metrics for parent selection
docs/design/fips-tun-driver.md
-236
View File
@@ -1,236 +0,0 @@
# FIPS TUN Driver Design
This document describes the design and implementation of the TUN interface
driver that connects FIPS to the local system's network stack.
## Overview
The TUN driver provides the interface between local applications and the FIPS
mesh network. It presents a virtual network interface (`fips0`) with the node's
FIPS address, allowing standard socket applications to communicate over the
mesh transparently.
## Architecture
```text
┌─────────────────────────────────────────────────────────────────┐
│ Local System │
│ ┌──────────────┐ │
│ │ Applications │ (sockets using fd00::/8 addresses) │
│ └──────┬───────┘ │
│ │ │
│ ┌──────▼───────┐ │
│ │ Kernel │ routing: fd00::/8 → fips0 │
│ │ IPv6 Stack │ local table intercepts traffic to self │
│ └──────┬───────┘ │
└─────────┼───────────────────────────────────────────────────────┘
│ raw IPv6 packets
┌─────────▼───────────────────────────────────────────────────────┐
│ TUN Device (fips0) │
│ ┌─────────────────────────────────────────────────────────────┐│
│ │ File Descriptor ││
│ │ (duplicated for reader/writer) ││
│ └──────────┬─────────────────────────────┬────────────────────┘│
│ │ │ │
│ ┌──────▼──────┐ ┌──────▼──────┐ │
│ │ Reader │ │ Writer │ │
│ │ Thread │ │ Thread │ │
│ │ (blocking) │ │ (blocking) │ │
│ └──────┬──────┘ └──────▲──────┘ │
│ │ │ │
└─────────────┼─────────────────────────────┼──────────────────────┘
│ │
▼ │
┌─────────────┐ ┌───────┴───────┐
│ Packet │ │ TX Queue │
│ Processing │──────────────▶ (mpsc) │
│ (routing) │ │ │
└─────────────┘ └───────▲───────┘
│
(future: transports)
```
## Components
### TunDevice (`src/tun.rs`)
The main TUN device wrapper that handles creation, configuration, and lifecycle.
**Responsibilities:**
- Create TUN interface via `tun` crate
- Configure IPv6 address via netlink (`rtnetlink`)
- Set MTU and bring interface up
- Provide read access for incoming packets
- Create writer handle via fd duplication
**Lifecycle:**
1. **Startup**: Delete existing interface if present, create new
2. **Active**: Reader and writer threads operate independently
3. **Shutdown**: Delete interface via netlink, threads exit on I/O error
### TunWriter (`src/tun.rs`)
Services a queue of outbound packets and writes them to the TUN device.
**Design rationale:**
Multiple sources will need to write to TUN:
- ICMPv6 error responses (from packet processing)
- Inbound mesh traffic from peers (future: transports)
- Locally-generated control traffic (future)
A single writer thread with an mpsc queue provides:
- No contention on TUN writes
- Clean separation of concerns
- Easy addition of new packet sources via `TunTx::clone()`
### ICMPv6 Module (`src/icmp.rs`)
Generates RFC 4443 compliant ICMPv6 error messages.
**Currently implemented:**
- Type 1 Code 0: Destination Unreachable - No route
**Validation (when NOT to send errors):**
- Original packet was an ICMPv6 error (types 0-127)
- Source address is multicast (0xff prefix)
- Source address is unspecified (::)
**Response format:**
- Total size ≤ 1280 bytes (IPv6 minimum MTU)
- Includes as much of original packet as fits
- Proper checksum with pseudo-header
## Packet Flow
### Outbound (local → mesh)
1. Application sends to `fd00::/8` address
2. Kernel routes to `fips0` (requires manual route addition)
3. TUN reader receives raw IPv6 packet
4. Packet processing determines next hop
5. If routable: forward to transport (future)
6. If not routable: send ICMPv6 Destination Unreachable via TX queue
### Inbound (mesh → local)
1. Transport receives packet from peer (future)
2. Check destination address
3. If destination is self: write to TUN via TX queue
4. If destination is other: forward to next hop (transit)
### Local Address Guarantee
Packets arriving at the TUN reader are guaranteed NOT to be destined for
local addresses. The Linux kernel routing order ensures this:
1. **Local routing table** - intercepts traffic to addresses on this machine
2. **Main routing table** - routes `fd00::/8` to fips0
This means every packet in the TUN reader requires a routing decision.
No "is this for me?" check needed on the read path.
## Configuration
From `fips.yaml`:
```yaml
tun:
enabled: true
name: fips0
mtu: 1400
```
**Parameters:**
| Field | Default | Description |
|-------|---------|-------------|
| `enabled` | `true` | Enable TUN interface |
| `name` | `fips0` | Interface name |
| `mtu` | `1400` | Maximum transmission unit |
## Privileges
TUN device creation requires `CAP_NET_ADMIN`. Options:
1. **Run as root**: `sudo ./fips`
2. **Set capability**: `sudo setcap cap_net_admin+ep ./target/debug/fips`
3. **Pre-created device**: Admin creates persistent TUN, FIPS just opens it
## Route Configuration
The kernel route must be added manually (not done by FIPS):
```bash
sudo ip -6 route add fd00::/8 dev fips0
```
This routes all FIPS addresses through the TUN interface.
## Implementation Status
### Completed
- [x] TUN device creation and configuration
- [x] IPv6 address assignment via netlink
- [x] Interface lifecycle (startup cleanup, graceful shutdown)
- [x] Reader thread with blocking I/O
- [x] Writer thread with mpsc queue
- [x] fd duplication for independent read/write
- [x] ICMPv6 Destination Unreachable (Type 1 Code 0)
- [x] Packet validation for ICMPv6 error generation
### Planned
- [ ] ICMPv6 Echo Reply (respond to ping)
- [ ] ICMPv6 Packet Too Big (PMTUD support)
- [ ] ICMPv6 Time Exceeded (hop limit)
- [ ] Rate limiting for ICMPv6 errors
- [ ] Integration with routing/forwarding logic
- [ ] Transit packet handling (decrement hop limit, forward)
- [ ] Automatic route management (add/remove fd00::/8 route)
## Testing
### Manual Testing
```bash
# Terminal 1: Run FIPS with debug logging
sudo RUST_LOG=debug ./target/debug/fips
# Terminal 2: Add route and test
sudo ip -6 route add fd00::/8 dev fips0
ping6 -c 1 fd00::1
# Expected: "Destination unreachable: No route" (not timeout)
```
### Verifying Local Routing
```bash
# Check that local address goes via loopback, not TUN
ip -6 route get <your-fips-address>
# Should show: local ... dev lo
```
## Dependencies
| Crate | Purpose |
|-------|---------|
| `tun` | TUN device creation |
| `rtnetlink` | Netlink interface configuration |
| `libc` | fd duplication (`dup`) |
| `futures` | Async netlink operations |
## References
- RFC 4443: ICMPv6 for IPv6
- RFC 4291: IPv6 Addressing Architecture
- Linux TUN/TAP documentation
docs/design/fips-wire-formats.md
+534
View File
@@ -0,0 +1,534 @@
# FIPS Wire Formats
This document is the comprehensive wire format reference for all three
protocol layers. It covers transport framing, link-layer message formats,
and session-layer message formats, with an encapsulation walkthrough showing
how application data is wrapped through each layer.
## Encoding Rules
- All multi-byte integers are **little-endian** (LE)
- NodeAddr is **16 bytes** — truncated SHA-256 hash of public key
- Signatures are **64 bytes** — secp256k1 Schnorr
- Variable-length arrays use a **2-byte u16 LE count prefix** followed by
that many items
- Public keys are **33 bytes** — compressed secp256k1 (02/03 prefix + 32
bytes)
## Transport Framing
### UDP
FIPS packets are carried directly in UDP datagrams. No additional framing is
needed — each UDP datagram contains exactly one FIPS link-layer packet.
### Stream Transports *(future direction)*
TCP, WebSocket, and Tor transports require length-prefix framing because
they provide a byte stream, not datagrams:
```text
┌────────────┬───────────────────────────────────┐
│ Length │ FIPS Packet │
│ 2 bytes LE │ Variable │
└────────────┴───────────────────────────────────┘
```
## Link-Layer Formats
All link-layer packets begin with a **discriminator byte** that determines
the payload format.
### Discriminator Table
| Byte | Type | Description |
| ---- | ---- | ----------- |
| 0x00 | Encrypted frame | Post-handshake encrypted traffic |
| 0x01 | Noise IK msg1 | Handshake initiation |
| 0x02 | Noise IK msg2 | Handshake response |
### Encrypted Frame (0x00)
All post-handshake traffic between authenticated peers. Contains one
encrypted link-layer message.
```text
┌────────┬──────────────┬──────────┬───────────────────────────┐
│ 0x00 │ receiver_idx │ counter │ ciphertext + AEAD tag │
│ 1 byte │ 4 bytes LE │ 8 bytes LE│ N + 16 bytes │
└────────┴──────────────┴──────────┴───────────────────────────┘
Total overhead: 29 bytes (1 + 4 + 8 + 16)
Minimum frame: 30 bytes (1-byte plaintext)
```
| Field | Size | Description |
| ----- | ---- | ----------- |
| discriminator | 1 byte | 0x00 |
| receiver_idx | 4 bytes LE | Session index for O(1) lookup |
| counter | 8 bytes LE | Monotonic nonce, used as AEAD nonce and for replay detection |
| ciphertext | N bytes | ChaCha20 encrypted payload |
| tag | 16 bytes | Poly1305 authentication tag |
The **plaintext** inside the encrypted frame begins with a message type byte:
| Type | Message |
| ---- | ------- |
| 0x10 | TreeAnnounce |
| 0x20 | FilterAnnounce |
| 0x30 | LookupRequest |
| 0x31 | LookupResponse |
| 0x40 | SessionDatagram |
| 0x50 | Disconnect |
### Noise IK Message 1 (0x01)
Handshake initiation from connecting party.
```text
┌────────┬─────────────┬─────────────────────────────────────────┐
│ 0x01 │ sender_idx │ Noise IK message 1 │
│ 1 byte │ 4 bytes LE │ 82 bytes │
└────────┴─────────────┴─────────────────────────────────────────┘
Total: 87 bytes
```
| Field | Size | Description |
| ----- | ---- | ----------- |
| discriminator | 1 byte | 0x01 |
| sender_idx | 4 bytes LE | Initiator's session index (becomes receiver's `receiver_idx`) |
| noise_msg1 | 82 bytes | Noise IK first message |
**Noise msg1 breakdown** (82 bytes):
| Offset | Field | Size | Description |
| ------ | ----- | ---- | ----------- |
| 0 | ephemeral_pubkey | 33 bytes | Initiator's ephemeral key (compressed secp256k1) |
| 33 | encrypted_static | 33 bytes | Initiator's static key (encrypted with es key) |
| 66 | tag | 16 bytes | AEAD tag for encrypted_static |
Noise pattern: `→ e, es, s, ss`
### Noise IK Message 2 (0x02)
Handshake response from responder.
```text
┌────────┬─────────────┬──────────────┬──────────────────────────┐
│ 0x02 │ sender_idx │ receiver_idx │ Noise IK message 2 │
│ 1 byte │ 4 bytes LE │ 4 bytes LE │ 33 bytes │
└────────┴─────────────┴──────────────┴──────────────────────────┘
Total: 42 bytes
```
| Field | Size | Description |
| ----- | ---- | ----------- |
| discriminator | 1 byte | 0x02 |
| sender_idx | 4 bytes LE | Responder's session index |
| receiver_idx | 4 bytes LE | Echo of initiator's sender_idx from msg1 |
| noise_msg2 | 33 bytes | Noise IK second message |
**Noise msg2 breakdown** (33 bytes):
| Offset | Field | Size | Description |
| ------ | ----- | ---- | ----------- |
| 0 | ephemeral_pubkey | 33 bytes | Responder's ephemeral key (compressed secp256k1) |
Noise pattern: `← e, ee, se`
After msg2, both parties derive identical symmetric session keys.
### Index Semantics
Each party in a link session maintains two indices:
| Index | Chosen By | Used By | Purpose |
| ----- | --------- | ------- | ------- |
| our_index | Us | Them | They include this as `receiver_idx` in packets to us |
| their_index | Them | Us | We include this as `receiver_idx` in packets to them |
### Handshake Flow
```text
Initiator Responder
───────── ─────────
generates sender_idx
generates ephemeral keypair
0x01 | sender_idx | noise_msg1
────────────────────────────────►
validates msg1
learns initiator's static key
generates sender_idx
generates ephemeral keypair
0x02 | sender_idx | receiver_idx | noise_msg2
◄────────────────────────────────
validates msg2
derives session keys
═══════════════ HANDSHAKE COMPLETE ═══════════════
First encrypted frame:
0x00 | receiver_idx | counter=0 | ciphertext+tag
────────────────────────────────►
```
## Link-Layer Message Types
These messages are carried as plaintext inside encrypted frames (0x00).
### TreeAnnounce (0x10)
Spanning tree state announcement, exchanged between direct peers only.
| Offset | Field | Size | Description |
| ------ | ----- | ---- | ----------- |
| 0 | msg_type | 1 byte | 0x10 |
| 1 | version | 1 byte | 0x01 (v1) |
| 2 | sequence | 8 bytes LE | Monotonic counter, increments on parent change |
| 10 | timestamp | 8 bytes LE | Unix seconds |
| 18 | parent | 16 bytes | NodeAddr of selected parent (self = root) |
| 34 | ancestry_count | 2 bytes LE | Number of AncestryEntry records |
| 36 | ancestry | 32 × n bytes | AncestryEntry array (self → root) |
| 36 + 32n | signature | 64 bytes | Schnorr signature over entire message |
**AncestryEntry** (32 bytes):
| Offset | Field | Size | Description |
| ------ | ----- | ---- | ----------- |
| 0 | node_addr | 16 bytes | Node's routing identifier |
| 16 | sequence | 8 bytes LE | Node's sequence number |
| 24 | timestamp | 8 bytes LE | Node's Unix timestamp |
**Size**: `100 + (n × 32)` bytes, where n = `ancestry_count` (depth + 1,
includes self)
| Tree Depth | Payload | With Link Overhead |
| ---------- | ------- | ------------------ |
| 0 (root) | 132 bytes | 161 bytes |
| 3 | 228 bytes | 257 bytes |
| 5 | 292 bytes | 321 bytes |
| 10 | 452 bytes | 481 bytes |
### FilterAnnounce (0x20)
Bloom filter reachability update, exchanged between direct peers only.
| Offset | Field | Size | Description |
| ------ | ----- | ---- | ----------- |
| 0 | msg_type | 1 byte | 0x20 |
| 1 | sequence | 8 bytes LE | Monotonic counter for freshness |
| 9 | hash_count | 1 byte | Number of hash functions (5 in v1) |
| 10 | size_class | 1 byte | Filter size: `512 << size_class` bytes |
| 11 | filter_bits | variable | Bloom filter bit array |
**Size class table**:
| size_class | Bytes | Bits | Status |
| ---------- | ----- | ---- | ------ |
| 0 | 512 | 4,096 | Reserved |
| 1 | 1,024 | 8,192 | **v1 (MUST use)** |
| 2 | 2,048 | 16,384 | Reserved |
| 3 | 4,096 | 32,768 | Reserved |
**v1 payload**: 1,035 bytes (11 header + 1,024 filter).
With link overhead: 1,064 bytes.
### LookupRequest (0x30)
Coordinate discovery request, flooded through the mesh.
| Offset | Field | Size | Description |
| ------ | ----- | ---- | ----------- |
| 0 | msg_type | 1 byte | 0x30 |
| 1 | request_id | 8 bytes LE | Unique random identifier |
| 9 | target | 16 bytes | NodeAddr being sought |
| 25 | origin | 16 bytes | Requester's NodeAddr |
| 41 | ttl | 1 byte | Remaining hops (default 64) |
| 42 | origin_coords_cnt | 2 bytes LE | Number of coordinate entries |
| 44 | origin_coords | 16 × n bytes | Requester's ancestry (NodeAddr only) |
| 44 + 16n | visited_hash_cnt | 1 byte | Hash count for visited filter |
| 45 + 16n | visited_bits | 256 bytes | Compact bloom of visited nodes |
**Size**: `301 + (n × 16)` bytes, where n = origin depth + 1
| Origin Depth | Payload |
| ------------ | ------- |
| 3 | 349 bytes |
| 5 | 381 bytes |
| 10 | 461 bytes |
### LookupResponse (0x31)
Coordinate discovery response, greedy-routed back to requester.
| Offset | Field | Size | Description |
| ------ | ----- | ---- | ----------- |
| 0 | msg_type | 1 byte | 0x31 |
| 1 | request_id | 8 bytes LE | Echoes the request's ID |
| 9 | target | 16 bytes | NodeAddr that was found |
| 25 | target_coords_cnt | 2 bytes LE | Number of coordinate entries |
| 27 | target_coords | 16 × n bytes | Target's ancestry (NodeAddr only) |
| 27 + 16n | proof | 64 bytes | Schnorr signature over `(request_id \|\| target)` |
**Size**: `91 + (n × 16)` bytes
| Target Depth | Payload |
| ------------ | ------- |
| 3 | 139 bytes |
| 5 | 171 bytes |
| 10 | 251 bytes |
**Proof coverage**: Signs `(request_id || target)` only — coordinates are
excluded so the proof survives tree reconvergence during the lookup
round-trip.
### SessionDatagram (0x40)
Encapsulated session-layer payload for multi-hop forwarding.
| Offset | Field | Size | Description |
| ------ | ----- | ---- | ----------- |
| 0 | msg_type | 1 byte | 0x40 |
| 1 | src_addr | 16 bytes | Source NodeAddr |
| 17 | dest_addr | 16 bytes | Destination NodeAddr |
| 33 | hop_limit | 1 byte | Decremented each hop |
| 34 | payload | variable | Session-layer message |
**Fixed header**: 34 bytes (`SESSION_DATAGRAM_HEADER_SIZE`)
The payload is opaque to transit nodes — session-layer encrypted
independently of link encryption.
### Disconnect (0x50)
Orderly link teardown with reason code.
| Offset | Field | Size | Description |
| ------ | ----- | ---- | ----------- |
| 0 | msg_type | 1 byte | 0x50 |
| 1 | reason | 1 byte | Disconnect reason code |
**Reason codes**:
| Code | Name | Description |
| ---- | ---- | ----------- |
| 0x00 | Shutdown | Normal operator-requested stop |
| 0x01 | Restart | Restarting, may reconnect soon |
| 0x02 | ProtocolError | Protocol error encountered |
| 0x03 | TransportFailure | Transport failure |
| 0x04 | ResourceExhaustion | Memory or connection limit |
| 0x05 | SecurityViolation | Authentication or policy violation |
| 0x06 | ConfigurationChange | Peer removed from configuration |
| 0x07 | Timeout | Keepalive or stale detection timeout |
| 0xFF | Other | Unspecified reason |
## Session-Layer Message Types
These messages are carried as the payload of a SessionDatagram (0x40).
### SessionSetup (0x00)
Establishes a session and warms transit coordinate caches.
| Offset | Field | Size | Description |
| ------ | ----- | ---- | ----------- |
| 0 | msg_type | 1 byte | 0x00 |
| 1 | flags | 1 byte | Bit 0: REQUEST_ACK, Bit 1: BIDIRECTIONAL |
| 2 | src_coords_count | 2 bytes LE | Number of source coordinate entries |
| 4 | src_coords | 16 × n bytes | Source's ancestry (NodeAddr, self → root) |
| ... | dest_coords_count | 2 bytes LE | Number of dest coordinate entries |
| ... | dest_coords | 16 × m bytes | Destination's ancestry |
| ... | handshake_len | 2 bytes LE | Noise payload length |
| ... | handshake_payload | variable | Noise IK msg1 (82 bytes typical) |
**Example** (depth-3 source, depth-4 destination):
```text
SessionDatagram header: 34 bytes
SessionSetup payload: 1 + 1 + 2 + 48 + 2 + 64 + 2 + 82 = 202 bytes
Total: 236 bytes
```
### SessionAck (0x01)
Confirms session establishment, completes the Noise handshake.
| Offset | Field | Size | Description |
| ------ | ----- | ---- | ----------- |
| 0 | msg_type | 1 byte | 0x01 |
| 1 | flags | 1 byte | Reserved |
| 2 | src_coords_count | 2 bytes LE | Number of coordinate entries |
| 4 | src_coords | 16 × n bytes | Acknowledger's ancestry (for cache warming) |
| ... | handshake_len | 2 bytes LE | Noise payload length |
| ... | handshake_payload | variable | Noise IK msg2 (33 bytes typical) |
### DataPacket (0x10)
Encrypted application data with explicit replay protection counter.
**Minimal header** (COORDS_PRESENT = 0):
| Offset | Field | Size | Description |
| ------ | ----- | ---- | ----------- |
| 0 | msg_type | 1 byte | 0x10 |
| 1 | flags | 1 byte | Bit 0: COORDS_PRESENT |
| 2 | counter | 8 bytes LE | Session encryption counter / replay nonce |
| 10 | payload_length | 2 bytes LE | Length of encrypted payload |
| 12 | payload | variable | Encrypted application data + 16-byte AEAD tag |
**Header size**: 12 bytes (`DATA_HEADER_SIZE`)
**With coordinates** (COORDS_PRESENT = 1):
| Offset | Field | Size | Description |
| ------ | ----- | ---- | ----------- |
| 0 | msg_type | 1 byte | 0x10 |
| 1 | flags | 1 byte | 0x01 (COORDS_PRESENT) |
| 2 | counter | 8 bytes LE | Session encryption counter |
| 10 | payload_length | 2 bytes LE | Length of encrypted payload |
| 12 | src_coords_count | 2 bytes LE | Source coordinate entries |
| 14 | src_coords | 16 × n bytes | Source's ancestry |
| ... | dest_coords_count | 2 bytes LE | Dest coordinate entries |
| ... | dest_coords | 16 × m bytes | Destination's ancestry |
| ... | payload | variable | Encrypted application data |
### CoordsRequired (0x20)
Link-layer error signal — transit node lacks coordinates for destination.
Plaintext (not end-to-end encrypted), generated by transit nodes.
| Offset | Field | Size | Description |
| ------ | ----- | ---- | ----------- |
| 0 | msg_type | 1 byte | 0x20 |
| 1 | flags | 1 byte | Reserved |
| 2 | dest_addr | 16 bytes | NodeAddr we couldn't route to |
| 18 | reporter | 16 bytes | NodeAddr of reporting router |
**Payload**: 34 bytes. Wrapped in SessionDatagram: 68 bytes total.
### PathBroken (0x21)
Link-layer error signal — greedy routing reached a dead end. Plaintext,
generated by transit nodes.
| Offset | Field | Size | Description |
| ------ | ----- | ---- | ----------- |
| 0 | msg_type | 1 byte | 0x21 |
| 1 | flags | 1 byte | Reserved |
| 2 | dest_addr | 16 bytes | Unreachable NodeAddr |
| 18 | reporter | 16 bytes | NodeAddr of reporting router |
| 34 | last_coords_count | 2 bytes LE | Number of stale coordinate entries |
| 36 | last_known_coords | 16 × n bytes | Stale coordinates that failed |
## Encapsulation Walkthrough
A complete picture of how application data is wrapped through each layer.
### Application Data → Wire
Starting with an application sending a 1024-byte payload to a destination:
```text
Layer 4: Application data
1024 bytes
Layer 3: Session encryption (FSP)
DataPacket header (12 bytes) + encrypted payload (1024) + AEAD tag (16)
= 1052 bytes
Layer 2: SessionDatagram envelope (FLP routing)
msg_type (1) + src_addr (16) + dest_addr (16) + hop_limit (1) + payload (1052)
= 1086 bytes
Layer 1: Link encryption (FLP per-hop)
discriminator (1) + receiver_idx (4) + counter (8) + ciphertext (1086) + tag (16)
= 1115 bytes
Layer 0: Transport
UDP datagram containing 1115 bytes
```
### Overhead Budget
| Layer | Overhead | Component |
| ----- | -------- | --------- |
| Link encryption | 29 bytes | 1 discriminator + 4 index + 8 counter + 16 AEAD tag |
| SessionDatagram | 34 bytes | 1 type + 16 src + 16 dest + 1 hop_limit |
| DataPacket header | 12 bytes | 1 type + 1 flags + 8 counter + 2 length |
| Session AEAD tag | 16 bytes | Poly1305 tag on session-encrypted payload |
| **Minimal total** | **91 bytes** | |
| Coordinates (if present) | ~44 bytes | Varies with tree depth |
| **Worst case** | **135 bytes** | `FIPS_OVERHEAD` constant |
### At Each Transit Node
```text
1. Receive UDP datagram
2. Read discriminator (0x00) → encrypted frame
3. Look up (transport_id, receiver_idx) → session
4. Check replay window (counter)
5. Decrypt with link keys → plaintext link message
6. Read msg_type (0x40) → SessionDatagram
7. Read dest_addr → routing decision
8. Decrement hop_limit
9. Re-encrypt with next-hop link keys
10. Send via next-hop transport
```
Transit nodes see the SessionDatagram envelope (src_addr, dest_addr,
hop_limit) but cannot read the session-layer payload (encrypted with
endpoint session keys).
## Size Summary
### Handshake Messages
| Message | Size |
| ------- | ---- |
| Noise IK msg1 | 87 bytes |
| Noise IK msg2 | 42 bytes |
### Link-Layer Messages (inside encrypted frame)
| Message | Size | Notes |
| ------- | ---- | ----- |
| TreeAnnounce | 100 + 32n bytes | n = depth + 1 |
| FilterAnnounce | 1,035 bytes | v1 (1KB filter) |
| LookupRequest | 301 + 16n bytes | n = origin depth + 1 |
| LookupResponse | 91 + 16n bytes | n = target depth + 1 |
| Disconnect | 2 bytes | |
### Session-Layer Messages (inside SessionDatagram)
| Message | Typical Size | Notes |
| ------- | ------------ | ----- |
| SessionSetup | ~200 bytes | Depth-dependent |
| SessionAck | ~80 bytes | Depth-dependent |
| DataPacket (minimal) | 12 + payload bytes | Steady state |
| DataPacket (with coords) | 12 + ~130 + payload bytes | Warmup/recovery |
| CoordsRequired | 34 bytes | Fixed |
| PathBroken | 36 + 16n bytes | Includes stale coords |
### Complete Packet Sizes (link + session)
| Scenario | Wire Size | Notes |
| -------- | --------- | ----- |
| Encrypted frame minimum | 30 bytes | 1-byte plaintext |
| SessionDatagram + DataPacket (minimal) | 29 + 34 + 12 + payload + 16 | 91 + payload |
| SessionDatagram + DataPacket (with coords) | ~135 + payload | Worst case |
| SessionDatagram + SessionSetup | ~265 bytes | Depth-3, both dirs |
| SessionDatagram + CoordsRequired | 29 + 34 + 34 = 97 bytes | Including link overhead |
## References
- [fips-link-layer.md](fips-link-layer.md) — FLP behavioral specification
- [fips-session-layer.md](fips-session-layer.md) — FSP behavioral specification
- [fips-transport-layer.md](fips-transport-layer.md) — Transport framing
- [fips-mesh-operation.md](fips-mesh-operation.md) — How messages work together
- [fips-ipv6-adapter.md](fips-ipv6-adapter.md) — MTU enforcement
docs/design/fips-wire-protocol.md
-1139
View File
File diff suppressed because it is too large Load Diff
docs/design/spanning-tree-dynamics.md
+59 -61
View File
@@ -5,7 +5,10 @@ operational behavior under various mesh conditions. This document complements
[fips-intro.md](fips-intro.md) with step-by-step walkthroughs of protocol
dynamics rather than message formats and data structures.
For wire formats, see [fips-gossip-protocol.md](fips-gossip-protocol.md) §2 (TreeAnnounce).
For wire formats, see [fips-wire-formats.md](fips-wire-formats.md) (TreeAnnounce section).
For spanning tree algorithms and data structures, see
[fips-spanning-tree.md](fips-spanning-tree.md). For how the spanning tree fits
into mesh routing, see [fips-mesh-operation.md](fips-mesh-operation.md).
The protocol is based on Yggdrasil v0.5's CRDT gossip design.
@@ -69,13 +72,6 @@ The root is deterministic: the node with the lexicographically smallest node_add
among all reachable nodes. No explicit election protocol exists—each node
independently derives the same answer from its local TreeState.
### Announcement Timing (from Yggdrasil v0.5)
- Root timestamp refresh: every 30 minutes
- Root timeout: 60 minutes without refresh
- Peer keepalive: triggered by data activity, not periodic
- TTL on tree entries: implementation-specific, typically minutes
---
## 2. Single Node Startup
@@ -109,9 +105,7 @@ At this point, node A is a fully functional single-node FIPS network. It can:
While isolated, A's state only changes on:
1. **Periodic refresh**: A regenerates its announcement with incremented sequence
and fresh timestamp (maintains liveness for future peers)
2. **Peer connection**: A new peer triggers gossip exchange (covered in Section 3)
1. **Peer connection**: A new peer triggers gossip exchange (covered in Section 3)
---
@@ -220,11 +214,12 @@ The propagation time is O(tree depth), not O(network size). In the example:
- B's coordinates are known immediately (B computes from D's ancestry)
- B's reachability propagates via bloom filter: D → A (1 hop to root)
- Any node wanting to reach B does a bloom filter lookup
- Any node wanting to reach B does a bloom filter lookup for candidate selection
- Total: 1-2 gossip rounds for B to be locatable
Note: Nodes A, C, E never add B to their TreeState. They can still route to B
by using bloom filter lookup to get B's coordinates, then greedy forwarding.
by using bloom filter lookup for candidate selection to get B's coordinates,
then coordinate-based greedy routing.
---
@@ -291,8 +286,8 @@ T7: Converged state
- Any two nodes can compute accurate distance via their coordinates
Nodes do *not* have global knowledge—a leaf node knows nothing about distant
subtrees. But any node can locate any other node via bloom filter lookup and
then route using coordinates.
subtrees. But any node can locate any other node via Bloom-guided candidate
selection and then route using tree coordinate distance.
**Convergence time**: Bounded by tree depth × gossip interval. For a tree of
depth D with gossip interval G:
@@ -312,7 +307,7 @@ During convergence, the network may temporarily have:
- **Multiple roots**: Different partitions with different root beliefs
- **Inconsistent coordinates**: Nodes computing distances from stale state
- **Routing failures**: Greedy routing may fail until coordinates stabilize
- **Routing failures**: Coordinate-based greedy routing may fail until coordinates stabilize
These are transient. The protocol guarantees eventual convergence, not instant
consistency.
@@ -424,18 +419,15 @@ Nodes detect they're partitioned when:
1. **Parent unreachable**: Direct link to parent fails
2. **Root unreachable**: No peer has path to current root
3. **Stale root timestamp**: Root's announcement exceeds timeout (60 min)
**Detection via gossip staleness**:
**Detection via gossip staleness** (not currently implemented — see
[Known Limitations](#known-limitations-v1-implementation)):
```
For each entry in TreeState:
if now - entry.timestamp > TTL:
expire(entry)
If root entry expires:
re-evaluate root from remaining entries
```
In principle, nodes would also detect partitions through root entry staleness:
if no fresh root announcements arrive within a timeout, the root is presumed
departed. This requires tracking root entry timestamps and enforcing expiration,
which is not yet implemented. Currently, only direct parent loss (case 1) and
absence of any peer with a path to root (case 2) trigger partition detection.
### Independent Operation
@@ -551,14 +543,14 @@ link_failed(peer):
- Short timeout: Quick failure detection, but transient issues cause flapping
- Long timeout: Stable under jitter, but slow to respond to real failures
**Typical values**:
**Typical values** (see [fips-spanning-tree.md](fips-spanning-tree.md) for
current FIPS-specific parameters):
```
peer_timeout: 10-30 seconds
keepalive_interval: peer_timeout / 3
gossip_interval: 1-5 seconds (or on-change)
tree_entry_ttl: 5-10 minutes
root_timeout: 60 minutes
gossip_interval: on topology change (no periodic refresh)
tree_entry_ttl: not currently enforced (see Known Limitations)
```
### Asymmetric Failures
@@ -637,9 +629,9 @@ the current parent (under the same root) to trigger a switch. Root changes
always trigger a switch regardless of depth.
**What this means for tree structure**: The v1 algorithm produces minimum-depth
trees, which minimizes coordinate path length and hop count for greedy routing.
However, it does not account for link quality—a high-latency or lossy link at
depth 1 is preferred over a fast link at depth 2.
trees, which minimizes coordinate path length and hop count for coordinate-based
greedy routing. However, it does not account for link quality—a high-latency or
lossy link at depth 1 is preferred over a fast link at depth 2.
### v2 Planned: Cost Metrics
@@ -699,32 +691,33 @@ Once converged, what does the network look like and how does it behave?
**Quiescent gossip**:
- Announcements only on periodic refresh (every few minutes)
- Delta encoding minimizes redundant information
- TreeAnnounce messages sent only on topology changes, not periodically
- No periodic root refresh — the tree is maintained purely by change-driven gossip
- In a stable network, gossip traffic drops to zero
- Bandwidth usage proportional to tree depth, not network size
**Consistent coordinates**:
- Every node knows its full path to root
- Distance calculations are accurate
- Greedy routing succeeds
- Coordinate-based greedy routing succeeds
### Steady State Gossip Pattern
```
Normal operation (no topology changes):
Root A: Refreshes timestamp every 30 minutes
└── Gossips refresh to children
Root A: No periodic announcements — announces only if topology changes
└── Root does not refresh its timestamp periodically
Each node: Forwards root's refresh when received
└── Only sends if peer's view is stale
Each node: Sends TreeAnnounce only when its own state changes
└── Parent change, new peer, or peer departure
Typical gossip per node:
├── Receive refresh from parent (periodic)
├── Forward to children if needed
├── Send own refresh periodically (separate from root's)
└── No gossip if nothing changed and peer is up-to-date
Typical gossip per node in steady state:
├── No periodic sends — tree gossip is entirely change-driven
├── Announce on parent selection change
├── Announce on peer link up/down
└── Zero gossip traffic when topology is stable
```
### Expected Steady State Properties
@@ -732,15 +725,14 @@ Typical gossip per node:
**Gossip volume**:
```
Per link, per refresh cycle:
├── Root timestamp update: ~100 bytes
├── Own declaration (if changed): ~100 bytes
Per topology change event:
├── Own declaration update: ~100 bytes
├── Delta of changed ancestors: varies
└── Total: O(100 bytes) to O(depth * 100 bytes)
For 1000-node network with depth ~10:
├── Each node sends O(1 KB) per refresh cycle
├── With 30-minute refresh: ~0.5 bytes/second per link
In steady state (no topology changes):
├── Zero gossip traffic — no periodic refreshes
├── Traffic resumes only when links change or nodes join/depart
└── Negligible compared to application traffic
```
@@ -774,10 +766,10 @@ In steady state with infrequent updates:
Indicators the network has converged:
1. **Root stability**: Same root for multiple refresh cycles
1. **Root stability**: Same root over extended period
2. **Parent stability**: No parent changes in recent interval
3. **Sequence number stability**: Sequence numbers increment slowly (refresh only)
4. **Routing success**: Greedy routing doesn't hit local minima
3. **Sequence number stability**: Sequence numbers increment only on topology changes
4. **Routing success**: Coordinate-based greedy routing doesn't hit local minima
Warning signs of instability:
@@ -832,10 +824,10 @@ T3: Converged tree (assuming equal link costs):
**Steady state**:
- A is root, refreshes every 30 min
- A is root
- B, C, D are direct children of A
- E is child of C (one hop to A through C)
- Gossip: Each refresh cycle propagates through 2 levels
- No periodic gossip — TreeAnnounce only on topology changes
**Link failure scenario**:
@@ -1035,9 +1027,8 @@ work.
### Known Limitation: Root Timeout Not Enforced
The design specifies a 60-minute root timeout (§1 timing parameters, §6
partition detection) after which nodes should treat the root as departed and
re-elect. The current implementation does not track root entry timestamps or
The design specifies a 60-minute root timeout (§6 partition detection) after
which nodes should treat the root as departed and re-elect. The current implementation does not track root entry timestamps or
perform staleness checks.
**Impact**: If the root node disappears permanently without a graceful
@@ -1137,8 +1128,9 @@ through:
7. **Stability thresholds** - Hysteresis prevents flapping on similar-cost paths
Each node maintains only its own ancestry and direct peer information—not global
topology. Reachability to arbitrary destinations is provided by bloom filters
propagating up the tree, with coordinate discovery via lookup protocol.
topology. Reachability to arbitrary destinations is provided by Bloom-guided
candidate selection (bloom filters propagating up the tree), with coordinate
discovery via lookup protocol and coordinate-based greedy routing for forwarding.
The protocol handles partitions gracefully (independent operation), heals
automatically when connectivity returns, and adapts to heterogeneous link
@@ -1148,6 +1140,12 @@ costs to form efficient tree structures.
## References
### FIPS Internal Documentation
- [fips-spanning-tree.md](fips-spanning-tree.md) — Spanning tree algorithms and data structures
- [fips-mesh-operation.md](fips-mesh-operation.md) — How the spanning tree fits into mesh routing
- [fips-wire-formats.md](fips-wire-formats.md) — TreeAnnounce wire format
### Yggdrasil Documentation
- [Yggdrasil v0.5 Release Notes](https://yggdrasil-network.github.io/2023/10/22/upcoming-v05-release.html)