From 20467f56502431c012f36c427277c1dbddbb0142 Mon Sep 17 00:00:00 2001 From: Johnathan Corgan Date: Sat, 14 Feb 2026 19:26:25 +0000 Subject: [PATCH] Design doc audit: correct 7 code/doc divergences across 5 documents MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Systematic review identified 12 divergences between design docs and implementation. Corrected 7, deferred 3 for further analysis. Changes: - Session tie-breaker: npub → node_addr ordering - Dual cache architecture: CoordCache (50K, TTL 300s) and RouteCache (10K, LRU) with correct names, sizes, and eviction policies - LookupResponse routing: greedy-only → find_next_hop + reverse-path - Discovery TTL default: 8 → 64 - Parent selection: v1 depth-only algorithm, cost metrics marked v2 - Leaf-only mode: implementation status note added - Data overhead: 36-byte → 38-byte header - Verification pass fixed stale cache/header refs in session protocol doc --- docs/design/fips-gossip-protocol.md | 32 +++-- docs/design/fips-routing.md | 83 +++++++---- docs/design/fips-session-protocol.md | 31 ++-- docs/design/fips-software-architecture.md | 45 +++--- docs/design/spanning-tree-dynamics.md | 167 +++++++++------------- 5 files changed, 190 insertions(+), 168 deletions(-) diff --git a/docs/design/fips-gossip-protocol.md b/docs/design/fips-gossip-protocol.md index 36092cd..ed07f17 100644 --- a/docs/design/fips-gossip-protocol.md +++ b/docs/design/fips-gossip-protocol.md @@ -319,8 +319,8 @@ duplicates. **origin**: The node_addr of the original requester. Used for response routing. -**origin_coords**: The requester's current tree coordinates. Enables greedy -routing of the response back to origin. +**origin_coords**: The requester's current tree coordinates. Used by the target +for the first hop of response routing (via `find_next_hop`). **ttl**: Propagation limit. Prevents unbounded flooding. @@ -384,22 +384,30 @@ LookupResponse { **target**: Confirms the identity found. **target_coords**: The target's current tree coordinates. This is the primary -payload - enables greedy routing to the target. +payload — cached by the originator to enable routing to the target. **proof**: Target's signature over `(request_id || target || target_coords)`. Prevents malicious nodes from claiming reachability and blackholing traffic. ### 5.3 Routing -LookupResponse uses greedy tree routing based on `origin_coords` from the -request: +LookupResponse uses a two-phase routing mechanism: ```text 1. Response created at target (or node with target in filter) -2. Each hop forwards toward origin using tree distance -3. Origin receives response, caches target_coords +2. First hop: target routes toward origin via find_next_hop + (standard bloom filter → tree routing path) +3. Subsequent hops: reverse-path forwarding via recent_requests + (each transit node recorded which peer sent the request) +4. Origin receives response, caches target_coords in RouteCache ``` +The first hop from the target uses `find_next_hop(origin)` because +the target was not a transit node for the request (it was the +destination). All transit nodes that forwarded the request stored a +`(request_id → from_peer)` entry in `recent_requests`, enabling +reverse-path forwarding for the response. + ### 5.4 Security The proof signature is critical: @@ -435,8 +443,8 @@ for the full link message type table. | ROOT_TIMEOUT | 60 min | Root declaration considered stale | | TREE_ENTRY_TTL | 5-10 min | Individual entry expiration | | ANNOUNCE_MIN_INTERVAL | 500 ms | Rate limit for announcements | -| LOOKUP_TTL | 8 | Discovery request propagation limit | -| LOOKUP_TIMEOUT | 5 sec | Time to wait for response | +| LOOKUP_TTL | 64 | Discovery request propagation limit | +| LOOKUP_TIMEOUT | 10 sec | Time to wait for response | --- @@ -669,7 +677,7 @@ PLAINTEXT BYTES: [8 bytes request_id] ← random unique ID [16 bytes target node_addr] ← who we're looking for [16 bytes origin node_addr] ← who's asking -08 ← ttl = 8 +40 ← ttl = 64 04 00 ← origin_coords_count = 4 [16 bytes] × 4 ← origin's ancestry (64 bytes) 07 ← visited hash_count = 7 @@ -748,13 +756,13 @@ Source S wants to reach distant destination D (not in local filters) │ ▼ │ │ ┌──────┬───────────────────────────────────┐ │ │ │ 0x30 │ LookupRequest payload │ │ - │ │ │ (target=D, origin=S, ttl=8, ...) │ │ + │ │ │ (target=D, origin=S, ttl=64, ...) │ │ │ └──────┴───────────────────────────────────┘ │ └──────────────────────────────────────────────────────────────┘ 2. Request propagates through network, reaches D -3. D creates LookupResponse, routes back via greedy routing: +3. D creates LookupResponse, routes back via find_next_hop + reverse-path: UDP DATAGRAM ┌──────────────────────────────────────────────────────────────┐ diff --git a/docs/design/fips-routing.md b/docs/design/fips-routing.md index bcabdde..18271e8 100644 --- a/docs/design/fips-routing.md +++ b/docs/design/fips-routing.md @@ -35,8 +35,8 @@ of forwarding decisions. split-horizon merge, so they cover the entire reachable network at steady state. - **Greedy tree routing**: Fallback forwarding during convergence windows - when bloom filters are incomplete. Also used for routing LookupResponse - messages back to the origin. + when bloom filters are incomplete. Also serves as tie-breaker among + bloom filter candidates (closest tree distance wins). - **Discovery protocol**: Populates the coordinate cache to enable greedy tree routing and to provide intermediate routers with coordinate data for more efficient path selection. Not required for basic reachability @@ -307,24 +307,27 @@ signature proves the target authorized the route. ### Caching -Discovered coordinates are cached: +Discovered coordinates are stored in the route cache (`RouteCache`): ```rust struct RouteCache { entries: HashMap, + max_entries: usize, // default: 10,000 } struct CachedCoords { - coords: Vec, + coords: TreeCoordinate, discovered_at: Timestamp, last_used: Timestamp, } ``` -- **Eviction**: LRU when cache full -- **Expiration**: TTL-based (coordinates may go stale if target moves in tree) +- **Eviction**: LRU when cache full (no automatic TTL expiration) - **Invalidation**: On route failure, evict and re-discover +This is distinct from the `CoordCache` used for session-populated +coordinates (see Part 4 for the full dual-cache architecture). + --- ## Part 3: Tree Coordinates and Greedy Routing @@ -446,15 +449,33 @@ with explicit error signaling over metadata privacy. --- -## Part 4: Route Cache Management +## Part 4: Coordinate Cache Architecture > **Wire formats**: For session layer message wire formats (SessionSetup, > SessionAck, DataPacket, CoordsRequired, PathBroken), see > [fips-session-protocol.md](fips-session-protocol.md) §8. -### Route Cache Purpose +### Dual Cache Architecture -The coordinate cache serves two functions: +FIPS uses two coordinate caches with different lifecycles: + +- **CoordCache** (session-populated): Stores coordinates learned from + SessionSetup and SessionAck coordinate fields during session establishment. + TTL-based expiration (300s). 50,000 entries max. Consulted first by + `find_next_hop()`. + +- **RouteCache** (discovery-populated): Stores coordinates learned from + LookupResponse during discovery protocol completion. LRU eviction only + (no automatic TTL). 10,000 entries max. Consulted as fallback when + CoordCache misses. + +Both caches store `TreeCoordinate` values (node_addr → coordinates). Neither +stores next-hop information — routing decisions are computed at lookup time by +`find_next_hop()` using the cached coordinates. + +### Cache Purpose + +The coordinate caches serve two functions: 1. **Greedy routing fallback** — when bloom filters haven't converged, cached coordinates enable tree-distance-based forwarding. @@ -575,30 +596,42 @@ impl Router { } ``` -### Cache Data Structure +### Cache Data Structures ```rust +/// Session-populated coordinate cache (TTL-based). struct CoordCache { entries: HashMap, - max_entries: usize, + max_entries: usize, // default: 50,000 + ttl_ms: u64, // default: 300,000 (5 minutes) } struct CacheEntry { - coords: Vec, - created: Timestamp, - last_used: Timestamp, - expires: Timestamp, + coords: TreeCoordinate, + created_at: u64, // Unix milliseconds + last_used: u64, + expires_at: u64, +} + +/// Discovery-populated coordinate cache (LRU-based, no TTL). +struct RouteCache { + entries: HashMap, + max_entries: usize, // default: 10,000 +} + +struct CachedCoords { + coords: TreeCoordinate, + discovered_at: u64, + last_used: u64, } ``` -**Eviction policy**: LRU (least recently used) when cache exceeds max_entries. +**CoordCache eviction**: Expired entries removed first, then LRU when cache +exceeds max_entries. Entries expire after TTL (300 seconds). Refreshed by +subsequent SessionSetup/SessionAck or DataPacket with COORDS_PRESENT. -**Expiration**: Entries expire after TTL (e.g., 300 seconds). Can be refreshed -by: - -- Subsequent SessionSetup -- SessionRefresh message (lightweight, just touches expiry) -- Data packet transit (optional: refresh on use) +**RouteCache eviction**: Pure LRU when cache exceeds max_entries. No automatic +TTL expiration. Invalidated explicitly on route failure. ### Cache Miss Recovery @@ -697,7 +730,7 @@ hop_limit). Sizes below include the SessionDatagram header. - **Bloom filter traffic**: Near zero (event-driven, no changes) - **Discovery traffic**: Rare (warm caches) - **Session traffic**: Rare (established sessions) -- **Data traffic**: Minimal overhead (36-byte header) +- **Data traffic**: Minimal overhead (38-byte header: 34 envelope + 4 DataPacket) ### Network Churn @@ -712,8 +745,8 @@ When nodes join/leave: | Resource | Full Participant | Leaf-Only | |----------|------------------|-----------| | Bloom filter storage | d × 1 KB (d = peer count) | None | -| Coordinate cache | 10K-100K entries | None | -| Route cache | 1K-10K entries | Minimal | +| CoordCache (session) | 50K entries, 300s TTL | None | +| RouteCache (discovery) | 10K entries, LRU | Minimal | | Bandwidth (idle) | < 1 KB/sec | Near zero | --- diff --git a/docs/design/fips-session-protocol.md b/docs/design/fips-session-protocol.md index f0b79be..2123b76 100644 --- a/docs/design/fips-session-protocol.md +++ b/docs/design/fips-session-protocol.md @@ -238,10 +238,11 @@ which also establish routing session state at intermediate nodes. ### 3.5 Simultaneous Session Initiation (Crossing Hellos) When both nodes attempt to establish a session simultaneously, a deterministic -tie-breaker resolves the conflict using npub ordering: +tie-breaker resolves the conflict using node_addr ordering (consistent +with the link-layer cross-connection tie-breaker): -- If local npub < remote npub: Continue as initiator, ignore incoming initiation -- If local npub > remote npub: Abort own initiation, switch to responder role +- If local node_addr < remote node_addr: Continue as initiator, ignore incoming setup +- If local node_addr > remote node_addr: Abort own initiation, switch to responder role This ensures exactly one handshake completes with minimal wasted effort. @@ -261,8 +262,9 @@ FIPS routing combines three mechanisms: 2. **Discovery protocol**: Query-based lookup for distant destinations 3. **Greedy tree routing**: Coordinate-based forwarding using spanning tree position -The routing layer maintains a route cache mapping `node_addr → (coordinates, -next_hop_peer)`. Cache hits enable immediate greedy routing; cache misses +The routing layer maintains coordinate caches mapping `node_addr → +TreeCoordinate` (see [fips-routing.md](fips-routing.md) §4 for details). +Cache hits enable immediate routing via `find_next_hop()`; cache misses trigger route discovery via bloom filter queries or LookupRequest flooding. ### 4.2 Packet Handling During Discovery @@ -270,13 +272,16 @@ trigger route discovery via bloom filter queries or LookupRequest flooding. Packets are queued (with bounded buffer) while route discovery is in progress and transmitted once coordinates are obtained. -### 4.3 Route Cache Lifetime +### 4.3 Coordinate Cache Lifetime -Route cache entries: +Two caches store `node_addr → TreeCoordinate` mappings (see +[fips-routing.md](fips-routing.md) §4): -- Expire after configurable timeout -- Refresh on successful packet delivery -- Invalidate when peer link goes down or spanning tree topology changes +- **CoordCache** (session-populated): TTL-based eviction (300s default), 50K entries +- **RouteCache** (discovery-populated): LRU eviction only, 10K entries + +Neither cache stores next-hop information — routing decisions are computed at +lookup time by `find_next_hop()` using cached coordinates. --- @@ -286,9 +291,9 @@ Route cache entries: The crypto session handshake (SessionSetup/SessionAck) warms route caches at intermediate routers as it transits. Each message carries the sender's -coordinates; routers extract and cache `(src_addr, dest_addr) → next_hop` for -both directions. After the handshake completes, data packets use minimal -36-byte headers and routers forward based on cached routes. +coordinates; routers extract and cache `node_addr → TreeCoordinate` in the +CoordCache. After the handshake completes, data packets use minimal 38-byte +headers (34 envelope + 4 DataPacket) and routers forward via `find_next_hop()`. ### 5.2 Cache Miss Recovery diff --git a/docs/design/fips-software-architecture.md b/docs/design/fips-software-architecture.md index 9266372..0a97529 100644 --- a/docs/design/fips-software-architecture.md +++ b/docs/design/fips-software-architecture.md @@ -372,8 +372,8 @@ handshake where the initiator knows the responder's static key. See When in `AwaitingMsg2` and we receive a msg1 from the same peer (both sides initiated simultaneously): -- If local npub < remote npub: Ignore incoming msg1, remain initiator -- If local npub > remote npub: Switch to responder role, send msg2, +- If local node_addr < remote node_addr: Ignore incoming msg1, remain initiator +- If local node_addr > remote node_addr: Switch to responder role, send msg2, transition to `ReceivedMsg1` **Events:** @@ -585,6 +585,14 @@ per-message reliability over unreliable links. ## Leaf-Only Operation +> **Implementation status**: Leaf-only mode is partially implemented. The +> `node.leaf_only` config flag and `BloomState::leaf_only()` (bloom filter +> suppression) exist. The following are **not yet implemented**: routing tunnel +> to upstream peer, tree announcement suppression, simplified peer structure, +> `leaf_dependents` tracking on the upstream side, and LookupRequest proxying. +> A leaf-only node currently runs the full Node code path with bloom filters +> disabled. + Leaf-only mode enables constrained devices (sensors, battery-powered nodes, mobile devices) to participate in FIPS without the overhead of full mesh routing. @@ -720,14 +728,15 @@ transport.*.auto_connect # single configured peer ### Resource Comparison -| Resource | Full Participant | Leaf-Only | -|---------------------|---------------------|-----------| -| RAM (Bloom filters) | d × 4KB (d = peers) | 0 | -| RAM (coord cache) | 10K-100K entries | 0 | -| RAM (tree state) | O(P × D) entries | 0 | -| Bandwidth (idle) | < 1 KB/sec | Near zero | -| CPU (filter ops) | Moderate | None | -| Peers | Multiple | One | +| Resource | Full Participant | Leaf-Only | +|---------------------|-----------------------|-----------| +| RAM (Bloom filters) | d × 4KB (d = peers) | 0 | +| RAM (CoordCache) | 50K entries, 300s TTL | 0 | +| RAM (RouteCache) | 10K entries, LRU | 0 | +| RAM (tree state) | O(P × D) entries | 0 | +| Bandwidth (idle) | < 1 KB/sec | Near zero | +| CPU (filter ops) | Moderate | None | +| Peers | Multiple | One | ### Use Cases @@ -932,14 +941,14 @@ present in the wire format for forward compatibility with larger filters. | Parameter | Type | Default | Description | |-----------|------|---------|-------------| -| `discovery.lookup.ttl` | u8 | 8 | Initial TTL for LookupRequest | +| `discovery.lookup.ttl` | u8 | 64 | Initial TTL for LookupRequest | | `discovery.lookup.timeout` | duration | 10s | Timeout waiting for response | | `discovery.lookup.retry_count` | u8 | 3 | Retries before giving up | | `discovery.cache.max_entries` | u32 | 10000 | Route cache size | -| `discovery.cache.ttl` | duration | 300s | Cached coordinates expiry | -The discovery cache stores coordinates learned from LookupResponses for destinations -this node wants to reach. This is the primary cache for endpoint nodes. +The discovery route cache (`RouteCache`) stores coordinates learned from +LookupResponses for destinations this node wants to reach. It uses LRU eviction +only (no automatic TTL expiration). This is the primary cache for endpoint nodes. ### Routing Session Management @@ -956,10 +965,10 @@ this node wants to reach. This is the primary cache for endpoint nodes. | `session.cache.ttl` | duration | 300s | Cached coordinates expiry | | `session.refresh.interval` | duration | 240s | Proactive session refresh | -The session cache stores coordinates learned from SessionSetup packets passing through -this node as a transit router. Larger than discovery cache since routers see traffic -for many destinations. Both caches are part of Node.coord_cache; these parameters -configure the same underlying cache but are grouped by purpose. +The session coordinate cache (`CoordCache`) stores coordinates learned from +SessionSetup/SessionAck packets passing through this node as a transit router. +Larger than the discovery route cache since routers see traffic for many +destinations. Uses TTL-based expiration (300s) with LRU fallback. ### Crypto Session Management diff --git a/docs/design/spanning-tree-dynamics.md b/docs/design/spanning-tree-dynamics.md index 4911463..7014fd0 100644 --- a/docs/design/spanning-tree-dynamics.md +++ b/docs/design/spanning-tree-dynamics.md @@ -18,7 +18,7 @@ The protocol is based on Yggdrasil v0.5's CRDT gossip design. 5. [Topology Changes and Reconvergence](#5-topology-changes-and-reconvergence) 6. [Partition Detection and Handling](#6-partition-detection-and-handling) 7. [Link Failure Detection](#7-link-failure-detection) -8. [Cost Metrics and Parent Selection](#8-cost-metrics-and-parent-selection) +8. [Parent Selection](#8-parent-selection) 9. [Steady State Behavior](#9-steady-state-behavior) 10. [Worked Examples](#10-worked-examples) 11. [Known Limitations (v1 Implementation)](#known-limitations-v1-implementation) @@ -384,15 +384,9 @@ Link B ←→ C fails: ### Reconvergence Dynamics -**Stability threshold**: To prevent flapping, a node only changes parent when: - -``` -improvement = current_cost - new_cost -if improvement > stability_threshold: - change_parent() -``` - -This hysteresis prevents oscillation when two paths have similar costs. +**Stability threshold**: To prevent flapping, a node only changes parent when +the improvement exceeds a threshold. In v1, this is a depth difference of at +least `PARENT_SWITCH_THRESHOLD` (1 hop). See §8 for details. **Sequence number advancement**: Each parent change increments the sequence number. Nodes observing rapid sequence increases can detect instability and @@ -589,39 +583,76 @@ see traffic to consider the link alive. --- -## 8. Cost Metrics and Parent Selection +## 8. Parent Selection Parent selection determines tree structure and routing efficiency. -### Cost Components +### v1 Implementation: Depth-Only Selection -**Latency** (primary metric): +The current implementation uses tree depth as the sole selection metric, with a +threshold to prevent thrashing between equivalent-depth paths. + +**Algorithm** (`TreeState::evaluate_parent()` in `tree.rs`): ``` -cost_latency = round_trip_time_ms +evaluate_parent(): + // 1. Find smallest root reachable through any peer + smallest_root = min(peer.root for peer in peers_with_coords) + + if self == smallest_root and is_root: + return None // Already root, no change + + // 2. Among peers reaching smallest_root, find shallowest + best_peer = min( + [p for p in peers if p.root == smallest_root], + key=lambda p: p.depth + ) + proposed_depth = best_peer.depth + 1 + + if best_peer == current_parent: + return None // Already using best + + // 3. Always switch if parent is gone or root is changing + if current_parent not in peers: + return best_peer // Path broken + if current_root != smallest_root: + return best_peer // Better root found + + // 4. For same root: require depth improvement ≥ threshold + current_depth = my_coords.depth() + if current_depth >= proposed_depth + PARENT_SWITCH_THRESHOLD: + return best_peer + + return None // Not enough improvement ``` -Measured via protocol message exchange timing. Lower is better. - -**Packet loss** (reliability): +**Constants**: ``` -cost_loss = 1 / (1 - loss_rate) +PARENT_SWITCH_THRESHOLD = 1 // Minimum depth improvement to switch parents ``` -Transforms loss rate into multiplicative cost. 10% loss → cost 1.11, 50% loss → cost 2. +This means a proposed parent must offer a path at least 1 hop shallower than +the current parent (under the same root) to trigger a switch. Root changes +always trigger a switch regardless of depth. -**Bandwidth** (capacity): +**What this means for tree structure**: The v1 algorithm produces minimum-depth +trees, which minimizes coordinate path length and hop count for greedy routing. +However, it does not account for link quality—a high-latency or lossy link at +depth 1 is preferred over a fast link at depth 2. -``` -cost_bandwidth = reference_bandwidth / actual_bandwidth -``` +### v2 Planned: Cost Metrics -Normalizes bandwidth to a reference value. Lower capacity → higher cost. +The following cost-based parent selection is planned but not yet implemented. -### Combined Cost +**Cost components**: -A weighted combination: +- **Latency** (primary): `cost_latency = round_trip_time_ms` +- **Packet loss** (reliability): `cost_loss = 1 / (1 - loss_rate)` — + transforms loss rate into multiplicative cost (10% loss → 1.11, 50% → 2.0) +- **Bandwidth** (capacity): `cost_bandwidth = reference_bandwidth / actual_bandwidth` + +**Combined cost**: Weighted combination with application-tunable weights: ``` effective_cost = w_latency * cost_latency @@ -629,91 +660,27 @@ effective_cost = w_latency * cost_latency + w_bandwidth * cost_bandwidth ``` -Weights depend on application priorities. Real-time traffic weights latency -heavily; bulk transfer weights bandwidth heavily. - -### Path Cost to Root - -The cost to reach the root through a peer: +**Path cost to root**: Recursive — each node advertises its cumulative cost, +allowing neighbors to compute total path cost: ``` path_cost(peer) = link_cost(self, peer) + peer.path_cost_to_root ``` -This is recursive—each node advertises its path cost to root, allowing -neighbors to compute their total path cost through that peer. - -### Parent Selection Algorithm - -``` -select_parent(): - candidates = [p for p in peers if p.has_path_to_root] - - if not candidates: - return self // Become own root - - best = min(candidates, key=lambda p: path_cost(p)) - - if current_parent is not None: - current_cost = path_cost(current_parent) - new_cost = path_cost(best) - improvement = current_cost - new_cost - - if improvement < stability_threshold: - return current_parent // Stay with current - - return best -``` - -### Stability Threshold - -Prevents flapping when paths have similar costs: +**Stability threshold**: Hysteresis with both absolute and relative components: ``` stability_threshold = base_threshold + current_cost * relative_threshold - -Example: - base_threshold = 5ms - relative_threshold = 0.1 (10%) - current_cost = 50ms - - threshold = 5 + 50 * 0.1 = 10ms - - New path must be >10ms better to trigger switch ``` -### Cost Measurement +**Cost measurement**: Active probing (periodic RTT measurement), passive +observation (inferred from protocol message timing), and exponential smoothing +(`alpha = 0.1–0.3`) to balance responsiveness with stability. -**Active probing**: - -``` -Every probe_interval: - for peer in peers: - send_probe(peer) - record_send_time() - -On probe_response: - rtt = now - send_time - update_latency_estimate(peer, rtt) -``` - -**Passive observation**: - -``` -On protocol_message_exchange: - infer_rtt_from_request_response_timing() - -On packet_loss_detected: - update_loss_estimate() -``` - -**Exponential smoothing**: - -``` -estimate = alpha * new_sample + (1 - alpha) * estimate - -alpha = 0.1-0.3 typical (higher = more responsive, less stable) -``` +**Implementation prerequisites**: The cost-based algorithm requires changes to +the TreeAnnounce wire format to carry path cost values, and a measurement +subsystem for link quality metrics. See Section 10, Example 2 for how +cost-based selection would affect tree structure in heterogeneous networks. ---