Design doc audit: correct 7 code/doc divergences across 5 documents

Systematic review identified 12 divergences between design docs and implementation. Corrected 7, deferred 3 for further analysis. Changes: - Session tie-breaker: npub → node_addr ordering - Dual cache architecture: CoordCache (50K, TTL 300s) and RouteCache (10K, LRU) with correct names, sizes, and eviction policies - LookupResponse routing: greedy-only → find_next_hop + reverse-path - Discovery TTL default: 8 → 64 - Parent selection: v1 depth-only algorithm, cost metrics marked v2 - Leaf-only mode: implementation status note added - Data overhead: 36-byte → 38-byte header - Verification pass fixed stale cache/header refs in session protocol doc
2026-08-09 00:04:54 +00:00 · 2026-02-14 19:26:25 +00:00
parent 5236fd02bf
commit 20467f5650
5 changed files with 190 additions and 168 deletions
@@ -319,8 +319,8 @@ duplicates.

 **origin**: The node_addr of the original requester. Used for response routing.

-**origin_coords**: The requester's current tree coordinates. Enables greedy
-routing of the response back to origin.
+**origin_coords**: The requester's current tree coordinates. Used by the target
+for the first hop of response routing (via `find_next_hop`).

 **ttl**: Propagation limit. Prevents unbounded flooding.

@@ -384,22 +384,30 @@ LookupResponse {
 **target**: Confirms the identity found.

 **target_coords**: The target's current tree coordinates. This is the primary
-payload - enables greedy routing to the target.
+payload — cached by the originator to enable routing to the target.

 **proof**: Target's signature over `(request_id || target || target_coords)`.
 Prevents malicious nodes from claiming reachability and blackholing traffic.

 ### 5.3 Routing

-LookupResponse uses greedy tree routing based on `origin_coords` from the
-request:
+LookupResponse uses a two-phase routing mechanism:

 ```text
 1. Response created at target (or node with target in filter)
-2. Each hop forwards toward origin using tree distance
-3. Origin receives response, caches target_coords
+2. First hop: target routes toward origin via find_next_hop
+   (standard bloom filter → tree routing path)
+3. Subsequent hops: reverse-path forwarding via recent_requests
+   (each transit node recorded which peer sent the request)
+4. Origin receives response, caches target_coords in RouteCache
 ```

+The first hop from the target uses `find_next_hop(origin)` because
+the target was not a transit node for the request (it was the
+destination). All transit nodes that forwarded the request stored a
+`(request_id → from_peer)` entry in `recent_requests`, enabling
+reverse-path forwarding for the response.
+
 ### 5.4 Security

 The proof signature is critical:
@@ -435,8 +443,8 @@ for the full link message type table.
 | ROOT_TIMEOUT | 60 min | Root declaration considered stale |
 | TREE_ENTRY_TTL | 5-10 min | Individual entry expiration |
 | ANNOUNCE_MIN_INTERVAL | 500 ms | Rate limit for announcements |
-| LOOKUP_TTL | 8 | Discovery request propagation limit |
-| LOOKUP_TIMEOUT | 5 sec | Time to wait for response |
+| LOOKUP_TTL | 64 | Discovery request propagation limit |
+| LOOKUP_TIMEOUT | 10 sec | Time to wait for response |

 ---

@@ -669,7 +677,7 @@ PLAINTEXT BYTES:
 [8 bytes request_id]             ← random unique ID
 [16 bytes target node_addr]        ← who we're looking for
 [16 bytes origin node_addr]        ← who's asking
-08                               ← ttl = 8
+40                               ← ttl = 64
 04 00                            ← origin_coords_count = 4
 [16 bytes] × 4                   ← origin's ancestry (64 bytes)
 07                               ← visited hash_count = 7
@@ -748,13 +756,13 @@ Source S wants to reach distant destination D (not in local filters)
   │                    ▼                                         │
   │              ┌──────┬───────────────────────────────────┐    │
   │              │ 0x30 │ LookupRequest payload             │    │
-   │              │      │ (target=D, origin=S, ttl=8, ...)  │    │
+   │              │      │ (target=D, origin=S, ttl=64, ...) │    │
   │              └──────┴───────────────────────────────────┘    │
   └──────────────────────────────────────────────────────────────┘

 2. Request propagates through network, reaches D

-3. D creates LookupResponse, routes back via greedy routing:
+3. D creates LookupResponse, routes back via find_next_hop + reverse-path:

   UDP DATAGRAM
   ┌──────────────────────────────────────────────────────────────┐
@@ -35,8 +35,8 @@ of forwarding decisions.
  split-horizon merge, so they cover the entire reachable network at
  steady state.
 - **Greedy tree routing**: Fallback forwarding during convergence windows
-  when bloom filters are incomplete. Also used for routing LookupResponse
-  messages back to the origin.
+  when bloom filters are incomplete. Also serves as tie-breaker among
+  bloom filter candidates (closest tree distance wins).
 - **Discovery protocol**: Populates the coordinate cache to enable greedy
  tree routing and to provide intermediate routers with coordinate data
  for more efficient path selection. Not required for basic reachability
@@ -307,24 +307,27 @@ signature proves the target authorized the route.

 ### Caching

-Discovered coordinates are cached:
+Discovered coordinates are stored in the route cache (`RouteCache`):

 ```rust
 struct RouteCache {
    entries: HashMap<NodeAddr, CachedCoords>,
+    max_entries: usize,  // default: 10,000
 }

 struct CachedCoords {
-    coords: Vec<NodeAddr>,
+    coords: TreeCoordinate,
    discovered_at: Timestamp,
    last_used: Timestamp,
 }
 ```

- **Eviction**: LRU when cache full
- **Expiration**: TTL-based (coordinates may go stale if target moves in tree)
+- **Eviction**: LRU when cache full (no automatic TTL expiration)
 - **Invalidation**: On route failure, evict and re-discover

+This is distinct from the `CoordCache` used for session-populated
+coordinates (see Part 4 for the full dual-cache architecture).
+
 ---

 ## Part 3: Tree Coordinates and Greedy Routing
@@ -446,15 +449,33 @@ with explicit error signaling over metadata privacy.

 ---

-## Part 4: Route Cache Management
+## Part 4: Coordinate Cache Architecture

 > **Wire formats**: For session layer message wire formats (SessionSetup,
 > SessionAck, DataPacket, CoordsRequired, PathBroken), see
 > [fips-session-protocol.md](fips-session-protocol.md) §8.

-### Route Cache Purpose
+### Dual Cache Architecture

-The coordinate cache serves two functions:
+FIPS uses two coordinate caches with different lifecycles:
+
+- **CoordCache** (session-populated): Stores coordinates learned from
+  SessionSetup and SessionAck coordinate fields during session establishment.
+  TTL-based expiration (300s). 50,000 entries max. Consulted first by
+  `find_next_hop()`.
+
+- **RouteCache** (discovery-populated): Stores coordinates learned from
+  LookupResponse during discovery protocol completion. LRU eviction only
+  (no automatic TTL). 10,000 entries max. Consulted as fallback when
+  CoordCache misses.
+
+Both caches store `TreeCoordinate` values (node_addr → coordinates). Neither
+stores next-hop information — routing decisions are computed at lookup time by
+`find_next_hop()` using the cached coordinates.
+
+### Cache Purpose
+
+The coordinate caches serve two functions:

 1. **Greedy routing fallback** — when bloom filters haven't converged,
   cached coordinates enable tree-distance-based forwarding.
@@ -575,30 +596,42 @@ impl Router {
 }
 ```

-### Cache Data Structure
+### Cache Data Structures

 ```rust
+/// Session-populated coordinate cache (TTL-based).
 struct CoordCache {
    entries: HashMap<NodeAddr, CacheEntry>,
-    max_entries: usize,
+    max_entries: usize,      // default: 50,000
+    ttl_ms: u64,             // default: 300,000 (5 minutes)
 }

 struct CacheEntry {
-    coords: Vec<NodeAddr>,
-    created: Timestamp,
-    last_used: Timestamp,
-    expires: Timestamp,
+    coords: TreeCoordinate,
+    created_at: u64,         // Unix milliseconds
+    last_used: u64,
+    expires_at: u64,
+}
+
+/// Discovery-populated coordinate cache (LRU-based, no TTL).
+struct RouteCache {
+    entries: HashMap<NodeAddr, CachedCoords>,
+    max_entries: usize,      // default: 10,000
+}
+
+struct CachedCoords {
+    coords: TreeCoordinate,
+    discovered_at: u64,
+    last_used: u64,
 }
 ```

-**Eviction policy**: LRU (least recently used) when cache exceeds max_entries.
+**CoordCache eviction**: Expired entries removed first, then LRU when cache
+exceeds max_entries. Entries expire after TTL (300 seconds). Refreshed by
+subsequent SessionSetup/SessionAck or DataPacket with COORDS_PRESENT.

-**Expiration**: Entries expire after TTL (e.g., 300 seconds). Can be refreshed
-by:
-
- Subsequent SessionSetup
- SessionRefresh message (lightweight, just touches expiry)
- Data packet transit (optional: refresh on use)
+**RouteCache eviction**: Pure LRU when cache exceeds max_entries. No automatic
+TTL expiration. Invalidated explicitly on route failure.

 ### Cache Miss Recovery

@@ -697,7 +730,7 @@ hop_limit). Sizes below include the SessionDatagram header.
 - **Bloom filter traffic**: Near zero (event-driven, no changes)
 - **Discovery traffic**: Rare (warm caches)
 - **Session traffic**: Rare (established sessions)
- **Data traffic**: Minimal overhead (36-byte header)
+- **Data traffic**: Minimal overhead (38-byte header: 34 envelope + 4 DataPacket)

 ### Network Churn

@@ -712,8 +745,8 @@ When nodes join/leave:
 | Resource | Full Participant | Leaf-Only |
 |----------|------------------|-----------|
 | Bloom filter storage | d × 1 KB (d = peer count) | None |
-| Coordinate cache | 10K-100K entries | None |
-| Route cache | 1K-10K entries | Minimal |
+| CoordCache (session) | 50K entries, 300s TTL | None |
+| RouteCache (discovery) | 10K entries, LRU | Minimal |
 | Bandwidth (idle) | < 1 KB/sec | Near zero |

 ---
@@ -238,10 +238,11 @@ which also establish routing session state at intermediate nodes.
 ### 3.5 Simultaneous Session Initiation (Crossing Hellos)

 When both nodes attempt to establish a session simultaneously, a deterministic
-tie-breaker resolves the conflict using npub ordering:
+tie-breaker resolves the conflict using node_addr ordering (consistent
+with the link-layer cross-connection tie-breaker):

- If local npub < remote npub: Continue as initiator, ignore incoming initiation
- If local npub > remote npub: Abort own initiation, switch to responder role
+- If local node_addr < remote node_addr: Continue as initiator, ignore incoming setup
+- If local node_addr > remote node_addr: Abort own initiation, switch to responder role

 This ensures exactly one handshake completes with minimal wasted effort.

@@ -261,8 +262,9 @@ FIPS routing combines three mechanisms:
 2. **Discovery protocol**: Query-based lookup for distant destinations
 3. **Greedy tree routing**: Coordinate-based forwarding using spanning tree position

-The routing layer maintains a route cache mapping `node_addr → (coordinates,
-next_hop_peer)`. Cache hits enable immediate greedy routing; cache misses
+The routing layer maintains coordinate caches mapping `node_addr →
+TreeCoordinate` (see [fips-routing.md](fips-routing.md) §4 for details).
+Cache hits enable immediate routing via `find_next_hop()`; cache misses
 trigger route discovery via bloom filter queries or LookupRequest flooding.

 ### 4.2 Packet Handling During Discovery
@@ -270,13 +272,16 @@ trigger route discovery via bloom filter queries or LookupRequest flooding.
 Packets are queued (with bounded buffer) while route discovery is in progress
 and transmitted once coordinates are obtained.

-### 4.3 Route Cache Lifetime
+### 4.3 Coordinate Cache Lifetime

-Route cache entries:
+Two caches store `node_addr → TreeCoordinate` mappings (see
+[fips-routing.md](fips-routing.md) §4):

- Expire after configurable timeout
- Refresh on successful packet delivery
- Invalidate when peer link goes down or spanning tree topology changes
+- **CoordCache** (session-populated): TTL-based eviction (300s default), 50K entries
+- **RouteCache** (discovery-populated): LRU eviction only, 10K entries
+
+Neither cache stores next-hop information — routing decisions are computed at
+lookup time by `find_next_hop()` using cached coordinates.

 ---

@@ -286,9 +291,9 @@ Route cache entries:

 The crypto session handshake (SessionSetup/SessionAck) warms route caches at
 intermediate routers as it transits. Each message carries the sender's
-coordinates; routers extract and cache `(src_addr, dest_addr) → next_hop` for
-both directions. After the handshake completes, data packets use minimal
-36-byte headers and routers forward based on cached routes.
+coordinates; routers extract and cache `node_addr → TreeCoordinate` in the
+CoordCache. After the handshake completes, data packets use minimal 38-byte
+headers (34 envelope + 4 DataPacket) and routers forward via `find_next_hop()`.

 ### 5.2 Cache Miss Recovery

@@ -372,8 +372,8 @@ handshake where the initiator knows the responder's static key. See
 When in `AwaitingMsg2` and we receive a msg1 from the same peer (both sides
 initiated simultaneously):

- If local npub < remote npub: Ignore incoming msg1, remain initiator
- If local npub > remote npub: Switch to responder role, send msg2,
+- If local node_addr < remote node_addr: Ignore incoming msg1, remain initiator
+- If local node_addr > remote node_addr: Switch to responder role, send msg2,
  transition to `ReceivedMsg1`

 **Events:**
@@ -585,6 +585,14 @@ per-message reliability over unreliable links.

 ## Leaf-Only Operation

+> **Implementation status**: Leaf-only mode is partially implemented. The
+> `node.leaf_only` config flag and `BloomState::leaf_only()` (bloom filter
+> suppression) exist. The following are **not yet implemented**: routing tunnel
+> to upstream peer, tree announcement suppression, simplified peer structure,
+> `leaf_dependents` tracking on the upstream side, and LookupRequest proxying.
+> A leaf-only node currently runs the full Node code path with bloom filters
+> disabled.
+
 Leaf-only mode enables constrained devices (sensors, battery-powered nodes, mobile
 devices) to participate in FIPS without the overhead of full mesh routing.

@@ -720,14 +728,15 @@ transport.*.auto_connect        # single configured peer

 ### Resource Comparison

-| Resource            | Full Participant    | Leaf-Only |
-|---------------------|---------------------|-----------|
-| RAM (Bloom filters) | d × 4KB (d = peers) | 0         |
-| RAM (coord cache)   | 10K-100K entries    | 0         |
-| RAM (tree state)    | O(P × D) entries    | 0         |
-| Bandwidth (idle)    | < 1 KB/sec          | Near zero |
-| CPU (filter ops)    | Moderate            | None      |
-| Peers               | Multiple            | One       |
+| Resource            | Full Participant      | Leaf-Only |
+|---------------------|-----------------------|-----------|
+| RAM (Bloom filters) | d × 4KB (d = peers)   | 0         |
+| RAM (CoordCache)    | 50K entries, 300s TTL | 0         |
+| RAM (RouteCache)    | 10K entries, LRU      | 0         |
+| RAM (tree state)    | O(P × D) entries      | 0         |
+| Bandwidth (idle)    | < 1 KB/sec            | Near zero |
+| CPU (filter ops)    | Moderate              | None      |
+| Peers               | Multiple              | One       |

 ### Use Cases

@@ -932,14 +941,14 @@ present in the wire format for forward compatibility with larger filters.

 | Parameter | Type | Default | Description |
 |-----------|------|---------|-------------|
-| `discovery.lookup.ttl` | u8 | 8 | Initial TTL for LookupRequest |
+| `discovery.lookup.ttl` | u8 | 64 | Initial TTL for LookupRequest |
 | `discovery.lookup.timeout` | duration | 10s | Timeout waiting for response |
 | `discovery.lookup.retry_count` | u8 | 3 | Retries before giving up |
 | `discovery.cache.max_entries` | u32 | 10000 | Route cache size |
-| `discovery.cache.ttl` | duration | 300s | Cached coordinates expiry |

-The discovery cache stores coordinates learned from LookupResponses for destinations
-this node wants to reach. This is the primary cache for endpoint nodes.
+The discovery route cache (`RouteCache`) stores coordinates learned from
+LookupResponses for destinations this node wants to reach. It uses LRU eviction
+only (no automatic TTL expiration). This is the primary cache for endpoint nodes.

 ### Routing Session Management

@@ -956,10 +965,10 @@ this node wants to reach. This is the primary cache for endpoint nodes.
 | `session.cache.ttl` | duration | 300s | Cached coordinates expiry |
 | `session.refresh.interval` | duration | 240s | Proactive session refresh |

-The session cache stores coordinates learned from SessionSetup packets passing through
-this node as a transit router. Larger than discovery cache since routers see traffic
-for many destinations. Both caches are part of Node.coord_cache; these parameters
-configure the same underlying cache but are grouped by purpose.
+The session coordinate cache (`CoordCache`) stores coordinates learned from
+SessionSetup/SessionAck packets passing through this node as a transit router.
+Larger than the discovery route cache since routers see traffic for many
+destinations. Uses TTL-based expiration (300s) with LRU fallback.

 ### Crypto Session Management

@@ -18,7 +18,7 @@ The protocol is based on Yggdrasil v0.5's CRDT gossip design.
 5. [Topology Changes and Reconvergence](#5-topology-changes-and-reconvergence)
 6. [Partition Detection and Handling](#6-partition-detection-and-handling)
 7. [Link Failure Detection](#7-link-failure-detection)
-8. [Cost Metrics and Parent Selection](#8-cost-metrics-and-parent-selection)
+8. [Parent Selection](#8-parent-selection)
 9. [Steady State Behavior](#9-steady-state-behavior)
 10. [Worked Examples](#10-worked-examples)
 11. [Known Limitations (v1 Implementation)](#known-limitations-v1-implementation)
@@ -384,15 +384,9 @@ Link B ←→ C fails:

 ### Reconvergence Dynamics

-**Stability threshold**: To prevent flapping, a node only changes parent when:
-
-```
-improvement = current_cost - new_cost
-if improvement > stability_threshold:
-    change_parent()
-```
-
-This hysteresis prevents oscillation when two paths have similar costs.
+**Stability threshold**: To prevent flapping, a node only changes parent when
+the improvement exceeds a threshold. In v1, this is a depth difference of at
+least `PARENT_SWITCH_THRESHOLD` (1 hop). See §8 for details.

 **Sequence number advancement**: Each parent change increments the sequence
 number. Nodes observing rapid sequence increases can detect instability and
@@ -589,39 +583,76 @@ see traffic to consider the link alive.

 ---

-## 8. Cost Metrics and Parent Selection
+## 8. Parent Selection

 Parent selection determines tree structure and routing efficiency.

-### Cost Components
+### v1 Implementation: Depth-Only Selection

-**Latency** (primary metric):
+The current implementation uses tree depth as the sole selection metric, with a
+threshold to prevent thrashing between equivalent-depth paths.
+
+**Algorithm** (`TreeState::evaluate_parent()` in `tree.rs`):

 ```
-cost_latency = round_trip_time_ms
+evaluate_parent():
+    // 1. Find smallest root reachable through any peer
+    smallest_root = min(peer.root for peer in peers_with_coords)
+
+    if self == smallest_root and is_root:
+        return None  // Already root, no change
+
+    // 2. Among peers reaching smallest_root, find shallowest
+    best_peer = min(
+        [p for p in peers if p.root == smallest_root],
+        key=lambda p: p.depth
+    )
+    proposed_depth = best_peer.depth + 1
+
+    if best_peer == current_parent:
+        return None  // Already using best
+
+    // 3. Always switch if parent is gone or root is changing
+    if current_parent not in peers:
+        return best_peer  // Path broken
+    if current_root != smallest_root:
+        return best_peer  // Better root found
+
+    // 4. For same root: require depth improvement ≥ threshold
+    current_depth = my_coords.depth()
+    if current_depth >= proposed_depth + PARENT_SWITCH_THRESHOLD:
+        return best_peer
+
+    return None  // Not enough improvement
 ```

-Measured via protocol message exchange timing. Lower is better.
-
-**Packet loss** (reliability):
+**Constants**:

 ```
-cost_loss = 1 / (1 - loss_rate)
+PARENT_SWITCH_THRESHOLD = 1  // Minimum depth improvement to switch parents
 ```

-Transforms loss rate into multiplicative cost. 10% loss → cost 1.11, 50% loss → cost 2.
+This means a proposed parent must offer a path at least 1 hop shallower than
+the current parent (under the same root) to trigger a switch. Root changes
+always trigger a switch regardless of depth.

-**Bandwidth** (capacity):
+**What this means for tree structure**: The v1 algorithm produces minimum-depth
+trees, which minimizes coordinate path length and hop count for greedy routing.
+However, it does not account for link quality—a high-latency or lossy link at
+depth 1 is preferred over a fast link at depth 2.

-```
-cost_bandwidth = reference_bandwidth / actual_bandwidth
-```
+### v2 Planned: Cost Metrics

-Normalizes bandwidth to a reference value. Lower capacity → higher cost.
+The following cost-based parent selection is planned but not yet implemented.

-### Combined Cost
+**Cost components**:

-A weighted combination:
+- **Latency** (primary): `cost_latency = round_trip_time_ms`
+- **Packet loss** (reliability): `cost_loss = 1 / (1 - loss_rate)` —
+  transforms loss rate into multiplicative cost (10% loss → 1.11, 50% → 2.0)
+- **Bandwidth** (capacity): `cost_bandwidth = reference_bandwidth / actual_bandwidth`
+
+**Combined cost**: Weighted combination with application-tunable weights:

 ```
 effective_cost = w_latency * cost_latency
@@ -629,91 +660,27 @@ effective_cost = w_latency * cost_latency
               + w_bandwidth * cost_bandwidth
 ```

-Weights depend on application priorities. Real-time traffic weights latency
-heavily; bulk transfer weights bandwidth heavily.
-
-### Path Cost to Root
-
-The cost to reach the root through a peer:
+**Path cost to root**: Recursive — each node advertises its cumulative cost,
+allowing neighbors to compute total path cost:

 ```
 path_cost(peer) = link_cost(self, peer) + peer.path_cost_to_root
 ```

-This is recursive—each node advertises its path cost to root, allowing
-neighbors to compute their total path cost through that peer.
-
-### Parent Selection Algorithm
-
-```
-select_parent():
-    candidates = [p for p in peers if p.has_path_to_root]
-
-    if not candidates:
-        return self  // Become own root
-
-    best = min(candidates, key=lambda p: path_cost(p))
-
-    if current_parent is not None:
-        current_cost = path_cost(current_parent)
-        new_cost = path_cost(best)
-        improvement = current_cost - new_cost
-
-        if improvement < stability_threshold:
-            return current_parent  // Stay with current
-
-    return best
-```
-
-### Stability Threshold
-
-Prevents flapping when paths have similar costs:
+**Stability threshold**: Hysteresis with both absolute and relative components:

 ```
 stability_threshold = base_threshold + current_cost * relative_threshold
-
-Example:
-    base_threshold = 5ms
-    relative_threshold = 0.1 (10%)
-    current_cost = 50ms
-
-    threshold = 5 + 50 * 0.1 = 10ms
-
-    New path must be >10ms better to trigger switch
 ```

-### Cost Measurement
+**Cost measurement**: Active probing (periodic RTT measurement), passive
+observation (inferred from protocol message timing), and exponential smoothing
+(`alpha = 0.1–0.3`) to balance responsiveness with stability.

-**Active probing**:
-
-```
-Every probe_interval:
-    for peer in peers:
-        send_probe(peer)
-        record_send_time()
-
-On probe_response:
-    rtt = now - send_time
-    update_latency_estimate(peer, rtt)
-```
-
-**Passive observation**:
-
-```
-On protocol_message_exchange:
-    infer_rtt_from_request_response_timing()
-
-On packet_loss_detected:
-    update_loss_estimate()
-```
-
-**Exponential smoothing**:
-
-```
-estimate = alpha * new_sample + (1 - alpha) * estimate
-
-alpha = 0.1-0.3 typical (higher = more responsive, less stable)
-```
+**Implementation prerequisites**: The cost-based algorithm requires changes to
+the TreeAnnounce wire format to carry path cost values, and a measurement
+subsystem for link quality metrics. See Section 10, Example 2 for how
+cost-based selection would affect tree structure in heterogeneous networks.

 ---