diff --git a/docs/design/fips-routing.md b/docs/design/fips-routing.md index e89aeef..ca55b22 100644 --- a/docs/design/fips-routing.md +++ b/docs/design/fips-routing.md @@ -1,25 +1,46 @@ # FIPS Routing Design This document describes the routing architecture for FIPS, including Bloom -filter reachability, discovery protocol, greedy tree routing, and routing -session establishment. +filter routing, greedy tree routing, discovery protocol, and routing session +establishment. For wire formats and exchange rules, see [fips-gossip-protocol.md](fips-gossip-protocol.md). For spanning tree dynamics and convergence, see [spanning-tree-dynamics.md](spanning-tree-dynamics.md). ## Overview -FIPS routing combines three mechanisms: +FIPS uses a layered routing strategy where each mechanism handles different +situations. In steady state, bloom filter routing handles the vast majority +of forwarding decisions. -1. **Bloom filters**: Fast reachability lookup for destinations reachable - through peers -2. **Discovery protocol**: Query-based lookup for distant destinations -3. **Greedy tree routing**: Coordinate-based forwarding using spanning tree - position +### Next-Hop Selection (in priority order) -The design separates discovery (finding where a destination is) from routing -(getting packets there). Bloom filters and discovery handle the former; tree -coordinates handle the latter. +1. **Local delivery** — destination is self +2. **Direct peer** — destination is an authenticated peer +3. **Bloom filter routing** — one or more peers' bloom filters contain the + destination; select the best candidate by `(link_cost, tree_distance, + node_addr)`. Since filters propagate unboundedly through the network, + every reachable destination eventually appears in at least one peer's + filter. This is the **primary routing path** for most traffic. +4. **Greedy tree routing** — fallback when bloom filters haven't yet + converged (transient condition during topology changes). Requires the + destination's tree coordinates to be in the local coordinate cache, + populated by a prior SessionSetup or LookupResponse. +5. **No route** — destination unreachable + +### Role of Each Mechanism + +- **Bloom filters**: Primary forwarding — tell each node which peer to + send through for a given destination. Propagate unboundedly via + split-horizon merge, so they cover the entire reachable network at + steady state. +- **Greedy tree routing**: Fallback forwarding during convergence windows + when bloom filters are incomplete. Also used for routing LookupResponse + messages back to the origin. +- **Discovery protocol**: Populates the coordinate cache to enable greedy + tree routing and to provide intermediate routers with coordinate data + for more efficient path selection. Not required for basic reachability + once bloom filters have converged. ## Design Goals @@ -33,11 +54,15 @@ coordinates handle the latter. | Scale | Nodes | Bloom Filter Role | |-------|-------|-------------------| -| Small private network | 100-1,000 | Covers entire network | -| Modest public network | ~1,000,000 | Covers transitive peer neighborhood | +| Small private network | 100-1,000 | Covers entire network with low FPR | +| Modest public network | ~1,000,000 | Covers entire network but FPR increases at hub nodes due to filter saturation | | Internet-scale | Billions | Out of scope (requires different architecture) | -The primary design target is networks up to ~1M nodes. +The primary design target is networks up to ~1M nodes. Since bloom filters +propagate unboundedly (no TTL), they converge to represent the entire +reachable network. At large scale, the fixed 1KB filter size means higher +false positive rates at well-connected hub nodes, which may trigger +unnecessary discovery queries but does not affect correctness. ## Node Participation Modes @@ -127,9 +152,12 @@ the node's reachable neighborhood. | 1,200 | 7.5% | Hub node | | 1,600 | 15% | Heavily loaded hub | -FPR above 5% triggers more LookupRequests but the discovery protocol handles -this gracefully. Hub nodes may benefit from larger filters in future protocol -versions (see §1.6). +Since filters propagate unboundedly, hub nodes with many peers will have +higher occupancy (more entries merged from more peers). FPR above 5% means +bloom filter routing may occasionally select a peer that can't actually +reach the destination (false positive), requiring fallback to greedy tree +routing or error recovery. Hub nodes may benefit from larger filters in +future protocol versions (see §1.6). ### Size Classes (Forward Compatibility) @@ -220,14 +248,16 @@ Bloom filters cannot remove individual entries. Expiration is handled via: ### Purpose -Discover the tree coordinates of distant destinations not covered by local -Bloom filters. +Discover the tree coordinates of a destination to enable greedy tree routing +and to populate coordinate caches at intermediate routers for more efficient +forwarding. In steady state, bloom filters handle reachability; discovery +provides the coordinate information that improves path selection quality. ### When Used -- Destination not found in any peer's Bloom filter -- Route cache miss -- After cached route failure +- During bloom filter convergence (destination not yet in any peer's filter) +- To populate coordinate caches for greedy tree routing (optimization) +- After cached route failure (coordinates may be stale) For wire formats, see [fips-gossip-protocol.md](fips-gossip-protocol.md) §4-5. @@ -299,6 +329,13 @@ struct CachedCoords { ## Part 3: Tree Coordinates and Greedy Routing +Tree coordinates and greedy routing serve two roles: + +1. **Fallback forwarding** during bloom filter convergence windows +2. **Tie-breaking** among bloom filter candidates — tree distance between + a candidate peer and the destination helps select the best path when + multiple peers advertise reachability + ### Tree Coordinates A node's coordinates are its ancestry path from self to root: @@ -326,39 +363,48 @@ Note: Coordinates are ordered self-to-root, so common ancestry is a suffix. ### Greedy Routing Algorithm +When used as a fallback (no bloom filter hits), greedy routing forwards to +the peer that minimizes tree distance to the destination. A self-distance +check ensures progress — the packet is only forwarded if the chosen peer is +strictly closer than the current node. + ```rust -fn greedy_next_hop(&self, dest_coords: &[NodeAddr]) -> NodeAddr { - // Check if we are the destination - if dest_coords[0] == self.node_addr { - return LOCAL_DELIVERY; +fn greedy_next_hop(&self, dest_coords: &TreeCoordinate) -> Option { + if self.my_coords.root_id() != dest_coords.root_id() { + return None; // different tree } - // Check if destination is a direct peer - for peer in &self.peers { - if peer.node_addr == dest_coords[0] { - return peer.node_addr; + let my_distance = self.my_coords.distance_to(dest_coords); + + // Find peer with minimum distance, tie-break by smallest node_addr + let best = self.peer_ancestry.iter() + .min_by(|(id_a, coords_a), (id_b, coords_b)| { + coords_a.distance_to(dest_coords) + .cmp(&coords_b.distance_to(dest_coords)) + .then_with(|| id_a.cmp(id_b)) + }); + + match best { + Some((peer_id, coords)) if coords.distance_to(dest_coords) < my_distance => { + Some(*peer_id) } + _ => None, // no peer is closer (local minimum) } - - // Forward to peer closest to destination - self.peers - .iter() - .min_by_key(|p| tree_distance(&p.coords, dest_coords)) - .map(|p| p.node_addr) - .expect("no peers") } ``` -### Guaranteed Progress +### Progress Guarantee -Greedy routing makes progress as long as: +When the coordinate cache is populated, greedy routing makes progress as +long as: 1. Tree is connected 2. Destination's coordinates are accurate -3. Current node is not the destination +3. A peer is closer to the destination than the current node -Unlike DHT routing, greedy tree routing cannot get stuck in local minima if -the tree is properly formed. +If no peer is closer (local minimum), routing returns `None` and the caller +generates a PathBroken error. In a properly formed tree this should not +occur, but the self-distance check provides a safety net. ### What Each Node Knows @@ -404,9 +450,13 @@ with explicit error signaling over metadata privacy. ### Route Cache Purpose -Intermediate routers cache coordinate mappings so that data packets can use -minimal headers (addresses only, no coordinates). This reduces per-packet -overhead from ~300 bytes to 38 bytes. +The coordinate cache serves two functions: + +1. **Greedy routing fallback** — when bloom filters haven't converged, + cached coordinates enable tree-distance-based forwarding. +2. **Reduced packet overhead** — once coordinates are cached at intermediate + routers, data packets can carry addresses only (38 bytes) rather than + full coordinates (~300 bytes). ### Cache Lifecycle @@ -476,7 +526,15 @@ impl Router { } } - // Route using cache (now populated if coords were present) + // Primary: bloom filter routing + let candidates = self.destination_in_filters(&packet.dest_addr); + if !candidates.is_empty() { + let next = self.select_best_candidate(&candidates); + self.forward(next, packet); + return; + } + + // Fallback: greedy tree routing via cached coordinates match self.coord_cache.get(&packet.dest_addr) { Some(entry) => { entry.last_used = now(); @@ -484,7 +542,7 @@ impl Router { self.forward(next, packet); } None => { - // Cache miss — request coordinates + // No bloom filter hit and no cached coordinates self.send_error(from, CoordsRequired { dest_addr: packet.dest_addr, reporter: self.node_addr, @@ -638,25 +696,28 @@ consulting the visited filter. This is referenced in the gossip protocol spec (section 4.4, rate limiting) but needs to be elevated to a protocol requirement, not an optimization. -### Known Limitation: Capacity-Blind Greedy Routing +### Known Limitation: Capacity-Blind Routing -Greedy routing selects the next hop by minimizing tree distance, which is -purely topological (hop count through the LCA). It does not account for link -capacity, latency, or loss. +Both bloom filter candidate selection and greedy tree routing currently +select next hops without considering link quality. When multiple peers +can reach a destination, the selection is based on tree distance and +node address tie-breaking — purely topological metrics that ignore link +capacity, latency, and loss. -This creates a problem when a topologically short but low-capacity link exists -alongside a longer but high-capacity path. Greedy routing will prefer the short -path, potentially saturating the slow link while the high-capacity path goes -underutilized. +This creates a problem when a topologically short but low-capacity link +exists alongside a longer but high-capacity path. The routing algorithm +will prefer the topologically closer peer, potentially saturating a slow +link while a higher-capacity path goes underutilized. -**Proposed mitigation**: Each node locally measures the quality of its direct -peer links (RTT, bandwidth, loss) and applies a cost adjustment to the -forwarding decision: +**Proposed mitigation**: Each node locally measures the quality of its +direct peer links (RTT, bandwidth, loss) and incorporates this into a +`link_cost()` metric. The next-hop selection uses a composite ordering +of `(link_cost, tree_distance, node_addr)` — link quality takes priority +over topological distance. -```text -effective_distance(peer, dest) = - tree_distance(peer, dest) × local_link_cost(peer) -``` +The `link_cost()` interface is implemented (currently returning a constant), +ready to be populated with real measurements using an established link +quality algorithm (ETX, Babel composite metric, etc.). This requires no protocol changes — link quality is measured locally, not advertised. Self-reported cost claims are intentionally excluded from the diff --git a/src/node/mod.rs b/src/node/mod.rs index a28f50b..3dac75f 100644 --- a/src/node/mod.rs +++ b/src/node/mod.rs @@ -706,14 +706,122 @@ impl Node { self.peers.values().filter(|p| p.can_send()).count() } - // === Routing (stubs) === + // === Routing === - /// Find next hop for a destination (stub). + /// Find next hop for a destination node address. /// - /// Returns the peer that minimizes tree distance to the destination. - pub fn find_next_hop(&self, _dest_node_addr: &NodeAddr) -> Option<&ActivePeer> { - // Stub: would implement greedy tree routing - None + /// Routing priority: + /// 1. Destination is self → `None` (local delivery) + /// 2. Destination is a direct peer → that peer + /// 3. Bloom filter candidates + greedy tree routing → among peers whose + /// bloom filter contains the destination, pick the one that minimizes + /// tree distance to the destination (if dest coords are cached), with + /// `(link_cost, tree_distance_to_dest, node_addr)` tie-breaking. + /// Falls back to greedy tree routing if no bloom filter hits. + /// 4. No route → `None` + /// + /// The self-distance check from greedy routing also applies to bloom + /// filter candidates: a peer is only selected if it is strictly closer + /// to the destination than we are (prevents routing loops). + pub fn find_next_hop(&self, dest_node_addr: &NodeAddr) -> Option<&ActivePeer> { + // 1. Local delivery + if dest_node_addr == self.node_addr() { + return None; + } + + // 2. Direct peer + if let Some(peer) = self.peers.get(dest_node_addr) { + if peer.can_send() { + return Some(peer); + } + } + + // Look up destination coords (used by both bloom and tree paths) + let now_ms = std::time::SystemTime::now() + .duration_since(std::time::UNIX_EPOCH) + .map(|d| d.as_millis() as u64) + .unwrap_or(0); + let dest_coords = self.coord_cache.get(dest_node_addr, now_ms); + + // 3. Bloom filter candidates, scored by tree distance to dest + let candidates: Vec<&ActivePeer> = self.destination_in_filters(dest_node_addr); + if !candidates.is_empty() { + return self.select_best_candidate(&candidates, dest_coords); + } + + // 4. Greedy tree routing fallback + let dest_coords = dest_coords?; + let next_hop_id = self.tree_state.find_next_hop(dest_coords)?; + + self.peers.get(&next_hop_id).filter(|p| p.can_send()) + } + + /// Select the best peer from a set of bloom filter candidates. + /// + /// When dest_coords are available, uses distance from each candidate's + /// coordinates to the destination as the primary metric (after link_cost). + /// Only selects peers that are strictly closer to the destination than + /// we are (self-distance check prevents loops). + /// + /// When dest_coords are not available, falls back to distance from us + /// to the candidate peer (a weaker heuristic — prefers closer peers on + /// the theory that shorter paths are better). + /// + /// Ordering: `(link_cost, distance_to_dest, node_addr)`. + fn select_best_candidate<'a>( + &'a self, + candidates: &[&'a ActivePeer], + dest_coords: Option<&crate::tree::TreeCoordinate>, + ) -> Option<&'a ActivePeer> { + let my_distance = dest_coords.map(|dc| self.tree_state.my_coords().distance_to(dc)); + + let mut best: Option<(&ActivePeer, f64, usize)> = None; + + for &candidate in candidates { + if !candidate.can_send() { + continue; + } + + let cost = candidate.link_cost(); + + // Compute distance: peer→dest if coords available, else us→peer + let dist = match dest_coords { + Some(dc) => self + .tree_state + .peer_coords(candidate.node_addr()) + .map(|pc| pc.distance_to(dc)) + .unwrap_or(usize::MAX), + None => self + .tree_state + .distance_to_peer(candidate.node_addr()) + .unwrap_or(usize::MAX), + }; + + // Self-distance check: when dest coords are available, + // only consider peers that are strictly closer than us + if let Some(my_dist) = my_distance { + if dist >= my_dist { + continue; + } + } + + let dominated = match &best { + None => true, + Some((_, best_cost, best_dist)) => { + cost < *best_cost + || (cost == *best_cost && dist < *best_dist) + || (cost == *best_cost + && dist == *best_dist + && candidate.node_addr() < best.as_ref().unwrap().0.node_addr()) + } + }; + + if dominated { + best = Some((candidate, cost, dist)); + } + } + + best.map(|(peer, _, _)| peer) } /// Check if a destination is in any peer's bloom filter. diff --git a/src/node/tests/mod.rs b/src/node/tests/mod.rs index 376233b..516b8d1 100644 --- a/src/node/tests/mod.rs +++ b/src/node/tests/mod.rs @@ -6,6 +6,7 @@ use std::time::Duration; mod bloom; mod handshake; +mod routing; mod spanning_tree; mod unit; diff --git a/src/node/tests/routing.rs b/src/node/tests/routing.rs new file mode 100644 index 0000000..9a886a6 --- /dev/null +++ b/src/node/tests/routing.rs @@ -0,0 +1,518 @@ +//! Routing integration tests. +//! +//! Tests the full Node::find_next_hop() routing logic including bloom +//! filter priority, greedy tree routing, and tie-breaking. + +use super::*; +use crate::bloom::BloomFilter; +use crate::tree::{ParentDeclaration, TreeCoordinate}; +use spanning_tree::{ + cleanup_nodes, drain_all_packets, generate_random_edges, initiate_handshake, make_test_node, + run_tree_test, verify_tree_convergence, TestNode, +}; +use std::collections::HashSet; + +// === Local delivery === + +#[test] +fn test_routing_local_delivery() { + let node = make_node(); + let my_addr = *node.node_addr(); + assert!(node.find_next_hop(&my_addr).is_none()); +} + +// === Direct peer === + +#[test] +fn test_routing_direct_peer() { + let mut node = make_node(); + let transport_id = TransportId::new(1); + let link_id = LinkId::new(1); + + let (conn, identity) = make_completed_connection(&mut node, link_id, transport_id, 1000); + let peer_addr = *identity.node_addr(); + node.add_connection(conn).unwrap(); + node.promote_connection(link_id, identity, 2000).unwrap(); + + let result = node.find_next_hop(&peer_addr); + assert!(result.is_some()); + assert_eq!(result.unwrap().node_addr(), &peer_addr); +} + +// === No route === + +#[test] +fn test_routing_unknown_destination() { + let node = make_node(); + let unknown = make_node_addr(99); + assert!(node.find_next_hop(&unknown).is_none()); +} + +// === Bloom filter priority === + +#[test] +fn test_routing_bloom_filter_hit() { + let mut node = make_node(); + let transport_id = TransportId::new(1); + + // Create two peers + let link_id1 = LinkId::new(1); + let (conn1, id1) = make_completed_connection(&mut node, link_id1, transport_id, 1000); + let peer1_addr = *id1.node_addr(); + node.add_connection(conn1).unwrap(); + node.promote_connection(link_id1, id1, 2000).unwrap(); + + let link_id2 = LinkId::new(2); + let (conn2, id2) = make_completed_connection(&mut node, link_id2, transport_id, 1000); + let peer2_addr = *id2.node_addr(); + node.add_connection(conn2).unwrap(); + node.promote_connection(link_id2, id2, 2000).unwrap(); + + // Destination not directly connected + let dest = make_node_addr(99); + + // Add dest to peer1's bloom filter only + let peer1 = node.get_peer_mut(&peer1_addr).unwrap(); + let mut filter = BloomFilter::new(); + filter.insert(&dest); + peer1.update_filter(filter, 1, 3000); + + // Should route through peer1 (bloom filter hit) + let result = node.find_next_hop(&dest); + assert!(result.is_some()); + assert_eq!(result.unwrap().node_addr(), &peer1_addr); + + // Peer2 should NOT be selected (no filter hit) + assert_ne!(result.unwrap().node_addr(), &peer2_addr); +} + +#[test] +fn test_routing_bloom_filter_multiple_hits_tiebreak() { + let mut node = make_node(); + let transport_id = TransportId::new(1); + + // Create three peers + let mut peer_addrs = Vec::new(); + for i in 1..=3 { + let link_id = LinkId::new(i); + let (conn, id) = make_completed_connection(&mut node, link_id, transport_id, 1000); + let addr = *id.node_addr(); + peer_addrs.push(addr); + node.add_connection(conn).unwrap(); + node.promote_connection(link_id, id, 2000).unwrap(); + } + + let dest = make_node_addr(99); + + // Add dest to ALL peers' bloom filters + for &addr in &peer_addrs { + let peer = node.get_peer_mut(&addr).unwrap(); + let mut filter = BloomFilter::new(); + filter.insert(&dest); + peer.update_filter(filter, 1, 3000); + } + + // All peers have equal link_cost (1.0) and no tree coords set, + // so tree distance is usize::MAX for all. Tie-break by smallest node_addr. + let result = node.find_next_hop(&dest); + assert!(result.is_some()); + + let smallest_addr = peer_addrs.iter().min().unwrap(); + assert_eq!(result.unwrap().node_addr(), smallest_addr); +} + +// === Greedy tree routing === + +#[test] +fn test_routing_tree_fallback() { + let mut node = make_node(); + let transport_id = TransportId::new(1); + let my_addr = *node.node_addr(); + + // Create a peer + let link_id = LinkId::new(1); + let (conn, id) = make_completed_connection(&mut node, link_id, transport_id, 1000); + let peer_addr = *id.node_addr(); + node.add_connection(conn).unwrap(); + node.promote_connection(link_id, id, 2000).unwrap(); + + // Set up tree state through the public API. + // We're root, peer is our child. The peer has a subtree below it. + // TreeState::new() already makes us the root with coords [my_addr]. + // Add peer as child of us. + let peer_coords = TreeCoordinate::from_addrs(vec![peer_addr, my_addr]).unwrap(); + node.tree_state_mut().update_peer( + ParentDeclaration::new(peer_addr, my_addr, 1, 1000), + peer_coords, + ); + + // Destination: a node under our peer in the tree + let dest = make_node_addr(99); + let dest_coords = + TreeCoordinate::from_addrs(vec![dest, peer_addr, my_addr]).unwrap(); + + // Put dest coords in the cache + let now_ms = std::time::SystemTime::now() + .duration_since(std::time::UNIX_EPOCH) + .map(|d| d.as_millis() as u64) + .unwrap_or(0); + node.coord_cache_mut().insert(dest, dest_coords, now_ms); + + // No bloom filter hit — should fall back to tree routing. + // Our distance to dest: 2 (root → peer → dest) + // Peer's distance to dest: 1 (peer → dest) + // Peer is closer, so it's the next hop. + let result = node.find_next_hop(&dest); + assert!(result.is_some()); + assert_eq!(result.unwrap().node_addr(), &peer_addr); +} + +#[test] +fn test_routing_tree_no_coords_in_cache() { + let mut node = make_node(); + let transport_id = TransportId::new(1); + + // Create a peer + let link_id = LinkId::new(1); + let (conn, id) = make_completed_connection(&mut node, link_id, transport_id, 1000); + node.add_connection(conn).unwrap(); + node.promote_connection(link_id, id, 2000).unwrap(); + + // Destination not in bloom filters and not in coord cache + let dest = make_node_addr(99); + assert!(node.find_next_hop(&dest).is_none()); +} + +// === Integration: converged network === + +#[tokio::test] +async fn test_routing_chain_topology() { + // Build a 4-node chain: 0 -- 1 -- 2 -- 3 + let mut nodes = vec![ + make_test_node().await, + make_test_node().await, + make_test_node().await, + make_test_node().await, + ]; + + // Connect the chain + initiate_handshake(&mut nodes, 0, 1).await; + initiate_handshake(&mut nodes, 1, 2).await; + initiate_handshake(&mut nodes, 2, 3).await; + + // Converge tree and bloom filters + drain_all_packets(&mut nodes, false).await; + + // Verify tree convergence + let root = nodes.iter().map(|n| *n.node.node_addr()).min().unwrap(); + for tn in &nodes { + assert_eq!( + *tn.node.tree_state().root(), + root, + "Tree not converged" + ); + } + + // Populate coord caches: each node caches the far-end node's coords + let now_ms = std::time::SystemTime::now() + .duration_since(std::time::UNIX_EPOCH) + .map(|d| d.as_millis() as u64) + .unwrap_or(0); + + let node3_addr = *nodes[3].node.node_addr(); + let node3_coords = nodes[3].node.tree_state().my_coords().clone(); + nodes[0] + .node + .coord_cache_mut() + .insert(node3_addr, node3_coords, now_ms); + + let node0_addr = *nodes[0].node.node_addr(); + let node0_coords = nodes[0].node.tree_state().my_coords().clone(); + nodes[3] + .node + .coord_cache_mut() + .insert(node0_addr, node0_coords, now_ms); + + // Node 0 should be able to route toward node 3. + // The next hop should be node 1 (only peer of node 0). + let hop = nodes[0].node.find_next_hop(&node3_addr); + assert!(hop.is_some(), "Node 0 should find route to node 3"); + let node1_addr = *nodes[1].node.node_addr(); + assert_eq!( + hop.unwrap().node_addr(), + &node1_addr, + "Node 0's next hop to node 3 should be node 1" + ); + + // Node 3 should route toward node 0 via node 2. + let hop = nodes[3].node.find_next_hop(&node0_addr); + assert!(hop.is_some(), "Node 3 should find route to node 0"); + let node2_addr = *nodes[2].node.node_addr(); + assert_eq!( + hop.unwrap().node_addr(), + &node2_addr, + "Node 3's next hop to node 0 should be node 2" + ); +} + +#[tokio::test] +async fn test_routing_bloom_preferred_over_tree() { + // Build a 3-node triangle: 0 -- 1, 0 -- 2, 1 -- 2 + let mut nodes = vec![ + make_test_node().await, + make_test_node().await, + make_test_node().await, + ]; + + initiate_handshake(&mut nodes, 0, 1).await; + initiate_handshake(&mut nodes, 0, 2).await; + initiate_handshake(&mut nodes, 1, 2).await; + + drain_all_packets(&mut nodes, false).await; + + // Create a destination beyond the network + let dest = make_node_addr(99); + + // Add dest to peer 2's bloom filter (from node 0's perspective) + let peer2_addr = *nodes[2].node.node_addr(); + let peer2 = nodes[0].node.get_peer_mut(&peer2_addr).unwrap(); + let mut filter = BloomFilter::new(); + filter.insert(&dest); + peer2.update_filter(filter, 100, 50000); + + // Even though we could use tree routing (if coords were cached), + // the bloom filter hit should be preferred. + let hop = nodes[0].node.find_next_hop(&dest); + assert!(hop.is_some(), "Should route via bloom filter"); + assert_eq!( + hop.unwrap().node_addr(), + &peer2_addr, + "Should pick peer with bloom filter hit" + ); +} + +// === Multi-hop forwarding simulation === + +/// Result of simulating multi-hop packet forwarding. +#[derive(Debug)] +enum ForwardResult { + /// Packet reached the destination in the given number of hops. + Delivered(usize), + /// Routing returned None at the given node index (no route). + NoRoute { at_node: usize, hops: usize }, + /// Routing loop detected (visited the same node twice). + Loop { at_node: usize, hops: usize }, +} + +/// Build a NodeAddr → node index lookup table. +fn build_addr_index(nodes: &[TestNode]) -> std::collections::HashMap { + nodes + .iter() + .enumerate() + .map(|(i, tn)| (*tn.node.node_addr(), i)) + .collect() +} + +/// Simulate multi-hop forwarding from source to destination. +/// +/// At each hop, calls `find_next_hop` on the current node and follows +/// the result to the next node. Terminates on delivery, routing failure, +/// or loop detection. +fn simulate_forwarding( + nodes: &[TestNode], + addr_index: &std::collections::HashMap, + src: usize, + dst: usize, +) -> ForwardResult { + let dest_addr = *nodes[dst].node.node_addr(); + let max_hops = nodes.len(); // can't take more hops than nodes + + let mut current = src; + let mut visited = HashSet::new(); + visited.insert(current); + + for hop in 0..max_hops { + let next = nodes[current].node.find_next_hop(&dest_addr); + + match next { + None => { + // find_next_hop returns None for local delivery (dest == self) + if *nodes[current].node.node_addr() == dest_addr { + return ForwardResult::Delivered(hop); + } + return ForwardResult::NoRoute { + at_node: current, + hops: hop, + }; + } + Some(peer) => { + let next_addr = *peer.node_addr(); + + // Is next hop the destination? + if next_addr == dest_addr { + return ForwardResult::Delivered(hop + 1); + } + + // Find the node index for the next hop + let next_idx = match addr_index.get(&next_addr) { + Some(&idx) => idx, + None => { + return ForwardResult::NoRoute { + at_node: current, + hops: hop, + }; + } + }; + + // Loop detection + if visited.contains(&next_idx) { + return ForwardResult::Loop { + at_node: next_idx, + hops: hop + 1, + }; + } + + visited.insert(next_idx); + current = next_idx; + } + } + } + + ForwardResult::NoRoute { + at_node: current, + hops: max_hops, + } +} + +/// 100-node random graph: verify all-pairs routing reachability. +/// +/// After tree and bloom filter convergence, simulates multi-hop packet +/// forwarding between every pair of nodes. Every packet must be delivered +/// without loops. +#[tokio::test] +async fn test_routing_reachability_100_nodes() { + const NUM_NODES: usize = 100; + const TARGET_EDGES: usize = 250; + const SEED: u64 = 42; + + let edges = generate_random_edges(NUM_NODES, TARGET_EDGES, SEED); + let mut nodes = run_tree_test(NUM_NODES, &edges, false).await; + verify_tree_convergence(&nodes); + + // Populate coord caches: every node learns every other node's coordinates. + // In production this happens via SessionSetup/LookupResponse; here we + // inject them directly so routing can make progress-based decisions. + let now_ms = std::time::SystemTime::now() + .duration_since(std::time::UNIX_EPOCH) + .map(|d| d.as_millis() as u64) + .unwrap_or(0); + + // Collect all (addr, coords) pairs first to avoid borrow issues + let all_coords: Vec<(NodeAddr, TreeCoordinate)> = nodes + .iter() + .map(|tn| (*tn.node.node_addr(), tn.node.tree_state().my_coords().clone())) + .collect(); + + for node in &mut nodes { + for &(ref addr, ref coords) in &all_coords { + if addr != node.node.node_addr() { + node.node.coord_cache_mut().insert(*addr, coords.clone(), now_ms); + } + } + } + + let addr_index = build_addr_index(&nodes); + + let mut total_pairs = 0; + let mut total_hops = 0usize; + let mut max_hops = 0usize; + let mut failures = Vec::new(); + let mut loops = Vec::new(); + + // Test all pairs + for src in 0..NUM_NODES { + for dst in 0..NUM_NODES { + if src == dst { + continue; + } + + total_pairs += 1; + + match simulate_forwarding(&nodes, &addr_index, src, dst) { + ForwardResult::Delivered(hops) => { + total_hops += hops; + if hops > max_hops { + max_hops = hops; + } + } + ForwardResult::NoRoute { at_node, hops } => { + failures.push((src, dst, at_node, hops)); + } + ForwardResult::Loop { at_node, hops } => { + loops.push((src, dst, at_node, hops)); + } + } + } + } + + let delivered = total_pairs - failures.len() - loops.len(); + let avg_hops = if delivered > 0 { + total_hops as f64 / delivered as f64 + } else { + 0.0 + }; + + eprintln!( + "\n === Routing Reachability ({} nodes) ===", + NUM_NODES + ); + eprintln!( + " Pairs tested: {} | Delivered: {} | Failed: {} | Loops: {}", + total_pairs, + delivered, + failures.len(), + loops.len() + ); + eprintln!( + " Hops: avg={:.1} max={}", + avg_hops, max_hops + ); + + if !failures.is_empty() { + let show = failures.len().min(10); + eprintln!(" First {} failures:", show); + for &(src, dst, at_node, hops) in &failures[..show] { + eprintln!( + " {} -> {}: stuck at node {} after {} hops", + src, dst, at_node, hops + ); + } + } + + if !loops.is_empty() { + let show = loops.len().min(10); + eprintln!(" First {} loops:", show); + for &(src, dst, at_node, hops) in &loops[..show] { + eprintln!( + " {} -> {}: loop at node {} after {} hops", + src, dst, at_node, hops + ); + } + } + + assert!( + loops.is_empty(), + "Detected {} routing loops out of {} pairs", + loops.len(), + total_pairs + ); + assert!( + failures.is_empty(), + "Detected {} routing failures out of {} pairs", + failures.len(), + total_pairs + ); + + cleanup_nodes(&mut nodes).await; +} + diff --git a/src/peer/active.rs b/src/peer/active.rs index 395c2bc..755fce2 100644 --- a/src/peer/active.rs +++ b/src/peer/active.rs @@ -391,6 +391,15 @@ impl ActivePeer { &mut self.link_stats } + /// Link cost for routing decisions. + /// + /// Returns a scalar cost where lower is better. Currently returns a + /// constant (all links equal). Future versions will compute from RTT, + /// loss rate, and throughput measurements. + pub fn link_cost(&self) -> f64 { + 1.0 + } + /// When this peer was authenticated. pub fn authenticated_at(&self) -> u64 { self.authenticated_at diff --git a/src/tree.rs b/src/tree.rs index 2f305a8..f094001 100644 --- a/src/tree.rs +++ b/src/tree.rs @@ -589,13 +589,46 @@ impl TreeState { .map(|coords| self.my_coords.distance_to(coords)) } - /// Find the best next hop toward a destination. + /// Find the best next hop toward a destination using greedy tree routing. /// - /// Returns the peer that minimizes tree distance to the destination. - /// This is a stub - full implementation requires greedy routing logic. - pub fn find_next_hop(&self, _dest_coords: &TreeCoordinate) -> Option { - // Stub: would implement greedy tree routing - None + /// Returns the peer that minimizes tree distance to the destination, + /// but only if that peer is strictly closer than we are (prevents + /// routing loops at local minima). Tie-breaks equal distance by + /// smallest node_addr. + /// + /// Returns `None` if: + /// - No peers have coordinates + /// - Destination is in a different tree (different root) + /// - No peer is closer to the destination than we are + pub fn find_next_hop(&self, dest_coords: &TreeCoordinate) -> Option { + if self.my_coords.root_id() != dest_coords.root_id() { + return None; + } + + let my_distance = self.my_coords.distance_to(dest_coords); + + let mut best: Option<(NodeAddr, usize)> = None; + + for (peer_id, peer_coords) in &self.peer_ancestry { + let distance = peer_coords.distance_to(dest_coords); + + let dominated = match &best { + None => true, + Some((best_id, best_dist)) => { + distance < *best_dist + || (distance == *best_dist && peer_id < best_id) + } + }; + + if dominated { + best = Some((*peer_id, distance)); + } + } + + match best { + Some((peer_id, distance)) if distance < my_distance => Some(peer_id), + _ => None, + } } /// Minimum depth improvement required to switch parents (same root). @@ -1301,4 +1334,142 @@ mod tests { assert!(state.my_declaration().sequence() > seq_before); assert_eq!(state.root(), &my_node); } + + // === find_next_hop tests === + + /// Build a TreeState with our own coordinates set. + fn make_tree_state(my_addr: u8, coord_path: &[u8]) -> TreeState { + let my_node = make_node_addr(my_addr); + let mut state = TreeState::new(my_node); + let coords = make_coords(coord_path); + state.root = *coords.root_id(); + state.my_coords = coords; + state + } + + /// Add a peer with given coordinates to the tree state. + fn add_peer(state: &mut TreeState, peer_addr: u8, coord_path: &[u8]) { + let peer = make_node_addr(peer_addr); + let parent = make_node_addr(coord_path[1]); + state.update_peer( + ParentDeclaration::new(peer, parent, 1, 1000), + make_coords(coord_path), + ); + } + + #[test] + fn test_find_next_hop_chain() { + // Chain: 0 (root) <- 5 (us) <- 1 <- 2 + // Both peers 1 and 2 are in our peer_ancestry. Peer 2 IS the + // destination (distance 0), so it's the best next hop. + let mut state = make_tree_state(5, &[5, 0]); + add_peer(&mut state, 1, &[1, 5, 0]); + add_peer(&mut state, 2, &[2, 1, 5, 0]); + + let dest = make_coords(&[2, 1, 5, 0]); + assert_eq!(state.find_next_hop(&dest), Some(make_node_addr(2))); + } + + #[test] + fn test_find_next_hop_chain_indirect() { + // Chain: 0 (root) <- 5 (us) <- 1 + // Dest is node 2 at [2, 1, 5, 0] but peer 2 is NOT in our peer + // list — only peer 1 is. So we route via peer 1 (distance 1). + let mut state = make_tree_state(5, &[5, 0]); + add_peer(&mut state, 1, &[1, 5, 0]); + + let dest = make_coords(&[2, 1, 5, 0]); + assert_eq!(state.find_next_hop(&dest), Some(make_node_addr(1))); + } + + #[test] + fn test_find_next_hop_toward_root() { + // Tree: 0 (root) <- 1 <- 5 (us) + // Routing toward root should pick node 1 (our parent). + let mut state = make_tree_state(5, &[5, 1, 0]); + add_peer(&mut state, 1, &[1, 0]); + + let dest = make_coords(&[0]); + assert_eq!(state.find_next_hop(&dest), Some(make_node_addr(1))); + } + + #[test] + fn test_find_next_hop_sibling() { + // Tree: 0 (root) <- 5 (us), 0 <- 3 + // Routing to sibling 3: should go through parent 0... but 0 is + // the root and not in our peer list. Our only peer is 3 itself. + // But 3 is not a "closer" peer in tree distance — distance from + // us to 3 is 2 (up to root, down to 3), and distance from 3 to + // 3 is 0, so 3 IS closer. Should pick 3. + let mut state = make_tree_state(5, &[5, 0]); + add_peer(&mut state, 3, &[3, 0]); + + let dest = make_coords(&[3, 0]); + assert_eq!(state.find_next_hop(&dest), Some(make_node_addr(3))); + } + + #[test] + fn test_find_next_hop_tie_breaking() { + // Tree: 0 (root) <- 5 (us), 0 <- 3, 0 <- 2 + // Both peers are siblings at depth 1, equidistant to a dest + // at [4, 0]. Should pick node 2 (smaller node_addr). + let mut state = make_tree_state(5, &[5, 0]); + add_peer(&mut state, 3, &[3, 0]); + add_peer(&mut state, 2, &[2, 0]); + + let dest = make_coords(&[4, 0]); + // Our distance: 2 (up to root, down to 4) + // Peer 3 distance: 2 (up to root, down to 4) + // Peer 2 distance: 2 (up to root, down to 4) + // All equal to our distance — no peer is strictly closer. + assert_eq!(state.find_next_hop(&dest), None); + } + + #[test] + fn test_find_next_hop_different_root() { + let mut state = make_tree_state(5, &[5, 0]); + add_peer(&mut state, 1, &[1, 0]); + + // Destination in a different tree (root = 9) + let dest = make_coords(&[3, 9]); + assert_eq!(state.find_next_hop(&dest), None); + } + + #[test] + fn test_find_next_hop_no_peers() { + let state = make_tree_state(5, &[5, 0]); + let dest = make_coords(&[3, 0]); + assert_eq!(state.find_next_hop(&dest), None); + } + + #[test] + fn test_find_next_hop_local_minimum() { + // Tree: 0 (root) <- 5 (us), 5 <- 8 + // Routing to node 3 at [3, 0]. Our distance = 2. + // Peer 8's distance = 4 (8→5→0→3 but via coords: [8,5,0] to [3,0] = 3). + // Actually: lca of [8,5,0] and [3,0] is root 0 at depth 0. + // dist = (2-0) + (1-0) = 3. Our dist = (1-0) + (1-0) = 2. + // Peer is farther, so no hop. + let mut state = make_tree_state(5, &[5, 0]); + add_peer(&mut state, 8, &[8, 5, 0]); + + let dest = make_coords(&[3, 0]); + assert_eq!(state.find_next_hop(&dest), None); + } + + #[test] + fn test_find_next_hop_best_of_multiple() { + // Tree: 0 (root) <- 1 <- 5 (us), 1 <- 3 <- 7 + // Dest is node 7 at [7, 3, 1, 0]. + // Peer 1 coords [1, 0]: dist to dest = 0 + 2 = 2 + // Peer 3 coords [3, 1, 0]: dist to dest = 0 + 1 = 1 + // Our coords [5, 1, 0]: dist to dest = 1 + 2 = 3 + // Peer 3 is closest. Should pick 3. + let mut state = make_tree_state(5, &[5, 1, 0]); + add_peer(&mut state, 1, &[1, 0]); + add_peer(&mut state, 3, &[3, 1, 0]); + + let dest = make_coords(&[7, 3, 1, 0]); + assert_eq!(state.find_next_hop(&dest), Some(make_node_addr(3))); + } }