Implement TCP transport for FIPS enabling firewall traversal and serving
as the foundation for future Tor transport. This is the first
connection-oriented transport in the system.
Key design decisions:
- FMP header-based framing: reuses existing 4-byte FMP common prefix for
packet boundary recovery with zero framing overhead
- Session survives TCP reconnection: Noise/MMP/FSP state bound to npub,
not TCP connection; MMP liveness is sole authority for peer death
- Connect-on-send: fresh connection on first send, transparent reconnect
- close_connection() trait method for cross-connection deduplication cleanup
New transport files:
- src/transport/tcp/mod.rs: TcpTransport, connection pool, accept loop
- src/transport/tcp/stream.rs: FMP-aware stream reader (shared with Tor)
Modified: transport trait (close_connection), TcpConfig, TransportHandle
match arms, create_transports(), initiate_connection() for connection-
oriented links, cross-connection tie-breaker cleanup, design docs.
Tree announce loop and TCP stability fixes:
- Preserve tree announce rate-limit state across reconnection: carry
forward last_tree_announce_sent_ms when a peer reconnects so the
rate-limit window isn't reset to zero
- Drop oversize TCP packets at sender: pre-send MTU check returns
MtuExceeded instead of writing to the stream, preventing receiver-side
connection teardown and reset-reconnect cycles
Chaos harness:
- TCP transport support: tcp_edges/has_tcp/tcp_peers in SimTopology,
transport-aware config_gen with per-edge transport type, TCP port 443,
pure-TCP node support
- Include all non-Ethernet edges in directed_outbound()
- Fix netem/links log messages to say "IP-based" instead of "UDP"
- Add tcp-chain, tcp-only, and tcp-mesh scenario files
Static harness:
- Transport-aware config generation (get_default_transport, transport_port)
- TCP transport injection via Python post-processing
- Add tcp-chain topology and docker-compose profile
Bump rand (0.8→0.10), rtnetlink (0.14→0.20), tun (0.7→0.8),
simple-dns (0.9→0.11), socket2 (0.5→0.6), and criterion (0.5→0.8).
Migrate all rand call sites: thread_rng()→rng(), gen()→random(),
gen_range()→random_range(), RngCore→Rng trait. Work around secp256k1
0.30 requiring rand 0.8 by generating random bytes directly and
constructing SecretKey from slice.
Migrate rtnetlink to builder-based API: LinkSetRequest replaced with
LinkUnspec builder + change(), RouteAddRequest replaced with
RouteMessageBuilder.
Remove bloom benchmark (criterion 0.8 incompatible with old harness
config).
Add per-peer replay suppression counter to ActivePeer. Log the first 3
replay detections at DEBUG, then suppress with a one-time notice. Emit
a summary count on session replacement or peer removal.
Non-replay decryption errors continue to be logged unconditionally.
Cost-based parent selection:
- Replace depth-only parent selection with effective_depth = depth + link_cost
- link_cost computed from locally measured MMP metrics: etx * (1.0 + srtt_ms / 100.0)
- Prevents bottleneck subtrees in heterogeneous networks where a LoRa link
at depth 1 would otherwise always beat fiber at depth 2
- Configurable hysteresis (default 0.2) prevents marginal parent switches
- Configurable hold-down timer (default 30s) suppresses re-evaluation
after parent switch
- Mandatory switches (parent lost, root change) bypass both safeguards
- Link costs passed as HashMap parameter to keep TreeState pure
Periodic re-evaluation:
- evaluate_parent() was only called on TreeAnnounce receipt or parent loss;
after tree stabilization, link degradation went undetected
- Added timer-based re-evaluation (reeval_interval_secs, default 60s) that
calls evaluate_parent() from the tick handler with current MMP link costs
- Respects existing hold-down and hysteresis safeguards
- Short-circuits when disabled or <2 peers
Design documentation:
- Update 7 design docs to reflect cost-based parent selection
- Replace depth-only algorithm descriptions with effective_depth model
- Replace rejected cumulative path cost spec with local-only design rationale
- Rewrite Example 2 (heterogeneous links) for local-only cost model
- Update config docs: parent_switch_threshold replaced by parent_hysteresis,
hold_down_secs, reeval_interval_secs
Chaos simulation enhancements:
- fips_overrides with deep merge for per-scenario FIPS config customization
- Explicit topology algorithm for deterministic test graphs
- Control socket querying via fipsctl for tree/MMP snapshot collection
- Edge existence validation in netem manager
- Per-link netem policy overrides
- 9 new chaos scenarios covering cost avoidance, depth-vs-cost tradeoffs,
stability, mixed topologies, periodic re-evaluation, and bottleneck parent
12 new unit tests, 667 total passing, clippy clean.
Each node generates a random 8-byte startup epoch, encrypted inside
both Noise IK handshake messages (msg1 and msg2). When a peer's msg1
arrives with a different epoch than the stored value, the node tears
down the stale session and processes the msg1 as a new connection,
enabling near-instant restart detection instead of the 30-second
dead timeout.
Wire format impact:
- msg1: 82 -> 106 bytes (added 24-byte encrypted epoch after ss DH)
- msg2: 33 -> 57 bytes (added 24-byte encrypted epoch after se DH)
- Wire msg1: 90 -> 114 bytes, wire msg2: 45 -> 69 bytes
Sends a 1-byte encrypted heartbeat (0x51) to each peer every 10s.
If no frame is received from a peer within 30s, the peer is removed
via remove_active_peer(), triggering tree reconvergence, coord cache
flush, and bloom filter recomputation.
This fixes the critical bug where UDP peers that silently died
(e.g., container stopped) were never detected or removed, leaving
the spanning tree permanently stale.
Both intervals are configurable via node.heartbeat_interval_secs
and node.link_dead_timeout_secs.
Add message-level retry for Noise IK handshake within the 30s timeout
window. Previously, a lost msg1 or msg2 required the full timeout to
expire before cleanup and retry. Under 10% bidirectional loss (~19%
per attempt), this made connection establishment unreliable.
Initiator resends stored msg1 bytes with exponential backoff (1s, 2s,
4s, 8s, 16s — 5 resends). Responder stores msg2 and resends on
duplicate msg1 receipt. Duplicate msg2 at initiator drops silently
via existing pending_outbound cleanup.
Config: handshake_resend_interval_ms (1000), handshake_resend_backoff
(2.0), handshake_max_resends (5) on node.rate_limit.
P(all 6 attempts fail) under 19% loss = 0.19^6 ≈ 0.005%.
Add the full next-hop routing algorithm to Node::find_next_hop():
- Local delivery, direct peer, bloom filter candidates, greedy tree
routing fallback, with (link_cost, tree_distance, node_addr) ordering
- select_best_candidate() scores by peer→dest distance (not us→peer)
with self-distance check to prevent routing loops
- TreeState::find_next_hop() for greedy tree routing with progress
guarantee
- ActivePeer::link_cost() placeholder (constant 1.0) for future link
quality metrics
Add routing tests including 100-node all-pairs reachability simulation
(9900/9900 delivered, 0 loops, avg 4.0 hops, max 8).
Update fips-routing.md to reflect bloom filter routing as the primary
forwarding mechanism, with greedy tree routing as fallback during
convergence windows.
When both nodes simultaneously initiate connections, each promotes its
inbound handshake first, leaving both sides with mismatched Noise sessions
and session indices. Resolve this in handle_msg2 using the existing
tie-breaker (smaller node_addr's outbound wins):
- Winner (smaller node): swaps ActivePeer to outbound session + indices
via new replace_session() method
- Loser (larger node): keeps inbound session and original their_index
(already references winner's outbound our_index from msg1)
Key insight: the loser must NOT update their_index from msg2, because the
msg2 sender_idx carries the winner's inbound index which becomes stale
after the winner swaps.
Defer outbound connection cleanup from promote_connection() to handle_msg2()
so outbound session state is available for the swap.
Verified with live two-node test over UDP in network namespaces: both nodes
connect, exchange TreeAnnounce, and converge to spanning tree within ~2s.
Fix promote_connection() to detect and clean up pending outbound
handshakes to the same peer, not just already-promoted peers. Previously,
when an inbound handshake completed while an outbound was still pending,
the outbound would linger until the 30s timeout.
Add auto-retry for failed outbound connections to auto-connect peers:
- New RetryState struct and node/retry.rs module
- Exponential backoff (default 5s base, max 5 attempts)
- Config: node.max_retries, node.base_retry_interval_secs
- check_timeouts() schedules retries, rx loop processes them
- promote_connection() clears retry state on success
- Remove unused PeerConnection retry fields (state now at Node level)
Move cleanup_stale_connection() logging to callers for context-appropriate
messages.
289 tests pass, clean build.
Rename NodeId to NodeAddr and npub to pubkey throughout documentation
and source code for consistency with the established identity model.
Add FIPS API vs IPv6 adapter overview to session protocol document.
Add transport address terminology note to wire protocol document.
Johnathan Corgan