Design documentation illustration and review pass

Wire format diagrams:
- Add 24 SVG diagrams covering every FMP and FSP wire format: common
  prefix, established frame headers, Noise IK handshake messages,
  handshake flow, TreeAnnounce, AncestryEntry, FilterAnnounce,
  LookupRequest/Response, SessionDatagram, Disconnect, SenderReport,
  ReceiverReport, FSP complete message, SessionSetup/Ack/Msg3,
  PathMtuNotification, CoordsRequired, PathBroken, and MtuExceeded
- Replace ASCII art in fips-wire-formats.md with SVG references
- Apply text edits to fips-mesh-layer.md, fips-mesh-operation.md,
  fips-transport-layer.md, and fips-ipv6-adapter.md

Spanning tree dynamics:
- Add 12 topology SVG diagrams: node join (overview + 3-panel steps),
  three-node convergence (4-panel), link addition with depth labels,
  link removal, partition formation, and 6 real-world example diagrams
  (office, mixed-link, two-site WAN topologies)
- Rewrite all code blocks to narrative prose with diagram references
- Add inline prior art attributions distinguishing Yggdrasil-derived
  concepts from FIPS-novel contributions
- Add 3 new references (De Couto ETX, IEEE 802.1D, RFC 2328 OSPF) and
  Prior Art summary
- Remove outdated sections: indirect partition note, integration test
  gaps, DHT-based lookup reference
- Change "must elect a new root" to "must rediscover its new root"

Spanning tree design review (fips-spanning-tree.md):
- Rename "Root Election" to "Root Discovery" across docs
- Add "What Is a Spanning Tree?" introductory section
- Add parent selection intro explaining self-organization role
- Fix tree distance example: 4 hops, not 2
- Clarify timestamp field as advisory only
- Remove unimplemented ROOT_TIMEOUT and TREE_ENTRY_TTL from timing
  parameters and implementation status tables

Bloom filter design review (fips-bloom-filters.md):
- Add "What Is a Bloom Filter?" intro section
- Rewrite Purpose section to frame filters as routing path identification
- Correct FPR analysis (old values were 3-50x overstated)
- Add Filter Occupancy Model based on network size and tree position
- Fix filter expiration to describe actual MMP-based cleanup
- Combine Scale Considerations with Size Classes after Wire Format
- Fix stale FPR values in src/bloom/mod.rs comments

Session layer review (fips-session-layer.md):
- Add inline prior art attributions: Noise Protocol Framework, WireGuard,
  DTLS (RFC 6347), IKEv2 (RFC 7296), RFC 1191 PMTUD, Yggdrasil, NIP-44
- Replace warmup state machine ASCII art with SVG diagram
- Convert CoordsWarmup wire format code block to prose
- Add External References section with full citations

Level 5 implementation doc cleanup:
- Delete fips-software-architecture.md (redundant with protocol layer docs)
- Delete fips-state-machines.md (Rust tutorial, not protocol design)
- Add fipsctl command reference to README.md
- Update cross-references in fips-intro.md, docs/design/README.md,
  fips-transport-layer.md, fips-configuration.md

Fixes:
- Correct fd::/8 to fd00::/8 in fips-session-layer.md,
  fips-identity-derivation.svg, and fips-node-architecture.svg
- Fix config example MTU: 1197 → 1472 in fips-configuration.md

File organization:
- Move all SVG diagrams into docs/design/diagrams/ subdirectory
- Update all diagram references to use new paths
This commit is contained in:
Johnathan Corgan
2026-02-24 17:32:37 +00:00
parent c1820e689d
commit 00e26765bd
62 changed files with 3232 additions and 1754 deletions
+101 -26
View File
@@ -23,8 +23,8 @@ Two complementary mechanisms provide the information each node needs:
ancestry path from itself to the root. These coordinates enable distance
calculations between any two nodes without global topology knowledge.
- **Bloom filters** summarize which destinations are reachable through each
peer. They provide candidate selection — narrowing which peers are worth
considering for forwarding a given destination.
peer. Because they propagate along tree edges, they encode directional
reachability — which subtree contains a given destination.
Together, they enable a routing decision process that is local, efficient,
and self-healing.
@@ -48,7 +48,7 @@ enable:
Nodes self-organize into a spanning tree through distributed parent selection:
1. **Root election**: The node with the smallest node_addr becomes the root.
1. **Root discovery**: The node with the smallest node_addr becomes the root.
No election protocol — this is a consequence of each node independently
preferring lower-addressed roots.
2. **Parent selection**: Each node selects a single parent from among its
@@ -75,10 +75,76 @@ Changes cascade through the tree:
TreeAnnounce propagation is rate-limited at 500ms minimum interval per peer.
A tree of depth D reconverges in roughly D×0.5s to D×1.0s.
### How the Tree Adapts to Link Quality
The initial tree forms based on hop count alone — all links default to a
cost of 1.0 before measurements are available. As the Metrics Measurement
Protocol (MMP) accumulates bidirectional delivery ratios and round-trip
time estimates, each node computes a per-link cost:
```text
link_cost = ETX × (1.0 + SRTT_ms / 100.0)
```
ETX (Expected Transmission Count) captures loss — a perfect link has
ETX = 1.0, while 10% loss in each direction yields ETX ≈ 1.23. The SRTT
term weights latency so that a low-loss but high-latency link (e.g., a
satellite hop) costs more than a low-loss, low-latency link.
Parent selection uses **effective depth** rather than raw hop count:
```text
effective_depth = peer.depth + link_cost_to_peer
```
This allows a node to trade a shorter but lossy path for a longer but
higher-quality one. A node two hops from the root over clean links
(effective depth ≈ 3.0) is preferred over a node one hop away over a
degraded link (effective depth ≈ 4.5).
Parent reselection is triggered by three paths:
1. **TreeAnnounce**: When a peer announces a new tree position, the node
re-evaluates using current link costs
2. **Periodic re-evaluation**: Every 60s (configurable), the node
re-evaluates its parent choice using the latest MMP metrics, catching
gradual link degradation that doesn't trigger TreeAnnounce
3. **Parent loss**: When the current parent is removed, the node
immediately selects the best alternative
To prevent oscillation from metric noise, parent switches are subject to
**hysteresis**: a candidate must offer an effective depth at least 20%
better than the current parent to trigger a switch. A **hold-down period**
(default 30s) suppresses non-mandatory re-evaluation after a switch,
allowing MMP metrics to stabilize on the new link before reconsidering.
### Flap Dampening
Unstable links that repeatedly connect and disconnect can cause cascading
tree reconvergence. The spanning tree uses flap dampening with hysteresis
and hold-down periods to suppress rapid parent oscillation. Links that flap
above a configurable threshold are temporarily penalized, preventing them
from being selected as parent until the link stabilizes.
### Link Liveness
Each node sends a dedicated **Heartbeat** message (0x51, 1 byte, no
payload) to every peer at a fixed interval (default 10s). Any
authenticated encrypted frame — heartbeat, MMP report, TreeAnnounce,
data packet — resets the peer's liveness timer. On an idle link with no
application data or topology changes, the heartbeat is the only traffic
that keeps the link alive.
Peers that are silent for a configurable dead timeout (default 30s) are
considered dead and removed from the peer table. With the default 10s
heartbeat interval, a peer must miss three consecutive heartbeats before
removal. This triggers tree reconvergence and bloom filter recomputation
for the affected subtree.
### Partition Handling
If the network partitions, each segment independently elects its own root
(the smallest node_addr in the segment) and reconverges. When segments
If the network partitions, each segment independently rediscovers its own
root (the smallest node_addr in the segment) and reconverges. When segments
rejoin, nodes discover the globally-smallest root through TreeAnnounce
exchange and reconverge to a single tree.
@@ -94,9 +160,10 @@ Each node maintains a bloom filter per peer, answering: "can peer P possibly
reach destination D?" The answer is either "no" (definitive) or "maybe"
(probabilistic — false positives are possible).
This is **candidate selection**, not routing. Bloom filters identify which
peers are worth considering for a destination, but the actual forwarding
decision uses tree coordinate distance to rank those candidates.
Because filters propagate along tree edges with split-horizon exclusion,
they encode directional reachability: a bloom hit on a tree peer reliably
indicates which subtree contains the destination. When multiple peers match,
tree coordinate distance ranks them.
### How Filters Propagate
@@ -135,9 +202,9 @@ Updates are rate-limited at 500ms to prevent storms during topology changes.
At moderate network sizes, bloom filters are highly accurate. At larger
scales (~1M nodes), hub nodes with many peers may see elevated false positive
rates (715% for nodes with 20+ peers). False positives cause unnecessary
discovery attempts but do not affect routing correctness — the tree distance
calculation makes the actual forwarding decision.
rates (715% for nodes with 20+ peers). False positives may cause a packet
to be forwarded toward the wrong subtree, but the self-distance check at
each hop prevents loops and the packet falls through to greedy tree routing.
See [fips-bloom-filters.md](fips-bloom-filters.md) for filter parameters,
FPR calculations, and size class folding.
@@ -155,8 +222,8 @@ priority chain. This is the core routing algorithm.
2. **Direct peer** — The destination is an authenticated neighbor. Forward
directly. No coordinates or bloom filters needed.
3. **Bloom-guided candidate selection** — One or more peers' bloom filters
contain the destination. Select the best candidate by composite key:
3. **Bloom-guided routing** — One or more peers' bloom filters contain the
destination. Select the best peer by composite key:
`(link_cost, tree_distance, node_addr)`. This requires the destination's
tree coordinates to be in the local coordinate cache.
@@ -189,10 +256,12 @@ through to greedy tree routing (functional but suboptimal).
When bloom filters identify multiple candidate peers, they are ranked by a
composite key:
1. **link_cost** — Per-link quality metric. ETX is computed from bidirectional
delivery ratios in MMP metrics and is used in cost-based parent selection
via `effective_depth = depth + link_cost`, but is not yet wired into
`find_next_hop()` candidate ranking.
1. **link_cost** — Per-link quality metric derived from ETX (Expected
Transmission Count), computed from bidirectional delivery ratios in MMP
metrics. In practice this is an uncommon tie-breaker: most forwarding
decisions are resolved by tree distance alone, and link_cost only
differentiates candidates when multiple peers offer the same tree distance
to the destination.
2. **tree_distance** — Coordinate-based distance to destination through this
peer
3. **node_addr** — Deterministic tie-breaker
@@ -285,9 +354,10 @@ to a visited filter (preventing loops on a single path), and forwards to all
peers not in the visited filter. Bloom filters may help direct the flood
toward likely candidates.
**Deduplication**: Nodes maintain a short-lived request_id dedup cache to
drop convergent duplicates (the same request arriving via different paths).
This is a protocol requirement, not an optimization.
**Deduplication**: Nodes maintain a short-lived request_id dedup cache
(default 10s window) to drop convergent duplicates (the same request
arriving via different paths). This is a protocol requirement, not an
optimization.
### LookupResponse
@@ -528,10 +598,12 @@ When traffic resumes:
3. Coordinates: discovery may be needed if cache has expired
4. SessionSetup re-warms transit caches on the new path
## Leaf-Only Operation *(future direction)*
## Leaf-Only Operation *(under development)*
Leaf-only operation is a planned optimization for resource-constrained nodes
(sensors, battery-powered devices). Not currently implemented.
Leaf-only operation is an optimization for resource-constrained nodes
(sensors, battery-powered devices). The core infrastructure exists (config
flag, node constructor, bloom filter support) but is not yet enabled in
normal operation.
### Concept
@@ -558,7 +630,7 @@ The upstream peer:
Even as a leaf-only node, it still:
- Maintains its own Noise IK link session with the upstream peer
- Maintains its own Noise IK link session with the upstream peer (FMP layer)
- Can establish end-to-end FSP sessions with arbitrary destinations
- Has its own identity (npub, node_addr)
@@ -621,9 +693,12 @@ recovery).
| LookupResponse proof verification | **Implemented** |
| Discovery reverse-path routing | **Implemented** |
| Error signal rate limiting | **Implemented** |
| Leaf-only operation | Future direction |
| Flap dampening (hysteresis + hold-down) | **Implemented** |
| Link liveness (dead timeout) | **Implemented** |
| Discovery request deduplication | **Implemented** |
| Leaf-only operation | Under development |
| Link cost in parent selection (ETX) | **Implemented** |
| Link cost in candidate ranking | Future direction |
| Link cost in candidate ranking | **Implemented** |
| Discovery path accumulation | Future direction |
## References