mirror of
https://github.com/jmcorgan/fips.git
synced 2026-07-30 19:46:15 +00:00
Implement cost-based parent selection with periodic re-evaluation
Cost-based parent selection: - Replace depth-only parent selection with effective_depth = depth + link_cost - link_cost computed from locally measured MMP metrics: etx * (1.0 + srtt_ms / 100.0) - Prevents bottleneck subtrees in heterogeneous networks where a LoRa link at depth 1 would otherwise always beat fiber at depth 2 - Configurable hysteresis (default 0.2) prevents marginal parent switches - Configurable hold-down timer (default 30s) suppresses re-evaluation after parent switch - Mandatory switches (parent lost, root change) bypass both safeguards - Link costs passed as HashMap parameter to keep TreeState pure Periodic re-evaluation: - evaluate_parent() was only called on TreeAnnounce receipt or parent loss; after tree stabilization, link degradation went undetected - Added timer-based re-evaluation (reeval_interval_secs, default 60s) that calls evaluate_parent() from the tick handler with current MMP link costs - Respects existing hold-down and hysteresis safeguards - Short-circuits when disabled or <2 peers Design documentation: - Update 7 design docs to reflect cost-based parent selection - Replace depth-only algorithm descriptions with effective_depth model - Replace rejected cumulative path cost spec with local-only design rationale - Rewrite Example 2 (heterogeneous links) for local-only cost model - Update config docs: parent_switch_threshold replaced by parent_hysteresis, hold_down_secs, reeval_interval_secs Chaos simulation enhancements: - fips_overrides with deep merge for per-scenario FIPS config customization - Explicit topology algorithm for deterministic test graphs - Control socket querying via fipsctl for tree/MMP snapshot collection - Edge existence validation in netem manager - Per-link netem policy overrides - 9 new chaos scenarios covering cost avoidance, depth-vs-cost tradeoffs, stability, mixed topologies, periodic re-evaluation, and bottleneck parent 12 new unit tests, 667 total passing, clippy clean.
This commit is contained in:
@@ -148,7 +148,9 @@ Controls tree construction and parent selection.
|
||||
| Parameter | Type | Default | Description |
|
||||
|----------------------------------------|-------|---------|--------------------------------------------------|
|
||||
| `node.tree.announce_min_interval_ms` | u64 | `500` | Per-peer TreeAnnounce rate limit |
|
||||
| `node.tree.parent_switch_threshold` | usize | `1` | Min depth improvement required to switch parents |
|
||||
| `node.tree.parent_hysteresis` | f64 | `0.2` | Cost improvement fraction required for same-root parent switch (0.0–1.0) |
|
||||
| `node.tree.hold_down_secs` | u64 | `30` | Suppress non-mandatory re-evaluation after parent switch |
|
||||
| `node.tree.reeval_interval_secs` | u64 | `60` | Periodic cost-based parent re-evaluation interval (0 = disabled) |
|
||||
|
||||
### Bloom Filter (`node.bloom.*`)
|
||||
|
||||
@@ -346,7 +348,9 @@ node:
|
||||
recent_expiry_secs: 10
|
||||
tree:
|
||||
announce_min_interval_ms: 500
|
||||
parent_switch_threshold: 1
|
||||
parent_hysteresis: 0.2 # cost improvement fraction for parent switch
|
||||
hold_down_secs: 30 # suppress re-evaluation after switch
|
||||
reeval_interval_secs: 60 # periodic cost-based re-evaluation (0 = disabled)
|
||||
bloom:
|
||||
update_debounce_ms: 500
|
||||
session:
|
||||
|
||||
@@ -460,11 +460,13 @@ low-latency links converge quickly. Each metric carries both short-term and
|
||||
long-term exponentially weighted moving averages, enabling detection of
|
||||
quality changes against a stable baseline.
|
||||
|
||||
MMP's primary role today is operator visibility — periodic log lines report
|
||||
per-link RTT, loss, jitter, and goodput. The metrics also provide the
|
||||
foundation for quality-aware routing: MMP computes an Expected Transmission
|
||||
Count (ETX) from bidirectional delivery ratios, which will feed into
|
||||
candidate ranking once link-cost routing is enabled.
|
||||
MMP serves dual roles: operator visibility and cost-based parent selection.
|
||||
Periodic log lines report per-link RTT, loss, jitter, and goodput. MMP
|
||||
computes an Expected Transmission Count (ETX) from bidirectional delivery
|
||||
ratios, which feeds into cost-based parent selection where each node
|
||||
evaluates `effective_depth = depth + link_cost` using
|
||||
`link_cost = etx * (1.0 + srtt_ms / 100.0)`. ETX is not yet used in
|
||||
`find_next_hop()` candidate ranking for data forwarding.
|
||||
|
||||
See [fips-mesh-layer.md](fips-mesh-layer.md) for MMP operating modes, report
|
||||
scheduling, and the spin bit design.
|
||||
|
||||
@@ -405,8 +405,8 @@ work together to build and maintain the mesh, and
|
||||
## Metrics Measurement Protocol (MMP)
|
||||
|
||||
Each active peer link runs an instance of the Metrics Measurement Protocol,
|
||||
providing per-link quality metrics to the operator and (in future) to the
|
||||
routing layer for link-cost decisions.
|
||||
providing per-link quality metrics to the operator and to the spanning tree
|
||||
layer for cost-based parent selection.
|
||||
|
||||
### Metrics Tracked
|
||||
|
||||
@@ -422,8 +422,9 @@ fields in the FMP wire format:
|
||||
- **OWD trend** — One-way delay trend (µs/s, signed). Indicates congestion
|
||||
buildup before loss occurs.
|
||||
- **ETX** — Expected Transmission Count, computed from bidirectional delivery
|
||||
ratios. Not yet wired into routing decisions (link cost is currently
|
||||
constant).
|
||||
ratios. Used in cost-based parent selection via
|
||||
`link_cost = etx * (1.0 + srtt_ms / 100.0)`; not yet used in
|
||||
`find_next_hop()` candidate ranking.
|
||||
- **Dual EWMA trends** — Short-term (α=1/4) and long-term (α=1/32) trend
|
||||
indicators for both RTT and loss, enabling change detection.
|
||||
|
||||
|
||||
@@ -52,8 +52,8 @@ Nodes self-organize into a spanning tree through distributed parent selection:
|
||||
No election protocol — this is a consequence of each node independently
|
||||
preferring lower-addressed roots.
|
||||
2. **Parent selection**: Each node selects a single parent from among its
|
||||
direct peers based on which offers the best path to root (considering
|
||||
depth improvement threshold).
|
||||
direct peers based on which offers the lowest effective depth (tree depth
|
||||
weighted by local link cost).
|
||||
3. **Coordinate computation**: Once a node has a parent, its coordinate is
|
||||
computed from its ancestry path.
|
||||
|
||||
@@ -190,8 +190,9 @@ When bloom filters identify multiple candidate peers, they are ranked by a
|
||||
composite key:
|
||||
|
||||
1. **link_cost** — Per-link quality metric. ETX is computed from bidirectional
|
||||
delivery ratios in MMP metrics but is not yet wired into `find_next_hop()`
|
||||
candidate ranking; link cost remains constant in the current implementation.
|
||||
delivery ratios in MMP metrics and is used in cost-based parent selection
|
||||
via `effective_depth = depth + link_cost`, but is not yet wired into
|
||||
`find_next_hop()` candidate ranking.
|
||||
2. **tree_distance** — Coordinate-based distance to destination through this
|
||||
peer
|
||||
3. **node_addr** — Deterministic tie-breaker
|
||||
@@ -621,7 +622,8 @@ recovery).
|
||||
| Discovery reverse-path routing | **Implemented** |
|
||||
| Error signal rate limiting | **Implemented** |
|
||||
| Leaf-only operation | Future direction |
|
||||
| Link cost metrics (ETX) | Future direction |
|
||||
| Link cost in parent selection (ETX) | **Implemented** |
|
||||
| Link cost in candidate ranking | Future direction |
|
||||
| Discovery path accumulation | Future direction |
|
||||
|
||||
## References
|
||||
|
||||
@@ -35,28 +35,46 @@ reconverge to a single tree.
|
||||
## Parent Selection
|
||||
|
||||
Each node selects a single parent from among its direct peers. Parent
|
||||
selection follows these rules:
|
||||
selection uses cost-weighted depth to balance tree depth against link
|
||||
quality.
|
||||
|
||||
### Selection Criteria
|
||||
|
||||
1. **Find the smallest root** visible across all peers' TreeAnnounce messages
|
||||
2. Among peers that can reach that root, prefer the one offering the
|
||||
**shallowest depth** (shortest path to root)
|
||||
3. Apply the **depth improvement threshold**: switching parents requires the
|
||||
proposed parent to offer a path at least 1 hop shallower than the current
|
||||
parent (when the root is the same)
|
||||
2. Compute **effective depth** for each candidate peer:
|
||||
`effective_depth = peer.depth + link_cost`, where
|
||||
`link_cost = etx * (1.0 + srtt_ms / 100.0)` using locally measured MMP
|
||||
metrics. When MMP metrics have not yet converged, `link_cost` defaults to
|
||||
1.0, preserving pure depth-based behavior as a graceful fallback.
|
||||
3. Apply **hysteresis**: switch parents only when the best candidate's
|
||||
effective depth is significantly better than the current parent's:
|
||||
`best_eff_depth < current_eff_depth * (1.0 - parent_hysteresis)`
|
||||
(default `parent_hysteresis = 0.2`, requiring 20% improvement)
|
||||
|
||||
### Immediate Switch Triggers
|
||||
### Mandatory Switch Triggers
|
||||
|
||||
Three conditions bypass the depth threshold and trigger immediate parent
|
||||
reselection:
|
||||
Two conditions bypass both hysteresis and the hold-down timer, triggering
|
||||
immediate parent reselection:
|
||||
|
||||
1. **Parent loss**: Current parent is no longer in the peer set (link
|
||||
broken, peer disconnected)
|
||||
2. **Better root**: A peer advertises a smaller root than the current
|
||||
tree's root — always switch regardless of depth
|
||||
3. **Depth improvement**: Same root, but the proposed parent offers depth
|
||||
at least `PARENT_SWITCH_THRESHOLD` (1 hop) better than the current parent
|
||||
tree's root — always switch regardless of effective depth
|
||||
|
||||
### Stability Mechanisms
|
||||
|
||||
- **Hold-down timer** (`hold_down_secs`, default 30s): After any parent
|
||||
switch, non-mandatory re-evaluation is suppressed to allow MMP metrics
|
||||
to stabilize on the new link. Mandatory switches (parent loss, root
|
||||
change) bypass the hold-down.
|
||||
- **Periodic re-evaluation** (`reeval_interval_secs`, default 60s):
|
||||
Re-evaluates parent selection using current MMP link costs, independent
|
||||
of TreeAnnounce traffic. This catches link degradation after the tree
|
||||
has stabilized and TreeAnnounce gossip has stopped.
|
||||
- **Local-only metrics**: Link costs use only locally measured MMP data
|
||||
(ETX and SRTT). No cumulative path costs are propagated and no wire
|
||||
format changes are required. This avoids the trust problems inherent
|
||||
in self-reported cost metrics in a permissionless network.
|
||||
|
||||
### After Parent Change
|
||||
|
||||
@@ -228,7 +246,9 @@ Example: In a 1000-node network with depth 10 and 5 peers, a node stores
|
||||
|
||||
| Parameter | Default | Description |
|
||||
| --------- | ------- | ----------- |
|
||||
| PARENT_SWITCH_THRESHOLD | 1 hop | Minimum depth improvement for same-root switch |
|
||||
| PARENT_HYSTERESIS | 0.2 (20%) | Fractional improvement in effective depth required for same-root switch |
|
||||
| HOLD_DOWN_SECS | 30s | Suppress non-mandatory re-evaluation after parent switch |
|
||||
| REEVAL_INTERVAL_SECS | 60s | Periodic cost-based parent re-evaluation interval |
|
||||
| ANNOUNCE_MIN_INTERVAL | 500ms | Minimum between announcements to same peer |
|
||||
| ROOT_TIMEOUT | 60 min | Root declaration considered stale (not yet enforced; heartbeat cascading covers common case) |
|
||||
| TREE_ENTRY_TTL | 5–10 min | Individual entry expiration |
|
||||
@@ -238,7 +258,9 @@ Example: In a 1000-node network with depth 10 and 5 peers, a node stores
|
||||
| Feature | Status |
|
||||
| ------- | ------ |
|
||||
| Root election (smallest node_addr) | **Implemented** |
|
||||
| Parent selection with depth threshold | **Implemented** |
|
||||
| Cost-based parent selection with hysteresis | **Implemented** |
|
||||
| Hold-down timer after parent change | **Implemented** |
|
||||
| Periodic cost-based parent re-evaluation | **Implemented** |
|
||||
| Coordinate computation | **Implemented** |
|
||||
| TreeAnnounce gossip | **Implemented** |
|
||||
| Signature verification (outer) | **Implemented** |
|
||||
@@ -247,7 +269,6 @@ Example: In a 1000-node network with depth 10 and 5 peers, a node stores
|
||||
| Coord cache flush on parent change | **Implemented** |
|
||||
| Root timeout enforcement | Planned |
|
||||
| Tree entry TTL enforcement | Planned |
|
||||
| Hold-down timer after parent change | Planned |
|
||||
| Per-ancestry-entry signatures | Future direction |
|
||||
|
||||
## References
|
||||
|
||||
@@ -410,9 +410,12 @@ MTU, and its own liveness tracking.
|
||||
### Transport Quality and Path Selection
|
||||
|
||||
Transport characteristics (latency, bandwidth, reliability) affect path
|
||||
quality but are not currently factored into routing decisions. The spanning
|
||||
tree parent selection uses a depth improvement threshold but does not
|
||||
consider transport quality. This is a potential area for future optimization.
|
||||
quality. The spanning tree parent selection factors in link quality through
|
||||
cost-based effective depth (`effective_depth = depth + link_cost`), where
|
||||
`link_cost` is derived from locally measured MMP metrics (ETX and SRTT).
|
||||
This allows the tree to prefer lower-latency, lower-loss links when the
|
||||
quality difference is significant. Link cost is not yet used in
|
||||
`find_next_hop()` candidate ranking for data forwarding.
|
||||
|
||||
## References
|
||||
|
||||
|
||||
@@ -24,7 +24,7 @@ The protocol is based on Yggdrasil v0.5's CRDT gossip design.
|
||||
8. [Parent Selection](#8-parent-selection)
|
||||
9. [Steady State Behavior](#9-steady-state-behavior)
|
||||
10. [Worked Examples](#10-worked-examples)
|
||||
11. [Known Limitations (v1 Implementation)](#known-limitations-v1-implementation)
|
||||
11. [Known Limitations](#known-limitations)
|
||||
|
||||
---
|
||||
|
||||
@@ -380,8 +380,10 @@ Link B ←→ C fails:
|
||||
### Reconvergence Dynamics
|
||||
|
||||
**Stability threshold**: To prevent flapping, a node only changes parent when
|
||||
the improvement exceeds a threshold. In v1, this is a depth difference of at
|
||||
least `PARENT_SWITCH_THRESHOLD` (1 hop). See §8 for details.
|
||||
the improvement exceeds cost-based hysteresis (`parent_hysteresis`, default
|
||||
0.2 = 20% improvement required). A hold-down timer (`hold_down_secs`,
|
||||
default 30s) further suppresses non-mandatory re-evaluation after a switch.
|
||||
See §8 for details.
|
||||
|
||||
**Sequence number advancement**: Each parent change increments the sequence
|
||||
number. Nodes observing rapid sequence increases can detect instability and
|
||||
@@ -421,7 +423,7 @@ Nodes detect they're partitioned when:
|
||||
2. **Root unreachable**: No peer has path to current root
|
||||
|
||||
**Detection via gossip staleness** (not currently implemented — see
|
||||
[Known Limitations](#known-limitations-v1-implementation)):
|
||||
[Known Limitations](#known-limitations)):
|
||||
|
||||
In principle, nodes would also detect partitions through root entry staleness:
|
||||
if no fresh root announcements arrive within a timeout, the root is presumed
|
||||
@@ -579,100 +581,114 @@ see traffic to consider the link alive.
|
||||
|
||||
Parent selection determines tree structure and routing efficiency.
|
||||
|
||||
### v1 Implementation: Depth-Only Selection
|
||||
### Cost-Based Selection with Effective Depth
|
||||
|
||||
The current implementation uses tree depth as the sole selection metric, with a
|
||||
threshold to prevent thrashing between equivalent-depth paths.
|
||||
The implementation uses cost-weighted depth to balance tree depth against link
|
||||
quality. Each candidate parent is evaluated by its **effective depth** — the
|
||||
tree depth plus a local link cost penalty derived from MMP metrics.
|
||||
|
||||
**Algorithm** (`TreeState::evaluate_parent()` in `tree.rs`):
|
||||
**Algorithm** (`TreeState::evaluate_parent()` in `tree/state.rs`):
|
||||
|
||||
```
|
||||
evaluate_parent():
|
||||
evaluate_parent(peer_costs: HashMap<NodeAddr, f64>):
|
||||
// 1. Find smallest root reachable through any peer
|
||||
smallest_root = min(peer.root for peer in peers_with_coords)
|
||||
|
||||
if self == smallest_root and is_root:
|
||||
return None // Already root, no change
|
||||
|
||||
// 2. Among peers reaching smallest_root, find shallowest
|
||||
best_peer = min(
|
||||
[p for p in peers if p.root == smallest_root],
|
||||
key=lambda p: p.depth
|
||||
)
|
||||
proposed_depth = best_peer.depth + 1
|
||||
// 2. Compute effective depth for each candidate
|
||||
for each peer with peer.root == smallest_root:
|
||||
link_cost = peer_costs.get(peer) or 1.0 // default optimistic
|
||||
peer.effective_depth = peer.depth + link_cost
|
||||
|
||||
best_peer = min(candidates, key=effective_depth, tiebreak=node_addr)
|
||||
|
||||
if best_peer == current_parent:
|
||||
return None // Already using best
|
||||
|
||||
// 3. Always switch if parent is gone or root is changing
|
||||
// 3. Mandatory switches — bypass hysteresis and hold-down
|
||||
if current_parent not in peers:
|
||||
return best_peer // Path broken
|
||||
return best_peer // Parent lost
|
||||
if current_root != smallest_root:
|
||||
return best_peer // Better root found
|
||||
|
||||
// 4. For same root: require depth improvement ≥ threshold
|
||||
current_depth = my_coords.depth()
|
||||
if current_depth >= proposed_depth + PARENT_SWITCH_THRESHOLD:
|
||||
// 4. Hold-down check — suppress non-mandatory switches
|
||||
if last_parent_switch + hold_down > now:
|
||||
return None // Too soon after last switch
|
||||
|
||||
// 5. Hysteresis — require significant improvement
|
||||
current_parent_eff = current_parent.depth + peer_costs.get(current_parent)
|
||||
if best_eff_depth < current_parent_eff * (1.0 - parent_hysteresis):
|
||||
return best_peer
|
||||
|
||||
return None // Not enough improvement
|
||||
```
|
||||
|
||||
**Constants**:
|
||||
**Parameters**:
|
||||
|
||||
```
|
||||
PARENT_SWITCH_THRESHOLD = 1 // Minimum depth improvement to switch parents
|
||||
parent_hysteresis = 0.2 // 20% improvement required for same-root switch
|
||||
hold_down_secs = 30 // Suppress re-evaluation after parent switch
|
||||
reeval_interval_secs = 60 // Periodic re-evaluation independent of TreeAnnounce
|
||||
```
|
||||
|
||||
This means a proposed parent must offer a path at least 1 hop shallower than
|
||||
the current parent (under the same root) to trigger a switch. Root changes
|
||||
always trigger a switch regardless of depth.
|
||||
|
||||
**What this means for tree structure**: The v1 algorithm produces minimum-depth
|
||||
trees, which minimizes coordinate path length and hop count for coordinate-based
|
||||
greedy routing. However, it does not account for link quality—a high-latency or
|
||||
lossy link at depth 1 is preferred over a fast link at depth 2.
|
||||
|
||||
### v2 Planned: Cost Metrics
|
||||
|
||||
The following cost-based parent selection is planned but not yet implemented.
|
||||
|
||||
**Cost components**:
|
||||
|
||||
- **Latency** (primary): `cost_latency = round_trip_time_ms`
|
||||
- **Packet loss** (reliability): `cost_loss = 1 / (1 - loss_rate)` —
|
||||
transforms loss rate into multiplicative cost (10% loss → 1.11, 50% → 2.0)
|
||||
- **Bandwidth** (capacity): `cost_bandwidth = reference_bandwidth / actual_bandwidth`
|
||||
|
||||
**Combined cost**: Weighted combination with application-tunable weights:
|
||||
**Link cost formula** (`ActivePeer::link_cost()` in `peer/active.rs`):
|
||||
|
||||
```
|
||||
effective_cost = w_latency * cost_latency
|
||||
+ w_loss * cost_loss
|
||||
+ w_bandwidth * cost_bandwidth
|
||||
link_cost = etx * (1.0 + srtt_ms / 100.0)
|
||||
```
|
||||
|
||||
**Path cost to root**: Recursive — each node advertises its cumulative cost,
|
||||
allowing neighbors to compute total path cost:
|
||||
Where ETX (Expected Transmission Count) comes from bidirectional MMP delivery
|
||||
ratios and SRTT (Smoothed Round-Trip Time) from MMP timestamp-echo. When MMP
|
||||
metrics have not yet converged, `link_cost` defaults to 1.0, preserving
|
||||
depth-only behavior as a graceful fallback.
|
||||
|
||||
```
|
||||
path_cost(peer) = link_cost(self, peer) + peer.path_cost_to_root
|
||||
```
|
||||
**What this means for tree structure**: The algorithm can prefer a deeper parent
|
||||
with a better link over a shallower parent with a poor link, when the effective
|
||||
depth difference is significant enough to overcome hysteresis. For example, a
|
||||
fiber link at depth 2 (effective depth ≈ 3.01) beats a LoRa link at depth 1
|
||||
(effective depth ≈ 7.32 with 500ms RTT and 5% loss). In homogeneous networks
|
||||
where all links have similar quality, effective depth tracks tree depth closely
|
||||
and the algorithm produces minimum-depth trees as before.
|
||||
|
||||
**Stability threshold**: Hysteresis with both absolute and relative components:
|
||||
**Periodic re-evaluation**: `evaluate_parent()` is event-driven — called on
|
||||
TreeAnnounce receipt or parent loss. After the tree stabilizes and TreeAnnounce
|
||||
traffic stops, link degradation goes undetected. The periodic re-evaluation
|
||||
timer (`reeval_interval_secs`) calls `evaluate_parent()` from the tick handler
|
||||
with current MMP link costs, independent of TreeAnnounce traffic.
|
||||
|
||||
```
|
||||
stability_threshold = base_threshold + current_cost * relative_threshold
|
||||
```
|
||||
### Design Rationale: Local-Only Cost Metrics
|
||||
|
||||
**Cost measurement**: Active probing (periodic RTT measurement), passive
|
||||
observation (inferred from protocol message timing), and exponential smoothing
|
||||
(`alpha = 0.1–0.3`) to balance responsiveness with stability.
|
||||
The original design considered cumulative path costs (OSPF-style, where each
|
||||
hop adds its link cost and the total is advertised in TreeAnnounce). This
|
||||
approach was rejected for three independent reasons:
|
||||
|
||||
**Implementation prerequisites**: The cost-based algorithm requires changes to
|
||||
the TreeAnnounce wire format to carry path cost values, and a measurement
|
||||
subsystem for link quality metrics. See Section 10, Example 2 for how
|
||||
cost-based selection would affect tree structure in heterogeneous networks.
|
||||
1. **Unverifiable self-reporting**: In a permissionless network, a node can
|
||||
claim any path cost. There is no mechanism for neighbors to verify that
|
||||
the reported cumulative cost is truthful. A malicious node advertising
|
||||
zero cost would attract traffic as a transit node.
|
||||
|
||||
2. **No shared metric semantics**: Different links measure different things.
|
||||
A LoRa link's 500ms RTT and a fiber link's 1ms RTT are both "round-trip
|
||||
time" but represent fundamentally different physical constraints.
|
||||
Accumulating them into a single path cost obscures per-hop information
|
||||
that is more useful when evaluated locally.
|
||||
|
||||
3. **Accumulation amplifies error**: Small measurement noise at each hop
|
||||
compounds across the path. A 5-hop path accumulates 5x the measurement
|
||||
error of a single hop, while providing no more actionable information
|
||||
than the local link cost to each candidate parent.
|
||||
|
||||
The local-only approach uses `link_cost = etx * (1.0 + srtt_ms / 100.0)`,
|
||||
where both components are locally measured via MMP. The RTT weighting
|
||||
addresses a blind spot in ETX alone: a clean-but-slow link (LoRa with 0%
|
||||
loss) gets ETX = 1.0, identical to fiber. The SRTT factor distinguishes them
|
||||
— a 500ms LoRa link gets cost ≈ 6.0 versus fiber at ≈ 1.01.
|
||||
|
||||
No wire format changes are required. TreeAnnounce messages continue to carry
|
||||
depth (not cost), and each node independently evaluates its direct links
|
||||
using trusted local measurements.
|
||||
|
||||
---
|
||||
|
||||
@@ -874,48 +890,48 @@ Physical topology:
|
||||
node_addr ordering: B < A < D < E < C
|
||||
```
|
||||
|
||||
**Cost calculation** (using bandwidth as primary):
|
||||
**Local link costs** (using `link_cost = etx * (1.0 + srtt_ms / 100.0)`):
|
||||
|
||||
```
|
||||
Link costs (normalized to 1 Gbps = 1):
|
||||
A ═ B: cost = 1
|
||||
B ═ D: cost = 1
|
||||
D ═ E: cost = 1
|
||||
C — D: cost = 1000 (1 Mbps)
|
||||
A ~ C: cost = 100000 (9600 bps)
|
||||
Assumed MMP measurements after convergence:
|
||||
A ═ B: fiber, 1ms RTT, 0% loss → link_cost = 1.0 * (1 + 1/100) ≈ 1.01
|
||||
B ═ D: fiber, 1ms RTT, 0% loss → link_cost = 1.0 * (1 + 1/100) ≈ 1.01
|
||||
D ═ E: fiber, 1ms RTT, 0% loss → link_cost = 1.0 * (1 + 1/100) ≈ 1.01
|
||||
C — D: DSL, 20ms RTT, 2% loss → link_cost = 1.04 * (1 + 20/100) ≈ 1.25
|
||||
A ~ C: radio, 500ms RTT, 5% loss→ link_cost = 1.11 * (1 + 500/100) ≈ 6.66
|
||||
```
|
||||
|
||||
**Tree formation with costs**:
|
||||
**Tree formation with effective depth**:
|
||||
|
||||
```
|
||||
Root = B (smallest node_addr)
|
||||
Root = B (smallest node_addr, depth 0)
|
||||
|
||||
Parent selection:
|
||||
├── A: peers are B (cost 1), C (cost 100000)
|
||||
│ └── Selects B (much lower cost)
|
||||
Parent selection (each node evaluates effective_depth = peer.depth + link_cost):
|
||||
├── A: peers are B (depth 0, cost 1.01 → eff 1.01), C (depth ?, cost 6.66)
|
||||
│ └── Selects B (lowest effective depth)
|
||||
│
|
||||
├── D: peers are B (cost 1), C (cost 1000), E (cost 1)
|
||||
│ └── Selects B (direct, cost 1)
|
||||
├── D: peers are B (depth 0, cost 1.01 → eff 1.01),
|
||||
│ C (depth ?, cost 1.25), E (depth ?, cost 1.01)
|
||||
│ └── Selects B (direct, lowest effective depth)
|
||||
│
|
||||
├── E: peer is D
|
||||
│ └── Path to B: E → D → B, cost = 1 + 1 = 2
|
||||
│ └── Selects D
|
||||
├── E: peer is D (depth 1, cost 1.01 → eff 2.01)
|
||||
│ └── Selects D (only candidate)
|
||||
│
|
||||
└── C: peers are A (cost 100000), D (cost 1000)
|
||||
└── Path through A: 100000 + 1 = 100001
|
||||
└── Path through D: 1000 + 1 = 1001
|
||||
└── Selects D (much lower cost despite higher local cost)
|
||||
└── C: peers are A (depth 1, cost 6.66 → eff 7.66),
|
||||
D (depth 1, cost 1.25 → eff 2.25)
|
||||
└── Selects D (eff 2.25 vs 7.66 — much lower)
|
||||
|
||||
Resulting tree:
|
||||
B (root)
|
||||
B (root, depth 0)
|
||||
/ \
|
||||
A D
|
||||
A D (both depth 1)
|
||||
|\
|
||||
E C
|
||||
E C (both depth 2)
|
||||
```
|
||||
|
||||
**Note**: C chooses D despite A being "closer" in hops, because total path
|
||||
cost through D is lower.
|
||||
**Note**: C chooses D despite both being at depth 1 — the DSL link to D
|
||||
(eff 2.25) far beats the radio link to A (eff 7.66). With local-only costs,
|
||||
each node evaluates only its direct link quality, not cumulative path cost.
|
||||
|
||||
**Radio link failure**:
|
||||
|
||||
@@ -925,11 +941,10 @@ If A ~ C radio fails:
|
||||
└── C loses a potential backup path, but current tree unchanged
|
||||
|
||||
If D — C DSL fails:
|
||||
├── C loses parent
|
||||
├── C loses parent (mandatory switch — bypasses hysteresis and hold-down)
|
||||
├── C's only remaining peer is A (radio)
|
||||
├── C selects A as parent
|
||||
├── C's path to root: C → A → B (cost 100001)
|
||||
└── Tree reconverges with C as child of A
|
||||
├── C selects A as parent (eff depth = 1 + 6.66 = 7.66)
|
||||
└── Tree reconverges with C as child of A at depth 2
|
||||
```
|
||||
|
||||
### Example 3: Network Partition and Healing
|
||||
@@ -1019,7 +1034,7 @@ initial exchange).
|
||||
|
||||
---
|
||||
|
||||
## Known Limitations (v1 Implementation)
|
||||
## Known Limitations
|
||||
|
||||
The following limitations exist in the current implementation relative to the
|
||||
design described in this document. They are documented here to guide future
|
||||
@@ -1079,26 +1094,27 @@ expires and has no peer with a fresher root declaration is partitioned. It
|
||||
becomes its own root and announces, allowing the partition to converge
|
||||
independently.
|
||||
|
||||
### Known Limitation: Limited Stability Mechanisms
|
||||
### Known Limitation: Remaining Stability Gaps
|
||||
|
||||
The implementation includes basic hysteresis (`PARENT_SWITCH_THRESHOLD = 1`
|
||||
depth difference required to switch parents), but the temporal stability
|
||||
mechanisms described in the design are not implemented:
|
||||
The primary stability mechanisms are implemented:
|
||||
|
||||
- **Cost-based hysteresis** (`parent_hysteresis = 0.2`): requires 20%
|
||||
effective depth improvement to switch parents under the same root
|
||||
- **Hold-down timer** (`hold_down_secs = 30`): suppresses non-mandatory
|
||||
re-evaluation after a parent switch, allowing MMP metrics to stabilize
|
||||
- **Periodic re-evaluation** (`reeval_interval_secs = 60`): catches link
|
||||
degradation after tree stabilization independent of TreeAnnounce traffic
|
||||
|
||||
Remaining gaps:
|
||||
|
||||
- No hold timer on parent changes (minimum time before next switch)
|
||||
- No sequence number advancement rate limiting
|
||||
- No announcement suppression during transient topology changes
|
||||
- No minimum stable state duration before re-announcing
|
||||
|
||||
**Impact**: Rapid topology changes (e.g., a flapping link) could cause
|
||||
excessive announcement traffic and repeated coordinate recomputation.
|
||||
Currently mitigated by per-peer rate limiting on TreeAnnounce sends, but
|
||||
the source node is not throttled.
|
||||
|
||||
**Proposed fix**: Add a hold-down timer (e.g., 5-10s) after each parent
|
||||
change during which further parent switches are suppressed unless the current
|
||||
parent is lost entirely. Track announcement rate and suppress if exceeding a
|
||||
threshold.
|
||||
**Impact**: A rapidly flapping link could still cause moderate announcement
|
||||
traffic. The hold-down timer limits the rate of parent switches (at most
|
||||
one non-mandatory switch per 30s), and per-peer rate limiting (500ms)
|
||||
bounds announcement frequency, but the source node's announcement rate is
|
||||
not independently throttled.
|
||||
|
||||
### Known Limitation: Integration Test Gaps
|
||||
|
||||
@@ -1123,7 +1139,7 @@ The gossip-based spanning tree protocol achieves distributed coordination
|
||||
through:
|
||||
|
||||
1. **Deterministic root election** - Smallest node_addr, no negotiation needed
|
||||
2. **Local parent selection** - Each node independently chooses best path to root
|
||||
2. **Cost-aware parent selection** - Each node independently chooses lowest effective depth to root using local link metrics
|
||||
3. **CRDT merge semantics** - Conflicts resolved by sequence number, then timestamp
|
||||
4. **Bounded state** - O(peers × depth) entries per node, not O(network size)
|
||||
5. **Depth-proportional convergence** - Scales with tree height, not node count
|
||||
|
||||
Reference in New Issue
Block a user