mirror of
https://github.com/jmcorgan/fips.git
synced 2026-08-09 08:14:42 +00:00
Johnathan Corgan
0d93a19e07
Cost-based parent selection: - Replace depth-only parent selection with effective_depth = depth + link_cost - link_cost computed from locally measured MMP metrics: etx * (1.0 + srtt_ms / 100.0) - Prevents bottleneck subtrees in heterogeneous networks where a LoRa link at depth 1 would otherwise always beat fiber at depth 2 - Configurable hysteresis (default 0.2) prevents marginal parent switches - Configurable hold-down timer (default 30s) suppresses re-evaluation after parent switch - Mandatory switches (parent lost, root change) bypass both safeguards - Link costs passed as HashMap parameter to keep TreeState pure Periodic re-evaluation: - evaluate_parent() was only called on TreeAnnounce receipt or parent loss; after tree stabilization, link degradation went undetected - Added timer-based re-evaluation (reeval_interval_secs, default 60s) that calls evaluate_parent() from the tick handler with current MMP link costs - Respects existing hold-down and hysteresis safeguards - Short-circuits when disabled or <2 peers Design documentation: - Update 7 design docs to reflect cost-based parent selection - Replace depth-only algorithm descriptions with effective_depth model - Replace rejected cumulative path cost spec with local-only design rationale - Rewrite Example 2 (heterogeneous links) for local-only cost model - Update config docs: parent_switch_threshold replaced by parent_hysteresis, hold_down_secs, reeval_interval_secs Chaos simulation enhancements: - fips_overrides with deep merge for per-scenario FIPS config customization - Explicit topology algorithm for deterministic test graphs - Control socket querying via fipsctl for tree/MMP snapshot collection - Edge existence validation in netem manager - Per-link netem policy overrides - 9 new chaos scenarios covering cost avoidance, depth-vs-cost tradeoffs, stability, mixed topologies, periodic re-evaluation, and bottleneck parent 12 new unit tests, 667 total passing, clippy clean.
Stochastic Network Simulation
Automated stochastic network testing for FIPS. Generates random topologies, spins up Docker containers, and applies configurable stressors (network impairment, link flaps, traffic generation, node churn) over a timed simulation run. Logs are collected and analyzed automatically.
Prerequisites
- Docker with the compose plugin
- Rust toolchain (for building the FIPS binary)
- Python 3 with
pyyamlandjinja2packages
Quick Start
./testing/chaos/scripts/build.sh
./testing/chaos/scripts/chaos.sh smoke-10
Available Scenarios
| Scenario | Nodes | Topology | Duration | Netem | Link Flaps | Traffic | Node Churn | Bandwidth |
|---|---|---|---|---|---|---|---|---|
| smoke-10 | 10 | random_geometric | 60s | -- | -- | -- | -- | -- |
| chaos-10 | 10 | random_geometric | 120s | yes | yes | yes | -- | -- |
| churn-10 | 10 | random_geometric | 600s | yes | yes | yes | yes | -- |
| churn-20 | 20 | erdos_renyi | 600s | yes | yes | yes | yes | yes |
CLI Options
| Option | Description |
|---|---|
-v, --verbose |
Enable debug logging |
--seed N |
Override the scenario's random seed |
--duration secs |
Override the scenario's duration |
--list |
List available scenarios |
The scenario argument accepts either a name (churn-10) or a file
path (scenarios/churn-10.yaml).
Scenario YAML Format
Annotated example based on churn-10.yaml:
scenario:
name: "churn-10"
seed: 42 # deterministic RNG seed
duration_secs: 600 # total simulation time
topology:
num_nodes: 10
algorithm: random_geometric # or erdos_renyi, chain
params:
radius: 0.5 # algorithm-specific parameter
ensure_connected: true # retry until graph is connected
subnet: "172.20.0.0/24"
ip_start: 10 # first node gets .10
netem:
enabled: true
default_policy:
delay_ms: { min: 5, max: 50 }
jitter_ms: { min: 1, max: 10 }
loss_pct: { min: 0, max: 2 }
mutation:
interval_secs: { min: 20, max: 45 } # re-roll interval
fraction: 0.3 # fraction of links mutated
policies: # named policy profiles
normal:
delay_ms: [5, 20]
loss_pct: [0, 1]
degraded:
delay_ms: [50, 100]
jitter_ms: [10, 30]
loss_pct: [3, 8]
link_flaps:
enabled: true
interval_secs: { min: 30, max: 60 }
max_down_links: 2
down_duration_secs: { min: 10, max: 30 }
protect_connectivity: true # never partition the graph
traffic:
enabled: true
max_concurrent: 3
interval_secs: { min: 10, max: 30 }
duration_secs: { min: 5, max: 15 }
parallel_streams: 4
node_churn:
enabled: true
interval_secs: { min: 60, max: 180 }
max_down_nodes: 1
down_duration_secs: { min: 30, max: 90 }
protect_connectivity: true # never kill the last path
bandwidth:
enabled: false # per-link HTB rate limiting
tiers_mbps: [1, 10, 100, 1000] # each link randomly assigned a tier
logging:
rust_log: "debug"
output_dir: "./sim-results"
Topology Algorithms
| Algorithm | Parameters | Description |
|---|---|---|
| random_geometric | radius (default 0.5) | Place nodes in unit square, connect pairs within radius |
| erdos_renyi | p (default 0.3) | Include each edge independently with probability p |
| chain | -- | Linear chain: n01--n02--...--nN |
When ensure_connected is true (default), the generator retries up to
50 times to produce a connected graph.
Directed Outbound Configs
The config generator assigns each edge to exactly one node for outbound connection using a BFS spanning tree rooted at the lowest node ID. Tree edges are assigned parent-to-child; non-tree edges are assigned from the lower node ID to the higher. This eliminates the dual-connect race condition where both sides initiate simultaneously, and creates a clear "owning side" for each link — relevant for auto-reconnect testing.
Output
Results written to sim-results/ (configurable via
logging.output_dir):
analysis.txt-- Summary: panics, errors, sessions, metricsmetadata.txt-- Seed, node count, edges, adjacency listrunner.log-- Orchestration events (topology, netem, churn, traffic) with timestampsfips-node-nXX.log-- Per-node log output
Exit code 0 on success, 2 if panics detected.
Creating Custom Scenarios
- Copy an existing scenario from
scenarios/. - Adjust topology size, algorithm, and stressor parameters.
- Run with
./testing/chaos/scripts/chaos.sh path/to/custom.yaml.