mirror of https://github.com/jmcorgan/fips.git synced 2026-07-30 19:46:15 +00:00

Files

T

Johnathan Corgan cfb087a95d Update design docs for heartbeat, auto-reconnect, handshake retry, and sim improvements

fips-link-layer.md:
- Rewrite Liveness Detection: explicit Heartbeat (0x51) with 10s interval
  and 30s dead timeout replaces vague gossip-as-heartbeat description
- Add Auto-Reconnect section: MMP dead timeout triggers retry with
  unlimited backoff for auto_reconnect peers
- Add Handshake Message Retry section: link + session layer resend with
  exponential backoff within timeout window
- Add Heartbeat to Link Message Types table
- Update Implementation Status with three new implemented features

fips-configuration.md:
- Add handshake_resend_interval_ms, handshake_resend_backoff,
  handshake_max_resends to rate_limit table
- Add heartbeat_interval_secs, link_dead_timeout_secs to general table
- Add peers[].auto_reconnect to peers table
- Note auto-reconnect bypasses max_retries in retry section
- Update complete reference YAML with all new parameters

fips-wire-formats.md:
- Rename 0x51 from reserved Keepalive to implemented Heartbeat
- Update Disconnect reason 0x07 to Heartbeat liveness timeout

testing/chaos/README.md:
- Add runner.log to output files
- Add Directed Outbound Configs subsection

2026-02-21 13:10:02 +00:00

5.1 KiB

Raw Blame History

Stochastic Network Simulation

Automated stochastic network testing for FIPS. Generates random topologies, spins up Docker containers, and applies configurable stressors (network impairment, link flaps, traffic generation, node churn) over a timed simulation run. Logs are collected and analyzed automatically.

Prerequisites

Docker with the compose plugin
Rust toolchain (for building the FIPS binary)
Python 3 with pyyaml and jinja2 packages

Quick Start

./testing/chaos/scripts/build.sh
./testing/chaos/scripts/chaos.sh smoke-10

Available Scenarios

Scenario	Nodes	Topology	Duration	Netem	Link Flaps	Traffic	Node Churn	Bandwidth
smoke-10	10	random_geometric	60s	--	--	--	--	--
chaos-10	10	random_geometric	120s	yes	yes	yes	--	--
churn-10	10	random_geometric	600s	yes	yes	yes	yes	--
churn-20	20	erdos_renyi	600s	yes	yes	yes	yes	yes

CLI Options

Option	Description
`-v`, `--verbose`	Enable debug logging
`--seed N`	Override the scenario's random seed
`--duration secs`	Override the scenario's duration
`--list`	List available scenarios

The scenario argument accepts either a name (churn-10) or a file path (scenarios/churn-10.yaml).

Scenario YAML Format

Annotated example based on churn-10.yaml:

scenario:
  name: "churn-10"
  seed: 42                          # deterministic RNG seed
  duration_secs: 600                # total simulation time

topology:
  num_nodes: 10
  algorithm: random_geometric       # or erdos_renyi, chain
  params:
    radius: 0.5                     # algorithm-specific parameter
  ensure_connected: true            # retry until graph is connected
  subnet: "172.20.0.0/24"
  ip_start: 10                      # first node gets .10

netem:
  enabled: true
  default_policy:
    delay_ms: { min: 5, max: 50 }
    jitter_ms: { min: 1, max: 10 }
    loss_pct: { min: 0, max: 2 }
  mutation:
    interval_secs: { min: 20, max: 45 }  # re-roll interval
    fraction: 0.3                         # fraction of links mutated
    policies:                             # named policy profiles
      normal:
        delay_ms: [5, 20]
        loss_pct: [0, 1]
      degraded:
        delay_ms: [50, 100]
        jitter_ms: [10, 30]
        loss_pct: [3, 8]

link_flaps:
  enabled: true
  interval_secs: { min: 30, max: 60 }
  max_down_links: 2
  down_duration_secs: { min: 10, max: 30 }
  protect_connectivity: true        # never partition the graph

traffic:
  enabled: true
  max_concurrent: 3
  interval_secs: { min: 10, max: 30 }
  duration_secs: { min: 5, max: 15 }
  parallel_streams: 4

node_churn:
  enabled: true
  interval_secs: { min: 60, max: 180 }
  max_down_nodes: 1
  down_duration_secs: { min: 30, max: 90 }
  protect_connectivity: true        # never kill the last path

bandwidth:
  enabled: false                    # per-link HTB rate limiting
  tiers_mbps: [1, 10, 100, 1000]   # each link randomly assigned a tier

logging:
  rust_log: "debug"
  output_dir: "./sim-results"

Topology Algorithms

Algorithm	Parameters	Description
random_geometric	radius (default 0.5)	Place nodes in unit square, connect pairs within radius
erdos_renyi	p (default 0.3)	Include each edge independently with probability p
chain	--	Linear chain: n01--n02--...--nN

When ensure_connected is true (default), the generator retries up to 50 times to produce a connected graph.

Directed Outbound Configs

The config generator assigns each edge to exactly one node for outbound connection using a BFS spanning tree rooted at the lowest node ID. Tree edges are assigned parent-to-child; non-tree edges are assigned from the lower node ID to the higher. This eliminates the dual-connect race condition where both sides initiate simultaneously, and creates a clear "owning side" for each link — relevant for auto-reconnect testing.

Output

Results written to sim-results/ (configurable via logging.output_dir):

analysis.txt -- Summary: panics, errors, sessions, metrics
metadata.txt -- Seed, node count, edges, adjacency list
runner.log -- Orchestration events (topology, netem, churn, traffic) with timestamps
fips-node-nXX.log -- Per-node log output

Exit code 0 on success, 2 if panics detected.

Creating Custom Scenarios

Copy an existing scenario from scenarios/.
Adjust topology size, algorithm, and stressor parameters.
Run with ./testing/chaos/scripts/chaos.sh path/to/custom.yaml.

5.1 KiB Raw Blame History