mirror of
https://github.com/jmcorgan/fips.git
synced 2026-07-30 19:46:15 +00:00
e9ca741e535fa2250cc9d9f5ad87cc3889cbb668
16
Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
 Johnathan Corgan
|
e9ca741e53 |
Assert parent selection where the cost scenarios actually specify it
Four scenarios exist to test cost-based parent selection and each named its expected outcome in a comment that nothing read. Add a tree_parents assertion mapping a node to the parent it must have in the final tree snapshot, and encode it for the two scenarios whose stated outcome the implementation meets: cost-avoidance (n04 takes the fiber n03) and bottleneck-parent (n06 takes the fiber n03, n09 keeps its only parent n05). Both hold in all six provably-completed archived runs. The other two are left unasserted on purpose, and each for a different reason worth keeping distinct. mixed-technology says n06 and n08 should both pick the fiber parent n03. They do not. Across the five completed archived runs n06 picks n10 three times and n08 picks n04 four times. Encoding the criterion as written would red the scenario most runs, and encoding the observed behaviour would bless something nobody specified, so the disagreement is recorded at the site as an open question. n06 preferring n10 may well be correct and the comment stale, since n10 reaches it by fiber too. depth-vs-cost is different: its validation line does not name an outcome at all, saying only that the choice "reflects the actual cost tradeoff", which either answer satisfies. The corpus shows both occurring under one seed, three runs to two. That scenario needs a protocol decision about which parent is correct before it can have an assertion. Compare parents by address resolved from the snapshot's own my_node_addr, and fail rather than skip when a node is absent from the snapshot or still claims to be its own root. Those two states are the common ones in the older corpus and both produce the same "no match" a wrong parent does, so only separating them keeps a harness problem from reading as a routing verdict. |
||
|
Johnathan Corgan
|
5a11cf091d |
Give congestion-stress the assertions its criteria described
The scenario listed four success criteria "verified via post-run congestion snapshot". Nothing read the snapshot, so the scenario could not fail on any of them. Add a congestion_signals assertion taking a floor on the number of nodes reporting each counter, and encode three of the four. The floors are one node each because that is what the criteria say; tightening them to the counts recently observed would assert something nobody wrote down. The fourth criterion is not encoded, and that is the finding rather than an omission. No node has reported a non-zero kernel_drop_events in any of the 182 archived runs, so asserting it would red the scenario permanently. It is recorded at the site as an unmet criterion and a coverage gap: the transport drop-detection path the scenario names as its second signal is exercised by nothing. Two things about the corpus are worth carrying, both recorded in the scenario. Only six archived runs carry a status.txt and are therefore provably completed; all six meet the three encoded criteria with five to nine nodes reporting each signal. The 176 older runs meet none of them, and they are not invalid samples: each reached teardown far enough to write an analysis.txt, and ECN landed before all but one of them. What changed on 2026-07-22 is not established, since neither the scenario nor netem.py, traffic.py or control.py has been touched. Count the nodes reporting a signal rather than the magnitude of any one counter, since a single node with a large count would satisfy a magnitude test while proving the signal never propagated. A missing snapshot fails rather than reading as an absence of congestion. |
||
|
Johnathan Corgan
|
aab149e215 |
Fail a chaos run whose nodes logged errors
The simulation exit ladder reported panics and failed assertions but said nothing about ERROR-level log lines, so a run in which every node errored on every line still exited 0. Add a max_errors assertion and apply it to every scenario by default rather than having each one opt in. This is a floor on what a green run means rather than a property of an individual scenario, and a scenario that has to ask for the floor is one that can forget to. The default ceiling of 0 is what the archived corpus supports: across 2416 result directories no node log contains an ERROR-level line, while the WARN counter extracted by the same code path ranges from 0 to 1135, so the counter is known to discriminate rather than merely known to read zero. A scenario that legitimately induces errors raises the ceiling in its own YAML and says there why. Reject rust_log: off alongside it, since that is the one level that would leave the ceiling counting zero whatever the mesh did. |
||
|
Johnathan Corgan
|
73e6917341 |
Give cost-stability the parent-switch ceiling its criterion described
The scenario's success criterion existed only as a comment: count "Parent switched" in n04's log, expect at most 5. Nothing read it, and the scenario declared no assertions at all, so its exit code could only report that the mesh came up and nothing crashed. Add a max_parent_switches assertion and wire that criterion to it. The assertion takes an optional node scope, which is the part that matters. The existing sibling counts mesh-wide, and a criterion written about one node's log is a different quantity from the sum over every node: across archived runs of this scenario n04 is 1 or 2 while the mesh-wide total ranges 3 to 6. Asserting the sum against a per-node threshold would check something other than what was specified, and at this threshold would also have failed several runs that were fine. Counting nothing must not read as stability. A node id absent from the topology fails explicitly rather than matching zero log lines and sailing under the ceiling, and a scenario that declares a parent-switch assertion while setting a log level that suppresses the events it counts is rejected at load rather than passing vacuously after a full run. The parse rejects a mistyped key, a missing, negative, boolean or non-integer ceiling, and a node written empty or as a non-string -- that last one because YAML renders a bare `node:` as null, which would silently revert to the mesh-wide count. The scenario comment now records what the threshold is worth against the 11 completed archived runs rather than a single sample: 5 is well above anything observed, the daemon's own parent hold-down caps switches near 6 per run, and nearly every switch lands during initial tree formation before any mutation fires. So this catches only near-pathological reparenting. Tightening to 3 would make it a real detector, but that changes the stated criterion rather than fixing it, so it is left as a decision. Verified against a live run: the assertion evaluates n04 at 2 against mesh-wide 4 and passes, and with the ceiling temporarily set to 0 the scenario exits 3, the assertion-failure code, which it previously could not reach. |
||
|
Johnathan Corgan
|
226c6994d3 |
Harvest chaos results only from a mesh that actually started
Teardown runs from a finally, so it also runs after a failed setup, and it was guarded only on the topology and the compose file, both of which are set well before any container exists. A scenario that died bringing its containers up therefore ran the whole harvest anyway: final snapshots, docker logs, the analysis, the assertions and the metadata, all addressing containers by names that are global to the host. What it left behind was not an empty directory an investigator would notice but a full and plausible one, in the recorded case ten node logs and an analysis reporting two promotions and two parent switches, every byte of it from a different scenario's mesh. Gate the harvest on whether the containers ever started, and leave the compose down outside that gate so a partly successful start still gets cleaned up. Record the outcome in a status file in every result directory, naming the run as completed, interrupted, aborted, setup-failed or teardown-failed, alongside the scenario, the seed and the container names it used. With the harvest gated, the presence of an analysis file is now itself proof that the scenario's own mesh existed, and the status file says which of the several ways a run can end applies. A run cut short by a signal keeps the existing exit codes rather than gaining one of its own: what it collected before stopping is real and still worth reporting, the status file records that it was truncated, and every wrapper already reports a Ctrl-C of its own. Log collection never looked at the status of docker logs. It concatenated stdout and stderr unconditionally, so collecting from containers that were not there wrote the daemon's "No such container" reply into each node log and analysed the result as a mesh with no panics, no errors and no sessions, which reads exactly like a clean run and exits zero. Check the return code, and treat a harvest that cannot read every container, or that reads none, as a failed teardown. A teardown that raises now also reports on the same footing as a run that never started, instead of escaping past the exit codes as a bare traceback and being read as a malformed command line. |
||
|
Johnathan Corgan
|
4f55a281ac |
Let a failed chaos scenario fail the run
The simulation caught every exception a scenario raised, logged it, and exited 0, so a scenario that died during setup was recorded as a pass. Local CI discarded the exit code in any case: run_chaos ends in a call whose own last statement is an echo, so it returned 0 whatever the scenario did, and the parallel launcher's wait therefore always succeeded and always recorded a pass. Between them no chaos failure of any kind could turn a local run red, and none could since the script was written. Carry the abort out of the run and give it its own exit code, ordered ahead of the two content codes so a run that never produced a mesh cannot report that mesh's panic and assertion counts. Return the scenario's status from run_chaos so the launcher sees it. The documented exit codes were stale before this change and are rewritten in full, including that the argument parser rejects a malformed command line with the same code that reports panics. Expect scenarios that have been reporting green to start reporting red. In the last recorded local run twelve of the thirteen scenarios never got past starting their containers, and all thirteen were counted as passes; those that now execute have not run for months, and their assertions have never been evaluated against a mesh of their own. |
||
|
Johnathan Corgan
|
5be7c6d0cb |
Give each chaos scenario its own container names and config directory
Chaos scenarios run four at a time under local CI, but every one of them claimed the container names fips-node-nNN and wrote its generated configs and compose file to the same generated-configs/sim directory. Container names are global in Docker and are not scoped by the compose project, so concurrent scenarios collided on both, and a scenario could start containers from a compose file another had overwritten. Thread the existing FIPS_CI_NAME_SUFFIX into the simulation. run_chaos narrows the run-wide suffix to the scenario, and the sim reads it once when the topology is built, applying it to the container names and to the config directory basename. The compose template renders the name through the topology accessor instead of duplicating the literal, so one expression produces every chaos container name. The suffix is empty when the variable is unset, so a bare chaos.sh run and the hosted CI jobs render byte-identical names and paths. Verified by rendering every scenario's compose file before and after with the variable unset and diffing. |
||
|
Johnathan Corgan
|
b3a1fb464f |
testing: add bloom-storm chaos scenario
Six-node depth-4 mesh with an induced upstream parent flap. Asserts a
trailing-window ceiling on per-node `stats.bloom.sent` and a sanity
floor on parent-switch count over a ~3-4 min observation window.
Guards against the regression class where a spanning-tree update that
changes only an internal path edge (no root or depth delta) fails to
be properly contained and instead propagates to leaves as a sustained
bloom-traffic oscillation, visible only at fleet scale and only after
several minutes of uptime.
Adds a new chaos primitive (`link_swap`) for deterministic asymmetric
link-cost flapping and a post-run assertion framework with two
checks:
- `bloom_send_rate.max_per_node`: trailing-window ceiling on the
`show_bloom` stats counter delta. Calibrated against the
post-mortem reproduction harness data (per-variant counter table
against pre-fix vs post-fix binaries).
- `min_parent_switches.min_total`: sanity guard against a
misconfigured harness where the flap inducer fires but the
topology never produces a real parent-switch event (e.g., wrong
root election from a different seed). Without this, the
bloom-rate assertion would trivially pass on any binary
including a regressed one.
The runner exits 3 on assertion failure (alongside 0 success and 2
panic-detected). Threshold derivation is documented in the scenario
README; the seed pin is also documented there since smallest-NodeAddr
root election is sensitive to the pubkey hash ordering.
Wired into ci-local.sh's chaos pool and the GitHub CI chaos matrix.
|
||
|
Johnathan Corgan
|
fed6cc6987 |
Fix chaos sim DNS pipeline, ephemeral npub tracking, and analysis
Three infrastructure bugs fixed: - DNS resolution never populated the identity cache because Docker overwrites /etc/resolv.conf at container start. Fix: bind-mount resolv.conf in the chaos compose template, matching the static and sidecar test configs. - Traffic generator used config-time npubs for iperf3 targets, but ephemeral nodes generate fresh keypairs at startup. Fix: share the peer churn manager's runtime npub cache with the traffic manager. - Log analysis had no discovery counters and used wrong patterns. Fix: add discovery section with correct log message matching. Also adds timestamped output directories (YYYYMMDD-HHMMSS-scenario), coord_ttl_secs override in maelstrom.yaml, FIPS_SIM_OUTPUT env var support, and maelstrom-sparse scenario for sparse topology testing. |
||
|
Johnathan Corgan
|
5053cf673d |
Add connect/disconnect control commands and maelstrom chaos scenario
Add runtime peer management to the FIPS daemon via control socket commands, and a new chaos simulation scenario that exercises dynamic topology mutation with ephemeral node identities. Daemon (connect/disconnect commands): - Extend control socket Request with optional params field - Add commands.rs module for mutating command dispatch, separate from read-only queries - Add api_connect() on Node: builds ephemeral PeerConfig (no auto- reconnect), pre-seeds identity cache, reuses initiate_peer_connection - Add api_disconnect() on Node: calls remove_active_peer(), clears retry_pending to suppress reconnection - Route non-show_* commands to async command dispatch in rx_loop fipsctl CLI: - Add Connect and Disconnect subcommands accepting npub or hostname - Resolve hostnames from /etc/fips/hosts before sending to daemon - Refactor socket I/O into reusable send_request helper Chaos simulator (maelstrom scenario): - Add PeerChurnManager: periodically disconnects a random active link and connects a random unconnected node pair via control socket - Add send_command() to control.py using base64-encoded JSON payloads to avoid shell quoting issues in docker exec - Add PeerChurnConfig to scenario with interval and ephemeral_fraction - Ephemeral identity support: nodes configured without nsec generate fresh keypairs on restart; simulator queries show_status for new npub and updates its cache via on_node_restart callback - Add maelstrom.yaml: all chaos dimensions (netem, link flaps, node churn, peer topology churn, traffic) with 50% ephemeral identity |
||
|
Johnathan Corgan
|
9e63b42bd9 |
Consolidate Docker test harness infrastructure
Replace 4 near-identical per-harness Docker setups with unified shared infrastructure. Net result: -463 lines across 55 files, faster CI (8.5 min vs ~13.5 min), 14 scenarios (down from 21). Unified Docker image (testing/docker/): - Single Dockerfile (trixie-slim) with FIPS_TEST_MODE env var for mode dispatch: default, chaos, sidecar, tor-socks5, tor-directory - Single entrypoint.sh with conditional logic per mode - Replaces 5 Dockerfiles, 3 entrypoints, 4 resolv.conf copies Shared build and libraries (testing/scripts/, testing/lib/): - testing/scripts/build.sh: single build script with macOS zigbuild support, replaces 4 per-harness copies - testing/lib/derive_keys.py: shared key derivation module, replaces 3 copies of derive-keys.py - testing/lib/log_analysis.py: shared log analysis extracted from chaos sim/logs.py, with CLI interface and rekey cutover tracking - testing/lib/wait-converge.sh: shared convergence wait helpers (wait_for_links, wait_for_peers) using fipsctl JSON polling Scenario consolidation: - Remove 7 redundant scenarios: tcp-chain (subsumed by tcp-mesh), tcp-only (subsumed by tcp-mesh), chaos-10 (replaced by churn-mixed --nodes 10), churn-10/churn-20/churn-20-mixed (subsumed by parameterized churn-mixed), cost-mixed-7node (overlaps mixed-technology) - Add churn-mixed scenario with --nodes flag for scale testing - Reduce idle scenario durations: smoke-10 60s→30s, ethernet-only 90s→30s, cost-avoidance 120s→45s, depth-vs-cost 120s→45s, bottleneck-parent 120s→60s, mixed-technology 180s→90s CI updates: - ci-local.sh: unified image build, structured chaos suite entries with per-scenario flags, --skip-build for sidecar - ci.yml: shared binary install + image build step, updated scenario matrix, chaos_flags support for parameterized scenarios - Add tcp-mesh and congestion-stress to CI matrix - Static ping test uses active peer convergence detection instead of hardcoded 5s sleep Chaos infrastructure improvements (from discovery-rework branch): - Pre-built Docker image instead of per-service build at scale - --nodes flag in chaos.sh for runtime topology size override |
||
|
Johnathan Corgan
|
56d39f223b |
Add ECN congestion signaling and transport congestion detection
Implement hop-by-hop ECN congestion signaling through the FMP layer, transport-level congestion detection via kernel drop counters, and chaos harness integration for end-to-end validation. FMP/session ECN plumbing: - Thread ce_flag parsed at link layer through dispatch_link_message, handle_session_datagram, handle_session_payload, and handle_encrypted_session_msg to session delivery - Replace hardcoded false in session-layer record_recv() with actual ce_flag, activating ecn_ce_count tracking in session MMP ECN congestion detection and CE relay: - Add EcnConfig (node.ecn.*) with configurable loss_threshold (5%) and etx_threshold (3.0) for transit congestion detection - Add send_encrypted_link_message_with_ce() that ORs FLAG_CE into FMP header flags; original method delegates with ce_flag=false - Compute outgoing_ce = incoming_ce || local congestion on next-hop link, enabling hop-by-hop CE relay through transit nodes IPv6 ECN-CE marking: - Mark ECN-CE (0b11) in IPv6 Traffic Class on received DataPackets before TUN delivery when FMP CE flag is set - Only marks ECN-capable packets (ECT(0)/ECT(1)); Not-ECT packets unchanged per RFC 3168 Transport congestion abstraction and UDP kernel drop detection: - Add TransportCongestion struct to transport layer for transport- agnostic local congestion indicators - Replace tokio::UdpSocket with AsyncFd<socket2::Socket> using libc::recvmsg() with ancillary data parsing - Enable SO_RXQ_OVFL for kernel receive buffer drop counter on every packet, wiring up previously-stubbed UdpStats.kernel_drops - Add TransportDropState for per-transport delta tracking with 1s tick sampling via sample_transport_congestion() - Extend detect_congestion() with transport kernel drop check alongside MMP loss metrics Congestion monitoring and control: - Add CongestionStats (ce_forwarded, ce_received, congestion_detected, kernel_drop_events) to NodeStats with snapshot serialization - Wire counters into forwarding path, session handler, and transport drop sampling with rate-limited warn logging (5s interval) - Expose congestion data in show_routing control query and ecn_ce_count in show_mmp peer entries - Add congestion counters to fipstop routing tab in two-column layout Chaos harness integration: - Add query_routing(), query_transports(), snapshot_all_congestion() to chaos control module - Add congestion/kernel-drop log analysis in logs module - Add congestion-stress scenario: 10-node tree, 1 Mbps bandwidth, 5-10% netem loss, heavy iperf3 traffic - Add IngressConfig for tc ingress policing with per-peer policer filters simulating upstream bandwidth bottlenecks - Add iperf3 JSON result capture to traffic manager for throughput measurement across scenarios - Add ECN A/B test scenarios (ecn-ab-on/off.yaml) with ingress policing and comparison script - Enable TCP ECN negotiation (tcp_ecn=1 sysctl) in container entrypoint for end-to-end CE propagation Tests: - 10 ECN unit/integration tests: mark_ipv6_ecn_ce variants, CE relay chain (3-node propagation), EcnConfig serde roundtrip - 3 transport drop congestion detection unit tests Documentation: - Update fips-mesh-layer.md: replace outdated CE Echo stub with full ECN Congestion Signaling section covering detection logic, CE relay, IPv6 marking, session tracking, and monitoring counters - Update fips-configuration.md: add node.ecn.* parameter table and ecn block in complete reference YAML - Update fips-transport-layer.md: add Congestion Reporting section with TransportCongestion struct, congestion() trait method, and per-transport status; document AsyncFd/recvmsg/SO_RXQ_OVFL in UDP - Update chaos README: add congestion/ECN scenario docs, ingress traffic control, and iperf3 JSON capture sections - Update README.md: add ECN to features list and "What works today"; update transport and tooling entries |
||
|
Johnathan Corgan
|
d29da442ac |
Add Ethernet transport with beacon discovery
Implement raw Ethernet transport using AF_PACKET SOCK_DGRAM on Linux with EtherType 0x88B5 (IEEE experimental range) and 1-byte frame type prefix (0x00=data, 0x01=beacon). Transport implementation: - EthernetConfig with interface, ethertype, MTU, buffer sizes, and four independent discovery knobs (discovery, announce, auto_connect, accept_connections) - PacketSocket/AsyncPacketSocket wrappers with ioctl helpers for interface index, MAC address, and MTU queries - EthernetTransport with Transport trait impl, async start/stop/send, receive loop dispatching data frames and discovery beacons - Discovery beacons (34 bytes: type + version + x-only pubkey) with DiscoveryBuffer for peer accumulation and dedup - Atomic statistics counters (frames, bytes, errors, beacons) - Platform-gated with #[cfg(target_os = "linux")] Transport-layer discovery integration: - Promote auto_connect() and accept_connections() to Transport trait with default implementations and TransportHandle dispatch - Extract initiate_connection() so both static peer config and discovery auto-connect share the same handshake initiation path - Add poll_transport_discovery() to the tick handler to drain discovery buffers and auto-connect to discovered peers - Enforce accept_connections() in handle_msg1() — transports with accept_connections=false silently drop inbound handshakes Node integration: - create_transports() handles Ethernet named instances - resolve_ethernet_addr() parses "interface/mac" address format - transport_mtu() generalized for multi-transport operation Test harness: - VethPair RAII struct for veth pair lifecycle management - Three #[ignore] integration tests requiring root/CAP_NET_RAW: two-node handshake, data exchange, mixed transport coexistence - Chaos harness: transport-aware topology model, VethManager for veth pairs between Docker containers, Ethernet-aware config gen, netem split (HTB+u32 for UDP, root netem for veth), transport-aware link flaps and node churn with veth re-setup - Container entrypoint waits for configured Ethernet interfaces before starting FIPS (handles veth creation timing) - New scenarios: ethernet-only (4-node ring), ethernet-mesh (6-node mixed UDP+Ethernet with netem and link flaps) Documentation: - fips-transport-layer.md: Ethernet section, beacon discovery, WiFi compatibility, updated discovery state, trait surface additions, implementation status table - fips-configuration.md: Ethernet parameter table, named instances, peer address format, mixed UDP+Ethernet example, complete reference - fips-wire-formats.md: Ethernet frame type prefix note |
||
|
Johnathan Corgan
|
0d93a19e07 |
Implement cost-based parent selection with periodic re-evaluation
Cost-based parent selection: - Replace depth-only parent selection with effective_depth = depth + link_cost - link_cost computed from locally measured MMP metrics: etx * (1.0 + srtt_ms / 100.0) - Prevents bottleneck subtrees in heterogeneous networks where a LoRa link at depth 1 would otherwise always beat fiber at depth 2 - Configurable hysteresis (default 0.2) prevents marginal parent switches - Configurable hold-down timer (default 30s) suppresses re-evaluation after parent switch - Mandatory switches (parent lost, root change) bypass both safeguards - Link costs passed as HashMap parameter to keep TreeState pure Periodic re-evaluation: - evaluate_parent() was only called on TreeAnnounce receipt or parent loss; after tree stabilization, link degradation went undetected - Added timer-based re-evaluation (reeval_interval_secs, default 60s) that calls evaluate_parent() from the tick handler with current MMP link costs - Respects existing hold-down and hysteresis safeguards - Short-circuits when disabled or <2 peers Design documentation: - Update 7 design docs to reflect cost-based parent selection - Replace depth-only algorithm descriptions with effective_depth model - Replace rejected cumulative path cost spec with local-only design rationale - Rewrite Example 2 (heterogeneous links) for local-only cost model - Update config docs: parent_switch_threshold replaced by parent_hysteresis, hold_down_secs, reeval_interval_secs Chaos simulation enhancements: - fips_overrides with deep merge for per-scenario FIPS config customization - Explicit topology algorithm for deterministic test graphs - Control socket querying via fipsctl for tree/MMP snapshot collection - Edge existence validation in netem manager - Per-link netem policy overrides - 9 new chaos scenarios covering cost avoidance, depth-vs-cost tradeoffs, stability, mixed topologies, periodic re-evaluation, and bottleneck parent 12 new unit tests, 667 total passing, clippy clean. |
||
|
Johnathan Corgan
|
78a73e1749 |
Auto-reconnect after MMP peer removal, directed outbound configs, sim improvements
Auto-reconnect: - Add per-peer auto_reconnect config (default true) to PeerConfig - schedule_reconnect() feeds removed peers back into retry system with unlimited retries and exponential backoff after MMP dead timeout - RetryState gains reconnect flag to distinguish startup retries (max_retries-limited) from auto-reconnect (unlimited) Retry re-fire fix: - process_pending_retries() now pushes retry_after_ms past the handshake timeout window after successful initiate_peer_connection(), preventing retries from firing every tick with no backoff Chaos sim improvements: - Directed outbound configs: BFS spanning tree + lower-ID-first assignment eliminates dual-connect race conditions in simulation - Save runner log (runner.log) alongside per-node logs for event correlation - Increase churn-20 traffic aggressiveness and node churn (max_down_nodes 3→5, traffic interval min 0s, duration max 90s, concurrent flows 5→10) |
||
|
Johnathan Corgan
|
66c268a564 |
Add static and stochastic Docker test harnesses
Move examples/docker-network/ to testing/static/ and add testing/chaos/ as a new stochastic simulation harness. testing/static/ — Static 5-node test harness: - Fixed mesh, chain, and mesh-public topologies with docker compose - Manual test scripts (ping, iperf, netem) - Build script, config generation, key derivation testing/chaos/ — Stochastic network simulation: - Python orchestrator generating N-node FIPS meshes with dynamic network conditions, driven by reproducible YAML scenarios - Topology generation: random geometric, Erdos-Renyi, or chain graphs with BFS connectivity guarantee - Per-link netem: HTB classful qdiscs with u32 filters for per-peer impairment (delay, loss, jitter), stochastic mutation across configurable policy profiles - Per-link bandwidth pacing: HTB rate limiting with configurable tiers (1/10/100/1000 mbps) randomly assigned per edge - Link flaps: tc netem 100% loss with graph connectivity protection - Node churn: docker stop/start with netem re-application on restart, shared down_nodes tracking across all managers - Traffic generation: random iperf3 sessions between node pairs - Down-node guards: all docker exec callers check container liveness, auto-detect crashed containers via is_container_running() safety net - Log collection and post-run analysis (panics, errors, sessions, MMP metrics, tree reconvergence) - chaos.sh wrapper with --seed, --duration, --verbose, --list options - Four scenarios: smoke-10, chaos-10, churn-10, churn-20 |