mirror of
https://github.com/jmcorgan/fips.git
synced 2026-08-09 08:14:42 +00:00
feat/platform-peer-queue
9
Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
 Johnathan Corgan
|
b3a1fb464f |
testing: add bloom-storm chaos scenario
Six-node depth-4 mesh with an induced upstream parent flap. Asserts a
trailing-window ceiling on per-node `stats.bloom.sent` and a sanity
floor on parent-switch count over a ~3-4 min observation window.
Guards against the regression class where a spanning-tree update that
changes only an internal path edge (no root or depth delta) fails to
be properly contained and instead propagates to leaves as a sustained
bloom-traffic oscillation, visible only at fleet scale and only after
several minutes of uptime.
Adds a new chaos primitive (`link_swap`) for deterministic asymmetric
link-cost flapping and a post-run assertion framework with two
checks:
- `bloom_send_rate.max_per_node`: trailing-window ceiling on the
`show_bloom` stats counter delta. Calibrated against the
post-mortem reproduction harness data (per-variant counter table
against pre-fix vs post-fix binaries).
- `min_parent_switches.min_total`: sanity guard against a
misconfigured harness where the flap inducer fires but the
topology never produces a real parent-switch event (e.g., wrong
root election from a different seed). Without this, the
bloom-rate assertion would trivially pass on any binary
including a regressed one.
The runner exits 3 on assertion failure (alongside 0 success and 2
panic-detected). Threshold derivation is documented in the scenario
README; the seed pin is also documented there since smallest-NodeAddr
root election is sensitive to the pubkey hash ordering.
Wired into ci-local.sh's chaos pool and the GitHub CI chaos matrix.
|
||
|
Johnathan Corgan
|
fed6cc6987 |
Fix chaos sim DNS pipeline, ephemeral npub tracking, and analysis
Three infrastructure bugs fixed: - DNS resolution never populated the identity cache because Docker overwrites /etc/resolv.conf at container start. Fix: bind-mount resolv.conf in the chaos compose template, matching the static and sidecar test configs. - Traffic generator used config-time npubs for iperf3 targets, but ephemeral nodes generate fresh keypairs at startup. Fix: share the peer churn manager's runtime npub cache with the traffic manager. - Log analysis had no discovery counters and used wrong patterns. Fix: add discovery section with correct log message matching. Also adds timestamped output directories (YYYYMMDD-HHMMSS-scenario), coord_ttl_secs override in maelstrom.yaml, FIPS_SIM_OUTPUT env var support, and maelstrom-sparse scenario for sparse topology testing. |
||
|
Johnathan Corgan
|
5053cf673d |
Add connect/disconnect control commands and maelstrom chaos scenario
Add runtime peer management to the FIPS daemon via control socket commands, and a new chaos simulation scenario that exercises dynamic topology mutation with ephemeral node identities. Daemon (connect/disconnect commands): - Extend control socket Request with optional params field - Add commands.rs module for mutating command dispatch, separate from read-only queries - Add api_connect() on Node: builds ephemeral PeerConfig (no auto- reconnect), pre-seeds identity cache, reuses initiate_peer_connection - Add api_disconnect() on Node: calls remove_active_peer(), clears retry_pending to suppress reconnection - Route non-show_* commands to async command dispatch in rx_loop fipsctl CLI: - Add Connect and Disconnect subcommands accepting npub or hostname - Resolve hostnames from /etc/fips/hosts before sending to daemon - Refactor socket I/O into reusable send_request helper Chaos simulator (maelstrom scenario): - Add PeerChurnManager: periodically disconnects a random active link and connects a random unconnected node pair via control socket - Add send_command() to control.py using base64-encoded JSON payloads to avoid shell quoting issues in docker exec - Add PeerChurnConfig to scenario with interval and ephemeral_fraction - Ephemeral identity support: nodes configured without nsec generate fresh keypairs on restart; simulator queries show_status for new npub and updates its cache via on_node_restart callback - Add maelstrom.yaml: all chaos dimensions (netem, link flaps, node churn, peer topology churn, traffic) with 50% ephemeral identity |
||
|
Johnathan Corgan
|
9e63b42bd9 |
Consolidate Docker test harness infrastructure
Replace 4 near-identical per-harness Docker setups with unified shared infrastructure. Net result: -463 lines across 55 files, faster CI (8.5 min vs ~13.5 min), 14 scenarios (down from 21). Unified Docker image (testing/docker/): - Single Dockerfile (trixie-slim) with FIPS_TEST_MODE env var for mode dispatch: default, chaos, sidecar, tor-socks5, tor-directory - Single entrypoint.sh with conditional logic per mode - Replaces 5 Dockerfiles, 3 entrypoints, 4 resolv.conf copies Shared build and libraries (testing/scripts/, testing/lib/): - testing/scripts/build.sh: single build script with macOS zigbuild support, replaces 4 per-harness copies - testing/lib/derive_keys.py: shared key derivation module, replaces 3 copies of derive-keys.py - testing/lib/log_analysis.py: shared log analysis extracted from chaos sim/logs.py, with CLI interface and rekey cutover tracking - testing/lib/wait-converge.sh: shared convergence wait helpers (wait_for_links, wait_for_peers) using fipsctl JSON polling Scenario consolidation: - Remove 7 redundant scenarios: tcp-chain (subsumed by tcp-mesh), tcp-only (subsumed by tcp-mesh), chaos-10 (replaced by churn-mixed --nodes 10), churn-10/churn-20/churn-20-mixed (subsumed by parameterized churn-mixed), cost-mixed-7node (overlaps mixed-technology) - Add churn-mixed scenario with --nodes flag for scale testing - Reduce idle scenario durations: smoke-10 60s→30s, ethernet-only 90s→30s, cost-avoidance 120s→45s, depth-vs-cost 120s→45s, bottleneck-parent 120s→60s, mixed-technology 180s→90s CI updates: - ci-local.sh: unified image build, structured chaos suite entries with per-scenario flags, --skip-build for sidecar - ci.yml: shared binary install + image build step, updated scenario matrix, chaos_flags support for parameterized scenarios - Add tcp-mesh and congestion-stress to CI matrix - Static ping test uses active peer convergence detection instead of hardcoded 5s sleep Chaos infrastructure improvements (from discovery-rework branch): - Pre-built Docker image instead of per-service build at scale - --nodes flag in chaos.sh for runtime topology size override |
||
|
Johnathan Corgan
|
56d39f223b |
Add ECN congestion signaling and transport congestion detection
Implement hop-by-hop ECN congestion signaling through the FMP layer, transport-level congestion detection via kernel drop counters, and chaos harness integration for end-to-end validation. FMP/session ECN plumbing: - Thread ce_flag parsed at link layer through dispatch_link_message, handle_session_datagram, handle_session_payload, and handle_encrypted_session_msg to session delivery - Replace hardcoded false in session-layer record_recv() with actual ce_flag, activating ecn_ce_count tracking in session MMP ECN congestion detection and CE relay: - Add EcnConfig (node.ecn.*) with configurable loss_threshold (5%) and etx_threshold (3.0) for transit congestion detection - Add send_encrypted_link_message_with_ce() that ORs FLAG_CE into FMP header flags; original method delegates with ce_flag=false - Compute outgoing_ce = incoming_ce || local congestion on next-hop link, enabling hop-by-hop CE relay through transit nodes IPv6 ECN-CE marking: - Mark ECN-CE (0b11) in IPv6 Traffic Class on received DataPackets before TUN delivery when FMP CE flag is set - Only marks ECN-capable packets (ECT(0)/ECT(1)); Not-ECT packets unchanged per RFC 3168 Transport congestion abstraction and UDP kernel drop detection: - Add TransportCongestion struct to transport layer for transport- agnostic local congestion indicators - Replace tokio::UdpSocket with AsyncFd<socket2::Socket> using libc::recvmsg() with ancillary data parsing - Enable SO_RXQ_OVFL for kernel receive buffer drop counter on every packet, wiring up previously-stubbed UdpStats.kernel_drops - Add TransportDropState for per-transport delta tracking with 1s tick sampling via sample_transport_congestion() - Extend detect_congestion() with transport kernel drop check alongside MMP loss metrics Congestion monitoring and control: - Add CongestionStats (ce_forwarded, ce_received, congestion_detected, kernel_drop_events) to NodeStats with snapshot serialization - Wire counters into forwarding path, session handler, and transport drop sampling with rate-limited warn logging (5s interval) - Expose congestion data in show_routing control query and ecn_ce_count in show_mmp peer entries - Add congestion counters to fipstop routing tab in two-column layout Chaos harness integration: - Add query_routing(), query_transports(), snapshot_all_congestion() to chaos control module - Add congestion/kernel-drop log analysis in logs module - Add congestion-stress scenario: 10-node tree, 1 Mbps bandwidth, 5-10% netem loss, heavy iperf3 traffic - Add IngressConfig for tc ingress policing with per-peer policer filters simulating upstream bandwidth bottlenecks - Add iperf3 JSON result capture to traffic manager for throughput measurement across scenarios - Add ECN A/B test scenarios (ecn-ab-on/off.yaml) with ingress policing and comparison script - Enable TCP ECN negotiation (tcp_ecn=1 sysctl) in container entrypoint for end-to-end CE propagation Tests: - 10 ECN unit/integration tests: mark_ipv6_ecn_ce variants, CE relay chain (3-node propagation), EcnConfig serde roundtrip - 3 transport drop congestion detection unit tests Documentation: - Update fips-mesh-layer.md: replace outdated CE Echo stub with full ECN Congestion Signaling section covering detection logic, CE relay, IPv6 marking, session tracking, and monitoring counters - Update fips-configuration.md: add node.ecn.* parameter table and ecn block in complete reference YAML - Update fips-transport-layer.md: add Congestion Reporting section with TransportCongestion struct, congestion() trait method, and per-transport status; document AsyncFd/recvmsg/SO_RXQ_OVFL in UDP - Update chaos README: add congestion/ECN scenario docs, ingress traffic control, and iperf3 JSON capture sections - Update README.md: add ECN to features list and "What works today"; update transport and tooling entries |
||
|
Johnathan Corgan
|
d29da442ac |
Add Ethernet transport with beacon discovery
Implement raw Ethernet transport using AF_PACKET SOCK_DGRAM on Linux with EtherType 0x88B5 (IEEE experimental range) and 1-byte frame type prefix (0x00=data, 0x01=beacon). Transport implementation: - EthernetConfig with interface, ethertype, MTU, buffer sizes, and four independent discovery knobs (discovery, announce, auto_connect, accept_connections) - PacketSocket/AsyncPacketSocket wrappers with ioctl helpers for interface index, MAC address, and MTU queries - EthernetTransport with Transport trait impl, async start/stop/send, receive loop dispatching data frames and discovery beacons - Discovery beacons (34 bytes: type + version + x-only pubkey) with DiscoveryBuffer for peer accumulation and dedup - Atomic statistics counters (frames, bytes, errors, beacons) - Platform-gated with #[cfg(target_os = "linux")] Transport-layer discovery integration: - Promote auto_connect() and accept_connections() to Transport trait with default implementations and TransportHandle dispatch - Extract initiate_connection() so both static peer config and discovery auto-connect share the same handshake initiation path - Add poll_transport_discovery() to the tick handler to drain discovery buffers and auto-connect to discovered peers - Enforce accept_connections() in handle_msg1() — transports with accept_connections=false silently drop inbound handshakes Node integration: - create_transports() handles Ethernet named instances - resolve_ethernet_addr() parses "interface/mac" address format - transport_mtu() generalized for multi-transport operation Test harness: - VethPair RAII struct for veth pair lifecycle management - Three #[ignore] integration tests requiring root/CAP_NET_RAW: two-node handshake, data exchange, mixed transport coexistence - Chaos harness: transport-aware topology model, VethManager for veth pairs between Docker containers, Ethernet-aware config gen, netem split (HTB+u32 for UDP, root netem for veth), transport-aware link flaps and node churn with veth re-setup - Container entrypoint waits for configured Ethernet interfaces before starting FIPS (handles veth creation timing) - New scenarios: ethernet-only (4-node ring), ethernet-mesh (6-node mixed UDP+Ethernet with netem and link flaps) Documentation: - fips-transport-layer.md: Ethernet section, beacon discovery, WiFi compatibility, updated discovery state, trait surface additions, implementation status table - fips-configuration.md: Ethernet parameter table, named instances, peer address format, mixed UDP+Ethernet example, complete reference - fips-wire-formats.md: Ethernet frame type prefix note |
||
|
Johnathan Corgan
|
0d93a19e07 |
Implement cost-based parent selection with periodic re-evaluation
Cost-based parent selection: - Replace depth-only parent selection with effective_depth = depth + link_cost - link_cost computed from locally measured MMP metrics: etx * (1.0 + srtt_ms / 100.0) - Prevents bottleneck subtrees in heterogeneous networks where a LoRa link at depth 1 would otherwise always beat fiber at depth 2 - Configurable hysteresis (default 0.2) prevents marginal parent switches - Configurable hold-down timer (default 30s) suppresses re-evaluation after parent switch - Mandatory switches (parent lost, root change) bypass both safeguards - Link costs passed as HashMap parameter to keep TreeState pure Periodic re-evaluation: - evaluate_parent() was only called on TreeAnnounce receipt or parent loss; after tree stabilization, link degradation went undetected - Added timer-based re-evaluation (reeval_interval_secs, default 60s) that calls evaluate_parent() from the tick handler with current MMP link costs - Respects existing hold-down and hysteresis safeguards - Short-circuits when disabled or <2 peers Design documentation: - Update 7 design docs to reflect cost-based parent selection - Replace depth-only algorithm descriptions with effective_depth model - Replace rejected cumulative path cost spec with local-only design rationale - Rewrite Example 2 (heterogeneous links) for local-only cost model - Update config docs: parent_switch_threshold replaced by parent_hysteresis, hold_down_secs, reeval_interval_secs Chaos simulation enhancements: - fips_overrides with deep merge for per-scenario FIPS config customization - Explicit topology algorithm for deterministic test graphs - Control socket querying via fipsctl for tree/MMP snapshot collection - Edge existence validation in netem manager - Per-link netem policy overrides - 9 new chaos scenarios covering cost avoidance, depth-vs-cost tradeoffs, stability, mixed topologies, periodic re-evaluation, and bottleneck parent 12 new unit tests, 667 total passing, clippy clean. |
||
|
Johnathan Corgan
|
78a73e1749 |
Auto-reconnect after MMP peer removal, directed outbound configs, sim improvements
Auto-reconnect: - Add per-peer auto_reconnect config (default true) to PeerConfig - schedule_reconnect() feeds removed peers back into retry system with unlimited retries and exponential backoff after MMP dead timeout - RetryState gains reconnect flag to distinguish startup retries (max_retries-limited) from auto-reconnect (unlimited) Retry re-fire fix: - process_pending_retries() now pushes retry_after_ms past the handshake timeout window after successful initiate_peer_connection(), preventing retries from firing every tick with no backoff Chaos sim improvements: - Directed outbound configs: BFS spanning tree + lower-ID-first assignment eliminates dual-connect race conditions in simulation - Save runner log (runner.log) alongside per-node logs for event correlation - Increase churn-20 traffic aggressiveness and node churn (max_down_nodes 3→5, traffic interval min 0s, duration max 90s, concurrent flows 5→10) |
||
|
Johnathan Corgan
|
66c268a564 |
Add static and stochastic Docker test harnesses
Move examples/docker-network/ to testing/static/ and add testing/chaos/ as a new stochastic simulation harness. testing/static/ — Static 5-node test harness: - Fixed mesh, chain, and mesh-public topologies with docker compose - Manual test scripts (ping, iperf, netem) - Build script, config generation, key derivation testing/chaos/ — Stochastic network simulation: - Python orchestrator generating N-node FIPS meshes with dynamic network conditions, driven by reproducible YAML scenarios - Topology generation: random geometric, Erdos-Renyi, or chain graphs with BFS connectivity guarantee - Per-link netem: HTB classful qdiscs with u32 filters for per-peer impairment (delay, loss, jitter), stochastic mutation across configurable policy profiles - Per-link bandwidth pacing: HTB rate limiting with configurable tiers (1/10/100/1000 mbps) randomly assigned per edge - Link flaps: tc netem 100% loss with graph connectivity protection - Node churn: docker stop/start with netem re-application on restart, shared down_nodes tracking across all managers - Traffic generation: random iperf3 sessions between node pairs - Down-node guards: all docker exec callers check container liveness, auto-detect crashed containers via is_container_running() safety net - Log collection and post-run analysis (panics, errors, sessions, MMP metrics, tree reconvergence) - chaos.sh wrapper with --seed, --duration, --verbose, --list options - Four scenarios: smoke-10, chaos-10, churn-10, churn-20 |