mirror of
https://github.com/jmcorgan/fips.git
synced 2026-08-09 16:24:45 +00:00
bb9bca4d83f23decbb3b0d487bf194b8690cf896
7
Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
 Johnathan Corgan
|
9b7a27f219 |
Claim the chaos network range instead of deriving it, ending the collision
Two concurrent ci-local runs could not both run chaos. Each child's subnet came from its position in the suite list, so both runs walked 10.30.0 through 10.30.12 and requested identical ranges; whichever reached the daemon first won and the other died with a pool overlap. A run index or hashed offset would only make that unlikely, and it is precisely the failure being removed. The simulator now claims its range: attempt-create on a candidate, advance on docker's own overlap error, fail loudly on anything else. Docker's address pool becomes the arbiter, so an overlap is impossible rather than improbable. The claim lives in the sim rather than in ci-local because only the process that creates the network can advance on conflict, and because it fixes the bare chaos.sh path too, which a ci-local-only fix would have left broken. The generated compose now declares the network external over the claimed one, and teardown releases the range from both paths, including the setup-failed path where a run that fell over after claiming still holds one. Ordering matters here and is not obvious: node IPs derive from the subnet during topology generation, and traffic shaping keys its filters on those addresses, so the claim happens before the topology exists. Claiming later would give a network on one range and filters on another, which does not fail at bring-up and instead leaves the shaping matching nothing. --subnet survives as an explicit pin for when a known range is wanted, and now fails loudly if that range is taken rather than silently overlapping. ci-local no longer passes it. Validated by running two instances of the same scenario deliberately concurrently: they claimed 10.30.1.0/24 and 10.30.0.0/24, both exited 0 with their assertions passing, and both released their networks. The negative control holds too: with a squatter on a pinned range the run aborts with a message naming the range rather than proceeding. |
||
|
Johnathan Corgan
|
5be7c6d0cb |
Give each chaos scenario its own container names and config directory
Chaos scenarios run four at a time under local CI, but every one of them claimed the container names fips-node-nNN and wrote its generated configs and compose file to the same generated-configs/sim directory. Container names are global in Docker and are not scoped by the compose project, so concurrent scenarios collided on both, and a scenario could start containers from a compose file another had overwritten. Thread the existing FIPS_CI_NAME_SUFFIX into the simulation. run_chaos narrows the run-wide suffix to the scenario, and the sim reads it once when the topology is built, applying it to the container names and to the config directory basename. The compose template renders the name through the topology accessor instead of duplicating the literal, so one expression produces every chaos container name. The suffix is empty when the variable is unset, so a bare chaos.sh run and the hosted CI jobs render byte-identical names and paths. Verified by rendering every scenario's compose file before and after with the variable unset and diffing. |
||
|
Johnathan Corgan
|
965de26239 |
testing: make ci-local.sh preemption-safe with per-run docker isolation
A CI worker may preempt an in-flight ci-local.sh run (SIGTERM, then SIGKILL after a grace period) to restart on a newer commit. For that kill to be safe, the script must clean up after itself and never let a dying run collide with its restart. It previously had no signal handling, shared the default compose project name across runs, and tore down each suite only at the suite end. - Derive a per-run id (honoring FIPS_CI_RUN_ID, else short-sha+random) and namespace every docker resource to it: a fipsci_<run>_<suite> compose project per suite and per parallel chaos child, and per-run image tags (fips-test:<run>, fips-test-app:<run>) retagged to :latest only after both builds succeed so :latest never points at a half-built image. - Install a bounded, idempotent teardown trap on SIGTERM/SIGINT (+ EXIT): reap parallel chaos children, then force-remove this run's docker resources via the new ci-cleanup.sh, wrapped in timeout so a stuck down cannot wedge it. - Exit 143 (SIGTERM) / 130 (SIGINT), distinct from 0 (pass) / 1 (failed), so a preempting worker tells a cancelled run from a real failure. - Add ci-cleanup.sh (also ci-local.sh --reap): force-removes leftover CI resources by the com.corganlabs.fips-ci=1 label and the fipsci_ project prefix, robust to however a prior run died. - Label every per-suite docker resource so the label sweep reaps it after a SIGKILL regardless of network name: direct docker run/network resources, the sidecar compose services, and every per-suite compose network (acl-allowlist, boringtun, firewall, nat, static, both tor suites, and the chaos generator template). Parametrize the static/sidecar compose image refs so the per-run tags are honored. - Give each parallel chaos child a unique /24 from 10.30.x (a new --subnet override on the sim CLI, assigned per-child in ci-local.sh) so parallel children never collide on a shared docker subnet, and a chaos net can never span a fixed-subnet suite (sidecar/static in 172.20.x). 10.30.x sits outside docker's default-address-pool range, so an auto-assigned net cannot land on it either; node IPs derive from the subnet, so no scenario config changes. |
||
|
Johnathan Corgan
|
fed6cc6987 |
Fix chaos sim DNS pipeline, ephemeral npub tracking, and analysis
Three infrastructure bugs fixed: - DNS resolution never populated the identity cache because Docker overwrites /etc/resolv.conf at container start. Fix: bind-mount resolv.conf in the chaos compose template, matching the static and sidecar test configs. - Traffic generator used config-time npubs for iperf3 targets, but ephemeral nodes generate fresh keypairs at startup. Fix: share the peer churn manager's runtime npub cache with the traffic manager. - Log analysis had no discovery counters and used wrong patterns. Fix: add discovery section with correct log message matching. Also adds timestamped output directories (YYYYMMDD-HHMMSS-scenario), coord_ttl_secs override in maelstrom.yaml, FIPS_SIM_OUTPUT env var support, and maelstrom-sparse scenario for sparse topology testing. |
||
|
Johnathan Corgan
|
9e63b42bd9 |
Consolidate Docker test harness infrastructure
Replace 4 near-identical per-harness Docker setups with unified shared infrastructure. Net result: -463 lines across 55 files, faster CI (8.5 min vs ~13.5 min), 14 scenarios (down from 21). Unified Docker image (testing/docker/): - Single Dockerfile (trixie-slim) with FIPS_TEST_MODE env var for mode dispatch: default, chaos, sidecar, tor-socks5, tor-directory - Single entrypoint.sh with conditional logic per mode - Replaces 5 Dockerfiles, 3 entrypoints, 4 resolv.conf copies Shared build and libraries (testing/scripts/, testing/lib/): - testing/scripts/build.sh: single build script with macOS zigbuild support, replaces 4 per-harness copies - testing/lib/derive_keys.py: shared key derivation module, replaces 3 copies of derive-keys.py - testing/lib/log_analysis.py: shared log analysis extracted from chaos sim/logs.py, with CLI interface and rekey cutover tracking - testing/lib/wait-converge.sh: shared convergence wait helpers (wait_for_links, wait_for_peers) using fipsctl JSON polling Scenario consolidation: - Remove 7 redundant scenarios: tcp-chain (subsumed by tcp-mesh), tcp-only (subsumed by tcp-mesh), chaos-10 (replaced by churn-mixed --nodes 10), churn-10/churn-20/churn-20-mixed (subsumed by parameterized churn-mixed), cost-mixed-7node (overlaps mixed-technology) - Add churn-mixed scenario with --nodes flag for scale testing - Reduce idle scenario durations: smoke-10 60s→30s, ethernet-only 90s→30s, cost-avoidance 120s→45s, depth-vs-cost 120s→45s, bottleneck-parent 120s→60s, mixed-technology 180s→90s CI updates: - ci-local.sh: unified image build, structured chaos suite entries with per-scenario flags, --skip-build for sidecar - ci.yml: shared binary install + image build step, updated scenario matrix, chaos_flags support for parameterized scenarios - Add tcp-mesh and congestion-stress to CI matrix - Static ping test uses active peer convergence detection instead of hardcoded 5s sleep Chaos infrastructure improvements (from discovery-rework branch): - Pre-built Docker image instead of per-service build at scale - --nodes flag in chaos.sh for runtime topology size override |
||
|
Johnathan Corgan
|
d29da442ac |
Add Ethernet transport with beacon discovery
Implement raw Ethernet transport using AF_PACKET SOCK_DGRAM on Linux with EtherType 0x88B5 (IEEE experimental range) and 1-byte frame type prefix (0x00=data, 0x01=beacon). Transport implementation: - EthernetConfig with interface, ethertype, MTU, buffer sizes, and four independent discovery knobs (discovery, announce, auto_connect, accept_connections) - PacketSocket/AsyncPacketSocket wrappers with ioctl helpers for interface index, MAC address, and MTU queries - EthernetTransport with Transport trait impl, async start/stop/send, receive loop dispatching data frames and discovery beacons - Discovery beacons (34 bytes: type + version + x-only pubkey) with DiscoveryBuffer for peer accumulation and dedup - Atomic statistics counters (frames, bytes, errors, beacons) - Platform-gated with #[cfg(target_os = "linux")] Transport-layer discovery integration: - Promote auto_connect() and accept_connections() to Transport trait with default implementations and TransportHandle dispatch - Extract initiate_connection() so both static peer config and discovery auto-connect share the same handshake initiation path - Add poll_transport_discovery() to the tick handler to drain discovery buffers and auto-connect to discovered peers - Enforce accept_connections() in handle_msg1() — transports with accept_connections=false silently drop inbound handshakes Node integration: - create_transports() handles Ethernet named instances - resolve_ethernet_addr() parses "interface/mac" address format - transport_mtu() generalized for multi-transport operation Test harness: - VethPair RAII struct for veth pair lifecycle management - Three #[ignore] integration tests requiring root/CAP_NET_RAW: two-node handshake, data exchange, mixed transport coexistence - Chaos harness: transport-aware topology model, VethManager for veth pairs between Docker containers, Ethernet-aware config gen, netem split (HTB+u32 for UDP, root netem for veth), transport-aware link flaps and node churn with veth re-setup - Container entrypoint waits for configured Ethernet interfaces before starting FIPS (handles veth creation timing) - New scenarios: ethernet-only (4-node ring), ethernet-mesh (6-node mixed UDP+Ethernet with netem and link flaps) Documentation: - fips-transport-layer.md: Ethernet section, beacon discovery, WiFi compatibility, updated discovery state, trait surface additions, implementation status table - fips-configuration.md: Ethernet parameter table, named instances, peer address format, mixed UDP+Ethernet example, complete reference - fips-wire-formats.md: Ethernet frame type prefix note |
||
|
Johnathan Corgan
|
66c268a564 |
Add static and stochastic Docker test harnesses
Move examples/docker-network/ to testing/static/ and add testing/chaos/ as a new stochastic simulation harness. testing/static/ — Static 5-node test harness: - Fixed mesh, chain, and mesh-public topologies with docker compose - Manual test scripts (ping, iperf, netem) - Build script, config generation, key derivation testing/chaos/ — Stochastic network simulation: - Python orchestrator generating N-node FIPS meshes with dynamic network conditions, driven by reproducible YAML scenarios - Topology generation: random geometric, Erdos-Renyi, or chain graphs with BFS connectivity guarantee - Per-link netem: HTB classful qdiscs with u32 filters for per-peer impairment (delay, loss, jitter), stochastic mutation across configurable policy profiles - Per-link bandwidth pacing: HTB rate limiting with configurable tiers (1/10/100/1000 mbps) randomly assigned per edge - Link flaps: tc netem 100% loss with graph connectivity protection - Node churn: docker stop/start with netem re-application on restart, shared down_nodes tracking across all managers - Traffic generation: random iperf3 sessions between node pairs - Down-node guards: all docker exec callers check container liveness, auto-detect crashed containers via is_container_running() safety net - Log collection and post-run analysis (panics, errors, sessions, MMP metrics, tree reconvergence) - chaos.sh wrapper with --seed, --duration, --verbose, --list options - Four scenarios: smoke-10, chaos-10, churn-10, churn-20 |