Commit Graph
3 Commits
Author SHA1 Message Date
Johnathan Corgan ab750f60b3 Record why a chaos docker command failed
Compose commands run with their output captured and with check set, so a failure raised a CalledProcessError and nothing ever looked at what the daemon had said. That exception reports the argv and the exit status and nothing else, so every scenario that died bringing its containers up left a runner log saying only that docker compose returned non-zero exit status 1, and the reason was captured and then discarded. In the run that prompted this work twelve scenarios failed that way and why they failed can no longer be established at all. Log the captured stderr and stdout before the error goes anywhere, keeping the tail of each so a verbose failure cannot fill the log file. A command that times out raises before its exit status is ever read, so that path records whatever it had emitted too, rather than leaving the same hole one shape over. Reproduced against both of the shapes this is meant to catch: an unusable network prefix now records the daemon's parse error, and a name already held by another container now records the conflict and names the container holding it. The success path logs nothing new, and a completed scenario's runner log is unchanged line for line.

The raising behaviour is deliberately untouched, since the abort flag and the exit code that came with it depend on it. The check is applied here rather than by subprocess so that a caller who passed check false gets the same record: both such callers are the compose down in teardown, which today can leave a mesh behind without saying so.

The same omission was costing the veth path its reasons. Failing ip commands had their stderr logged at debug, which the runner does not emit unless asked for verbose output, so a failed pair creation reached the log as nothing more than the names of the two interfaces it could not create. Raise that to a warning. The deletes issued to clear a stale pair are expected to fail and still say nothing.
2026-07-22 08:26:59 +00:00
c1589c02af Cross-compilation and macOS compatibility fixes (#1)
- Ethernet ioctl: cfg-gate the ioctl request parameter type — c_int on
  musl, c_ulong on gnu — so the same code compiles on both
  x86_64-unknown-linux-gnu and x86_64-unknown-linux-musl targets

- Chaos harness macOS support: replace direct host 'ip link' invocations
  with a privileged Docker container helper (--net=host --pid=host) that
  shares the Docker VM's namespaces, making veth pair setup work on both
  Linux and macOS (where host 'ip' is unavailable and container PIDs
  live inside the Docker Desktop VM)

- Python 3.9 compat: add 'from __future__ import annotations' to
  docker_exec.py for X|Y union syntax support

- Add deploy/ to .gitignore

Co-authored-by: origami74 <origami74@gmail.com>
2026-02-26 03:48:51 -08:00
Johnathan Corgan 66c268a564 Add static and stochastic Docker test harnesses
Move examples/docker-network/ to testing/static/ and add testing/chaos/
as a new stochastic simulation harness.

testing/static/ — Static 5-node test harness:
- Fixed mesh, chain, and mesh-public topologies with docker compose
- Manual test scripts (ping, iperf, netem)
- Build script, config generation, key derivation

testing/chaos/ — Stochastic network simulation:
- Python orchestrator generating N-node FIPS meshes with dynamic
  network conditions, driven by reproducible YAML scenarios
- Topology generation: random geometric, Erdos-Renyi, or chain
  graphs with BFS connectivity guarantee
- Per-link netem: HTB classful qdiscs with u32 filters for per-peer
  impairment (delay, loss, jitter), stochastic mutation across
  configurable policy profiles
- Per-link bandwidth pacing: HTB rate limiting with configurable
  tiers (1/10/100/1000 mbps) randomly assigned per edge
- Link flaps: tc netem 100% loss with graph connectivity protection
- Node churn: docker stop/start with netem re-application on restart,
  shared down_nodes tracking across all managers
- Traffic generation: random iperf3 sessions between node pairs
- Down-node guards: all docker exec callers check container liveness,
  auto-detect crashed containers via is_container_running() safety net
- Log collection and post-run analysis (panics, errors, sessions,
  MMP metrics, tree reconvergence)
- chaos.sh wrapper with --seed, --duration, --verbose, --list options
- Four scenarios: smoke-10, chaos-10, churn-10, churn-20
2026-02-20 13:35:57 +00:00