mirror of
https://github.com/jmcorgan/fips.git
synced 2026-07-30 19:46:15 +00:00
Johnathan Corgan
cdda660f10
The run built per-run images and then retagged them to fips-test:latest as a compatibility bridge for the consumers that had never been migrated. The bridge was the defect: while it existed, two concurrent runs shared one binding, so a suite could start containers from the other run's binaries and the verdict was recorded against a commit whose code never ran. It fails silently by construction — the run is green either way, and nothing compares a running container's binary against the commit under test. Every consumer now reads the run's image, so the retag is deleted rather than kept. That is the point of deleting it: a consumer that was missed fails loudly instead of quietly resolving whichever run wrote the tag last. Two consequences handled here. The cleanup script ran ip(8) inside fips-test:latest to reap simulation interfaces, and nothing writes that tag any more, so it now takes the caller's image, then the run's, then any surviving test image — the last of which is what keeps an unscoped --reap working, since that path execs before the run identity is exported. And a guard checks statically that nothing names the shared tag, because on a host with a hand-built copy lying around a reintroduction would run green; it is break-checked against a reintroduced compose consumer and a reintroduced default. Both runners gate on it, as they do the other guards.
180 lines
8.4 KiB
Markdown
180 lines
8.4 KiB
Markdown
# FIPS Testing
|
|
|
|
Integration and simulation test harnesses for FIPS, using Docker
|
|
containers running the full protocol stack.
|
|
|
|
## Test Harnesses
|
|
|
|
### [static/](static/) -- Static Docker Network
|
|
|
|
Fixed topologies with manual scripts for building, config generation,
|
|
connectivity tests (ping, iperf), and network impairment (netem).
|
|
Useful for deterministic debugging and validating specific topology
|
|
configurations.
|
|
|
|
| Topology | Nodes | Transport | Description |
|
|
| ----------- | ----- | --------- | -------------------------------- |
|
|
| mesh | 5 | UDP | Sparse mesh, 6 links, multi-hop |
|
|
| chain | 5 | UDP | Linear chain, max 4-hop paths |
|
|
| rekey | 5 | UDP | Rekey integration test topology |
|
|
|
|
### [tor/](tor/) -- Tor Transport Integration
|
|
|
|
End-to-end Tor transport testing with Docker containers running real
|
|
Tor daemons. Requires internet access for Tor bootstrapping.
|
|
|
|
| Scenario | Description |
|
|
| -------------- | -------------------------------------------------------- |
|
|
| socks5-outbound | Outbound SOCKS5 connections through Tor to clearnet peer |
|
|
| directory-mode | Inbound via HiddenServiceDir onion service (co-located) |
|
|
|
|
### [nat/](nat/) -- NAT Traversal Lab
|
|
|
|
Real Docker NAT traversal tests for the Nostr/STUN bootstrap path,
|
|
using router containers with `iptables`-based NAT, a local Nostr relay,
|
|
and a local STUN responder.
|
|
|
|
| Scenario | Description |
|
|
| --------- | ------------------------------------------------------------ |
|
|
| cone | Two NATed peers establish a UDP traversal path |
|
|
| symmetric | UDP traversal fails under symmetric NAT, TCP fallback wins |
|
|
| lan | Peers on the same LAN prefer local addresses over reflexive |
|
|
|
|
### [chaos/](chaos/) -- Stochastic Simulation
|
|
|
|
Automated network testing with configurable node counts, topology
|
|
algorithms (random geometric, Erdos-Renyi, chain, explicit), and fault
|
|
injection (netem mutation, link flaps, traffic generation, node
|
|
churn). 20 scenarios covering general stress testing, cost-based parent
|
|
selection, mixed link technologies (fiber/Bluetooth/WiFi),
|
|
transport-specific validation (UDP, TCP, Ethernet), and ECN/congestion
|
|
testing. Scenarios are
|
|
defined in YAML and executed via a Python harness that manages the full
|
|
lifecycle: topology generation, Docker orchestration, fault scheduling,
|
|
log collection, and analysis.
|
|
|
|
### [interop/](interop/) -- Mixed-Version Interop Harness
|
|
|
|
On-demand harness that runs an N-node full mesh from a node-spec where
|
|
each node can run a different build of the FIPS daemon, then attributes
|
|
every FMP/FSP/rekey/connectivity failure to a specific version pair
|
|
(same-version vs MIXED). Used to catch interop regressions between
|
|
builds, not as a per-commit CI gate; not part of `ci-local.sh`.
|
|
|
|
### [mesh-lab/](mesh-lab/) -- Mesh Reliability Lab
|
|
|
|
On-demand harness that runs a chosen integration suite N times under a
|
|
configurable host-pressure profile (idle / light / github-runner-
|
|
equivalent / heavy via `stress-ng`), per-container netem impairment,
|
|
and optional trace-level RUST_LOG, capturing per-rep diagnostics and a
|
|
mechanism-match summary across the run. Used for statistical reliability
|
|
characterization of known flake classes under calibrated stress, not as
|
|
a per-commit gate; not part of `ci-local.sh`.
|
|
|
|
## Running CI locally (`ci-local.sh`)
|
|
|
|
[`ci-local.sh`](ci-local.sh) runs the full local CI pipeline — build,
|
|
clippy, unit tests, and the integration suites (including the chaos
|
|
scenarios) — mirroring the GitHub `ci.yml` integration matrix. Run
|
|
`./ci-local.sh --help` for the full option list and `--list` for the
|
|
available suites. Every run starts with a parity check that verifies the
|
|
local suite set covers the same work as the GitHub matrix, per scenario for
|
|
chaos and per distro for deb-install; a divergence fails the run. GitHub
|
|
runs the same check as its own `ci-parity` job. `--check-parity` runs it
|
|
alone (see [check-ci-parity.sh](check-ci-parity.sh)).
|
|
|
|
### Per-run isolation and the `FIPS_CI_RUN_ID` override
|
|
|
|
Every invocation derives a **run id** and scopes all of its Docker
|
|
resources to it, so two simultaneous runs on the same host (for example,
|
|
one per git worktree, or an operator testing by hand while CI is in
|
|
flight) never collide:
|
|
|
|
- **Compose projects** are named `fipsci_<run-id>_<suite>`, so
|
|
container, network, and volume names are all prefixed per run.
|
|
- **Build images** are tagged `fips-test:<run-id>` and
|
|
`fips-test-app:<run-id>`, exported as `FIPS_TEST_IMAGE` /
|
|
`FIPS_TEST_APP_IMAGE`, and **every** compose file and suite script reads
|
|
those. The run does not write `fips-test:latest` at all: a bridge back to
|
|
that shared mutable name would let a consumer that had been missed keep
|
|
working while resolving whichever concurrent run wrote the tag last.
|
|
`:latest` stays the hand-build name, produced by
|
|
`testing/scripts/build.sh`, and remains the default every consumer falls
|
|
back to when the variables are unset.
|
|
- **The build context** is a per-run copy at `testing/docker-<run-id>/`,
|
|
exported as `FIPS_BUILD_CONTEXT`. It is absolute because compose resolves
|
|
a relative build context against the compose file's own directory rather
|
|
than the working directory. `testing/docker/` is the hand-run context and
|
|
a CI run does not write to it. Without this, two runs race on the contents
|
|
of one directory and either can build a correctly-per-run-tagged image
|
|
from the other's binaries.
|
|
- Each parallel chaos child gets a unique, non-overlapping `/24` in
|
|
`10.30.x` (via the sim `--subnet` override). `10.30.x` sits outside
|
|
Docker's default address pool and the fixed-subnet suites' `172.x`
|
|
ranges, so neither a sibling chaos child nor an auto-assigned network
|
|
can swallow a pinned subnet.
|
|
|
|
By default the run id is `<short-git-sha>-<random>` — the SHA portion
|
|
records *what code* a container is testing, the random suffix keeps
|
|
simultaneous runs of the same SHA disjoint. Override it for a
|
|
reproducible, attach-by-name debug session:
|
|
|
|
```sh
|
|
FIPS_CI_RUN_ID=mydebug ./ci-local.sh --only static-mesh
|
|
# containers are named fipsci_mydebug_static_fips-node-a, etc.
|
|
```
|
|
|
|
### Preemption-safety and exit codes
|
|
|
|
`ci-local.sh` is safe to cancel mid-run. A signal trap tears down *every*
|
|
compose project the run started (not just the current suite) and reaps
|
|
any in-flight parallel chaos children, bounded by a `timeout` so a stuck
|
|
`compose down` cannot wedge the trap. Exit codes distinguish a cancelled
|
|
run from a failing one:
|
|
|
|
| Code | Meaning |
|
|
| ---- | ------- |
|
|
| `0` | all stages passed |
|
|
| `1` | one or more stages failed |
|
|
| `130` | interrupted by SIGINT — cancelled, not a failure |
|
|
| `143` | terminated by SIGTERM — cancelled, not a failure |
|
|
|
|
A preempting CI worker (the push-triggered, CI-gated build pipeline that
|
|
kills an in-flight run when a newer same-branch tip arrives) maps
|
|
`130`/`143` → *cancelled* (discard, do not record a failing commit), `0`
|
|
→ green, any other non-zero → red.
|
|
|
|
### Cleaning up leftover resources
|
|
|
|
Every CI-created container, network, and volume carries the label
|
|
`com.corganlabs.fips-ci=1`. If a run is hard-killed (SIGKILL, OOM, crash)
|
|
and leaves resources behind, reap them with:
|
|
|
|
```sh
|
|
./ci-local.sh --reap # or: ./ci-cleanup.sh
|
|
```
|
|
|
|
[`ci-cleanup.sh`](ci-cleanup.sh) force-removes everything bearing the CI
|
|
label or a `fipsci_` compose-project prefix; it is safe to run when there
|
|
is nothing to reap and safe to run repeatedly. Pass `--project-prefix` to
|
|
scope the sweep to a single run.
|
|
|
|
It also removes the chaos simulation's leftover host-namespace veth
|
|
interfaces (`vh…a`/`vh…b`), the one resource it touches that is neither a
|
|
docker object nor labelled — a host interface can carry neither a label
|
|
nor a compose project, so it is matched by name shape alone. That makes
|
|
the reach here asymmetric with everything above, and worth stating
|
|
plainly:
|
|
|
|
- A bare `chaos.sh` run's **containers** survive a broad reap. Its
|
|
compose project is not `fipsci_`, and the simulation labels only the
|
|
network, not the services.
|
|
- A bare `chaos.sh` run's **veth interfaces do not.** An unscoped reap
|
|
deletes them while they are in use, severing the Ethernet links of a
|
|
live simulation and leaving its containers running.
|
|
|
|
So do not run a broad `--reap` while a bare simulation is up. Scope the
|
|
interface sweep with `--veth-suffixes` (which is what `ci-local.sh`'s own
|
|
teardown passes) or wait for the simulation to finish. `--project-prefix`
|
|
does not help here: it scopes only the compose-project sweep.
|