6 Commits

Include renames

Author	SHA1	Message	Date
Johnathan Corgan	fa49dc1210	Retire the mesh-public static test topology ... The mesh-public profile ran in neither runner: ci-local's static suite list carries only static-mesh and static-chain, and the GitHub matrix has only mesh and chain. It also added no coverage over static-mesh. ping-test.sh and iperf-test.sh branched mesh and mesh-public together and exercised the same 20 directed pairs among node-a through node-e, and no script referenced the external node at all. The convergence waits even used mesh's peer counts rather than mesh-public's, so the extra link to the public node was never counted, let alone asserted. Running it would therefore have added a dependency on a live internet host (test-us01.fips.network) in exchange for zero additional assertions. Remove the topology, its five compose services, the script branches that aliased it to mesh, and the documentation rows. An invocation using the old profile name now fails on ping-test.sh's unknown-profile guard instead of silently behaving as mesh. The config generator's external-node support (external_ip, is_external_node) stays. It has no consumer now, but it is woven into the config path every remaining topology uses, so removing it would put the gating suites at risk for no present gain.	2026-07-26 13:54:14 +00:00
Johnathan Corgan	6c52b0e01e	Let the static test network float so concurrent CI runs cannot collide ... Two local CI runs on one host both asked docker for 172.20.0.0/24 and the second lost its whole static family to "Pool overlaps". Docker honours a fixed subnet request verbatim, so the only robust fix is to stop making one: fips-net now requests no subnet and docker assigns from its own pool, which cannot hand the same range to two runs. That means node addresses are not known before `up`, so peers address each other by container hostname instead. The generator emits node-<id>, or the topology's docker_host where the compose hostname differs — only the gateway profile, whose services are gw-*. External peers keep the address the topology gives them, since it is not ours to assign. The resolv.conf mount stays: dnsmasq is what forwards these names to docker's resolver and .fips to the daemon, so removing it would take out every .fips assertion. generated-configs is now per-run as well. A shared directory let two runs overwrite each other's node configs, which the subnet collision had been hiding by killing runs before that window opened. The generator, the compose bind mounts and env_file, the six scripts that read it, and teardown all follow FIPS_CI_NAME_SUFFIX; unset, every path renders as before. Teardown keeps the directory after a failed run, where it is the evidence of what the failing nodes were configured with. Three things this exposed that were wrong independently: admission-cap built its tcpdump patterns from the topology file's docker_ip literals. Floating the subnet makes those match nothing, which would have left its expect-zero "no Msg2 leaked" assertion passing because it could no longer see anything at all. It now reads addresses from the running containers. Restarting the denied peers together also made them swap addresses, so each peer's counts were really the pair's total; they are restarted one at a time now, and a check fails the suite outright if two denied peers ever share an address, because per-peer attribution is impossible once they do. Attribute lookups in the generator used a fixed ten-line window and read the next node's fields when a node omitted an attribute. An external node followed by an internal one was classified as internal, which under hostname peering would emit a name that resolves nowhere. Lookups are bounded to the node's own block; generated output is byte-identical for all eight topologies. The rekey outbound-only variant used to rewrite peer addresses to hostnames to set up its scenario. The generator now does that everywhere, so the rewrite matched nothing and was silently doing no work. It asserts the premise instead, and fails if a numeric address ever reappears. Verified by running three instances of this compose at once — tcp-chain plus two independent meshes — which drew 10.128.2/3/4.0/24 with no overlap while both meshes passed ping-test 20/20 over the real .fips path. tcp-chain is run by neither CI runner, so it was checked by hand: chain peer counts 1/2/1 and multi-hop .fips reachable both directions over TCP. gateway-lan still pins its own IPv4 and fd02:: ranges and is unchanged here, so the gateway profile is not yet concurrency-safe.	2026-07-23 02:05:56 +00:00
Johnathan Corgan	e4a854f6b0	Stop concurrent local CI runs from clobbering each other's containers ... Two runs on one host destroyed each other's containers, producing mid-test "No such container" failures that look like real defects. The automated builder runs a full local CI on the same box every few minutes, so the machine is contended almost always and this has red-ed both a hand run and an automated gate. Two independent causes are fixed here. Container names were hardcoded, and docker names are global rather than scoped by compose project, so two runs collided on the same name; every name now takes an optional suffix that the harness sets from the run id. And the cleanup sweep matched a label shared by every run, so one run's teardown force-removed another's containers; resources now also carry a per-run label and the sweep can be narrowed to it. A third hazard turned up that was not in the original report: the sidecar suite passes explicit compose project names, which override the run-scoped project and put it outside the shared prefix entirely. Its project names, network, and derived container references are now scoped too. Both are default-off. With the suffix unset, names render exactly as they do today and a bare compose invocation is unchanged, which is what keeps the hosted CI and the documentation correct. A cleanup run with no run id still reaps everything, which is what a manual "clear the box" wants. The literal-name sweep was not sufficient: six scripts build container names dynamically from node labels, and two suites create their own containers outside compose. Those are handled at their construction sites. Verified: syntax check on all modified scripts; compose validation on every modified file with the suffix both set and unset; and a synthetic two-run reproduction that shows the old cleanup destroying a bystander run and the new one leaving it alone. Known gap: subnets are still hardcoded, so two concurrent full runs will still collide on address-pool overlap. That fix reaches into topology configs, chaos scenarios, diagrams and production source, so it is left for its own change rather than half-done here.	2026-07-19 06:41:25 +00:00
Martti MalmiandJohnathan Corgan	0a5c367edc	data-plane perf overhaul: off-task encrypt + decrypt, GSO, connected UDP ... Moves both AEAD layers (ChaCha20-Poly1305, one round per layer per packet) plus the sendmsg syscall off the rx_loop task onto a per-shard worker pool, adds per-peer connect(2)-ed UDP with SO_REUSEPORT, and uses Linux UDP GSO (sendmsg+UDP_SEGMENT — kernel splits one super-skb into N on-the-wire datagrams in a single TX-stack walk) when packets in a batch are uniform-size. Same kernel primitive WireGuard's in-kernel module and BoringTun use to hit 2.5–3.2 Gbps single-stream. Single TCP stream on a 5-node docker-bridge mesh, 5 x 15 s x P=1: A→D: 1379 → 2708 Mbps (1.96x, RTT +0.12 ms) A→E: 1394 → 2663 Mbps (1.91x, RTT +0.11 ms) E→A: 1406 → 2624 Mbps (1.87x, RTT +0.19 ms) Static-peer pairs only — every CoV under 3%, 0 outliers, 0% ICMP loss. The ~+100 µs RTT is the worker queue handoff cost; AEAD + sendmmsg now run on a separate core in exchange. What lands: - src/node/encrypt_worker.rs: std::thread + crossbeam_channel workers; hash-by-destination dispatch pins a TCP flow to one worker so wire ordering is preserved; per-worker sendmmsg(2) batching up to 32; Linux uses sendmsg(2)+UDP_SEGMENT when packets in a group are uniform-size. - src/node/decrypt_worker.rs: receive-side mirror. Each shard owns its session's recv cipher + replay window in a thread-local HashMap (no shared RwLock/Mutex). Sessions are handed off at promote_connection and re-registered on K-bit flip / rekey cutover. - src/node/handlers/session.rs try_send_session_data_pipelined: FSP+FMP both seal in-place in the worker on one wire-buffer alloc; no intermediate inner_plaintext / fsp_payload Vecs. - src/transport/udp/connected_peer.rs + peer_drain.rs: per-peer connect(2)-ed UDP socket with SO_REUSEPORT (set on the listen socket too — without that, EADDRINUSE on activation and every packet falls back to the wildcard path); the worker sends with msg_name=NULL and the kernel uses its cached 5-tuple. Tick- driven activation in handlers/connected_udp.rs, idempotent. - src/transport/udp/mod.rs: mem::replace the recvmmsg backing buffer instead of buf.to_vec() per packet — single pointer swap, no MTU-sized memcpy. - src/protocol/link.rs SessionDatagramRef: zero-copy borrowed view used by handle_session_datagram for the bulk local-delivery path; handle_session_payload takes the borrowed payload directly (no payload[35..].to_vec()). - src/transport/mod.rs TransportAddr::from_socket_addr: collapses the two-alloc from_string(addr.to_string()) pattern to one. - src/node/handlers/rx_loop.rs: decrypt-fallback drain promoted ahead of packet_rx in the select! (TCP ACK starvation fix); interleaved fallback drain every 32 packets inside the rx burst loop. - noise::Session: send_cipher_clone / recv_cipher_clone / recv_replay_snapshot_owned / take_send_counter / accept_replay so off-task workers can hold a cloned cipher + reserved counter while the dispatcher keeps replay/counter sequencing serial. CipherState::cipher_clone returns a refcount-bumped LessSafeKey. AsyncUdpSocket: AsRawFd so workers issue raw sendmmsg / sendmsg without going through the tokio reactor. - Worker pool sizing: both default to num_cpus, overridable via FIPS_ENCRYPT_WORKERS=N / FIPS_DECRYPT_WORKERS=N. Per-peer connected UDP can be disabled via FIPS_CONNECTED_UDP=0. - src/perf_profile.rs: optional per-stage timing reporter under FIPS_PERF=1 (or FIPS_PIPELINE_TRACE=1). Off by default; zero overhead when disabled. - All cfg(unix)-gated. Windows continues on the existing tokio- based send/recv. Decrypt worker session lifecycle: - Node::unregister_decrypt_worker_session mirrors the existing register helper. Wired at the two natural sites that already iterate peers_by_index: the rekey drain-completion block in handlers/rekey.rs (drops the worker entry for the old our_index once the drain window has expired and the cache_key is unreachable to any in-flight OLD-K packet), and remove_active_peer in handlers/dispatch.rs (drops the worker entry for each of the four index slots: current, rekey, pending, previous). Only our_index is normally registered; unregister_session is fire- and-forget for missing entries, so calling unconditionally on all four slots is correct and bounds the cleanup without per- slot accounting. Without these callers the per-worker sessions HashMap and the Node's decrypt_registered_sessions set would grow monotonically per rekey on long-lived peers. Testing: - testing/static/scripts/bench-multirun.sh: multi-run iperf3 + ping bench. N reruns (default 5), median / min / max / CoV % / per-run outlier flag, avg ping RTT, ICMP loss %, TCP retransmit total. Plain client→dest labels + topology header. Pre-bench peer-convergence check (FIPS_BENCH_CONVERGE_SECS, default 15); per-path route verification via stats.bytes_sent deltas — fails fast if traffic exits via a non-static-peer link. - testing/static/docker-compose.yml: passes FIPS_ENCRYPT_WORKERS / FIPS_DECRYPT_WORKERS / FIPS_PERF through to containers for A/B benchmarking without rebuilds. - testing/static/scripts/iperf-test.sh: same plain client→dest labels + topology header (was multihop/direct/N hop, which conflated topology distance with on-wire path). - .config/nextest.toml: synthetic UDP node tests serialized through a max-threads=1 test group. Localhost handshakes drop on shared CI runners under parallel load; one-at-a-time keeps assertions reliable. - src/node/tests/spanning_tree.rs: repair_missing_edge_handshakes — retries up to 5 times for synthetic edges whose msg1 was dropped, with a drain after each edge retry instead of after each attempt's full burst. - src/node/decrypt_worker.rs::tests: two unit tests asserting WorkerMsg::UnregisterSession removes the worker-thread session HashMap entry (handle_msg_unregister_session_removes_entry) and is a no-op for never-seen cache_keys (handle_msg_unregister_session_idempotent_on_unknown_key), which is the safety invariant the unconditional unregister calls at the four index slots in remove_active_peer rely on. - src/node/encrypt_worker.rs::unix_tests pipelined_send_wire_layout_roundtrips_canonical_decoders: mirrors the encoder geometry of try_send_session_data_pipelined (no coords, the common established-session path), runs the worker's real seal + send via flush_direct_batch_sync, and decodes the resulting wire packet using only canonical receive-side decoders (EncryptedHeader::parse, SessionDatagramRef::decode, FSP header parse, noise::open). Any divergence between the hand-rolled encoder offsets (fsp_aad_offset, fsp_plaintext_offset) and the decoders fails at one of the parse / open / decode steps before the inner-plaintext assertion fires. Complements the existing fsp_preseal_runs_before_outer_fmp_seal test which covers the seal-ordering invariant with synthetic headers but does not exercise the wire-layout invariant. CHANGELOG.md [Unreleased] # Changed entry added describing the worker-pool threading model, hash-by-destination dispatch, sendmmsg/UDP_GSO, per-peer connected UDP, the operator-facing env vars, and the bench numbers above. Cherry-picks from mmalmi/master (paths translated from crates/fips-core/src/ to src/): 9b7c723, 0deb5cb, 13f7339, e036c0e, 3740a68, 3792f83, 8510193, 4910b07, e53f545, e4e2896, 5fe4af5, 1d01ada, 8c37008, e12469e, 6eb2860. Co-authored-by: Johnathan Corgan <johnathan@corganlabs.com>	2026-05-19 20:53:31 +00:00
Martti MalmiandJohnathan Corgan	b05c80e5f5	testing: add boringtun throughput benchmark and iperf ref-compare harness ... New testing/boringtun/ harness runs two Cloudflare BoringTun userspace WireGuard containers with iperf3 between them, giving a single-hop userspace tunnel baseline for comparison against FIPS throughput numbers. Local WG key generation runs through the harness image so the host needs no wireguard-tools. New testing/static/scripts/iperf-compare-refs.sh builds two git refs into separate fips-test:* images via git worktree and runs the same static iperf topology against both, with RUNS-based repetition and aggregate avg/min/max reporting. testing/static/scripts/iperf-test.sh gains DURATION, PARALLEL, SETTLE_SECONDS, IPERF_TIMEOUT env knobs and a per-path iperf timeout. testing/static/docker-compose.yml selects the image under test via FIPS_TEST_IMAGE; testing/scripts/build.sh respects CARGO_TARGET_DIR. Author benchmark on aarch64 Docker Desktop: boringtun bob -> alice : 1000.13 Mbits/sec	2026-05-15 18:03:42 +00:00
Johnathan Corgan	66c268a564	Add static and stochastic Docker test harnesses ... Move examples/docker-network/ to testing/static/ and add testing/chaos/ as a new stochastic simulation harness. testing/static/ — Static 5-node test harness: - Fixed mesh, chain, and mesh-public topologies with docker compose - Manual test scripts (ping, iperf, netem) - Build script, config generation, key derivation testing/chaos/ — Stochastic network simulation: - Python orchestrator generating N-node FIPS meshes with dynamic network conditions, driven by reproducible YAML scenarios - Topology generation: random geometric, Erdos-Renyi, or chain graphs with BFS connectivity guarantee - Per-link netem: HTB classful qdiscs with u32 filters for per-peer impairment (delay, loss, jitter), stochastic mutation across configurable policy profiles - Per-link bandwidth pacing: HTB rate limiting with configurable tiers (1/10/100/1000 mbps) randomly assigned per edge - Link flaps: tc netem 100% loss with graph connectivity protection - Node churn: docker stop/start with netem re-application on restart, shared down_nodes tracking across all managers - Traffic generation: random iperf3 sessions between node pairs - Down-node guards: all docker exec callers check container liveness, auto-detect crashed containers via is_container_running() safety net - Log collection and post-run analysis (panics, errors, sessions, MMP metrics, tree reconvergence) - chaos.sh wrapper with --seed, --duration, --verbose, --list options - Four scenarios: smoke-10, chaos-10, churn-10, churn-20	2026-02-20 13:35:57 +00:00