mirror of
https://github.com/jmcorgan/fips.git
synced 2026-10-05 11:08:25 +00:00
ArjenandJohnathan Corgan
95866b7c7c
Two daemons whose only transports are interface-bound, run against a veth pair the harness creates, downs, deletes and recreates underneath them. Asserts the boot race (a daemon whose only interface is missing starts, reports the transport absent and the node Degraded, rather than exiting on NoTransports or skipping the transport for the life of the process), the late attach and discovery over it, the flap in both directions, destroy-and-recreate, and that an optional interface which never appears never moves node health. Also the log policy, which is the half that is easy to regress silently: absence is logged once on the edge and not once per retry; a required interface still absent past the ten-second bring-up window errors exactly once, while the optional one — absent just as long — stays silent; and that error is not repeated on a schedule. The detach edge is checked not to error, guarded by how long detection actually took, so a slow runner skips the check rather than failing on the harness's own latency. The containers run FIPS_TEST_MODE=default, not chaos. The chaos entrypoint waits up to 30 s for every configured Ethernet interface before starting the daemon, which is precisely the workaround under test — the daemon has to do its own waiting here or the suite proves nothing. Host-namespace ip(8) runs in a short-lived privileged container sharing the host network and PID namespaces, for the reason chaos/sim/veth.py documents: on macOS the containers live in the Docker VM, so ip(8) run on the macOS host could never reach them. Chaos ethernet transports are marked optional: true. In that harness a neighbour's interface disappearing is the scenario, not a fault — node_churn stops a container, which destroys its netns and with it both ends of every veth it held, so a surviving node watches a required interface vanish for the 30-90 s the neighbour is down, once per churn event. Reporting that at error is right for a deployment and wrong for a harness that tears the interface down on purpose; the mesh-wide zero-ERROR ceiling would have failed on injected chaos rather than on a defect. test(iface-binding): cover an interface present before the daemon starts Every scenario in the suite created its interface after the daemons were already running — that ordering is the boot race the suite was written for. But it means both nodes could only ever reach Present through binder_loop, so the inline bind in start_async, which is the ordinary case on a booted router, had no end-to-end coverage at all. That is where the churn guard went unseeded and the first detach stopped reaching node health, and no existing case could reach it: they all detach from a binding the loop created, which seeds the guard as a side effect. Case (f) adds a third node whose single required interface exists before its daemon does. The gate is what buys that ordering — the harness needs a running container to have a netns to move a veth into, but the daemon must not start until after the move, so node-c comes up parked on a file and the harness releases it once the interface is in place. Then one detach, on a binding the loop did not create, and the node must degrade. Verified against the defect rather than only against the fix: with the guard seed reverted, cases (a) through (e) all still pass and (f) is the only failure. A regression test that has never been seen to fail is a claim, not a test. It also asserts the reverse edge, so Degraded stays a level rather than a latch on this path too. test(iface-binding): assert the fast path and the churn guard Three gaps, two of them in tests that existed and asserted nothing. **The netlink path was never asserted to be in use.** The 1 s poll is a complete fallback and covers every wait in the suite, so the whole thing passed with `open_link_socket()` hardcoded to Err — the fast path could have been dead for a release and no test would have said so. The binder reports which backing it got at startup, so case (g) asks it directly rather than inferring from timing the poll would also satisfy, and the unit test that used to write `let _ = w.is_event_driven();` now asserts it on Linux, where the source is an unprivileged `AF_NETLINK` socket and falling back is a real loss rather than a sandbox's prerogative. **Churn damping had no end-to-end coverage**, which now matters twice over: it bounds the recovery announcements, and since the detach edge withdraws peers it is also the only thing bounding how often that withdrawal fires. Every flap elsewhere in the suite is a single down/up with long settles either side — exactly the shape the damper ignores. Case (h) drives four bindings that each die inside `MIN_STABLE_BINDING`, asserts the guard engages, asserts it then *suppresses* rather than merely counting, and asserts it is not a latch. **`a_poisoned_binding_does_not_strand_the_transport` discarded its result.** `let _ = eth.binding.tasks_alive();` left the entire point unasserted: reading a poisoned lock as "alive" would have the binder believe a dead binding healthy and never rebind, and treating it as an error would strand the transport. `false` is what routes it back through detach and rebind, so say so. `a_stop_racing_a_bind_leaves_nothing_behind` now asserts the error *kind*. `bind_and_spawn` refuses at its presence probe long before the post-store shutdown check, so `is_err()` alone passed on absence and would still pass with that check deleted. The test keeps the coverage it genuinely has — stop raises the flag before teardown, teardown leaves no socket and no loops — and says plainly that the race it is named for needs a bind that succeeds, which needs privilege no unit test has. Both new cases were verified against the defect: with the netlink source forced to Err, (g) fails; with `CHURN_THRESHOLD` raised out of reach, (h) fails. Nothing else in the suite notices either. One case was attempted and removed rather than shipped: `"interface replaced"` cannot be produced deterministically, because the delete that changes an ifindex fires a netlink event the binder acts on within microseconds, so `gone` wins the race. It passed about one run in three. reference/notes.md records the measurement and the two approaches that could work. Also fixes a real bug in the harness: `grep -q` under `set -o pipefail` exits on its first match, `docker logs` takes SIGPIPE, and the pipeline reports failure even though the line was found. That cost two false failures before it was spotted; `log_count` reads the stream to the end. test(chaos): cover an Ethernet rebind under active traffic The one case dynamic interface binding had no coverage for anywhere: a datagram crossing an Ethernet link while the interface underneath it goes away and comes back. No existing scenario could reach it, for two separate reasons. `ethernet-only` and `ethernet-mesh` both run with `traffic.enabled: false`, so no datagram crosses an Ethernet link in any test — `ethernet-only`'s own comment says exactly that, and names framing, the length field that trims NIC minimum-frame padding, and AEAD over Ethernet as unexercised because of it. And `link_flaps` cannot produce a rebind whatever it is pointed at: it simulates a down link with netem 100% loss, so the interface stays IFF_UP and the presence machine never sees an edge. `ethernet-mesh` has had link flaps enabled all along without once exercising a rebind. `node_churn` is what actually moves an interface. Stopping a container destroys its network namespace, deleting every veth in it — and deleting one end of a veth deletes its peer — so a *surviving* node watches its Ethernet interface disappear outright, and watches it return when the harness recreates the pair on restart. That is a real detach and a real rebind, driven from outside the daemon. The new scenario is a 4-node Ethernet ring with traffic on and one node churned at a time, with link flaps deliberately off so the only outage is a genuine interface removal and a traffic shortfall cannot be ambiguous between the two. Measured across four runs: 206-388 MB moved over Ethernet links while interfaces were being taken away underneath. It also needed an assertion that did not exist. Traffic results have always been written to `iperf3-results.json` and never read, so a scenario carrying `traffic.enabled: true` could have every session fail and still exit 0 on a green control plane — and a rebind under load is precisely what a tree snapshot cannot see. `min_traffic` counts sessions that finished with bytes actually received, treating iperf3's top-level `error` and a missing `end` block as zero, so a session only counts when it moved data. The baseline is calibrated against four runs rather than assumed: `max_roots` starts at the observed maximum plus one, and the site records the sample, its size, and why four runs is thin. The first draft asserted a single root and failed every run — the harness restores stopped nodes immediately before the final snapshot, so a just-restarted node has not re-parented yet and is briefly its own root. That is the scenario working. Wired into both runners, since a chaos scenario on one side only makes "local green" and "GitHub green" stop meaning the same thing; check-ci-parity was confirmed to fail on a one-sided addition before this was committed. The iface-binding suite's entry in the GitHub workflow's integration matrix moves here from the commit that introduced the presence machine. That commit declared the suite on GitHub before testing/iface-binding/ existed and before testing/ci-local.sh knew about it, so testing/check-ci-parity.sh failed there and the three workflow steps named files that were not yet in the tree. Registering both runners in the commit that adds the suite settles both.
129 lines
4.7 KiB
YAML
129 lines
4.7 KiB
YAML
# Ethernet rebind under active traffic
|
|
#
|
|
# The one case dynamic interface binding has no coverage for anywhere:
|
|
# traffic crossing an Ethernet link while the interface underneath it goes
|
|
# away and comes back.
|
|
#
|
|
# The other Ethernet scenarios cannot reach it. `ethernet-only` and
|
|
# `ethernet-mesh` both run with `traffic.enabled: false`, so no datagram
|
|
# crosses an Ethernet link in either — framing, the length field that trims
|
|
# NIC minimum-frame padding, and AEAD over Ethernet are all control-plane
|
|
# assumptions there. And `link_flaps` cannot produce a rebind whatever it is
|
|
# pointed at: it simulates a down link with netem 100% loss, so the interface
|
|
# stays IFF_UP and the presence machine never sees an edge.
|
|
#
|
|
# `node_churn` is what actually moves an interface. Stopping a container
|
|
# destroys its network namespace, which deletes every veth in it — and
|
|
# deleting one end of a veth deletes its peer — so a *surviving* node watches
|
|
# its Ethernet interface disappear outright. On restart the harness recreates
|
|
# the pair (`NodeChurnManager._start_node`), and the survivor watches it come
|
|
# back. That is a real detach and a real rebind, driven from outside the
|
|
# daemon, with iperf3 running across the mesh throughout.
|
|
#
|
|
# Topology: a 4-node ring, so removing any single node leaves the remaining
|
|
# three connected in a line and `protect_connectivity` has something to
|
|
# protect.
|
|
#
|
|
# n01 ---eth--- n02
|
|
# | |
|
|
# eth eth
|
|
# | |
|
|
# n04 ---eth--- n03
|
|
|
|
scenario:
|
|
name: "ethernet-churn"
|
|
seed: 42
|
|
duration_secs: 240
|
|
|
|
topology:
|
|
algorithm: explicit
|
|
num_nodes: 4
|
|
default_transport: ethernet
|
|
params:
|
|
adjacency:
|
|
- [n01, n02]
|
|
- [n02, n03]
|
|
- [n03, n04]
|
|
- [n04, n01]
|
|
|
|
# Mild, and deliberately so. The variable under test is the interface going
|
|
# away, not the link being bad while it is there; heavy loss here would make a
|
|
# traffic shortfall ambiguous between the two.
|
|
netem:
|
|
enabled: true
|
|
default_policy:
|
|
delay_ms: [1, 5]
|
|
jitter_ms: [0, 1]
|
|
loss_pct: [0, 0.5]
|
|
|
|
# Off on purpose. A netem-simulated down link would add outage that never
|
|
# reaches the presence machine, which is the opposite of what this isolates.
|
|
link_flaps:
|
|
enabled: false
|
|
|
|
traffic:
|
|
enabled: true
|
|
max_concurrent: 2
|
|
interval_secs: {min: 10, max: 20}
|
|
duration_secs: {min: 15, max: 25}
|
|
parallel_streams: 2
|
|
|
|
# The mechanism. One node down at a time, long enough to outlast the
|
|
# ten-second bring-up window so the absence is a real one rather than a race,
|
|
# and to give traffic time to run against the reduced mesh before it returns.
|
|
node_churn:
|
|
enabled: true
|
|
interval_secs: {min: 45, max: 60}
|
|
max_down_nodes: 1
|
|
down_duration_secs: {min: 20, max: 35}
|
|
protect_connectivity: true
|
|
|
|
assertions:
|
|
# The mesh re-forms after each interface comes back.
|
|
#
|
|
# Calibrated 2026-09-02 against four runs, all at this file's fixed seed 42
|
|
# and therefore an identical churn schedule, so the spread is container
|
|
# timing rather than differing scenarios:
|
|
#
|
|
# nodes answering 4 in all four runs
|
|
# distinct roots 2 in all four runs
|
|
# nodes parented 2 in all four runs
|
|
# traffic 3-5 sessions, 206-388 MB
|
|
#
|
|
# `max_roots: 1` was wrong and failed every run: the harness restores
|
|
# stopped nodes immediately before the final snapshot, so a node that has
|
|
# just restarted has not re-parented yet and is briefly its own root. That
|
|
# is the scenario working, not failing.
|
|
#
|
|
# The ceiling is 3, one step beyond the observed maximum of 2, with the
|
|
# parented floor at its complement. A mesh that genuinely collapsed — every
|
|
# node islanded — still fails, which is all this assertion is for. Do not
|
|
# read a pass as convergence.
|
|
#
|
|
# READ BEFORE RETUNING: four runs is a thin sample. `churn-mixed` documents
|
|
# what happens when a threshold is set one step outside a small one — it
|
|
# ends up inside the real distribution and reddens runs whatever the daemon
|
|
# does. Widen on evidence; tighten only against a much larger sample.
|
|
baseline:
|
|
min_nodes_reporting: 3
|
|
max_roots: 3
|
|
min_nodes_parented: 2
|
|
|
|
# The point of the scenario. Traffic must actually have crossed Ethernet
|
|
# links while interfaces were being taken away underneath it — a green tree
|
|
# with zero bytes moved is the failure this catches, and is exactly what a
|
|
# control-plane-only assertion would have called a pass.
|
|
min_traffic:
|
|
min_sessions_ok: 2
|
|
|
|
# Chaos Ethernet transports are `optional: true` (see config_gen), precisely
|
|
# because a neighbour's interface disappearing is the scenario here rather
|
|
# than a fault. So absence must stay silent: any ERROR means something other
|
|
# than the churn went wrong.
|
|
max_errors:
|
|
max_total: 0
|
|
|
|
logging:
|
|
rust_log: "info"
|
|
output_dir: "./sim-results"
|