Reconciles the tick profiler with this line's extra tick step. `next`
drives `resend_pending_fmp_rekey_msg3` from the tick arm and the master
line does not, so the step table gains a variant here and the arm gains
one more instrumented call.
The pinned row-count test is what caught it: it failed on the merge
rather than letting the emitted table and the call sites drift apart
silently, which is the failure mode it was written for.
The mesh rehearsal could not exercise the write-error stop at all, and
that is not a gap in the rehearsal. Removing the sink file leaves the
writer's descriptor valid, so writes keep succeeding into the unlinked
inode and the capture runs on; mounting a tiny filesystem inside the test
container is refused outright. So the terminal state added for a failing
sink had no coverage from either direction.
Split one flush cycle out of the writer loop as a function returning what
it decided, so the loop owns the waiting and the state transition while
the decision is testable on its own. A sink that fails every write now
drives the error outcome directly, and a working sink is asserted to
continue so the first test cannot pass for a writer that always stops.
The residual gap is recorded at the test: what is covered is that an
error from the sink produces the error outcome rather than the cap
outcome, not that a real full disk reaches that branch.
The rehearsal itself was otherwise clean on a live three-node mesh. The
header carries node, build, platform, tick period and the reading
caveats; every step reports every interval; stopping returns in 53 ms
rather than waiting out the flush interval; an idle status reports no
path and no bytes; arming twice is refused naming the active file; an
unwritable directory fails the command; and the node kept serving
throughout. On an idle node the entry gap sits at one tick period and
arm starvation reads about 1.4 ms, which is small but not zero, so the
measurement is live rather than degenerate.
The tick arm runs twenty-six housekeeping steps in sequence on the one
runtime thread, and is polled last, so anything slow in it holds up
inbound packets, TUN traffic and control commands behind it. Field
evidence says that happens for over a second at a time, but the
attribution behind that is two months old and predates the
connect-on-send gate, the control read isolation, and the peer lifecycle
rework. This measures it rather than continuing to reason about it.
Per step it records exact count, max and total into fixed static
counters; a dedicated writer thread drains them every ten seconds to a
TSV under /var/log/fips, one file per capture, capped at 32 MB. Nothing
accumulates: the counters are swapped to zero each interval and the
thread holds no history. Arming is `fipsctl profile tick on`, served in
the control accept task so the toggle cannot queue behind the very
behaviour it measures, and it does not survive a restart.
The whole thing is behind a Cargo feature that is off by default,
because the risk worth eliminating is the twenty-six edited call sites
in the hot loop. With the feature off the macro expands to the bare
expression, which makes the default build's neutrality something you
read off the generated code rather than something a benchmark fails to
disprove.
The measurement that matters is how late each tick is against the
deadline it was scheduled for, since the arm is polled last and that
lateness is the delay. Two earlier designs derived it from the interval
between entries and both under-reported: the schedule is fixed, so a
steady delay leaves every gap exactly one period and any gap-derived
figure reads zero under precisely the sustained overload this is meant
to find. The interval hands back its own deadline, so the delay is now a
subtraction with no model behind it, and a test drives three late ticks
at a constant gap to keep it that way.
CI gains a default-features clippy and a feature-on build and test on
both runners, closing the gap left by clippy already running with all
features.
The NAT lab was the last suite pinning fixed IPv4 subnets, so two overlapping
runs collided on the wan and shared-lan bridges.
Each run now claims a free /24 for each bridge, scanning candidates and
advancing on an overlap while still failing fast on any other network-create
error. The claim exports NAT_WAN_PREFIX and NAT_LAN_PREFIX, and every routable
address in the compose file and the suite scripts derives from them, with
defaults that render exactly what the lab used before. The router-side LANs are
deliberately left pinned: they live inside per-container network namespaces,
never become docker networks, and cannot collide.
An external-network overlay lets the suites attach to the networks the run
already claimed instead of creating their own. Two of the three suite scripts
had no overlay hook, so they would have requested the claimed range a second
time; both now have one.
Host veth names are scoped by a short token rather than the full run id, which
overruns the fifteen-character interface-name limit at the default run-id
length, and the cleanup reaper's pattern is widened to match the new shape.
Networks are released inline on every exit path rather than in a trap, since a
trap written inside a shell function replaces the script-level handler and
would disable the whole run's teardown.
The NAT lab wrote its generated node configs to one shared directory, unlike
the static and firewall labs which already scope theirs by run. The directory
is bind-mounted by compose and read back by the suite scripts after the
containers are up, so two overlapping runs let the second run's generator
overwrite the npubs the first is about to ping.
Scope the directory with FIPS_CI_NAME_SUFFIX at all three places that have to
agree: the generator's output path, the ten compose bind-mounts, and the
CONFIG_DIR the suite scripts read back from. An unset suffix renders the plain
path the lab has always used, so a bare invocation and the GitHub matrix are
unaffected. The run teardown removes the per-run directory alongside the static
and firewall ones, and the gitignore is widened to cover the suffixed form.
Rendering the compose file with every profile and no suffix set reproduces
today's exact ten paths; setting a suffix moves all ten.
This lands on its own, ahead of the network and address work, so that a failure
of the two-overlapping-runs acceptance test can be bisected between the config
fix and the address conversion.
Windows unit tests have been reding on next at a rekey-drain assertion that
reads like a defect in the drain logic. The drain logic is correct; the test
never reached the state it asserts on.
All three test-only backdating seams shared the fallback
checked_sub(age).unwrap_or_else(Instant::now). When the requested backdate
falls outside the monotonic clock's representable range, that assigns the
present instant, moving the timestamp forward rather than backward. A fresh
Windows CI runner has been up for less than the 600 seconds the drain test
asked for, so the window looked brand new, check_rekey correctly declined to
release the demoted session, and the assertion fired against innocent code.
The seams now route through a shared helper that panics naming the field, the
requested age and the real cause. The drain test asks for 30 seconds, three
times the ten-second window it needs to clear, rather than sixty times it.
Both properties were established by breaking them. Backdating by five seconds,
inside the window, reds the test, so it still depends on expiry rather than
passing vacuously at the smaller margin. Forcing the subtraction to fail
produces the new panic and its explanation rather than a silent reset. Full
cargo test --lib green at 1742, fmt and clippy clean.
No step ran on Windows, so the platform mechanism is inferred from the split
between runners and confirmed only when that job goes green on this tip.
Carries today's maint batch up: the local CI image-scoping work, which
gives each run its own build context and test image tag instead of
writing the shared mutable one, and the retirement of the bloom-storm
chaos scenario.
Resolved as the earlier per-commit merges did, in the two files where
this line has diverged — the node test module list and the static suite
compose file. Parity holds at 21 legs a side, matching master.
Carries today's maint batch: the local CI image-scoping work, which gives
each run its own build context and test image tag instead of writing the
shared mutable one, and the retirement of the bloom-storm chaos scenario.
Both apply unchanged here — the compose files, suite lists and matrix legs
they touch are identical on the two lines, so no branch adaptation was
needed. Parity stays symmetric across runners at 21 legs a side on this
branch and 24 on maint, the gap being the three rekey Docker suites maint
keeps by design.
The scenario guarded a real regression: a mid-chain tree update that
changed neither root nor depth leaking downstream as a sustained bloom
announce storm. But it was never once run against the regressed binary,
and its per-node ceiling was inferred from a post-mortem harness that no
longer exists in the tree. On the only surviving regressed measurement
the tail node's rate scales to roughly 7 sends per 30s, well under the
scenario's ceiling of 40, so it was never established that the assertion
could fire on its own bug class. The ceiling is also uniform per node,
calibrated against the flap target that is legitimately busy rather than
against the tail where the storm actually shows.
This removes coverage rather than relocating it, unlike the two earlier
retirements above it, and the comment in ci-local.sh records that gap
explicitly. The scenario, its README, the link_swap sim primitive and the
mesh-lab dispatch all stay on disk, so it remains runnable by hand.
Parity holds at 24 legs a side. The parity guard was break-checked by
re-adding the GitHub leg alone, which correctly reported the asymmetry.
The run built per-run images and then retagged them to fips-test:latest as
a compatibility bridge for the consumers that had never been migrated. The
bridge was the defect: while it existed, two concurrent runs shared one
binding, so a suite could start containers from the other run's binaries
and the verdict was recorded against a commit whose code never ran. It
fails silently by construction — the run is green either way, and nothing
compares a running container's binary against the commit under test.
Every consumer now reads the run's image, so the retag is deleted rather
than kept. That is the point of deleting it: a consumer that was missed
fails loudly instead of quietly resolving whichever run wrote the tag last.
Two consequences handled here. The cleanup script ran ip(8) inside
fips-test:latest to reap simulation interfaces, and nothing writes that tag
any more, so it now takes the caller's image, then the run's, then any
surviving test image — the last of which is what keeps an unscoped --reap
working, since that path execs before the run identity is exported. And a
guard checks statically that nothing names the shared tag, because on a
host with a hand-built copy lying around a reintroduction would run green;
it is break-checked against a reintroduced compose consumer and a
reintroduced default. Both runners gate on it, as they do the other guards.
Scoping the image tag was never sufficient on its own. Every build read one
unscoped directory in the working tree, into which each run copies the
binaries it just built, so two concurrent runs raced on the contents of the
context as well as on the name of the result — and a run could produce a
correctly-per-run-tagged image built from the other run's binaries.
The run now copies the context's tracked files into its own directory,
installs its binaries there, and builds from it, exporting the path so every
other consumer follows. Deliberately not carried over: a previous run's
binaries, since inheriting them is the failure this prevents. The path is
absolute because compose resolves a relative build context against the
compose file's own directory rather than the working directory, which was
measured rather than assumed.
The chaos entry script gated the whole simulation on a binary in the shared
directory by literal path, so it moves in this same commit: left behind, it
would have failed on a clean checkout and, worse on a host with leftovers,
passed while reading a binary that was not the one under test.
Teardown removes the directory on red runs as well as green, since it holds
only reproducible content and is never the evidence of a failure. The
worker's SIGKILL runs no trap, so the cleanup script also sweeps contexts
left by a preempted run, and the ignore rule keeps a concurrent run from
showing up as untracked working-tree noise.
The runner rebuilt the shared tag from the shared build context at the
start of every scenario, unconditionally. Local CI runs chaos scenarios in
parallel, so that was several concurrent builds into one name inside a
single run, before any second run is considered, and it silently replaced
whatever image the harness had already built and handed over.
It now uses the caller's image when one is named, and asserts the image is
present rather than building a substitute, because under a harness a miss
means something upstream is broken and building would hide it. A bare run
still builds the shared tag exactly as before, reading the run's own build
context when one is set.
Only the static family read FIPS_TEST_IMAGE; the firewall, ACL, nat,
sidecar and both Tor compose files named the shared mutable tag directly,
which is why the local CI runner has to retag its per-run image to that
name. They now take the same defaulted form the static file already uses,
so a harness run resolves the image it built and a bare hand run still
resolves the shared tag exactly as before.
Checked by rendering each file through `docker compose config` with the
variables set: every service resolves to the supplied name and none is
left naming the shared tag.
Both suites ran `docker compose up -d --build` unconditionally, and their
compose files carry both a build context and an image name, so the flag
rebuilt and retagged the shared test image from the shared build context on
every run. Their --skip-build flag did not cover it: it guarded only the
Rust build above. Under a harness that has already built the image and
handed it over, that rebuild can only replace the binaries under test with
whatever the shared context currently holds.
The nat scripts had the mirror-image problem. Their image guard built a
replacement whenever the image was missing, which is right by hand and
wrong under a harness: when the caller names an image it built, a miss
means something upstream is broken, and manufacturing a substitute hides
that. They now fail loudly in that case and keep building only on the hand
path. The mesh-lab guard reads the same variable and its build hint moves
to the script that still produces the shared tag.
The root-count ceiling of 4 was set from six runs whose observed maximum
was 3. Fourteen runs that recorded a baseline verdict put the real
distribution at 1 to 5 roots, so the ceiling sat at roughly the 93rd
percentile of the scenario's own variance: one sample in fourteen
exceeded it and four sat at or above it. It therefore reddened a share of
runs whatever the daemon did, which is a calibration fault rather than a
convergence one. Every sample ran the file's fixed seed, so the spread is
container timing and not a differing chaos schedule.
The ceiling is now 6, one step beyond the observed maximum, and the
parented floor moves to its complement at 4. That still fails a mesh
where seven or more of ten nodes are islanded, which is the collapse case
the assertion exists to catch; driving evaluate_baseline directly
confirms all fourteen samples pass and 7, 8 and 10 roots still fail.
min_sessions is left at 10 against an observed minimum of 12. It has
never fired, but the comment now records the margin as thin so a future
failure there is read as calibration first.
All three required root or CAP_NET_RAW, neither runner passes --ignored,
and so none had ever executed in CI. Before deleting them I checked what
each covered and whether anything else covered it:
- The two-node handshake is covered by the ethernet-only chaos scenario,
whose baseline assertion cannot pass unless handshakes complete over
AF_PACKET.
- Tree convergence and root election by smallest NodeAddr are covered by
ethernet-only's single-root assertion and by the in-process
spanning-tree tests respectively.
- The mixed-transport coexistence property is covered twice over by
test_tcp_mixed_transport_coexistence and test_ble_mixed_transport, which
call the same verify_tree_convergence_components helper with the same
two-component shape and both run today.
So the only thing running them would have added is an AF_PACKET variant of
a property already proven on two other transports.
The counts confirm nothing running was lost: 1392 tests pass before and
after, and the ignored count falls from 7 to 4.
The ethernet-only comment block cited these tests as the reason its
assertion was the only Ethernet coverage in CI. It now records the sharper
fact found while reading them: that coverage is control plane only. Both
Ethernet scenarios disable traffic, so no datagram crosses an Ethernet
link anywhere, which leaves the frame length field that trims NIC minimum
frame padding ahead of AEAD verification unexercised. Deleting these tests
does not widen that gap, because the test named for data exchange did not
exchange any.
The shared analyzer matched two strings for one event: "Peer promoted to
active", emitted at info, and "Outbound handshake completed", emitted at
debug. Both come from handle_msg2 on the same call path for the same
outbound promotion, so any run at debug level counted every outbound
promotion twice.
Measured rather than argued. On the archived debug-level churn-mixed run
of 2026-07-25, the node logs carry 20 of each string; the old matcher
reported 40 promotions and the new one reports 20. The neighbouring
counters are unmoved at 13 sessions and 33 parent switches.
Info-level runs are unaffected, which was checked rather than assumed: on
the tcp-mesh run of the same date the debug string occurs zero times, and
the old and new matchers both report 7 against 7 raw occurrences. Three
scenarios run at debug, one of them gating, so this corrects the promotion
figure in their artifacts. No assertion reads the counter, so no suite's
verdict changes.
The earlier fix that introduced this added the live string alongside the
dead one where it needed to replace it.
The tcp-chain profile has never been runnable. ping-test.sh dispatches on
chain and mesh only, and the string tcp-chain has never appeared in that
script in its history, so invoking the profile has always fallen through
to the unknown-profile branch: silently asserting nothing before that
branch was made to fail, and exiting 2 since. Neither runner referenced
it either.
What it would have covered is already covered. The chaos tcp-mesh
scenario runs in local CI and gives TCP a discriminating gate: n04's only
edges are TCP, so if the transport were broken n04 could not parent and
the scenario's baseline assertion would fail. It also carries a pure-TCP
two-hop path, n01 to n04 to n05, under netem and link flaps. The only
residual tcp-chain would have added is a mesh with no UDP present
anywhere, which is not worth a fixture that has never run.
Remove the topology, its three compose services, and the documentation
rows. Every profile the static compose file still defines is now
exercised by a suite.
Also correct the chaos README, which has listed tcp-only and tcp-chain in
its transport table since those two scenarios were deleted in 9e63b42.
Neither has existed for months.
The mesh-public profile ran in neither runner: ci-local's static suite
list carries only static-mesh and static-chain, and the GitHub matrix has
only mesh and chain. It also added no coverage over static-mesh.
ping-test.sh and iperf-test.sh branched mesh and mesh-public together and
exercised the same 20 directed pairs among node-a through node-e, and no
script referenced the external node at all. The convergence waits even
used mesh's peer counts rather than mesh-public's, so the extra link to
the public node was never counted, let alone asserted.
Running it would therefore have added a dependency on a live internet
host (test-us01.fips.network) in exchange for zero additional assertions.
Remove the topology, its five compose services, the script branches that
aliased it to mesh, and the documentation rows. An invocation using the
old profile name now fails on ping-test.sh's unknown-profile guard
instead of silently behaving as mesh.
The config generator's external-node support (external_ip,
is_external_node) stays. It has no consumer now, but it is woven into the
config path every remaining topology uses, so removing it would put the
gating suites at risk for no present gain.
wait_for_peers reads the connected-peer count through a pipeline ending in
`|| echo 0`, so a container that never answers contributes 0. That is safe
against a floor of 1 or more, where 0 reads as "not converged yet" and the wait
eventually times out, and it is unsafe against a floor of 0, where the first
read from a dead container satisfies the wait and the caller proceeds as though
convergence had been observed.
No caller passes 0 today; every one passes 1 or more, and the single variable
minimum is advisory. Rather than harden the reader, reject the input: a floor
below 1 is now an error. That makes the shape unreachable instead of repairing
instances of it, and it costs a future caller nothing except a clear message
saying that asserting "exactly zero peers" needs a reader that distinguishes no
answer from zero, which this floor is not.
wait_for_links goes with it. It had no caller anywhere in the tree on any
branch, and it carried the identical fallback, so the only thing it could do was
hand the hazard to whoever called it first. An uncalled helper cannot be wrong
today, which is exactly why it was the risk worth removing rather than the one
worth keeping for symmetry.
The hermetic gate suite gains a case covering both directions, with docker
stubbed so it stays container-free: a floor of 0 is refused with a message
naming why, and the same unreachable container with a floor of 1 still polls its
full budget and times out. Verified by removing the guard and confirming exactly
the three zero-floor assertions red while both floor-of-one controls stay green.
wait_for_peers_exact read the connected-peer count through a pipeline ending in
`|| echo 0`, so a container that never answered and a daemon that answered zero
produced the same value. That is only harmless while every caller expects a
non-zero count, which is true here today and is the reason this copy was left
alone when the acl suite's copy was fixed. It leaves the trap armed for whoever
adds the first caller expecting zero: the check would be satisfied on its first
iteration without the property it exists to verify ever being observed.
The read moves into its own function that returns the empty string when the
container does not answer, and the caller treats empty as "no answer" rather
than as a count. A run that never gets an answer now fails saying so, distinctly
from one that answered the wrong count, and the diagnostic peer dump runs only
on the latter, since dumping from a container that cannot answer prints a docker
error rather than evidence.
This is the shape the acl-allowlist suite already uses. The two copies had
diverged on whether a silent container counts as an answer, which is the kind of
disagreement that decides a security assertion in whichever file is read last.
Checked by construction rather than by a green run: driving the old form against
an absent container with an expected count of zero returns success on the first
iteration, and the new form fails; with an expected count of one, the shape both
real callers use, both forms still fail, and a genuine zero from a live daemon
is still reported as zero.
The compose pinned 172.32.0.0/24 with a per-container ipv4_address, so two
concurrent runs asked docker for the same address space and the second failed
with a pool-overlap error. Request no subnet and let docker assign one from the
daemon pool. Peers address each other by the docker hostname the compose already
sets (host-a, host-b) rather than by literal IP; docker's embedded DNS is
per-network, so the same hostname in two runs resolves inside each run's own
subnet. Nothing this suite asserts on moves as a result: its checks run over the
fips0 overlay, whose addresses are derived from node npubs and are independent of
docker addressing, and the packaged nftables ruleset matches on interface rather
than on any address.
The generated config directory moves under the run suffix for the same reason it
did in the acl suite, and the run teardown gains the matching removal so a run
no longer leaves its directory behind.
Case (b) also wrote curl's output to a fixed path under /tmp. Two concurrent
runs shared that one file, and either run's cleanup landing between the other's
write and read left an empty read, failing the http_code check for a reason
having nothing to do with the firewall. It uses mktemp now.
As in the acl suite, the floating subnet removes one of the two obstacles to
concurrent runs. The compose project name is still fixed; the local CI runner
scopes it externally, a bare hand run does not, and the comment at the site says
so rather than claiming the file is self-sufficient.
The compose pinned 172.31.0.0/24 with a per-container ipv4_address, so two
concurrent runs of this suite asked docker for the same address space and the
second failed with a pool-overlap error. Request no subnet instead and let
docker assign one from the daemon pool, which leaves nothing for two runs to
contend for. Peers now address each other by the docker hostname the compose
already sets (host-a through host-f) rather than by literal IP, and docker's
embedded DNS is per-network, so the same hostname in two runs resolves inside
each run's own subnet.
The generated config directory moves under the run suffix in the same change,
because it has to: the generator does rm -rf on it, so a shared directory means
one run deletes the fixtures another is still using, which fails silently rather
than loudly at bring-up. Scoping it requires the generator output path, all five
bind-mount sources per service, and the gitignore entry together; scoping only
the generator leaves the compose reading the unscoped path.
The floating subnet removes one of the two obstacles to running this suite
twice at once, not both: the compose project name is still fixed, and nothing
scopes it for this suite. Recorded at the site so the comment does not claim
more than the change delivers.
Everything this suite asserted is now covered in-process over loopback by
mixed_profile_nodes_converge_and_forward: mixed Full, NonRouting and Leaf
convergence, the peer degrees, every direct reachability pair, and the
Leaf-to-Full pair that routes through the Full node between them. The
in-process test covers a superset, asserting that multi-hop path in both
directions where the Docker suite covered one.
Retired on evidence that the multi-hop assertion discriminates rather than on a
green run: reverting the leaf root-election gate reds it in roughly a third of
runs, with the leaf reporting itself as tree root. That was the property the
suite was being kept for, and it is the property now demonstrated elsewhere.
Removed from the local runner's suite list, its runner function, the default
flow and the single-suite dispatch, and from the workflow matrix and its five
steps. Parity reports 22 legs a side, one fewer than before and equal across
the two runners. The suite's script, topology and compose profile stay on disk
and it remains runnable by hand; the retirement is recorded with the others in
the runner's deliberately-not-run block.
The rekey tests configured a one-second interval to put the timer arm within
reach, then backdated the session two minutes so the jitter draw could not
decide the outcome. That worked, but it asks for a configuration the daemon now
refuses to build: an interval at or below the jitter bound saturates to zero on
a negative draw and rekeys on sight for roughly half of sessions, which
validation rejects.
Thirty seconds clears the bound and leaves the existing backdate a margin of
seventy-five seconds against the worst draw, so the tests keep the property
they were relying on. The helper that drives these handshakes now records why
the value cannot simply be made smaller again.
Surfaced by merging the deployed line up, which is the first point at which
these tests met the new validation.
Clean merge, no conflicts. The hop-limit helper corrections and the rekey
config validation arrive from the deployed line and apply here unchanged.
Checked rather than assumed, since this branch changed the same subsystem in
the same session: next keeps its own split of the rekey cadence, where the
config flag rides into the core as RekeyCfg::initiate and check_rekey no longer
returns early, and the incoming validation of the same config block sits
alongside it without overlap.
Both maint changes carry over unchanged. The hop-limit helpers follow the
module rename from src/protocol/link.rs to src/proto/link.rs, applied by
rename detection and verified at the renamed path rather than assumed; the old
path is not recreated. The rekey config validation and its jitter-constant
import apply as written, the node module exposing the constant on this branch
as well.
The only conflict was the unreleased changelog, where master's own Changed
entries and the incoming hop-limit entry were both correct for their branch.
Resolved as the union, with the incoming entry appended to master's Changed
list.
The in-process mixed-profile test asserted convergence, peer degrees and every
direct pair, but deliberately left out the Leaf-to-Full pair that routes
through the Full node between them: a leaf holding a small enough address used
to self-elect as tree root and partition the mesh, so the assertion would have
been flaky. That defect is fixed, so assert the pair here, in both directions.
The leaf's peer degree is already pinned at one in the same test, so neither
direction can be satisfied by a direct link.
Failures in this test are root-election partitions keyed on the random
per-node address ordering, which are undiagnosable from a bare assertion
message. Each of the three failure paths now reports every node's address,
elected root and self-rooted flag.
Reverting the self-election gate reds the new pair in roughly a third of runs
with the leaf reporting itself as root, and the assertion is clean over
repeated runs with the gate in place.
Config validation said nothing about the rekey block, so two settings that
disable rekey in appearance and hammer it in practice were accepted silently.
after_messages of zero makes the message-count arm true on every poll, since
the trigger tests the counter against it with a greater-or-equal. It reads like
a way to switch the arm off and does the opposite.
after_secs at or below the per-session jitter is the same trap on the timer
arm. Each session offsets the interval by a random value drawn from plus or
minus the jitter bound, so a smaller interval saturates to zero on a negative
draw and rekeys on sight, for roughly half of sessions and not the other half.
The rule is expressed against the jitter constant rather than its current
value, so it tracks if the bound ever moves.
Both are checked whether or not rekey is enabled, so turning it on later cannot
surface a configuration error at a surprising moment. Neither has an upper
bound: a very large value is the established way to disable one arm of the
trigger and stays legal.
The forwarding path was corrected to full IP semantics: local delivery is not
hop-limit gated, and forwarding decrements first and drops at zero. Two helpers
on SessionDatagram were left implementing the old rule. decrement_ttl still
checked before decrementing, and can_forward still answered true at a hop limit
of one, where the decrement leaves zero and the datagram is dropped. Neither is
called anywhere today, which is precisely why they are worth correcting rather
than leaving as an invitation to reintroduce the defect.
The TtlExhausted reject doc described behaviour that no longer exists. The
reject is charged for a transit arrival at one as well as at zero, and is never
charged for a datagram addressed to this node, whose delivery is decided ahead
of the test.
Both helpers are pinned by tests at the boundary, which fail against the
previous implementations.
The responder half of the establish decision was gated on this node's own
node.rekey.enabled. With the rekey now declared in the msg3 negotiation
payload, that flag was the only thing that could divert a msg3 whose marker
matched the session we hold, and it diverted it to a msg2 resend while the
initiator had already installed its pending session and would cut over on its
own timer regardless. A pair configured with the flag true on one end and
false on the other therefore parted company at the initiator's cutover and
carried no traffic in either direction until the link-dead timer.
Drop the flag from that gate. The arm now turns on the sender's declaration
matching the session we hold, which is the only signal that was ever
authoritative for the decision, and the snapshot field it read is gone since
every initiator-side use reads the config directly.
Removing the gate exposed a second reading of the same flag. check_rekey
returned early when rekey was disabled, and the drain-expiry arm of the rekey
poll is the only thing that releases a demoted session and its index. A node
that does not initiate had never accepted a rekey before, so it never
accumulated a drain; now that it does, the early return would pin the previous
session and its allocator index forever. The flag rides into the core as
RekeyCfg::initiate instead: the polled cutover and the trigger are gated on it,
the drain is not, and the shell runs the cadence unconditionally.
The cutover arm has to stay gated. The peer snapshot carries no rekey role, so
a pending session stored as responder satisfies that arm exactly as one we
initiated does; ungating it would move a non-initiating node's cutover off the
peer's k-bit flip and onto the poll.
Covered by a two-ended test asserting the pair converges on the same session
indices under an asymmetric setting, a node-level test that a non-initiating
responder actually releases the demoted session and index once the drain window
expires, and a core test pinning which arms a non-initiating snapshot yields.
The existing msg2-resend test reached its arm only through the removed gate and
now reaches it through the health conjunct instead.
The doc comment merged up from the maintenance line lists the map's live
readers, and on this branch there is one more: the anonymous-beacon dial
dedup in poll_transport_discovery, which has no counterpart there.
It reads the map with an address supplied by transport discovery, and the
shared-media dial paths resolve before registering, so it compares a key
written in the same form it reads and is not exposed to the mismatch the
comment warns about.
Carries the address-to-link map's corrected doc comment. The only
conflict was the adjacent line naming the discovery config keys, which
this line renamed to node.lookup.* and the maintenance line did not; kept
this branch's names.
The doc said the map "enables dispatching incoming packets to the right
connection before authentication completes." It does not: find_link_by_addr
has no callers outside its own tests on any branch, and encrypted frames
dispatch by session index. Its live readers are the msg1 admission fast
path and the duplicate-inbound-handshake check.
The comment mattered because the map has a trap that invites exactly the
misuse it was advertising. An outbound dial registers the literal
configured address string, which may be a hostname, while an inbound
packet carries the resolved form; TransportAddr compares byte-wise, so
the two never match and a lookup keyed on an inbound address returns "no
such link" for every hostname-configured peer rather than failing. The
entry is also single-valued per key, so an inbound handshake overwrites
an outbound dial's entry for the same address.
Both current readers survive this because each compares a key written in
the same form it reads. That is a property of those two call sites and
not of the map, so the comment now says so, and says not to key a
peer-identity question on it.
Documentation only; no behaviour changes.
Carries the rekey counter-arm boundary test and the removal of the
closure-based trigger test. Applied cleanly with no adaptation: the
trigger predicate is byte-identical on both lines, and the claim the new
test rests on was re-measured here rather than assumed to carry — widening
the bound to `p.counter >= 1` reds exactly one test on this line too.
The send-counter arm of the rekey trigger had no test that could fail for
the reason it existed. The integration suites never reached it: they
inject after_messages = 65536 against a ping-driven workload, so every
rekey they observe is time-triggered. The unit test that looked like the
guard reproduced the trigger's OR predicate as a local closure and
asserted against its own copy, so deleting the arm from the real
predicate left it green.
That is the coverage hole that let the data-plane drop across a
message-count-triggered rekey survive: the defect is precisely a rekey of
a young session, which only the counter arm can produce.
Replaces the existing at-threshold check with a boundary test driving
poll_rekey, holding the time arm off so the counter is provably what
decides. The silence-below half is the part that is new, and it earns its
place by measurement: widening the bound to `p.counter >= 1` reds exactly
one test in the library suite, this one. Deletion, narrowing to `>` and
inversion are caught here too, though the time arm's negative tests
already catch a bound removed altogether.
Removes the closure-based test rather than repairing it. Its one
assertion over real code, that a fresh session entry's jitter lies within
the symmetric bound, is already covered over 100 samples elsewhere.
Note this leaves the deployed maintenance line untested on the same arm:
its trigger is inline in an async method that reads the clock directly,
with no pure function to drive, so the same test cannot be written there
without the sans-IO seam this line already has.
Carries the in-process rekey continuity test and the retirement of the
three rekey Docker suites onto the XX line.
The continuity test was written against master's IK handshake, where the
responder decided rekey-versus-fresh-dial from session age and the test
had to backdate both sessions past a 30s acceptance gate. Here the
responder classifies from the rekey marker the initiator declares in its
msg3 negotiation TLV and reads no clock at all, so the backdating is
dropped: it would age the session past nothing, and leaving it in would
tell a reader a gate exists that does not.
Checked that the test still discriminates rather than merely observing a
session index change. Suppressing the declaration so the responder falls
back to the cross-connection path reds the post-cutover assertion in 8 of
16 runs, which is the address-ordering coin flip the original data-plane
drop had; the healthy tree is 0 of 16 red. The 30s backdate makes no
difference to either rate, which is what confirms it was vestigial here.
The suite wiring keeps mixed-profile, which master does not have, and
drops the three rekey suites from the local suite list, the runner
functions, the default flow, the --only dispatch and the GitHub matrix
together, so check-ci-parity.sh stays green at 23 legs a side. The
retirement note is retargeted at this branch's coverage: the establish
decision tests live in src/proto/fmp/tests/core.rs and src/peer/machine.rs
here, not in the establish_chartests.rs file master keeps.
The rekey marker's cross-connection arm declined to swap while a rekey of
ours was in flight, reasoning that the swap rewrites the session and both
indices while touching no rekey state, so it would orphan that rekey. The
reasoning was one-sided and the arm was wrong: this is half of a two-sided
tie-break whose other half reads only whether a peer exists and the address
ordering, and cannot see our rekey. Declining here while the peer's outbound
half swaps anyway leaves the two ends on four distinct indices with neither
holding a pending session, dead both directions until the link-dead timer.
Measured against the pre-marker tree, which converges: it holds the crossing
dial as pending and adopts it at cutover.
The orphaned-rekey hazard is real but belongs to the executor. The swap now
abandons the rekey it displaces and frees that index first, exactly as the
rekey-responder path already does when it loses the dual-rekey tie-break, so
the classifier can take the arm unconditionally.
Two further failure paths of the marker are tightened. A malformed marker
tore the handshake down without freeing the index msg1 allocated, and that
arm sits ahead of the ACL gate, so any peer able to complete a msg3 could
grow the allocator set one entry at a time. A rekey with no session index to
declare silently omitted the marker, which puts a real rekey back on the
cross-connection path; it is unreachable today and is now asserted rather
than skipped.
On the test side, the marker's wire contract gains coverage: the accessor
round-trips, an unknown TLV field leaves it alone in both directions, and a
well-formed marker of the wrong length is an error rather than an absence,
which is the case the container codec cannot see and the whole reason the
accessor is fallible.
Two node-level tests now assert both ends rather than one. A rekey fired by
message count on a zero-age session is the mechanism behind the datagram
drops, and nothing exercised it: it fails on the pre-marker tree and passes
here. The mutual-dial test gains an index-pair assertion, since every
per-end predicate it checked holds when the ends have diverged.
One existing test had stopped reaching its own arm. Driving a bare second
handshake into a rekey-disabled peer now lands on the cross-connection arm,
which that flag does not gate, and every assertion held there too; it is
retargeted onto a declared rekey, which is the real route to the duplicate
arm, and asserts the session is left alone so the two arms are told apart.
Documentation that described the removed session-age floor is rewritten
around the declaration.
A rekey and a fresh dial both arrive as a new msg1 on a new link, so the
responder has to tell them apart at msg3. It cannot: the distinguishing
fact is internal to the initiator and nothing on the wire carries it.
Session age stood in for it and could not do the job - a
message-count-triggered rekey fires on a young session, so a real rekey
landed below the floor, was resolved as a cross-connection, and left the
two ends of the link on different session indices.
Carry the answer instead. The initiator adds a negotiation TLV to its
rekey msg3 naming the session it replaces, using the index the receiver
allocated so the receiver can match it against its own. The shell
resolves that into a three-valued claim - none, matching, mismatched -
and the classifier branches on it.
A mismatched marker resends msg2 rather than rejecting: the initiator has
already installed its pending session and will cut over on its own timer,
and the reject path sends nothing back, so tearing down would strand the
link in the way this defect already does. A fresh dial arriving while our
own rekey is in flight also resends, because the cross-connection swap
rewrites both indices and touches nothing else, which would leave that
rekey pointing at a session the counterpart no longer holds. The age
floor was excluding that case incidentally.
The floor and the session-age snapshot field are gone with it, so the
establish snapshot no longer reads the clock at all.
Both node-level rekey tests now drive a real rekey through the
initiator's own trigger. They previously reached the rekey arms by
backdating a session and re-handshaking, which tested the age proxy this
replaces; one of them was reaching its assertions through the duplicate
arm rather than the arm it names. Age them well past the trigger, since
the jitter is +/-15s and a smaller margin makes firing depend on the draw.
Incomplete: no TLV round-trip test, no two-ended convergence test, no
rekey-crossed-with-fresh-dial case, and no integration run yet.
The static suites hardcoded RUST_LOG=info, so raising the log level for a
node under test meant editing the compose file. Take it from the
environment with info as the default, matching the passthrough the same
block already uses for the worker-pool overrides.
Rendering is unchanged when RUST_LOG is unset, which is how CI invokes it.
A leaf-profile node that held the smallest NodeAddr would self-elect as
tree root, but its peers refuse a non-full node as a parent, so it formed
an isolated second root and partitioned the mesh: a multi-hop session from
the leaf to a non-adjacent full node then failed because the far node could
not route a handshake reply back into the leaf's separate coordinate tree.
Gate tree-state self-election on a new self_is_leaf flag, set from the node
profile at construction. A leaf now attaches under its full upstream and
holds that subtree's coordinate for its own routing. That coordinate's
self < root would be rejected by the root-min wire check, but a leaf never
announces it; it reaches peers only via the coordinates carried on the
leaf's session frames, and the upstream already advertises the leaf in its
bloom filter, so the far node routes back through the upstream.
Also fix Node::with_identity to derive the node profile, leaf-only flag,
and bloom state from the config, matching Node::new; it previously
hardcoded the full profile and silently dropped a configured leaf or
non-routing profile.
Covered by sans-IO decision tests (leaf does not self-elect; leaf keeps its
coordinate under a larger-rooted parent) and an end-to-end multi-hop
regression, each confirmed to fail if the gate is reverted.
Brings up a Full/Full/NonRouting/Leaf mesh matching the mixed-profile
Docker topology, asserts each node peers to the degree its role and
topology dictate (A=3, B=2, C=2, D=1), and verifies end-to-end data
between every directly reachable pair (Full-Full, Full-NonRouting,
Full-Leaf). Adds run_tree_test_with_profiles, a per-node-profile sibling
of run_tree_test.
The Leaf-to-Full multi-hop the Docker suite also checks is deliberately
left out for now: a Leaf node's multi-hop session initiation on the XX
line is not yet reliable, so the Docker mixed-profile suite is retained
for that path rather than replaced with a flaky assertion.
The three rekey integration suites (rekey, rekey-accept-off,
rekey-outbound-only) are dropped from both runners. Their coverage now
lives in fast, deterministic in-process tests:
- rekey timing and choreography (trigger, K-bit cutover, drain, jitter,
guards) in the sans-IO poll_rekey tests (proto/fsp, proto/fmp);
- data-plane continuity across a real cutover in
rekey_cutover_preserves_data_plane (a real IK rekey over loopback);
- the accept-off dual-init regression and the udp.outbound_only rekey
loop in the should_admit_msg1 and dual-init characterization tests.
Drop them from both runners in lockstep so the parity guard stays green,
and record the retirement in the deliberately-not-run block with a pointer
to the standalone runner. The rekey-test.sh script stays on disk.
Drives a real IK rekey handshake to K-bit cutover between two loopback
nodes and asserts an encrypted datagram sent after the cutover decodes on
the new session, with no spurious peer teardown. The rekey is forced
deterministically: rekey.after_messages = 1 crosses the initiator's
trigger on the first sent datagram, and both sessions are backdated past
the responder's 30s rekey-acceptance gate, so no wall clock is read.
The cutover is proven to actually occur (the live session index changes),
which is what makes the continuity assertion meaningful rather than
vacuous. Adds a make_test_node_with_config loopback helper for setting the
rekey thresholds. The rekey timing and choreography decisions themselves
are already covered exhaustively by the sans-IO poll_rekey tests.
Johnathan Corgan