Johnathan Corgan
9f8b266e5d
The decrypt worker's module doc and its fallback-path comment framed the code as a step in an architectural plan this repository does not carry, and named a "DataShard end-state" that exists nowhere in the tree. State the property directly instead: each worker owns its session state, and a worker that also owned the rx_loop side could restore the EndpointData fast path. The reworded comments are rewrapped to the surrounding width. The metrics registry docs described off-thread reads as a later step, but the control read handle already serves show_metrics from the registry off the rx_loop task. Put both the module doc and the Padded doc in the present tense, with the single writer the code has. Three source comments cited commits by hash (an abandoned republish design, the change that made sessions shard-owned, and the fix that moved encrypt dispatch off round-robin). None of those hashes resolves in this repository, so a reader cannot follow them. Say what the commit did instead. The one hash cited in the routing tests is on every branch and stays. The bloom-storm scenario README pointed readers at a reproduction harness and results file outside this repository, told them to check out a commit and a branch that are not here, and named the regressing commit, which is not here either. Describe the harness and the regressed build without those references and anchor the confirmation procedure on the fix commit, which every branch contains. The off-loop snapshot dispatch test's name carried a stage label from an outside plan. Rename it to say what it checks: show_acl, show_stats_peers and show_stats_history_all_peers are served off-loop through snapshot_dispatch and render byte-identically to their on-loop forms. Nothing refers to the old name.
6.7 KiB
bloom-storm — bloom announce storm regression scenario
Six-node depth-4 mesh with an induced upstream parent flap. Asserts a
trailing-window ceiling on per-node stats.bloom.sent to catch a
regression class where a localized spanning-tree update at an
internal mid-chain edge fails to be properly contained and instead
propagates as a bloom announce storm down to the leaves.
Topology
n01 (root, depth 0)
/ \
n02 n03 (parent candidates pa/pb, depth 1)
\ /
n04 (flap, depth 2 — induced parent flap)
|
n05 (leaf, depth 3)
|
n06 (tail, depth 4)
n01 is expected to win the root election under the deterministic key
derivation used by the chaos runner; the run's tree snapshots
(tree-snapshot-warmup.json, tree-snapshot-final.json) record the
actual assignment.
Bug class guarded against
A spanning-tree update that changes only an internal path edge — no
root change, no depth change — must not produce a sustained bloom
announce storm at downstream nodes. The original regression
(since rolled back, and fixed in master 4cdf382) had this property:
in the field, a single mid-chain ancestor swap on an upstream node
caused every downstream node in its subtree to issue a bloom
announce on every parent re-evaluation tick of the upstream node,
resulting in a ~480x mesh-wide elevation in FilterAnnounce traffic
and a perfect bimodal flip on est_entries.
Mechanism
The new link_swap chaos primitive deterministically alternates the
netem delay on n02-n04 (5ms vs 100ms) and n03-n04 (100ms vs 5ms)
every 4 seconds. Combined with the FIPS overrides
(parent_hysteresis: 0.0, reeval_interval_secs: 1,
flap_threshold: 9999, hold_down_secs: 0), this forces n04 to
switch parents on every swap. The whole point of the scenario is to
force sustained parent flapping at n04 and assert that the bloom
layer doesn't amplify it.
Assertions
assertions:
bloom_send_rate:
window_secs: 30
max_per_node: 40
min_parent_switches:
min_total: 10
bloom_send_rate is the load-bearing assertion: per-node delta of
stats.bloom.sent over the trailing 30s of the run must be at most
40. Per-node deltas and the offending node IDs are written to
assertions.txt and the runner exits 3 on failure.
min_parent_switches is a sanity guard. It fails if the run did not
record at least 10 parent switches across all nodes, which would
mean the harness fired its link swaps but the topology never produced
a real parent-switch event (e.g., wrong root election made the flap
target's parent candidates structurally non-equivalent). Without this
guard, the bloom-rate assertion would trivially pass on any binary,
including a regressed one.
Threshold derivation
The original reproduction harness, kept outside this repository, measured (90s flap window, ~21 induced parent switches at the mid-chain node):
| binary | tail bloom_sent / 90s | rate scaled to 30s |
|---|---|---|
| pre-fix | 21 | ~7 |
| current fix | 0 | 0 |
Observed on this scenario at master db5b6b1 (180s run, 35 parent
switches, 41 link swaps), per-node bloom_sent deltas over the
trailing 30s:
n01=5 n02=5 n03=4 n04=12 n05=6 n06=0
n04 (the flapping node) is the highest because it is legitimately
re-sending its filter on its own parent changes. n06 (the depth-4
"tail") sees 0, matching the calm post-fix behavior that harness recorded
for its fix2 variant.
In the field, the regression's mesh-wide rate scaled ~480x above
steady state. A 30 / 30s / node ceiling sits ~2.5x above the
observed maximum on fixed master and well below the
deployment-scale storm rate, giving headroom for harness jitter
without losing the ability to fail loud on the regression class.
If link_swap.interval_secs or the netem delta is changed,
recalibrate. The threshold is calibrated against the values in
scenarios/bloom-storm.yaml as committed and the seed: 31 pin.
Lab-data ceiling re-tune (2026-05-24)
The ceiling was bumped from 30 to 40 after a 59-rep characterization run
under the github-runner-equivalent pressure profile with per-container
CPU pinning to cpuset=0,1 (mimicking a 2-core ubuntu-latest GitHub
runner). Combined pinned distribution on n04 (the structural max-spike
node — flap target at depth 2):
| metric | value |
|---|---|
| mean | 24.4 |
| sd | 4.7 |
| P90 | 28 |
| P95 | 29 |
| P99 | 29 |
| max | 30 |
The lab's structural ceiling at 30 corresponds to the bloom-advertise
rate-limit token bucket's steady-state cap of ~1 send per second over the
trailing 30 s assertion window. GHA fires at n04=34 represent transient
release of queued bloom-sends during flap-recovery windows and do not
reproduce on this lab host even with CPU-pinning sidecar (cpuset=0,1)
applied to every chaos-spawned container.
Rationale for ceiling = 40: lab max 30 + ~2σ headroom (≈ 39.4) rounds to 40, giving 33 % margin over the observed lab maximum while still firing loud on a regression-class storm (the original regression scaled mesh-wide bloom traffic ~480× above steady state, far above any plausible jitter band).
Limitations
-
The bloom-storm regression has not been confirmed-failing here on a regressed binary in this harness directly; the threshold is inferred from the values measured in the dedicated post-mortem harness against a regressed build whose commit is not in this repository. To gain that confirmation, build a binary that carries the regression and predates the
4cdf382fix, copy binaries intotesting/docker/, and rerun this scenario; the bloom-rate assertion is expected to fail loud with n05/n06 deltas well above 40. -
Root-election outcome is sensitive to the seed (smallest
NodeAddrwins, whereNodeAddr = SHA-256(pubkey)[..16]). The seed value31is pinned for this reason. Themin_parent_switchesassertion catches drift if the seed is changed without re-validating the topology.
Running locally
# From the source repo root, with binaries already built and copied
# into testing/docker/ (see testing/scripts/build.sh).
./testing/chaos/scripts/chaos.sh bloom-storm
Run output is in testing/chaos/sim-results/<timestamp>-bloom-storm/.
Key artifacts:
analysis.txt— log analysis (panics, errors, parent switches).assertions.txt— per-assertion pass/fail with per-node deltas.tree-snapshot-warmup.json,tree-snapshot-final.json— control socket tree state at warmup end and at run end.runner.log— full orchestration log.
Total runtime: ~3.5 minutes (25s warmup + 180s scenario + ~30s teardown).