laantungir/fips
1
0
Fork 0
mirror of https://github.com/jmcorgan/fips.git synced 2026-08-09 08:14:42 +00:00
Code Issues Packages Projects Releases Wiki Activity

Merge branch 'master' into next

Browse Source 
Forward-merge of 12 master commits past the previous merge
(823b830, master @ 18019bb): dep-audit bumps (rand,
clap, tun, rtnetlink, windows-service, plus the bump-safe
lockfile batch), bloom-storm chaos scenario, control-socket
resolver consolidation, gateway dns.listen default change,
OpenWrt ipk README refresh, gateway tutorial review, Ethernet
MTU rustdoc fix, dead session-variant drop, CHANGELOG prep.

Conflict resolution:

- CHANGELOG.md: both bullets kept under [Unreleased] / Fixed.
  Master's spanning-tree internal-path-propagation fix precedes
  next's tree-ancestry-test determinism entry and the
  responder-Disconnect XX-handshake entry.
- src/protocol/session.rs: kept next's SessionSetup/SessionAck
  variants and rustdoc. Master's drop of those variants suits
  v0.3.0's FSP phase-byte dispatch but is undone by next's
  v0.4.0 wire format, which retains the variants and uses the
  inner msg_type byte for handshake identification.
- src/transport/ethernet/mod.rs: kept next's "interface MTU - 4"
  comment. Master corrected the v0.3.0 3-byte rustdoc; next
  redesigned the framing to a 4-byte header (type/flags/length)
  for shared-media beacons, so master's correction does not
  apply to next's format.

Auto-merged cleanly: Cargo.toml (next's 0.4.0-dev + the new dep
pins from master), Cargo.lock, all gateway docs,
docs/reference/configuration.md, packaging files,
.github/workflows/ci.yml, testing/chaos/sim/* and
testing/ci-local.sh (bloom-storm additions), src/config/*.

Local verification: cargo build --release, cargo test (1252
passed, 4 ignored), cargo clippy -D warnings, cargo fmt --check
all green.
This commit is contained in:
Johnathan Corgan
2026-05-08 18:47:13 +00:00
parent 823b830289 b3a1fb464f
commit 45a8a4cd9b
26 changed files with 1518 additions and 434 deletions
Download Patch File Download Diff File Expand all files Collapse all files
.github/workflows/ci.yml
+3
View File
@@ -366,6 +366,9 @@ jobs:
- suite: congestion-stress
type: chaos
scenario: congestion-stress
- suite: bloom-storm
type: chaos
scenario: bloom-storm
# ── Sidecar deployment ──────────────────────────────────────────
- suite: sidecar
type: sidecar
CHANGELOG.md
+41
View File
@@ -379,9 +379,45 @@ with v0.2.x peers.
`max_fpr` and returns `Option<f64>`, returning `None` for
saturated filters; this propagates through `compute_mesh_size`
into `estimated_mesh_size` (already `Option<u64>`)
- The `docs/` tree is reorganised so readers can find content by
what they're trying to do: tutorials for new users, how-to guides
for specific tasks, reference material for configuration and
protocol details, and design discussion for architectural
background. New top-level `getting-started.md` and per-section
landing pages anchor the entry points. Content was reconciled
against current source: protocol layer details, wire-format
diagrams, configuration knobs, and CLI references were brought
back into agreement with the implementation. Gateway feature-set
documentation was rewritten end-to-end.
- Test coverage was substantially expanded for the new release
surface (discovery state machine, control-socket query handlers,
decrypt-failure thresholds, STUN parser, gateway, NAT traversal,
packaging install paths) alongside CI-side hardening for the new
Windows and macOS platforms.
- Gateway `dns.listen` source default changed from `[::]:53` to
`[::1]:5353` to match the canonical deployment model (a host
already serving DHCP/DNS to a LAN segment, where port 53 is
taken by the existing resolver and `.fips` queries are forwarded
to the gateway over loopback). The OpenWrt ipk previously
overrode this in its packaged config; the override is now
redundant and has been dropped. Operators on a host without a
pre-existing resolver on port 53 can opt back into the wildcard
bind by setting `dns.listen: "[::]:53"` explicitly. The new
default binds IPv6 loopback only — forwarders that reach the
gateway over IPv4 loopback need an explicit IPv4 listen address.
### Fixed
- Default control-socket path resolution: daemon and client tools now
use a shared resolver, eliminating a divergence where `fipsctl` /
`fipstop` could connect to a socket the daemon never bound (notably
on dev runs with `XDG_RUNTIME_DIR` set, or after a prior packaged
install left a root-owned `/run/fips` behind). Canonical order is
`/run/fips` → `$XDG_RUNTIME_DIR/fips/` → `/tmp/fips-<name>`, with
writability of `/run/fips` probed via tempfile create (ACL- and
group-aware) and `XDG_RUNTIME_DIR` validated as an existing
directory before being used. The deployed fleet is unaffected:
packaged configs set `node.control.socket_path` explicitly.
- UDP transport with `advertise_on_nostr: true` + `public: true` +
a wildcard `bind_addr` (e.g. `0.0.0.0:2121`) is now advertised
with its STUN-discovered public IPv4 instead of being silently
@@ -532,6 +568,11 @@ with v0.2.x peers.
tree specification. The receive path now verifies that the
ancestry is structurally consistent with the signed parent
declaration before mutating tree state.
- Spanning-tree updates that change only the internal path between
root and leaf — without changing the root or the depth — now
propagate to leaves correctly. Previously a leaf could continue
routing against a stale internal path until the parent or depth
also changed.
- Make the tree ancestry acceptance unit test deterministic.
`test_tree_announce_validate_semantics_accepts_valid_non_root`
generated a random signing identity while pinning the fixed root
Cargo.lock
Generated
+286 -340
View File
File diff suppressed because it is too large Load Diff
Cargo.toml
+5 -5
View File
@@ -14,7 +14,7 @@ secp256k1 = { version = "0.30", features = ["rand", "global-context"] }
sha2 = "0.10"
hkdf = "0.12"
chacha20poly1305 = "0.10"
rand = "0.10.0"
rand = "0.10.1"
thiserror = "2.0"
bech32 = "0.11"
serde = { version = "1.0", features = ["derive"] }
@@ -22,7 +22,7 @@ serde_json = "1.0"
serde_yaml = "0.9"
dirs = "6.0"
hex = "0.4"
clap = { version = "4.5", features = ["derive"] }
clap = { version = "4.6", features = ["derive"] }
tracing = "0.1"
tracing-subscriber = { version = "0.3", features = ["env-filter"] }
tokio = { version = "1", features = ["rt", "macros", "signal", "sync", "net", "time", "process", "io-util"] }
@@ -36,11 +36,11 @@ nostr = { version = "0.44", features = ["std", "nip59"] }
nostr-sdk = "0.44"
[target.'cfg(unix)'.dependencies]
tun = { version = "0.8.5", features = ["async"] }
tun = { version = "0.8.7", features = ["async"] }
libc = "0.2"
[target.'cfg(target_os = "linux")'.dependencies]
rtnetlink = "0.20.0"
rtnetlink = "0.21.0"
rustables = "0.8.7"
# bluer/BlueZ needs glibc — see build.rs `bluer_available` cfg gate.
@@ -49,7 +49,7 @@ bluer = { version = "0.17", features = ["bluetoothd", "l2cap"] }
[target.'cfg(windows)'.dependencies]
wintun = "0.5"
windows-service = "0.7"
windows-service = "0.8.1"
[package.metadata.deb]
maintainer = "Johnathan Corgan <jcorgan@corganlabs.com>"
docs/design/fips-gateway.md
+10 -5
View File
@@ -130,7 +130,7 @@ There is no `fipsctl gateway` subcommand; clients (including
│ │
│ ┌──────────────┐ ┌───────────┐ │
│ │ DNS proxy │ │ Virtual │ │
│ │ ([::]:53) │─▶│ IP pool │ │
│ │ ([::1]:5353) │─▶│ IP pool │ │
│ │ .fips only │ │ (state │ │
│ └──────┬───────┘ │ machine) │ │
│ │ └─────┬─────┘ │
@@ -182,13 +182,18 @@ involving the DNS proxy or the pool.
### DNS Resolution Flow
1. A LAN client sends a DNS query to the gateway's listener (default
`[::]:53`, configurable via `gateway.dns.listen`).
`[::1]:5353`, configurable via `gateway.dns.listen`). The default
is loopback-only on an unprivileged port: the canonical deployment
has another resolver on the host (dnsmasq, systemd-resolved, BIND)
holding port 53 and forwarding `.fips` queries to the gateway over
loopback. Operators on a host without a pre-existing resolver on
53 can override the listen value to `"[::]:53"` to let LAN clients
query the gateway directly.
2. If the question is not for a `.fips` domain, the gateway replies
`REFUSED`. The proxy is intentionally narrow — it does not resolve
public DNS, and the LAN's primary resolver should hold port 53 on
the gateway host (the OpenWrt init script wires this up by binding
the gateway listener to a non-conflicting port and configuring
dnsmasq to forward `.fips` queries there).
the gateway host (the OpenWrt init script wires dnsmasq to forward
`.fips` queries to the loopback listener automatically).
3. The gateway forwards the query to the daemon resolver
(`gateway.dns.upstream`, default `[::1]:5354`). The daemon must
match: an IPv6 socket bound to `[::1]` does not accept v4-mapped
docs/how-to/deploy-gateway.md
+17 -9
View File
@@ -142,7 +142,7 @@ pick a different `fdXX::/N`). The `/112` size yields 65 536 virtual
IPs, which is the gateway's hard cap regardless of CIDR width.
This minimum config is enough to start the gateway. The `dns.*` block
is optional and defaults to `listen: "[::]:53"` and
is optional and defaults to `listen: "[::1]:5353"` and
`upstream: "[::1]:5354"`. The full block — including `dns.*`,
`pool_grace_period`, `conntrack.*`, and `port_forwards[]` — is
documented in
@@ -195,20 +195,28 @@ Constraints:
```yaml
gateway:
dns:
listen: "[::]:53"
listen: "[::1]:5353"
upstream: "[::1]:5354"
ttl: 60
```
Common cases:
- **No other resolver on the host:** `listen: "[::]:53"` is the
default and works.
- **systemd-resolved is on port 53:** either disable its stub
listener (`DNSStubListener=no` in
`/etc/systemd/resolved.conf`) or move the gateway to a different
port (e.g., `[::]:5353`) and put a forwarder on 53 that delegates
`.fips` to the gateway. See
- **Another resolver on the host (the canonical case):** the default
`listen: "[::1]:5353"` is loopback-only on an unprivileged port,
so it never conflicts with dnsmasq, systemd-resolved, or BIND
holding 53. Configure the existing resolver to forward `.fips`
queries to `[::1]:5353` and you are done — this is what the
OpenWrt ipk does automatically.
- **No other resolver on the host:** set `listen: "[::]:53"`
explicitly and LAN clients can query the gateway directly.
- **systemd-resolved is on port 53:** the default already side-steps
this — leave the listen address at `[::1]:5353` and configure the
stub or a small forwarder to delegate `.fips` to the gateway. If
you would rather have the gateway on 53 directly, disable the
systemd stub listener (`DNSStubListener=no` in
`/etc/systemd/resolved.conf`) and switch `listen` to `"[::]:53"`.
See
[troubleshoot-gateway.md](troubleshoot-gateway.md#port-conflict-on-the-dns-listen-port).
- **Bind on the LAN address only:** `listen: "192.168.1.1:53"`
exposes the resolver only to LAN clients, not loopback.
docs/how-to/troubleshoot-gateway.md
+11 -16
View File
@@ -82,10 +82,13 @@ for the full flag list.
### Port conflict on the DNS listen port
Symptom: gateway fails to start with "address already in use" on
port 53 (or whatever `gateway.dns.listen` is set to).
the configured `gateway.dns.listen` address.
Another DNS server (systemd-resolved, dnsmasq, BIND) is bound to
the port. Identify it:
The default `[::1]:5353` is loopback-only on an unprivileged port and
should not collide with any standard resolver. If you have overridden
`dns.listen` to bind port 53 (or a LAN-side address) and another DNS
server (systemd-resolved, dnsmasq, BIND) is already bound there,
identify it:
```sh
sudo ss -tulnp | grep ':53'
@@ -93,18 +96,10 @@ sudo ss -tulnp | grep ':53'
Two options:
- **Use an alternate listen address.** Pick a non-conflicting port
and update the gateway config:
```yaml
gateway:
dns:
listen: "192.168.1.1:5353"
```
Then either point LAN clients at the alternate port directly, or
run a forwarding stub on port 53 that delegates `.fips` queries to
the gateway.
- **Stay on the loopback default.** Drop the override and let the
gateway use `[::1]:5353`. Configure the existing resolver to
forward `.fips` queries to it (the canonical OpenWrt deployment
works this way out of the box).
- **Relocate the conflicting resolver.** Move it to a different port
(or disable it if not needed) and let the gateway bind 53.
@@ -189,7 +184,7 @@ not running or not enabled. Check that the daemon config has
**Step 2.** Verify the gateway is listening on its DNS port:
```sh
sudo ss -tulnp | grep -E ':53\b'
sudo ss -tulnp | grep -E ':(53|5353)\b'
```
If nothing is listening on the configured `dns.listen` address, the
docs/reference/configuration.md
+2 -2
View File
@@ -700,7 +700,7 @@ Non-`.fips` queries are answered with `REFUSED`.
| Parameter | Type | Default | Description |
|-----------|------|---------|-------------|
| `gateway.dns.listen` | string | `"[::]:53"` | LAN-facing DNS listen address. Bind on the LAN-side IP (e.g., `"192.168.1.1:53"`) or on all interfaces (`"[::]:53"`) for LAN clients to query. Bind to a non-53 port if another resolver already owns 53 on the host (see [../how-to/troubleshoot-gateway.md](../how-to/troubleshoot-gateway.md)). |
| `gateway.dns.listen` | string | `"[::1]:5353"` | DNS listen address. The default binds IPv6 loopback on an unprivileged port, matching the canonical deployment where another resolver on the host (dnsmasq, systemd-resolved, BIND) holds port 53 and forwards `.fips` queries to the gateway over loopback. Bind on the LAN-side IP (e.g., `"192.168.1.1:53"`) or wildcard (`"[::]:53"`) only on hosts with no other resolver on 53 and where LAN clients query the gateway directly. See [../how-to/troubleshoot-gateway.md](../how-to/troubleshoot-gateway.md). |
| `gateway.dns.upstream` | string | `"[::1]:5354"` | Upstream FIPS daemon resolver. **Must match the daemon's `dns.bind_addr` and `dns.port`.** Defaults match the daemon defaults (`::1:5354`). A v4 upstream (`"127.0.0.1:5354"`) cannot reach a daemon bound on `[::1]:5354` — Linux IPv6 sockets bound to explicit `::1` do not accept v4-mapped traffic. If you change the daemon's `dns.bind_addr`, update this field accordingly. |
| `gateway.dns.ttl` | u32 | `60` | TTL in seconds on AAAA responses returned to LAN clients. Smaller values let the gateway recycle pool addresses faster; larger values reduce LAN-side query traffic. |
@@ -747,7 +747,7 @@ gateway:
pool: "fd01::/112"
lan_interface: "enp3s0"
dns:
listen: "[::]:53"
listen: "[::1]:5353"
upstream: "[::1]:5354"
ttl: 60
pool_grace_period: 60
docs/tutorials/deploy-fips-gateway.md
+21 -9
View File
@@ -132,7 +132,6 @@ gateway:
pool: "fd01::/112" # virtual IP range (up to 65535 addresses)
lan_interface: "br-lan" # LAN-facing interface for proxy NDP
dns:
listen: "[::1]:5353" # gateway DNS listener (dnsmasq forwards here)
upstream: "[::1]:5354" # FIPS daemon DNS resolver (matches daemon default)
ttl: 60 # DNS TTL and mapping lifetime (seconds)
pool_grace_period: 60 # seconds after last session before reclaiming
@@ -147,10 +146,11 @@ Three things to notice:
- `lan_interface: "br-lan"` — the OpenWrt LAN bridge. The gateway
installs proxy-NDP entries on this interface so LAN clients can
ARP-equivalent for pool addresses.
- `dns.listen: "[::1]:5353"` — the gateway's DNS listener is bound
to IPv6 loopback only. dnsmasq, which owns LAN port 53, forwards
`.fips` queries to it. The init script wires up that forwarding;
you don't bind to a LAN address yourself.
- No `dns.listen` line — the source default `[::1]:5353` is exactly
what OpenWrt wants. The gateway listens on IPv6 loopback only;
dnsmasq, which owns LAN port 53, forwards `.fips` queries to it.
The init script wires up that forwarding; you don't bind to a LAN
address yourself.
For the full reference, see
[../reference/configuration.md § Gateway](../reference/configuration.md#gateway-gateway).
@@ -207,8 +207,21 @@ table install, and pool initialisation.
## Step 4: Test the outbound half from a LAN client
From a phone or laptop on the AP's LAN — anything that does IPv6 and
DNS, with no FIPS software installed — try one of the public test
Before bringing a LAN client into the picture, confirm from the AP
itself that the mesh side is still healthy after the gateway start:
```sh
ping6 -c 2 test-us01.fips
```
This isolates the router-to-mesh path before involving the LAN
segment. If this fails, the troubleshooting target is the daemon /
mesh side, not the gateway-to-client side. If it succeeds and the
LAN-client test below fails, the target is the LAN segment —
`proxy_ndp`, the RA pool route, or DNS forwarding through dnsmasq.
Now from a phone or laptop on the AP's LAN — anything that does IPv6
and DNS, with no FIPS software installed — try one of the public test
mesh nodes:
```sh
@@ -272,7 +285,7 @@ nft list table inet fips_gateway
You will see DNAT, SNAT, and masquerade chains populated with one
rule per active mapping.
## Step 6: Add an inbound port-forward for a LAN service
## Step 6 (Optional): Add an inbound port-forward for a LAN service
The outbound half is the steady-state use of a gateway. The inbound
half — exposing a LAN service to mesh peers — is a separate decision,
@@ -321,7 +334,6 @@ gateway:
pool: "fd01::/112"
lan_interface: "br-lan"
dns:
listen: "[::1]:5353"
upstream: "[::1]:5354"
ttl: 60
pool_grace_period: 60
packaging/common/fips.yaml
+1 -1
View File
@@ -94,7 +94,7 @@ transports:
# pool: "fd01::/112"
# lan_interface: "eth0"
# dns:
# listen: "[::]:53"
# listen: "[::1]:5353"
# # upstream must match the daemon's dns.bind_addr above. The
# # default "[::1]:5354" matches the daemon's default. If you set
# # the daemon to bind on a wildcard ("::") or specific address,
packaging/openwrt-ipk/README.md
+31 -7
View File
@@ -11,13 +11,16 @@ For ad-hoc deployment without the build system, see
| Installed path | Purpose |
|---|---|
| `/usr/bin/fips` | Mesh daemon |
| `/usr/bin/fipsctl` | CLI control tool (`fipsctl peers`, `fipsctl links`, …) |
| `/usr/bin/fipsctl` | CLI control tool (`fipsctl show peers`, `fipsctl show links`, …) |
| `/usr/bin/fipstop` | Live TUI dashboard |
| `/etc/init.d/fips` | procd service (auto-start, crash respawn) |
| `/usr/bin/fips-gateway` | Outbound LAN gateway service (not started by default) |
| `/etc/init.d/fips` | procd service for the daemon (auto-start, crash respawn) |
| `/etc/init.d/fips-gateway` | procd service for the gateway (disabled by default) |
| `/etc/fips/fips.yaml` | Node configuration (edit before first start) |
| `/etc/fips/firewall.sh` | Firewall helper — accepts traffic on `fips0` |
| `/etc/dnsmasq.d/fips.conf` | Forwards `.fips` DNS queries to the daemon |
| `/etc/sysctl.d/fips-bridge.conf` | `br_netfilter` settings for Ethernet transport |
| `/etc/sysctl.d/fips-gateway.conf` | `proxy_ndp` and IPv6 forwarding for the gateway |
| `/etc/hotplug.d/net/99-fips` | Applies firewall rules when `fips0` comes up |
| `/etc/uci-defaults/90-fips-setup` | First-boot kernel module and firewall setup |
| `/lib/upgrade/keep.d/fips` | Preserves `/etc/fips/` across `sysupgrade` |
@@ -142,17 +145,35 @@ physical interface names for your router. **Always use physical port names
/etc/init.d/fips disable
```
### Outbound LAN gateway (optional)
The `fips-gateway` service is installed but disabled by default. It
turns the router into an outbound gateway that bridges LAN clients
onto the FIPS mesh. Enable only after configuring a `gateway:`
section in `/etc/fips/fips.yaml`:
```bash
/etc/init.d/fips-gateway enable
/etc/init.d/fips-gateway start
```
See `docs/tutorials/deploy-fips-gateway.md` in the source tree for
the full walkthrough.
## Inspection and logs
```bash
# Peer table
fipsctl peers
# Node-level status overview
fipsctl show status
# Active sessions
fipsctl sessions
# Peer table
fipsctl show peers
# Transport links
fipsctl links
fipsctl show links
# Active end-to-end sessions
fipsctl show sessions
# Live TUI dashboard
fipstop
@@ -161,6 +182,9 @@ fipstop
logread | grep fips
```
See [`docs/reference/cli-fipsctl.md`](../../docs/reference/cli-fipsctl.md)
for the full subcommand list.
## Upgrading
Install the new `.ipk` over the existing one:
packaging/openwrt-ipk/files/etc/fips/fips.yaml
+3 -4
View File
@@ -76,16 +76,15 @@ peers: []
# Allows unmodified LAN hosts to reach FIPS mesh destinations via
# DNS-allocated virtual IPs and kernel nftables NAT.
#
# The gateway DNS listens on port 5353 by default so it does not conflict
# with dnsmasq on port 53. The init script configures dnsmasq to forward
# .fips queries to the gateway automatically.
# The gateway DNS listens on `[::1]:5353` by default (set in source) so it
# does not conflict with dnsmasq on port 53. The init script configures
# dnsmasq to forward .fips queries to the gateway automatically.
gateway:
enabled: true
pool: "fd01::/112" # virtual IP range (up to 65535 addresses)
lan_interface: "br-lan" # LAN-facing interface for proxy NDP
dns:
listen: "[::1]:5353" # gateway DNS listener (dnsmasq forwards here)
upstream: "[::1]:5354" # FIPS daemon DNS resolver (matches daemon default)
ttl: 60 # DNS TTL and mapping lifetime (seconds)
pool_grace_period: 60 # seconds after last session before reclaiming
src/config/gateway.rs
+17 -4
View File
@@ -8,7 +8,20 @@ use std::net::SocketAddrV6;
use serde::{Deserialize, Serialize};
/// Default gateway DNS listen address.
const DEFAULT_DNS_LISTEN: &str = "[::]:53";
///
/// Loopback-only on the unprivileged port 5353. The canonical
/// gateway deployment is a host already serving DHCP/DNS to a LAN
/// segment (e.g., an OpenWrt AP), where port 53 is taken by the
/// existing resolver and `.fips` queries are forwarded to the
/// gateway over loopback. Operators on a host without a pre-existing
/// resolver on 53 can opt back into the wildcard bind by setting
/// `dns.listen: "[::]:53"` explicitly.
///
/// `[::1]` is IPv6 loopback only; Linux IPv6 sockets bound to
/// explicit `::1` do not accept v4-mapped traffic. Forwarders that
/// reach the gateway over IPv4 loopback (`127.0.0.1`) need to be
/// pointed at an explicit IPv4 listen address instead.
const DEFAULT_DNS_LISTEN: &str = "[::1]:5353";
/// Default upstream DNS resolver (FIPS daemon).
///
@@ -118,7 +131,7 @@ pub struct PortForward {
/// Gateway DNS resolver configuration (`gateway.dns.*`).
#[derive(Debug, Clone, Default, Serialize, Deserialize)]
pub struct GatewayDnsConfig {
/// Listen address and port (default: `0.0.0.0:53`).
/// Listen address and port (default: `[::1]:5353`).
#[serde(default, skip_serializing_if = "Option::is_none")]
pub listen: Option<String>,
@@ -133,7 +146,7 @@ pub struct GatewayDnsConfig {
}
impl GatewayDnsConfig {
/// Get the listen address (default: `0.0.0.0:53`).
/// Get the listen address (default: `[::1]:5353`).
pub fn listen(&self) -> &str {
self.listen.as_deref().unwrap_or(DEFAULT_DNS_LISTEN)
}
@@ -205,7 +218,7 @@ lan_interface: "eth0"
assert!(!config.enabled);
assert_eq!(config.pool, "fd01::/112");
assert_eq!(config.lan_interface, "eth0");
assert_eq!(config.dns.listen(), "[::]:53");
assert_eq!(config.dns.listen(), "[::1]:5353");
assert_eq!(config.dns.upstream(), "[::1]:5354");
assert_eq!(config.dns.ttl(), 60);
assert_eq!(config.grace_period(), 60);
src/config/mod.rs
+186 -19
View File
@@ -90,21 +90,77 @@ pub fn pub_file_path(config_path: &Path) -> PathBuf {
.join(PUB_FILENAME)
}
/// Resolve a default Unix-socket path under the canonical order:
/// `/run/fips/<filename>` → `$XDG_RUNTIME_DIR/fips/<filename>` → `/tmp/fips-<filename>`.
///
/// `/run/fips` is the packaged convention (`root:fips 0770` directory created
/// by the daemon at bind time). `XDG_RUNTIME_DIR` covers non-root dev runs
/// where `/run/fips` does not exist or is not writable. `/tmp` is the
/// last-resort fallback.
///
/// Hardening notes:
/// - `/run/fips` is accepted only if the directory exists and is writable by
/// the current process. `create_dir_all` reporting `Ok(())` is *not*
/// sufficient: it returns `Ok` for an existing root-owned dir that we
/// cannot write to, which would silently steer a non-root daemon onto a
/// path that fails at bind time. Writability is probed via tempfile create
/// rather than mode bits so ACLs and group membership (the dir is
/// `root:fips 0770`) are honored.
/// - `XDG_RUNTIME_DIR` is validated as an existing directory before being
/// used; a stale post-logout value (after `pam_systemd` reaps the dir) is
/// treated as missing.
#[cfg(unix)]
pub(crate) fn resolve_default_socket(filename: &str) -> String {
// 1. /run/fips — accept only if the directory exists and is writable.
let run_fips = Path::new("/run/fips");
if run_fips.is_dir() && is_writable_dir(run_fips) {
return format!("/run/fips/{filename}");
}
// Also accept /run/fips if we can create it (covers the first-boot
// daemon-as-root case before the directory has been materialized). The
// actual chown happens at bind time.
if std::fs::create_dir_all(run_fips).is_ok() && is_writable_dir(run_fips) {
return format!("/run/fips/{filename}");
}
// 2. $XDG_RUNTIME_DIR/fips/ — only if the variable points at an existing
// directory.
if let Ok(xdg) = std::env::var("XDG_RUNTIME_DIR") {
let xdg_path = Path::new(&xdg);
if xdg_path.is_dir() {
return format!("{xdg}/fips/{filename}");
}
}
// 3. Last resort: /tmp with a name-mangled prefix so multiple users
// don't collide.
format!("/tmp/fips-{filename}")
}
#[cfg(unix)]
fn is_writable_dir(path: &Path) -> bool {
// Probe via tempfile creation rather than mode bits: mode-bit checks miss
// ACLs and group-membership effects (the /run/fips dir is `root:fips
// 0770` and the daemon may run as a user that's in the `fips` group).
let probe = path.join(format!(".fips-write-probe-{}", std::process::id()));
match std::fs::File::create(&probe) {
Ok(_) => {
let _ = std::fs::remove_file(&probe);
true
}
Err(_) => false,
}
}
/// Default control socket path for fipsctl / fipstop.
///
/// On Unix, checks the system-wide path first (used when the daemon runs as
/// a systemd service), then falls back to the user's XDG runtime directory.
/// On Windows, returns the default TCP port ("21210").
/// On Unix, delegates to [`resolve_default_socket`] for the canonical
/// `/run/fips` → `XDG_RUNTIME_DIR` → `/tmp` order. On Windows, returns the
/// default TCP port ("21210").
pub fn default_control_path() -> PathBuf {
#[cfg(unix)]
{
if Path::new("/run/fips").exists() {
PathBuf::from("/run/fips/control.sock")
} else if let Ok(runtime_dir) = std::env::var("XDG_RUNTIME_DIR") {
PathBuf::from(format!("{runtime_dir}/fips/control.sock"))
} else {
PathBuf::from("/tmp/fips-control.sock")
}
PathBuf::from(resolve_default_socket("control.sock"))
}
#[cfg(windows)]
{
@@ -114,18 +170,16 @@ pub fn default_control_path() -> PathBuf {
/// Default gateway control socket path.
///
/// On Unix, follows the same pattern as the main control socket.
/// On Windows, returns a placeholder TCP port ("21211").
/// On Unix, delegates to [`resolve_default_socket`] (same canonical order as
/// the main control socket). The gateway daemon itself uses a hardcoded
/// `/run/fips/gateway.sock` since gateway operation requires root for
/// NAT/conntrack management; this client-side resolver falls through
/// gracefully for non-root dev runs that need a gateway socket path. On
/// Windows, returns a placeholder TCP port ("21211").
pub fn default_gateway_path() -> PathBuf {
#[cfg(unix)]
{
if Path::new("/run/fips").exists() {
PathBuf::from("/run/fips/gateway.sock")
} else if let Ok(runtime_dir) = std::env::var("XDG_RUNTIME_DIR") {
PathBuf::from(format!("{runtime_dir}/fips/gateway.sock"))
} else {
PathBuf::from("/tmp/fips-gateway.sock")
}
PathBuf::from(resolve_default_socket("gateway.sock"))
}
#[cfg(windows)]
{
@@ -1466,4 +1520,117 @@ peers:
let cfg = UdpConfig::default();
assert!(cfg.accept_connections());
}
/// Mutex serializing tests that mutate `XDG_RUNTIME_DIR`. `cargo test`
/// runs tests on multiple threads in the same process, and env mutation
/// is process-global, so concurrent env-touching tests would race.
#[cfg(unix)]
static ENV_MUTEX: std::sync::Mutex<()> = std::sync::Mutex::new(());
#[cfg(unix)]
#[test]
fn test_resolve_default_socket_call_sites_agree() {
// The three resolver call sites must all produce strings that agree
// on the directory, differing only in the filename suffix.
let _g = ENV_MUTEX.lock().unwrap();
let control_client = default_control_path().to_string_lossy().into_owned();
let gateway_client = default_gateway_path().to_string_lossy().into_owned();
let control_daemon = ControlConfig::default().socket_path;
// Daemon-side and client-side control paths must be identical.
assert_eq!(
control_daemon, control_client,
"daemon and client default control-socket paths diverged: \
daemon={control_daemon}, client={control_client}"
);
// Control and gateway must share a parent directory (or /tmp prefix).
let control_dir = std::path::Path::new(&control_client)
.parent()
.map(|p| p.to_string_lossy().into_owned())
.unwrap_or_default();
let gateway_dir = std::path::Path::new(&gateway_client)
.parent()
.map(|p| p.to_string_lossy().into_owned())
.unwrap_or_default();
assert_eq!(
control_dir, gateway_dir,
"control and gateway default-socket paths picked different directories: \
control={control_client}, gateway={gateway_client}"
);
}
#[cfg(unix)]
#[test]
fn test_resolve_default_socket_xdg_when_no_run_fips() {
// With /run/fips unwritable (non-root tests) and XDG_RUNTIME_DIR
// pointing at an existing directory, the resolver picks XDG.
let _g = ENV_MUTEX.lock().unwrap();
let temp_dir = TempDir::new().unwrap();
let prev_xdg = std::env::var("XDG_RUNTIME_DIR").ok();
// SAFETY: serialized via ENV_MUTEX above.
unsafe {
std::env::set_var("XDG_RUNTIME_DIR", temp_dir.path());
}
let path = resolve_default_socket("control.sock");
// Restore env before asserting so a panic doesn't leak state.
unsafe {
match prev_xdg {
Some(v) => std::env::set_var("XDG_RUNTIME_DIR", v),
None => std::env::remove_var("XDG_RUNTIME_DIR"),
}
}
// If /run/fips happens to be writable in the test environment (CI
// running as root, for instance), the resolver legitimately picks
// /run/fips and skips XDG entirely. Accept either outcome but
// demand that one of the two canonical prefixes is chosen — never
// /tmp when XDG was valid.
assert!(
path.starts_with("/run/fips/")
|| path.starts_with(&format!("{}/fips/", temp_dir.path().display())),
"expected /run/fips or XDG path, got: {path}"
);
}
#[cfg(unix)]
#[test]
fn test_resolve_default_socket_tmp_when_xdg_invalid() {
// With XDG_RUNTIME_DIR pointing at a non-existent directory and
// /run/fips unwritable, the resolver falls through to /tmp.
let _g = ENV_MUTEX.lock().unwrap();
let prev_xdg = std::env::var("XDG_RUNTIME_DIR").ok();
// Use a path that definitely does not exist.
let bogus = "/nonexistent-xdg-runtime-dir-for-fips-test-zzz";
// SAFETY: serialized via ENV_MUTEX.
unsafe {
std::env::set_var("XDG_RUNTIME_DIR", bogus);
}
let path = resolve_default_socket("gateway.sock");
unsafe {
match prev_xdg {
Some(v) => std::env::set_var("XDG_RUNTIME_DIR", v),
None => std::env::remove_var("XDG_RUNTIME_DIR"),
}
}
// Accept either /run/fips/ (test running as root with that dir
// writable) or /tmp/fips-... (the dev-machine fallback). Never
// accept the bogus XDG dir leaking through.
assert!(
path.starts_with("/run/fips/") || path == "/tmp/fips-gateway.sock",
"expected /run/fips or /tmp fallback, got: {path}"
);
assert!(
!path.starts_with(bogus),
"stale/invalid XDG_RUNTIME_DIR leaked into resolver: {path}"
);
}
}
src/config/node.rs
+6 -11
View File
@@ -784,20 +784,15 @@ impl ControlConfig {
/// Default control socket path.
///
/// On Unix, returns a Unix domain socket path (XDG_RUNTIME_DIR, /run/fips,
/// or /tmp fallback). On Windows, returns a TCP port number as a string
/// since Windows does not support Unix domain sockets; the control socket
/// listens on localhost at this port.
/// On Unix, delegates to [`super::resolve_default_socket`] for the
/// canonical `/run/fips` → `XDG_RUNTIME_DIR` → `/tmp` order shared with
/// the client-side `default_control_path`. On Windows, returns a TCP
/// port number as a string since Windows does not support Unix domain
/// sockets; the control socket listens on localhost at this port.
fn default_socket_path() -> String {
#[cfg(unix)]
{
if let Ok(runtime_dir) = std::env::var("XDG_RUNTIME_DIR") {
format!("{runtime_dir}/fips/control.sock")
} else if std::fs::create_dir_all("/run/fips").is_ok() {
"/run/fips/control.sock".to_string()
} else {
"/tmp/fips-control.sock".to_string()
}
super::resolve_default_socket("control.sock")
}
#[cfg(windows)]
{
src/config/transport.rs
+4 -2
View File
@@ -267,8 +267,10 @@ pub struct EthernetConfig {
#[serde(default, skip_serializing_if = "Option::is_none")]
pub ethertype: Option<u16>,
/// MTU override. Defaults to the interface's MTU minus 1 (for frame type prefix).
/// Cannot exceed the interface's actual MTU.
/// MTU override. Defaults to the interface's MTU minus 3 bytes of frame
/// header (`[type:1][length:2 LE][payload]`). The 2-byte length field is
/// required to trim NIC minimum-frame padding before AEAD verification.
/// Cannot exceed the interface's actual MTU minus 3.
#[serde(default, skip_serializing_if = "Option::is_none")]
pub mtu: Option<u16>,
src/gateway/control.rs
+8
View File
@@ -15,6 +15,14 @@ use tokio::sync::watch;
use tracing::{debug, info, warn};
/// Socket path for the gateway control socket.
///
/// Hardcoded to `/run/fips/gateway.sock`: gateway operation requires root
/// for NAT/conntrack management, so the daemon side never needs to fall
/// back to `XDG_RUNTIME_DIR` or `/tmp`. Client tools resolve the path via
/// [`crate::config::default_gateway_path`], which uses the shared
/// [`crate::config::resolve_default_socket`] helper and falls through to
/// `XDG_RUNTIME_DIR` / `/tmp` for non-root dev runs that don't have
/// `/run/fips` writable.
pub const GATEWAY_SOCKET_PATH: &str = "/run/fips/gateway.sock";
/// Maximum request size in bytes (4 KB).
testing/chaos/scenarios/bloom-storm.README.md
+145
View File
@@ -0,0 +1,145 @@
# bloom-storm — bloom announce storm regression scenario
Six-node depth-4 mesh with an induced upstream parent flap. Asserts a
trailing-window ceiling on per-node `stats.bloom.sent` to catch a
regression class where a localized spanning-tree update at an
internal mid-chain edge fails to be properly contained and instead
propagates as a bloom announce storm down to the leaves.
## Topology
```text
n01 (root, depth 0)
/ \
n02 n03 (parent candidates pa/pb, depth 1)
\ /
n04 (flap, depth 2 — induced parent flap)
|
n05 (leaf, depth 3)
|
n06 (tail, depth 4)
```
n01 is expected to win the root election under the deterministic key
derivation used by the chaos runner; the run's tree snapshots
(`tree-snapshot-warmup.json`, `tree-snapshot-final.json`) record the
actual assignment.
## Bug class guarded against
A spanning-tree update that changes only an internal path edge — no
root change, no depth change — must not produce a sustained bloom
announce storm at downstream nodes. The original regression
(rolled-back `0caef2a`, fixed in master `4cdf382`) had this property:
in the field, a single mid-chain ancestor swap on an upstream node
caused every downstream node in its subtree to issue a bloom
announce on every parent re-evaluation tick of the upstream node,
resulting in a ~480x mesh-wide elevation in `FilterAnnounce` traffic
and a perfect bimodal flip on `est_entries`.
## Mechanism
The new `link_swap` chaos primitive deterministically alternates the
netem delay on `n02-n04` (5ms vs 100ms) and `n03-n04` (100ms vs 5ms)
every 4 seconds. Combined with the FIPS overrides
(`parent_hysteresis: 0.0`, `reeval_interval_secs: 1`,
`flap_threshold: 9999`, `hold_down_secs: 0`), this forces n04 to
switch parents on every swap. The whole point of the scenario is to
*force* sustained parent flapping at n04 and assert that the bloom
layer doesn't amplify it.
## Assertions
```yaml
assertions:
bloom_send_rate:
window_secs: 30
max_per_node: 30
min_parent_switches:
min_total: 10
```
`bloom_send_rate` is the load-bearing assertion: per-node delta of
`stats.bloom.sent` over the trailing 30s of the run must be at most
30. Per-node deltas and the offending node IDs are written to
`assertions.txt` and the runner exits 3 on failure.
`min_parent_switches` is a sanity guard. It fails if the run did not
record at least 10 parent switches across all nodes, which would
mean the harness fired its link swaps but the topology never produced
a real parent-switch event (e.g., wrong root election made the flap
target's parent candidates structurally non-equivalent). Without this
guard, the bloom-rate assertion would trivially pass on any binary,
including a regressed one.
## Threshold derivation
The original `issues/2026-0019-repro/` reproduction harness measured
(90s flap window, ~21 induced parent switches at the mid-chain
node):
| binary | tail bloom_sent / 90s | rate scaled to 30s |
| ----------- | --------------------: | -----------------: |
| pre-fix | 21 | ~7 |
| current fix | 0 | 0 |
Observed on this scenario at master `db5b6b1` (180s run, 35 parent
switches, 41 link swaps), per-node `bloom_sent` deltas over the
trailing 30s:
```text
n01=5 n02=5 n03=4 n04=12 n05=6 n06=0
```
n04 (the flapping node) is the highest because it is legitimately
re-sending its filter on its own parent changes. n06 (the depth-4
"tail") sees 0, matching the calm post-fix behavior recorded in
`issues/2026-0019-repro/RESULTS.md` for the `fix2` variant.
In the field, the regression's mesh-wide rate scaled ~480x above
steady state. A `30 / 30s / node` ceiling sits ~2.5x above the
observed maximum on fixed master and well below the
deployment-scale storm rate, giving headroom for harness jitter
without losing the ability to fail loud on the regression class.
If `link_swap.interval_secs` or the netem delta is changed,
recalibrate. The threshold is calibrated against the values in
`scenarios/bloom-storm.yaml` as committed and the `seed: 31` pin.
## Limitations
- The bloom-storm regression has not been confirmed-failing here
on a regressed binary in this harness directly; the threshold is
inferred from the values measured in the dedicated
`issues/2026-0019-repro/` post-mortem harness against
`0caef2a`. To gain that confirmation, check out `0caef2a`
(or the `backup-broadcast-gate-bloom-storm` branch if still
retained), build, copy binaries into `testing/docker/`, and rerun
this scenario; the bloom-rate assertion is expected to fail loud
with n05/n06 deltas well above 30.
- Root-election outcome is sensitive to the seed (smallest
`NodeAddr` wins, where `NodeAddr = SHA-256(pubkey)[..16]`). The
seed value `31` is pinned for this reason. The
`min_parent_switches` assertion catches drift if the seed is
changed without re-validating the topology.
## Running locally
```bash
# From the source repo root, with binaries already built and copied
# into testing/docker/ (see testing/scripts/build.sh).
./testing/chaos/scripts/chaos.sh bloom-storm
```
Run output is in `testing/chaos/sim-results/<timestamp>-bloom-storm/`.
Key artifacts:
- `analysis.txt` — log analysis (panics, errors, parent switches).
- `assertions.txt` — per-assertion pass/fail with per-node deltas.
- `tree-snapshot-warmup.json`, `tree-snapshot-final.json` — control
socket tree state at warmup end and at run end.
- `runner.log` — full orchestration log.
Total runtime: ~3.5 minutes (25s warmup + 180s scenario + ~30s
teardown).
testing/chaos/scenarios/bloom-storm.yaml
+142
View File
@@ -0,0 +1,142 @@
# Bloom-storm regression scenario.
#
# Six-node depth-4 mesh with an induced upstream parent flap. Guards
# against a regression class where a spanning-tree update that changes
# only an internal path edge (no root or depth delta) fails to
# propagate to leaves, producing a sustained bloom-traffic
# oscillation visible only at fleet scale and only after several
# minutes of uptime. See README.md alongside this file for the
# bug-class description and threshold derivation.
#
# Topology (n01 should win the root election under deterministic
# key derivation; the validation snapshot at end-of-run will confirm
# the assigned root):
#
# n01 (root, depth 0)
# / \
# n02 n03 (parent candidates pa/pb, depth 1)
# \ /
# n04 (flap, depth 2 — induced parent flap)
# |
# n05 (leaf, depth 3 — observes mid-chain swap)
# |
# n06 (tail, depth 4 — sees the cascade)
#
# The flap is induced by alternating the netem delay on n02-n04 and
# n03-n04 every 4 seconds. Because the FIPS overrides below disable
# parent-flap dampening, n04 switches parents on each swap. The bug
# class under test is whether this localized swap at depth 2 leaks
# downstream as a bloom-announce storm at depth 3 and depth 4.
#
# Assertions (bloom_send_rate.max_per_node_per_window) are evaluated
# over the trailing 30s of the run, after sustained flapping has had
# time to expose any cascade behavior.
scenario:
name: "bloom-storm"
# Seed pinned so deterministic key derivation makes n01 win the
# root election and orders the other NodeAddrs as n02 < n03 <
# ... < n04 (so n04 is at the bottom of the tiebreak and its
# parent candidates n02/n03 are both at depth 1). The warmup tree
# snapshot in sim-results/ is the authoritative check.
#
# Other seeds may elect a different node as root or order n04
# ahead of n02/n03, in which case the induced flap will not
# exercise the cascade path the assertion is written to catch.
seed: 31
duration_secs: 180
topology:
algorithm: explicit
num_nodes: 6
params:
adjacency:
- [n01, n02]
- [n01, n03]
- [n02, n04]
- [n03, n04]
- [n04, n05]
- [n05, n06]
subnet: "172.20.0.0/24"
ip_start: 10
netem:
enabled: true
default_policy:
# Calm baseline so steady-state bloom traffic is at its floor.
delay_ms: [1, 1]
jitter_ms: [0, 0]
loss_pct: [0, 0]
# Asymmetric link-cost flap on n04's two upstream candidates. Every
# `interval_secs`, the policies on the two listed edges are swapped.
# This drives n04 to alternate parents between n02 and n03 each
# round. The 4s cadence and 5ms-vs-100ms delta come from the
# original ISSUE-2026-0019 reproduction harness — they are
# calibrated to produce a parent switch per round under the
# zero-hysteresis FIPS overrides below.
link_swap:
enabled: true
interval_secs: 4
policies:
fast:
delay_ms: [5, 5]
jitter_ms: [0, 0]
loss_pct: [0, 0]
slow:
delay_ms: [100, 100]
jitter_ms: [0, 0]
loss_pct: [0, 0]
# Initial assignment: n02-n04 fast, n03-n04 slow. After each
# interval the assignments are swapped.
edges:
- edge: "n02-n04"
policy: fast
- edge: "n03-n04"
policy: slow
link_flaps:
enabled: false
traffic:
enabled: false
logging:
rust_log: "info"
output_dir: "./sim-results"
# Per-node assertions evaluated at the end of the run. Each asserts
# on the delta of a control-socket counter over a trailing window.
assertions:
bloom_send_rate:
# The trailing window is the last `window_secs` of the run.
# `max_per_node` is the ceiling on stats.bloom.sent delta over
# that window, per node. The threshold is set ~5x above the
# observed steady-state ceiling on fixed master and well below
# the storm-state value on the regressed binary.
window_secs: 30
max_per_node: 30
min_parent_switches:
# Sanity guard: detect a misconfigured harness where the flap
# inducer fires but the topology never produces a real parent
# switch (e.g., wrong root election made n04's parent
# candidates structurally non-equivalent). Without this, the
# bloom-rate assertion would trivially pass on any binary,
# including a regressed one.
#
# Floor calibrated against expected behavior under the chosen
# seed: 41 link swaps over 180s should produce >= 10 parent
# switches at n04 alone if the topology is right.
min_total: 10
# FIPS overrides to suppress parent-flap dampening so the asymmetric
# link-cost flap reliably produces a parent switch per round. The
# whole point of the scenario is to *force* sustained parent
# flapping at n04 and assert that bloom layer doesn't amplify it.
fips_overrides:
node:
tree:
parent_hysteresis: 0.0
reeval_interval_secs: 1
flap_threshold: 9999
hold_down_secs: 0
testing/chaos/sim/__main__.py
+2
View File
@@ -54,6 +54,8 @@ def main():
if result and result.panics:
sys.exit(2)
if runner.assertions_failed:
sys.exit(3)
sys.exit(0)
testing/chaos/sim/assertions.py
+163
View File
@@ -0,0 +1,163 @@
"""Post-run scenario assertions evaluated via control-socket data.
Assertions are declared in the scenario YAML under ``assertions:`` and
are evaluated near the end of the simulation, before teardown begins.
Each failing assertion is recorded with a clear pass/fail message; the
runner exits non-zero when any assertion fails.
Currently supported assertions:
- ``bloom_send_rate``: per-node trailing-window ceiling on
``stats.bloom.sent`` delta. Calibrated for the bloom-storm
regression scenario but generally usable.
"""
from __future__ import annotations
import logging
from dataclasses import dataclass
from .control import snapshot_all_bloom
from .scenario import BloomSendRateAssertion, MinParentSwitchesAssertion
from .topology import SimTopology
log = logging.getLogger(__name__)
@dataclass
class AssertionOutcome:
name: str
passed: bool
detail: str
def _bloom_sent_total(node_data: dict) -> int | None:
"""Extract stats.bloom.sent from a show_bloom response."""
stats = node_data.get("stats") or {}
sent = stats.get("sent")
if sent is None:
return None
try:
return int(sent)
except (TypeError, ValueError):
return None
class BloomSendRateMonitor:
"""Samples per-node ``stats.bloom.sent`` to evaluate a trailing-window
ceiling assertion at end-of-run.
Usage:
m = BloomSendRateMonitor(topology, cfg)
m.sample_window_start() # called window_secs before scenario end
...
m.sample_end() # called at scenario end
outcome = m.evaluate()
"""
def __init__(self, topology: SimTopology, cfg: BloomSendRateAssertion):
self.topology = topology
self.cfg = cfg
self.window_start: dict[str, int] = {}
self.window_end: dict[str, int] = {}
def sample_window_start(self) -> None:
snap = snapshot_all_bloom(self.topology)
for nid, data in snap.items():
v = _bloom_sent_total(data)
if v is not None:
self.window_start[nid] = v
def sample_end(self) -> None:
snap = snapshot_all_bloom(self.topology)
for nid, data in snap.items():
v = _bloom_sent_total(data)
if v is not None:
self.window_end[nid] = v
def evaluate(self) -> AssertionOutcome:
max_per_node = self.cfg.max_per_node
window_secs = self.cfg.window_secs
if not self.window_start or not self.window_end:
return AssertionOutcome(
name="bloom_send_rate",
passed=False,
detail=(
f"FAIL bloom_send_rate: failed to sample window endpoints "
f"(start={len(self.window_start)} nodes, "
f"end={len(self.window_end)} nodes)"
),
)
per_node_deltas: dict[str, int] = {}
for nid, end_v in self.window_end.items():
start_v = self.window_start.get(nid)
if start_v is None:
continue
per_node_deltas[nid] = end_v - start_v
offenders = {
nid: d for nid, d in per_node_deltas.items() if d > max_per_node
}
max_obs = max(per_node_deltas.values()) if per_node_deltas else 0
if offenders:
sorted_off = sorted(offenders.items(), key=lambda kv: -kv[1])
details = ", ".join(f"{nid}={d}" for nid, d in sorted_off)
detail = (
f"FAIL bloom_send_rate: {len(offenders)} node(s) exceeded "
f"ceiling of {max_per_node} bloom_sent over trailing "
f"{window_secs}s — offenders: {details} "
f"(all per-node deltas: "
f"{', '.join(f'{n}={v}' for n, v in sorted(per_node_deltas.items()))})"
)
return AssertionOutcome(
name="bloom_send_rate",
passed=False,
detail=detail,
)
detail = (
f"PASS bloom_send_rate: max per-node delta {max_obs} <= "
f"ceiling {max_per_node} over trailing {window_secs}s "
f"(per-node: "
f"{', '.join(f'{n}={v}' for n, v in sorted(per_node_deltas.items()))})"
)
return AssertionOutcome(
name="bloom_send_rate",
passed=True,
detail=detail,
)
def evaluate_min_parent_switches(
cfg: MinParentSwitchesAssertion,
parent_switch_count: int,
) -> AssertionOutcome:
"""Sanity guard: fail the scenario if the harness-induced flap did
not produce at least ``cfg.min_total`` parent switches across the
run. Detects misconfiguration (e.g., wrong root election) where
the bloom-rate assertion would otherwise trivially pass on any
binary including the regressed one.
"""
if parent_switch_count >= cfg.min_total:
return AssertionOutcome(
name="min_parent_switches",
passed=True,
detail=(
f"PASS min_parent_switches: {parent_switch_count} switches "
f">= floor {cfg.min_total}"
),
)
return AssertionOutcome(
name="min_parent_switches",
passed=False,
detail=(
f"FAIL min_parent_switches: {parent_switch_count} switches "
f"< floor {cfg.min_total} — harness did not induce sufficient "
f"parent flapping; bloom-rate assertion would be trivially "
f"true. Check tree-snapshot-warmup.json: did the expected "
f"node win the root election?"
),
)
testing/chaos/sim/control.py
+21
View File
@@ -149,6 +149,27 @@ def snapshot_all_mmp(topology: SimTopology) -> dict[str, dict]:
return result
def query_bloom(container: str) -> dict | None:
"""Query a node's bloom filter state and stats."""
return query_node(container, "show_bloom")
def snapshot_all_bloom(topology: SimTopology) -> dict[str, dict]:
"""Query show_bloom on all nodes, return {node_id: bloom_data}.
Nodes that fail to respond are omitted from the result.
"""
result = {}
for node_id in sorted(topology.nodes):
container = topology.container_name(node_id)
data = query_bloom(container)
if data is not None:
result[node_id] = data
else:
log.warning("No bloom data from %s", node_id)
return result
def query_routing(container: str) -> dict | None:
"""Query a node's routing stats (includes congestion counters)."""
return query_node(container, "show_routing")
testing/chaos/sim/link_swap.py
+151
View File
@@ -0,0 +1,151 @@
"""Deterministic asymmetric link-cost flapping.
Periodically rotates a fixed assignment of netem policies across a
fixed set of edges. Unlike ``LinkManager`` (random link-down events)
or the ``netem.mutation`` block (random per-edge policy mutation),
this is a deterministic, periodic flip between named policies on
named edges — useful for forcing a downstream node to switch parents
on a fixed cadence.
Rotation is a simple one-position cyclic shift: the policy assigned
to edges[0] moves to edges[1], edges[1] -> edges[2], ..., edges[N-1]
-> edges[0]. With two edges this degenerates to a swap, which is the
intended use for the bloom-storm scenario.
"""
from __future__ import annotations
import logging
import random
import time
from .netem import NetemManager, NetemParams
from .scenario import LinkSwapConfig, NetemPolicy
from .topology import SimTopology
log = logging.getLogger(__name__)
def _canonical_edge(edge_str: str) -> tuple[str, str]:
"""Parse "nXX-nYY" into a canonical (a, b) tuple sorted alphabetically."""
parts = edge_str.split("-")
if len(parts) != 2:
raise ValueError(f"link_swap edge '{edge_str}' is not 'nXX-nYY' form")
a, b = sorted(parts)
return a, b
class LinkSwapManager:
"""Manages deterministic netem-policy rotation across a fixed edge set."""
def __init__(
self,
topology: SimTopology,
config: LinkSwapConfig,
netem_mgr: NetemManager,
rng: random.Random,
):
self.topology = topology
self.config = config
self.netem_mgr = netem_mgr
self.rng = rng
# Resolve edges and validate they exist in the topology
topo_edges = {tuple(sorted([a, b])) for a, b in topology.edges}
self._edges: list[tuple[str, str]] = []
for entry in config.edges:
edge = _canonical_edge(entry.edge)
if edge not in topo_edges:
raise ValueError(
f"link_swap edge {entry.edge} not present in topology"
)
self._edges.append(edge)
# Current per-position policy name. Index i in this list is
# the policy currently applied to self._edges[i]. Each rotate()
# cyclically shifts these by one position.
self._current_policies: list[str] = [e.policy for e in config.edges]
self.swap_count = 0
# Last apply timestamp; the runner uses this together with
# config.interval_secs to schedule rotations.
self.last_swap_at: float | None = None
def policies(self) -> dict[str, NetemPolicy]:
return self.config.policies
def setup_initial(self) -> None:
"""Apply the initial policy assignment to each edge.
Called once after NetemManager.setup_initial() so the link's
per-direction tc class state already exists. The initial
assignment is taken straight from the config (no rotation).
"""
for edge, policy_name in zip(self._edges, self._current_policies):
policy = self.config.policies[policy_name]
self._apply(edge, policy)
self.last_swap_at = time.time()
log.info(
"Link swap initialized: %d edges, interval %.1fs, policies=%s",
len(self._edges),
self.config.interval_secs,
list(self.config.policies.keys()),
)
def maybe_swap(self, now: float) -> bool:
"""If the swap interval has elapsed, rotate policies. Return whether
a swap occurred so the runner can reschedule.
"""
if self.last_swap_at is None:
self.last_swap_at = now
return False
if (now - self.last_swap_at) < self.config.interval_secs:
return False
# Cyclic shift by one position
self._current_policies = (
[self._current_policies[-1]] + self._current_policies[:-1]
)
for edge, policy_name in zip(self._edges, self._current_policies):
policy = self.config.policies[policy_name]
self._apply(edge, policy)
self.swap_count += 1
self.last_swap_at = now
log.debug(
"Link swap #%d: %s",
self.swap_count,
", ".join(
f"{a}-{b}={p}"
for (a, b), p in zip(self._edges, self._current_policies)
),
)
return True
def _apply(self, edge: tuple[str, str], policy: NetemPolicy) -> None:
"""Apply a policy to both directions of an edge using NetemManager."""
params = self._sample_policy(policy)
# Drive both directions through NetemManager's per-link state so
# the rotation persists across mutate() runs and survives node
# restarts via setup_node().
self.netem_mgr._update_link(edge[0], edge[1], params)
def _sample_policy(self, policy: NetemPolicy) -> NetemParams:
"""Sample concrete params from a policy's ranges (deterministic
when min == max, which is the bloom-storm scenario's contract).
"""
return NetemParams(
delay_ms=int(self.rng.uniform(policy.delay_ms[0], policy.delay_ms[1])),
jitter_ms=int(self.rng.uniform(policy.jitter_ms[0], policy.jitter_ms[1])),
loss_pct=round(
self.rng.uniform(policy.loss_pct[0], policy.loss_pct[1]), 1
),
duplicate_pct=round(
self.rng.uniform(policy.duplicate_pct[0], policy.duplicate_pct[1]), 1
),
reorder_pct=round(
self.rng.uniform(policy.reorder_pct[0], policy.reorder_pct[1]), 1
),
corrupt_pct=round(
self.rng.uniform(policy.corrupt_pct[0], policy.corrupt_pct[1]), 1
),
)
testing/chaos/sim/runner.py
+117
View File
@@ -12,10 +12,12 @@ import sys
import time
from datetime import datetime
from .assertions import AssertionOutcome, BloomSendRateMonitor, evaluate_min_parent_switches
from .compose import generate_compose
from .config_gen import write_configs
from .control import snapshot_all_congestion, snapshot_all_mmp, snapshot_all_trees
from .docker_exec import docker_compose
from .link_swap import LinkSwapManager
from .links import LinkManager
from .logs import AnalysisResult, analyze_logs, collect_logs, write_sim_metadata
from .netem import NetemManager
@@ -46,10 +48,15 @@ class SimRunner:
self.veth_mgr: VethManager | None = None
self.netem_mgr: NetemManager | None = None
self.link_mgr: LinkManager | None = None
self.link_swap_mgr: LinkSwapManager | None = None
self.traffic_mgr: TrafficManager | None = None
self.node_mgr: NodeManager | None = None
self.peer_churn_mgr: PeerChurnManager | None = None
# Post-run assertion monitors (sampled near end of run).
self.bloom_rate_monitor: BloomSendRateMonitor | None = None
self.assertion_outcomes: list[AssertionOutcome] = []
@staticmethod
def _resolve_output_dir(scenario: Scenario) -> str:
"""Build a timestamped output directory path.
@@ -190,6 +197,20 @@ class SimRunner:
self.topology, s.link_flaps, self.rng, netem_mgr=self.netem_mgr
)
if s.link_swap.enabled:
if not self.netem_mgr:
raise RuntimeError(
"link_swap requires netem.enabled (depends on per-link tc state)"
)
self.link_swap_mgr = LinkSwapManager(
self.topology, s.link_swap, self.netem_mgr, self.rng,
)
if s.assertions.bloom_send_rate is not None:
self.bloom_rate_monitor = BloomSendRateMonitor(
self.topology, s.assertions.bloom_send_rate,
)
if s.traffic.enabled:
self.traffic_mgr = TrafficManager(
self.topology, s.traffic, self.rng, down_nodes=self._down_nodes
@@ -226,6 +247,12 @@ class SimRunner:
if self.peer_churn_mgr:
self.peer_churn_mgr.refresh_all_npubs()
# Initial link swap policy assignment (after warmup so the netem
# tc state is fully set up and the daemons have already
# discovered each other under the calm baseline).
if self.link_swap_mgr:
self.link_swap_mgr.setup_initial()
def _handle_node_restart(self, node_id: str):
"""Called after a node container is restarted.
@@ -260,12 +287,38 @@ class SimRunner:
next_churn = self._schedule_next(start, s.node_churn.interval_secs) if self.node_mgr else float("inf")
next_peer_churn = self._schedule_next(start, s.peer_churn.interval_secs) if self.peer_churn_mgr else float("inf")
# Bloom-send-rate assertion: sample at window_secs before end.
bloom_window_start_at = float("inf")
if self.bloom_rate_monitor is not None:
bloom_window_start_at = (
start + duration - s.assertions.bloom_send_rate.window_secs
)
bloom_window_started = False
while not self._interrupted:
now = time.time()
elapsed = now - start
if elapsed >= duration:
break
# Bloom-rate window-start sampling
if (
self.bloom_rate_monitor is not None
and not bloom_window_started
and now >= bloom_window_start_at
):
log.info(
"Sampling bloom-rate window start (last %ds of run)...",
s.assertions.bloom_send_rate.window_secs,
)
self.bloom_rate_monitor.sample_window_start()
bloom_window_started = True
# Deterministic link swap (before netem mutation so a
# mutation round can't clobber the swap mid-tick).
if self.link_swap_mgr:
self.link_swap_mgr.maybe_swap(now)
# Netem mutation
if self.netem_mgr and now >= next_netem:
self.netem_mgr.mutate()
@@ -304,6 +357,8 @@ class SimRunner:
active = self.traffic_mgr.active_count if self.traffic_mgr else 0
peer_churns = self.peer_churn_mgr.churn_count if self.peer_churn_mgr else 0
status_extra = f" peer_churns={peer_churns}" if self.peer_churn_mgr else ""
if self.link_swap_mgr:
status_extra += f" swaps={self.link_swap_mgr.swap_count}"
print(
f"\r [{elapsed:.0f}s/{duration}s] "
f"nodes={len(self.topology.nodes)} "
@@ -320,11 +375,49 @@ class SimRunner:
print() # Clear status line
# Bloom-rate window-end sampling.
# Done before teardown so containers are still running.
if self.bloom_rate_monitor is not None:
if not bloom_window_started:
# Loop exited before the window-start mark (e.g.,
# interrupted). Take both samples now so we still
# produce a finite outcome rather than an empty dict.
log.warning(
"Bloom-rate window did not start during run; "
"sampling both endpoints at end (delta will be 0)."
)
self.bloom_rate_monitor.sample_window_start()
log.info("Sampling bloom-rate window end...")
self.bloom_rate_monitor.sample_end()
def _evaluate_assertions(self) -> None:
"""Evaluate post-run assertions and stash outcomes on self.
Called from teardown while containers are still running so the
assertion outcomes (which include per-node detail) can be
written alongside the run artifacts.
"""
if self.bloom_rate_monitor is not None:
outcome = self.bloom_rate_monitor.evaluate()
self.assertion_outcomes.append(outcome)
if outcome.passed:
log.info("%s", outcome.detail)
else:
log.error("%s", outcome.detail)
@property
def assertions_failed(self) -> bool:
return any(not o.passed for o in self.assertion_outcomes)
def _teardown(self) -> AnalysisResult | None:
"""Stop dynamic elements, collect logs, analyze, stop containers."""
result = None
if self.topology and self.compose_file:
# Evaluate post-run assertions before doing any teardown so
# control sockets are still reachable.
self._evaluate_assertions()
# Stop traffic
if self.traffic_mgr:
log.info("Stopping traffic sessions...")
@@ -366,6 +459,30 @@ class SimRunner:
f.write(result.summary())
print(result.summary())
# Log-derived assertions (evaluated after analyze_logs so
# parent_switches and similar are populated).
mps_cfg = self.scenario.assertions.min_parent_switches
if mps_cfg is not None:
outcome = evaluate_min_parent_switches(
mps_cfg, len(result.parent_switches)
)
self.assertion_outcomes.append(outcome)
if outcome.passed:
log.info("%s", outcome.detail)
else:
log.error("%s", outcome.detail)
# Write assertion outcomes
if self.assertion_outcomes:
assertions_path = os.path.join(self.output_dir, "assertions.txt")
with open(assertions_path, "w") as f:
for o in self.assertion_outcomes:
f.write(o.detail + "\n")
print("=== Assertions ===")
for o in self.assertion_outcomes:
print(o.detail)
print()
# Write metadata
write_sim_metadata(
self.output_dir,
testing/chaos/sim/scenario.py
+123
View File
@@ -150,6 +150,67 @@ class IngressConfig:
burst_bytes: int = 32000
@dataclass
class LinkSwapEdge:
"""One edge in a link-swap rotation.
Edge is a canonical "nXX-nYY" string. ``policy`` names a policy
in ``LinkSwapConfig.policies``.
"""
edge: str = ""
policy: str = ""
@dataclass
class LinkSwapConfig:
"""Deterministic asymmetric link-cost flapping.
On each ``interval_secs``, the policies on every pair of edges in
``edges`` are swapped (cyclically rotated by one position). This
differs from ``link_flaps`` (random link-down events) and
``netem.mutation`` (random per-edge policy mutation) by being a
deterministic, periodic flip between two named netem policies on
a fixed set of edges.
Used to drive a downstream node to repeatedly switch parents on
a fixed cadence — exercises the spanning-tree rebalance path
without the noise of a random mutation walk.
"""
enabled: bool = False
interval_secs: float = 4.0
policies: dict[str, NetemPolicy] = field(default_factory=dict)
edges: list[LinkSwapEdge] = field(default_factory=list)
@dataclass
class BloomSendRateAssertion:
"""Trailing-window ceiling on per-node ``stats.bloom.sent`` delta."""
window_secs: int = 30
max_per_node: int = 30
@dataclass
class MinParentSwitchesAssertion:
"""Sanity guard: total parent switches across the run must be at
least ``min_total``. Used to detect a misconfigured harness where
the flap inducer is firing but the topology never produces a real
parent-switch event (e.g., wrong root election).
"""
min_total: int = 1
@dataclass
class AssertionsConfig:
"""Optional post-run assertions evaluated against control-socket data."""
bloom_send_rate: BloomSendRateAssertion | None = None
min_parent_switches: MinParentSwitchesAssertion | None = None
@dataclass
class LoggingConfig:
rust_log: str = "info"
@@ -169,6 +230,8 @@ class Scenario:
peer_churn: PeerChurnConfig = field(default_factory=PeerChurnConfig)
bandwidth: BandwidthConfig = field(default_factory=BandwidthConfig)
ingress: IngressConfig = field(default_factory=IngressConfig)
link_swap: LinkSwapConfig = field(default_factory=LinkSwapConfig)
assertions: AssertionsConfig = field(default_factory=AssertionsConfig)
logging: LoggingConfig = field(default_factory=LoggingConfig)
# Raw YAML dict appended to each generated FIPS node config.
# Allows scenarios to override any FIPS config parameter
@@ -320,6 +383,40 @@ def load_scenario(path: str) -> Scenario:
s.ingress.tiers_kbps = [int(t) for t in tiers]
s.ingress.burst_bytes = int(ig.get("burst_bytes", 32000))
# Link swap section (deterministic asymmetric link-cost flapping).
ls = raw.get("link_swap", {})
s.link_swap.enabled = ls.get("enabled", False)
if "interval_secs" in ls:
s.link_swap.interval_secs = float(ls["interval_secs"])
if "policies" in ls:
s.link_swap.policies = {
name: _parse_netem_policy(pdata)
for name, pdata in ls["policies"].items()
}
if "edges" in ls:
for edata in ls["edges"]:
if not isinstance(edata, dict):
raise ValueError("link_swap.edges entries must be dicts")
edge = str(edata.get("edge", ""))
policy = str(edata.get("policy", ""))
if not edge or not policy:
raise ValueError("link_swap.edges entries require 'edge' and 'policy'")
s.link_swap.edges.append(LinkSwapEdge(edge=edge, policy=policy))
# Assertions section (post-run control-socket-based checks).
asrt = raw.get("assertions", {})
if "bloom_send_rate" in asrt:
bsr = asrt["bloom_send_rate"]
s.assertions.bloom_send_rate = BloomSendRateAssertion(
window_secs=int(bsr.get("window_secs", 30)),
max_per_node=int(bsr.get("max_per_node", 30)),
)
if "min_parent_switches" in asrt:
mps = asrt["min_parent_switches"]
s.assertions.min_parent_switches = MinParentSwitchesAssertion(
min_total=int(mps.get("min_total", 1)),
)
# Logging section
lg = raw.get("logging", {})
s.logging.rust_log = lg.get("rust_log", "info")
@@ -435,3 +532,29 @@ def _validate(s: Scenario):
raise ValueError(f"ingress.tiers_kbps: all values must be > 0, got {tier}")
if s.ingress.burst_bytes <= 0:
raise ValueError(f"ingress.burst_bytes must be > 0, got {s.ingress.burst_bytes}")
# Validate link_swap
if s.link_swap.enabled:
if s.link_swap.interval_secs <= 0:
raise ValueError("link_swap.interval_secs must be > 0")
if len(s.link_swap.edges) < 2:
raise ValueError("link_swap.edges must list at least 2 edges to swap")
if not s.link_swap.policies:
raise ValueError("link_swap.policies must not be empty when link_swap.enabled")
for entry in s.link_swap.edges:
if entry.policy not in s.link_swap.policies:
raise ValueError(
f"link_swap.edges: policy '{entry.policy}' not in link_swap.policies"
)
# Validate assertions
if s.assertions.bloom_send_rate is not None:
bsr = s.assertions.bloom_send_rate
if bsr.window_secs < 1:
raise ValueError("assertions.bloom_send_rate.window_secs must be >= 1")
if bsr.max_per_node < 0:
raise ValueError("assertions.bloom_send_rate.max_per_node must be >= 0")
if bsr.window_secs > s.duration_secs:
raise ValueError(
"assertions.bloom_send_rate.window_secs must not exceed scenario duration"
)
testing/ci-local.sh
+2
View File
@@ -23,6 +23,7 @@
# chaos-ethernet-only, chaos-tcp-mesh, chaos-bottleneck-parent,
# chaos-cost-avoidance, chaos-cost-reeval, chaos-cost-stability,
# chaos-depth-vs-cost, chaos-mixed-technology, chaos-congestion-stress,
# chaos-bloom-storm,
# sidecar, dns-resolver, deb-install
#
# Opt-in (require --with-tor; depend on live Tor network):
@@ -70,6 +71,7 @@ CHAOS_SUITES=(
"depth-vs-cost depth-vs-cost"
"mixed-technology mixed-technology"
"congestion-stress congestion-stress"
"bloom-storm bloom-storm"
)
SIDECAR_SUITES=(sidecar)
GATEWAY_SUITES=(gateway)