feat(transport): bind and rebind network interfaces dynamically

An interface-bound transport is a long-lived object that is *sometimes
bound*. The interface it names may not exist when the daemon starts, may
appear minutes later, may vanish and return mid-operation, and may never
appear at all. Until now the first observation was final: an interface
missing at start was logged once and skipped for the life of the process,
and one that disappeared at runtime published a health change but was never
rebound.

On OpenWrt that is a live bug. procd starts fips before wifi has created
fips-mesh0 / fips-ap0, both transports are skipped and never retried, and
the 802.11s peer link forms anyway (that is mac80211, not the daemon) — so
the node looks healthy and reaches nothing.

Give every interface-bound transport presence state and a binder task.
start_async now returns Ok with the transport ABSENT rather than failing;
the binder binds when the interface appears, tears the socket down when it
goes away, and rebinds when it returns. Start-time absence and runtime
detach are one transition and one code path.

TransportError::InterfaceUnavailable makes absence branchable: a missing
interface and a typo'd interface name were the same flat StartFailed
(String), so nothing downstream could tell a state from a fault. A bind
failure that is *not* absence — no CAP_NET_RAW, no readable /dev/bpf* —
still fails the start, because retrying a socket that can never open behind
a Degraded nobody is watching is worse than dying loudly at boot.

**Presence is IFF_UP, not IFF_UP|IFF_RUNNING.** Carrier and bindability are
different questions and only the second belongs in a bind gate. An
AF_PACKET socket on a carrier-less bridge is valid and starts carrying
traffic the instant a member port comes up, with no rebind. Gating on
carrier would report a wifi-only router Degraded forever for an empty
br-lan, turn every carrier flap the socket would have survived into an
unbind/rebind cycle, and deadlock an 802.11s interface that reports RUNNING
only once it has peered — peering needs beacons, beacons need a bound
socket. Carrier is reported beside presence in show_transports instead.

**A name is not a device.** Both backends bind by device: AF_PACKET stores
sll_ifindex, a BPF descriptor follows the interface it was attached to. A
netdev deleted and recreated under the same name leaves the socket attached
to something gone while the name resolves perfectly well, and nothing
notices — a stale AF_PACKET socket never becomes readable, so the receive
loop neither errors nor exits, and send failures go to the caller rather
than the binder. A listen-only node (announce: false, no beacon sender to
fail) sat present and deaf indefinitely after a wifi reload. The bound index
is captured at bind and compared on every poll. A name reappearing with a
different MAC is different hardware: cached neighbors are dropped rather
than resumed onto.

**Detection is event-driven** where the kernel offers a source — netlink
RTNLGRP_LINK on Linux, PF_ROUTE on macOS and FreeBSD — with a 1 s
getifaddrs poll underneath as a backstop. Link-event payloads are not
parsed: an event is a hint to re-run the probe, which is cheap and
authoritative, and a parser's bugs would be presence bugs. Probes are
coalesced to ten a second because PF_ROUTE has no group filter and delivers
every routing message on the host; a persistently failing event source is
logged once, backed off, and abandoned for the poll after five errors
rather than spinning a core silently on ENOBUFS.

**Degraded becomes a level, not a latch.** The supervisor's reason set was
monotonic, which was correct while no child could recover; with recovery it
would have come to mean "something broke at some point since boot" rather
than "something is broken now". Absence lives in its own reversible set,
health is recomputed on every transition in both directions, and an absent
transport still counts as up so a single-interface node that boots before
its wifi degrades rather than exiting on NoTransports. There is deliberately
no restart action in the FSM: rebinding belongs next to the file descriptor,
and a supervisor-authored retry would be a second mechanism racing the first
for the same socket.

**The node's egress MTU follows the bound set.** transport_mtu filters on
is_bound(), not is_operational() — an interface-bound transport is
operational from the moment it starts, so the weaker predicate let hardware
that had never appeared clamp the whole node's IPv6 MTU. Because that
minimum now moves at runtime, the TUN reader and writer read the TCP MSS
ceiling from a shared atomic instead of a u16 captured at spawn; every other
consumer (show_status, the snapshot, the session-layer fragmentation check)
already read it live, so the clamp was the one place the daemon could report
one effective MTU and enforce another. It moves in both directions: a narrow
interface appearing tightens it, its departure releases it. MSS is
negotiated per connection, so a change binds connections opened after it.

**Logging is per edge, never per attempt**, with one deadline after it. The
edge itself is not an error — info at boot, warn on a runtime detach, since
an interface bound a moment later is the ordinary case this mechanism exists
to absorb and crying error at t=0 then "recovered" at t=0.2s is the failure
the rule exists to prevent. Ten seconds is the whole grace: past it, absence
is no longer a race against a radio or a container, so a required interface
still missing is reported once at error. Start-time absence and a runtime
detach share that one deadline, as they share everything else here. Said
once, not repeated: how long an absence has lasted is state, published as
interface.since_secs and as Degraded, and a monitor can threshold it per
deployment rather than the daemon compiling a schedule in.

Successful rebinds are damped. Backoff covers failed binds; the nastier case
is binds that keep succeeding into a socket that dies moments later, which a
receive loop giving up on a persistent error while the interface stays UP
produces once per second forever. Consecutive bindings dying inside ten
seconds back off on the 1 s → 30 s curve, and past three the binder stops
announcing each bind as a recovery until one lasts.

The receive loop backs off and exits on a dead socket instead of spinning on
Err with a warn per iteration, and the ad-hoc ENXIO socket reopen in the
beacon sender is gone: both hand recovery to the presence machine, one
mechanism for every cause rather than one hack per symptom. Beacons pause
while absent because the task simply does not exist then.

transports.ethernet.*.optional (default false) decides how loudly absence is
reported. Naming an interface is a statement that you expect it, so the
default complains: Degraded, and the error at the deadline. optional: true
is silent — info on the edge, no health impact, no error — for hardware
legitimately not always there. It describes the interface's presence, not
the transport's importance, and no value of it makes a missing interface
fatal at startup.

show_transports grows an `interface` block — presence, carrier, policy, how
long the phase has been held, bind count, failed attempts. The original boot
race was expensive precisely because nothing an operator could see said the
node was deaf.

ci: exercise the presence probe against musl and an address-less interface

Two legs, both about the assumption the presence probe rests on.

OpenWrt — the platform dynamic interface binding exists for — is musl, and
musl reimplements getifaddrs independently of glibc. `interface_present`
reads ifa_flags out of it, and the interfaces this feature exists for
(fips-mesh0, fips-ap0) are unbridged with no IP address at all, which is
exactly where getifaddrs implementations differ. Every other leg is glibc,
so without this one the probe was asserted on a libc no test had ever run it
against, on the target it was written for. Building for the musl target
rather than inside an Alpine container keeps the leg to a cross-build plus a
run, without a second toolchain image to maintain.

The address-less interface is created on both this leg and the glibc one.
Pinning the contract on both libcs is what turns "glibc and musl agree about
getifaddrs" from an assumption into a checked fact, and makes the glibc leg
fail first if glibc is the one that changes. Loopback cannot stand in: it
has 127.0.0.1, so probing it asks whether getifaddrs works rather than
whether it reports this. Deliberately not tolerant of failure — a guard that
quietly does not run is worse than no guard.

The iface-binding suite is deliberately not added to the integration
matrix here. Its files and its testing/ci-local.sh registration arrive
in the next commit, and the GitHub workflow's matrix entry now goes with
them, so both runners gain the suite together and
testing/check-ci-parity.sh holds at every point in this history.

docs: document dynamic interface binding

The transport-layer design gains an Interface Presence section: the three
deployment scenarios that motivated it, the presence machine and its two
invariants, why presence is IFF_UP and not IFF_UP|IFF_RUNNING, interface
identity and the recreated-netdev case, the detection sources and their two
rate limits, the optional policy, health as a level rather than a latch, the
egress-MTU consequence, the logging rules, and what the mechanism retires.
It records why the error deadline is a single ten-second window rather than
a repeating severity ladder, so the ladder does not come back.

The control-socket and fipsctl references described show_transports without
its `interface` block, and the transport-layer state machine still implied
that Up meant bound. Both now say what Up and presence each describe and how
they differ, since "transport up, interface absent" is a normal state an
operator will meet and would otherwise read as a contradiction.

The configuration reference documents `optional` with the absence, log and
retry behaviour of each setting, and the ground-up tutorial's dongle example
gains optional: true, which is what that example is actually for.

fix(transport): pair every presence edge the binder publishes

Six defects, all in the wiring around the presence machine rather than in
the machine itself. The state machines were well tested; what went untested
was how the binder feeds them, and every one of these lives there.

**The first detach after a clean start never reached node health.**
start_async binds inline and publishes that edge itself, before binder_loop
exists. The loop then built a fresh ChurnGuard, which had therefore recorded
no bind — and detached() takes `announced` to decide whether an edge is
owed, so it asked for no retraction. A node that booted with its interface
present and then lost it kept reporting Full for as long as the interface
stayed away, while show_transports said absent and the log said detached.
stabilized() could not repair it either: it early-returns while bound_at is
None, which it is for a bind the loop did not perform, so the guard stayed
unseeded until a second detach happened to fix it. The loop now seeds the
guard from the binding it inherits.

This is not the cable-unplug case. Presence is IFF_UP, so a carrier loss is
correctly not a detach at all; it takes an admin down, a netdev delete, or a
device removal — a wifi reload, a hostapd restart, a dongle pulled.

**An optional interface published nothing, and the MTU floor rode the same
channel.** Filtering the edge at the source conflated two questions that
happen to share a transport: whether node health should move, and whether
the bound set changed. Only the first is policy. The second determines the
node's egress MTU floor, and refresh_tun_mss_ceiling has no other trigger —
so an optional transport binding or unbinding at runtime left the TUN MSS
clamp derived from a bound set that no longer existed, reporting one
effective MTU and clamping to another. That is the defect the MssCeiling
work was written to remove, reintroduced for exactly the transports the
shipped OpenWrt config marks optional: five of seven. TransportPresence
gains health_relevant; the edge always goes out and only health is filtered.

**A permanent bind fault went unlogged if any absence race preceded it.**
record_attempt ran ahead of the InterfaceUnavailable arm, so a probe that
won a race the bind then lost burned the counter that gates the only error
a real fault ever gets — emitted on attempts == 1. A flapping interface
followed by CAP_NET_RAW being dropped or /dev/bpf* exhausting therefore
reported nothing at all, for the life of the process, and the operator got
report_sustained_absence's "still missing" instead: wrong, for an interface
that is present. An absence race is not a bind attempt and no longer counts
as one.

**The watcher's give-up did not stick.** Abandoning the event source parked
on pending() *inside the changed() future*, and that future is constructed
fresh on every pass of the binder's select! and dropped whenever the poll
ticker wins. So the next pass re-read the dead socket, re-counted the error
and re-logged "not recoverable" — once per wake-up, forever, which at a 1 s
tick across the shipped seven-transport config is seven warnings and seven
failing syscalls a second on flash-backed logging. The 100 ms backoff lived
in the dropped future too and never applied across passes. Abandonment is
now state on the watcher.

**A zero-length read livelocked the binder.** try_io clears readiness only
on WouldBlock, which the sibling error arm handles by hand and this one did
not — so breaking out left readable() instantly ready with nothing to read,
and the loop never returned Pending. That starves the select! of its ticker
entirely and takes presence detection down with it. Cleared and treated as a
fault so the give-up path applies.

**A failed presence probe read as an absent interface.** getifaddrs is a
netlink dump and fails for reasons that have nothing to do with the
interface: ENOBUFS under memory pressure, EMFILE or ENFILE under fd
exhaustion, since it opens a socket of its own. Answering "not present"
there tore down a working socket and degraded health over a transient
syscall failure, undiagnosably, and under fd exhaustion the rebind could not
have succeeded anyway. interface_has_flags now distinguishes the two; the
detach gate holds its binding when the kernel will not answer, while the
bind gate still treats it as absence and retries.

**On macOS the reader thread could not die, so a dead socket read as live.**
Any read() failure other than EBADF — ENXIO being the one that matters,
which is what BPF answers once the interface it was attached to is torn away
— reset the parse buffer and continued. The thread never returned, so the
channel never closed, so recv_from never failed, so the tokio task never
exited, so tasks_alive() reported a dead socket as a live one. Detach
detection on macOS reduced to the name and the index, and an interface reset
in place left the transport present and deaf. It now gives up after a
streak, and the return closes the channel the binder is actually watching.

Two smaller pairings while here: stop_async retracts the edge it would
otherwise leave standing for a socket that is gone, and start_async hands a
refused edge to the binder to retry rather than dropping the only edge
either consumer will see until the interface next moves. The absence
deadline is stamped at start rather than at construction, so a transport
staged for longer than the bring-up window no longer reports sustained
absence on its first tick having given the interface no window at all.

feat(config): reject impossible ethernet interface names at load

Config::validate never inspected transports.ethernet, so an empty interface
name, a name past the kernel's 15-byte limit, and two transports naming the
same netdev all loaded cleanly and failed only at runtime — the first two as
a permanent absence indistinguishable from an interface that has not been
created yet.

That indistinguishability is deliberate and worth keeping: waiting is the
right answer for an interface the operator has not made yet, and the daemon
cannot know which of the two it is looking at. Which is exactly why the
syntactic gate earns its place. A name that is *impossible* is the one case
still separable from "not there yet", and without the check a typo costs a
permanently Degraded node whose only symptom is an interface that never
arrives — the failure mode the presence machine exists to make legible,
reintroduced one level up.

Syntax only. Whether a well-formed name exists stays the binder's question,
asked once a second, forever.

The duplicate check is a different fault: two transports on one netdev means
two sockets on the same device at the same ethertype, each receiving every
frame the other does.

test(transport): close the vacuous and uncovered branches

Six tests that asserted nothing, or asserted less than they claimed.

`a_bind_fault_still_fails_the_start` returned early whenever the socket
*could* be opened, so it was vacuous as root and on any developer machine with
a group-readable /dev/bpf* — the fail-fast path it is named for went unchecked
exactly where someone was most likely to run it. It now asserts in both
halves, and the privileged half is worth more than the fix: a present,
bindable interface binding inline is the ordinary case on a booted router, and
no other unit test reaches it, because every other one here names an interface
that does not exist. The bind-success path had no unit coverage at all.

`an_interface_with_no_addresses_is_still_present` returned early when the
fixture was absent — no fixture, pass. It still has to skip on a machine with
no address-less interface, so the guard is the runner declaring that it has
fixtures: CI now sets FIPS_TEST_REQUIRE_FIXTURES beside
FIPS_TEST_ADDRLESS_IFACE, and the test fails rather than skips if the fixture
step is ever removed or renamed. Its `let _ = interface_carrier(...)` is now
asserted too: if presence and carrier ever collapsed into one read, a
carrier-less bridge would report absent and the whole IFF_UP-not-IFF_RUNNING
decision would be silently undone.

`policy_labels` checked `Required.as_str()` and not `Optional.as_str()`, so a
swapped pair would paint every expected interface as the tolerated kind and
stay green. `Presence::as_str` had no test at all — `binding` was never
observed by anything, anywhere.

Three binder branches had no coverage: a transport restarting (the second
`start_async` clearing the previous run's stop flag — only a second *stop* was
tested, so a transport that could never restart passed everything), the
episode clock being restamped at start rather than at construction, and the
refused-edge retry actually delivering. The last one matters most: the
existing test filled the channel and dropped the receiver, so a slot that
captured an edge and never re-sent it would pass while health sat on a stale
level forever.

And the hardware-change boundary `record_bind` returns, which the neighbour
flush hangs off: false on a first bind (or every clean start would drop a
cache it had just built) and true once, not stickily, on a MAC change.

Each new test was verified against the defect it guards — comment out the
`shutdown.store(false)`, the `mark_starting()`, or the seeded `unpublished`,
and the corresponding test goes red while the rest stay green.

fix(test): correct the dummy-carrier assertion, and pin Darwin's presence probe

The carrier assertion added in d6240698 was wrong and would have failed both
Linux legs. A `dummy` interface brought up reports `<BROADCAST,NOARP,UP,
LOWER_UP>` — `IFF_RUNNING` is set, so it *has* carrier. Verified against the
exact fixture CI builds, `addrgenmode none` and all, rather than against the
comment: the original code discarded the result and its comment claimed "up
but not running", which is what made asserting it look safe.

So the fixture pins address-less *presence* and cannot demonstrate the
presence-vs-carrier split at all — an interface up with no carrier is a bridge
with nothing plugged in, which no fixture here creates.
`carrier_is_reported_separately_from_presence` pins that split from the other
side. The corrected assertion is Linux-only, because the expected answer is a
property of the fixture device rather than of the code.

That is also what lets the macOS fixture land. The Linux legs pin the
address-less contract on glibc and musl, but the BSD-derived `getifaddrs` the
macOS backend actually calls had no coverage — the test skipped itself
silently on that runner, which is precisely the shape the previous commit was
removing. `feth` is macOS's fake-Ethernet pseudo-interface and is created
address-less; the step fails the leg rather than testing the wrong thing if the
runner hands it an address anyway, mirroring why the Linux step needs
`addrgenmode none`.

Both branches of the fixture guard were exercised: unset skips and passes, and
declared-but-missing fails loudly. The corrected test was run against a real
Linux dummy inside a container, not reasoned about.

fix(ci): put the macOS presence fixture on the macOS job

The `feth` fixture step added in 6b9faa0c landed on the Linux `test` job, not
on `test-macos`, and has failed CI ever since:

    create: Host name lookup failure
    ifconfig: `--help' gives usage information.

That is Linux net-tools `ifconfig`, which has no `create` subcommand. So the
Linux job ran two address-less-fixture steps — its own correct `ip link add
type dummy` one, then a macOS one that cannot work there — while `test-macos`
had none at all, leaving the Darwin `getifaddrs` path exactly as uncovered as
before.

Cause was a pattern-anchored edit: `Install cargo-nextest` followed by `Run
unit tests` appears in three jobs, and the insert hit the first match. Moving
it then hit the *last* match, which is `test-windows`. It is now placed by job
boundary rather than by pattern, and verified per job: `test` and `test-musl`
carry the `ip link`/dummy fixture, `test-macos` carries `ifconfig`/`feth`, and
`feth` appears exactly once in the file.

The verification that missed this was counting steps in `test-macos` and
reading 6 as confirmation. Six was the count *before* the insert; seven is what
a successful insert looks like. Now asserted by job and by which tool each
fixture step uses, so a step in the wrong place fails the check rather than
matching a total.

test(transport): cover the detach branch and the stop-race check

Two of the three untested branches, by two different routes.

**The detach decision is now a pure function.** `classify_detach(gone,
replaced, dead)` replaces the inline three-way `if` in the binder loop, and all
eight input combinations are asserted, plus the precedence between them and the
`reason=` labels the integration suite and operators grep for.

This does not make `Replaced` or `SocketDied` reachable from a test — both need
a bind that succeeded and then a specific external event, and the integration
suite cannot arrange the recreate deterministically either, for the reason
recorded in reference/notes.md: the netlink event from a delete is acted on
within microseconds, so `Gone` wins that race in practice. What it does is
split the untested thing in two. The three inputs each already had tests
(`interface_present`, `device_replaced`, `tasks_alive`); the branch between them
did not, and that half is now total and exhaustive.

Precedence is asserted rather than assumed: an interface that has gone away has
also trivially been "replaced" and its socket is also dead, so the most
specific true statement has to win, and callers pass `replaced` already masked
by `!gone` — the function no longer depends on them having masked correctly.

**The post-store shutdown check is now tested directly.** It needs a bind that
*succeeds*, which is why `a_stop_racing_a_bind_leaves_nothing_behind` could
never reach it: that test's interface does not exist, so `bind_and_spawn`
refuses at the presence probe several steps earlier. Binding loopback as root
reaches it, and the assertion is that a bind completing after a stop undoes its
own socket, its own loops, and its own presence.

Unprivileged runners skip it, but loudly: a runner declaring
`FIPS_TEST_PRIVILEGED` and unable to open a raw socket fails instead of
skipping, the same guard shape as `FIPS_TEST_REQUIRE_FIXTURES`. No CI leg sets
that yet — unit tests run unprivileged — so today it exercises the branch only
under a root container. Verified there, including the negative control: with
the post-store check removed the test fails, and with it present it passes.

test(transport): run the bind-success half in CI, and cover the replaced device

Adds the privileged unit-test step the rest of this depends on, then uses it.

**The privileged step.** Every unit-test leg runs unprivileged, so
`PacketSocket::open` cannot succeed on any of them and everything past a
successful bind runs nowhere in CI: the post-store shutdown check, the
`Present` arm of the binder loop, `bind_now` itself. The step builds as the
runner user and executes only the test binary under sudo — running `cargo` as
root would use root's CARGO_HOME and discard the cache the job just restored.
The binary-path extraction was verified locally before being written into the
workflow.

`FIPS_TEST_PRIVILEGED` is what makes it honest. Tests needing a raw socket skip
quietly without it; with it set, a test that cannot open one fails and says so.
A runner that stops granting the capability shows up as a red leg rather than
as silence.

**The replaced device.** `"interface replaced"` — a netdev recreated under the
same name, the `wifi reload` case in #125 — could not be reached by the
integration suite: with link events live the kernel's `RTM_DELLINK` is acted on
within microseconds, so the binder observes "gone" first and takes the branch
already covered. Attempting it there passed about one run in three.

Rather than race the binder, the test asserts the *inputs*: after a real
delete-and-recreate the name still resolves and the bound index no longer
matches. Paired with the exhaustive classifier test, which pins that
`(gone: false, replaced: true)` maps to `Replaced`, the path is covered without
depending on scheduling. A watcher-disable hook was written for the racing
approach and removed once this one made it unnecessary.

**Two smaller gaps.** `report_sustained_absence` had no direct test — the
integration suite infers it from counting ERROR lines, and ten seconds of real
time is how a deadline ends up asserted by proxy. A test-only `backdate_for_test`
ages the episode clock instead. And the probe floor was asserted only as a
relation between two constants; it now has behaviour, via a `probe_delay`
helper extracted from the loop.

That extraction was not neutral, and its own test caught it: `checked_sub`
yields `Some(0)` at exact equality where the loop used a strict `<`, so a
zero-length sleep would have replaced no sleep at all. Harmless in effect,
wrong in meaning, and fixed.

Every new test was run against the defect it guards, in a root container:
remove the post-store shutdown check, or make `device_replaced` always answer
false, and the corresponding test fails.
Mark publish_presence must_use. The defect this commit fixes was a discarded
return value, so the fix is made self-guarding: a future call site that drops
the pending edge instead of storing it now fails the lint rather than silently
reintroducing the bug. Option is not must_use in std, unlike Result, so the
attribute has to be explicit.
This commit is contained in:
Arjen
2026-09-10 19:18:09 +00:00
committed by Johnathan Corgan
parent 268e529125
commit abe3f0b40a
24 changed files with 5370 additions and 338 deletions
+162
View File
@@ -284,6 +284,30 @@ jobs:
- name: Install system dependencies
run: sudo apt-get update && sudo apt-get install -y libdbus-1-dev
# The same address-less interface the musl leg creates. Pinning the
# contract on both libcs is what turns "glibc and musl agree about
# `getifaddrs`" from an assumption into a checked fact — and makes this
# leg fail first if glibc is the one that changes.
- name: Create an address-less interface for the presence probe
run: |
sudo ip link add fips-probe0 type dummy
# `addrgenmode none` before bringing it up: the kernel hands an IPv6
# link-local to any interface that comes up, and an interface with a
# link-local is not address-less — the fixture would have quietly
# tested nothing.
sudo ip link set fips-probe0 addrgenmode none
sudo ip link set fips-probe0 up
ip addr show fips-probe0
# Fail rather than test the wrong thing if it acquired one anyway.
if ip addr show fips-probe0 | grep -qE "inet6? "; then
echo "fips-probe0 has an address; it cannot test the address-less case" >&2
exit 1
fi
echo "FIPS_TEST_ADDRLESS_IFACE=fips-probe0" >> "$GITHUB_ENV"
# Declare that this runner has fixtures, so a test that depends on
# one fails when the fixture is missing instead of skipping silently.
echo "FIPS_TEST_REQUIRE_FIXTURES=1" >> "$GITHUB_ENV"
- name: Install Rust toolchain
uses: actions-rust-lang/setup-rust-toolchain@166cdcfd11aee3cb47222f9ddb555ce30ddb9659 # v1
with:
@@ -304,9 +328,39 @@ jobs:
- name: Install cargo-nextest
uses: taiki-e/install-action@nextest
- name: Run unit tests
run: cargo nextest run --all --profile ci
# The bind-success half. Every other unit-test leg runs unprivileged, so
# `PacketSocket::open` cannot succeed on any of them and everything past
# a successful bind — the post-store shutdown check, the `Present` arm of
# the binder loop, `bind_now` itself — runs nowhere in CI.
#
# Built as the runner user and only *executed* under sudo: `cargo` run as
# root would use root's CARGO_HOME and discard the cache this job just
# restored.
#
# `FIPS_TEST_PRIVILEGED` is what makes this leg honest. A test that needs
# a raw socket skips quietly without it; with it set, a test that cannot
# open one fails and says so, so a runner that stops granting the
# capability shows up as a red leg rather than as silence.
- name: Run interface-binding tests with privilege
run: |
cargo test --lib --no-run
BIN=$(cargo test --lib --no-run --message-format=json \
| jq -r 'select(.reason == "compiler-artifact")
| select(.executable != null)
| select(.target.kind[0] == "lib")
| .executable' \
| tail -1)
if [ -z "$BIN" ] || [ ! -x "$BIN" ]; then
echo "could not locate the lib test binary" >&2
exit 1
fi
echo "running $BIN as root"
sudo -E env FIPS_TEST_PRIVILEGED=1 "$BIN" transport::ethernet --test-threads=1
- name: Publish test report (Checks tab)
uses: dorny/test-reporter@4a2e97665d5fa767581ef38eca97b9694bd4eef4 # v2
if: always()
@@ -363,9 +417,116 @@ jobs:
- name: Install cargo-nextest
uses: taiki-e/install-action@nextest
# The Darwin half of the address-less presence contract. The Linux legs
# pin that `getifaddrs` reports an interface with no addresses as
# present, on both glibc and musl; without this the same claim on the
# BSD-derived implementation the macOS backend actually calls was
# untested, and the test skipped itself silently on this runner.
#
# `feth` is macOS's fake-Ethernet pseudo-interface. It is created
# address-less, and the check below fails the leg rather than testing the
# wrong thing if this runner hands it one anyway — the same shape as the
# Linux fixture step, which needs `addrgenmode none` for exactly that
# reason.
- name: Create an address-less interface for the presence probe
run: |
sudo ifconfig feth0 create
sudo ifconfig feth0 up
ifconfig feth0
if ifconfig feth0 | grep -qE "^[[:space:]]*inet6? "; then
echo "feth0 has an address; it cannot test the address-less case" >&2
exit 1
fi
echo "FIPS_TEST_ADDRLESS_IFACE=feth0" >> "$GITHUB_ENV"
# Declare that this runner has fixtures, so a test that depends on
# one fails when the fixture is missing instead of skipping silently.
echo "FIPS_TEST_REQUIRE_FIXTURES=1" >> "$GITHUB_ENV"
- name: Run unit tests
run: cargo nextest run --all --profile ci
# ─────────────────────────────────────────────────────────────────────────────
# Job 2bb – Unit tests (musl)
#
# OpenWrt — the platform the Ethernet transport's dynamic interface binding
# exists for — is musl, and musl reimplements the libc calls that binding is
# built on rather than sharing glibc's. `interface_present` reads `ifa_flags`
# out of `getifaddrs`, and the interfaces it has to see (`fips-mesh0`,
# `fips-ap0`) are deliberately unbridged with no IP address at all, which is
# exactly where getifaddrs implementations differ. Every other leg is glibc, so
# without this one the presence probe is asserted on a libc no test has ever
# run it against, on the target it was written for.
#
# Built for the musl target on a glibc host rather than inside an Alpine
# container. The test binary links musl statically and runs natively on the
# runner, so musl's `getifaddrs` is the one under test — while the build
# scripts stay host artifacts, which keeps rustables' bindgen on the same
# libclang the glibc leg already builds with. Building inside Alpine put
# bindgen on a musl toolchain it does not work on: statically linked build
# scripts cannot `dlopen` libclang, and turning the static CRT off then left
# it loading libclang but unable to parse. None of that is anything this leg
# is trying to test.
# ─────────────────────────────────────────────────────────────────────────────
test-musl:
name: Unit tests (musl)
runs-on: ubuntu-latest
needs: [build]
steps:
- uses: actions/checkout@d23441a48e516b6c34aea4fa41551a30e30af803 # v6
- name: Set SOURCE_DATE_EPOCH from git
run: echo "SOURCE_DATE_EPOCH=$(git log -1 --format=%ct)" >> "$GITHUB_ENV"
# libdbus for the host build scripts; musl-tools for the musl C
# toolchain the `cc`-driven dependencies link against. BLE is excluded on
# musl by a Cargo.toml cfg, so bluer is not in this build at all.
- name: Install system dependencies
run: sudo apt-get update && sudo apt-get install -y libdbus-1-dev musl-tools
# The address-less interface the presence probe has to be tested
# against; see the matching step on the glibc leg for why loopback
# cannot stand in for it.
- name: Create an address-less interface for the presence probe
run: |
sudo ip link add fips-probe0 type dummy
# `addrgenmode none` before bringing it up: the kernel hands an IPv6
# link-local to any interface that comes up, and an interface with a
# link-local is not address-less — the fixture would have quietly
# tested nothing.
sudo ip link set fips-probe0 addrgenmode none
sudo ip link set fips-probe0 up
ip addr show fips-probe0
# Fail rather than test the wrong thing if it acquired one anyway.
if ip addr show fips-probe0 | grep -qE "inet6? "; then
echo "fips-probe0 has an address; it cannot test the address-less case" >&2
exit 1
fi
echo "FIPS_TEST_ADDRLESS_IFACE=fips-probe0" >> "$GITHUB_ENV"
# Declare that this runner has fixtures, so a test that depends on
# one fails when the fixture is missing instead of skipping silently.
echo "FIPS_TEST_REQUIRE_FIXTURES=1" >> "$GITHUB_ENV"
- name: Install Rust toolchain
uses: actions-rust-lang/setup-rust-toolchain@166cdcfd11aee3cb47222f9ddb555ce30ddb9659 # v1
with:
cache: false
rustflags: ''
target: x86_64-unknown-linux-musl
- name: Cache Cargo registry + build
uses: actions/cache@caa296126883cff596d87d8935842f9db880ef25 # v5
with:
path: |
~/.cargo/registry
~/.cargo/git
target
key: musl-cargo-${{ hashFiles('**/Cargo.lock') }}
restore-keys: |
musl-cargo-
- name: Run library tests
run: cargo test --lib --target x86_64-unknown-linux-musl
# ─────────────────────────────────────────────────────────────────────────────
# Job 2c – Unit tests (Windows)
# ─────────────────────────────────────────────────────────────────────────────
@@ -395,6 +556,7 @@ jobs:
- name: Install cargo-nextest
uses: taiki-e/install-action@nextest
- name: Run unit tests
run: cargo nextest run --all --profile ci
+226 -58
View File
@@ -7,6 +7,232 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
## [Unreleased]
### Added
- Dynamic interface binding for the Ethernet transport. An interface-bound
transport is now a long-lived object that is *sometimes bound*: the interface
it names need not exist when the daemon starts, may appear minutes later, and
may vanish and return mid-operation. `start_async` returns `Ok` with the
transport **absent** rather than failing, and a per-transport binder task
binds when the interface appears, unbinds when it goes away, and rebinds when
it returns. Start-time absence and runtime detach are one code path.
Detection is event-driven where the kernel offers a source — netlink
`RTNLGRP_LINK` on Linux, `PF_ROUTE` on macOS and FreeBSD — with a 1 s
`getifaddrs` poll underneath as a backstop. Presence means
`IFF_UP` — the interface exists and is administratively up — and
deliberately not `IFF_RUNNING`. Binding needs no carrier and a socket
outlives a carrier flap, so a bridge with nothing plugged into it (`br-lan`
on a wifi-only router) is bound and healthy rather than permanently
`Degraded`, and starts carrying traffic the moment a port comes up. Whether
an interface has carrier is reported separately as `interface.carrier` in
`show_transports`, never acted on.
This closes the OpenWrt boot race (procd starts `fips` before wifi has
created `fips-mesh0` / `fips-ap0`; both transports were skipped for the life
of the process while the 802.11s peer link formed anyway, so the node looked
healthy and reached nothing), the intermittent-adapter case, and the
mid-operation `wifi reload` that destroyed and recreated an interface under a
live socket.
- `transports.ethernet.*.optional` (bool, default `false`). Naming an interface
in configuration is a statement that you expect it, so the default is to
complain: while a required interface is missing the node reports `Degraded`
and logs the edge, at a severity that follows how long the absence lasts
(see below). `optional: true` makes absence silent (`info` on the edge, no
health impact) for hardware that is legitimately not always there. It describes the interface's *presence*, not the transport's
importance — an optional interface that is present is used exactly as hard as
any other — and no value of it makes a missing interface fatal at startup.
- `fipsctl show transports` reports interface presence per transport under a new
`interface` block: `name`, `presence` (`absent` / `binding` / `present`),
`carrier`, `policy` (`required` / `optional`), `since_secs`, `binds` and
`failed_attempts`. The original boot-race bug was expensive precisely because
nothing an operator could see said the node was deaf.
- `testing/iface-binding/` integration suite (`ci-local.sh --only
iface-binding`, and a GitHub matrix leg): two daemons whose only transports
are interface-bound, run against a veth pair the harness creates, downs,
deletes and recreates underneath them. Asserts the boot race, the late
attach and peering over it, the flap in both directions,
destroy-and-recreate, that an `optional` interface never moves node health,
and that absence is logged once on the edge rather than once per retry.
#### Node lifecycle
- Transport-medium change detection, controlled by the new `node.netmon.*`
block (on by default). For each peer whose transport address is a numeric IP
endpoint, the node asks the kernel which local address it would use to reach
*that peer* — a `connect(2)` on a UDP socket, which resolves the route and
sends nothing — and reports a change once some peer held across two
consecutive samples is reached from a different local address, or has stopped
being reachable at all. Asking the question per peer rather than about the
host is what keeps it quiet: a container bridge, a VPN, a `veth` pair or a
tunnel appearing is not the route to any peer and cannot move the
fingerprint, while a peer on the same LAN — reached by its subnet route, not
the default route — is covered, as is a more specific route moving under a
single peer. Peers joining and leaving are ignored on their own, being
ordinary node behaviour rather than a statement about the medium — except
that a peer seen for the first time is checked against its own
`connect()`-ed socket, and reported if that socket is pinned to a source the
routing table would no longer choose, so a medium change in the window
between a peer authenticating and the next sample is not adopted silently
while that peer sits stranded on the old path. A peer
addressed by MAC, by `.onion` or Nym recipient, by a scoped IPv6 literal, or
by a hostname it has not yet been heard from on, has no route to ask about
and contributes nothing; a node with no peers detects nothing, having nothing
bound to the old path to repair. The peer table is read through the node's
existing lock-free entity snapshot, so the detector stays a detached task
holding no node state. A handover is not atomic (the route goes, briefly
there is none, the new one arrives), so a short debounce coalesces the burst
into one event and a fingerprint that settles back where it started reports
nothing. Linux subscribes to `NETLINK_ROUTE`
multicast (the groups `ip monitor` uses) and macOS and FreeBSD to a
`PF_ROUTE` socket, both reacting to the kernel event in milliseconds; every
other platform samples on a timer at `node.netmon.poll_interval_secs`, which
also runs underneath the kernel sources as a backstop, since a netlink socket
drops messages under memory pressure and the subscription can be refused in a
restricted sandbox. A backend only decides *when to look* — the fingerprint
comparison, the debounce and the settled-back suppression are shared — so the
remaining backends (`NotifyIpInterfaceChange` on Windows, an embedder push on
iOS) land behind the same seam without touching the reaction. Android takes
the netlink source, and falls back to the timer where policy refuses the
group bind. What the node does with the signal is the connected-socket
rebind described under Fixed above.
Bluetooth is not covered: an adapter's state is not an IP attachment and is
invisible to this detector.
**Upgrade note: this couples a new key to one that has already shipped.** A
handover is ridden out for up to eight settling rounds of
`node.netmon.debounce_ms` before a change is reported, and if that worst case
reaches `node.link_dead_timeout_secs` the reaper tears the peering down
before the change is ever acted on, so the node refuses to start rather than
run in that shape. At the shipped defaults the margin is wide (8 × 250ms = 2s
against 30s), but detection is on by default, so **a node that shortened
`node.link_dead_timeout_secs` to 1 or 2 seconds for fast failover will be
refused at startup after the upgrade**, naming a `node.netmon.*` key its
operator never set. Raise the timeout, lower `debounce_ms` so eight rounds
stay under it, or set `node.netmon.enabled: false`. A
`node.link_dead_timeout_secs` of 0 is exempt from the check.
#### Library surface and internals
- `TransportError::InterfaceUnavailable { interface }`. A missing interface and
a typo'd interface name were previously the same flat
`StartFailed(String)`; nothing downstream could branch on absence.
### Changed
- `Degraded` is now a level rather than a latch. The supervisor's reason set
was monotonic, which was correct while no child could recover; with recovery
it would have meant "something broke at some point since boot" rather than
"something is broken now". Interface absence is tracked in its own reversible
set and node health is recomputed on every transition **in both directions**,
so plugging the WAN back in clears `Degraded` without a restart. A transport
whose interface is absent still counts as up, so a single-interface node that
boots before its wifi degrades rather than exiting on "no transports".
- The OpenWrt package ships the `mesh0`/`mesh1` and `ap0`/`ap1` Ethernet
transports **enabled** with `optional: true`, instead of commented out.
`fips-mesh-setup` and `fips-ap-setup` no longer comment-toggle blocks in
`fips.yaml`, and no longer tell the operator to restart the daemon after
creating an interface — the daemon binds it on its own. `phy0-sta0` (`wwan`)
is marked `optional: true` for the same reason: it only exists while a radio
is in station mode.
- The Ethernet receive loop backs off and exits on a dead socket instead of
spinning on `Err` with a `warn!` per iteration, and the ad-hoc ENXIO
socket-reopen in the beacon sender is gone. Both hand recovery to the
presence machine: one mechanism for every cause rather than one hack per
symptom. Beacons pause while an interface is absent.
- The absence edge is not itself an error, and there is exactly one deadline
after it. An interface missing when the daemon starts logs at `info` — that
is the boot race the mechanism exists to absorb, not a fault — and a runtime
detach at `warn`, because a link coming and going is ordinary weather for a
mesh daemon. Ten seconds is the whole grace: past it, absence is no longer a
race against a radio or a container, so a **required** interface still
missing is reported once at `error`. Start-time absence and a runtime detach
share that one deadline rather than getting one each. An `optional`
interface never reaches `error`. Node health does not wait for any of it,
publishing `Degraded` on the first edge either way.
- The TUN boundary's TCP MSS clamp now tracks the node's egress MTU at
runtime instead of freezing it at startup. `transport_mtu()` is the minimum
across *bound* transports, so a transport that binds minutes after start can
be the narrow one — but the TUN reader and writer were handed a `u16`
computed once when they spawned, while every other consumer
(`show_status`, the control-socket snapshot, the session-layer fragmentation
check) read it live. A node could therefore report one effective IPv6 MTU
and clamp to another. The ceiling is now shared with those threads and
recomputed whenever the bound set changes, in both directions: a narrow
interface appearing tightens it, and its departure releases it. MSS is
negotiated per connection, so a change applies to connections opened after
it; existing ones are not disturbed.
- Rebinds that keep succeeding into a socket that dies moments later are
damped: consecutive bindings shorter than ten seconds back off on the
1 s → 30 s curve, and past three of them the binder stops announcing each
bind as a recovery until one lasts. Undamped, a persistently broken socket
behind a healthy interface produced a log pair and a `Degraded`→`Running`
health flap every second.
- A bind failure that is **not** absence — no `CAP_NET_RAW`, no readable
`/dev/bpf*`, a buffer the kernel refused — fails the daemon's start as it
always has, rather than being waited out. It is a fault, not a state, and
will not resolve on its own; only a missing interface is retried at start.
A non-absence failure during a later rebind still backs off, since the node
is serving by then.
- The binder cannot outlive its transport, and a teardown that races a bind
cannot leave a live receive loop on a socket nothing owns. A shared stop flag
is raised before teardown and checked by the binder after it stores a
binding, so whichever order the two interleave exactly one of them cleans up;
`EthernetTransport` gained a `Drop` that raises the flag, aborts the binder
and releases the socket, for handles dropped without `stop_async`.
- Presence edges are published with `try_send` and retried on the next tick
rather than awaited. A bounded channel could previously park the binder
mid-publish — a health channel able to deadlock the machine whose health it
carries — freezing the interface in whatever state it held.
- `TransportHandle::is_bound()` joins `is_operational()`: the latter means the
transport was *started*, which for an interface-bound transport no longer
implies a live socket. `Node::transport_mtu` now filters on the former,
because an interface that has never existed was clamping the whole node's
IPv6 MTU to a number derived from absent hardware.
- An interface deleted and recreated under the same name is detected as a
detach. Both backends bind by device rather than by name, so the old socket
is attached to nothing while the name still resolves — and a stale
`AF_PACKET` socket never becomes readable, so nothing errors and nothing
exits. Detection previously rested entirely on the beacon sender failing,
which a node with `announce: false` does not have. The bound interface index
is now captured at bind and compared on every poll.
- The link-event watcher distinguishes a genuine receive error from
`WouldBlock`. `try_io` clears readiness only on the latter, so a persistent
error — `ENOBUFS` after a burst of link events overflows the socket buffer —
span a core flat with nothing logged. Errors are now counted, logged once,
backed off, and after five the source is abandoned for the presence poll.
- Presence probes are coalesced to at most ten a second. Linux netlink is
filtered to `RTNLGRP_LINK`, but `PF_ROUTE` has no group filter, so the macOS
source delivers every routing message on the host — route churn, ARP, DHCP
renewals, a VPN going up and down — and each would otherwise drive a full
`getifaddrs` walk.
- CI runs the library tests on musl (Alpine) as well as glibc. Presence is
built on `getifaddrs` and `ifa_flags`, musl reimplements both independently,
and the interfaces this feature exists for (`fips-mesh0`, `fips-ap0` on
OpenWrt) are unbridged with no IP address at all — the case where
implementations most plausibly differ. It was previously asserted on a libc
no test had ever run it against, on the target it was written for.
- Interface presence state ignores lock poisoning. Treating a poisoned lock as
a failure meant reading "no socket, tasks dead", which is the destructive
direction: a transport reporting itself present while every send fails, or a
binder tearing down and rebinding every second while teardown silently
declined to abort anything.
### Fixed
#### Node lifecycle
@@ -84,64 +310,6 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
platforms with the connected-socket fast path); elsewhere the heartbeat alone
carries the new address.
### Added
#### Node lifecycle
- Transport-medium change detection, controlled by the new `node.netmon.*`
block (on by default). For each peer whose transport address is a numeric IP
endpoint, the node asks the kernel which local address it would use to reach
*that peer* — a `connect(2)` on a UDP socket, which resolves the route and
sends nothing — and reports a change once some peer held across two
consecutive samples is reached from a different local address, or has stopped
being reachable at all. Asking the question per peer rather than about the
host is what keeps it quiet: a container bridge, a VPN, a `veth` pair or a
tunnel appearing is not the route to any peer and cannot move the
fingerprint, while a peer on the same LAN — reached by its subnet route, not
the default route — is covered, as is a more specific route moving under a
single peer. Peers joining and leaving are ignored on their own, being
ordinary node behaviour rather than a statement about the medium — except
that a peer seen for the first time is checked against its own
`connect()`-ed socket, and reported if that socket is pinned to a source the
routing table would no longer choose, so a medium change in the window
between a peer authenticating and the next sample is not adopted silently
while that peer sits stranded on the old path. A peer
addressed by MAC, by `.onion` or Nym recipient, by a scoped IPv6 literal, or
by a hostname it has not yet been heard from on, has no route to ask about
and contributes nothing; a node with no peers detects nothing, having nothing
bound to the old path to repair. The peer table is read through the node's
existing lock-free entity snapshot, so the detector stays a detached task
holding no node state. A handover is not atomic (the route goes, briefly
there is none, the new one arrives), so a short debounce coalesces the burst
into one event and a fingerprint that settles back where it started reports
nothing. Linux subscribes to `NETLINK_ROUTE`
multicast (the groups `ip monitor` uses) and macOS and FreeBSD to a
`PF_ROUTE` socket, both reacting to the kernel event in milliseconds; every
other platform samples on a timer at `node.netmon.poll_interval_secs`, which
also runs underneath the kernel sources as a backstop, since a netlink socket
drops messages under memory pressure and the subscription can be refused in a
restricted sandbox. A backend only decides *when to look* — the fingerprint
comparison, the debounce and the settled-back suppression are shared — so the
remaining backends (`NotifyIpInterfaceChange` on Windows, an embedder push on
iOS) land behind the same seam without touching the reaction. Android takes
the netlink source, and falls back to the timer where policy refuses the
group bind. What the node does with the signal is the connected-socket
rebind described under Fixed above.
Bluetooth is not covered: an adapter's state is not an IP attachment and is
invisible to this detector.
**Upgrade note: this couples a new key to one that has already shipped.** A
handover is ridden out for up to eight settling rounds of
`node.netmon.debounce_ms` before a change is reported, and if that worst case
reaches `node.link_dead_timeout_secs` the reaper tears the peering down
before the change is ever acted on, so the node refuses to start rather than
run in that shape. At the shipped defaults the margin is wide (8 × 250ms = 2s
against 30s), but detection is on by default, so **a node that shortened
`node.link_dead_timeout_secs` to 1 or 2 seconds for fast failover will be
refused at startup after the upgrade**, naming a `node.netmon.*` key its
operator never set. Raise the timeout, lower `debounce_ms` so eight rounds
stay under it, or set `node.netmon.enabled: false`. A
`node.link_dead_timeout_secs` of 0 is exempt from the check.
### Changed
- The lockfile moves `chacha20` from 0.10.1 to 0.10.2, because 0.10.1 is yanked.
+307
View File
@@ -1010,6 +1010,313 @@ Transports begin in `Configured` state with all parameters set. `start()`
transitions through `Starting` to `Up` (operational). `stop()` moves to
`Down`. Transport failures move to `Failed`.
## Interface Presence
`Up` describes the *transport*, not the socket. An interface-bound transport
(today: Ethernet) carries a second, orthogonal state — whether it is bound
right now — and the two are independent: a transport is `Up` from the moment
it starts, whether or not the interface it names exists.
### The Gap This Closes
Three deployment scenarios exercise one missing mechanism:
- **Boot ordering.** On OpenWrt, procd starts `fips` before wifi has created
`fips-mesh0` / `fips-ap0`. Both transports were skipped and never retried,
while the 802.11s peer link formed anyway — that is mac80211, not the
daemon — so the node looked healthy and reached nothing. The failure was
expensive precisely because nothing an operator could see said the node was
deaf.
- **Intermittent hardware.** A USB ethernet adapter named in `fips.yaml` is
plugged in some days and not others. Its absence is normal and must be
silent; its arrival must bind without operator action.
- **Mid-operation restart.** `wifi reload` for a channel change destroys and
recreates the mesh interface within a couple of seconds. The socket dies,
the receive loop spun on `Err` with no backoff and no exit, and nothing
rebound.
These are not three features. They are one presence machine plus one policy
field. Before it existed, the first observation was final: an interface
missing at start was logged once and skipped for the life of the process, and
one that disappeared at runtime published a health change but was never
rebound.
### The Presence Machine
```text
Absent ──attach──> Binding ──ok──> Present
^ │ │
└──── fail/backoff ─┘ │
└──────────── detach ──────────────┘
```
`start_async` binds if it can and otherwise returns `Ok` with the transport
`Up` and `Absent`; a per-transport binder task then binds when the interface
appears, tears the socket down when it goes away, and rebinds when it
returns. Two invariants do the work:
- **The transport object survives detach.** Config, `TransportId`,
statistics and the neighbor buffer persist; only the file descriptor and
its loops go. A transport is never destroyed because its interface went
away.
- **Start-time absence and runtime detach are the same transition.** A node
that boots before its wifi and a node whose wifi reloads at 03:00 take one
code path. The old asymmetry — skip forever at start, publish health at
runtime — is gone.
`TransportError::InterfaceUnavailable` is what makes absence branchable. A
missing interface and a typo'd interface name were the same flat
`StartFailed(String)`, so nothing downstream could tell a state from a fault.
### What Counts as Present
Presence means `IFF_UP` — the interface exists and the operator has enabled
it — and deliberately **not** `IFF_RUNNING`.
Carrier and bindability are different questions, and only the second belongs
in a bind gate. An `AF_PACKET` socket on a carrier-less bridge is valid and
starts carrying traffic the instant a member port comes up, with no rebind:
the socket outlives the carrier. Gating on `IFF_RUNNING` bought nothing and
cost three things:
- `br-lan` on a router with nothing in its LAN ports is `UP` with
`NO-CARRIER`, so a healthy wifi-only router reported `Degraded` forever and
errored for a fault it did not have;
- every carrier flap the socket would have survived became an unbind/rebind
cycle — churn the presence machine then has to damp, a mechanism
compensating for a policy error;
- an 802.11s interface that reports `RUNNING` only once it has peered cannot
peer, because peering needs beacons, beacons need a bound socket, and the
gate refuses to bind. A deadlock reachable on the hardware this mechanism
was written for.
The signal `IFF_RUNNING` carries is not lost: `show_transports` reports
`interface.carrier` beside presence, so an operator can still tell a bound
transport carrying nothing from a working one. It is reported rather than
obeyed.
The probe is `getifaddrs` plus `ifa_flags` rather than an `SIOCGIFFLAGS`
ioctl: it needs no socket, so the watcher can probe before any file
descriptor exists, and it is spelled the same on Linux and the BSDs.
### Interface Identity
The configured name is the key, but a name is not a device. Both backends
bind by *device* — `AF_PACKET` stores `sll_ifindex`, a BPF descriptor follows
the interface it was attached to — so an interface deleted and recreated
under the same name leaves the socket attached to something that no longer
exists while the name resolves perfectly well.
Nothing else notices. A stale `AF_PACKET` socket never becomes readable, so
the receive loop neither errors nor exits, and send failures go to the caller
rather than to the binder. A listen-only node (`announce: false`, so no
beacon sender to fail) therefore sat `present` and deaf indefinitely after a
`wifi reload` — the original bug wearing a different hat. The bound index is
captured at bind and compared on every poll; a mismatch is a detach.
Hardware can also change underneath a name. If the name reappears with a MAC
other than the one last bound, that is a different device, so the cached
neighbor entries for that transport are dropped rather than resumed onto, and
the swap is logged at `warn`. Richer selectors (`match: { name | mac |
id_path }`) are deliberately deferred; the requirement here is only that FIPS
never silently resumes onto different hardware.
### Detection
| Platform | Source |
| -------- | ------ |
| Linux | netlink `RTNLGRP_LINK` (`RTM_NEWLINK` / `RTM_DELLINK`) |
| macOS, FreeBSD† | `PF_ROUTE` socket, `RTM_IFINFO` |
| Fallback | poll `getifaddrs` + flags, 1 s |
† Aspirational: the Ethernet transport is
`cfg(any(target_os = "linux", target_os = "macos"))`, so FreeBSD has no
interface-bound transport for a watcher to serve. The `PF_ROUTE` branch
compiles for the BSD family, but only macOS reaches it.
Where an event source exists, detection is sub-second. The poll stays
underneath as a backstop rather than as the mechanism, and must stay at ~1 s:
the probe is cheap, and letting the interval drift to tens of seconds
reintroduces exactly the latency the event source was added to remove.
Construction is best-effort — a kernel or sandbox that refuses the socket
yields a watcher that never fires, and the binder degrades to its poll.
Link-event payloads are **not parsed**. An event is a hint to re-run the
presence probe, which is cheap and authoritative; decoding
`nlmsghdr`/`ifinfomsg` to reach the same answer would add a parser whose bugs
would be presence bugs.
Two rate limits protect the binder from its own event source. Probes are
coalesced to ten a second, because `PF_ROUTE` has no group filter and
delivers every routing message on the host — route churn, ARP, DHCP renewals,
a VPN going up and down — each of which would otherwise drive a full
`getifaddrs` walk. And a persistently failing event source is counted, logged
once, backed off, and after five consecutive errors abandoned for the poll:
losing events is survivable because the poll is the backstop, but burning a
core on a socket that is readable-but-erroring is not.
Bind failures that are *not* absence back off 1 s → 30 s. Absence itself does
not back off; there is nothing to poll but the probe.
### Policy: `optional`
One field per transport, `transports.ethernet.*.optional`, default `false`:
| | absence | log | retries |
| --- | --- | --- | --- |
| `optional: false` (default) | node reports `Degraded` | `info` at boot / `warn` on a runtime detach, then `error` once if it lasts past 10 s | forever |
| `optional: true` | no health impact | `info`, and nothing after | forever |
Naming an interface in configuration is a statement that you expect it, so
the default is to complain; silence is opted into.
`optional` describes **the interface's presence, not the transport's
importance**. An optional interface that is present is used exactly as hard
as any other. No value of it makes a missing interface fatal at startup: the
only fatal case remains "no transports at all came up". If a deployment ever
needs absence to abort startup, that arrives as an explicit `on_absent: exit`
— never as a second meaning for `optional`.
A bind failure that is not absence — no `CAP_NET_RAW`, no readable
`/dev/bpf*`, a buffer the kernel refused — is a fault, not a state, and still
fails the daemon's start. Retrying those forever would convert a hard,
actionable deployment error into a daemon that retries a socket it can never
open behind a `Degraded` nobody is watching. Only absence is waited out at
start; a non-absence failure during a later *rebind* does back off, since by
then the node is serving and killing it would be the worse answer.
### Health
Node health is recomputed on every presence transition **in both
directions**, so a returning interface clears `Degraded` without a restart.
That makes `Degraded` a level rather than a latch: the supervisor's reason set
was monotonic, which was correct while no child could recover, but with
recovery it would have come to mean "something broke at some point since boot"
rather than "something is broken now". Absence lives in its own reversible
set, separate from the one-way `failed` set a start failure enters.
An absent transport still counts as *up*. It came up — `start_async` returned
`Ok` — so it does not push a single-transport node into the fatal
`NoTransports`, which would make a node that merely booted before its wifi
exit instead of waiting. Absence degrades; it never kills.
There is deliberately **no restart action in the supervisor FSM.** The
presence watcher and the rebind loop live inside the transport, next to the
file descriptor they manage, and once that exists a supervisor-authored retry
has nothing left to do — it would be a second mechanism racing the first for
the same socket. The supervisor learns about presence
(`Event::ChildAbsent` / `Event::ChildPresent`) and republishes health; it does
not drive rebinding.
Peer state needs no separate grace period. A send over an absent interface
returns `InterfaceUnavailable` and the peer entry survives untouched, so peers
are already held across a detach and resume when the interface returns; the
liveness reaper is the effective linger bound. A recreated mesh interface
comes back with the same MAC (it is derived from the phy) and Noise sessions
are keyed on the remote peer, so a local rebind is invisible to peers. A
dedicated linger timer would be a second answer to a question already
answered.
### Logging
Edges, never attempts. A loop that logs per attempt reproduces the hot log
spin this mechanism removed, at 1–30 s intervals forever on any router with
an unplugged WAN — and operators learn to filter it, which is how the next
real failure gets missed.
The edge itself is not an error. An interface missing when the daemon starts
and bound a moment later is the ordinary case the mechanism exists to absorb,
so it is `info`; calling it an error at t=0 and "recovered" at t=0.2 s is the
cry-wolf failure this rule exists to prevent. A runtime detach is `warn` — a
link coming and going is ordinary weather for a mesh daemon.
There is exactly one deadline. Ten seconds is the window in which absence
could still be a race — a radio, a container, a veth arriving late. Past it a
**required** interface is a fault an operator has to fix, and it is reported
once at `error`. Start-time absence and a runtime detach share that deadline
rather than getting one each, for the same reason they share a code path
everywhere else here. An `optional` interface never reaches `error`; that is
what `optional` means.
Once, not repeated. This was a 1 m / 10 m / 1 h ladder that re-announced the
same fact at rising severity and then went permanently quiet after an hour,
which got both halves wrong: it used the log as a store for something already
published continuously as state, and it stopped mentioning a fault that was
still live. Duration belongs in `interface.since_secs` and in how long
`Degraded` has been held, where a monitor can threshold it per deployment
instead of the daemon compiling one in.
Node health does not wait for the deadline. `Degraded` publishes on the first
edge, which is the signal an operator actually watches.
Successful rebinds are damped. Backoff covers failed binds; the opposite and
nastier case is binds that keep *succeeding* into a socket that dies moments
later, which a receive loop giving up on a persistent error while the
interface stays `UP` produces once per second, forever. The binder counts
consecutive bindings that die inside ten seconds, backs off on the same
1 s → 30 s curve, and past three of them stops announcing each bind as a
recovery — holding health where it is until a binding lasts.
### Egress MTU
`transport_mtu()` is the minimum across *bound* transports — `is_bound()`,
not `is_operational()`, because an interface-bound transport is operational
from the moment it starts whether or not it holds a socket. Filtering on the
weaker predicate let a transport whose interface had never appeared set the
whole node's IPv6 MTU from hardware that was not present.
Since a transport can now bind long after start, that minimum moves at
runtime, and every consumer has to read it live. `show_status`, the
control-socket snapshot and the session-layer fragmentation check always did.
The TUN reader and writer did not: they were handed a `u16` at spawn, so a
narrow interface binding later never tightened the TCP MSS clamp and the node
reported one effective MTU while clamping to another. The ceiling is now
shared with those threads — an atomic beside the per-destination
`path_mtu_lookup` they already read on the same packet — and recomputed on
every change to the bound set.
Both directions, for the same reason `Degraded` is a level rather than a
latch: a narrow interface arriving must tighten the clamp or traffic
egressing over it is clamped too loose, and that interface leaving must
release it or unplugging a low-MTU adapter leaves the node over-clamped until
it restarts. MSS is negotiated per connection at SYN time, so a change binds
connections opened after it and leaves established ones alone.
### Observability
`show_transports` carries an `interface` block per interface-bound transport:
netdev name, `presence` (`absent` / `binding` / `present`), `carrier`,
`policy` (`required` / `optional`), `since_secs`, `binds` and
`failed_attempts`. The two counters separate an interface that is flapping
from one that is there and refusing to bind, and `since_secs` measures the
absence *episode* rather than the phase — a bind that fails walks
`Absent → Binding → Absent`, and restarting the clock on those edges would
report a permanently unbindable interface as one second old forever.
`fipstop`'s transports view names the netdev and the absence policy in their
own columns and shows presence in the State column for these transports,
because `state` reads `up` from the moment the transport starts and is
therefore precisely the wrong answer in the one case someone is scanning that
column for. Both render sites sort by ascending transport id — creation
order, and so grouped by transport type — rather than by `HashMap` iteration
order, which was arbitrary and differed on every daemon restart.
### What This Retires
- The `hotplug.d/net` rule that restarted the daemon when the FIPS radio
interfaces appeared, and the `wifi down; wifi up; sleep` dance provisioning
performed to sequence around the race.
- The YAML comment-toggling in `fips-mesh-setup` / `fips-ap-setup` — the mesh
and AP blocks ship enabled with `optional: true` and simply wait.
- The ad-hoc ENXIO socket reopen in the beacon sender: beacons now pause while
absent because the task does not exist then, and recovery is the presence
machine's job. One mechanism for every cause rather than one hack per
symptom.
- The start-time versus runtime asymmetry in the supervisor.
It also covers the case none of those workarounds did: an interface that flaps
while the daemon is running.
## Implementation Status
| Transport | Status | Notes |
+1 -1
View File
@@ -52,7 +52,7 @@ prints the response's `data` object as pretty JSON.
| `show mmp` | `show_mmp` | MMP metrics summary: per-peer link-layer metrics and per-session session-layer metrics. |
| `show cache` | `show_cache` | Coordinate cache: TTL, fill ratio, per-destination coords and path MTU. |
| `show connections` | `show_connections` | Pending handshake connections: state, idle time, resend count. |
| `show transports` | `show_transports` | Transport instances: type, state, MTU, local address, per-transport stats. |
| `show transports` | `show_transports` | Transport instances: type, state, MTU, local address, per-transport stats, and — for interface-bound transports — interface presence (`absent` / `binding` / `present`), carrier, absence policy (`required` / `optional`), time in the current phase, and bind/failed-attempt counts. |
| `show routing` | `show_routing` | Routing summary: pending lookups, retry state, forwarding/discovery/error/congestion counters. |
| `show identity-cache` | `show_identity_cache` | Cached `(node_addr → npub)` entries with last-seen timestamps. |
| `show native-flows` | `show_native_flows` | Native datagram API: open and pending flows with their ports, queue depth and age, bound listeners with their backlog, and the `native` counters. |
+74
View File
@@ -673,6 +673,80 @@ root; on macOS it requires read/write access to a `/dev/bpf*` device.
| `auto_connect` | bool | `false` | Auto-connect to discovered peers |
| `accept_connections` | bool | `false` | Accept incoming connection attempts from discovered peers |
| `beacon_interval_secs` | u64 | `30` | Announcement beacon interval in seconds (minimum 10) |
| `optional` | bool | `false` | Whether absence of the interface is normal. See below |
**Dynamic binding.** The interface does not have to exist when the daemon
starts. A transport whose interface is missing comes up *absent*: it is not a
start failure, it is not skipped, and it binds on its own the moment the
interface appears — sub-second where the kernel offers link events (netlink on
Linux, `PF_ROUTE` on the BSDs), within a second otherwise. An interface that
goes away at runtime unbinds and rebinds by the same path, so a `wifi reload`
or an unplugged adapter needs no restart. Presence means `IFF_UP` — the
interface exists and is administratively up — and deliberately not
`IFF_RUNNING`: binding needs no carrier, and the socket keeps working across a
carrier flap without rebinding. A bridge with nothing plugged into it, such as
`br-lan` on a wifi-only router, is therefore bound and healthy rather than
permanently `Degraded`, and starts carrying traffic the moment a port comes up.
Whether an interface has carrier is reported separately, as `interface.carrier`
in `show_transports`.
`optional` selects how that absence is reported:
| | absence | log | retries |
| --- | --- | --- | --- |
| `optional: false` (default) | node reports `Degraded` | `info` at boot / `warn` on a runtime detach, then `error` once if it lasts past 10 s | forever |
| `optional: true` | no health impact | `info`, and nothing after | forever |
Naming an interface in configuration is a statement that you expect it, so the
default is to complain; silence is opted into. Set `optional: true` for
hardware that is legitimately not always there — a dock adapter, a radio only
some boards carry.
`optional` describes **the interface's presence, not the transport's
importance**. An optional interface that is present is used exactly as hard as
any other. No value of it makes a missing interface fatal at startup: the only
fatal case remains "no transports at all came up".
Either way the edge is logged once, on entering absence and on recovery —
never once per retry.
The edge itself is not an error. An interface missing when the daemon starts
and bound a moment later is the ordinary boot race this mechanism exists to
absorb, so it is `info`; a runtime detach is `warn`, because a link coming and
going is ordinary weather for a mesh daemon and a cable unplugged for two
seconds does not need a human.
Ten seconds is the whole grace. Past that it is no longer a race against a
radio or a container coming up, so a **required** interface still missing is
reported once at `error` and stays `Degraded` until it returns. Start-time
absence and a runtime detach share the one deadline — they are the same
transition throughout this mechanism. An `optional` interface never reaches
`error`; that is what `optional` means.
Said once, not repeated. How long the absence has lasted is a *state*, and it
is published as one: `interface.since_secs` in `show_transports`, and
`Degraded` for as long as it holds. Re-announcing it on a timer would put a
second, lossier copy of that in the log.
Node health does not wait for the ten seconds — `Degraded` is published on the
first edge, and that is the signal to watch.
If a binding keeps dying moments after it is established (a socket that errors
persistently while the interface stays up), the binder stops treating each
bind as a recovery: it backs off on the same 1 s → 30 s curve, holds node
health at its degraded reading, and stays quiet until a binding survives ten
seconds. Without that damping a broken socket produces a health flap and a log
pair every second, which is the same cry-wolf failure the edge-only logging
rule exists to prevent.
A bind failure that is **not** absence — no `CAP_NET_RAW`, no readable
`/dev/bpf*`, a buffer the kernel refused — is a fault, not a state, and fails
the daemon's start as it always has. Only a missing interface is waited out.
`fipsctl show transports` reports the current state per transport under
`interface`: `presence` (`absent` / `binding` / `present`), `carrier`,
`policy` (`required` / `optional`), `since_secs`, `binds`, and
`failed_attempts`.
**Named instances.** Multiple Ethernet interfaces can be configured by
using named sub-keys instead of flat parameters:
+1 -1
View File
@@ -124,7 +124,7 @@ table below lists every command currently registered.
| `show_mmp` | — | `peers[]` (link-layer per peer), `sessions[]` (session-layer per session). Each entry includes loss/RTT/ETX/goodput, smoothed values, trends. |
| `show_cache` | — | `count`, `max_entries`, `fill_ratio`, `default_ttl_ms`, `expired`, `avg_age_ms`, `entries[]` — per-destination coords, depth, age, last-used, optional `path_mtu`. |
| `show_connections` | — | `connections[]` — pending handshakes: `link_id`, `direction`, `handshake_state`, `started_at_ms`, `idle_ms`, `resend_count`, optional `expected_peer`. |
| `show_transports` | — | `transports[]` — `transport_id`, `type`, `state`, `mtu`, `name`, `local_addr`, optional `tor_mode`, `onion_address`, `tor_monitoring`, `stats`. |
| `show_transports` | — | `transports[]` — `transport_id`, `type`, `state`, `mtu`, `name`, `local_addr`, optional `tor_mode`, `onion_address`, `tor_monitoring`, `stats`, and `interface` for interface-bound transports. `interface` carries `name` (the configured netdev), `presence` (`absent` / `binding` / `present`), `carrier` (whether the link has `IFF_RUNNING` — reported only, never acted on: presence is `IFF_UP`, so a bound interface with no carrier is normal), `policy` (`required` / `optional`), `since_secs` (how long the current presence phase has been held), `binds` (successful binds since the transport was created — `1` after a clean start, more means it has rebound) and `failed_attempts` (failed binds since the last success). Absent entirely for transports that are not bound to a named interface, rather than reported as a permanently-`present` interface named `""`. Note that `state` describes the *transport* (`up` once started) and `interface.presence` describes the *socket*: an `up` transport whose interface is `absent` is started and waiting, which is a normal state and not a failure. |
| `show_routing` | — | `coord_cache_entries`, `identity_cache_entries`, `pending_lookups[]`, `pending_tun_destinations`, `pending_tun_packets`, `recent_requests`, `retries[]`, `forwarding`, `discovery` (request/response sub-counters; includes `req_deduplicated` — requests suppressed as recent duplicates — and `req_dedup_cache_full` — requests admitted because the dedup cache was full), `error_signals`, `congestion`. |
| `show_identity_cache` | — | `entries[]`, `count`, `max_entries`. Each entry: `node_addr`, `npub`, `display_name`, `ipv6_addr`, `last_seen_ms`, `age_ms`. |
| `show_native_flows` | — | `flows[]`, `listeners[]`, `stats` (the `native` counter family). Each flow: `flow_id`, `peer` (the peer's npub, which is its address; always present, because the flow carries the key its client named or its session authenticated), `peer_addr` (the 16-byte node address in hex — a truncated hash of the same key, kept because it is what `show_sessions` and `show_routing` key on), `local_port`, `remote_port`, `state` (`established` / `pending_accept`), `queued` (datagrams the node is holding for the flow), `age_ms` (time since the flow reached its current state: opened for a flow this node opened, accepted for one taken off a listener, announced for one still pending — accepting a pending flow restarts the clock). Each listener: `local_port`, `backlog`. |
+11
View File
@@ -223,6 +223,7 @@ is "all four flags on both ends."
> accept_connections: true
> dongle:
> interface: "enx00aabbccddee"
> optional: true
> announce: true
> # ...
> ```
@@ -231,6 +232,16 @@ is "all four flags on both ends."
> A single ground-up link only needs the flat form shown
> first; named instances become useful when the same node
> bridges multiple physical segments.
>
> `optional: true` on the dongle says its absence is normal —
> a USB adapter that is plugged in some days and not others.
> Without it, naming an interface is a statement that you
> expect it, and while it is missing the node reports
> `Degraded` and logs at `error`. Either way the interface
> does not have to exist when the daemon starts: a transport
> whose interface is missing waits and binds when it appears,
> and rebinds if it later goes away. Watch that with
> `fipsctl show transports`.
## Step 3: Grant the daemon permission to open raw sockets
+62
View File
@@ -1042,6 +1042,68 @@ impl Config {
/// Validate cross-field configuration invariants.
pub fn validate(&self) -> Result<(), ConfigError> {
self.validate_ethernet_interfaces()?;
self.validate_rendezvous()
}
/// Reject interface names no kernel could ever hand back.
///
/// The presence machine deliberately cannot tell a typo from an interface
/// that has not been created yet — both are simply absent, and waiting is
/// the right answer for the second. That is what makes this check worth
/// having: a name that is *impossible* is the one case still separable
/// from "not there yet", and without it a typo costs a permanently
/// `Degraded` node whose only symptom is an interface that never arrives.
///
/// Syntax only. Whether a well-formed name exists is the binder's
/// question, asked once a second, forever.
fn validate_ethernet_interfaces(&self) -> Result<(), ConfigError> {
// Kernel limit: `IFNAMSIZ` is 16 including the terminating NUL, on
// both Linux and the BSDs.
const MAX_INTERFACE_NAME: usize = 15;
let mut seen: std::collections::HashMap<&str, &str> = std::collections::HashMap::new();
for (name, cfg) in self.transports.ethernet.iter() {
let label = name.unwrap_or("ethernet");
let iface = cfg.interface.as_str();
if iface.is_empty() {
return Err(ConfigError::Validation(format!(
"transport `{label}` has an empty `interface`"
)));
}
if iface.len() > MAX_INTERFACE_NAME {
return Err(ConfigError::Validation(format!(
"transport `{label}` interface `{iface}` is {} bytes; \
the kernel limit is {MAX_INTERFACE_NAME}, so no such \
interface can exist",
iface.len()
)));
}
if iface.contains('/') || iface.chars().any(char::is_whitespace) {
return Err(ConfigError::Validation(format!(
"transport `{label}` interface `{iface}` contains a \
character no interface name may hold"
)));
}
// Two transports on one netdev means two sockets on the same
// device at the same ethertype, each receiving every frame the
// other does.
if let Some(prior) = seen.insert(iface, label) {
return Err(ConfigError::Validation(format!(
"transports `{prior}` and `{label}` both bind interface \
`{iface}`"
)));
}
}
Ok(())
}
/// Cross-checks between transports, peers and the Nostr rendezvous.
fn validate_rendezvous(&self) -> Result<(), ConfigError> {
let nostr = &self.node.rendezvous.nostr;
let any_transport_advertises_on_nostr = self
+60
View File
@@ -302,6 +302,25 @@ pub struct EthernetConfig {
/// Announcement beacon interval in seconds. Default: 30.
#[serde(default, skip_serializing_if = "Option::is_none")]
pub beacon_interval_secs: Option<u64>,
/// Whether the absence of this interface is normal. Default: false.
///
/// Naming an interface in configuration is a statement that you expect it,
/// so the default is to complain: while the interface is missing the node
/// reports `Degraded`, the edge is logged (`info` at startup, `warn` on a
/// runtime detach), and an absence that outlasts the bring-up window — ten
/// seconds, past which it is no longer a race against a radio or a
/// container — is reported once at `error`. Set `optional: true` for
/// hardware that is legitimately not always there — a dock adapter, a
/// radio that only some boards carry — and its absence becomes silent
/// (`info` on the edge, no health impact, no error).
///
/// This describes **the interface's presence, not the transport's
/// importance**. An optional interface that is present is used exactly as
/// hard as any other; setting it does not deprioritize the transport, and
/// no value of this field makes a missing interface fatal at startup.
#[serde(default, skip_serializing_if = "Option::is_none")]
pub optional: Option<bool>,
}
impl EthernetConfig {
@@ -340,6 +359,11 @@ impl EthernetConfig {
self.accept_connections.unwrap_or(false)
}
/// Whether absence of the interface is normal. Default: false.
pub fn optional(&self) -> bool {
self.optional.unwrap_or(false)
}
/// Get the beacon interval, clamped to minimum. Default: 30s.
pub fn beacon_interval_secs(&self) -> u64 {
self.beacon_interval_secs
@@ -1071,4 +1095,40 @@ mod tests {
serde_yaml::from_str("interface: eth0\nbogus: true\n");
assert!(bogus.is_err());
}
#[test]
fn ethernet_absence_is_an_error_unless_opted_out() {
// Naming an interface is a statement that you expect it, so the
// default has to be the loud one. A default of `true` here would make
// every missing interface silent, which is the failure mode the
// whole presence mechanism exists to stop hiding.
let bare: EthernetConfig = serde_yaml::from_str("interface: eth0\n").unwrap();
assert_eq!(bare.optional, None);
assert!(!bare.optional(), "absence must default to an error");
let opted: EthernetConfig =
serde_yaml::from_str("interface: enx00e04c680001\noptional: true\n").unwrap();
assert!(opted.optional());
let explicit: EthernetConfig =
serde_yaml::from_str("interface: eth0\noptional: false\n").unwrap();
assert!(!explicit.optional());
}
#[test]
fn ethernet_optional_survives_a_round_trip() {
// The packaged OpenWrt config ships `optional: true` on the mesh and
// access blocks; a serializer that dropped it would silently turn
// every stock router Degraded.
let opted: EthernetConfig =
serde_yaml::from_str("interface: fips-mesh0\noptional: true\n").unwrap();
let round: EthernetConfig =
serde_yaml::from_str(&serde_yaml::to_string(&opted).unwrap()).unwrap();
assert!(round.optional());
// ... and the default stays absent from the output rather than being
// written back as an explicit `false`.
let bare: EthernetConfig = serde_yaml::from_str("interface: eth0\n").unwrap();
assert!(!serde_yaml::to_string(&bare).unwrap().contains("optional"));
}
}
+38 -2
View File
@@ -1361,8 +1361,17 @@ pub(crate) fn show_connections_from_handle(
/// `show_transports` — Transport instances.
pub fn show_transports(node: &Node) -> Value {
let transports: Vec<Value> = node
.transport_ids()
// Ascending id, which is creation order: UDP, then Ethernet, then TCP,
// Tor, Nym, BLE, with each type's instances in config order. The map
// behind `transport_ids` is a `HashMap`, so without this the array order
// is whatever the hash seed produced — arbitrary, and different on every
// daemon restart. Anything scripting against this output, and every view
// rendering it, inherits that. Sorting by id groups the list by transport
// type for free, because the ids were handed out that way.
let mut ids: Vec<_> = node.transport_ids().copied().collect();
ids.sort_by_key(|id| id.as_u32());
let transports: Vec<Value> = ids
.iter()
.map(|id| {
let handle = node.get_transport(id).unwrap();
let mut t_json = json!({
@@ -1390,6 +1399,21 @@ pub fn show_transports(node: &Node) -> Value {
t_json["tor_monitoring"] = serde_json::to_value(&monitoring).unwrap_or_default();
}
// Interface presence, for the transports that have an interface.
// Absent from the payload entirely for the ones that do not, rather
// than reported as a permanently-`present` interface named "".
if let Some(p) = handle.interface_presence() {
t_json["interface"] = json!({
"name": handle.interface_name().unwrap_or_default(),
"presence": p.presence,
"carrier": p.carrier,
"policy": p.policy,
"since_secs": p.since_secs,
"binds": p.binds,
"failed_attempts": p.failed_attempts,
});
}
t_json["stats"] = handle.transport_stats();
t_json
@@ -1434,6 +1458,18 @@ pub(crate) fn show_transports_from_handle(handle: &super::read_handle::ControlRe
t_json["tor_monitoring"] = monitoring.clone();
}
if let Some(iface) = &t.interface {
t_json["interface"] = json!({
"name": iface.name,
"presence": iface.presence,
"carrier": iface.carrier,
"policy": iface.policy,
"since_secs": iface.since_secs,
"binds": iface.binds,
"failed_attempts": iface.failed_attempts,
});
}
t_json["stats"] = t.stats.clone();
t_json
+15
View File
@@ -778,6 +778,21 @@ pub(crate) struct TransportRow {
pub onion_address: Option<String>,
pub tor_monitoring: Option<serde_json::Value>,
pub stats: serde_json::Value,
/// Interface presence for interface-bound transports; `None` for the rest.
pub interface: Option<InterfaceRow>,
}
/// Interface name, presence and policy for an interface-bound transport, as
/// `show_transports` renders it.
#[derive(Clone, PartialEq)]
pub(crate) struct InterfaceRow {
pub name: String,
pub presence: &'static str,
pub carrier: bool,
pub policy: &'static str,
pub since_secs: u64,
pub binds: u64,
pub failed_attempts: u32,
}
/// MMP trend labels for a peer's link-layer block in `show_mmp` (each present
+53
View File
@@ -109,6 +109,16 @@ impl Node {
}
};
// Interface-presence receiver, or a dummy channel — same pattern and
// same reason as the child-liveness receiver above.
let (mut presence_rx, _presence_guard) = match self.transport_presence_rx.take() {
Some(rx) => (rx, None),
None => {
let (tx, rx) = tokio::sync::mpsc::channel(1);
(rx, Some(tx))
}
};
let tick_period = Duration::from_secs(self.config().node.tick_interval_secs);
let mut tick = tokio::time::interval(tick_period);
@@ -346,6 +356,49 @@ impl Node {
self.supervisor.state = ns;
}
}
// A transport child exiting leaves the bound set, so
// it can be the one that was holding the node's egress
// MTU down. `is_bound()` is `is_operational()` plus the
// presence refinement, and this moves the first half.
self.refresh_tun_mss_ceiling();
}
}
// Interface presence. An interface-bound transport's binder
// reports attach and detach; the FSM folds it into health.
// Unlike `ChildExited` this is reversible in both directions —
// the interface coming back republishes `Running` — which is
// the whole point of `Degraded` being a level rather than a
// latch.
maybe_presence = presence_rx.recv() => {
if let Some(edge) = maybe_presence {
// Health is policy-filtered; the MTU floor below is
// not. An `optional` interface's absence is normal and
// must not move the node off `Full`, but it changes
// the bound set all the same.
if edge.health_relevant {
let child = crate::node::lifecycle::supervisor::Child::Transport(
edge.transport_id,
);
let event = if edge.present {
crate::node::lifecycle::supervisor::Event::ChildPresent { child }
} else {
crate::node::lifecycle::supervisor::Event::ChildAbsent { child }
};
let actions = self.supervisor.fsm.step(event);
for action in actions {
if let crate::node::lifecycle::supervisor::Action::PublishState(
ns,
) = action
{
self.supervisor.state = ns;
}
}
}
// The bound set just changed, so the node's egress MTU
// floor may have. Both directions: an interface that
// binds can be the narrow one, and one that detaches
// can be the reason the clamp was tight.
self.refresh_tun_mss_ceiling();
}
}
Some(ipv6_packet) = tun_outbound_rx.recv() => {
+90 -10
View File
@@ -1381,6 +1381,17 @@ impl Node {
self.child_exit_tx = Some(child_exit_tx);
self.child_exit_rx = Some(child_exit_rx);
// Interface-presence channel. Created before `create_transports` so
// every interface-bound transport gets the sender at construction and
// its very first bind attempt — the one `start_async` makes inline —
// is already reportable. A boot race therefore reaches the FSM while
// it is still `Starting`, and start-completion health resolves to
// `Degraded` on the first publish rather than publishing `Full` and
// correcting it a moment later.
let (presence_tx, presence_rx) = tokio::sync::mpsc::channel(16);
self.transport_presence_tx = Some(presence_tx);
self.transport_presence_rx = Some(presence_rx);
// Initialize transports first (before TUN, before Nostr discovery).
// Creation allocates each transport's id; the supervisor FSM authors
// the start order over those ids.
@@ -1642,12 +1653,20 @@ impl Node {
info!(" address: {}", device.address());
info!(" mtu: {}", mtu);
// Calculate max MSS for TCP clamping
// Seed the shared MSS ceiling from whatever is bound
// right now. Both TUN threads read it live from here
// on, so a transport binding or unbinding later moves
// the clamp instead of leaving it at this instant's
// value — see `crate::upper::tun::MssCeiling`.
self.refresh_tun_mss_ceiling();
let max_mss = self.tun_mss_ceiling.clone();
let effective_mtu = self.effective_ipv6_mtu();
let max_mss = effective_mtu.saturating_sub(40).saturating_sub(20); // IPv6 + TCP headers
info!("effective MTU: {} bytes", effective_mtu);
debug!(" max TCP MSS: {} bytes", max_mss);
debug!(
" max TCP MSS: {} bytes",
max_mss.load(std::sync::atomic::Ordering::Relaxed)
);
// On macOS and FreeBSD, create a shutdown pipe. Writing to it
// unblocks the reader thread's select() loop without closing
@@ -1671,8 +1690,8 @@ impl Node {
// Create writer (dups the fd for independent write access).
// Pass path_mtu_lookup so inbound SYN-ACK clamp can read
// per-destination path MTU learned via discovery.
let (writer, tun_tx) =
device.create_writer(max_mss, self.path_mtu_lookup.clone())?;
let (writer, tun_tx) = device
.create_writer(max_mss.clone(), self.path_mtu_lookup.clone())?;
// Spawn writer thread. On exit it self-reports
// `Child::Tun` (sync context → `blocking_send`); TUN
@@ -1698,7 +1717,6 @@ impl Node {
// self-reports `Child::Tun` on exit (sync context →
// `blocking_send`). Exactly one cfg variant compiles,
// so the single clone is moved into that closure.
let transport_mtu = self.transport_mtu();
let path_mtu_lookup = self.path_mtu_lookup.clone();
let reader_child_tx = self.child_exit_tx.clone();
#[cfg(any(target_os = "macos", target_os = "freebsd"))]
@@ -1709,7 +1727,7 @@ impl Node {
our_addr,
reader_tun_tx,
outbound_tx,
transport_mtu,
max_mss,
path_mtu_lookup,
shutdown_read_fd,
);
@@ -1725,7 +1743,7 @@ impl Node {
our_addr,
reader_tun_tx,
outbound_tx,
transport_mtu,
max_mss,
path_mtu_lookup,
);
if let Some(tx) = &reader_child_tx {
@@ -1856,6 +1874,16 @@ impl Node {
}
};
// Drain any presence edges this child's start produced *before*
// reporting the child itself. An interface-bound transport whose
// interface is missing reports absence from inside `start_async`
// and then reports `SubstrateUp` (absence is a state, not a start
// failure), so ordering the drain first means start-completion
// health already knows about the absence when `pending` empties.
// Otherwise a boot race publishes `Full` and corrects itself a
// moment later, and every consumer sees a spurious transition.
let _ = self.drain_transport_presence();
let feedback_actions = self.supervisor.fsm.step(feedback);
for action in &feedback_actions {
if let Action::PublishState(ns) = action {
@@ -1864,6 +1892,12 @@ impl Node {
}
}
// Late edges: a transport that bound after its `SubstrateUp` was
// reported, or one that detached during a later child's bring-up.
if let Some(ns) = self.drain_transport_presence() {
start_outcome = Some(ns);
}
// Seams that never triggered inside the loop: the "Transports
// initialized" info! when there was no non-transport child, and the
// peer-connect when there was no Tun/Dns child (today it still runs,
@@ -1902,8 +1936,10 @@ impl Node {
// children. Enumerate them for the operator, then proceed —
// a degraded node serves traffic.
warn!(
degraded_children = ?self.supervisor.fsm.failed(),
"Node started DEGRADED: one or more configured optional children failed to start"
degraded_children = ?self.supervisor.fsm.degraded_children(),
absent_interfaces = ?self.supervisor.fsm.absent(),
"Node started DEGRADED: one or more configured optional children failed to \
start, or a configured interface is absent"
);
}
_ => {}
@@ -2262,6 +2298,50 @@ impl Node {
}
}
/// Feed every queued interface-presence edge to the supervisor FSM,
/// returning the last [`NodeState`] it asked to publish (if any).
///
/// Non-blocking: it drains what is already queued and returns. Used during
/// bring-up, where the rx_loop's presence arm is not running yet — from
/// then on that arm owns the same translation.
pub(in crate::node) fn drain_transport_presence(&mut self) -> Option<NodeState> {
let mut edges = Vec::new();
if let Some(rx) = self.transport_presence_rx.as_mut() {
while let Ok(edge) = rx.try_recv() {
edges.push(edge);
}
}
let mut published = None;
let saw_edge = !edges.is_empty();
for edge in edges {
// Health is policy-filtered; the MTU refresh below is not. See
// `saw_edge`.
if !edge.health_relevant {
continue;
}
let child = Child::Transport(edge.transport_id);
let event = if edge.present {
Event::ChildPresent { child }
} else {
Event::ChildAbsent { child }
};
for action in self.supervisor.fsm.step(event) {
if let Action::PublishState(ns) = action {
published = Some(ns);
}
}
}
if saw_edge {
// A bind or unbind changes which transports are bound, and so the
// node's egress MTU floor. During bring-up this runs before the
// TUN threads exist, which is exactly when it must: they read the
// ceiling this leaves behind.
self.refresh_tun_mss_ceiling();
}
published
}
/// Reconstruct the supervised up-set from observed runtime presence, so the
/// FSM authors the teardown order regardless of how the node reached
/// `Running`. Worker pools are deliberately excluded: today's teardown never
+434 -12
View File
@@ -67,6 +67,39 @@
//! when a task/thread dies at runtime) is **deferred**: start-completion health
//! resolution is start-framed, and liveness monitoring is a substantial unbuilt
//! mechanism. This commit is start-time health only.
//!
//! ## Scope: interface presence, and `Degraded` as a level (this commit)
//!
//! Interface-bound transports are now *sometimes bound*: a transport whose
//! interface is missing at start comes up [`Absent`] and binds later, and one
//! whose interface goes away at runtime unbinds and rebinds when it returns.
//! Two things follow for this machine.
//!
//! - **A second reason set.** [`Event::ChildAbsent`] / [`Event::ChildPresent`]
//! move a child in and out of `absent`, which feeds `Degraded` exactly like
//! `failed` does. It is kept separate because it is *reversible* and `failed`
//! is not: a child that failed to start stays failed for the bring-up, while
//! an absent interface is expected to come back.
//! - **`Degraded` is a level, not a latch.** Nothing ever removed from `failed`,
//! which was correct while no child could recover — a monotonic set accurately
//! described a one-way door. Once recovery exists the assumption inverts: plug
//! the WAN back in and the node would stay `Degraded` until the process
//! restarted, and `Degraded` would come to mean "something broke at some point
//! since boot" rather than "something is broken now".
//! [`Self::classify_health`](SupervisorFsm::classify_health) is therefore
//! recomputed on every transition **in both directions**.
//!
//! An absent transport still counts as *up*. It came up — `start_async` returns
//! `Ok` with the transport absent — so it does not push a single-transport node
//! into the fatal [`FailReason::NoTransports`], which would make a node that
//! merely booted before its wifi exit instead of waiting. Absence degrades; it
//! never kills.
//!
//! There is deliberately **no restart action**. Rebinding is owned by the
//! transport's own binder task, which is where the file descriptor and the
//! presence watcher live; the FSM is told what happened and republishes health.
//!
//! [`Absent`]: crate::transport::ethernet::Presence::Absent
use std::collections::HashSet;
use std::sync::Arc;
@@ -174,6 +207,22 @@ pub(crate) enum Event {
/// The child whose task/thread exited.
child: Child,
},
/// An interface-bound child lost its interface — it was never there at
/// start, or it went away at runtime. The child stays *up* (the transport
/// object survives detach, only its socket goes) but contributes
/// `Degraded`. Valid while `Running`; while `Starting` the edge is recorded
/// so start-completion health already reflects it.
ChildAbsent {
/// The child whose interface is absent.
child: Child,
},
/// An interface-bound child's interface came back and it rebound. Clears
/// the absence and republishes health, which is how `Degraded` becomes
/// reversible.
ChildPresent {
/// The child whose interface is present again.
child: Child,
},
}
/// A driver-scheduled timer the supervisor can arm. Only the
@@ -320,7 +369,16 @@ pub(crate) struct SupervisorFsm {
up: HashSet<Child>,
/// Configured children that failed to start during the current bring-up.
/// Feeds the `Degraded` health determination when `pending` empties.
///
/// One-way within a bring-up: a start failure is not recoverable, so
/// nothing removes from this set until the next `Start`.
failed: HashSet<Child>,
/// Children that are up but whose network interface is currently absent.
///
/// Reversible, unlike [`Self::failed`] — that is the whole reason it is a
/// second set rather than more entries in the first. Feeds `Degraded` the
/// same way, and empties as interfaces come back.
absent: HashSet<Child>,
}
impl SupervisorFsm {
@@ -330,6 +388,7 @@ impl SupervisorFsm {
state: SupState::Created,
up: HashSet::new(),
failed: HashSet::new(),
absent: HashSet::new(),
}
}
@@ -348,6 +407,7 @@ impl SupervisorFsm {
},
up: up.into_iter().collect(),
failed: HashSet::new(),
absent: HashSet::new(),
}
}
@@ -357,13 +417,26 @@ impl SupervisorFsm {
&self.state
}
/// The configured children that failed to start during bring-up. The driver
/// reads this on the `Degraded` start outcome to enumerate the degraded
/// children in an operator-visible `warn!`.
/// The configured children that failed to start during bring-up. Kept for
/// the tests that pin the failure-vs-absence split; the driver reports
/// [`Self::degraded_children`], which is the union of the two.
#[cfg(test)]
pub(in crate::node) fn failed(&self) -> &HashSet<Child> {
&self.failed
}
/// Children whose interface is currently absent.
pub(in crate::node) fn absent(&self) -> &HashSet<Child> {
&self.absent
}
/// Every child currently contributing `Degraded` — the ones that failed to
/// start plus the ones whose interface is away. This is what an operator
/// wants named when the node reports `Degraded`.
pub(in crate::node) fn degraded_children(&self) -> HashSet<Child> {
self.failed.union(&self.absent).copied().collect()
}
/// Whether the machine is in the bounded-drain window. The driver uses this
/// after the rx loop returns to decide between the drain-teardown path and
/// the immediate-`stop()` fallback.
@@ -398,6 +471,8 @@ impl SupervisorFsm {
Event::DrainDeadlineElapsed => self.on_drain_deadline_elapsed(),
Event::ChildStopped { child } => self.on_child_stopped(child),
Event::ChildExited { child } => self.on_child_exited(child),
Event::ChildAbsent { child } => self.on_child_absent(child),
Event::ChildPresent { child } => self.on_child_present(child),
}
}
@@ -444,6 +519,7 @@ impl SupervisorFsm {
self.up.clear();
self.failed.clear();
self.absent.clear();
// A node with no children at all resolves health immediately. Zero
// transports up → `Failed` (this is the behavioral
@@ -497,19 +573,30 @@ impl SupervisorFsm {
self.classify_health()
}
/// Classify health from the current `up` / `failed` sets and set the
/// resulting state, returning the [`NodeState`] the driver should publish.
/// Shared by start-completion ([`Self::resolve_start_health`]) and runtime
/// child-exit ([`Self::on_child_exited`]):
/// Classify health from the current `up` / `failed` / `absent` sets and set
/// the resulting state, returning the [`NodeState`] the driver should
/// publish. Shared by start-completion ([`Self::resolve_start_health`]),
/// runtime child-exit ([`Self::on_child_exited`]) and the presence edges
/// ([`Self::on_child_absent`] / [`Self::on_child_present`]):
///
/// - zero transports up → [`SupState::Failed`] / [`NodeState::Failed`];
/// - ≥1 transport up but some child in `failed` → [`Health::Degraded`] /
/// [`NodeState::Degraded`];
/// - everything up and nothing failed → [`Health::Full`] / [`NodeState::Running`].
/// - ≥1 transport up but some child in `failed` or `absent` →
/// [`Health::Degraded`] / [`NodeState::Degraded`];
/// - everything up, nothing failed, nothing absent → [`Health::Full`] /
/// [`NodeState::Running`].
///
/// Worker-pool failures are captured in `failed` like any other optional
/// child, so they contribute `Degraded` (never `Failed`) — the inline crypto
/// fallback keeps the node correct without the pools.
///
/// A transport whose interface is absent is still counted among
/// `transports_up`: it came up, it is simply not bound. Excluding it would
/// make a single-ethernet node that booted before its wifi resolve to the
/// fatal [`FailReason::NoTransports`] and exit — which is the failure this
/// whole mechanism exists to remove. Absence degrades; it never kills.
///
/// Recomputed in **both** directions. This function is the reason
/// `Degraded` is a level rather than a latch.
fn classify_health(&mut self) -> NodeState {
let transports_up = self
.up
@@ -521,10 +608,10 @@ impl SupervisorFsm {
reason: FailReason::NoTransports,
};
NodeState::Failed
} else if !self.failed.is_empty() {
} else if !self.failed.is_empty() || !self.absent.is_empty() {
self.state = SupState::Running {
health: Health::Degraded {
reasons: self.failed.clone(),
reasons: self.degraded_children(),
},
};
NodeState::Degraded
@@ -536,6 +623,53 @@ impl SupervisorFsm {
}
}
/// An interface-bound child lost its interface.
///
/// The child stays in the up-set: the transport object survives detach —
/// config, id, statistics and neighbor buffer persist, only the descriptor
/// and its loops go. Only the reason set changes.
///
/// While `Starting` the edge is recorded silently; start-completion health
/// picks it up when `pending` empties, so a boot race resolves to
/// `Degraded` on the first publish rather than publishing `Full` and
/// immediately correcting it. Inert outside `Starting` / `Running`: a
/// teardown in flight owns its own bookkeeping.
fn on_child_absent(&mut self, child: Child) -> Vec<Action> {
match self.state {
SupState::Starting { .. } => {
self.absent.insert(child);
Vec::new()
}
SupState::Running { .. } => {
if !self.absent.insert(child) {
return Vec::new();
}
vec![Action::PublishState(self.classify_health())]
}
_ => Vec::new(),
}
}
/// An interface-bound child's interface came back.
///
/// Clears the absence and republishes. A child that is not currently
/// recorded absent produces nothing — a duplicate edge is not an event.
fn on_child_present(&mut self, child: Child) -> Vec<Action> {
match self.state {
SupState::Starting { .. } => {
self.absent.remove(&child);
Vec::new()
}
SupState::Running { .. } => {
if !self.absent.remove(&child) {
return Vec::new();
}
vec![Action::PublishState(self.classify_health())]
}
_ => Vec::new(),
}
}
fn on_stop(&mut self) -> Vec<Action> {
if !matches!(self.state, SupState::Running { .. }) {
return Vec::new();
@@ -588,6 +722,7 @@ impl SupervisorFsm {
if let SupState::Stopping { pending } = &mut self.state {
pending.remove(&child);
self.up.remove(&child);
self.absent.remove(&child);
if pending.is_empty() {
self.state = SupState::Stopped;
}
@@ -617,6 +752,8 @@ impl SupervisorFsm {
if !self.up.remove(&child) {
return Vec::new();
}
// An exit supersedes an absence: the child is gone, not waiting.
self.absent.remove(&child);
self.failed.insert(child);
vec![Action::PublishState(self.classify_health())]
}
@@ -1393,4 +1530,289 @@ mod tests {
);
assert!(s.failed().is_empty());
}
// ── Interface presence ────────────────────────────────────────────────
//
// The absence set is separate from `failed` because it is reversible, and
// reversibility is what turns `Degraded` from a latch into a level.
/// Helper: bring a full node up cleanly and leave it `Running{Full}`.
fn running_node() -> SupervisorFsm {
let mut s = SupervisorFsm::new();
s.step(start_full());
for child in [
Child::Transport(tid(1)),
Child::Transport(tid(2)),
Child::EncryptWorkers,
Child::DecryptWorkers,
Child::Nostr,
Child::Mdns,
Child::Tun,
Child::Dns,
] {
s.step(Event::SubstrateUp { child });
}
assert_eq!(
s.state(),
&SupState::Running {
health: Health::Full
}
);
s
}
#[test]
fn an_absent_interface_degrades_a_running_node() {
let mut s = running_node();
assert_eq!(
s.step(Event::ChildAbsent {
child: Child::Transport(tid(1))
}),
vec![Action::PublishState(NodeState::Degraded)]
);
assert!(s.absent().contains(&Child::Transport(tid(1))));
// Absence is not failure: the two sets stay distinct.
assert!(s.failed().is_empty());
}
#[test]
fn a_returning_interface_clears_degraded() {
// The regression this whole design turns on: plug the WAN back in and
// the node must leave `Degraded`, not carry it until the process
// restarts. `Degraded` is a level, not a latch.
let mut s = running_node();
s.step(Event::ChildAbsent {
child: Child::Transport(tid(1)),
});
assert_eq!(
s.step(Event::ChildPresent {
child: Child::Transport(tid(1))
}),
vec![Action::PublishState(NodeState::Running)]
);
assert_eq!(
s.state(),
&SupState::Running {
health: Health::Full
}
);
assert!(s.absent().is_empty());
}
#[test]
fn health_reflects_the_last_interface_still_away() {
// Two interfaces away, one returns: still Degraded. Only the empty
// absence set publishes Full.
let mut s = running_node();
s.step(Event::ChildAbsent {
child: Child::Transport(tid(1)),
});
s.step(Event::ChildAbsent {
child: Child::Transport(tid(2)),
});
assert_eq!(
s.step(Event::ChildPresent {
child: Child::Transport(tid(1))
}),
vec![Action::PublishState(NodeState::Degraded)]
);
assert_eq!(
s.step(Event::ChildPresent {
child: Child::Transport(tid(2))
}),
vec![Action::PublishState(NodeState::Running)]
);
}
#[test]
fn a_duplicate_presence_edge_is_not_an_event() {
let mut s = running_node();
s.step(Event::ChildAbsent {
child: Child::Transport(tid(1)),
});
assert_eq!(
s.step(Event::ChildAbsent {
child: Child::Transport(tid(1))
}),
vec![],
"re-reporting the same absence must not republish"
);
s.step(Event::ChildPresent {
child: Child::Transport(tid(1)),
});
assert_eq!(
s.step(Event::ChildPresent {
child: Child::Transport(tid(1))
}),
vec![],
"re-reporting the same return must not republish"
);
}
#[test]
fn absence_during_bringup_resolves_degraded_on_the_first_publish() {
// The boot race. The transport reports absence from inside its own
// start, then reports up. Start-completion health must already know,
// so the node never publishes `Full` and corrects itself.
let mut s = SupervisorFsm::new();
s.step(start_full());
s.step(Event::ChildAbsent {
child: Child::Transport(tid(1)),
});
for child in [
Child::Transport(tid(1)),
Child::Transport(tid(2)),
Child::EncryptWorkers,
Child::DecryptWorkers,
Child::Nostr,
Child::Mdns,
Child::Tun,
] {
assert_eq!(s.step(Event::SubstrateUp { child }), vec![]);
}
assert_eq!(
s.step(Event::SubstrateUp { child: Child::Dns }),
vec![Action::PublishState(NodeState::Degraded)]
);
let mut reasons = HashSet::new();
reasons.insert(Child::Transport(tid(1)));
assert_eq!(
s.state(),
&SupState::Running {
health: Health::Degraded { reasons }
}
);
}
#[test]
fn a_lone_absent_transport_degrades_rather_than_fails() {
// A single-ethernet node that boots before its wifi. The transport
// came up — absence is a state, not a start failure — so it counts
// among the transports up and the node serves. Resolving to `Failed`
// here would make the daemon exit on the very race this mechanism
// exists to survive.
let mut s = SupervisorFsm::new();
s.step(Event::Start {
transports: vec![tid(1)],
encrypt_workers: false,
decrypt_workers: false,
nostr: false,
mdns: false,
tun: false,
dns: false,
});
s.step(Event::ChildAbsent {
child: Child::Transport(tid(1)),
});
assert_eq!(
s.step(Event::SubstrateUp {
child: Child::Transport(tid(1))
}),
vec![Action::PublishState(NodeState::Degraded)]
);
assert!(
!matches!(s.state(), SupState::Failed { .. }),
"an absent interface must never be the fatal no-transports case"
);
}
#[test]
fn an_exit_supersedes_an_absence() {
// A transport that is away and then exits is gone, not waiting: it
// leaves the up-set and moves from `absent` to `failed`, so a later
// spurious `ChildPresent` cannot resurrect it into `Full`.
let mut s = running_node();
s.step(Event::ChildAbsent {
child: Child::Transport(tid(1)),
});
s.step(Event::ChildExited {
child: Child::Transport(tid(1)),
});
assert!(s.absent().is_empty());
assert!(s.failed().contains(&Child::Transport(tid(1))));
assert_eq!(
s.step(Event::ChildPresent {
child: Child::Transport(tid(1))
}),
vec![]
);
assert!(matches!(
s.state(),
SupState::Running {
health: Health::Degraded { .. }
}
));
}
#[test]
fn presence_edges_are_inert_outside_starting_and_running() {
// Teardown and drain own their own bookkeeping; a late edge from a
// binder that has not noticed the stop must not author actions.
let mut created = SupervisorFsm::new();
assert_eq!(
created.step(Event::ChildAbsent {
child: Child::Transport(tid(1))
}),
vec![]
);
let mut draining = running_node();
draining.step(Event::Drain { deadline_ms: 1 });
assert_eq!(
draining.step(Event::ChildAbsent {
child: Child::Transport(tid(1))
}),
vec![]
);
assert!(
draining.is_draining(),
"a presence edge must not end a drain"
);
}
#[test]
fn a_new_start_clears_the_previous_absences() {
let mut s = running_node();
s.step(Event::ChildAbsent {
child: Child::Transport(tid(1)),
});
s.step(Event::Stop);
for child in [
Child::Dns,
Child::Nostr,
Child::Mdns,
Child::Transport(tid(1)),
Child::Transport(tid(2)),
Child::Tun,
] {
s.step(Event::ChildStopped { child });
}
s.step(start_full());
assert!(s.absent().is_empty());
}
#[test]
fn degraded_children_is_the_union_of_both_reason_sets() {
let mut s = SupervisorFsm::new();
s.step(start_full());
s.step(Event::ChildAbsent {
child: Child::Transport(tid(1)),
});
s.step(Event::SubstrateFailed { child: Child::Mdns });
for child in [
Child::Transport(tid(1)),
Child::Transport(tid(2)),
Child::EncryptWorkers,
Child::DecryptWorkers,
Child::Nostr,
Child::Tun,
Child::Dns,
] {
s.step(Event::SubstrateUp { child });
}
let degraded = s.degraded_children();
assert!(degraded.contains(&Child::Mdns));
assert!(degraded.contains(&Child::Transport(tid(1))));
assert_eq!(degraded.len(), 2);
}
}
+109 -3
View File
@@ -375,6 +375,16 @@ pub struct Node {
/// SYN/SYN-ACK clamp can use the smaller of the local-egress floor
/// and the learned per-destination path MTU.
path_mtu_lookup: crate::upper::tun::PathMtuLookup,
/// Node-global TCP MSS ceiling, shared live with the TUN reader and writer
/// threads and recomputed whenever the set of *bound* transports changes.
///
/// Sits beside `path_mtu_lookup` because it answers the other half of the
/// same question at the same moment: that map supplies the per-destination
/// ceiling, this supplies the local-egress one, and the clamp takes the
/// smaller. Both have to be read live — a transport that binds after start
/// can be the narrow one, and one that unbinds can be the reason the node
/// was clamped at all.
tun_mss_ceiling: crate::upper::tun::MssCeiling,
/// Which transport last supplied a *link seed* into `path_mtu_lookup`,
/// per destination.
///
@@ -419,6 +429,20 @@ pub struct Node {
/// rx_loop select arm that feeds `Event::ChildExited` to the supervisor FSM.
child_exit_rx: Option<tokio::sync::mpsc::Receiver<crate::node::lifecycle::supervisor::Child>>,
// === Interface Presence Channel ===
/// Sender half of the interface-presence channel, cloned into every
/// interface-bound transport so its binder task can report attach and
/// detach. Held on `self` for the rx_loop's lifetime as the keep-alive
/// sender, exactly like [`Self::child_exit_tx`].
///
/// Separate from the child-exit channel because presence is *reversible*:
/// an exit is one-way, an interface comes back.
transport_presence_tx: Option<crate::transport::PresenceTx>,
/// Receiver half of the interface-presence channel, `take()`-en by the
/// rx_loop select arm that feeds `Event::ChildAbsent` / `Event::ChildPresent`
/// to the supervisor FSM.
transport_presence_rx: Option<crate::transport::PresenceRx>,
// === Per-Peer Control Machines ===
/// Per-peer lifecycle control FSMs, keyed by the stable `LinkId` that spans
/// the handshake→active lifetime. Each machine owns its handshake crypto
@@ -814,6 +838,8 @@ impl Node {
packet_rx: None,
child_exit_tx: None,
child_exit_rx: None,
transport_presence_tx: None,
transport_presence_rx: None,
peer_machines: HashMap::new(),
peer_timers: HashMap::new(),
peers: HashMap::new(),
@@ -881,6 +907,12 @@ impl Node {
peer_acl,
host_map,
path_mtu_lookup: Arc::new(std::sync::RwLock::new(HashMap::new())),
// Seeded at the IPv6 minimum, which is what `transport_mtu()`
// itself falls back to when nothing is bound. Refreshed before
// the TUN threads start and on every change to the bound set.
tun_mss_ceiling: Arc::new(std::sync::atomic::AtomicU16::new(
crate::upper::icmp::mss_ceiling(crate::upper::tun::IPV6_MIN_MTU),
)),
path_mtu_seeded_by: Arc::new(std::sync::RwLock::new(HashMap::new())),
#[cfg(unix)]
decrypt_registered_sessions: std::collections::HashSet::new(),
@@ -979,6 +1011,8 @@ impl Node {
packet_rx: None,
child_exit_tx: None,
child_exit_rx: None,
transport_presence_tx: None,
transport_presence_rx: None,
peer_machines: HashMap::new(),
peer_timers: HashMap::new(),
peers: HashMap::new(),
@@ -1043,6 +1077,12 @@ impl Node {
peer_acl,
host_map,
path_mtu_lookup: Arc::new(std::sync::RwLock::new(HashMap::new())),
// Seeded at the IPv6 minimum, which is what `transport_mtu()`
// itself falls back to when nothing is bound. Refreshed before
// the TUN threads start and on every change to the bound set.
tun_mss_ceiling: Arc::new(std::sync::atomic::AtomicU16::new(
crate::upper::icmp::mss_ceiling(crate::upper::tun::IPV6_MIN_MTU),
)),
path_mtu_seeded_by: Arc::new(std::sync::RwLock::new(HashMap::new())),
#[cfg(unix)]
decrypt_registered_sessions: std::collections::HashSet::new(),
@@ -1099,6 +1139,11 @@ impl Node {
let mut eth =
EthernetTransport::new(transport_id, name, eth_config, packet_tx.clone());
eth.set_local_pubkey(xonly);
// The binder task reports attach and detach here, so node
// health tracks the interface in both directions.
if let Some(tx) = self.transport_presence_tx.clone() {
eth.set_presence_tx(tx);
}
transports.push(TransportHandle::Ethernet(eth));
}
}
@@ -1412,6 +1457,46 @@ impl Node {
crate::upper::icmp::effective_ipv6_mtu(self.transport_mtu())
}
/// The TCP MSS ceiling the TUN threads are currently clamping to.
#[cfg(test)]
pub(crate) fn tun_mss_ceiling(&self) -> u16 {
self.tun_mss_ceiling
.load(std::sync::atomic::Ordering::Relaxed)
}
/// Recompute the shared TUN MSS ceiling from the currently bound
/// transports, and log it if it moved.
///
/// Called wherever the bound set can change — the presence edges that
/// bind and unbind an interface-bound transport, and a child exiting —
/// so the clamp the TUN threads apply keeps agreeing with the
/// `effective_ipv6_mtu` this node reports in `show_status`.
///
/// Moves in **both** directions, deliberately. A narrow interface
/// appearing has to tighten the ceiling or the clamp is wrong for
/// traffic that will egress over it; that same interface going away has
/// to release it, or unplugging a low-MTU adapter leaves the node
/// over-clamped until it restarts. It is the same argument that makes
/// `Degraded` a level rather than a latch: nothing here is one-way once
/// a transport can come back.
///
/// Existing flows are not re-clamped — MSS is negotiated per connection
/// at SYN time, so a change applies to connections opened after it.
pub(crate) fn refresh_tun_mss_ceiling(&self) {
use std::sync::atomic::Ordering;
let ceiling = crate::upper::icmp::mss_ceiling(self.transport_mtu());
let previous = self.tun_mss_ceiling.swap(ceiling, Ordering::Relaxed);
if previous != ceiling {
tracing::info!(
previous_max_mss = previous,
max_mss = ceiling,
effective_ipv6_mtu = self.effective_ipv6_mtu(),
"Node egress MTU changed; TCP MSS ceiling updated for new connections"
);
}
}
/// Get the transport MTU governing the global TUN-boundary MSS clamp.
///
/// Returns the **minimum** MTU across all operational transports, or
@@ -1427,10 +1512,17 @@ impl Node {
/// to vary across HashMap iteration order + async-startup race) makes
/// the clamp deterministic across daemon restarts.
pub fn transport_mtu(&self) -> u16 {
// `is_bound`, not `is_operational`. An interface-bound transport is
// "operational" from the moment it starts, whether or not its
// interface exists — so filtering on that let a transport whose
// interface has never appeared clamp the whole node's IPv6 MTU to a
// number derived from hardware that is not present. Before dynamic
// binding a transport that could not bind was never inserted here at
// all, so the distinction did not exist to get wrong.
let min_operational = self
.transports
.values()
.filter(|h| h.is_operational())
.filter(|h| h.is_bound())
.map(|h| h.mtu())
.min();
if let Some(mtu) = min_operational {
@@ -2336,8 +2428,13 @@ impl Node {
.collect();
// --- transports (show_transports) ---
let transport_rows: Vec<snap::TransportRow> = self
.transport_ids()
// Ascending id, matching `show_transports`; see the note there. The
// off-loop renderer reads this table verbatim, so the two paths would
// otherwise disagree about ordering as well as being arbitrary.
let mut transport_ids: Vec<_> = self.transport_ids().copied().collect();
transport_ids.sort_by_key(|id| id.as_u32());
let transport_rows: Vec<snap::TransportRow> = transport_ids
.iter()
.map(|id| {
let handle = self.get_transport(id).unwrap();
snap::TransportRow {
@@ -2353,6 +2450,15 @@ impl Node {
.tor_monitoring()
.map(|m| serde_json::to_value(&m).unwrap_or_default()),
stats: handle.transport_stats(),
interface: handle.interface_presence().map(|p| snap::InterfaceRow {
name: handle.interface_name().unwrap_or_default().to_string(),
presence: p.presence,
carrier: p.carrier,
policy: p.policy,
since_secs: p.since_secs,
binds: p.binds,
failed_attempts: p.failed_attempts,
}),
}
})
.collect();
+109
View File
@@ -1842,6 +1842,115 @@ async fn test_transport_mtu_returns_min_across_operational() {
}
}
#[tokio::test]
async fn the_tun_mss_ceiling_follows_a_transport_arriving_and_leaving() {
// The TUN reader and writer used to be handed a `u16` computed once at
// spawn. Every other consumer of `transport_mtu()` reads it live, so once
// a transport could bind minutes after start the daemon reported one
// effective MTU in `show_status` and clamped to another.
//
// Both directions. A narrow transport arriving has to tighten the ceiling
// or traffic egressing over it is clamped too loose; the same transport
// leaving has to release it, or unplugging a low-MTU adapter leaves the
// node over-clamped until it restarts.
let mut node = make_node();
let (packet_tx, packet_rx) = packet_channel(64);
node.supervisor.packet_tx = Some(packet_tx);
node.packet_rx = Some(packet_rx);
let wide = make_udp_transport_with_mtu(1, 1452).await;
node.transports.insert(TransportId::new(1), wide);
node.refresh_tun_mss_ceiling();
let wide_ceiling = node.tun_mss_ceiling();
assert_eq!(
wide_ceiling,
crate::upper::icmp::mss_ceiling(1452),
"the seeded ceiling must match the only bound transport"
);
// A narrower transport arrives after the TUN threads would already be
// running. The shared ceiling has to tighten.
let narrow = make_udp_transport_with_mtu(2, 1280).await;
node.transports.insert(TransportId::new(2), narrow);
node.refresh_tun_mss_ceiling();
let narrow_ceiling = node.tun_mss_ceiling();
assert_eq!(narrow_ceiling, crate::upper::icmp::mss_ceiling(1280));
assert!(
narrow_ceiling < wide_ceiling,
"a narrower transport must tighten the clamp, not be ignored"
);
assert_eq!(
narrow_ceiling,
crate::upper::icmp::mss_ceiling(node.transport_mtu()),
"the clamp and the reported MTU must not disagree"
);
// ...and leaving has to release it again.
if let Some(mut gone) = node.transports.remove(&TransportId::new(2)) {
gone.stop().await.ok();
}
node.refresh_tun_mss_ceiling();
assert_eq!(
node.tun_mss_ceiling(),
wide_ceiling,
"the ceiling must rise again when the narrow transport goes away"
);
for transport in node.transports.values_mut() {
transport.stop().await.ok();
}
}
#[tokio::test]
async fn a_presence_edge_refreshes_the_tun_mss_ceiling_without_being_asked() {
// The one above pins the arithmetic; this pins the wiring. A presence
// edge arriving has to refresh the ceiling on its own — if the refresh is
// dropped from the edge handlers the value silently stops tracking, which
// is the defect in its original form.
let mut node = make_node();
let (packet_tx, packet_rx) = packet_channel(64);
node.supervisor.packet_tx = Some(packet_tx);
node.packet_rx = Some(packet_rx);
let (presence_tx, presence_rx) = tokio::sync::mpsc::channel(4);
node.transport_presence_tx = Some(presence_tx.clone());
node.transport_presence_rx = Some(presence_rx);
// Nothing bound: the conservative seed.
node.refresh_tun_mss_ceiling();
let seeded = node.tun_mss_ceiling();
assert_eq!(seeded, crate::upper::icmp::mss_ceiling(1280));
// A wide transport appears, and an edge announces it. No explicit
// refresh call here — draining the edge is the whole trigger.
let wide = make_udp_transport_with_mtu(1, 1452).await;
node.transports.insert(TransportId::new(1), wide);
presence_tx
.send(crate::transport::TransportPresence {
transport_id: TransportId::new(1),
present: true,
health_relevant: true,
})
.await
.expect("presence edge queued");
node.drain_transport_presence();
assert_eq!(
node.tun_mss_ceiling(),
crate::upper::icmp::mss_ceiling(1452),
"draining a presence edge must refresh the ceiling on its own"
);
assert_ne!(
node.tun_mss_ceiling(),
seeded,
"the ceiling stayed at its seed, so the edge did not refresh it"
);
for transport in node.transports.values_mut() {
transport.stop().await.ok();
}
}
#[tokio::test]
async fn test_transport_mtu_fallback_when_no_operational_transports() {
// No transports configured at all → falls back to 1280 (IPv6 minimum).
+143
View File
@@ -9,6 +9,120 @@ use crate::transport::TransportError;
/// Broadcast MAC address.
pub const ETHERNET_BROADCAST: [u8; 6] = [0xff; 6];
/// Whether the named interface exists and is administratively up.
///
/// Presence is `IFF_UP` — the interface exists and the operator has enabled
/// it — and deliberately **not** `IFF_RUNNING`.
///
/// Carrier is a different question from bindability, and only the second one
/// belongs in a bind gate. An `AF_PACKET` socket on a carrier-less bridge is
/// perfectly valid and starts carrying traffic the instant a member port comes
/// up, with no rebind: the socket outlives the carrier. Gating on `IFF_RUNNING`
/// bought nothing and cost three things —
///
/// - `br-lan` on a router with nothing plugged into its LAN ports is `UP` with
/// `NO-CARRIER`, so a perfectly healthy wifi-only router reported `Degraded`
/// forever;
/// - every carrier flap the socket would have survived became an unbind /
/// rebind cycle, which is churn the presence machine then has to damp;
/// - an 802.11s mesh interface that reports `RUNNING` only once it has peered
/// cannot peer, because peering needs beacons, which need a bound socket,
/// which the gate refuses. A deadlock reachable on shipped hardware.
///
/// The signal `IFF_RUNNING` does carry — "is anything plugged in" — is not
/// lost; it is reported alongside presence by [`interface_carrier`] and
/// surfaced in `show_transports`, where an operator can read it without it
/// steering the daemon.
///
/// `getifaddrs` rather than an `SIOCGIFFLAGS` ioctl: it needs no socket, so
/// the presence watcher can poll before any file descriptor exists, and it is
/// spelled the same on Linux and the BSDs.
#[cfg(unix)]
pub fn interface_present(interface: &str) -> bool {
interface_present_probe(interface).unwrap_or(false)
}
/// [`interface_present`], keeping "the probe failed" distinct from "absent".
///
/// `None` means the kernel would not answer. A caller deciding whether to
/// *bind* can treat that as absence and retry on the next tick, which is what
/// [`interface_present`] does. A caller deciding whether to *unbind* must not:
/// see [`interface_has_flags`].
#[cfg(unix)]
pub fn interface_present_probe(interface: &str) -> Option<bool> {
interface_has_flags(interface, libc::IFF_UP as u32)
}
/// The kernel's index for the named interface, or `None` if it does not exist.
///
/// A name is not a device, and neither is a name that is still there. Both
/// backends bind by index — `AF_PACKET` stores `sll_ifindex`, and a BPF
/// descriptor follows the device it was attached to — so an interface deleted
/// and recreated under the same name leaves the socket attached to a device
/// that no longer exists while the *name* resolves perfectly well. Comparing
/// the live index against the one captured at bind is what tells those apart.
#[cfg(unix)]
pub fn interface_index(interface: &str) -> Option<u32> {
let c_name = std::ffi::CString::new(interface).ok()?;
// Cheaper than `getifaddrs`: one syscall, no allocation, no walk.
match unsafe { libc::if_nametoindex(c_name.as_ptr()) } {
0 => None,
idx => Some(idx),
}
}
/// Whether the named interface currently has carrier (`IFF_RUNNING`).
///
/// Reported, never acted on — see [`interface_present`]. `false` for an
/// interface that does not exist, which keeps "no carrier" and "no interface"
/// from being told apart here; presence answers that.
#[cfg(unix)]
pub fn interface_carrier(interface: &str) -> bool {
// Report-only, so a probe failure reads the same as no carrier.
interface_has_flags(interface, (libc::IFF_UP | libc::IFF_RUNNING) as u32).unwrap_or(false)
}
/// Whether the named interface exists and has every flag in `wanted` set, or
/// `None` if the question could not be asked.
///
/// The `None` matters. `getifaddrs` is a netlink dump on Linux and it does
/// fail for reasons that have nothing to do with the interface — `ENOBUFS`
/// under memory pressure or a busy netlink socket, `EMFILE`/`ENFILE` under fd
/// exhaustion, since it opens a socket of its own. Answering `false` there
/// reports a present interface as gone, and a caller holding a live binding
/// would tear a working socket down over a transient syscall failure. Callers
/// that can tell the two apart should.
#[cfg(unix)]
fn interface_has_flags(interface: &str, wanted: u32) -> Option<bool> {
let Ok(c_name) = std::ffi::CString::new(interface) else {
// An interior NUL is not a probe failure — no such interface can
// exist, and no retry will change that.
return Some(false);
};
let mut addrs: *mut libc::ifaddrs = std::ptr::null_mut();
if unsafe { libc::getifaddrs(&mut addrs) } != 0 {
return None;
}
let mut matched = false;
let mut cur = addrs;
while !cur.is_null() {
let entry = unsafe { &*cur };
if !entry.ifa_name.is_null()
&& unsafe { libc::strcmp(entry.ifa_name, c_name.as_ptr()) } == 0
&& entry.ifa_flags & wanted == wanted
{
matched = true;
break;
}
cur = entry.ifa_next;
}
unsafe { libc::freeifaddrs(addrs) };
Some(matched)
}
// Platform-specific PacketSocket implementation.
#[cfg(target_os = "linux")]
#[path = "io_linux.rs"]
@@ -193,6 +307,12 @@ mod async_impl {
/// A received frame: (payload, source_mac).
type Frame = (Vec<u8>, [u8; 6]);
/// Consecutive failed BPF reads before the reader thread gives up.
///
/// Mirrors the receive loop's own error threshold: the point is not to
/// tolerate errors but to end the task so the binder can rebind.
const READ_ERROR_EXIT_THRESHOLD: u32 = 5;
pub struct AsyncPacketSocket {
inner: Arc<PacketSocket>,
/// `None` once shutdown has taken the receiver, which is what makes
@@ -219,6 +339,9 @@ mod async_impl {
let mut parse_buf = vec![0u8; bpf_buflen];
let mut parse_offset: usize = 0;
let mut parse_len: usize = 0;
// Consecutive failed reads, to bound a socket whose
// interface went away underneath it.
let mut read_errors: u32 = 0;
let nfds = bpf_fd.max(shutdown_fd) + 1;
loop {
@@ -286,11 +409,31 @@ mod async_impl {
if err.raw_os_error() == Some(libc::EBADF) {
break;
}
if err.kind() == std::io::ErrorKind::Interrupted {
continue;
}
}
// Anything else — `ENXIO` is the one that matters,
// which is what BPF answers once the interface it
// was attached to is torn away — used to loop here
// forever. That mattered beyond the spin: the
// binder's detach check asks whether this thread is
// still running, so a thread that never returns
// reports a dead socket as a live one, and the
// transport sits `present` and deaf until the name
// or index happens to change too. Give up after a
// streak and let the return close the channel,
// which fails `recv_from`, which ends the tokio
// task the binder is actually watching.
read_errors += 1;
if read_errors >= READ_ERROR_EXIT_THRESHOLD {
break;
}
parse_len = 0;
parse_offset = 0;
continue;
}
read_errors = 0;
parse_len = ret as usize;
parse_offset = 0;
}
+13 -5
View File
@@ -59,6 +59,14 @@ impl PacketSocket {
if ret < 0 {
let err = std::io::Error::last_os_error();
unsafe { libc::close(fd) };
// The interface can disappear between the index lookup and the
// bind. That is absence arriving a few microseconds late, not a
// configuration fault, so it reports as absence.
if matches!(err.raw_os_error(), Some(libc::ENODEV) | Some(libc::ENXIO)) {
return Err(TransportError::InterfaceUnavailable {
interface: interface.to_string(),
});
}
return Err(TransportError::StartFailed(format!(
"bind(AF_PACKET, {}) failed: {}",
interface, err
@@ -231,11 +239,11 @@ fn get_if_index(_fd: RawFd, interface: &str) -> Result<i32, TransportError> {
let idx = unsafe { libc::if_nametoindex(c_name.as_ptr()) };
if idx == 0 {
return Err(TransportError::StartFailed(format!(
"interface not found: {} ({})",
interface,
std::io::Error::last_os_error()
)));
// Absence, not a fault: the caller's presence watcher rebinds when the
// interface shows up. See `TransportError::InterfaceUnavailable`.
return Err(TransportError::InterfaceUnavailable {
interface: interface.to_string(),
});
}
Ok(idx as i32)
}
+15 -7
View File
@@ -393,10 +393,18 @@ fn bind_to_interface(fd: RawFd, interface: &str) -> Result<(), TransportError> {
let ret = unsafe { libc::ioctl(fd, BIOCSETIF, ifreq.as_ptr()) };
if ret < 0 {
let err = std::io::Error::last_os_error();
// BIOCSETIF answers ENXIO for an interface that is not there. The
// interface can also vanish between the index lookup and this ioctl,
// so absence is reported as absence rather than as a bind fault.
if matches!(err.raw_os_error(), Some(libc::ENXIO) | Some(libc::ENODEV)) {
return Err(TransportError::InterfaceUnavailable {
interface: interface.to_string(),
});
}
return Err(TransportError::StartFailed(format!(
"BIOCSETIF({}) failed: {}",
interface,
std::io::Error::last_os_error()
interface, err
)));
}
Ok(())
@@ -498,11 +506,11 @@ fn get_if_index(interface: &str) -> Result<i32, TransportError> {
let idx = unsafe { libc::if_nametoindex(c_name.as_ptr()) };
if idx == 0 {
return Err(TransportError::StartFailed(format!(
"interface not found: {} ({})",
interface,
std::io::Error::last_os_error()
)));
// Absence, not a fault — see the Linux twin and
// `TransportError::InterfaceUnavailable`.
return Err(TransportError::InterfaceUnavailable {
interface: interface.to_string(),
});
}
Ok(idx as i32)
}
File diff suppressed because it is too large Load Diff
+851
View File
@@ -0,0 +1,851 @@
//! Interface presence state for interface-bound transports.
//!
//! A transport bound to a network interface is a long-lived object that is
//! *sometimes bound*. The interface it names may not exist when the daemon
//! starts, may appear minutes later, may vanish and return mid-operation, and
//! may never appear at all. This module holds the state that makes that
//! observable and drives the rebind loop in [`super`].
//!
//! ```text
//! Absent ──attach──> Binding ──ok──> Present
//! ^ │ │
//! └──── fail/backoff ─┘ │
//! └──────────── detach ──────────────┘
//! ```
//!
//! Two invariants do the work:
//!
//! - **The transport object survives detach.** Config, `TransportId`,
//! statistics, and the neighbor buffer persist; only the file descriptor and
//! its loops go. A transport is never destroyed because its interface went
//! away.
//! - **Start-time absence and runtime detach are the same transition.** A node
//! that boots before wifi and a node whose wifi reloads at 03:00 take one
//! code path.
//!
//! Presence tracks `IFF_UP` — the interface exists and the operator has
//! enabled it — and deliberately not `IFF_RUNNING`: binding needs no carrier,
//! and a socket outlives a carrier flap. Carrier is reported alongside it
//! rather than steering it. See
//! [`interface_present`](super::io::interface_present) for why.
use std::sync::atomic::{AtomicU8, AtomicU32, AtomicU64, Ordering};
use std::sync::{PoisonError, RwLock, RwLockReadGuard, RwLockWriteGuard};
use std::time::{Duration, Instant};
/// Read a lock, ignoring poisoning.
///
/// Every value guarded in this module is plain data — an `Instant`, an
/// `Option<[u8; 6]>` — that a panic mid-write cannot leave logically
/// inconsistent, so poisoning carries no information worth propagating.
/// Treating it as a failure is what would hurt: the callers here are on the
/// presence path, and "assume the worst" there means a transport that reports
/// itself bound while every send fails, or a binder that tears down and
/// rebinds every second forever. A stuck state is a worse outcome than
/// reading a byte written by a thread that later panicked.
fn read<T>(lock: &RwLock<T>) -> RwLockReadGuard<'_, T> {
lock.read().unwrap_or_else(PoisonError::into_inner)
}
/// Write a lock, ignoring poisoning. See [`read`].
fn write<T>(lock: &RwLock<T>) -> RwLockWriteGuard<'_, T> {
lock.write().unwrap_or_else(PoisonError::into_inner)
}
/// How absence of the configured interface is reported.
///
/// Describes *the interface's presence*, not the transport's importance: an
/// optional interface that is present is used exactly as hard as any other.
#[derive(Clone, Copy, Debug, PartialEq, Eq)]
pub enum AbsencePolicy {
/// Naming an interface in configuration is a statement that you expect it,
/// so the default is to complain: absence degrades node health from the
/// first edge, and if it outlasts [`ABSENCE_ERROR_AFTER`] — the window in
/// which it could still have been an ordinary bring-up race — it is
/// reported once at `error`.
Required,
/// Absence is normal for this interface (a dock adapter, a radio that only
/// exists on some hardware): no health impact, `info` on the edge.
Optional,
}
impl AbsencePolicy {
/// `optional: true` in configuration selects [`AbsencePolicy::Optional`].
pub fn from_optional(optional: bool) -> Self {
if optional {
Self::Optional
} else {
Self::Required
}
}
/// Whether absence should be hidden from node health.
pub fn is_optional(self) -> bool {
matches!(self, Self::Optional)
}
/// Operator-facing label, used by `show_transports`.
pub fn as_str(self) -> &'static str {
match self {
Self::Required => "required",
Self::Optional => "optional",
}
}
}
/// Where a transport sits in the presence cycle.
#[derive(Clone, Copy, Debug, PartialEq, Eq)]
pub enum Presence {
/// The interface is not there (or is there without carrier). No socket, no
/// loops; the watcher is waiting.
Absent,
/// The interface appeared and a bind is in flight, or a non-absence bind
/// failure is backing off.
Binding,
/// Bound, with a live socket and running loops.
Present,
}
impl Presence {
fn from_u8(v: u8) -> Self {
match v {
1 => Self::Binding,
2 => Self::Present,
_ => Self::Absent,
}
}
fn as_u8(self) -> u8 {
match self {
Self::Absent => 0,
Self::Binding => 1,
Self::Present => 2,
}
}
/// Operator-facing label, used by `show_transports`.
pub fn as_str(self) -> &'static str {
match self {
Self::Absent => "absent",
Self::Binding => "binding",
Self::Present => "present",
}
}
}
impl std::fmt::Display for Presence {
fn fmt(&self, f: &mut std::fmt::Formatter<'_>) -> std::fmt::Result {
f.write_str(self.as_str())
}
}
/// Shared, lock-light presence state.
///
/// Written by the transport's binder task, read by `send`, by the control
/// plane, and by tests. Held behind an `Arc` so the binder task can outlive
/// any particular borrow of the transport.
#[derive(Debug)]
pub struct PresenceState {
phase: AtomicU8,
/// Successful binds since the transport was created. `1` after a clean
/// start; every increment past that is a rebind.
binds: AtomicU64,
/// Failed bind attempts since the last successful bind. Reset on bind.
attempts: AtomicU32,
/// When the current phase was entered.
since: RwLock<Instant>,
/// MAC observed at the last successful bind. A name reappearing with a
/// different MAC is different hardware, not the same device returning.
last_mac: RwLock<Option<[u8; 6]>>,
}
impl Default for PresenceState {
fn default() -> Self {
Self::new()
}
}
impl PresenceState {
/// A fresh tracker in [`Presence::Absent`].
pub fn new() -> Self {
Self {
phase: AtomicU8::new(Presence::Absent.as_u8()),
binds: AtomicU64::new(0),
attempts: AtomicU32::new(0),
since: RwLock::new(Instant::now()),
last_mac: RwLock::new(None),
}
}
/// Current phase.
pub fn presence(&self) -> Presence {
Presence::from_u8(self.phase.load(Ordering::Acquire))
}
/// Whether the transport currently holds a bound socket.
pub fn is_present(&self) -> bool {
self.presence() == Presence::Present
}
/// How long the current presence *episode* has been held.
///
/// Episode, not phase: `Binding` is part of the absence episode until it
/// succeeds. An interface that has been gone for a week while a bind is
/// retried and refused every second must report a week, not one second —
/// otherwise [`ABSENCE_ERROR_AFTER`] is never reached and the
/// operator-facing `since_secs` reads as a healthy young absence forever.
pub fn since(&self) -> Duration {
read(&self.since).elapsed()
}
/// Restart the episode clock, for a transport that is about to start.
///
/// `new()` stamps the clock at construction, but construction and
/// `start_async` need not be adjacent — config load and supervisor staging
/// sit between them. Left alone, a transport staged for longer than
/// [`ABSENCE_ERROR_AFTER`] logs the sustained-absence error on its very
/// first binder tick, having given the interface no bring-up window at
/// all. The window is supposed to absorb exactly that race.
pub fn mark_starting(&self) {
*write(&self.since) = Instant::now();
}
/// Test-only: age the episode clock, so deadline behaviour can be tested
/// without sleeping through it.
///
/// `ABSENCE_ERROR_AFTER` is ten seconds. A test that waited it out would
/// be ten seconds of nothing, which is how deadline logic ends up
/// untested.
#[cfg(test)]
pub(crate) fn backdate_for_test(&self, by: Duration) {
let mut since = write(&self.since);
*since = since.checked_sub(by).unwrap_or(*since);
}
/// Successful binds since creation (`1` after a clean start).
pub fn binds(&self) -> u64 {
self.binds.load(Ordering::Relaxed)
}
/// Failed bind attempts since the last successful bind.
pub fn attempts(&self) -> u32 {
self.attempts.load(Ordering::Relaxed)
}
/// MAC observed at the last successful bind, if any.
pub fn last_mac(&self) -> Option<[u8; 6]> {
*read(&self.last_mac)
}
/// Move to `phase`, returning `true` if this was an actual edge.
///
/// Edge-vs-level is what the logging policy keys on: logged once on
/// entering absence and once on recovery, never per retry attempt.
///
/// The episode clock ([`Self::since`]) restarts only when *boundness*
/// changes — Present↔not-Present. A failed bind walks
/// `Absent → Binding → Absent`, and resetting the clock on those would
/// hide a permanent absence behind a timer that never gets past one
/// second.
pub fn transition(&self, phase: Presence) -> bool {
let prev = Presence::from_u8(self.phase.swap(phase.as_u8(), Ordering::AcqRel));
if prev == phase {
return false;
}
if (prev == Presence::Present) != (phase == Presence::Present) {
*write(&self.since) = Instant::now();
}
true
}
/// Record a successful bind at `mac`.
///
/// Returns `true` when the interface came back as *different hardware* —
/// the name reappeared with a MAC other than the one last bound. The
/// caller drops cached neighbor state rather than silently resuming onto
/// a different adapter.
pub fn record_bind(&self, mac: [u8; 6]) -> bool {
let changed = match *read(&self.last_mac) {
Some(prev) => prev != mac,
None => false,
};
*write(&self.last_mac) = Some(mac);
self.binds.fetch_add(1, Ordering::Relaxed);
self.attempts.store(0, Ordering::Relaxed);
self.transition(Presence::Present);
changed
}
/// Record a failed bind attempt, returning the new attempt count.
pub fn record_attempt(&self) -> u32 {
self.attempts.fetch_add(1, Ordering::Relaxed) + 1
}
}
/// Backoff for bind failures that are *not* absence — permission denied,
/// buffer sizing, a BPF device shortage. Absence itself does not back off
/// where an event source is available: there is nothing to poll.
///
/// 1 s doubling to a 30 s ceiling.
pub fn bind_backoff(attempts: u32) -> Duration {
const BASE_SECS: u64 = 1;
const CEILING_SECS: u64 = 30;
let shift = attempts.saturating_sub(1).min(5);
Duration::from_secs((BASE_SECS << shift).min(CEILING_SECS))
}
/// How long a *required* interface may be absent before it is an error.
///
/// Absence is a state the presence machine handles, so it is not an error for
/// happening — a daemon that wins the race against its own radio, or a cable
/// out for two seconds, is the ordinary case this mechanism exists to absorb,
/// and calling that an error at t=0 and "recovered" at t=0.2 s is cry-wolf.
/// Past this window it is no longer a race: something an operator has to fix
/// is wrong, and the log should say so once.
///
/// One window for both shapes of absence. A node that boots before its wifi
/// and a node whose wifi reloads at 03:00 take one code path everywhere else
/// in this module; giving them different deadlines would reintroduce exactly
/// the start-versus-runtime asymmetry the presence machine removed.
///
/// Tuned against the platforms this exists for: comfortably past a veth or a
/// container coming up, short enough that a mesh radio which never appears is
/// named while somebody is still watching the boot. Raising it hides a real
/// fault for longer; lowering it starts reporting ordinary bring-up races.
pub const ABSENCE_ERROR_AFTER: Duration = Duration::from_secs(10);
/// Minimum lifetime for a binding to count as a real recovery.
///
/// A socket that dies sooner than this never really came back.
pub const MIN_STABLE_BINDING: Duration = Duration::from_secs(10);
/// Consecutive short-lived bindings before the binder stops treating a
/// successful bind as a recovery.
pub const CHURN_THRESHOLD: u32 = 3;
/// Damping for the rebind loop.
///
/// Backoff covers *failed* binds; this covers the opposite and nastier case —
/// binds that keep **succeeding** into a socket that dies moments later. A
/// receive loop that gives up on a persistent error while the interface stays
/// `UP` produces exactly that: tear down, rebind, succeed, fail again, once
/// per second, forever. Undamped it is an `error!`/`info!` pair and a
/// `Degraded`→`Running` health flap every cycle, which defeats both the
/// "log edges, not attempts" rule and the meaning of `Degraded`.
///
/// So: count consecutive bindings that die young, back off between them on
/// the same 1 s → 30 s curve, and once the streak reaches
/// [`CHURN_THRESHOLD`] stop announcing each bind as a recovery — hold the
/// node at its degraded reading until a binding actually survives
/// [`MIN_STABLE_BINDING`]. A binding that holds ends the streak.
///
/// Pure state, driven by an injected clock, so the policy is testable without
/// a network interface.
#[derive(Debug, Default)]
pub struct ChurnGuard {
/// Consecutive bindings that died younger than [`MIN_STABLE_BINDING`].
streak: u32,
/// When the current binding was established.
bound_at: Option<Instant>,
/// Whether the current binding was announced as a recovery.
announced: bool,
}
/// What the caller should do about a successful bind.
#[derive(Clone, Copy, Debug, PartialEq, Eq)]
pub struct BindOutcome {
/// Log the recovery and publish presence now. `false` while churning:
/// the bind is held back until it proves it will last.
pub announce: bool,
}
/// What the caller should do about a detach.
#[derive(Clone, Copy, Debug, PartialEq, Eq)]
pub struct DetachOutcome {
/// Publish absence. `false` when the binding that just died was never
/// announced, so there is nothing to retract.
pub retract: bool,
/// Log the detach edge. `false` once churning — the fact has been said.
pub log_edge: bool,
/// This detach is the one that crossed [`CHURN_THRESHOLD`]; say so once.
pub entered_churn: bool,
/// Wait this long before trying to bind again.
pub backoff: Option<Duration>,
}
impl ChurnGuard {
/// A guard with no history.
pub fn new() -> Self {
Self::default()
}
/// Consecutive short-lived bindings, for logging and tests.
pub fn streak(&self) -> u32 {
self.streak
}
/// Record a successful bind.
pub fn bound(&mut self, now: Instant) -> BindOutcome {
self.bound_at = Some(now);
let announce = self.streak < CHURN_THRESHOLD;
if announce {
self.announced = true;
}
BindOutcome { announce }
}
/// Called on every tick while bound. Returns `true` exactly once, at the
/// moment a held-back binding has proved stable and should be announced.
pub fn stabilized(&mut self, now: Instant) -> bool {
let Some(bound_at) = self.bound_at else {
return false;
};
if now.duration_since(bound_at) < MIN_STABLE_BINDING {
return false;
}
let newly_announced = !self.announced;
self.streak = 0;
self.announced = true;
newly_announced
}
/// Record a detach.
pub fn detached(&mut self, now: Instant) -> DetachOutcome {
let young = self
.bound_at
.is_some_and(|t| now.duration_since(t) < MIN_STABLE_BINDING);
self.bound_at = None;
if young {
self.streak += 1;
} else {
self.streak = 0;
}
DetachOutcome {
retract: std::mem::take(&mut self.announced),
log_edge: self.streak < CHURN_THRESHOLD,
entered_churn: self.streak == CHURN_THRESHOLD,
backoff: (self.streak > 0).then(|| bind_backoff(self.streak)),
}
}
}
#[cfg(test)]
mod tests {
use super::*;
use std::sync::Arc;
#[test]
fn presence_starts_absent() {
let p = PresenceState::new();
assert_eq!(p.presence(), Presence::Absent);
assert_eq!(p.binds(), 0);
assert_eq!(p.attempts(), 0);
assert!(p.last_mac().is_none());
}
#[test]
fn transition_reports_edges_only() {
let p = PresenceState::new();
assert!(p.transition(Presence::Binding));
assert!(!p.transition(Presence::Binding));
assert!(p.transition(Presence::Present));
}
#[test]
fn bind_records_mac_and_clears_attempts() {
let p = PresenceState::new();
p.record_attempt();
p.record_attempt();
assert_eq!(p.attempts(), 2);
let mac = [0x02, 0, 0, 0, 0, 1];
assert!(!p.record_bind(mac), "first bind is not a hardware change");
assert_eq!(p.attempts(), 0);
assert_eq!(p.binds(), 1);
assert_eq!(p.presence(), Presence::Present);
assert_eq!(p.last_mac(), Some(mac));
}
#[test]
fn rebind_on_same_mac_is_not_a_hardware_change() {
let p = PresenceState::new();
let mac = [0x02, 0, 0, 0, 0, 1];
p.record_bind(mac);
p.transition(Presence::Absent);
assert!(!p.record_bind(mac));
assert_eq!(p.binds(), 2);
}
#[test]
fn rebind_on_different_mac_is_a_hardware_change() {
let p = PresenceState::new();
p.record_bind([0x02, 0, 0, 0, 0, 1]);
p.transition(Presence::Absent);
assert!(p.record_bind([0x02, 0, 0, 0, 0, 2]));
}
#[test]
fn backoff_climbs_to_a_ceiling() {
assert_eq!(bind_backoff(0), Duration::from_secs(1));
assert_eq!(bind_backoff(1), Duration::from_secs(1));
assert_eq!(bind_backoff(2), Duration::from_secs(2));
assert_eq!(bind_backoff(3), Duration::from_secs(4));
assert_eq!(bind_backoff(6), Duration::from_secs(30));
assert_eq!(bind_backoff(u32::MAX), Duration::from_secs(30));
}
#[test]
fn the_error_deadline_outlasts_an_ordinary_bring_up_race() {
// The window has to clear the races it exists to absorb — a veth
// arriving a fraction of a second late, a container starting — while
// staying short enough that a radio which never appears is named
// during the boot somebody is watching. It is also the one deadline:
// start-time absence and a runtime detach share it.
assert!(
ABSENCE_ERROR_AFTER >= Duration::from_secs(5),
"shorter than a bring-up race would report the ordinary case"
);
assert!(
ABSENCE_ERROR_AFTER <= Duration::from_secs(60),
"longer and a required interface that never appears goes unsaid \
for the whole boot"
);
}
// ── The absence clock measures an episode, not a phase ────────────────
#[test]
fn a_failed_bind_does_not_restart_the_absence_clock() {
// An interface that is present but refuses to bind walks
// Absent → Binding → Absent on every retry. If those edges reset the
// clock, `since_secs` reads as a one-second-old absence forever and
// the error deadline is never reached — so a permission error would
// sit silently behind a healthy-looking counter.
let p = PresenceState::new();
std::thread::sleep(Duration::from_millis(30));
let before = p.since();
assert!(p.transition(Presence::Binding));
assert!(p.transition(Presence::Absent));
assert!(p.transition(Presence::Binding));
assert!(p.transition(Presence::Absent));
assert!(
p.since() >= before,
"the absence clock ran backwards across failed binds"
);
}
#[test]
fn the_clock_restarts_only_when_boundness_changes() {
let p = PresenceState::new();
std::thread::sleep(Duration::from_millis(30));
// Absent → Present restarts it: a new episode began.
p.record_bind([0x02, 0, 0, 0, 0, 1]);
assert!(p.since() < Duration::from_millis(30));
std::thread::sleep(Duration::from_millis(30));
let bound_for = p.since();
// Present → Present is not an edge at all.
assert!(!p.transition(Presence::Present));
assert!(p.since() >= bound_for);
// Present → Absent restarts it: the episode ended.
assert!(p.transition(Presence::Absent));
assert!(p.since() < Duration::from_millis(30));
}
// ── Poisoning must not be a stuck state ───────────────────────────────
#[test]
fn a_poisoned_lock_still_reports_presence() {
// `.ok()`-style handling would make a poisoned lock read as "no MAC,
// no socket, tasks dead" — a transport reporting itself present while
// every send fails, and a binder rebinding once a second forever.
// Poisoning carries no information about plain data, so it is ignored.
let p = Arc::new(PresenceState::new());
p.record_bind([0x02, 0, 0, 0, 0, 7]);
let poisoner = Arc::clone(&p);
let panicked = std::thread::spawn(move || {
let _guard = poisoner.last_mac.write().unwrap();
panic!("poison the lock while holding it");
})
.join();
assert!(panicked.is_err(), "the helper thread was supposed to panic");
assert!(
p.last_mac.is_poisoned(),
"the lock was supposed to be poisoned"
);
assert_eq!(
p.last_mac(),
Some([0x02, 0, 0, 0, 0, 7]),
"a poisoned lock must not erase the binding"
);
// And the clock still answers rather than collapsing to zero.
let _ = p.since();
}
// ── Rebind churn ──────────────────────────────────────────────────────
#[test]
fn a_healthy_bind_and_detach_is_not_churn() {
let mut g = ChurnGuard::new();
let t0 = Instant::now();
assert!(g.bound(t0).announce, "a first bind is a recovery");
// Held well past the stability floor, then lost.
let out = g.detached(t0 + MIN_STABLE_BINDING + Duration::from_secs(60));
assert!(out.retract, "an announced binding must be retracted");
assert!(out.log_edge, "an isolated detach is worth a line");
assert!(!out.entered_churn);
assert_eq!(out.backoff, None, "one clean outage must not back off");
assert_eq!(g.streak(), 0);
}
#[test]
fn an_unseeded_guard_retracts_nothing_and_never_repairs_itself() {
// Why `binder_loop` seeds the guard when it inherits a binding from
// `start_async`, rather than leaving it fresh.
//
// A guard that was never told about a bind believes it has announced
// nothing, so it asks for no retraction — and health, which learned
// `present: true` from the inline bind, would keep reading `Full` with
// the interface gone. `stabilized` cannot rescue it either: with no
// `bound_at` there is nothing for it to judge stable.
let mut g = ChurnGuard::new();
let t0 = Instant::now();
assert!(
!g.stabilized(t0 + MIN_STABLE_BINDING + Duration::from_secs(60)),
"a guard with no recorded bind has nothing to stabilize"
);
let out = g.detached(t0 + Duration::from_secs(60));
assert!(
!out.retract,
"an unseeded guard retracts nothing — which is exactly why the \
binder must seed it from the inline bind"
);
}
#[test]
fn a_seeded_guard_retracts_the_edge_the_inline_bind_published() {
// The fix, from the binder's angle: seeding with `bound` is what makes
// the first detach after a clean start reach node health.
let mut g = ChurnGuard::new();
let t0 = Instant::now();
g.bound(t0);
let out = g.detached(t0 + MIN_STABLE_BINDING + Duration::from_secs(60));
assert!(
out.retract,
"the edge `start_async` published must be retracted on detach"
);
assert!(out.log_edge);
}
#[test]
fn short_lived_bindings_back_off() {
// The failure this guards: a receive loop that gives up on a
// persistent error while the interface stays UP. Bind succeeds, dies,
// rebinds, dies — once per second, forever, undamped.
let mut g = ChurnGuard::new();
let mut t = Instant::now();
for expected in [1u64, 2, 4] {
g.bound(t);
t += Duration::from_secs(1);
let out = g.detached(t);
assert_eq!(
out.backoff,
Some(Duration::from_secs(expected)),
"streak {} should back off {expected}s",
g.streak()
);
}
assert_eq!(g.streak(), 3);
}
#[test]
fn churn_stops_announcing_and_stops_logging() {
let mut g = ChurnGuard::new();
let mut t = Instant::now();
// The detaches below the threshold are still news and still logged.
for _ in 0..CHURN_THRESHOLD - 1 {
assert!(g.bound(t).announce);
t += Duration::from_secs(1);
let out = g.detached(t);
assert!(out.log_edge, "the first few detaches are still news");
assert!(!out.entered_churn);
}
// The detach that crosses the threshold reports the churn instead of
// the edge: one line saying "this keeps happening", not two saying
// "it happened" and "it keeps happening".
assert!(g.bound(t).announce);
t += Duration::from_secs(1);
let crossing = g.detached(t);
assert!(crossing.entered_churn, "crossing must be announced once");
assert!(!crossing.log_edge, "the churn line replaces the edge line");
// Past the threshold: bindings are no longer announced as recoveries,
// so node health stays put instead of flapping every second, and the
// edges stop being logged.
for _ in 0..5 {
assert!(!g.bound(t).announce, "a churning bind is not a recovery");
t += Duration::from_secs(1);
let out = g.detached(t);
assert!(!out.log_edge, "churn must not log per cycle");
assert!(!out.retract, "nothing was announced, so nothing to retract");
assert!(
!out.entered_churn,
"the threshold is crossed once, not repeatedly"
);
}
}
#[test]
fn backoff_during_churn_is_capped() {
let mut g = ChurnGuard::new();
let mut t = Instant::now();
let mut last = None;
for _ in 0..12 {
g.bound(t);
t += Duration::from_secs(1);
last = g.detached(t).backoff;
}
assert_eq!(
last,
Some(Duration::from_secs(30)),
"churn backoff must climb to the ceiling and stop"
);
}
#[test]
fn a_binding_that_lasts_ends_the_streak_and_announces_once() {
let mut g = ChurnGuard::new();
let mut t = Instant::now();
// Churn into the held-back state.
for _ in 0..CHURN_THRESHOLD + 1 {
g.bound(t);
t += Duration::from_secs(1);
g.detached(t);
}
assert!(!g.bound(t).announce);
// Not yet stable: still nothing to say.
assert!(!g.stabilized(t + Duration::from_secs(1)));
// Survived the floor: announce exactly once, and the streak is over.
let stable_at = t + MIN_STABLE_BINDING;
assert!(
g.stabilized(stable_at),
"a binding that lasts is a recovery"
);
assert!(
!g.stabilized(stable_at + Duration::from_secs(60)),
"recovery is announced once, not on every tick"
);
assert_eq!(g.streak(), 0);
// And the next detach behaves like an ordinary one again.
let out = g.detached(stable_at + Duration::from_secs(60));
assert!(out.retract);
assert!(out.log_edge);
assert_eq!(out.backoff, None);
}
#[test]
fn stabilized_is_silent_for_an_ordinary_binding() {
// A bind that was announced immediately must not be announced again
// when it passes the stability floor.
let mut g = ChurnGuard::new();
let t = Instant::now();
assert!(g.bound(t).announce);
assert!(!g.stabilized(t + MIN_STABLE_BINDING + Duration::from_secs(1)));
}
#[test]
fn policy_labels() {
assert_eq!(AbsencePolicy::from_optional(true), AbsencePolicy::Optional);
assert_eq!(AbsencePolicy::from_optional(false), AbsencePolicy::Required);
assert!(AbsencePolicy::Optional.is_optional());
assert!(!AbsencePolicy::Required.is_optional());
assert_eq!(AbsencePolicy::Required.as_str(), "required");
// Both labels, not just one. `show_transports` renders this string and
// fipstop's severity split keys on it, so a swapped pair would paint
// every expected interface as the tolerated kind and vice versa —
// while a test that checks only `Required` stays green through it.
assert_eq!(AbsencePolicy::Optional.as_str(), "optional");
}
#[test]
fn a_first_bind_is_not_a_hardware_change() {
// The boundary the flush hangs off. `record_bind` returns "different
// hardware", and on the very first bind there is no previous MAC to
// differ from — so it must answer false, or every clean start would
// drop a neighbour cache it had just built and log a hardware swap
// that never happened.
let state = PresenceState::new();
assert!(
!state.record_bind([1, 2, 3, 4, 5, 6]),
"the first bind has nothing to differ from"
);
assert_eq!(state.binds(), 1);
assert_eq!(state.presence(), Presence::Present);
}
#[test]
fn a_rebind_on_new_hardware_reports_the_change_once() {
// And it reports the change once, not on every subsequent bind: the
// caller drops its cached neighbours on a `true`, so a sticky answer
// would flush the cache on every rebind forever.
let state = PresenceState::new();
state.record_bind([1, 2, 3, 4, 5, 6]);
assert!(
state.record_bind([0xaa, 0xbb, 0xcc, 0xdd, 0xee, 0xff]),
"a name returning on a different MAC is different hardware"
);
assert!(
!state.record_bind([0xaa, 0xbb, 0xcc, 0xdd, 0xee, 0xff]),
"the same hardware rebinding is not a change"
);
assert_eq!(state.binds(), 3);
}
#[test]
fn every_presence_label_is_distinct_and_round_trips() {
// The labels are the operator-facing vocabulary — `show_transports`
// emits them and the fipstop State column renders them — and
// `Presence::as_str` had no test at all, so `binding` in particular was
// never observed by anything.
use std::collections::HashSet;
let all = [Presence::Absent, Presence::Binding, Presence::Present];
let labels: HashSet<&str> = all.iter().map(|p| p.as_str()).collect();
assert_eq!(labels.len(), 3, "each phase needs its own label");
assert!(labels.contains("binding"));
for phase in all {
assert_eq!(
Presence::from_u8(phase.as_u8()),
phase,
"{phase} must survive the atomic round trip the state uses"
);
assert_eq!(phase.to_string(), phase.as_str(), "Display must agree");
}
// Anything outside the enum reads as absent rather than panicking: the
// byte comes out of an AtomicU8 that a torn write could leave at any
// value, and the safe answer there is "not bound".
assert_eq!(Presence::from_u8(99), Presence::Absent);
}
}
+337
View File
@@ -0,0 +1,337 @@
//! Link-event sources for the interface presence watcher.
//!
//! The presence machine works on a 1-second poll alone. This module removes
//! the latency: where the kernel offers a link-event source, the binder blocks
//! on it and reacts in sub-second time, and the poll stays underneath as a
//! backstop rather than as the mechanism.
//!
//! | Platform | Source |
//! | -------- | ------ |
//! | Linux | netlink `RTNLGRP_LINK` (`RTM_NEWLINK` / `RTM_DELLINK`) |
//! | macOS, FreeBSD | `PF_ROUTE` socket, `RTM_IFINFO` |
//! | Fallback | poll `getifaddrs` + flags, 1 s |
//!
//! The messages themselves are deliberately **not parsed**. A link event is a
//! hint to re-run the presence probe, which is cheap and authoritative;
//! decoding `nlmsghdr`/`ifinfomsg` payloads to reach the same answer would add
//! a parser whose bugs would be presence bugs. Any event on the socket wakes
//! the binder, which then asks
//! [`interface_present`](super::io::interface_present).
//!
//! Construction is best-effort. A kernel or sandbox that refuses the socket
//! yields a watcher that never fires, and the binder degrades to its poll.
use std::os::unix::io::{AsRawFd, RawFd};
use std::sync::atomic::{AtomicBool, AtomicU32, Ordering};
use std::time::Duration;
use tokio::io::unix::AsyncFd;
use tracing::{debug, warn};
/// Consecutive receive errors before the event source is abandoned for the
/// caller's poll.
const ERROR_GIVE_UP: u32 = 5;
/// An owned link-event socket. Closes its descriptor on drop.
struct LinkEventSocket {
fd: RawFd,
}
impl AsRawFd for LinkEventSocket {
fn as_raw_fd(&self) -> RawFd {
self.fd
}
}
impl Drop for LinkEventSocket {
fn drop(&mut self) {
unsafe { libc::close(self.fd) };
}
}
impl LinkEventSocket {
fn recv(&self, buf: &mut [u8]) -> std::io::Result<usize> {
let n = unsafe { libc::recv(self.fd, buf.as_mut_ptr() as *mut libc::c_void, buf.len(), 0) };
if n < 0 {
Err(std::io::Error::last_os_error())
} else {
Ok(n as usize)
}
}
}
/// Open the platform's link-event socket, non-blocking.
#[cfg(target_os = "linux")]
fn open_link_socket() -> std::io::Result<LinkEventSocket> {
// RTMGRP_LINK. Spelled as a literal because the constant's name and
// availability differ across libc versions; the value is ABI.
const RTMGRP_LINK: u32 = 1;
let fd = unsafe {
libc::socket(
libc::AF_NETLINK,
libc::SOCK_RAW | libc::SOCK_NONBLOCK | libc::SOCK_CLOEXEC,
libc::NETLINK_ROUTE,
)
};
if fd < 0 {
return Err(std::io::Error::last_os_error());
}
let socket = LinkEventSocket { fd };
let mut sa: libc::sockaddr_nl = unsafe { std::mem::zeroed() };
sa.nl_family = libc::AF_NETLINK as u16;
sa.nl_groups = RTMGRP_LINK;
let ret = unsafe {
libc::bind(
fd,
&sa as *const libc::sockaddr_nl as *const libc::sockaddr,
std::mem::size_of::<libc::sockaddr_nl>() as libc::socklen_t,
)
};
if ret < 0 {
return Err(std::io::Error::last_os_error());
}
Ok(socket)
}
/// Open the platform's link-event socket, non-blocking.
#[cfg(not(target_os = "linux"))]
fn open_link_socket() -> std::io::Result<LinkEventSocket> {
// PF_ROUTE delivers RTM_IFINFO (and the rest of the routing messages) to
// every reader; no bind and no group selection exist for it.
let fd = unsafe { libc::socket(libc::PF_ROUTE, libc::SOCK_RAW, libc::AF_UNSPEC) };
if fd < 0 {
return Err(std::io::Error::last_os_error());
}
let socket = LinkEventSocket { fd };
let flags = unsafe { libc::fcntl(fd, libc::F_GETFL) };
if flags < 0 {
return Err(std::io::Error::last_os_error());
}
if unsafe { libc::fcntl(fd, libc::F_SETFL, flags | libc::O_NONBLOCK) } < 0 {
return Err(std::io::Error::last_os_error());
}
Ok(socket)
}
/// A source of "something about the links changed" wake-ups.
pub(crate) struct LinkWatcher {
/// `None` when no event source could be opened — the caller's poll is then
/// the whole mechanism, which is exactly the documented fallback.
inner: Option<AsyncFd<LinkEventSocket>>,
/// Consecutive receive errors. Reset by any successful read.
errors: AtomicU32,
/// Set once the source has been abandoned for good.
///
/// Abandonment has to outlive the future that decided it. `changed()` is
/// called fresh on every pass of the caller's `select!` and dropped
/// whenever the poll ticker wins, so a `pending()` inside that future
/// parks nothing beyond the current pass — without this flag the next pass
/// re-reads the dead socket, re-counts the error, and re-logs the
/// give-up warning, once per wake-up, forever.
given_up: AtomicBool,
}
impl LinkWatcher {
/// Open the platform link-event source, falling back to nothing.
pub(crate) fn new() -> Self {
let inner = match open_link_socket() {
Ok(socket) => match AsyncFd::new(socket) {
Ok(afd) => Some(afd),
Err(e) => {
debug!(error = %e, "Link event socket not registrable; polling instead");
None
}
},
Err(e) => {
debug!(error = %e, "No link event source available; polling instead");
None
}
};
Self {
inner,
errors: AtomicU32::new(0),
given_up: AtomicBool::new(false),
}
}
/// Whether an event source is actually backing this watcher.
pub(crate) fn is_event_driven(&self) -> bool {
self.inner.is_some()
}
/// Resolve when the kernel reports a link change.
///
/// Never resolves when no event source is available, which makes it safe
/// to `select!` against the poll ticker: the ticker simply always wins.
pub(crate) async fn changed(&self) {
let Some(afd) = &self.inner else {
std::future::pending::<()>().await;
unreachable!("pending never resolves")
};
// Already abandoned on an earlier pass. Park without touching the
// socket, so giving up costs one syscall in total rather than one per
// caller wake-up for the life of the process.
if self.given_up.load(Ordering::Relaxed) {
std::future::pending::<()>().await;
unreachable!("pending never resolves")
}
loop {
let Ok(mut guard) = afd.readable().await else {
// The registration died. Stop firing rather than spinning; the
// caller's poll continues to cover presence.
std::future::pending::<()>().await;
unreachable!("pending never resolves")
};
// Drain to WouldBlock so a burst of link messages is one wake-up
// and the socket buffer does not fill behind us.
let mut buf = [0u8; 4096];
let mut saw_event = false;
let mut failure = None;
loop {
match guard.try_io(|inner| inner.get_ref().recv(&mut buf)) {
Ok(Ok(n)) if n > 0 => saw_event = true,
// A zero-length read. Readiness is *not* cleared by
// `try_io` here — it clears only on `WouldBlock` — so
// breaking out plainly would leave `readable()` instantly
// ready with nothing to read, and this loop would spin
// without ever returning `Pending`. That starves the
// caller's `select!` of its poll ticker entirely, which
// takes presence detection down with it. Clear it by hand
// and treat it as a fault, so the give-up path applies.
Ok(Ok(_)) => {
guard.clear_ready();
failure = Some(std::io::Error::from(std::io::ErrorKind::UnexpectedEof));
break;
}
// A genuine socket error. Distinct from WouldBlock, and
// the distinction is the whole point: `try_io` clears
// readiness only on WouldBlock, so breaking out of a real
// error leaves `readable()` instantly ready, `recv`
// failing again, and the loop spinning a core flat with
// nothing logged. Clear it by hand and back off.
Ok(Err(e)) => {
guard.clear_ready();
failure = Some(e);
break;
}
// WouldBlock — readiness is cleared, drain complete.
Err(_) => break,
}
}
if saw_event {
self.errors.store(0, Ordering::Relaxed);
return;
}
if let Some(e) = failure {
let errors = self.errors.fetch_add(1, Ordering::Relaxed) + 1;
if errors == 1 {
// ENOBUFS is the realistic one: a burst of link events
// overflowed the socket buffer, so the kernel dropped some.
// Losing events is survivable — the caller polls — but the
// spin is not, and neither is doing it silently.
warn!(error = %e, "Link event source read failed");
}
if errors >= ERROR_GIVE_UP {
self.given_up.store(true, Ordering::Relaxed);
warn!(
errors,
"Link event source is not recoverable; falling back to \
polling for interface presence"
);
std::future::pending::<()>().await;
unreachable!("pending never resolves")
}
tokio::time::sleep(Duration::from_millis(100) * errors).await;
}
}
}
}
#[cfg(test)]
mod tests {
use super::*;
/// The watcher must construct on any host, with or without a usable event
/// source, because the binder builds one unconditionally.
#[tokio::test]
async fn watcher_constructs_and_reports_its_backing() {
let w = LinkWatcher::new();
// Both answers are legitimate — a sandbox may refuse the socket — so
// this pins that asking is safe, not which answer comes back.
let _ = w.is_event_driven();
}
/// A descriptor whose `recv` always fails must not become a busy loop.
///
/// `try_io` clears readiness only on `WouldBlock`. Breaking out of a real
/// error left `readable()` instantly ready, `recv` failing again, and the
/// loop spinning a core flat with nothing logged — the realistic trigger
/// being `ENOBUFS` when a burst of link events overflows the socket
/// buffer. A pipe stands in for that here: `recv` on one answers
/// `ENOTSOCK`, every time, which is exactly the shape of a persistent
/// error.
#[tokio::test]
async fn a_persistently_failing_source_gives_up_instead_of_spinning() {
let mut fds = [0i32; 2];
assert_eq!(unsafe { libc::pipe(fds.as_mut_ptr()) }, 0, "pipe()");
let (read_fd, write_fd) = (fds[0], fds[1]);
// AsyncFd requires a non-blocking descriptor.
let flags = unsafe { libc::fcntl(read_fd, libc::F_GETFL) };
assert!(unsafe { libc::fcntl(read_fd, libc::F_SETFL, flags | libc::O_NONBLOCK) } >= 0);
let watcher = LinkWatcher {
inner: Some(AsyncFd::new(LinkEventSocket { fd: read_fd }).expect("register")),
errors: AtomicU32::new(0),
given_up: AtomicBool::new(false),
};
// Keep producing readiness edges. The error arm calls `clear_ready`,
// and a descriptor that was already readable before re-registration
// may never deliver another edge on its own — which would stall the
// loop at one error and hide whether the give-up path works. A steady
// trickle stands in for the burst of link events that provokes the
// real failure.
let writer = tokio::task::spawn_blocking(move || {
for _ in 0..200 {
if unsafe { libc::write(write_fd, b"x".as_ptr().cast(), 1) } < 0 {
break;
}
std::thread::sleep(Duration::from_millis(25));
}
unsafe { libc::close(write_fd) };
});
// Never resolves — there is no event to report — but it must reach the
// give-up state rather than burn until the timeout.
let fired = tokio::time::timeout(Duration::from_secs(5), watcher.changed()).await;
assert!(fired.is_err(), "a failing source must not report an event");
assert!(
watcher.errors.load(Ordering::Relaxed) >= ERROR_GIVE_UP,
"the error path must count, back off and stop, not spin silently"
);
writer.abort();
}
/// A watcher with no event source must never resolve, so a `select!`
/// against the poll ticker degrades cleanly instead of spinning.
#[tokio::test]
async fn a_sourceless_watcher_never_fires() {
let w = LinkWatcher {
inner: None,
errors: AtomicU32::new(0),
given_up: AtomicBool::new(false),
};
let fired = tokio::time::timeout(std::time::Duration::from_millis(50), w.changed()).await;
assert!(fired.is_err(), "sourceless watcher resolved");
}
}
+116
View File
@@ -112,6 +112,62 @@ pub fn packet_channel(buffer: usize) -> (PacketTx, PacketRx) {
tokio::sync::mpsc::channel(buffer)
}
/// Operator-visible interface presence, rendered by `show_transports`.
///
/// Worth as much as the retry itself. The original boot-race bug was expensive
/// precisely because the 802.11s peer link formed regardless of the daemon, so
/// nothing an operator could see said the node was deaf.
#[derive(Clone, Copy, Debug, PartialEq, Eq)]
pub struct InterfacePresence {
/// `absent`, `binding`, or `present`.
pub presence: &'static str,
/// Whether the interface currently has carrier (`IFF_RUNNING`).
///
/// Reported, never acted on. Presence is `IFF_UP`, because binding does
/// not need carrier and a socket outlives a carrier flap — but "is
/// anything plugged in" is still what an operator wants to know when a
/// bound transport is carrying nothing, so it is reported here instead of
/// steering the daemon.
pub carrier: bool,
/// `required` or `optional`.
pub policy: &'static str,
/// How long the current phase has been held.
pub since_secs: u64,
/// Successful binds since the transport was created (`1` after a clean
/// start; more means it has rebound).
pub binds: u64,
/// Failed bind attempts since the last successful bind.
pub failed_attempts: u32,
}
/// A presence edge published by an interface-bound transport.
///
/// Absence and return are the same transition seen from two sides, so one
/// event type carries both: `present: false` on detach (including a start
/// where the interface was never there), `present: true` on every successful
/// bind after the first observation.
#[derive(Clone, Copy, Debug, PartialEq, Eq)]
pub struct TransportPresence {
/// The transport whose interface changed presence.
pub transport_id: TransportId,
/// Whether the interface is now bound.
pub present: bool,
/// Whether this edge should move node health.
///
/// `false` for an `optional` interface, whose absence is normal and must
/// not take the node off `Full`. The edge is still published, because the
/// bound set changed either way and the node's egress MTU floor is derived
/// from it — health and MTU are two different questions riding one
/// channel, and only the first one is policy-filtered.
pub health_relevant: bool,
}
/// Channel sender for transport presence edges.
pub type PresenceTx = tokio::sync::mpsc::Sender<TransportPresence>;
/// Channel receiver for transport presence edges.
pub type PresenceRx = tokio::sync::mpsc::Receiver<TransportPresence>;
// ============================================================================
// Errors
// ============================================================================
@@ -128,6 +184,20 @@ pub enum TransportError {
#[error("transport failed to start: {0}")]
StartFailed(String),
/// The named network interface is not usable right now: it does not exist,
/// or it exists but is administratively down (no `IFF_UP`).
///
/// Distinct from [`TransportError::StartFailed`] because absence is a
/// *state*, not a fault. Interface-bound transports treat it as "not bound
/// yet" and keep a presence watcher running; a `StartFailed` carrying the
/// same text could not be told apart from a typo'd interface name or a
/// missing capability.
#[error("interface unavailable: {interface}")]
InterfaceUnavailable {
/// The configured interface name.
interface: String,
},
#[error("transport shutdown failed: {0}")]
ShutdownFailed(String),
@@ -874,6 +944,52 @@ impl TransportHandle {
}
}
/// Interface presence for interface-bound transports: the phase label, the
/// absence policy, how long the phase has been held, and the failed-bind
/// count since the last successful bind.
///
/// `None` for transports that are not bound to a named interface — for
/// those, presence is not a concept and an operator should not be shown an
/// always-`present` column.
pub fn interface_presence(&self) -> Option<InterfacePresence> {
match self {
#[cfg(any(target_os = "linux", target_os = "macos"))]
TransportHandle::Ethernet(t) => {
let state = t.presence_state();
Some(InterfacePresence {
presence: t.presence().as_str(),
carrier: t.has_carrier(),
policy: t.absence_policy().as_str(),
since_secs: state.since().as_secs(),
binds: state.binds(),
failed_attempts: state.attempts(),
})
}
_ => None,
}
}
/// Whether this transport can actually put a frame on the wire *now*.
///
/// [`Self::is_operational`] answers a different question: it means the
/// transport was started, which for an interface-bound transport no longer
/// implies a live socket — that is the whole point of presence. Callers
/// that are choosing a transport to use, or deriving a value from one,
/// want this; callers reasoning about lifecycle want `is_operational`.
///
/// `true` for every transport that is not interface-bound, so this is
/// `is_operational` with the presence refinement applied where it exists.
pub fn is_bound(&self) -> bool {
if !self.is_operational() {
return false;
}
match self {
#[cfg(any(target_os = "linux", target_os = "macos"))]
TransportHandle::Ethernet(t) => t.presence() == ethernet::Presence::Present,
_ => true,
}
}
/// Get the interface name (Ethernet only, returns None for other transports).
pub fn interface_name(&self) -> Option<&str> {
match self {
+81 -42
View File
@@ -80,6 +80,35 @@ impl PathMtuEntry {
/// address).
pub type PathMtuLookup = Arc<RwLock<HashMap<FipsAddress, PathMtuEntry>>>;
/// The node-global TCP MSS ceiling, shared live with the TUN reader and
/// writer threads.
///
/// Shared rather than copied because the value it is derived from moves at
/// runtime. `Node::transport_mtu()` is the minimum across *bound* transports,
/// and since dynamic interface binding a transport can bind minutes after
/// start or unbind mid-operation — so a narrow interface appearing must
/// tighten the clamp, and its departure must release it. Every other consumer
/// of `transport_mtu()` already reads it live (`show_status`, the snapshot,
/// the session-layer fragmentation check); these two threads captured a `u16`
/// at spawn and were the only place left where the daemon could report one
/// effective MTU and clamp to another.
///
/// A relaxed load per packet, beside the `PathMtuLookup` read that already
/// happens on the same packet — strictly the cheaper of the two. Ordering is
/// irrelevant: this is a clamp, and a packet that reads the previous value
/// during the store is clamped by the ceiling that was correct a microsecond
/// earlier. The per-flow ceiling has always had that property.
/// The IPv6 minimum link MTU (RFC 8200): every compliant path accepts a packet
/// this large, so an MSS derived from it fits anywhere.
///
/// Two callers, and they must not disagree: the cold-flow fallback in
/// [`per_flow_max_mss`], and the seed for the node's [`MssCeiling`] before any
/// transport has bound — which is the same value `Node::transport_mtu()` falls
/// back to when nothing is bound.
pub const IPV6_MIN_MTU: u16 = 1280;
pub type MssCeiling = Arc<std::sync::atomic::AtomicU16>;
/// Compute the effective TCP MSS ceiling for a packet given its peer
/// address bytes (a 16-byte IPv6 destination on outbound, source on
/// inbound). Returns `min(global_max_mss, learned_path_max_mss)` when
@@ -115,7 +144,6 @@ pub(crate) fn per_flow_max_mss(
// RFC 8200 IPv6-minimum MTU (1280) → effective FIPS-encapsulated
// payload (1203) → TCP segment after IPv6+TCP headers (1143).
// Used as the conservative ceiling for empty-lookup destinations.
const IPV6_MIN_MTU: u16 = 1280;
let conservative_max_mss = mss_ceiling(IPV6_MIN_MTU);
let empty_lookup_ceiling = std::cmp::min(global_max_mss, conservative_max_mss);
@@ -435,13 +463,14 @@ impl TunDevice {
/// a channel sender for submitting packets to be written.
///
/// `max_mss` is the global TCP MSS ceiling derived from the local
/// `transport_mtu()` floor. `path_mtu_lookup` is a read-only handle to
/// the per-destination path MTU map populated by discovery; the writer
/// reads it on each inbound SYN-ACK to compute a per-flow ceiling that
/// honors learned narrow paths through the mesh.
/// `transport_mtu()` floor, shared live so a transport that binds or
/// unbinds after start moves it (see [`MssCeiling`]). `path_mtu_lookup`
/// is a read-only handle to the per-destination path MTU map populated by
/// discovery; the writer reads both on each inbound SYN-ACK to compute a
/// per-flow ceiling that honors learned narrow paths through the mesh.
pub fn create_writer(
&self,
max_mss: u16,
max_mss: MssCeiling,
path_mtu_lookup: PathMtuLookup,
) -> Result<(TunWriter, TunTx), TunError> {
let fd = self.device.as_raw_fd();
@@ -517,7 +546,7 @@ pub struct TunWriter {
file: File,
rx: mpsc::Receiver<Vec<u8>>,
name: String,
max_mss: u16,
max_mss: MssCeiling,
path_mtu_lookup: PathMtuLookup,
}
@@ -531,16 +560,23 @@ impl TunWriter {
pub fn run(mut self) {
use super::tcp_mss::clamp_tcp_mss;
debug!(name = %self.name, max_mss = self.max_mss, "TUN writer starting");
debug!(
name = %self.name,
max_mss = self.max_mss.load(std::sync::atomic::Ordering::Relaxed),
"TUN writer starting"
);
for mut packet in self.rx {
// Read per packet, not once: a transport binding or unbinding
// moves the node's egress floor at runtime. See `MssCeiling`.
let global_max_mss = self.max_mss.load(std::sync::atomic::Ordering::Relaxed);
// Per-destination clamp: peer IPv6 source address (bytes 8..24)
// identifies the flow's remote end. If discovery has learned a
// smaller path MTU for that peer, tighten the ceiling.
let effective_max_mss = if packet.len() >= 24 {
per_flow_max_mss(&self.path_mtu_lookup, &packet[8..24], self.max_mss)
per_flow_max_mss(&self.path_mtu_lookup, &packet[8..24], global_max_mss)
} else {
self.max_mss
global_max_mss
};
// Clamp TCP MSS on inbound SYN-ACK packets
if clamp_tcp_mss(&mut packet, effective_max_mss) {
@@ -622,17 +658,17 @@ pub fn run_tun_reader(
our_addr: FipsAddress,
tun_tx: TunTx,
outbound_tx: TunOutboundTx,
transport_mtu: u16,
max_mss: MssCeiling,
path_mtu_lookup: PathMtuLookup,
) {
let (name, mut buf, max_mss) = tun_reader_setup(device.name(), mtu, transport_mtu);
let (name, mut buf) = tun_reader_setup(device.name(), mtu, &max_mss);
loop {
match device.read_packet(&mut buf) {
Ok(n) if n > 0 => {
if !handle_tun_packet(
&mut buf[..n],
max_mss,
max_mss.load(std::sync::atomic::Ordering::Relaxed),
&name,
our_addr,
&tun_tx,
@@ -685,13 +721,13 @@ pub fn run_tun_reader(
our_addr: FipsAddress,
tun_tx: TunTx,
outbound_tx: TunOutboundTx,
transport_mtu: u16,
max_mss: MssCeiling,
path_mtu_lookup: PathMtuLookup,
shutdown_fd: std::os::unix::io::RawFd,
) {
let _shutdown_fd = ShutdownFd(shutdown_fd);
let tun_fd = device.device().as_raw_fd();
let (name, mut buf, max_mss) = tun_reader_setup(device.name(), mtu, transport_mtu);
let (name, mut buf) = tun_reader_setup(device.name(), mtu, &max_mss);
// Set TUN fd to non-blocking so we can use select + read without blocking
// past the point where select returns readable.
@@ -741,7 +777,7 @@ pub fn run_tun_reader(
Ok(n) if n > 0 => {
if !handle_tun_packet(
&mut buf[..n],
max_mss,
max_mss.load(std::sync::atomic::Ordering::Relaxed),
&name,
our_addr,
&tun_tx,
@@ -774,30 +810,25 @@ pub fn run_tun_reader(
// _shutdown_fd closes on drop
}
/// Common setup for TUN reader: allocates buffer, computes max MSS.
fn tun_reader_setup(device_name: &str, mtu: u16, transport_mtu: u16) -> (String, Vec<u8>, u16) {
use super::icmp::effective_ipv6_mtu;
/// Common setup for TUN reader: allocates the buffer and names the device.
///
/// The MSS ceiling is deliberately *not* returned. It is read from the shared
/// [`MssCeiling`] on every packet, because a transport binding or unbinding
/// moves it after this function has run; returning it here is what let the
/// reader clamp to a floor derived from the transports that happened to be
/// bound at spawn.
fn tun_reader_setup(device_name: &str, mtu: u16, max_mss: &MssCeiling) -> (String, Vec<u8>) {
let name = device_name.to_string();
let buf = vec![0u8; mtu as usize + 100];
const IPV6_HEADER: u16 = 40;
const TCP_HEADER: u16 = 20;
let effective_mtu = effective_ipv6_mtu(transport_mtu);
let max_mss = effective_mtu
.saturating_sub(IPV6_HEADER)
.saturating_sub(TCP_HEADER);
debug!(
name = %name,
tun_mtu = mtu,
transport_mtu = transport_mtu,
effective_mtu = effective_mtu,
max_mss = max_mss,
max_mss = max_mss.load(std::sync::atomic::Ordering::Relaxed),
"TUN reader starting"
);
(name, buf, max_mss)
(name, buf)
}
/// Process a single TUN packet. Returns `false` if the reader should exit.
@@ -1076,12 +1107,13 @@ mod windows_tun {
/// packets independently. Returns the writer and a channel sender for
/// submitting packets to be written.
///
/// `max_mss` is the global TCP MSS ceiling. `path_mtu_lookup` is a
/// read-only handle to per-destination path MTU learned via
/// discovery.
/// `max_mss` is the global TCP MSS ceiling, shared live so a transport
/// that binds or unbinds after start moves it (see [`MssCeiling`]).
/// `path_mtu_lookup` is a read-only handle to per-destination path MTU
/// learned via discovery.
pub fn create_writer(
&self,
max_mss: u16,
max_mss: MssCeiling,
path_mtu_lookup: PathMtuLookup,
) -> Result<(TunWriter, TunTx), TunError> {
let (tx, rx) = mpsc::channel();
@@ -1119,7 +1151,7 @@ mod windows_tun {
session: Arc<wintun::Session>,
rx: mpsc::Receiver<Vec<u8>>,
name: String,
max_mss: u16,
max_mss: MssCeiling,
path_mtu_lookup: PathMtuLookup,
}
@@ -1132,14 +1164,21 @@ mod windows_tun {
use super::per_flow_max_mss;
use crate::upper::tcp_mss::clamp_tcp_mss;
debug!(name = %self.name, max_mss = self.max_mss, "TUN writer starting");
debug!(
name = %self.name,
max_mss = self.max_mss.load(std::sync::atomic::Ordering::Relaxed),
"TUN writer starting"
);
for mut packet in self.rx {
// Read per packet, not once: a transport binding or unbinding
// moves the node's egress floor. See `MssCeiling`.
let global_max_mss = self.max_mss.load(std::sync::atomic::Ordering::Relaxed);
// Per-destination clamp (peer source IPv6 = bytes 8..24)
let effective_max_mss = if packet.len() >= 24 {
per_flow_max_mss(&self.path_mtu_lookup, &packet[8..24], self.max_mss)
per_flow_max_mss(&self.path_mtu_lookup, &packet[8..24], global_max_mss)
} else {
self.max_mss
global_max_mss
};
// Clamp TCP MSS on inbound SYN-ACK packets
if clamp_tcp_mss(&mut packet, effective_max_mss) {
@@ -1188,17 +1227,17 @@ mod windows_tun {
our_addr: FipsAddress,
tun_tx: TunTx,
outbound_tx: TunOutboundTx,
transport_mtu: u16,
max_mss: MssCeiling,
path_mtu_lookup: PathMtuLookup,
) {
let (name, mut buf, max_mss) = super::tun_reader_setup(device.name(), mtu, transport_mtu);
let (name, mut buf) = super::tun_reader_setup(device.name(), mtu, &max_mss);
loop {
match device.read_packet(&mut buf) {
Ok(n) if n > 0 => {
if !super::handle_tun_packet(
&mut buf[..n],
max_mss,
max_mss.load(std::sync::atomic::Ordering::Relaxed),
&name,
our_addr,
&tun_tx,