laantungir/fips
1
0
Fork 0
mirror of https://github.com/jmcorgan/fips.git synced 2026-10-05 19:18:25 +00:00
Code Issues Packages Projects Releases Wiki Activity
Files
1f7c57beb1f780a47d31e3be09b3bfb8ef835031
fips/src/node/lifecycle/supervisor.rs
T
Johnathan Corgan a9423f801a Report crypto worker deaths as degraded health instead of hiding them 
The encrypt and decrypt worker pools discarded their thread handles, so a
worker that panicked left the node reporting full health while every packet
hashed to that worker was dropped behind a DEBUG line. A worker thread that
could not be started panicked start-up instead.

The pools now keep each worker's thread handle and report how many are
live. A worker that cannot be started is logged and left dead, and the
node starts degraded instead of aborting; a pool with no live worker is not
installed, so its traffic takes the main-loop path. A sweep on the rx loop
tick notices a worker that exits at runtime, logs a warning naming the pool
and the live and configured counts, and reports the pool to the supervisor,
which publishes Degraded. Losing workers never fails the node: the pools
are an offload, and Windows runs without them.

A dispatch refused by an exited worker is now counted and logged at WARN,
rate-limited, in place of the DEBUG line.
2026-10-02 18:14:01 +00:00

1877 lines
71 KiB
Rust
Raw Blame History

//! Node lifecycle supervisor — sans-IO core.
//!
//! A synchronous `step(event) -> Vec<Action>` finite-state machine over the
//! fixed set of substrate children. It owns the *decision* of what to bring up
//! and tear down and in what order; the async driver in [`super`]
//! (`start()`/`stop()`) performs the actual I/O each [`Action`] names and reports
//! results back as [`Event`]s. The core reads no clock, performs no I/O, and
//! holds no runtime handles — time enters only as inputs (a future `Tick`/
//! `DrainDeadlineElapsed`, added with the `Draining` phase) — so it is
//! unit-testable with synthetic sequences and survives a later thread-boundary
//! move (cores are sans-IO).
//!
//! ## Scope: the behavior-neutral rewrite
//!
//! This module is strictly
//! behavior-preserving. The machine mirrors today's `start()`/`stop()` exactly:
//!
//! - every configured child is spawned in the current order, and optional
//! failures are warn/debug-and-continue (there is no `Degraded` yet — a
//! failed child simply drains from `pending` and start still reaches
//! `Running`, as today an even-zero-transport node does);
//! - teardown runs in today's order and, faithfully, does **not** stop the
//! encrypt/decrypt worker pools (they are spawned in `start()` but never torn
//! down in `stop()`);
//! - the machine authors only the `SpawnChild`/`StopChild` *ordering*, and the
//! driver keeps its `self.state` writes at their current positions. The
//! behavior-neutral relocation left the published `NodeState` transitions
//! byte-for-byte unchanged; the bounded-drain phase below adds exactly one new
//! published transition (`Draining`), written directly like the others.
//!
//! ## Scope: the bounded `Draining` phase (this commit)
//!
//! This commit adds the operator-visible bounded-drain additions and nothing
//! else: the [`SupState::Draining`] state, the [`Event::Drain`] /
//! [`Event::DrainDeadlineElapsed`] events, the drain [`Action`]s
//! ([`Action::BroadcastDisconnect`], [`Action::SetTimer`],
//! [`Action::SetPeeringDesired`], [`Action::SuspendReplenish`]), and the new
//! published [`NodeState::Draining`](crate::node::NodeState::Draining) —
//! written directly by the driver at drain entry, exactly like the other
//! `self.state` transitions this module uses. The existing immediate `Stop`
//! path is untouched. `Draining` and `Stop` share a single teardown-plan author
//! (`begin_stopping`), so the teardown ordering is defined once.
//!
//! ## Scope: the `Running{Full|Degraded}` + `Failed` health split (this commit)
//!
//! This module implements the operator-visible start-completion health policy
//! and, with it, the FSM-owned [`Action::PublishState`]:
//!
//! - [`SupState::Running`] now carries a [`Health`] (`Full` or `Degraded`), and
//! [`SupState::Failed`] is the fatal path. When `Starting.pending` empties (or
//! the degenerate no-children path), the machine resolves health once
//! ([`SupervisorFsm::resolve_start_health`]): **zero transports up → `Failed`**
//! (fatal); **≥1 transport up but a configured optional child failed →
//! `Degraded`**; **everything configured came up → `Full`**. Not-configured
//! children never count (a node never asked to run DNS is not degraded for
//! lacking it); worker-pool failures are `Degraded` at most, never `Failed`.
//! - the health outcome is a fork that a single direct `self.state` write cannot
//! express, so the machine emits [`Action::PublishState`] carrying the resolved
//! [`NodeState`]; the driver writes it. The non-forking transitions
//! (`Starting`/`Draining`/`Stopping`/`Stopped`) keep their direct `self.state`
//! writes — only the start-completion health outcome routes through
//! `PublishState`, to minimize churn.
//! - the degenerate no-children path now resolves to `Failed` (zero transports),
//! **not** the old immediate-`Running`.
//!
//! Runtime child-liveness (a `ChildExited` event re-routing health when a
//! task/thread dies at runtime) came later. Its producers are the children that
//! report their own exit (TUN threads, the DNS task, the mDNS/Nostr monitor)
//! and, for the crypto worker pools, the rx loop tick's liveness sweep, which
//! reports a pool's child as exited when it loses a worker.
//!
//! ## Scope: interface presence, and `Degraded` as a level (this commit)
//!
//! Interface-bound transports are now *sometimes bound*: a transport whose
//! interface is missing at start comes up [`Absent`] and binds later, and one
//! whose interface goes away at runtime unbinds and rebinds when it returns.
//! Two things follow for this machine.
//!
//! - **A second reason set.** [`Event::ChildAbsent`] / [`Event::ChildPresent`]
//! move a child in and out of `absent`, which feeds `Degraded` exactly like
//! `failed` does. It is kept separate because it is *reversible* and `failed`
//! is not: a child that failed to start stays failed for the bring-up, while
//! an absent interface is expected to come back.
//! - **`Degraded` is a level, not a latch.** Nothing ever removed from `failed`,
//! which was correct while no child could recover — a monotonic set accurately
//! described a one-way door. Once recovery exists the assumption inverts: plug
//! the WAN back in and the node would stay `Degraded` until the process
//! restarted, and `Degraded` would come to mean "something broke at some point
//! since boot" rather than "something is broken now".
//! [`Self::classify_health`](SupervisorFsm::classify_health) is therefore
//! recomputed on every transition **in both directions**.
//!
//! An absent transport still counts as *up*. It came up — `start_async` returns
//! `Ok` with the transport absent — so it does not push a single-transport node
//! into the fatal [`FailReason::NoTransports`], which would make a node that
//! merely booted before its wifi exit instead of waiting. Absence degrades; it
//! never kills.
//!
//! There is deliberately **no restart action**. Rebinding is owned by the
//! transport's own binder task, which is where the file descriptor and the
//! presence watcher live; the FSM is told what happened and republishes health.
//!
//! [`Absent`]: crate::transport::ethernet::Presence::Absent
use std::collections::HashSet;
use std::sync::Arc;
use std::thread::JoinHandle;
use crate::node::NodeState;
use crate::transport::{PacketTx, TransportId};
use crate::upper::tun::{TunOutboundRx, TunTx};
/// A supervised substrate child.
///
/// Each transport is an individual child keyed by its id so the later
/// required-vs-optional health policy can reason about partial N-of-M bring-up.
/// The TUN device is a compound unit at the driver (a reader thread plus a
/// writer thread); the supervisor tracks it as the single `Tun` child, and the
/// driver joins both threads when it executes `StopChild(Tun)`.
#[derive(Clone, Copy, Debug, PartialEq, Eq, Hash)]
pub(crate) enum Child {
/// A transport instance (UDP / TCP / Ethernet), keyed by its runtime id.
Transport(TransportId),
/// The off-task FMP-encrypt + UDP-send worker pool (`#[cfg(unix)]`).
EncryptWorkers,
/// The off-task FMP-decrypt worker pool (`#[cfg(unix)]`).
DecryptWorkers,
/// Nostr overlay rendezvous/discovery.
Nostr,
/// LAN mDNS / DNS-SD rendezvous.
Mdns,
/// The TUN device (reader + writer threads).
Tun,
/// The `.fips` DNS responder task.
Dns,
}
/// An input to the supervisor. Results of executing [`Action`]s are fed back as
/// `SubstrateUp` / `SubstrateFailed` / `ChildStopped`.
///
/// `Tick` is **deferred** (the per-tick reconciler backstop lands
/// with the cadence work). `ChildExited` is present: it feeds runtime
/// child-liveness monitoring, routing health the same way a start failure does.
#[derive(Clone, Debug, PartialEq, Eq)]
pub(crate) enum Event {
/// Begin bring-up. `transports` are the ids the driver has already created
/// (their ids are allocated at creation), in creation order; the booleans
/// mark which singleton children are configured. Valid from `Created` or
/// `Stopped`.
Start {
/// Created transport ids, in the order they must be started.
transports: Vec<TransportId>,
/// The `#[cfg(unix)]` encrypt worker pool is configured.
encrypt_workers: bool,
/// The `#[cfg(unix)]` decrypt worker pool is configured.
decrypt_workers: bool,
/// Nostr overlay discovery is enabled.
nostr: bool,
/// LAN mDNS discovery is enabled.
mdns: bool,
/// The TUN device is enabled.
tun: bool,
/// The DNS responder is enabled.
dns: bool,
},
/// A child the driver was asked to spawn came up.
SubstrateUp {
/// The child that started successfully.
child: Child,
},
/// A child the driver was asked to spawn failed to start. In the
/// behavior-neutral rewrite this is warn/debug-and-continue: the child
/// drains from `pending` and never joins the up-set (matching today), and
/// start still proceeds to `Running`.
SubstrateFailed {
/// The child that failed to start.
child: Child,
},
/// Begin an immediate teardown (no drain). Valid from `Running`. This is
/// the path `node.stop()` uses; unchanged from the behavior-neutral rewrite.
Stop,
/// Begin a bounded graceful drain. Valid from `Running`. Emits the drain
/// entry actions (broadcast Disconnect, arm the deadline timer, gate the
/// reconciler off) and moves to `Draining`; the driver then runs the
/// bounded drain window before feeding `DrainDeadlineElapsed`.
Drain {
/// Absolute drain deadline in driver-clock milliseconds, carried into
/// `Draining` and the `SetTimer` action for observability. The driver
/// owns the actual bounded wait.
deadline_ms: u64,
},
/// The bounded drain window has closed — either the deadline elapsed or all
/// peers drained early. Valid from `Draining`; begins the (shared) teardown
/// plan, transitioning to `Stopping`.
DrainDeadlineElapsed,
/// A child the driver was asked to stop has finished stopping.
ChildStopped {
/// The child that has been torn down.
child: Child,
},
/// A supervised child's task or thread exited on its own at runtime — not
/// in response to a `StopChild`. Valid from `Running`; routes
/// health the same way a start failure does (the last transport out →
/// `Failed`, an optional child out → `Degraded`), but at runtime `Failed` is
/// a published health signal, not a teardown — the driver keeps serving. No
/// restart (the FSM has no restart action). Inert outside `Running`.
ChildExited {
/// The child whose task/thread exited.
child: Child,
},
/// An interface-bound child lost its interface — it was never there at
/// start, or it went away at runtime. The child stays *up* (the transport
/// object survives detach, only its socket goes) but contributes
/// `Degraded`. Valid while `Running`; while `Starting` the edge is recorded
/// so start-completion health already reflects it.
ChildAbsent {
/// The child whose interface is absent.
child: Child,
},
/// An interface-bound child's interface came back and it rebound. Clears
/// the absence and republishes health, which is how `Degraded` becomes
/// reversible.
ChildPresent {
/// The child whose interface is present again.
child: Child,
},
}
/// A driver-scheduled timer the supervisor can arm. Only the
/// drain deadline exists for now; the handshake/rekey/liveness timers
/// arrive with later cores.
#[derive(Clone, Copy, Debug, PartialEq, Eq)]
pub(crate) enum Timer {
/// Fires when the bounded drain window closes. The driver feeds
/// [`Event::DrainDeadlineElapsed`] when it elapses (or earlier, when all
/// peers have drained).
DrainDeadline,
}
/// The reconciler's desired peering set (drain gate). Only
/// `Empty` is needed for now; the populated variants that the
/// homeostatic reconciler converges toward land with that core.
#[derive(Clone, Debug, PartialEq, Eq)]
pub(crate) enum PeeringDesired {
/// No peers desired. Entering `Draining` sets this so the reconciler stops
/// reconnecting the peers the drain just closed ("Draining switches the
/// homeostat off").
Empty,
}
/// An effect the driver must perform. The core never performs I/O itself.
///
/// `PublishState` lands here, with the `Running{Full|Degraded}`
/// health split: the start-completion health outcome is a fork
/// (`Full`/`Degraded`/`Failed`) that a single direct `self.state` write cannot
/// express, so the machine authors it as an action. The driver keeps its direct
/// `self.state` writes for the non-forking transitions (`Starting`/`Draining`/
/// `Stopping`/`Stopped`); only the start-completion health outcome routes through
/// `PublishState`.
#[derive(Clone, Debug, PartialEq, Eq)]
pub(crate) enum Action {
/// Bring up this child (the driver performs the spawn / start I/O and
/// reports `SubstrateUp` or `SubstrateFailed`).
SpawnChild(Child),
/// Publish the given operator-visible [`NodeState`]. Emitted at start
/// completion (when `Starting.pending` empties, or the degenerate
/// no-children path) to carry the resolved health outcome —
/// [`NodeState::Running`] (Full), [`NodeState::Degraded`], or
/// [`NodeState::Failed`] — to the driver, which writes it to the published
/// state.
PublishState(NodeState),
/// Tear down this child (the driver performs the stop / join I/O and
/// reports `ChildStopped`).
StopChild(Child),
/// Broadcast a shutdown `Disconnect` to all sendable peers. Emitted once,
/// at drain entry; the drain teardown does not re-broadcast.
BroadcastDisconnect,
/// Arm a driver timer at the given absolute driver-clock milliseconds. In
/// this commit only `DrainDeadline` exists; the driver notes the deadline
/// and owns the bounded drain wait, so this is a documented no-op beyond
/// bookkeeping.
SetTimer(Timer, u64),
/// Set the reconciler's desired peering set (drain gate). Documented
/// **no-op for now** — the reconciler that consumes it is not yet
/// built; the driver logs/ignores it for now.
SetPeeringDesired(PeeringDesired),
/// Suspend peer replenishment (drain gate). Documented **no-op for
/// now** for the same reason as `SetPeeringDesired`.
SuspendReplenish,
}
/// Start-completion health. Resolved once when
/// `Starting.pending` empties: `Full` iff every configured child came up,
/// `Degraded` iff ≥1 transport is up but some configured optional child failed.
/// Zero transports up is not a health — it is the fatal [`SupState::Failed`].
#[derive(Clone, Debug, PartialEq, Eq)]
pub(crate) enum Health {
/// Every configured child came up.
Full,
/// ≥1 transport is up, but one or more configured optional children failed
/// to start (a transport beyond the first, Nostr, mDNS, TUN, DNS, or a
/// worker pool that did not start every worker). The node is operational
/// (serving) but degraded.
Degraded {
/// The configured children that failed to start.
reasons: HashSet<Child>,
},
}
/// Reason for the fatal [`SupState::Failed`] state.
#[derive(Clone, Copy, Debug, PartialEq, Eq)]
pub(crate) enum FailReason {
/// Zero transports came up at start completion. Without a transport the node
/// cannot serve, so this is fatal (the driver tears down and returns an
/// error), unlike the degraded-but-serving optional-child failures.
NoTransports,
}
/// Internal supervisor state. Richer than the published
/// [`NodeState`](crate::node::NodeState): `Starting`/`Stopping` carry the set of
/// children still resolving, and `Running` carries the resolved [`Health`].
#[derive(Clone, Debug, PartialEq, Eq)]
pub(crate) enum SupState {
/// Constructed but not started.
Created,
/// Bringing children up; `pending` is the set not yet resolved.
Starting {
/// Children asked to spawn that have not yet reported up-or-failed.
pending: HashSet<Child>,
},
/// All children resolved and ≥1 transport up; node operational. Carries the
/// resolved [`Health`] (`Full` or `Degraded`).
Running {
/// Resolved start-completion health.
health: Health,
},
/// Start completed with zero transports up — fatal. The
/// driver tears down any children that did come up and returns an error.
Failed {
/// Why the start failed.
reason: FailReason,
},
/// Bounded graceful-drain window. Broadcast Disconnect
/// has gone out and the reconciler is gated off (desired peering set
/// emptied, replenishment suspended); teardown begins when
/// `DrainDeadlineElapsed` arrives. Logically sits between `Running` and
/// `Stopping`.
Draining {
/// Absolute drain deadline in driver-clock milliseconds (carried for
/// observability; the driver owns the actual wait).
deadline_ms: u64,
},
/// Tearing children down; `pending` is the set not yet stopped.
Stopping {
/// Children asked to stop that have not yet reported stopped.
pending: HashSet<Child>,
},
/// Fully torn down.
Stopped,
}
/// The lifecycle supervisor FSM.
///
/// Construct with [`SupervisorFsm::new`], feed [`Event`]s via [`SupervisorFsm::step`],
/// and execute the returned [`Action`]s. See the module docs for the
/// behavior-neutral scope.
#[derive(Clone, Debug)]
pub(crate) struct SupervisorFsm {
state: SupState,
/// Children currently up (present). Drives the teardown plan.
up: HashSet<Child>,
/// Configured children that failed to start during the current bring-up.
/// Feeds the `Degraded` health determination when `pending` empties.
///
/// One-way within a bring-up: a start failure is not recoverable, so
/// nothing removes from this set until the next `Start`.
failed: HashSet<Child>,
/// Children that are up but whose network interface is currently absent.
///
/// Reversible, unlike [`Self::failed`] — that is the whole reason it is a
/// second set rather than more entries in the first. Feeds `Degraded` the
/// same way, and empties as interfaces come back.
absent: HashSet<Child>,
}
impl SupervisorFsm {
/// A fresh supervisor in `Created`.
pub(crate) fn new() -> Self {
Self {
state: SupState::Created,
up: HashSet::new(),
failed: HashSet::new(),
absent: HashSet::new(),
}
}
/// A supervisor seeded directly into `Running` with a known up-set.
///
/// The teardown driver (`stop()`) reconstructs the up-set from observed
/// runtime presence (`dns_task.is_some()`, transports keys, etc.) rather
/// than relying on a live machine persisted across start/stop, so that
/// teardown ordering is authored here regardless of how the node reached
/// `Running`. Feeding `Event::Stop` then yields the ordered `StopChild`
/// plan over exactly the present children.
pub(crate) fn running_with(up: impl IntoIterator<Item = Child>) -> Self {
Self {
state: SupState::Running {
health: Health::Full,
},
up: up.into_iter().collect(),
failed: HashSet::new(),
absent: HashSet::new(),
}
}
/// Current internal state (for the driver's bookkeeping and for tests).
#[cfg(test)]
pub(crate) fn state(&self) -> &SupState {
&self.state
}
/// The configured children that failed to start during bring-up. Kept for
/// the tests that pin the failure-vs-absence split; the driver reports
/// [`Self::degraded_children`], which is the union of the two.
#[cfg(test)]
pub(in crate::node) fn failed(&self) -> &HashSet<Child> {
&self.failed
}
/// Children whose interface is currently absent.
pub(in crate::node) fn absent(&self) -> &HashSet<Child> {
&self.absent
}
/// Every child currently contributing `Degraded` — the ones that failed to
/// start plus the ones whose interface is away. This is what an operator
/// wants named when the node reports `Degraded`.
pub(in crate::node) fn degraded_children(&self) -> HashSet<Child> {
self.failed.union(&self.absent).copied().collect()
}
/// Whether the machine is in the bounded-drain window. The driver uses this
/// after the rx loop returns to decide between the drain-teardown path and
/// the immediate-`stop()` fallback.
pub(crate) fn is_draining(&self) -> bool {
matches!(self.state, SupState::Draining { .. })
}
/// Advance the machine by one event, returning the effects to perform.
pub(crate) fn step(&mut self, event: Event) -> Vec<Action> {
match event {
Event::Start {
transports,
encrypt_workers,
decrypt_workers,
nostr,
mdns,
tun,
dns,
} => self.on_start(
transports,
encrypt_workers,
decrypt_workers,
nostr,
mdns,
tun,
dns,
),
Event::SubstrateUp { child } => self.on_substrate_up(child),
Event::SubstrateFailed { child } => self.on_substrate_failed(child),
Event::Stop => self.on_stop(),
Event::Drain { deadline_ms } => self.on_drain(deadline_ms),
Event::DrainDeadlineElapsed => self.on_drain_deadline_elapsed(),
Event::ChildStopped { child } => self.on_child_stopped(child),
Event::ChildExited { child } => self.on_child_exited(child),
Event::ChildAbsent { child } => self.on_child_absent(child),
Event::ChildPresent { child } => self.on_child_present(child),
}
}
#[allow(clippy::too_many_arguments)]
fn on_start(
&mut self,
transports: Vec<TransportId>,
encrypt_workers: bool,
decrypt_workers: bool,
nostr: bool,
mdns: bool,
tun: bool,
dns: bool,
) -> Vec<Action> {
// Only meaningful from a not-running state (the driver also guards on
// `can_start`). Ignore otherwise.
if !matches!(self.state, SupState::Created | SupState::Stopped) {
return Vec::new();
}
// Canonical spawn order, mirroring today's `start()`:
// transports (creation order) → encrypt → decrypt → nostr → mdns →
// tun → dns. (The driver performs the peer-connect between mdns and
// tun; it is not a supervised child.)
let mut order: Vec<Child> = transports.into_iter().map(Child::Transport).collect();
if encrypt_workers {
order.push(Child::EncryptWorkers);
}
if decrypt_workers {
order.push(Child::DecryptWorkers);
}
if nostr {
order.push(Child::Nostr);
}
if mdns {
order.push(Child::Mdns);
}
if tun {
order.push(Child::Tun);
}
if dns {
order.push(Child::Dns);
}
self.up.clear();
self.failed.clear();
self.absent.clear();
// A node with no children at all resolves health immediately. Zero
// transports up → `Failed` (this is the behavioral
// change from the old immediate-`Running`).
if order.is_empty() {
return vec![Action::PublishState(self.resolve_start_health())];
}
self.state = SupState::Starting {
pending: order.iter().copied().collect(),
};
order.into_iter().map(Action::SpawnChild).collect()
}
fn on_substrate_up(&mut self, child: Child) -> Vec<Action> {
let SupState::Starting { pending } = &mut self.state else {
return Vec::new();
};
pending.remove(&child);
let emptied = pending.is_empty();
self.up.insert(child);
if emptied {
vec![Action::PublishState(self.resolve_start_health())]
} else {
Vec::new()
}
}
fn on_substrate_failed(&mut self, child: Child) -> Vec<Action> {
// Record the failed child: a configured child that
// failed to start drives the `Degraded` determination when `pending`
// empties. It drains from `pending` and never joins the up-set.
let SupState::Starting { pending } = &mut self.state else {
return Vec::new();
};
pending.remove(&child);
let emptied = pending.is_empty();
self.failed.insert(child);
if emptied {
vec![Action::PublishState(self.resolve_start_health())]
} else {
Vec::new()
}
}
/// Resolve start-completion health. Called once when
/// `Starting.pending` empties (or the degenerate no-children path); the
/// classification is shared with runtime child-exit via
/// [`Self::classify_health`].
fn resolve_start_health(&mut self) -> NodeState {
self.classify_health()
}
/// Classify health from the current `up` / `failed` / `absent` sets and set
/// the resulting state, returning the [`NodeState`] the driver should
/// publish. Shared by start-completion ([`Self::resolve_start_health`]),
/// runtime child-exit ([`Self::on_child_exited`]) and the presence edges
/// ([`Self::on_child_absent`] / [`Self::on_child_present`]):
///
/// - zero transports up → [`SupState::Failed`] / [`NodeState::Failed`];
/// - ≥1 transport up but some child in `failed` or `absent` →
/// [`Health::Degraded`] / [`NodeState::Degraded`];
/// - everything up, nothing failed, nothing absent → [`Health::Full`] /
/// [`NodeState::Running`].
///
/// Worker-pool failures are captured in `failed` like any other optional
/// child, so they contribute `Degraded` (never `Failed`) — the inline crypto
/// fallback keeps the node correct without the pools.
///
/// A transport whose interface is absent is still counted among
/// `transports_up`: it came up, it is simply not bound. Excluding it would
/// make a single-ethernet node that booted before its wifi resolve to the
/// fatal [`FailReason::NoTransports`] and exit — which is the failure this
/// whole mechanism exists to remove. Absence degrades; it never kills.
///
/// Recomputed in **both** directions. This function is the reason
/// `Degraded` is a level rather than a latch.
fn classify_health(&mut self) -> NodeState {
let transports_up = self
.up
.iter()
.filter(|c| matches!(c, Child::Transport(_)))
.count();
if transports_up == 0 {
self.state = SupState::Failed {
reason: FailReason::NoTransports,
};
NodeState::Failed
} else if !self.failed.is_empty() || !self.absent.is_empty() {
self.state = SupState::Running {
health: Health::Degraded {
reasons: self.degraded_children(),
},
};
NodeState::Degraded
} else {
self.state = SupState::Running {
health: Health::Full,
};
NodeState::Running
}
}
/// An interface-bound child lost its interface.
///
/// The child stays in the up-set: the transport object survives detach —
/// config, id, statistics and neighbor buffer persist, only the descriptor
/// and its loops go. Only the reason set changes.
///
/// While `Starting` the edge is recorded silently; start-completion health
/// picks it up when `pending` empties, so a boot race resolves to
/// `Degraded` on the first publish rather than publishing `Full` and
/// immediately correcting it. Inert outside `Starting` / `Running`: a
/// teardown in flight owns its own bookkeeping.
fn on_child_absent(&mut self, child: Child) -> Vec<Action> {
match self.state {
SupState::Starting { .. } => {
self.absent.insert(child);
Vec::new()
}
SupState::Running { .. } => {
if !self.absent.insert(child) {
return Vec::new();
}
vec![Action::PublishState(self.classify_health())]
}
_ => Vec::new(),
}
}
/// An interface-bound child's interface came back.
///
/// Clears the absence and republishes. A child that is not currently
/// recorded absent produces nothing — a duplicate edge is not an event.
fn on_child_present(&mut self, child: Child) -> Vec<Action> {
match self.state {
SupState::Starting { .. } => {
self.absent.remove(&child);
Vec::new()
}
SupState::Running { .. } => {
if !self.absent.remove(&child) {
return Vec::new();
}
vec![Action::PublishState(self.classify_health())]
}
_ => Vec::new(),
}
}
fn on_stop(&mut self) -> Vec<Action> {
if !matches!(self.state, SupState::Running { .. }) {
return Vec::new();
}
self.begin_stopping()
}
fn on_drain(&mut self, deadline_ms: u64) -> Vec<Action> {
// Only a graceful drain from a running node (either health). Inert
// otherwise (matching `Stop`'s guard).
if !matches!(self.state, SupState::Running { .. }) {
return Vec::new();
}
self.state = SupState::Draining { deadline_ms };
// Drain entry, in order: broadcast the shutdown Disconnect, arm the
// deadline timer, then gate the reconciler off (desired = ∅, suspend
// replenishment) so it cannot reconnect the peers the drain just
// closed. The up-set is left intact for the eventual teardown plan.
vec![
Action::BroadcastDisconnect,
Action::SetTimer(Timer::DrainDeadline, deadline_ms),
Action::SetPeeringDesired(PeeringDesired::Empty),
Action::SuspendReplenish,
]
}
fn on_drain_deadline_elapsed(&mut self) -> Vec<Action> {
// The bounded drain window closed (deadline or all-peers-gone). Author
// the same teardown plan `Stop` produces.
if !matches!(self.state, SupState::Draining { .. }) {
return Vec::new();
}
self.begin_stopping()
}
/// Author the teardown plan over the current up-set, transition to
/// `Stopping`, and return the ordered `StopChild` actions. Shared by the
/// immediate `Stop` path ([`Self::on_stop`]) and the drain-window-close path
/// ([`Self::on_drain_deadline_elapsed`]) so the teardown ordering is defined
/// exactly once.
fn begin_stopping(&mut self) -> Vec<Action> {
let order = self.teardown_order();
self.state = SupState::Stopping {
pending: order.iter().copied().collect(),
};
order.into_iter().map(Action::StopChild).collect()
}
fn on_child_stopped(&mut self, child: Child) -> Vec<Action> {
if let SupState::Stopping { pending } = &mut self.state {
pending.remove(&child);
self.up.remove(&child);
self.absent.remove(&child);
if pending.is_empty() {
self.state = SupState::Stopped;
}
}
Vec::new()
}
/// A supervised child exited on its own at runtime. Only
/// meaningful while `Running`: startup (`Starting`), drain (`Draining`), and
/// teardown (`Stopping`) own their own child bookkeeping through the
/// `pending` / `up` sets and the `SubstrateUp` / `SubstrateFailed` /
/// `ChildStopped` events, so a stray exit signal in those states is ignored.
///
/// The exit is routed exactly like a start-time failure via
/// [`Self::classify_health`] — the last transport out → `Failed`, an optional
/// child out → `Degraded{+child}` — the runtime analogue of the
/// start-time policy. Two differences from start: `Failed` here is a
/// published health signal only (the driver keeps serving, per the resolved
/// runtime policy — no auto-teardown), and there is no restart (the FSM has
/// no restart action). If the exiting child is not currently up (a duplicate
/// signal, or one that never came up) the machine is unchanged and emits
/// nothing.
fn on_child_exited(&mut self, child: Child) -> Vec<Action> {
if !matches!(self.state, SupState::Running { .. }) {
return Vec::new();
}
if !self.up.remove(&child) {
return Vec::new();
}
// An exit supersedes an absence: the child is gone, not waiting.
self.absent.remove(&child);
self.failed.insert(child);
vec![Action::PublishState(self.classify_health())]
}
/// Teardown order over the up-set, mirroring today's `stop()`:
/// dns → nostr → mdns → transports (ascending id) → tun.
///
/// The encrypt/decrypt worker pools are deliberately excluded: today's
/// `stop()` spawns them in `start()` but never tears them down. Transports
/// are ordered by ascending id for determinism (today's `stop()` iterates
/// them in nondeterministic `HashMap` order, so this is neutral).
fn teardown_order(&self) -> Vec<Child> {
let mut order = Vec::new();
if self.up.contains(&Child::Dns) {
order.push(Child::Dns);
}
if self.up.contains(&Child::Nostr) {
order.push(Child::Nostr);
}
if self.up.contains(&Child::Mdns) {
order.push(Child::Mdns);
}
let mut transports: Vec<TransportId> = self
.up
.iter()
.filter_map(|c| match c {
Child::Transport(id) => Some(*id),
_ => None,
})
.collect();
transports.sort_by_key(|id| id.as_u32());
order.extend(transports.into_iter().map(Child::Transport));
if self.up.contains(&Child::Tun) {
order.push(Child::Tun);
}
order
}
}
/// Owner of the node's lifecycle-managed substrate handles plus the sans-IO
/// [`SupervisorFsm`] that authors their spawn/teardown ordering.
///
/// The fields moved here off `Node` are exactly the children the supervisor
/// governs — the packet-send channel, the TUN reader/writer plumbing, the DNS
/// responder task, the Nostr/LAN rendezvous drivers, and (on unix) the
/// encrypt/decrypt worker pools — together with the published `NodeState`.
/// This is a pure relocation: the driver (`start()`/`stop()`) reaches each
/// field through `self.supervisor.*`, and the initializers are the same ones
/// `Node::new` used.
pub(crate) struct Supervisor {
/// Node operational state (the published `NodeState`; the driver keeps its
/// verbatim writes here at their current positions).
pub(in crate::node) state: NodeState,
/// Packet sender for transports.
pub(in crate::node) packet_tx: Option<PacketTx>,
/// TUN packet sender channel.
pub(in crate::node) tun_tx: Option<TunTx>,
/// Receiver for outbound packets from the TUN reader.
pub(in crate::node) tun_outbound_rx: Option<TunOutboundRx>,
/// TUN reader thread handle.
pub(in crate::node) tun_reader_handle: Option<JoinHandle<()>>,
/// TUN writer thread handle.
pub(in crate::node) tun_writer_handle: Option<JoinHandle<()>>,
/// Shutdown pipe: writing to this fd unblocks the TUN reader thread on
/// macOS and FreeBSD. On Linux, deleting the interface via netlink
/// serves the same purpose.
#[cfg(any(target_os = "macos", target_os = "freebsd"))]
pub(in crate::node) tun_shutdown_fd: Option<std::os::unix::io::RawFd>,
/// Receiver for resolved identities from the DNS responder.
pub(in crate::node) dns_identity_rx: Option<crate::upper::dns::DnsIdentityRx>,
/// DNS responder task handle.
pub(in crate::node) dns_task: Option<tokio::task::JoinHandle<()>>,
/// Address the DNS responder actually bound, read back from the socket
/// after `bind` so a port-0 config resolves to the assigned port. `Some`
/// only while the responder is up; published to embedders through
/// [`Node::dns_local_addr`](crate::Node::dns_local_addr).
pub(in crate::node) dns_local_addr: Option<std::net::SocketAddr>,
/// Sends a new peer-alias base to the running DNS responder; `Some` only
/// while it runs.
pub(in crate::node) dns_aliases:
Option<tokio::sync::watch::Sender<crate::upper::hosts::HostMap>>,
/// Sender for each UDP listen socket the transport spawn binds — its raw
/// fd and the instance name it was configured under — armed by
/// [`Node::enable_app_owned_udp_fd`](crate::Node::enable_app_owned_udp_fd)
/// and fired from the transport-spawn arm of `start()` once the socket is
/// bound. `None` unless the embedder armed it.
#[cfg(unix)]
pub(in crate::node) udp_fd_tx: Option<std::sync::mpsc::Sender<crate::node::AppOwnedUdpSocket>>,
/// Node-side driver state for the Nostr overlay peer-rendezvous
/// subsystem: the engine handle, its startup timestamp, the one-shot
/// startup-sweep latch, and the per-peer bootstrap-transport bookkeeping
/// adopted from NAT-traversal handoffs.
pub(in crate::node) nostr_rendezvous: crate::nostr::RendezvousDriver,
/// mDNS / DNS-SD responder + browser for local-link peer discovery.
/// Identity is unverified at this layer — the Noise XX handshake
/// initiated against an mDNS-observed endpoint is what proves the
/// peer holds the matching private key.
pub(in crate::node) lan_rendezvous: Option<Arc<crate::mdns::LanRendezvous>>,
/// Off-task FMP-encrypt + UDP-send worker pool. Unix-only —
/// the worker issues direct sendmmsg(2) / sendmsg+UDP_GSO calls
/// on raw fds via `AsRawFd`. None on Windows or when no worker
/// thread could be started.
#[cfg(unix)]
pub(crate) encrypt_workers: Option<crate::node::encrypt_worker::EncryptWorkerPool>,
/// Off-task FMP decrypt worker pool — receiver-side mirror of
/// `encrypt_workers`. Workers are shards: each owns its session
/// state directly in a thread-local `HashMap` (no `RwLock`,
/// no `Mutex` per packet). Hash-by-cache-key dispatch.
/// None on Windows, with `FIPS_DECRYPT_WORKERS=0`, or when no worker
/// thread could be started.
#[cfg(unix)]
pub(crate) decrypt_workers: Option<crate::node::decrypt_worker::DecryptWorkerPool>,
/// Pools a test has built for start-up to install in place of spawning
/// its own.
#[cfg(all(test, unix))]
pub(in crate::node) staged_pools: StagedPools,
/// Transport-medium change detection: the receiver the rx loop drains and
/// the detector task behind it.
///
/// Deliberately **not** a [`Child`]. The FSM's children are the substrate
/// the node's health is defined against — a detector that dies makes
/// recovery slower, not the node unhealthy, and giving it a `Child` would
/// put it in the start plan, the teardown order and the N-of-M health
/// policy for no gain. It is spawned after bring-up and aborted in
/// teardown, like the runtime child-liveness monitor.
pub(in crate::node) netmon_rx: Option<crate::node::netmon::NetChangeRx>,
/// Handle for the detector task, aborted at teardown.
pub(in crate::node) netmon_task: Option<tokio::task::JoinHandle<()>>,
/// The sans-IO lifecycle FSM authoring spawn/teardown ordering.
pub(in crate::node) fsm: SupervisorFsm,
}
/// Crypto worker pools a test hands to start-up, each taken by the first
/// start of its pool's child.
#[cfg(all(test, unix))]
#[derive(Default)]
pub(in crate::node) struct StagedPools {
pub encrypt: Option<crate::node::encrypt_worker::EncryptWorkerPool>,
pub decrypt: Option<crate::node::decrypt_worker::DecryptWorkerPool>,
}
impl Supervisor {
/// A fresh supervisor with all handles empty and the FSM in `Created`,
/// matching the field initializers `Node::new` previously used.
pub(crate) fn new() -> Self {
Self {
state: NodeState::Created,
packet_tx: None,
tun_tx: None,
tun_outbound_rx: None,
tun_reader_handle: None,
tun_writer_handle: None,
#[cfg(any(target_os = "macos", target_os = "freebsd"))]
tun_shutdown_fd: None,
dns_identity_rx: None,
dns_task: None,
dns_local_addr: None,
dns_aliases: None,
#[cfg(unix)]
udp_fd_tx: None,
nostr_rendezvous: crate::nostr::RendezvousDriver::default(),
lan_rendezvous: None,
#[cfg(unix)]
encrypt_workers: None,
#[cfg(unix)]
decrypt_workers: None,
#[cfg(all(test, unix))]
staged_pools: StagedPools::default(),
netmon_rx: None,
netmon_task: None,
fsm: SupervisorFsm::new(),
}
}
}
#[cfg(test)]
mod tests {
use super::*;
fn tid(n: u32) -> TransportId {
TransportId::new(n)
}
fn start_full() -> Event {
Event::Start {
transports: vec![tid(1), tid(2)],
encrypt_workers: true,
decrypt_workers: true,
nostr: true,
mdns: true,
tun: true,
dns: true,
}
}
#[test]
fn start_emits_spawn_in_canonical_order() {
let mut s = SupervisorFsm::new();
let actions = s.step(start_full());
assert_eq!(
actions,
vec![
Action::SpawnChild(Child::Transport(tid(1))),
Action::SpawnChild(Child::Transport(tid(2))),
Action::SpawnChild(Child::EncryptWorkers),
Action::SpawnChild(Child::DecryptWorkers),
Action::SpawnChild(Child::Nostr),
Action::SpawnChild(Child::Mdns),
Action::SpawnChild(Child::Tun),
Action::SpawnChild(Child::Dns),
]
);
assert!(matches!(s.state(), SupState::Starting { .. }));
}
#[test]
fn all_configured_up_reaches_full() {
// Everything configured came up (2 transports + all optional children)
// → Full; the pending-emptying step publishes `Running`.
let mut s = SupervisorFsm::new();
let spawns = s.step(start_full());
let children: Vec<Child> = spawns
.into_iter()
.map(|a| match a {
Action::SpawnChild(c) => c,
_ => panic!("unexpected action"),
})
.collect();
let last = children.len() - 1;
for (i, child) in children.into_iter().enumerate() {
let out = s.step(Event::SubstrateUp { child });
if i == last {
assert_eq!(out, vec![Action::PublishState(NodeState::Running)]);
} else {
assert_eq!(out, vec![]);
}
}
assert_eq!(
s.state(),
&SupState::Running {
health: Health::Full
}
);
assert!(s.failed().is_empty());
}
#[test]
fn configured_optional_child_failure_is_degraded() {
// A configured optional child (mDNS) fails but ≥1 transport is up →
// Degraded, with the failed child in `reasons`. The node stays
// operational and tears down cleanly (the failed child never joined the
// up-set, so it is excluded from teardown; workers excluded by design).
let mut s = SupervisorFsm::new();
s.step(start_full());
for child in [
Child::Transport(tid(1)),
Child::Transport(tid(2)),
Child::EncryptWorkers,
Child::DecryptWorkers,
Child::Nostr,
] {
assert_eq!(s.step(Event::SubstrateUp { child }), vec![]);
}
// mdns fails, tun comes up, dns comes up last (empties pending).
assert_eq!(
s.step(Event::SubstrateFailed { child: Child::Mdns }),
vec![]
);
assert_eq!(s.step(Event::SubstrateUp { child: Child::Tun }), vec![]);
assert_eq!(
s.step(Event::SubstrateUp { child: Child::Dns }),
vec![Action::PublishState(NodeState::Degraded)]
);
let mut expected_reasons = HashSet::new();
expected_reasons.insert(Child::Mdns);
assert_eq!(
s.state(),
&SupState::Running {
health: Health::Degraded {
reasons: expected_reasons.clone()
}
}
);
assert_eq!(s.failed(), &expected_reasons);
let stops = s.step(Event::Stop);
// mdns must not appear in teardown; workers excluded by design.
assert_eq!(
stops,
vec![
Action::StopChild(Child::Dns),
Action::StopChild(Child::Nostr),
Action::StopChild(Child::Transport(tid(1))),
Action::StopChild(Child::Transport(tid(2))),
Action::StopChild(Child::Tun),
]
);
}
#[test]
fn worker_pool_failure_is_degraded_not_failed() {
// A worker-pool spawn failure is Degraded at most, never Failed
// (inline crypto fallback keeps the node correct). One transport is up.
let mut s = SupervisorFsm::new();
s.step(Event::Start {
transports: vec![tid(1)],
encrypt_workers: true,
decrypt_workers: false,
nostr: false,
mdns: false,
tun: false,
dns: false,
});
assert_eq!(
s.step(Event::SubstrateUp {
child: Child::Transport(tid(1))
}),
vec![]
);
assert_eq!(
s.step(Event::SubstrateFailed {
child: Child::EncryptWorkers
}),
vec![Action::PublishState(NodeState::Degraded)]
);
let mut expected = HashSet::new();
expected.insert(Child::EncryptWorkers);
assert_eq!(
s.state(),
&SupState::Running {
health: Health::Degraded { reasons: expected }
}
);
}
#[test]
fn not_configured_child_does_not_cause_degraded() {
// A node that never asked to run mDNS/TUN/DNS/Nostr is not degraded for
// lacking them: only a configured-and-failed child counts. One transport
// configured and up, nothing else configured → Full.
let mut s = SupervisorFsm::new();
s.step(Event::Start {
transports: vec![tid(1)],
encrypt_workers: false,
decrypt_workers: false,
nostr: false,
mdns: false,
tun: false,
dns: false,
});
assert_eq!(
s.step(Event::SubstrateUp {
child: Child::Transport(tid(1))
}),
vec![Action::PublishState(NodeState::Running)]
);
assert_eq!(
s.state(),
&SupState::Running {
health: Health::Full
}
);
}
#[test]
fn zero_transports_up_is_failed_via_child_failures() {
// Transports were configured but all failed → zero transports up →
// Failed (fatal), even though other children came up. Failed takes
// priority over Degraded.
let mut s = SupervisorFsm::new();
s.step(Event::Start {
transports: vec![tid(1)],
encrypt_workers: false,
decrypt_workers: false,
nostr: true,
mdns: false,
tun: false,
dns: false,
});
assert_eq!(
s.step(Event::SubstrateFailed {
child: Child::Transport(tid(1))
}),
vec![]
);
assert_eq!(
s.step(Event::SubstrateUp {
child: Child::Nostr
}),
vec![Action::PublishState(NodeState::Failed)]
);
assert_eq!(
s.state(),
&SupState::Failed {
reason: FailReason::NoTransports
}
);
}
#[test]
fn no_children_is_failed_immediately() {
// The degenerate empty-`Start` path: zero transports → Failed (the
// behavioral change from the old immediate-Running).
let mut s = SupervisorFsm::new();
let actions = s.step(Event::Start {
transports: vec![],
encrypt_workers: false,
decrypt_workers: false,
nostr: false,
mdns: false,
tun: false,
dns: false,
});
assert_eq!(actions, vec![Action::PublishState(NodeState::Failed)]);
assert_eq!(
s.state(),
&SupState::Failed {
reason: FailReason::NoTransports
}
);
}
#[test]
fn stop_teardown_order_excludes_workers() {
let mut s = SupervisorFsm::new();
s.step(start_full());
for child in [
Child::Transport(tid(2)),
Child::Transport(tid(1)),
Child::EncryptWorkers,
Child::DecryptWorkers,
Child::Nostr,
Child::Mdns,
Child::Tun,
Child::Dns,
] {
s.step(Event::SubstrateUp { child });
}
let stops = s.step(Event::Stop);
assert_eq!(
stops,
vec![
Action::StopChild(Child::Dns),
Action::StopChild(Child::Nostr),
Action::StopChild(Child::Mdns),
// transports ascending by id regardless of up-report order
Action::StopChild(Child::Transport(tid(1))),
Action::StopChild(Child::Transport(tid(2))),
Action::StopChild(Child::Tun),
]
);
assert!(matches!(s.state(), SupState::Stopping { .. }));
}
#[test]
fn all_children_stopped_reaches_stopped() {
let mut s = SupervisorFsm::new();
s.step(start_full());
// Every spawned child reports an outcome: five come up, three fail.
// `pending` drains fully; one transport (tid(1)) is up so the node
// reaches `Running`, but a configured transport (tid(2)) and two
// configured optional children failed → Degraded.
for child in [
Child::Transport(tid(1)),
Child::EncryptWorkers,
Child::Nostr,
Child::Tun,
Child::Dns,
] {
s.step(Event::SubstrateUp { child });
}
for child in [Child::Transport(tid(2)), Child::DecryptWorkers, Child::Mdns] {
s.step(Event::SubstrateFailed { child });
}
assert!(matches!(
s.state(),
SupState::Running {
health: Health::Degraded { .. }
}
));
// Only the children that came up are torn down; the failed ones never
// joined the up-set.
let stops = s.step(Event::Stop);
assert_eq!(
stops,
vec![
Action::StopChild(Child::Dns),
Action::StopChild(Child::Nostr),
Action::StopChild(Child::Transport(tid(1))),
Action::StopChild(Child::Tun),
]
);
for a in stops {
let child = match a {
Action::StopChild(c) => c,
_ => panic!("unexpected action"),
};
assert_eq!(s.step(Event::ChildStopped { child }), vec![]);
}
assert_eq!(s.state(), &SupState::Stopped);
}
#[test]
fn late_substrate_up_in_running_is_inert() {
let mut s = SupervisorFsm::new();
s.step(Event::Start {
transports: vec![tid(1)],
encrypt_workers: false,
decrypt_workers: false,
nostr: false,
mdns: false,
tun: false,
dns: false,
});
s.step(Event::SubstrateUp {
child: Child::Transport(tid(1)),
});
assert_eq!(
s.state(),
&SupState::Running {
health: Health::Full
}
);
// A stray event in Running produces nothing and does not change state.
assert_eq!(
s.step(Event::SubstrateUp {
child: Child::Nostr
}),
vec![]
);
assert_eq!(
s.state(),
&SupState::Running {
health: Health::Full
}
);
}
#[test]
fn drain_from_running_emits_entry_actions_and_enters_draining() {
let mut s = SupervisorFsm::running_with([
Child::Dns,
Child::Nostr,
Child::Transport(tid(1)),
Child::Tun,
]);
let actions = s.step(Event::Drain { deadline_ms: 5_000 });
// Order matters: broadcast → arm timer → gate reconciler off.
assert_eq!(
actions,
vec![
Action::BroadcastDisconnect,
Action::SetTimer(Timer::DrainDeadline, 5_000),
Action::SetPeeringDesired(PeeringDesired::Empty),
Action::SuspendReplenish,
]
);
assert_eq!(s.state(), &SupState::Draining { deadline_ms: 5_000 });
}
#[test]
fn drain_deadline_elapsed_yields_teardown_and_enters_stopping() {
let mut s = SupervisorFsm::running_with([
Child::Dns,
Child::Nostr,
Child::Mdns,
Child::Transport(tid(2)),
Child::Transport(tid(1)),
Child::Tun,
]);
s.step(Event::Drain { deadline_ms: 2_000 });
let stops = s.step(Event::DrainDeadlineElapsed);
// Same ordering the immediate `Stop` path authors: dns → nostr → mdns →
// transports (ascending id) → tun.
assert_eq!(
stops,
vec![
Action::StopChild(Child::Dns),
Action::StopChild(Child::Nostr),
Action::StopChild(Child::Mdns),
Action::StopChild(Child::Transport(tid(1))),
Action::StopChild(Child::Transport(tid(2))),
Action::StopChild(Child::Tun),
]
);
assert!(matches!(s.state(), SupState::Stopping { .. }));
}
#[test]
fn drain_teardown_matches_immediate_stop_teardown() {
// The drain path and the immediate-stop path must produce the identical
// StopChild plan over the same up-set (single teardown author).
let up = [
Child::Dns,
Child::Nostr,
Child::Mdns,
Child::Transport(tid(1)),
Child::Transport(tid(3)),
Child::Tun,
];
let mut immediate = SupervisorFsm::running_with(up);
let stop_plan = immediate.step(Event::Stop);
let mut drained = SupervisorFsm::running_with(up);
drained.step(Event::Drain { deadline_ms: 1_000 });
let drain_plan = drained.step(Event::DrainDeadlineElapsed);
assert_eq!(stop_plan, drain_plan);
}
#[test]
fn drain_is_inert_from_non_running() {
// From `Created`.
let mut s = SupervisorFsm::new();
assert_eq!(s.step(Event::Drain { deadline_ms: 1_000 }), vec![]);
assert_eq!(s.state(), &SupState::Created);
// From `Stopping` (seed a drain, close its window, then try to drain
// again — inert).
let mut s2 = SupervisorFsm::running_with([Child::Transport(tid(1))]);
s2.step(Event::Drain { deadline_ms: 1_000 });
s2.step(Event::DrainDeadlineElapsed);
assert!(matches!(s2.state(), SupState::Stopping { .. }));
assert_eq!(s2.step(Event::Drain { deadline_ms: 1_000 }), vec![]);
assert!(matches!(s2.state(), SupState::Stopping { .. }));
}
#[test]
fn drain_deadline_elapsed_is_inert_outside_draining() {
// Inert from `Running` (no drain in progress).
let mut s = SupervisorFsm::running_with([Child::Transport(tid(1))]);
assert_eq!(s.step(Event::DrainDeadlineElapsed), vec![]);
assert_eq!(
s.state(),
&SupState::Running {
health: Health::Full
}
);
}
#[test]
fn child_exited_optional_is_degraded() {
// An optional child (Nostr) exits at runtime with a transport still up →
// Degraded, the exited child recorded in `reasons` and dropped from `up`.
let mut s = SupervisorFsm::running_with([
Child::Transport(tid(1)),
Child::Nostr,
Child::Tun,
Child::Dns,
]);
assert_eq!(
s.step(Event::ChildExited {
child: Child::Nostr
}),
vec![Action::PublishState(NodeState::Degraded)]
);
let mut reasons = HashSet::new();
reasons.insert(Child::Nostr);
assert_eq!(
s.state(),
&SupState::Running {
health: Health::Degraded {
reasons: reasons.clone()
}
}
);
assert_eq!(s.failed(), &reasons);
}
#[test]
fn child_exited_transport_beyond_first_is_degraded() {
// One of several transports exits → Degraded (≥1 transport remains up),
// not Failed.
let mut s =
SupervisorFsm::running_with([Child::Transport(tid(1)), Child::Transport(tid(2))]);
assert_eq!(
s.step(Event::ChildExited {
child: Child::Transport(tid(1))
}),
vec![Action::PublishState(NodeState::Degraded)]
);
assert!(matches!(
s.state(),
SupState::Running {
health: Health::Degraded { .. }
}
));
}
#[test]
fn child_exited_last_transport_is_failed() {
// The only transport exits → Failed (published), but no teardown action:
// per the runtime policy the driver keeps serving on the Failed health
// signal; the FSM emits only `PublishState(Failed)`.
let mut s = SupervisorFsm::running_with([Child::Transport(tid(1)), Child::Dns]);
assert_eq!(
s.step(Event::ChildExited {
child: Child::Transport(tid(1))
}),
vec![Action::PublishState(NodeState::Failed)]
);
assert_eq!(
s.state(),
&SupState::Failed {
reason: FailReason::NoTransports
}
);
}
#[test]
fn child_exited_accumulates_reasons_then_fails_on_last_transport() {
// Runtime exits accumulate in `reasons` and stay Degraded while a
// transport survives; the last transport out flips to Failed.
let mut s = SupervisorFsm::running_with([
Child::Transport(tid(1)),
Child::Transport(tid(2)),
Child::Nostr,
]);
assert_eq!(
s.step(Event::ChildExited {
child: Child::Nostr
}),
vec![Action::PublishState(NodeState::Degraded)]
);
assert_eq!(
s.step(Event::ChildExited {
child: Child::Transport(tid(1))
}),
vec![Action::PublishState(NodeState::Degraded)]
);
// Still Degraded (tid(2) up); reasons carry both prior exits.
let mut reasons = HashSet::new();
reasons.insert(Child::Nostr);
reasons.insert(Child::Transport(tid(1)));
assert_eq!(s.failed(), &reasons);
// Last transport out → Failed.
assert_eq!(
s.step(Event::ChildExited {
child: Child::Transport(tid(2))
}),
vec![Action::PublishState(NodeState::Failed)]
);
assert!(matches!(s.state(), SupState::Failed { .. }));
}
#[test]
fn child_exited_is_inert_outside_running() {
// From `Created`: no producer should fire pre-Running, but a stray signal
// is ignored.
let mut created = SupervisorFsm::new();
assert_eq!(
created.step(Event::ChildExited {
child: Child::Nostr
}),
vec![]
);
assert_eq!(created.state(), &SupState::Created);
// From `Draining`: a child exiting during the drain window is the drain's
// own teardown, owned by the `ChildStopped` path — ignore it here.
let mut draining = SupervisorFsm::running_with([Child::Transport(tid(1)), Child::Dns]);
draining.step(Event::Drain { deadline_ms: 1_000 });
assert_eq!(
draining.step(Event::ChildExited { child: Child::Dns }),
vec![]
);
assert!(matches!(draining.state(), SupState::Draining { .. }));
}
#[test]
fn child_exited_unknown_child_is_noop() {
// A child not in the up-set (duplicate signal, or never up) leaves the
// machine unchanged and emits nothing.
let mut s = SupervisorFsm::running_with([Child::Transport(tid(1))]);
assert_eq!(
s.step(Event::ChildExited {
child: Child::Nostr
}),
vec![]
);
assert_eq!(
s.state(),
&SupState::Running {
health: Health::Full
}
);
assert!(s.failed().is_empty());
}
// ── Interface presence ────────────────────────────────────────────────
//
// The absence set is separate from `failed` because it is reversible, and
// reversibility is what turns `Degraded` from a latch into a level.
/// Helper: bring a full node up cleanly and leave it `Running{Full}`.
fn running_node() -> SupervisorFsm {
let mut s = SupervisorFsm::new();
s.step(start_full());
for child in [
Child::Transport(tid(1)),
Child::Transport(tid(2)),
Child::EncryptWorkers,
Child::DecryptWorkers,
Child::Nostr,
Child::Mdns,
Child::Tun,
Child::Dns,
] {
s.step(Event::SubstrateUp { child });
}
assert_eq!(
s.state(),
&SupState::Running {
health: Health::Full
}
);
s
}
#[test]
fn an_absent_interface_degrades_a_running_node() {
let mut s = running_node();
assert_eq!(
s.step(Event::ChildAbsent {
child: Child::Transport(tid(1))
}),
vec![Action::PublishState(NodeState::Degraded)]
);
assert!(s.absent().contains(&Child::Transport(tid(1))));
// Absence is not failure: the two sets stay distinct.
assert!(s.failed().is_empty());
}
#[test]
fn a_returning_interface_clears_degraded() {
// The regression this whole design turns on: plug the WAN back in and
// the node must leave `Degraded`, not carry it until the process
// restarts. `Degraded` is a level, not a latch.
let mut s = running_node();
s.step(Event::ChildAbsent {
child: Child::Transport(tid(1)),
});
assert_eq!(
s.step(Event::ChildPresent {
child: Child::Transport(tid(1))
}),
vec![Action::PublishState(NodeState::Running)]
);
assert_eq!(
s.state(),
&SupState::Running {
health: Health::Full
}
);
assert!(s.absent().is_empty());
}
#[test]
fn health_reflects_the_last_interface_still_away() {
// Two interfaces away, one returns: still Degraded. Only the empty
// absence set publishes Full.
let mut s = running_node();
s.step(Event::ChildAbsent {
child: Child::Transport(tid(1)),
});
s.step(Event::ChildAbsent {
child: Child::Transport(tid(2)),
});
assert_eq!(
s.step(Event::ChildPresent {
child: Child::Transport(tid(1))
}),
vec![Action::PublishState(NodeState::Degraded)]
);
assert_eq!(
s.step(Event::ChildPresent {
child: Child::Transport(tid(2))
}),
vec![Action::PublishState(NodeState::Running)]
);
}
#[test]
fn a_duplicate_presence_edge_is_not_an_event() {
let mut s = running_node();
s.step(Event::ChildAbsent {
child: Child::Transport(tid(1)),
});
assert_eq!(
s.step(Event::ChildAbsent {
child: Child::Transport(tid(1))
}),
vec![],
"re-reporting the same absence must not republish"
);
s.step(Event::ChildPresent {
child: Child::Transport(tid(1)),
});
assert_eq!(
s.step(Event::ChildPresent {
child: Child::Transport(tid(1))
}),
vec![],
"re-reporting the same return must not republish"
);
}
#[test]
fn absence_during_bringup_resolves_degraded_on_the_first_publish() {
// The boot race. The transport reports absence from inside its own
// start, then reports up. Start-completion health must already know,
// so the node never publishes `Full` and corrects itself.
let mut s = SupervisorFsm::new();
s.step(start_full());
s.step(Event::ChildAbsent {
child: Child::Transport(tid(1)),
});
for child in [
Child::Transport(tid(1)),
Child::Transport(tid(2)),
Child::EncryptWorkers,
Child::DecryptWorkers,
Child::Nostr,
Child::Mdns,
Child::Tun,
] {
assert_eq!(s.step(Event::SubstrateUp { child }), vec![]);
}
assert_eq!(
s.step(Event::SubstrateUp { child: Child::Dns }),
vec![Action::PublishState(NodeState::Degraded)]
);
let mut reasons = HashSet::new();
reasons.insert(Child::Transport(tid(1)));
assert_eq!(
s.state(),
&SupState::Running {
health: Health::Degraded { reasons }
}
);
}
#[test]
fn a_lone_absent_transport_degrades_rather_than_fails() {
// A single-ethernet node that boots before its wifi. The transport
// came up — absence is a state, not a start failure — so it counts
// among the transports up and the node serves. Resolving to `Failed`
// here would make the daemon exit on the very race this mechanism
// exists to survive.
let mut s = SupervisorFsm::new();
s.step(Event::Start {
transports: vec![tid(1)],
encrypt_workers: false,
decrypt_workers: false,
nostr: false,
mdns: false,
tun: false,
dns: false,
});
s.step(Event::ChildAbsent {
child: Child::Transport(tid(1)),
});
assert_eq!(
s.step(Event::SubstrateUp {
child: Child::Transport(tid(1))
}),
vec![Action::PublishState(NodeState::Degraded)]
);
assert!(
!matches!(s.state(), SupState::Failed { .. }),
"an absent interface must never be the fatal no-transports case"
);
}
#[test]
fn an_exit_supersedes_an_absence() {
// A transport that is away and then exits is gone, not waiting: it
// leaves the up-set and moves from `absent` to `failed`, so a later
// spurious `ChildPresent` cannot resurrect it into `Full`.
let mut s = running_node();
s.step(Event::ChildAbsent {
child: Child::Transport(tid(1)),
});
s.step(Event::ChildExited {
child: Child::Transport(tid(1)),
});
assert!(s.absent().is_empty());
assert!(s.failed().contains(&Child::Transport(tid(1))));
assert_eq!(
s.step(Event::ChildPresent {
child: Child::Transport(tid(1))
}),
vec![]
);
assert!(matches!(
s.state(),
SupState::Running {
health: Health::Degraded { .. }
}
));
}
#[test]
fn presence_edges_are_inert_outside_starting_and_running() {
// Teardown and drain own their own bookkeeping; a late edge from a
// binder that has not noticed the stop must not author actions.
let mut created = SupervisorFsm::new();
assert_eq!(
created.step(Event::ChildAbsent {
child: Child::Transport(tid(1))
}),
vec![]
);
let mut draining = running_node();
draining.step(Event::Drain { deadline_ms: 1 });
assert_eq!(
draining.step(Event::ChildAbsent {
child: Child::Transport(tid(1))
}),
vec![]
);
assert!(
draining.is_draining(),
"a presence edge must not end a drain"
);
}
#[test]
fn a_new_start_clears_the_previous_absences() {
let mut s = running_node();
s.step(Event::ChildAbsent {
child: Child::Transport(tid(1)),
});
s.step(Event::Stop);
for child in [
Child::Dns,
Child::Nostr,
Child::Mdns,
Child::Transport(tid(1)),
Child::Transport(tid(2)),
Child::Tun,
] {
s.step(Event::ChildStopped { child });
}
s.step(start_full());
assert!(s.absent().is_empty());
}
/// The crypto worker pools are a performance offload with a main-loop path
/// behind them, so losing one degrades the node and can never be what
/// fails it.
#[test]
fn a_worker_pool_exit_is_degraded_and_never_failed() {
let mut s = SupervisorFsm::running_with([
Child::Transport(tid(1)),
Child::EncryptWorkers,
Child::DecryptWorkers,
]);
assert_eq!(
s.step(Event::ChildExited {
child: Child::EncryptWorkers
}),
vec![Action::PublishState(NodeState::Degraded)]
);
assert_eq!(
s.step(Event::ChildExited {
child: Child::DecryptWorkers
}),
vec![Action::PublishState(NodeState::Degraded)]
);
assert!(matches!(
s.state(),
SupState::Running {
health: Health::Degraded { .. }
}
));
let degraded = s.degraded_children();
assert!(degraded.contains(&Child::EncryptWorkers));
assert!(degraded.contains(&Child::DecryptWorkers));
}
#[test]
fn degraded_children_is_the_union_of_both_reason_sets() {
let mut s = SupervisorFsm::new();
s.step(start_full());
s.step(Event::ChildAbsent {
child: Child::Transport(tid(1)),
});
s.step(Event::SubstrateFailed { child: Child::Mdns });
for child in [
Child::Transport(tid(1)),
Child::Transport(tid(2)),
Child::EncryptWorkers,
Child::DecryptWorkers,
Child::Nostr,
Child::Tun,
Child::Dns,
] {
s.step(Event::SubstrateUp { child });
}
let degraded = s.degraded_children();
assert!(degraded.contains(&Child::Mdns));
assert!(degraded.contains(&Child::Transport(tid(1))));
assert_eq!(degraded.len(), 2);
}
}
Reference in New Issue View Git Blame Copy Permalink