mirror of
https://github.com/jmcorgan/fips.git
synced 2026-10-05 19:18:25 +00:00
Add a fipsctl probe diagnostic for reachability and tree position
fipsctl probe <npub|hostname> answers, for one target, where it sits in the spanning tree relative to us and whether we can actually reach it. It reports our coordinates, the target's, the walk between them and the next hop we would select, then opens an FSP session, waits for one MMP receiver report to yield a round-trip time, and tears down what it opened. Nothing here changes the wire format. The probe is built entirely from messages that already exist, and the control socket carries the new request triplet. The work runs as five stages that report separately: bloom, discovery, path, session and rtt. One verdict covering several findings is what makes an operator read source, and the distinctions are real ones. "No peer's filter claims this address" says the mesh has never heard of the target; "a filter claimed it and nothing answered" says the opposite, that somebody believes the address is reachable and the lookup went unanswered anyway. Bloom emits the lookup and settles on the gate's answer, where a miss, a backoff suppression or a zero fanout ends the probe; discovery waits for the coordinates and owns the ladder timeout. "Lookup never resolved" and "resolved but the handshake never completed" part the same way further down. Each stage keeps the reasons it owns, so no discriminator is lost and none sits on a stage that cannot produce it. The path is computed from coordinates, not observed. The output says so in those words: nothing traverses the mesh to confirm the hops, and a route display that reads like traceroute output would be believed as one. A real per-hop trace needs a new wire message, so it is not on this branch. The probe is a daemon-side job advanced on the tick, not a blocking control call. The control socket has a five second timeout and its dispatch is awaited inline in the rx loop, so a handler that waits for a handshake would stall the data plane. Start, poll and cancel each return immediately and fipsctl hides the polling. The job is stepped once at the end of admission rather than left for the next tick, which admission can do because the probe commands take the command path and therefore already run on the rx loop; otherwise every probe spent up to a full tick period of its own budget before a single message left the node, which against a one-second tick meant the first three polls of an already-cached target showed nothing happening. It cleans up after itself, and that is the part built to be defended rather than assumed. A session that existed before the probe started is never torn down, ownership is decided at the moment of action rather than once at the beginning, re-checked before teardown, and dropped if our entry is replaced or adopted by traffic underneath us. Removing the ownership guard reds eleven tests. The client renders each poll rather than waiting for the end. The daemon was already progressive, returning the whole report on every poll with each stage carrying its own verdict as it reaches one, so a client that waited for `state == "done"` made a probe spending seventeen seconds in a lookup ladder look identical to one that was hung. On a terminal the stage block is redrawn in place with a spinner and a running elapsed on whichever stage is working. Piped or redirected there is no cursor to move, so each row prints once, at the moment it settles, and the transcript ends up the same block a terminal leaves behind. `--json` is untouched and still emits exactly one document at the end, so a script parsing the report does not have to skip past progress output. Four things the rendering has to get right, none of them automatic: - A running stage may only report what the daemon has observed, and must never preview an outcome. Every settled text keys on `reason`, which is null while a stage runs, so the success arm renders for a stage that has not succeeded and a running session row would claim the handshake completed. - The elapsed column comes from the daemon's clock throughout, the running stage's figure being the report's elapsed less the stages already accounted for, so the numbers a viewer watches are the ones the final report prints. - A frame shorter than the last one blanks the rows it no longer covers and walks the cursor back over them, or the previous frame's tail stays on screen under a report that has stopped mentioning it. - The discovery ladder is read from the report rather than assumed, since it is configuration and a node may not be using the default. One line per request sent, with the timeout that attempt was given and whether it drew a reply, the last animating while it is in flight. Below the block, the tree walk is one line: self, up through the least common ancestor, down to the target, with the ancestor emphasised on a terminal and left plain in a pipe or a file. Naming the ancestor alone left the reader to assemble the route from it and the two coordinate lines above. Where the target is itself the ancestor there is no descent and the line ends on the emphasised address. Stages that were never attempted print no row. A failure marks everything behind it not reached, and saying that three more times adds nothing to the failed row that already said it. The rule keys on `not_reached` rather than on the position of the failure, because those are not the same set: a failed path stage does not stop the probe, since the preview touches nothing and the session can still succeed where it named no next hop, so the rows behind that one describe work that really happened. A skip keeps its row for the same reason, being a result naming why a stage was unnecessary rather than an absence. A probe that fails before the path stage prints no path section, which had been restating the failure as "no coords" and "no next hop". Two counts the discovery stage gets right that are easy to get wrong. It marks itself running while it waits, where publishing `pending` throughout would read to a poller as a stage that has not started. And the first attempt is counted when the request is sent rather than when the pending table is next observed, since a lookup answered inside one tick never appears in that table and the fastest case would report no attempts at all. Two honest gaps: the HopNotSendReady branch is not reached by any test, and the concurrent-probe cap counts only unfinished jobs without a test covering that filter. Adds 63 tests across 32 files. One changelog entry under Added, describing the released state: the five stages and why they are separate, the session the probe opens and the one it must not tear down, the path being computed rather than observed, the three control commands and why they cannot block, and the two rendering modes. It says in as many words that the wire format is unchanged.
This commit is contained in:
Johnathan Corgan
@@ -190,6 +190,37 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
|
||||
invisible for a peer that has a second address that works: the lane would
|
||||
never carry traffic and nothing above debug logging would say so.
|
||||
|
||||
- `fipsctl probe <npub|hostname>` answers, for one target, where it sits in the
|
||||
spanning tree relative to this node and whether this node can actually reach
|
||||
it. The work runs as five stages that report separately, `bloom`,
|
||||
`discovery`, `path`, `session` and `rtt`, because one verdict covering
|
||||
several findings is what sends an operator to the source: "no peer's filter
|
||||
claims this address" says the mesh has never heard of the target, while "a
|
||||
filter claimed it and nothing answered" says the opposite. The probe opens an
|
||||
FSP session, waits for one MMP receiver report to yield a round-trip time,
|
||||
and tears down only what it opened. A session that existed before the probe
|
||||
started is never torn down, ownership is decided at the moment of action
|
||||
rather than once at the start, and it is re-checked before teardown, so a
|
||||
session adopted by traffic underneath the probe is left alone. **The path is
|
||||
computed from coordinates rather than observed**, and the output says so in
|
||||
those words: nothing traverses the mesh to confirm the hops, and a display
|
||||
that read like traceroute output would be believed as one. A real per-hop
|
||||
trace needs a wire message that does not exist. **Nothing here changes the
|
||||
wire format**; the probe is built from messages that already exist. The
|
||||
control socket carries three new commands, `probe_start`, `probe_poll` and
|
||||
`probe_cancel`, each returning in well under a millisecond with the stages
|
||||
advanced on the daemon's tick, because a probe needs a mesh lookup, a Noise
|
||||
XK handshake and at least one remote MMP tick, which no single control
|
||||
round-trip could survive inside the socket's five-second timeout. A probe
|
||||
that runs and finds a problem is not an error response: the status is `ok`
|
||||
and the failure sits in the per-stage verdicts, and error responses stay
|
||||
reserved for malformed or inadmissible requests. On a terminal the stage
|
||||
block is redrawn in place with a running elapsed on whichever stage is
|
||||
working; piped or redirected there is no cursor to move, so each row prints
|
||||
once, at the moment it settles, and the transcript ends up the same block a
|
||||
terminal leaves behind. `--json` emits exactly one document at the end, so a
|
||||
script parsing the report does not have to skip past progress output.
|
||||
|
||||
### Changed
|
||||
|
||||
- Node health is determined at start completion instead of unconditionally
|
||||
|
||||
@@ -115,6 +115,104 @@ Tell the daemon to drop a peer link.
|
||||
| -------- | ----------- |
|
||||
| `peer` | npub (bech32) or hostname from `/etc/fips/hosts`. |
|
||||
|
||||
### `probe <target>`
|
||||
|
||||
Diagnose whether a mesh endpoint is reachable, in five stages, and
|
||||
report a per-stage verdict so a partial failure localizes itself.
|
||||
|
||||
| Argument | Description |
|
||||
| -------- | ----------- |
|
||||
| `target` | npub (bech32) or hostname from `/etc/fips/hosts`. |
|
||||
| `--json` | Emit the report as JSON instead of human-readable text. |
|
||||
| `--timeout <secs>` | Client-side ceiling. Defaults to the budget the daemon computed, which scales with its tick interval. |
|
||||
|
||||
The stages are:
|
||||
|
||||
1. **bloom** — does any peer's announced filter claim the target, and
|
||||
did a LookupRequest therefore go out? A miss ends the probe here and
|
||||
is a statement about the mesh's own knowledge: nobody has heard of
|
||||
this address. Skipped when the coordinates are already cached, and
|
||||
when the target is a directly connected peer.
|
||||
2. **discovery** — waiting for a LookupResponse to answer with
|
||||
coordinates. Each request the node sends gets its own line under the
|
||||
stage, with the timeout that attempt was given and whether it drew a
|
||||
reply. Failing here is the opposite finding to a bloom miss: a peer's
|
||||
filter did claim the address, and nothing answered for it.
|
||||
3. **path** — the least-common-ancestor walk between the two
|
||||
coordinates, plus the next hop this node would select. This is a
|
||||
**local computation, not a traceroute**: no hop beyond the first is
|
||||
contacted, and the tree distance is an upper bound on the real hop
|
||||
count because a crosslink cut-through can deliver in fewer hops.
|
||||
When no coordinates were available the walk is not computed at all:
|
||||
`path.coords_known` is false and every tree field is null, rather
|
||||
than a default that would read as a finding about the spanning tree.
|
||||
4. **session** — a full Noise XK handshake over FSP. Completing it is
|
||||
genuine end-to-end evidence: our route reached them, their route
|
||||
reached us, and the remote holds the expected static key.
|
||||
5. **rtt** — one MMP sender/receiver report exchange, for a real
|
||||
round-trip time.
|
||||
|
||||
Exit status is 0 only for an overall verdict of `ok`; `partial`,
|
||||
`failed` and `cancelled` all exit 1.
|
||||
|
||||
**The stage block fills in as the probe runs.** Each stage reports its
|
||||
verdict at the moment it reaches one, rather than the whole report
|
||||
arriving at the end, so a slow stage is visible as the stage that is
|
||||
slow. On a terminal the block is redrawn in place, with a spinner and a
|
||||
running elapsed on whichever stage is working; piped or redirected, each
|
||||
row is printed once, when it settles, and the transcript ends up the
|
||||
same block. A running stage reports only what the daemon has observed —
|
||||
which requests have gone unanswered so far, whether the handshake is in
|
||||
flight, how many receiver reports have arrived — and never previews an
|
||||
outcome it does not have yet. The per-request lines under the discovery
|
||||
stage carry the configured timeout of each attempt in parentheses, which
|
||||
is the node's own ladder rather than a measurement.
|
||||
|
||||
The elapsed column comes from the daemon's clock throughout: a finished
|
||||
stage carries its own tick-quantized figure, and the stage still running
|
||||
carries the report's elapsed less the stages already accounted for.
|
||||
Nothing in that column is measured client-side.
|
||||
|
||||
**Stages that were never attempted get no row.** A failure marks
|
||||
everything behind it as not reached, and the report says that once, in
|
||||
the failed row, rather than three more times. Two cases deliberately keep
|
||||
their rows: a *skipped* stage, because a skip is a result naming why that
|
||||
stage was unnecessary, and everything after a *failed path* stage,
|
||||
because the path preview touches nothing and the session can still
|
||||
succeed where the preview named no next hop.
|
||||
|
||||
Below the stage block, the `path:` line renders the whole tree walk on
|
||||
one line, from this node to the target, through the least common
|
||||
ancestor, which is emphasised on a terminal. It is the same computed
|
||||
walk the `ours`, `theirs` and `tree walk` lines describe, read in one
|
||||
piece.
|
||||
|
||||
`--json` is unaffected and still emits exactly one document, when the
|
||||
probe ends, so a script parsing the report does not have to skip past
|
||||
progress output.
|
||||
|
||||
**What the probe leaves behind.** It tears down a session it opened
|
||||
itself and never touches one that already existed. Three residues are
|
||||
deliberate and worth knowing about:
|
||||
|
||||
- The coordinate-cache and identity-cache entries a lookup produced are
|
||||
not evicted. They are TTL-bounded shared read caches, and evicting
|
||||
them could strand an unrelated flow mid-route.
|
||||
- The remote's half of a probe-created session persists until its own
|
||||
idle timeout (default 90s). There is no teardown wire message. In the
|
||||
window between our removal and its idle purge, any session frame the
|
||||
remote sends lands here as one unknown-session reject.
|
||||
- To obtain a round-trip time the probe sends a `CoordsWarmup`, which
|
||||
starts MMP reporting on the session. On a session the probe does not
|
||||
own, that reporting continues until the idle purge — the same traffic
|
||||
any single data packet would cause, and bounded, but a real change to
|
||||
a session the probe did not create. The report names it under
|
||||
`cleanup.warmups_sent`.
|
||||
|
||||
Running a probe against a production node is safe: the job carries its
|
||||
own deadline daemon-side, so it cleans up whether or not the client is
|
||||
still there.
|
||||
|
||||
### `profile tick <on|off|status>`
|
||||
|
||||
> **Reading the output.** Step durations are wall clock measured across `await`
|
||||
|
||||
@@ -94,6 +94,10 @@ daemon closes it with no response.
|
||||
| `unknown command: <name>` | Command not registered with this daemon. |
|
||||
| `missing params for <name>` | Command requires `params` but none were provided. |
|
||||
| `missing '<field>' parameter` | Required parameter missing. |
|
||||
| `invalid peer npub: <e>` | `probe_start` could not decode the bech32 npub. |
|
||||
| `cannot probe this node` | `probe_start` was given this daemon's own npub. |
|
||||
| `unknown probe id: <n>` | `probe_poll` / `probe_cancel` named a job that does not exist, or one already collected by a previous poll. |
|
||||
| `too many probes in flight` | The probe registry is at its concurrency cap. |
|
||||
| `query timeout` | Internal handler did not respond within 5 seconds. |
|
||||
| `node shutting down` | Daemon is exiting. |
|
||||
| `gateway not yet initialized` | (Gateway socket only) snapshot has not been published yet. |
|
||||
@@ -163,6 +167,9 @@ not reproduced here to avoid duplicating the source.
|
||||
| ------- | --------------- | --------- |
|
||||
| `connect` | `npub` (bech32), `address` (transport endpoint), `transport` (`udp`, `tcp`, `tor`, `nym`, `ethernet`) | Asks the node to dial the peer over the named transport. The named transport must be configured and running. Returns the API result on success or an error string on failure. |
|
||||
| `disconnect` | `npub` (bech32) | Asks the node to drop the link to the named peer. |
|
||||
| `probe_start` | `npub` (bech32) | Admits a diagnostic probe job and returns immediately. `data`: `probe_id`, `npub`, `node_addr`, `display_name`, `budget_ms`. |
|
||||
| `probe_poll` | `probe_id` (integer) | Reports a probe's progress. `data`: `state` (`running` / `done`) and `report`. A terminal job is removed on the poll that observes it, so the report is delivered once. |
|
||||
| `probe_cancel` | `probe_id` (integer) | Runs the probe's terminal actions immediately, without the teardown grace tick. |
|
||||
|
||||
`connect` on a peer the node is **already connected to** neither tears the
|
||||
live link down nor ignores the address: the address is tried as an alternate
|
||||
@@ -175,8 +182,27 @@ dial to a peer the node does not yet hold also reports `refreshed: false`.
|
||||
`connect` is ephemeral either way: the peer is not written to the config file
|
||||
and gets no auto-reconnect, so an attempt that fails leaves no residue.
|
||||
|
||||
Both commands run on the daemon's main task and may block briefly
|
||||
while the node mutates its state.
|
||||
`connect` and `disconnect` run on the daemon's main task and may block
|
||||
briefly while the node mutates its state. The probe triplet does not:
|
||||
each call returns in well under a millisecond, and the stages run on
|
||||
the daemon's tick. That split exists because a probe needs a mesh
|
||||
lookup, a Noise XK handshake and at least one remote MMP tick, which no
|
||||
single control round-trip could survive inside the 5-second I/O
|
||||
timeout.
|
||||
|
||||
A probe that runs and finds a problem is **not** an error response: the
|
||||
status is `ok` and the failure is in the report's per-stage verdicts.
|
||||
Error responses are reserved for malformed or inadmissible requests.
|
||||
|
||||
The report carries one block per stage — `bloom`, `discovery`, `path`,
|
||||
`session`, `rtt` — plus `target`, `cleanup` and the overall verdict.
|
||||
The two lookup stages are separate because they fail for unrelated
|
||||
reasons: `bloom` says whether any peer's filter claimed the address and
|
||||
a request therefore went out, and `discovery` says whether anything
|
||||
answered. `discovery.attempts` is the request in flight, or the last
|
||||
one tried, and `discovery.attempt_timeouts_secs` is this node's
|
||||
configured ladder, published so a client can say how long each attempt
|
||||
was given rather than guessing.
|
||||
|
||||
#### Profiler toggle (`--features profiling` builds only)
|
||||
|
||||
|
||||
+1094
-1
File diff suppressed because it is too large
Load Diff
@@ -13,6 +13,9 @@ pub async fn dispatch(node: &mut Node, command: &str, params: Option<&Value>) ->
|
||||
match command {
|
||||
"connect" => connect(node, params).await,
|
||||
"disconnect" => disconnect(node, params).await,
|
||||
"probe_start" => probe_start(node, params).await,
|
||||
"probe_poll" => probe_poll(node, params),
|
||||
"probe_cancel" => probe_cancel(node, params).await,
|
||||
_ => Response::error(format!("unknown command: {command}")),
|
||||
}
|
||||
}
|
||||
@@ -66,3 +69,133 @@ async fn disconnect(node: &mut Node, params: Option<&Value>) -> Response {
|
||||
Err(msg) => Response::error(msg),
|
||||
}
|
||||
}
|
||||
|
||||
/// Start a diagnostic probe. Returns as soon as the job is admitted; the
|
||||
/// stages run on the tick and the client polls for the report.
|
||||
///
|
||||
/// Params: `{"npub": "npub1..."}`
|
||||
async fn probe_start(node: &mut Node, params: Option<&Value>) -> Response {
|
||||
let Some(params) = params else {
|
||||
return Response::error("missing params for probe_start");
|
||||
};
|
||||
|
||||
let npub = match params.get("npub").and_then(|v| v.as_str()) {
|
||||
Some(v) => v,
|
||||
None => return Response::error("missing 'npub' parameter"),
|
||||
};
|
||||
|
||||
debug!(npub = %npub, "API probe requested");
|
||||
|
||||
match node.api_probe_start(npub).await {
|
||||
Ok(data) => Response::ok(data),
|
||||
Err(msg) => Response::error(msg),
|
||||
}
|
||||
}
|
||||
|
||||
/// Read a probe's progress.
|
||||
///
|
||||
/// Params: `{"probe_id": 7}`
|
||||
fn probe_poll(node: &mut Node, params: Option<&Value>) -> Response {
|
||||
let Some(id) = probe_id(params) else {
|
||||
return Response::error("missing 'probe_id' parameter");
|
||||
};
|
||||
|
||||
match node.api_probe_poll(id) {
|
||||
Ok(data) => Response::ok(data),
|
||||
Err(msg) => Response::error(msg),
|
||||
}
|
||||
}
|
||||
|
||||
/// Cancel a probe.
|
||||
///
|
||||
/// Params: `{"probe_id": 7}`
|
||||
async fn probe_cancel(node: &mut Node, params: Option<&Value>) -> Response {
|
||||
let Some(id) = probe_id(params) else {
|
||||
return Response::error("missing 'probe_id' parameter");
|
||||
};
|
||||
|
||||
match node.api_probe_cancel(id).await {
|
||||
Ok(data) => Response::ok(data),
|
||||
Err(msg) => Response::error(msg),
|
||||
}
|
||||
}
|
||||
|
||||
fn probe_id(params: Option<&Value>) -> Option<u64> {
|
||||
params?.get("probe_id").and_then(|v| v.as_u64())
|
||||
}
|
||||
|
||||
#[cfg(test)]
|
||||
mod tests {
|
||||
use super::*;
|
||||
use crate::config::Config;
|
||||
use serde_json::json;
|
||||
|
||||
fn test_node() -> Node {
|
||||
Node::new(Config::new()).expect("default config is valid")
|
||||
}
|
||||
|
||||
/// A known-good npub that is not this node's own.
|
||||
const OTHER_NPUB: &str = "npub180cvv07tjdrrgpa0j7j7tmnyl2yr6yr7l8j4s3evf6u64th6gkwsyjh6w6";
|
||||
|
||||
#[tokio::test]
|
||||
async fn probe_poll_unknown_id_errors() {
|
||||
let mut node = test_node();
|
||||
let resp = dispatch(&mut node, "probe_poll", Some(&json!({"probe_id": 999}))).await;
|
||||
assert_eq!(resp.status, "error");
|
||||
}
|
||||
|
||||
#[tokio::test]
|
||||
async fn probe_start_rejects_bad_npub() {
|
||||
let mut node = test_node();
|
||||
let resp = dispatch(
|
||||
&mut node,
|
||||
"probe_start",
|
||||
Some(&json!({"npub": "npub1nonsense"})),
|
||||
)
|
||||
.await;
|
||||
assert_eq!(resp.status, "error");
|
||||
assert!(
|
||||
resp.message
|
||||
.as_deref()
|
||||
.is_some_and(|m| m.contains("invalid peer npub")),
|
||||
"message was {:?}",
|
||||
resp.message
|
||||
);
|
||||
}
|
||||
|
||||
#[tokio::test]
|
||||
async fn probe_start_missing_npub_errors() {
|
||||
let mut node = test_node();
|
||||
let resp = dispatch(&mut node, "probe_start", Some(&json!({}))).await;
|
||||
assert_eq!(resp.status, "error");
|
||||
}
|
||||
|
||||
#[tokio::test]
|
||||
async fn probe_start_rejects_self() {
|
||||
// The most natural first thing a user tries. Unguarded it produces an
|
||||
// uninterpretable result, since routing returns no next hop for the
|
||||
// local address.
|
||||
let mut node = test_node();
|
||||
let own = node.identity().npub();
|
||||
let resp = dispatch(&mut node, "probe_start", Some(&json!({"npub": own}))).await;
|
||||
assert_eq!(resp.status, "error");
|
||||
assert_eq!(resp.message.as_deref(), Some("cannot probe this node"));
|
||||
}
|
||||
|
||||
#[tokio::test]
|
||||
async fn probe_start_admits_a_valid_target() {
|
||||
let mut node = test_node();
|
||||
let resp = dispatch(&mut node, "probe_start", Some(&json!({"npub": OTHER_NPUB}))).await;
|
||||
assert_eq!(resp.status, "ok", "message: {:?}", resp.message);
|
||||
let data = resp.data.expect("probe_start returns data");
|
||||
assert_eq!(data["probe_id"], 1);
|
||||
assert!(data["budget_ms"].as_u64().is_some_and(|b| b > 0));
|
||||
}
|
||||
|
||||
#[tokio::test]
|
||||
async fn probe_cancel_missing_id_errors() {
|
||||
let mut node = test_node();
|
||||
let resp = dispatch(&mut node, "probe_cancel", None).await;
|
||||
assert_eq!(resp.status, "error");
|
||||
}
|
||||
}
|
||||
|
||||
@@ -11,6 +11,7 @@
|
||||
pub mod commands;
|
||||
pub mod firewall_state;
|
||||
pub mod listening;
|
||||
pub mod probe;
|
||||
pub mod protocol;
|
||||
pub mod queries;
|
||||
pub mod read_handle;
|
||||
|
||||
@@ -0,0 +1,267 @@
|
||||
//! Control-socket projection of a probe report.
|
||||
//!
|
||||
//! Plain serde structs mirroring the sans-IO core's stage records. Only
|
||||
//! `rtt_ms`, `srtt_ms` and `path_mtu` vanish when unmeasured; every other key
|
||||
//! is always present so a script need not guard. The `path` block is
|
||||
//! deliberately explicit that it is a local computation: `computed_locally` is
|
||||
//! always true and `observed` always false, so a machine reader cannot mistake
|
||||
//! it for a traceroute.
|
||||
|
||||
use serde::Serialize;
|
||||
|
||||
use crate::node::Node;
|
||||
use crate::node::ProbeJob;
|
||||
use crate::proto::probe::{ProbeSnapshot, StageRecord};
|
||||
|
||||
/// Common to every stage. `verdict` is the per-stage answer.
|
||||
#[derive(Serialize)]
|
||||
pub struct StageStatus {
|
||||
/// "pending" | "running" | "ok" | "skipped" | "failed"
|
||||
pub verdict: &'static str,
|
||||
/// Machine-stable failure or skip discriminator; `null` when the stage is
|
||||
/// ok, pending or running.
|
||||
pub reason: Option<&'static str>,
|
||||
/// Free text for the one case carrying a real message: the error string
|
||||
/// out of session initiation.
|
||||
pub detail: Option<String>,
|
||||
pub elapsed_ms: Option<u64>,
|
||||
}
|
||||
|
||||
impl From<&StageRecord> for StageStatus {
|
||||
fn from(record: &StageRecord) -> Self {
|
||||
Self {
|
||||
verdict: record.verdict.name(),
|
||||
reason: record.reason.map(|r| r.name()),
|
||||
detail: record.detail.clone(),
|
||||
elapsed_ms: record.elapsed_ms,
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
#[derive(Serialize)]
|
||||
pub struct TargetInfo {
|
||||
pub npub: String,
|
||||
pub node_addr: String,
|
||||
pub display_name: String,
|
||||
pub ipv6_addr: String,
|
||||
}
|
||||
|
||||
/// Whether any peer's announced filter claims the target, which decides
|
||||
/// whether a LookupRequest is sent at all.
|
||||
#[derive(Serialize)]
|
||||
pub struct BloomStage {
|
||||
#[serde(flatten)]
|
||||
pub status: StageStatus,
|
||||
/// Peers the LookupRequest actually reached; null when none was issued.
|
||||
pub fanout: Option<usize>,
|
||||
}
|
||||
|
||||
/// Waiting for a LookupResponse to answer with coordinates.
|
||||
#[derive(Serialize)]
|
||||
pub struct DiscoveryStage {
|
||||
#[serde(flatten)]
|
||||
pub status: StageStatus,
|
||||
/// "cache" | "lookup" | "direct_peer" | null
|
||||
pub source: Option<&'static str>,
|
||||
/// The attempt in flight, or the last one tried. Counts from 1.
|
||||
pub attempts: Option<u8>,
|
||||
/// This node's lookup ladder: the timeout of each attempt, in order. It
|
||||
/// is configuration rather than measurement, and is published so a client
|
||||
/// can say how long each attempt was given without guessing.
|
||||
pub attempt_timeouts_secs: Vec<u64>,
|
||||
}
|
||||
|
||||
#[derive(Serialize)]
|
||||
pub struct NextHopInfo {
|
||||
pub node_addr: String,
|
||||
pub display_name: String,
|
||||
pub class: &'static str,
|
||||
pub direct_peer: bool,
|
||||
/// True when the first hop leaves the up-then-down tree walk that
|
||||
/// `tree_hops_*` describes.
|
||||
pub leaves_tree_walk: bool,
|
||||
}
|
||||
|
||||
#[derive(Serialize)]
|
||||
pub struct PathStage {
|
||||
#[serde(flatten)]
|
||||
pub status: StageStatus,
|
||||
/// Always true: these facts are computed here from coordinates.
|
||||
pub computed_locally: bool,
|
||||
/// Always false: no hop beyond the first was contacted.
|
||||
pub observed: bool,
|
||||
/// False when no coordinates were available, in which case every field
|
||||
/// below that describes the tree is null rather than a computed value. A
|
||||
/// resolve failure and a fresh direct peer both land here, and neither
|
||||
/// justifies a claim about the spanning tree.
|
||||
pub coords_known: bool,
|
||||
pub our_coords: Vec<String>,
|
||||
pub their_coords: Vec<String>,
|
||||
pub our_depth: Option<usize>,
|
||||
pub their_depth: Option<usize>,
|
||||
pub same_root: Option<bool>,
|
||||
pub lca: Option<String>,
|
||||
pub lca_depth: Option<usize>,
|
||||
/// Tree metric, not a predicted hop count.
|
||||
pub tree_hops_up: Option<usize>,
|
||||
pub tree_hops_down: Option<usize>,
|
||||
/// Upper bound on the real hop count; a crosslink cut-through shortens it.
|
||||
pub tree_distance: Option<usize>,
|
||||
pub next_hop: Option<NextHopInfo>,
|
||||
pub no_hop_reason: Option<&'static str>,
|
||||
}
|
||||
|
||||
#[derive(Serialize)]
|
||||
pub struct SessionStage {
|
||||
#[serde(flatten)]
|
||||
pub status: StageStatus,
|
||||
/// An entry existed at the moment the probe would have opened one, so the
|
||||
/// probe neither opened nor closed it.
|
||||
pub preexisting: bool,
|
||||
pub established: bool,
|
||||
#[serde(skip_serializing_if = "Option::is_none")]
|
||||
pub path_mtu: Option<u32>,
|
||||
}
|
||||
|
||||
#[derive(Serialize)]
|
||||
pub struct RttStage {
|
||||
#[serde(flatten)]
|
||||
pub status: StageStatus,
|
||||
#[serde(skip_serializing_if = "Option::is_none")]
|
||||
pub rtt_ms: Option<u32>,
|
||||
#[serde(skip_serializing_if = "Option::is_none")]
|
||||
pub srtt_ms: Option<f64>,
|
||||
/// Deltas measured across the probe.
|
||||
pub reports_seen: u64,
|
||||
pub samples: u64,
|
||||
pub zero_samples: u64,
|
||||
pub arith_failures: u64,
|
||||
}
|
||||
|
||||
#[derive(Serialize)]
|
||||
pub struct CleanupInfo {
|
||||
pub session_created_and_torn_down: bool,
|
||||
pub session_left_intact: bool,
|
||||
/// "preexisting" | "adopted_by_traffic" | "replaced" | null
|
||||
pub left_intact_reason: Option<&'static str>,
|
||||
pub lookup_issued: bool,
|
||||
/// The coord-cache and identity-cache entries a lookup produced are
|
||||
/// deliberately not evicted; these say what was already there.
|
||||
pub coords_were_cached: bool,
|
||||
pub identity_was_cached: bool,
|
||||
pub warmups_sent: u8,
|
||||
}
|
||||
|
||||
#[derive(Serialize)]
|
||||
pub struct ProbeReport {
|
||||
pub probe_id: u64,
|
||||
pub target: TargetInfo,
|
||||
/// "running" | "ok" | "partial" | "failed" | "cancelled"
|
||||
pub overall: &'static str,
|
||||
pub elapsed_ms: u64,
|
||||
/// Tick period in ms. Every stage duration is quantized to this.
|
||||
pub tick_ms: u64,
|
||||
pub bloom: BloomStage,
|
||||
pub discovery: DiscoveryStage,
|
||||
pub path: PathStage,
|
||||
pub session: SessionStage,
|
||||
pub rtt: RttStage,
|
||||
pub cleanup: CleanupInfo,
|
||||
}
|
||||
|
||||
impl ProbeReport {
|
||||
/// Project a live job into its report. `now_ms` is monotonic, used only
|
||||
/// for the elapsed figure of a job that has not finished.
|
||||
pub(crate) fn build(node: &Node, job: &ProbeJob, now_ms: u64) -> Self {
|
||||
let snap = job.probe().snapshot();
|
||||
let target = *job.target();
|
||||
Self {
|
||||
probe_id: job.id(),
|
||||
target: TargetInfo {
|
||||
npub: job.npub().to_string(),
|
||||
node_addr: target.to_string(),
|
||||
display_name: node.peer_display_name(&target),
|
||||
ipv6_addr: crate::FipsAddress::from_node_addr(&target).to_string(),
|
||||
},
|
||||
overall: snap.overall.name(),
|
||||
elapsed_ms: job.probe().elapsed_ms(now_ms),
|
||||
tick_ms: snap.tick_ms,
|
||||
bloom: BloomStage {
|
||||
status: (&snap.bloom).into(),
|
||||
fanout: snap.lookup_fanout,
|
||||
},
|
||||
discovery: DiscoveryStage {
|
||||
status: (&snap.discovery).into(),
|
||||
source: snap.resolve_source.map(|s| s.name()),
|
||||
attempts: job.lookup_attempts(),
|
||||
attempt_timeouts_secs: node.config().node.lookup.attempt_timeouts_secs.clone(),
|
||||
},
|
||||
path: path_stage(node, &snap),
|
||||
session: SessionStage {
|
||||
status: (&snap.session).into(),
|
||||
preexisting: snap.session_preexisting,
|
||||
established: snap.session_established,
|
||||
path_mtu: snap.path_mtu,
|
||||
},
|
||||
rtt: RttStage {
|
||||
status: (&snap.rtt).into(),
|
||||
rtt_ms: snap.rtt_ms,
|
||||
srtt_ms: snap.srtt_ms,
|
||||
reports_seen: snap.counters.reports_seen,
|
||||
samples: snap.counters.samples,
|
||||
zero_samples: snap.counters.zero,
|
||||
arith_failures: snap.counters.arith_fail,
|
||||
},
|
||||
cleanup: cleanup_info(job, &snap),
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
fn path_stage(node: &Node, snap: &ProbeSnapshot) -> PathStage {
|
||||
let facts = snap.path_facts.as_ref();
|
||||
PathStage {
|
||||
status: (&snap.path).into(),
|
||||
computed_locally: true,
|
||||
observed: false,
|
||||
coords_known: facts.is_some(),
|
||||
our_coords: facts.map_or_else(Vec::new, |f| {
|
||||
f.our_coords.iter().map(|a| a.to_string()).collect()
|
||||
}),
|
||||
their_coords: facts.map_or_else(Vec::new, |f| {
|
||||
f.their_coords.iter().map(|a| a.to_string()).collect()
|
||||
}),
|
||||
our_depth: facts.map(|f| f.our_depth),
|
||||
their_depth: facts.map(|f| f.their_depth),
|
||||
same_root: facts.map(|f| f.same_root),
|
||||
lca: facts.and_then(|f| f.lca.map(|a| a.to_string())),
|
||||
lca_depth: facts.and_then(|f| f.lca_depth),
|
||||
tree_hops_up: facts.and_then(|f| f.tree_hops_up),
|
||||
tree_hops_down: facts.and_then(|f| f.tree_hops_down),
|
||||
tree_distance: facts.and_then(|f| f.tree_distance),
|
||||
next_hop: snap.next_hop.as_ref().map(|h| NextHopInfo {
|
||||
node_addr: h.node_addr.to_string(),
|
||||
display_name: node.peer_display_name(&h.node_addr),
|
||||
class: h.class.name(),
|
||||
direct_peer: h.direct_peer,
|
||||
leaves_tree_walk: h.leaves_tree_walk,
|
||||
}),
|
||||
no_hop_reason: snap.no_hop_reason.map(|r| r.name()),
|
||||
}
|
||||
}
|
||||
|
||||
fn cleanup_info(job: &ProbeJob, snap: &ProbeSnapshot) -> CleanupInfo {
|
||||
// A drive-time refusal overrides the core's decision: the action was
|
||||
// emitted but the guard declined, so the session is still there.
|
||||
let refused = job.teardown_refused();
|
||||
let torn_down = snap.torn_down && refused.is_none();
|
||||
let left_intact = refused.or(snap.left_intact);
|
||||
CleanupInfo {
|
||||
session_created_and_torn_down: torn_down,
|
||||
session_left_intact: left_intact.is_some(),
|
||||
left_intact_reason: left_intact.map(|r| r.name()),
|
||||
lookup_issued: snap.lookup_issued,
|
||||
coords_were_cached: snap.coords_were_cached,
|
||||
identity_was_cached: snap.identity_was_cached,
|
||||
warmups_sent: snap.warmups_sent,
|
||||
}
|
||||
}
|
||||
+105
-2
@@ -2767,8 +2767,17 @@ mod tests {
|
||||
assert_eq!(resp.status, "ok", "{cmd} off-loop response not ok");
|
||||
}
|
||||
|
||||
// Mutations are NOT served off-loop; they take the rx_loop COMMAND path.
|
||||
for cmd in ["connect", "disconnect"] {
|
||||
// Mutations are NOT served off-loop; they take the rx_loop COMMAND
|
||||
// path. A probe served from the one-tick-stale off-loop snapshot would
|
||||
// silently never run, so the three probe commands are asserted here
|
||||
// alongside connect/disconnect.
|
||||
for cmd in [
|
||||
"connect",
|
||||
"disconnect",
|
||||
"probe_start",
|
||||
"probe_poll",
|
||||
"probe_cancel",
|
||||
] {
|
||||
let req = Request {
|
||||
command: cmd.to_string(),
|
||||
params: None,
|
||||
@@ -2780,6 +2789,100 @@ mod tests {
|
||||
}
|
||||
}
|
||||
|
||||
/// Schema guard for the probe report. Built by hand from a terminal
|
||||
/// outcome so the fixture pins field names, nesting and the flattened
|
||||
/// stage status.
|
||||
#[test]
|
||||
fn probe_report_json_shape() {
|
||||
use crate::control::probe::{
|
||||
BloomStage, CleanupInfo, DiscoveryStage, NextHopInfo, PathStage, ProbeReport, RttStage,
|
||||
SessionStage, StageStatus, TargetInfo,
|
||||
};
|
||||
|
||||
let status = |verdict, elapsed| StageStatus {
|
||||
verdict,
|
||||
reason: None,
|
||||
detail: None,
|
||||
elapsed_ms: Some(elapsed),
|
||||
};
|
||||
let report = ProbeReport {
|
||||
probe_id: 7,
|
||||
target: TargetInfo {
|
||||
npub: "npub1qq8s4f7x".to_string(),
|
||||
node_addr: "4a1f9c22e08b5d3714aa06fb92c1de55".to_string(),
|
||||
display_name: "hydra".to_string(),
|
||||
ipv6_addr: "fd00:4a1f:9c22:e08b:5d37:14aa:06fb:92c1".to_string(),
|
||||
},
|
||||
overall: "ok",
|
||||
elapsed_ms: 4000,
|
||||
tick_ms: 1000,
|
||||
bloom: BloomStage {
|
||||
status: status("ok", 0),
|
||||
fanout: Some(3),
|
||||
},
|
||||
discovery: DiscoveryStage {
|
||||
status: status("ok", 2000),
|
||||
source: Some("lookup"),
|
||||
attempts: Some(2),
|
||||
attempt_timeouts_secs: vec![1, 2, 4, 8],
|
||||
},
|
||||
path: PathStage {
|
||||
status: status("ok", 0),
|
||||
computed_locally: true,
|
||||
observed: false,
|
||||
coords_known: true,
|
||||
our_coords: vec!["4d2201ff".to_string(), "91b40c6e".to_string()],
|
||||
their_coords: vec!["4a1f9c22".to_string(), "91b40c6e".to_string()],
|
||||
our_depth: Some(1),
|
||||
their_depth: Some(1),
|
||||
same_root: Some(true),
|
||||
lca: Some("91b40c6e".to_string()),
|
||||
lca_depth: Some(0),
|
||||
tree_hops_up: Some(1),
|
||||
tree_hops_down: Some(1),
|
||||
tree_distance: Some(2),
|
||||
next_hop: Some(NextHopInfo {
|
||||
node_addr: "91b40c6e".to_string(),
|
||||
display_name: "relay-a".to_string(),
|
||||
class: "tree_up",
|
||||
direct_peer: false,
|
||||
leaves_tree_walk: false,
|
||||
}),
|
||||
no_hop_reason: None,
|
||||
},
|
||||
session: SessionStage {
|
||||
status: status("ok", 1000),
|
||||
preexisting: false,
|
||||
established: true,
|
||||
path_mtu: Some(1420),
|
||||
},
|
||||
rtt: RttStage {
|
||||
status: status("ok", 1000),
|
||||
rtt_ms: Some(18),
|
||||
srtt_ms: Some(18.0),
|
||||
reports_seen: 1,
|
||||
samples: 1,
|
||||
zero_samples: 0,
|
||||
arith_failures: 0,
|
||||
},
|
||||
cleanup: CleanupInfo {
|
||||
session_created_and_torn_down: true,
|
||||
session_left_intact: false,
|
||||
left_intact_reason: None,
|
||||
lookup_issued: true,
|
||||
coords_were_cached: false,
|
||||
identity_was_cached: false,
|
||||
warmups_sent: 1,
|
||||
},
|
||||
};
|
||||
|
||||
let mut value = serde_json::to_value(&report).expect("report serializes");
|
||||
normalize_value(&mut value);
|
||||
let sorted = sort_object_keys(&value);
|
||||
let actual = serde_json::to_string_pretty(&sorted).expect("pretty");
|
||||
assert_snapshot("probe_report", &actual);
|
||||
}
|
||||
|
||||
/// Structural confirmation that the rx_loop no longer dispatches `show_*`:
|
||||
/// the rx_loop source carries no `queries::dispatch` call and no
|
||||
/// `starts_with("show_")` routing branch. Reads the committed source of
|
||||
|
||||
@@ -0,0 +1,96 @@
|
||||
{
|
||||
"bloom": {
|
||||
"detail": null,
|
||||
"elapsed_ms": 0,
|
||||
"fanout": 3,
|
||||
"reason": null,
|
||||
"verdict": "ok"
|
||||
},
|
||||
"cleanup": {
|
||||
"coords_were_cached": false,
|
||||
"identity_was_cached": false,
|
||||
"left_intact_reason": null,
|
||||
"lookup_issued": true,
|
||||
"session_created_and_torn_down": true,
|
||||
"session_left_intact": false,
|
||||
"warmups_sent": 1
|
||||
},
|
||||
"discovery": {
|
||||
"attempt_timeouts_secs": [
|
||||
1,
|
||||
2,
|
||||
4,
|
||||
8
|
||||
],
|
||||
"attempts": 2,
|
||||
"detail": null,
|
||||
"elapsed_ms": 2000,
|
||||
"reason": null,
|
||||
"source": "lookup",
|
||||
"verdict": "ok"
|
||||
},
|
||||
"elapsed_ms": 4000,
|
||||
"overall": "ok",
|
||||
"path": {
|
||||
"computed_locally": true,
|
||||
"coords_known": true,
|
||||
"detail": null,
|
||||
"elapsed_ms": 0,
|
||||
"lca": "91b40c6e",
|
||||
"lca_depth": 0,
|
||||
"next_hop": {
|
||||
"class": "tree_up",
|
||||
"direct_peer": false,
|
||||
"display_name": "relay-a",
|
||||
"leaves_tree_walk": false,
|
||||
"node_addr": "91b40c6e"
|
||||
},
|
||||
"no_hop_reason": null,
|
||||
"observed": false,
|
||||
"our_coords": [
|
||||
"4d2201ff",
|
||||
"91b40c6e"
|
||||
],
|
||||
"our_depth": 1,
|
||||
"reason": null,
|
||||
"same_root": true,
|
||||
"their_coords": [
|
||||
"4a1f9c22",
|
||||
"91b40c6e"
|
||||
],
|
||||
"their_depth": 1,
|
||||
"tree_distance": 2,
|
||||
"tree_hops_down": 1,
|
||||
"tree_hops_up": 1,
|
||||
"verdict": "ok"
|
||||
},
|
||||
"probe_id": 7,
|
||||
"rtt": {
|
||||
"arith_failures": 0,
|
||||
"detail": null,
|
||||
"elapsed_ms": 1000,
|
||||
"reason": null,
|
||||
"reports_seen": 1,
|
||||
"rtt_ms": 18,
|
||||
"samples": 1,
|
||||
"srtt_ms": 18.0,
|
||||
"verdict": "ok",
|
||||
"zero_samples": 0
|
||||
},
|
||||
"session": {
|
||||
"detail": null,
|
||||
"elapsed_ms": 1000,
|
||||
"established": true,
|
||||
"path_mtu": 1420,
|
||||
"preexisting": false,
|
||||
"reason": null,
|
||||
"verdict": "ok"
|
||||
},
|
||||
"target": {
|
||||
"display_name": "hydra",
|
||||
"ipv6_addr": "fd00:4a1f:9c22:e08b:5d37:14aa:06fb:92c1",
|
||||
"node_addr": "4a1f9c22e08b5d3714aa06fb92c1de55",
|
||||
"npub": "npub1qq8s4f7x"
|
||||
},
|
||||
"tick_ms": 1000
|
||||
}
|
||||
@@ -69,6 +69,7 @@ pub(crate) enum Step {
|
||||
CheckRekey,
|
||||
CheckSessionRekey,
|
||||
CheckPendingLookups,
|
||||
PollProbes,
|
||||
PollTransportDiscovery,
|
||||
SampleTransportCongestion,
|
||||
ActivateConnectedUdpSessions,
|
||||
@@ -107,6 +108,7 @@ pub(crate) const STEPS: [Step; N_STEPS] = [
|
||||
Step::CheckRekey,
|
||||
Step::CheckSessionRekey,
|
||||
Step::CheckPendingLookups,
|
||||
Step::PollProbes,
|
||||
Step::PollTransportDiscovery,
|
||||
Step::SampleTransportCongestion,
|
||||
Step::ActivateConnectedUdpSessions,
|
||||
@@ -140,6 +142,7 @@ impl Step {
|
||||
Step::CheckRekey => "check_rekey",
|
||||
Step::CheckSessionRekey => "check_session_rekey",
|
||||
Step::CheckPendingLookups => "check_pending_lookups",
|
||||
Step::PollProbes => "poll_probes",
|
||||
Step::PollTransportDiscovery => "poll_transport_discovery",
|
||||
Step::SampleTransportCongestion => "sample_transport_congestion",
|
||||
Step::ActivateConnectedUdpSessions => "activate_connected_udp_sessions",
|
||||
@@ -378,9 +381,9 @@ mod tests {
|
||||
#[test]
|
||||
fn emitted_row_count_matches_build() {
|
||||
let emitted = STEPS.iter().filter(|s| s.emitted()).count();
|
||||
// 25 unconditional subsystem steps + the whole-tick span, plus the two
|
||||
// 26 unconditional subsystem steps + the whole-tick span, plus the two
|
||||
// conditionally-compiled steps where this build has them.
|
||||
let mut expected = 26;
|
||||
let mut expected = 27;
|
||||
if cfg!(any(target_os = "linux", target_os = "macos")) {
|
||||
expected += 1;
|
||||
}
|
||||
|
||||
@@ -403,6 +403,11 @@ impl Node {
|
||||
self.check_session_rekey().await);
|
||||
instr_step!(instr_on, crate::instr::Domain::Tick, crate::instr::Step::CheckPendingLookups,
|
||||
self.check_pending_lookups(now_ms).await);
|
||||
// After CheckPendingLookups so a probe's resolve stage
|
||||
// observes this tick's lookup progress rather than the
|
||||
// previous tick's.
|
||||
instr_step!(instr_on, crate::instr::Domain::Tick, crate::instr::Step::PollProbes,
|
||||
self.poll_probes().await);
|
||||
instr_step!(instr_on, crate::instr::Domain::Tick, crate::instr::Step::PollTransportDiscovery,
|
||||
self.poll_transport_discovery().await);
|
||||
instr_step!(instr_on, crate::instr::Domain::Tick, crate::instr::Step::SampleTransportCongestion,
|
||||
|
||||
@@ -10,6 +10,7 @@ use crate::node::reject::DiscoveryReject;
|
||||
use crate::proto::lookup::{
|
||||
LookupAction, LookupRequest, LookupResponse, MAX_RECENT_LOOKUP_REQUESTS,
|
||||
};
|
||||
use crate::proto::probe::LookupOutcomeKind;
|
||||
use crate::transport::{TransportAddr, TransportId};
|
||||
use crate::{NodeAddr, PeerIdentity};
|
||||
use tracing::{debug, info, trace, warn};
|
||||
@@ -26,6 +27,41 @@ struct NodeRoutingView<'a> {
|
||||
node: &'a Node,
|
||||
}
|
||||
|
||||
/// What [`Node::maybe_initiate_lookup`] did, for callers that report on the
|
||||
/// lookup rather than merely triggering it. The three existing data-path call
|
||||
/// sites are statement-position and discard it unchanged.
|
||||
pub(in crate::node) struct LookupInitiateOutcome {
|
||||
kind: LookupOutcomeKind,
|
||||
/// Peers the LookupRequest actually reached; `None` when the gate declined
|
||||
/// and no request was built.
|
||||
sent: Option<usize>,
|
||||
}
|
||||
|
||||
impl LookupInitiateOutcome {
|
||||
fn gated(kind: LookupOutcomeKind) -> Self {
|
||||
Self { kind, sent: None }
|
||||
}
|
||||
|
||||
fn sent(sent: usize) -> Self {
|
||||
Self {
|
||||
kind: if sent == 0 {
|
||||
LookupOutcomeKind::ZeroFanout
|
||||
} else {
|
||||
LookupOutcomeKind::Sent
|
||||
},
|
||||
sent: Some(sent),
|
||||
}
|
||||
}
|
||||
|
||||
pub(in crate::node) fn kind(&self) -> LookupOutcomeKind {
|
||||
self.kind
|
||||
}
|
||||
|
||||
pub(in crate::node) fn fanout(&self) -> Option<usize> {
|
||||
self.sent
|
||||
}
|
||||
}
|
||||
|
||||
impl crate::proto::lookup::RoutingView for NodeRoutingView<'_> {
|
||||
fn is_tree_peer(&self, addr: &NodeAddr) -> bool {
|
||||
self.node.is_tree_peer(addr)
|
||||
@@ -592,7 +628,10 @@ impl Node {
|
||||
/// Subsequent attempts (with fresh request_ids) are scheduled by
|
||||
/// [`Self::check_pending_lookups`] when each attempt's per-attempt timeout
|
||||
/// expires, using the sequence in `node.lookup.attempt_timeouts_secs`.
|
||||
pub(in crate::node) async fn maybe_initiate_lookup(&mut self, dest: &NodeAddr) {
|
||||
pub(in crate::node) async fn maybe_initiate_lookup(
|
||||
&mut self,
|
||||
dest: &NodeAddr,
|
||||
) -> LookupInitiateOutcome {
|
||||
let now_ms = Self::now_ms();
|
||||
|
||||
// Bloom filter pre-check (view read) BEFORE the core call: if no peer's
|
||||
@@ -609,6 +648,7 @@ impl Node {
|
||||
target_node = %self.peer_display_name(dest),
|
||||
"Discovery lookup deduplicated, already pending"
|
||||
);
|
||||
LookupInitiateOutcome::gated(LookupOutcomeKind::Deduplicated)
|
||||
}
|
||||
InitiateDecision::Suppressed { failures } => {
|
||||
self.metrics().lookup.req_backoff_suppressed.inc();
|
||||
@@ -617,6 +657,7 @@ impl Node {
|
||||
failures = failures,
|
||||
"Discovery lookup suppressed by backoff"
|
||||
);
|
||||
LookupInitiateOutcome::gated(LookupOutcomeKind::Suppressed)
|
||||
}
|
||||
InitiateDecision::BloomMiss => {
|
||||
self.metrics().lookup.req_bloom_miss.inc();
|
||||
@@ -624,6 +665,7 @@ impl Node {
|
||||
target_node = %self.peer_display_name(dest),
|
||||
"Discovery skipped, target not in any peer bloom filter"
|
||||
);
|
||||
LookupInitiateOutcome::gated(LookupOutcomeKind::BloomMiss)
|
||||
}
|
||||
InitiateDecision::Proceed => {
|
||||
let ttl = self.config().node.lookup.ttl;
|
||||
@@ -637,6 +679,7 @@ impl Node {
|
||||
"Discovery failed, no tree peers with bloom match"
|
||||
);
|
||||
}
|
||||
LookupInitiateOutcome::sent(sent)
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
@@ -3,6 +3,7 @@
|
||||
pub(in crate::node) mod handshake;
|
||||
pub(crate) mod lookup;
|
||||
mod mmp;
|
||||
pub(crate) mod probe;
|
||||
mod rekey;
|
||||
pub(in crate::node) mod session;
|
||||
mod timeout;
|
||||
|
||||
@@ -0,0 +1,541 @@
|
||||
//! Async driver for the `probe` diagnostic.
|
||||
//!
|
||||
//! The probe needs a mesh lookup, a Noise XK handshake and at least one remote
|
||||
//! MMP tick, so it cannot be one control round-trip: `IO_TIMEOUT` is 5s and the
|
||||
//! command dispatcher is awaited inline inside the rx-loop `select!`. It is
|
||||
//! therefore a daemon-side job driven on the tick, with a start/poll/cancel
|
||||
//! control triplet. Every round-trip returns immediately; `fipsctl` hides the
|
||||
//! polling.
|
||||
//!
|
||||
//! Because the job carries its own deadline it self-cleans when the client
|
||||
//! goes away, which is what makes running a probe against a production node
|
||||
//! safe. All protocol and stage decisions live in the sans-IO
|
||||
//! [`crate::proto::probe`] core; this file performs I/O, reads the clocks, and
|
||||
//! executes the returned actions.
|
||||
|
||||
use std::collections::{BTreeMap, HashSet};
|
||||
|
||||
use secp256k1::PublicKey;
|
||||
use tracing::{debug, info};
|
||||
|
||||
use crate::node::Node;
|
||||
use crate::node::session::SessionEntry;
|
||||
use crate::proto::probe::{
|
||||
Budgets, LeftIntact, LookupOutcomeKind, MAX_CONCURRENT_PROBES, NextHopFacts, NoHopReason,
|
||||
Observation, Preflight, Probe, ProbeAction, REAP_MS, RttCounters, describe_path,
|
||||
};
|
||||
use crate::proto::routing::{self, RouteClass};
|
||||
use crate::{NodeAddr, PeerIdentity};
|
||||
|
||||
/// Why a live session entry is not the probe's to remove.
|
||||
#[derive(Copy, Clone, Debug, PartialEq, Eq)]
|
||||
enum Disowned {
|
||||
/// The probe never created one.
|
||||
NeverOwned,
|
||||
/// A different entry took its place, or the peer became the initiator.
|
||||
Replaced,
|
||||
/// Real application traffic moved on the session during the probe.
|
||||
Adopted,
|
||||
/// The entry is no longer there at all.
|
||||
Gone,
|
||||
}
|
||||
|
||||
/// One in-flight probe plus the shell-side facts the core cannot read.
|
||||
pub(crate) struct ProbeJob {
|
||||
id: u64,
|
||||
target: NodeAddr,
|
||||
pubkey: PublicKey,
|
||||
npub: String,
|
||||
probe: Probe,
|
||||
/// Set only while this job holds the registry's target claim.
|
||||
holds_claim: bool,
|
||||
/// `created_at` of the entry `initiate_session` inserted for us. The
|
||||
/// teardown identity check is an equality against this, not an inequality
|
||||
/// against the start time — a `>=` still admits a replacement entry.
|
||||
owned_created_at: Option<u64>,
|
||||
activity_at_establish: Option<u64>,
|
||||
traffic_at_establish: Option<(u64, u64, u64, u64)>,
|
||||
lookup_outcome: Option<LookupOutcomeKind>,
|
||||
lookup_fanout: Option<usize>,
|
||||
lookup_attempts: Option<u8>,
|
||||
session_error: Option<String>,
|
||||
/// Set when the drive-time guard refused a teardown the core had already
|
||||
/// decided on, so the report says the session was left in place.
|
||||
teardown_refused: Option<LeftIntact>,
|
||||
reap_at_ms: Option<u64>,
|
||||
}
|
||||
|
||||
impl ProbeJob {
|
||||
pub(crate) fn id(&self) -> u64 {
|
||||
self.id
|
||||
}
|
||||
|
||||
pub(crate) fn target(&self) -> &NodeAddr {
|
||||
&self.target
|
||||
}
|
||||
|
||||
pub(crate) fn npub(&self) -> &str {
|
||||
&self.npub
|
||||
}
|
||||
|
||||
pub(crate) fn probe(&self) -> &Probe {
|
||||
&self.probe
|
||||
}
|
||||
|
||||
pub(crate) fn lookup_attempts(&self) -> Option<u8> {
|
||||
self.lookup_attempts
|
||||
}
|
||||
|
||||
pub(crate) fn teardown_refused(&self) -> Option<LeftIntact> {
|
||||
self.teardown_refused
|
||||
}
|
||||
}
|
||||
|
||||
/// Probe jobs in flight, with a per-target ownership claim.
|
||||
///
|
||||
/// The claim is what stops two probes started between ticks from both taking
|
||||
/// ownership of one target's session: the first job to reach `OpenSession`
|
||||
/// holds it, and the second behaves exactly as it would for a pre-existing
|
||||
/// session.
|
||||
pub(in crate::node) struct ProbeRegistry {
|
||||
jobs: BTreeMap<u64, ProbeJob>,
|
||||
claims: HashSet<NodeAddr>,
|
||||
next_id: u64,
|
||||
}
|
||||
|
||||
impl ProbeRegistry {
|
||||
pub(in crate::node) fn new() -> Self {
|
||||
Self {
|
||||
jobs: BTreeMap::new(),
|
||||
claims: HashSet::new(),
|
||||
next_id: 1,
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
impl Node {
|
||||
// === Control API methods ===
|
||||
|
||||
/// Start a probe toward `npub` and return immediately.
|
||||
pub(crate) async fn api_probe_start(
|
||||
&mut self,
|
||||
npub: &str,
|
||||
) -> Result<serde_json::Value, String> {
|
||||
let identity =
|
||||
PeerIdentity::from_npub(npub).map_err(|e| format!("invalid peer npub: {e}"))?;
|
||||
let target = *identity.node_addr();
|
||||
if target == *self.node_addr() {
|
||||
return Err("cannot probe this node".to_string());
|
||||
}
|
||||
// Only unfinished jobs count against the cap. A terminal job is
|
||||
// retained for `REAP_MS` so a late poll still gets its report, and
|
||||
// counting those would lock a caller that never polls out of new
|
||||
// probes for 30s with an error that says the opposite of what is true.
|
||||
let in_flight = self
|
||||
.probes
|
||||
.jobs
|
||||
.values()
|
||||
.filter(|j| !j.probe.is_finished())
|
||||
.count();
|
||||
if in_flight >= MAX_CONCURRENT_PROBES {
|
||||
return Err("too many probes in flight".to_string());
|
||||
}
|
||||
|
||||
// Seeding the identity cache is mandatory, not a convenience: the
|
||||
// originator path aborts a LookupResponse it cannot verify. `connect`
|
||||
// seeds identically for the same reason.
|
||||
self.peer_aliases.insert(target, identity.short_npub());
|
||||
self.register_identity(target, identity.pubkey_full());
|
||||
|
||||
let wall_ms = Self::now_ms();
|
||||
let now_ms = crate::time::mono_ms();
|
||||
let tick_ms = self.config().node.tick_interval_secs * 1000;
|
||||
let ladder = self.config().node.lookup.attempt_timeouts_secs.clone();
|
||||
let budgets = Budgets::derive(tick_ms, &ladder);
|
||||
let budget_ms = budgets.total_ms();
|
||||
|
||||
let claimed = !self.probes.claims.insert(target);
|
||||
let preflight = Preflight {
|
||||
session_present: self.sessions.contains_key(&target),
|
||||
coords_cached: self.coord_cache.get(&target, wall_ms).is_some(),
|
||||
identity_cached: self.has_cached_identity(&target),
|
||||
target_claimed: claimed,
|
||||
};
|
||||
|
||||
let id = self.probes.next_id;
|
||||
self.probes.next_id += 1;
|
||||
let mut job = ProbeJob {
|
||||
id,
|
||||
target,
|
||||
pubkey: identity.pubkey_full(),
|
||||
npub: npub.to_string(),
|
||||
probe: Probe::new(now_ms, budgets, preflight),
|
||||
holds_claim: !claimed,
|
||||
owned_created_at: None,
|
||||
activity_at_establish: None,
|
||||
traffic_at_establish: None,
|
||||
lookup_outcome: None,
|
||||
lookup_fanout: None,
|
||||
lookup_attempts: None,
|
||||
session_error: None,
|
||||
teardown_refused: None,
|
||||
reap_at_ms: None,
|
||||
};
|
||||
let data = serde_json::json!({
|
||||
"probe_id": id,
|
||||
"npub": npub,
|
||||
"node_addr": target.to_string(),
|
||||
"display_name": self.peer_display_name(&target),
|
||||
"budget_ms": budget_ms,
|
||||
});
|
||||
|
||||
// Step it here rather than leaving it for the tick. Admission runs on
|
||||
// the rx loop already, so this is the same context the tick driver
|
||||
// uses, and without it the first stage does not begin until the next
|
||||
// tick fires -- a whole tick of a probe's budget spent before anything
|
||||
// is sent, and a client watching four pending stages for as long.
|
||||
self.drive_probe(&mut job, now_ms, wall_ms).await;
|
||||
self.probes.jobs.insert(id, job);
|
||||
|
||||
info!(npub = %npub, probe_id = id, budget_ms, "Probe started");
|
||||
Ok(data)
|
||||
}
|
||||
|
||||
/// Report a probe's progress. A terminal job is removed on the poll that
|
||||
/// observes it, so the report is delivered exactly once.
|
||||
pub(crate) fn api_probe_poll(&mut self, probe_id: u64) -> Result<serde_json::Value, String> {
|
||||
let now_ms = crate::time::mono_ms();
|
||||
let Some(job) = self.probes.jobs.get(&probe_id) else {
|
||||
return Err(format!("unknown probe id: {probe_id}"));
|
||||
};
|
||||
let done = job.probe.is_finished();
|
||||
let report = crate::control::probe::ProbeReport::build(self, job, now_ms);
|
||||
let data = serde_json::json!({
|
||||
"state": if done { "done" } else { "running" },
|
||||
"report": report,
|
||||
});
|
||||
if done {
|
||||
self.probes.jobs.remove(&probe_id);
|
||||
}
|
||||
Ok(data)
|
||||
}
|
||||
|
||||
/// Read a job without consuming it, for tests that assert on a probe's
|
||||
/// progress without the poll API's remove-on-done behaviour.
|
||||
#[cfg(test)]
|
||||
pub(crate) fn probe_job(&self, probe_id: u64) -> Option<&ProbeJob> {
|
||||
self.probes.jobs.get(&probe_id)
|
||||
}
|
||||
|
||||
/// Cancel a probe: run its terminal actions now, without the grace tick.
|
||||
pub(crate) async fn api_probe_cancel(
|
||||
&mut self,
|
||||
probe_id: u64,
|
||||
) -> Result<serde_json::Value, String> {
|
||||
let now_ms = crate::time::mono_ms();
|
||||
let Some(mut job) = self.probes.jobs.remove(&probe_id) else {
|
||||
return Err(format!("unknown probe id: {probe_id}"));
|
||||
};
|
||||
for action in job.probe.cancel(now_ms) {
|
||||
self.drive_probe_action(&mut job, action, now_ms).await;
|
||||
}
|
||||
self.probes.jobs.insert(probe_id, job);
|
||||
Ok(serde_json::json!({ "probe_id": probe_id, "cancelled": true }))
|
||||
}
|
||||
|
||||
// === Tick driver ===
|
||||
|
||||
/// Advance every in-flight probe by one observation, then reap the
|
||||
/// terminal jobs whose retention window has passed.
|
||||
pub(in crate::node) async fn poll_probes(&mut self) {
|
||||
if self.probes.jobs.is_empty() {
|
||||
return;
|
||||
}
|
||||
let now_ms = crate::time::mono_ms();
|
||||
let wall_ms = Self::now_ms();
|
||||
let ids: Vec<u64> = self.probes.jobs.keys().copied().collect();
|
||||
|
||||
for id in ids {
|
||||
// Take the job out of the registry so the drive path can hold
|
||||
// `&mut self` without an outstanding borrow of the map.
|
||||
let Some(mut job) = self.probes.jobs.remove(&id) else {
|
||||
continue;
|
||||
};
|
||||
if job.probe.is_finished() {
|
||||
if job.reap_at_ms.is_some_and(|t| now_ms >= t) {
|
||||
self.release_probe_claim(&mut job);
|
||||
continue;
|
||||
}
|
||||
self.probes.jobs.insert(id, job);
|
||||
continue;
|
||||
}
|
||||
|
||||
self.drive_probe(&mut job, now_ms, wall_ms).await;
|
||||
self.probes.jobs.insert(id, job);
|
||||
}
|
||||
}
|
||||
|
||||
/// Step one job and perform whatever it emits.
|
||||
///
|
||||
/// The caller owns the job for the duration, because driving an action
|
||||
/// needs `&mut self` and the job cannot stay borrowed out of the registry
|
||||
/// at the same time.
|
||||
async fn drive_probe(&mut self, job: &mut ProbeJob, now_ms: u64, wall_ms: u64) {
|
||||
let obs = self.observe_probe(job, now_ms, wall_ms);
|
||||
for action in job.probe.step(&obs) {
|
||||
self.drive_probe_action(job, action, now_ms).await;
|
||||
}
|
||||
}
|
||||
|
||||
/// Build one observation. Every read here is deliberately non-mutating:
|
||||
/// `coord_cache.get` rather than `get_and_touch`, `has_cached_identity`
|
||||
/// rather than `lookup_by_fips_prefix`.
|
||||
fn observe_probe(&self, job: &mut ProbeJob, now_ms: u64, wall_ms: u64) -> Observation {
|
||||
let target = job.target;
|
||||
let entry = self.sessions.get(&target);
|
||||
let session_present = entry.is_some();
|
||||
let session_established = entry.is_some_and(SessionEntry::is_established);
|
||||
let is_ours = self.probe_session_is_ours(job);
|
||||
|
||||
// Baseline for the adoption check: real user traffic arriving after
|
||||
// this point means the session is no longer the probe's to remove.
|
||||
if is_ours
|
||||
&& session_established
|
||||
&& job.activity_at_establish.is_none()
|
||||
&& let Some(entry) = entry
|
||||
{
|
||||
job.activity_at_establish = Some(entry.last_activity());
|
||||
job.traffic_at_establish = Some(entry.traffic_counters());
|
||||
}
|
||||
|
||||
let mmp = entry.and_then(SessionEntry::mmp);
|
||||
let counters = mmp.map_or(RttCounters::default(), |m| RttCounters {
|
||||
reports_seen: m.metrics.reports_seen(),
|
||||
samples: m.metrics.rtt_samples(),
|
||||
zero: m.metrics.rtt_zero(),
|
||||
arith_fail: m.metrics.rtt_arith_fail(),
|
||||
});
|
||||
let path_mtu = mmp
|
||||
.map(|m| m.path_mtu.last_observed_mtu())
|
||||
.filter(|mtu| *mtu != u16::MAX)
|
||||
.map(u32::from);
|
||||
|
||||
let mut lookup_pending = false;
|
||||
for (addr, pending) in self.pending_lookups_iter() {
|
||||
if *addr == target {
|
||||
lookup_pending = true;
|
||||
job.lookup_attempts = Some(pending.attempt);
|
||||
}
|
||||
}
|
||||
|
||||
let coords = self.coord_cache.get(&target, wall_ms).cloned();
|
||||
let path = coords
|
||||
.as_ref()
|
||||
.map(|c| describe_path(self.tree_state().my_coords(), c));
|
||||
let (next_hop, mut no_hop_reason) = self.preview_next_hop(&target, wall_ms);
|
||||
if path.as_ref().is_some_and(|p| !p.same_root) {
|
||||
no_hop_reason = Some(NoHopReason::DisjointTrees);
|
||||
}
|
||||
|
||||
Observation {
|
||||
now_ms,
|
||||
coords_cached: coords.is_some(),
|
||||
lookup_pending,
|
||||
lookup_outcome: job.lookup_outcome.take(),
|
||||
lookup_fanout: job.lookup_fanout.take(),
|
||||
path,
|
||||
next_hop,
|
||||
no_hop_reason,
|
||||
session_present,
|
||||
session_established,
|
||||
session_is_ours: is_ours,
|
||||
session_error: job.session_error.take(),
|
||||
target_is_direct_peer: self.peers.get(&target).is_some_and(|p| p.can_send()),
|
||||
counters,
|
||||
last_rtt_ms: mmp.and_then(|m| m.metrics.last_rtt_ms()),
|
||||
srtt_ms: mmp.and_then(|m| m.metrics.srtt_ms()),
|
||||
path_mtu,
|
||||
}
|
||||
}
|
||||
|
||||
async fn drive_probe_action(&mut self, job: &mut ProbeJob, action: ProbeAction, now_ms: u64) {
|
||||
match action {
|
||||
ProbeAction::InitiateLookup => {
|
||||
let outcome = self.maybe_initiate_lookup(&job.target).await;
|
||||
job.lookup_outcome = Some(outcome.kind());
|
||||
job.lookup_fanout = outcome.fanout();
|
||||
// Count the first attempt here rather than waiting to see it
|
||||
// in the pending table. A lookup answered inside one tick
|
||||
// never appears there, so the observation path alone reports
|
||||
// no attempts at all for the fastest case there is.
|
||||
if outcome.fanout().is_some_and(|f| f > 0) {
|
||||
job.lookup_attempts = Some(1);
|
||||
}
|
||||
}
|
||||
ProbeAction::OpenSession => {
|
||||
// Re-check at the moment of action: `api_probe_start` runs on
|
||||
// the control arm and can interleave with ticks, and an
|
||||
// inbound handshake can land between the observation and here.
|
||||
if self.sessions.contains_key(&job.target) {
|
||||
debug!(
|
||||
probe_id = job.id,
|
||||
"Probe declined to open an existing session"
|
||||
);
|
||||
return;
|
||||
}
|
||||
match self.initiate_session(job.target, job.pubkey).await {
|
||||
Ok(()) => {
|
||||
job.owned_created_at =
|
||||
self.sessions.get(&job.target).map(SessionEntry::created_at);
|
||||
}
|
||||
Err(e) => job.session_error = Some(e.to_string()),
|
||||
}
|
||||
}
|
||||
ProbeAction::SendWarmup => {
|
||||
if let Err(e) = self.send_coords_warmup(&job.target).await {
|
||||
debug!(probe_id = job.id, error = %e, "Probe warmup send failed");
|
||||
}
|
||||
}
|
||||
ProbeAction::TeardownSession => self.probe_teardown(job),
|
||||
ProbeAction::Finish => {
|
||||
job.reap_at_ms = Some(now_ms + REAP_MS);
|
||||
self.release_probe_claim(job);
|
||||
debug!(probe_id = job.id, "Probe finished");
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
/// The entry is ours only if it is byte-for-byte the one `initiate_session`
|
||||
/// inserted, we are still the initiator, and no application traffic has
|
||||
/// adopted it.
|
||||
fn probe_session_is_ours(&self, job: &ProbeJob) -> bool {
|
||||
self.probe_session_disowned(job).is_none()
|
||||
}
|
||||
|
||||
/// Why the live entry is not the probe's to remove, or `None` when it is.
|
||||
///
|
||||
/// The three causes are distinct and an operator debugging a surviving
|
||||
/// session needs to be told which one applied: a replacement entry, a
|
||||
/// peer-driven takeover and a session adopted by real traffic call for
|
||||
/// different next steps.
|
||||
fn probe_session_disowned(&self, job: &ProbeJob) -> Option<Disowned> {
|
||||
let Some(created) = job.owned_created_at else {
|
||||
return Some(Disowned::NeverOwned);
|
||||
};
|
||||
let Some(entry) = self.sessions.get(&job.target) else {
|
||||
return Some(Disowned::Gone);
|
||||
};
|
||||
if entry.created_at() != created || !entry.is_initiator() {
|
||||
return Some(Disowned::Replaced);
|
||||
}
|
||||
match (job.activity_at_establish, job.traffic_at_establish) {
|
||||
(Some(activity), Some(traffic))
|
||||
if entry.last_activity() != activity || entry.traffic_counters() != traffic =>
|
||||
{
|
||||
Some(Disowned::Adopted)
|
||||
}
|
||||
_ => None,
|
||||
}
|
||||
}
|
||||
|
||||
/// Remove a probe-created session, or decline and say why.
|
||||
///
|
||||
/// `pending_tun_packets` is deliberately untouched: the idle-purge path
|
||||
/// removes it because it runs after the idle timeout, but here the map can
|
||||
/// only hold real user packets queued while the session was `Initiating`.
|
||||
/// The probe queues none, so it has none to remove.
|
||||
fn probe_teardown(&mut self, job: &mut ProbeJob) {
|
||||
if let Some(cause) = self.probe_session_disowned(job) {
|
||||
// `Gone` leaves the refusal unset: there is no session left in
|
||||
// place, so reporting one would be its own falsehood.
|
||||
job.teardown_refused = match cause {
|
||||
Disowned::NeverOwned => Some(LeftIntact::Preexisting),
|
||||
Disowned::Replaced => Some(LeftIntact::Replaced),
|
||||
Disowned::Adopted => Some(LeftIntact::AdoptedByTraffic),
|
||||
Disowned::Gone => None,
|
||||
};
|
||||
return;
|
||||
}
|
||||
let name = self.peer_display_name(&job.target);
|
||||
if let Some(entry) = self.sessions.get(&job.target)
|
||||
&& let Some(mmp) = entry.mmp()
|
||||
{
|
||||
Self::log_session_mmp_teardown(&name, mmp);
|
||||
}
|
||||
self.sessions.remove(&job.target);
|
||||
debug!(probe_id = job.id, dest = %name, "Probe tore down the session it opened");
|
||||
}
|
||||
|
||||
fn release_probe_claim(&mut self, job: &mut ProbeJob) {
|
||||
if job.holds_claim {
|
||||
self.probes.claims.remove(&job.target);
|
||||
job.holds_claim = false;
|
||||
}
|
||||
}
|
||||
|
||||
/// Non-touching mirror of [`Node::find_next_hop`]: the same five steps in
|
||||
/// the same order, but reading the coord cache without the LRU touch and
|
||||
/// taking `wall_ms` as a parameter instead of reading the clock inline.
|
||||
/// A diagnostic must not perturb the state it reports on.
|
||||
pub(in crate::node) fn preview_next_hop(
|
||||
&self,
|
||||
dest: &NodeAddr,
|
||||
wall_ms: u64,
|
||||
) -> (Option<NextHopFacts>, Option<NoHopReason>) {
|
||||
if dest == self.node_addr() {
|
||||
return (None, Some(NoHopReason::Local));
|
||||
}
|
||||
if let Some(peer) = self.peers.get(dest)
|
||||
&& peer.can_send()
|
||||
{
|
||||
return (
|
||||
Some(NextHopFacts {
|
||||
node_addr: *dest,
|
||||
class: RouteClass::DirectPeer,
|
||||
direct_peer: true,
|
||||
leaves_tree_walk: false,
|
||||
}),
|
||||
None,
|
||||
);
|
||||
}
|
||||
let Some(dest_coords) = self.coord_cache.get(dest, wall_ms).cloned() else {
|
||||
return (None, Some(NoHopReason::NoCoords));
|
||||
};
|
||||
|
||||
let selected = {
|
||||
let view = crate::node::NodeRoutingView {
|
||||
coord_cache: &self.coord_cache,
|
||||
peers: &self.peers,
|
||||
tree_state: &self.tree_state,
|
||||
congested: false,
|
||||
};
|
||||
routing::select_best_candidate(&view, dest, &dest_coords, self.tree_state.my_coords())
|
||||
}
|
||||
.or_else(|| {
|
||||
self.tree_state
|
||||
.find_next_hop(&dest_coords, &std::collections::BTreeSet::new())
|
||||
});
|
||||
|
||||
let Some(hop) = selected else {
|
||||
return (None, Some(NoHopReason::NoCloserPeer));
|
||||
};
|
||||
if !self.peers.get(&hop).is_some_and(|p| p.can_send()) {
|
||||
return (None, Some(NoHopReason::HopNotSendReady));
|
||||
}
|
||||
let class = self.classify_forward(dest, &hop);
|
||||
(
|
||||
Some(NextHopFacts {
|
||||
node_addr: hop,
|
||||
class,
|
||||
direct_peer: hop == *dest,
|
||||
leaves_tree_walk: matches!(
|
||||
class,
|
||||
RouteClass::TreeDownCross
|
||||
| RouteClass::CrosslinkDescend
|
||||
| RouteClass::CrosslinkAscend
|
||||
),
|
||||
}),
|
||||
None,
|
||||
)
|
||||
}
|
||||
}
|
||||
@@ -2633,7 +2633,10 @@ impl Node {
|
||||
/// coordinates via `try_warm_coord_cache()` (same as CP-flagged data
|
||||
/// packets). The encrypted inner payload is the 6-byte inner header
|
||||
/// with no application data.
|
||||
async fn send_coords_warmup(&mut self, dest_addr: &NodeAddr) -> Result<(), NodeError> {
|
||||
pub(in crate::node) async fn send_coords_warmup(
|
||||
&mut self,
|
||||
dest_addr: &NodeAddr,
|
||||
) -> Result<(), NodeError> {
|
||||
let now_ms = Self::now_ms();
|
||||
|
||||
let my_coords = self.tree_state.my_coords().clone();
|
||||
|
||||
@@ -22,6 +22,8 @@ mod rate_limit;
|
||||
pub(crate) mod reject;
|
||||
mod reloadable;
|
||||
pub(crate) mod session;
|
||||
pub(crate) use handlers::probe::ProbeJob;
|
||||
|
||||
pub(crate) mod stats;
|
||||
pub(crate) mod stats_history;
|
||||
#[cfg(test)]
|
||||
@@ -460,6 +462,11 @@ pub struct Node {
|
||||
/// lookups, originator-side backoff, and transit-side forward limiter.
|
||||
lookup: Lookup,
|
||||
|
||||
// === Diagnostics ===
|
||||
/// In-flight `probe` jobs plus their per-target ownership claims. Driven
|
||||
/// once per tick by `poll_probes`; see `node::handlers::probe`.
|
||||
probes: handlers::probe::ProbeRegistry,
|
||||
|
||||
// === Counters ===
|
||||
/// Next link ID to allocate.
|
||||
next_link_id: u64,
|
||||
@@ -782,6 +789,7 @@ impl Node {
|
||||
coords_response_rate_limiter: RoutingErrorRateLimiter::with_interval_ms(
|
||||
coords_response_interval_ms,
|
||||
),
|
||||
probes: handlers::probe::ProbeRegistry::new(),
|
||||
lookup: Lookup::new(
|
||||
LookupBackoff::with_params(backoff_base_secs, backoff_max_secs),
|
||||
LookupForwardRateLimiter::with_interval_ms(forward_min_interval_secs * 1000),
|
||||
@@ -929,6 +937,7 @@ impl Node {
|
||||
coords_response_rate_limiter: RoutingErrorRateLimiter::with_interval_ms(
|
||||
coords_response_interval_ms,
|
||||
),
|
||||
probes: handlers::probe::ProbeRegistry::new(),
|
||||
lookup: Lookup::new(LookupBackoff::new(), LookupForwardRateLimiter::new()),
|
||||
peering: peering::reconcile::Peering::new(),
|
||||
last_parent_reeval: None,
|
||||
@@ -2874,6 +2883,16 @@ impl Node {
|
||||
self.pending_tun_packets.len()
|
||||
}
|
||||
|
||||
/// Queue a TUN packet for a destination directly (for tests that need a
|
||||
/// pending queue without driving the whole outbound path).
|
||||
#[cfg(test)]
|
||||
pub(crate) fn queue_pending_tun_packet_for_test(&mut self, dest: NodeAddr, packet: Vec<u8>) {
|
||||
self.pending_tun_packets
|
||||
.entry(dest)
|
||||
.or_default()
|
||||
.push_back(packet);
|
||||
}
|
||||
|
||||
/// Total TUN packets queued across all destinations.
|
||||
pub fn pending_tun_total_packets(&self) -> usize {
|
||||
self.pending_tun_packets.values().map(|q| q.len()).sum()
|
||||
|
||||
@@ -284,8 +284,8 @@ impl SessionEntry {
|
||||
self.state.as_ref().is_some_and(|s| s.is_awaiting_msg3())
|
||||
}
|
||||
|
||||
/// Get creation time.
|
||||
#[cfg(test)]
|
||||
/// Get creation time. Used as the probe's session-identity check: an
|
||||
/// entry with a different creation stamp is a replacement, not ours.
|
||||
pub(crate) fn created_at(&self) -> u64 {
|
||||
self.created_at
|
||||
}
|
||||
|
||||
@@ -11,7 +11,8 @@ use crate::proto::link::SessionDatagram;
|
||||
use crate::proto::stp::TreeCoordinate;
|
||||
use crate::proto::stp::encode_coords;
|
||||
use spanning_tree::{
|
||||
TestNode, cleanup_nodes, process_available_packets, run_tree_test, verify_tree_convergence,
|
||||
TestNode, cleanup_nodes, populate_all_coord_caches, process_available_packets, run_tree_test,
|
||||
verify_tree_convergence,
|
||||
};
|
||||
|
||||
// ============================================================================
|
||||
@@ -475,35 +476,6 @@ async fn test_coord_cache_warming_ttl_zero_transit_drop() {
|
||||
// Integration Tests
|
||||
// ============================================================================
|
||||
|
||||
/// Helper: populate all coordinate caches across a set of test nodes.
|
||||
fn populate_all_coord_caches(nodes: &mut [TestNode]) {
|
||||
let now_ms = std::time::SystemTime::now()
|
||||
.duration_since(std::time::UNIX_EPOCH)
|
||||
.unwrap()
|
||||
.as_millis() as u64;
|
||||
|
||||
// Collect all coords first to avoid borrow conflicts
|
||||
let all_coords: Vec<(NodeAddr, TreeCoordinate)> = nodes
|
||||
.iter()
|
||||
.map(|tn| {
|
||||
(
|
||||
*tn.node.node_addr(),
|
||||
tn.node.tree_state().my_coords().clone(),
|
||||
)
|
||||
})
|
||||
.collect();
|
||||
|
||||
for tn in nodes.iter_mut() {
|
||||
for (addr, coords) in &all_coords {
|
||||
if addr != tn.node.node_addr() {
|
||||
tn.node
|
||||
.coord_cache_mut()
|
||||
.insert(*addr, coords.clone(), now_ms);
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
#[tokio::test]
|
||||
async fn test_forwarding_single_hop() {
|
||||
// 3-node chain: 0 -- 1 -- 2
|
||||
|
||||
@@ -20,6 +20,7 @@ mod forwarding;
|
||||
mod handshake;
|
||||
mod heartbeat;
|
||||
mod mmp_chartests;
|
||||
mod probe;
|
||||
mod routing;
|
||||
mod session;
|
||||
mod spanning_tree;
|
||||
|
||||
@@ -0,0 +1,504 @@
|
||||
//! Integration tests for the `probe` diagnostic.
|
||||
//!
|
||||
//! Each runs a two-node topology, starts a probe on node 0 toward node 1, then
|
||||
//! alternates packet delivery with explicit `poll_probes` calls in a bounded
|
||||
//! loop — the harness has no rx-loop tick, so the driver is invoked by hand.
|
||||
//!
|
||||
//! Stated coverage gap, not discharged: a same-process harness has a
|
||||
//! sub-millisecond path, so `MmpMetrics`'s `rtt_ms > 0` guard can reject every
|
||||
//! sample and the rtt stage then reports `sub_millisecond` rather than `ok`.
|
||||
//! A real round-trip measurement is exercisable only on a live multi-host
|
||||
//! mesh, and the `rtt: ok` path stays unexercised here.
|
||||
|
||||
use super::*;
|
||||
use crate::proto::probe::StageVerdict;
|
||||
use spanning_tree::{
|
||||
TestNode, cleanup_nodes, populate_all_coord_caches, process_available_packets, run_tree_test,
|
||||
verify_tree_convergence,
|
||||
};
|
||||
|
||||
/// Two peered nodes with coordinates seeded on both sides.
|
||||
async fn two_peered_nodes() -> Vec<TestNode> {
|
||||
let mut nodes = run_tree_test(2, &[(0, 1)], false).await;
|
||||
verify_tree_convergence(&nodes);
|
||||
populate_all_coord_caches(&mut nodes);
|
||||
nodes
|
||||
}
|
||||
|
||||
/// Deliver packets, let both nodes emit their MMP reports, and advance node
|
||||
/// 0's probes. One call is one simulated tick.
|
||||
async fn pump(nodes: &mut [TestNode]) {
|
||||
tokio::time::sleep(Duration::from_millis(10)).await;
|
||||
process_available_packets(nodes).await;
|
||||
for tn in nodes.iter_mut() {
|
||||
tn.node.check_session_mmp_reports().await;
|
||||
}
|
||||
nodes[0].node.poll_probes().await;
|
||||
}
|
||||
|
||||
/// Pump until the probe reaches a terminal state, or give up. Returns whether
|
||||
/// it finished, so a caller can assert rather than hang.
|
||||
async fn pump_until_done(nodes: &mut [TestNode], id: u64, rounds: usize) -> bool {
|
||||
for _ in 0..rounds {
|
||||
pump(nodes).await;
|
||||
if nodes[0]
|
||||
.node
|
||||
.probe_job(id)
|
||||
.is_none_or(|job| job.probe().is_finished())
|
||||
{
|
||||
return true;
|
||||
}
|
||||
}
|
||||
false
|
||||
}
|
||||
|
||||
fn npub_of(nodes: &[TestNode], idx: usize) -> String {
|
||||
nodes[idx].node.identity().npub()
|
||||
}
|
||||
|
||||
async fn start_probe(nodes: &mut [TestNode], target: usize) -> u64 {
|
||||
let npub = npub_of(nodes, target);
|
||||
let data = nodes[0]
|
||||
.node
|
||||
.api_probe_start(&npub)
|
||||
.await
|
||||
.expect("probe admitted");
|
||||
data["probe_id"].as_u64().expect("probe_id is a number")
|
||||
}
|
||||
|
||||
#[tokio::test]
|
||||
async fn a_probe_is_stepped_at_admission_rather_than_waiting_for_the_next_tick() {
|
||||
// Nothing here pumps: the assertions are about the state `api_probe_start`
|
||||
// leaves behind. Left to the tick driver, the first stage does not begin
|
||||
// until the next tick fires, which spends a whole tick period of the
|
||||
// probe's budget before a single message is sent.
|
||||
let mut nodes = two_peered_nodes().await;
|
||||
let id = start_probe(&mut nodes, 1).await;
|
||||
|
||||
let snap = nodes[0].node.probe_job(id).unwrap().probe().snapshot();
|
||||
assert_eq!(
|
||||
snap.bloom.verdict,
|
||||
StageVerdict::Skipped,
|
||||
"the bloom stage should have run and skipped a direct peer: {:?}",
|
||||
snap.bloom
|
||||
);
|
||||
assert_eq!(
|
||||
snap.discovery.verdict,
|
||||
StageVerdict::Skipped,
|
||||
"discovery should have been skipped with it: {:?}",
|
||||
snap.discovery
|
||||
);
|
||||
assert_ne!(
|
||||
snap.path.verdict,
|
||||
StageVerdict::Pending,
|
||||
"path should have been computed in the same step: {:?}",
|
||||
snap.path
|
||||
);
|
||||
assert_eq!(
|
||||
snap.session.verdict,
|
||||
StageVerdict::Running,
|
||||
"the session stage should be in flight already: {:?}",
|
||||
snap.session
|
||||
);
|
||||
|
||||
cleanup_nodes(&mut nodes).await;
|
||||
}
|
||||
|
||||
#[tokio::test]
|
||||
async fn probe_reaches_session_stage_against_direct_peer() {
|
||||
let mut nodes = two_peered_nodes().await;
|
||||
let node1_addr = *nodes[1].node.node_addr();
|
||||
let node0_addr = *nodes[0].node.node_addr();
|
||||
let id = start_probe(&mut nodes, 1).await;
|
||||
|
||||
for _ in 0..20 {
|
||||
pump(&mut nodes).await;
|
||||
let snap = nodes[0].node.probe_job(id).unwrap().probe().snapshot();
|
||||
if snap.session.verdict == StageVerdict::Ok {
|
||||
// The initiator flips to Established on sending msg3; the far end
|
||||
// needs one more delivery pass to process it.
|
||||
pump(&mut nodes).await;
|
||||
pump(&mut nodes).await;
|
||||
break;
|
||||
}
|
||||
}
|
||||
|
||||
let snap = nodes[0].node.probe_job(id).unwrap().probe().snapshot();
|
||||
assert_eq!(
|
||||
snap.session.verdict,
|
||||
StageVerdict::Ok,
|
||||
"session stage: {:?}",
|
||||
snap.session
|
||||
);
|
||||
assert!(nodes[0].node.get_session(&node1_addr).is_some());
|
||||
assert!(
|
||||
nodes[1]
|
||||
.node
|
||||
.get_session(&node0_addr)
|
||||
.is_some_and(|e| e.is_established()),
|
||||
"the far end must hold an established session too"
|
||||
);
|
||||
|
||||
cleanup_nodes(&mut nodes).await;
|
||||
}
|
||||
|
||||
#[tokio::test]
|
||||
async fn probe_tears_down_the_session_it_opened() {
|
||||
let mut nodes = two_peered_nodes().await;
|
||||
let node1_addr = *nodes[1].node.node_addr();
|
||||
let id = start_probe(&mut nodes, 1).await;
|
||||
|
||||
assert!(
|
||||
pump_until_done(&mut nodes, id, 600).await,
|
||||
"probe must reach a terminal state"
|
||||
);
|
||||
let snap = nodes[0].node.probe_job(id).unwrap().probe().snapshot();
|
||||
assert!(snap.torn_down, "cleanup: {:?}", snap.left_intact);
|
||||
assert!(
|
||||
nodes[0].node.get_session(&node1_addr).is_none(),
|
||||
"the probe-created session must be gone"
|
||||
);
|
||||
|
||||
// The rtt stage must terminate rather than hang, either with a real
|
||||
// measurement or with one of the four discriminated failure reasons.
|
||||
assert!(
|
||||
matches!(snap.rtt.verdict, StageVerdict::Ok | StageVerdict::Failed),
|
||||
"rtt stage: {:?}",
|
||||
snap.rtt
|
||||
);
|
||||
|
||||
// The projection is what a script actually reads, and the tests above all
|
||||
// bypass it by reading the core snapshot directly. Assert on the published
|
||||
// JSON, including the poll API's remove-on-done behaviour.
|
||||
let data = nodes[0].node.api_probe_poll(id).expect("poll accepted");
|
||||
assert_eq!(data["state"], "done");
|
||||
let report = &data["report"];
|
||||
assert_eq!(report["probe_id"].as_u64(), Some(id));
|
||||
assert_eq!(report["cleanup"]["session_created_and_torn_down"], true);
|
||||
assert_eq!(report["cleanup"]["session_left_intact"], false);
|
||||
assert_eq!(
|
||||
report["cleanup"]["left_intact_reason"],
|
||||
serde_json::Value::Null
|
||||
);
|
||||
assert_eq!(report["session"]["preexisting"], false);
|
||||
assert_eq!(report["session"]["established"], true);
|
||||
assert_eq!(report["path"]["computed_locally"], true);
|
||||
assert_eq!(report["path"]["observed"], false);
|
||||
assert!(
|
||||
nodes[0].node.api_probe_poll(id).is_err(),
|
||||
"a terminal report is delivered exactly once"
|
||||
);
|
||||
|
||||
cleanup_nodes(&mut nodes).await;
|
||||
}
|
||||
|
||||
#[tokio::test]
|
||||
async fn probe_leaves_a_preexisting_session_intact() {
|
||||
let mut nodes = two_peered_nodes().await;
|
||||
let node0_addr = *nodes[0].node.node_addr();
|
||||
let node1_addr = *nodes[1].node.node_addr();
|
||||
let node1_pubkey = nodes[1].node.identity().pubkey_full();
|
||||
|
||||
nodes[0]
|
||||
.node
|
||||
.initiate_session(node1_addr, node1_pubkey)
|
||||
.await
|
||||
.expect("initiate_session failed");
|
||||
for _ in 0..6 {
|
||||
tokio::time::sleep(Duration::from_millis(10)).await;
|
||||
process_available_packets(&mut nodes).await;
|
||||
}
|
||||
assert!(
|
||||
nodes[0]
|
||||
.node
|
||||
.get_session(&node1_addr)
|
||||
.is_some_and(|e| e.is_established())
|
||||
);
|
||||
|
||||
let id = start_probe(&mut nodes, 1).await;
|
||||
assert!(pump_until_done(&mut nodes, id, 600).await);
|
||||
|
||||
let snap = nodes[0].node.probe_job(id).unwrap().probe().snapshot();
|
||||
assert!(!snap.torn_down, "must not remove a session it did not open");
|
||||
assert!(snap.session_preexisting);
|
||||
assert_eq!(
|
||||
snap.left_intact.map(|r| r.name()),
|
||||
Some("preexisting"),
|
||||
"session stage: {:?}",
|
||||
snap.session
|
||||
);
|
||||
assert!(
|
||||
nodes[0]
|
||||
.node
|
||||
.get_session(&node1_addr)
|
||||
.is_some_and(|e| e.is_established()),
|
||||
"our end of the pre-existing session must survive"
|
||||
);
|
||||
assert!(
|
||||
nodes[1]
|
||||
.node
|
||||
.get_session(&node0_addr)
|
||||
.is_some_and(|e| e.is_established()),
|
||||
"the far end must survive too"
|
||||
);
|
||||
|
||||
let data = nodes[0].node.api_probe_poll(id).expect("poll accepted");
|
||||
assert_eq!(data["state"], "done");
|
||||
let report = &data["report"];
|
||||
assert_eq!(report["cleanup"]["session_created_and_torn_down"], false);
|
||||
assert_eq!(report["cleanup"]["session_left_intact"], true);
|
||||
assert_eq!(report["cleanup"]["left_intact_reason"], "preexisting");
|
||||
assert_eq!(report["session"]["preexisting"], true);
|
||||
|
||||
cleanup_nodes(&mut nodes).await;
|
||||
}
|
||||
|
||||
#[tokio::test]
|
||||
async fn probe_declines_to_tear_down_a_session_adopted_by_traffic() {
|
||||
// The drive-time half of the ownership guard. Cancel drives the terminal
|
||||
// actions with no fresh observation, so the core still believes it owns the
|
||||
// session and the refusal has to come from the driver's own identity check.
|
||||
let mut nodes = two_peered_nodes().await;
|
||||
let node1_addr = *nodes[1].node.node_addr();
|
||||
let id = start_probe(&mut nodes, 1).await;
|
||||
|
||||
for _ in 0..20 {
|
||||
pump(&mut nodes).await;
|
||||
let snap = nodes[0].node.probe_job(id).unwrap().probe().snapshot();
|
||||
if snap.session.verdict == StageVerdict::Ok {
|
||||
break;
|
||||
}
|
||||
}
|
||||
assert!(nodes[0].node.get_session(&node1_addr).is_some());
|
||||
|
||||
// Real application traffic moves on the session after the probe's last
|
||||
// observation: it is no longer the probe's to remove.
|
||||
nodes[0]
|
||||
.node
|
||||
.get_session_mut(&node1_addr)
|
||||
.expect("probe session")
|
||||
.record_recv(512);
|
||||
|
||||
nodes[0]
|
||||
.node
|
||||
.api_probe_cancel(id)
|
||||
.await
|
||||
.expect("cancel accepted");
|
||||
|
||||
assert!(
|
||||
nodes[0]
|
||||
.node
|
||||
.get_session(&node1_addr)
|
||||
.is_some_and(|e| e.is_established()),
|
||||
"an adopted session must survive the probe's teardown"
|
||||
);
|
||||
let data = nodes[0].node.api_probe_poll(id).expect("poll accepted");
|
||||
let cleanup = &data["report"]["cleanup"];
|
||||
assert_eq!(cleanup["session_created_and_torn_down"], false);
|
||||
assert_eq!(cleanup["left_intact_reason"], "adopted_by_traffic");
|
||||
|
||||
cleanup_nodes(&mut nodes).await;
|
||||
}
|
||||
|
||||
#[tokio::test]
|
||||
async fn probe_names_a_tree_next_hop_for_a_target_two_hops_away() {
|
||||
// The two-node topologies above always take the direct-peer short-circuit,
|
||||
// so everything past it — candidate selection, the tree-state fallback and
|
||||
// the forward classification — goes unrun. A chain exercises it, and a
|
||||
// multi-hop target is the case the feature exists to diagnose.
|
||||
let mut nodes = run_tree_test(3, &[(0, 1), (1, 2)], false).await;
|
||||
verify_tree_convergence(&nodes);
|
||||
populate_all_coord_caches(&mut nodes);
|
||||
|
||||
let node1_addr = *nodes[1].node.node_addr();
|
||||
let node2_addr = *nodes[2].node.node_addr();
|
||||
let wall_ms = Node::now_ms();
|
||||
let (hop, reason) = nodes[0].node.preview_next_hop(&node2_addr, wall_ms);
|
||||
|
||||
let hop = hop.unwrap_or_else(|| panic!("no next hop toward a two-hop target: {reason:?}"));
|
||||
assert_eq!(reason, None);
|
||||
assert_eq!(hop.node_addr, node1_addr, "the relay is the first hop");
|
||||
assert!(!hop.direct_peer, "a two-hop target is not a direct peer");
|
||||
|
||||
cleanup_nodes(&mut nodes).await;
|
||||
}
|
||||
|
||||
#[tokio::test]
|
||||
async fn preview_next_hop_reports_why_it_could_name_no_hop() {
|
||||
let mut nodes = two_peered_nodes().await;
|
||||
let node0_addr = *nodes[0].node.node_addr();
|
||||
let wall_ms = Node::now_ms();
|
||||
|
||||
// The local address routes nowhere.
|
||||
let (hop, reason) = nodes[0].node.preview_next_hop(&node0_addr, wall_ms);
|
||||
assert!(hop.is_none());
|
||||
assert_eq!(reason.map(|r| r.name()), Some("local"));
|
||||
|
||||
// An address that is neither a peer nor in the coord cache.
|
||||
let stranger = crate::NodeAddr::from_bytes([0x5a; 16]);
|
||||
let (hop, reason) = nodes[0].node.preview_next_hop(&stranger, wall_ms);
|
||||
assert!(hop.is_none());
|
||||
assert_eq!(reason.map(|r| r.name()), Some("no_coords"));
|
||||
|
||||
// Coordinates under a root nobody here can reach: no peer is closer.
|
||||
let alien_root = crate::NodeAddr::from_bytes([0x77; 16]);
|
||||
let coords = crate::proto::stp::TreeCoordinate::from_addrs(vec![stranger, alien_root])
|
||||
.expect("non-empty coordinate");
|
||||
nodes[0]
|
||||
.node
|
||||
.coord_cache_mut()
|
||||
.insert(stranger, coords, wall_ms);
|
||||
let (hop, reason) = nodes[0].node.preview_next_hop(&stranger, wall_ms);
|
||||
assert!(hop.is_none(), "no peer makes progress toward another tree");
|
||||
assert_eq!(reason.map(|r| r.name()), Some("no_closer_peer"));
|
||||
|
||||
cleanup_nodes(&mut nodes).await;
|
||||
}
|
||||
|
||||
#[tokio::test]
|
||||
async fn probe_does_not_own_a_session_replaced_after_start() {
|
||||
let mut nodes = two_peered_nodes().await;
|
||||
let node0_addr = *nodes[0].node.node_addr();
|
||||
let node1_addr = *nodes[1].node.node_addr();
|
||||
let node0_pubkey = nodes[0].node.identity().pubkey_full();
|
||||
|
||||
let id = start_probe(&mut nodes, 1).await;
|
||||
for _ in 0..20 {
|
||||
pump(&mut nodes).await;
|
||||
let snap = nodes[0].node.probe_job(id).unwrap().probe().snapshot();
|
||||
if snap.session.verdict == StageVerdict::Ok {
|
||||
break;
|
||||
}
|
||||
}
|
||||
assert!(nodes[0].node.get_session(&node1_addr).is_some());
|
||||
|
||||
// The probe opens its session at admission, so the entry it must not own
|
||||
// is one that arrives in place of its own. Dropping both halves and
|
||||
// letting node 1 dial produces exactly what a peer's inbound handshake
|
||||
// does under simultaneous initiation — an established entry node 0 did
|
||||
// not initiate — without depending on which side wins a tie-break.
|
||||
nodes[0].node.remove_session(&node1_addr);
|
||||
nodes[1].node.remove_session(&node0_addr);
|
||||
nodes[1]
|
||||
.node
|
||||
.initiate_session(node0_addr, node0_pubkey)
|
||||
.await
|
||||
.expect("initiate_session failed");
|
||||
for _ in 0..6 {
|
||||
tokio::time::sleep(Duration::from_millis(10)).await;
|
||||
process_available_packets(&mut nodes).await;
|
||||
}
|
||||
assert!(
|
||||
nodes[0]
|
||||
.node
|
||||
.get_session(&node1_addr)
|
||||
.is_some_and(|e| !e.is_initiator()),
|
||||
"the replacement must be the peer's entry, not ours"
|
||||
);
|
||||
|
||||
nodes[0]
|
||||
.node
|
||||
.api_probe_cancel(id)
|
||||
.await
|
||||
.expect("cancel accepted");
|
||||
|
||||
let data = nodes[0].node.api_probe_poll(id).expect("poll accepted");
|
||||
let cleanup = &data["report"]["cleanup"];
|
||||
assert_eq!(cleanup["session_created_and_torn_down"], false);
|
||||
assert_eq!(cleanup["left_intact_reason"], "replaced");
|
||||
assert!(
|
||||
nodes[0]
|
||||
.node
|
||||
.get_session(&node1_addr)
|
||||
.is_some_and(|e| e.is_established()),
|
||||
"the peer-driven session must survive the probe"
|
||||
);
|
||||
assert!(
|
||||
nodes[1]
|
||||
.node
|
||||
.get_session(&node0_addr)
|
||||
.is_some_and(|e| e.is_established()),
|
||||
"node 1's own session must survive"
|
||||
);
|
||||
|
||||
cleanup_nodes(&mut nodes).await;
|
||||
}
|
||||
|
||||
#[tokio::test]
|
||||
async fn probe_does_not_discard_queued_tun_packets() {
|
||||
let mut nodes = two_peered_nodes().await;
|
||||
let node1_addr = *nodes[1].node.node_addr();
|
||||
let id = start_probe(&mut nodes, 1).await;
|
||||
|
||||
// Establish the probe's session first. The normal establish and inbound
|
||||
// paths legitimately flush anything queued for a destination, so the
|
||||
// packet is queued after the last delivery pass and the teardown is driven
|
||||
// by cancelling — the queue is then observed across the teardown alone.
|
||||
for _ in 0..20 {
|
||||
pump(&mut nodes).await;
|
||||
let snap = nodes[0].node.probe_job(id).unwrap().probe().snapshot();
|
||||
if snap.session.verdict == StageVerdict::Ok {
|
||||
break;
|
||||
}
|
||||
}
|
||||
nodes[0]
|
||||
.node
|
||||
.queue_pending_tun_packet_for_test(node1_addr, vec![0u8; 64]);
|
||||
assert_eq!(nodes[0].node.pending_tun_total_packets(), 1);
|
||||
|
||||
nodes[0]
|
||||
.node
|
||||
.api_probe_cancel(id)
|
||||
.await
|
||||
.expect("cancel accepted");
|
||||
assert!(
|
||||
nodes[0].node.get_session(&node1_addr).is_none(),
|
||||
"cancel must still tear the probe's own session down"
|
||||
);
|
||||
assert_eq!(
|
||||
nodes[0].node.pending_tun_total_packets(),
|
||||
1,
|
||||
"the probe must not touch queued user packets"
|
||||
);
|
||||
|
||||
cleanup_nodes(&mut nodes).await;
|
||||
}
|
||||
|
||||
#[tokio::test]
|
||||
async fn probe_terminates_without_a_poller() {
|
||||
let mut nodes = two_peered_nodes().await;
|
||||
let node1_addr = *nodes[1].node.node_addr();
|
||||
let id = start_probe(&mut nodes, 1).await;
|
||||
|
||||
// Never call the poll API: the job must still be driven to terminal on the
|
||||
// tick, which is what makes the feature safe when the client goes away.
|
||||
assert!(pump_until_done(&mut nodes, id, 600).await);
|
||||
assert!(nodes[0].node.probe_job(id).unwrap().probe().is_finished());
|
||||
assert!(nodes[0].node.get_session(&node1_addr).is_none());
|
||||
|
||||
cleanup_nodes(&mut nodes).await;
|
||||
}
|
||||
|
||||
#[tokio::test]
|
||||
async fn concurrent_probes_to_one_target_do_not_both_own_it() {
|
||||
let mut nodes = two_peered_nodes().await;
|
||||
let node1_addr = *nodes[1].node.node_addr();
|
||||
let first = start_probe(&mut nodes, 1).await;
|
||||
let second = start_probe(&mut nodes, 1).await;
|
||||
assert_ne!(first, second);
|
||||
|
||||
assert!(pump_until_done(&mut nodes, first, 600).await);
|
||||
assert!(pump_until_done(&mut nodes, second, 600).await);
|
||||
|
||||
let a = nodes[0].node.probe_job(first).unwrap().probe().snapshot();
|
||||
let b = nodes[0].node.probe_job(second).unwrap().probe().snapshot();
|
||||
assert_eq!(
|
||||
usize::from(a.torn_down) + usize::from(b.torn_down),
|
||||
1,
|
||||
"exactly one job may own and tear down the session"
|
||||
);
|
||||
assert!(a.session_preexisting || b.session_preexisting);
|
||||
assert!(nodes[0].node.get_session(&node1_addr).is_none());
|
||||
|
||||
cleanup_nodes(&mut nodes).await;
|
||||
}
|
||||
@@ -4,8 +4,9 @@ use super::*;
|
||||
use crate::node::session::EndToEndState;
|
||||
use crate::node::tests::spanning_tree::{
|
||||
TestNode, cleanup_nodes, drain_all_packets, generate_random_edges, initiate_handshake,
|
||||
lock_large_network_test, make_test_node_with_config, process_available_packets, run_tree_test,
|
||||
run_tree_test_with_configs, run_tree_test_with_mtus, verify_tree_convergence,
|
||||
lock_large_network_test, make_test_node_with_config, populate_all_coord_caches,
|
||||
process_available_packets, run_tree_test, run_tree_test_with_configs, run_tree_test_with_mtus,
|
||||
verify_tree_convergence,
|
||||
};
|
||||
use crate::proto::fsp::{SessionAck, SessionMsg3};
|
||||
use crate::proto::link::SessionDatagram;
|
||||
@@ -19,37 +20,6 @@ fn stub_link_peer() -> NodeAddr {
|
||||
make_node_addr(0xFE)
|
||||
}
|
||||
|
||||
/// Populate all nodes' coordinate caches with each other's coords.
|
||||
///
|
||||
/// This enables routing between non-adjacent nodes (bloom filter + tree
|
||||
/// routing both require cached destination coordinates).
|
||||
fn populate_all_coord_caches(nodes: &mut [TestNode]) {
|
||||
let now_ms = std::time::SystemTime::now()
|
||||
.duration_since(std::time::UNIX_EPOCH)
|
||||
.unwrap()
|
||||
.as_millis() as u64;
|
||||
|
||||
let all_coords: Vec<(NodeAddr, crate::proto::stp::TreeCoordinate)> = nodes
|
||||
.iter()
|
||||
.map(|tn| {
|
||||
(
|
||||
*tn.node.node_addr(),
|
||||
tn.node.tree_state().my_coords().clone(),
|
||||
)
|
||||
})
|
||||
.collect();
|
||||
|
||||
for tn in nodes.iter_mut() {
|
||||
for (addr, coords) in &all_coords {
|
||||
if addr != tn.node.node_addr() {
|
||||
tn.node
|
||||
.coord_cache_mut()
|
||||
.insert(*addr, coords.clone(), now_ms);
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
// ============================================================================
|
||||
// Unit tests: SessionEntry data structure
|
||||
// ============================================================================
|
||||
|
||||
@@ -850,6 +850,39 @@ async fn converge_nodes(mut nodes: Vec<TestNode>, edges: &[(usize, usize)]) -> V
|
||||
nodes
|
||||
}
|
||||
|
||||
/// Populate every node's coordinate cache with every other node's coords.
|
||||
///
|
||||
/// Routing between non-adjacent nodes needs cached destination coordinates on
|
||||
/// both the bloom-filter and greedy-tree paths, so tests that skip discovery
|
||||
/// seed them here.
|
||||
pub(super) fn populate_all_coord_caches(nodes: &mut [TestNode]) {
|
||||
let now_ms = std::time::SystemTime::now()
|
||||
.duration_since(std::time::UNIX_EPOCH)
|
||||
.unwrap()
|
||||
.as_millis() as u64;
|
||||
|
||||
// Collect all coords first to avoid borrow conflicts
|
||||
let all_coords: Vec<(NodeAddr, crate::proto::stp::TreeCoordinate)> = nodes
|
||||
.iter()
|
||||
.map(|tn| {
|
||||
(
|
||||
*tn.node.node_addr(),
|
||||
tn.node.tree_state().my_coords().clone(),
|
||||
)
|
||||
})
|
||||
.collect();
|
||||
|
||||
for tn in nodes.iter_mut() {
|
||||
for (addr, coords) in &all_coords {
|
||||
if addr != tn.node.node_addr() {
|
||||
tn.node
|
||||
.coord_cache_mut()
|
||||
.insert(*addr, coords.clone(), now_ms);
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
/// Clean up transports for all test nodes.
|
||||
pub(super) async fn cleanup_nodes(nodes: &mut [TestNode]) {
|
||||
for tn in nodes.iter_mut() {
|
||||
|
||||
@@ -87,6 +87,24 @@ pub struct MmpMetrics {
|
||||
prev_reverse_highest: u64,
|
||||
/// Whether we have a previous reverse-side snapshot for delta computation.
|
||||
has_prev_reverse: bool,
|
||||
|
||||
// --- Report-processing counters (in-memory process state only) ---
|
||||
/// ReceiverReports that arrived at all, counted before the stale/duplicate
|
||||
/// early return. A diagnostic that reads only the accepted-sample count
|
||||
/// cannot tell "nothing came back" from "nothing usable came back", and
|
||||
/// those mean opposite things about reverse-path reachability.
|
||||
reports_seen: u64,
|
||||
/// Reports that produced an accepted RTT sample (`rtt_ms > 0`).
|
||||
rtt_samples: u64,
|
||||
/// Echoes whose derived RTT truncated to 0 ms — the expected outcome on
|
||||
/// loopback and same-host meshes, not a fault.
|
||||
rtt_zero: u64,
|
||||
/// Echoes whose arithmetic failed: the session-relative timestamp wrapped,
|
||||
/// or the peer echoed a bogus or stale value. A real anomaly.
|
||||
rtt_arith_fail: u64,
|
||||
/// The most recent accepted per-exchange RTT in ms, which the SRTT
|
||||
/// estimator otherwise smooths away.
|
||||
last_rtt_ms: Option<u32>,
|
||||
}
|
||||
|
||||
impl MmpMetrics {
|
||||
@@ -132,6 +150,11 @@ impl MmpMetrics {
|
||||
prev_reverse_packets: 0,
|
||||
prev_reverse_highest: 0,
|
||||
has_prev_reverse: false,
|
||||
reports_seen: 0,
|
||||
rtt_samples: 0,
|
||||
rtt_zero: 0,
|
||||
rtt_arith_fail: 0,
|
||||
last_rtt_ms: None,
|
||||
}
|
||||
}
|
||||
|
||||
@@ -151,6 +174,9 @@ impl MmpMetrics {
|
||||
now_ms: u64,
|
||||
) -> (bool, RrLog) {
|
||||
let had_srtt = self.srtt.initialized();
|
||||
// Counted before the stale/duplicate return below: this counter means
|
||||
// "a report arrived at all", not "a report was useful".
|
||||
self.reports_seen += 1;
|
||||
|
||||
if self.has_prev_rr {
|
||||
let counters_regressed = rr.highest_counter < self.prev_rr_highest_counter
|
||||
@@ -195,10 +221,19 @@ impl MmpMetrics {
|
||||
// trace read `srtt` before `update`).
|
||||
let srtt_ms = self.srtt.srtt_us() as f64 / 1000.0;
|
||||
log = RrLog::RttSample { rtt_ms, srtt_ms };
|
||||
self.rtt_samples += 1;
|
||||
self.last_rtt_ms = Some(rtt_ms);
|
||||
self.srtt.update(rtt_us);
|
||||
self.rtt_trend.update(rtt_us as f64);
|
||||
}
|
||||
_ => {
|
||||
Some(_) => {
|
||||
// A sample that truncated to 0 ms: sub-millisecond path.
|
||||
self.rtt_zero += 1;
|
||||
log = RrLog::InvalidRtt;
|
||||
}
|
||||
None => {
|
||||
// `checked_add` / `checked_sub` failed: wrapped or bogus.
|
||||
self.rtt_arith_fail += 1;
|
||||
log = RrLog::InvalidRtt;
|
||||
}
|
||||
}
|
||||
@@ -324,6 +359,31 @@ impl MmpMetrics {
|
||||
pub fn last_ecn_ce_count(&self) -> u32 {
|
||||
self.prev_rr_ecn_ce
|
||||
}
|
||||
|
||||
/// ReceiverReports processed, including stale and duplicate ones.
|
||||
pub fn reports_seen(&self) -> u64 {
|
||||
self.reports_seen
|
||||
}
|
||||
|
||||
/// Reports that yielded an accepted RTT sample.
|
||||
pub fn rtt_samples(&self) -> u64 {
|
||||
self.rtt_samples
|
||||
}
|
||||
|
||||
/// Echoes whose derived RTT truncated to 0 ms.
|
||||
pub fn rtt_zero(&self) -> u64 {
|
||||
self.rtt_zero
|
||||
}
|
||||
|
||||
/// Echoes whose RTT arithmetic failed.
|
||||
pub fn rtt_arith_fail(&self) -> u64 {
|
||||
self.rtt_arith_fail
|
||||
}
|
||||
|
||||
/// The most recent accepted per-exchange RTT in ms.
|
||||
pub fn last_rtt_ms(&self) -> Option<u32> {
|
||||
self.last_rtt_ms
|
||||
}
|
||||
}
|
||||
|
||||
impl Default for MmpMetrics {
|
||||
|
||||
@@ -565,6 +565,60 @@ fn test_ignores_future_rtt_sample() {
|
||||
assert!(m.srtt_ms().is_none());
|
||||
}
|
||||
|
||||
/// The four report counters must land in the arms they name. `rtt_zero` means
|
||||
/// "expected on loopback" and `rtt_arith_fail` means "a real anomaly": swapping
|
||||
/// them steers an operator away from a genuine fault, and nothing else in the
|
||||
/// suite observes either of them incrementing from a real report.
|
||||
#[test]
|
||||
fn test_report_counters_discriminate_the_rtt_outcomes() {
|
||||
// A positive sample: echo 1000 + dwell 5, our clock at 1050 => 45 ms.
|
||||
let mut m = MmpMetrics::new();
|
||||
m.process_receiver_report(&make_rr(10, 10, 5_000, 1_000, 5, 0), 1_050, 0);
|
||||
assert_eq!(m.reports_seen(), 1);
|
||||
assert_eq!(m.rtt_samples(), 1);
|
||||
assert_eq!(m.last_rtt_ms(), Some(45));
|
||||
assert_eq!(m.rtt_zero(), 0);
|
||||
assert_eq!(m.rtt_arith_fail(), 0);
|
||||
|
||||
// A sample that truncates to exactly 0 ms: echo 1000 + dwell 5 == our 1005.
|
||||
let mut m = MmpMetrics::new();
|
||||
m.process_receiver_report(&make_rr(10, 10, 5_000, 1_000, 5, 0), 1_005, 0);
|
||||
assert_eq!(m.reports_seen(), 1);
|
||||
assert_eq!(m.rtt_zero(), 1);
|
||||
assert_eq!(m.rtt_samples(), 0);
|
||||
assert_eq!(m.rtt_arith_fail(), 0);
|
||||
assert_eq!(m.last_rtt_ms(), None);
|
||||
|
||||
// Arithmetic failure: echo + dwell overflows u32.
|
||||
let mut m = MmpMetrics::new();
|
||||
m.process_receiver_report(&make_rr(10, 10, 5_000, u32::MAX - 10, 20, 0), 15, 0);
|
||||
assert_eq!(m.reports_seen(), 1);
|
||||
assert_eq!(m.rtt_arith_fail(), 1);
|
||||
assert_eq!(m.rtt_zero(), 0);
|
||||
assert_eq!(m.rtt_samples(), 0);
|
||||
}
|
||||
|
||||
/// `reports_seen` is counted before the stale/duplicate early return, which is
|
||||
/// what lets the probe's `no_report` be a reachability claim rather than a
|
||||
/// measurement claim. Moving it below the return would make the probe assert
|
||||
/// one-way reachability against a peer that is in fact answering.
|
||||
#[test]
|
||||
fn test_reports_seen_counts_a_stale_report_the_rtt_arms_reject() {
|
||||
let mut m = MmpMetrics::new();
|
||||
let rr = make_rr(10, 10, 5_000, 1_000, 5, 0);
|
||||
m.process_receiver_report(&rr, 1_050, 0);
|
||||
let after_first = (m.rtt_samples(), m.rtt_zero(), m.rtt_arith_fail());
|
||||
|
||||
// The identical report again is stale and returns before the RTT arms.
|
||||
m.process_receiver_report(&rr, 6_000, 5_000);
|
||||
assert_eq!(m.reports_seen(), 2, "a report arrived, usable or not");
|
||||
assert_eq!(
|
||||
(m.rtt_samples(), m.rtt_zero(), m.rtt_arith_fail()),
|
||||
after_first,
|
||||
"a stale report must move no RTT counter"
|
||||
);
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn test_loss_rate_computation() {
|
||||
let mut m = MmpMetrics::new();
|
||||
|
||||
@@ -15,6 +15,7 @@ pub(crate) mod link;
|
||||
pub(crate) mod lookup;
|
||||
pub(crate) mod math;
|
||||
pub(crate) mod mmp;
|
||||
pub(crate) mod probe;
|
||||
pub(crate) mod rate_limit;
|
||||
pub(crate) mod routing;
|
||||
pub(crate) mod stp;
|
||||
|
||||
@@ -0,0 +1,785 @@
|
||||
//! Sans-IO probe stage machine.
|
||||
//!
|
||||
//! Pure, runtime-agnostic decisions for the `probe` diagnostic: which stage is
|
||||
//! current, whether its budget has expired, whether the probe owns the session
|
||||
//! it is about to tear down, and what the overall verdict is. The async driver
|
||||
//! in `node::handlers::probe` builds a plain-data [`Observation`] (every clock
|
||||
//! read pre-resolved into `u64`, every map read into `bool`/`Option`), calls
|
||||
//! [`Probe::step`], and performs the returned [`ProbeAction`]s. No I/O, no
|
||||
//! clock, no logging here.
|
||||
//!
|
||||
//! Time enters in three ways and is never read inside the core: `now_ms` on
|
||||
//! every observation, the budgets computed once by the shell from config, and
|
||||
//! the four MMP report counters — which are *counters* rather than timestamps
|
||||
//! precisely so the core need not reason about when a report arrived.
|
||||
|
||||
use super::limits::{
|
||||
BLOOM_MIN_TICKS, RESOLVE_SLACK_TICKS, RTT_FLOOR_MS, RTT_MIN_TICKS, SESSION_FLOOR_MS,
|
||||
SESSION_MIN_TICKS, TEARDOWN_GRACE_TICKS, WARMUP_RETRY_MS,
|
||||
};
|
||||
use super::state::{
|
||||
FailKind, LeftIntact, LookupOutcomeKind, NextHopFacts, NoHopReason, Overall, PathFacts,
|
||||
Preflight, ProbeSnapshot, ResolveSource, RttCounters, Stage, StageRecord, StageVerdict,
|
||||
};
|
||||
use crate::proto::stp::TreeCoordinate;
|
||||
|
||||
/// Per-stage budgets, all monotonic milliseconds and all tick-quantized by the
|
||||
/// shell before they get here.
|
||||
#[derive(Copy, Clone, Debug, PartialEq, Eq)]
|
||||
pub(crate) struct Budgets {
|
||||
pub(crate) bloom_ms: u64,
|
||||
pub(crate) discovery_ms: u64,
|
||||
pub(crate) session_ms: u64,
|
||||
pub(crate) rtt_ms: u64,
|
||||
pub(crate) tick_ms: u64,
|
||||
}
|
||||
|
||||
impl Budgets {
|
||||
/// Derive the stage budgets from the rx-loop tick period and the lookup
|
||||
/// attempt ladder. Each stage is `max(floor, N * tick_ms)` so a node with a
|
||||
/// long tick does not expire a stage between two observations.
|
||||
pub(crate) fn derive(tick_ms: u64, attempt_timeouts_secs: &[u64]) -> Self {
|
||||
let ladder_ms: u64 = attempt_timeouts_secs.iter().sum::<u64>() * 1000;
|
||||
Self {
|
||||
bloom_ms: BLOOM_MIN_TICKS * tick_ms,
|
||||
discovery_ms: ladder_ms + RESOLVE_SLACK_TICKS * tick_ms,
|
||||
session_ms: SESSION_FLOOR_MS.max(SESSION_MIN_TICKS * tick_ms),
|
||||
rtt_ms: RTT_FLOOR_MS.max(RTT_MIN_TICKS * tick_ms),
|
||||
tick_ms,
|
||||
}
|
||||
}
|
||||
|
||||
/// Total worst-case budget, which is what the client sizes its own ceiling
|
||||
/// from.
|
||||
pub(crate) fn total_ms(&self) -> u64 {
|
||||
self.bloom_ms + self.discovery_ms + self.session_ms + self.rtt_ms
|
||||
}
|
||||
}
|
||||
|
||||
/// Live state as of one tick, with every clock read already resolved.
|
||||
#[derive(Clone, Debug)]
|
||||
pub(crate) struct Observation {
|
||||
/// Monotonic. All budget arithmetic uses this.
|
||||
///
|
||||
/// Deliberately the monotonic clock and not the wall clock: an NTP step or
|
||||
/// a resume from suspend must not invent a handshake timeout against a
|
||||
/// healthy peer. Wall time is read shell-side only, where the surrounding
|
||||
/// API requires it (the coord cache's TTLs are keyed to it).
|
||||
pub(crate) now_ms: u64,
|
||||
pub(crate) coords_cached: bool,
|
||||
pub(crate) lookup_pending: bool,
|
||||
pub(crate) lookup_outcome: Option<LookupOutcomeKind>,
|
||||
pub(crate) lookup_fanout: Option<usize>,
|
||||
pub(crate) path: Option<PathFacts>,
|
||||
pub(crate) next_hop: Option<NextHopFacts>,
|
||||
pub(crate) no_hop_reason: Option<NoHopReason>,
|
||||
pub(crate) session_present: bool,
|
||||
pub(crate) session_established: bool,
|
||||
/// The identity check passed at drive time: the live entry is the one this
|
||||
/// probe created and still exclusively owns.
|
||||
pub(crate) session_is_ours: bool,
|
||||
pub(crate) session_error: Option<String>,
|
||||
pub(crate) target_is_direct_peer: bool,
|
||||
pub(crate) counters: RttCounters,
|
||||
pub(crate) last_rtt_ms: Option<u32>,
|
||||
pub(crate) srtt_ms: Option<f64>,
|
||||
pub(crate) path_mtu: Option<u32>,
|
||||
}
|
||||
|
||||
/// An effect the async driver performs on the core's behalf.
|
||||
#[derive(Clone, Debug, PartialEq, Eq)]
|
||||
pub(crate) enum ProbeAction {
|
||||
/// Call the gated lookup entry point for the target.
|
||||
InitiateLookup,
|
||||
/// Open an FSP session, after re-checking that no entry exists.
|
||||
OpenSession,
|
||||
/// Send one standalone `CoordsWarmup` on the session.
|
||||
SendWarmup,
|
||||
/// Remove the probe-created session, after the identity and adoption
|
||||
/// checks pass.
|
||||
TeardownSession,
|
||||
/// Mark the job terminal and release its target claim.
|
||||
Finish,
|
||||
}
|
||||
|
||||
/// The probe stage machine.
|
||||
pub(crate) struct Probe {
|
||||
started_ms: u64,
|
||||
ended_ms: Option<u64>,
|
||||
deadline_ms: u64,
|
||||
budgets: Budgets,
|
||||
preflight: Preflight,
|
||||
|
||||
stage: Stage,
|
||||
stage_started_ms: u64,
|
||||
bloom: StageRecord,
|
||||
discovery: StageRecord,
|
||||
path: StageRecord,
|
||||
session: StageRecord,
|
||||
rtt: StageRecord,
|
||||
overall: Overall,
|
||||
|
||||
// --- ownership ---
|
||||
owns_session: bool,
|
||||
open_requested: bool,
|
||||
teardown_emitted: bool,
|
||||
teardown_at_ms: Option<u64>,
|
||||
left_intact: Option<LeftIntact>,
|
||||
|
||||
// --- latched observations ---
|
||||
resolve_source: Option<ResolveSource>,
|
||||
lookup_issued: bool,
|
||||
lookup_fanout: Option<usize>,
|
||||
lookup_was_pending: bool,
|
||||
lookup_outcome: Option<LookupOutcomeKind>,
|
||||
path_facts: Option<PathFacts>,
|
||||
next_hop: Option<NextHopFacts>,
|
||||
no_hop_reason: Option<NoHopReason>,
|
||||
session_preexisting: bool,
|
||||
session_established: bool,
|
||||
path_mtu: Option<u32>,
|
||||
rtt_ms: Option<u32>,
|
||||
srtt_ms: Option<f64>,
|
||||
baseline: RttCounters,
|
||||
counters: RttCounters,
|
||||
warmups_sent: u8,
|
||||
last_warmup_ms: Option<u64>,
|
||||
finished: bool,
|
||||
}
|
||||
|
||||
impl Probe {
|
||||
/// Start a probe. Records the deadline; it does **not** decide ownership,
|
||||
/// which is latched at action time (see [`Probe::step`]).
|
||||
pub(crate) fn new(now_ms: u64, budgets: Budgets, preflight: Preflight) -> Self {
|
||||
Self {
|
||||
started_ms: now_ms,
|
||||
ended_ms: None,
|
||||
deadline_ms: now_ms + budgets.total_ms(),
|
||||
budgets,
|
||||
preflight,
|
||||
stage: Stage::Bloom,
|
||||
stage_started_ms: now_ms,
|
||||
bloom: StageRecord::pending(),
|
||||
discovery: StageRecord::pending(),
|
||||
path: StageRecord::pending(),
|
||||
session: StageRecord::pending(),
|
||||
rtt: StageRecord::pending(),
|
||||
overall: Overall::Running,
|
||||
owns_session: false,
|
||||
open_requested: false,
|
||||
teardown_emitted: false,
|
||||
teardown_at_ms: None,
|
||||
left_intact: None,
|
||||
resolve_source: None,
|
||||
lookup_issued: false,
|
||||
lookup_fanout: None,
|
||||
lookup_was_pending: false,
|
||||
lookup_outcome: None,
|
||||
path_facts: None,
|
||||
next_hop: None,
|
||||
no_hop_reason: None,
|
||||
session_preexisting: preflight.session_present,
|
||||
session_established: false,
|
||||
path_mtu: None,
|
||||
rtt_ms: None,
|
||||
srtt_ms: None,
|
||||
baseline: RttCounters::default(),
|
||||
counters: RttCounters::default(),
|
||||
warmups_sent: 0,
|
||||
last_warmup_ms: None,
|
||||
finished: false,
|
||||
}
|
||||
}
|
||||
|
||||
/// Whether the probe currently holds exclusive ownership of the session.
|
||||
#[cfg(test)]
|
||||
pub(crate) fn owns_session(&self) -> bool {
|
||||
self.owns_session
|
||||
}
|
||||
|
||||
pub(crate) fn is_finished(&self) -> bool {
|
||||
self.finished
|
||||
}
|
||||
|
||||
/// Advance the machine by one observation.
|
||||
pub(crate) fn step(&mut self, obs: &Observation) -> Vec<ProbeAction> {
|
||||
let mut actions = Vec::new();
|
||||
if self.finished {
|
||||
return actions;
|
||||
}
|
||||
|
||||
self.absorb(obs);
|
||||
|
||||
// Ownership can be lost but never regained. A live entry that is no
|
||||
// longer byte-for-byte the one we created is not ours to remove.
|
||||
//
|
||||
// The discriminator is `open_requested`, not `confirmed_ours`: under
|
||||
// simultaneous initiation the peer's inbound handshake can replace our
|
||||
// entry before the first observation that would have confirmed it, and
|
||||
// an entry the probe asked for and obtained was still replaced rather
|
||||
// than pre-existing.
|
||||
if self.owns_session && !obs.session_is_ours {
|
||||
self.owns_session = false;
|
||||
self.left_intact = Some(if self.open_requested {
|
||||
LeftIntact::Replaced
|
||||
} else {
|
||||
LeftIntact::Preexisting
|
||||
});
|
||||
}
|
||||
|
||||
if self.stage == Stage::Terminal {
|
||||
return self.drive_terminal(obs, actions);
|
||||
}
|
||||
|
||||
// Total-deadline backstop, independent of the per-stage budgets.
|
||||
if obs.now_ms >= self.deadline_ms {
|
||||
self.expire_running_stage();
|
||||
self.enter_terminal(obs);
|
||||
return self.drive_terminal(obs, actions);
|
||||
}
|
||||
|
||||
// Run the stage machine until it blocks. A stage that completes
|
||||
// without needing I/O — the path stage always, and a skipped resolve —
|
||||
// hands straight on to the next one inside the same tick, so a probe
|
||||
// against a cached peer does not spend three ticks doing arithmetic.
|
||||
loop {
|
||||
let before = self.stage;
|
||||
match self.stage {
|
||||
Stage::Bloom => self.step_bloom(obs, &mut actions),
|
||||
Stage::Discovery => self.step_discovery(obs),
|
||||
Stage::Path => self.step_path(obs),
|
||||
Stage::Session => self.step_session(obs, &mut actions),
|
||||
Stage::Rtt => self.step_rtt(obs, &mut actions),
|
||||
Stage::Terminal => {}
|
||||
}
|
||||
if self.stage == before || self.stage == Stage::Terminal {
|
||||
break;
|
||||
}
|
||||
}
|
||||
|
||||
if self.stage == Stage::Terminal {
|
||||
return self.drive_terminal(obs, actions);
|
||||
}
|
||||
actions
|
||||
}
|
||||
|
||||
/// Cancel the probe: mark it cancelled and emit the terminal actions
|
||||
/// without waiting out the teardown grace, since the caller wants the job
|
||||
/// gone now.
|
||||
pub(crate) fn cancel(&mut self, now_ms: u64) -> Vec<ProbeAction> {
|
||||
let mut actions = Vec::new();
|
||||
if self.finished {
|
||||
return actions;
|
||||
}
|
||||
self.expire_running_stage();
|
||||
self.overall = Overall::Cancelled;
|
||||
self.stage = Stage::Terminal;
|
||||
self.ended_ms = Some(now_ms);
|
||||
if self.owns_session && !self.teardown_emitted {
|
||||
self.teardown_emitted = true;
|
||||
actions.push(ProbeAction::TeardownSession);
|
||||
}
|
||||
self.finished = true;
|
||||
actions.push(ProbeAction::Finish);
|
||||
actions
|
||||
}
|
||||
|
||||
/// Project the report the control socket publishes.
|
||||
pub(crate) fn snapshot(&self) -> ProbeSnapshot {
|
||||
ProbeSnapshot {
|
||||
overall: self.overall,
|
||||
tick_ms: self.budgets.tick_ms,
|
||||
bloom: self.bloom.clone(),
|
||||
discovery: self.discovery.clone(),
|
||||
path: self.path.clone(),
|
||||
session: self.session.clone(),
|
||||
rtt: self.rtt.clone(),
|
||||
resolve_source: self.resolve_source,
|
||||
lookup_fanout: self.lookup_fanout,
|
||||
path_facts: self.path_facts.clone(),
|
||||
next_hop: self.next_hop.clone(),
|
||||
no_hop_reason: self.no_hop_reason,
|
||||
session_preexisting: self.session_preexisting,
|
||||
session_established: self.session_established,
|
||||
path_mtu: self.path_mtu,
|
||||
rtt_ms: self.rtt_ms,
|
||||
srtt_ms: self.srtt_ms,
|
||||
counters: self.counters,
|
||||
torn_down: self.teardown_emitted,
|
||||
left_intact: self.left_intact,
|
||||
lookup_issued: self.lookup_issued,
|
||||
coords_were_cached: self.preflight.coords_cached,
|
||||
identity_was_cached: self.preflight.identity_cached,
|
||||
warmups_sent: self.warmups_sent,
|
||||
}
|
||||
}
|
||||
|
||||
/// Elapsed wall of the job so far, for a poll that arrives mid-run.
|
||||
pub(crate) fn elapsed_ms(&self, now_ms: u64) -> u64 {
|
||||
self.ended_ms
|
||||
.unwrap_or(now_ms)
|
||||
.saturating_sub(self.started_ms)
|
||||
}
|
||||
|
||||
// ---- stages ----------------------------------------------------------
|
||||
|
||||
/// Is the target claimed by anyone, and did a request therefore go out?
|
||||
///
|
||||
/// The gate is evaluated shell-side inside the action this stage emits, so
|
||||
/// its answer arrives on the following observation rather than this one.
|
||||
/// Every outcome that means *no request was sent* ends the probe here,
|
||||
/// which is the whole reason this is its own stage: "nobody claims this
|
||||
/// address" and "nobody answered for it" are different findings and used
|
||||
/// to share one verdict.
|
||||
fn step_bloom(&mut self, obs: &Observation, actions: &mut Vec<ProbeAction>) {
|
||||
if !self.lookup_issued {
|
||||
if obs.coords_cached {
|
||||
self.finish_stage(FailKind::Cached.into_skip(), obs);
|
||||
self.resolve_source = Some(ResolveSource::Cache);
|
||||
self.advance(obs);
|
||||
return;
|
||||
}
|
||||
if obs.target_is_direct_peer {
|
||||
// Routing short-circuits on a send-ready direct peer before it
|
||||
// reads the coord cache, so a neighbour needs no lookup at all.
|
||||
self.finish_stage(FailKind::DirectPeer.into_skip(), obs);
|
||||
self.resolve_source = Some(ResolveSource::DirectPeer);
|
||||
self.advance(obs);
|
||||
return;
|
||||
}
|
||||
self.bloom.verdict = StageVerdict::Running;
|
||||
self.lookup_issued = true;
|
||||
actions.push(ProbeAction::InitiateLookup);
|
||||
return;
|
||||
}
|
||||
|
||||
match self.lookup_outcome {
|
||||
Some(LookupOutcomeKind::BloomMiss) => self.fail_stage(FailKind::BloomMiss, obs),
|
||||
Some(LookupOutcomeKind::Suppressed) => {
|
||||
self.fail_stage(FailKind::BackoffSuppressed, obs)
|
||||
}
|
||||
Some(LookupOutcomeKind::ZeroFanout) => self.fail_stage(FailKind::NoTreePeers, obs),
|
||||
// A request is on the wire, or this probe joined one already in
|
||||
// flight. Either way the filters claimed the target.
|
||||
Some(LookupOutcomeKind::Sent) | Some(LookupOutcomeKind::Deduplicated) => {
|
||||
self.finish_stage(ok_record(), obs);
|
||||
self.advance(obs);
|
||||
}
|
||||
// Can only be reached if the action never ran. The total deadline
|
||||
// backstops it; this bounds the wait to something legible.
|
||||
None => {
|
||||
if obs.now_ms.saturating_sub(self.stage_started_ms) >= self.budgets.bloom_ms {
|
||||
self.fail_stage(FailKind::NoResponse, obs);
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
/// Wait for a LookupResponse to put coordinates in the cache.
|
||||
fn step_discovery(&mut self, obs: &Observation) {
|
||||
// Nothing was asked, so there is nothing to wait for: the bloom stage
|
||||
// already settled the cases that need no lookup, and this stage
|
||||
// records the same reason rather than inventing a second one.
|
||||
if !self.lookup_issued {
|
||||
let kind = match self.resolve_source {
|
||||
Some(ResolveSource::DirectPeer) => FailKind::DirectPeer,
|
||||
_ => FailKind::Cached,
|
||||
};
|
||||
self.finish_stage(kind.into_skip(), obs);
|
||||
self.advance(obs);
|
||||
return;
|
||||
}
|
||||
|
||||
if obs.coords_cached {
|
||||
self.finish_stage(ok_record(), obs);
|
||||
self.resolve_source = Some(ResolveSource::Lookup);
|
||||
self.advance(obs);
|
||||
return;
|
||||
}
|
||||
|
||||
// The request is out and the answer has not arrived. Say so: a stage
|
||||
// left `Pending` while it is in fact waiting reads to a poller as one
|
||||
// that has not started, and this one waits the longest of any.
|
||||
self.discovery.verdict = StageVerdict::Running;
|
||||
|
||||
// The ladder gave up: the pending entry existed and is now gone with
|
||||
// no coordinates. The answer is already known, so do not hold the
|
||||
// caller for the rest of the budget.
|
||||
let gave_up = self.lookup_was_pending && !obs.lookup_pending;
|
||||
if gave_up || obs.now_ms.saturating_sub(self.stage_started_ms) >= self.budgets.discovery_ms
|
||||
{
|
||||
let reason = if self.lookup_outcome == Some(LookupOutcomeKind::Deduplicated) {
|
||||
FailKind::AlreadyPending
|
||||
} else {
|
||||
FailKind::NoResponse
|
||||
};
|
||||
self.fail_stage(reason, obs);
|
||||
}
|
||||
}
|
||||
|
||||
fn step_path(&mut self, obs: &Observation) {
|
||||
// Path is pure computation over what resolve established; it never
|
||||
// halts the probe, because `initiate_session`'s own routing may
|
||||
// succeed where this non-touching preview did not.
|
||||
let record = match &self.path_facts {
|
||||
None => {
|
||||
// A send-ready direct peer already latched a next hop from the
|
||||
// routing short-circuit, which reads no coordinates at all. Only
|
||||
// claim "no coords" when there is genuinely no hop to name.
|
||||
if self.next_hop.is_none() {
|
||||
self.no_hop_reason = Some(NoHopReason::NoCoords);
|
||||
}
|
||||
if obs.target_is_direct_peer {
|
||||
FailKind::DirectPeer.into_skip()
|
||||
} else {
|
||||
FailKind::NoNextHop.into_fail()
|
||||
}
|
||||
}
|
||||
Some(facts) if !facts.same_root => FailKind::DisjointTrees.into_fail(),
|
||||
Some(_) if self.next_hop.is_none() => FailKind::NoNextHop.into_fail(),
|
||||
Some(_) => ok_record(),
|
||||
};
|
||||
self.finish_stage(record, obs);
|
||||
self.advance(obs);
|
||||
}
|
||||
|
||||
fn step_session(&mut self, obs: &Observation, actions: &mut Vec<ProbeAction>) {
|
||||
if obs.session_established && self.owns_session {
|
||||
self.session_established = true;
|
||||
self.finish_stage(ok_record(), obs);
|
||||
self.advance(obs);
|
||||
return;
|
||||
}
|
||||
|
||||
if let Some(err) = &obs.session_error {
|
||||
// `initiate_session` inserts its entry only after the send
|
||||
// succeeds, so an error means nothing was created.
|
||||
self.owns_session = false;
|
||||
let mut record = FailKind::SendError.into_fail();
|
||||
record.detail = Some(err.clone());
|
||||
self.finish_stage(record, obs);
|
||||
self.skip_rest(FailKind::NotReached);
|
||||
self.enter_terminal(obs);
|
||||
return;
|
||||
}
|
||||
|
||||
if obs.session_present {
|
||||
// Only an entry the probe never asked for is pre-existing. The
|
||||
// ticks between our own `OpenSession` and establishment also see
|
||||
// `session_present`, and reporting those as pre-existing would
|
||||
// contradict the teardown the same report carries.
|
||||
if !self.open_requested {
|
||||
self.session_preexisting = true;
|
||||
}
|
||||
if !self.owns_session {
|
||||
if obs.session_established {
|
||||
self.session_established = true;
|
||||
self.finish_stage(FailKind::Preexisting.into_skip(), obs);
|
||||
self.left_intact = Some(self.left_intact.unwrap_or(LeftIntact::Preexisting));
|
||||
self.advance(obs);
|
||||
return;
|
||||
}
|
||||
// Someone else's handshake is in flight. Wait it out, but do
|
||||
// not touch it.
|
||||
if obs.now_ms.saturating_sub(self.stage_started_ms) >= self.budgets.session_ms {
|
||||
self.finish_stage(FailKind::Preexisting.into_skip(), obs);
|
||||
self.left_intact = Some(self.left_intact.unwrap_or(LeftIntact::Preexisting));
|
||||
self.skip_rest(FailKind::NotReached);
|
||||
self.enter_terminal(obs);
|
||||
}
|
||||
return;
|
||||
}
|
||||
} else if !self.open_requested && !self.preflight.target_claimed {
|
||||
// The one place ownership is taken, and only on a step where no
|
||||
// entry exists at all — broader than production's
|
||||
// established-or-initiating predicate, so an inbound handshake
|
||||
// awaiting msg3 also counts as "not mine".
|
||||
self.session.verdict = StageVerdict::Running;
|
||||
self.open_requested = true;
|
||||
self.owns_session = true;
|
||||
actions.push(ProbeAction::OpenSession);
|
||||
return;
|
||||
}
|
||||
|
||||
if obs.now_ms.saturating_sub(self.stage_started_ms) >= self.budgets.session_ms {
|
||||
let record = if self.owns_session {
|
||||
FailKind::HandshakeTimeout.into_fail()
|
||||
} else {
|
||||
FailKind::Preexisting.into_skip()
|
||||
};
|
||||
self.finish_stage(record, obs);
|
||||
self.skip_rest(FailKind::NotReached);
|
||||
self.enter_terminal(obs);
|
||||
}
|
||||
}
|
||||
|
||||
fn step_rtt(&mut self, obs: &Observation, actions: &mut Vec<ProbeAction>) {
|
||||
if self.warmups_sent == 0 {
|
||||
self.rtt.verdict = StageVerdict::Running;
|
||||
self.baseline = obs.counters;
|
||||
self.warmups_sent = 1;
|
||||
self.last_warmup_ms = Some(obs.now_ms);
|
||||
actions.push(ProbeAction::SendWarmup);
|
||||
return;
|
||||
}
|
||||
|
||||
let delta = obs.counters.delta(self.baseline);
|
||||
self.counters = delta;
|
||||
|
||||
// A pre-existing session may already carry an SRTT, and SRTT survives
|
||||
// rekey, so `srtt_ms.is_some()` proves nothing about this probe. A new
|
||||
// accepted sample is the only attributable signal.
|
||||
if delta.samples > 0 {
|
||||
self.rtt_ms = obs.last_rtt_ms;
|
||||
self.finish_stage(ok_record(), obs);
|
||||
self.enter_terminal(obs);
|
||||
return;
|
||||
}
|
||||
|
||||
let retry_after = WARMUP_RETRY_MS.max(self.budgets.tick_ms);
|
||||
if self.warmups_sent == 1
|
||||
&& delta.reports_seen == 0
|
||||
&& obs.now_ms.saturating_sub(self.last_warmup_ms.unwrap_or(0)) >= retry_after
|
||||
{
|
||||
self.warmups_sent = 2;
|
||||
self.last_warmup_ms = Some(obs.now_ms);
|
||||
actions.push(ProbeAction::SendWarmup);
|
||||
}
|
||||
|
||||
if obs.now_ms.saturating_sub(self.stage_started_ms) >= self.budgets.rtt_ms {
|
||||
self.finish_stage(classify_rtt_failure(delta).into_fail(), obs);
|
||||
self.enter_terminal(obs);
|
||||
}
|
||||
}
|
||||
|
||||
// ---- helpers ---------------------------------------------------------
|
||||
|
||||
/// Latch the report-only facts every stage publishes.
|
||||
fn absorb(&mut self, obs: &Observation) {
|
||||
if let Some(kind) = obs.lookup_outcome {
|
||||
self.lookup_outcome = Some(kind);
|
||||
}
|
||||
if obs.lookup_fanout.is_some() {
|
||||
self.lookup_fanout = obs.lookup_fanout;
|
||||
}
|
||||
if obs.lookup_pending {
|
||||
self.lookup_was_pending = true;
|
||||
}
|
||||
if obs.path.is_some() {
|
||||
self.path_facts = obs.path.clone();
|
||||
}
|
||||
if obs.next_hop.is_some() {
|
||||
self.next_hop = obs.next_hop.clone();
|
||||
}
|
||||
if obs.no_hop_reason.is_some() {
|
||||
self.no_hop_reason = obs.no_hop_reason;
|
||||
}
|
||||
if obs.path_mtu.is_some() {
|
||||
self.path_mtu = obs.path_mtu;
|
||||
}
|
||||
if obs.srtt_ms.is_some() {
|
||||
self.srtt_ms = obs.srtt_ms;
|
||||
}
|
||||
if obs.session_established {
|
||||
self.session_established = true;
|
||||
}
|
||||
}
|
||||
|
||||
/// End the probe on the current stage's failure: record it, mark
|
||||
/// everything behind it not reached, and go terminal.
|
||||
fn fail_stage(&mut self, kind: FailKind, obs: &Observation) {
|
||||
self.finish_stage(kind.into_fail(), obs);
|
||||
self.skip_rest(FailKind::NotReached);
|
||||
self.enter_terminal(obs);
|
||||
}
|
||||
|
||||
fn finish_stage(&mut self, mut record: StageRecord, obs: &Observation) {
|
||||
record.elapsed_ms = Some(obs.now_ms.saturating_sub(self.stage_started_ms));
|
||||
*self.current_record_mut() = record;
|
||||
}
|
||||
|
||||
fn current_record_mut(&mut self) -> &mut StageRecord {
|
||||
match self.stage {
|
||||
Stage::Bloom => &mut self.bloom,
|
||||
Stage::Discovery => &mut self.discovery,
|
||||
Stage::Path => &mut self.path,
|
||||
Stage::Session => &mut self.session,
|
||||
Stage::Rtt | Stage::Terminal => &mut self.rtt,
|
||||
}
|
||||
}
|
||||
|
||||
fn advance(&mut self, obs: &Observation) {
|
||||
self.stage = match self.stage {
|
||||
Stage::Bloom => Stage::Discovery,
|
||||
Stage::Discovery => Stage::Path,
|
||||
Stage::Path => Stage::Session,
|
||||
Stage::Session => Stage::Rtt,
|
||||
Stage::Rtt | Stage::Terminal => Stage::Terminal,
|
||||
};
|
||||
self.stage_started_ms = obs.now_ms;
|
||||
}
|
||||
|
||||
/// Mark every stage after the current one as not reached.
|
||||
fn skip_rest(&mut self, kind: FailKind) {
|
||||
let record = kind.into_skip();
|
||||
let current = self.stage;
|
||||
let mut mark = false;
|
||||
for (stage, slot) in [
|
||||
(Stage::Bloom, &mut self.bloom),
|
||||
(Stage::Discovery, &mut self.discovery),
|
||||
(Stage::Path, &mut self.path),
|
||||
(Stage::Session, &mut self.session),
|
||||
(Stage::Rtt, &mut self.rtt),
|
||||
] {
|
||||
if mark {
|
||||
*slot = record.clone();
|
||||
}
|
||||
if stage == current {
|
||||
mark = true;
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
/// The deadline fired mid-stage: record the running stage as failed for
|
||||
/// the reason its own budget would have given.
|
||||
fn expire_running_stage(&mut self) {
|
||||
let kind = match self.stage {
|
||||
Stage::Bloom | Stage::Discovery => FailKind::NoResponse,
|
||||
Stage::Path => FailKind::NoNextHop,
|
||||
Stage::Session => {
|
||||
if self.owns_session {
|
||||
FailKind::HandshakeTimeout
|
||||
} else {
|
||||
FailKind::Preexisting
|
||||
}
|
||||
}
|
||||
Stage::Rtt => classify_rtt_failure(self.counters),
|
||||
Stage::Terminal => return,
|
||||
};
|
||||
let record = if self.stage == Stage::Session && !self.owns_session {
|
||||
kind.into_skip()
|
||||
} else {
|
||||
kind.into_fail()
|
||||
};
|
||||
if self.current_record_mut().verdict != StageVerdict::Ok {
|
||||
*self.current_record_mut() = record;
|
||||
}
|
||||
self.skip_rest(FailKind::NotReached);
|
||||
}
|
||||
|
||||
fn enter_terminal(&mut self, obs: &Observation) {
|
||||
self.stage = Stage::Terminal;
|
||||
self.teardown_at_ms = Some(obs.now_ms + TEARDOWN_GRACE_TICKS * self.budgets.tick_ms.max(1));
|
||||
self.overall = self.compute_overall();
|
||||
}
|
||||
|
||||
fn drive_terminal(
|
||||
&mut self,
|
||||
obs: &Observation,
|
||||
mut actions: Vec<ProbeAction>,
|
||||
) -> Vec<ProbeAction> {
|
||||
if self.owns_session && !self.teardown_emitted {
|
||||
if obs.now_ms < self.teardown_at_ms.unwrap_or(obs.now_ms) {
|
||||
// Absorb the grace tick so a receiver report still in flight
|
||||
// lands on a session that exists.
|
||||
return actions;
|
||||
}
|
||||
self.teardown_emitted = true;
|
||||
actions.push(ProbeAction::TeardownSession);
|
||||
}
|
||||
self.finished = true;
|
||||
self.ended_ms = Some(obs.now_ms);
|
||||
self.overall = self.compute_overall();
|
||||
actions.push(ProbeAction::Finish);
|
||||
actions
|
||||
}
|
||||
|
||||
fn compute_overall(&self) -> Overall {
|
||||
if self.overall == Overall::Cancelled {
|
||||
return Overall::Cancelled;
|
||||
}
|
||||
if self.bloom.is_failed() || self.discovery.is_failed() || self.session.is_failed() {
|
||||
return Overall::Failed;
|
||||
}
|
||||
if self.rtt.verdict == StageVerdict::Ok {
|
||||
return Overall::Ok;
|
||||
}
|
||||
Overall::Partial
|
||||
}
|
||||
}
|
||||
|
||||
impl FailKind {
|
||||
fn into_skip(self) -> StageRecord {
|
||||
StageRecord {
|
||||
verdict: StageVerdict::Skipped,
|
||||
reason: Some(self),
|
||||
detail: None,
|
||||
elapsed_ms: None,
|
||||
}
|
||||
}
|
||||
|
||||
fn into_fail(self) -> StageRecord {
|
||||
StageRecord {
|
||||
verdict: StageVerdict::Failed,
|
||||
reason: Some(self),
|
||||
detail: None,
|
||||
elapsed_ms: None,
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
fn ok_record() -> StageRecord {
|
||||
StageRecord {
|
||||
verdict: StageVerdict::Ok,
|
||||
reason: None,
|
||||
detail: None,
|
||||
elapsed_ms: None,
|
||||
}
|
||||
}
|
||||
|
||||
/// Discriminate the four rtt failure modes, which mean opposite things: only
|
||||
/// `NoReport` leaves reachability genuinely in doubt, and only
|
||||
/// `BadTimestampEcho` is an anomaly rather than an artifact.
|
||||
fn classify_rtt_failure(delta: RttCounters) -> FailKind {
|
||||
if delta.arith_fail > 0 {
|
||||
FailKind::BadTimestampEcho
|
||||
} else if delta.zero > 0 {
|
||||
FailKind::SubMillisecond
|
||||
} else if delta.reports_seen > 0 {
|
||||
FailKind::NoEcho
|
||||
} else {
|
||||
FailKind::NoReport
|
||||
}
|
||||
}
|
||||
|
||||
/// Pure LCA arithmetic over two coordinates.
|
||||
///
|
||||
/// `TreeCoordinate::lca` is the primary discriminator because it returns
|
||||
/// `None` for different roots. `lca_depth` alone returns `0` there via its
|
||||
/// `saturating_sub(1)`, indistinguishable from "the LCA is the root", so the
|
||||
/// core never keys off it directly.
|
||||
pub(crate) fn describe_path(my: &TreeCoordinate, their: &TreeCoordinate) -> PathFacts {
|
||||
let our_depth = my.depth();
|
||||
let their_depth = their.depth();
|
||||
let lca = my.lca(their).copied();
|
||||
let same_root = lca.is_some();
|
||||
let lca_depth = if same_root {
|
||||
Some(my.lca_depth(their))
|
||||
} else {
|
||||
None
|
||||
};
|
||||
let up = lca_depth.map(|d| our_depth.saturating_sub(d));
|
||||
let down = lca_depth.map(|d| their_depth.saturating_sub(d));
|
||||
PathFacts {
|
||||
our_coords: my.node_addrs().copied().collect(),
|
||||
their_coords: their.node_addrs().copied().collect(),
|
||||
our_depth,
|
||||
their_depth,
|
||||
same_root,
|
||||
lca,
|
||||
lca_depth,
|
||||
tree_hops_up: up,
|
||||
tree_hops_down: down,
|
||||
tree_distance: up.zip(down).map(|(u, d)| u + d),
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,53 @@
|
||||
//! Probe budget floors and latch counts.
|
||||
//!
|
||||
//! Every value here is either derived from a protocol constant or from the
|
||||
//! rx-loop tick period. Budgets are **monotonic** (never wall-clock) and
|
||||
//! tick-quantized: the shell computes each stage budget as
|
||||
//! `max(floor, N * tick_ms)` so a node with a long `tick_interval_secs` does
|
||||
//! not expire a stage between two consecutive observations.
|
||||
|
||||
/// Floor for the session (handshake) stage. `handshake_resend_interval_ms`
|
||||
/// is 1000 with `handshake_resend_backoff` 2.0, putting resends at t=1s and
|
||||
/// t=3s, so 5s covers two resends. The node's own `handshake_timeout_secs`
|
||||
/// is far longer; the probe cuts sooner on purpose.
|
||||
pub(crate) const SESSION_FLOOR_MS: u64 = 5_000;
|
||||
|
||||
/// Floor for the rtt stage. The remote's session receiver cold-starts at
|
||||
/// `SESSION_COLD_START_INTERVAL_MS` (1000 ms) and emits on its own tick, so
|
||||
/// 3s covers two remote ticks plus the cold start. It does not cover a peer
|
||||
/// sitting at the 10s report-interval ceiling; that is a deliberate trade.
|
||||
pub(crate) const RTT_FLOOR_MS: u64 = 3_000;
|
||||
|
||||
/// Ticks added to `sum(attempt_timeouts_secs)` for the discovery budget, so
|
||||
/// the ladder's own final attempt can land before the budget cuts it off.
|
||||
pub(crate) const RESOLVE_SLACK_TICKS: u64 = 2;
|
||||
|
||||
/// Ticks allowed for the bloom stage. Nothing here waits on the wire: the
|
||||
/// gate's answer is produced by the action the stage emits, and the shell
|
||||
/// performs that after the step returns, so the verdict lands on the next
|
||||
/// observation. Two ticks is one spare, and the total deadline backstops it.
|
||||
pub(crate) const BLOOM_MIN_TICKS: u64 = 2;
|
||||
|
||||
/// Minimum session-stage budget in ticks.
|
||||
pub(crate) const SESSION_MIN_TICKS: u64 = 3;
|
||||
|
||||
/// Minimum rtt-stage budget in ticks.
|
||||
pub(crate) const RTT_MIN_TICKS: u64 = 4;
|
||||
|
||||
/// Delay before the single allowed warmup retransmit. FSP rides UDP, so one
|
||||
/// lost datagram would otherwise leave the remote's `interval_has_data`
|
||||
/// unlatched and report `no_report` against a healthy peer. Two packets total
|
||||
/// is the whole allowance — resending every tick would make a diagnostic into
|
||||
/// a traffic source.
|
||||
pub(crate) const WARMUP_RETRY_MS: u64 = 1_000;
|
||||
|
||||
/// Ticks between the last observation and the teardown of a probe-created
|
||||
/// session. The remote may have a receiver report in flight; tearing down
|
||||
/// first makes that report land as an `UnknownSession` reject here.
|
||||
pub(crate) const TEARDOWN_GRACE_TICKS: u64 = 1;
|
||||
|
||||
/// How long a terminal job is retained so a late `probe_poll` can read it.
|
||||
pub(crate) const REAP_MS: u64 = 30_000;
|
||||
|
||||
/// Registry-wide cap on jobs in flight.
|
||||
pub(crate) const MAX_CONCURRENT_PROBES: usize = 4;
|
||||
@@ -0,0 +1,28 @@
|
||||
//! Sans-IO diagnostic-probe state.
|
||||
//!
|
||||
//! Pure, runtime-agnostic decisions for the `probe` diagnostic: a four-stage
|
||||
//! machine (resolve, path, session, rtt) driven one step per rx-loop tick. The
|
||||
//! async adapter lives in `node::handlers::probe`; the control-socket
|
||||
//! projection lives in `control::probe`.
|
||||
//!
|
||||
//! - `core.rs` — the [`Probe`] stage machine, its [`Observation`] input and
|
||||
//! [`ProbeAction`] output, and the pure [`describe_path`] LCA arithmetic.
|
||||
//! - `state.rs` — the stage/verdict/reason vocabulary and the plain-data
|
||||
//! preflight and report snapshots.
|
||||
//! - `limits.rs` — budget floors, latch counts, and the registry cap.
|
||||
|
||||
mod core;
|
||||
mod limits;
|
||||
mod state;
|
||||
|
||||
#[cfg(test)]
|
||||
mod tests;
|
||||
|
||||
pub(crate) use core::{Budgets, Observation, Probe, ProbeAction, describe_path};
|
||||
pub(crate) use limits::{MAX_CONCURRENT_PROBES, REAP_MS};
|
||||
#[cfg(test)]
|
||||
pub(crate) use state::StageVerdict;
|
||||
pub(crate) use state::{
|
||||
LeftIntact, LookupOutcomeKind, NextHopFacts, NoHopReason, Preflight, ProbeSnapshot,
|
||||
RttCounters, StageRecord,
|
||||
};
|
||||
@@ -0,0 +1,324 @@
|
||||
//! Plain data for the probe stage machine: stage identity, per-stage verdicts
|
||||
//! and their machine-stable discriminators, the coordinate-derived path facts,
|
||||
//! and the preflight/report snapshots the shell exchanges with the core.
|
||||
//!
|
||||
//! Every enum carries a `const fn name()` returning the exact string the
|
||||
//! control-socket JSON publishes, so the wire vocabulary has one home.
|
||||
|
||||
use crate::NodeAddr;
|
||||
use crate::proto::routing::RouteClass;
|
||||
|
||||
/// The five stages, in order, plus the terminal drain.
|
||||
#[derive(Copy, Clone, Debug, PartialEq, Eq)]
|
||||
pub(crate) enum Stage {
|
||||
/// Does any peer's announced filter claim the target, and did a request
|
||||
/// therefore go out? Separate from `Discovery` because the two fail for
|
||||
/// unrelated reasons: nobody claims the address, against nobody answers
|
||||
/// for it.
|
||||
Bloom,
|
||||
/// Waiting for a LookupResponse to put coordinates in the cache.
|
||||
Discovery,
|
||||
Path,
|
||||
Session,
|
||||
Rtt,
|
||||
/// Stages are done; teardown (if owed) and `Finish` remain.
|
||||
Terminal,
|
||||
}
|
||||
|
||||
/// Per-stage answer. `Pending` means the stage has not been reached.
|
||||
#[derive(Copy, Clone, Debug, PartialEq, Eq)]
|
||||
pub(crate) enum StageVerdict {
|
||||
Pending,
|
||||
Running,
|
||||
Ok,
|
||||
Skipped,
|
||||
Failed,
|
||||
}
|
||||
|
||||
impl StageVerdict {
|
||||
pub(crate) const fn name(self) -> &'static str {
|
||||
match self {
|
||||
StageVerdict::Pending => "pending",
|
||||
StageVerdict::Running => "running",
|
||||
StageVerdict::Ok => "ok",
|
||||
StageVerdict::Skipped => "skipped",
|
||||
StageVerdict::Failed => "failed",
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
/// Machine-stable failure or skip discriminator.
|
||||
#[derive(Copy, Clone, Debug, PartialEq, Eq)]
|
||||
pub(crate) enum FailKind {
|
||||
/// bloom and discovery skipped: coordinates already cached.
|
||||
Cached,
|
||||
/// bloom and discovery skipped: the target is a send-ready direct peer.
|
||||
DirectPeer,
|
||||
/// bloom: no peer's filter holds the target, so no request was sent.
|
||||
BloomMiss,
|
||||
/// bloom: this node is in post-failure lookup backoff.
|
||||
BackoffSuppressed,
|
||||
/// discovery: joined an in-flight lookup, which then failed.
|
||||
AlreadyPending,
|
||||
/// bloom: the gate proceeded but the fanout was zero.
|
||||
NoTreePeers,
|
||||
/// discovery: the attempt ladder was exhausted, or the budget expired.
|
||||
NoResponse,
|
||||
/// path: the two coordinates have different spanning-tree roots.
|
||||
DisjointTrees,
|
||||
/// path: no send-ready peer is strictly closer to the target.
|
||||
NoNextHop,
|
||||
/// session skipped: an entry already existed at action time.
|
||||
Preexisting,
|
||||
/// session: `initiate_session` returned an error.
|
||||
SendError,
|
||||
/// session: the budget expired with the handshake incomplete.
|
||||
HandshakeTimeout,
|
||||
/// rtt: no receiver report of any kind arrived.
|
||||
NoReport,
|
||||
/// rtt: reports arrived, none carried a usable timestamp echo.
|
||||
NoEcho,
|
||||
/// rtt: echoes arrived and every sample truncated to 0 ms.
|
||||
SubMillisecond,
|
||||
/// rtt: echo arithmetic failed — a real anomaly.
|
||||
BadTimestampEcho,
|
||||
/// the stage was skipped because an earlier stage failed.
|
||||
NotReached,
|
||||
}
|
||||
|
||||
impl FailKind {
|
||||
pub(crate) const fn name(self) -> &'static str {
|
||||
match self {
|
||||
FailKind::Cached => "cached",
|
||||
FailKind::DirectPeer => "direct_peer",
|
||||
FailKind::BloomMiss => "bloom_miss",
|
||||
FailKind::BackoffSuppressed => "backoff_suppressed",
|
||||
FailKind::AlreadyPending => "already_pending",
|
||||
FailKind::NoTreePeers => "no_tree_peers",
|
||||
FailKind::NoResponse => "no_response",
|
||||
FailKind::DisjointTrees => "disjoint_trees",
|
||||
FailKind::NoNextHop => "no_next_hop",
|
||||
FailKind::Preexisting => "preexisting",
|
||||
FailKind::SendError => "send_error",
|
||||
FailKind::HandshakeTimeout => "handshake_timeout",
|
||||
FailKind::NoReport => "no_report",
|
||||
FailKind::NoEcho => "no_echo",
|
||||
FailKind::SubMillisecond => "sub_millisecond",
|
||||
FailKind::BadTimestampEcho => "bad_timestamp_echo",
|
||||
FailKind::NotReached => "not_reached",
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
/// Where the target's coordinates came from.
|
||||
#[derive(Copy, Clone, Debug, PartialEq, Eq)]
|
||||
pub(crate) enum ResolveSource {
|
||||
Cache,
|
||||
Lookup,
|
||||
DirectPeer,
|
||||
}
|
||||
|
||||
impl ResolveSource {
|
||||
pub(crate) const fn name(self) -> &'static str {
|
||||
match self {
|
||||
ResolveSource::Cache => "cache",
|
||||
ResolveSource::Lookup => "lookup",
|
||||
ResolveSource::DirectPeer => "direct_peer",
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
/// One stage's record. `elapsed_ms` is filled when the stage leaves `Running`.
|
||||
#[derive(Clone, Debug, PartialEq, Eq)]
|
||||
pub(crate) struct StageRecord {
|
||||
pub(crate) verdict: StageVerdict,
|
||||
pub(crate) reason: Option<FailKind>,
|
||||
/// Free text for the one case carrying a real message: the error string
|
||||
/// out of `initiate_session`.
|
||||
pub(crate) detail: Option<String>,
|
||||
pub(crate) elapsed_ms: Option<u64>,
|
||||
}
|
||||
|
||||
impl StageRecord {
|
||||
pub(crate) fn pending() -> Self {
|
||||
Self {
|
||||
verdict: StageVerdict::Pending,
|
||||
reason: None,
|
||||
detail: None,
|
||||
elapsed_ms: None,
|
||||
}
|
||||
}
|
||||
|
||||
pub(crate) fn is_failed(&self) -> bool {
|
||||
self.verdict == StageVerdict::Failed
|
||||
}
|
||||
}
|
||||
|
||||
/// The probe's overall answer.
|
||||
#[derive(Copy, Clone, Debug, PartialEq, Eq)]
|
||||
pub(crate) enum Overall {
|
||||
Running,
|
||||
Ok,
|
||||
Partial,
|
||||
Failed,
|
||||
Cancelled,
|
||||
}
|
||||
|
||||
impl Overall {
|
||||
pub(crate) const fn name(self) -> &'static str {
|
||||
match self {
|
||||
// A job still in flight publishes its stages, not a verdict.
|
||||
Overall::Running => "running",
|
||||
Overall::Ok => "ok",
|
||||
Overall::Partial => "partial",
|
||||
Overall::Failed => "failed",
|
||||
Overall::Cancelled => "cancelled",
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
/// Coordinate-derived path facts. Purely local arithmetic over two
|
||||
/// coordinates — nothing here was observed on the wire.
|
||||
#[derive(Clone, Debug, PartialEq, Eq)]
|
||||
pub(crate) struct PathFacts {
|
||||
/// Self → root, as `TreeCoordinate` stores them.
|
||||
pub(crate) our_coords: Vec<NodeAddr>,
|
||||
pub(crate) their_coords: Vec<NodeAddr>,
|
||||
pub(crate) our_depth: usize,
|
||||
pub(crate) their_depth: usize,
|
||||
pub(crate) same_root: bool,
|
||||
pub(crate) lca: Option<NodeAddr>,
|
||||
/// Root-relative: 0 is the root.
|
||||
pub(crate) lca_depth: Option<usize>,
|
||||
/// Tree metric, not a predicted hop count.
|
||||
pub(crate) tree_hops_up: Option<usize>,
|
||||
pub(crate) tree_hops_down: Option<usize>,
|
||||
/// Upper bound on the real hop count: selection requires only strict
|
||||
/// progress, so a crosslink cut-through routinely delivers in fewer hops.
|
||||
pub(crate) tree_distance: Option<usize>,
|
||||
}
|
||||
|
||||
/// The first hop this node would select right now.
|
||||
#[derive(Clone, Debug, PartialEq, Eq)]
|
||||
pub(crate) struct NextHopFacts {
|
||||
pub(crate) node_addr: NodeAddr,
|
||||
pub(crate) class: RouteClass,
|
||||
pub(crate) direct_peer: bool,
|
||||
/// True when the class is a crosslink form: the first hop leaves the
|
||||
/// up-then-down walk `tree_hops_*` describes.
|
||||
pub(crate) leaves_tree_walk: bool,
|
||||
}
|
||||
|
||||
/// Why no next hop could be named.
|
||||
#[derive(Copy, Clone, Debug, PartialEq, Eq)]
|
||||
pub(crate) enum NoHopReason {
|
||||
Local,
|
||||
NoCoords,
|
||||
DisjointTrees,
|
||||
NoCloserPeer,
|
||||
HopNotSendReady,
|
||||
}
|
||||
|
||||
impl NoHopReason {
|
||||
pub(crate) const fn name(self) -> &'static str {
|
||||
match self {
|
||||
NoHopReason::Local => "local",
|
||||
NoHopReason::NoCoords => "no_coords",
|
||||
NoHopReason::DisjointTrees => "disjoint_trees",
|
||||
NoHopReason::NoCloserPeer => "no_closer_peer",
|
||||
NoHopReason::HopNotSendReady => "hop_not_send_ready",
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
/// What the gated lookup entry point decided, flattened for the core.
|
||||
#[derive(Copy, Clone, Debug, PartialEq, Eq)]
|
||||
pub(crate) enum LookupOutcomeKind {
|
||||
BloomMiss,
|
||||
Suppressed,
|
||||
Deduplicated,
|
||||
/// The gate proceeded but the request reached zero peers.
|
||||
ZeroFanout,
|
||||
/// The gate proceeded and the request went to at least one peer.
|
||||
Sent,
|
||||
}
|
||||
|
||||
/// Live state read once, at `probe_start`, before any stage runs.
|
||||
#[derive(Copy, Clone, Debug, Default)]
|
||||
pub(crate) struct Preflight {
|
||||
pub(crate) session_present: bool,
|
||||
pub(crate) coords_cached: bool,
|
||||
pub(crate) identity_cached: bool,
|
||||
/// Another probe already holds this target's ownership claim.
|
||||
pub(crate) target_claimed: bool,
|
||||
}
|
||||
|
||||
/// The four MMP report counters, sampled together.
|
||||
#[derive(Copy, Clone, Debug, Default, PartialEq, Eq)]
|
||||
pub(crate) struct RttCounters {
|
||||
pub(crate) reports_seen: u64,
|
||||
pub(crate) samples: u64,
|
||||
pub(crate) zero: u64,
|
||||
pub(crate) arith_fail: u64,
|
||||
}
|
||||
|
||||
impl RttCounters {
|
||||
/// Per-field saturating difference, so a rekey or a wrap cannot produce a
|
||||
/// negative delta that reads as "nothing happened".
|
||||
pub(crate) fn delta(self, base: RttCounters) -> RttCounters {
|
||||
RttCounters {
|
||||
reports_seen: self.reports_seen.saturating_sub(base.reports_seen),
|
||||
samples: self.samples.saturating_sub(base.samples),
|
||||
zero: self.zero.saturating_sub(base.zero),
|
||||
arith_fail: self.arith_fail.saturating_sub(base.arith_fail),
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
/// Why a session the probe touched was left in place.
|
||||
#[derive(Copy, Clone, Debug, PartialEq, Eq)]
|
||||
pub(crate) enum LeftIntact {
|
||||
Preexisting,
|
||||
AdoptedByTraffic,
|
||||
Replaced,
|
||||
}
|
||||
|
||||
impl LeftIntact {
|
||||
pub(crate) const fn name(self) -> &'static str {
|
||||
match self {
|
||||
LeftIntact::Preexisting => "preexisting",
|
||||
LeftIntact::AdoptedByTraffic => "adopted_by_traffic",
|
||||
LeftIntact::Replaced => "replaced",
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
/// Everything the control-socket projection needs from the core, as plain
|
||||
/// data. The core hands one of these out; it never serializes anything itself.
|
||||
#[derive(Clone, Debug)]
|
||||
pub(crate) struct ProbeSnapshot {
|
||||
pub(crate) overall: Overall,
|
||||
pub(crate) tick_ms: u64,
|
||||
pub(crate) bloom: StageRecord,
|
||||
pub(crate) discovery: StageRecord,
|
||||
pub(crate) path: StageRecord,
|
||||
pub(crate) session: StageRecord,
|
||||
pub(crate) rtt: StageRecord,
|
||||
pub(crate) resolve_source: Option<ResolveSource>,
|
||||
pub(crate) lookup_fanout: Option<usize>,
|
||||
pub(crate) path_facts: Option<PathFacts>,
|
||||
pub(crate) next_hop: Option<NextHopFacts>,
|
||||
pub(crate) no_hop_reason: Option<NoHopReason>,
|
||||
pub(crate) session_preexisting: bool,
|
||||
pub(crate) session_established: bool,
|
||||
pub(crate) path_mtu: Option<u32>,
|
||||
pub(crate) rtt_ms: Option<u32>,
|
||||
pub(crate) srtt_ms: Option<f64>,
|
||||
pub(crate) counters: RttCounters,
|
||||
pub(crate) torn_down: bool,
|
||||
pub(crate) left_intact: Option<LeftIntact>,
|
||||
pub(crate) lookup_issued: bool,
|
||||
pub(crate) coords_were_cached: bool,
|
||||
pub(crate) identity_was_cached: bool,
|
||||
pub(crate) warmups_sent: u8,
|
||||
}
|
||||
@@ -0,0 +1,752 @@
|
||||
//! Stage-machine tests. Each drives `Probe::step` with synthetic `now_ms` and
|
||||
//! hand-built `Observation`s and asserts on the returned action list.
|
||||
|
||||
use crate::NodeAddr;
|
||||
use crate::proto::probe::core::{Budgets, Observation, Probe, ProbeAction};
|
||||
use crate::proto::probe::state::{
|
||||
FailKind, LeftIntact, LookupOutcomeKind, NextHopFacts, PathFacts, Preflight, ResolveSource,
|
||||
RttCounters, StageVerdict,
|
||||
};
|
||||
use crate::proto::routing::RouteClass;
|
||||
|
||||
const T0: u64 = 1_000_000;
|
||||
|
||||
fn budgets() -> Budgets {
|
||||
// The default ladder [1,2,4,8] at a 1s tick.
|
||||
Budgets::derive(1_000, &[1, 2, 4, 8])
|
||||
}
|
||||
|
||||
fn addr(v: u8) -> NodeAddr {
|
||||
let mut bytes = [0u8; 16];
|
||||
bytes[0] = v;
|
||||
NodeAddr::from_bytes(bytes)
|
||||
}
|
||||
|
||||
/// A path that resolves cleanly, so the path stage never fails and never
|
||||
/// masks what a test is actually asserting on.
|
||||
fn good_path() -> PathFacts {
|
||||
PathFacts {
|
||||
our_coords: vec![addr(2), addr(1)],
|
||||
their_coords: vec![addr(3), addr(1)],
|
||||
our_depth: 1,
|
||||
their_depth: 1,
|
||||
same_root: true,
|
||||
lca: Some(addr(1)),
|
||||
lca_depth: Some(0),
|
||||
tree_hops_up: Some(1),
|
||||
tree_hops_down: Some(1),
|
||||
tree_distance: Some(2),
|
||||
}
|
||||
}
|
||||
|
||||
fn good_hop() -> NextHopFacts {
|
||||
NextHopFacts {
|
||||
node_addr: addr(3),
|
||||
class: RouteClass::DirectPeer,
|
||||
direct_peer: true,
|
||||
leaves_tree_walk: false,
|
||||
}
|
||||
}
|
||||
|
||||
/// An observation with everything healthy and nothing happening.
|
||||
fn obs(now_ms: u64) -> Observation {
|
||||
Observation {
|
||||
now_ms,
|
||||
coords_cached: true,
|
||||
lookup_pending: false,
|
||||
lookup_outcome: None,
|
||||
lookup_fanout: None,
|
||||
path: Some(good_path()),
|
||||
next_hop: Some(good_hop()),
|
||||
no_hop_reason: None,
|
||||
session_present: false,
|
||||
session_established: false,
|
||||
session_is_ours: false,
|
||||
session_error: None,
|
||||
target_is_direct_peer: false,
|
||||
counters: RttCounters::default(),
|
||||
last_rtt_ms: None,
|
||||
srtt_ms: None,
|
||||
path_mtu: None,
|
||||
}
|
||||
}
|
||||
|
||||
fn preflight() -> Preflight {
|
||||
Preflight {
|
||||
session_present: false,
|
||||
coords_cached: true,
|
||||
identity_cached: true,
|
||||
target_claimed: false,
|
||||
}
|
||||
}
|
||||
|
||||
/// Drive a probe from a fresh start to an established, owned session, and
|
||||
/// return it sitting at the top of the rtt stage.
|
||||
fn probe_at_rtt() -> (Probe, u64) {
|
||||
let mut probe = Probe::new(T0, budgets(), preflight());
|
||||
// Resolve (skipped, cached) then path, then session opens.
|
||||
let actions = probe.step(&obs(T0));
|
||||
assert_eq!(actions, vec![ProbeAction::OpenSession]);
|
||||
let mut o = obs(T0 + 1_000);
|
||||
o.session_present = true;
|
||||
o.session_established = true;
|
||||
o.session_is_ours = true;
|
||||
let actions = probe.step(&o);
|
||||
assert_eq!(actions, vec![ProbeAction::SendWarmup]);
|
||||
(probe, T0 + 1_000)
|
||||
}
|
||||
|
||||
// ---- bloom and discovery ------------------------------------------------------------
|
||||
|
||||
#[test]
|
||||
fn both_lookup_stages_are_skipped_when_coords_are_cached() {
|
||||
let mut probe = Probe::new(T0, budgets(), preflight());
|
||||
let actions = probe.step(&obs(T0));
|
||||
assert!(!actions.contains(&ProbeAction::InitiateLookup));
|
||||
let snap = probe.snapshot();
|
||||
// Neither stage ran, and both name the same reason: the second must not
|
||||
// invent a reason of its own for work the first already declined.
|
||||
assert_eq!(snap.bloom.verdict, StageVerdict::Skipped);
|
||||
assert_eq!(snap.bloom.reason, Some(FailKind::Cached));
|
||||
assert_eq!(snap.discovery.verdict, StageVerdict::Skipped);
|
||||
assert_eq!(snap.discovery.reason, Some(FailKind::Cached));
|
||||
assert_eq!(snap.resolve_source, Some(ResolveSource::Cache));
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn both_lookup_stages_are_skipped_for_a_direct_peer() {
|
||||
let mut pre = preflight();
|
||||
pre.coords_cached = false;
|
||||
let mut probe = Probe::new(T0, budgets(), pre);
|
||||
let mut o = obs(T0);
|
||||
o.coords_cached = false;
|
||||
o.target_is_direct_peer = true;
|
||||
let actions = probe.step(&o);
|
||||
assert!(!actions.contains(&ProbeAction::InitiateLookup));
|
||||
let snap = probe.snapshot();
|
||||
assert_eq!(snap.bloom.verdict, StageVerdict::Skipped);
|
||||
assert_eq!(snap.bloom.reason, Some(FailKind::DirectPeer));
|
||||
assert_eq!(snap.discovery.reason, Some(FailKind::DirectPeer));
|
||||
// The probe proceeds rather than declaring a neighbour unreachable.
|
||||
assert_eq!(actions, vec![ProbeAction::OpenSession]);
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn discovery_times_out_at_its_own_budget_measured_from_the_request() {
|
||||
let b = budgets();
|
||||
let mut pre = preflight();
|
||||
pre.coords_cached = false;
|
||||
let mut probe = Probe::new(T0, b, pre);
|
||||
|
||||
let mut o = obs(T0);
|
||||
o.coords_cached = false;
|
||||
assert_eq!(probe.step(&o), vec![ProbeAction::InitiateLookup]);
|
||||
|
||||
// The gate's answer arrives on the next observation and closes the bloom
|
||||
// stage; the discovery budget runs from there, not from the probe's start.
|
||||
let sent_ms = T0 + 1_000;
|
||||
let mut o = obs(sent_ms);
|
||||
o.coords_cached = false;
|
||||
o.lookup_pending = true;
|
||||
o.lookup_outcome = Some(LookupOutcomeKind::Sent);
|
||||
assert!(probe.step(&o).is_empty());
|
||||
assert_eq!(probe.snapshot().bloom.verdict, StageVerdict::Ok);
|
||||
assert_eq!(probe.snapshot().discovery.verdict, StageVerdict::Running);
|
||||
|
||||
// One millisecond short of the budget the machine must still be running.
|
||||
let mut o = obs(sent_ms + b.discovery_ms - 1);
|
||||
o.coords_cached = false;
|
||||
o.lookup_pending = true;
|
||||
assert!(probe.step(&o).is_empty());
|
||||
assert_eq!(probe.snapshot().discovery.verdict, StageVerdict::Running);
|
||||
|
||||
let mut o = obs(sent_ms + b.discovery_ms);
|
||||
o.coords_cached = false;
|
||||
o.lookup_pending = true;
|
||||
assert_eq!(probe.step(&o), vec![ProbeAction::Finish]);
|
||||
let snap = probe.snapshot();
|
||||
assert_eq!(snap.discovery.verdict, StageVerdict::Failed);
|
||||
assert_eq!(snap.discovery.reason, Some(FailKind::NoResponse));
|
||||
assert_eq!(
|
||||
snap.bloom.verdict,
|
||||
StageVerdict::Ok,
|
||||
"the request did go out"
|
||||
);
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn discovery_fails_early_when_the_pending_entry_clears() {
|
||||
let b = budgets();
|
||||
let mut pre = preflight();
|
||||
pre.coords_cached = false;
|
||||
let mut probe = Probe::new(T0, b, pre);
|
||||
|
||||
let mut o = obs(T0);
|
||||
o.coords_cached = false;
|
||||
probe.step(&o);
|
||||
|
||||
let mut o = obs(T0 + 1_000);
|
||||
o.coords_cached = false;
|
||||
o.lookup_pending = true;
|
||||
o.lookup_outcome = Some(LookupOutcomeKind::Sent);
|
||||
assert!(probe.step(&o).is_empty());
|
||||
|
||||
// The ladder gave up well inside the budget; the answer is already known.
|
||||
let mut o = obs(T0 + 2_000);
|
||||
o.coords_cached = false;
|
||||
o.lookup_pending = false;
|
||||
assert_eq!(probe.step(&o), vec![ProbeAction::Finish]);
|
||||
assert_eq!(
|
||||
probe.snapshot().discovery.reason,
|
||||
Some(FailKind::NoResponse)
|
||||
);
|
||||
assert!(T0 + 2_000 < T0 + b.discovery_ms);
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn each_gate_decision_lands_on_the_stage_that_owns_it() {
|
||||
let cases = [
|
||||
(LookupOutcomeKind::BloomMiss, FailKind::BloomMiss),
|
||||
(LookupOutcomeKind::Suppressed, FailKind::BackoffSuppressed),
|
||||
(LookupOutcomeKind::ZeroFanout, FailKind::NoTreePeers),
|
||||
(LookupOutcomeKind::Deduplicated, FailKind::AlreadyPending),
|
||||
];
|
||||
for (kind, expected) in cases {
|
||||
let mut pre = preflight();
|
||||
pre.coords_cached = false;
|
||||
let mut probe = Probe::new(T0, budgets(), pre);
|
||||
let mut o = obs(T0);
|
||||
o.coords_cached = false;
|
||||
probe.step(&o);
|
||||
|
||||
let mut o = obs(T0 + 1_000);
|
||||
o.coords_cached = false;
|
||||
o.lookup_outcome = Some(kind);
|
||||
o.lookup_pending = kind == LookupOutcomeKind::Deduplicated;
|
||||
probe.step(&o);
|
||||
|
||||
if kind == LookupOutcomeKind::Deduplicated {
|
||||
// Joined an in-flight lookup; it has to fail before the reason is
|
||||
// known, so clear the pending entry.
|
||||
let mut o = obs(T0 + 2_000);
|
||||
o.coords_cached = false;
|
||||
o.lookup_pending = false;
|
||||
probe.step(&o);
|
||||
}
|
||||
// A gate decision that stopped the request is the bloom stage's
|
||||
// finding; only the joined-lookup case gets far enough to be
|
||||
// discovery's.
|
||||
let snap = probe.snapshot();
|
||||
let actual = if kind == LookupOutcomeKind::Deduplicated {
|
||||
snap.discovery.reason
|
||||
} else {
|
||||
snap.bloom.reason
|
||||
};
|
||||
assert_eq!(
|
||||
actual,
|
||||
Some(expected),
|
||||
"outcome {kind:?} must not collapse into one reason"
|
||||
);
|
||||
}
|
||||
}
|
||||
|
||||
// ---- ownership ----------------------------------------------------------
|
||||
|
||||
#[test]
|
||||
fn session_skipped_when_entry_present_at_preflight() {
|
||||
let mut pre = preflight();
|
||||
pre.session_present = true;
|
||||
let mut probe = Probe::new(T0, budgets(), pre);
|
||||
|
||||
let mut saw_teardown = false;
|
||||
let mut saw_open = false;
|
||||
for tick in 0..40u64 {
|
||||
let mut o = obs(T0 + tick * 1_000);
|
||||
o.session_present = true;
|
||||
o.session_established = true;
|
||||
for a in probe.step(&o) {
|
||||
saw_open |= a == ProbeAction::OpenSession;
|
||||
saw_teardown |= a == ProbeAction::TeardownSession;
|
||||
}
|
||||
}
|
||||
assert!(!saw_open, "must never open an existing session");
|
||||
assert!(!saw_teardown, "must never tear down someone else's session");
|
||||
assert!(!probe.owns_session());
|
||||
assert_eq!(probe.snapshot().session.reason, Some(FailKind::Preexisting));
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn session_present_but_not_established_is_still_not_owned() {
|
||||
// The inbound-handshake case: an AwaitingMsg3 entry does not satisfy
|
||||
// production's established-or-initiating predicate, so a probe copying
|
||||
// that predicate would clobber it.
|
||||
let mut pre = preflight();
|
||||
pre.session_present = true;
|
||||
let mut probe = Probe::new(T0, budgets(), pre);
|
||||
|
||||
let mut saw_open = false;
|
||||
let mut saw_teardown = false;
|
||||
for tick in 0..40u64 {
|
||||
let mut o = obs(T0 + tick * 1_000);
|
||||
o.session_present = true;
|
||||
o.session_established = false;
|
||||
for a in probe.step(&o) {
|
||||
saw_open |= a == ProbeAction::OpenSession;
|
||||
saw_teardown |= a == ProbeAction::TeardownSession;
|
||||
}
|
||||
}
|
||||
assert!(!saw_open);
|
||||
assert!(!saw_teardown);
|
||||
assert!(!probe.owns_session());
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn session_appearing_mid_probe_is_never_owned() {
|
||||
let mut pre = preflight();
|
||||
pre.coords_cached = false;
|
||||
let mut probe = Probe::new(T0, budgets(), pre);
|
||||
|
||||
// Still resolving; no session anywhere.
|
||||
let mut o = obs(T0);
|
||||
o.coords_cached = false;
|
||||
assert_eq!(probe.step(&o), vec![ProbeAction::InitiateLookup]);
|
||||
|
||||
let mut saw_open = false;
|
||||
let mut saw_teardown = false;
|
||||
for tick in 1..40u64 {
|
||||
// A session appears before the probe ever asked for one.
|
||||
let mut o = obs(T0 + tick * 1_000);
|
||||
o.session_present = true;
|
||||
o.session_established = true;
|
||||
for a in probe.step(&o) {
|
||||
saw_open |= a == ProbeAction::OpenSession;
|
||||
saw_teardown |= a == ProbeAction::TeardownSession;
|
||||
}
|
||||
}
|
||||
assert!(!saw_open, "ownership must be latched at action time");
|
||||
assert!(!saw_teardown);
|
||||
assert!(!probe.owns_session());
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn ownership_is_dropped_when_our_entry_is_replaced() {
|
||||
let mut probe = Probe::new(T0, budgets(), preflight());
|
||||
assert_eq!(probe.step(&obs(T0)), vec![ProbeAction::OpenSession]);
|
||||
|
||||
// Confirm it was ours once...
|
||||
let mut o = obs(T0 + 1_000);
|
||||
o.session_present = true;
|
||||
o.session_established = true;
|
||||
o.session_is_ours = true;
|
||||
probe.step(&o);
|
||||
assert!(probe.owns_session());
|
||||
|
||||
// ...then the driver's identity check stops matching.
|
||||
let mut saw_teardown = false;
|
||||
for tick in 2..40u64 {
|
||||
let mut o = obs(T0 + tick * 1_000);
|
||||
o.session_present = true;
|
||||
o.session_established = true;
|
||||
o.session_is_ours = false;
|
||||
for a in probe.step(&o) {
|
||||
saw_teardown |= a == ProbeAction::TeardownSession;
|
||||
}
|
||||
}
|
||||
assert!(!saw_teardown, "must not remove a replacement entry");
|
||||
assert!(!probe.owns_session());
|
||||
assert_eq!(probe.snapshot().left_intact, Some(LeftIntact::Replaced));
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn owned_session_torn_down_on_every_terminal_path() {
|
||||
// 1. rtt succeeds.
|
||||
let (mut probe, t) = probe_at_rtt();
|
||||
let mut o = obs(t + 1_000);
|
||||
o.session_present = true;
|
||||
o.session_established = true;
|
||||
o.session_is_ours = true;
|
||||
o.counters = RttCounters {
|
||||
reports_seen: 1,
|
||||
samples: 1,
|
||||
zero: 0,
|
||||
arith_fail: 0,
|
||||
};
|
||||
o.last_rtt_ms = Some(18);
|
||||
probe.step(&o);
|
||||
assert_teardown_then_finish(&mut probe, t + 2_000, 40);
|
||||
|
||||
// 2. rtt times out.
|
||||
let (mut probe, t) = probe_at_rtt();
|
||||
assert_teardown_then_finish(&mut probe, t + 1_000, 40);
|
||||
|
||||
// 3. the handshake never completes. This path must keep its own loop: the
|
||||
// shared helper forces `session_established`, which would silently turn
|
||||
// this case into a repeat of case 2 and leave the probe's own half-open
|
||||
// Initiating entry untested. A leaked entry is worse than none, because
|
||||
// a later `initiate_session` then returns Ok on a dead session.
|
||||
let mut probe = Probe::new(T0, budgets(), preflight());
|
||||
assert_eq!(probe.step(&obs(T0)), vec![ProbeAction::OpenSession]);
|
||||
let mut seen: Vec<ProbeAction> = Vec::new();
|
||||
for tick in 1..40u64 {
|
||||
let mut o = obs(T0 + tick * 1_000);
|
||||
o.session_present = true;
|
||||
o.session_established = false;
|
||||
o.session_is_ours = true;
|
||||
seen.extend(probe.step(&o));
|
||||
}
|
||||
assert_eq!(
|
||||
seen.iter()
|
||||
.filter(|a| **a == ProbeAction::TeardownSession)
|
||||
.count(),
|
||||
1,
|
||||
"the half-open entry must be removed exactly once: {seen:?}"
|
||||
);
|
||||
let td = seen
|
||||
.iter()
|
||||
.position(|a| *a == ProbeAction::TeardownSession)
|
||||
.unwrap();
|
||||
let fin = seen
|
||||
.iter()
|
||||
.position(|a| *a == ProbeAction::Finish)
|
||||
.expect("must finish");
|
||||
assert!(td < fin, "teardown must precede finish: {seen:?}");
|
||||
assert_eq!(
|
||||
probe.snapshot().session.reason,
|
||||
Some(FailKind::HandshakeTimeout)
|
||||
);
|
||||
|
||||
// 4. cancel.
|
||||
let mut probe = Probe::new(T0, budgets(), preflight());
|
||||
assert_eq!(probe.step(&obs(T0)), vec![ProbeAction::OpenSession]);
|
||||
let mut o = obs(T0 + 1_000);
|
||||
o.session_present = true;
|
||||
o.session_established = true;
|
||||
o.session_is_ours = true;
|
||||
probe.step(&o);
|
||||
assert_eq!(
|
||||
probe.cancel(T0 + 2_000),
|
||||
vec![ProbeAction::TeardownSession, ProbeAction::Finish]
|
||||
);
|
||||
assert!(probe.cancel(T0 + 3_000).is_empty());
|
||||
}
|
||||
|
||||
/// Step forward until the probe finishes, asserting exactly one teardown and
|
||||
/// that it precedes `Finish`.
|
||||
fn assert_teardown_then_finish(probe: &mut Probe, from_ms: u64, ticks: u64) {
|
||||
let mut seen: Vec<ProbeAction> = Vec::new();
|
||||
for tick in 0..ticks {
|
||||
let mut o = obs(from_ms + tick * 1_000);
|
||||
o.session_present = true;
|
||||
o.session_established = true;
|
||||
o.session_is_ours = true;
|
||||
seen.extend(probe.step(&o));
|
||||
}
|
||||
let teardowns = seen
|
||||
.iter()
|
||||
.filter(|a| **a == ProbeAction::TeardownSession)
|
||||
.count();
|
||||
assert_eq!(teardowns, 1, "actions seen: {seen:?}");
|
||||
let td = seen
|
||||
.iter()
|
||||
.position(|a| *a == ProbeAction::TeardownSession)
|
||||
.unwrap();
|
||||
let fin = seen
|
||||
.iter()
|
||||
.position(|a| *a == ProbeAction::Finish)
|
||||
.expect("must finish");
|
||||
assert!(td < fin, "teardown must precede finish: {seen:?}");
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn a_session_the_probe_opened_is_never_reported_preexisting() {
|
||||
// The handshake normally spans several ticks, and every one of them sees
|
||||
// `session_present`. Reporting those as pre-existing would contradict the
|
||||
// teardown the same report carries.
|
||||
let mut probe = Probe::new(T0, budgets(), preflight());
|
||||
assert_eq!(probe.step(&obs(T0)), vec![ProbeAction::OpenSession]);
|
||||
|
||||
let mut o = obs(T0 + 1_000);
|
||||
o.session_present = true;
|
||||
o.session_established = false;
|
||||
o.session_is_ours = true;
|
||||
probe.step(&o);
|
||||
assert!(probe.owns_session());
|
||||
assert!(
|
||||
!probe.snapshot().session_preexisting,
|
||||
"an entry the probe opened is not pre-existing"
|
||||
);
|
||||
|
||||
let mut o = obs(T0 + 2_000);
|
||||
o.session_present = true;
|
||||
o.session_established = true;
|
||||
o.session_is_ours = true;
|
||||
probe.step(&o);
|
||||
assert!(!probe.snapshot().session_preexisting);
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn ownership_lost_before_confirmation_reads_as_replaced() {
|
||||
// Simultaneous initiation: the peer's inbound handshake replaces our entry
|
||||
// before the first observation that would have confirmed it was ours. The
|
||||
// probe did create one, so "preexisting" would be the wrong account.
|
||||
let mut probe = Probe::new(T0, budgets(), preflight());
|
||||
assert_eq!(probe.step(&obs(T0)), vec![ProbeAction::OpenSession]);
|
||||
|
||||
let mut o = obs(T0 + 1_000);
|
||||
o.session_present = true;
|
||||
o.session_established = true;
|
||||
o.session_is_ours = false;
|
||||
probe.step(&o);
|
||||
|
||||
assert!(!probe.owns_session());
|
||||
assert_eq!(probe.snapshot().left_intact, Some(LeftIntact::Replaced));
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn session_send_error_drops_ownership_and_carries_the_message() {
|
||||
let mut probe = Probe::new(T0, budgets(), preflight());
|
||||
assert_eq!(probe.step(&obs(T0)), vec![ProbeAction::OpenSession]);
|
||||
assert!(probe.owns_session());
|
||||
|
||||
let mut o = obs(T0 + 1_000);
|
||||
o.session_error = Some("no route to destination".to_string());
|
||||
let actions = probe.step(&o);
|
||||
|
||||
assert!(
|
||||
!actions.contains(&ProbeAction::TeardownSession),
|
||||
"a failed send created nothing to tear down: {actions:?}"
|
||||
);
|
||||
assert_eq!(actions, vec![ProbeAction::Finish]);
|
||||
assert!(!probe.owns_session());
|
||||
let snap = probe.snapshot();
|
||||
assert_eq!(snap.session.reason, Some(FailKind::SendError));
|
||||
assert_eq!(
|
||||
snap.session.detail.as_deref(),
|
||||
Some("no route to destination")
|
||||
);
|
||||
assert_eq!(snap.overall.name(), "failed");
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn direct_peer_with_no_coords_names_a_hop_and_no_missing_coords_reason() {
|
||||
// The routing short-circuit names a direct peer before reading the coord
|
||||
// cache, so the report must not carry a reason why no hop could be named
|
||||
// alongside a named hop.
|
||||
let mut pre = preflight();
|
||||
pre.coords_cached = false;
|
||||
let mut probe = Probe::new(T0, budgets(), pre);
|
||||
|
||||
let mut o = obs(T0);
|
||||
o.coords_cached = false;
|
||||
o.target_is_direct_peer = true;
|
||||
o.path = None;
|
||||
probe.step(&o);
|
||||
|
||||
let snap = probe.snapshot();
|
||||
assert!(snap.next_hop.is_some());
|
||||
assert_eq!(snap.no_hop_reason, None, "a named hop needs no excuse");
|
||||
}
|
||||
|
||||
// ---- rtt ----------------------------------------------------------------
|
||||
|
||||
#[test]
|
||||
fn rtt_requires_a_new_sample_not_a_present_srtt() {
|
||||
// A pre-existing session already carries an SRTT, and SRTT survives rekey,
|
||||
// so `srtt_ms.is_some()` is not attributable to this probe. Only a delta on
|
||||
// the accepted-sample counter is.
|
||||
let base = RttCounters {
|
||||
reports_seen: 5,
|
||||
samples: 5,
|
||||
zero: 0,
|
||||
arith_fail: 0,
|
||||
};
|
||||
let established = |now_ms: u64, counters: RttCounters| {
|
||||
let mut o = obs(now_ms);
|
||||
o.session_present = true;
|
||||
o.session_established = true;
|
||||
o.session_is_ours = true;
|
||||
o.counters = counters;
|
||||
o.srtt_ms = Some(12.0);
|
||||
o
|
||||
};
|
||||
|
||||
let mut probe = Probe::new(T0, budgets(), preflight());
|
||||
assert_eq!(probe.step(&obs(T0)), vec![ProbeAction::OpenSession]);
|
||||
// Entering the rtt stage latches the baseline from this observation.
|
||||
assert_eq!(
|
||||
probe.step(&established(T0 + 1_000, base)),
|
||||
vec![ProbeAction::SendWarmup]
|
||||
);
|
||||
|
||||
for tick in 2..4u64 {
|
||||
probe.step(&established(T0 + tick * 1_000, base));
|
||||
assert_eq!(probe.snapshot().rtt.verdict, StageVerdict::Running);
|
||||
}
|
||||
|
||||
let sampled = RttCounters { samples: 6, ..base };
|
||||
let mut o = established(T0 + 4_000, sampled);
|
||||
o.last_rtt_ms = Some(18);
|
||||
probe.step(&o);
|
||||
let snap = probe.snapshot();
|
||||
assert_eq!(snap.rtt.verdict, StageVerdict::Ok);
|
||||
assert_eq!(snap.rtt_ms, Some(18));
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn rtt_four_failure_modes_are_distinct() {
|
||||
let cases = [
|
||||
(RttCounters::default(), FailKind::NoReport),
|
||||
(
|
||||
RttCounters {
|
||||
reports_seen: 2,
|
||||
..RttCounters::default()
|
||||
},
|
||||
FailKind::NoEcho,
|
||||
),
|
||||
(
|
||||
RttCounters {
|
||||
reports_seen: 2,
|
||||
zero: 2,
|
||||
..RttCounters::default()
|
||||
},
|
||||
FailKind::SubMillisecond,
|
||||
),
|
||||
(
|
||||
RttCounters {
|
||||
reports_seen: 2,
|
||||
arith_fail: 2,
|
||||
..RttCounters::default()
|
||||
},
|
||||
FailKind::BadTimestampEcho,
|
||||
),
|
||||
];
|
||||
for (counters, expected) in cases {
|
||||
let (mut probe, t) = probe_at_rtt();
|
||||
for tick in 1..12u64 {
|
||||
let mut o = obs(t + tick * 1_000);
|
||||
o.session_present = true;
|
||||
o.session_established = true;
|
||||
o.session_is_ours = true;
|
||||
o.counters = counters;
|
||||
probe.step(&o);
|
||||
}
|
||||
assert_eq!(
|
||||
probe.snapshot().rtt.reason,
|
||||
Some(expected),
|
||||
"counters {counters:?} must not be conflated"
|
||||
);
|
||||
}
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn warmup_sent_at_most_twice() {
|
||||
let (mut probe, t) = probe_at_rtt();
|
||||
let mut warmups = 1; // the entry warmup, from probe_at_rtt
|
||||
for tick in 1..11u64 {
|
||||
let mut o = obs(t + tick * 1_000);
|
||||
o.session_present = true;
|
||||
o.session_established = true;
|
||||
o.session_is_ours = true;
|
||||
warmups += probe
|
||||
.step(&o)
|
||||
.iter()
|
||||
.filter(|a| **a == ProbeAction::SendWarmup)
|
||||
.count();
|
||||
}
|
||||
assert_eq!(warmups, 2, "a diagnostic must not become a traffic source");
|
||||
|
||||
// A report arriving after the first warmup cancels the retransmit.
|
||||
let (mut probe, t) = probe_at_rtt();
|
||||
let mut warmups = 1;
|
||||
for tick in 1..11u64 {
|
||||
let mut o = obs(t + tick * 1_000);
|
||||
o.session_present = true;
|
||||
o.session_established = true;
|
||||
o.session_is_ours = true;
|
||||
o.counters = RttCounters {
|
||||
reports_seen: 1,
|
||||
..RttCounters::default()
|
||||
};
|
||||
warmups += probe
|
||||
.step(&o)
|
||||
.iter()
|
||||
.filter(|a| **a == ProbeAction::SendWarmup)
|
||||
.count();
|
||||
}
|
||||
assert_eq!(warmups, 1);
|
||||
}
|
||||
|
||||
// ---- verdicts and budgets ----------------------------------------------
|
||||
|
||||
#[test]
|
||||
fn overall_ok_when_all_stages_pass() {
|
||||
let (mut probe, t) = probe_at_rtt();
|
||||
let mut o = obs(t + 1_000);
|
||||
o.session_present = true;
|
||||
o.session_established = true;
|
||||
o.session_is_ours = true;
|
||||
o.counters = RttCounters {
|
||||
reports_seen: 1,
|
||||
samples: 1,
|
||||
zero: 0,
|
||||
arith_fail: 0,
|
||||
};
|
||||
o.last_rtt_ms = Some(18);
|
||||
probe.step(&o);
|
||||
assert_teardown_then_finish(&mut probe, t + 2_000, 10);
|
||||
assert_eq!(probe.snapshot().overall.name(), "ok");
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn overall_partial_when_session_ok_and_rtt_failed() {
|
||||
let (mut probe, t) = probe_at_rtt();
|
||||
assert_teardown_then_finish(&mut probe, t + 1_000, 40);
|
||||
assert_eq!(probe.snapshot().overall.name(), "partial");
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn overall_failed_when_session_failed() {
|
||||
let mut probe = Probe::new(T0, budgets(), preflight());
|
||||
assert_eq!(probe.step(&obs(T0)), vec![ProbeAction::OpenSession]);
|
||||
// The handshake never completes.
|
||||
for tick in 1..40u64 {
|
||||
let mut o = obs(T0 + tick * 1_000);
|
||||
o.session_present = true;
|
||||
o.session_is_ours = true;
|
||||
probe.step(&o);
|
||||
}
|
||||
assert_eq!(
|
||||
probe.snapshot().session.reason,
|
||||
Some(FailKind::HandshakeTimeout)
|
||||
);
|
||||
assert_eq!(probe.snapshot().overall.name(), "failed");
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn total_deadline_forces_terminal() {
|
||||
let b = budgets();
|
||||
let mut pre = preflight();
|
||||
pre.coords_cached = false;
|
||||
let mut probe = Probe::new(T0, b, pre);
|
||||
let mut o = obs(T0);
|
||||
o.coords_cached = false;
|
||||
probe.step(&o);
|
||||
|
||||
// Jump straight past the total budget while still in resolve.
|
||||
let mut o = obs(T0 + b.total_ms() + 1);
|
||||
o.coords_cached = false;
|
||||
o.lookup_pending = true;
|
||||
assert_eq!(probe.step(&o), vec![ProbeAction::Finish]);
|
||||
assert!(probe.is_finished());
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn budgets_scale_with_tick() {
|
||||
// A node with a 10s tick would otherwise expire every stage between two
|
||||
// consecutive observations.
|
||||
let b = Budgets::derive(10_000, &[1, 2, 4, 8]);
|
||||
assert!(b.session_ms >= 30_000, "session_ms = {}", b.session_ms);
|
||||
assert!(b.rtt_ms >= 40_000, "rtt_ms = {}", b.rtt_ms);
|
||||
assert_eq!(b.discovery_ms, 15_000 + 2 * 10_000);
|
||||
assert_eq!(b.bloom_ms, 2 * 10_000);
|
||||
}
|
||||
@@ -0,0 +1,6 @@
|
||||
//! Sans-IO probe tests: the stage machine driven with synthetic timestamps and
|
||||
//! hand-built observations, and the pure LCA arithmetic. No runtime, no
|
||||
//! sleeps, no network.
|
||||
|
||||
mod core;
|
||||
mod path;
|
||||
@@ -0,0 +1,75 @@
|
||||
//! Pure LCA arithmetic. Expected values are derived from
|
||||
//! `TreeCoordinate::depth()`, which is `entries - 1`, and worked out in each
|
||||
//! test's comment — never read off a rendered example.
|
||||
|
||||
use crate::NodeAddr;
|
||||
use crate::proto::probe::core::describe_path;
|
||||
use crate::proto::stp::TreeCoordinate;
|
||||
|
||||
fn addr(v: u8) -> NodeAddr {
|
||||
let mut bytes = [0u8; 16];
|
||||
bytes[0] = v;
|
||||
NodeAddr::from_bytes(bytes)
|
||||
}
|
||||
|
||||
/// Coordinates are stored self → root.
|
||||
fn coord(path: &[u8]) -> TreeCoordinate {
|
||||
TreeCoordinate::from_addrs(path.iter().copied().map(addr).collect()).unwrap()
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn path_facts_same_tree() {
|
||||
// ours [4d, 91, 7c] : self 4d, parent 91, root 7c depth 2
|
||||
// theirs [4a, e5, 91, 7c] : self 4a, e5, 91, root 7c depth 3
|
||||
// common suffix is [7c, 91], so the LCA is 91 at root-relative depth 1.
|
||||
// up = our_depth - lca_depth = 2 - 1 = 1
|
||||
// down = their_depth - lca_depth = 3 - 1 = 2
|
||||
let facts = describe_path(
|
||||
&coord(&[0x4d, 0x91, 0x7c]),
|
||||
&coord(&[0x4a, 0xe5, 0x91, 0x7c]),
|
||||
);
|
||||
assert!(facts.same_root);
|
||||
assert_eq!(facts.our_depth, 2);
|
||||
assert_eq!(facts.their_depth, 3);
|
||||
assert_eq!(facts.lca, Some(addr(0x91)));
|
||||
assert_eq!(facts.lca_depth, Some(1));
|
||||
assert_eq!(facts.tree_hops_up, Some(1));
|
||||
assert_eq!(facts.tree_hops_down, Some(2));
|
||||
assert_eq!(facts.tree_distance, Some(3));
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn path_facts_different_roots_is_disjoint_not_depth_zero() {
|
||||
// `lca_depth` alone returns 0 here through its `saturating_sub(1)`, which
|
||||
// would render as "the LCA is the root" — a wrong answer that looks
|
||||
// plausible. `lca()` returning None is the only sound discriminator.
|
||||
let facts = describe_path(&coord(&[0x11, 0x01]), &coord(&[0x22, 0x02]));
|
||||
assert!(!facts.same_root);
|
||||
assert_eq!(facts.lca, None);
|
||||
assert_eq!(facts.lca_depth, None);
|
||||
assert_eq!(facts.tree_hops_up, None);
|
||||
assert_eq!(facts.tree_hops_down, None);
|
||||
assert_eq!(facts.tree_distance, None);
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn path_facts_ancestor_pair() {
|
||||
// theirs [91, 7c] is a strict ancestor of ours [4d, 91, 7c]:
|
||||
// the LCA is 91 itself, at depth 1, so nothing goes back down.
|
||||
let facts = describe_path(&coord(&[0x4d, 0x91, 0x7c]), &coord(&[0x91, 0x7c]));
|
||||
assert_eq!(facts.lca, Some(addr(0x91)));
|
||||
assert_eq!(facts.lca_depth, Some(1));
|
||||
assert_eq!(facts.tree_hops_up, Some(1));
|
||||
assert_eq!(facts.tree_hops_down, Some(0));
|
||||
assert_eq!(facts.tree_distance, Some(1));
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn depth_matches_entry_count_minus_one() {
|
||||
let facts = describe_path(
|
||||
&coord(&[0x4d, 0x91, 0x7c]),
|
||||
&coord(&[0x4a, 0xe5, 0x91, 0x7c]),
|
||||
);
|
||||
assert_eq!(facts.our_depth, facts.our_coords.len() - 1);
|
||||
assert_eq!(facts.their_depth, facts.their_coords.len() - 1);
|
||||
}
|
||||
@@ -273,6 +273,20 @@ pub(crate) enum RouteClass {
|
||||
DirectPeer,
|
||||
}
|
||||
|
||||
impl RouteClass {
|
||||
/// The stable string form, for diagnostics that publish the class.
|
||||
pub(crate) const fn name(self) -> &'static str {
|
||||
match self {
|
||||
RouteClass::TreeUp => "tree_up",
|
||||
RouteClass::TreeDown => "tree_down",
|
||||
RouteClass::TreeDownCross => "tree_down_cross",
|
||||
RouteClass::CrosslinkDescend => "crosslink_descend",
|
||||
RouteClass::CrosslinkAscend => "crosslink_ascend",
|
||||
RouteClass::DirectPeer => "direct_peer",
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
/// Select the best next hop from the active peers that may reach `dest`.
|
||||
///
|
||||
/// Enumerates borrowed peers through [`RoutingView`], applies the bloom and
|
||||
|
||||
Reference in New Issue
Block a user