mirror of
https://github.com/jmcorgan/fips.git
synced 2026-10-05 11:08:25 +00:00
feat(peer): a peer with a session is never dialled; a handshake creates no path state
Three ways an address for a peer we already hold a session with used to reach the dialler — a beacon on a new transport, `update_peers` or `fipsctl connect`, a configured address whose transport came up later — and each was a second handshake, which the far side read as a rekey and this side resolved as a cross-connection, the two not composing. Two phones hearing the same Wi-Fi return at the same moment both dialled at once; one side swapped to its outbound session and freed the index it had just handed out in the rekey reply, the other kept its inbound session and the pre-rekey index, every frame between them was dropped, and the link-dead reap tore the peer down. About a minute dark on every Wi-Fi return. A peer that holds a session is never dialled now. An address on a transport it has no path over becomes a path candidate under that session; one on a transport whose path is not eligible re-points that path (the active path included: it is not answering, that is why we are here); one on a transport whose path is carrying acknowledged traffic changes nothing. The heartbeat tick probes the candidate under the existing session — one authenticated, replay-checked round trip — and the mandatory switch takes it if the current path stops answering. Nothing is lost against the dial: a session that is truly gone answers no probe either, is reaped by the link-dead timeout, and is dialled then; a peer that restarted dials us with a new epoch and wins promotion outright, as before. Applies to the control API's connect, to update_peers, to configured addresses (checked once a tick) and to transport discovery alike. The counterpart: a handshake creates no path state. A dial that does reach a peer with a session — a startup that lists two addresses dials both, a caller that still dials by hand, an older node dialling us — is classified and resolved exactly as before this work: rekey, duplicate or restart on the responder, whichever transport the msg1 arrived on; the cross-connection tie-break on the initiator. The address it ran to is left as a candidate for the probe exchange. Two reasons. Both ends must resolve a handshake on the same information, and "is this a new transport to a live peer" was a fact only one end could see. And the IK responder commits at msg1, which carries no freshness beyond the startup epoch: a captured msg1 replayed from any address would otherwise have planted a path, probed full-size for the life of the peering and counting as a transport the peer is on for the decrypt-failure gate. So that gate now counts garbage only on the active path or one the peer has acknowledged. On a connection-oriented transport the connection a dial opened is kept as the candidate's socket rather than closed as the losing leg: the probe rides it, and closing it would only have the first probe dial again — or, at the responder, find an ephemeral port that cannot be dialled at all. `api_disconnect` closes every path's connection, the standby's included; loopback records the closes it is asked for so a test can say so. Path heartbeats are gated and bounded. A peer with one live path is not path-heartbeated: selection has nothing to move to, the link heartbeat keeps liveness, and five probes a second on every single-path link was cost without a decision behind it. A standby the peer never acknowledges is given up after eight discovery probes, Dead and pruned after the grace; the active path is never given up. The active path's first probe is small, the handshake having proved it and seeded its MTU. And a Dead path is probed again when its transport returns: nothing on our side ever re-probed one, so after a NIC replug traffic stayed on the standby until the grace pruned the path and a beacon found it with no history. The presence edge now revives every Dead path on the transport as Probing, RTT window and ETX kept. Smaller: `add_path_candidate` re-points a known transport's path at a moved address (`refresh_path_addr`), for a Wi-Fi Aware data path that re-forms with a new link-local; `api_disconnect` closes every path's connection, not the active one alone; `path_show` is built from the `show_peers` path projection plus the three now-relative fields; `PathState` and `TransportRole` render through `as_str()`; `node.path.switch_margin` is validated finite and at least 1.0; `PathPolicy::PERMISSIVE` had no users; four doc comments an inserted function had split are put back on the function they describe. The dual-udp-flap scenario is config-driven: the dial owner lists udp/main and udp/<veth>, both dial at startup, and the second is proven as a path under the first's session by the probe exchange.
This commit is contained in:
@@ -44,6 +44,23 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
|
||||
a path tells the peer with a `PathClose` on a surviving path, so the
|
||||
peer moves at once rather than after its own timeout.
|
||||
|
||||
- Connect semantics, for multi-path. A peer that holds a session is never
|
||||
dialled again: an address for it on a transport it has no path over —
|
||||
from a beacon, `update_peers`, `fipsctl connect`, a runtime peer lane,
|
||||
or a configured address whose transport came up later — becomes a
|
||||
candidate path, probed under the session by the next heartbeat tick;
|
||||
one on a transport whose path has stopped answering re-points that
|
||||
path; one on a transport whose path carries acknowledged traffic changes
|
||||
nothing. A handshake never creates path state: a second handshake to a
|
||||
peer with a session (two configured addresses dialled at startup)
|
||||
resolves as it always did — rekey, duplicate or cross-connection
|
||||
tie-break, whichever transport it ran over — and the address it ran to
|
||||
is left as a candidate for the probe exchange. A standby the peer never
|
||||
answers on is given up after eight probes, and until it is answered it
|
||||
does not count as a transport the peer is on for the decrypt-failure
|
||||
gate. On a connection-oriented transport the connection a dial opened
|
||||
is kept as the candidate's socket rather than closed.
|
||||
|
||||
- Operator surface, for multi-path: `role: backup` on any transport (never
|
||||
carries a peer's traffic while a normal path is eligible);
|
||||
`fipsctl path show|pin|unpin` and the `path_show`, `path_pin`,
|
||||
|
||||
@@ -116,7 +116,8 @@ table below lists every command currently registered.
|
||||
| ------- | ------ | ----------------------------- |
|
||||
| `show_status` | — | `version`, `npub`, `node_addr`, `ipv6_addr`, `state`, `is_leaf_only`, `is_root` (bool — this node is the spanning-tree root), `root` (hex node-addr of the current tree root), `persistent` (bool — identity is persisted, i.e. `persistent` set or an `nsec` configured), `peer_count`, `session_count`, `link_count`, `transport_count`, `connection_count`, `transport_peer_counts` (object mapping transport-type name to its connected-peer count; configured transports appear with `0`), `tun_state`, `tun_name`, `effective_ipv6_mtu`, `control_socket`, `pid`, `exe_path`, `uptime_secs`, `estimated_mesh_size`, `forwarding`, `sparklines`. |
|
||||
| `show_acl` | — | `allow_file`, `deny_file`, `enforcement_active`, `effective_mode`, `default_decision`, `allow_all`, `deny_all`, `allow_file_entries`, `deny_file_entries`, `allow_entries`, `deny_entries`. |
|
||||
| `show_peers` | — | `peers[]` — per-peer object: `node_addr`, `npub`, `display_name`, `ipv6_addr`, `connectivity`, `link_id`, `direction`, `transport_addr`, `transport_type`, `is_parent`, `is_child`, `tree_depth`, `effective_depth` (`tree_depth + link_cost` — the metric `evaluate_parent` ranks on; `null` when the peer has no coords, or is unmeasured while another peer has an SRTT sample, per the cold-start gate), `stats`, `noise`, `current_k_bit`, `mmp`, `paths[]` (every path to the peer: `transport_id`, `transport`, `transport_type`, `addr`, `state`, `active`, `remote_active`, `role`, `pinned`, `last_rtt_ms`, `min_rtt_ms`, `rtt_samples`, `etx`, `score` — the `path_show` fields minus the now-relative ages), plus optional `nostr_traversal`, `rekey_in_progress`, `rekey_draining`. |
|
||||
| `show_peers` | — | `peers[]` — per-peer object: `node_addr`, `npub`, `display_name`, `ipv6_addr`, `connectivity`, `link_id`, `direction`, `transport_addr`, `transport_type`, `is_parent`, `is_child`, `tree_depth`, `effective_depth` (`tree_depth + link_cost` — the metric `evaluate_parent` ranks on; `null` when the peer has no coords, or is unmeasured while another peer has an SRTT sample, per the cold-start gate), `stats`, `noise`, `current_k_bit`, `mmp`, `paths[]` (every path to the peer: `transport_id`, `transport` (instance name or null), `transport_type`, `addr`, `state` (`probing` / `live` / `suspect` / `dead`), `active`, `remote_active`, `role` (`normal` / `backup`), `pinned`, `last_rtt_ms`, `min_rtt_ms`, `rtt_samples`, `etx`, `score`), plus optional `nostr_traversal`, `rekey_in_progress`, `rekey_draining`. |
|
||||
| `path_show` | `npub` (bech32) | Every path to one peer. `data`: `peer`, `link_cost`, `link_cost_held`, and `paths[]` — the `show_peers` per-path object plus `rx_live_ms_ago`, `tx_live_ms_ago` (ms since the last authentic frame heard there / the last ack proving the peer hears us there; `null` if never) and `acked_once`. Takes a parameter, so it is served on the daemon's main task like the mutating commands, not from the snapshot. |
|
||||
| `show_links` | — | `links[]` — `link_id`, `transport_id`, `remote_addr`, `direction`, `state`, `created_at_ms`, `stats`. |
|
||||
| `show_tree` | — | `my_node_addr`, `root`, `root_npub` (bech32 npub of the current tree root), `is_root`, `depth`, `my_coords[]`, `parent`, `parent_display_name`, `declaration_sequence`, `declaration_signed`, `peer_tree_count`, `peers[]`, `stats`. |
|
||||
| `show_sessions` | — | `sessions[]` — `remote_addr`, `npub`, `display_name`, `state` (`established`, `initiating`, `awaiting_msg3`, `unknown`), `is_initiator`, `last_activity_ms`, `stats`, optional `mmp`, `current_k_bit`, `is_draining`. |
|
||||
@@ -173,17 +174,24 @@ not reproduced here to avoid duplicating the source.
|
||||
| `probe_start` | `npub` (bech32) | Admits a diagnostic probe job and returns immediately. `data`: `probe_id`, `npub`, `node_addr`, `display_name`, `budget_ms`. |
|
||||
| `probe_poll` | `probe_id` (integer) | Reports a probe's progress. `data`: `state` (`running` / `done`) and `report`. A terminal job is removed on the poll that observes it, so the report is delivered once. |
|
||||
| `probe_cancel` | `probe_id` (integer) | Runs the probe's terminal actions immediately, without the teardown grace tick. |
|
||||
| `path_show` | `npub` (bech32) | Every path to the peer. `data`: `peer`, `link_cost`, `link_cost_held`, and `paths[]` with `transport_id`, `transport` (instance name or null), `addr`, `state`, `active`, `remote_active`, `role`, `pinned`, `rx_live_ms_ago`, `tx_live_ms_ago`, `acked_once`, `last_rtt_ms`, `min_rtt_ms`, `rtt_samples`, `etx`, `score`. |
|
||||
| `path_pin` | `npub` (bech32), `transport` (instance name or numeric id) | Pins this node's traffic to the peer to that transport's path. Applies on the next selection tick. Error if the peer has no path there. |
|
||||
| `path_unpin` | `npub` (bech32) | Clears the pin. |
|
||||
| `path_pin` | `npub` (bech32), `transport` (instance name or numeric id) | Pins this node's traffic to the peer to that transport's path. Applies on the next selection tick, and is suspended while that path is not eligible and re-applied when it is again. `data`: `{"pinned": <transport_id>}`. Error if the peer has no path there. |
|
||||
| `path_unpin` | `npub` (bech32) | Clears the pin. `data`: `{"pinned": null}`. |
|
||||
|
||||
`connect` on a peer the node is **already connected to** neither tears the
|
||||
live link down nor ignores the address: the address is tried as an alternate
|
||||
path alongside the existing one, and the peer moves to it only if that
|
||||
handshake authenticates. The response carries `refreshed` — `true` when such a
|
||||
handshake was started, `false` when the peer is already on this exact path and
|
||||
that path is fresh (a successful no-op). A `connect` that starts an ordinary
|
||||
dial to a peer the node does not yet hold also reports `refreshed: false`.
|
||||
`connect` has three outcomes, told apart by whether the node already holds
|
||||
a session with the peer and by the response's `refreshed` field:
|
||||
|
||||
- **No session:** an ordinary dial over the named transport. `refreshed:
|
||||
false`; the peer appears in `show_peers` once the handshake completes.
|
||||
- **Session, and the address is on a transport the peer has no path over,
|
||||
or one whose path has stopped answering:** no handshake. The address
|
||||
becomes a path candidate under the existing session (or re-points the
|
||||
unanswering path), the next heartbeat tick probes it, and selection
|
||||
moves traffic to it if it measures better or the current path stops
|
||||
answering. `refreshed: true`. `path_show` lists it as `probing` until
|
||||
the peer acknowledges, `live` after.
|
||||
- **Session, and the peer is already reachable at exactly that address,
|
||||
or that transport's path is carrying acknowledged traffic:** nothing
|
||||
changes. `refreshed: false`.
|
||||
|
||||
`connect` is ephemeral either way: the peer is not written to the config file
|
||||
and gets no auto-reconnect, so an attempt that fails leaves no residue.
|
||||
|
||||
@@ -1241,6 +1241,18 @@ impl Config {
|
||||
}
|
||||
}
|
||||
|
||||
// Path selection. The margin is the whole fail-back policy: a
|
||||
// standby must beat the active path's score by this factor before
|
||||
// traffic moves. Below 1.0 (or NaN, which compares false both ways)
|
||||
// two paths of equal score would swap after every dwell, each swap
|
||||
// re-seeding the path MTU and holding the tree-visible link cost.
|
||||
let margin = self.node.path.switch_margin;
|
||||
if !margin.is_finite() || margin < 1.0 {
|
||||
return Err(ConfigError::Validation(format!(
|
||||
"`node.path.switch_margin` = {margin} must be a finite number of at least 1.0: it is the factor a standby's score must beat the active path's by, and anything less makes two equal paths swap after every dwell"
|
||||
)));
|
||||
}
|
||||
|
||||
let native = &self.node.native_api;
|
||||
// Both floors refuse a node that would start, answer every setup call
|
||||
// and then drop every datagram a peer sent. A zero `backlog` makes the
|
||||
@@ -2557,6 +2569,20 @@ node:
|
||||
assert!(config.node.discovery.is_none());
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn a_switch_margin_below_one_is_refused() {
|
||||
// Two paths of equal score would swap after every dwell.
|
||||
for bad in [0.9, 0.0, -1.0, f64::NAN, f64::INFINITY] {
|
||||
let mut config = Config::default();
|
||||
config.node.path.switch_margin = bad;
|
||||
let err = config.validate().expect_err("validation should fail");
|
||||
assert!(err.to_string().contains("switch_margin"), "{bad}: {err}");
|
||||
}
|
||||
let mut config = Config::default();
|
||||
config.node.path.switch_margin = 1.0;
|
||||
config.validate().expect("1.0 means any better path wins");
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn test_a_zero_netmon_poll_interval_is_refused() {
|
||||
// It was silently clamped to 1s, so a typo produced a node that polled
|
||||
|
||||
@@ -58,6 +58,18 @@ pub enum TransportRole {
|
||||
Backup,
|
||||
}
|
||||
|
||||
impl TransportRole {
|
||||
/// The config and control-socket spelling: `normal`, `backup`. Same
|
||||
/// strings serde reads and writes, fixed here so a variant rename cannot
|
||||
/// silently change what `show_peers` and `path_show` emit.
|
||||
pub fn as_str(self) -> &'static str {
|
||||
match self {
|
||||
Self::Normal => "normal",
|
||||
Self::Backup => "backup",
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
/// UDP transport instance configuration.
|
||||
#[derive(Debug, Clone, Default, Serialize, Deserialize)]
|
||||
#[serde(deny_unknown_fields)]
|
||||
|
||||
@@ -408,13 +408,9 @@ pub fn show_peers(node: &Node) -> Value {
|
||||
json!({ "peers": peers })
|
||||
}
|
||||
|
||||
/// Render a snapshot [`EntityMmp`](super::snapshot::EntityMmp) into the inline
|
||||
/// MMP JSON block, with the quality-index key named `quality_key` (`lqi` for
|
||||
/// peers, `sqi` for sessions). Reproduces the on-loop key insertion order
|
||||
/// exactly. `path_mtu` is emitted (inside the leading literal) only when
|
||||
/// present (session-layer); for peers it is `None` and omitted.
|
||||
/// Render a peer's path rows as the `paths` array of `show_peers`.
|
||||
fn render_peer_paths(paths: &[super::snapshot::PeerPathRow]) -> Value {
|
||||
/// Render a peer's path rows as the `paths` array of `show_peers`. Also the
|
||||
/// base of `path_show`'s rows, which add the now-relative fields on top.
|
||||
pub(crate) fn render_peer_paths(paths: &[super::snapshot::PeerPathRow]) -> Value {
|
||||
Value::Array(
|
||||
paths
|
||||
.iter()
|
||||
@@ -440,6 +436,11 @@ fn render_peer_paths(paths: &[super::snapshot::PeerPathRow]) -> Value {
|
||||
)
|
||||
}
|
||||
|
||||
/// Render a snapshot [`EntityMmp`](super::snapshot::EntityMmp) into the inline
|
||||
/// MMP JSON block, with the quality-index key named `quality_key` (`lqi` for
|
||||
/// peers, `sqi` for sessions). Reproduces the on-loop key insertion order
|
||||
/// exactly. `path_mtu` is emitted (inside the leading literal) only when
|
||||
/// present (session-layer); for peers it is `None` and omitted.
|
||||
fn render_entity_mmp(mmp: &super::snapshot::EntityMmp, quality_key: &str) -> Value {
|
||||
// The on-loop `show_sessions` block places loss_rate/etx/goodput_bps/
|
||||
// delivery ratios/path_mtu in the leading json! literal, while `show_peers`
|
||||
|
||||
@@ -551,6 +551,12 @@ impl Node {
|
||||
/// landing on an index the allocator has already handed to a new owner:
|
||||
/// index reuse is immediate, with no quarantine.
|
||||
///
|
||||
/// Only a path the peer has proven counts: the active one, or one the
|
||||
/// peer has acknowledged a probe on. A `Probing` path is an address we
|
||||
/// were *told* about — a beacon, a config entry, a handshake source —
|
||||
/// and until the peer answers there, garbage arriving on its transport
|
||||
/// says nothing about the peer.
|
||||
///
|
||||
/// A peer with no transport bound yet is charged unconditionally, as
|
||||
/// before.
|
||||
pub(in crate::node) fn charge_decrypt_failure(
|
||||
@@ -558,10 +564,13 @@ impl Node {
|
||||
node_addr: &crate::NodeAddr,
|
||||
transport_id: crate::transport::TransportId,
|
||||
) {
|
||||
let on_path = self
|
||||
.peers
|
||||
.get(node_addr)
|
||||
.is_some_and(|peer| peer.paths().is_empty() || peer.path_on(transport_id).is_some());
|
||||
let on_path = self.peers.get(node_addr).is_some_and(|peer| {
|
||||
peer.paths().is_empty()
|
||||
|| peer.transport_id() == Some(transport_id)
|
||||
|| peer
|
||||
.path_on(transport_id)
|
||||
.is_some_and(|path| path.acked_once())
|
||||
});
|
||||
if !on_path {
|
||||
trace!(
|
||||
peer = %self.peer_display_name(node_addr),
|
||||
|
||||
@@ -345,8 +345,9 @@ impl Node {
|
||||
"executor CrossConnectionLost is unreachable on \
|
||||
driven net-new establish paths"
|
||||
);
|
||||
// Close this (losing) connection, drop its link,
|
||||
// and restore `addr_to_link` to the winner.
|
||||
// Close this connection, drop its link, and
|
||||
// point `addr_to_link` for the new address at the
|
||||
// winner, so a msg1 from it is recognised.
|
||||
if let Some(transport) =
|
||||
self.transports.get(&ambient.transport_id)
|
||||
{
|
||||
|
||||
@@ -1573,13 +1573,24 @@ impl Node {
|
||||
|
||||
// Clean up outbound connection state
|
||||
self.pending_outbound.remove(&key);
|
||||
// Close the losing TCP connection (no-op for connectionless)
|
||||
|
||||
// The handshake ran over some (transport, address). Whichever
|
||||
// session won, that is where the peer answered just now — but a
|
||||
// handshake creates no path state: the probe exchange, which is
|
||||
// authenticated and replay-checked under the surviving session,
|
||||
// is the one way a path is proven. Leave the address as a
|
||||
// candidate for the heartbeat tick. The link record goes (the
|
||||
// peer's link is the one its session rides), but the transport
|
||||
// connection stays: on TCP, Tor or Nym that socket is what the
|
||||
// probe will go out on, and closing it would only have the
|
||||
// first probe dial it again — or, on the responder, find that
|
||||
// our ephemeral port cannot be dialled at all. `api_disconnect`
|
||||
// closes every path's connection. A connectionless close was a
|
||||
// no-op either way.
|
||||
if let Some(link) = self.links.get(&link_id) {
|
||||
let tid = link.transport_id();
|
||||
let addr = link.remote_addr().clone();
|
||||
if let Some(transport) = self.transports.get(&tid) {
|
||||
transport.close_connection(&addr).await;
|
||||
}
|
||||
self.add_path_candidate(peer_node_addr, tid, addr);
|
||||
}
|
||||
self.remove_link(&link_id);
|
||||
|
||||
|
||||
+52
-75
@@ -121,39 +121,24 @@ impl Node {
|
||||
}
|
||||
|
||||
/// `fipsctl path show <peer>`: every path to the peer, per direction.
|
||||
///
|
||||
/// The per-path object is the `show_peers` one (`project_peer_paths`,
|
||||
/// rendered by `render_peer_paths`) plus the three fields only a
|
||||
/// now-relative read can give: the liveness ages and `acked_once`.
|
||||
/// Built from the same projection so the two field lists cannot drift.
|
||||
pub(crate) fn api_path_show(&self, npub: &str) -> Result<serde_json::Value, String> {
|
||||
let node_addr = self.resolve_peer_npub(npub)?;
|
||||
let peer = &self.peers[&node_addr];
|
||||
let now_ms = crate::time::mono_ms();
|
||||
let active = peer.transport_id();
|
||||
let paths: Vec<serde_json::Value> = peer
|
||||
.paths()
|
||||
.iter()
|
||||
.map(|path| {
|
||||
let ago = |at: Option<u64>| at.map(|t| now_ms.saturating_sub(t));
|
||||
serde_json::json!({
|
||||
"transport_id": path.transport_id().as_u32(),
|
||||
"transport": self
|
||||
.transports
|
||||
.get(&path.transport_id())
|
||||
.and_then(|t| t.name().map(str::to_string)),
|
||||
"addr": path.addr().to_string(),
|
||||
"state": format!("{:?}", path.state()).to_lowercase(),
|
||||
"active": Some(path.transport_id()) == active,
|
||||
"remote_active": path.remote_active(),
|
||||
"role": format!("{:?}", path.role()).to_lowercase(),
|
||||
"pinned": path.pinned(),
|
||||
"rx_live_ms_ago": ago(path.rx_live_at_ms()),
|
||||
"tx_live_ms_ago": ago(path.tx_live_at_ms()),
|
||||
"acked_once": path.acked_once(),
|
||||
"last_rtt_ms": path.last_rtt_ms(),
|
||||
"min_rtt_ms": path.min_rtt_ms(),
|
||||
"rtt_samples": path.rtt_samples(),
|
||||
"etx": path.etx(),
|
||||
"score": path.score(),
|
||||
})
|
||||
})
|
||||
.collect();
|
||||
let ago = |at: Option<u64>| at.map(|t| now_ms.saturating_sub(t));
|
||||
let mut paths = crate::control::queries::render_peer_paths(&self.project_peer_paths(peer));
|
||||
if let Some(rows) = paths.as_array_mut() {
|
||||
for (row, path) in rows.iter_mut().zip(peer.paths()) {
|
||||
row["rx_live_ms_ago"] = serde_json::json!(ago(path.rx_live_at_ms()));
|
||||
row["tx_live_ms_ago"] = serde_json::json!(ago(path.tx_live_at_ms()));
|
||||
row["acked_once"] = serde_json::json!(path.acked_once());
|
||||
}
|
||||
}
|
||||
Ok(serde_json::json!({
|
||||
"peer": npub,
|
||||
"link_cost": peer.link_cost(now_ms),
|
||||
@@ -204,13 +189,28 @@ impl Node {
|
||||
let Some(peer) = self.peers.get_mut(&node_addr) else {
|
||||
return;
|
||||
};
|
||||
if peer.transport_id() == Some(transport_id) {
|
||||
// The active path: the handshake proved it.
|
||||
if peer.transport_id() == Some(transport_id)
|
||||
&& peer.path_on(transport_id).is_some_and(|p| p.is_eligible())
|
||||
{
|
||||
// The active path, and it is answering: nothing to add, and an
|
||||
// address that has proven nothing does not displace it. An
|
||||
// active path that has stopped answering falls through to the
|
||||
// re-pointing below like any other.
|
||||
return;
|
||||
}
|
||||
let was_new = peer.path_on(transport_id).is_none();
|
||||
peer.add_path(transport_id, remote_addr.clone())
|
||||
.set_role(role);
|
||||
// A known transport at a new address: the peer moved there (an
|
||||
// Aware data path that re-formed, a DHCP lease that changed) and
|
||||
// the old address answers nothing. Re-point the path; the heartbeat
|
||||
// tick probes it from here. Only while the path is not eligible: an
|
||||
// address that is carrying acknowledged traffic is not displaced by
|
||||
// one that has proven nothing — a probe from the new address
|
||||
// (`note_path_probe`) is what moves a working path.
|
||||
let moved = !was_new
|
||||
&& peer.path_on(transport_id).is_some_and(|p| !p.is_eligible())
|
||||
&& peer.refresh_path_addr(transport_id, remote_addr.clone());
|
||||
if was_new {
|
||||
debug!(
|
||||
peer = %self.peer_display_name(&node_addr),
|
||||
@@ -218,47 +218,13 @@ impl Node {
|
||||
remote_addr = %remote_addr,
|
||||
"Peer beaconed on a new transport; path added, probing"
|
||||
);
|
||||
}
|
||||
}
|
||||
|
||||
/// Tests: add the path and probe it at once, as one heartbeat tick
|
||||
/// would, without the tick's other sends. Uses the test-only
|
||||
/// `take_probe`, which honours the per-path backoff.
|
||||
#[cfg(test)]
|
||||
pub(in crate::node) async fn maybe_probe_path(
|
||||
&mut self,
|
||||
node_addr: NodeAddr,
|
||||
transport_id: TransportId,
|
||||
remote_addr: TransportAddr,
|
||||
) {
|
||||
self.add_path_candidate(node_addr, transport_id, remote_addr.clone());
|
||||
let now_ms = crate::time::mono_ms();
|
||||
let timing = self.heartbeat_timing();
|
||||
let Some(peer) = self.peers.get_mut(&node_addr) else {
|
||||
return;
|
||||
};
|
||||
if peer.transport_id() == Some(transport_id) {
|
||||
return;
|
||||
}
|
||||
let Some((probe_id, remote_active, path_id)) = peer.take_probe(
|
||||
transport_id,
|
||||
now_ms,
|
||||
timing.fast_ms,
|
||||
timing.discovery_cap_ms,
|
||||
) else {
|
||||
return;
|
||||
};
|
||||
let probe = PathMessage {
|
||||
probe_id,
|
||||
remote_active,
|
||||
path_id,
|
||||
};
|
||||
let wire = self.pad_to_link_mtu(probe.encode_probe().to_vec(), transport_id, &remote_addr);
|
||||
if let Err(e) = self
|
||||
.send_encrypted_link_message_on_path(&node_addr, &wire, transport_id, remote_addr)
|
||||
.await
|
||||
{
|
||||
debug!(peer = %self.peer_display_name(&node_addr), error = %e, "Path probe send failed");
|
||||
} else if moved {
|
||||
debug!(
|
||||
peer = %self.peer_display_name(&node_addr),
|
||||
transport_id = %transport_id,
|
||||
remote_addr = %remote_addr,
|
||||
"Peer beaconed at a new address on a known transport; path re-addressed, probing"
|
||||
);
|
||||
}
|
||||
}
|
||||
|
||||
@@ -766,10 +732,21 @@ impl Node {
|
||||
}
|
||||
|
||||
/// A transport's presence came back: clear the probe backoff on every
|
||||
/// path over it so the next discovery tick may probe at once.
|
||||
/// path over it so the next heartbeat tick may probe at once, and
|
||||
/// revive every path that went `Dead` with it — a replugged NIC, a
|
||||
/// wifi interface that cycled — as `Probing`, history kept. Without
|
||||
/// this nothing on our side ever probed a `Dead` path again: traffic
|
||||
/// stayed on the standby until the grace pruned the path and a fresh
|
||||
/// beacon found it with no history.
|
||||
pub(in crate::node) fn reset_probe_backoff_on_transport(&mut self, transport_id: TransportId) {
|
||||
for peer in self.peers.values_mut() {
|
||||
peer.reset_probe_backoff_on(transport_id);
|
||||
for (node_addr, peer) in self.peers.iter_mut() {
|
||||
if peer.reset_probe_backoff_on(transport_id) {
|
||||
debug!(
|
||||
peer = %node_addr,
|
||||
%transport_id,
|
||||
"Transport returned: dead path probing again"
|
||||
);
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
+242
-163
@@ -464,6 +464,21 @@ impl Node {
|
||||
.map(|t| t.transport_type().connection_oriented)
|
||||
.unwrap_or(false);
|
||||
|
||||
// A dial to a peer we already hold a session with (a startup that
|
||||
// lists two addresses, a caller that still dials by hand) proves
|
||||
// nothing by itself: the handshake creates no path state, and its
|
||||
// outcome is the cross-connection tie-break as ever. The address
|
||||
// is still a fact worth holding — leave it as a candidate for the
|
||||
// heartbeat tick to probe under the session, on a datagram
|
||||
// transport, so it becomes a path whichever way the dial goes.
|
||||
if !is_connection_oriented && self.peers.contains_key(peer_identity.node_addr()) {
|
||||
self.add_path_candidate(
|
||||
*peer_identity.node_addr(),
|
||||
transport_id,
|
||||
remote_addr.clone(),
|
||||
);
|
||||
}
|
||||
|
||||
// Allocate link ID and create link
|
||||
let link_id = self.allocate_link_id();
|
||||
|
||||
@@ -812,51 +827,38 @@ impl Node {
|
||||
let connected = self.peers.contains_key(&node_addr);
|
||||
|
||||
if connected {
|
||||
// Active peer: skip every candidate while the link we
|
||||
// already hold is live — the current path *and* any
|
||||
// alternate one.
|
||||
// Active peer: never a dial, whatever the state of the
|
||||
// link we hold. The address is a path to add or
|
||||
// re-point, and the heartbeat tick probes it under the
|
||||
// session we have — one round trip, authenticated, and
|
||||
// selection moves traffic if the path proves better or
|
||||
// the current one is not answering.
|
||||
//
|
||||
// Only the same-path case used to be skipped, which left
|
||||
// the stated intent ("avoid churning a healthy link")
|
||||
// covering exactly the case that could not churn anything.
|
||||
// A peer reachable twice — the ordinary result of two
|
||||
// machines sharing a LAN and a cable, since each beacons on
|
||||
// both — was therefore re-dialled on its alternate path
|
||||
// every discovery tick, forever. Each dial that completed
|
||||
// promoted and displaced the incumbent, so the peer's link
|
||||
// migrated back and forth on a fixed cadence, tearing down
|
||||
// and re-establishing its session each time. Measured on
|
||||
// real hardware: seventeen dials to one peer in fifteen
|
||||
// minutes, alternating wifi and cable, displacing a link
|
||||
// reporting `etx = 1.0` and `loss = 0.0`.
|
||||
// Dialling an active peer used to be the fallback for one
|
||||
// that had gone quiet ("liveness is the gate"). Two
|
||||
// reasons it is not any more. A handshake to a peer that
|
||||
// already holds a session is read by the far side as a
|
||||
// rekey, and when both ends do it at once — the ordinary
|
||||
// case, since both hear the same medium come back — the
|
||||
// rekey and the cross-connection resolution overlap and
|
||||
// the two sides part on different session indices, dead
|
||||
// to each other until the link timeout reaps them. And
|
||||
// the dial buys nothing the probe does not: a session
|
||||
// that is truly gone answers no probe either, is reaped
|
||||
// by the link-dead timeout, and is dialled then; a peer
|
||||
// that restarted dials us itself with a new epoch and
|
||||
// wins the promotion outright.
|
||||
//
|
||||
// When that peer is the parent — which the best path
|
||||
// usually is — every migration also switched parents,
|
||||
// invalidating the downstream coordinate cache and
|
||||
// re-announcing to every peer. The cost of the churn was
|
||||
// therefore mesh-wide while the benefit was nil: the link
|
||||
// being replaced was already perfect.
|
||||
//
|
||||
// Failover is unaffected. Liveness is the gate, so a peer
|
||||
// that stops answering goes stale within a heartbeat
|
||||
// interval and every path, alternate included, is dialled
|
||||
// again. What is given up is switching away from a link
|
||||
// that is working, which is not a thing worth doing.
|
||||
if self.active_peer_link_is_live(&node_addr) {
|
||||
// A live peer beaconing on a transport we hold no
|
||||
// path to it over is a path to add, not a link to
|
||||
// replace: the heartbeat tick probes it under the
|
||||
// existing session instead of dialling.
|
||||
path_candidates.push((node_addr, candidate_transport_id, remote_addr));
|
||||
continue;
|
||||
}
|
||||
if self.is_connecting_to_peer_on_path(
|
||||
&node_addr,
|
||||
candidate_transport_id,
|
||||
&remote_addr,
|
||||
) {
|
||||
continue;
|
||||
}
|
||||
// (History: only the same-path case used to be skipped,
|
||||
// so a peer reachable twice — two machines sharing a LAN
|
||||
// and a cable — was re-dialled on its alternate path
|
||||
// every discovery tick, each completed dial displacing
|
||||
// the incumbent: seventeen dials to one peer in fifteen
|
||||
// minutes, alternating wifi and cable, over a link
|
||||
// reporting `etx = 1.0`. Liveness gating fixed that and
|
||||
// left the quiet-peer dial; this removes the last of it.)
|
||||
path_candidates.push((node_addr, candidate_transport_id, remote_addr));
|
||||
continue;
|
||||
} else if self.is_connecting_to_peer_on_path(
|
||||
&node_addr,
|
||||
candidate_transport_id,
|
||||
@@ -878,6 +880,7 @@ impl Node {
|
||||
for (node_addr, transport_id, remote_addr) in path_candidates {
|
||||
self.add_path_candidate(node_addr, transport_id, remote_addr);
|
||||
}
|
||||
self.add_configured_path_candidates();
|
||||
|
||||
if transport_neighbors.is_empty() {
|
||||
return;
|
||||
@@ -2560,6 +2563,126 @@ impl Node {
|
||||
.collect()
|
||||
}
|
||||
|
||||
/// Configured addresses of live peers on transports they have no path
|
||||
/// over become paths. Runs every discovery tick, idempotent and cheap:
|
||||
/// a configured address whose transport was down at dial time (wifi
|
||||
/// joined later, Tor came up) is otherwise never looked at again, since
|
||||
/// a peer that is already active is not re-dialled.
|
||||
fn add_configured_path_candidates(&mut self) {
|
||||
let configs: Vec<PeerConfig> = self.config().auto_connect_peers().cloned().collect();
|
||||
for peer_config in configs {
|
||||
let Ok(identity) = PeerIdentity::from_npub(&peer_config.npub) else {
|
||||
continue;
|
||||
};
|
||||
let node_addr = *identity.node_addr();
|
||||
if !self.peers.contains_key(&node_addr) || !self.active_peer_link_is_live(&node_addr) {
|
||||
continue;
|
||||
}
|
||||
for addr in peer_config.addresses_by_priority() {
|
||||
if addr.transport == "udp" && addr.addr.eq_ignore_ascii_case("nat") {
|
||||
continue;
|
||||
}
|
||||
let Some((transport_id, remote_addr)) = self.resolve_peer_address(addr) else {
|
||||
continue;
|
||||
};
|
||||
let has_path = self
|
||||
.peers
|
||||
.get(&node_addr)
|
||||
.is_some_and(|p| p.path_on(transport_id).is_some());
|
||||
if !has_path {
|
||||
self.add_path_candidate(node_addr, transport_id, remote_addr);
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
/// The transport and address a configured peer address dials to, or
|
||||
/// `None` (logged at debug) if no operational transport can carry it.
|
||||
///
|
||||
/// The transport field may name a specific instance (`"udp/aware"`):
|
||||
/// the type half picks the resolver, the instance half is handed to
|
||||
/// whichever resolver can honour it, and only the UDP one can. The
|
||||
/// `"nat"` pseudo-address is not resolved here.
|
||||
fn resolve_peer_address(&self, addr: &PeerAddress) -> Option<(TransportId, TransportAddr)> {
|
||||
let spec = addr.spec();
|
||||
if addr.transport == "ethernet" {
|
||||
return match self.resolve_ethernet_addr(&addr.addr) {
|
||||
Ok(result) => Some(result),
|
||||
Err(e) => {
|
||||
debug!(
|
||||
transport = %addr.transport,
|
||||
addr = %addr.addr,
|
||||
error = %e,
|
||||
"Failed to resolve Ethernet address"
|
||||
);
|
||||
None
|
||||
}
|
||||
};
|
||||
}
|
||||
if addr.transport == "ble" {
|
||||
#[cfg(ble_available)]
|
||||
{
|
||||
return match self.resolve_ble_addr(&addr.addr) {
|
||||
Ok(result) => Some(result),
|
||||
Err(e) => {
|
||||
debug!(
|
||||
transport = %addr.transport,
|
||||
addr = %addr.addr,
|
||||
error = %e,
|
||||
"Failed to resolve BLE address"
|
||||
);
|
||||
None
|
||||
}
|
||||
};
|
||||
}
|
||||
#[cfg(not(ble_available))]
|
||||
{
|
||||
debug!(transport = %addr.transport, "BLE transport not available on this build");
|
||||
return None;
|
||||
}
|
||||
}
|
||||
let tid = if spec.kind == "udp"
|
||||
&& let Ok(remote_socket_addr) = addr.addr.parse::<SocketAddr>()
|
||||
{
|
||||
match self.find_udp_transport_for_remote_addr(remote_socket_addr, spec.instance) {
|
||||
Some((id, _)) => id,
|
||||
None => {
|
||||
debug!(
|
||||
transport = %addr.transport,
|
||||
addr = %addr.addr,
|
||||
"No compatible operational UDP transport for address"
|
||||
);
|
||||
return None;
|
||||
}
|
||||
}
|
||||
} else if spec.instance.is_some() {
|
||||
// Only the UDP resolver above can honour an instance name.
|
||||
// Matching any instance of the type here would be the silent
|
||||
// wrong-lane substitution this whole mechanism exists to
|
||||
// prevent, so refuse instead.
|
||||
debug!(
|
||||
transport = %addr.transport,
|
||||
addr = %addr.addr,
|
||||
"Instance-qualified address for a transport type that \
|
||||
does not support instance selection"
|
||||
);
|
||||
return None;
|
||||
} else {
|
||||
match self.find_transport_for_type(spec.kind) {
|
||||
Some(id) => id,
|
||||
None => {
|
||||
debug!(
|
||||
transport = %addr.transport,
|
||||
addr = %addr.addr,
|
||||
"No operational transport for address type"
|
||||
);
|
||||
return None;
|
||||
}
|
||||
}
|
||||
};
|
||||
Some((tid, TransportAddr::from_string(&addr.addr)))
|
||||
}
|
||||
|
||||
async fn attempt_peer_address_list(
|
||||
&mut self,
|
||||
peer_config: &PeerConfig,
|
||||
@@ -2581,9 +2704,6 @@ impl Node {
|
||||
if attempted >= max_attempts {
|
||||
break;
|
||||
}
|
||||
// The transport field may name a specific instance
|
||||
// (`"udp/aware"`); everything below dispatches on the type half
|
||||
// and hands the instance half to whichever resolver can honour it.
|
||||
let spec = addr.spec();
|
||||
|
||||
if spec.kind == "udp" && addr.addr.eq_ignore_ascii_case("nat") {
|
||||
@@ -2601,82 +2721,8 @@ impl Node {
|
||||
continue;
|
||||
}
|
||||
|
||||
let (transport_id, remote_addr) = if addr.transport == "ethernet" {
|
||||
match self.resolve_ethernet_addr(&addr.addr) {
|
||||
Ok(result) => result,
|
||||
Err(e) => {
|
||||
debug!(
|
||||
transport = %addr.transport,
|
||||
addr = %addr.addr,
|
||||
error = %e,
|
||||
"Failed to resolve Ethernet address"
|
||||
);
|
||||
continue;
|
||||
}
|
||||
}
|
||||
} else if addr.transport == "ble" {
|
||||
#[cfg(ble_available)]
|
||||
{
|
||||
match self.resolve_ble_addr(&addr.addr) {
|
||||
Ok(result) => result,
|
||||
Err(e) => {
|
||||
debug!(
|
||||
transport = %addr.transport,
|
||||
addr = %addr.addr,
|
||||
error = %e,
|
||||
"Failed to resolve BLE address"
|
||||
);
|
||||
continue;
|
||||
}
|
||||
}
|
||||
}
|
||||
#[cfg(not(ble_available))]
|
||||
{
|
||||
debug!(transport = %addr.transport, "BLE transport not available on this build");
|
||||
continue;
|
||||
}
|
||||
} else {
|
||||
let tid = if spec.kind == "udp"
|
||||
&& let Ok(remote_socket_addr) = addr.addr.parse::<SocketAddr>()
|
||||
{
|
||||
match self.find_udp_transport_for_remote_addr(remote_socket_addr, spec.instance)
|
||||
{
|
||||
Some((id, _)) => id,
|
||||
None => {
|
||||
debug!(
|
||||
transport = %addr.transport,
|
||||
addr = %addr.addr,
|
||||
"No compatible operational UDP transport for address"
|
||||
);
|
||||
continue;
|
||||
}
|
||||
}
|
||||
} else if spec.instance.is_some() {
|
||||
// Only the UDP resolver above can honour an instance name.
|
||||
// Matching any instance of the type here would be the
|
||||
// silent wrong-lane substitution this whole mechanism
|
||||
// exists to prevent, so refuse instead.
|
||||
debug!(
|
||||
transport = %addr.transport,
|
||||
addr = %addr.addr,
|
||||
"Instance-qualified address for a transport type that \
|
||||
does not support instance selection"
|
||||
);
|
||||
continue;
|
||||
} else {
|
||||
match self.find_transport_for_type(spec.kind) {
|
||||
Some(id) => id,
|
||||
None => {
|
||||
debug!(
|
||||
transport = %addr.transport,
|
||||
addr = %addr.addr,
|
||||
"No operational transport for address type"
|
||||
);
|
||||
continue;
|
||||
}
|
||||
}
|
||||
};
|
||||
(tid, TransportAddr::from_string(&addr.addr))
|
||||
let Some((transport_id, remote_addr)) = self.resolve_peer_address(addr) else {
|
||||
continue;
|
||||
};
|
||||
|
||||
if self.is_connecting_to_peer_on_path(&peer_node_addr, transport_id, &remote_addr) {
|
||||
@@ -3243,27 +3289,55 @@ impl Node {
|
||||
.into_iter()
|
||||
.filter(|addr| !(addr.transport == "udp" && addr.addr.eq_ignore_ascii_case("nat")))
|
||||
.collect();
|
||||
let has_alternative = concrete
|
||||
.iter()
|
||||
.any(|addr| !self.active_peer_matches_candidate(&peer_node_addr, addr));
|
||||
let attempt_candidates: Vec<_> = if has_alternative {
|
||||
concrete
|
||||
.into_iter()
|
||||
.filter(|addr| !self.active_peer_matches_candidate(&peer_node_addr, addr))
|
||||
.collect()
|
||||
} else if self.active_peer_needs_same_path_refresh(&peer_node_addr) {
|
||||
concrete
|
||||
} else {
|
||||
Vec::new()
|
||||
};
|
||||
// Every address the peer is not already on. (The same-path case —
|
||||
// the one address it is on, gone quiet — was once re-dialled from
|
||||
// here; the path's own heartbeat and the link-dead reap own that
|
||||
// now, see below.)
|
||||
let attempt_candidates: Vec<_> = concrete
|
||||
.into_iter()
|
||||
.filter(|addr| !self.active_peer_matches_candidate(&peer_node_addr, addr))
|
||||
.collect();
|
||||
|
||||
if attempt_candidates.is_empty() {
|
||||
return Ok(false);
|
||||
// A peer we hold a session to gets a *path* at each address, under
|
||||
// that session, never a second handshake: the probe exchange proves
|
||||
// the path and selection moves traffic if it measures better or the
|
||||
// current path stops answering. A handshake to a peer that already
|
||||
// has one is read by the far side as a rekey, and two ends doing it
|
||||
// at once — both hearing the same medium return — leave the rekey
|
||||
// and the cross-connection resolution overlapping and the sides on
|
||||
// different session indices. An address the peer is already
|
||||
// reachable at is nothing to do; one on a transport that already
|
||||
// has a path re-points that path (the peer moved); one on a new
|
||||
// transport adds a path. A peer that has gone quiet is not dialled
|
||||
// either: a dead session answers no probe, is reaped by the
|
||||
// link-dead timeout, and is dialled then. See `poll_discovered_peers`.
|
||||
let mut paths_added = false;
|
||||
for addr in attempt_candidates {
|
||||
let Some((transport_id, remote_addr)) = self.resolve_peer_address(&addr) else {
|
||||
continue;
|
||||
};
|
||||
let Some(peer) = self.peers.get(&peer_node_addr) else {
|
||||
continue;
|
||||
};
|
||||
if peer.is_reachable_at(transport_id, &remote_addr) {
|
||||
continue;
|
||||
}
|
||||
let known_transport = peer.path_on(transport_id).is_some();
|
||||
info!(
|
||||
peer = %self.peer_display_name(&peer_node_addr),
|
||||
%transport_id,
|
||||
addr = %remote_addr,
|
||||
"{}",
|
||||
if known_transport {
|
||||
"Configured address moved on a known transport: path re-pointed, not dialled"
|
||||
} else {
|
||||
"Configured address on a new transport: added as a path, not dialled"
|
||||
}
|
||||
);
|
||||
self.add_path_candidate(peer_node_addr, transport_id, remote_addr);
|
||||
paths_added = true;
|
||||
}
|
||||
|
||||
self.attempt_peer_address_list(peer_config, peer_identity, false, &attempt_candidates)
|
||||
.await?;
|
||||
Ok(true)
|
||||
Ok(paths_added)
|
||||
}
|
||||
|
||||
async fn peer_address_candidates(&self, peer_config: &PeerConfig) -> Vec<PeerAddress> {
|
||||
@@ -3362,14 +3436,17 @@ impl Node {
|
||||
/// auto-reconnect). Reuses the same connection path as auto-connect
|
||||
/// peers. Returns JSON data on success or an error message.
|
||||
///
|
||||
/// For a peer the node is already connected to, the supplied address is
|
||||
/// tried as an *alternate path* rather than ignored — the same treatment
|
||||
/// [`Node::update_peers`] gives a refreshed runtime peer. The handshake
|
||||
/// runs in parallel with the live link and promotion happens only once it
|
||||
/// authenticates, so an address the caller got wrong cannot displace a
|
||||
/// healthy path. The response's `refreshed` field reports whether such a
|
||||
/// handshake was started; it is `false` when the peer is already on this
|
||||
/// exact path and that path is fresh.
|
||||
/// For a peer the node already holds a session with, the supplied
|
||||
/// address is never dialled: it becomes a path candidate under that
|
||||
/// session (or re-points a path that has stopped answering), the
|
||||
/// heartbeat tick probes it, and selection moves traffic there only
|
||||
/// once the peer has answered — the same treatment
|
||||
/// [`Node::update_peers`] gives a refreshed runtime peer, so an address
|
||||
/// the caller got wrong cannot displace a healthy path. The response's
|
||||
/// `refreshed` field reports whether a path was added or re-pointed;
|
||||
/// it is `false` when the peer is already reachable at that address or
|
||||
/// that transport's path is carrying acknowledged traffic, and for an
|
||||
/// ordinary dial to a peer the node does not yet hold.
|
||||
pub(crate) async fn api_connect(
|
||||
&mut self,
|
||||
npub: &str,
|
||||
@@ -3451,16 +3528,18 @@ impl Node {
|
||||
return Err(format!("peer not found: {npub}"));
|
||||
};
|
||||
|
||||
// Read the transport path the peer is actually sending over BEFORE the
|
||||
// teardown below drops the peer and its link — afterwards there is
|
||||
// nothing left to derive it from. `current_addr` rather than the
|
||||
// link's remote address, because roaming updates the former and it is
|
||||
// the address the pool entry (and its inbound-slot accounting) is
|
||||
// keyed by.
|
||||
let transport_path = match (peer.transport_id(), peer.current_addr()) {
|
||||
(Some(transport_id), Some(addr)) => Some((transport_id, addr.clone())),
|
||||
_ => None,
|
||||
};
|
||||
// Read every path the peer holds BEFORE the teardown below drops
|
||||
// the peer and its link — afterwards there is nothing left to
|
||||
// derive them from. The path's address rather than the link's
|
||||
// remote address, because roaming updates the former and it is the
|
||||
// address the pool entry (and its inbound-slot accounting) is keyed
|
||||
// by. Every path, not the active one alone: a standby on a
|
||||
// connection-oriented transport holds a pool entry of its own.
|
||||
let transport_paths: Vec<(TransportId, TransportAddr)> = peer
|
||||
.paths()
|
||||
.iter()
|
||||
.map(|path| (path.transport_id(), path.addr().clone()))
|
||||
.collect();
|
||||
|
||||
// Notify the peer before we tear down the link, so it drops its own
|
||||
// session and re-handshakes symmetrically rather than holding a stale
|
||||
@@ -3483,10 +3562,10 @@ impl Node {
|
||||
// verbatim: closing twice is harmless, because every
|
||||
// `close_connection` implementation is `if let Some(conn) =
|
||||
// pool.remove(addr)` and the connectionless default is a no-op.
|
||||
if let Some((transport_id, addr)) = transport_path
|
||||
&& let Some(transport) = self.transports.get(&transport_id)
|
||||
{
|
||||
transport.close_connection(&addr).await;
|
||||
for (transport_id, addr) in transport_paths {
|
||||
if let Some(transport) = self.transports.get(&transport_id) {
|
||||
transport.close_connection(&addr).await;
|
||||
}
|
||||
}
|
||||
|
||||
// Suppress any pending auto-reconnect
|
||||
|
||||
+2
-2
@@ -2697,10 +2697,10 @@ impl Node {
|
||||
transport: handle.and_then(|t| t.name().map(str::to_string)),
|
||||
transport_type: handle.map(|t| t.transport_type().name.to_string()),
|
||||
addr: path.addr().to_string(),
|
||||
state: format!("{:?}", path.state()).to_lowercase(),
|
||||
state: path.state().as_str().to_string(),
|
||||
active: Some(path.transport_id()) == active,
|
||||
remote_active: path.remote_active(),
|
||||
role: format!("{:?}", path.role()).to_lowercase(),
|
||||
role: path.role().as_str().to_string(),
|
||||
pinned: path.pinned(),
|
||||
last_rtt_ms: path.last_rtt_ms(),
|
||||
min_rtt_ms: path.min_rtt_ms(),
|
||||
|
||||
+14
-17
@@ -150,13 +150,11 @@ async fn test_api_connect_on_current_fresh_path_is_a_no_op() {
|
||||
}
|
||||
|
||||
/// `connect` naming a *different* address for a peer the node is already
|
||||
/// connected to starts an alternate-path handshake instead of silently doing
|
||||
/// nothing — the fix.
|
||||
///
|
||||
/// The existing peer stays put while that handshake runs: promotion is the
|
||||
/// handshake's job, not the command's.
|
||||
/// connected to takes it as a path under the session it has — never a
|
||||
/// second handshake, which the far side would read as a rekey. The peer and
|
||||
/// its link stay put; the heartbeat tick probes the path from here.
|
||||
#[tokio::test]
|
||||
async fn test_api_connect_starts_alternate_path_for_active_peer() {
|
||||
async fn test_api_connect_takes_an_alternate_address_as_a_path() {
|
||||
let mut nodes = run_tree_test(2, &[(0, 1)], false).await;
|
||||
|
||||
let node1_addr = *nodes[1].node.node_addr();
|
||||
@@ -182,26 +180,25 @@ async fn test_api_connect_starts_alternate_path_for_active_peer() {
|
||||
|
||||
assert_eq!(
|
||||
data["refreshed"], true,
|
||||
"a new path for an active peer must start a refresh"
|
||||
"a new address for an active peer is taken as a path"
|
||||
);
|
||||
assert!(
|
||||
nodes[0]
|
||||
!nodes[0]
|
||||
.node
|
||||
.is_connecting_to_peer_on_path(&node1_addr, transport_id, &alternate),
|
||||
"an outbound leg should exist on the alternate path"
|
||||
"no handshake: a peer with a session is probed, not dialled"
|
||||
);
|
||||
assert_eq!(nodes[0].node.connection_count(), 0);
|
||||
let peer = nodes[0]
|
||||
.node
|
||||
.get_peer(&node1_addr)
|
||||
.expect("the existing peer must survive the parallel handshake");
|
||||
assert_eq!(
|
||||
peer.link_id(),
|
||||
link_before,
|
||||
"the alternate handshake must not tear the live link down before it authenticates"
|
||||
.expect("the existing peer is untouched");
|
||||
assert_eq!(peer.link_id(), link_before, "the live link must not change");
|
||||
assert!(
|
||||
peer.path_on(transport_id).is_some(),
|
||||
"the transport still has its one path"
|
||||
);
|
||||
|
||||
// Let the alternate handshake run to completion; the peer must still be
|
||||
// there afterwards.
|
||||
for _ in 0..20 {
|
||||
if process_available_packets(&mut nodes).await == 0 {
|
||||
break;
|
||||
@@ -209,7 +206,7 @@ async fn test_api_connect_starts_alternate_path_for_active_peer() {
|
||||
}
|
||||
assert!(
|
||||
nodes[0].node.get_peer(&node1_addr).is_some(),
|
||||
"node 1 should still be a peer after the alternate path resolves"
|
||||
"node 1 is still a peer"
|
||||
);
|
||||
|
||||
cleanup_nodes(&mut nodes).await;
|
||||
|
||||
+509
-83
@@ -307,6 +307,22 @@ async fn dual_homed_pair() -> (Vec<TestNode>, TransportAddr, TransportAddr) {
|
||||
(nodes, wifi_0, wifi_1)
|
||||
}
|
||||
|
||||
/// Hand `nodes[i]` the address `addr` for `peer` on `transport` and run one
|
||||
/// heartbeat tick: the production route by which a path gets probed
|
||||
/// (`add_path_candidate` is what a beacon, a config entry or a completed
|
||||
/// dial leaves behind; the tick is the one issuer of probes). Nothing is
|
||||
/// delivered here — the caller drives `process_available_packets`.
|
||||
async fn probe_candidate(
|
||||
nodes: &mut [TestNode],
|
||||
i: usize,
|
||||
peer: NodeAddr,
|
||||
transport: TransportId,
|
||||
addr: TransportAddr,
|
||||
) {
|
||||
nodes[i].node.add_path_candidate(peer, transport, addr);
|
||||
nodes[i].node.run_path_heartbeats().await;
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn path_message_round_trips_on_the_wire() {
|
||||
let probe = PathMessage {
|
||||
@@ -345,10 +361,7 @@ async fn a_probe_adds_a_path_at_both_ends_and_the_ack_makes_it_live() {
|
||||
let cable = nodes[0].transport_id;
|
||||
|
||||
// Node 1 probes node 0 over the wifi.
|
||||
nodes[1]
|
||||
.node
|
||||
.maybe_probe_path(addr_0, wifi(), wifi_0.clone())
|
||||
.await;
|
||||
probe_candidate(&mut nodes, 1, addr_0, wifi(), wifi_0.clone()).await;
|
||||
{
|
||||
let peer = nodes[1].node.get_peer(&addr_0).unwrap();
|
||||
let path = peer
|
||||
@@ -358,8 +371,9 @@ async fn a_probe_adds_a_path_at_both_ends_and_the_ack_makes_it_live() {
|
||||
assert!(path.tx_live_at_ms().is_none());
|
||||
}
|
||||
|
||||
// Probe reaches node 0.
|
||||
assert_eq!(process_available_packets(&mut nodes).await, 1);
|
||||
// The probe reaches node 0 (the tick heartbeats the active path too,
|
||||
// now that the peer has two).
|
||||
assert!(process_available_packets(&mut nodes).await >= 1);
|
||||
{
|
||||
let peer = nodes[0].node.get_peer(&addr_1).unwrap();
|
||||
let path = peer.path_on(wifi()).expect("the receiver adds the path");
|
||||
@@ -374,8 +388,8 @@ async fn a_probe_adds_a_path_at_both_ends_and_the_ack_makes_it_live() {
|
||||
assert_eq!(peer.paths().len(), 2);
|
||||
}
|
||||
|
||||
// Ack reaches node 1.
|
||||
assert_eq!(process_available_packets(&mut nodes).await, 1);
|
||||
// The ack reaches node 1.
|
||||
assert!(process_available_packets(&mut nodes).await >= 1);
|
||||
{
|
||||
let peer = nodes[1].node.get_peer(&addr_0).unwrap();
|
||||
let path = peer.path_on(wifi()).unwrap();
|
||||
@@ -446,6 +460,121 @@ async fn a_beacon_from_a_live_peer_on_a_new_transport_probes_instead_of_dialling
|
||||
);
|
||||
}
|
||||
|
||||
#[tokio::test]
|
||||
async fn a_second_handshake_creates_no_path_state_and_the_address_is_probed_instead() {
|
||||
let (mut nodes, wifi_0, _wifi_1) = dual_homed_pair().await;
|
||||
let addr_0 = *nodes[0].node.node_addr();
|
||||
let addr_1 = *nodes[1].node.node_addr();
|
||||
let cable = nodes[0].transport_id;
|
||||
let session_before = (
|
||||
nodes[0].node.get_peer(&addr_1).unwrap().our_index(),
|
||||
nodes[1].node.get_peer(&addr_0).unwrap().our_index(),
|
||||
);
|
||||
|
||||
// Node 1 dials node 0 over the wifi, as a static config listing both
|
||||
// addresses does at startup, while the cable session is live. The
|
||||
// responder answers it as it would any msg1 from a peer it holds a
|
||||
// session with — a duplicate here, the session being seconds old —
|
||||
// and neither end derives a path from the handshake.
|
||||
let identity_0 = PeerIdentity::from_pubkey_full(nodes[0].node.identity().pubkey_full());
|
||||
nodes[1]
|
||||
.node
|
||||
.initiate_connection(wifi(), wifi_0.clone(), identity_0)
|
||||
.await
|
||||
.expect("dial starts");
|
||||
for _ in 0..8 {
|
||||
if process_available_packets(&mut nodes).await == 0 {
|
||||
break;
|
||||
}
|
||||
}
|
||||
|
||||
// Neither end re-peered: one peer each, the session untouched.
|
||||
assert_eq!(nodes[0].node.peer_count(), 1);
|
||||
assert_eq!(nodes[1].node.peer_count(), 1);
|
||||
assert_eq!(
|
||||
(
|
||||
nodes[0].node.get_peer(&addr_1).unwrap().our_index(),
|
||||
nodes[1].node.get_peer(&addr_0).unwrap().our_index(),
|
||||
),
|
||||
session_before,
|
||||
"the handshake did not replace the session"
|
||||
);
|
||||
// The dialler holds the address as an unproven candidate, traffic on
|
||||
// the cable; the responder learned nothing from the handshake.
|
||||
let p1 = nodes[1].node.get_peer(&addr_0).unwrap();
|
||||
assert_eq!(p1.transport_id(), Some(cable), "traffic stays on the cable");
|
||||
let wifi_path = p1
|
||||
.path_on(wifi())
|
||||
.expect("the dialled address is a candidate");
|
||||
assert_eq!(wifi_path.state(), PathState::Probing);
|
||||
assert!(!wifi_path.acked_once(), "a handshake proves no path");
|
||||
assert!(
|
||||
nodes[0]
|
||||
.node
|
||||
.get_peer(&addr_1)
|
||||
.unwrap()
|
||||
.path_on(wifi())
|
||||
.is_none(),
|
||||
"the responder derives no path from a handshake"
|
||||
);
|
||||
|
||||
// The connection the dial opened is left for the probe to ride: on a
|
||||
// connection-oriented transport it is the path's socket, and closing
|
||||
// it would have the first probe dial again (or, at the responder, find
|
||||
// an ephemeral port that cannot be dialled). Loopback records the
|
||||
// close it would have been asked for.
|
||||
for node in &nodes {
|
||||
let closed = match node.node.transports.get(&wifi()).expect("wifi transport") {
|
||||
TransportHandle::Loopback(t) => t.closed(),
|
||||
_ => unreachable!("tests run over loopback"),
|
||||
};
|
||||
assert!(
|
||||
closed.is_empty(),
|
||||
"the dial's connection is kept for the candidate: {closed:?}"
|
||||
);
|
||||
}
|
||||
|
||||
// The heartbeat tick proves it, at both ends, under the shared session.
|
||||
nodes[1].node.run_path_heartbeats().await;
|
||||
for _ in 0..8 {
|
||||
if process_available_packets(&mut nodes).await == 0 {
|
||||
break;
|
||||
}
|
||||
}
|
||||
let p1 = nodes[1].node.get_peer(&addr_0).unwrap();
|
||||
assert_eq!(p1.path_on(wifi()).unwrap().state(), PathState::Live);
|
||||
assert_eq!(p1.transport_id(), Some(cable), "traffic still on the cable");
|
||||
let p0 = nodes[0].node.get_peer(&addr_1).unwrap();
|
||||
assert!(
|
||||
p0.path_on(wifi()).is_some(),
|
||||
"the probe taught the responder the path"
|
||||
);
|
||||
assert_eq!(p0.transport_id(), Some(cable));
|
||||
|
||||
// An operator disconnect closes every path's connection, the standby's
|
||||
// included: on a connection-oriented transport each holds a pool entry.
|
||||
let npub_0 = nodes[0].node.identity().npub();
|
||||
nodes[1]
|
||||
.node
|
||||
.api_disconnect(&npub_0)
|
||||
.await
|
||||
.expect("disconnect");
|
||||
let closed_wifi = match nodes[1].node.transports.get(&wifi()).unwrap() {
|
||||
TransportHandle::Loopback(t) => t.closed(),
|
||||
_ => unreachable!(),
|
||||
};
|
||||
assert_eq!(
|
||||
closed_wifi,
|
||||
vec![wifi_0.clone()],
|
||||
"the standby's connection is closed too"
|
||||
);
|
||||
let closed_cable = match nodes[1].node.transports.get(&cable).unwrap() {
|
||||
TransportHandle::Loopback(t) => t.closed(),
|
||||
_ => unreachable!(),
|
||||
};
|
||||
assert_eq!(closed_cable.len(), 1, "and the active path's");
|
||||
}
|
||||
|
||||
#[tokio::test]
|
||||
async fn a_beaconed_path_is_probed_by_the_next_heartbeat_tick_and_once_only() {
|
||||
let (mut nodes, wifi_0, _wifi_1) = dual_homed_pair().await;
|
||||
@@ -509,24 +638,6 @@ async fn an_ack_for_no_outstanding_probe_changes_nothing() {
|
||||
);
|
||||
}
|
||||
|
||||
#[tokio::test]
|
||||
async fn the_active_path_is_not_probed() {
|
||||
let (mut nodes, _wifi_0, _wifi_1) = dual_homed_pair().await;
|
||||
let addr_0 = *nodes[0].node.node_addr();
|
||||
let cable = nodes[0].transport_id;
|
||||
let cable_addr_0 = nodes[0].addr.clone();
|
||||
|
||||
nodes[1]
|
||||
.node
|
||||
.maybe_probe_path(addr_0, cable, cable_addr_0)
|
||||
.await;
|
||||
assert_eq!(
|
||||
nodes[0].packet_rx.len(),
|
||||
0,
|
||||
"the handshake proved the active path"
|
||||
);
|
||||
}
|
||||
|
||||
// ============================================================================
|
||||
// Presence loss withdraws a path (design §5–6)
|
||||
// ============================================================================
|
||||
@@ -537,14 +648,8 @@ async fn pair_with_wifi_live() -> (Vec<TestNode>, TransportAddr, TransportAddr)
|
||||
let (mut nodes, wifi_0, wifi_1) = dual_homed_pair().await;
|
||||
let addr_0 = *nodes[0].node.node_addr();
|
||||
let addr_1 = *nodes[1].node.node_addr();
|
||||
nodes[1]
|
||||
.node
|
||||
.maybe_probe_path(addr_0, wifi(), wifi_0.clone())
|
||||
.await;
|
||||
nodes[0]
|
||||
.node
|
||||
.maybe_probe_path(addr_1, wifi(), wifi_1.clone())
|
||||
.await;
|
||||
probe_candidate(&mut nodes, 1, addr_0, wifi(), wifi_0.clone()).await;
|
||||
probe_candidate(&mut nodes, 0, addr_1, wifi(), wifi_1.clone()).await;
|
||||
for _ in 0..4 {
|
||||
if process_available_packets(&mut nodes).await == 0 {
|
||||
break;
|
||||
@@ -595,7 +700,9 @@ async fn losing_the_active_transport_moves_traffic_to_the_live_standby() {
|
||||
assert_eq!(seeded_by, Some(wifi()));
|
||||
|
||||
// The same session carries on: a frame sent now goes out on the wifi
|
||||
// and decrypts at the far end, which never saw a switch.
|
||||
// and decrypts at the far end. (The withdrawal also told the far end,
|
||||
// with a PathClose on the wifi, that the cable is gone: it moves its
|
||||
// own traffic to the wifi on hearing it, and re-peers nowhere.)
|
||||
let before = nodes[0].packet_rx.len();
|
||||
nodes[1]
|
||||
.node
|
||||
@@ -603,15 +710,21 @@ async fn losing_the_active_transport_moves_traffic_to_the_live_standby() {
|
||||
.await
|
||||
.expect("send over the standby");
|
||||
assert_eq!(nodes[0].packet_rx.len(), before + 1);
|
||||
let packet = nodes[0].packet_rx.try_recv().unwrap();
|
||||
assert_eq!(packet.transport_id, wifi());
|
||||
nodes[0].node.handle_encrypted_frame(packet).await;
|
||||
while let Ok(packet) = nodes[0].packet_rx.try_recv() {
|
||||
assert_eq!(packet.transport_id, wifi());
|
||||
nodes[0].node.handle_encrypted_frame(packet).await;
|
||||
}
|
||||
let far = nodes[0]
|
||||
.node
|
||||
.get_peer(nodes[1].node.node_addr())
|
||||
.expect("no re-peering");
|
||||
assert_eq!(far.consecutive_decrypt_failures(), 0);
|
||||
assert_eq!(far.transport_id(), Some(cable), "the far end did not move");
|
||||
assert_eq!(nodes[0].node.peer_count(), 1);
|
||||
assert_eq!(
|
||||
far.transport_id(),
|
||||
Some(wifi()),
|
||||
"told the cable is gone, the far end moved too"
|
||||
);
|
||||
}
|
||||
|
||||
#[tokio::test]
|
||||
@@ -621,10 +734,7 @@ async fn losing_the_active_transport_with_only_a_probing_standby_reaps() {
|
||||
let cable = nodes[1].transport_id;
|
||||
|
||||
// Probe sent, ack never processed: the wifi path is unproven.
|
||||
nodes[1]
|
||||
.node
|
||||
.maybe_probe_path(addr_0, wifi(), wifi_0.clone())
|
||||
.await;
|
||||
probe_candidate(&mut nodes, 1, addr_0, wifi(), wifi_0.clone()).await;
|
||||
assert_eq!(
|
||||
nodes[1]
|
||||
.node
|
||||
@@ -662,21 +772,39 @@ async fn a_dead_path_is_reprobed_when_its_transport_returns_and_forgotten_after_
|
||||
let (mut nodes, wifi_0, _wifi_1) = pair_with_wifi_live().await;
|
||||
let addr_0 = *nodes[0].node.node_addr();
|
||||
|
||||
nodes[1].node.withdraw_transport(wifi()).await;
|
||||
|
||||
// Presence returns: the next discovery tick probes it again, and the
|
||||
// ack brings it back Live with its history. (The withdrawal also sent
|
||||
// node 0 a PathClose on the cable, which it processes here too.)
|
||||
nodes[1].node.reset_probe_backoff_on_transport(wifi());
|
||||
nodes[1]
|
||||
let samples_before = nodes[1]
|
||||
.node
|
||||
.maybe_probe_path(addr_0, wifi(), wifi_0.clone())
|
||||
.await;
|
||||
.get_peer(&addr_0)
|
||||
.unwrap()
|
||||
.path_on(wifi())
|
||||
.unwrap()
|
||||
.rtt_samples();
|
||||
assert!(samples_before > 0);
|
||||
nodes[1].node.withdraw_transport(wifi()).await;
|
||||
// (The withdrawal also sent node 0 a PathClose on the cable.)
|
||||
for _ in 0..4 {
|
||||
if process_available_packets(&mut nodes).await == 0 {
|
||||
break;
|
||||
}
|
||||
}
|
||||
let dead = nodes[1]
|
||||
.node
|
||||
.get_peer(&addr_0)
|
||||
.unwrap()
|
||||
.path_on(wifi())
|
||||
.unwrap();
|
||||
assert_eq!(dead.state(), PathState::Dead);
|
||||
assert_eq!(dead.addr(), &wifi_0);
|
||||
// Dead: the heartbeat tick leaves it alone.
|
||||
nodes[1].node.run_path_heartbeats().await;
|
||||
assert!(
|
||||
nodes[0].packet_rx.try_recv().is_err(),
|
||||
"a Dead path is not probed"
|
||||
);
|
||||
|
||||
// Presence returns: the path is Probing again, the next heartbeat tick
|
||||
// probes it, and the ack brings it back Live with its history.
|
||||
nodes[1].node.reset_probe_backoff_on_transport(wifi());
|
||||
assert_eq!(
|
||||
nodes[1]
|
||||
.node
|
||||
@@ -685,7 +813,24 @@ async fn a_dead_path_is_reprobed_when_its_transport_returns_and_forgotten_after_
|
||||
.path_on(wifi())
|
||||
.unwrap()
|
||||
.state(),
|
||||
PathState::Live
|
||||
PathState::Probing
|
||||
);
|
||||
nodes[1].node.run_path_heartbeats().await;
|
||||
for _ in 0..4 {
|
||||
if process_available_packets(&mut nodes).await == 0 {
|
||||
break;
|
||||
}
|
||||
}
|
||||
let revived = nodes[1]
|
||||
.node
|
||||
.get_peer(&addr_0)
|
||||
.unwrap()
|
||||
.path_on(wifi())
|
||||
.unwrap();
|
||||
assert_eq!(revived.state(), PathState::Live);
|
||||
assert!(
|
||||
revived.rtt_samples() > samples_before,
|
||||
"re-proved on top of its history, not from nothing"
|
||||
);
|
||||
|
||||
// Dead again, and this time the grace expires.
|
||||
@@ -751,6 +896,49 @@ async fn garbage_on_a_standby_path_counts_against_the_peer() {
|
||||
);
|
||||
}
|
||||
|
||||
#[tokio::test]
|
||||
async fn garbage_on_a_path_the_peer_never_acknowledged_is_not_counted() {
|
||||
// A candidate is an address we were told about — a beacon, a config
|
||||
// entry, a handshake source, possibly a replayed one. Until the peer
|
||||
// answers a probe there, garbage on its transport says nothing about
|
||||
// the peer, and cannot tear the peering down.
|
||||
let (mut nodes, _wifi_0, wifi_1) = dual_homed_pair().await;
|
||||
let addr_1 = *nodes[1].node.node_addr();
|
||||
nodes[0]
|
||||
.node
|
||||
.add_path_candidate(addr_1, wifi(), wifi_1.clone());
|
||||
assert!(
|
||||
!nodes[0]
|
||||
.node
|
||||
.get_peer(&addr_1)
|
||||
.unwrap()
|
||||
.path_on(wifi())
|
||||
.unwrap()
|
||||
.acked_once()
|
||||
);
|
||||
let our_index = nodes[0]
|
||||
.node
|
||||
.get_peer(&addr_1)
|
||||
.unwrap()
|
||||
.our_index()
|
||||
.unwrap();
|
||||
for counter in 0..(THRESHOLD * 2) as u64 {
|
||||
nodes[0]
|
||||
.node
|
||||
.handle_encrypted_frame(ReceivedPacket::new(
|
||||
wifi(),
|
||||
wifi_1.clone(),
|
||||
garbage_frame(our_index, counter),
|
||||
))
|
||||
.await;
|
||||
}
|
||||
let peer = nodes[0]
|
||||
.node
|
||||
.get_peer(&addr_1)
|
||||
.expect("an unproven path is not a transport the peer is on");
|
||||
assert_eq!(peer.consecutive_decrypt_failures(), 0);
|
||||
}
|
||||
|
||||
// ============================================================================
|
||||
// Selection (design §8)
|
||||
// ============================================================================
|
||||
@@ -1039,24 +1227,124 @@ fn a_timed_out_echo_on_an_acknowledged_path_makes_it_suspect_and_selection_leave
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn a_path_the_peer_never_acknowledged_backs_off_instead_of_going_suspect() {
|
||||
fn a_peer_with_one_live_path_is_not_heartbeated() {
|
||||
let mut peer = ActivePeer::new(make_peer_identity(), LinkId::new(1), 0);
|
||||
peer.rebind_transport(tid(CABLE), TransportAddr::from_string("10.0.0.1:1"));
|
||||
// Promotion-style: Live and tx_live from the handshake, never acked.
|
||||
// Promotion-style: one Live path from the handshake. Nothing to decide,
|
||||
// nothing sent — the link heartbeat keeps liveness.
|
||||
let t0 = 1_000_000;
|
||||
for k in 0..20 {
|
||||
assert!(
|
||||
peer.plan_heartbeats(t0 + k * FAST, &TIMING)
|
||||
.sends
|
||||
.is_empty(),
|
||||
"a single-path peer gets no path probes"
|
||||
);
|
||||
}
|
||||
// A candidate makes it two: probes start, on the candidate and on the
|
||||
// active path alike.
|
||||
peer.add_path(tid(WIFI), TransportAddr::from_string("10.0.0.7:1"));
|
||||
let plan = peer.plan_heartbeats(t0 + 21 * FAST, &TIMING);
|
||||
assert_eq!(plan.sends.len(), 2);
|
||||
assert!(
|
||||
plan.sends
|
||||
.iter()
|
||||
.any(|s| s.transport_id == tid(WIFI) && s.full_size),
|
||||
"the candidate's first probe is the full-size discovery probe"
|
||||
);
|
||||
assert!(
|
||||
plan.sends
|
||||
.iter()
|
||||
.any(|s| s.transport_id == tid(CABLE) && !s.full_size),
|
||||
"the handshake proved the active path: its first probe is small"
|
||||
);
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn an_active_path_the_peer_never_acknowledged_backs_off_instead_of_going_suspect() {
|
||||
let mut peer = ActivePeer::new(make_peer_identity(), LinkId::new(1), 0);
|
||||
peer.rebind_transport(tid(CABLE), TransportAddr::from_string("10.0.0.1:1"));
|
||||
// A candidate on the wifi makes the peer two-path, so the cable — Live
|
||||
// from the handshake, never acked: an old node — is probed too.
|
||||
peer.add_path(tid(WIFI), TransportAddr::from_string("10.0.0.7:1"));
|
||||
let t0 = 1_000_000;
|
||||
let plan = peer.plan_heartbeats(t0, &TIMING);
|
||||
assert_eq!(plan.sends.len(), 1);
|
||||
assert!(plan.sends.iter().any(|s| s.transport_id == tid(CABLE)));
|
||||
let plan = peer.plan_heartbeats(t0 + TIMEOUT, &TIMING);
|
||||
assert!(plan.suspects.is_empty(), "an old node is not a dead path");
|
||||
let cable = peer.path_on(tid(CABLE)).unwrap();
|
||||
assert_eq!(cable.state(), PathState::Live);
|
||||
assert_eq!(plan.sends.len(), 1, "tried again, with the backoff doubled");
|
||||
assert!(
|
||||
peer.plan_heartbeats(t0 + TIMEOUT + 2 * FAST - 1, &TIMING)
|
||||
plan.sends.iter().any(|s| s.transport_id == tid(CABLE)),
|
||||
"tried again, with the backoff doubled"
|
||||
);
|
||||
assert!(
|
||||
!peer
|
||||
.plan_heartbeats(t0 + TIMEOUT + 2 * FAST - 1, &TIMING)
|
||||
.sends
|
||||
.is_empty(),
|
||||
.iter()
|
||||
.any(|s| s.transport_id == tid(CABLE)),
|
||||
"the unanswered probe pushed the next one out"
|
||||
);
|
||||
// However long it goes unanswered, the active path is never given up:
|
||||
// the handshake proved it, and an old node answers no probe.
|
||||
let mut t = t0 + TIMEOUT + 2 * FAST;
|
||||
for _ in 0..200 {
|
||||
peer.plan_heartbeats(t, &TIMING);
|
||||
t += TIMEOUT;
|
||||
}
|
||||
assert_eq!(peer.path_on(tid(CABLE)).unwrap().state(), PathState::Live);
|
||||
assert_eq!(peer.transport_id(), Some(tid(CABLE)));
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn a_standby_the_peer_never_acknowledges_is_given_up_after_the_discovery_budget() {
|
||||
let mut peer = dual_path_peer(1, 5);
|
||||
// A third address the peer never answers on: a NIC it no longer sends
|
||||
// from, a replayed source, an old node's transport.
|
||||
peer.add_path(tid(3), TransportAddr::from_string("10.0.0.9:1"));
|
||||
let mut t = 1_000_000;
|
||||
let mut probes = 0;
|
||||
for _ in 0..400 {
|
||||
let plan = peer.plan_heartbeats(t, &TIMING);
|
||||
assert!(
|
||||
plan.suspects.is_empty(),
|
||||
"never acknowledged: never Suspect"
|
||||
);
|
||||
probes += plan
|
||||
.sends
|
||||
.iter()
|
||||
.filter(|s| s.transport_id == tid(3))
|
||||
.count();
|
||||
for s in plan.sends {
|
||||
if s.transport_id != tid(3) {
|
||||
peer.note_path_ack(s.transport_id, s.probe_id, false, 1, t + 1, u64::MAX);
|
||||
}
|
||||
}
|
||||
if peer.path_on(tid(3)).unwrap().state() == PathState::Dead {
|
||||
break;
|
||||
}
|
||||
t += TIMEOUT;
|
||||
}
|
||||
assert_eq!(
|
||||
peer.path_on(tid(3)).unwrap().state(),
|
||||
PathState::Dead,
|
||||
"given up after the budget"
|
||||
);
|
||||
assert_eq!(probes, crate::peer::MAX_DISCOVERY_PROBES as usize);
|
||||
// Dead: not probed again, and forgotten after the grace.
|
||||
assert!(
|
||||
!peer
|
||||
.plan_heartbeats(t + TIMEOUT, &TIMING)
|
||||
.sends
|
||||
.iter()
|
||||
.any(|s| s.transport_id == tid(3))
|
||||
);
|
||||
peer.prune_dead_paths(t + 10 * 60_000, 5 * 60_000);
|
||||
assert!(peer.path_on(tid(3)).is_none());
|
||||
// The proven paths are untouched.
|
||||
assert_eq!(peer.path_on(tid(CABLE)).unwrap().state(), PathState::Live);
|
||||
assert_eq!(peer.path_on(tid(WIFI)).unwrap().state(), PathState::Live);
|
||||
}
|
||||
|
||||
#[test]
|
||||
@@ -1097,22 +1385,31 @@ fn withdrawing_our_only_path_leaves_it_suspect_and_still_probed() {
|
||||
fn presence_return_resets_the_discovery_backoff() {
|
||||
let mut peer = ActivePeer::new(make_peer_identity(), LinkId::new(1), 0);
|
||||
peer.rebind_transport(tid(CABLE), TransportAddr::from_string("10.0.0.1:1"));
|
||||
// Two-path, so the never-acked cable is probed at all.
|
||||
peer.add_path(tid(WIFI), TransportAddr::from_string("10.0.0.7:1"));
|
||||
let t0 = 1_000_000;
|
||||
let on_cable = |plan: &crate::peer::HeartbeatPlan| {
|
||||
plan.sends
|
||||
.iter()
|
||||
.filter(|s| s.transport_id == tid(CABLE))
|
||||
.count()
|
||||
};
|
||||
// Never answered: each timeout doubles the wait.
|
||||
assert_eq!(peer.plan_heartbeats(t0, &TIMING).sends.len(), 1);
|
||||
assert_eq!(peer.plan_heartbeats(t0 + TIMEOUT, &TIMING).sends.len(), 1);
|
||||
assert_eq!(on_cable(&peer.plan_heartbeats(t0, &TIMING)), 1);
|
||||
assert_eq!(on_cable(&peer.plan_heartbeats(t0 + TIMEOUT, &TIMING)), 1);
|
||||
assert_eq!(
|
||||
peer.plan_heartbeats(t0 + 2 * TIMEOUT, &TIMING).sends.len(),
|
||||
on_cable(&peer.plan_heartbeats(t0 + 2 * TIMEOUT, &TIMING)),
|
||||
1
|
||||
);
|
||||
let t = t0 + 3 * TIMEOUT;
|
||||
assert!(
|
||||
peer.plan_heartbeats(t, &TIMING).sends.is_empty(),
|
||||
assert_eq!(
|
||||
on_cable(&peer.plan_heartbeats(t, &TIMING)),
|
||||
0,
|
||||
"the third timeout pushed the next probe past now"
|
||||
);
|
||||
// The transport's presence cycles: probed at once.
|
||||
peer.reset_probe_backoff_on(tid(CABLE));
|
||||
assert_eq!(peer.plan_heartbeats(t, &TIMING).sends.len(), 1);
|
||||
assert_eq!(on_cable(&peer.plan_heartbeats(t, &TIMING)), 1);
|
||||
}
|
||||
|
||||
#[test]
|
||||
@@ -1127,7 +1424,7 @@ fn the_discovery_backoff_is_capped_and_never_goes_suspect() {
|
||||
discovery_cap_ms: 2_000,
|
||||
..TIMING
|
||||
};
|
||||
// Drive the wifi standby through many unanswered probes.
|
||||
// Drive the wifi standby through its unanswered probes.
|
||||
let mut t = 1_000_000;
|
||||
let mut sent_at = Vec::new();
|
||||
for _ in 0..40 {
|
||||
@@ -1136,9 +1433,19 @@ fn the_discovery_backoff_is_capped_and_never_goes_suspect() {
|
||||
plan.suspects.is_empty(),
|
||||
"never acknowledged: not a dead path"
|
||||
);
|
||||
assert!(
|
||||
plan.sends
|
||||
.iter()
|
||||
.filter(|s| s.transport_id == tid(WIFI))
|
||||
.all(|s| s.full_size),
|
||||
"every probe on an unproven path is full-size"
|
||||
);
|
||||
if plan.sends.iter().any(|s| s.transport_id == tid(WIFI)) {
|
||||
sent_at.push(t);
|
||||
}
|
||||
if peer.path_on(tid(WIFI)).unwrap().state() == PathState::Dead {
|
||||
break;
|
||||
}
|
||||
t += TIMEOUT;
|
||||
}
|
||||
let gaps: Vec<u64> = sent_at.windows(2).map(|w| w[1] - w[0]).collect();
|
||||
@@ -1147,14 +1454,12 @@ fn the_discovery_backoff_is_capped_and_never_goes_suspect() {
|
||||
gaps.iter().all(|g| *g <= 2_000 + TIMEOUT),
|
||||
"the backoff is capped at discovery_cap_ms: {gaps:?}"
|
||||
);
|
||||
assert!(
|
||||
peer.plan_heartbeats(t, &timing)
|
||||
.sends
|
||||
.iter()
|
||||
.filter(|s| s.transport_id == tid(WIFI))
|
||||
.all(|s| s.full_size),
|
||||
"every probe on an unproven path is full-size"
|
||||
assert_eq!(
|
||||
sent_at.len(),
|
||||
crate::peer::MAX_DISCOVERY_PROBES as usize,
|
||||
"and the budget bounds it: the path is given up, not probed forever"
|
||||
);
|
||||
assert_eq!(peer.path_on(tid(WIFI)).unwrap().state(), PathState::Dead);
|
||||
}
|
||||
|
||||
#[test]
|
||||
@@ -1298,7 +1603,7 @@ fn unreachable_send_errors_are_classified() {
|
||||
|
||||
#[tokio::test]
|
||||
async fn the_fast_tick_heartbeats_the_active_path_and_the_ack_measures_it() {
|
||||
let (mut nodes, _wifi_0, _wifi_1) = pair_with_wifi_live().await;
|
||||
let (mut nodes, wifi_0, _wifi_1) = dual_homed_pair().await;
|
||||
let addr_0 = *nodes[0].node.node_addr();
|
||||
let cable = nodes[1].transport_id;
|
||||
assert!(
|
||||
@@ -1312,6 +1617,11 @@ async fn the_fast_tick_heartbeats_the_active_path_and_the_ack_measures_it() {
|
||||
"the handshake proved the cable; no probe has yet"
|
||||
);
|
||||
|
||||
// A second path makes the peer worth heartbeating; the tick then
|
||||
// probes the active cable as well as the wifi candidate.
|
||||
nodes[1]
|
||||
.node
|
||||
.add_path_candidate(addr_0, wifi(), wifi_0.clone());
|
||||
nodes[1].node.run_path_heartbeats().await;
|
||||
let queued = nodes[0].packet_rx.len();
|
||||
assert!(queued >= 1, "a heartbeat probe went out on the cable");
|
||||
@@ -1556,11 +1866,16 @@ fn a_probe_reaching_a_dead_path_revives_it() {
|
||||
async fn the_first_probe_on_a_path_is_full_size_and_so_is_its_ack() {
|
||||
let (mut nodes, wifi_0, _wifi_1) = dual_homed_pair().await;
|
||||
let addr_0 = *nodes[0].node.node_addr();
|
||||
nodes[1]
|
||||
.node
|
||||
.maybe_probe_path(addr_0, wifi(), wifi_0.clone())
|
||||
.await;
|
||||
let probe = nodes[0].packet_rx.try_recv().expect("probe queued");
|
||||
probe_candidate(&mut nodes, 1, addr_0, wifi(), wifi_0.clone()).await;
|
||||
// The tick also heartbeated the active cable, small; the wifi probe
|
||||
// is the discovery one.
|
||||
let mut probe = None;
|
||||
while let Ok(packet) = nodes[0].packet_rx.try_recv() {
|
||||
if packet.transport_id == wifi() {
|
||||
probe = Some(packet);
|
||||
}
|
||||
}
|
||||
let probe = probe.expect("probe queued");
|
||||
let mtu = usize::from(
|
||||
nodes[1]
|
||||
.node
|
||||
@@ -1592,9 +1907,14 @@ fn a_proven_path_is_probed_full_size_once_a_minute() {
|
||||
let t0 = 1_000_000;
|
||||
let plan = peer.plan_heartbeats(t0, &TIMING);
|
||||
// dual_path_peer acked via take_probe, which sets no full-size stamp,
|
||||
// so the first heartbeat is full-size; after that, small until a
|
||||
// minute has passed.
|
||||
assert!(plan.sends.iter().all(|s| s.full_size));
|
||||
// so the standby's first heartbeat is full-size — the active path's is
|
||||
// not, the handshake having proved it; after that, small until a
|
||||
// minute has passed, then full-size on both.
|
||||
assert!(
|
||||
plan.sends
|
||||
.iter()
|
||||
.all(|s| s.full_size == (s.transport_id == tid(WIFI)))
|
||||
);
|
||||
for s in plan.sends {
|
||||
peer.note_path_ack(s.transport_id, s.probe_id, false, 1, t0 + 1, u64::MAX);
|
||||
}
|
||||
@@ -1611,7 +1931,8 @@ fn a_proven_path_is_probed_full_size_once_a_minute() {
|
||||
);
|
||||
}
|
||||
let plan = peer.plan_heartbeats(t0 + 61_000, &TIMING);
|
||||
assert!(plan.sends.iter().any(|s| s.full_size));
|
||||
assert_eq!(plan.sends.len(), 2);
|
||||
assert!(plan.sends.iter().all(|s| s.full_size));
|
||||
}
|
||||
|
||||
#[tokio::test]
|
||||
@@ -1773,3 +2094,108 @@ fn an_outage_does_not_keep_charging_the_path_s_etx() {
|
||||
"an outage is one event, not a lossy medium"
|
||||
);
|
||||
}
|
||||
|
||||
// ---------------------------------------------------------------------------
|
||||
// A handshake proves a path
|
||||
// ---------------------------------------------------------------------------
|
||||
|
||||
#[test]
|
||||
fn a_known_transport_at_a_new_address_is_re_pointed_there() {
|
||||
let mut peer = ActivePeer::new(make_peer_identity(), LinkId::new(1), 0);
|
||||
peer.rebind_transport(tid(CABLE), TransportAddr::from_string("10.0.0.1:1"));
|
||||
peer.add_path(tid(WIFI), TransportAddr::from_string("10.0.0.7:1"));
|
||||
|
||||
// `add_path` is "add or return": the address it carries is ignored for
|
||||
// a path that exists. A move is a separate, explicit step.
|
||||
peer.add_path(tid(WIFI), TransportAddr::from_string("10.0.0.8:1"));
|
||||
assert_eq!(
|
||||
peer.path_on(tid(WIFI)).unwrap().addr(),
|
||||
&TransportAddr::from_string("10.0.0.7:1")
|
||||
);
|
||||
assert!(peer.refresh_path_addr(tid(WIFI), TransportAddr::from_string("10.0.0.8:1")));
|
||||
assert_eq!(
|
||||
peer.path_on(tid(WIFI)).unwrap().addr(),
|
||||
&TransportAddr::from_string("10.0.0.8:1")
|
||||
);
|
||||
assert!(
|
||||
!peer.refresh_path_addr(tid(WIFI), TransportAddr::from_string("10.0.0.8:1")),
|
||||
"the same address is not a move"
|
||||
);
|
||||
assert!(
|
||||
!peer.refresh_path_addr(tid(3), TransportAddr::from_string("10.0.0.9:1")),
|
||||
"a transport with no path is not re-pointed: that is add_path's job"
|
||||
);
|
||||
assert!(peer.path_on(tid(3)).is_none());
|
||||
}
|
||||
|
||||
#[tokio::test]
|
||||
async fn a_dial_to_a_stale_peer_over_another_transport_is_probed_and_taken_when_it_answers() {
|
||||
let (mut nodes, wifi_0, _wifi_1) = dual_homed_pair().await;
|
||||
let addr_0 = *nodes[0].node.node_addr();
|
||||
let cable = nodes[0].transport_id;
|
||||
|
||||
// Node 1 has not heard node 0 in a long time: the cable is not live.
|
||||
nodes[1].node.get_peer_mut(&addr_0).unwrap().touch(0);
|
||||
assert!(!nodes[1].node.active_peer_link_is_live(&addr_0));
|
||||
|
||||
// An address on another transport arrives. Discovery never dials a
|
||||
// peer with a session; a caller that does gets the same outcome: the
|
||||
// handshake settles as a cross-connection, the address is a candidate.
|
||||
let identity_0 = PeerIdentity::from_pubkey_full(nodes[0].node.identity().pubkey_full());
|
||||
nodes[1]
|
||||
.node
|
||||
.initiate_connection(wifi(), wifi_0.clone(), identity_0)
|
||||
.await
|
||||
.expect("dial starts");
|
||||
for _ in 0..8 {
|
||||
if process_available_packets(&mut nodes).await == 0 {
|
||||
break;
|
||||
}
|
||||
}
|
||||
assert_eq!(nodes[0].node.peer_count(), 1);
|
||||
assert_eq!(nodes[1].node.peer_count(), 1);
|
||||
let p = nodes[1].node.get_peer(&addr_0).unwrap();
|
||||
assert_eq!(
|
||||
p.transport_id(),
|
||||
Some(cable),
|
||||
"nothing moved on the handshake"
|
||||
);
|
||||
assert_eq!(p.path_on(wifi()).unwrap().state(), PathState::Probing);
|
||||
|
||||
// The heartbeat tick proves the wifi. The cable's own echoes then time
|
||||
// out (the peer is silent there, which is why we are here): a hard
|
||||
// signal, and the mandatory switch takes the path that answered.
|
||||
nodes[1].node.run_path_heartbeats().await;
|
||||
for _ in 0..8 {
|
||||
if process_available_packets(&mut nodes).await == 0 {
|
||||
break;
|
||||
}
|
||||
}
|
||||
assert!(
|
||||
nodes[1]
|
||||
.node
|
||||
.get_peer_mut(&addr_0)
|
||||
.unwrap()
|
||||
.mark_path_suspect(cable)
|
||||
);
|
||||
nodes[1].node.run_path_selection();
|
||||
let p = nodes[1].node.get_peer(&addr_0).unwrap();
|
||||
assert_eq!(p.path_on(wifi()).unwrap().state(), PathState::Live);
|
||||
assert_eq!(
|
||||
p.transport_id(),
|
||||
Some(wifi()),
|
||||
"traffic moved to the path that answered"
|
||||
);
|
||||
assert_eq!(p.current_addr(), Some(&wifi_0));
|
||||
// The link record followed the traffic.
|
||||
let link = nodes[1].node.links.get(&p.link_id()).expect("peer's link");
|
||||
assert_eq!(link.transport_id(), wifi());
|
||||
assert_eq!(link.remote_addr(), &wifi_0);
|
||||
assert!(
|
||||
p.path_on(cable).is_some(),
|
||||
"the cable stays for the heartbeat tick to probe"
|
||||
);
|
||||
// And the session is still shared.
|
||||
assert_eq!(nodes[0].node.peer_count(), 1);
|
||||
assert_eq!(nodes[1].node.peer_count(), 1);
|
||||
}
|
||||
|
||||
@@ -1433,7 +1433,7 @@ async fn node_context_mirrors_config_and_immutable_facades() {
|
||||
}
|
||||
|
||||
#[tokio::test]
|
||||
async fn update_peers_races_new_alternative_without_dropping_active_peer() {
|
||||
async fn update_peers_takes_a_new_alternative_as_a_path_without_dropping_active_peer() {
|
||||
// The node's *current* (pre-update) peer set must contain `old_peer`, so it
|
||||
// is baked into the Config at construction (immutable context = sole store).
|
||||
let peer_full = Identity::generate();
|
||||
@@ -1497,16 +1497,18 @@ async fn update_peers_races_new_alternative_without_dropping_active_peer() {
|
||||
|
||||
assert_eq!(outcome.updated, 1);
|
||||
assert_eq!(node.peer_count(), 1, "existing link must stay live");
|
||||
assert_eq!(node.connection_count(), 1);
|
||||
assert_eq!(
|
||||
node.connections()
|
||||
.next()
|
||||
.and_then(|(_, machine)| machine.conn_source_addr()),
|
||||
Some(&new_addr)
|
||||
node.connection_count(),
|
||||
0,
|
||||
"a peer with a session is not dialled on a new address: the address is a path"
|
||||
);
|
||||
let active = node.get_peer(&peer_node_addr).unwrap();
|
||||
assert_eq!(active.link_id(), old_link_id);
|
||||
assert_eq!(active.current_addr(), Some(¤t_addr));
|
||||
// The path on this transport was never acknowledged (bound by
|
||||
// `set_current_addr`, no ack), so it is not eligible and the new address
|
||||
// re-points it; the heartbeat tick probes it there. An eligible path
|
||||
// would have kept its address.
|
||||
assert_eq!(active.current_addr(), Some(&new_addr));
|
||||
|
||||
for transport in node.transports.values_mut() {
|
||||
transport.stop().await.ok();
|
||||
|
||||
+124
-28
@@ -22,6 +22,16 @@ use std::time::{Duration, Instant};
|
||||
/// How often a full-size (MTU-padded) probe goes out on a proven path.
|
||||
const FULL_SIZE_PROBE_INTERVAL_MS: u64 = 60_000;
|
||||
|
||||
/// Discovery probes a never-acknowledged standby path gets before it is
|
||||
/// given up as `Dead`. At the doubling backoff from the fast interval,
|
||||
/// capped at the heartbeat interval, that is about half a minute. An
|
||||
/// address we were told about but the peer never answers on — an old
|
||||
/// node's transport, a replayed handshake source, a beacon from a NIC the
|
||||
/// peer no longer sends on — is then no longer probed, and no longer
|
||||
/// counts as a transport the peer is on. `prune_dead_paths` forgets it
|
||||
/// after the grace; a fresh candidate starts the count over.
|
||||
pub const MAX_DISCOVERY_PROBES: u32 = 8;
|
||||
|
||||
/// Fold one probe outcome into a path's ETX: the long EWMA (α = 1/32) of
|
||||
/// the delivery ratio, inverted and clamped like the link ETX. Per report a
|
||||
/// raw value is a flap generator on a lightly loaded link; the long average
|
||||
@@ -86,6 +96,20 @@ pub enum PathState {
|
||||
Dead,
|
||||
}
|
||||
|
||||
impl PathState {
|
||||
/// The control-socket spelling: `probing`, `live`, `suspect`, `dead`.
|
||||
/// A fixed string per variant, so a rename here cannot silently change
|
||||
/// what `show_peers` and `path_show` emit and what fipstop matches.
|
||||
pub fn as_str(self) -> &'static str {
|
||||
match self {
|
||||
Self::Probing => "probing",
|
||||
Self::Live => "live",
|
||||
Self::Suspect => "suspect",
|
||||
Self::Dead => "dead",
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
/// Probe bookkeeping for one path: what is outstanding, and when the next
|
||||
/// one may go.
|
||||
#[derive(Clone, Copy, Debug, Default)]
|
||||
@@ -120,16 +144,6 @@ pub struct PathPolicy {
|
||||
pub rtt_window_ms: u64,
|
||||
}
|
||||
|
||||
impl PathPolicy {
|
||||
/// Everything selectable at once; for tests.
|
||||
pub const PERMISSIVE: Self = Self {
|
||||
margin: 1.5,
|
||||
dwell_ms: 0,
|
||||
min_samples: 0,
|
||||
rtt_window_ms: u64::MAX,
|
||||
};
|
||||
}
|
||||
|
||||
/// A post-switch hold on the link cost the tree sees.
|
||||
#[derive(Clone, Copy, Debug)]
|
||||
struct CostHold {
|
||||
@@ -1111,6 +1125,34 @@ impl ActivePeer {
|
||||
&mut self.send.paths[idx]
|
||||
}
|
||||
|
||||
/// The peer's address on `transport_id` is `addr` now. Roams the path
|
||||
/// there, if any, without touching its state; a `Dead` path brought back
|
||||
/// at a new address is `Probing` again. Returns whether anything changed.
|
||||
/// Nothing happens for a transport the peer has no path on — that is
|
||||
/// [`add_path`](Self::add_path)'s job — nor for the same address.
|
||||
///
|
||||
/// A peer's address on a transport does change under it: a Wi-Fi Aware
|
||||
/// data path that re-forms comes up with a new link-local, and an
|
||||
/// address that has moved is not one a probe can reach. Without this the
|
||||
/// path kept the dead address for as long as it lived.
|
||||
pub fn refresh_path_addr(&mut self, transport_id: TransportId, addr: TransportAddr) -> bool {
|
||||
let Some(path) = self.path_on_mut(transport_id) else {
|
||||
return false;
|
||||
};
|
||||
if path.addr == addr {
|
||||
return false;
|
||||
}
|
||||
path.addr = addr;
|
||||
#[cfg(any(target_os = "linux", target_os = "macos"))]
|
||||
path.clear_connected_udp();
|
||||
if path.state == PathState::Dead {
|
||||
path.state = PathState::Probing;
|
||||
path.dead_since_ms = None;
|
||||
}
|
||||
path.probe.next_at_ms = 0;
|
||||
true
|
||||
}
|
||||
|
||||
/// An authentic frame arrived on `transport_id`: the path there, if any,
|
||||
/// is `rx_live` as of `now_ms`.
|
||||
pub fn note_path_rx(&mut self, transport_id: TransportId, now_ms: u64) {
|
||||
@@ -1125,9 +1167,10 @@ impl ActivePeer {
|
||||
/// backoff has not expired. `backoff_cap_ms` bounds the retry interval,
|
||||
/// which doubles from `base_ms` per unanswered probe.
|
||||
///
|
||||
/// Tests only. In production [`plan_heartbeats`](Self::plan_heartbeats)
|
||||
/// is the one issuer of probes, so no two writers race for
|
||||
/// `probe.outstanding`.
|
||||
/// Unit-test sampler for the selection tests, which need a path fed a
|
||||
/// chosen round trip without driving a whole heartbeat schedule. Every
|
||||
/// node-level test goes through [`plan_heartbeats`](Self::plan_heartbeats),
|
||||
/// the one issuer of probes in production.
|
||||
#[cfg(test)]
|
||||
pub fn take_probe(
|
||||
&mut self,
|
||||
@@ -1163,12 +1206,11 @@ impl ActivePeer {
|
||||
remote_id: u32,
|
||||
now_ms: u64,
|
||||
) {
|
||||
let path = self.add_path(transport_id, addr.clone());
|
||||
if path.addr != addr {
|
||||
path.addr = addr;
|
||||
#[cfg(any(target_os = "linux", target_os = "macos"))]
|
||||
path.clear_connected_udp();
|
||||
}
|
||||
self.add_path(transport_id, addr.clone());
|
||||
self.refresh_path_addr(transport_id, addr);
|
||||
let path = self
|
||||
.path_on_mut(transport_id)
|
||||
.expect("path added just above");
|
||||
path.remote_id = Some(remote_id);
|
||||
if path.state == PathState::Dead {
|
||||
// The peer is probing a path we had given up on: it is back,
|
||||
@@ -1497,6 +1539,15 @@ impl ActivePeer {
|
||||
/// The silence hint: our active path silent for two of the peer's
|
||||
/// intervals on it while a standby hears the peer triggers a probe now,
|
||||
/// never `Suspect` (see §7 for the loop that would otherwise follow).
|
||||
///
|
||||
/// A peer with one `Live` path is not heartbeated here at all.
|
||||
/// Selection has nothing to move to, so a `Suspect` mark on it changes
|
||||
/// nothing, and the link heartbeat already keeps its liveness; five
|
||||
/// probes a second on every single-path link would be cost without a
|
||||
/// decision behind it. Probes start the moment a second path — a
|
||||
/// candidate included — exists, which is when a verdict can act; and a
|
||||
/// lone path that is `Suspect` (the peer closed it, nowhere to go) is
|
||||
/// probed so its ack can bring it back.
|
||||
pub fn plan_heartbeats(&mut self, now_ms: u64, timing: &HeartbeatTiming) -> HeartbeatPlan {
|
||||
let HeartbeatTiming {
|
||||
fast_ms,
|
||||
@@ -1506,6 +1557,23 @@ impl ActivePeer {
|
||||
} = *timing;
|
||||
let mut plan = HeartbeatPlan::default();
|
||||
let active = self.send.active;
|
||||
let alone_and_live = self
|
||||
.send
|
||||
.paths
|
||||
.iter()
|
||||
.filter(|p| p.state != PathState::Dead)
|
||||
.count()
|
||||
< 2
|
||||
&& active
|
||||
.and_then(|i| self.send.paths.get(i))
|
||||
.is_some_and(|p| p.state == PathState::Live);
|
||||
if alone_and_live {
|
||||
for path in self.send.paths.iter_mut() {
|
||||
path.probe.outstanding = None;
|
||||
path.probe.timed_out = None;
|
||||
}
|
||||
return plan;
|
||||
}
|
||||
let newest_rx = self.send.paths.iter().filter_map(|p| p.rx_live_at_ms).max();
|
||||
for (i, path) in self.send.paths.iter_mut().enumerate() {
|
||||
if path.state == PathState::Dead {
|
||||
@@ -1549,6 +1617,16 @@ impl ActivePeer {
|
||||
path.probe.next_at_ms = 0;
|
||||
} else {
|
||||
path.probe.unanswered = path.probe.unanswered.saturating_add(1);
|
||||
// A standby the peer has never answered on is given up
|
||||
// after the discovery budget. Never the active path:
|
||||
// an old node answers no probe there either, and the
|
||||
// handshake proved it.
|
||||
if !ours && path.probe.unanswered >= MAX_DISCOVERY_PROBES {
|
||||
path.state = PathState::Dead;
|
||||
path.dead_since_ms = Some(now_ms);
|
||||
path.probe.timed_out = None;
|
||||
continue;
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
@@ -1582,11 +1660,16 @@ impl ActivePeer {
|
||||
.min(discovery_cap_ms.max(interval))
|
||||
};
|
||||
path.probe.next_at_ms = now_ms.saturating_add(delay.max(1));
|
||||
let full_size = !path.acked_once
|
||||
|| path
|
||||
.last_full_probe_ms
|
||||
.is_none_or(|t| now_ms.saturating_sub(t) >= FULL_SIZE_PROBE_INTERVAL_MS);
|
||||
if full_size {
|
||||
// Discovery probes are full-size so a medium that passes small
|
||||
// frames and drops large ones never proves itself — except on
|
||||
// the active path, which the handshake proved and whose MTU it
|
||||
// seeded; there the once-a-minute full-size probe is enough.
|
||||
let full_size = (!path.acked_once && !ours)
|
||||
|| match path.last_full_probe_ms {
|
||||
Some(t) => now_ms.saturating_sub(t) >= FULL_SIZE_PROBE_INTERVAL_MS,
|
||||
None => !ours,
|
||||
};
|
||||
if full_size || path.last_full_probe_ms.is_none() {
|
||||
path.last_full_probe_ms = Some(now_ms);
|
||||
}
|
||||
plan.sends.push(HeartbeatSend {
|
||||
@@ -1658,11 +1741,24 @@ impl ActivePeer {
|
||||
self.send.active = Some(new_active);
|
||||
}
|
||||
|
||||
/// Clear the probe backoff on every path over `transport_id`.
|
||||
pub fn reset_probe_backoff_on(&mut self, transport_id: TransportId) {
|
||||
if let Some(path) = self.path_on_mut(transport_id) {
|
||||
path.reset_probe_backoff();
|
||||
/// The transport `transport_id` is back: clear the probe backoff on
|
||||
/// its path, and a path given up as `Dead` while it was gone is
|
||||
/// `Probing` again, its RTT window and ETX intact, so the next
|
||||
/// heartbeat tick re-proves it rather than measuring it from nothing.
|
||||
/// Returns whether a `Dead` path was revived.
|
||||
pub fn reset_probe_backoff_on(&mut self, transport_id: TransportId) -> bool {
|
||||
let Some(path) = self.path_on_mut(transport_id) else {
|
||||
return false;
|
||||
};
|
||||
path.reset_probe_backoff();
|
||||
if path.state == PathState::Dead {
|
||||
path.state = PathState::Probing;
|
||||
path.dead_since_ms = None;
|
||||
path.probe.outstanding = None;
|
||||
path.probe.timed_out = None;
|
||||
return true;
|
||||
}
|
||||
false
|
||||
}
|
||||
|
||||
// === Handshake Resend ===
|
||||
|
||||
+3
-2
@@ -9,8 +9,9 @@ mod active;
|
||||
pub(crate) mod machine;
|
||||
|
||||
pub use active::{
|
||||
ActivePeer, ConnectivityState, HeartbeatPlan, HeartbeatSend, HeartbeatTiming, PathPolicy,
|
||||
PathState, PathSwitch, PathWithdrawal, PeerPath, SwitchReason,
|
||||
ActivePeer, ConnectivityState, HeartbeatPlan, HeartbeatSend, HeartbeatTiming,
|
||||
MAX_DISCOVERY_PROBES, PathPolicy, PathState, PathSwitch, PathWithdrawal, PeerPath,
|
||||
SwitchReason,
|
||||
};
|
||||
|
||||
use crate::NodeAddr;
|
||||
|
||||
@@ -646,6 +646,13 @@ impl Fmp {
|
||||
|
||||
if snap.has_existing_peer {
|
||||
let peer_addr = *wire.peer_identity.node_addr();
|
||||
// Which transport the msg1 arrived on plays no part here: a
|
||||
// handshake never creates path state. A peer dialling us over
|
||||
// a second transport gets the same answer as one dialling over
|
||||
// the first — a rekey or a duplicate — and the transport becomes
|
||||
// a path only through the authenticated, replay-checked probe
|
||||
// exchange. Both ends then resolve on the same information; a
|
||||
// rule that read this end's view of its own liveness split them.
|
||||
match (snap.existing_peer_epoch, wire.remote_epoch) {
|
||||
(Some(existing), Some(new)) if existing != new => {
|
||||
// Epoch mismatch → peer restart.
|
||||
|
||||
@@ -707,3 +707,29 @@ fn retirements_are_grouped_after_drains_and_before_rekey_initiations() {
|
||||
matches!(actions[4], ConnAction::InitiateRekey { peer } if peer == make_node_addr(0x03))
|
||||
);
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn a_live_peer_s_msg1_is_classified_the_same_on_every_transport() {
|
||||
// Same epoch, session old enough to rekey. The transport the msg1
|
||||
// arrived on is not an input: a handshake never creates path state, so
|
||||
// a live peer dialling over a second transport is the rekey it looks
|
||||
// like, exactly as a dial over the first would be. Both ends then
|
||||
// decide from the same facts.
|
||||
let mut snap = establish_snapshot();
|
||||
snap.has_existing_peer = true;
|
||||
snap.existing_peer_epoch = Some([1u8; 8]);
|
||||
snap.has_session = true;
|
||||
snap.existing_session_age_secs = 31;
|
||||
let wire = wire_outcome(Some([1u8; 8]));
|
||||
assert!(matches!(
|
||||
Fmp::new().establish_inbound(&snap, &wire),
|
||||
InboundDecision::RekeyRespond { .. }
|
||||
));
|
||||
|
||||
// A restart is a restart, whatever transport it arrives on.
|
||||
let restarted = wire_outcome(Some([2u8; 8]));
|
||||
assert!(matches!(
|
||||
Fmp::new().establish_inbound(&snap, &restarted),
|
||||
InboundDecision::RestartThenPromote { .. }
|
||||
));
|
||||
}
|
||||
|
||||
@@ -341,7 +341,6 @@ impl MmpMetrics {
|
||||
}
|
||||
}
|
||||
|
||||
/// Smoothed ETX (long-term EWMA), or `None` if not yet initialized.
|
||||
/// The quality index, [`quality_index`](super::quality_index) of the
|
||||
/// smoothed ETX and the smoothed RTT. `None` until both are measured.
|
||||
pub fn quality_index(&self) -> Option<f64> {
|
||||
@@ -351,6 +350,7 @@ impl MmpMetrics {
|
||||
}
|
||||
}
|
||||
|
||||
/// Smoothed ETX (long-term EWMA), or `None` if not yet initialized.
|
||||
pub fn smoothed_etx(&self) -> Option<f64> {
|
||||
if self.etx_trend.initialized() {
|
||||
Some(self.etx_trend.long())
|
||||
|
||||
@@ -57,6 +57,11 @@ pub struct LoopbackTransport {
|
||||
/// transport's `IFF_RUNNING`. `None`: not interface-bound, no presence
|
||||
/// reported, which is the default.
|
||||
carrier: Mutex<Option<bool>>,
|
||||
/// Every `close_connection` the node asked for, in order. Loopback is
|
||||
/// connectionless so the close itself does nothing; the record lets a
|
||||
/// test say whether a socket a connection-oriented transport would own
|
||||
/// was kept or closed.
|
||||
closed: Mutex<Vec<TransportAddr>>,
|
||||
}
|
||||
|
||||
impl LoopbackTransport {
|
||||
@@ -84,9 +89,20 @@ impl LoopbackTransport {
|
||||
registry,
|
||||
discovered: Mutex::new(Vec::new()),
|
||||
carrier: Mutex::new(None),
|
||||
closed: Mutex::new(Vec::new()),
|
||||
}
|
||||
}
|
||||
|
||||
/// Record a `close_connection` call.
|
||||
pub fn record_close(&self, addr: &TransportAddr) {
|
||||
self.closed.lock().unwrap().push(addr.clone());
|
||||
}
|
||||
|
||||
/// The addresses the node has asked to close on this transport.
|
||||
pub fn closed(&self) -> Vec<TransportAddr> {
|
||||
self.closed.lock().unwrap().clone()
|
||||
}
|
||||
|
||||
/// Pretend this transport is bound to an interface with (or without)
|
||||
/// carrier; `None` returns it to reporting no presence at all.
|
||||
pub fn set_carrier(&self, carrier: Option<bool>) {
|
||||
|
||||
+15
-15
@@ -248,6 +248,20 @@ pub enum TransportError {
|
||||
}
|
||||
|
||||
impl TransportError {
|
||||
/// Whether the kernel refused the send for want of a route: the
|
||||
/// interface is up but nothing is reachable through it. A hard signal
|
||||
/// that the path is gone (`ENETUNREACH`, `EHOSTUNREACH`), distinct from
|
||||
/// `is_transient`: the binder is not going to fix this.
|
||||
pub fn is_unreachable(&self) -> bool {
|
||||
match self {
|
||||
Self::Io(e) => matches!(
|
||||
e.kind(),
|
||||
std::io::ErrorKind::NetworkUnreachable | std::io::ErrorKind::HostUnreachable
|
||||
),
|
||||
_ => false,
|
||||
}
|
||||
}
|
||||
|
||||
/// Whether this failure is expected to clear on its own.
|
||||
///
|
||||
/// The distinction callers need is not *what* went wrong but whether
|
||||
@@ -268,20 +282,6 @@ impl TransportError {
|
||||
/// which is a statement about the peer rather than about this node's
|
||||
/// ability to transmit, and the existing retry paths for them already sit
|
||||
/// at a different layer.
|
||||
/// Whether the kernel refused the send for want of a route: the
|
||||
/// interface is up but nothing is reachable through it. A hard signal
|
||||
/// that the path is gone (`ENETUNREACH`, `EHOSTUNREACH`), distinct from
|
||||
/// `is_transient`: the binder is not going to fix this.
|
||||
pub fn is_unreachable(&self) -> bool {
|
||||
match self {
|
||||
Self::Io(e) => matches!(
|
||||
e.kind(),
|
||||
std::io::ErrorKind::NetworkUnreachable | std::io::ErrorKind::HostUnreachable
|
||||
),
|
||||
_ => false,
|
||||
}
|
||||
}
|
||||
|
||||
pub fn is_transient(&self) -> bool {
|
||||
match self {
|
||||
// The interface is absent or mid-rebind. The binder is polling for
|
||||
@@ -1250,7 +1250,7 @@ impl TransportHandle {
|
||||
#[cfg(ble_available)]
|
||||
TransportHandle::Ble(t) => t.close_connection_async(addr).await,
|
||||
#[cfg(test)]
|
||||
TransportHandle::Loopback(_) => {} // connectionless no-op
|
||||
TransportHandle::Loopback(t) => t.record_close(addr), // connectionless; recorded for tests
|
||||
}
|
||||
}
|
||||
|
||||
|
||||
@@ -115,11 +115,10 @@ Explicit topologies exercising non-UDP transports.
|
||||
`node.path.*`; see the file header for what to read from a run.
|
||||
- **dual-udp-flap**: the all-IP twin (`[n01, n02, udp-veth+udp]`): a veth
|
||||
carrying IP with an interface-bound UDP instance at each end, plus UDP
|
||||
over the bridge. The veth half is handed to the daemon by the runner
|
||||
(`fipsctl connect ... udp/<veth>`) once the pair has peered, and becomes a
|
||||
path under the existing session. Exercises `udp.interface` on the listen
|
||||
and per-peer connected sockets, and a configured address on a new
|
||||
transport becoming a path.
|
||||
over the bridge. Both are static addresses on the dial owner, so two
|
||||
handshakes run at startup and the second is kept as a path under the
|
||||
first's session. Exercises `udp.interface` on the listen and per-peer
|
||||
connected sockets, and a second handshake to a live peer becoming a path.
|
||||
- **tcp-mesh**: 6-node mesh with 4 UDP and 3 TCP edges. Both transports use
|
||||
static peer config. Netem mutation (30% fraction, every 20-40s) and link
|
||||
flaps (1 link max, 10-20s down).
|
||||
|
||||
@@ -58,24 +58,31 @@ def generate_peers_block(
|
||||
for peer_id in sorted(outbound_peers):
|
||||
peer = topology.nodes[peer_id]
|
||||
transport = topology.transport_for_edge(node_id, peer_id)
|
||||
if topology.is_dual_udp_edge(node_id, peer_id):
|
||||
# The veth half of a dual edge is found by beacon (Ethernet) or
|
||||
# added as a path by the runner after the pair has peered
|
||||
# (udp-veth); the bridge half is dialled from here.
|
||||
transport = "udp"
|
||||
dual = topology.is_dual_udp_edge(node_id, peer_id)
|
||||
# (transport, addr, priority) per address. A dual edge's Ethernet
|
||||
# half is found by beacon, so only its bridge half is listed; a dual
|
||||
# udp-veth edge lists both halves, bridge first, and the daemon takes
|
||||
# the second completed handshake as a path under the first's session.
|
||||
addresses = []
|
||||
if transport == UDP_VETH:
|
||||
link = next(l for l in topology.udp_veth_links(node_id) if l.peer_id == peer_id)
|
||||
transport = f"udp/{link.instance}"
|
||||
addr = link.peer_addr
|
||||
if dual:
|
||||
addresses.append((bridge, f"{peer.docker_ip}:{_TRANSPORT_PORTS['udp']}", 1))
|
||||
addresses.append((f"udp/{link.instance}", link.peer_addr, 10 if dual else 1))
|
||||
else:
|
||||
if dual:
|
||||
transport = "udp"
|
||||
addr = f"{peer.docker_ip}:{_TRANSPORT_PORTS.get(transport, 2121)}"
|
||||
if transport == "udp":
|
||||
transport = bridge
|
||||
addresses.append((transport, addr, 1))
|
||||
lines.append(f' - npub: "{peer.npub}"')
|
||||
lines.append(f' alias: "{peer_id}"')
|
||||
lines.append(f" addresses:")
|
||||
lines.append(f" - transport: {transport}")
|
||||
lines.append(f' addr: "{addr}"')
|
||||
for transport, addr, priority in addresses:
|
||||
lines.append(f" - transport: {transport}")
|
||||
lines.append(f' addr: "{addr}"')
|
||||
lines.append(f" priority: {priority}")
|
||||
lines.append(f" connect_policy: auto_connect")
|
||||
return "\n".join(lines)
|
||||
|
||||
|
||||
@@ -389,11 +389,6 @@ class SimRunner:
|
||||
self._sleep(wait)
|
||||
self._take_snapshot("warmup")
|
||||
|
||||
# The veth half of every udp-veth+udp edge: UDP has no beacon, so
|
||||
# the runner hands the daemon the address once the pair has peered
|
||||
# over the bridge, and it becomes a path under that session.
|
||||
self._add_udp_veth_paths()
|
||||
|
||||
# Populate npub cache after convergence (nodes must be running)
|
||||
if self.peer_churn_mgr:
|
||||
self.peer_churn_mgr.refresh_all_npubs()
|
||||
@@ -404,67 +399,13 @@ class SimRunner:
|
||||
if self.link_swap_mgr:
|
||||
self.link_swap_mgr.setup_initial()
|
||||
|
||||
def _add_udp_veth_paths(self, only_node: str | None = None):
|
||||
"""Give each dual udp-veth edge its veth path.
|
||||
|
||||
Sent from the edge's dial owner (the side whose static config holds
|
||||
the bridge address) as a control-socket ``connect`` naming the
|
||||
interface-bound instance: to a peer it already holds a session with,
|
||||
the daemon adds that as a path rather than dialling. Waits for the
|
||||
bridge session first, so the command cannot become the first dial.
|
||||
"""
|
||||
from .control import send_command
|
||||
|
||||
outbound = self.topology.directed_outbound()
|
||||
for node_id in sorted(self.topology.nodes):
|
||||
if only_node is not None and node_id != only_node:
|
||||
continue
|
||||
for link in self.topology.udp_veth_links(node_id):
|
||||
if not self.topology.is_dual_udp_edge(node_id, link.peer_id):
|
||||
continue
|
||||
if link.peer_id not in outbound.get(node_id, []):
|
||||
continue
|
||||
if node_id in self._down_nodes or link.peer_id in self._down_nodes:
|
||||
continue
|
||||
container = self.topology.container_name(node_id)
|
||||
npub = self.topology.nodes[link.peer_id].npub
|
||||
params = {
|
||||
"npub": npub,
|
||||
"address": link.peer_addr,
|
||||
"transport": f"udp/{link.instance}",
|
||||
}
|
||||
added = None
|
||||
for _ in range(30):
|
||||
if send_command(container, "path_show", {"npub": npub}) is None:
|
||||
time.sleep(1) # not peered over the bridge yet
|
||||
continue
|
||||
added = send_command(container, "connect", params)
|
||||
break
|
||||
if added is None:
|
||||
log.warning(
|
||||
"udp-veth path %s -> %s via %s not added",
|
||||
node_id, link.peer_id, link.instance,
|
||||
)
|
||||
else:
|
||||
log.info(
|
||||
"udp-veth path %s -> %s via %s (%s)",
|
||||
node_id, link.peer_id, link.instance, link.peer_addr,
|
||||
)
|
||||
|
||||
def _handle_node_restart(self, node_id: str):
|
||||
"""Called after a node container is restarted.
|
||||
|
||||
Re-adds the node's udp-veth paths once it has re-peered, and for
|
||||
ephemeral identity nodes waits briefly for the daemon to start,
|
||||
then queries its new npub and updates the peer churn manager's
|
||||
cache.
|
||||
For ephemeral identity nodes, waits briefly for the daemon to
|
||||
start, then queries its new npub and updates the peer churn
|
||||
manager's cache.
|
||||
"""
|
||||
if any(
|
||||
self.topology.is_dual_udp_edge(node_id, link.peer_id)
|
||||
for link in self.topology.udp_veth_links(node_id)
|
||||
):
|
||||
time.sleep(2)
|
||||
self._add_udp_veth_paths(only_node=node_id)
|
||||
if not self.peer_churn_mgr:
|
||||
return
|
||||
if node_id not in self.peer_churn_mgr.ephemeral_nodes:
|
||||
|
||||
Reference in New Issue
Block a user