mirror of
https://github.com/jmcorgan/fips.git
synced 2026-10-05 11:08:25 +00:00
feat(peer): a peer with a session is never dialled; a handshake creates no path state
Three ways an address for a peer we already hold a session with used to reach the dialler — a beacon on a new transport, `update_peers` or `fipsctl connect`, a configured address whose transport came up later — and each was a second handshake, which the far side read as a rekey and this side resolved as a cross-connection, the two not composing. Two phones hearing the same Wi-Fi return at the same moment both dialled at once; one side swapped to its outbound session and freed the index it had just handed out in the rekey reply, the other kept its inbound session and the pre-rekey index, every frame between them was dropped, and the link-dead reap tore the peer down. About a minute dark on every Wi-Fi return. A peer that holds a session is never dialled now. An address on a transport it has no path over becomes a path candidate under that session; one on a transport whose path is not eligible re-points that path (the active path included: it is not answering, that is why we are here); one on a transport whose path is carrying acknowledged traffic changes nothing. The heartbeat tick probes the candidate under the existing session — one authenticated, replay-checked round trip — and the mandatory switch takes it if the current path stops answering. Nothing is lost against the dial: a session that is truly gone answers no probe either, is reaped by the link-dead timeout, and is dialled then; a peer that restarted dials us with a new epoch and wins promotion outright, as before. Applies to the control API's connect, to update_peers, to configured addresses (checked once a tick) and to transport discovery alike. The counterpart: a handshake creates no path state. A dial that does reach a peer with a session — a startup that lists two addresses dials both, a caller that still dials by hand, an older node dialling us — is classified and resolved exactly as before this work: rekey, duplicate or restart on the responder, whichever transport the msg1 arrived on; the cross-connection tie-break on the initiator. The address it ran to is left as a candidate for the probe exchange. Two reasons. Both ends must resolve a handshake on the same information, and "is this a new transport to a live peer" was a fact only one end could see. And the IK responder commits at msg1, which carries no freshness beyond the startup epoch: a captured msg1 replayed from any address would otherwise have planted a path, probed full-size for the life of the peering and counting as a transport the peer is on for the decrypt-failure gate. So that gate now counts garbage only on the active path or one the peer has acknowledged. On a connection-oriented transport the connection a dial opened is kept as the candidate's socket rather than closed as the losing leg: the probe rides it, and closing it would only have the first probe dial again — or, at the responder, find an ephemeral port that cannot be dialled at all. `api_disconnect` closes every path's connection, the standby's included; loopback records the closes it is asked for so a test can say so. Path heartbeats are gated and bounded. A peer with one live path is not path-heartbeated: selection has nothing to move to, the link heartbeat keeps liveness, and five probes a second on every single-path link was cost without a decision behind it. A standby the peer never acknowledges is given up after eight discovery probes, Dead and pruned after the grace; the active path is never given up. The active path's first probe is small, the handshake having proved it and seeded its MTU. And a Dead path is probed again when its transport returns: nothing on our side ever re-probed one, so after a NIC replug traffic stayed on the standby until the grace pruned the path and a beacon found it with no history. The presence edge now revives every Dead path on the transport as Probing, RTT window and ETX kept. Smaller: `add_path_candidate` re-points a known transport's path at a moved address (`refresh_path_addr`), for a Wi-Fi Aware data path that re-forms with a new link-local; `api_disconnect` closes every path's connection, not the active one alone; `path_show` is built from the `show_peers` path projection plus the three now-relative fields; `PathState` and `TransportRole` render through `as_str()`; `node.path.switch_margin` is validated finite and at least 1.0; `PathPolicy::PERMISSIVE` had no users; four doc comments an inserted function had split are put back on the function they describe. The dual-udp-flap scenario is config-driven: the dial owner lists udp/main and udp/<veth>, both dial at startup, and the second is proven as a path under the first's session by the probe exchange.
This commit is contained in:
@@ -44,6 +44,23 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
|
|||||||
a path tells the peer with a `PathClose` on a surviving path, so the
|
a path tells the peer with a `PathClose` on a surviving path, so the
|
||||||
peer moves at once rather than after its own timeout.
|
peer moves at once rather than after its own timeout.
|
||||||
|
|
||||||
|
- Connect semantics, for multi-path. A peer that holds a session is never
|
||||||
|
dialled again: an address for it on a transport it has no path over —
|
||||||
|
from a beacon, `update_peers`, `fipsctl connect`, a runtime peer lane,
|
||||||
|
or a configured address whose transport came up later — becomes a
|
||||||
|
candidate path, probed under the session by the next heartbeat tick;
|
||||||
|
one on a transport whose path has stopped answering re-points that
|
||||||
|
path; one on a transport whose path carries acknowledged traffic changes
|
||||||
|
nothing. A handshake never creates path state: a second handshake to a
|
||||||
|
peer with a session (two configured addresses dialled at startup)
|
||||||
|
resolves as it always did — rekey, duplicate or cross-connection
|
||||||
|
tie-break, whichever transport it ran over — and the address it ran to
|
||||||
|
is left as a candidate for the probe exchange. A standby the peer never
|
||||||
|
answers on is given up after eight probes, and until it is answered it
|
||||||
|
does not count as a transport the peer is on for the decrypt-failure
|
||||||
|
gate. On a connection-oriented transport the connection a dial opened
|
||||||
|
is kept as the candidate's socket rather than closed.
|
||||||
|
|
||||||
- Operator surface, for multi-path: `role: backup` on any transport (never
|
- Operator surface, for multi-path: `role: backup` on any transport (never
|
||||||
carries a peer's traffic while a normal path is eligible);
|
carries a peer's traffic while a normal path is eligible);
|
||||||
`fipsctl path show|pin|unpin` and the `path_show`, `path_pin`,
|
`fipsctl path show|pin|unpin` and the `path_show`, `path_pin`,
|
||||||
|
|||||||
@@ -116,7 +116,8 @@ table below lists every command currently registered.
|
|||||||
| ------- | ------ | ----------------------------- |
|
| ------- | ------ | ----------------------------- |
|
||||||
| `show_status` | — | `version`, `npub`, `node_addr`, `ipv6_addr`, `state`, `is_leaf_only`, `is_root` (bool — this node is the spanning-tree root), `root` (hex node-addr of the current tree root), `persistent` (bool — identity is persisted, i.e. `persistent` set or an `nsec` configured), `peer_count`, `session_count`, `link_count`, `transport_count`, `connection_count`, `transport_peer_counts` (object mapping transport-type name to its connected-peer count; configured transports appear with `0`), `tun_state`, `tun_name`, `effective_ipv6_mtu`, `control_socket`, `pid`, `exe_path`, `uptime_secs`, `estimated_mesh_size`, `forwarding`, `sparklines`. |
|
| `show_status` | — | `version`, `npub`, `node_addr`, `ipv6_addr`, `state`, `is_leaf_only`, `is_root` (bool — this node is the spanning-tree root), `root` (hex node-addr of the current tree root), `persistent` (bool — identity is persisted, i.e. `persistent` set or an `nsec` configured), `peer_count`, `session_count`, `link_count`, `transport_count`, `connection_count`, `transport_peer_counts` (object mapping transport-type name to its connected-peer count; configured transports appear with `0`), `tun_state`, `tun_name`, `effective_ipv6_mtu`, `control_socket`, `pid`, `exe_path`, `uptime_secs`, `estimated_mesh_size`, `forwarding`, `sparklines`. |
|
||||||
| `show_acl` | — | `allow_file`, `deny_file`, `enforcement_active`, `effective_mode`, `default_decision`, `allow_all`, `deny_all`, `allow_file_entries`, `deny_file_entries`, `allow_entries`, `deny_entries`. |
|
| `show_acl` | — | `allow_file`, `deny_file`, `enforcement_active`, `effective_mode`, `default_decision`, `allow_all`, `deny_all`, `allow_file_entries`, `deny_file_entries`, `allow_entries`, `deny_entries`. |
|
||||||
| `show_peers` | — | `peers[]` — per-peer object: `node_addr`, `npub`, `display_name`, `ipv6_addr`, `connectivity`, `link_id`, `direction`, `transport_addr`, `transport_type`, `is_parent`, `is_child`, `tree_depth`, `effective_depth` (`tree_depth + link_cost` — the metric `evaluate_parent` ranks on; `null` when the peer has no coords, or is unmeasured while another peer has an SRTT sample, per the cold-start gate), `stats`, `noise`, `current_k_bit`, `mmp`, `paths[]` (every path to the peer: `transport_id`, `transport`, `transport_type`, `addr`, `state`, `active`, `remote_active`, `role`, `pinned`, `last_rtt_ms`, `min_rtt_ms`, `rtt_samples`, `etx`, `score` — the `path_show` fields minus the now-relative ages), plus optional `nostr_traversal`, `rekey_in_progress`, `rekey_draining`. |
|
| `show_peers` | — | `peers[]` — per-peer object: `node_addr`, `npub`, `display_name`, `ipv6_addr`, `connectivity`, `link_id`, `direction`, `transport_addr`, `transport_type`, `is_parent`, `is_child`, `tree_depth`, `effective_depth` (`tree_depth + link_cost` — the metric `evaluate_parent` ranks on; `null` when the peer has no coords, or is unmeasured while another peer has an SRTT sample, per the cold-start gate), `stats`, `noise`, `current_k_bit`, `mmp`, `paths[]` (every path to the peer: `transport_id`, `transport` (instance name or null), `transport_type`, `addr`, `state` (`probing` / `live` / `suspect` / `dead`), `active`, `remote_active`, `role` (`normal` / `backup`), `pinned`, `last_rtt_ms`, `min_rtt_ms`, `rtt_samples`, `etx`, `score`), plus optional `nostr_traversal`, `rekey_in_progress`, `rekey_draining`. |
|
||||||
|
| `path_show` | `npub` (bech32) | Every path to one peer. `data`: `peer`, `link_cost`, `link_cost_held`, and `paths[]` — the `show_peers` per-path object plus `rx_live_ms_ago`, `tx_live_ms_ago` (ms since the last authentic frame heard there / the last ack proving the peer hears us there; `null` if never) and `acked_once`. Takes a parameter, so it is served on the daemon's main task like the mutating commands, not from the snapshot. |
|
||||||
| `show_links` | — | `links[]` — `link_id`, `transport_id`, `remote_addr`, `direction`, `state`, `created_at_ms`, `stats`. |
|
| `show_links` | — | `links[]` — `link_id`, `transport_id`, `remote_addr`, `direction`, `state`, `created_at_ms`, `stats`. |
|
||||||
| `show_tree` | — | `my_node_addr`, `root`, `root_npub` (bech32 npub of the current tree root), `is_root`, `depth`, `my_coords[]`, `parent`, `parent_display_name`, `declaration_sequence`, `declaration_signed`, `peer_tree_count`, `peers[]`, `stats`. |
|
| `show_tree` | — | `my_node_addr`, `root`, `root_npub` (bech32 npub of the current tree root), `is_root`, `depth`, `my_coords[]`, `parent`, `parent_display_name`, `declaration_sequence`, `declaration_signed`, `peer_tree_count`, `peers[]`, `stats`. |
|
||||||
| `show_sessions` | — | `sessions[]` — `remote_addr`, `npub`, `display_name`, `state` (`established`, `initiating`, `awaiting_msg3`, `unknown`), `is_initiator`, `last_activity_ms`, `stats`, optional `mmp`, `current_k_bit`, `is_draining`. |
|
| `show_sessions` | — | `sessions[]` — `remote_addr`, `npub`, `display_name`, `state` (`established`, `initiating`, `awaiting_msg3`, `unknown`), `is_initiator`, `last_activity_ms`, `stats`, optional `mmp`, `current_k_bit`, `is_draining`. |
|
||||||
@@ -173,17 +174,24 @@ not reproduced here to avoid duplicating the source.
|
|||||||
| `probe_start` | `npub` (bech32) | Admits a diagnostic probe job and returns immediately. `data`: `probe_id`, `npub`, `node_addr`, `display_name`, `budget_ms`. |
|
| `probe_start` | `npub` (bech32) | Admits a diagnostic probe job and returns immediately. `data`: `probe_id`, `npub`, `node_addr`, `display_name`, `budget_ms`. |
|
||||||
| `probe_poll` | `probe_id` (integer) | Reports a probe's progress. `data`: `state` (`running` / `done`) and `report`. A terminal job is removed on the poll that observes it, so the report is delivered once. |
|
| `probe_poll` | `probe_id` (integer) | Reports a probe's progress. `data`: `state` (`running` / `done`) and `report`. A terminal job is removed on the poll that observes it, so the report is delivered once. |
|
||||||
| `probe_cancel` | `probe_id` (integer) | Runs the probe's terminal actions immediately, without the teardown grace tick. |
|
| `probe_cancel` | `probe_id` (integer) | Runs the probe's terminal actions immediately, without the teardown grace tick. |
|
||||||
| `path_show` | `npub` (bech32) | Every path to the peer. `data`: `peer`, `link_cost`, `link_cost_held`, and `paths[]` with `transport_id`, `transport` (instance name or null), `addr`, `state`, `active`, `remote_active`, `role`, `pinned`, `rx_live_ms_ago`, `tx_live_ms_ago`, `acked_once`, `last_rtt_ms`, `min_rtt_ms`, `rtt_samples`, `etx`, `score`. |
|
| `path_pin` | `npub` (bech32), `transport` (instance name or numeric id) | Pins this node's traffic to the peer to that transport's path. Applies on the next selection tick, and is suspended while that path is not eligible and re-applied when it is again. `data`: `{"pinned": <transport_id>}`. Error if the peer has no path there. |
|
||||||
| `path_pin` | `npub` (bech32), `transport` (instance name or numeric id) | Pins this node's traffic to the peer to that transport's path. Applies on the next selection tick. Error if the peer has no path there. |
|
| `path_unpin` | `npub` (bech32) | Clears the pin. `data`: `{"pinned": null}`. |
|
||||||
| `path_unpin` | `npub` (bech32) | Clears the pin. |
|
|
||||||
|
|
||||||
`connect` on a peer the node is **already connected to** neither tears the
|
`connect` has three outcomes, told apart by whether the node already holds
|
||||||
live link down nor ignores the address: the address is tried as an alternate
|
a session with the peer and by the response's `refreshed` field:
|
||||||
path alongside the existing one, and the peer moves to it only if that
|
|
||||||
handshake authenticates. The response carries `refreshed` — `true` when such a
|
- **No session:** an ordinary dial over the named transport. `refreshed:
|
||||||
handshake was started, `false` when the peer is already on this exact path and
|
false`; the peer appears in `show_peers` once the handshake completes.
|
||||||
that path is fresh (a successful no-op). A `connect` that starts an ordinary
|
- **Session, and the address is on a transport the peer has no path over,
|
||||||
dial to a peer the node does not yet hold also reports `refreshed: false`.
|
or one whose path has stopped answering:** no handshake. The address
|
||||||
|
becomes a path candidate under the existing session (or re-points the
|
||||||
|
unanswering path), the next heartbeat tick probes it, and selection
|
||||||
|
moves traffic to it if it measures better or the current path stops
|
||||||
|
answering. `refreshed: true`. `path_show` lists it as `probing` until
|
||||||
|
the peer acknowledges, `live` after.
|
||||||
|
- **Session, and the peer is already reachable at exactly that address,
|
||||||
|
or that transport's path is carrying acknowledged traffic:** nothing
|
||||||
|
changes. `refreshed: false`.
|
||||||
|
|
||||||
`connect` is ephemeral either way: the peer is not written to the config file
|
`connect` is ephemeral either way: the peer is not written to the config file
|
||||||
and gets no auto-reconnect, so an attempt that fails leaves no residue.
|
and gets no auto-reconnect, so an attempt that fails leaves no residue.
|
||||||
|
|||||||
@@ -1241,6 +1241,18 @@ impl Config {
|
|||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
|
// Path selection. The margin is the whole fail-back policy: a
|
||||||
|
// standby must beat the active path's score by this factor before
|
||||||
|
// traffic moves. Below 1.0 (or NaN, which compares false both ways)
|
||||||
|
// two paths of equal score would swap after every dwell, each swap
|
||||||
|
// re-seeding the path MTU and holding the tree-visible link cost.
|
||||||
|
let margin = self.node.path.switch_margin;
|
||||||
|
if !margin.is_finite() || margin < 1.0 {
|
||||||
|
return Err(ConfigError::Validation(format!(
|
||||||
|
"`node.path.switch_margin` = {margin} must be a finite number of at least 1.0: it is the factor a standby's score must beat the active path's by, and anything less makes two equal paths swap after every dwell"
|
||||||
|
)));
|
||||||
|
}
|
||||||
|
|
||||||
let native = &self.node.native_api;
|
let native = &self.node.native_api;
|
||||||
// Both floors refuse a node that would start, answer every setup call
|
// Both floors refuse a node that would start, answer every setup call
|
||||||
// and then drop every datagram a peer sent. A zero `backlog` makes the
|
// and then drop every datagram a peer sent. A zero `backlog` makes the
|
||||||
@@ -2557,6 +2569,20 @@ node:
|
|||||||
assert!(config.node.discovery.is_none());
|
assert!(config.node.discovery.is_none());
|
||||||
}
|
}
|
||||||
|
|
||||||
|
#[test]
|
||||||
|
fn a_switch_margin_below_one_is_refused() {
|
||||||
|
// Two paths of equal score would swap after every dwell.
|
||||||
|
for bad in [0.9, 0.0, -1.0, f64::NAN, f64::INFINITY] {
|
||||||
|
let mut config = Config::default();
|
||||||
|
config.node.path.switch_margin = bad;
|
||||||
|
let err = config.validate().expect_err("validation should fail");
|
||||||
|
assert!(err.to_string().contains("switch_margin"), "{bad}: {err}");
|
||||||
|
}
|
||||||
|
let mut config = Config::default();
|
||||||
|
config.node.path.switch_margin = 1.0;
|
||||||
|
config.validate().expect("1.0 means any better path wins");
|
||||||
|
}
|
||||||
|
|
||||||
#[test]
|
#[test]
|
||||||
fn test_a_zero_netmon_poll_interval_is_refused() {
|
fn test_a_zero_netmon_poll_interval_is_refused() {
|
||||||
// It was silently clamped to 1s, so a typo produced a node that polled
|
// It was silently clamped to 1s, so a typo produced a node that polled
|
||||||
|
|||||||
@@ -58,6 +58,18 @@ pub enum TransportRole {
|
|||||||
Backup,
|
Backup,
|
||||||
}
|
}
|
||||||
|
|
||||||
|
impl TransportRole {
|
||||||
|
/// The config and control-socket spelling: `normal`, `backup`. Same
|
||||||
|
/// strings serde reads and writes, fixed here so a variant rename cannot
|
||||||
|
/// silently change what `show_peers` and `path_show` emit.
|
||||||
|
pub fn as_str(self) -> &'static str {
|
||||||
|
match self {
|
||||||
|
Self::Normal => "normal",
|
||||||
|
Self::Backup => "backup",
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
/// UDP transport instance configuration.
|
/// UDP transport instance configuration.
|
||||||
#[derive(Debug, Clone, Default, Serialize, Deserialize)]
|
#[derive(Debug, Clone, Default, Serialize, Deserialize)]
|
||||||
#[serde(deny_unknown_fields)]
|
#[serde(deny_unknown_fields)]
|
||||||
|
|||||||
@@ -408,13 +408,9 @@ pub fn show_peers(node: &Node) -> Value {
|
|||||||
json!({ "peers": peers })
|
json!({ "peers": peers })
|
||||||
}
|
}
|
||||||
|
|
||||||
/// Render a snapshot [`EntityMmp`](super::snapshot::EntityMmp) into the inline
|
/// Render a peer's path rows as the `paths` array of `show_peers`. Also the
|
||||||
/// MMP JSON block, with the quality-index key named `quality_key` (`lqi` for
|
/// base of `path_show`'s rows, which add the now-relative fields on top.
|
||||||
/// peers, `sqi` for sessions). Reproduces the on-loop key insertion order
|
pub(crate) fn render_peer_paths(paths: &[super::snapshot::PeerPathRow]) -> Value {
|
||||||
/// exactly. `path_mtu` is emitted (inside the leading literal) only when
|
|
||||||
/// present (session-layer); for peers it is `None` and omitted.
|
|
||||||
/// Render a peer's path rows as the `paths` array of `show_peers`.
|
|
||||||
fn render_peer_paths(paths: &[super::snapshot::PeerPathRow]) -> Value {
|
|
||||||
Value::Array(
|
Value::Array(
|
||||||
paths
|
paths
|
||||||
.iter()
|
.iter()
|
||||||
@@ -440,6 +436,11 @@ fn render_peer_paths(paths: &[super::snapshot::PeerPathRow]) -> Value {
|
|||||||
)
|
)
|
||||||
}
|
}
|
||||||
|
|
||||||
|
/// Render a snapshot [`EntityMmp`](super::snapshot::EntityMmp) into the inline
|
||||||
|
/// MMP JSON block, with the quality-index key named `quality_key` (`lqi` for
|
||||||
|
/// peers, `sqi` for sessions). Reproduces the on-loop key insertion order
|
||||||
|
/// exactly. `path_mtu` is emitted (inside the leading literal) only when
|
||||||
|
/// present (session-layer); for peers it is `None` and omitted.
|
||||||
fn render_entity_mmp(mmp: &super::snapshot::EntityMmp, quality_key: &str) -> Value {
|
fn render_entity_mmp(mmp: &super::snapshot::EntityMmp, quality_key: &str) -> Value {
|
||||||
// The on-loop `show_sessions` block places loss_rate/etx/goodput_bps/
|
// The on-loop `show_sessions` block places loss_rate/etx/goodput_bps/
|
||||||
// delivery ratios/path_mtu in the leading json! literal, while `show_peers`
|
// delivery ratios/path_mtu in the leading json! literal, while `show_peers`
|
||||||
|
|||||||
@@ -551,6 +551,12 @@ impl Node {
|
|||||||
/// landing on an index the allocator has already handed to a new owner:
|
/// landing on an index the allocator has already handed to a new owner:
|
||||||
/// index reuse is immediate, with no quarantine.
|
/// index reuse is immediate, with no quarantine.
|
||||||
///
|
///
|
||||||
|
/// Only a path the peer has proven counts: the active one, or one the
|
||||||
|
/// peer has acknowledged a probe on. A `Probing` path is an address we
|
||||||
|
/// were *told* about — a beacon, a config entry, a handshake source —
|
||||||
|
/// and until the peer answers there, garbage arriving on its transport
|
||||||
|
/// says nothing about the peer.
|
||||||
|
///
|
||||||
/// A peer with no transport bound yet is charged unconditionally, as
|
/// A peer with no transport bound yet is charged unconditionally, as
|
||||||
/// before.
|
/// before.
|
||||||
pub(in crate::node) fn charge_decrypt_failure(
|
pub(in crate::node) fn charge_decrypt_failure(
|
||||||
@@ -558,10 +564,13 @@ impl Node {
|
|||||||
node_addr: &crate::NodeAddr,
|
node_addr: &crate::NodeAddr,
|
||||||
transport_id: crate::transport::TransportId,
|
transport_id: crate::transport::TransportId,
|
||||||
) {
|
) {
|
||||||
let on_path = self
|
let on_path = self.peers.get(node_addr).is_some_and(|peer| {
|
||||||
.peers
|
peer.paths().is_empty()
|
||||||
.get(node_addr)
|
|| peer.transport_id() == Some(transport_id)
|
||||||
.is_some_and(|peer| peer.paths().is_empty() || peer.path_on(transport_id).is_some());
|
|| peer
|
||||||
|
.path_on(transport_id)
|
||||||
|
.is_some_and(|path| path.acked_once())
|
||||||
|
});
|
||||||
if !on_path {
|
if !on_path {
|
||||||
trace!(
|
trace!(
|
||||||
peer = %self.peer_display_name(node_addr),
|
peer = %self.peer_display_name(node_addr),
|
||||||
|
|||||||
@@ -345,8 +345,9 @@ impl Node {
|
|||||||
"executor CrossConnectionLost is unreachable on \
|
"executor CrossConnectionLost is unreachable on \
|
||||||
driven net-new establish paths"
|
driven net-new establish paths"
|
||||||
);
|
);
|
||||||
// Close this (losing) connection, drop its link,
|
// Close this connection, drop its link, and
|
||||||
// and restore `addr_to_link` to the winner.
|
// point `addr_to_link` for the new address at the
|
||||||
|
// winner, so a msg1 from it is recognised.
|
||||||
if let Some(transport) =
|
if let Some(transport) =
|
||||||
self.transports.get(&ambient.transport_id)
|
self.transports.get(&ambient.transport_id)
|
||||||
{
|
{
|
||||||
|
|||||||
@@ -1573,13 +1573,24 @@ impl Node {
|
|||||||
|
|
||||||
// Clean up outbound connection state
|
// Clean up outbound connection state
|
||||||
self.pending_outbound.remove(&key);
|
self.pending_outbound.remove(&key);
|
||||||
// Close the losing TCP connection (no-op for connectionless)
|
|
||||||
|
// The handshake ran over some (transport, address). Whichever
|
||||||
|
// session won, that is where the peer answered just now — but a
|
||||||
|
// handshake creates no path state: the probe exchange, which is
|
||||||
|
// authenticated and replay-checked under the surviving session,
|
||||||
|
// is the one way a path is proven. Leave the address as a
|
||||||
|
// candidate for the heartbeat tick. The link record goes (the
|
||||||
|
// peer's link is the one its session rides), but the transport
|
||||||
|
// connection stays: on TCP, Tor or Nym that socket is what the
|
||||||
|
// probe will go out on, and closing it would only have the
|
||||||
|
// first probe dial it again — or, on the responder, find that
|
||||||
|
// our ephemeral port cannot be dialled at all. `api_disconnect`
|
||||||
|
// closes every path's connection. A connectionless close was a
|
||||||
|
// no-op either way.
|
||||||
if let Some(link) = self.links.get(&link_id) {
|
if let Some(link) = self.links.get(&link_id) {
|
||||||
let tid = link.transport_id();
|
let tid = link.transport_id();
|
||||||
let addr = link.remote_addr().clone();
|
let addr = link.remote_addr().clone();
|
||||||
if let Some(transport) = self.transports.get(&tid) {
|
self.add_path_candidate(peer_node_addr, tid, addr);
|
||||||
transport.close_connection(&addr).await;
|
|
||||||
}
|
|
||||||
}
|
}
|
||||||
self.remove_link(&link_id);
|
self.remove_link(&link_id);
|
||||||
|
|
||||||
|
|||||||
+52
-75
@@ -121,39 +121,24 @@ impl Node {
|
|||||||
}
|
}
|
||||||
|
|
||||||
/// `fipsctl path show <peer>`: every path to the peer, per direction.
|
/// `fipsctl path show <peer>`: every path to the peer, per direction.
|
||||||
|
///
|
||||||
|
/// The per-path object is the `show_peers` one (`project_peer_paths`,
|
||||||
|
/// rendered by `render_peer_paths`) plus the three fields only a
|
||||||
|
/// now-relative read can give: the liveness ages and `acked_once`.
|
||||||
|
/// Built from the same projection so the two field lists cannot drift.
|
||||||
pub(crate) fn api_path_show(&self, npub: &str) -> Result<serde_json::Value, String> {
|
pub(crate) fn api_path_show(&self, npub: &str) -> Result<serde_json::Value, String> {
|
||||||
let node_addr = self.resolve_peer_npub(npub)?;
|
let node_addr = self.resolve_peer_npub(npub)?;
|
||||||
let peer = &self.peers[&node_addr];
|
let peer = &self.peers[&node_addr];
|
||||||
let now_ms = crate::time::mono_ms();
|
let now_ms = crate::time::mono_ms();
|
||||||
let active = peer.transport_id();
|
let ago = |at: Option<u64>| at.map(|t| now_ms.saturating_sub(t));
|
||||||
let paths: Vec<serde_json::Value> = peer
|
let mut paths = crate::control::queries::render_peer_paths(&self.project_peer_paths(peer));
|
||||||
.paths()
|
if let Some(rows) = paths.as_array_mut() {
|
||||||
.iter()
|
for (row, path) in rows.iter_mut().zip(peer.paths()) {
|
||||||
.map(|path| {
|
row["rx_live_ms_ago"] = serde_json::json!(ago(path.rx_live_at_ms()));
|
||||||
let ago = |at: Option<u64>| at.map(|t| now_ms.saturating_sub(t));
|
row["tx_live_ms_ago"] = serde_json::json!(ago(path.tx_live_at_ms()));
|
||||||
serde_json::json!({
|
row["acked_once"] = serde_json::json!(path.acked_once());
|
||||||
"transport_id": path.transport_id().as_u32(),
|
}
|
||||||
"transport": self
|
}
|
||||||
.transports
|
|
||||||
.get(&path.transport_id())
|
|
||||||
.and_then(|t| t.name().map(str::to_string)),
|
|
||||||
"addr": path.addr().to_string(),
|
|
||||||
"state": format!("{:?}", path.state()).to_lowercase(),
|
|
||||||
"active": Some(path.transport_id()) == active,
|
|
||||||
"remote_active": path.remote_active(),
|
|
||||||
"role": format!("{:?}", path.role()).to_lowercase(),
|
|
||||||
"pinned": path.pinned(),
|
|
||||||
"rx_live_ms_ago": ago(path.rx_live_at_ms()),
|
|
||||||
"tx_live_ms_ago": ago(path.tx_live_at_ms()),
|
|
||||||
"acked_once": path.acked_once(),
|
|
||||||
"last_rtt_ms": path.last_rtt_ms(),
|
|
||||||
"min_rtt_ms": path.min_rtt_ms(),
|
|
||||||
"rtt_samples": path.rtt_samples(),
|
|
||||||
"etx": path.etx(),
|
|
||||||
"score": path.score(),
|
|
||||||
})
|
|
||||||
})
|
|
||||||
.collect();
|
|
||||||
Ok(serde_json::json!({
|
Ok(serde_json::json!({
|
||||||
"peer": npub,
|
"peer": npub,
|
||||||
"link_cost": peer.link_cost(now_ms),
|
"link_cost": peer.link_cost(now_ms),
|
||||||
@@ -204,13 +189,28 @@ impl Node {
|
|||||||
let Some(peer) = self.peers.get_mut(&node_addr) else {
|
let Some(peer) = self.peers.get_mut(&node_addr) else {
|
||||||
return;
|
return;
|
||||||
};
|
};
|
||||||
if peer.transport_id() == Some(transport_id) {
|
if peer.transport_id() == Some(transport_id)
|
||||||
// The active path: the handshake proved it.
|
&& peer.path_on(transport_id).is_some_and(|p| p.is_eligible())
|
||||||
|
{
|
||||||
|
// The active path, and it is answering: nothing to add, and an
|
||||||
|
// address that has proven nothing does not displace it. An
|
||||||
|
// active path that has stopped answering falls through to the
|
||||||
|
// re-pointing below like any other.
|
||||||
return;
|
return;
|
||||||
}
|
}
|
||||||
let was_new = peer.path_on(transport_id).is_none();
|
let was_new = peer.path_on(transport_id).is_none();
|
||||||
peer.add_path(transport_id, remote_addr.clone())
|
peer.add_path(transport_id, remote_addr.clone())
|
||||||
.set_role(role);
|
.set_role(role);
|
||||||
|
// A known transport at a new address: the peer moved there (an
|
||||||
|
// Aware data path that re-formed, a DHCP lease that changed) and
|
||||||
|
// the old address answers nothing. Re-point the path; the heartbeat
|
||||||
|
// tick probes it from here. Only while the path is not eligible: an
|
||||||
|
// address that is carrying acknowledged traffic is not displaced by
|
||||||
|
// one that has proven nothing — a probe from the new address
|
||||||
|
// (`note_path_probe`) is what moves a working path.
|
||||||
|
let moved = !was_new
|
||||||
|
&& peer.path_on(transport_id).is_some_and(|p| !p.is_eligible())
|
||||||
|
&& peer.refresh_path_addr(transport_id, remote_addr.clone());
|
||||||
if was_new {
|
if was_new {
|
||||||
debug!(
|
debug!(
|
||||||
peer = %self.peer_display_name(&node_addr),
|
peer = %self.peer_display_name(&node_addr),
|
||||||
@@ -218,47 +218,13 @@ impl Node {
|
|||||||
remote_addr = %remote_addr,
|
remote_addr = %remote_addr,
|
||||||
"Peer beaconed on a new transport; path added, probing"
|
"Peer beaconed on a new transport; path added, probing"
|
||||||
);
|
);
|
||||||
}
|
} else if moved {
|
||||||
}
|
debug!(
|
||||||
|
peer = %self.peer_display_name(&node_addr),
|
||||||
/// Tests: add the path and probe it at once, as one heartbeat tick
|
transport_id = %transport_id,
|
||||||
/// would, without the tick's other sends. Uses the test-only
|
remote_addr = %remote_addr,
|
||||||
/// `take_probe`, which honours the per-path backoff.
|
"Peer beaconed at a new address on a known transport; path re-addressed, probing"
|
||||||
#[cfg(test)]
|
);
|
||||||
pub(in crate::node) async fn maybe_probe_path(
|
|
||||||
&mut self,
|
|
||||||
node_addr: NodeAddr,
|
|
||||||
transport_id: TransportId,
|
|
||||||
remote_addr: TransportAddr,
|
|
||||||
) {
|
|
||||||
self.add_path_candidate(node_addr, transport_id, remote_addr.clone());
|
|
||||||
let now_ms = crate::time::mono_ms();
|
|
||||||
let timing = self.heartbeat_timing();
|
|
||||||
let Some(peer) = self.peers.get_mut(&node_addr) else {
|
|
||||||
return;
|
|
||||||
};
|
|
||||||
if peer.transport_id() == Some(transport_id) {
|
|
||||||
return;
|
|
||||||
}
|
|
||||||
let Some((probe_id, remote_active, path_id)) = peer.take_probe(
|
|
||||||
transport_id,
|
|
||||||
now_ms,
|
|
||||||
timing.fast_ms,
|
|
||||||
timing.discovery_cap_ms,
|
|
||||||
) else {
|
|
||||||
return;
|
|
||||||
};
|
|
||||||
let probe = PathMessage {
|
|
||||||
probe_id,
|
|
||||||
remote_active,
|
|
||||||
path_id,
|
|
||||||
};
|
|
||||||
let wire = self.pad_to_link_mtu(probe.encode_probe().to_vec(), transport_id, &remote_addr);
|
|
||||||
if let Err(e) = self
|
|
||||||
.send_encrypted_link_message_on_path(&node_addr, &wire, transport_id, remote_addr)
|
|
||||||
.await
|
|
||||||
{
|
|
||||||
debug!(peer = %self.peer_display_name(&node_addr), error = %e, "Path probe send failed");
|
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
@@ -766,10 +732,21 @@ impl Node {
|
|||||||
}
|
}
|
||||||
|
|
||||||
/// A transport's presence came back: clear the probe backoff on every
|
/// A transport's presence came back: clear the probe backoff on every
|
||||||
/// path over it so the next discovery tick may probe at once.
|
/// path over it so the next heartbeat tick may probe at once, and
|
||||||
|
/// revive every path that went `Dead` with it — a replugged NIC, a
|
||||||
|
/// wifi interface that cycled — as `Probing`, history kept. Without
|
||||||
|
/// this nothing on our side ever probed a `Dead` path again: traffic
|
||||||
|
/// stayed on the standby until the grace pruned the path and a fresh
|
||||||
|
/// beacon found it with no history.
|
||||||
pub(in crate::node) fn reset_probe_backoff_on_transport(&mut self, transport_id: TransportId) {
|
pub(in crate::node) fn reset_probe_backoff_on_transport(&mut self, transport_id: TransportId) {
|
||||||
for peer in self.peers.values_mut() {
|
for (node_addr, peer) in self.peers.iter_mut() {
|
||||||
peer.reset_probe_backoff_on(transport_id);
|
if peer.reset_probe_backoff_on(transport_id) {
|
||||||
|
debug!(
|
||||||
|
peer = %node_addr,
|
||||||
|
%transport_id,
|
||||||
|
"Transport returned: dead path probing again"
|
||||||
|
);
|
||||||
|
}
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|||||||
+242
-163
@@ -464,6 +464,21 @@ impl Node {
|
|||||||
.map(|t| t.transport_type().connection_oriented)
|
.map(|t| t.transport_type().connection_oriented)
|
||||||
.unwrap_or(false);
|
.unwrap_or(false);
|
||||||
|
|
||||||
|
// A dial to a peer we already hold a session with (a startup that
|
||||||
|
// lists two addresses, a caller that still dials by hand) proves
|
||||||
|
// nothing by itself: the handshake creates no path state, and its
|
||||||
|
// outcome is the cross-connection tie-break as ever. The address
|
||||||
|
// is still a fact worth holding — leave it as a candidate for the
|
||||||
|
// heartbeat tick to probe under the session, on a datagram
|
||||||
|
// transport, so it becomes a path whichever way the dial goes.
|
||||||
|
if !is_connection_oriented && self.peers.contains_key(peer_identity.node_addr()) {
|
||||||
|
self.add_path_candidate(
|
||||||
|
*peer_identity.node_addr(),
|
||||||
|
transport_id,
|
||||||
|
remote_addr.clone(),
|
||||||
|
);
|
||||||
|
}
|
||||||
|
|
||||||
// Allocate link ID and create link
|
// Allocate link ID and create link
|
||||||
let link_id = self.allocate_link_id();
|
let link_id = self.allocate_link_id();
|
||||||
|
|
||||||
@@ -812,51 +827,38 @@ impl Node {
|
|||||||
let connected = self.peers.contains_key(&node_addr);
|
let connected = self.peers.contains_key(&node_addr);
|
||||||
|
|
||||||
if connected {
|
if connected {
|
||||||
// Active peer: skip every candidate while the link we
|
// Active peer: never a dial, whatever the state of the
|
||||||
// already hold is live — the current path *and* any
|
// link we hold. The address is a path to add or
|
||||||
// alternate one.
|
// re-point, and the heartbeat tick probes it under the
|
||||||
|
// session we have — one round trip, authenticated, and
|
||||||
|
// selection moves traffic if the path proves better or
|
||||||
|
// the current one is not answering.
|
||||||
//
|
//
|
||||||
// Only the same-path case used to be skipped, which left
|
// Dialling an active peer used to be the fallback for one
|
||||||
// the stated intent ("avoid churning a healthy link")
|
// that had gone quiet ("liveness is the gate"). Two
|
||||||
// covering exactly the case that could not churn anything.
|
// reasons it is not any more. A handshake to a peer that
|
||||||
// A peer reachable twice — the ordinary result of two
|
// already holds a session is read by the far side as a
|
||||||
// machines sharing a LAN and a cable, since each beacons on
|
// rekey, and when both ends do it at once — the ordinary
|
||||||
// both — was therefore re-dialled on its alternate path
|
// case, since both hear the same medium come back — the
|
||||||
// every discovery tick, forever. Each dial that completed
|
// rekey and the cross-connection resolution overlap and
|
||||||
// promoted and displaced the incumbent, so the peer's link
|
// the two sides part on different session indices, dead
|
||||||
// migrated back and forth on a fixed cadence, tearing down
|
// to each other until the link timeout reaps them. And
|
||||||
// and re-establishing its session each time. Measured on
|
// the dial buys nothing the probe does not: a session
|
||||||
// real hardware: seventeen dials to one peer in fifteen
|
// that is truly gone answers no probe either, is reaped
|
||||||
// minutes, alternating wifi and cable, displacing a link
|
// by the link-dead timeout, and is dialled then; a peer
|
||||||
// reporting `etx = 1.0` and `loss = 0.0`.
|
// that restarted dials us itself with a new epoch and
|
||||||
|
// wins the promotion outright.
|
||||||
//
|
//
|
||||||
// When that peer is the parent — which the best path
|
// (History: only the same-path case used to be skipped,
|
||||||
// usually is — every migration also switched parents,
|
// so a peer reachable twice — two machines sharing a LAN
|
||||||
// invalidating the downstream coordinate cache and
|
// and a cable — was re-dialled on its alternate path
|
||||||
// re-announcing to every peer. The cost of the churn was
|
// every discovery tick, each completed dial displacing
|
||||||
// therefore mesh-wide while the benefit was nil: the link
|
// the incumbent: seventeen dials to one peer in fifteen
|
||||||
// being replaced was already perfect.
|
// minutes, alternating wifi and cable, over a link
|
||||||
//
|
// reporting `etx = 1.0`. Liveness gating fixed that and
|
||||||
// Failover is unaffected. Liveness is the gate, so a peer
|
// left the quiet-peer dial; this removes the last of it.)
|
||||||
// that stops answering goes stale within a heartbeat
|
path_candidates.push((node_addr, candidate_transport_id, remote_addr));
|
||||||
// interval and every path, alternate included, is dialled
|
continue;
|
||||||
// again. What is given up is switching away from a link
|
|
||||||
// that is working, which is not a thing worth doing.
|
|
||||||
if self.active_peer_link_is_live(&node_addr) {
|
|
||||||
// A live peer beaconing on a transport we hold no
|
|
||||||
// path to it over is a path to add, not a link to
|
|
||||||
// replace: the heartbeat tick probes it under the
|
|
||||||
// existing session instead of dialling.
|
|
||||||
path_candidates.push((node_addr, candidate_transport_id, remote_addr));
|
|
||||||
continue;
|
|
||||||
}
|
|
||||||
if self.is_connecting_to_peer_on_path(
|
|
||||||
&node_addr,
|
|
||||||
candidate_transport_id,
|
|
||||||
&remote_addr,
|
|
||||||
) {
|
|
||||||
continue;
|
|
||||||
}
|
|
||||||
} else if self.is_connecting_to_peer_on_path(
|
} else if self.is_connecting_to_peer_on_path(
|
||||||
&node_addr,
|
&node_addr,
|
||||||
candidate_transport_id,
|
candidate_transport_id,
|
||||||
@@ -878,6 +880,7 @@ impl Node {
|
|||||||
for (node_addr, transport_id, remote_addr) in path_candidates {
|
for (node_addr, transport_id, remote_addr) in path_candidates {
|
||||||
self.add_path_candidate(node_addr, transport_id, remote_addr);
|
self.add_path_candidate(node_addr, transport_id, remote_addr);
|
||||||
}
|
}
|
||||||
|
self.add_configured_path_candidates();
|
||||||
|
|
||||||
if transport_neighbors.is_empty() {
|
if transport_neighbors.is_empty() {
|
||||||
return;
|
return;
|
||||||
@@ -2560,6 +2563,126 @@ impl Node {
|
|||||||
.collect()
|
.collect()
|
||||||
}
|
}
|
||||||
|
|
||||||
|
/// Configured addresses of live peers on transports they have no path
|
||||||
|
/// over become paths. Runs every discovery tick, idempotent and cheap:
|
||||||
|
/// a configured address whose transport was down at dial time (wifi
|
||||||
|
/// joined later, Tor came up) is otherwise never looked at again, since
|
||||||
|
/// a peer that is already active is not re-dialled.
|
||||||
|
fn add_configured_path_candidates(&mut self) {
|
||||||
|
let configs: Vec<PeerConfig> = self.config().auto_connect_peers().cloned().collect();
|
||||||
|
for peer_config in configs {
|
||||||
|
let Ok(identity) = PeerIdentity::from_npub(&peer_config.npub) else {
|
||||||
|
continue;
|
||||||
|
};
|
||||||
|
let node_addr = *identity.node_addr();
|
||||||
|
if !self.peers.contains_key(&node_addr) || !self.active_peer_link_is_live(&node_addr) {
|
||||||
|
continue;
|
||||||
|
}
|
||||||
|
for addr in peer_config.addresses_by_priority() {
|
||||||
|
if addr.transport == "udp" && addr.addr.eq_ignore_ascii_case("nat") {
|
||||||
|
continue;
|
||||||
|
}
|
||||||
|
let Some((transport_id, remote_addr)) = self.resolve_peer_address(addr) else {
|
||||||
|
continue;
|
||||||
|
};
|
||||||
|
let has_path = self
|
||||||
|
.peers
|
||||||
|
.get(&node_addr)
|
||||||
|
.is_some_and(|p| p.path_on(transport_id).is_some());
|
||||||
|
if !has_path {
|
||||||
|
self.add_path_candidate(node_addr, transport_id, remote_addr);
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
/// The transport and address a configured peer address dials to, or
|
||||||
|
/// `None` (logged at debug) if no operational transport can carry it.
|
||||||
|
///
|
||||||
|
/// The transport field may name a specific instance (`"udp/aware"`):
|
||||||
|
/// the type half picks the resolver, the instance half is handed to
|
||||||
|
/// whichever resolver can honour it, and only the UDP one can. The
|
||||||
|
/// `"nat"` pseudo-address is not resolved here.
|
||||||
|
fn resolve_peer_address(&self, addr: &PeerAddress) -> Option<(TransportId, TransportAddr)> {
|
||||||
|
let spec = addr.spec();
|
||||||
|
if addr.transport == "ethernet" {
|
||||||
|
return match self.resolve_ethernet_addr(&addr.addr) {
|
||||||
|
Ok(result) => Some(result),
|
||||||
|
Err(e) => {
|
||||||
|
debug!(
|
||||||
|
transport = %addr.transport,
|
||||||
|
addr = %addr.addr,
|
||||||
|
error = %e,
|
||||||
|
"Failed to resolve Ethernet address"
|
||||||
|
);
|
||||||
|
None
|
||||||
|
}
|
||||||
|
};
|
||||||
|
}
|
||||||
|
if addr.transport == "ble" {
|
||||||
|
#[cfg(ble_available)]
|
||||||
|
{
|
||||||
|
return match self.resolve_ble_addr(&addr.addr) {
|
||||||
|
Ok(result) => Some(result),
|
||||||
|
Err(e) => {
|
||||||
|
debug!(
|
||||||
|
transport = %addr.transport,
|
||||||
|
addr = %addr.addr,
|
||||||
|
error = %e,
|
||||||
|
"Failed to resolve BLE address"
|
||||||
|
);
|
||||||
|
None
|
||||||
|
}
|
||||||
|
};
|
||||||
|
}
|
||||||
|
#[cfg(not(ble_available))]
|
||||||
|
{
|
||||||
|
debug!(transport = %addr.transport, "BLE transport not available on this build");
|
||||||
|
return None;
|
||||||
|
}
|
||||||
|
}
|
||||||
|
let tid = if spec.kind == "udp"
|
||||||
|
&& let Ok(remote_socket_addr) = addr.addr.parse::<SocketAddr>()
|
||||||
|
{
|
||||||
|
match self.find_udp_transport_for_remote_addr(remote_socket_addr, spec.instance) {
|
||||||
|
Some((id, _)) => id,
|
||||||
|
None => {
|
||||||
|
debug!(
|
||||||
|
transport = %addr.transport,
|
||||||
|
addr = %addr.addr,
|
||||||
|
"No compatible operational UDP transport for address"
|
||||||
|
);
|
||||||
|
return None;
|
||||||
|
}
|
||||||
|
}
|
||||||
|
} else if spec.instance.is_some() {
|
||||||
|
// Only the UDP resolver above can honour an instance name.
|
||||||
|
// Matching any instance of the type here would be the silent
|
||||||
|
// wrong-lane substitution this whole mechanism exists to
|
||||||
|
// prevent, so refuse instead.
|
||||||
|
debug!(
|
||||||
|
transport = %addr.transport,
|
||||||
|
addr = %addr.addr,
|
||||||
|
"Instance-qualified address for a transport type that \
|
||||||
|
does not support instance selection"
|
||||||
|
);
|
||||||
|
return None;
|
||||||
|
} else {
|
||||||
|
match self.find_transport_for_type(spec.kind) {
|
||||||
|
Some(id) => id,
|
||||||
|
None => {
|
||||||
|
debug!(
|
||||||
|
transport = %addr.transport,
|
||||||
|
addr = %addr.addr,
|
||||||
|
"No operational transport for address type"
|
||||||
|
);
|
||||||
|
return None;
|
||||||
|
}
|
||||||
|
}
|
||||||
|
};
|
||||||
|
Some((tid, TransportAddr::from_string(&addr.addr)))
|
||||||
|
}
|
||||||
|
|
||||||
async fn attempt_peer_address_list(
|
async fn attempt_peer_address_list(
|
||||||
&mut self,
|
&mut self,
|
||||||
peer_config: &PeerConfig,
|
peer_config: &PeerConfig,
|
||||||
@@ -2581,9 +2704,6 @@ impl Node {
|
|||||||
if attempted >= max_attempts {
|
if attempted >= max_attempts {
|
||||||
break;
|
break;
|
||||||
}
|
}
|
||||||
// The transport field may name a specific instance
|
|
||||||
// (`"udp/aware"`); everything below dispatches on the type half
|
|
||||||
// and hands the instance half to whichever resolver can honour it.
|
|
||||||
let spec = addr.spec();
|
let spec = addr.spec();
|
||||||
|
|
||||||
if spec.kind == "udp" && addr.addr.eq_ignore_ascii_case("nat") {
|
if spec.kind == "udp" && addr.addr.eq_ignore_ascii_case("nat") {
|
||||||
@@ -2601,82 +2721,8 @@ impl Node {
|
|||||||
continue;
|
continue;
|
||||||
}
|
}
|
||||||
|
|
||||||
let (transport_id, remote_addr) = if addr.transport == "ethernet" {
|
let Some((transport_id, remote_addr)) = self.resolve_peer_address(addr) else {
|
||||||
match self.resolve_ethernet_addr(&addr.addr) {
|
continue;
|
||||||
Ok(result) => result,
|
|
||||||
Err(e) => {
|
|
||||||
debug!(
|
|
||||||
transport = %addr.transport,
|
|
||||||
addr = %addr.addr,
|
|
||||||
error = %e,
|
|
||||||
"Failed to resolve Ethernet address"
|
|
||||||
);
|
|
||||||
continue;
|
|
||||||
}
|
|
||||||
}
|
|
||||||
} else if addr.transport == "ble" {
|
|
||||||
#[cfg(ble_available)]
|
|
||||||
{
|
|
||||||
match self.resolve_ble_addr(&addr.addr) {
|
|
||||||
Ok(result) => result,
|
|
||||||
Err(e) => {
|
|
||||||
debug!(
|
|
||||||
transport = %addr.transport,
|
|
||||||
addr = %addr.addr,
|
|
||||||
error = %e,
|
|
||||||
"Failed to resolve BLE address"
|
|
||||||
);
|
|
||||||
continue;
|
|
||||||
}
|
|
||||||
}
|
|
||||||
}
|
|
||||||
#[cfg(not(ble_available))]
|
|
||||||
{
|
|
||||||
debug!(transport = %addr.transport, "BLE transport not available on this build");
|
|
||||||
continue;
|
|
||||||
}
|
|
||||||
} else {
|
|
||||||
let tid = if spec.kind == "udp"
|
|
||||||
&& let Ok(remote_socket_addr) = addr.addr.parse::<SocketAddr>()
|
|
||||||
{
|
|
||||||
match self.find_udp_transport_for_remote_addr(remote_socket_addr, spec.instance)
|
|
||||||
{
|
|
||||||
Some((id, _)) => id,
|
|
||||||
None => {
|
|
||||||
debug!(
|
|
||||||
transport = %addr.transport,
|
|
||||||
addr = %addr.addr,
|
|
||||||
"No compatible operational UDP transport for address"
|
|
||||||
);
|
|
||||||
continue;
|
|
||||||
}
|
|
||||||
}
|
|
||||||
} else if spec.instance.is_some() {
|
|
||||||
// Only the UDP resolver above can honour an instance name.
|
|
||||||
// Matching any instance of the type here would be the
|
|
||||||
// silent wrong-lane substitution this whole mechanism
|
|
||||||
// exists to prevent, so refuse instead.
|
|
||||||
debug!(
|
|
||||||
transport = %addr.transport,
|
|
||||||
addr = %addr.addr,
|
|
||||||
"Instance-qualified address for a transport type that \
|
|
||||||
does not support instance selection"
|
|
||||||
);
|
|
||||||
continue;
|
|
||||||
} else {
|
|
||||||
match self.find_transport_for_type(spec.kind) {
|
|
||||||
Some(id) => id,
|
|
||||||
None => {
|
|
||||||
debug!(
|
|
||||||
transport = %addr.transport,
|
|
||||||
addr = %addr.addr,
|
|
||||||
"No operational transport for address type"
|
|
||||||
);
|
|
||||||
continue;
|
|
||||||
}
|
|
||||||
}
|
|
||||||
};
|
|
||||||
(tid, TransportAddr::from_string(&addr.addr))
|
|
||||||
};
|
};
|
||||||
|
|
||||||
if self.is_connecting_to_peer_on_path(&peer_node_addr, transport_id, &remote_addr) {
|
if self.is_connecting_to_peer_on_path(&peer_node_addr, transport_id, &remote_addr) {
|
||||||
@@ -3243,27 +3289,55 @@ impl Node {
|
|||||||
.into_iter()
|
.into_iter()
|
||||||
.filter(|addr| !(addr.transport == "udp" && addr.addr.eq_ignore_ascii_case("nat")))
|
.filter(|addr| !(addr.transport == "udp" && addr.addr.eq_ignore_ascii_case("nat")))
|
||||||
.collect();
|
.collect();
|
||||||
let has_alternative = concrete
|
// Every address the peer is not already on. (The same-path case —
|
||||||
.iter()
|
// the one address it is on, gone quiet — was once re-dialled from
|
||||||
.any(|addr| !self.active_peer_matches_candidate(&peer_node_addr, addr));
|
// here; the path's own heartbeat and the link-dead reap own that
|
||||||
let attempt_candidates: Vec<_> = if has_alternative {
|
// now, see below.)
|
||||||
concrete
|
let attempt_candidates: Vec<_> = concrete
|
||||||
.into_iter()
|
.into_iter()
|
||||||
.filter(|addr| !self.active_peer_matches_candidate(&peer_node_addr, addr))
|
.filter(|addr| !self.active_peer_matches_candidate(&peer_node_addr, addr))
|
||||||
.collect()
|
.collect();
|
||||||
} else if self.active_peer_needs_same_path_refresh(&peer_node_addr) {
|
|
||||||
concrete
|
|
||||||
} else {
|
|
||||||
Vec::new()
|
|
||||||
};
|
|
||||||
|
|
||||||
if attempt_candidates.is_empty() {
|
// A peer we hold a session to gets a *path* at each address, under
|
||||||
return Ok(false);
|
// that session, never a second handshake: the probe exchange proves
|
||||||
|
// the path and selection moves traffic if it measures better or the
|
||||||
|
// current path stops answering. A handshake to a peer that already
|
||||||
|
// has one is read by the far side as a rekey, and two ends doing it
|
||||||
|
// at once — both hearing the same medium return — leave the rekey
|
||||||
|
// and the cross-connection resolution overlapping and the sides on
|
||||||
|
// different session indices. An address the peer is already
|
||||||
|
// reachable at is nothing to do; one on a transport that already
|
||||||
|
// has a path re-points that path (the peer moved); one on a new
|
||||||
|
// transport adds a path. A peer that has gone quiet is not dialled
|
||||||
|
// either: a dead session answers no probe, is reaped by the
|
||||||
|
// link-dead timeout, and is dialled then. See `poll_discovered_peers`.
|
||||||
|
let mut paths_added = false;
|
||||||
|
for addr in attempt_candidates {
|
||||||
|
let Some((transport_id, remote_addr)) = self.resolve_peer_address(&addr) else {
|
||||||
|
continue;
|
||||||
|
};
|
||||||
|
let Some(peer) = self.peers.get(&peer_node_addr) else {
|
||||||
|
continue;
|
||||||
|
};
|
||||||
|
if peer.is_reachable_at(transport_id, &remote_addr) {
|
||||||
|
continue;
|
||||||
|
}
|
||||||
|
let known_transport = peer.path_on(transport_id).is_some();
|
||||||
|
info!(
|
||||||
|
peer = %self.peer_display_name(&peer_node_addr),
|
||||||
|
%transport_id,
|
||||||
|
addr = %remote_addr,
|
||||||
|
"{}",
|
||||||
|
if known_transport {
|
||||||
|
"Configured address moved on a known transport: path re-pointed, not dialled"
|
||||||
|
} else {
|
||||||
|
"Configured address on a new transport: added as a path, not dialled"
|
||||||
|
}
|
||||||
|
);
|
||||||
|
self.add_path_candidate(peer_node_addr, transport_id, remote_addr);
|
||||||
|
paths_added = true;
|
||||||
}
|
}
|
||||||
|
Ok(paths_added)
|
||||||
self.attempt_peer_address_list(peer_config, peer_identity, false, &attempt_candidates)
|
|
||||||
.await?;
|
|
||||||
Ok(true)
|
|
||||||
}
|
}
|
||||||
|
|
||||||
async fn peer_address_candidates(&self, peer_config: &PeerConfig) -> Vec<PeerAddress> {
|
async fn peer_address_candidates(&self, peer_config: &PeerConfig) -> Vec<PeerAddress> {
|
||||||
@@ -3362,14 +3436,17 @@ impl Node {
|
|||||||
/// auto-reconnect). Reuses the same connection path as auto-connect
|
/// auto-reconnect). Reuses the same connection path as auto-connect
|
||||||
/// peers. Returns JSON data on success or an error message.
|
/// peers. Returns JSON data on success or an error message.
|
||||||
///
|
///
|
||||||
/// For a peer the node is already connected to, the supplied address is
|
/// For a peer the node already holds a session with, the supplied
|
||||||
/// tried as an *alternate path* rather than ignored — the same treatment
|
/// address is never dialled: it becomes a path candidate under that
|
||||||
/// [`Node::update_peers`] gives a refreshed runtime peer. The handshake
|
/// session (or re-points a path that has stopped answering), the
|
||||||
/// runs in parallel with the live link and promotion happens only once it
|
/// heartbeat tick probes it, and selection moves traffic there only
|
||||||
/// authenticates, so an address the caller got wrong cannot displace a
|
/// once the peer has answered — the same treatment
|
||||||
/// healthy path. The response's `refreshed` field reports whether such a
|
/// [`Node::update_peers`] gives a refreshed runtime peer, so an address
|
||||||
/// handshake was started; it is `false` when the peer is already on this
|
/// the caller got wrong cannot displace a healthy path. The response's
|
||||||
/// exact path and that path is fresh.
|
/// `refreshed` field reports whether a path was added or re-pointed;
|
||||||
|
/// it is `false` when the peer is already reachable at that address or
|
||||||
|
/// that transport's path is carrying acknowledged traffic, and for an
|
||||||
|
/// ordinary dial to a peer the node does not yet hold.
|
||||||
pub(crate) async fn api_connect(
|
pub(crate) async fn api_connect(
|
||||||
&mut self,
|
&mut self,
|
||||||
npub: &str,
|
npub: &str,
|
||||||
@@ -3451,16 +3528,18 @@ impl Node {
|
|||||||
return Err(format!("peer not found: {npub}"));
|
return Err(format!("peer not found: {npub}"));
|
||||||
};
|
};
|
||||||
|
|
||||||
// Read the transport path the peer is actually sending over BEFORE the
|
// Read every path the peer holds BEFORE the teardown below drops
|
||||||
// teardown below drops the peer and its link — afterwards there is
|
// the peer and its link — afterwards there is nothing left to
|
||||||
// nothing left to derive it from. `current_addr` rather than the
|
// derive them from. The path's address rather than the link's
|
||||||
// link's remote address, because roaming updates the former and it is
|
// remote address, because roaming updates the former and it is the
|
||||||
// the address the pool entry (and its inbound-slot accounting) is
|
// address the pool entry (and its inbound-slot accounting) is keyed
|
||||||
// keyed by.
|
// by. Every path, not the active one alone: a standby on a
|
||||||
let transport_path = match (peer.transport_id(), peer.current_addr()) {
|
// connection-oriented transport holds a pool entry of its own.
|
||||||
(Some(transport_id), Some(addr)) => Some((transport_id, addr.clone())),
|
let transport_paths: Vec<(TransportId, TransportAddr)> = peer
|
||||||
_ => None,
|
.paths()
|
||||||
};
|
.iter()
|
||||||
|
.map(|path| (path.transport_id(), path.addr().clone()))
|
||||||
|
.collect();
|
||||||
|
|
||||||
// Notify the peer before we tear down the link, so it drops its own
|
// Notify the peer before we tear down the link, so it drops its own
|
||||||
// session and re-handshakes symmetrically rather than holding a stale
|
// session and re-handshakes symmetrically rather than holding a stale
|
||||||
@@ -3483,10 +3562,10 @@ impl Node {
|
|||||||
// verbatim: closing twice is harmless, because every
|
// verbatim: closing twice is harmless, because every
|
||||||
// `close_connection` implementation is `if let Some(conn) =
|
// `close_connection` implementation is `if let Some(conn) =
|
||||||
// pool.remove(addr)` and the connectionless default is a no-op.
|
// pool.remove(addr)` and the connectionless default is a no-op.
|
||||||
if let Some((transport_id, addr)) = transport_path
|
for (transport_id, addr) in transport_paths {
|
||||||
&& let Some(transport) = self.transports.get(&transport_id)
|
if let Some(transport) = self.transports.get(&transport_id) {
|
||||||
{
|
transport.close_connection(&addr).await;
|
||||||
transport.close_connection(&addr).await;
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
// Suppress any pending auto-reconnect
|
// Suppress any pending auto-reconnect
|
||||||
|
|||||||
+2
-2
@@ -2697,10 +2697,10 @@ impl Node {
|
|||||||
transport: handle.and_then(|t| t.name().map(str::to_string)),
|
transport: handle.and_then(|t| t.name().map(str::to_string)),
|
||||||
transport_type: handle.map(|t| t.transport_type().name.to_string()),
|
transport_type: handle.map(|t| t.transport_type().name.to_string()),
|
||||||
addr: path.addr().to_string(),
|
addr: path.addr().to_string(),
|
||||||
state: format!("{:?}", path.state()).to_lowercase(),
|
state: path.state().as_str().to_string(),
|
||||||
active: Some(path.transport_id()) == active,
|
active: Some(path.transport_id()) == active,
|
||||||
remote_active: path.remote_active(),
|
remote_active: path.remote_active(),
|
||||||
role: format!("{:?}", path.role()).to_lowercase(),
|
role: path.role().as_str().to_string(),
|
||||||
pinned: path.pinned(),
|
pinned: path.pinned(),
|
||||||
last_rtt_ms: path.last_rtt_ms(),
|
last_rtt_ms: path.last_rtt_ms(),
|
||||||
min_rtt_ms: path.min_rtt_ms(),
|
min_rtt_ms: path.min_rtt_ms(),
|
||||||
|
|||||||
+14
-17
@@ -150,13 +150,11 @@ async fn test_api_connect_on_current_fresh_path_is_a_no_op() {
|
|||||||
}
|
}
|
||||||
|
|
||||||
/// `connect` naming a *different* address for a peer the node is already
|
/// `connect` naming a *different* address for a peer the node is already
|
||||||
/// connected to starts an alternate-path handshake instead of silently doing
|
/// connected to takes it as a path under the session it has — never a
|
||||||
/// nothing — the fix.
|
/// second handshake, which the far side would read as a rekey. The peer and
|
||||||
///
|
/// its link stay put; the heartbeat tick probes the path from here.
|
||||||
/// The existing peer stays put while that handshake runs: promotion is the
|
|
||||||
/// handshake's job, not the command's.
|
|
||||||
#[tokio::test]
|
#[tokio::test]
|
||||||
async fn test_api_connect_starts_alternate_path_for_active_peer() {
|
async fn test_api_connect_takes_an_alternate_address_as_a_path() {
|
||||||
let mut nodes = run_tree_test(2, &[(0, 1)], false).await;
|
let mut nodes = run_tree_test(2, &[(0, 1)], false).await;
|
||||||
|
|
||||||
let node1_addr = *nodes[1].node.node_addr();
|
let node1_addr = *nodes[1].node.node_addr();
|
||||||
@@ -182,26 +180,25 @@ async fn test_api_connect_starts_alternate_path_for_active_peer() {
|
|||||||
|
|
||||||
assert_eq!(
|
assert_eq!(
|
||||||
data["refreshed"], true,
|
data["refreshed"], true,
|
||||||
"a new path for an active peer must start a refresh"
|
"a new address for an active peer is taken as a path"
|
||||||
);
|
);
|
||||||
assert!(
|
assert!(
|
||||||
nodes[0]
|
!nodes[0]
|
||||||
.node
|
.node
|
||||||
.is_connecting_to_peer_on_path(&node1_addr, transport_id, &alternate),
|
.is_connecting_to_peer_on_path(&node1_addr, transport_id, &alternate),
|
||||||
"an outbound leg should exist on the alternate path"
|
"no handshake: a peer with a session is probed, not dialled"
|
||||||
);
|
);
|
||||||
|
assert_eq!(nodes[0].node.connection_count(), 0);
|
||||||
let peer = nodes[0]
|
let peer = nodes[0]
|
||||||
.node
|
.node
|
||||||
.get_peer(&node1_addr)
|
.get_peer(&node1_addr)
|
||||||
.expect("the existing peer must survive the parallel handshake");
|
.expect("the existing peer is untouched");
|
||||||
assert_eq!(
|
assert_eq!(peer.link_id(), link_before, "the live link must not change");
|
||||||
peer.link_id(),
|
assert!(
|
||||||
link_before,
|
peer.path_on(transport_id).is_some(),
|
||||||
"the alternate handshake must not tear the live link down before it authenticates"
|
"the transport still has its one path"
|
||||||
);
|
);
|
||||||
|
|
||||||
// Let the alternate handshake run to completion; the peer must still be
|
|
||||||
// there afterwards.
|
|
||||||
for _ in 0..20 {
|
for _ in 0..20 {
|
||||||
if process_available_packets(&mut nodes).await == 0 {
|
if process_available_packets(&mut nodes).await == 0 {
|
||||||
break;
|
break;
|
||||||
@@ -209,7 +206,7 @@ async fn test_api_connect_starts_alternate_path_for_active_peer() {
|
|||||||
}
|
}
|
||||||
assert!(
|
assert!(
|
||||||
nodes[0].node.get_peer(&node1_addr).is_some(),
|
nodes[0].node.get_peer(&node1_addr).is_some(),
|
||||||
"node 1 should still be a peer after the alternate path resolves"
|
"node 1 is still a peer"
|
||||||
);
|
);
|
||||||
|
|
||||||
cleanup_nodes(&mut nodes).await;
|
cleanup_nodes(&mut nodes).await;
|
||||||
|
|||||||
+509
-83
@@ -307,6 +307,22 @@ async fn dual_homed_pair() -> (Vec<TestNode>, TransportAddr, TransportAddr) {
|
|||||||
(nodes, wifi_0, wifi_1)
|
(nodes, wifi_0, wifi_1)
|
||||||
}
|
}
|
||||||
|
|
||||||
|
/// Hand `nodes[i]` the address `addr` for `peer` on `transport` and run one
|
||||||
|
/// heartbeat tick: the production route by which a path gets probed
|
||||||
|
/// (`add_path_candidate` is what a beacon, a config entry or a completed
|
||||||
|
/// dial leaves behind; the tick is the one issuer of probes). Nothing is
|
||||||
|
/// delivered here — the caller drives `process_available_packets`.
|
||||||
|
async fn probe_candidate(
|
||||||
|
nodes: &mut [TestNode],
|
||||||
|
i: usize,
|
||||||
|
peer: NodeAddr,
|
||||||
|
transport: TransportId,
|
||||||
|
addr: TransportAddr,
|
||||||
|
) {
|
||||||
|
nodes[i].node.add_path_candidate(peer, transport, addr);
|
||||||
|
nodes[i].node.run_path_heartbeats().await;
|
||||||
|
}
|
||||||
|
|
||||||
#[test]
|
#[test]
|
||||||
fn path_message_round_trips_on_the_wire() {
|
fn path_message_round_trips_on_the_wire() {
|
||||||
let probe = PathMessage {
|
let probe = PathMessage {
|
||||||
@@ -345,10 +361,7 @@ async fn a_probe_adds_a_path_at_both_ends_and_the_ack_makes_it_live() {
|
|||||||
let cable = nodes[0].transport_id;
|
let cable = nodes[0].transport_id;
|
||||||
|
|
||||||
// Node 1 probes node 0 over the wifi.
|
// Node 1 probes node 0 over the wifi.
|
||||||
nodes[1]
|
probe_candidate(&mut nodes, 1, addr_0, wifi(), wifi_0.clone()).await;
|
||||||
.node
|
|
||||||
.maybe_probe_path(addr_0, wifi(), wifi_0.clone())
|
|
||||||
.await;
|
|
||||||
{
|
{
|
||||||
let peer = nodes[1].node.get_peer(&addr_0).unwrap();
|
let peer = nodes[1].node.get_peer(&addr_0).unwrap();
|
||||||
let path = peer
|
let path = peer
|
||||||
@@ -358,8 +371,9 @@ async fn a_probe_adds_a_path_at_both_ends_and_the_ack_makes_it_live() {
|
|||||||
assert!(path.tx_live_at_ms().is_none());
|
assert!(path.tx_live_at_ms().is_none());
|
||||||
}
|
}
|
||||||
|
|
||||||
// Probe reaches node 0.
|
// The probe reaches node 0 (the tick heartbeats the active path too,
|
||||||
assert_eq!(process_available_packets(&mut nodes).await, 1);
|
// now that the peer has two).
|
||||||
|
assert!(process_available_packets(&mut nodes).await >= 1);
|
||||||
{
|
{
|
||||||
let peer = nodes[0].node.get_peer(&addr_1).unwrap();
|
let peer = nodes[0].node.get_peer(&addr_1).unwrap();
|
||||||
let path = peer.path_on(wifi()).expect("the receiver adds the path");
|
let path = peer.path_on(wifi()).expect("the receiver adds the path");
|
||||||
@@ -374,8 +388,8 @@ async fn a_probe_adds_a_path_at_both_ends_and_the_ack_makes_it_live() {
|
|||||||
assert_eq!(peer.paths().len(), 2);
|
assert_eq!(peer.paths().len(), 2);
|
||||||
}
|
}
|
||||||
|
|
||||||
// Ack reaches node 1.
|
// The ack reaches node 1.
|
||||||
assert_eq!(process_available_packets(&mut nodes).await, 1);
|
assert!(process_available_packets(&mut nodes).await >= 1);
|
||||||
{
|
{
|
||||||
let peer = nodes[1].node.get_peer(&addr_0).unwrap();
|
let peer = nodes[1].node.get_peer(&addr_0).unwrap();
|
||||||
let path = peer.path_on(wifi()).unwrap();
|
let path = peer.path_on(wifi()).unwrap();
|
||||||
@@ -446,6 +460,121 @@ async fn a_beacon_from_a_live_peer_on_a_new_transport_probes_instead_of_dialling
|
|||||||
);
|
);
|
||||||
}
|
}
|
||||||
|
|
||||||
|
#[tokio::test]
|
||||||
|
async fn a_second_handshake_creates_no_path_state_and_the_address_is_probed_instead() {
|
||||||
|
let (mut nodes, wifi_0, _wifi_1) = dual_homed_pair().await;
|
||||||
|
let addr_0 = *nodes[0].node.node_addr();
|
||||||
|
let addr_1 = *nodes[1].node.node_addr();
|
||||||
|
let cable = nodes[0].transport_id;
|
||||||
|
let session_before = (
|
||||||
|
nodes[0].node.get_peer(&addr_1).unwrap().our_index(),
|
||||||
|
nodes[1].node.get_peer(&addr_0).unwrap().our_index(),
|
||||||
|
);
|
||||||
|
|
||||||
|
// Node 1 dials node 0 over the wifi, as a static config listing both
|
||||||
|
// addresses does at startup, while the cable session is live. The
|
||||||
|
// responder answers it as it would any msg1 from a peer it holds a
|
||||||
|
// session with — a duplicate here, the session being seconds old —
|
||||||
|
// and neither end derives a path from the handshake.
|
||||||
|
let identity_0 = PeerIdentity::from_pubkey_full(nodes[0].node.identity().pubkey_full());
|
||||||
|
nodes[1]
|
||||||
|
.node
|
||||||
|
.initiate_connection(wifi(), wifi_0.clone(), identity_0)
|
||||||
|
.await
|
||||||
|
.expect("dial starts");
|
||||||
|
for _ in 0..8 {
|
||||||
|
if process_available_packets(&mut nodes).await == 0 {
|
||||||
|
break;
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
// Neither end re-peered: one peer each, the session untouched.
|
||||||
|
assert_eq!(nodes[0].node.peer_count(), 1);
|
||||||
|
assert_eq!(nodes[1].node.peer_count(), 1);
|
||||||
|
assert_eq!(
|
||||||
|
(
|
||||||
|
nodes[0].node.get_peer(&addr_1).unwrap().our_index(),
|
||||||
|
nodes[1].node.get_peer(&addr_0).unwrap().our_index(),
|
||||||
|
),
|
||||||
|
session_before,
|
||||||
|
"the handshake did not replace the session"
|
||||||
|
);
|
||||||
|
// The dialler holds the address as an unproven candidate, traffic on
|
||||||
|
// the cable; the responder learned nothing from the handshake.
|
||||||
|
let p1 = nodes[1].node.get_peer(&addr_0).unwrap();
|
||||||
|
assert_eq!(p1.transport_id(), Some(cable), "traffic stays on the cable");
|
||||||
|
let wifi_path = p1
|
||||||
|
.path_on(wifi())
|
||||||
|
.expect("the dialled address is a candidate");
|
||||||
|
assert_eq!(wifi_path.state(), PathState::Probing);
|
||||||
|
assert!(!wifi_path.acked_once(), "a handshake proves no path");
|
||||||
|
assert!(
|
||||||
|
nodes[0]
|
||||||
|
.node
|
||||||
|
.get_peer(&addr_1)
|
||||||
|
.unwrap()
|
||||||
|
.path_on(wifi())
|
||||||
|
.is_none(),
|
||||||
|
"the responder derives no path from a handshake"
|
||||||
|
);
|
||||||
|
|
||||||
|
// The connection the dial opened is left for the probe to ride: on a
|
||||||
|
// connection-oriented transport it is the path's socket, and closing
|
||||||
|
// it would have the first probe dial again (or, at the responder, find
|
||||||
|
// an ephemeral port that cannot be dialled). Loopback records the
|
||||||
|
// close it would have been asked for.
|
||||||
|
for node in &nodes {
|
||||||
|
let closed = match node.node.transports.get(&wifi()).expect("wifi transport") {
|
||||||
|
TransportHandle::Loopback(t) => t.closed(),
|
||||||
|
_ => unreachable!("tests run over loopback"),
|
||||||
|
};
|
||||||
|
assert!(
|
||||||
|
closed.is_empty(),
|
||||||
|
"the dial's connection is kept for the candidate: {closed:?}"
|
||||||
|
);
|
||||||
|
}
|
||||||
|
|
||||||
|
// The heartbeat tick proves it, at both ends, under the shared session.
|
||||||
|
nodes[1].node.run_path_heartbeats().await;
|
||||||
|
for _ in 0..8 {
|
||||||
|
if process_available_packets(&mut nodes).await == 0 {
|
||||||
|
break;
|
||||||
|
}
|
||||||
|
}
|
||||||
|
let p1 = nodes[1].node.get_peer(&addr_0).unwrap();
|
||||||
|
assert_eq!(p1.path_on(wifi()).unwrap().state(), PathState::Live);
|
||||||
|
assert_eq!(p1.transport_id(), Some(cable), "traffic still on the cable");
|
||||||
|
let p0 = nodes[0].node.get_peer(&addr_1).unwrap();
|
||||||
|
assert!(
|
||||||
|
p0.path_on(wifi()).is_some(),
|
||||||
|
"the probe taught the responder the path"
|
||||||
|
);
|
||||||
|
assert_eq!(p0.transport_id(), Some(cable));
|
||||||
|
|
||||||
|
// An operator disconnect closes every path's connection, the standby's
|
||||||
|
// included: on a connection-oriented transport each holds a pool entry.
|
||||||
|
let npub_0 = nodes[0].node.identity().npub();
|
||||||
|
nodes[1]
|
||||||
|
.node
|
||||||
|
.api_disconnect(&npub_0)
|
||||||
|
.await
|
||||||
|
.expect("disconnect");
|
||||||
|
let closed_wifi = match nodes[1].node.transports.get(&wifi()).unwrap() {
|
||||||
|
TransportHandle::Loopback(t) => t.closed(),
|
||||||
|
_ => unreachable!(),
|
||||||
|
};
|
||||||
|
assert_eq!(
|
||||||
|
closed_wifi,
|
||||||
|
vec![wifi_0.clone()],
|
||||||
|
"the standby's connection is closed too"
|
||||||
|
);
|
||||||
|
let closed_cable = match nodes[1].node.transports.get(&cable).unwrap() {
|
||||||
|
TransportHandle::Loopback(t) => t.closed(),
|
||||||
|
_ => unreachable!(),
|
||||||
|
};
|
||||||
|
assert_eq!(closed_cable.len(), 1, "and the active path's");
|
||||||
|
}
|
||||||
|
|
||||||
#[tokio::test]
|
#[tokio::test]
|
||||||
async fn a_beaconed_path_is_probed_by_the_next_heartbeat_tick_and_once_only() {
|
async fn a_beaconed_path_is_probed_by_the_next_heartbeat_tick_and_once_only() {
|
||||||
let (mut nodes, wifi_0, _wifi_1) = dual_homed_pair().await;
|
let (mut nodes, wifi_0, _wifi_1) = dual_homed_pair().await;
|
||||||
@@ -509,24 +638,6 @@ async fn an_ack_for_no_outstanding_probe_changes_nothing() {
|
|||||||
);
|
);
|
||||||
}
|
}
|
||||||
|
|
||||||
#[tokio::test]
|
|
||||||
async fn the_active_path_is_not_probed() {
|
|
||||||
let (mut nodes, _wifi_0, _wifi_1) = dual_homed_pair().await;
|
|
||||||
let addr_0 = *nodes[0].node.node_addr();
|
|
||||||
let cable = nodes[0].transport_id;
|
|
||||||
let cable_addr_0 = nodes[0].addr.clone();
|
|
||||||
|
|
||||||
nodes[1]
|
|
||||||
.node
|
|
||||||
.maybe_probe_path(addr_0, cable, cable_addr_0)
|
|
||||||
.await;
|
|
||||||
assert_eq!(
|
|
||||||
nodes[0].packet_rx.len(),
|
|
||||||
0,
|
|
||||||
"the handshake proved the active path"
|
|
||||||
);
|
|
||||||
}
|
|
||||||
|
|
||||||
// ============================================================================
|
// ============================================================================
|
||||||
// Presence loss withdraws a path (design §5–6)
|
// Presence loss withdraws a path (design §5–6)
|
||||||
// ============================================================================
|
// ============================================================================
|
||||||
@@ -537,14 +648,8 @@ async fn pair_with_wifi_live() -> (Vec<TestNode>, TransportAddr, TransportAddr)
|
|||||||
let (mut nodes, wifi_0, wifi_1) = dual_homed_pair().await;
|
let (mut nodes, wifi_0, wifi_1) = dual_homed_pair().await;
|
||||||
let addr_0 = *nodes[0].node.node_addr();
|
let addr_0 = *nodes[0].node.node_addr();
|
||||||
let addr_1 = *nodes[1].node.node_addr();
|
let addr_1 = *nodes[1].node.node_addr();
|
||||||
nodes[1]
|
probe_candidate(&mut nodes, 1, addr_0, wifi(), wifi_0.clone()).await;
|
||||||
.node
|
probe_candidate(&mut nodes, 0, addr_1, wifi(), wifi_1.clone()).await;
|
||||||
.maybe_probe_path(addr_0, wifi(), wifi_0.clone())
|
|
||||||
.await;
|
|
||||||
nodes[0]
|
|
||||||
.node
|
|
||||||
.maybe_probe_path(addr_1, wifi(), wifi_1.clone())
|
|
||||||
.await;
|
|
||||||
for _ in 0..4 {
|
for _ in 0..4 {
|
||||||
if process_available_packets(&mut nodes).await == 0 {
|
if process_available_packets(&mut nodes).await == 0 {
|
||||||
break;
|
break;
|
||||||
@@ -595,7 +700,9 @@ async fn losing_the_active_transport_moves_traffic_to_the_live_standby() {
|
|||||||
assert_eq!(seeded_by, Some(wifi()));
|
assert_eq!(seeded_by, Some(wifi()));
|
||||||
|
|
||||||
// The same session carries on: a frame sent now goes out on the wifi
|
// The same session carries on: a frame sent now goes out on the wifi
|
||||||
// and decrypts at the far end, which never saw a switch.
|
// and decrypts at the far end. (The withdrawal also told the far end,
|
||||||
|
// with a PathClose on the wifi, that the cable is gone: it moves its
|
||||||
|
// own traffic to the wifi on hearing it, and re-peers nowhere.)
|
||||||
let before = nodes[0].packet_rx.len();
|
let before = nodes[0].packet_rx.len();
|
||||||
nodes[1]
|
nodes[1]
|
||||||
.node
|
.node
|
||||||
@@ -603,15 +710,21 @@ async fn losing_the_active_transport_moves_traffic_to_the_live_standby() {
|
|||||||
.await
|
.await
|
||||||
.expect("send over the standby");
|
.expect("send over the standby");
|
||||||
assert_eq!(nodes[0].packet_rx.len(), before + 1);
|
assert_eq!(nodes[0].packet_rx.len(), before + 1);
|
||||||
let packet = nodes[0].packet_rx.try_recv().unwrap();
|
while let Ok(packet) = nodes[0].packet_rx.try_recv() {
|
||||||
assert_eq!(packet.transport_id, wifi());
|
assert_eq!(packet.transport_id, wifi());
|
||||||
nodes[0].node.handle_encrypted_frame(packet).await;
|
nodes[0].node.handle_encrypted_frame(packet).await;
|
||||||
|
}
|
||||||
let far = nodes[0]
|
let far = nodes[0]
|
||||||
.node
|
.node
|
||||||
.get_peer(nodes[1].node.node_addr())
|
.get_peer(nodes[1].node.node_addr())
|
||||||
.expect("no re-peering");
|
.expect("no re-peering");
|
||||||
assert_eq!(far.consecutive_decrypt_failures(), 0);
|
assert_eq!(far.consecutive_decrypt_failures(), 0);
|
||||||
assert_eq!(far.transport_id(), Some(cable), "the far end did not move");
|
assert_eq!(nodes[0].node.peer_count(), 1);
|
||||||
|
assert_eq!(
|
||||||
|
far.transport_id(),
|
||||||
|
Some(wifi()),
|
||||||
|
"told the cable is gone, the far end moved too"
|
||||||
|
);
|
||||||
}
|
}
|
||||||
|
|
||||||
#[tokio::test]
|
#[tokio::test]
|
||||||
@@ -621,10 +734,7 @@ async fn losing_the_active_transport_with_only_a_probing_standby_reaps() {
|
|||||||
let cable = nodes[1].transport_id;
|
let cable = nodes[1].transport_id;
|
||||||
|
|
||||||
// Probe sent, ack never processed: the wifi path is unproven.
|
// Probe sent, ack never processed: the wifi path is unproven.
|
||||||
nodes[1]
|
probe_candidate(&mut nodes, 1, addr_0, wifi(), wifi_0.clone()).await;
|
||||||
.node
|
|
||||||
.maybe_probe_path(addr_0, wifi(), wifi_0.clone())
|
|
||||||
.await;
|
|
||||||
assert_eq!(
|
assert_eq!(
|
||||||
nodes[1]
|
nodes[1]
|
||||||
.node
|
.node
|
||||||
@@ -662,21 +772,39 @@ async fn a_dead_path_is_reprobed_when_its_transport_returns_and_forgotten_after_
|
|||||||
let (mut nodes, wifi_0, _wifi_1) = pair_with_wifi_live().await;
|
let (mut nodes, wifi_0, _wifi_1) = pair_with_wifi_live().await;
|
||||||
let addr_0 = *nodes[0].node.node_addr();
|
let addr_0 = *nodes[0].node.node_addr();
|
||||||
|
|
||||||
nodes[1].node.withdraw_transport(wifi()).await;
|
let samples_before = nodes[1]
|
||||||
|
|
||||||
// Presence returns: the next discovery tick probes it again, and the
|
|
||||||
// ack brings it back Live with its history. (The withdrawal also sent
|
|
||||||
// node 0 a PathClose on the cable, which it processes here too.)
|
|
||||||
nodes[1].node.reset_probe_backoff_on_transport(wifi());
|
|
||||||
nodes[1]
|
|
||||||
.node
|
.node
|
||||||
.maybe_probe_path(addr_0, wifi(), wifi_0.clone())
|
.get_peer(&addr_0)
|
||||||
.await;
|
.unwrap()
|
||||||
|
.path_on(wifi())
|
||||||
|
.unwrap()
|
||||||
|
.rtt_samples();
|
||||||
|
assert!(samples_before > 0);
|
||||||
|
nodes[1].node.withdraw_transport(wifi()).await;
|
||||||
|
// (The withdrawal also sent node 0 a PathClose on the cable.)
|
||||||
for _ in 0..4 {
|
for _ in 0..4 {
|
||||||
if process_available_packets(&mut nodes).await == 0 {
|
if process_available_packets(&mut nodes).await == 0 {
|
||||||
break;
|
break;
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
let dead = nodes[1]
|
||||||
|
.node
|
||||||
|
.get_peer(&addr_0)
|
||||||
|
.unwrap()
|
||||||
|
.path_on(wifi())
|
||||||
|
.unwrap();
|
||||||
|
assert_eq!(dead.state(), PathState::Dead);
|
||||||
|
assert_eq!(dead.addr(), &wifi_0);
|
||||||
|
// Dead: the heartbeat tick leaves it alone.
|
||||||
|
nodes[1].node.run_path_heartbeats().await;
|
||||||
|
assert!(
|
||||||
|
nodes[0].packet_rx.try_recv().is_err(),
|
||||||
|
"a Dead path is not probed"
|
||||||
|
);
|
||||||
|
|
||||||
|
// Presence returns: the path is Probing again, the next heartbeat tick
|
||||||
|
// probes it, and the ack brings it back Live with its history.
|
||||||
|
nodes[1].node.reset_probe_backoff_on_transport(wifi());
|
||||||
assert_eq!(
|
assert_eq!(
|
||||||
nodes[1]
|
nodes[1]
|
||||||
.node
|
.node
|
||||||
@@ -685,7 +813,24 @@ async fn a_dead_path_is_reprobed_when_its_transport_returns_and_forgotten_after_
|
|||||||
.path_on(wifi())
|
.path_on(wifi())
|
||||||
.unwrap()
|
.unwrap()
|
||||||
.state(),
|
.state(),
|
||||||
PathState::Live
|
PathState::Probing
|
||||||
|
);
|
||||||
|
nodes[1].node.run_path_heartbeats().await;
|
||||||
|
for _ in 0..4 {
|
||||||
|
if process_available_packets(&mut nodes).await == 0 {
|
||||||
|
break;
|
||||||
|
}
|
||||||
|
}
|
||||||
|
let revived = nodes[1]
|
||||||
|
.node
|
||||||
|
.get_peer(&addr_0)
|
||||||
|
.unwrap()
|
||||||
|
.path_on(wifi())
|
||||||
|
.unwrap();
|
||||||
|
assert_eq!(revived.state(), PathState::Live);
|
||||||
|
assert!(
|
||||||
|
revived.rtt_samples() > samples_before,
|
||||||
|
"re-proved on top of its history, not from nothing"
|
||||||
);
|
);
|
||||||
|
|
||||||
// Dead again, and this time the grace expires.
|
// Dead again, and this time the grace expires.
|
||||||
@@ -751,6 +896,49 @@ async fn garbage_on_a_standby_path_counts_against_the_peer() {
|
|||||||
);
|
);
|
||||||
}
|
}
|
||||||
|
|
||||||
|
#[tokio::test]
|
||||||
|
async fn garbage_on_a_path_the_peer_never_acknowledged_is_not_counted() {
|
||||||
|
// A candidate is an address we were told about — a beacon, a config
|
||||||
|
// entry, a handshake source, possibly a replayed one. Until the peer
|
||||||
|
// answers a probe there, garbage on its transport says nothing about
|
||||||
|
// the peer, and cannot tear the peering down.
|
||||||
|
let (mut nodes, _wifi_0, wifi_1) = dual_homed_pair().await;
|
||||||
|
let addr_1 = *nodes[1].node.node_addr();
|
||||||
|
nodes[0]
|
||||||
|
.node
|
||||||
|
.add_path_candidate(addr_1, wifi(), wifi_1.clone());
|
||||||
|
assert!(
|
||||||
|
!nodes[0]
|
||||||
|
.node
|
||||||
|
.get_peer(&addr_1)
|
||||||
|
.unwrap()
|
||||||
|
.path_on(wifi())
|
||||||
|
.unwrap()
|
||||||
|
.acked_once()
|
||||||
|
);
|
||||||
|
let our_index = nodes[0]
|
||||||
|
.node
|
||||||
|
.get_peer(&addr_1)
|
||||||
|
.unwrap()
|
||||||
|
.our_index()
|
||||||
|
.unwrap();
|
||||||
|
for counter in 0..(THRESHOLD * 2) as u64 {
|
||||||
|
nodes[0]
|
||||||
|
.node
|
||||||
|
.handle_encrypted_frame(ReceivedPacket::new(
|
||||||
|
wifi(),
|
||||||
|
wifi_1.clone(),
|
||||||
|
garbage_frame(our_index, counter),
|
||||||
|
))
|
||||||
|
.await;
|
||||||
|
}
|
||||||
|
let peer = nodes[0]
|
||||||
|
.node
|
||||||
|
.get_peer(&addr_1)
|
||||||
|
.expect("an unproven path is not a transport the peer is on");
|
||||||
|
assert_eq!(peer.consecutive_decrypt_failures(), 0);
|
||||||
|
}
|
||||||
|
|
||||||
// ============================================================================
|
// ============================================================================
|
||||||
// Selection (design §8)
|
// Selection (design §8)
|
||||||
// ============================================================================
|
// ============================================================================
|
||||||
@@ -1039,24 +1227,124 @@ fn a_timed_out_echo_on_an_acknowledged_path_makes_it_suspect_and_selection_leave
|
|||||||
}
|
}
|
||||||
|
|
||||||
#[test]
|
#[test]
|
||||||
fn a_path_the_peer_never_acknowledged_backs_off_instead_of_going_suspect() {
|
fn a_peer_with_one_live_path_is_not_heartbeated() {
|
||||||
let mut peer = ActivePeer::new(make_peer_identity(), LinkId::new(1), 0);
|
let mut peer = ActivePeer::new(make_peer_identity(), LinkId::new(1), 0);
|
||||||
peer.rebind_transport(tid(CABLE), TransportAddr::from_string("10.0.0.1:1"));
|
peer.rebind_transport(tid(CABLE), TransportAddr::from_string("10.0.0.1:1"));
|
||||||
// Promotion-style: Live and tx_live from the handshake, never acked.
|
// Promotion-style: one Live path from the handshake. Nothing to decide,
|
||||||
|
// nothing sent — the link heartbeat keeps liveness.
|
||||||
|
let t0 = 1_000_000;
|
||||||
|
for k in 0..20 {
|
||||||
|
assert!(
|
||||||
|
peer.plan_heartbeats(t0 + k * FAST, &TIMING)
|
||||||
|
.sends
|
||||||
|
.is_empty(),
|
||||||
|
"a single-path peer gets no path probes"
|
||||||
|
);
|
||||||
|
}
|
||||||
|
// A candidate makes it two: probes start, on the candidate and on the
|
||||||
|
// active path alike.
|
||||||
|
peer.add_path(tid(WIFI), TransportAddr::from_string("10.0.0.7:1"));
|
||||||
|
let plan = peer.plan_heartbeats(t0 + 21 * FAST, &TIMING);
|
||||||
|
assert_eq!(plan.sends.len(), 2);
|
||||||
|
assert!(
|
||||||
|
plan.sends
|
||||||
|
.iter()
|
||||||
|
.any(|s| s.transport_id == tid(WIFI) && s.full_size),
|
||||||
|
"the candidate's first probe is the full-size discovery probe"
|
||||||
|
);
|
||||||
|
assert!(
|
||||||
|
plan.sends
|
||||||
|
.iter()
|
||||||
|
.any(|s| s.transport_id == tid(CABLE) && !s.full_size),
|
||||||
|
"the handshake proved the active path: its first probe is small"
|
||||||
|
);
|
||||||
|
}
|
||||||
|
|
||||||
|
#[test]
|
||||||
|
fn an_active_path_the_peer_never_acknowledged_backs_off_instead_of_going_suspect() {
|
||||||
|
let mut peer = ActivePeer::new(make_peer_identity(), LinkId::new(1), 0);
|
||||||
|
peer.rebind_transport(tid(CABLE), TransportAddr::from_string("10.0.0.1:1"));
|
||||||
|
// A candidate on the wifi makes the peer two-path, so the cable — Live
|
||||||
|
// from the handshake, never acked: an old node — is probed too.
|
||||||
|
peer.add_path(tid(WIFI), TransportAddr::from_string("10.0.0.7:1"));
|
||||||
let t0 = 1_000_000;
|
let t0 = 1_000_000;
|
||||||
let plan = peer.plan_heartbeats(t0, &TIMING);
|
let plan = peer.plan_heartbeats(t0, &TIMING);
|
||||||
assert_eq!(plan.sends.len(), 1);
|
assert!(plan.sends.iter().any(|s| s.transport_id == tid(CABLE)));
|
||||||
let plan = peer.plan_heartbeats(t0 + TIMEOUT, &TIMING);
|
let plan = peer.plan_heartbeats(t0 + TIMEOUT, &TIMING);
|
||||||
assert!(plan.suspects.is_empty(), "an old node is not a dead path");
|
assert!(plan.suspects.is_empty(), "an old node is not a dead path");
|
||||||
let cable = peer.path_on(tid(CABLE)).unwrap();
|
let cable = peer.path_on(tid(CABLE)).unwrap();
|
||||||
assert_eq!(cable.state(), PathState::Live);
|
assert_eq!(cable.state(), PathState::Live);
|
||||||
assert_eq!(plan.sends.len(), 1, "tried again, with the backoff doubled");
|
|
||||||
assert!(
|
assert!(
|
||||||
peer.plan_heartbeats(t0 + TIMEOUT + 2 * FAST - 1, &TIMING)
|
plan.sends.iter().any(|s| s.transport_id == tid(CABLE)),
|
||||||
|
"tried again, with the backoff doubled"
|
||||||
|
);
|
||||||
|
assert!(
|
||||||
|
!peer
|
||||||
|
.plan_heartbeats(t0 + TIMEOUT + 2 * FAST - 1, &TIMING)
|
||||||
.sends
|
.sends
|
||||||
.is_empty(),
|
.iter()
|
||||||
|
.any(|s| s.transport_id == tid(CABLE)),
|
||||||
"the unanswered probe pushed the next one out"
|
"the unanswered probe pushed the next one out"
|
||||||
);
|
);
|
||||||
|
// However long it goes unanswered, the active path is never given up:
|
||||||
|
// the handshake proved it, and an old node answers no probe.
|
||||||
|
let mut t = t0 + TIMEOUT + 2 * FAST;
|
||||||
|
for _ in 0..200 {
|
||||||
|
peer.plan_heartbeats(t, &TIMING);
|
||||||
|
t += TIMEOUT;
|
||||||
|
}
|
||||||
|
assert_eq!(peer.path_on(tid(CABLE)).unwrap().state(), PathState::Live);
|
||||||
|
assert_eq!(peer.transport_id(), Some(tid(CABLE)));
|
||||||
|
}
|
||||||
|
|
||||||
|
#[test]
|
||||||
|
fn a_standby_the_peer_never_acknowledges_is_given_up_after_the_discovery_budget() {
|
||||||
|
let mut peer = dual_path_peer(1, 5);
|
||||||
|
// A third address the peer never answers on: a NIC it no longer sends
|
||||||
|
// from, a replayed source, an old node's transport.
|
||||||
|
peer.add_path(tid(3), TransportAddr::from_string("10.0.0.9:1"));
|
||||||
|
let mut t = 1_000_000;
|
||||||
|
let mut probes = 0;
|
||||||
|
for _ in 0..400 {
|
||||||
|
let plan = peer.plan_heartbeats(t, &TIMING);
|
||||||
|
assert!(
|
||||||
|
plan.suspects.is_empty(),
|
||||||
|
"never acknowledged: never Suspect"
|
||||||
|
);
|
||||||
|
probes += plan
|
||||||
|
.sends
|
||||||
|
.iter()
|
||||||
|
.filter(|s| s.transport_id == tid(3))
|
||||||
|
.count();
|
||||||
|
for s in plan.sends {
|
||||||
|
if s.transport_id != tid(3) {
|
||||||
|
peer.note_path_ack(s.transport_id, s.probe_id, false, 1, t + 1, u64::MAX);
|
||||||
|
}
|
||||||
|
}
|
||||||
|
if peer.path_on(tid(3)).unwrap().state() == PathState::Dead {
|
||||||
|
break;
|
||||||
|
}
|
||||||
|
t += TIMEOUT;
|
||||||
|
}
|
||||||
|
assert_eq!(
|
||||||
|
peer.path_on(tid(3)).unwrap().state(),
|
||||||
|
PathState::Dead,
|
||||||
|
"given up after the budget"
|
||||||
|
);
|
||||||
|
assert_eq!(probes, crate::peer::MAX_DISCOVERY_PROBES as usize);
|
||||||
|
// Dead: not probed again, and forgotten after the grace.
|
||||||
|
assert!(
|
||||||
|
!peer
|
||||||
|
.plan_heartbeats(t + TIMEOUT, &TIMING)
|
||||||
|
.sends
|
||||||
|
.iter()
|
||||||
|
.any(|s| s.transport_id == tid(3))
|
||||||
|
);
|
||||||
|
peer.prune_dead_paths(t + 10 * 60_000, 5 * 60_000);
|
||||||
|
assert!(peer.path_on(tid(3)).is_none());
|
||||||
|
// The proven paths are untouched.
|
||||||
|
assert_eq!(peer.path_on(tid(CABLE)).unwrap().state(), PathState::Live);
|
||||||
|
assert_eq!(peer.path_on(tid(WIFI)).unwrap().state(), PathState::Live);
|
||||||
}
|
}
|
||||||
|
|
||||||
#[test]
|
#[test]
|
||||||
@@ -1097,22 +1385,31 @@ fn withdrawing_our_only_path_leaves_it_suspect_and_still_probed() {
|
|||||||
fn presence_return_resets_the_discovery_backoff() {
|
fn presence_return_resets_the_discovery_backoff() {
|
||||||
let mut peer = ActivePeer::new(make_peer_identity(), LinkId::new(1), 0);
|
let mut peer = ActivePeer::new(make_peer_identity(), LinkId::new(1), 0);
|
||||||
peer.rebind_transport(tid(CABLE), TransportAddr::from_string("10.0.0.1:1"));
|
peer.rebind_transport(tid(CABLE), TransportAddr::from_string("10.0.0.1:1"));
|
||||||
|
// Two-path, so the never-acked cable is probed at all.
|
||||||
|
peer.add_path(tid(WIFI), TransportAddr::from_string("10.0.0.7:1"));
|
||||||
let t0 = 1_000_000;
|
let t0 = 1_000_000;
|
||||||
|
let on_cable = |plan: &crate::peer::HeartbeatPlan| {
|
||||||
|
plan.sends
|
||||||
|
.iter()
|
||||||
|
.filter(|s| s.transport_id == tid(CABLE))
|
||||||
|
.count()
|
||||||
|
};
|
||||||
// Never answered: each timeout doubles the wait.
|
// Never answered: each timeout doubles the wait.
|
||||||
assert_eq!(peer.plan_heartbeats(t0, &TIMING).sends.len(), 1);
|
assert_eq!(on_cable(&peer.plan_heartbeats(t0, &TIMING)), 1);
|
||||||
assert_eq!(peer.plan_heartbeats(t0 + TIMEOUT, &TIMING).sends.len(), 1);
|
assert_eq!(on_cable(&peer.plan_heartbeats(t0 + TIMEOUT, &TIMING)), 1);
|
||||||
assert_eq!(
|
assert_eq!(
|
||||||
peer.plan_heartbeats(t0 + 2 * TIMEOUT, &TIMING).sends.len(),
|
on_cable(&peer.plan_heartbeats(t0 + 2 * TIMEOUT, &TIMING)),
|
||||||
1
|
1
|
||||||
);
|
);
|
||||||
let t = t0 + 3 * TIMEOUT;
|
let t = t0 + 3 * TIMEOUT;
|
||||||
assert!(
|
assert_eq!(
|
||||||
peer.plan_heartbeats(t, &TIMING).sends.is_empty(),
|
on_cable(&peer.plan_heartbeats(t, &TIMING)),
|
||||||
|
0,
|
||||||
"the third timeout pushed the next probe past now"
|
"the third timeout pushed the next probe past now"
|
||||||
);
|
);
|
||||||
// The transport's presence cycles: probed at once.
|
// The transport's presence cycles: probed at once.
|
||||||
peer.reset_probe_backoff_on(tid(CABLE));
|
peer.reset_probe_backoff_on(tid(CABLE));
|
||||||
assert_eq!(peer.plan_heartbeats(t, &TIMING).sends.len(), 1);
|
assert_eq!(on_cable(&peer.plan_heartbeats(t, &TIMING)), 1);
|
||||||
}
|
}
|
||||||
|
|
||||||
#[test]
|
#[test]
|
||||||
@@ -1127,7 +1424,7 @@ fn the_discovery_backoff_is_capped_and_never_goes_suspect() {
|
|||||||
discovery_cap_ms: 2_000,
|
discovery_cap_ms: 2_000,
|
||||||
..TIMING
|
..TIMING
|
||||||
};
|
};
|
||||||
// Drive the wifi standby through many unanswered probes.
|
// Drive the wifi standby through its unanswered probes.
|
||||||
let mut t = 1_000_000;
|
let mut t = 1_000_000;
|
||||||
let mut sent_at = Vec::new();
|
let mut sent_at = Vec::new();
|
||||||
for _ in 0..40 {
|
for _ in 0..40 {
|
||||||
@@ -1136,9 +1433,19 @@ fn the_discovery_backoff_is_capped_and_never_goes_suspect() {
|
|||||||
plan.suspects.is_empty(),
|
plan.suspects.is_empty(),
|
||||||
"never acknowledged: not a dead path"
|
"never acknowledged: not a dead path"
|
||||||
);
|
);
|
||||||
|
assert!(
|
||||||
|
plan.sends
|
||||||
|
.iter()
|
||||||
|
.filter(|s| s.transport_id == tid(WIFI))
|
||||||
|
.all(|s| s.full_size),
|
||||||
|
"every probe on an unproven path is full-size"
|
||||||
|
);
|
||||||
if plan.sends.iter().any(|s| s.transport_id == tid(WIFI)) {
|
if plan.sends.iter().any(|s| s.transport_id == tid(WIFI)) {
|
||||||
sent_at.push(t);
|
sent_at.push(t);
|
||||||
}
|
}
|
||||||
|
if peer.path_on(tid(WIFI)).unwrap().state() == PathState::Dead {
|
||||||
|
break;
|
||||||
|
}
|
||||||
t += TIMEOUT;
|
t += TIMEOUT;
|
||||||
}
|
}
|
||||||
let gaps: Vec<u64> = sent_at.windows(2).map(|w| w[1] - w[0]).collect();
|
let gaps: Vec<u64> = sent_at.windows(2).map(|w| w[1] - w[0]).collect();
|
||||||
@@ -1147,14 +1454,12 @@ fn the_discovery_backoff_is_capped_and_never_goes_suspect() {
|
|||||||
gaps.iter().all(|g| *g <= 2_000 + TIMEOUT),
|
gaps.iter().all(|g| *g <= 2_000 + TIMEOUT),
|
||||||
"the backoff is capped at discovery_cap_ms: {gaps:?}"
|
"the backoff is capped at discovery_cap_ms: {gaps:?}"
|
||||||
);
|
);
|
||||||
assert!(
|
assert_eq!(
|
||||||
peer.plan_heartbeats(t, &timing)
|
sent_at.len(),
|
||||||
.sends
|
crate::peer::MAX_DISCOVERY_PROBES as usize,
|
||||||
.iter()
|
"and the budget bounds it: the path is given up, not probed forever"
|
||||||
.filter(|s| s.transport_id == tid(WIFI))
|
|
||||||
.all(|s| s.full_size),
|
|
||||||
"every probe on an unproven path is full-size"
|
|
||||||
);
|
);
|
||||||
|
assert_eq!(peer.path_on(tid(WIFI)).unwrap().state(), PathState::Dead);
|
||||||
}
|
}
|
||||||
|
|
||||||
#[test]
|
#[test]
|
||||||
@@ -1298,7 +1603,7 @@ fn unreachable_send_errors_are_classified() {
|
|||||||
|
|
||||||
#[tokio::test]
|
#[tokio::test]
|
||||||
async fn the_fast_tick_heartbeats_the_active_path_and_the_ack_measures_it() {
|
async fn the_fast_tick_heartbeats_the_active_path_and_the_ack_measures_it() {
|
||||||
let (mut nodes, _wifi_0, _wifi_1) = pair_with_wifi_live().await;
|
let (mut nodes, wifi_0, _wifi_1) = dual_homed_pair().await;
|
||||||
let addr_0 = *nodes[0].node.node_addr();
|
let addr_0 = *nodes[0].node.node_addr();
|
||||||
let cable = nodes[1].transport_id;
|
let cable = nodes[1].transport_id;
|
||||||
assert!(
|
assert!(
|
||||||
@@ -1312,6 +1617,11 @@ async fn the_fast_tick_heartbeats_the_active_path_and_the_ack_measures_it() {
|
|||||||
"the handshake proved the cable; no probe has yet"
|
"the handshake proved the cable; no probe has yet"
|
||||||
);
|
);
|
||||||
|
|
||||||
|
// A second path makes the peer worth heartbeating; the tick then
|
||||||
|
// probes the active cable as well as the wifi candidate.
|
||||||
|
nodes[1]
|
||||||
|
.node
|
||||||
|
.add_path_candidate(addr_0, wifi(), wifi_0.clone());
|
||||||
nodes[1].node.run_path_heartbeats().await;
|
nodes[1].node.run_path_heartbeats().await;
|
||||||
let queued = nodes[0].packet_rx.len();
|
let queued = nodes[0].packet_rx.len();
|
||||||
assert!(queued >= 1, "a heartbeat probe went out on the cable");
|
assert!(queued >= 1, "a heartbeat probe went out on the cable");
|
||||||
@@ -1556,11 +1866,16 @@ fn a_probe_reaching_a_dead_path_revives_it() {
|
|||||||
async fn the_first_probe_on_a_path_is_full_size_and_so_is_its_ack() {
|
async fn the_first_probe_on_a_path_is_full_size_and_so_is_its_ack() {
|
||||||
let (mut nodes, wifi_0, _wifi_1) = dual_homed_pair().await;
|
let (mut nodes, wifi_0, _wifi_1) = dual_homed_pair().await;
|
||||||
let addr_0 = *nodes[0].node.node_addr();
|
let addr_0 = *nodes[0].node.node_addr();
|
||||||
nodes[1]
|
probe_candidate(&mut nodes, 1, addr_0, wifi(), wifi_0.clone()).await;
|
||||||
.node
|
// The tick also heartbeated the active cable, small; the wifi probe
|
||||||
.maybe_probe_path(addr_0, wifi(), wifi_0.clone())
|
// is the discovery one.
|
||||||
.await;
|
let mut probe = None;
|
||||||
let probe = nodes[0].packet_rx.try_recv().expect("probe queued");
|
while let Ok(packet) = nodes[0].packet_rx.try_recv() {
|
||||||
|
if packet.transport_id == wifi() {
|
||||||
|
probe = Some(packet);
|
||||||
|
}
|
||||||
|
}
|
||||||
|
let probe = probe.expect("probe queued");
|
||||||
let mtu = usize::from(
|
let mtu = usize::from(
|
||||||
nodes[1]
|
nodes[1]
|
||||||
.node
|
.node
|
||||||
@@ -1592,9 +1907,14 @@ fn a_proven_path_is_probed_full_size_once_a_minute() {
|
|||||||
let t0 = 1_000_000;
|
let t0 = 1_000_000;
|
||||||
let plan = peer.plan_heartbeats(t0, &TIMING);
|
let plan = peer.plan_heartbeats(t0, &TIMING);
|
||||||
// dual_path_peer acked via take_probe, which sets no full-size stamp,
|
// dual_path_peer acked via take_probe, which sets no full-size stamp,
|
||||||
// so the first heartbeat is full-size; after that, small until a
|
// so the standby's first heartbeat is full-size — the active path's is
|
||||||
// minute has passed.
|
// not, the handshake having proved it; after that, small until a
|
||||||
assert!(plan.sends.iter().all(|s| s.full_size));
|
// minute has passed, then full-size on both.
|
||||||
|
assert!(
|
||||||
|
plan.sends
|
||||||
|
.iter()
|
||||||
|
.all(|s| s.full_size == (s.transport_id == tid(WIFI)))
|
||||||
|
);
|
||||||
for s in plan.sends {
|
for s in plan.sends {
|
||||||
peer.note_path_ack(s.transport_id, s.probe_id, false, 1, t0 + 1, u64::MAX);
|
peer.note_path_ack(s.transport_id, s.probe_id, false, 1, t0 + 1, u64::MAX);
|
||||||
}
|
}
|
||||||
@@ -1611,7 +1931,8 @@ fn a_proven_path_is_probed_full_size_once_a_minute() {
|
|||||||
);
|
);
|
||||||
}
|
}
|
||||||
let plan = peer.plan_heartbeats(t0 + 61_000, &TIMING);
|
let plan = peer.plan_heartbeats(t0 + 61_000, &TIMING);
|
||||||
assert!(plan.sends.iter().any(|s| s.full_size));
|
assert_eq!(plan.sends.len(), 2);
|
||||||
|
assert!(plan.sends.iter().all(|s| s.full_size));
|
||||||
}
|
}
|
||||||
|
|
||||||
#[tokio::test]
|
#[tokio::test]
|
||||||
@@ -1773,3 +2094,108 @@ fn an_outage_does_not_keep_charging_the_path_s_etx() {
|
|||||||
"an outage is one event, not a lossy medium"
|
"an outage is one event, not a lossy medium"
|
||||||
);
|
);
|
||||||
}
|
}
|
||||||
|
|
||||||
|
// ---------------------------------------------------------------------------
|
||||||
|
// A handshake proves a path
|
||||||
|
// ---------------------------------------------------------------------------
|
||||||
|
|
||||||
|
#[test]
|
||||||
|
fn a_known_transport_at_a_new_address_is_re_pointed_there() {
|
||||||
|
let mut peer = ActivePeer::new(make_peer_identity(), LinkId::new(1), 0);
|
||||||
|
peer.rebind_transport(tid(CABLE), TransportAddr::from_string("10.0.0.1:1"));
|
||||||
|
peer.add_path(tid(WIFI), TransportAddr::from_string("10.0.0.7:1"));
|
||||||
|
|
||||||
|
// `add_path` is "add or return": the address it carries is ignored for
|
||||||
|
// a path that exists. A move is a separate, explicit step.
|
||||||
|
peer.add_path(tid(WIFI), TransportAddr::from_string("10.0.0.8:1"));
|
||||||
|
assert_eq!(
|
||||||
|
peer.path_on(tid(WIFI)).unwrap().addr(),
|
||||||
|
&TransportAddr::from_string("10.0.0.7:1")
|
||||||
|
);
|
||||||
|
assert!(peer.refresh_path_addr(tid(WIFI), TransportAddr::from_string("10.0.0.8:1")));
|
||||||
|
assert_eq!(
|
||||||
|
peer.path_on(tid(WIFI)).unwrap().addr(),
|
||||||
|
&TransportAddr::from_string("10.0.0.8:1")
|
||||||
|
);
|
||||||
|
assert!(
|
||||||
|
!peer.refresh_path_addr(tid(WIFI), TransportAddr::from_string("10.0.0.8:1")),
|
||||||
|
"the same address is not a move"
|
||||||
|
);
|
||||||
|
assert!(
|
||||||
|
!peer.refresh_path_addr(tid(3), TransportAddr::from_string("10.0.0.9:1")),
|
||||||
|
"a transport with no path is not re-pointed: that is add_path's job"
|
||||||
|
);
|
||||||
|
assert!(peer.path_on(tid(3)).is_none());
|
||||||
|
}
|
||||||
|
|
||||||
|
#[tokio::test]
|
||||||
|
async fn a_dial_to_a_stale_peer_over_another_transport_is_probed_and_taken_when_it_answers() {
|
||||||
|
let (mut nodes, wifi_0, _wifi_1) = dual_homed_pair().await;
|
||||||
|
let addr_0 = *nodes[0].node.node_addr();
|
||||||
|
let cable = nodes[0].transport_id;
|
||||||
|
|
||||||
|
// Node 1 has not heard node 0 in a long time: the cable is not live.
|
||||||
|
nodes[1].node.get_peer_mut(&addr_0).unwrap().touch(0);
|
||||||
|
assert!(!nodes[1].node.active_peer_link_is_live(&addr_0));
|
||||||
|
|
||||||
|
// An address on another transport arrives. Discovery never dials a
|
||||||
|
// peer with a session; a caller that does gets the same outcome: the
|
||||||
|
// handshake settles as a cross-connection, the address is a candidate.
|
||||||
|
let identity_0 = PeerIdentity::from_pubkey_full(nodes[0].node.identity().pubkey_full());
|
||||||
|
nodes[1]
|
||||||
|
.node
|
||||||
|
.initiate_connection(wifi(), wifi_0.clone(), identity_0)
|
||||||
|
.await
|
||||||
|
.expect("dial starts");
|
||||||
|
for _ in 0..8 {
|
||||||
|
if process_available_packets(&mut nodes).await == 0 {
|
||||||
|
break;
|
||||||
|
}
|
||||||
|
}
|
||||||
|
assert_eq!(nodes[0].node.peer_count(), 1);
|
||||||
|
assert_eq!(nodes[1].node.peer_count(), 1);
|
||||||
|
let p = nodes[1].node.get_peer(&addr_0).unwrap();
|
||||||
|
assert_eq!(
|
||||||
|
p.transport_id(),
|
||||||
|
Some(cable),
|
||||||
|
"nothing moved on the handshake"
|
||||||
|
);
|
||||||
|
assert_eq!(p.path_on(wifi()).unwrap().state(), PathState::Probing);
|
||||||
|
|
||||||
|
// The heartbeat tick proves the wifi. The cable's own echoes then time
|
||||||
|
// out (the peer is silent there, which is why we are here): a hard
|
||||||
|
// signal, and the mandatory switch takes the path that answered.
|
||||||
|
nodes[1].node.run_path_heartbeats().await;
|
||||||
|
for _ in 0..8 {
|
||||||
|
if process_available_packets(&mut nodes).await == 0 {
|
||||||
|
break;
|
||||||
|
}
|
||||||
|
}
|
||||||
|
assert!(
|
||||||
|
nodes[1]
|
||||||
|
.node
|
||||||
|
.get_peer_mut(&addr_0)
|
||||||
|
.unwrap()
|
||||||
|
.mark_path_suspect(cable)
|
||||||
|
);
|
||||||
|
nodes[1].node.run_path_selection();
|
||||||
|
let p = nodes[1].node.get_peer(&addr_0).unwrap();
|
||||||
|
assert_eq!(p.path_on(wifi()).unwrap().state(), PathState::Live);
|
||||||
|
assert_eq!(
|
||||||
|
p.transport_id(),
|
||||||
|
Some(wifi()),
|
||||||
|
"traffic moved to the path that answered"
|
||||||
|
);
|
||||||
|
assert_eq!(p.current_addr(), Some(&wifi_0));
|
||||||
|
// The link record followed the traffic.
|
||||||
|
let link = nodes[1].node.links.get(&p.link_id()).expect("peer's link");
|
||||||
|
assert_eq!(link.transport_id(), wifi());
|
||||||
|
assert_eq!(link.remote_addr(), &wifi_0);
|
||||||
|
assert!(
|
||||||
|
p.path_on(cable).is_some(),
|
||||||
|
"the cable stays for the heartbeat tick to probe"
|
||||||
|
);
|
||||||
|
// And the session is still shared.
|
||||||
|
assert_eq!(nodes[0].node.peer_count(), 1);
|
||||||
|
assert_eq!(nodes[1].node.peer_count(), 1);
|
||||||
|
}
|
||||||
|
|||||||
@@ -1433,7 +1433,7 @@ async fn node_context_mirrors_config_and_immutable_facades() {
|
|||||||
}
|
}
|
||||||
|
|
||||||
#[tokio::test]
|
#[tokio::test]
|
||||||
async fn update_peers_races_new_alternative_without_dropping_active_peer() {
|
async fn update_peers_takes_a_new_alternative_as_a_path_without_dropping_active_peer() {
|
||||||
// The node's *current* (pre-update) peer set must contain `old_peer`, so it
|
// The node's *current* (pre-update) peer set must contain `old_peer`, so it
|
||||||
// is baked into the Config at construction (immutable context = sole store).
|
// is baked into the Config at construction (immutable context = sole store).
|
||||||
let peer_full = Identity::generate();
|
let peer_full = Identity::generate();
|
||||||
@@ -1497,16 +1497,18 @@ async fn update_peers_races_new_alternative_without_dropping_active_peer() {
|
|||||||
|
|
||||||
assert_eq!(outcome.updated, 1);
|
assert_eq!(outcome.updated, 1);
|
||||||
assert_eq!(node.peer_count(), 1, "existing link must stay live");
|
assert_eq!(node.peer_count(), 1, "existing link must stay live");
|
||||||
assert_eq!(node.connection_count(), 1);
|
|
||||||
assert_eq!(
|
assert_eq!(
|
||||||
node.connections()
|
node.connection_count(),
|
||||||
.next()
|
0,
|
||||||
.and_then(|(_, machine)| machine.conn_source_addr()),
|
"a peer with a session is not dialled on a new address: the address is a path"
|
||||||
Some(&new_addr)
|
|
||||||
);
|
);
|
||||||
let active = node.get_peer(&peer_node_addr).unwrap();
|
let active = node.get_peer(&peer_node_addr).unwrap();
|
||||||
assert_eq!(active.link_id(), old_link_id);
|
assert_eq!(active.link_id(), old_link_id);
|
||||||
assert_eq!(active.current_addr(), Some(¤t_addr));
|
// The path on this transport was never acknowledged (bound by
|
||||||
|
// `set_current_addr`, no ack), so it is not eligible and the new address
|
||||||
|
// re-points it; the heartbeat tick probes it there. An eligible path
|
||||||
|
// would have kept its address.
|
||||||
|
assert_eq!(active.current_addr(), Some(&new_addr));
|
||||||
|
|
||||||
for transport in node.transports.values_mut() {
|
for transport in node.transports.values_mut() {
|
||||||
transport.stop().await.ok();
|
transport.stop().await.ok();
|
||||||
|
|||||||
+124
-28
@@ -22,6 +22,16 @@ use std::time::{Duration, Instant};
|
|||||||
/// How often a full-size (MTU-padded) probe goes out on a proven path.
|
/// How often a full-size (MTU-padded) probe goes out on a proven path.
|
||||||
const FULL_SIZE_PROBE_INTERVAL_MS: u64 = 60_000;
|
const FULL_SIZE_PROBE_INTERVAL_MS: u64 = 60_000;
|
||||||
|
|
||||||
|
/// Discovery probes a never-acknowledged standby path gets before it is
|
||||||
|
/// given up as `Dead`. At the doubling backoff from the fast interval,
|
||||||
|
/// capped at the heartbeat interval, that is about half a minute. An
|
||||||
|
/// address we were told about but the peer never answers on — an old
|
||||||
|
/// node's transport, a replayed handshake source, a beacon from a NIC the
|
||||||
|
/// peer no longer sends on — is then no longer probed, and no longer
|
||||||
|
/// counts as a transport the peer is on. `prune_dead_paths` forgets it
|
||||||
|
/// after the grace; a fresh candidate starts the count over.
|
||||||
|
pub const MAX_DISCOVERY_PROBES: u32 = 8;
|
||||||
|
|
||||||
/// Fold one probe outcome into a path's ETX: the long EWMA (α = 1/32) of
|
/// Fold one probe outcome into a path's ETX: the long EWMA (α = 1/32) of
|
||||||
/// the delivery ratio, inverted and clamped like the link ETX. Per report a
|
/// the delivery ratio, inverted and clamped like the link ETX. Per report a
|
||||||
/// raw value is a flap generator on a lightly loaded link; the long average
|
/// raw value is a flap generator on a lightly loaded link; the long average
|
||||||
@@ -86,6 +96,20 @@ pub enum PathState {
|
|||||||
Dead,
|
Dead,
|
||||||
}
|
}
|
||||||
|
|
||||||
|
impl PathState {
|
||||||
|
/// The control-socket spelling: `probing`, `live`, `suspect`, `dead`.
|
||||||
|
/// A fixed string per variant, so a rename here cannot silently change
|
||||||
|
/// what `show_peers` and `path_show` emit and what fipstop matches.
|
||||||
|
pub fn as_str(self) -> &'static str {
|
||||||
|
match self {
|
||||||
|
Self::Probing => "probing",
|
||||||
|
Self::Live => "live",
|
||||||
|
Self::Suspect => "suspect",
|
||||||
|
Self::Dead => "dead",
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
/// Probe bookkeeping for one path: what is outstanding, and when the next
|
/// Probe bookkeeping for one path: what is outstanding, and when the next
|
||||||
/// one may go.
|
/// one may go.
|
||||||
#[derive(Clone, Copy, Debug, Default)]
|
#[derive(Clone, Copy, Debug, Default)]
|
||||||
@@ -120,16 +144,6 @@ pub struct PathPolicy {
|
|||||||
pub rtt_window_ms: u64,
|
pub rtt_window_ms: u64,
|
||||||
}
|
}
|
||||||
|
|
||||||
impl PathPolicy {
|
|
||||||
/// Everything selectable at once; for tests.
|
|
||||||
pub const PERMISSIVE: Self = Self {
|
|
||||||
margin: 1.5,
|
|
||||||
dwell_ms: 0,
|
|
||||||
min_samples: 0,
|
|
||||||
rtt_window_ms: u64::MAX,
|
|
||||||
};
|
|
||||||
}
|
|
||||||
|
|
||||||
/// A post-switch hold on the link cost the tree sees.
|
/// A post-switch hold on the link cost the tree sees.
|
||||||
#[derive(Clone, Copy, Debug)]
|
#[derive(Clone, Copy, Debug)]
|
||||||
struct CostHold {
|
struct CostHold {
|
||||||
@@ -1111,6 +1125,34 @@ impl ActivePeer {
|
|||||||
&mut self.send.paths[idx]
|
&mut self.send.paths[idx]
|
||||||
}
|
}
|
||||||
|
|
||||||
|
/// The peer's address on `transport_id` is `addr` now. Roams the path
|
||||||
|
/// there, if any, without touching its state; a `Dead` path brought back
|
||||||
|
/// at a new address is `Probing` again. Returns whether anything changed.
|
||||||
|
/// Nothing happens for a transport the peer has no path on — that is
|
||||||
|
/// [`add_path`](Self::add_path)'s job — nor for the same address.
|
||||||
|
///
|
||||||
|
/// A peer's address on a transport does change under it: a Wi-Fi Aware
|
||||||
|
/// data path that re-forms comes up with a new link-local, and an
|
||||||
|
/// address that has moved is not one a probe can reach. Without this the
|
||||||
|
/// path kept the dead address for as long as it lived.
|
||||||
|
pub fn refresh_path_addr(&mut self, transport_id: TransportId, addr: TransportAddr) -> bool {
|
||||||
|
let Some(path) = self.path_on_mut(transport_id) else {
|
||||||
|
return false;
|
||||||
|
};
|
||||||
|
if path.addr == addr {
|
||||||
|
return false;
|
||||||
|
}
|
||||||
|
path.addr = addr;
|
||||||
|
#[cfg(any(target_os = "linux", target_os = "macos"))]
|
||||||
|
path.clear_connected_udp();
|
||||||
|
if path.state == PathState::Dead {
|
||||||
|
path.state = PathState::Probing;
|
||||||
|
path.dead_since_ms = None;
|
||||||
|
}
|
||||||
|
path.probe.next_at_ms = 0;
|
||||||
|
true
|
||||||
|
}
|
||||||
|
|
||||||
/// An authentic frame arrived on `transport_id`: the path there, if any,
|
/// An authentic frame arrived on `transport_id`: the path there, if any,
|
||||||
/// is `rx_live` as of `now_ms`.
|
/// is `rx_live` as of `now_ms`.
|
||||||
pub fn note_path_rx(&mut self, transport_id: TransportId, now_ms: u64) {
|
pub fn note_path_rx(&mut self, transport_id: TransportId, now_ms: u64) {
|
||||||
@@ -1125,9 +1167,10 @@ impl ActivePeer {
|
|||||||
/// backoff has not expired. `backoff_cap_ms` bounds the retry interval,
|
/// backoff has not expired. `backoff_cap_ms` bounds the retry interval,
|
||||||
/// which doubles from `base_ms` per unanswered probe.
|
/// which doubles from `base_ms` per unanswered probe.
|
||||||
///
|
///
|
||||||
/// Tests only. In production [`plan_heartbeats`](Self::plan_heartbeats)
|
/// Unit-test sampler for the selection tests, which need a path fed a
|
||||||
/// is the one issuer of probes, so no two writers race for
|
/// chosen round trip without driving a whole heartbeat schedule. Every
|
||||||
/// `probe.outstanding`.
|
/// node-level test goes through [`plan_heartbeats`](Self::plan_heartbeats),
|
||||||
|
/// the one issuer of probes in production.
|
||||||
#[cfg(test)]
|
#[cfg(test)]
|
||||||
pub fn take_probe(
|
pub fn take_probe(
|
||||||
&mut self,
|
&mut self,
|
||||||
@@ -1163,12 +1206,11 @@ impl ActivePeer {
|
|||||||
remote_id: u32,
|
remote_id: u32,
|
||||||
now_ms: u64,
|
now_ms: u64,
|
||||||
) {
|
) {
|
||||||
let path = self.add_path(transport_id, addr.clone());
|
self.add_path(transport_id, addr.clone());
|
||||||
if path.addr != addr {
|
self.refresh_path_addr(transport_id, addr);
|
||||||
path.addr = addr;
|
let path = self
|
||||||
#[cfg(any(target_os = "linux", target_os = "macos"))]
|
.path_on_mut(transport_id)
|
||||||
path.clear_connected_udp();
|
.expect("path added just above");
|
||||||
}
|
|
||||||
path.remote_id = Some(remote_id);
|
path.remote_id = Some(remote_id);
|
||||||
if path.state == PathState::Dead {
|
if path.state == PathState::Dead {
|
||||||
// The peer is probing a path we had given up on: it is back,
|
// The peer is probing a path we had given up on: it is back,
|
||||||
@@ -1497,6 +1539,15 @@ impl ActivePeer {
|
|||||||
/// The silence hint: our active path silent for two of the peer's
|
/// The silence hint: our active path silent for two of the peer's
|
||||||
/// intervals on it while a standby hears the peer triggers a probe now,
|
/// intervals on it while a standby hears the peer triggers a probe now,
|
||||||
/// never `Suspect` (see §7 for the loop that would otherwise follow).
|
/// never `Suspect` (see §7 for the loop that would otherwise follow).
|
||||||
|
///
|
||||||
|
/// A peer with one `Live` path is not heartbeated here at all.
|
||||||
|
/// Selection has nothing to move to, so a `Suspect` mark on it changes
|
||||||
|
/// nothing, and the link heartbeat already keeps its liveness; five
|
||||||
|
/// probes a second on every single-path link would be cost without a
|
||||||
|
/// decision behind it. Probes start the moment a second path — a
|
||||||
|
/// candidate included — exists, which is when a verdict can act; and a
|
||||||
|
/// lone path that is `Suspect` (the peer closed it, nowhere to go) is
|
||||||
|
/// probed so its ack can bring it back.
|
||||||
pub fn plan_heartbeats(&mut self, now_ms: u64, timing: &HeartbeatTiming) -> HeartbeatPlan {
|
pub fn plan_heartbeats(&mut self, now_ms: u64, timing: &HeartbeatTiming) -> HeartbeatPlan {
|
||||||
let HeartbeatTiming {
|
let HeartbeatTiming {
|
||||||
fast_ms,
|
fast_ms,
|
||||||
@@ -1506,6 +1557,23 @@ impl ActivePeer {
|
|||||||
} = *timing;
|
} = *timing;
|
||||||
let mut plan = HeartbeatPlan::default();
|
let mut plan = HeartbeatPlan::default();
|
||||||
let active = self.send.active;
|
let active = self.send.active;
|
||||||
|
let alone_and_live = self
|
||||||
|
.send
|
||||||
|
.paths
|
||||||
|
.iter()
|
||||||
|
.filter(|p| p.state != PathState::Dead)
|
||||||
|
.count()
|
||||||
|
< 2
|
||||||
|
&& active
|
||||||
|
.and_then(|i| self.send.paths.get(i))
|
||||||
|
.is_some_and(|p| p.state == PathState::Live);
|
||||||
|
if alone_and_live {
|
||||||
|
for path in self.send.paths.iter_mut() {
|
||||||
|
path.probe.outstanding = None;
|
||||||
|
path.probe.timed_out = None;
|
||||||
|
}
|
||||||
|
return plan;
|
||||||
|
}
|
||||||
let newest_rx = self.send.paths.iter().filter_map(|p| p.rx_live_at_ms).max();
|
let newest_rx = self.send.paths.iter().filter_map(|p| p.rx_live_at_ms).max();
|
||||||
for (i, path) in self.send.paths.iter_mut().enumerate() {
|
for (i, path) in self.send.paths.iter_mut().enumerate() {
|
||||||
if path.state == PathState::Dead {
|
if path.state == PathState::Dead {
|
||||||
@@ -1549,6 +1617,16 @@ impl ActivePeer {
|
|||||||
path.probe.next_at_ms = 0;
|
path.probe.next_at_ms = 0;
|
||||||
} else {
|
} else {
|
||||||
path.probe.unanswered = path.probe.unanswered.saturating_add(1);
|
path.probe.unanswered = path.probe.unanswered.saturating_add(1);
|
||||||
|
// A standby the peer has never answered on is given up
|
||||||
|
// after the discovery budget. Never the active path:
|
||||||
|
// an old node answers no probe there either, and the
|
||||||
|
// handshake proved it.
|
||||||
|
if !ours && path.probe.unanswered >= MAX_DISCOVERY_PROBES {
|
||||||
|
path.state = PathState::Dead;
|
||||||
|
path.dead_since_ms = Some(now_ms);
|
||||||
|
path.probe.timed_out = None;
|
||||||
|
continue;
|
||||||
|
}
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
@@ -1582,11 +1660,16 @@ impl ActivePeer {
|
|||||||
.min(discovery_cap_ms.max(interval))
|
.min(discovery_cap_ms.max(interval))
|
||||||
};
|
};
|
||||||
path.probe.next_at_ms = now_ms.saturating_add(delay.max(1));
|
path.probe.next_at_ms = now_ms.saturating_add(delay.max(1));
|
||||||
let full_size = !path.acked_once
|
// Discovery probes are full-size so a medium that passes small
|
||||||
|| path
|
// frames and drops large ones never proves itself — except on
|
||||||
.last_full_probe_ms
|
// the active path, which the handshake proved and whose MTU it
|
||||||
.is_none_or(|t| now_ms.saturating_sub(t) >= FULL_SIZE_PROBE_INTERVAL_MS);
|
// seeded; there the once-a-minute full-size probe is enough.
|
||||||
if full_size {
|
let full_size = (!path.acked_once && !ours)
|
||||||
|
|| match path.last_full_probe_ms {
|
||||||
|
Some(t) => now_ms.saturating_sub(t) >= FULL_SIZE_PROBE_INTERVAL_MS,
|
||||||
|
None => !ours,
|
||||||
|
};
|
||||||
|
if full_size || path.last_full_probe_ms.is_none() {
|
||||||
path.last_full_probe_ms = Some(now_ms);
|
path.last_full_probe_ms = Some(now_ms);
|
||||||
}
|
}
|
||||||
plan.sends.push(HeartbeatSend {
|
plan.sends.push(HeartbeatSend {
|
||||||
@@ -1658,11 +1741,24 @@ impl ActivePeer {
|
|||||||
self.send.active = Some(new_active);
|
self.send.active = Some(new_active);
|
||||||
}
|
}
|
||||||
|
|
||||||
/// Clear the probe backoff on every path over `transport_id`.
|
/// The transport `transport_id` is back: clear the probe backoff on
|
||||||
pub fn reset_probe_backoff_on(&mut self, transport_id: TransportId) {
|
/// its path, and a path given up as `Dead` while it was gone is
|
||||||
if let Some(path) = self.path_on_mut(transport_id) {
|
/// `Probing` again, its RTT window and ETX intact, so the next
|
||||||
path.reset_probe_backoff();
|
/// heartbeat tick re-proves it rather than measuring it from nothing.
|
||||||
|
/// Returns whether a `Dead` path was revived.
|
||||||
|
pub fn reset_probe_backoff_on(&mut self, transport_id: TransportId) -> bool {
|
||||||
|
let Some(path) = self.path_on_mut(transport_id) else {
|
||||||
|
return false;
|
||||||
|
};
|
||||||
|
path.reset_probe_backoff();
|
||||||
|
if path.state == PathState::Dead {
|
||||||
|
path.state = PathState::Probing;
|
||||||
|
path.dead_since_ms = None;
|
||||||
|
path.probe.outstanding = None;
|
||||||
|
path.probe.timed_out = None;
|
||||||
|
return true;
|
||||||
}
|
}
|
||||||
|
false
|
||||||
}
|
}
|
||||||
|
|
||||||
// === Handshake Resend ===
|
// === Handshake Resend ===
|
||||||
|
|||||||
+3
-2
@@ -9,8 +9,9 @@ mod active;
|
|||||||
pub(crate) mod machine;
|
pub(crate) mod machine;
|
||||||
|
|
||||||
pub use active::{
|
pub use active::{
|
||||||
ActivePeer, ConnectivityState, HeartbeatPlan, HeartbeatSend, HeartbeatTiming, PathPolicy,
|
ActivePeer, ConnectivityState, HeartbeatPlan, HeartbeatSend, HeartbeatTiming,
|
||||||
PathState, PathSwitch, PathWithdrawal, PeerPath, SwitchReason,
|
MAX_DISCOVERY_PROBES, PathPolicy, PathState, PathSwitch, PathWithdrawal, PeerPath,
|
||||||
|
SwitchReason,
|
||||||
};
|
};
|
||||||
|
|
||||||
use crate::NodeAddr;
|
use crate::NodeAddr;
|
||||||
|
|||||||
@@ -646,6 +646,13 @@ impl Fmp {
|
|||||||
|
|
||||||
if snap.has_existing_peer {
|
if snap.has_existing_peer {
|
||||||
let peer_addr = *wire.peer_identity.node_addr();
|
let peer_addr = *wire.peer_identity.node_addr();
|
||||||
|
// Which transport the msg1 arrived on plays no part here: a
|
||||||
|
// handshake never creates path state. A peer dialling us over
|
||||||
|
// a second transport gets the same answer as one dialling over
|
||||||
|
// the first — a rekey or a duplicate — and the transport becomes
|
||||||
|
// a path only through the authenticated, replay-checked probe
|
||||||
|
// exchange. Both ends then resolve on the same information; a
|
||||||
|
// rule that read this end's view of its own liveness split them.
|
||||||
match (snap.existing_peer_epoch, wire.remote_epoch) {
|
match (snap.existing_peer_epoch, wire.remote_epoch) {
|
||||||
(Some(existing), Some(new)) if existing != new => {
|
(Some(existing), Some(new)) if existing != new => {
|
||||||
// Epoch mismatch → peer restart.
|
// Epoch mismatch → peer restart.
|
||||||
|
|||||||
@@ -707,3 +707,29 @@ fn retirements_are_grouped_after_drains_and_before_rekey_initiations() {
|
|||||||
matches!(actions[4], ConnAction::InitiateRekey { peer } if peer == make_node_addr(0x03))
|
matches!(actions[4], ConnAction::InitiateRekey { peer } if peer == make_node_addr(0x03))
|
||||||
);
|
);
|
||||||
}
|
}
|
||||||
|
|
||||||
|
#[test]
|
||||||
|
fn a_live_peer_s_msg1_is_classified_the_same_on_every_transport() {
|
||||||
|
// Same epoch, session old enough to rekey. The transport the msg1
|
||||||
|
// arrived on is not an input: a handshake never creates path state, so
|
||||||
|
// a live peer dialling over a second transport is the rekey it looks
|
||||||
|
// like, exactly as a dial over the first would be. Both ends then
|
||||||
|
// decide from the same facts.
|
||||||
|
let mut snap = establish_snapshot();
|
||||||
|
snap.has_existing_peer = true;
|
||||||
|
snap.existing_peer_epoch = Some([1u8; 8]);
|
||||||
|
snap.has_session = true;
|
||||||
|
snap.existing_session_age_secs = 31;
|
||||||
|
let wire = wire_outcome(Some([1u8; 8]));
|
||||||
|
assert!(matches!(
|
||||||
|
Fmp::new().establish_inbound(&snap, &wire),
|
||||||
|
InboundDecision::RekeyRespond { .. }
|
||||||
|
));
|
||||||
|
|
||||||
|
// A restart is a restart, whatever transport it arrives on.
|
||||||
|
let restarted = wire_outcome(Some([2u8; 8]));
|
||||||
|
assert!(matches!(
|
||||||
|
Fmp::new().establish_inbound(&snap, &restarted),
|
||||||
|
InboundDecision::RestartThenPromote { .. }
|
||||||
|
));
|
||||||
|
}
|
||||||
|
|||||||
@@ -341,7 +341,6 @@ impl MmpMetrics {
|
|||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
/// Smoothed ETX (long-term EWMA), or `None` if not yet initialized.
|
|
||||||
/// The quality index, [`quality_index`](super::quality_index) of the
|
/// The quality index, [`quality_index`](super::quality_index) of the
|
||||||
/// smoothed ETX and the smoothed RTT. `None` until both are measured.
|
/// smoothed ETX and the smoothed RTT. `None` until both are measured.
|
||||||
pub fn quality_index(&self) -> Option<f64> {
|
pub fn quality_index(&self) -> Option<f64> {
|
||||||
@@ -351,6 +350,7 @@ impl MmpMetrics {
|
|||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
|
/// Smoothed ETX (long-term EWMA), or `None` if not yet initialized.
|
||||||
pub fn smoothed_etx(&self) -> Option<f64> {
|
pub fn smoothed_etx(&self) -> Option<f64> {
|
||||||
if self.etx_trend.initialized() {
|
if self.etx_trend.initialized() {
|
||||||
Some(self.etx_trend.long())
|
Some(self.etx_trend.long())
|
||||||
|
|||||||
@@ -57,6 +57,11 @@ pub struct LoopbackTransport {
|
|||||||
/// transport's `IFF_RUNNING`. `None`: not interface-bound, no presence
|
/// transport's `IFF_RUNNING`. `None`: not interface-bound, no presence
|
||||||
/// reported, which is the default.
|
/// reported, which is the default.
|
||||||
carrier: Mutex<Option<bool>>,
|
carrier: Mutex<Option<bool>>,
|
||||||
|
/// Every `close_connection` the node asked for, in order. Loopback is
|
||||||
|
/// connectionless so the close itself does nothing; the record lets a
|
||||||
|
/// test say whether a socket a connection-oriented transport would own
|
||||||
|
/// was kept or closed.
|
||||||
|
closed: Mutex<Vec<TransportAddr>>,
|
||||||
}
|
}
|
||||||
|
|
||||||
impl LoopbackTransport {
|
impl LoopbackTransport {
|
||||||
@@ -84,9 +89,20 @@ impl LoopbackTransport {
|
|||||||
registry,
|
registry,
|
||||||
discovered: Mutex::new(Vec::new()),
|
discovered: Mutex::new(Vec::new()),
|
||||||
carrier: Mutex::new(None),
|
carrier: Mutex::new(None),
|
||||||
|
closed: Mutex::new(Vec::new()),
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
|
/// Record a `close_connection` call.
|
||||||
|
pub fn record_close(&self, addr: &TransportAddr) {
|
||||||
|
self.closed.lock().unwrap().push(addr.clone());
|
||||||
|
}
|
||||||
|
|
||||||
|
/// The addresses the node has asked to close on this transport.
|
||||||
|
pub fn closed(&self) -> Vec<TransportAddr> {
|
||||||
|
self.closed.lock().unwrap().clone()
|
||||||
|
}
|
||||||
|
|
||||||
/// Pretend this transport is bound to an interface with (or without)
|
/// Pretend this transport is bound to an interface with (or without)
|
||||||
/// carrier; `None` returns it to reporting no presence at all.
|
/// carrier; `None` returns it to reporting no presence at all.
|
||||||
pub fn set_carrier(&self, carrier: Option<bool>) {
|
pub fn set_carrier(&self, carrier: Option<bool>) {
|
||||||
|
|||||||
+15
-15
@@ -248,6 +248,20 @@ pub enum TransportError {
|
|||||||
}
|
}
|
||||||
|
|
||||||
impl TransportError {
|
impl TransportError {
|
||||||
|
/// Whether the kernel refused the send for want of a route: the
|
||||||
|
/// interface is up but nothing is reachable through it. A hard signal
|
||||||
|
/// that the path is gone (`ENETUNREACH`, `EHOSTUNREACH`), distinct from
|
||||||
|
/// `is_transient`: the binder is not going to fix this.
|
||||||
|
pub fn is_unreachable(&self) -> bool {
|
||||||
|
match self {
|
||||||
|
Self::Io(e) => matches!(
|
||||||
|
e.kind(),
|
||||||
|
std::io::ErrorKind::NetworkUnreachable | std::io::ErrorKind::HostUnreachable
|
||||||
|
),
|
||||||
|
_ => false,
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
/// Whether this failure is expected to clear on its own.
|
/// Whether this failure is expected to clear on its own.
|
||||||
///
|
///
|
||||||
/// The distinction callers need is not *what* went wrong but whether
|
/// The distinction callers need is not *what* went wrong but whether
|
||||||
@@ -268,20 +282,6 @@ impl TransportError {
|
|||||||
/// which is a statement about the peer rather than about this node's
|
/// which is a statement about the peer rather than about this node's
|
||||||
/// ability to transmit, and the existing retry paths for them already sit
|
/// ability to transmit, and the existing retry paths for them already sit
|
||||||
/// at a different layer.
|
/// at a different layer.
|
||||||
/// Whether the kernel refused the send for want of a route: the
|
|
||||||
/// interface is up but nothing is reachable through it. A hard signal
|
|
||||||
/// that the path is gone (`ENETUNREACH`, `EHOSTUNREACH`), distinct from
|
|
||||||
/// `is_transient`: the binder is not going to fix this.
|
|
||||||
pub fn is_unreachable(&self) -> bool {
|
|
||||||
match self {
|
|
||||||
Self::Io(e) => matches!(
|
|
||||||
e.kind(),
|
|
||||||
std::io::ErrorKind::NetworkUnreachable | std::io::ErrorKind::HostUnreachable
|
|
||||||
),
|
|
||||||
_ => false,
|
|
||||||
}
|
|
||||||
}
|
|
||||||
|
|
||||||
pub fn is_transient(&self) -> bool {
|
pub fn is_transient(&self) -> bool {
|
||||||
match self {
|
match self {
|
||||||
// The interface is absent or mid-rebind. The binder is polling for
|
// The interface is absent or mid-rebind. The binder is polling for
|
||||||
@@ -1250,7 +1250,7 @@ impl TransportHandle {
|
|||||||
#[cfg(ble_available)]
|
#[cfg(ble_available)]
|
||||||
TransportHandle::Ble(t) => t.close_connection_async(addr).await,
|
TransportHandle::Ble(t) => t.close_connection_async(addr).await,
|
||||||
#[cfg(test)]
|
#[cfg(test)]
|
||||||
TransportHandle::Loopback(_) => {} // connectionless no-op
|
TransportHandle::Loopback(t) => t.record_close(addr), // connectionless; recorded for tests
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
|
|||||||
@@ -115,11 +115,10 @@ Explicit topologies exercising non-UDP transports.
|
|||||||
`node.path.*`; see the file header for what to read from a run.
|
`node.path.*`; see the file header for what to read from a run.
|
||||||
- **dual-udp-flap**: the all-IP twin (`[n01, n02, udp-veth+udp]`): a veth
|
- **dual-udp-flap**: the all-IP twin (`[n01, n02, udp-veth+udp]`): a veth
|
||||||
carrying IP with an interface-bound UDP instance at each end, plus UDP
|
carrying IP with an interface-bound UDP instance at each end, plus UDP
|
||||||
over the bridge. The veth half is handed to the daemon by the runner
|
over the bridge. Both are static addresses on the dial owner, so two
|
||||||
(`fipsctl connect ... udp/<veth>`) once the pair has peered, and becomes a
|
handshakes run at startup and the second is kept as a path under the
|
||||||
path under the existing session. Exercises `udp.interface` on the listen
|
first's session. Exercises `udp.interface` on the listen and per-peer
|
||||||
and per-peer connected sockets, and a configured address on a new
|
connected sockets, and a second handshake to a live peer becoming a path.
|
||||||
transport becoming a path.
|
|
||||||
- **tcp-mesh**: 6-node mesh with 4 UDP and 3 TCP edges. Both transports use
|
- **tcp-mesh**: 6-node mesh with 4 UDP and 3 TCP edges. Both transports use
|
||||||
static peer config. Netem mutation (30% fraction, every 20-40s) and link
|
static peer config. Netem mutation (30% fraction, every 20-40s) and link
|
||||||
flaps (1 link max, 10-20s down).
|
flaps (1 link max, 10-20s down).
|
||||||
|
|||||||
@@ -58,24 +58,31 @@ def generate_peers_block(
|
|||||||
for peer_id in sorted(outbound_peers):
|
for peer_id in sorted(outbound_peers):
|
||||||
peer = topology.nodes[peer_id]
|
peer = topology.nodes[peer_id]
|
||||||
transport = topology.transport_for_edge(node_id, peer_id)
|
transport = topology.transport_for_edge(node_id, peer_id)
|
||||||
if topology.is_dual_udp_edge(node_id, peer_id):
|
dual = topology.is_dual_udp_edge(node_id, peer_id)
|
||||||
# The veth half of a dual edge is found by beacon (Ethernet) or
|
# (transport, addr, priority) per address. A dual edge's Ethernet
|
||||||
# added as a path by the runner after the pair has peered
|
# half is found by beacon, so only its bridge half is listed; a dual
|
||||||
# (udp-veth); the bridge half is dialled from here.
|
# udp-veth edge lists both halves, bridge first, and the daemon takes
|
||||||
transport = "udp"
|
# the second completed handshake as a path under the first's session.
|
||||||
|
addresses = []
|
||||||
if transport == UDP_VETH:
|
if transport == UDP_VETH:
|
||||||
link = next(l for l in topology.udp_veth_links(node_id) if l.peer_id == peer_id)
|
link = next(l for l in topology.udp_veth_links(node_id) if l.peer_id == peer_id)
|
||||||
transport = f"udp/{link.instance}"
|
if dual:
|
||||||
addr = link.peer_addr
|
addresses.append((bridge, f"{peer.docker_ip}:{_TRANSPORT_PORTS['udp']}", 1))
|
||||||
|
addresses.append((f"udp/{link.instance}", link.peer_addr, 10 if dual else 1))
|
||||||
else:
|
else:
|
||||||
|
if dual:
|
||||||
|
transport = "udp"
|
||||||
addr = f"{peer.docker_ip}:{_TRANSPORT_PORTS.get(transport, 2121)}"
|
addr = f"{peer.docker_ip}:{_TRANSPORT_PORTS.get(transport, 2121)}"
|
||||||
if transport == "udp":
|
if transport == "udp":
|
||||||
transport = bridge
|
transport = bridge
|
||||||
|
addresses.append((transport, addr, 1))
|
||||||
lines.append(f' - npub: "{peer.npub}"')
|
lines.append(f' - npub: "{peer.npub}"')
|
||||||
lines.append(f' alias: "{peer_id}"')
|
lines.append(f' alias: "{peer_id}"')
|
||||||
lines.append(f" addresses:")
|
lines.append(f" addresses:")
|
||||||
lines.append(f" - transport: {transport}")
|
for transport, addr, priority in addresses:
|
||||||
lines.append(f' addr: "{addr}"')
|
lines.append(f" - transport: {transport}")
|
||||||
|
lines.append(f' addr: "{addr}"')
|
||||||
|
lines.append(f" priority: {priority}")
|
||||||
lines.append(f" connect_policy: auto_connect")
|
lines.append(f" connect_policy: auto_connect")
|
||||||
return "\n".join(lines)
|
return "\n".join(lines)
|
||||||
|
|
||||||
|
|||||||
@@ -389,11 +389,6 @@ class SimRunner:
|
|||||||
self._sleep(wait)
|
self._sleep(wait)
|
||||||
self._take_snapshot("warmup")
|
self._take_snapshot("warmup")
|
||||||
|
|
||||||
# The veth half of every udp-veth+udp edge: UDP has no beacon, so
|
|
||||||
# the runner hands the daemon the address once the pair has peered
|
|
||||||
# over the bridge, and it becomes a path under that session.
|
|
||||||
self._add_udp_veth_paths()
|
|
||||||
|
|
||||||
# Populate npub cache after convergence (nodes must be running)
|
# Populate npub cache after convergence (nodes must be running)
|
||||||
if self.peer_churn_mgr:
|
if self.peer_churn_mgr:
|
||||||
self.peer_churn_mgr.refresh_all_npubs()
|
self.peer_churn_mgr.refresh_all_npubs()
|
||||||
@@ -404,67 +399,13 @@ class SimRunner:
|
|||||||
if self.link_swap_mgr:
|
if self.link_swap_mgr:
|
||||||
self.link_swap_mgr.setup_initial()
|
self.link_swap_mgr.setup_initial()
|
||||||
|
|
||||||
def _add_udp_veth_paths(self, only_node: str | None = None):
|
|
||||||
"""Give each dual udp-veth edge its veth path.
|
|
||||||
|
|
||||||
Sent from the edge's dial owner (the side whose static config holds
|
|
||||||
the bridge address) as a control-socket ``connect`` naming the
|
|
||||||
interface-bound instance: to a peer it already holds a session with,
|
|
||||||
the daemon adds that as a path rather than dialling. Waits for the
|
|
||||||
bridge session first, so the command cannot become the first dial.
|
|
||||||
"""
|
|
||||||
from .control import send_command
|
|
||||||
|
|
||||||
outbound = self.topology.directed_outbound()
|
|
||||||
for node_id in sorted(self.topology.nodes):
|
|
||||||
if only_node is not None and node_id != only_node:
|
|
||||||
continue
|
|
||||||
for link in self.topology.udp_veth_links(node_id):
|
|
||||||
if not self.topology.is_dual_udp_edge(node_id, link.peer_id):
|
|
||||||
continue
|
|
||||||
if link.peer_id not in outbound.get(node_id, []):
|
|
||||||
continue
|
|
||||||
if node_id in self._down_nodes or link.peer_id in self._down_nodes:
|
|
||||||
continue
|
|
||||||
container = self.topology.container_name(node_id)
|
|
||||||
npub = self.topology.nodes[link.peer_id].npub
|
|
||||||
params = {
|
|
||||||
"npub": npub,
|
|
||||||
"address": link.peer_addr,
|
|
||||||
"transport": f"udp/{link.instance}",
|
|
||||||
}
|
|
||||||
added = None
|
|
||||||
for _ in range(30):
|
|
||||||
if send_command(container, "path_show", {"npub": npub}) is None:
|
|
||||||
time.sleep(1) # not peered over the bridge yet
|
|
||||||
continue
|
|
||||||
added = send_command(container, "connect", params)
|
|
||||||
break
|
|
||||||
if added is None:
|
|
||||||
log.warning(
|
|
||||||
"udp-veth path %s -> %s via %s not added",
|
|
||||||
node_id, link.peer_id, link.instance,
|
|
||||||
)
|
|
||||||
else:
|
|
||||||
log.info(
|
|
||||||
"udp-veth path %s -> %s via %s (%s)",
|
|
||||||
node_id, link.peer_id, link.instance, link.peer_addr,
|
|
||||||
)
|
|
||||||
|
|
||||||
def _handle_node_restart(self, node_id: str):
|
def _handle_node_restart(self, node_id: str):
|
||||||
"""Called after a node container is restarted.
|
"""Called after a node container is restarted.
|
||||||
|
|
||||||
Re-adds the node's udp-veth paths once it has re-peered, and for
|
For ephemeral identity nodes, waits briefly for the daemon to
|
||||||
ephemeral identity nodes waits briefly for the daemon to start,
|
start, then queries its new npub and updates the peer churn
|
||||||
then queries its new npub and updates the peer churn manager's
|
manager's cache.
|
||||||
cache.
|
|
||||||
"""
|
"""
|
||||||
if any(
|
|
||||||
self.topology.is_dual_udp_edge(node_id, link.peer_id)
|
|
||||||
for link in self.topology.udp_veth_links(node_id)
|
|
||||||
):
|
|
||||||
time.sleep(2)
|
|
||||||
self._add_udp_veth_paths(only_node=node_id)
|
|
||||||
if not self.peer_churn_mgr:
|
if not self.peer_churn_mgr:
|
||||||
return
|
return
|
||||||
if node_id not in self.peer_churn_mgr.ephemeral_nodes:
|
if node_id not in self.peer_churn_mgr.ephemeral_nodes:
|
||||||
|
|||||||
Reference in New Issue
Block a user