fix(control): make connect and disconnect do what they report

Two control-socket commands answered success without doing what the
caller asked. They land as one commit because they share a test file:
the connect fix creates `src/node/tests/control.rs` and the `mod
control;` line that declares it, and the disconnect fix adds to that
file without declaring it, so the disconnect fix on its own does not
build.

Each original commit message follows in full.

--- connect must refresh the path of an already-connected peer ---

`connect` on a peer the node already holds a session to is silently a
no-op. `api_connect` builds an ephemeral PeerConfig and hands it to
`initiate_peer_connection`, which returns Ok(()) as soon as
`self.peers.contains_key(&peer_node_addr)`. The control socket answers
`{"status":"ok"}` and `fipsctl connect` prints success, but the node
never tries the address it was given.

That is the wrong answer whenever the caller knows a path the node does
not. An operator moving a peer onto a freshly-provisioned link, or a
supervising process that has just observed a second, faster path come
up, has no way to make the node use it — the peer stays on whatever path
it first authenticated over until that path dies and the ordinary retry
machinery rediscovers it.

`update_peers`, the other runtime peer-mutation entry point, already
gets this right: for a peer that is currently active it calls
`try_active_peer_alternative_addresses`, which drops candidates matching
the peer's current path, keeps the rest, and starts a parallel
handshake — promotion happens only after that handshake authenticates,
so a bad or spoofed address cannot displace a healthy link. Route
`api_connect`'s already-connected case through the same helper rather
than growing a second mechanism beside it.

Behaviour for an unknown or merely-connecting peer is unchanged, and
`connect` stays ephemeral: the peer is not persisted to config and gets
no auto-reconnect, so a refresh that fails leaves no residue. The
response gains one additive field, `refreshed`, so the caller can tell
"started an alternate-path handshake" from "already on this exact path
and it is fresh", which was previously indistinguishable from a dial.
`fipsctl` pretty-prints the whole `data` object and `fipstop` reads only
`status`, so neither is disturbed.

Note that `connect` deliberately bypasses the reconciler's opportunistic
discovery budget — it is a manual command and the caller is treated as
authoritative about what it can see — so it is bounded only by
`path_candidate_attempt_budget`. A caller that re-announces the same
peer on the same fresh path every cycle now provably costs nothing:
that case is a no-op with a regression test.

--- disconnect must close the transport connection, not just the peer ---

`api_disconnect` notifies the peer and calls `remove_active_peer`, which
frees every node-side structure — session, indices, link, address
mapping, tree and bloom state. It never touches the transport. On a
connection-oriented transport (TCP, Tor, Nym, BLE) the pool entry, the
underlying socket and its inbound-slot accounting therefore survive the
peer the node has just forgotten, until the far end closes or the
receive loop errors. An operator who disconnects a peer to free a slot
does not free the slot.

This is the same hazard `cleanup_stale_connection` was fixed for, and
the reasoning there applies verbatim: closing twice is harmless, because
every `close_connection` implementation is `if let Some(conn) =
pool.remove(addr)` and the connectionless default is a no-op. Read the
peer's transport id and current address before removal and close the
connection after it, mirroring that path. `current_addr` rather than the
link's remote address, because roaming updates the former and it is the
address the pool entry is keyed by.

No effect on UDP, Ethernet or loopback, whose `close_connection` is the
connectionless no-op — the change is a real leak fix on TCP and Tor
today, and on pooled link transports generally.

Not addressed here: `disconnect` still errors with `peer not found` for
an identity that is only mid-handshake, so withdrawing a peer during its
handshake leaves that leg resending msg1 until the handshake timeout
bounds it. That is a separate, timeout-bounded case.

Tests: a TCP two-node test asserts the pool entry is gone after
`api_disconnect` (it fails on the pre-fix code with `Connected`); a
connectionless test asserts the no-op default neither errors nor panics
and that a repeat withdrawal is a clean `peer not found`.

--- changelog ---

Both fixes get an entry under Fixed. Each names the mechanism and what
stays unchanged, and the disconnect entry carries the case its commit
says it does not fix. A duplicate blank line in the Changed section,
left there by an earlier entry, is removed here as well.
This commit is contained in:
Arjen
2026-08-20 21:50:30 +00:00
committed by Johnathan Corgan
parent 23ec0a7b81
commit 4ecc192456
7 changed files with 511 additions and 7 deletions
+29 -1
View File
@@ -191,7 +191,6 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
folded into the new tables with a one-time deprecation warning; migrate your
`fips.yaml` to the new keys.
- Inbound traversal offers are now admitted against a per-sender allowance as
well as the global pool. The intake path previously took a permit from a
single semaphore before any identity check, with the sender's npub used only
@@ -566,6 +565,35 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
counter now charges at the node that makes the decision rather than at the
hop after it.
- `disconnect` on the control socket now closes the transport connection
rather than only the peer. It notified the peer and freed every node-side
structure, sessions, indices, links, address mapping, tree and bloom state,
and never touched the transport, so on a connection-oriented transport (TCP,
Tor, Nym, BLE) the pool entry, the socket and its inbound-slot accounting
survived the peer the node had just forgotten, until the far end closed or
the receive loop errored. An operator who disconnected a peer to free a slot
did not free the slot. No effect on UDP, Ethernet or loopback, whose
`close_connection` is the connectionless no-op. Still not addressed:
`disconnect` reports `peer not found` for an identity that is only
mid-handshake, so withdrawing a peer during its handshake leaves that leg
resending msg1 until the handshake timeout bounds it.
- `connect` on the control socket now tries the address it was given for a
peer the node is already connected to, instead of reporting success without
doing anything. The command built an ephemeral peer configuration and handed
it to the ordinary dial path, which returns success the moment the peer is
already held, so `fipsctl connect` printed success and the node never
attempted the path. An operator moving a peer onto a freshly provisioned
link, or a supervising process that has just seen a second path come up, had
no way to make the node use it: the peer stayed where it first authenticated
until that path died. The address is now tried as an alternate path
alongside the live one, through the same helper a runtime peer refresh uses,
so promotion happens only after the alternate handshake authenticates and a
wrong address cannot displace a healthy link. The response gains an additive
`refreshed` field distinguishing "started an alternate-path handshake" from
"already on this exact path and it is fresh". `connect` stays ephemeral: the
peer is not written to configuration and gets no auto-reconnect.
### Security
- The influence a remote party has over path MTU is now bounded, and the
+11
View File
@@ -164,6 +164,17 @@ not reproduced here to avoid duplicating the source.
| `connect` | `npub` (bech32), `address` (transport endpoint), `transport` (`udp`, `tcp`, `tor`, `nym`, `ethernet`) | Asks the node to dial the peer over the named transport. The named transport must be configured and running. Returns the API result on success or an error string on failure. |
| `disconnect` | `npub` (bech32) | Asks the node to drop the link to the named peer. |
`connect` on a peer the node is **already connected to** neither tears the
live link down nor ignores the address: the address is tried as an alternate
path alongside the existing one, and the peer moves to it only if that
handshake authenticates. The response carries `refreshed` — `true` when such a
handshake was started, `false` when the peer is already on this exact path and
that path is fresh (a successful no-op). A `connect` that starts an ordinary
dial to a peer the node does not yet hold also reports `refreshed: false`.
`connect` is ephemeral either way: the peer is not written to the config file
and gets no auto-reconnect, so an attempt that fails leaves no residue.
Both commands run on the daemon's main task and may block briefly
while the node mutates its state.
+73 -5
View File
@@ -3167,6 +3167,15 @@ impl Node {
/// Creates an ephemeral peer connection (not persisted to config, no
/// auto-reconnect). Reuses the same connection path as auto-connect
/// peers. Returns JSON data on success or an error message.
///
/// For a peer the node is already connected to, the supplied address is
/// tried as an *alternate path* rather than ignored — the same treatment
/// [`Node::update_peers`] gives a refreshed runtime peer. The handshake
/// runs in parallel with the live link and promotion happens only once it
/// authenticates, so an address the caller got wrong cannot displace a
/// healthy path. The response's `refreshed` field reports whether such a
/// handshake was started; it is `false` when the peer is already on this
/// exact path and that path is fresh.
pub(crate) async fn api_connect(
&mut self,
npub: &str,
@@ -3182,13 +3191,43 @@ impl Node {
via_nostr: false,
};
// Pre-seed identity cache (same as initiate_peer_connections does)
if let Ok(identity) = PeerIdentity::from_npub(npub) {
// Pre-seed identity cache (same as initiate_peer_connections does).
// An unparseable npub is left to `initiate_peer_connection` below,
// which reports it as `InvalidPeerNpub`.
let peer_identity = PeerIdentity::from_npub(npub).ok();
if let Some(identity) = peer_identity.as_ref() {
self.peer_aliases
.insert(*identity.node_addr(), identity.short_npub());
self.register_identity(*identity.node_addr(), identity.pubkey_full());
}
// A peer we already hold a session to must not fall through to
// `initiate_peer_connection`: that returns Ok(()) the moment the peer
// is in `self.peers`, so the command would report success without ever
// trying the address it was handed. Route it through the same
// alternate-path helper `update_peers` uses instead.
if let Some(identity) = peer_identity
&& self.peers.contains_key(identity.node_addr())
{
let refreshed = self
.try_active_peer_alternative_addresses(&peer_config, identity)
.await
.map_err(|e| e.to_string())?;
info!(
npub = %npub,
address = %address,
transport = %transport,
refreshed = refreshed,
"API connect resolved against an already-connected peer"
);
return Ok(serde_json::json!({
"npub": npub,
"address": address,
"transport": transport,
"refreshed": refreshed,
}));
}
self.initiate_peer_connection(&peer_config)
.await
.map(|()| {
@@ -3202,6 +3241,7 @@ impl Node {
"npub": npub,
"address": address,
"transport": transport,
"refreshed": false,
})
})
.map_err(|e| e.to_string())
@@ -3209,15 +3249,27 @@ impl Node {
/// Disconnect a peer via the control API.
///
/// Notifies the peer, removes it locally, and suppresses auto-reconnect.
/// Notifies the peer, removes it locally, closes the transport connection
/// it was using, and suppresses auto-reconnect.
pub(crate) async fn api_disconnect(&mut self, npub: &str) -> Result<serde_json::Value, String> {
let peer_identity =
PeerIdentity::from_npub(npub).map_err(|e| format!("invalid npub '{npub}': {e}"))?;
let node_addr = *peer_identity.node_addr();
if !self.peers.contains_key(&node_addr) {
let Some(peer) = self.peers.get(&node_addr) else {
return Err(format!("peer not found: {npub}"));
}
};
// Read the transport path the peer is actually sending over BEFORE the
// teardown below drops the peer and its link — afterwards there is
// nothing left to derive it from. `current_addr` rather than the
// link's remote address, because roaming updates the former and it is
// the address the pool entry (and its inbound-slot accounting) is
// keyed by.
let transport_path = match (peer.transport_id(), peer.current_addr()) {
(Some(transport_id), Some(addr)) => Some((transport_id, addr.clone())),
_ => None,
};
// Notify the peer before we tear down the link, so it drops its own
// session and re-handshakes symmetrically rather than holding a stale
@@ -3230,6 +3282,22 @@ impl Node {
// Remove the peer (full cleanup: sessions, indices, links, tree, bloom)
self.remove_active_peer(&node_addr);
// Tear down the transport connection, not just the node-side state.
// `remove_active_peer` frees every node-side structure but never
// touches the transport, so on a connection-oriented transport the
// pool entry, the socket and its inbound-slot accounting would
// otherwise survive the peer the node has just forgotten — an operator
// who disconnects a peer to free a slot would not free the slot. This
// mirrors `cleanup_stale_connection`, and the reasoning there applies
// verbatim: closing twice is harmless, because every
// `close_connection` implementation is `if let Some(conn) =
// pool.remove(addr)` and the connectionless default is a no-op.
if let Some((transport_id, addr)) = transport_path
&& let Some(transport) = self.transports.get(&transport_id)
{
transport.close_connection(&addr).await;
}
// Suppress any pending auto-reconnect
self.peering.reconciler.retry_pending.remove(&node_addr);
+313
View File
@@ -0,0 +1,313 @@
//! Control API (`connect` / `disconnect`) behaviour tests.
//!
//! These drive `Node::api_connect` and `Node::api_disconnect` directly —
//! the same entry points the control socket's mutating commands dispatch to
//! (`src/control/commands.rs`) — so the assertions are about node state, not
//! socket framing.
use super::*;
use spanning_tree::{
TestNode, add_loopback_alias, cleanup_nodes, drain_all_packets, make_test_node,
process_available_packets, run_tree_test,
};
/// Count the in-flight handshake legs a node is running toward `peer`.
fn outbound_leg_count(node: &Node, peer: &NodeAddr) -> usize {
node.peer_machines
.values()
.filter(|machine| {
machine.leg().is_some()
&& machine
.conn_expected_identity()
.map(|id| id.node_addr() == peer)
.unwrap_or(false)
})
.count()
}
/// The loopback address string the control API would be handed for a node.
fn loopback_address(node: &TestNode) -> String {
node.addr.to_string()
}
/// `connect` for a peer the node does not know dials it and the handshake
/// completes: the baseline the other tests are contrasted against.
#[tokio::test]
async fn test_api_connect_dials_an_unknown_peer() {
let mut nodes = vec![make_test_node().await, make_test_node().await];
let node0_addr = *nodes[0].node.node_addr();
let node1_addr = *nodes[1].node.node_addr();
let node1_npub = nodes[1].node.npub();
let node1_address = loopback_address(&nodes[1]);
let data = nodes[0]
.node
.api_connect(&node1_npub, &node1_address, "loopback")
.await
.expect("api_connect should dial an unknown peer");
assert_eq!(
data["refreshed"], false,
"a first dial is not an alternate-path refresh"
);
let total = drain_all_packets(&mut nodes, false).await;
assert!(total > 0, "the dial should have produced packets");
assert!(
nodes[0].node.get_peer(&node1_addr).is_some(),
"node 0 should have node 1 as a peer after api_connect"
);
assert!(
nodes[1].node.get_peer(&node0_addr).is_some(),
"node 1 should have node 0 as a peer after api_connect"
);
cleanup_nodes(&mut nodes).await;
}
/// A second `connect` while the first handshake is still in flight must not
/// start a second leg — the `is_connecting_to_peer` guard.
#[tokio::test]
async fn test_api_connect_duplicate_while_connecting_starts_one_leg() {
let mut nodes = vec![make_test_node().await, make_test_node().await];
let node1_addr = *nodes[1].node.node_addr();
let node1_npub = nodes[1].node.npub();
let node1_address = loopback_address(&nodes[1]);
nodes[0]
.node
.api_connect(&node1_npub, &node1_address, "loopback")
.await
.expect("first api_connect should succeed");
assert_eq!(
outbound_leg_count(&nodes[0].node, &node1_addr),
1,
"the first connect should start exactly one handshake leg"
);
// Deliberately do not pump packets: the peer is still mid-handshake.
nodes[0]
.node
.api_connect(&node1_npub, &node1_address, "loopback")
.await
.expect("second api_connect should succeed");
assert_eq!(
outbound_leg_count(&nodes[0].node, &node1_addr),
1,
"a duplicate connect must not start a second handshake leg"
);
cleanup_nodes(&mut nodes).await;
}
/// `connect` naming the path an active peer is already on, while that path is
/// fresh, is a successful no-op — and says so.
///
/// This is the regression guard for the alternate-path fix: a caller that
/// re-announces the same peer at the same address every discovery cycle must
/// not churn a healthy link.
#[tokio::test]
async fn test_api_connect_on_current_fresh_path_is_a_no_op() {
let mut nodes = run_tree_test(2, &[(0, 1)], false).await;
let node1_addr = *nodes[1].node.node_addr();
let node1_npub = nodes[1].node.npub();
let node1_address = loopback_address(&nodes[1]);
let link_before = nodes[0]
.node
.get_peer(&node1_addr)
.expect("node 0 should have node 1")
.link_id();
let legs_before = outbound_leg_count(&nodes[0].node, &node1_addr);
let data = nodes[0]
.node
.api_connect(&node1_npub, &node1_address, "loopback")
.await
.expect("api_connect on the current path should succeed");
assert_eq!(
data["refreshed"], false,
"re-announcing the current fresh path is a no-op"
);
assert_eq!(
outbound_leg_count(&nodes[0].node, &node1_addr),
legs_before,
"no new handshake leg for the path the peer is already on"
);
let peer = nodes[0]
.node
.get_peer(&node1_addr)
.expect("the live peer must survive a no-op connect");
assert_eq!(peer.link_id(), link_before, "the live link must not change");
cleanup_nodes(&mut nodes).await;
}
/// `connect` naming a *different* address for a peer the node is already
/// connected to starts an alternate-path handshake instead of silently doing
/// nothing — the fix.
///
/// The existing peer stays put while that handshake runs: promotion is the
/// handshake's job, not the command's.
#[tokio::test]
async fn test_api_connect_starts_alternate_path_for_active_peer() {
let mut nodes = run_tree_test(2, &[(0, 1)], false).await;
let node1_addr = *nodes[1].node.node_addr();
let node1_npub = nodes[1].node.npub();
let transport_id = nodes[0].transport_id;
// A second address that reaches node 1, standing in for a second path
// coming up.
let alternate = add_loopback_alias(&nodes[1].addr);
assert_ne!(alternate, nodes[1].addr);
let link_before = nodes[0]
.node
.get_peer(&node1_addr)
.expect("node 0 should have node 1")
.link_id();
let data = nodes[0]
.node
.api_connect(&node1_npub, &alternate.to_string(), "loopback")
.await
.expect("api_connect on an alternate path should succeed");
assert_eq!(
data["refreshed"], true,
"a new path for an active peer must start a refresh"
);
assert!(
nodes[0]
.node
.is_connecting_to_peer_on_path(&node1_addr, transport_id, &alternate),
"an outbound leg should exist on the alternate path"
);
let peer = nodes[0]
.node
.get_peer(&node1_addr)
.expect("the existing peer must survive the parallel handshake");
assert_eq!(
peer.link_id(),
link_before,
"the alternate handshake must not tear the live link down before it authenticates"
);
// Let the alternate handshake run to completion; the peer must still be
// there afterwards.
for _ in 0..20 {
if process_available_packets(&mut nodes).await == 0 {
break;
}
}
assert!(
nodes[0].node.get_peer(&node1_addr).is_some(),
"node 1 should still be a peer after the alternate path resolves"
);
cleanup_nodes(&mut nodes).await;
}
/// An unparseable npub is rejected and changes nothing.
#[tokio::test]
async fn test_api_connect_rejects_invalid_npub() {
let mut node = make_node();
let err = node
.api_connect("notanpub", "loopback:0", "loopback")
.await
.expect_err("an invalid npub must be rejected");
assert!(
err.contains("notanpub"),
"the error should name the bad npub, got: {err}"
);
assert_eq!(node.peer_count(), 0);
assert!(node.peer_machines.is_empty());
}
/// `connect` naming a transport the node does not have fails cleanly rather
/// than half-registering a peer. This is also the pre-start case: a node with
/// no transports yet cannot dial anything.
#[tokio::test]
async fn test_api_connect_without_a_matching_transport_fails_cleanly() {
let mut node = make_node();
let peer = make_node();
let peer_npub = peer.npub();
let peer_addr = *peer.node_addr();
let err = node
.api_connect(&peer_npub, "127.0.0.1:1", "tor")
.await
.expect_err("no tor transport is configured");
assert!(
err.contains("no operational transport"),
"unexpected error: {err}"
);
assert_eq!(node.peer_count(), 0, "no peer may be registered");
assert!(
node.peer_machines.is_empty(),
"no handshake leg may be left behind"
);
assert_eq!(outbound_leg_count(&node, &peer_addr), 0);
}
/// `disconnect` on a connectionless transport removes the peer and the
/// transport close degrades to the no-op trait default — no error, no panic.
///
/// Disconnecting again reports `peer not found`, which is also the
/// double-close path: the first call already closed the connection.
#[tokio::test]
async fn test_api_disconnect_on_a_connectionless_transport() {
let mut nodes = run_tree_test(2, &[(0, 1)], false).await;
let node1_addr = *nodes[1].node.node_addr();
let node1_npub = nodes[1].node.npub();
nodes[0]
.node
.api_disconnect(&node1_npub)
.await
.expect("api_disconnect should succeed");
assert!(
nodes[0].node.get_peer(&node1_addr).is_none(),
"the peer must be gone"
);
let err = nodes[0]
.node
.api_disconnect(&node1_npub)
.await
.expect_err("a second disconnect has no peer to remove");
assert!(err.contains("peer not found"), "unexpected error: {err}");
cleanup_nodes(&mut nodes).await;
}
/// `disconnect` for a peer the node does not hold is rejected without any
/// partial teardown.
#[tokio::test]
async fn test_api_disconnect_unknown_peer_changes_nothing() {
let mut node = make_node();
let stranger = make_node();
let peers_before = node.peer_count();
let machines_before = node.peer_machines.len();
let links_before = node.links.len();
let err = node
.api_disconnect(&stranger.npub())
.await
.expect_err("an unknown peer cannot be disconnected");
assert!(err.contains("peer not found"), "unexpected error: {err}");
assert_eq!(node.peer_count(), peers_before);
assert_eq!(node.peer_machines.len(), machines_before);
assert_eq!(node.links.len(), links_before);
}
+1
View File
@@ -11,6 +11,7 @@ mod ble;
mod bloom;
mod bloom_poison;
mod bootstrap;
mod control;
mod decrypt_failure;
mod disconnect;
mod discovery;
+20
View File
@@ -56,6 +56,26 @@ pub(super) async fn lock_large_network_test() -> tokio::sync::MutexGuard<'static
LARGE_NETWORK_TEST_LOCK.lock().await
}
/// Register a second loopback address that delivers to the node already
/// listening on `existing`.
///
/// Gives a test node a second reachable path without a second transport:
/// packets sent to the returned address land in the same node's receive
/// channel. Used by the alternate-path tests, whose point is that the node
/// dialing it must treat the new address as a different path to the same
/// peer.
pub(super) fn add_loopback_alias(existing: &TransportAddr) -> TransportAddr {
let tx = LOOPBACK_REGISTRY
.lock()
.unwrap()
.get(existing)
.expect("existing loopback address must be registered")
.clone();
let alias = next_loopback_addr();
LOOPBACK_REGISTRY.lock().unwrap().insert(alias.clone(), tx);
alias
}
/// A test node bundling a Node with its transport and packet channel.
pub(super) struct TestNode {
pub(super) node: Node,
+64 -1
View File
@@ -8,7 +8,9 @@
use super::*;
use crate::config::{Config, TcpConfig};
use crate::transport::tcp::TcpTransport;
use crate::transport::{TransportAddr, TransportHandle, TransportId, packet_channel};
use crate::transport::{
ConnectionState, TransportAddr, TransportHandle, TransportId, packet_channel,
};
use spanning_tree::{
TestNode, cleanup_nodes, drain_all_packets, initiate_handshake, verify_tree_convergence,
};
@@ -396,3 +398,64 @@ async fn test_tcp_oriented_connect_failure_tears_down_link() {
cleanup_nodes(&mut nodes).await;
}
/// `api_disconnect` closes the pooled TCP connection, not just the peer.
///
/// Regression test: node-side teardown used to leave the socket, its pool
/// entry and its inbound-slot accounting alive after the node had forgotten
/// the peer they belonged to.
#[tokio::test]
async fn test_api_disconnect_closes_the_tcp_connection() {
let mut nodes = vec![make_test_node_tcp().await, make_test_node_tcp().await];
initiate_handshake(&mut nodes, 0, 1).await;
let total = drain_all_packets(&mut nodes, false).await;
assert!(total > 0, "should have processed packets");
let addr_1 = *nodes[1].node.node_addr();
let node1_npub = nodes[1].node.npub();
let peer = nodes[0]
.node
.get_peer(&addr_1)
.expect("node 0 should have node 1 as peer");
let peer_transport_id = peer.transport_id().expect("peer should have a transport");
let peer_addr = peer
.current_addr()
.expect("peer should have a current address")
.clone();
assert_eq!(
nodes[0]
.node
.transports
.get(&peer_transport_id)
.unwrap()
.connection_state(&peer_addr),
ConnectionState::Connected,
"the TCP connection should be pooled while the peer is up"
);
nodes[0]
.node
.api_disconnect(&node1_npub)
.await
.expect("api_disconnect should succeed");
assert!(
nodes[0].node.get_peer(&addr_1).is_none(),
"node 0 should have removed node 1"
);
assert_eq!(
nodes[0]
.node
.transports
.get(&peer_transport_id)
.unwrap()
.connection_state(&peer_addr),
ConnectionState::None,
"the pooled TCP connection must not outlive the peer"
);
cleanup_nodes(&mut nodes).await;
}