node: don't drive connect-on-send from the rx_loop tick path

The tick body's per-peer check_* loops (heartbeats, bloom
announces, MMP reports, tree announces) called transport.send
for every active peer, which on TCP/Tor fell through to a 5 s
connect-on-send wait for any peer whose pool entry was not yet
established. That wedged the entire tick body for the full
connect_timeout_ms per unreachable peer; under post-restart
convergence on a high-peer mesh, this cascaded into multi-
second tick stalls. On master, the same mechanism also starved
the per-tick control-snapshot republish and pushed fipsctl
queries onto an mpsc fallback that was itself queued behind
the wedged rx_loop, producing the 5-second fipsctl head-of-line
pattern operators observed on loaded nodes.

Gate send_encrypted_link_message_with_ce on
transport.connection_state before the send: proceed only when
Connected; on None, kick off a non-blocking background connect
(idempotent — TransportHandle::connect dedupes against the
connecting pool and spawns the timeout-bounded TcpStream::connect
inside its own tokio task) and fail this send fast with a
clear "transport connection not ready" error. A subsequent
tick retries once the pool has an entry. The reconnect
lifecycle (check_link_heartbeats, process_pending_retries,
poll_pending_connects) is unchanged. The connect-on-send
branch in transport.send_async itself remains in place for
code paths that legitimately need synchronous connect (e.g.,
explicit operator-driven fipsctl connect).
This commit is contained in:
Johnathan Corgan
2026-05-25 04:50:30 +00:00
parent f396d71826
commit 4d5380604a
2 changed files with 54 additions and 1 deletions
+23 -1
View File
@@ -45,7 +45,8 @@ use crate::transport::tcp::TcpTransport;
use crate::transport::tor::TorTransport;
use crate::transport::udp::UdpTransport;
use crate::transport::{
Link, LinkId, PacketRx, PacketTx, TransportAddr, TransportError, TransportHandle, TransportId,
ConnectionState, Link, LinkId, PacketRx, PacketTx, TransportAddr, TransportError,
TransportHandle, TransportId,
};
use crate::tree::TreeState;
use crate::upper::hosts::HostMap;
@@ -1967,6 +1968,27 @@ impl Node {
.get(&transport_id)
.ok_or(NodeError::TransportNotFound(transport_id))?;
// Gate: don't drive connect-on-send from the tick path. If the
// transport's connection isn't ready, kick off a non-blocking
// background connect (no-op if already in flight or pooled) and
// fail this send fast. A subsequent tick will retry once the
// pool entry exists. The historical connect-on-send wedged the
// rx_loop tick body for up to `connect_timeout_ms` (5 s default)
// per unreachable peer, which under convergence-phase mesh
// pressure cascaded into multi-tick stalls and control-RPC HOL.
match transport.connection_state(&remote_addr) {
ConnectionState::Connected => {}
other => {
if matches!(other, ConnectionState::None) {
let _ = transport.connect(&remote_addr).await;
}
return Err(NodeError::SendFailed {
node_addr: *node_addr,
reason: format!("transport connection not ready: {:?}", other),
});
}
}
let bytes_sent = transport
.send(&remote_addr, &wire_packet)
.await