mirror of
https://github.com/jmcorgan/fips.git
synced 2026-08-09 08:14:42 +00:00
node: don't drive connect-on-send from the rx_loop tick path
The tick body's per-peer check_* loops (heartbeats, bloom announces, MMP reports, tree announces) called transport.send for every active peer, which on TCP/Tor fell through to a 5 s connect-on-send wait for any peer whose pool entry was not yet established. That wedged the entire tick body for the full connect_timeout_ms per unreachable peer; under post-restart convergence on a high-peer mesh, this cascaded into multi- second tick stalls. On master, the same mechanism also starved the per-tick control-snapshot republish and pushed fipsctl queries onto an mpsc fallback that was itself queued behind the wedged rx_loop, producing the 5-second fipsctl head-of-line pattern operators observed on loaded nodes. Gate send_encrypted_link_message_with_ce on transport.connection_state before the send: proceed only when Connected; on None, kick off a non-blocking background connect (idempotent — TransportHandle::connect dedupes against the connecting pool and spawns the timeout-bounded TcpStream::connect inside its own tokio task) and fail this send fast with a clear "transport connection not ready" error. A subsequent tick retries once the pool has an entry. The reconnect lifecycle (check_link_heartbeats, process_pending_retries, poll_pending_connects) is unchanged. The connect-on-send branch in transport.send_async itself remains in place for code paths that legitimately need synchronous connect (e.g., explicit operator-driven fipsctl connect).
This commit is contained in:
+23
-1
@@ -45,7 +45,8 @@ use crate::transport::tcp::TcpTransport;
|
||||
use crate::transport::tor::TorTransport;
|
||||
use crate::transport::udp::UdpTransport;
|
||||
use crate::transport::{
|
||||
Link, LinkId, PacketRx, PacketTx, TransportAddr, TransportError, TransportHandle, TransportId,
|
||||
ConnectionState, Link, LinkId, PacketRx, PacketTx, TransportAddr, TransportError,
|
||||
TransportHandle, TransportId,
|
||||
};
|
||||
use crate::tree::TreeState;
|
||||
use crate::upper::hosts::HostMap;
|
||||
@@ -1967,6 +1968,27 @@ impl Node {
|
||||
.get(&transport_id)
|
||||
.ok_or(NodeError::TransportNotFound(transport_id))?;
|
||||
|
||||
// Gate: don't drive connect-on-send from the tick path. If the
|
||||
// transport's connection isn't ready, kick off a non-blocking
|
||||
// background connect (no-op if already in flight or pooled) and
|
||||
// fail this send fast. A subsequent tick will retry once the
|
||||
// pool entry exists. The historical connect-on-send wedged the
|
||||
// rx_loop tick body for up to `connect_timeout_ms` (5 s default)
|
||||
// per unreachable peer, which under convergence-phase mesh
|
||||
// pressure cascaded into multi-tick stalls and control-RPC HOL.
|
||||
match transport.connection_state(&remote_addr) {
|
||||
ConnectionState::Connected => {}
|
||||
other => {
|
||||
if matches!(other, ConnectionState::None) {
|
||||
let _ = transport.connect(&remote_addr).await;
|
||||
}
|
||||
return Err(NodeError::SendFailed {
|
||||
node_addr: *node_addr,
|
||||
reason: format!("transport connection not ready: {:?}", other),
|
||||
});
|
||||
}
|
||||
}
|
||||
|
||||
let bytes_sent = transport
|
||||
.send(&remote_addr, &wire_packet)
|
||||
.await
|
||||
|
||||
Reference in New Issue
Block a user