mirror of
https://github.com/jmcorgan/fips.git
synced 2026-07-30 19:46:15 +00:00
node: don't drive connect-on-send from the rx_loop tick path
The tick body's per-peer check_* loops (heartbeats, bloom announces, MMP reports, tree announces) called transport.send for every active peer, which on TCP/Tor fell through to a 5 s connect-on-send wait for any peer whose pool entry was not yet established. That wedged the entire tick body for the full connect_timeout_ms per unreachable peer; under post-restart convergence on a high-peer mesh, this cascaded into multi- second tick stalls. On master, the same mechanism also starved the per-tick control-snapshot republish and pushed fipsctl queries onto an mpsc fallback that was itself queued behind the wedged rx_loop, producing the 5-second fipsctl head-of-line pattern operators observed on loaded nodes. Gate send_encrypted_link_message_with_ce on transport.connection_state before the send: proceed only when Connected; on None, kick off a non-blocking background connect (idempotent — TransportHandle::connect dedupes against the connecting pool and spawns the timeout-bounded TcpStream::connect inside its own tokio task) and fail this send fast with a clear "transport connection not ready" error. A subsequent tick retries once the pool has an entry. The reconnect lifecycle (check_link_heartbeats, process_pending_retries, poll_pending_connects) is unchanged. The connect-on-send branch in transport.send_async itself remains in place for code paths that legitimately need synchronous connect (e.g., explicit operator-driven fipsctl connect).
This commit is contained in:
@@ -60,6 +60,37 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
|
|||||||
|
|
||||||
### Fixed
|
### Fixed
|
||||||
|
|
||||||
|
- `rx_loop` tick-arm stall under convergence-phase mesh pressure
|
||||||
|
is eliminated. Previously, the tick body's per-peer `check_*`
|
||||||
|
loops (heartbeats, bloom announces, MMP reports, tree announces)
|
||||||
|
called `transport.send` directly for every active peer. For
|
||||||
|
TCP/Tor peers whose pool entry was not yet established,
|
||||||
|
`send_async` fell through to a synchronous connect-on-send
|
||||||
|
branch that wrapped `TcpStream::connect` in
|
||||||
|
`tokio::time::timeout(connect_timeout_ms, …)` — 5 seconds by
|
||||||
|
default — and blocked the entire tick body for the duration per
|
||||||
|
unreachable peer. Under post-restart convergence on a high-peer
|
||||||
|
mesh, this cascaded into multi-second tick stalls; the same
|
||||||
|
mechanism also starved the master-only per-tick control-snapshot
|
||||||
|
republish and pushed `fipsctl show *` queries onto an mpsc
|
||||||
|
fallback that was itself queued behind the wedged `rx_loop`,
|
||||||
|
producing the five-second `fipsctl` head-of-line pattern
|
||||||
|
operators observed on loaded nodes. The send path now gates on
|
||||||
|
`transport.connection_state(addr)` before sending: proceed only
|
||||||
|
when `Connected`; on `None`, kick off a non-blocking background
|
||||||
|
`connect` (idempotent — deduplicates against the connecting
|
||||||
|
pool, spawns the timeout-bounded `TcpStream::connect` inside its
|
||||||
|
own tokio task) and fail this send fast with a clear
|
||||||
|
`transport connection not ready` error. A subsequent tick
|
||||||
|
retries once the pool has an entry. The existing reconnect
|
||||||
|
lifecycle (heartbeat-dead detection in `check_link_heartbeats`,
|
||||||
|
scheduled retries via `process_pending_retries`, background-
|
||||||
|
connect polling via `poll_pending_connects`) is unchanged.
|
||||||
|
The connect-on-send branch in `transport.send_async` itself
|
||||||
|
remains in place for code paths that legitimately need
|
||||||
|
synchronous connect (e.g., explicit operator-driven
|
||||||
|
`fipsctl connect`); the tick path just no longer trips it.
|
||||||
|
|
||||||
- NAT-traversal cross-init adoption is now deterministic under
|
- NAT-traversal cross-init adoption is now deterministic under
|
||||||
simultaneous dual-initiation. Previously, when two peers'
|
simultaneous dual-initiation. Previously, when two peers'
|
||||||
Nostr-mediated UDP punches completed within the same scheduling
|
Nostr-mediated UDP punches completed within the same scheduling
|
||||||
|
|||||||
+23
-1
@@ -45,7 +45,8 @@ use crate::transport::tcp::TcpTransport;
|
|||||||
use crate::transport::tor::TorTransport;
|
use crate::transport::tor::TorTransport;
|
||||||
use crate::transport::udp::UdpTransport;
|
use crate::transport::udp::UdpTransport;
|
||||||
use crate::transport::{
|
use crate::transport::{
|
||||||
Link, LinkId, PacketRx, PacketTx, TransportAddr, TransportError, TransportHandle, TransportId,
|
ConnectionState, Link, LinkId, PacketRx, PacketTx, TransportAddr, TransportError,
|
||||||
|
TransportHandle, TransportId,
|
||||||
};
|
};
|
||||||
use crate::tree::TreeState;
|
use crate::tree::TreeState;
|
||||||
use crate::upper::hosts::HostMap;
|
use crate::upper::hosts::HostMap;
|
||||||
@@ -1967,6 +1968,27 @@ impl Node {
|
|||||||
.get(&transport_id)
|
.get(&transport_id)
|
||||||
.ok_or(NodeError::TransportNotFound(transport_id))?;
|
.ok_or(NodeError::TransportNotFound(transport_id))?;
|
||||||
|
|
||||||
|
// Gate: don't drive connect-on-send from the tick path. If the
|
||||||
|
// transport's connection isn't ready, kick off a non-blocking
|
||||||
|
// background connect (no-op if already in flight or pooled) and
|
||||||
|
// fail this send fast. A subsequent tick will retry once the
|
||||||
|
// pool entry exists. The historical connect-on-send wedged the
|
||||||
|
// rx_loop tick body for up to `connect_timeout_ms` (5 s default)
|
||||||
|
// per unreachable peer, which under convergence-phase mesh
|
||||||
|
// pressure cascaded into multi-tick stalls and control-RPC HOL.
|
||||||
|
match transport.connection_state(&remote_addr) {
|
||||||
|
ConnectionState::Connected => {}
|
||||||
|
other => {
|
||||||
|
if matches!(other, ConnectionState::None) {
|
||||||
|
let _ = transport.connect(&remote_addr).await;
|
||||||
|
}
|
||||||
|
return Err(NodeError::SendFailed {
|
||||||
|
node_addr: *node_addr,
|
||||||
|
reason: format!("transport connection not ready: {:?}", other),
|
||||||
|
});
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
let bytes_sent = transport
|
let bytes_sent = transport
|
||||||
.send(&remote_addr, &wire_packet)
|
.send(&remote_addr, &wire_packet)
|
||||||
.await
|
.await
|
||||||
|
|||||||
Reference in New Issue
Block a user