Files
ngit-grasp/tests/common/flapping_relay.rs
T
DanConwayDev bccbb54b62 fix(sync): require stable recovery before resetting backoff
Production showed nostr-pub.wellorder.net repeatedly completing a WebSocket handshake, resetting under normal REQ load 30-42 seconds later, and returning to the five-second base reconnect delay. The health tracker cleared its failure streak at handshake time even though its state model and documentation already required five minutes of stable recovery.

Preserve connection-failure history across recovered handshakes, promote a connected relay only after the existing five-minute stability period, and remove the redundant quick-reconnect success record. A relay continues normal live and historic work while degraded; only its next reconnect delay escalates. Active rate-limit cooldown accounting remains independent.

A WebSocket/REQ fixture reproduces the short-lived-success lifecycle and verifies the recovered session retains its failure count. Unit tests cover escalated backoff, stable promotion, and the rule that disconnected relays cannot age into recovery.

Correctness assumes an uninterrupted five-minute session under normal workload is sufficient evidence to forgive the streak. Intermittent sessions remain one instability episode, including for the existing 24-hour dead-relay threshold. Changing workload pace, stability duration, backoff configuration, or relay subscription policy is deliberately excluded.

Validated with cargo test --lib (650 passed), cargo test --test sync (88 passed, 1 ignored), the focused reconnect scenario, and nix build .#ngit-grasp.
2026-08-07 16:25:21 +00:00

116 lines
4.1 KiB
Rust

//! WebSocket relay fixture that accepts normal setup and then drops the session.
use std::sync::{Arc, Mutex};
use std::time::{Duration, Instant};
use futures_util::StreamExt;
use tokio::net::TcpListener;
use tokio::sync::{oneshot, Notify};
use tokio_tungstenite::tungstenite::Message;
/// Relay-shaped endpoint for exercising reconnect health accounting.
pub struct FlappingRelay {
url: String,
connected_at: Arc<Mutex<Vec<Instant>>>,
connection_observed: Arc<Notify>,
shutdown_tx: Option<oneshot::Sender<()>>,
handle: Option<tokio::task::JoinHandle<()>>,
}
impl FlappingRelay {
/// Start an endpoint that drops each WebSocket after the first REQ frame.
pub async fn start() -> Self {
let listener = TcpListener::bind("127.0.0.1:0")
.await
.expect("FlappingRelay failed to bind");
let port = listener.local_addr().expect("flapping local_addr").port();
let connected_at = Arc::new(Mutex::new(Vec::new()));
let connection_observed = Arc::new(Notify::new());
let (shutdown_tx, mut shutdown_rx) = oneshot::channel();
let observed = connected_at.clone();
let notify = connection_observed.clone();
let handle = tokio::spawn(async move {
loop {
tokio::select! {
accepted = listener.accept() => {
let Ok((stream, _)) = accepted else { break };
let observed = observed.clone();
let notify = notify.clone();
tokio::spawn(async move {
let Ok(mut websocket) = tokio_tungstenite::accept_async(stream).await
else {
// NIP-11 HTTP probes are expected to fail this
// deliberately WebSocket-only fixture.
return;
};
observed.lock().expect("flapping timestamps poisoned").push(Instant::now());
notify.notify_waiters();
while let Some(message) = websocket.next().await {
let Ok(message) = message else { break };
if matches!(message, Message::Text(ref text) if text.contains("\"REQ\"")) {
// Dropping the socket after observable normal
// setup models the production peer reset.
return;
}
}
});
}
_ = &mut shutdown_rx => break,
}
}
});
Self {
url: format!("ws://127.0.0.1:{port}"),
connected_at,
connection_observed,
shutdown_tx: Some(shutdown_tx),
handle: Some(handle),
}
}
pub fn url(&self) -> &str {
&self.url
}
/// Wait for `count` successful WebSocket sessions using notifications,
/// bounded by `timeout`.
pub async fn wait_for_connections(&self, count: usize, timeout: Duration) -> Vec<Instant> {
tokio::time::timeout(timeout, async {
loop {
let notified = self.connection_observed.notified();
let timestamps = self
.connected_at
.lock()
.expect("flapping timestamps poisoned")
.clone();
if timestamps.len() >= count {
return timestamps;
}
notified.await;
}
})
.await
.expect("recovered relay did not make the expected connection attempts")
}
pub async fn stop(mut self) {
if let Some(tx) = self.shutdown_tx.take() {
let _ = tx.send(());
}
if let Some(handle) = self.handle.take() {
let _ = handle.await;
}
}
}
impl Drop for FlappingRelay {
fn drop(&mut self) {
if let Some(tx) = self.shutdown_tx.take() {
let _ = tx.send(());
}
}
}