mirror of
https://relay.ngit.dev/npub15qydau2hjma6ngxkl2cyar74wzyjshvl65za5k5rl69264ar2exs5cyejr/ngit-grasp.git
synced 2026-10-05 15:08:24 +00:00
docs(tests): replace catchup.rs test with documentation
The catchup sync mechanism (reconnection with since filter) is implemented in src/sync/mod.rs handle_connect_or_reconnect(), but cannot be reliably integration tested with current infrastructure: - TestRelay uses in-memory database (events lost on stop) - No way to force WebSocket disconnection without stopping relay - Stopping syncing relay creates new instance (fresh sync, not catchup) Convert the skeleton test file to comprehensive documentation explaining: - How catchup sync works (since filter on reconnect) - The 15-minute quick reconnect window logic - Why integration testing is not feasible - Alternative approaches that could enable testing - Related tests that cover adjacent functionality
This commit is contained in:
DanConwayDev
+100
-191
@@ -1,201 +1,110 @@
|
|||||||
//! Catchup Sync Tests
|
//! Catchup Sync - Documentation Only
|
||||||
//!
|
//!
|
||||||
//! Tests for the catchup synchronization feature (Test 0).
|
//! This file documents the catchup sync mechanism. No integration tests are included
|
||||||
|
//! because the functionality cannot be reliably tested with current test infrastructure.
|
||||||
//!
|
//!
|
||||||
//! # Catchup Sync Overview
|
//! # What is Catchup Sync?
|
||||||
//!
|
//!
|
||||||
//! Catchup sync refers to the ability of a relay to synchronize historical events
|
//! Catchup sync ensures that when a relay's WebSocket connection to another relay drops
|
||||||
//! that were published while it was offline or unreachable. This is critical for
|
//! and reconnects, any events the source relay received during the disconnection are
|
||||||
//! ensuring data consistency across the relay network.
|
//! fetched using a `since` filter based on the last connection timestamp.
|
||||||
//!
|
//!
|
||||||
//! ## Expected Behavior
|
//! # Implementation Status: ✅ IMPLEMENTED
|
||||||
//!
|
//!
|
||||||
//! When a relay comes back online after being offline:
|
//! The catchup sync mechanism is fully implemented in the sync module via:
|
||||||
//! 1. Detect gap in event history by comparing timestamps
|
|
||||||
//! 2. Query connected relays for events in the gap period
|
|
||||||
//! 3. Backfill Layer 2 events (kind 1618) from bootstrap relays
|
|
||||||
//! 4. Discover and sync Layer 3 events (kinds 1, 1111) referencing Layer 2 events
|
|
||||||
//! 5. Maintain chronological ordering during backfill
|
|
||||||
//!
|
//!
|
||||||
//! ## Implementation Status
|
//! - [`handle_connect_or_reconnect()`](../../src/sync/mod.rs) - Detects reconnection and
|
||||||
|
//! applies appropriate sync strategy
|
||||||
|
//! - [`RelayState.last_connected`](../../src/sync/mod.rs) - Tracks when we last connected
|
||||||
|
//! to each relay
|
||||||
|
//! - [`filters::build_announcement_filter(since)`](../../src/sync/filters.rs) - Builds
|
||||||
|
//! Layer 1 filters with `since` timestamp
|
||||||
|
//! - [`filters::build_layer2_and_layer3_filters(since)`](../../src/sync/filters.rs) -
|
||||||
|
//! Builds Layer 2/3 filters with `since` timestamp
|
||||||
//!
|
//!
|
||||||
//! ⚠ **NOT YET IMPLEMENTED** - Tests marked with `#[ignore]`
|
//! ## Reconnection Logic
|
||||||
//!
|
//!
|
||||||
//! These tests are ready to enable once catchup sync is implemented in the relay.
|
//! When a relay reconnects to a source relay, the sync manager uses smart reconnection:
|
||||||
//!
|
//!
|
||||||
//! ## See Also
|
//! | Scenario | Behavior |
|
||||||
|
//! |----------|----------|
|
||||||
|
//! | First connection ever | Full sync (no `since` filter) |
|
||||||
|
//! | Reconnect within 15 min | Quick reconnect with `since = last_connected - 15min` |
|
||||||
|
//! | Reconnect after >15 min | Full sync (clear state, treat as fresh connection) |
|
||||||
//!
|
//!
|
||||||
//! - Bootstrap sync: [`tests/sync/bootstrap.rs`](bootstrap.rs)
|
//! The 15-minute buffer on the `since` filter accounts for clock drift and ensures
|
||||||
//! - Live sync: [`tests/sync/live_sync.rs`](live_sync.rs)
|
//! no events are missed at the boundary.
|
||||||
//! - Discovery sync: [`tests/sync/discovery.rs`](discovery.rs)
|
//!
|
||||||
|
//! # Why No Integration Tests?
|
||||||
use std::time::Duration;
|
//!
|
||||||
|
//! Testing catchup sync in integration tests is not feasible with current infrastructure:
|
||||||
use nostr_sdk::prelude::*;
|
//!
|
||||||
|
//! ## 1. Cannot Force WebSocket Disconnection
|
||||||
use crate::common::{sync_helpers::*, TestRelay};
|
//!
|
||||||
|
//! The catchup mechanism is designed for same-process reconnection scenarios, such as:
|
||||||
/// Test that relay performs catchup sync after being offline
|
//! - Network hiccup causing temporary disconnection
|
||||||
///
|
//! - Source relay temporarily unreachable
|
||||||
/// # Scenario
|
//! - WebSocket connection timeout
|
||||||
///
|
//!
|
||||||
/// 1. Start two relays (relay1, relay2) with discovery configured
|
//! Our [`TestRelay`](../common/relay.rs) fixture doesn't provide a way to force a
|
||||||
/// 2. Publish several Layer 2 events to relay2
|
//! WebSocket disconnection without stopping the relay entirely.
|
||||||
/// 3. Stop relay1 (simulating offline state)
|
//!
|
||||||
/// 4. Publish more Layer 2 events to relay2 while relay1 is offline
|
//! ## 2. Stopping a Relay Loses Events (In-Memory Database)
|
||||||
/// 5. Restart relay1
|
//!
|
||||||
/// 6. Verify relay1 catches up and syncs events it missed
|
//! `TestRelay` uses `NGIT_DATABASE_BACKEND=memory` for test isolation. If we stop
|
||||||
///
|
//! the source relay (to simulate disconnection), all events are lost. When a new
|
||||||
/// # Expected Result
|
//! instance starts, there's nothing to "catch up" on.
|
||||||
///
|
//!
|
||||||
/// All events published while relay1 was offline should be synced
|
//! ## 3. Stopping the Syncing Relay Creates a New Instance
|
||||||
/// to relay1 after it comes back online, maintaining chronological order.
|
//!
|
||||||
///
|
//! If we stop the syncing relay and start a new one:
|
||||||
/// # TODO
|
//! - `last_connected` is lost (in-memory state)
|
||||||
///
|
//! - New instance does a fresh full sync, not a `since`-filtered catchup
|
||||||
/// - Implement catchup sync mechanism in relay
|
//! - This is correct behavior, but tests the bootstrap path, not catchup
|
||||||
/// - Add timestamp-based gap detection
|
//!
|
||||||
/// - Add backfill query generation
|
//! # Alternative Testing Approaches (Not Implemented)
|
||||||
/// - Enable this test by removing `#[ignore]`
|
//!
|
||||||
#[tokio::test]
|
//! These could enable catchup testing but add significant complexity:
|
||||||
#[ignore = "Catchup sync not yet implemented"]
|
//!
|
||||||
async fn test_catchup_sync_after_relay_restart() {
|
//! 1. **Persistent database for source relay** - Use SQLite instead of in-memory,
|
||||||
// NOTE: This is a skeleton implementation ready for when catchup sync is added
|
//! allowing relay restart without data loss
|
||||||
|
//!
|
||||||
// 1. Start two relays
|
//! 2. **TestRelay restart capability** - Add `restart()` method that preserves the
|
||||||
let relay1 = TestRelay::start().await;
|
//! same port and database path
|
||||||
let relay2 = TestRelay::start().await;
|
//!
|
||||||
|
//! 3. **Network simulation** - Add ability to inject network failures between specific
|
||||||
// 2. Set up discovery between relays via shared announcement
|
//! relay pairs without stopping either relay
|
||||||
let keys = Keys::generate();
|
//!
|
||||||
let identifier = "catchup-test-repo";
|
//! 4. **Internal sync manager API** - Expose methods to force reconnection without
|
||||||
|
//! network-level disruption
|
||||||
// Create announcement listing both relays
|
//!
|
||||||
let domain1 = relay1.domain();
|
//! # Related Tests
|
||||||
let domain2 = relay2.domain();
|
//!
|
||||||
let announcement = create_repo_announcement(
|
//! While catchup sync itself isn't directly tested, related functionality is covered:
|
||||||
&keys,
|
//!
|
||||||
&[&domain1, &domain2],
|
//! - [`bootstrap.rs`](bootstrap.rs) - Tests that a new relay syncs existing events
|
||||||
identifier,
|
//! from a bootstrap relay (fresh full sync path)
|
||||||
);
|
//! - [`live_sync.rs`](live_sync.rs) - Tests real-time sync of new events after
|
||||||
|
//! connection is established
|
||||||
// Publish announcement to both relays
|
//! - [`discovery.rs`](discovery.rs) - Tests that relays discover each other via
|
||||||
let client1 = TestClient::new(relay1.url(), keys.clone())
|
//! repository announcements
|
||||||
.await
|
//!
|
||||||
.expect("Failed to connect to relay1");
|
//! # Design Rationale
|
||||||
let client2 = TestClient::new(relay2.url(), keys.clone())
|
//!
|
||||||
.await
|
//! The catchup mechanism prioritizes simplicity and correctness:
|
||||||
.expect("Failed to connect to relay2");
|
//!
|
||||||
|
//! - **Correctness over testing**: The `since` filter logic is straightforward and
|
||||||
client1
|
//! uses well-tested nostr-sdk primitives. The risk of bugs is low.
|
||||||
.send_event(&announcement)
|
//!
|
||||||
.await
|
//! - **15-minute quick reconnect window**: Balances efficiency (avoid full resync for
|
||||||
.expect("Failed to send announcement to relay1");
|
//! brief outages) with simplicity (don't track complex state for long outages).
|
||||||
client2
|
//!
|
||||||
.send_event(&announcement)
|
//! - **Full sync fallback**: After 15 minutes, the relay does a complete resync.
|
||||||
.await
|
//! This guarantees no events are missed, at the cost of redundant transfers.
|
||||||
.expect("Failed to send announcement to relay2");
|
//!
|
||||||
|
//! # See Also
|
||||||
// Wait for discovery connections to establish
|
//!
|
||||||
tokio::time::sleep(Duration::from_secs(2)).await;
|
//! - [`src/sync/mod.rs`](../../src/sync/mod.rs) - Main sync module with reconnection logic
|
||||||
|
//! - [`src/sync/filters.rs`](../../src/sync/filters.rs) - Filter builders with `since` support
|
||||||
// 3. Publish initial Layer 2 event (while both relays are online)
|
//! - [`src/sync/metrics.rs`](../../src/sync/metrics.rs) - Metrics tracking event sources
|
||||||
let repo_coord_str = repo_coord(&keys, identifier);
|
//! including `RECONNECT` for catchup events
|
||||||
let event1 = build_layer2_issue_event(&keys, &repo_coord_str, "Issue 1 - before offline")
|
|
||||||
.expect("Failed to build event1");
|
|
||||||
let event1_id = client2
|
|
||||||
.send_event(&event1)
|
|
||||||
.await
|
|
||||||
.expect("Failed to send event1");
|
|
||||||
|
|
||||||
// Verify initial sync works (baseline check)
|
|
||||||
let synced = wait_for_event_on_relay(
|
|
||||||
relay1.url(),
|
|
||||||
Filter::new().id(event1_id),
|
|
||||||
Duration::from_secs(5),
|
|
||||||
)
|
|
||||||
.await;
|
|
||||||
assert!(synced, "Initial event should sync normally via live sync");
|
|
||||||
|
|
||||||
// 4. Stop relay1 (simulating offline state)
|
|
||||||
// Note: In a real implementation, we'd need a way to stop and restart a relay
|
|
||||||
// For now, this skeleton demonstrates the intended test flow
|
|
||||||
relay1.stop().await;
|
|
||||||
|
|
||||||
// Small delay to ensure relay1 is fully stopped
|
|
||||||
tokio::time::sleep(Duration::from_millis(500)).await;
|
|
||||||
|
|
||||||
// 5. Publish events while relay1 is offline
|
|
||||||
let event2 = build_layer2_issue_event(&keys, &repo_coord_str, "Issue 2 - during offline")
|
|
||||||
.expect("Failed to build event2");
|
|
||||||
let event2_id = client2
|
|
||||||
.send_event(&event2)
|
|
||||||
.await
|
|
||||||
.expect("Failed to send event2");
|
|
||||||
|
|
||||||
let event3 = build_layer2_issue_event(&keys, &repo_coord_str, "Issue 3 - during offline")
|
|
||||||
.expect("Failed to build event3");
|
|
||||||
let event3_id = client2
|
|
||||||
.send_event(&event3)
|
|
||||||
.await
|
|
||||||
.expect("Failed to send event3");
|
|
||||||
|
|
||||||
// Give time for events to be stored in relay2
|
|
||||||
tokio::time::sleep(Duration::from_secs(1)).await;
|
|
||||||
|
|
||||||
// 6. Restart relay1
|
|
||||||
// Note: TestRelay doesn't currently support restart, so we start a new instance
|
|
||||||
// A real implementation would need persistent storage and relay restart capability
|
|
||||||
let relay1_restarted = TestRelay::start().await;
|
|
||||||
|
|
||||||
// Reconnect client to the new relay instance
|
|
||||||
let client1_restarted = TestClient::new(relay1_restarted.url(), keys.clone())
|
|
||||||
.await
|
|
||||||
.expect("Failed to connect to restarted relay1");
|
|
||||||
|
|
||||||
// Re-publish announcement to establish discovery
|
|
||||||
let domain1_restarted = relay1_restarted.domain();
|
|
||||||
let announcement_restarted = create_repo_announcement(
|
|
||||||
&keys,
|
|
||||||
&[&domain1_restarted, &domain2],
|
|
||||||
identifier,
|
|
||||||
);
|
|
||||||
client1_restarted
|
|
||||||
.send_event(&announcement_restarted)
|
|
||||||
.await
|
|
||||||
.expect("Failed to send announcement to restarted relay1");
|
|
||||||
|
|
||||||
// 7. Wait for catchup sync to complete
|
|
||||||
// This is where the catchup sync mechanism would kick in
|
|
||||||
tokio::time::sleep(Duration::from_secs(5)).await;
|
|
||||||
|
|
||||||
// 8. Verify missed events were synced via catchup
|
|
||||||
let event2_synced = wait_for_event_on_relay(
|
|
||||||
relay1_restarted.url(),
|
|
||||||
Filter::new().id(event2_id),
|
|
||||||
Duration::from_secs(5),
|
|
||||||
)
|
|
||||||
.await;
|
|
||||||
|
|
||||||
let event3_synced = wait_for_event_on_relay(
|
|
||||||
relay1_restarted.url(),
|
|
||||||
Filter::new().id(event3_id),
|
|
||||||
Duration::from_secs(5),
|
|
||||||
)
|
|
||||||
.await;
|
|
||||||
|
|
||||||
assert!(
|
|
||||||
event2_synced,
|
|
||||||
"Event 2 (missed while offline) should be synced via catchup"
|
|
||||||
);
|
|
||||||
assert!(
|
|
||||||
event3_synced,
|
|
||||||
"Event 3 (missed while offline) should be synced via catchup"
|
|
||||||
);
|
|
||||||
|
|
||||||
// 9. Cleanup
|
|
||||||
client1_restarted.disconnect().await;
|
|
||||||
client2.disconnect().await;
|
|
||||||
relay1_restarted.stop().await;
|
|
||||||
relay2.stop().await;
|
|
||||||
}
|
|
||||||
|
|||||||
Reference in New Issue
Block a user