ngit-grasp/docs/explanation/grasp-02-proactive-sync-v2.md

GRASP-02: Proactive Sync v2 - Simplified Design

Overview

This document presents a simplified redesign of the proactive sync module. The key insight is that all sync filters can be derived from two database queries, with incremental updates via self-subscription.

Goals (Same as v1)

1. Data Availability: Ensure we have all relevant events for repositories we host
2. Resilience: Handle relay failures gracefully with backoff and health tracking
3. Efficiency: Minimize connections and bandwidth through filter consolidation
4. Consistency: Use unified filters for both live sync and catchup

Scale Targets & Upper Bounds

This design targets the following scale:

Metric	Target	Notes
Repositories	1,000	Repos we host/track
Root events per repo	50 (avg)	PRs, Issues, Patches per repo
Total relays in ecosystem	100	Unique relays across all repos
Relays per repo	5 (avg)	Relays listed in each repo's announcement
Total root events	~50,000	1,000 repos × 50 events
Sync connections	~50-100	Based on relay overlap

Memory Estimate (in-memory HashMaps):

• FollowingRepoRootEvents: ~1,000 entries × 50 EventIds = ~3-5 MB
• SyncRelays: ~100 entries × varying repo counts = ~2-3 MB
• Total in-memory state: ~10 MB (well within acceptable limits)

Upper Bounds (redesign triggers):

• 10,000+ repos: Consider database-backed state instead of in-memory HashMaps
• 500+ sync relays: Consider connection pooling or relay prioritization
• 500+ root events per repo: Consider per-repo pagination in Layer 3 filters
• Sustained >100 events/second: Consider write batching to database

Beyond these limits, the in-memory HashMap model may need to evolve to a database-backed approach with lazy loading.

Core Data Structures

The entire sync filter state is captured in two HashMaps, initialized from database queries at startup:

/// Repository root events we're following
/// Key: repo addressable reference (e.g., "30617:pubkey:identifier")
/// Value: Set of event IDs (kinds 1617, 1618, 1619, 1621) that tag this repo
///
/// Note: May include a few extra repo refs that aren't in sync_relays.
/// This is acceptable - we won't query other relays for them.
type FollowingRepoRootEvents = Arc<RwLock<HashMap<String, HashSet<EventId>>>>;

/// Relays we sync from, including their repos and events
/// Key: relay URL
/// Value: Map of repo_ref -> event IDs for repos that list both this relay AND our service
///
/// Note: Bootstrap relay (if configured) is always present and excluded from removal logic.
type SyncRelays = Arc<RwLock<HashMap<String, HashMap<String, HashSet<EventId>>>>>;

Architecture Overview

flowchart TB
    subgraph Startup
        DB[(Database)]
        Q1[Query kinds 1617/1618/1619/1621]
        Q2[Query kind 30617]
        DB --> Q1 --> FRRE[following_repo_root_events]
        DB --> Q2 --> SR[sync_relays]
        BR[Bootstrap Relay] --> SR
    end

    subgraph SyncManager
        SS[Self-Subscriber]
        FRRE --> SS
        SR --> SS
    end

    subgraph SyncRelays
        R1[Relay Connection 1]
        R2[Relay Connection 2]
        RN[Relay Connection N]
    end

    SS -->|spawn/update| R1
    SS -->|spawn/update| R2
    SS -->|spawn/update| RN

    R1 -->|events| AP[Acceptance Policy]
    R2 -->|events| AP
    RN -->|events| AP
    AP -->|store| DB

Module Structure

The sync module is organized following the pattern used by src/http/mod.rs and src/metrics/mod.rs where the primary struct lives in mod.rs:

src/sync/
├── mod.rs              # SyncManager + state types (FollowingRepoRootEvents, SyncRelays)
├── self_subscriber.rs  # SelfSubscriber struct and batching logic
├── relay_connection.rs # Per-relay WebSocket connection management
├── health.rs           # RelayHealthTracker for backoff and dead relay detection
└── metrics.rs          # SyncMetrics for Prometheus integration

Rationale: The state type aliases (FollowingRepoRootEvents, SyncRelays) are simple Arc<RwLock<HashMap<...>>> wrappers owned by SyncManager. Rather than creating a separate state.rs for two type aliases, they are colocated with SyncManager in mod.rs to reduce file count while maintaining clarity.

Design Decision: No Jitter

We considered adding jitter to prevent thundering herd scenarios when:

• Multiple relay connections initialize simultaneously
• Batched updates affect multiple relays
• Filter consolidation triggers across connections

Decision: No jitter implemented.

Rationale:

• Our GRASP server should handle the load of simultaneous operations
• Jitter would lead to more orphan filters (filters added one at a time rather than atomically)
• Jitter creates inefficiency - partial subscriptions miss events during the stagger window
• The batching window (5s) already provides natural smoothing without the downsides

Health Tracking & Backoff

/// Health state machine for relay connections
enum HealthState {
    Healthy,           // Connected and working
    Backoff(u32),      // Failed, attempt count for exponential backoff
    Dead,              // 24h+ continuous failures
}

impl RelayHealthTracker {
    /// Backoff durations:
    /// - Attempt 1: 5s
    /// - Attempt 2: 10s
    /// - Attempt 3: 20s
    /// - Attempt 4: 40s
    /// - ... exponential up to 1h max
    /// - After reaching 1h, continue hourly until 24h total failure time
    /// - After 24h: marked Dead, retry once per 24h
    fn get_backoff(&self, relay_url: &str) -> Duration;
}

State	Retry Behavior
Healthy	Immediate reconnect on disconnect
Backoff	5s → 10s → 20s → ... → 1h max (exponential)
Hourly	Once per hour after hitting 1h cap
Dead	After 24h total failures, retry once per 24h

Startup Initialization

At startup, two database queries initialize the sync state:

impl SyncManager {
    async fn initialize_from_database(&mut self) -> Result<()> {
        // Initialize bootstrap relay if configured (never removed)
        if let Some(bootstrap_url) = &self.config.bootstrap_relay_url {
            self.sync_relays.write().await.insert(
                bootstrap_url.clone(),
                HashMap::new()  // Repos potentially populated below but may stay empty (Layer 1 only)
            );
        }

        // Query 1: Build following_repo_root_events
        // Find all 1617/1618/1619/1621 events and extract their repo references
        let root_events = self.database
            .query(Filter::new().kinds([
                Kind::GitPatch,      // 1617
                Kind::Custom(1618),  // PRs
                Kind::Custom(1619),  // PR updates
                Kind::Custom(1621),  // Issues
            ]))
            .await?;

        for event in root_events {
            // An event may have multiple 'a' tags pointing to different repos
            let repo_refs = self.extract_all_repo_refs(&event);
            for repo_ref in repo_refs {
                self.following_repo_root_events
                    .write().await
                    .entry(repo_ref)
                    .or_default()
                    .insert(event.id);
            }
        }

        // Query 2: Build sync_relays from kind 30617 announcements
        let announcements = self.database
            .query(Filter::new().kind(Kind::Custom(30617)))
            .await?;

        for event in announcements {
            let repo_ref = self.build_repo_ref(&event);
            let relay_urls = self.extract_relay_urls(&event);

            // Only track repos that list BOTH a remote relay AND our service
            if self.lists_our_service(&event) {
                for relay_url in relay_urls {
                    if !self.is_own_relay(&relay_url) {
                        // Get events for this repo from following_repo_root_events
                        let events = self.following_repo_root_events
                            .read().await
                            .get(&repo_ref)
                            .cloned()
                            .unwrap_or_default();

                        self.sync_relays
                            .write().await
                            .entry(relay_url)
                            .or_default()
                            .insert(repo_ref.clone(), events);
                    }
                }
            }
        }

        Ok(())
    }

    /// Extract ALL repo refs from an event (it may tag multiple repos)
    fn extract_all_repo_refs(&self, event: &Event) -> Vec<String> {
        event.tags.iter()
            .filter_map(|tag| {
                let tag_vec = tag.clone().to_vec();
                if tag_vec.len() >= 2 && tag_vec[0] == "a" {
                    // Validate it's a 30617 reference
                    if tag_vec[1].starts_with("30617:") {
                        Some(tag_vec[1].clone())
                    } else {
                        None
                    }
                } else {
                    None
                }
            })
            .collect()
    }
}

Self-Subscriber: Event-Driven Updates

A single self-subscriber watches for new events from our own relay and updates the HashMaps.

Important: The self-subscriber does NOT subscribe to kind 30618 as this would never lead to refreshing the sync filters. Those events are synced from remote relays only (via Layer 1 filter on sync relay connections).

Batching Strategy

The batch timer starts only when the first event arrives, not on a fixed interval. This prevents the scenario where an event arriving at second 4 of a 5-second interval only gets 1 second before the batch fires.

Important: Once the batch timer starts, it does NOT reset when additional events arrive. The batch will fire exactly 5 seconds after the first event, regardless of how many subsequent events are queued. This ensures predictable latency and prevents indefinite batching during high-activity periods.

impl SelfSubscriber {
    async fn run(&self) {
        // Subscribe to our own relay for relevant kinds
        // Note: 30618 NOT included - synced from remote relays only
        let filter = Filter::new()
            .kinds([
                Kind::Custom(30617),  // Repository announcements
                Kind::GitPatch,       // 1617 Patches
                Kind::Custom(1618),   // PRs
                Kind::Custom(1619),   // PR updates
                Kind::Custom(1621),   // Issues
            ]);

        let mut pending_updates: Vec<PendingUpdate> = Vec::new();
        let mut batch_deadline: Option<Instant> = None;

        loop {
            let timeout = batch_deadline
                .map(|d| d.saturating_duration_since(Instant::now()))
                .unwrap_or(Duration::MAX);

            tokio::select! {
                Some(event) = self.event_receiver.recv() => {
                    pending_updates.push(self.classify_update(&event));

                    // Start batch timer on first event
                    if batch_deadline.is_none() {
                        batch_deadline = Some(Instant::now() + Duration::from_secs(5));
                    }
                }
                _ = tokio::time::sleep(timeout), if batch_deadline.is_some() => {
                    // Batch window elapsed - apply all pending updates
                    self.apply_batched_updates(pending_updates.drain(..).collect()).await;
                    batch_deadline = None;
                }
            }
        }
    }

    fn classify_update(&self, event: &Event) -> PendingUpdate {
        match event.kind.as_u16() {
            30617 => PendingUpdate::NewAnnouncement(event.clone()),
            1617 | 1618 | 1619 | 1621 => PendingUpdate::NewRootEvent(event.clone()),
            _ => PendingUpdate::None,
        }
    }
}

Applying Batched Updates

When the batch window closes, we process all pending updates together:

/// Batched updates grouped by relay
struct RelayUpdateBatch {
    /// New repo refs to subscribe to (Layer 2)
    new_repo_refs: HashSet<String>,
    /// New event IDs to subscribe to (Layer 3)
    new_event_ids: HashSet<EventId>,
    /// Whether this is a newly discovered relay
    is_new_relay: bool,
}

impl SelfSubscriber {
    async fn apply_batched_updates(&mut self, updates: Vec<PendingUpdate>) {
        // Step 1: Process all updates and update HashMaps
        // Build batched actions per relay
        let mut relay_batches: HashMap<String, RelayUpdateBatch> = HashMap::new();

        for update in updates {
            match update {
                PendingUpdate::NewAnnouncement(event) => {
                    self.process_announcement(&event, &mut relay_batches).await;
                }
                PendingUpdate::NewRootEvent(event) => {
                    self.process_root_event(&event, &mut relay_batches).await;
                }
                PendingUpdate::None => {}
            }
        }

        // Step 2: Apply batched updates to each relay
        for (relay_url, batch) in relay_batches {
            self.apply_batch_to_relay(&relay_url, batch).await;
        }

        // Step 3: Check for relay removal (repos removed from announcements)
        self.check_relay_removal().await;
    }

    async fn apply_batch_to_relay(&mut self, relay_url: &str, batch: RelayUpdateBatch) {
        if batch.is_new_relay {
            // Spawn new relay connection with full filters
            self.spawn_sync_relay(relay_url.to_string()).await;
            return;
        }

        // Build incremental filters for new content (NO since - get historical)
        let incremental_filters = self.build_incremental_filters(&batch);

        if incremental_filters.is_empty() {
            return;
        }

        // Check if we need to consolidate
        let current_filter_count = self.get_filter_count_for_relay(relay_url).await;
        let new_filter_count = current_filter_count + incremental_filters.len();

        // Note: 70 is a conservative threshold that may need tuning based on
        // production observations. It was chosen to trigger consolidation earlier
        // than v1's 150, but optimal value depends on relay behavior.
        if new_filter_count > 70 {
            // Consolidate: add incremental filters first (no since), wait for EOSE,
            // then close all and resubscribe with consolidated filters (with since)
            self.consolidate_relay_subscription(relay_url, incremental_filters).await;
        } else {
            // Just add incremental filters (no since - to get historical events)
            self.send_filters_to_relay(relay_url, incremental_filters).await;
        }
    }

    fn build_incremental_filters(&self, batch: &RelayUpdateBatch) -> Vec<Filter> {
        let mut filters = Vec::new();

        // Layer 2: New repo refs (for ALL kinds that tag repos with 'a' tags)
        if !batch.new_repo_refs.is_empty() {
            let refs: Vec<String> = batch.new_repo_refs.iter().cloned().collect();
            for chunk in refs.chunks(100) {
                // All kinds with lowercase 'a' tag
                filters.push(
                    Filter::new()
                        .custom_tag(SingleLetterTag::lowercase(Alphabet::A), chunk.to_vec())
                );
                // All kinds with uppercase 'A' tag
                filters.push(
                    Filter::new()
                        .custom_tag(SingleLetterTag::uppercase(Alphabet::A), chunk.to_vec())
                );
                // All kinds with 'q' tag (quote)
                filters.push(
                    Filter::new()
                        .custom_tag(SingleLetterTag::lowercase(Alphabet::Q), chunk.to_vec())
                );
            }
        }

        // Layer 3: New event IDs
        if !batch.new_event_ids.is_empty() {
            let ids: Vec<String> = batch.new_event_ids.iter()
                .map(|id| id.to_hex())
                .collect();
            for chunk in ids.chunks(100) {
                filters.push(
                    Filter::new()
                        .custom_tag(SingleLetterTag::lowercase(Alphabet::E), chunk.to_vec())
                );
                filters.push(
                    Filter::new()
                        .custom_tag(SingleLetterTag::uppercase(Alphabet::E), chunk.to_vec())
                );
                filters.push(
                    Filter::new()
                        .custom_tag(SingleLetterTag::lowercase(Alphabet::Q), chunk.to_vec())
                );
            }
        }

        filters
    }
}

Consolidation Strategy

When consolidating, we need a two-phase approach:

1. First, subscribe with incremental filters (no since) to get any historical events we missed
2. After receiving EOSE, close all subscriptions and resubscribe with consolidated filters (with since)

async fn consolidate_relay_subscription(
    &mut self,
    relay_url: &str,
    incremental_filters: Vec<Filter>,
) {
    // Phase 1: Add incremental filters WITHOUT since to catch up on new content
    // These filters are for new repo_refs / event_ids we just discovered
    let phase1_sub_id = self.send_filters_to_relay_and_wait_eose(
        relay_url,
        incremental_filters
    ).await;

    // Phase 2: After EOSE, consolidate everything
    // Close ALL existing subscriptions for this relay
    self.close_all_subscriptions(relay_url).await;

    // Build fresh consolidated filters using current HashMap state
    let consolidated_filters = self.build_three_layer_filters_for_relay(relay_url).await;

    // Resubscribe with since = now - 15 minutes
    let since = Timestamp::now() - 900;
    let filters_with_since: Vec<Filter> = consolidated_filters
        .into_iter()
        .map(|f| f.since(since))
        .collect();

    self.send_filters_to_relay(relay_url, filters_with_since).await;
}

Daily Full Catchup

To capture events that may have taken longer than 15 minutes to propagate through the nostr network, we perform a daily full catchup:

impl SyncManager {
    /// Runs approximately every 24 hours per relay connection
    async fn daily_catchup(&mut self, relay_url: &str) {
        // Close all current subscriptions for this relay
        self.close_all_subscriptions(relay_url).await;

        // Rebuild fresh filters from current HashMap state
        let filters = self.build_three_layer_filters_for_relay(relay_url).await;

        // Subscribe WITHOUT since filter to get full historical sync
        for filter in filters {
            self.subscribe_to_relay(relay_url, filter).await;
        }

        // After EOSE, switch back to live mode with since filter
        self.wait_for_eose(relay_url).await;

        // Re-add since filter for ongoing live sync
        let since = Timestamp::now() - 900; // 15 minutes ago
        self.resubscribe_with_since(relay_url, since).await;
    }
}

Rationale: The 15-minute reconnection window is standard for nostr event propagation, but some events may take longer. Rather than increasing the window (which would cause more duplicate processing), we do a daily full catchup to ensure nothing is missed. This adds minimal complexity while providing comprehensive coverage.

Sync Relay Connections

Each sync relay connection uses the three-layer filter strategy:

impl SyncRelayConnection {
    async fn start(&mut self) {
        loop {
            match self.connect_and_subscribe().await {
                Ok(()) => {
                    // Record successful connection
                    self.last_successful_connection = Instant::now();
                    self.health_tracker.record_success(&self.url);

                    // Run event loop until disconnect
                    self.run_event_loop().await;
                }
                Err(e) => {
                    self.health_tracker.record_failure(&self.url);
                }
            }

            // Reconnect with backoff and since filter
            let backoff = self.health_tracker.get_backoff(&self.url);
            tokio::time::sleep(backoff).await;

            // On reconnect, use since = last_successful - 15 minutes
            self.reconnect_since = Some(
                Timestamp::from(self.last_successful_connection - Duration::from_secs(900))
            );
        }
    }

    async fn connect_and_subscribe(&mut self) -> Result<()> {
        self.client.connect().await?;

        let filters = self.build_three_layer_filters().await;

        // Apply since filter if reconnecting
        let filters = if let Some(since) = self.reconnect_since {
            filters.into_iter().map(|f| f.since(since)).collect()
        } else {
            filters
        };

        for filter in filters {
            self.client.subscribe(filter, None).await?;
        }

        Ok(())
    }
}

Three-Layer Filter Strategy

impl SyncRelayConnection {
    async fn build_three_layer_filters(&self) -> Vec<Filter> {
        let mut filters = Vec::new();

        // Get repos for this relay
        let repos = self.sync_relays.read().await
            .get(&self.url)
            .cloned()
            .unwrap_or_default();

        // Layer 1: Announcements (kinds 30617 + 30618)
        // Note: 30618 is ONLY synced from remote relays, not self-subscribed
        // Always included even if relay has no repos (bootstrap relay case)
        filters.push(
            Filter::new().kinds([Kind::Custom(30617), Kind::Custom(30618)])
        );

        // Layer 2: Events tagging repos with 'a' tags (ALL kinds)
        // Batched per 100 repo refs
        let repo_refs: Vec<String> = repos.keys().cloned().collect();
        for chunk in repo_refs.chunks(100) {
            filters.push(
                Filter::new()
                    .custom_tag(SingleLetterTag::lowercase(Alphabet::A), chunk.to_vec())
            );
            filters.push(
                Filter::new()
                    .custom_tag(SingleLetterTag::uppercase(Alphabet::A), chunk.to_vec())
            );
            filters.push(
                Filter::new()
                    .custom_tag(SingleLetterTag::lowercase(Alphabet::Q), chunk.to_vec())
            );
        }

        // Layer 3: Events tagging root events (batch per 100 event IDs)
        let all_event_ids: HashSet<EventId> = repos.values()
            .flat_map(|ids| ids.iter().cloned())
            .collect();

        let event_id_strs: Vec<String> = all_event_ids
            .iter()
            .map(|id| id.to_hex())
            .collect();

        for chunk in event_id_strs.chunks(100) {
            filters.push(
                Filter::new()
                    .custom_tag(SingleLetterTag::lowercase(Alphabet::E), chunk.to_vec())
            );
            filters.push(
                Filter::new()
                    .custom_tag(SingleLetterTag::uppercase(Alphabet::E), chunk.to_vec())
            );
            filters.push(
                Filter::new()
                    .custom_tag(SingleLetterTag::lowercase(Alphabet::Q), chunk.to_vec())
            );
        }

        filters
    }
}

Relay Removal

async fn check_relay_removal(&mut self) {
    let relays_to_check: Vec<String> = self.sync_relays.read().await
        .keys()
        .cloned()
        .collect();

    for relay_url in relays_to_check {
        // Never remove bootstrap relay
        if Some(relay_url.as_str()) == self.config.bootstrap_relay_url.as_deref() {
            continue;
        }

        // Check if relay has any repos left
        let should_remove = {
            let sync_relays = self.sync_relays.read().await;
            sync_relays.get(&relay_url)
                .map(|repos| repos.is_empty())
                .unwrap_or(true)
        };

        if should_remove {
            // Remove from HashMap
            self.sync_relays.write().await.remove(&relay_url);

            // Close connection
            self.close_relay_connection(&relay_url).await;
        }
    }
}

Prometheus Metrics (Same as v1)

Metric	Type	Labels	Description
ngit_sync_relay_connected	Gauge	relay	Connection status 1/0
ngit_sync_connection_attempts_total	Counter	relay, result	Attempts by outcome
ngit_sync_relay_status	Gauge	relay	Health state 1/2/3
ngit_sync_relay_failures	Gauge	relay	Consecutive failures
ngit_sync_events_total	Counter	source	Events by source type
ngit_sync_gap_events_total	Counter	relay	Gap events filled
ngit_sync_relays_tracked_total	Gauge	-	Total relays discovered
ngit_sync_relays_connected_total	Gauge	-	Currently connected
ngit_sync_relays_dead_total	Gauge	-	Dead relay count

Module Structure (Simplified)

src/sync/
├── mod.rs              # SyncManager + state types (FollowingRepoRootEvents, SyncRelays)
├── self_subscriber.rs  # SelfSubscriber + batching logic
├── relay_connection.rs # Per-relay WebSocket + filters
├── health.rs           # RelayHealthTracker (reuse from v1)
└── metrics.rs          # SyncMetrics (reuse from v1)

Note: SyncManager and state type aliases are colocated in mod.rs following the pattern of src/http/mod.rs (HttpService) and src/metrics/mod.rs (Metrics). See the earlier "Module Structure" section for rationale.

Comparison: v1 vs v2

Aspect	v1 (Current)	v2 (Simplified)
State Model	Spread across FilterService, SubscriptionManager, ConnectionState	Two HashMaps derived from DB
Relay Discovery	Multiple paths: bootstrap, DB query, self-subscribe, remote events	Single path: DB init + self-subscribe
Filter Updates	Dynamic per-event subscription adds	Batched updates (5s window, starts on event)
Consolidation	Per-connection at 150 filters	Per-connection at 70 filters
Batching	Per 100 tags	Per 100 tags
Reconnection	Various backoff strategies	Unified: since = last_success - 15min
Jitter	Startup jitter	None (see design decision)
30618 Handling	Synced everywhere	Remote relays only, not self-subscribed
1621 (Issues)	Not included	Included with 1617/1618/1619
Layer 2 Scope	Specific NIP-34 kinds	ALL kinds with 'a' tags
Health Backoff	Variable	5s → exp → 1h max → hourly → dead@24h → daily

Key Design Decisions

1. Single Source of Truth: Two HashMaps represent all sync state, initialized from database
2. Event-Driven Updates: Self-subscriber updates HashMaps; relay connections read from them
3. Batched Filter Updates: 5-second window that starts on first event (timer does NOT reset on subsequent events)
4. Uniform Reconnection: Always use since = last_successful - 15min
5. No Jitter: Trade-offs not worth it - orphan filters and inefficiency outweigh thundering herd concerns
6. Bootstrap Relay Protected: Never removed from sync_relays - ensures at least one sync connection exists even when no repositories currently list our service (cold start / recovery scenario)
7. 30618 Remote-Only: Maintainer state synced from remote relays, not self-subscribed
8. 70 Filter Consolidation Threshold: Lower than v1's 150 for earlier consolidation (conservative value that may need tuning based on production observation)
9. 100-Tag Batching: Consistent batch size for Layer 2 and Layer 3 filters
10. Layer 2 All Kinds: Subscribe to ALL events with 'a' tags, not just NIP-34 kinds
11. Two-Phase Consolidation: Incremental filters WITHOUT since first, then consolidated WITH since
12. Multiple Repo Refs: Handle events that tag multiple repos correctly
13. Daily Full Catchup: Periodic sync restart without since filter (~24h) to catch slow-propagating events
14. Dual DB Queries at Startup: Separate queries for root events and announcements. Could be combined into a single query, but some relays cache by kind which may make separate queries more efficient. Trade-off deferred for future optimization.

---

Detailed Flow Diagram

sequenceDiagram
    participant DB as Database
    participant SM as SyncManager
    participant SS as Self-Subscriber
    participant RC as RelayConnection

    Note over SM: Startup
    SM->>SM: Add bootstrap relay to sync_relays
    SM->>DB: Query kinds 1617/1618/1619/1621
    DB-->>SM: Root events
    SM->>SM: Build following_repo_root_events
    SM->>SM: Handle multi-repo events

    SM->>DB: Query kind 30617
    DB-->>SM: Announcements
    SM->>SM: Build sync_relays

    SM->>RC: Spawn connections for each relay
    RC->>RC: Build 3-layer filters from sync_relays
    RC->>RC: Connect and subscribe

    Note over SS: Event-Driven Updates
    SS->>SS: Subscribe to 30617/1617/1618/1619/1621
    SS->>SS: Receive event - start 5s batch timer
    SS->>SS: Collect more events in batch window
    SS->>SS: Batch window closes
    SS->>SM: Apply batched updates to HashMaps

    alt New Relay Discovered
        SM->>RC: Spawn new connection
    else New Content for Existing Relay
        alt Under 70 filter limit
            SM->>RC: Add incremental filter - no since
        else Over 70 filter limit
            SM->>RC: Add incremental filters - no since
            RC-->>SM: EOSE received
            SM->>RC: Close all subscriptions
            SM->>RC: Resubscribe consolidated - with since
        end
    else Relay Has No More Repos
        alt Is Bootstrap Relay
            SM->>SM: Keep connection - Layer 1 only
        else Not Bootstrap Relay
            SM->>RC: Close connection
        end
    end

    Note over RC: Connection Lifecycle
    RC->>RC: Process incoming events
    RC->>DB: Store via acceptance policy

    RC->>RC: Connection drops
    RC->>RC: Wait backoff - 5s to 1h exponential
    RC->>RC: Reconnect with since = last_success - 15min

    Note over RC: If failures continue 24h
    RC->>RC: Mark dead - retry once per 24h