ngit-grasp/docs/explanation/grasp-02-proactive-sync.md

GRASP-02: Proactive Sync - Design & Implementation

Overview

This document explains the proactive sync system that synchronizes repository data from external relays based on relay URLs listed in 30617 repository announcements. Key principles:

1. Triggers call compute_actions → sync_computed_filters - Self-subscriber batches and connect/reconnect events trigger this flow
2. Clear separation of live vs historic sync - Two distinct primitives with different purposes
3. Layer 1 on connect, Layer 2+3 via AddFilters - L1 handled at connection time, L2+L3 flow through compute_actions
4. Always clear PendingSyncIndex first - Before any reconnect/consolidate operation
5. NIP-77 negentropy for historical sync - Efficient set reconciliation, fallback to REQ if unsupported

---

Data Model

RepoSyncIndex (Source of Truth)

/// What we WANT to sync - derived from events received via self-subscription.
/// Updated immediately when self-subscriber batch fires.
/// Key: repo addressable ref - 30617:pubkey:identifier
pub type RepoSyncIndex = Arc<RwLock<HashMap<String, RepoSyncNeeds>>>;

#[derive(Debug, Clone, Default)]
pub struct RepoSyncNeeds {
    /// Relay URLs listed in this repo's 30617 announcement
    pub relays: HashSet<String>,
    /// Root event IDs - 1617/1618/1619/1621 - that reference this repo
    pub root_events: HashSet<EventId>,
}

RelaySyncIndex (Confirmed State + Connection)

/// What we have CONFIRMED syncing - includes connection state for integrated lifecycle.
/// Key: relay URL
pub type RelaySyncIndex = Arc<RwLock<HashMap<String, RelayState>>>;

/// Connection status for a relay
#[derive(Debug, Clone, Copy, PartialEq, Eq)]
pub enum ConnectionStatus {
    /// Not currently connected
    Disconnected,
    /// Connection attempt in progress
    Connecting,
    /// Successfully connected and subscribed
    Connected,
}

/// Complete state for a single relay - combines sync needs with connection lifecycle
#[derive(Debug)]
pub struct RelayState {
    /// Repos we have confirmed syncing from this relay
    pub repos: HashSet<String>,
    /// Root events we have confirmed tracking
    pub root_events: HashSet<EventId>,
    /// If true, never disconnect this relay
    pub is_bootstrap: bool,
    /// Current connection status
    pub connection_status: ConnectionStatus,
    /// When we last successfully connected - used for since filter on reconnect
    pub last_connected: Option<Timestamp>,
    /// When we disconnected - for 15-minute state retention rule
    pub disconnected_at: Option<Timestamp>,
    /// Whether announcement filter historic sync has completed for this relay
    /// Used to determine if we can use `since` filter on reconnect for Layer 1
    pub announcements_synced: bool,
}

impl RelayState {
    /// Check if state should be cleared based on 15-minute rule
    pub fn should_clear_state(&self) -> bool {
        match self.disconnected_at {
            Some(disconnected) => {
                let now = Timestamp::now();
                now.as_secs().saturating_sub(disconnected.as_secs()) > 900 // 15 minutes
            }
            None => false, // Still connected or never connected
        }
    }

    /// Clear repos and root_events - called when reconnect takes > 15 minutes
    pub fn clear_sync_state(&mut self) {
        self.repos.clear();
        self.root_events.clear();
    }
}

PendingSyncIndex (In-Flight Batches)

/// Method used for synchronization
#[derive(Debug, Clone, Copy, PartialEq, Eq)]
pub enum SyncMethod {
    /// Traditional REQ+EOSE flow - waits for EOSE on subscriptions
    ReqEose,
    /// NIP-77 negentropy sync - confirms immediately after sync completes
    Negentropy,
}

/// Tracks batches of subscriptions that are in-flight, awaiting EOSE.
/// Each batch has its own ID and can confirm independently.
/// Key: relay URL
pub type PendingSyncIndex = Arc<RwLock<HashMap<String, Vec<PendingBatch>>>>;

pub struct PendingBatch {
    /// Unique ID for this batch - for debugging/logging
    pub batch_id: u64,
    /// The items this batch is syncing
    pub items: PendingItems,
    /// Subscription IDs that must ALL receive EOSE before confirming (for ReqEose)
    /// Empty for Negentropy sync method until missing event ids identified
    pub outstanding_subs: HashSet<SubscriptionId>,
    /// The sync method used for this batch
    pub sync_method: SyncMethod,
}

#[derive(Debug, Clone, Default)]
pub struct PendingItems {
    pub repos: HashSet<String>,
    pub root_events: HashSet<EventId>,
}

---

Connection Lifecycle

Object vs Connection Lifecycle

Key Principle: RelayConnection objects persist forever, WebSocket connections are transient.

• RelayConnection object: Created once via register_relay(), stored in HashMap permanently
• WebSocket connection: Transient, established via try_connect_relay(), dies on disconnect
• Event loop: Spawned by handle_connect_or_reconnect(), must be respawned after every reconnection

Connection State Machine

stateDiagram-v2
    [*] --> Disconnected: discover relay → register_relay()
    Disconnected --> Connecting: retry_disconnected_relays → try_connect_relay
    Connecting --> Connected: success → handle_connect_or_reconnect
    Connecting --> Disconnected: failure + record in health tracker
    Connected --> Disconnected: connection lost → handle_disconnect
    Connected --> [*]: intentional disconnect via check_disconnects

    note right of Disconnected: disconnected_at set for 15min rule<br/>RelayConnection kept in HashMap
    note right of Connected: last_connected tracked for since filter<br/>Event loop spawned here
    note right of Connecting: connection attempt with timeout

Connection Flow Methods

Method	Purpose	When Called	Actions
register_relay()	Initialize relay tracking	Discovery via RepoSyncIndex	Creates RelayConnection, stores in HashMap, returns immediately
try_connect_relay()	Attempt connection	Periodic retry (500ms)	Calls connect_and_subscribe, sends notification on success
handle_connect_or_reconnect()	Setup after connection	ConnectNotification received	Spawns event loop, updates state, decides sync strategy
handle_disconnect()	Cleanup after disconnect	DisconnectNotification received	Updates state, clears pending, KEEPS RelayConnection
retry_disconnected_relays()	Periodic reconnection	Every 500ms	For each ready relay: try_connect_relay()

Event Loop Lifecycle

Critical: Event loops die on disconnect and cannot be reused.

flowchart LR
    CONN[Connection Success] --> SPAWN[handle_connect_or_reconnect<br/>spawns event loop]
    SPAWN --> RUN[run_event_loop active]
    RUN --> DISC[Disconnect detected]
    DISC --> EXIT[Event loop breaks + task exits]
    EXIT --> RETRY[retry_disconnected_relays]
    RETRY --> RECONN[try_connect_relay]
    RECONN --> |success| SPAWN

Why respawn is required:

• run_event_loop() breaks on RelayStatus::Disconnected
• The spawned task completely exits
• Cannot resume terminated task - must spawn fresh
• Happens for both initial connection AND every reconnect

---

Core Architecture: Live vs Historic Sync

The sync system is built on two fundamental primitives that are clearly separated:

Sync Primitives

Primitive	Purpose	Filter Modifier	Tracking
sync_live()	Ongoing event stream	limit: 0	Not tracked
historic_sync()	Catch up on past events	Optional since	PendingSyncIndex

Why limit: 0 for Live Sync?

Approach	Pros	Cons
since: now	Intuitive	Time-sensitive, clock skew issues
limit: 0	Deterministic, mirrors filter structure	Less intuitive name

limit: 0 is better because:

1. No time dependency: Doesn't depend on synchronized clocks
2. Mirrors historic filters: Same tag structure, just different limit
3. State reconstruction: Can rebuild from repo/event lists without timestamps

Layer Strategy

Layer	Content	When Subscribed	Managed By
Layer 1	30617 Announcements, 30618 Maintainers	On connect (any type)	Connection lifecycle
Layer 2	Events tagging our repos (a/A/q tags)	Via AddFilters	compute_actions
Layer 3	Events tagging root events (e/E/q tags)	Via AddFilters	compute_actions

Key insight: Layer 1 is connection-level (handled at connect time), Layer 2+3 are item-level (flow through compute_actions → sync_computed_filters).

---

Triggers and Flow

What Triggers compute_actions → sync_computed_filters?

Trigger	When	What Happens
Self-subscriber batch fires	New events discovered on own relay	Update RepoSyncIndex → compute_actions
fresh_start()	Initial connect, long_reconnect, daily	After L1 setup → compute_actions
quick_reconnect()	Reconnect < 15 minutes	After L1+L2+L3 catchup → compute_actions
consolidate()	Filter count > threshold	After live rebuild → compute_actions

The Core Flow

flowchart TB
    TRIGGER[Trigger fires] --> CA[compute_actions]
    CA --> |derives from| RSI[RepoSyncIndex]
    CA --> |subtracts| RLI[RelaySyncIndex]
    CA --> |subtracts| PSI[PendingSyncIndex]
    CA --> |produces| AF[AddFilters actions]
    AF --> SFRE[sync_computed_filters]
    SFRE --> LIVE[sync_live - L2+L3]
    SFRE --> HIST[historic_sync - L2+L3]
    HIST --> PSI_UPDATE[Update PendingSyncIndex]
    PSI_UPDATE --> |EOSE received| CONFIRM[Move to RelaySyncIndex]

---

Flow Scenarios

Scenario 1: Fresh Start (Initial Connect / Long Reconnect / Daily Sync)

flowchart TB
    DISC[Relay discovered via RepoSyncIndex] --> REG[register_relay]
    REG --> CREATE[Create RelayConnection, store in HashMap]
    CREATE --> RET[Returns immediately]
    RET --> LOOP[retry_disconnected_relays - 500ms periodic]
    LOOP --> CHECK[health_tracker.should_attempt_connection?]
    CHECK --> |ready| TRY[try_connect_relay]
    TRY --> CONN[connection.connect_and_subscribe]
    CONN --> |success| NOTIFY[Send ConnectNotification]
    NOTIFY --> HANDLE[handle_connect_or_reconnect called]
    HANDLE --> UPD[Update state to Connected]
    UPD --> SPAWN[Spawn event loop + processor]
    SPAWN --> STRAT[Decide strategy: fresh_start]
    STRAT --> CLEAR_PSI[Clear PendingSyncIndex]
    CLEAR_PSI --> CLEAR_RSI[Clear RelaySyncIndex]
    CLEAR_RSI --> L1_LIVE[L1: sync_live - announcements]
    L1_LIVE --> L1_HIST[L1: historic_sync - no since]
    L1_HIST --> NEG{NIP-77 supported?}
    NEG --> |yes| NEGENTROPY[negentropy sync]
    NEG --> |no| REQ[REQ+EOSE]
    NEGENTROPY --> CA[compute_actions]
    REQ --> CA
    CA --> |empty RelaySyncIndex| AF[AddFilters for ALL repos]
    AF --> SFRE[sync_computed_filters]
    SFRE --> L23_LIVE[L2+L3: sync_live]
    SFRE --> L23_HIST[L2+L3: historic_sync]
    L23_HIST --> PB[Create PendingBatch]
    PB --> EOSE[Wait for EOSE]
    EOSE --> CONFIRM[Move items to RelaySyncIndex]

Key points:

• Always clear PendingSyncIndex first, then RelaySyncIndex
• L1 live + L1 historic (uses negentropy if available)
• Empty RelaySyncIndex means diff produces AddFilters for everything
• L2+L3 flow through sync_computed_filters with proper pending tracking

Scenario 2: Quick Reconnect (< 15 minutes)

flowchart TB
    DISC[Connection lost detected] --> LOOP_EXIT[Event loop breaks]
    LOOP_EXIT --> TASK_EXIT[Event processor task exits]
    TASK_EXIT --> NOTIFY_DISC[Send DisconnectNotification]
    NOTIFY_DISC --> HANDLE_DISC[handle_disconnect called]
    HANDLE_DISC --> UPD_STATE[Update state to Disconnected]
    UPD_STATE --> MARK[Set disconnected_at = now]
    MARK --> CLEAR[Clear pending batches]
    CLEAR --> KEEP[Keep RelayConnection in HashMap]
    KEEP --> WAIT[Wait < 15min]
    WAIT --> RETRY[retry_disconnected_relays - 500ms]
    RETRY --> CHECK[health_tracker checks backoff]
    CHECK --> |ready| TRY[try_connect_relay]
    TRY --> CONN[connection.connect_and_subscribe]
    CONN --> |success| NOTIFY[Send ConnectNotification]
    NOTIFY --> RECONN[handle_connect_or_reconnect]
    RECONN --> UPD_CONN[Update state to Connected]
    UPD_CONN --> SPAWN[Spawn NEW event loop + processor]
    SPAWN --> STRAT[Decide strategy: quick_reconnect]
    STRAT --> CLEAR_PSI[Clear PendingSyncIndex]
    CLEAR_PSI --> L1_LIVE[L1: sync_live - announcements]
    L1_LIVE --> L1_HIST[L1: historic_sync WITH since]
    L1_HIST --> RECON[reconstruct_filters from RelaySyncIndex]
    RECON --> L23_LIVE[L2+L3: sync_live]
    RECON --> L23_HIST[L2+L3: historic_sync WITH since]
    L23_HIST --> CA[compute_actions]
    CA --> |check for new items| AF{New items?}
    AF --> |yes| SFRE[sync_computed_filters]
    AF --> |no| DONE[Done]
    SFRE --> PB[Create PendingBatch]

Key points:

• Clear PendingSyncIndex first (old subscriptions are dead)
• L1 live (always on any connection)
• L1 historic WITH since (catches up missed announcements)
• L2+L3 rebuilt from RelaySyncIndex (confirmed state preserved)
• compute_actions checks for any NEW items discovered during catchup

Scenario 3: Long Reconnect (> 15 minutes)

flowchart TB
    RECONN[Connection restored > 15min] --> METRIC[Record disconnect/reconnect metric]
    METRIC --> FRESH[fresh_start]
    FRESH --> |same as initial connect| DONE[Full sync initiated]

Key points:

• Records disconnect/reconnect as a metric
• Delegates to fresh_start() - same as initial connect
• State too stale to trust, start fresh

Scenario 4: Consolidation (Filter Count > Threshold)

flowchart TB
    CHECK[Filter count check] --> THRESHOLD{count > 70?}
    THRESHOLD --> |yes| CLEAR_PSI[Clear PendingSyncIndex]
    CLEAR_PSI --> UNSUB[unsubscribe_all]
    UNSUB --> RECON[reconstruct_filters from RelaySyncIndex]
    RECON --> L1_LIVE[L1: sync_live]
    RECON --> L23_LIVE[L2+L3: sync_live]
    L23_LIVE --> CA[compute_actions]
    CA --> |check for new items| AF{New items?}
    AF --> |yes| SFRE[sync_computed_filters]
    AF --> |no| DONE[Done]
    THRESHOLD --> |no| SKIP[Continue normally]

Key points:

• Clear PendingSyncIndex first
• NO historic sync needed - items already synced/syncing
• Only rebuilds live subscriptions from confirmed state
• compute_actions catches any new items that need syncing

Scenario 5: Daily Sync (23-25h Random Timer)

flowchart TB
    TIMER[Daily timer fires] --> FRESH[fresh_start]
    FRESH --> |NO disconnect metric| DONE[Full sync initiated]

Key points:

• Same as fresh_start() but WITHOUT recording disconnect/reconnect metric
• Ensures consistency, detects any drift accumulated over 24 hours

Scenario 6: Self-Subscriber Batch

flowchart TB
    EVENTS[Events from own relay] --> QUEUE[Queue to pending batch]
    QUEUE --> TIMER[Batch timer fires - 5 seconds]
    TIMER --> UPDATE[Update RepoSyncIndex]
    UPDATE --> CA[compute_actions]
    CA --> |new repos/events discovered| AF[AddFilters]
    AF --> SFRE[sync_computed_filters]
    SFRE --> LIVE[sync_live - L2+L3]
    SFRE --> HIST[historic_sync - L2+L3]

Key points:

• Self-subscriber monitors own relay for 30617, 1617, 1618, 1619, 1621
• Batches events (5 second window)
• Updates RepoSyncIndex, then compute_actions finds new work
• New items flow through sync_computed_filters

---

Core Algorithms

derive_relay_targets

Transforms the repo-centric RepoSyncIndex into a relay-centric view. For each relay URL mentioned in any repo's announcements, collects all the repos and root events that should be synced from that relay.

// Conceptual: inverts repo → relays to relay → repos
fn derive_relay_targets(repo_index: &HashMap<String, RepoSyncNeeds>)
    -> HashMap<String, RelaySyncNeeds>

compute_actions (Three-Way Diff)

This is the ONLY decision point for what NEW subscriptions to create.

Performs a three-way diff: target - pending - confirmed = new

• targets: What we want (from derive_relay_targets)
• pending: What's already in-flight awaiting EOSE
• confirmed: What's already confirmed syncing

Only creates AddFilters actions for items not already pending or confirmed. Skips disconnected relays (they will get AddFilters on reconnect).

fn compute_actions(
    targets: &HashMap<String, RelaySyncNeeds>,
    pending: &PendingSyncIndex,
    confirmed: &RelaySyncIndex,
) -> Vec<AddFilters>

---

Method Specifications

Primitives

sync_live() - Live Subscriptions

/// Set up live subscription (filters with limit: 0)
///
/// - Uses `limit: 0` to receive only new events
/// - NOT tracked in PendingSyncIndex (state reconstructable)
async fn sync_live(&self, relay_url: &str, filters: &[Filter])

historic_sync() - Historical Sync Dispatcher

/// Dispatch to appropriate historic sync method based on relay capabilities
///
/// Both paths update PendingSyncIndex to ensure consistent lifecycle tracking.
async fn historic_sync(
    &mut self,
    relay_url: &str,
    filters: Vec<Filter>,
    items: PendingItems,
    since: Option<Timestamp>,
) -> Option<u64>  // Returns batch_id

Dispatches to:

• historic_sync_negentropy() - NIP-77 parallel sync (if supported)
• historic_sync_legacy() - REQ+EOSE fallback

Building Blocks

reconstruct_filters() - Rebuild from Confirmed State

/// Reconstruct filters from RelaySyncIndex (confirmed state ONLY)
///
/// Returns raw Vec<Filter> for L1+L2+L3.
/// Used by: quick_reconnect, consolidate
/// Does NOT include pending items - those flow through AddFilters path.
async fn reconstruct_filters(&self, relay_url: &str) -> Vec<Filter>

sync_computed_filters() - Handle New AddFilters

/// Process AddFilters action (from compute_actions)
///
/// Orchestrates both live and historic sync for NEW items:
/// 1. sync_live() - set up permanent L2+L3 subscriptions
/// 2. historic_sync() - catch up on past events
///
/// This is specifically for NEW filter discovery.
async fn sync_computed_filters(
    &mut self,
    action: AddFilters,
    since: Option<Timestamp>,
) -> Option<u64>

Top-Level Entry Points

fresh_start() - Clean Slate Sync

/// Fresh start - clears state and does full sync
///
/// Called by: initial connect, long_reconnect, daily_sync
///
/// Flow:
/// 1. Clear PendingSyncIndex
/// 2. Clear RelaySyncIndex
/// 3. L1 live + L1 historic (negentropy if available)
/// 4. compute_actions → AddFilters → sync_computed_filters for L2+L3
async fn fresh_start(&mut self, relay_url: &str)

quick_reconnect() - Short Disconnection Recovery

/// Quick reconnect - for disconnections < 15 minutes
///
/// Flow:
/// 1. Clear PendingSyncIndex
/// 2. L1 live + L1 historic(since)
/// 3. reconstruct_filters → L2+L3 live + L2+L3 historic(since)
/// 4. compute_actions for any new items
async fn quick_reconnect(&mut self, relay_url: &str, since: Timestamp)

long_reconnect() - Extended Disconnection Recovery

/// Long reconnect - for disconnections > 15 minutes
///
/// Flow:
/// 1. Record disconnect/reconnect metric
/// 2. fresh_start()
async fn long_reconnect(&mut self, relay_url: &str)

daily_sync() - Scheduled Full Refresh

/// Daily sync - full refresh without disconnect metrics
///
/// Flow: fresh_start() (no disconnect metric recorded)
async fn daily_sync(&mut self, relay_url: &str)

consolidate() - Filter Count Reduction

/// Consolidate subscriptions when filter count exceeds threshold
///
/// Flow:
/// 1. Clear PendingSyncIndex
/// 2. unsubscribe_all
/// 3. reconstruct_filters → sync_live only (L1+L2+L3)
/// 4. compute_actions for any new items
///
/// NO historic sync - items already synced, just reducing subscriptions
async fn consolidate(&mut self, relay_url: &str)

handle_new_sync_filters() - New Filter Discovery

/// Handle AddFilters action from compute_actions
///
/// Flow:
/// 1. Check/spawn connection if needed
/// 2. maybe_consolidate (check filter threshold)
/// 3. sync_computed_filters
async fn handle_new_sync_filters(&mut self, action: AddFilters)

---

Method Relationships Summary

fresh_start(relay_url)                          // Initial/long_reconnect/daily
    ├──> Clear PendingSyncIndex
    ├──> Clear RelaySyncIndex
    ├──> L1: sync_live(announcement_filter)
    ├──> L1: historic_sync(announcement_filter, None)
    └──> compute_actions → AddFilters → sync_computed_filters (L2+L3)

quick_reconnect(relay_url, since)               // Disconnected < 15 min
    ├──> Clear PendingSyncIndex
    ├──> L1: sync_live(announcement_filter)
    ├──> L1: historic_sync(announcement_filter, since)
    ├──> reconstruct_filters() → L2+L3 filters
    ├──> L2+L3: sync_live(filters)
    ├──> L2+L3: historic_sync(filters, since)
    └──> compute_actions → AddFilters → sync_computed_filters (new items only)

long_reconnect(relay_url)                       // Disconnected > 15 min
    ├──> Record disconnect/reconnect metric
    └──> fresh_start()

daily_sync(relay_url)                           // Timer fires
    └──> fresh_start()  // No disconnect metric

consolidate(relay_url)                          // Filter count > threshold
    ├──> Clear PendingSyncIndex
    ├──> unsubscribe_all()
    ├──> reconstruct_filters() → L1+L2+L3 filters
    ├──> sync_live(filters)  // Live only, NO historic
    └──> compute_actions → AddFilters → sync_computed_filters (new items only)

handle_new_sync_filters(action)                      // New filter discovery
    ├──> Check/spawn connection
    ├──> maybe_consolidate()
    └──> sync_computed_filters(action, None)

sync_computed_filters(action, since)            // Process AddFilters
    ├──> sync_live(action.filters)              // L2+L3 live
    └──> historic_sync(action.filters, since)   // L2+L3 historic
              ├── historic_sync_negentropy()    // Parallel, updates Pending
              └── historic_sync_legacy()        // REQ+EOSE, updates Pending

---

Filter Building (Three-Layer Strategy)

Layer 1: Announcements

• Kinds: 30617 (Repository Announcements), 30618 (Maintainer Lists)
• When subscribed: On connect (any type) - handled by connection lifecycle
• Function: build_announcement_filter(since: Option<Timestamp>)
• 30618 is ONLY synced from remote relays, not self-subscribed

Layer 2: Events Tagging Our Repos

• Tags: lowercase a, uppercase A, and q tags for comprehensive coverage
• Batching: Per 100 repo refs
• Function: build_repo_tag_filters(repos, since)

Layer 3: Events Tagging Our Root Events

• Tags: lowercase e, uppercase E, and q tags for comprehensive coverage
• Batching: Per 100 event IDs
• Function: build_root_event_tag_filters(root_events, since)

Combined Layer 2+3

The build_layer2_and_layer3_filters() function combines both layers. Used by:

• sync_computed_filters for new item subscriptions
• reconstruct_filters for rebuilding from confirmed state

---

NIP-77 Negentropy Sync

What is Negentropy?

NIP-77 defines the negentropy protocol for efficient event set comparison. Instead of requesting all events matching a filter (REQ+EOSE), negentropy allows relays to compare fingerprints of their event sets and only transfer the differences.

When Negentropy is Used

Negentropy sync is attempted for:

• fresh_start() - Full sync without since
• daily_sync() - Periodic full refresh (via fresh_start)
• long_reconnect() - Via fresh_start

Negentropy is NOT used for:

• quick_reconnect() - Uses REQ with since (more efficient for small gaps)
• Live subscriptions - Always use REQ with limit: 0

Fallback Behavior

If negentropy fails (relay doesn't support NIP-77, network error, etc.):

1. A warning is logged (once per relay to avoid spam)
2. The sync falls back to traditional REQ+EOSE
3. No error is raised - fallback is automatic

---

State Flow Summary

flowchart TB
    subgraph Input
        SS[SelfSubscriber]
        OWN[Own Relay]
    end

    subgraph RepoSyncIndex - What We Want
        RSI[HashMap: Repo → Relays+Events]
    end

    subgraph Triggers
        T1[Self-subscriber batch]
        T2[fresh_start after L1]
        T3[quick_reconnect after catchup]
        T4[consolidate after live rebuild]
    end

    subgraph compute_actions - Decision Point
        CA[Three-way diff: target - pending - confirmed]
    end

    subgraph PendingSyncIndex - In Flight
        PSI[Vec PendingBatch per relay]
    end

    subgraph RelaySyncIndex - Confirmed State
        RLI[RelayState per relay]
    end

    SS -->|subscribe| OWN
    OWN -->|events| SS
    SS -->|batch fires| RSI
    RSI --> T1
    T1 --> CA
    T2 --> CA
    T3 --> CA
    T4 --> CA
    PSI --> CA
    RLI --> CA
    CA -->|new items| AF[AddFilters]
    AF --> SFRE[sync_computed_filters]
    SFRE --> LIVE[sync_live L2+L3]
    SFRE --> HIST[historic_sync L2+L3]
    HIST --> PSI
    PSI -->|EOSE| RLI

---

Key Design Decisions

Decision	Choice	Rationale
Live vs Historic separation	Two distinct primitives	Clear responsibilities, easier reasoning about state
Live sync method	limit: 0 not since: now	No clock dependency, deterministic, mirrors filter structure
Layer 1 handling	On connect, separate from AddFilters	Connection-level concern, not item-level
Layer 2+3 handling	Via compute_actions → sync_computed_filters	Item-level, proper pending tracking
Clear PendingSyncIndex	Always first	Old subscriptions are dead, must clear before any operation
fresh_start vs long_reconnect	Same flow, different metrics	Reuse logic, distinguish intentional refresh from failure recovery
Consolidation	Live only, no historic	Items already synced, just reducing subscription count
compute_actions role	ONLY decision point for new work	Single place to reason about what needs syncing
NIP-77 negentropy	Try first on full sync, fallback	Efficient for large sets, graceful degradation

---

Module Structure

src/sync/
├── mod.rs              # SyncManager, main loop, data structures
├── algorithms.rs       # derive_relay_targets(), compute_actions()
├── filters.rs          # build_announcement_filter(), build_layer2_and_layer3_filters()
├── health.rs           # RelayHealthTracker with exponential backoff
├── relay_connection.rs # RelayConnection, RelayEvent handling
├── self_subscriber.rs  # SelfSubscriber with batching
└── metrics.rs          # SyncMetrics for Prometheus

---

Health Tracking

The RelayHealthTracker manages connection health with exponential backoff:

• States: Healthy, Degraded, Dead
• Backoff: base * 2^(failures-1), capped at max_backoff
• Dead threshold: 24 hours of continuous failures
• Dead relay retry: Once per 24 hours

Bootstrap relays are never disconnected by the cleanup system, even if empty.

---

Self-Subscriber

The SelfSubscriber monitors our own relay for repository announcements and root events, updating the RepoSyncIndex.

Event Kinds Monitored

• 30617 - Repository Announcements (triggers discovery of repos listing our relay)
• 1617 - Patches (root events referencing repos)
• 1618 - Issues
• 1619 - Replies/Status
• 1621 - Pull Requests

Note: 30618 (Maintainer Lists) is NOT self-subscribed - only synced from remote relays.

Batching Flow

1. Receive events from own relay subscription
2. Queue to pending - announcements get repo ID + relay URLs; root events get repo ref + event ID
3. Timer fires (configurable window, default 5 seconds) - does NOT reset on new events
4. Process batch:
   • Update RepoSyncIndex with discovered repos and root events
   • Call compute_actions()
   • Send AddFilters actions to SyncManager → sync_computed_filters()

---

Disconnect Handling

The disconnect checker runs periodically (default: 60 seconds) to clean up empty relays:

• Finds relays with repos.is_empty() && root_events.is_empty()
• Skips bootstrap relays (is_bootstrap == true)
• Removes from relay_sync_index, pending_sync_index, and connections
• Disconnects the WebSocket connection

Also triggers reconnection attempts for disconnected relays that have pending work.