docs: update sync docs post implementation

2026-10-05 23:18:24 +00:00 · 2025-12-10 13:08:41 +00:00
parent 8cfe8546e5
commit 67f7ebdaef
2 changed files with 151 additions and 6 deletions
@@ -11,6 +11,7 @@
 After examining both the reference implementation and HTTP server options, we have two options:

 #### Option 1: Hook-Based (Reference Implementation Approach)
+
 - Use standard Git HTTP backend
 - Create pre-receive and post-receive hooks
 - Hooks query the Nostr relay and validate pushes
@@ -18,6 +19,7 @@ After examining both the reference implementation and HTTP server options, we ha
 - **Cons**: Requires hook management, harder to test, less Rust-native

 #### Option 2: Inline Authorization (Recommended)
+
 - Intercept Git receive-pack requests in the HTTP handler
 - Validate against Nostr state before spawning Git process
 - Only forward valid pushes to Git
@@ -30,13 +32,15 @@ After examining both the reference implementation and HTTP server options, we ha

 1. **Full control over HTTP layer**: Using Hyper directly gives us complete control over request handling, WebSocket upgrades, and CORS headers.

-2. **Better Developer Experience**: 
+2. **Better Developer Experience**:
+
   - Validation errors can be returned as proper HTTP responses
   - No need to parse hook stderr output
   - Shared state between Git and Nostr components
   - Pure Rust testing without shell scripts

 3. **Simpler Deployment**:
+
   - Single binary
   - No hook symlinks or permissions to manage
   - No multi-process coordination
@@ -93,6 +97,7 @@ After examining both the reference implementation and HTTP server options, we ha
 ### 1. Main Server ([`src/main.rs`](src/main.rs))

 **Responsibilities:**
+
 - Initialize configuration from environment (clap + dotenvy)
 - Set up Hyper HTTP server with request routing
 - Initialize Nostr relay builder with custom [`Nip34WritePolicy`](src/nostr/builder.rs:51)
@@ -101,6 +106,7 @@ After examining both the reference implementation and HTTP server options, we ha
 - Handle graceful shutdown

 **Key Dependencies:**
+
 ```rust
 hyper = "1"
 tokio = { version = "1", features = ["full"] }
@@ -112,6 +118,7 @@ nostr-lmdb = "0.43"
 ### 2. HTTP Module ([`src/http/mod.rs`](src/http/mod.rs))

 **Responsibilities:**
+
 - Route HTTP requests to appropriate handlers
 - WebSocket upgrade for Nostr relay at `/`
 - Git Smart HTTP endpoints at `/<npub>/<identifier>.git/*`
@@ -228,7 +235,7 @@ Provides structures for parsing NIP-34 events:
 /// Parsed repository announcement (Kind 30617)
 pub struct RepositoryAnnouncement { ... }

-/// Parsed repository state (Kind 30618)  
+/// Parsed repository state (Kind 30618)
 pub struct RepositoryState { ... }
 ```

@@ -332,10 +339,12 @@ See [test-strategy.md](../reference/test-strategy.md) for comprehensive testing
 ### Quick Overview

 **Integration Tests** ([`tests/`](tests/)):
+
 - Use [`TestRelay`](tests/common/relay.rs:14) fixture for automatic relay lifecycle
 - Each test file in [`tests/`](tests/) covers a GRASP-01 requirement

 **Audit Tests** ([`grasp-audit/`](grasp-audit/)):
+
 - Reusable compliance testing for any GRASP implementation
 - Spec-mirrored structure in [`grasp-audit/src/specs/grasp01/`](grasp-audit/src/specs/grasp01/)

@@ -367,11 +376,52 @@ async fn test_nip01_websocket_connection() {
 - Maintainer sets computed once per event validation
 - State lookups use database indexes

+## Proactive Sync (GRASP-02)
+
+The ngit-grasp relay implements **Proactive Sync of Nostr Eevents**, which synchronizes repository data from external relays listed in 30617 repository announcements. This enables the relay to maintain complete repository graphs even when events are published to other listed relays.
+
+**Key Features:**
+
+- **Self-subscription** discovery - monitors own relay for announcements and root events to follow
+- **Three-way diff** (`compute_actions`) determines new subscriptions needed
+- **Smart reconnection** - uses `since` filter for quick reconnects (<15 min), fresh sync otherwise
+- **Health tracking** with exponential backoff for failing relays
+- **Daily sync** with random 23-25h timer to detect state drift
+- **Filter consolidation** when count exceeds 70 to prevent subscription explosion
+
+**Architecture:**
+
+```
+┌─────────────────────────────────────────────────────────────┐
+│                       SyncManager                            │
+├─────────────────────────────────────────────────────────────┤
+│                                                             │
+│  ┌──────────────────┐              ┌──────────────────┐    │
+│  │  SelfSubscriber  │──actions──▶  │  Main Event Loop │    │
+│  │  (own relay)     │              │  (Arc<Mutex>)    │    │
+│  └──────────────────┘              └────────┬─────────┘    │
+│                                             │              │
+│  ┌──────────────────┐              ┌────────▼─────────┐    │
+│  │  Daily Timer     │──────────────▶  RelayConnection │    │
+│  │  (23-25h random) │              │  per external    │    │
+│  └──────────────────┘              │  relay           │    │
+│                                    └──────────────────┘    │
+│  ┌──────────────────┐                                      │
+│  │  Health Tracker  │  Exponential backoff, dead detection │
+│  │  (DashMap)       │                                      │
+│  └──────────────────┘                                      │
+└─────────────────────────────────────────────────────────────┘
+```
+
+**Source Code:** [`src/sync/`](src/sync/)
+
+For full design details, see [grasp-02-proactive-sync-v4.md](grasp-02-proactive-sync-v4.md).
+
 ## Future Extensions

 ### GRASP-02: Proactive Sync

-See [grasp-02-proactive-sync.md](grasp-02-proactive-sync.md) for detailed design.
+GRASP-02 is only partially implemented. still outstanding is the proactive sync of git data for 1. state event and 2. PRs / PR Update refs.

 ### GRASP-05: Archive

@@ -437,6 +487,6 @@ The key insight is that we don't need to rely on Git's hook mechanism when we ha
 ## Related Documentation

 - [Inline Authorization Explanation](inline-authorization.md) - Why we chose this approach
- [GRASP-02 Proactive Sync](grasp-02-proactive-sync.md) - Future work design
+- [GRASP-02 Proactive Sync v4 Design](grasp-02-proactive-sync-v4.md) - Current production sync implementation
 - [Test Strategy](../reference/test-strategy.md) - Comprehensive testing documentation
- [GRASP-01 Implementation Learnings](../learnings/grasp-01-implementation.md) - Patterns and lessons learned
+- [GRASP-01 Implementation Learnings](../learnings/grasp-01-implementation.md) - Patterns and lessons learned
@@ -1226,4 +1226,99 @@ impl SelfSubscriber {
        }
    }
 }
-```
+```
+
+---
+
+## Implementation Notes
+
+This section documents the actual implementation details as of December 2024 (Phases 1-10 complete).
+
+### Architectural Decisions During Implementation
+
+**Phase 7 Refactoring**: The `SyncManager::run()` method required refactoring to use `Arc<Mutex<SyncManager>>` for shared access. The daily timer and disconnect checker tasks need to access the manager, so `self` is wrapped after initial setup:
+
+```rust
+// 7. Wrap self in Arc<Mutex> for sharing with timer task
+let sync_manager = Arc::new(Mutex::new(self));
+```
+
+This allows background tasks (daily timer, disconnect checker) to acquire the lock when needed while the main event loop handles actions from the self-subscriber.
+
+**Health Module**: The health tracking module was adapted from the v3 implementation at `work/sync-v3/health.rs`. The implementation uses:
+- `DashMap` for thread-safe concurrent access without external locking
+- Three states: `Healthy`, `Degraded`, `Dead`
+- Exponential backoff: `base * 2^(failures-1)`, capped at max_backoff
+- Dead threshold: 24 hours of continuous failures
+- Dead relay retry: Once per 24 hours
+
+### Implementation Constants
+
+| Constant | Value | Purpose |
+|----------|-------|---------|
+| `CONSOLIDATION_THRESHOLD` | 70 filters | Maximum filters before triggering consolidation |
+| `CONSOLIDATION_WAIT_TIMEOUT_SECS` | 30 seconds | Timeout for pending batches during consolidation |
+| `QUICK_RECONNECT_WINDOW_SECS` | 15 minutes | Window for quick reconnect vs fresh sync |
+| `DISCONNECT_CHECK_INTERVAL_SECS` | 60 seconds | Interval for checking empty relays to disconnect |
+| `DEAD_THRESHOLD_HOURS` | 24 hours | Time before relay marked as dead |
+| `BASE_BACKOFF_SECS` | 5 seconds | Base duration for exponential backoff |
+
+### Daily Timer Randomization
+
+The daily timer uses randomization between 23-25 hours to prevent thundering herd effects when multiple ngit-grasp instances are running:
+
+```rust
+let hours = 23.0 + rand::thread_rng().gen::<f64>() * 2.0;
+```
+
+### Bootstrap Relay Protection
+
+Bootstrap relays are never disconnected by the cleanup system. The `check_disconnects()` method explicitly filters them out:
+
+```rust
+.filter(|(_, state)| {
+    !state.is_bootstrap &&
+    state.repos.is_empty() &&
+    state.root_events.is_empty()
+})
+```
+
+### Graceful Shutdown
+
+Shutdown uses a tokio broadcast channel for coordinated termination:
+
+```rust
+let (shutdown_tx, _shutdown_rx) = broadcast::channel(1);
+```
+
+Each background task (self-subscriber, daily timer, disconnect checker) receives its own `broadcast::Receiver` subscription and monitors for the shutdown signal in its main loop.
+
+### Actual Module Structure
+
+The implemented module structure differs from the original spec:
+
+```
+src/sync/
+├── mod.rs              # SyncManager, main loop, index types, metrics
+├── algorithms.rs       # derive_relay_targets, compute_actions, AddFilters
+├── filters.rs          # build_announcement_filter, build_layer2_and_layer3_filters
+├── health.rs           # RelayHealthTracker, HealthState, exponential backoff
+├── relay_connection.rs # RelayConnection, RelayEvent, WebSocket handling
+└── self_subscriber.rs  # SelfSubscriber, RelayAction, batching logic
+```
+
+Key differences from spec:
+- No separate `state.rs` - types are defined in `mod.rs`
+- No separate `actions.rs` - moved to `algorithms.rs`
+- No separate `consolidation.rs` - consolidation logic in `mod.rs`
+- No separate `metrics.rs` - `SyncMetrics` defined in `mod.rs`
+
+### Deviations from Original v4 Spec
+
+1. **RelayState lacks `connection` field**: The spec showed `connection: Option<RelayConnection>` in `RelayState`, but the implementation stores connections in a separate `HashMap<String, RelayConnection>` in `SyncManager`.
+
+2. **SelfSubscriber simplified**: The actual implementation uses `RelayAction` enum (SpawnRelay/AddFilters) rather than directly using `AddFilters` struct.
+
+3. **Consolidation wait_pending_complete**: The spec described a `wait_pending_complete()` method, but the implementation uses a simpler timeout-based approach checking pending batches.
+
+4. **Timestamp API**: Uses `Timestamp::now().as_secs()` instead of `.as_u64()` due to nostr-sdk 0.43 API.