diff --git a/820a-relay-ngit-dev-migration.md b/820a-relay-ngit-dev-migration.md index da879b1..f7f25c0 100644 --- a/820a-relay-ngit-dev-migration.md +++ b/820a-relay-ngit-dev-migration.md @@ -550,6 +550,62 @@ docker start ngit-relay 4. Verify state event count matches production (>95% coverage) 5. Proceed to Phase 3 validation once sync complete +### 2026-01-21 [Session 15:00] - Negentropy Fix Applied & DEEPER ISSUE DISCOVERED ⚠️ +- Completed: Applied negentropy fix, verified working, but discovered sync query design issue +- **Negentropy Fix Applied Successfully:** + - ✅ Issue b4d0 created: "Failed historic sync batches not retried after negentropy timeout" + - ✅ Config updated: Added `syncDisableNegentropy = true` to archive service + - ✅ Deployed and restarted: Service running with REQ+EOSE sync + - ✅ Fix verified: Logs show `use_negentropy=false`, state events flowing + - ✅ State event count: 336 → 339 (increasing) + - ✅ Status: `Connected` (not `ConnectedHistoricSyncFailures`) +- **1-man-1-tank Investigation - CRITICAL FINDING:** + - ❌ State event still NOT received after negentropy fix + - ✅ Announcement (kind 30617) exists in archive + - ❌ State event (kind 30618) does NOT exist in archive + - ✅ State event exists on relay.ngit.dev and git.shakespeare.diy + - ❌ Git repo empty (no refs, no objects) +- **ROOT CAUSE: Sync Query Design Flaw** + - Announcements: Broad query `{"kinds": [10317, 30617]}` (no d-tag filter) ✅ + - State events: Narrow query `{"kinds": [30618], "#d": [list of known repos]}` ❌ + - **Problem:** State events only queried for repos archive already knows about + - **Chicken-and-egg:** 1-man-1-tank state event never synced → not in query list → never queried → never synced +- **Why This Happens:** + 1. Archive receives announcement for 1-man-1-tank (broad query) + 2. Archive creates empty git repo + 3. Archive queries state events only for repos it already has state events for + 4. 1-man-1-tank NOT in list → state event never queried + 5. State event never arrives → git data never fetched → repo stays empty +- **Evidence from Logs:** + ``` + # Broad announcement query (works) + Sending '["REQ",...,{"kinds":[10317,30617]}]' to relay.ngit.dev + + # Narrow state event query (misses 1-man-1-tank) + Sending '["REQ",...,{"#d":["BeatsBars","storyforge-ai",...], + "kinds":[30618]}]' to relay.ngit.dev + # Note: "1-man-1-tank" NOT in d-tag list + ``` +- **Impact Assessment:** + - Negentropy fix IS working (REQ+EOSE active, other repos syncing) + - But won't help repos whose state events were never queried + - Affects ALL repos where announcement arrived but state event didn't + - Explains persistent gap between announcement count and state event count +- **Recommended Fixes:** + 1. **Option 1 (Best):** Add broad state event query `{"kinds": [30618]}` (no d-tag filter) + 2. **Option 2:** Query state events for all known announcements (when announcement received) + 3. **Option 3 (Immediate):** Manual re-sync by publishing state events to archive +- **Files Created:** + - `work/NEGENTROPY-FIX-VERIFICATION.md` - Verification of negentropy fix + - `work/1-MAN-1-TANK-SYNC-STATUS.md` - Detailed investigation of sync query issue +- **Migration Status:** ❌ **BLOCKED** - Sync query design prevents complete state event coverage +- **Next Steps:** + 1. Create issue for sync query design flaw + 2. Implement broad state event query (Option 1) + 3. Re-sync archive with fixed query design + 4. Verify 1-man-1-tank and other missing repos sync successfully + 5. Achieve >95% state event coverage before proceeding to Phase 3 + ### 2026-01-20 [Session 14:30] - Naughty List Blocker Identified - Completed: Extensive state validation and root cause analysis - **Blocker identified:** relay.ngit.dev on git "naughty list" preventing archive sync