ngit-grasp/4bc5-relay-ngit-dev-migration-v2.md

# Migrate relay.ngit.dev from ngit-relay to ngit-grasp (v2)

**ID:** 4bc5

## Problem

relay.ngit.dev currently runs ngit-relay (reference implementation). We want to consolidate on ngit-grasp as the production implementation.

**Goal:** Replace an ngit-relay instance on a VPS running NixOS with ngit-grasp.

**Context:** This is a fresh start after issue 820a became too complex with extensive investigation history. We're starting from scratch with a focus on creating a small, lightweight, easy-to-implement how-to document.

## Plan

### Script Development (Modular Architecture)

- [x] **Phase 1: Fetch Events** (~30s, local) - `01-fetch-events.sh`
  - Fetch kind 30618 (state), 30617 (announcement), 5 (deletion) from relay
  - Run for both prod and archive relays
- [ ] **Phase 2: Git Sync Check** (~20 mins, VPS) - `10-check-git-sync.sh`
  - Compare state event refs to actual git data on disk
  - Run for both prod and archive git directories
  - **Note:** Existing Jan 22 data available, script not yet created
- [x] **Phase 3: Categorize & Compare** (fast, local) - `20-categorize.sh`, `21-compare-relays.sh`
  - Apply 4-category logic (complete/empty/partial/no-match)
  - Find gaps between prod and archive
- [x] **Phase 4: Log-Based Categories** (VPS) - `30-extract-parse-failures.sh`, `31-extract-purgatory-expiry.sh`
  - Extract parse failures and purgatory expiry from logs
  - **Dependency:** Logging improvements in ngit-grasp - IMPLEMENTED
- [x] **Phase 5: Final Classification** (fast, local) - `40-classify-actions.sh`
  - Combine all data to produce: no-action, action-required, manual-investigation
- [x] **Orchestration Script** - `run-migration-analysis.sh`
  - Runs all phases with proper error handling and progress reporting
  - Supports phase control (skip, only, from-phase options)
  - Dry-run mode, timing information, summary display

### Migration Execution

- [ ] Run analysis scripts on relay.ngit.dev
- [ ] Review action-required repos, make decisions
- [ ] Execute migration (switch domain, disable archive mode)
- [ ] Validate migration success

## Progress

### 2026-01-23 [Session 16:00]
- Created: Fresh v2 issue to replace complex 820a migration
- Context: Previous issue (820a) paused due to complexity
- Approach: Start from scratch, potentially reuse scripts from 820a worktree
- Goal: Create small, lightweight, easy-to-implement how-to document
- Started work: Created worktree for issue 4bc5
- Completed: Created initial how-to document at `docs/how-to/migrate-ngit-relay-to-ngit-grasp.md`
- Document includes: Approach, challenges, analysis categories, gotchas
- Next: User requested NOT to do planning for migration script yet

### 2026-01-23 [Session 17:30]
- Reviewed existing scripts from 820a worktree:
  - `analyze-git-state-sync.sh` - monolithic, takes ~20 mins (git sync is slow part)
  - `compare-categories.sh` - compares prod vs archive categories
  - `migration-validation-guide.md` - comprehensive troubleshooting guide
- Reviewed existing analysis output (Jan 22):
  - Prod: 654 repos (509 complete, 114 empty, 25 partial, 6 no-match)
  - Archive: 263 repos (247 complete, 9 empty, 5 partial, 2 no-match)
- Designed modular script architecture for fast iteration:
  - Split into 5 phases with clear inputs/outputs
  - Phases 1, 3, 5 can run locally; Phases 2, 4 need VPS
  - Can use cached data from Jan 22 to develop categorization logic
- Added log-based categories (scriptable):
  - Parse failures: `[PARSE_FAIL] kind=X event_id=Y reason=Z`
  - Purgatory expiry: `[PURGATORY_EXPIRED] repo=X npub=Y`
- Updated how-to doc with full architecture diagram
- Next: Implement Phase 1 (fetch events) to get fresh data

### 2026-01-23 [Session 18:45]
- Reviewed Phase 2 outputs from Jan 22 (820a worktree):
  - Prod: 654 repos (509 complete, 114 empty, 25 partial, 6 no-match)
  - Archive: 263 repos (247 complete, 9 empty, 5 partial, 2 no-match)
  - Format: `repo | npub | state_refs=N | git_refs=N | matches=N [| reason=X]`
- Decision: Phase 2 outputs ARE sufficient for Phase 3 processing
  - Existing data already categorized into 4 files
  - No need to create Phase 2 script immediately (can use Jan 22 data)
- Implemented Phase 3 scripts:
  - `20-categorize.sh` - Takes TSV input, outputs 4 category files
  - `21-compare-relays.sh` - Compares prod vs archive categories
- Tested both scripts successfully:
  - `20-categorize.sh` correctly categorizes sample TSV data
  - `21-compare-relays.sh` produces comparison with Jan 22 data:
    - Complete in both: 231 (no action needed)
    - Complete in prod, MISSING from archive: 276 (needs investigation)
    - Complete in prod, incomplete in archive: 2
    - Incomplete in both: 131
    - In archive only: 5
- Updated how-to doc with correct script paths and output structure
- Next: Phase 4 (log extraction) or Phase 5 (final classification)

### 2026-01-23 [Session 20:00]
- Implemented structured debug logging for Phase 4 migration scripts
- Added `[PARSE_FAIL]` log entries in `src/nostr/builder.rs`:
  - Format: `[PARSE_FAIL] kind=X event_id=Y... reason="Z" repo=R npub=N`
  - Logged when: announcement parsing fails, state event parsing fails, PR git data check fails
  - Includes repo identifier extracted from 'd' tag (announcements/states) or 'a' tag (PRs)
- Added `[PURGATORY_EXPIRED]` log entries in `src/purgatory/mod.rs`:
  - Format: `[PURGATORY_EXPIRED] repo=X npub=Y event_id=Z... kind=K reason="..."`
  - Logged when: state events or PR events expire from purgatory without git data
  - Includes all fields needed by Phase 4 scripts
- All 382 tests pass
- Log format matches what Phase 4 scripts expect (30-extract-parse-failures.sh, 31-extract-purgatory-expiry.sh)
- Next: Commit changes, then Phase 5 (final classification)

### 2026-01-23 [Session 11:40]
- Implemented Phase 5 final classification script (`40-classify-actions.sh`)
  - Combines all data sources from Phases 1-4
  - Produces three output files: no-action-required.txt, action-required.txt, manual-investigation.txt
  - Generates summary.txt with breakdown by category and reason
- Created orchestration script (`run-migration-analysis.sh`)
  - Runs all 5 phases in sequence with proper error handling
  - Parameterized inputs: relay URLs, git paths, service name, output directory
  - Phase control: --skip-phase-N, --only-phase-N, --from-phase-N
  - Dry-run mode to preview execution
  - Progress indicators and timing information
  - Auto-detects available features (git paths, journalctl)
- Restructured migration guide (`docs/how-to/migrate-ngit-relay-to-ngit-grasp.md`)
  - Added Quick Start section with copy-paste commands
  - Added Prerequisites section with verification steps
  - Added Running the Analysis section with all options
  - Added Understanding Results section explaining output files
  - Added Troubleshooting section for common issues
  - Moved Architecture section (was at top) for those wanting details
  - Added Next Steps section for post-analysis workflow
- All scripts committed:
  - `5dfd1cb` - Add orchestration script for migration analysis pipeline
  - `d8a88e8` - Restructure migration guide for practical usage
- Next: Run analysis on relay.ngit.dev, review results

### 2026-01-23 [Session 11:50]
- Reviewed structured logging implementation (commit 807961b)
- **Fixed multi-repo PR event handling:**
  - PR events can reference multiple repositories (via multiple `a` tags)
  - Original code only logged the FIRST repo identifier
  - Updated `extract_repos_from_pr_event` to return ALL unique repos
  - Now logs once per repo for both `[PARSE_FAIL]` and `[PURGATORY_EXPIRED]`
- **Structured logging assessment:**
  - Current format uses formatted strings (not tracing structured fields)
  - This is intentional - designed for grep/awk parsing by Phase 4 scripts
  - Proper structured logging would require script updates
  - Recommendation: Keep current format for migration, consider structured logging as future improvement
- **Script compatibility verified:**
  - Phase 4 scripts will correctly parse multi-repo PR events as separate entries
  - No script changes needed
- All 382 unit tests + 38 integration tests pass
- **Recommendation:** Create low-priority issue for broader structured logging adoption (better observability, log aggregation)
- Next: Commit multi-repo fix, then run analysis on relay.ngit.dev

## Notes

- **Related issue:** 820a-relay-ngit-dev-migration.md (paused, in paused/ directory)
- **VPS:** Running NixOS
- **Old worktree:** Can reference `/persistent/dcdev/clones/ngit-grasp/worktrees/820a-relay-ngit-dev-migration/` for existing scripts and learnings
- **Target:** Simple, practical migration guide that works