Migrate relay.ngit.dev from ngit-relay to ngit-grasp (v2)

ID: 4bc5

Problem

relay.ngit.dev currently runs ngit-relay (reference implementation). We want to consolidate on ngit-grasp as the production implementation.

Goal: Replace an ngit-relay instance on a VPS running NixOS with ngit-grasp.

Context: This is a fresh start after issue 820a became too complex with extensive investigation history. We're starting from scratch with a focus on creating a small, lightweight, easy-to-implement how-to document.

Plan

Script Development (Modular Architecture)

Phase 1: Fetch Events (~30s, local) - 01-fetch-events.sh
- Fetch kind 30618 (state), 30617 (announcement), 5 (deletion) from relay
- Run for both prod and archive relays
Phase 2: Git Sync Check (~20 mins, VPS) - 10-check-git-sync.sh
- Compare state event refs to actual git data on disk
- Run for both prod and archive git directories
- Note: Existing Jan 22 data available, script not yet created
Phase 3: Categorize & Compare (fast, local) - 20-categorize.sh, 21-compare-relays.sh
- Apply 4-category logic (complete/empty/partial/no-match)
- Find gaps between prod and archive
Phase 4: Log-Based Categories (VPS) - 30-extract-parse-failures.sh, 31-extract-purgatory-expiry.sh
- Extract parse failures and purgatory expiry from logs
- Dependency: Logging improvements in ngit-grasp - IMPLEMENTED
Phase 5: Final Classification (fast, local) - 40-classify-actions.sh
- Combine all data to produce: no-action, action-required, manual-investigation
Orchestration Script - run-migration-analysis.sh
- Runs all phases with proper error handling and progress reporting
- Supports phase control (skip, only, from-phase options)
- Dry-run mode, timing information, summary display

Migration Execution

Run analysis scripts on relay.ngit.dev
Review action-required repos, make decisions
Execute migration (switch domain, disable archive mode)
Validate migration success

Progress

2026-01-23 [Session 16:00]

Created: Fresh v2 issue to replace complex 820a migration
Context: Previous issue (820a) paused due to complexity
Approach: Start from scratch, potentially reuse scripts from 820a worktree
Goal: Create small, lightweight, easy-to-implement how-to document
Started work: Created worktree for issue 4bc5
Completed: Created initial how-to document at docs/how-to/migrate-ngit-relay-to-ngit-grasp.md
Document includes: Approach, challenges, analysis categories, gotchas
Next: User requested NOT to do planning for migration script yet

2026-01-23 [Session 17:30]

Reviewed existing scripts from 820a worktree:
- analyze-git-state-sync.sh - monolithic, takes ~20 mins (git sync is slow part)
- compare-categories.sh - compares prod vs archive categories
- migration-validation-guide.md - comprehensive troubleshooting guide
Reviewed existing analysis output (Jan 22):
- Prod: 654 repos (509 complete, 114 empty, 25 partial, 6 no-match)
- Archive: 263 repos (247 complete, 9 empty, 5 partial, 2 no-match)
Designed modular script architecture for fast iteration:
- Split into 5 phases with clear inputs/outputs
- Phases 1, 3, 5 can run locally; Phases 2, 4 need VPS
- Can use cached data from Jan 22 to develop categorization logic
Added log-based categories (scriptable):
- Parse failures: [PARSE_FAIL] kind=X event_id=Y reason=Z
- Purgatory expiry: [PURGATORY_EXPIRED] repo=X npub=Y
Updated how-to doc with full architecture diagram
Next: Implement Phase 1 (fetch events) to get fresh data

2026-01-23 [Session 18:45]

Reviewed Phase 2 outputs from Jan 22 (820a worktree):
- Prod: 654 repos (509 complete, 114 empty, 25 partial, 6 no-match)
- Archive: 263 repos (247 complete, 9 empty, 5 partial, 2 no-match)
- Format: repo | npub | state_refs=N | git_refs=N | matches=N [| reason=X]
Decision: Phase 2 outputs ARE sufficient for Phase 3 processing
- Existing data already categorized into 4 files
- No need to create Phase 2 script immediately (can use Jan 22 data)
Implemented Phase 3 scripts:
- 20-categorize.sh - Takes TSV input, outputs 4 category files
- 21-compare-relays.sh - Compares prod vs archive categories
Tested both scripts successfully:
- 20-categorize.sh correctly categorizes sample TSV data
- 21-compare-relays.sh produces comparison with Jan 22 data:
  - Complete in both: 231 (no action needed)
  - Complete in prod, MISSING from archive: 276 (needs investigation)
  - Complete in prod, incomplete in archive: 2
  - Incomplete in both: 131
  - In archive only: 5
Updated how-to doc with correct script paths and output structure
Next: Phase 4 (log extraction) or Phase 5 (final classification)

2026-01-23 [Session 20:00]

Implemented structured debug logging for Phase 4 migration scripts
Added [PARSE_FAIL] log entries in src/nostr/builder.rs:
- Format: [PARSE_FAIL] kind=X event_id=Y... reason="Z" repo=R npub=N
- Logged when: announcement parsing fails, state event parsing fails, PR git data check fails
- Includes repo identifier extracted from 'd' tag (announcements/states) or 'a' tag (PRs)
Added [PURGATORY_EXPIRED] log entries in src/purgatory/mod.rs:
- Format: [PURGATORY_EXPIRED] repo=X npub=Y event_id=Z... kind=K reason="..."
- Logged when: state events or PR events expire from purgatory without git data
- Includes all fields needed by Phase 4 scripts
All 382 tests pass
Log format matches what Phase 4 scripts expect (30-extract-parse-failures.sh, 31-extract-purgatory-expiry.sh)
Next: Commit changes, then Phase 5 (final classification)

2026-01-23 [Session 11:40]

Implemented Phase 5 final classification script (40-classify-actions.sh)
- Combines all data sources from Phases 1-4
- Produces three output files: no-action-required.txt, action-required.txt, manual-investigation.txt
- Generates summary.txt with breakdown by category and reason
Created orchestration script (run-migration-analysis.sh)
- Runs all 5 phases in sequence with proper error handling
- Parameterized inputs: relay URLs, git paths, service name, output directory
- Phase control: --skip-phase-N, --only-phase-N, --from-phase-N
- Dry-run mode to preview execution
- Progress indicators and timing information
- Auto-detects available features (git paths, journalctl)
Restructured migration guide (docs/how-to/migrate-ngit-relay-to-ngit-grasp.md)
- Added Quick Start section with copy-paste commands
- Added Prerequisites section with verification steps
- Added Running the Analysis section with all options
- Added Understanding Results section explaining output files
- Added Troubleshooting section for common issues
- Moved Architecture section (was at top) for those wanting details
- Added Next Steps section for post-analysis workflow
All scripts committed:
- 5dfd1cb - Add orchestration script for migration analysis pipeline
- d8a88e8 - Restructure migration guide for practical usage
Next: Run analysis on relay.ngit.dev, review results

2026-01-23 [Session 11:50]

Reviewed structured logging implementation (commit 807961b)
Fixed multi-repo PR event handling:
- PR events can reference multiple repositories (via multiple a tags)
- Original code only logged the FIRST repo identifier
- Updated extract_repos_from_pr_event to return ALL unique repos
- Now logs once per repo for both [PARSE_FAIL] and [PURGATORY_EXPIRED]
Structured logging assessment:
- Current format uses formatted strings (not tracing structured fields)
- This is intentional - designed for grep/awk parsing by Phase 4 scripts
- Proper structured logging would require script updates
- Recommendation: Keep current format for migration, consider structured logging as future improvement
Script compatibility verified:
- Phase 4 scripts will correctly parse multi-repo PR events as separate entries
- No script changes needed
All 382 unit tests + 38 integration tests pass
Recommendation: Create low-priority issue for broader structured logging adoption (better observability, log aggregation)
Next: Commit multi-repo fix, then run analysis on relay.ngit.dev

Notes

Related issue: 820a-relay-ngit-dev-migration.md (paused, in paused/ directory)
VPS: Running NixOS
Old worktree: Can reference /persistent/dcdev/clones/ngit-grasp/worktrees/820a-relay-ngit-dev-migration/ for existing scripts and learnings
Target: Simple, practical migration guide that works

8.5 KiB Raw Blame History