mirror of
https://relay.ngit.dev/npub15qydau2hjma6ngxkl2cyar74wzyjshvl65za5k5rl69264ar2exs5cyejr/ngit-grasp.git
synced 2026-10-05 15:08:24 +00:00
issue: create 1ce6 - purgatory retry mechanism fails during initial sync
This commit is contained in:
DanConwayDev
@@ -0,0 +1,174 @@
|
|||||||
|
# Purgatory retry mechanism fails during initial sync when clone URLs are unreachable
|
||||||
|
|
||||||
|
**ID:** 1ce6
|
||||||
|
|
||||||
|
## Problem
|
||||||
|
|
||||||
|
During initial sync (archive/migration scenarios), ngit-grasp's purgatory mechanism can permanently reject events that should be retried, causing significant data loss.
|
||||||
|
|
||||||
|
### How Purgatory Works
|
||||||
|
|
||||||
|
1. Events arrive at the relay and go to purgatory (30-minute expiry) while git data is fetched
|
||||||
|
2. If git fetch succeeds within 30 minutes, event is processed
|
||||||
|
3. If git fetch fails, event expires and its ID is added to `expired_events`
|
||||||
|
4. **Critical:** Once in `expired_events`, the event will NOT be retried until the 7-day cleanup
|
||||||
|
|
||||||
|
### The Problem
|
||||||
|
|
||||||
|
During initial sync, archive fetches events from source relay. When clone URLs are unreachable or throttled:
|
||||||
|
|
||||||
|
1. Git fetch fails for the event
|
||||||
|
2. After 30 minutes, event expires
|
||||||
|
3. Event ID added to `expired_events` (7-day rejection)
|
||||||
|
4. **Event will NOT be retried even if clone URLs become available**
|
||||||
|
|
||||||
|
### Real-World Impact (relay.ngit.dev Migration)
|
||||||
|
|
||||||
|
- Archive synced 1,858 repos from relay.ngit.dev
|
||||||
|
- **1,581 repos (85.1%) stuck in purgatory** - git data never fetched
|
||||||
|
- Only 277 repos (14.9%) have git data
|
||||||
|
- State event coverage: 47.0% instead of expected >99%
|
||||||
|
- Migration blocked until purgatory state manually cleared
|
||||||
|
|
||||||
|
### Root Cause Analysis
|
||||||
|
|
||||||
|
1. **External relay unavailability:** Archive tried to fetch from external relays (git.shakespeare.diy, etc.) that were unreachable or throttling
|
||||||
|
2. **No fallback logic:** Archive doesn't try alternative clone URLs from the same event
|
||||||
|
3. **Permanent rejection:** Once event expires, it's rejected for 7 days (no retry mechanism)
|
||||||
|
4. **Silent failure:** No prominent logging or metrics for purgatory accumulation
|
||||||
|
|
||||||
|
## Why This Matters
|
||||||
|
|
||||||
|
1. **Initial sync is fragile:** If external relays are down during initial sync, repos are stuck for 7 days
|
||||||
|
2. **No retry mechanism:** Once event expires, it won't be retried even if clone URLs become available
|
||||||
|
3. **No fallback logic:** Archive doesn't try alternative clone URLs from the same event
|
||||||
|
4. **Silent failure:** No prominent logging or metrics for purgatory accumulation
|
||||||
|
5. **Manual intervention required:** Only way to retry is to delete purgatory state file and restart
|
||||||
|
6. **Not scalable:** Manual intervention doesn't scale for production deployments
|
||||||
|
|
||||||
|
## Plan
|
||||||
|
|
||||||
|
### Phase 1: Retry with Exponential Backoff
|
||||||
|
- [ ] Don't permanently reject events after first failure
|
||||||
|
- [ ] Implement retry with increasing delays (1h, 4h, 12h, 24h, 7d)
|
||||||
|
- [ ] Only permanently reject after multiple failures over 7 days
|
||||||
|
- [ ] Track retry count and last attempt time per event
|
||||||
|
|
||||||
|
### Phase 2: Clone URL Fallback Logic
|
||||||
|
- [ ] Parse all clone URLs from event (not just first one)
|
||||||
|
- [ ] Try secondary URLs if primary fails
|
||||||
|
- [ ] Log which URL succeeded for debugging
|
||||||
|
- [ ] Consider URL priority/preference ordering
|
||||||
|
|
||||||
|
### Phase 3: Purgatory Metrics and Alerts
|
||||||
|
- [ ] Expose `ngit_purgatory_count` metric
|
||||||
|
- [ ] Expose `ngit_expired_events_count` metric
|
||||||
|
- [ ] Expose `ngit_purgatory_retry_count` metric
|
||||||
|
- [ ] Log warnings when purgatory accumulates (>10%, >50%, >80% of total events)
|
||||||
|
- [ ] Add structured logging for purgatory state changes
|
||||||
|
|
||||||
|
### Phase 4: Configuration Options
|
||||||
|
- [ ] `purgatory_retry_enabled` (default: true)
|
||||||
|
- [ ] `purgatory_retry_max_attempts` (default: 5)
|
||||||
|
- [ ] `purgatory_retry_backoff_multiplier` (default: 2)
|
||||||
|
- [ ] `purgatory_expiry_duration` (default: 30min, allow longer for initial sync)
|
||||||
|
- [ ] Document all options in configuration reference
|
||||||
|
|
||||||
|
### Phase 5: Initial Sync Mode
|
||||||
|
- [ ] Detect initial sync (empty database or explicit flag)
|
||||||
|
- [ ] Use longer purgatory expiry (4 hours instead of 30 minutes)
|
||||||
|
- [ ] More aggressive retry logic during initial sync
|
||||||
|
- [ ] Better logging and progress reporting
|
||||||
|
- [ ] Consider `NGIT_INITIAL_SYNC_MODE=true` environment variable
|
||||||
|
|
||||||
|
## Acceptance Criteria
|
||||||
|
|
||||||
|
- [ ] Events are retried multiple times before permanent rejection
|
||||||
|
- [ ] All clone URLs are attempted before giving up
|
||||||
|
- [ ] Purgatory metrics are exposed (count, expired, retries)
|
||||||
|
- [ ] Initial sync has special handling (longer expiry, more retries)
|
||||||
|
- [ ] Manual intervention not required for transient failures
|
||||||
|
- [ ] Configuration options documented and tested
|
||||||
|
- [ ] Integration tests cover retry scenarios
|
||||||
|
|
||||||
|
## Technical Notes
|
||||||
|
|
||||||
|
### Current Purgatory Implementation
|
||||||
|
|
||||||
|
- **Location:** `src/purgatory/mod.rs`
|
||||||
|
- **Expiry tracking:** `expired_events: DashMap<EventId, Instant>` (line 79)
|
||||||
|
- **Expiry duration:** 30 minutes (hardcoded)
|
||||||
|
- **Cleanup:** 7-day expiry for `expired_events` entries
|
||||||
|
|
||||||
|
### Proposed Changes
|
||||||
|
|
||||||
|
1. **New fields in purgatory:**
|
||||||
|
```rust
|
||||||
|
struct RetryState {
|
||||||
|
attempts: u32,
|
||||||
|
last_attempt: Instant,
|
||||||
|
next_retry: Instant,
|
||||||
|
failed_urls: Vec<String>,
|
||||||
|
}
|
||||||
|
retry_state: DashMap<EventId, RetryState>
|
||||||
|
```
|
||||||
|
|
||||||
|
2. **Retry backoff schedule:**
|
||||||
|
- Attempt 1: Immediate
|
||||||
|
- Attempt 2: 1 hour
|
||||||
|
- Attempt 3: 4 hours
|
||||||
|
- Attempt 4: 12 hours
|
||||||
|
- Attempt 5: 24 hours
|
||||||
|
- After 5 failures: Permanent rejection (7-day expiry)
|
||||||
|
|
||||||
|
3. **Clone URL fallback:**
|
||||||
|
- Parse `clone` tags from event
|
||||||
|
- Try URLs in order until one succeeds
|
||||||
|
- Track which URLs failed for debugging
|
||||||
|
|
||||||
|
## Related Context
|
||||||
|
|
||||||
|
- **Migration issue:** 820a-relay-ngit-dev-migration
|
||||||
|
- **Purgatory persistence:** a6b6-purgatory-surive-reboot (completed)
|
||||||
|
- **Archive sync results:** 1,858 repos, 1,581 in purgatory (85.1%)
|
||||||
|
- **Workaround:** Delete `purgatory-state.json` and restart relay
|
||||||
|
- **Investigation:** work/state-validation/ARCHIVE-PURGATORY-ANALYSIS.md
|
||||||
|
|
||||||
|
## Workaround (Until Fixed)
|
||||||
|
|
||||||
|
For operators experiencing this issue:
|
||||||
|
|
||||||
|
```bash
|
||||||
|
# 1. Stop the relay
|
||||||
|
systemctl stop ngit-grasp
|
||||||
|
|
||||||
|
# 2. Delete purgatory state (forces re-sync of all events)
|
||||||
|
rm /path/to/data/purgatory-state.json
|
||||||
|
|
||||||
|
# 3. Optionally delete rejected events cache
|
||||||
|
rm /path/to/data/rejected-events-cache.json
|
||||||
|
|
||||||
|
# 4. Restart relay
|
||||||
|
systemctl start ngit-grasp
|
||||||
|
|
||||||
|
# 5. Monitor purgatory accumulation
|
||||||
|
journalctl -u ngit-grasp -f | grep -i purgatory
|
||||||
|
```
|
||||||
|
|
||||||
|
**Note:** This workaround requires manual intervention and may cause duplicate processing. The proper fix is implementing the retry mechanism described above.
|
||||||
|
|
||||||
|
## Progress
|
||||||
|
|
||||||
|
### 2026-01-20 [Session 12:00]
|
||||||
|
- Created: Issue documenting purgatory retry problem discovered during relay.ngit.dev migration
|
||||||
|
- Context: Archive sync showed 85.1% of repos stuck in purgatory due to unreachable clone URLs
|
||||||
|
- Root cause: No retry mechanism after initial git fetch failure
|
||||||
|
- Impact: Migration blocked, manual intervention required
|
||||||
|
- Next: Prioritize based on migration timeline
|
||||||
|
|
||||||
|
## Notes
|
||||||
|
|
||||||
|
- Priority: High (blocks migration scenarios)
|
||||||
|
- Discovered during: 820a relay.ngit.dev migration
|
||||||
|
- Affects: Archive mode, initial sync, any scenario with unreliable external relays
|
||||||
|
- Related to but distinct from: a6b6 (persistence) - that issue ensures purgatory survives restarts, this issue addresses retry logic
|
||||||
Reference in New Issue
Block a user