diff --git a/ddbf-enhance-deletion-requests.md b/ddbf-enhance-deletion-requests.md new file mode 100644 index 0000000..bd5d2ef --- /dev/null +++ b/ddbf-enhance-deletion-requests.md @@ -0,0 +1,273 @@ +# Enhanced Deletion Request Features + +**ID:** ddbf + +**Status:** Not Started +**Priority:** Medium +**Complexity:** High +**Estimated Timeline:** 4-6 weeks + +## Problem Statement + +The core NIP-09 deletion request implementation (b905) is complete with Phases 1-5 delivering: +- Basic deletion with cascade for single and multi-maintainer repositories +- Git archival and background cleanup +- Multi-maintainer graph-based retention algorithm +- Recovery mechanism for accidentally deleted repositories +- Extended cascade deletion for all NIP-34 event types + +This issue tracks enhancements, edge cases, and production hardening features that were deferred from the initial implementation. + +## Dependencies + +- **Requires:** b905 (Deletion Request Support) - Phases 1-5 complete +- **Blocks:** None + +## Implementation Phases + +### Phase 1: Background Cleanup & rust-nostr Investigation +**Goal:** Resolve cleanup timing issues and understand rust-nostr deletion behavior + +**Tasks:** +- [ ] Background cleanup task investigation: + - [ ] Daily cleanup doesn't work well with 3-second retention tests + - [ ] Design alternative: trigger-based cleanup for tests? + - [ ] Or: configurable cleanup interval (separate from retention)? + - [ ] Test with both short intervals (tests) and daily (production) +- [ ] rust-nostr deletion behavior investigation: + - [ ] Check if nostr-relay-builder automatically deletes events tagged in kind 5 + - [ ] Check if it stops serving events tagged in deletion requests + - [ ] If YES: need to override/disable this behavior when `deletion_request_disrespector = true` + - [ ] Document any hooks or callbacks we need to implement + - [ ] Ensure archival mode truly ignores deletions at relay library level +- [ ] Author validation enforcement: + - [ ] Verify we only honor deletion requests where author matches deleted event author + - [ ] Add comprehensive tests for author mismatch rejection + - [ ] Document this requirement clearly in NIP-11 and README + +**Exit Criteria:** +- Cleanup timing strategy documented and tested +- rust-nostr deletion behavior fully understood +- Author validation comprehensively tested +- Documentation updated + +--- + +### Phase 2: Edge Cases & Performance Analysis +**Goal:** Document and test edge cases, analyze performance characteristics + +**Tasks:** +- [ ] Max depth edge case analysis: + - [ ] Document scenarios where depth limit matters + - [ ] Recommend alternative approaches if needed + - [ ] Add configuration guidance +- [ ] Large-scale scenario testing: + - [ ] Repository with 1000+ PRs/issues + - [ ] Deep dependency chains (50+ levels) + - [ ] Performance profiling + - [ ] Memory usage analysis +- [ ] Race condition investigation: + - [ ] Deletion during active sync + - [ ] Concurrent deletions of shared repo + - [ ] Lock strategy for git archival + - [ ] Document mitigation strategies + +**Exit Criteria:** +- Edge cases documented with recommendations +- Performance acceptable for production scale +- Race conditions understood and mitigated +- Configuration guidance complete + +--- + +### Phase 3: Blacklist Integration +**Goal:** Integrate deletion mechanism with blacklist functionality + +**Tasks:** +- [ ] Blacklist retroactive deletion investigation: + - [ ] Should adding to blacklist move existing repos to 90-day holding area? + - [ ] **Decision:** YES - use same archive mechanism as deletion requests + - [ ] Detect on startup: scan main DB for repos matching blacklist + - [ ] Move to holding area with metadata marking blacklist trigger + - [ ] Same cascade delete logic as NIP-09 deletions + - [ ] Archive git data before moving to holding area +- [ ] Blacklist + disrespector mode interaction: + - [ ] **Decision:** `deletion_request_disrespector` does NOT prevent blacklist deletion + - [ ] Blacklist is moderation/operational decision, not user-initiated deletion + - [ ] Archival relays can still blacklist spam/malware/abuse + - [ ] Only NIP-09 user deletions are ignored in disrespector mode +- [ ] Blacklist removal recovery: + - [ ] Should removing from blacklist auto-restore from holding DB? + - [ ] Or require manual operator intervention? + - [ ] What about the git data archive? + - [ ] Design recovery flow for unblacklisted repos + +**Exit Criteria:** +- Blacklist deletion working with same archival mechanism +- Disrespector mode interaction clearly defined +- Blacklist removal recovery flow designed +- All behaviors tested and documented + +--- + +### Phase 4: Manual Ejection & Delayed Archival +**Goal:** Add operator controls and delayed archival for active repositories + +**Tasks:** +- [ ] Manual ejection from holding area: + - [ ] Operator needs ability to force-delete from holding area before expiry + - [ ] Use case: large repos consuming excessive storage + - [ ] Use case: confirmed malware/abuse that shouldn't be recoverable + - [ ] Design mechanism: admin CLI command? config flag? database operation? + - [ ] Should manual ejection delete git archive immediately or wait for cleanup? + - [ ] Log manual ejections for audit trail +- [ ] Delayed archival strategy design (defer full implementation): + - [ ] Activity threshold evaluation algorithm + - [ ] Count issues/PRs/patches (< 5 = low activity) + - [ ] Count unique creator pubkeys (1-2 = low activity) + - [ ] Both conditions evaluated (either triggers immediate archival) + - [ ] Grace period implementation (2 minutes default) + - [ ] Prevent accidental deletions + - [ ] Allow user to publish new announcement to cancel + - [ ] Handle cancellation during grace period + - [ ] Notification issue creation + - [ ] Sign issue with repository owner pubkey + - [ ] Content: deletion warning, 14-day timeline, affected content list + - [ ] Store notification issue reference for cleanup + - [ ] Archive delay implementation (10 days default) + - [ ] Community response window + - [ ] Scheduled task system for delayed execution + - [ ] Cancellation check before archival + - [ ] Configuration options: + - [ ] `deletion_notification_delay_secs` (default 120) + - [ ] `deletion_archive_delay_secs` (default 864000) + - [ ] `deletion_activity_threshold` (default 5) + - [ ] `deletion_creator_threshold` (default 3) + - [ ] Add config to all 4 required sources (CRITICAL): + - [ ] `src/config.rs` + - [ ] `docs/reference/configuration.md` + - [ ] `nix/module.nix` + - [ ] `.env.example` + - [ ] Timeline calculation and validation + - [ ] Total for active repos: 2min + 10days + 90days ≈ 100 days + - [ ] Low-activity repos: immediate + 90 days + - [ ] Testing considerations + - [ ] Override delays for test scenarios (5-10 seconds) + - [ ] Test immediate vs delayed paths + - [ ] Test cancellation during both grace and delay periods + +**Exit Criteria:** +- Manual ejection mechanism designed and implemented +- Delayed archival strategy fully designed +- Configuration options added to all 4 sources +- Testing strategy defined + +--- + +### Phase 5: Metrics & Monitoring +**Goal:** Add comprehensive metrics for production monitoring + +**Tasks:** +- [ ] Metrics implementation: + - [ ] `ngit_deletion_requests_total` + - [ ] `ngit_deletion_requests_processed` + - [ ] `ngit_deletion_requests_immediate` (new - low activity) + - [ ] `ngit_deletion_requests_delayed` (new - active repos) + - [ ] `ngit_deletion_notifications_created` (new) + - [ ] `ngit_deletion_requests_cancelled` (new) + - [ ] `ngit_blacklist_deletions_total` + - [ ] `ngit_holding_database_events` + - [ ] `ngit_holding_database_size_bytes` + - [ ] `ngit_archive_files_total` + - [ ] `ngit_archive_size_bytes` + - [ ] `ngit_recoveries_total` + - [ ] `ngit_permanent_deletions_total` + - [ ] `ngit_manual_ejections_total` + +**Exit Criteria:** +- All metrics implemented and tested +- Metrics exposed via Prometheus endpoint +- Dashboard examples provided +- Alerting recommendations documented + +--- + +### Phase 6: Master Branch Integration & Documentation +**Goal:** Ensure compatibility with latest master and complete documentation + +**Tasks:** +- [ ] Master branch integration review: + - [ ] Review implications of deletion requests with latest master changes + - [ ] Verify compatibility with purgatory persistence feature + - [ ] Verify compatibility with defensive measures (rate limiting) + - [ ] Test interaction between holding database and purgatory + - [ ] Ensure no conflicts with rejected events index + - [ ] Document any integration considerations +- [ ] Documentation: + - [ ] Update `docs/explanation/deletion-requests.md` with findings + - [ ] Add blacklist deletion behavior documentation + - [ ] Add manual ejection mechanism documentation + - [ ] Add delayed archival strategy documentation + - [ ] Add edge case documentation + - [ ] Performance tuning guide + +**Exit Criteria:** +- Master branch integration verified +- All documentation complete and reviewed +- Production deployment guide ready + +--- + +### Phase 7: Advanced Deletion Features (Future Exploration) +**Goal:** Explore advanced deletion scenarios and recovery mechanisms + +**Tasks:** +- [ ] Event ID deletion (vs address deletion): + - [ ] When a specific announcement or state event is deleted by event ID (`e` tag) rather than address (`a` tag) + - [ ] Should NOT cascade delete + - [ ] Instead: try to rollback to previous replaceable event version (if available) + - [ ] Only proceed with deletion if no previous version can be found + - [ ] Explore recovery mechanisms +- [ ] PR deletion handling: + - [ ] When a PR (kind 1617/1618/1619) is deleted + - [ ] For now: should NOT cascade delete dependent events + - [ ] Needs more thought - revisit later + +**Exit Criteria:** +- Event ID deletion behavior designed +- PR deletion strategy defined +- Implementation plan created + +## Success Criteria + +1. Background cleanup timing optimized for both tests and production +2. rust-nostr deletion behavior fully understood and controlled +3. Edge cases documented with mitigation strategies +4. Performance acceptable at production scale (1000+ events) +5. Blacklist integration working with same archival mechanism +6. Manual ejection mechanism available for operators +7. Delayed archival strategy designed and documented +8. Comprehensive metrics for production monitoring +9. Master branch integration verified +10. Complete documentation for all features + +## Progress + +### 2026-01-15 [Session 11:30] +- **Issue Created:** Extracted Phase 6+ content from b905 issue + - Core implementation (Phases 1-5) complete in b905 + - This issue tracks enhancements, edge cases, and production hardening + - 7 implementation phases defined + - Dependencies: requires b905 completion +- **Next:** Not started - waiting for b905 integration tests to complete + +## Notes + +- **Related Issue:** b905 (Deletion Request Support) - core implementation +- **Reference:** `docs/explanation/deletion-requests.md` - comprehensive architecture +- **Configuration:** All config changes must update 4 sources (per AGENTS.md) +- **Future Enhancements:** + - GRASP-05 archive mode integration + - Selective disrespect (based on popularity/criteria) + - Distributed archive network coordination + - Recovery notifications to repository owners