laantungir/ngit-grasp
1
0
Fork 0
mirror of https://relay.ngit.dev/npub15qydau2hjma6ngxkl2cyar74wzyjshvl65za5k5rl69264ar2exs5cyejr/ngit-grasp.git synced 2026-10-05 23:18:24 +00:00
Code Issues Packages Projects Releases Wiki Activity
Files
f01f65d49cf5e1cbb78582a724515dfc5a6e83d5
ngit-grasp/b905-deletion-request-support.md
T

26 KiB
Raw Blame History

Deletion Request Support (NIP-09)

ID: b905

Status: Planning Complete, Ready for Implementation
Priority: Medium
Complexity: High (graph algorithms, multi-database coordination)
Estimated Timeline: 6 weeks (phased approach with full test coverage)

Problem Statement

ngit-grasp currently has no mechanism to handle NIP-09 deletion requests. Repository owners cannot remove their repositories from the relay, and there's no protection against "left-pad" scenarios where critical repositories are deleted, breaking dependent projects.

Objectives

  1. ✅ Respect deletion requests for repository announcements (kind 30617) deleted by address
  2. ✅ Archive git data before deletion with configurable retention (default 90 days)
  3. ✅ Cascade delete ALL dependent events (PRs, issues, patches, comments)
  4. ✅ Provide recovery mechanism via holding database
  5. ✅ Configurable "deletion request disrespector" mode for archival relays (prevents left-pad)
  6. ✅ Handle multi-maintainer repositories with graph-based retention algorithm

Design Decisions

Three-Database Architecture

  • Main Database: Live events actively served in queries
  • Holding Database: Archived events during retention window (separate DB file, same backend type)
  • Archive Filesystem: Compressed git data (.archive/ subdirectory)

Cascade Delete Strategy

Delete ALL dependent events when announcement is deleted (not just owner's events). Rationale:

  • Matches user expectation of "delete everything"
  • Orphaned PRs/issues confusing without context
  • Recovery available via holding database
  • Archival relays protect community work

Multi-Maintainer Handling

Graph-based retention algorithm:

  1. Archive deleting maintainer's git directory
  2. Re-evaluate all events through acceptance policy WITHOUT deleted announcement
  3. Build dependency graph showing retention reasons
  4. Detect circular dependencies
  5. Delete events that would fail acceptance

Configuration

  • deletion_request_disrespector (bool, default false) - Archival relay mode
  • archive_retention_secs (u64, default 7776000 = 90 days) - Configurable in seconds for testing

Implementation Phases

Phase 1: Core Deletion + Simple Cascade (Week 1)

Goal: Basic deletion working for single-maintainer repositories

Tasks:

  • Add config options to src/config.rs:
    • deletion_request_disrespector: bool
    • archive_retention_secs: u64 (default 7776000)
  • Update configuration files (CRITICAL - must update all 4):
    • src/config.rs - Config struct fields
    • docs/reference/configuration.md - Documentation
    • nix/module.nix - NixOS options
    • .env.example - Example with comments
  • Create holding database infrastructure:
    • src/database/holding.rs - Holding database wrapper
    • Support same backend types (LMDB/Memory/NostrDB)
    • Separate file: <relay_data_path>/holding-<backend>
  • Implement src/nostr/policy/deletion.rs:
    • DeletionPolicy::validate() - Parse kind 5, extract a/e tags
    • process_address_deletion() - Handle a tag deletions
    • Author pubkey matching validation
    • Check deletion_request_disrespector config
  • Integrate into src/nostr/builder.rs:
    • Add Kind::EventDeletion (5) case in admit_event()
    • Route to DeletionPolicy
    • If disrespector mode: store event, return early
  • Implement event dependency query:
    • query_dependent_events() - Recursive traversal
    • Find events with a tags referencing announcement
    • Find events with e tags referencing above events
    • Simple cascade (no graph complexity yet)
  • Move events between databases:
    • move_to_holding_database() - Atomic operation
    • Move announcement + all dependents
    • Store deletion timestamp, deletion event ID
    • Delete from main database
  • Tests:
    • Config loading and validation
    • Kind 5 parsing and validation
    • Author pubkey matching (accept/reject)
    • Disrespector mode behavior
    • Simple cascade deletion (single maintainer)
    • Events moved to holding DB correctly
    • Events no longer in main DB (queries return nothing)
    • Use 3-5 second retention for fast tests

Exit Criteria:

  • All tests passing with 100% coverage
  • Single-maintainer repository deletion fully working
  • Events properly moved between databases
  • Disrespector mode prevents deletion

Phase 2: Git Archival & Cleanup (Week 2)

Goal: Archive git data and implement cleanup task

Tasks:

  • Implement src/git/archive.rs:
    • archive_repository() - Create tar.gz of git directory
    • Archive path: .archive/<npub>/<identifier>-<timestamp>.tar.gz
    • create_archive_metadata() - Store metadata JSON
    • Compression using flate2 crate
  • Archive metadata structure:
    • Deletion timestamp
    • Deletion event ID
    • Maintainer pubkey
    • Identifier
    • Archive file path
    • Expiry timestamp (deletion_ts + retention_secs)
  • Integrate archival into deletion flow:
    • Archive git data BEFORE moving events to holding DB
    • Handle archive failures (rollback? log error?)
  • Background cleanup task:
    • Spawn tokio task in main.rs
    • Run daily (24-hour interval)
    • Also run on startup (catch-up for offline periods)
    • cleanup_archived_data():
      • Query holding DB for expired entries
      • Delete events from holding DB
      • Delete archive tar.gz files
      • Delete archive metadata
  • Tests:
    • Archive creation and compression
    • Archive metadata storage
    • Archive extraction (verify integrity)
    • Background cleanup with 5-second retention
    • Cleanup on startup (simulate offline period)
    • Disk space reclamation verification

Exit Criteria:

  • Git data properly archived before deletion
  • Background cleanup working reliably
  • No disk space leaks
  • All tests passing

Phase 3: Multi-Maintainer Graph Algorithm (Week 3)

Goal: Handle complex multi-maintainer deletion scenarios

Tasks:

  • Implement dependency graph builder:
    • build_event_graph() - Create directed graph
    • Nodes: Events
    • Edges: References (a/e/q tags)
    • Track retention reasons for each event
  • Re-evaluation engine:
    • reevaluate_events_without_announcement()
    • Query all events referencing deleted announcement
    • Run each through acceptance policy WITHOUT deleted announcement
    • Track which announcements/events make it acceptable
  • Graph traversal:
    • topological_traverse() - Start from announcements
    • Mark reachable events as "keep"
    • Mark unreachable events as "delete"
    • Configurable max depth (default 100)
  • Circular dependency detection:
    • Detect mutual references (A→B, B→A)
    • Mark both for deletion if no external anchor
  • Integration:
    • Replace simple cascade with graph algorithm
    • Handle edge cases (isolated subgraphs)
  • Tests:
    • Two maintainers, one deletes (events preserved)
    • Two maintainers, both delete (events deleted)
    • Circular dependencies (both deleted)
    • Complex reference graphs (3+ levels deep)
    • Max depth exceeded (logged warning)

Exit Criteria:

  • Multi-maintainer scenarios handled correctly
  • Graph algorithm thoroughly tested
  • Max depth configurable and tested
  • All tests passing

Phase 4: Recovery Mechanism (Week 4)

Goal: Allow owners to recover accidentally deleted repositories

Tasks:

  • Implement recovery detection:
    • check_for_recovery() in announcement processing
    • Query holding DB for matching identifier + pubkey
    • Check if within retention period
  • Git data restoration:
    • Extract tar.gz to temp directory
    • Verify integrity
    • Move to <git_data_path>/<npub>/<identifier>.git
  • Event restoration:
    • Query holding DB for all events
    • Re-run acceptance policy (should now pass)
    • Move from holding DB → main DB
    • Count restored events
  • Archive cleanup after recovery:
    • Delete archive tar.gz
    • Delete archive metadata
    • Remove holding DB entries
  • Response to client:
    • Success message with count: "Restored 47 events"
    • Or normal new repo: "New repository created"
  • Tests:
    • Full recovery workflow
    • Partial recovery (some events expired)
    • Recovery at edge of retention window
    • Recovery after retention expired (new repo)
    • Corrupt archive (graceful failure)

Exit Criteria:

  • Recovery fully functional
  • Edge cases handled gracefully
  • User-friendly response messages
  • All tests passing

Phase 5: Extended Cascade Deletion (Week 5)

Goal: Complete cascade deletion for all event types

Tasks:

  • Extend cascade delete to all NIP-34 event types:
    • Patches (1617) - tag repos via a
    • Issues (1621) - tag repos via a
    • PR Updates (1619) - tag PRs via e
    • Status events (1630-1633) - tag issues/PRs via e
  • Update dependency graph to include all types
  • Tests for each event type:
    • Patches cascade delete
    • Issues cascade delete
    • PR Updates cascade delete
    • Status events cascade delete
    • Mixed event types (comprehensive)

Exit Criteria:

  • All NIP-34 event types properly cascade deleted
  • Comprehensive test coverage
  • All tests passing

Phase 6: Analysis & Edge Cases (Week 6)

Goal: Production hardening and edge case analysis

Tasks:

  • Background cleanup task investigation:
    • Daily cleanup doesn't work well with 3-second retention tests
    • Design alternative: trigger-based cleanup for tests?
    • Or: configurable cleanup interval (separate from retention)?
    • Test with both short intervals (tests) and daily (production)
  • rust-nostr deletion behavior investigation:
    • Check if nostr-relay-builder automatically deletes events tagged in kind 5
    • Check if it stops serving events tagged in deletion requests
    • If YES: need to override/disable this behavior when deletion_request_disrespector = true
    • Document any hooks or callbacks we need to implement
    • Ensure archival mode truly ignores deletions at relay library level
  • Author validation enforcement:
    • Verify we only honor deletion requests where author matches deleted event author
    • Add comprehensive tests for author mismatch rejection
    • Document this requirement clearly in NIP-11 and README
  • Max depth edge case analysis:
    • Document scenarios where depth limit matters
    • Recommend alternative approaches if needed
    • Add configuration guidance
  • Large-scale scenario testing:
    • Repository with 1000+ PRs/issues
    • Deep dependency chains (50+ levels)
    • Performance profiling
    • Memory usage analysis
  • Race condition investigation:
    • Deletion during active sync
    • Concurrent deletions of shared repo
    • Lock strategy for git archival
    • Document mitigation strategies
  • Blacklist retroactive deletion investigation:
    • Should adding to blacklist move existing repos to 90-day holding area?
    • Decision: YES - use same archive mechanism as deletion requests
    • Detect on startup: scan main DB for repos matching blacklist
    • Move to holding area with metadata marking blacklist trigger
    • Same cascade delete logic as NIP-09 deletions
    • Archive git data before moving to holding area
  • Blacklist + disrespector mode interaction:
    • Decision: deletion_request_disrespector does NOT prevent blacklist deletion
    • Blacklist is moderation/operational decision, not user-initiated deletion
    • Archival relays can still blacklist spam/malware/abuse
    • Only NIP-09 user deletions are ignored in disrespector mode
  • Blacklist removal recovery:
    • Should removing from blacklist auto-restore from holding DB?
    • Or require manual operator intervention?
    • What about the git data archive?
    • Design recovery flow for unblacklisted repos
  • Manual ejection from holding area:
    • Operator needs ability to force-delete from holding area before expiry
    • Use case: large repos consuming excessive storage
    • Use case: confirmed malware/abuse that shouldn't be recoverable
    • Design mechanism: admin CLI command? config flag? database operation?
    • Should manual ejection delete git archive immediately or wait for cleanup?
    • Log manual ejections for audit trail
  • Delayed archival strategy design (defer full implementation):
    • Activity threshold evaluation algorithm
      • Count issues/PRs/patches (< 5 = low activity)
      • Count unique creator pubkeys (1-2 = low activity)
      • Both conditions evaluated (either triggers immediate archival)
    • Grace period implementation (2 minutes default)
      • Prevent accidental deletions
      • Allow user to publish new announcement to cancel
      • Handle cancellation during grace period
    • Notification issue creation
      • Sign issue with repository owner pubkey
      • Content: deletion warning, 14-day timeline, affected content list
      • Store notification issue reference for cleanup
    • Archive delay implementation (10 days default)
      • Community response window
      • Scheduled task system for delayed execution
      • Cancellation check before archival
    • Configuration options:
      • deletion_notification_delay_secs (default 120)
      • deletion_archive_delay_secs (default 864000)
      • deletion_activity_threshold (default 5)
      • deletion_creator_threshold (default 3)
    • Add config to all 4 required sources (CRITICAL):
      • src/config.rs
      • docs/reference/configuration.md
      • nix/module.nix
      • .env.example
    • Timeline calculation and validation
      • Total for active repos: 2min + 10days + 90days ≈ 100 days
      • Low-activity repos: immediate + 90 days
    • Testing considerations
      • Override delays for test scenarios (5-10 seconds)
      • Test immediate vs delayed paths
      • Test cancellation during both grace and delay periods
  • Metrics implementation:
    • ngit_deletion_requests_total
    • ngit_deletion_requests_processed
    • ngit_deletion_requests_immediate (new - low activity)
    • ngit_deletion_requests_delayed (new - active repos)
    • ngit_deletion_notifications_created (new)
    • ngit_deletion_requests_cancelled (new)
    • ngit_blacklist_deletions_total
    • ngit_holding_database_events
    • ngit_holding_database_size_bytes
    • ngit_archive_files_total
    • ngit_archive_size_bytes
    • ngit_recoveries_total
    • ngit_permanent_deletions_total
    • ngit_manual_ejections_total
  • Documentation:
    • Update docs/explanation/deletion-requests.md with findings
    • Add blacklist deletion behavior documentation
    • Add manual ejection mechanism documentation
    • Add delayed archival strategy documentation
    • Add edge case documentation
    • Performance tuning guide

Exit Criteria:

  • Edge cases documented
  • Performance acceptable for production
  • Metrics implemented and tested
  • Documentation complete

Phase 7: Integration & Final Testing (Week 7)

Goal: End-to-end testing and documentation

Tasks:

  • End-to-end integration tests:
    • Full deletion workflow (all phases)
    • Multi-maintainer + recovery + cleanup
    • Disrespector mode comprehensive test
  • grasp-audit compliance tests:
    • NIP-09 validation
    • Event re-submission after deletion (rejected)
    • Deletion request event storage
    • Archival mode behavior
  • NIP-11 relay information updates:
    • Add "deletion" to supported NIPs array (only if deletion_request_disrespector = false)
    • If disrespector mode enabled, do NOT advertise NIP-09 support
    • Update src/http/nip11.rs to conditionally include based on config
  • README.md updates:
    • Add NIP-09 deletion request support to feature list
    • Document cascade deletion behavior
    • Link to explanation document
    • Update "Delete Events" roadmap section (mark as complete)
  • Architecture documentation updates:
    • Add deletion request section to docs/explanation/architecture.md
    • Document cascade deletion strategy
    • Reference deletion-requests.md for details
  • User documentation:
    • How-to: "Deleting a Repository"
    • How-to: "Running an Archival Relay"
    • How-to: "Recovering Deleted Repositories"
  • Reference documentation:
    • Configuration reference (update all 4 sources!)
    • NIP-09 feature matrix
    • Database schema documentation
  • Code review and cleanup:
    • Remove debug logging
    • Add production logging (info level)
    • Code cleanup and refactoring
    • Performance optimizations

Exit Criteria:

  • All integration tests passing
  • Documentation complete and reviewed
  • Code ready for production
  • Ready to merge

Technical Architecture

See docs/explanation/deletion-requests.md for comprehensive architecture documentation.

Key Components:

  • src/nostr/policy/deletion.rs - Deletion request validation
  • src/git/archive.rs - Git repository archival
  • src/database/holding.rs - Holding database wrapper
  • Background cleanup task in main.rs

Data Flow:

Kind 5 Event → Validate → Query Dependents → Archive Git → Move Events → Delete from Main DB
                                                    ↓
                                         Background Task (daily)
                                                    ↓
                              Expired? → Delete from Holding DB → Delete Archive

Configuration Updates Required

CRITICAL: Must update all 4 configuration sources (per AGENTS.md):

  1. ✅ src/config.rs
  2. ✅ docs/reference/configuration.md
  3. ✅ nix/module.nix
  4. ✅ .env.example

Testing Notes

  • Use 3-5 second retention for tests (not 90 days!)
  • Real temp directories (not mocked filesystem)
  • Full test coverage at each phase before proceeding
  • Integration tests after each phase completion
  • Background cleanup timing: Daily cleanup doesn't work with 3-second tests - investigate in Phase 6

Critical Requirements

Author Validation (NIP-09 Spec):

  • We will ONLY honor deletion requests where the deletion request author matches the deleted event author
  • This is a fundamental security requirement
  • Must be clearly documented in NIP-11, README, and user-facing docs

rust-nostr Behavior:

  • Need to verify nostr-relay-builder doesn't automatically process deletions
  • If it does, must override this when deletion_request_disrespector = true
  • Investigate in Phase 6

Open Questions & Future Investigation

Deferred to Phase 6 Analysis:

  • Background cleanup timing strategy (daily vs configurable interval)
  • rust-nostr automatic deletion behavior and override mechanisms
  • Author validation comprehensive test coverage
  • Max depth edge cases and alternatives
  • Large-scale performance characteristics (1000+ events)
  • Race conditions during concurrent operations
  • Lock strategies for git repository access

Future Enhancements:

  • GRASP-05 archive mode integration
  • Selective disrespect (based on popularity/criteria)
  • Distributed archive network coordination
  • Recovery notifications to repository owners

Success Criteria

  1. ✅ Repository owners can delete via NIP-09
  2. ✅ Git data archived for configurable retention
  3. ✅ All dependent events cascade deleted
  4. ✅ Recovery mechanism working
  5. ✅ Archival mode prevents left-pad
  6. ✅ Multi-maintainer scenarios handled
  7. ✅ Full test coverage (unit, integration, audit)
  8. ✅ Documentation complete
  9. ✅ Production-ready metrics

Progress

2026-01-13 [Session 14:30]

  • Design Update: Added delayed archival strategy for active repositories
    • Immediate archival for repos with < 5 items OR 1-2 pubkeys
    • Delayed archival for active repos: 2min grace + 10 day delay + notification issue
    • Total timeline: ~100 days for active repos (2min + 10d + 90d retention)
    • Added 4 new configuration options: notification delay, archive delay, activity threshold, creator threshold
  • Decision: Defer full implementation to Phase 6 as design consideration
    • Core deletion flow (Phases 1-5) remains simpler with immediate archival
    • Delayed strategy adds significant complexity (scheduled tasks, cancellation, notifications)
    • Can evaluate necessity after basic deletion working
  • Updated: Both docs/explanation/deletion-requests.md and issue with new strategy
  • Next: Commit changes and continue with Phase 1 implementation planning

2026-01-14 [Session 10:00]

  • Started Phase 1 Implementation: Created 3 parallel phase worktrees
    • Phase 1A (b905-phase1a-config): Configuration setup - add config options to all 4 required sources
    • Phase 1B (b905-phase1b-holding-db): Holding database infrastructure - separate DB for archived events
    • Phase 1C (b905-phase1c-deletion-policy): Deletion policy validation and integration into event admission
  • Strategy: Parallel implementation using subagents, each in dedicated worktree
  • Next: Launch subagents for each phase, then merge back using finish-phase skill

2026-01-14 [Session 10:30]

  • Completed Phase 1A (Configuration): All 4 configuration sources updated
    • Added deletion_request_disrespector (bool, default false) - archival relay mode
    • Added archive_retention_secs (u64, default 7776000 = 90 days) - retention period
    • Updated src/config.rs with new fields and test config
    • Updated docs/reference/configuration.md with comprehensive documentation
    • Updated nix/module.nix with NixOS options and environment mappings
    • Updated .env.example with commented examples and explanations
    • All changes compile successfully
    • 4 commits created following AGENTS.md guidelines
  • Completed Phase 1B (Holding Database): Holding database infrastructure implemented
    • Created src/database/holding.rs with HoldingDatabase wrapper
    • Supports same backend types as main database (LMDB/Memory/NostrDB)
    • Separate file path: <relay_data_path>/holding-<backend>
    • Methods: new(), store_event(), query_events(), delete_event(), query_expired()
    • Deletion metadata stored as custom tags: deletion-ts, deletion-event, expiry-ts
    • Created src/database/mod.rs and integrated into src/lib.rs
    • All 5 tests passing (memory DB, LMDB, store/query, expired events, metadata preservation)
    • Code compiles successfully with only unused import warnings
  • Completed Phase 1C (Deletion Policy): Deletion request validation and integration complete
    • Created src/nostr/policy/deletion.rs with DeletionPolicy struct
    • Implemented validate() method: parses kind 5, extracts e/a tags, validates author matching
    • Implemented validate_event_author() and validate_address_author() for NIP-09 compliance
    • Implemented process_address_deletion() for kind 30617 repository announcement deletions
    • Added deletion_request_disrespector config check (store event but ignore deletion)
    • Exported DeletionPolicy and DeletionResult from src/nostr/policy/mod.rs
    • Integrated into src/nostr/builder.rs: added Kind::from(5) case in admit_event()
    • Added handle_deletion() method routing to DeletionPolicy
    • Uses nostr-sdk 0.43 API correctly (fields not methods, .as_slice(), single Filter in query)
    • All 337 unit tests passing
    • 2 commits created following AGENTS.md guidelines
  • All 3 Phases Merged: Successfully merged all phases back to parent worktree using finish-phase skill
    • Phase 1A: 4 commits (config struct, docs, nix module, env example)
    • Phase 1B: 1 commit (holding database infrastructure)
    • Phase 1C: 2 commits (deletion policy, integration)
    • All merges used rebase + fast-forward (clean linear history, no merge commits)
    • All 342 tests passing (5 new tests from holding database)
  • Next: Implement remaining Phase 1 tasks:
    • Event dependency query (recursive traversal to find dependent events)
    • Database move operations (move_to_holding_database() for atomic event movement)
    • Comprehensive tests for deletion validation and cascade behavior

2026-01-14 [Session 11:00]

  • Completed Event Dependency Queries: Created src/nostr/policy/deletion_ops.rs
    • query_dependent_events() - Recursive traversal to find all dependent events
    • Queries events with a tags referencing addresses (kind 30617 announcements)
    • Queries events with e tags referencing other events (PRs, issues, patches, status)
    • Simple cascade for Phase 1 (graph algorithm deferred to Phase 3)
    • Helper functions: query_events_by_address_tag(), query_events_by_event_tag()
    • Tag checking: has_address_tag(), has_event_tag()
  • Completed Database Migration: Implemented move_to_holding_database()
    • Atomic operation to move events from main DB to holding DB
    • Creates DeletionMetadata with deletion timestamp, event ID, and expiry
    • Stores events in holding DB with metadata as custom tags
    • Logs warnings about API limitation (can't delete from main DB yet)
    • Returns count of successfully moved events
  • Integrated Deletion Processing: Updated src/nostr/builder.rs
    • Create holding database in create_relay() (unless disrespector mode)
    • Add holding_database field to Nip34WritePolicy
    • Implement full deletion processing in handle_deletion():
      • Query dependent events using query_dependent_events()
      • Move all events to holding database with move_to_holding_database()
      • Log success/failure with event counts
      • Handle missing holding database gracefully
    • All 344 tests passing
  • Next: Write comprehensive integration tests for Phase 1 deletion flow

References

  • Explanation: docs/explanation/deletion-requests.md
  • NIP-09 Spec: /persistent/dcdev/clones/nips/09.md
  • Roadmap: README.md lines 198-206
  • AGENTS.md: Configuration sync requirements
Reference in New Issue View Git Blame Copy Permalink