ngit-grasp/closed/3ca0-announcements-purgatory.md

# Announcements Purgatory System

**ID:** 3ca0

## Problem

Currently, acceptable announcements that don't replace existing announcements are immediately accepted and served to clients. We need a "purgatory" system where new announcements are held in a pending state until they become relevant (when a corresponding state event arrives).

This prevents serving announcements for repositories that may never receive state events, while still maintaining the git infrastructure in case they do.

## Requirements

### 1. Purgatory Instead of Immediate Acceptance
- Acceptable announcements that do not replace an existing announcement should go to purgatory instead of being immediately accepted and served
- The bare git repo should still be created when an announcement enters purgatory
- Announcements should not be served to clients while in purgatory

### 2. Expiry and Cleanup
- If an announcement expires while in purgatory, the bare git repo should be deleted
- Need to track expiry times for announcements in purgatory
- Background cleanup mechanism to handle expired purgatory announcements

### 3. State Event Interaction
- When a state event arrives, any announcements in purgatory should have their expiry extended to match the state event's expiry
- This keeps purgatory announcements alive as long as they might become relevant
- Need to determine when/how announcements transition from purgatory to active

### 4. State Event Acceptability
- State event acceptability checks must consider announcements in purgatory
- A state event should be rejected if it would conflict with announcements in purgatory (not just active announcements)
- Validation logic needs to check both active and purgatory announcements

## Plan

- [x] Phase 1: Design
  - Design doc: `docs/explanation/announcements-purgatory-design.md`
  - Implementation doc: `docs/explanation/announcements-purgatory-implementation.md`

- [x] Phase 2: Core implementation (commit 1d09e4b)
  - AnnouncementPurgatoryEntry type and DashMap store
  - Route new announcements to purgatory (replacements skip)
  - Promote on git data arrival (process_purgatory_announcements)
  - Authorization checks purgatory (fetch_repository_data_with_purgatory)
  - State policy uses purgatory for maintainer validation
  - Cleanup task handles announcement expiry
  - Updated count()/cleanup() to 3-tuples

- [ ] Phase 3: Implement SyncLevel and fix sync integration (ROOT CAUSE of test failure)
  - The sync module doesn't fetch state events for purgatory announcements
    because they're not saved to DB and thus never "confirmed" as repos
  - The fix per design doc (decision #6) is SyncLevel, NOT treating
    purgatory as fully confirmed. Full confirmation would over-fetch
    L2/L3 events (PRs, issues) that get rejected anyway.
  - Add SyncLevel enum (StateOnly vs Full) to RepoSyncNeeds
  - When announcement enters purgatory, register in sync index with
    SyncLevel::StateOnly (only fetches kind 30618 state events)
  - On promotion, upgrade sync level to Full
  - Modify filter building in src/sync/filters.rs to partition repos
    by sync level and build appropriate filters
  - Revert the partial fix in commit 1d09e4b that treats
    ProcessResult::Purgatory as confirmed in pending_sync_index
    (wrong approach - would trigger full L2/L3 sync)
  - Keep the hot cache invalidation for rejected state events (correct)
  - This should fix test_archive_read_only_creates_bare_repo

- [ ] Phase 4: Announcement persistence (save/restore)
  - save_to_disk() and restore_from_disk() skip announcement_purgatory
  - PurgatoryState struct has no announcement field
  - Need SerializableAnnouncementPurgatoryEntry type
  - Need save/restore logic with Instant offset conversion
  - Need persistence tests (purgatory_persistence.rs currently asserts 0)

- [ ] Phase 5: Remaining design features
  - Soft expiry two-phase - delete bare repo but retain event for 24h,
    allow revival on state event arrival
  - Expiry extension - extend_announcement_expiry() exists but is never
    called from state policy or git auth
  - Newer announcement replaces older in purgatory (not just DB check)
  - Service change clears purgatory entry
  - Deletion event (kind 5) removes from purgatory

- [ ] Phase 6: Testing
  - Fix test_archive_read_only_creates_bare_repo
  - Add persistence round-trip tests for announcements
  - Test new functionality from Phase 5 as implemented

## Progress

### 2026-01-23 [Session 00:00]
- Started: Created issue with requirements and initial plan
- Created worktree: worktrees/3ca0-announcements-purgatory
- Ready to begin implementation

### 2026-01-23 [Design Session]
- Conducted extensive architectural review with multiple iterations
- Clarified requirements based on user feedback:
  - Problem: Prevent serving empty bare repos indefinitely (misleads clients)
  - Trigger: Git data arrival promotes announcements (not state events)
  - Indexing: Must use (pubkey, identifier) tuple (identifier not unique)
  - Expiry extension: Two places (state event arrival AND git auth)
  - Edge cases: Newer announcements replace older, service list changes
- Created design document: `docs/explanation/announcements-purgatory-design.md`
- OpenCode conversation: https://opencode.ai/c/01JJBXPFVVQVX5RVQF2KQBQZJR

### 2026-02-05 [Design Completion]
- Completed high-level design with user review
- Key decisions finalized:
  - **Sync integration**: Purgatory announcements sync state events only (SyncLevel::StateOnly)
  - **Authorization**: Must check purgatory announcements in addition to DB
  - **Soft expiry**: Delete bare repo but retain event for 24h (allows revival on state event)
- Created implementation details document: `docs/explanation/announcements-purgatory-implementation.md`
- Design documents committed: 603c87f

### 2026-02-13 [Core Implementation]
- Implemented core announcement purgatory (commit 1d09e4b)
- Happy path works: announcement -> purgatory -> git push -> promotion -> served
- All purgatory integration tests pass (44/44)
- One existing test broken: test_archive_read_only_creates_bare_repo
- Root cause identified: sync module does not treat purgatory announcements
  as "confirmed repos", so per-repo sync never fetches state events for them.
  The sync flow is: fetch announcements via negentropy -> save to DB marks them
  "confirmed" -> for each confirmed repo, fetch state/PR events. Purgatory
  announcements skip the DB save, so they're never confirmed.
- Partial fix applied: ProcessResult::Purgatory now counts as confirmed in
  pending_sync_index, and announcements entering purgatory invalidate
  previously-rejected state events from the hot cache
- Also identified: announcement persistence not implemented (save_to_disk
  and restore_from_disk skip announcement_purgatory entirely)

## Notes

- This feature prevents resource waste on announcements that never receive git data
- Maintains quick response time for repositories that do become active
- Soft expiry solves the failed_events problem without permanent blocklisting