fix(purgatory): preserve events during active git sync

Motivation: the archive cold-start burn-in on gitnostr.com expired eight synced state events at the 30-minute boundary. At least the trezor-firmware event was a false expiry: its healthy gitnostr.com source advertised required objects while a roughly 1.3 GiB clone was still actively building packs. The fixed wall-clock sweep overtook valid Git work and permanently excluded the event from later relay sync.

Approach: make the purgatory expiry sweep consult the existing per-identifier sync queue. Matching announcements, state events, and PR events are retained only while sync_identifier is concretely in progress. When that operation completes, normal processing promotes satisfied events; otherwise the next sweep may expire them.

Correctness: all event types for an identifier share the same active-work guard, preventing announcement cleanup from deleting a repository beneath its state or PR fetch. Queue membership, scheduled retries, and backoff do not extend retention, so unreachable or inconsistent repositories still age out. A multi-repository PR is retained while any referenced repository has active work.

Excluded scope: the nominal 30-minute and announcement 24-hour soft-expiry windows are unchanged, as are retry cadence, Git concurrency, and unhealthy-peer classification.

Validation: nix develop -c cargo test --lib (667 passed). The focused regression expires an announcement, state event, and PR synthetically, verifies active Git work protects all three, then verifies idle/backoff state permits ordinary expiry.
This commit is contained in:
DanConwayDev
2026-08-08 14:19:19 +00:00
parent 9ab07840fc
commit 5b19c2ff36
4 changed files with 135 additions and 2 deletions
+5
View File
@@ -20,6 +20,11 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
### Fixed
- Keep purgatory announcements, state events, and PR events alive while a
concrete background Git sync for their repository is actively running. A
large healthy clone can exceed the nominal 30-minute purgatory window;
queued and backoff-only work remains eligible for normal expiry so broken
repositories cannot extend retention indefinitely.
- Preserve purgatory and rejected-event recovery across abrupt termination.
Their snapshots now remain after restore and are atomically replaced every
60 seconds, bounding crash loss to one interval instead of consuming the
+2
View File
@@ -307,6 +307,8 @@ pub struct Purgatory {
5. **Automatic Expiry**: 30-minute default expiry, extensible during processing
- Background cleanup task runs every 60 seconds
- Removes expired entries from all stores
- Defers matching expiry while a concrete background Git sync is actively
running; queued/backoff work receives no extension
#### Data Types
+10
View File
@@ -348,6 +348,8 @@ fetch_repository_data_with_purgatory() checks DB + purgatory
The protocol specifies 30-minute expiry for announcements. We implement a two-phase soft expiry:
**Phase 1 — Initial 30-minute expiry (`soft_expired == false`):**
- Defer the expiry sweep while a concrete background Git sync for the
repository is actively running
- Delete the bare git repo (frees disk space, respects protocol expiry)
- Set `soft_expired = true`
- Extend `expires_at` by 24 hours (`SOFT_EXPIRY_EXTENDED`)
@@ -378,6 +380,14 @@ The 30-minute purgatory timer is reset (extended) in three scenarios:
All three call `purgatory.extend_announcement_expiry(owner, identifier, 1800s)`.
Background Git synchronization does not reset the 30-minute deadline. Instead,
the expiry sweep skips matching announcements, state events and PR events only
while `sync_identifier` is actively running. This lets a large but healthy
clone finish without allowing an unavailable repository to live forever:
queue entries waiting for their next attempt or in backoff remain eligible for
normal expiry. When active work finishes, it either promotes the satisfied
events or the next sweep expires any still-unsatisfied entries.
### Purgatory Lifecycle
```
+118 -2
View File
@@ -344,6 +344,35 @@ impl Purgatory {
false
}
/// Whether a concrete background Git sync is currently running for an
/// identifier.
///
/// Queue membership alone is deliberately insufficient: entries waiting
/// for their next backoff attempt must still expire normally when remote
/// Git data remains unavailable. Only the interval in which
/// `sync_identifier` is actively working protects matching purgatory
/// entries from the expiry sweep.
fn identifier_sync_in_progress(&self, identifier: &str) -> bool {
self.sync_queue
.get(identifier)
.is_some_and(|entry| entry.in_progress)
}
/// Whether any repository referenced by an event has active Git sync.
fn event_sync_in_progress(&self, event: &Event) -> bool {
event.tags.iter().any(|tag| {
let values = tag.clone().to_vec();
if values.len() < 2 || values[0] != "a" || !values[1].starts_with("30617:") {
return false;
}
values[1]
.splitn(3, ':')
.nth(2)
.is_some_and(|identifier| self.identifier_sync_in_progress(identifier))
})
}
/// Get a reference to the sync queue (for the sync loop).
pub fn sync_queue(&self) -> &Arc<DashMap<String, SyncQueueEntry>> {
&self.sync_queue
@@ -1143,7 +1172,9 @@ impl Purgatory {
let expired_announcements: Vec<(PublicKey, String, PathBuf, EventId, bool)> = self
.announcement_purgatory
.iter()
.filter(|entry| entry.value().expires_at <= now)
.filter(|entry| {
entry.value().expires_at <= now && !self.identifier_sync_in_progress(&entry.key().1)
})
.map(|entry| {
let key = entry.key();
let v = entry.value();
@@ -1224,6 +1255,10 @@ impl Purgatory {
// Remove expired state events and mark them as expired
self.state_events.retain(|identifier, entries| {
if self.identifier_sync_in_progress(identifier) {
return true;
}
let original_len = entries.len();
// Log and collect expired entries before removing
@@ -1267,7 +1302,14 @@ impl Purgatory {
let expired_prs: Vec<_> = self
.pr_events
.iter()
.filter(|entry| entry.value().expires_at <= now)
.filter(|entry| {
let value = entry.value();
value.expires_at <= now
&& !value
.event
.as_ref()
.is_some_and(|event| self.event_sync_in_progress(event))
})
.map(|entry| {
let pr_entry = entry.value();
let event_id_str = entry.key().clone();
@@ -2146,6 +2188,80 @@ fn test_cleanup_preserves_non_expired_entries() {
assert_eq!(pr_count, 1);
}
#[test]
fn cleanup_defers_expiry_only_while_repository_sync_is_active() {
let purgatory = Purgatory::new(PathBuf::new());
let keys = Keys::generate();
let identifier = "large-repository";
let announcement = EventBuilder::new(Kind::GitRepoAnnouncement, "")
.finalize(&keys)
.unwrap();
purgatory.add_announcement(
announcement,
identifier.to_string(),
keys.public_key(),
PathBuf::from("/path/that/does/not/exist"),
HashSet::new(),
);
let state = EventBuilder::new(Kind::RepoState, "")
.finalize(&keys)
.unwrap();
purgatory.add_state(state, identifier.to_string(), keys.public_key(), true);
let pr = EventBuilder::new(Kind::GitPatch, "")
.tag(Tag::custom(
"a",
vec![format!("30617:{}:{identifier}", keys.public_key().to_hex())],
))
.finalize(&keys)
.unwrap();
purgatory.add_pr(pr, "pr-event".to_string(), "commit".to_string(), true);
let expired_at = Instant::now() - Duration::from_secs(1);
purgatory
.announcement_purgatory
.get_mut(&(keys.public_key(), identifier.to_string()))
.unwrap()
.expires_at = expired_at;
purgatory.state_events.get_mut(identifier).unwrap()[0].expires_at = expired_at;
purgatory.pr_events.get_mut("pr-event").unwrap().expires_at = expired_at;
purgatory
.sync_queue
.get_mut(identifier)
.unwrap()
.in_progress = true;
let removed = purgatory.cleanup();
assert_eq!(removed, (0, 0, 0));
assert!(
!purgatory
.find_announcement(&keys.public_key(), identifier)
.unwrap()
.soft_expired
);
assert_eq!(purgatory.count(), (1, 1, 1));
assert_eq!(purgatory.expired_count(), 0);
// An entry merely waiting in the queue or in backoff is not active work.
// Once the concrete Git operation ends, an unsatisfied event may expire.
purgatory
.sync_queue
.get_mut(identifier)
.unwrap()
.in_progress = false;
let removed = purgatory.cleanup();
assert_eq!(removed, (0, 1, 1));
assert!(
purgatory
.find_announcement(&keys.public_key(), identifier)
.unwrap()
.soft_expired
);
assert_eq!(purgatory.expired_count(), 2);
}
#[test]
fn test_cleanup_mixed_expired_and_fresh() {
use std::time::Duration;