Files
ngit-grasp/src/grasp06/cleanup.rs
T
DanConwayDev 629be17e64 fix(git): stage unsigned uploads until a signed event accepts them
A push to refs/nostr/<event-id> is accepted before its PR event is known,
and its objects went straight into the identifier family. The family is
never garbage-collected, so anyone could fill permanent storage without
signing anything, and an upload whose event never arrived stayed forever.

Approach

Route by what push authorization already decides. A ref named by a signed
State or PR event, accepted or in purgatory, is received into the family as
before. A refs/nostr ref with no event, or only a placeholder, is received
into the view's own object directory. Pre-validation now reports whether a
match was signed, because a placeholder match was indistinguishable from a
stored event.

Accepting a PR event first promotes its tip: fetch the history from the
view into the family, check the family alone holds it down to existing
retained roots, then install the usual retained and base roots. On failure
the event is rejected and its placeholder kept, so the upload expires
normally and the event can be sent again.

While a view holds staged objects every push to it is staged, because the
view advertises pending refs and a client may omit objects only staging
holds. Signed tips of such a push are recorded as owed before Git runs and
promoted when it finishes. Compaction refuses to run while anything is
owed: rollback after a State deletion needs history no ref names, so a
missing ref never proves history is disposable. Objects a view holds before
it is first staged are moved into the family.

Staging is reclaimed with git repack -a -d -l and git prune. Git can
install a ref whose parent a concurrent repack removed (see
tests/git_cruft_concurrency.rs), so compaction takes the family write lease
that every push already holds. It waits at most 250ms and retries with
backoff. One worker handles requests from pushes, promotions and ref
deletions, and reads the persistent registry at startup.

The /prs/ handler now releases the family lease before post-push
processing, as the standard handler does. Promotion of a waiting PR event
re-enters the family and would otherwise deadlock.

Assumptions

- Views and their family are on one filesystem; moving pre-existing objects
  uses hard links.
- Retained roots are complete. A damaged root fails promotion and is left
  to the integrity pass.
- One server process per storage root, as the family lease already assumes.

Excluded

- Storage quotas. Staging bounds how long an unsigned upload is kept, not
  its size.
- A pack from a signed push is stored whole. A signer can make any object
  reachable from their own tip, so filtering it would protect nothing.
- Fetches take no lease. A fetch of a pending ref that expires while being
  served may fail.
- Archives store a view as it is; restoring one imports staged objects.
- State acceptance, rollback and purgatory sync are unchanged.

Validation

- nix develop -c cargo test --lib git::staging: 13 passed, covering
  reclamation, a pending sibling keeping its history, promotion, refusal of
  incomplete history, owed history surviving ref deletion, restart recovery
  and a busy family.
- nix develop -c cargo test --test pending_upload_staging: 4 end-to-end
  tests passed, including an unsigned upload across a relay crash and a
  signed push that omits objects only staging holds.
- cargo clippy --workspace --all-targets -- -D warnings and cargo fmt
  --check were clean.

Assisted-by: Claude Fable 5.1
2026-09-29 10:21:41 +00:00

165 lines
6.2 KiB
Rust

//! Startup recovery for the GRASP-06 `/prs/` subtree.
//!
//! The inline cleanup paths in [`crate::grasp06::receive`],
//! [`crate::nostr::policy::pr_event`], and [`crate::purgatory::Purgatory::cleanup`]
//! remove `/prs/<submitter>/<identifier>.git` directories the moment they
//! go zero-ref while the process is running. They cannot, however, deal
//! with directories left zero-ref by a previous run:
//!
//! - A crash between writing a ref and the end-of-push cleanup.
//! - A crash between [`crate::git::delete_ref`] and `remove_dir_all` in
//! one of the off-push cleanup paths.
//! - A clean shutdown with a scoped placeholder still in memory whose
//! matching event then never arrives in the next run, after the
//! purgatory state has been dropped or aged out.
//!
//! Without recovery the bare directory and any dangling refs persist
//! indefinitely. The standalone `cleanup-empty-repos` CLI tool
//! explicitly skips `/prs/` (see
//! [`crate::grasp06::paths::is_prs_repo_path`]) because its event-driven
//! model does not apply, so there is no operational lifeline either.
//!
//! [`scan_on_startup`] walks `<git_data_path>/prs/<hex>/<id>.git` once,
//! before the HTTP server starts accepting requests, and removes any
//! bare repository with zero refs. At startup nothing is in flight, so
//! the [`PrsPathState`] mutex and `in_flight` counter are unnecessary —
//! a direct filesystem scan is safe and the same `list_refs` /
//! `remove_dir_all` shape the runtime cleanup paths use applies.
//!
//! Empty `<hex>/` parent directories are removed too so `/prs/` does not
//! accumulate submitter dirs over time.
//!
//! [`PrsPathState`]: crate::grasp06::receive::PrsPathState
use std::path::Path;
use nostr_sdk::prelude::*;
use tracing::{debug, info, warn};
use crate::git::list_refs;
use crate::grasp06::paths::prs_base_path;
/// Walk `<git_data_path>/prs/` once and remove any bare repository
/// directory with zero refs, plus any submitter directory left empty as
/// a result. Returns `(repos_removed, submitter_dirs_removed)`.
///
/// Must be called *before* the HTTP server starts accepting requests —
/// the scan does no locking and assumes no concurrent writers to the
/// `/prs/` subtree.
pub fn scan_on_startup(git_data_path: &Path) -> (usize, usize) {
let base = prs_base_path(git_data_path);
let hex_entries = match std::fs::read_dir(&base) {
Ok(entries) => entries,
Err(e) if e.kind() == std::io::ErrorKind::NotFound => return (0, 0),
Err(e) => {
warn!(
"/prs/ startup scan: failed to read {}: {}",
base.display(),
e
);
return (0, 0);
}
};
let mut repos_removed = 0usize;
let mut submitter_dirs_removed = 0usize;
for hex_entry in hex_entries.flatten() {
let hex_path = hex_entry.path();
if !hex_path.is_dir() {
continue;
}
let submitter_hex = hex_entry.file_name().to_string_lossy().into_owned();
if PublicKey::from_hex(&submitter_hex).is_err() {
// Skip directories whose name is not a valid pubkey — these
// shouldn't exist under /prs/, but we don't want to delete
// something we don't recognise.
debug!(
"/prs/ startup scan: skipping non-hex entry {}",
hex_path.display()
);
continue;
}
let id_entries = match std::fs::read_dir(&hex_path) {
Ok(entries) => entries,
Err(e) => {
warn!(
"/prs/ startup scan: failed to read {}: {}",
hex_path.display(),
e
);
continue;
}
};
for id_entry in id_entries.flatten() {
let repo_path = id_entry.path();
if !repo_path.is_dir() {
continue;
}
// Must end in `.git` to match the on-disk shape produced by
// `prs_repo_path` — anything else is suspicious; leave it
// alone.
if repo_path
.file_name()
.and_then(|s| s.to_str())
.is_none_or(|s| !s.ends_with(".git"))
{
continue;
}
match list_refs(&repo_path) {
// Signed history still owed to the family outlives its refs.
Ok(refs) if refs.is_empty() && crate::git::staging::owes_history(&repo_path) => {}
Ok(refs) if refs.is_empty() => {
if let Err(e) = std::fs::remove_dir_all(&repo_path) {
warn!(
"/prs/ startup scan: failed to remove zero-ref repo {}: {}",
repo_path.display(),
e
);
} else {
debug!(
"/prs/ startup scan: removed zero-ref repo {}",
repo_path.display()
);
repos_removed += 1;
}
}
Ok(_) => {
// Has refs — leave alone.
}
Err(e) => {
warn!(
"/prs/ startup scan: list_refs failed for {}: {}",
repo_path.display(),
e
);
}
}
}
// If we just emptied this submitter dir, drop it. `remove_dir`
// fails if non-empty so we don't need an explicit check.
if std::fs::remove_dir(&hex_path).is_ok() {
debug!(
"/prs/ startup scan: removed empty submitter dir {}",
hex_path.display()
);
submitter_dirs_removed += 1;
}
}
if repos_removed > 0 || submitter_dirs_removed > 0 {
info!(
"/prs/ startup scan: removed {} zero-ref repo(s) and {} empty submitter dir(s)",
repos_removed, submitter_dirs_removed
);
} else {
debug!("/prs/ startup scan: nothing to clean up");
}
(repos_removed, submitter_dirs_removed)
}