diff --git a/CHANGELOG.md b/CHANGELOG.md index 6f248fd..f210e45 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -9,7 +9,7 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0 ### Added -- **GRASP-06 contributor PR submission endpoint** (`NGIT_GRASP06_ENABLE`, default off). When enabled, the relay accepts unauthenticated `git push` of `refs/nostr/` to `/prs//.git` from any contributor, even for repositories this relay has no accepted announcement for. The corresponding PR (kind 1618) or PR Update (kind 1619) event is accepted into purgatory when its `clone` tag names this relay's `/prs//.git` endpoint and its `a` tag's d-tag matches the URL identifier. When the event and the push match (signer, d-tag, c-tag commit) the event is released from purgatory and the ref is mirrored into any accepted-announcement repos on this relay. Empty `/prs/` repos (probe pushes, mismatched events) are garbage-collected inline at the three sites that can leave them empty: the receive handler at the end of a push, the PR-event policy when discarding a mismatched scoped placeholder, and the purgatory sweep when a scoped placeholder expires without a matching event. GRASP-06 is advertised in NIP-11 `supported_grasps` when enabled. See [how-to/enable-grasp-06.md](docs/how-to/enable-grasp-06.md) and [explanation/grasp-06-contributor-pr-submission.md](docs/explanation/grasp-06-contributor-pr-submission.md). +- **GRASP-06 contributor PR submission endpoint** (`NGIT_GRASP06_ENABLE`, default off). When enabled, the relay accepts unauthenticated `git push` of `refs/nostr/` to `/prs//.git` from any contributor, even for repositories this relay has no accepted announcement for. The corresponding PR (kind 1618) or PR Update (kind 1619) event is accepted into purgatory when its `clone` tag names this relay's `/prs//.git` endpoint and its `a` tag's d-tag matches the URL identifier. When the event and the push match (signer, d-tag, c-tag commit) the event is released from purgatory and the ref is mirrored into any accepted-announcement repos on this relay. Empty `/prs/` repos (probe pushes, mismatched events) are garbage-collected inline at the three runtime sites that can leave them empty (receive handler at end of push, PR-event policy when discarding a mismatched scoped placeholder, purgatory sweep when a scoped placeholder expires without a matching event) plus a one-shot startup scan that removes any zero-ref `/prs/` bare repos left behind by a previous run (crash, mid-cleanup failure, or shutdown with unresolved scoped placeholders). GRASP-06 is advertised in NIP-11 `supported_grasps` when enabled. See [how-to/enable-grasp-06.md](docs/how-to/enable-grasp-06.md) and [explanation/grasp-06-contributor-pr-submission.md](docs/explanation/grasp-06-contributor-pr-submission.md). ## [1.0.2] - 2026-04-10 diff --git a/docs/explanation/architecture.md b/docs/explanation/architecture.md index 9b0e5dc..5952e4d 100644 --- a/docs/explanation/architecture.md +++ b/docs/explanation/architecture.md @@ -556,6 +556,7 @@ Optional endpoint at `/prs//.git`, gated on `NGIT_GRASP06_ENAB - [`src/grasp06/fetch.rs`](../../src/grasp06/fetch.rs) — empty-repo synthesis for `info/refs` and `git-upload-pack` against repos that don't yet exist on disk. - [`src/grasp06/receive.rs`](../../src/grasp06/receive.rs) — `git-receive-pack` with init-on-push, strict `refs/nostr/` ref-name validation, and per-ref post-push validation against the database and purgatory. Uses a per-`(submitter, identifier)` `PrsPathState` (mutex + `in_flight` counter) so concurrent pushes to the same path proceed in parallel: the mutex is held only for init+register and decrement+end-of-push cleanup, not across `git-receive-pack` itself. The same state is consulted (with `try_lock` in synchronous contexts) by the PR-event policy and the purgatory expiry sweep, which only remove a `/prs/` ref or zero-ref bare repo when `in_flight == 0`. - [`src/grasp06/policy.rs`](../../src/grasp06/policy.rs) — strict clone-tag URL comparator used by the PR-event acceptance relaxation. +- [`src/grasp06/cleanup.rs`](../../src/grasp06/cleanup.rs) — one-shot startup scan over `/prs/` that removes zero-ref bare repos left behind by a previous run (crash mid-push, crash mid-cleanup, or shutdown with unresolved scoped placeholders). Runs before the HTTP server starts accepting requests; no locking is needed because nothing else is touching `/prs/` yet. `/prs/` repos are intentionally isolated from other subsystems: empty-repo cleanup skips the `/prs/` subtree, the proactive-sync subsystem never discovers them because subscriptions are built from DB-resident announcements, and the standard repo landing page guards against ever matching a `/prs/` path. Full design: [GRASP-06 Contributor Pull Request Submission](grasp-06-contributor-pr-submission.md). Operator how-to: [Enable GRASP-06](../how-to/enable-grasp-06.md). diff --git a/docs/explanation/grasp-06-contributor-pr-submission.md b/docs/explanation/grasp-06-contributor-pr-submission.md index 09f3820..fdb1a3b 100644 --- a/docs/explanation/grasp-06-contributor-pr-submission.md +++ b/docs/explanation/grasp-06-contributor-pr-submission.md @@ -156,11 +156,12 @@ Placeholder entries created at `/prs/` should be validated against `(submitter = ### Zero-ref `/prs/` cleanup -A `/prs//.git` bare repo can become zero-ref through three independent paths. Each is cleaned up inline at the site where the last ref is removed; there is no separate periodic sweep over `/prs/`: +A `/prs//.git` bare repo can become zero-ref through three runtime paths — each cleaned up inline at the site where the last ref is removed — plus a startup recovery pass that catches anything left behind by a previous run. There is no periodic sweep over `/prs/`: 1. **Probe push leaves no valid refs.** The `/prs/` receive handler validates each pushed `refs/nostr/` against the database and purgatory. If every ref fails validation, the bare repo is empty at the end of the push. The handler removes it before returning, under a brief end-of-push critical section on the per-path mutex (the mutex is *not* held across `git-receive-pack` itself — only across the init+register and decrement+cleanup windows). 2. **Scoped placeholder fails validation against a later-arriving event.** When a PR event arrives whose `(signer, identifier, commit)` does not match the placeholder a `/prs/` push registered, the PR-event policy ([`src/nostr/policy/pr_event.rs`](../../src/nostr/policy/pr_event.rs)) takes the per-path mutex, deletes the corresponding `refs/nostr/` ref, and discards the placeholder. It removes the bare directory only when `in_flight == 0` (no push is currently mid-receive) and `list_refs` reports empty. 3. **Scoped placeholder expires without a matching event.** The standard purgatory sweep ([`src/purgatory/mod.rs`](../../src/purgatory/mod.rs), every 60 seconds) walks expiring `PrPurgatoryEntry` rows. For entries with a `prs_scope`, the sweep best-effort deletes the dangling `refs/nostr/` ref and — under the same `in_flight == 0` guard — the bare repo itself. The sweep is synchronous, so it uses `try_lock` on the per-path mutex: if a push is briefly holding the mutex (during init or end-of-push), the cleanup is skipped this cycle. In the worst case a dangling ref is left on disk; this is harmless because any future push to the same path simply ignores it, and the next purgatory sweep will retry. +4. **Startup recovery.** Before the HTTP server starts accepting requests, [`src/grasp06/cleanup.rs::scan_on_startup`](../../src/grasp06/cleanup.rs) walks `/prs//.git` once and removes any bare repo with zero refs, plus any submitter directories left empty as a result. This catches dirs left behind by a previous run — crashes mid-push, crashes mid-cleanup (between `delete_ref` and `remove_dir_all`), or clean shutdowns with unresolved scoped placeholders whose in-memory state was lost. Nothing is in flight at startup so no locking is needed; the same `list_refs` / `remove_dir_all` shape as the runtime paths applies. Gated on `grasp06_enable` — operators who have turned the feature off do not have their existing `/prs/` data scanned. The shared lock map is wired through: diff --git a/docs/how-to/enable-grasp-06.md b/docs/how-to/enable-grasp-06.md index eb5ce2c..c7377b1 100644 --- a/docs/how-to/enable-grasp-06.md +++ b/docs/how-to/enable-grasp-06.md @@ -59,11 +59,12 @@ The push succeeds, the relay creates `/prs//.git` ## Storage cost -One bare repo per `(submitter, identifier)` combination under `/prs//.git`. Repos are garbage-collected inline at the three sites that can leave one empty — there is no separate periodic sweep over `/prs/`: +One bare repo per `(submitter, identifier)` combination under `/prs//.git`. Repos are garbage-collected inline at the three runtime sites that can leave one empty, plus a startup scan that recovers anything left behind by a previous run — there is no separate periodic sweep over `/prs/`: - After receive-pack, the repo is removed immediately if it has zero refs left (probe pushes that produced no valid state). - When a PR event arrives that fails validation against a scoped `/prs/` placeholder, the corresponding `refs/nostr/` ref is deleted; if that leaves the repo with zero refs the directory is removed in the same step. - When a scoped placeholder expires from purgatory without a matching PR event (default 30 minutes), the standard purgatory sweep deletes the dangling ref and, if it leaves the repo zero-ref, the bare repo itself. +- On startup, before the HTTP server accepts requests, the `/prs/` subtree is scanned once and any zero-ref bare repos (left by a crash mid-push, crash mid-cleanup, or shutdown with in-memory state lost) are removed. All three sites coordinate through a per-`(submitter, identifier)` mutex and an `in_flight` counter; cleanup paths only `rm -rf` the bare repo while `in_flight == 0`, so an in-flight push cannot have its repo deleted out from under it. The mutex is held only briefly at each site — pack uploads and per-ref validation run lock-free, so concurrent pushes by multiple agents using the same identity do not serialise behind each other. diff --git a/src/grasp06/cleanup.rs b/src/grasp06/cleanup.rs new file mode 100644 index 0000000..52d85f1 --- /dev/null +++ b/src/grasp06/cleanup.rs @@ -0,0 +1,162 @@ +//! Startup recovery for the GRASP-06 `/prs/` subtree. +//! +//! The inline cleanup paths in [`crate::grasp06::receive`], +//! [`crate::nostr::policy::pr_event`], and [`crate::purgatory::Purgatory::cleanup`] +//! remove `/prs//.git` directories the moment they +//! go zero-ref while the process is running. They cannot, however, deal +//! with directories left zero-ref by a previous run: +//! +//! - A crash between writing a ref and the end-of-push cleanup. +//! - A crash between [`crate::git::delete_ref`] and `remove_dir_all` in +//! one of the off-push cleanup paths. +//! - A clean shutdown with a scoped placeholder still in memory whose +//! matching event then never arrives in the next run, after the +//! purgatory state has been dropped or aged out. +//! +//! Without recovery the bare directory and any dangling refs persist +//! indefinitely. The standalone `cleanup-empty-repos` CLI tool +//! explicitly skips `/prs/` (see +//! [`crate::grasp06::paths::is_prs_repo_path`]) because its event-driven +//! model does not apply, so there is no operational lifeline either. +//! +//! [`scan_on_startup`] walks `/prs//.git` once, +//! before the HTTP server starts accepting requests, and removes any +//! bare repository with zero refs. At startup nothing is in flight, so +//! the [`PrsPathState`] mutex and `in_flight` counter are unnecessary — +//! a direct filesystem scan is safe and the same `list_refs` / +//! `remove_dir_all` shape the runtime cleanup paths use applies. +//! +//! Empty `/` parent directories are removed too so `/prs/` does not +//! accumulate submitter dirs over time. +//! +//! [`PrsPathState`]: crate::grasp06::receive::PrsPathState + +use std::path::Path; + +use nostr_sdk::prelude::*; +use tracing::{debug, info, warn}; + +use crate::git::list_refs; +use crate::grasp06::paths::prs_base_path; + +/// Walk `/prs/` once and remove any bare repository +/// directory with zero refs, plus any submitter directory left empty as +/// a result. Returns `(repos_removed, submitter_dirs_removed)`. +/// +/// Must be called *before* the HTTP server starts accepting requests — +/// the scan does no locking and assumes no concurrent writers to the +/// `/prs/` subtree. +pub fn scan_on_startup(git_data_path: &Path) -> (usize, usize) { + let base = prs_base_path(git_data_path); + let hex_entries = match std::fs::read_dir(&base) { + Ok(entries) => entries, + Err(e) if e.kind() == std::io::ErrorKind::NotFound => return (0, 0), + Err(e) => { + warn!( + "/prs/ startup scan: failed to read {}: {}", + base.display(), + e + ); + return (0, 0); + } + }; + + let mut repos_removed = 0usize; + let mut submitter_dirs_removed = 0usize; + + for hex_entry in hex_entries.flatten() { + let hex_path = hex_entry.path(); + if !hex_path.is_dir() { + continue; + } + let submitter_hex = hex_entry.file_name().to_string_lossy().into_owned(); + if PublicKey::from_hex(&submitter_hex).is_err() { + // Skip directories whose name is not a valid pubkey — these + // shouldn't exist under /prs/, but we don't want to delete + // something we don't recognise. + debug!( + "/prs/ startup scan: skipping non-hex entry {}", + hex_path.display() + ); + continue; + } + + let id_entries = match std::fs::read_dir(&hex_path) { + Ok(entries) => entries, + Err(e) => { + warn!( + "/prs/ startup scan: failed to read {}: {}", + hex_path.display(), + e + ); + continue; + } + }; + + for id_entry in id_entries.flatten() { + let repo_path = id_entry.path(); + if !repo_path.is_dir() { + continue; + } + // Must end in `.git` to match the on-disk shape produced by + // `prs_repo_path` — anything else is suspicious; leave it + // alone. + if repo_path + .file_name() + .and_then(|s| s.to_str()) + .is_none_or(|s| !s.ends_with(".git")) + { + continue; + } + + match list_refs(&repo_path) { + Ok(refs) if refs.is_empty() => { + if let Err(e) = std::fs::remove_dir_all(&repo_path) { + warn!( + "/prs/ startup scan: failed to remove zero-ref repo {}: {}", + repo_path.display(), + e + ); + } else { + debug!( + "/prs/ startup scan: removed zero-ref repo {}", + repo_path.display() + ); + repos_removed += 1; + } + } + Ok(_) => { + // Has refs — leave alone. + } + Err(e) => { + warn!( + "/prs/ startup scan: list_refs failed for {}: {}", + repo_path.display(), + e + ); + } + } + } + + // If we just emptied this submitter dir, drop it. `remove_dir` + // fails if non-empty so we don't need an explicit check. + if std::fs::remove_dir(&hex_path).is_ok() { + debug!( + "/prs/ startup scan: removed empty submitter dir {}", + hex_path.display() + ); + submitter_dirs_removed += 1; + } + } + + if repos_removed > 0 || submitter_dirs_removed > 0 { + info!( + "/prs/ startup scan: removed {} zero-ref repo(s) and {} empty submitter dir(s)", + repos_removed, submitter_dirs_removed + ); + } else { + debug!("/prs/ startup scan: nothing to clean up"); + } + + (repos_removed, submitter_dirs_removed) +} diff --git a/src/grasp06/mod.rs b/src/grasp06/mod.rs index 2ffc434..68094bd 100644 --- a/src/grasp06/mod.rs +++ b/src/grasp06/mod.rs @@ -21,6 +21,7 @@ //! Cross-service mirroring into matching announced repos is not yet //! implemented. +pub mod cleanup; pub mod endpoint; pub mod fetch; pub mod paths; diff --git a/src/main.rs b/src/main.rs index ab080ba..54c383c 100644 --- a/src/main.rs +++ b/src/main.rs @@ -131,13 +131,35 @@ async fn run_relay(config: Config) -> Result<()> { } } - // Shared per-path init mutexes for the GRASP-06 `/prs/` endpoint. Lives + // Shared per-path state for the GRASP-06 `/prs/` endpoint (mutex + + // in-flight counter, see [`grasp06::receive::PrsPathState`]). Lives // for the lifetime of the process so concurrent pushes to the same - // `/prs//.git` path — and policy / purgatory code paths - // that may delete a `/prs/` bare repo — serialise against in-flight - // pushes via the same DashMap. + // `/prs//.git` path — and policy / purgatory code + // paths that may delete a `/prs/` bare repo — coordinate through + // the same DashMap. let repo_init_locks = grasp06::receive::new_repo_init_locks(); + // Startup recovery for the GRASP-06 `/prs/` subtree: remove any + // zero-ref bare repos and empty submitter dirs left behind by a + // previous run (crashes mid-push, mid-cleanup, or shutdowns with + // unresolved scoped placeholders). Inline cleanup paths only fire + // while the process is running, so without this scan abandoned + // dirs persist indefinitely — the `cleanup-empty-repos` CLI tool + // explicitly skips `/prs/`. Gated on `grasp06_enable` so an + // operator who has turned the feature off does not start removing + // their existing `/prs/` data on the next restart. Runs before + // anything that could write to `/prs/` so no locking is needed. + if config.grasp06_enable { + let git_data_path = PathBuf::from(config.effective_git_data_path()); + let (repos, dirs) = grasp06::cleanup::scan_on_startup(&git_data_path); + if repos > 0 || dirs > 0 { + info!( + "GRASP-06 /prs/ startup recovery: removed {} zero-ref repo(s), {} empty submitter dir(s)", + repos, dirs + ); + } + } + // Create Nostr relay with NIP-34 validation // Returns both the relay and database for direct queries in handlers if let Ok(relay_with_db) =