mirror of
https://relay.ngit.dev/npub15qydau2hjma6ngxkl2cyar74wzyjshvl65za5k5rl69264ar2exs5cyejr/ngit-grasp.git
synced 2026-10-05 15:08:24 +00:00
A push to refs/nostr/<event-id> is accepted before its PR event is known, and its objects went straight into the identifier family. The family is never garbage-collected, so anyone could fill permanent storage without signing anything, and an upload whose event never arrived stayed forever. Approach Route by what push authorization already decides. A ref named by a signed State or PR event, accepted or in purgatory, is received into the family as before. A refs/nostr ref with no event, or only a placeholder, is received into the view's own object directory. Pre-validation now reports whether a match was signed, because a placeholder match was indistinguishable from a stored event. Accepting a PR event first promotes its tip: fetch the history from the view into the family, check the family alone holds it down to existing retained roots, then install the usual retained and base roots. On failure the event is rejected and its placeholder kept, so the upload expires normally and the event can be sent again. While a view holds staged objects every push to it is staged, because the view advertises pending refs and a client may omit objects only staging holds. Signed tips of such a push are recorded as owed before Git runs and promoted when it finishes. Compaction refuses to run while anything is owed: rollback after a State deletion needs history no ref names, so a missing ref never proves history is disposable. Objects a view holds before it is first staged are moved into the family. Staging is reclaimed with git repack -a -d -l and git prune. Git can install a ref whose parent a concurrent repack removed (see tests/git_cruft_concurrency.rs), so compaction takes the family write lease that every push already holds. It waits at most 250ms and retries with backoff. One worker handles requests from pushes, promotions and ref deletions, and reads the persistent registry at startup. The /prs/ handler now releases the family lease before post-push processing, as the standard handler does. Promotion of a waiting PR event re-enters the family and would otherwise deadlock. Assumptions - Views and their family are on one filesystem; moving pre-existing objects uses hard links. - Retained roots are complete. A damaged root fails promotion and is left to the integrity pass. - One server process per storage root, as the family lease already assumes. Excluded - Storage quotas. Staging bounds how long an unsigned upload is kept, not its size. - A pack from a signed push is stored whole. A signer can make any object reachable from their own tip, so filtering it would protect nothing. - Fetches take no lease. A fetch of a pending ref that expires while being served may fail. - Archives store a view as it is; restoring one imports staged objects. - State acceptance, rollback and purgatory sync are unchanged. Validation - nix develop -c cargo test --lib git::staging: 13 passed, covering reclamation, a pending sibling keeping its history, promotion, refusal of incomplete history, owed history surviving ref deletion, restart recovery and a busy family. - nix develop -c cargo test --test pending_upload_staging: 4 end-to-end tests passed, including an unsigned upload across a relay crash and a signed push that omits objects only staging holds. - cargo clippy --workspace --all-targets -- -D warnings and cargo fmt --check were clean. Assisted-by: Claude Fable 5.1
165 lines
6.2 KiB
Rust
165 lines
6.2 KiB
Rust
//! Startup recovery for the GRASP-06 `/prs/` subtree.
|
|
//!
|
|
//! The inline cleanup paths in [`crate::grasp06::receive`],
|
|
//! [`crate::nostr::policy::pr_event`], and [`crate::purgatory::Purgatory::cleanup`]
|
|
//! remove `/prs/<submitter>/<identifier>.git` directories the moment they
|
|
//! go zero-ref while the process is running. They cannot, however, deal
|
|
//! with directories left zero-ref by a previous run:
|
|
//!
|
|
//! - A crash between writing a ref and the end-of-push cleanup.
|
|
//! - A crash between [`crate::git::delete_ref`] and `remove_dir_all` in
|
|
//! one of the off-push cleanup paths.
|
|
//! - A clean shutdown with a scoped placeholder still in memory whose
|
|
//! matching event then never arrives in the next run, after the
|
|
//! purgatory state has been dropped or aged out.
|
|
//!
|
|
//! Without recovery the bare directory and any dangling refs persist
|
|
//! indefinitely. The standalone `cleanup-empty-repos` CLI tool
|
|
//! explicitly skips `/prs/` (see
|
|
//! [`crate::grasp06::paths::is_prs_repo_path`]) because its event-driven
|
|
//! model does not apply, so there is no operational lifeline either.
|
|
//!
|
|
//! [`scan_on_startup`] walks `<git_data_path>/prs/<hex>/<id>.git` once,
|
|
//! before the HTTP server starts accepting requests, and removes any
|
|
//! bare repository with zero refs. At startup nothing is in flight, so
|
|
//! the [`PrsPathState`] mutex and `in_flight` counter are unnecessary —
|
|
//! a direct filesystem scan is safe and the same `list_refs` /
|
|
//! `remove_dir_all` shape the runtime cleanup paths use applies.
|
|
//!
|
|
//! Empty `<hex>/` parent directories are removed too so `/prs/` does not
|
|
//! accumulate submitter dirs over time.
|
|
//!
|
|
//! [`PrsPathState`]: crate::grasp06::receive::PrsPathState
|
|
|
|
use std::path::Path;
|
|
|
|
use nostr_sdk::prelude::*;
|
|
use tracing::{debug, info, warn};
|
|
|
|
use crate::git::list_refs;
|
|
use crate::grasp06::paths::prs_base_path;
|
|
|
|
/// Walk `<git_data_path>/prs/` once and remove any bare repository
|
|
/// directory with zero refs, plus any submitter directory left empty as
|
|
/// a result. Returns `(repos_removed, submitter_dirs_removed)`.
|
|
///
|
|
/// Must be called *before* the HTTP server starts accepting requests —
|
|
/// the scan does no locking and assumes no concurrent writers to the
|
|
/// `/prs/` subtree.
|
|
pub fn scan_on_startup(git_data_path: &Path) -> (usize, usize) {
|
|
let base = prs_base_path(git_data_path);
|
|
let hex_entries = match std::fs::read_dir(&base) {
|
|
Ok(entries) => entries,
|
|
Err(e) if e.kind() == std::io::ErrorKind::NotFound => return (0, 0),
|
|
Err(e) => {
|
|
warn!(
|
|
"/prs/ startup scan: failed to read {}: {}",
|
|
base.display(),
|
|
e
|
|
);
|
|
return (0, 0);
|
|
}
|
|
};
|
|
|
|
let mut repos_removed = 0usize;
|
|
let mut submitter_dirs_removed = 0usize;
|
|
|
|
for hex_entry in hex_entries.flatten() {
|
|
let hex_path = hex_entry.path();
|
|
if !hex_path.is_dir() {
|
|
continue;
|
|
}
|
|
let submitter_hex = hex_entry.file_name().to_string_lossy().into_owned();
|
|
if PublicKey::from_hex(&submitter_hex).is_err() {
|
|
// Skip directories whose name is not a valid pubkey — these
|
|
// shouldn't exist under /prs/, but we don't want to delete
|
|
// something we don't recognise.
|
|
debug!(
|
|
"/prs/ startup scan: skipping non-hex entry {}",
|
|
hex_path.display()
|
|
);
|
|
continue;
|
|
}
|
|
|
|
let id_entries = match std::fs::read_dir(&hex_path) {
|
|
Ok(entries) => entries,
|
|
Err(e) => {
|
|
warn!(
|
|
"/prs/ startup scan: failed to read {}: {}",
|
|
hex_path.display(),
|
|
e
|
|
);
|
|
continue;
|
|
}
|
|
};
|
|
|
|
for id_entry in id_entries.flatten() {
|
|
let repo_path = id_entry.path();
|
|
if !repo_path.is_dir() {
|
|
continue;
|
|
}
|
|
// Must end in `.git` to match the on-disk shape produced by
|
|
// `prs_repo_path` — anything else is suspicious; leave it
|
|
// alone.
|
|
if repo_path
|
|
.file_name()
|
|
.and_then(|s| s.to_str())
|
|
.is_none_or(|s| !s.ends_with(".git"))
|
|
{
|
|
continue;
|
|
}
|
|
|
|
match list_refs(&repo_path) {
|
|
// Signed history still owed to the family outlives its refs.
|
|
Ok(refs) if refs.is_empty() && crate::git::staging::owes_history(&repo_path) => {}
|
|
Ok(refs) if refs.is_empty() => {
|
|
if let Err(e) = std::fs::remove_dir_all(&repo_path) {
|
|
warn!(
|
|
"/prs/ startup scan: failed to remove zero-ref repo {}: {}",
|
|
repo_path.display(),
|
|
e
|
|
);
|
|
} else {
|
|
debug!(
|
|
"/prs/ startup scan: removed zero-ref repo {}",
|
|
repo_path.display()
|
|
);
|
|
repos_removed += 1;
|
|
}
|
|
}
|
|
Ok(_) => {
|
|
// Has refs — leave alone.
|
|
}
|
|
Err(e) => {
|
|
warn!(
|
|
"/prs/ startup scan: list_refs failed for {}: {}",
|
|
repo_path.display(),
|
|
e
|
|
);
|
|
}
|
|
}
|
|
}
|
|
|
|
// If we just emptied this submitter dir, drop it. `remove_dir`
|
|
// fails if non-empty so we don't need an explicit check.
|
|
if std::fs::remove_dir(&hex_path).is_ok() {
|
|
debug!(
|
|
"/prs/ startup scan: removed empty submitter dir {}",
|
|
hex_path.display()
|
|
);
|
|
submitter_dirs_removed += 1;
|
|
}
|
|
}
|
|
|
|
if repos_removed > 0 || submitter_dirs_removed > 0 {
|
|
info!(
|
|
"/prs/ startup scan: removed {} zero-ref repo(s) and {} empty submitter dir(s)",
|
|
repos_removed, submitter_dirs_removed
|
|
);
|
|
} else {
|
|
debug!("/prs/ startup scan: nothing to clean up");
|
|
}
|
|
|
|
(repos_removed, submitter_dirs_removed)
|
|
}
|