mirror of
https://relay.ngit.dev/npub15qydau2hjma6ngxkl2cyar74wzyjshvl65za5k5rl69264ar2exs5cyejr/ngit-grasp.git
synced 2026-10-05 23:18:24 +00:00
Merge #283c9227: Stage unsigned uploads until a signed event accepts th…
Stage unsigned uploads until a signed event accepts them nostr:nevent1qgsx2lyl2e4zvfadwcvkd9fkrcwczj7mf858hy85mwqclwgut8wpg2spz3mhxue69uhhyetvv9ujumn8d96zuer9wcq3yamnwvaz7tm8d96xummnw3ezucm0d5q3kamnwvaz7tmwva5hgtnyv9hxxmmwwashjer9wchxxmmdqqszs0yjy7u3p6g7f8k4fqtxz45qlwdzlwtvdrhxjz45fe3yk3ngmqg9cs9xm PR-Author: DanConwayDev's Agent nostr:npub1v47f74n2ycn66asev62nv8sas99akj0g0wg0fkup37u3ckwuzs4q7cwtp0 CoverNote: A push to `refs/nostr/<event-id>` is accepted before its PR event is known, and its objects went straight into the identifier family. The family is never garbage-collected, so anyone could fill permanent storage without signing anything. Such an upload is now staged in the view that received it, promoted into the family when a signed event accepts it, and reclaimed after its pending ref expires. This replaces nostr:nevent1qqsv6famykjg4jvlcrphuz0j4eehwdzxfdw64jsvclk9hmcl4es5rxcpz3mhxue69uhhyetvv9ujumn8d96zuer9wcdek4dx. It uses the existing family write lease. ## Design - **A signed event earns family storage at push time.** Refs named by a State or PR event, accepted or in purgatory, are received into the family as before. Only refs with no event, or just a placeholder, are staged. Maintainer pushes are unchanged unless the view already holds staged objects. - **PR and State acceptance promote first.** Staged history is fetched into the family and checked there before accepting a PR or State event, or releasing a waiting event from purgatory. This includes a State adopting an earlier unsigned upload without another push. Failed promotion prevents acceptance; waiting events remain in purgatory. - **Deletion needs proof.** Signed tips pushed into a staged view are recorded as owed before Git runs. Compaction refuses to run while anything is owed, because rollback after a State deletion needs history that no ref names. - **Expiry survives crashes.** Unsigned ref intentions and absolute deadlines are fsynced before receive-pack. Startup restores missing placeholders independently of the purgatory checkpoint. A bad individual record is logged and skipped, preserving its metadata and data while healthy records recover. - **Git does the compaction.** `git repack -a -d -l` and `git prune`, under the family write lease that every push already holds. Trade-off: a pack from a signed push is stored whole. A signer can make any object reachable from their own tip, so filtering it would protect nothing. ## Review order 1. `fa86e72b` expire abandoned normal-endpoint PR refs (unchanged from the previous PR) 2. `0edc7b83` characterise the Git races that make the lease necessary 3. `dcf36a22` rejection test fixture selects its hook explicitly 4. `c1255eac` recover `/prs/` placeholder scope after a crash 5. `1559698d` kill and reap Git children when their handle is dropped 6. `3ff84070` keep pending PR refs across graceful shutdown 7. `feb0d499` write copied history to the family, not the target view 8. `201ed7c2` one helper for removing empty `/prs/` repositories 9. `629be17e` stage unsigned uploads until a signed event accepts them 10. `6de1d473` promote staged history before State acceptance 11. `c8fee2ce` recover unsigned upload expiry after crashes 12. `8c2ee32e` isolate startup expiry recovery failures per record Commits 3 to 8 are independent of staging and can be reviewed alone. Commit 7 fixes existing behaviour: three fetches wrote accepted history into a view's own object directory. ## Validation On `8c2ee32e`: 18 staging unit tests and 62 integration/helper tests passed. The new regressions reproduce corrupt metadata, verify healthy-record recovery and data preservation, and restart a relay with a corrupt staging record. Workspace all-target clippy with warnings denied and `cargo fmt --check` passed. On `c8fee2ce`: 1,179 tests passed across the library, pending-upload staging, purgatory and GRASP-06 hosting suites. A subsequent reviewer reported 3,713 passing tests plus clean clippy and formatting on that revision. ## Not verified - The push and repack race is reproduced against plain Git and the lease exclusion is tested directly, but the interleaving was not replayed against a running relay. - Rejection of a PR event after a failed promotion has no end-to-end test. - Crash-expiry reclamation is exercised with a missing checkpoint and an injected elapsed deadline; the tests do not wait out the real 30-minute interval. ## Known limits - Staging bounds how long an unsigned upload is kept, not its size. - Restoring an archive imports any staged objects it contained. - A damaged retained root blocks promotion for that family until the integrity pass repairs it. - A fetch of a pending ref that expires while being served may fail. - An unreadable registry directory still prevents startup. Individual bad records are skipped; uploads without recovered placeholders remain retained for inspection. - Staged States with many tips still install roots through separate Git processes; batching is deferred.
This commit is contained in:
DanConwayDev
@@ -7,6 +7,21 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
|
||||
|
||||
## [Unreleased]
|
||||
|
||||
### Fixed
|
||||
|
||||
- Keep Git uploads that no signed event names out of permanent storage. A push
|
||||
to `refs/nostr/<event-id>` whose PR event is not yet known is staged in the
|
||||
repository that received it, promoted into shared storage when the event is
|
||||
accepted, and reclaimed after its pending ref expires. Pushes backed by a
|
||||
signed State or PR event are stored as before.
|
||||
|
||||
- Accept a GRASP-06 `/prs/` PR event whose pushed ref survived a crash that
|
||||
lost its purgatory placeholder, matching it by the event's service-local
|
||||
clone URL and exact `refs/nostr/<event-id>` ref.
|
||||
|
||||
- Remove abandoned normal-endpoint PR refs when their pending event expires,
|
||||
retaining exact repository scopes across restarts and retrying failed cleanup.
|
||||
|
||||
## [3.0.5] - 2026-09-25
|
||||
|
||||
Improve live sync startup, bound retries against unhealthy relays, fix Git push
|
||||
|
||||
@@ -113,7 +113,7 @@ runtime:
|
||||
pair in check-only or repair mode
|
||||
- Serve HTTP + WebSocket until a caller-supplied shutdown future
|
||||
resolves, then stop background mutation, persist a final state snapshot
|
||||
(purgatory and rejected-events cache), and clean up placeholder refs
|
||||
(purgatory and rejected-events cache), preserving pending PR refs for restart and expiry
|
||||
|
||||
**Key Dependencies:**
|
||||
|
||||
@@ -525,7 +525,17 @@ See [`types.rs`](../../src/purgatory/types.rs) for complete definitions:
|
||||
- Creates `Arc<Purgatory>` at startup
|
||||
- Passes purgatory to both write policy and git handlers
|
||||
- Passes `RepositoryLifecycle` directly to HTTP git serving and deletion/recovery
|
||||
- Spawns background cleanup task (60-second interval)
|
||||
- Spawns background cleanup task (60-second interval). Normal-endpoint PR
|
||||
placeholders persist every pushed owner/identifier and commit. After 30 minutes
|
||||
without the event, expiry takes repository lifecycle write locks, rechecks the
|
||||
placeholder, and compare-and-deletes only those recorded refs. Failed deletions
|
||||
keep the placeholder for retry. `/prs/` copies retain their separate cleanup.
|
||||
Old unscoped placeholders cannot safely identify a repository for online cleanup;
|
||||
the authorization-integrity checker preserves unmatched refs for inspection.
|
||||
- Spawns the staging maintenance worker. Uploads that no signed event names
|
||||
are staged in their view; the worker promotes history owed to the family and
|
||||
reclaims staged objects once their pending ref is gone. See
|
||||
[Staging of unsigned uploads](git-family-object-storage.md#staging-of-unsigned-uploads).
|
||||
|
||||
#### Thread Safety
|
||||
|
||||
|
||||
@@ -158,19 +158,159 @@ authorization remain view-specific.
|
||||
|
||||
For local writes, receive-pack writes its quarantine and final objects into the
|
||||
family object directory while it updates refs in the selected view. Other Git
|
||||
commands that can create objects, including proactive fetch and archive
|
||||
restore, must use the same family object directory. Read-only commands can use
|
||||
the view normally because its alternate resolves the inventory.
|
||||
commands that can create objects, including proactive fetch, copies between
|
||||
views and archive restore, must use the same family object directory. Read-only
|
||||
commands can use the view normally because its alternate resolves the
|
||||
inventory. The one exception is an upload that no signed event names yet,
|
||||
described next.
|
||||
|
||||
## Staging of unsigned uploads
|
||||
|
||||
**Added:** 2026-09-29
|
||||
|
||||
A push to `refs/nostr/<event-id>` is accepted before its PR event is known.
|
||||
Until this change its objects went straight into the family, which is never
|
||||
garbage-collected, so anyone could fill permanent storage without signing
|
||||
anything. Such an upload is now **staged**: receive-pack writes its objects to
|
||||
the view's own object directory, and they reach the family only when a signed
|
||||
event accepts them.
|
||||
|
||||
### What earns family storage
|
||||
|
||||
A signed event earns family storage at push time. Push authorization already
|
||||
decides, for each pushed ref, whether a signed event names it:
|
||||
|
||||
| Pushed ref | Objects go to |
|
||||
| ------------------------------------------------------------ | ------------- |
|
||||
| Branch or tag named by a State, accepted or in purgatory | family |
|
||||
| `refs/nostr/<id>` whose PR event is accepted or in purgatory | family |
|
||||
| `refs/nostr/<id>` with no event, or only a placeholder | view staging |
|
||||
|
||||
A push that carries any unsigned ref is staged as a whole, because one push is
|
||||
one pack. While a view holds staged objects, every push to it is staged too:
|
||||
the view advertises its pending refs, so a client may omit objects that only
|
||||
staging holds, and receive-pack could not check such a push against the family
|
||||
alone.
|
||||
|
||||
Trade-off: a pack from a signed push is stored whole, including any object in
|
||||
it that the signed tip does not reach. This is accepted because a signer can
|
||||
make any object reachable from their own tip. Staging defends against uploads
|
||||
nobody signed for, not against a signer's choice of content. Purgatory sync
|
||||
fetches and integrity repair fetch history for signed events and keep writing
|
||||
to the family.
|
||||
|
||||
### Promotion
|
||||
|
||||
Promotion copies a tip's history from the view into the family with
|
||||
`git fetch`, checks that the family alone holds it, and installs the usual
|
||||
retained and base roots. The check walks from the tip down to existing retained
|
||||
roots. Those were complete when installed and the family is append-only, so the
|
||||
cost follows the new history instead of the whole repository. Detecting damage
|
||||
behind a root remains the integrity pass's job; a damaged root fails the check
|
||||
and therefore the promotion.
|
||||
|
||||
Promotion happens at these boundaries:
|
||||
|
||||
- **PR acceptance.** A PR or PR Update event is accepted only after its tip is
|
||||
promoted. On failure the event is rejected and its placeholder is kept, so
|
||||
the upload expires normally and the client can send the event again.
|
||||
- **Signed push into a staged view.** Its signed tips are promoted when
|
||||
receive-pack finishes, while the push still holds the family lease.
|
||||
|
||||
- **State acceptance and purgatory release.** All concrete branch and tag tips
|
||||
are promoted before ref alignment and acceptance. A State can adopt an
|
||||
unsigned upload already in the view without another push. Failure rejects
|
||||
a directly submitted State or leaves a waiting State in purgatory; its
|
||||
history must become durable before later ref changes can make it disposable.
|
||||
|
||||
### Owed history
|
||||
|
||||
Rollback after a State deletion depends on history that no current ref names.
|
||||
The absence of a ref therefore never proves that staged history is disposable.
|
||||
|
||||
Before receive-pack runs, the signed tips of a staged push are recorded as
|
||||
**owed** in the view's registry record. A tip stays owed until its history is
|
||||
complete in the family alone. If promotion at the end of the push fails, or
|
||||
the server stops before it runs, the tip is still owed after restart and
|
||||
maintenance promotes it by object ID, whether or not a ref still names it. An
|
||||
owed tip whose object is in neither store belongs to a push that never
|
||||
completed and is dropped. An empty `/prs/` view is not removed while it owes
|
||||
history.
|
||||
|
||||
A view that already holds objects when it is first staged has them moved into
|
||||
the family first. They predate staging and cannot be told apart from accepted
|
||||
history.
|
||||
|
||||
### Compaction
|
||||
|
||||
Staging is reclaimed by Git itself. For one view, maintenance:
|
||||
|
||||
1. promotes owed tips, and stops if any remain owed;
|
||||
2. runs `git repack -a -d -l` and `git prune --expire=now`, which keep the
|
||||
history live refs need and the family lacks, and drop everything else;
|
||||
3. removes loose objects the family also holds;
|
||||
4. removes the registry record if nothing is left, and the view itself if it
|
||||
is an empty `/prs/` view.
|
||||
|
||||
Every live ref is a root, so a pending upload keeps exactly its own history
|
||||
and an abandoned one disappears once its ref expires.
|
||||
|
||||
Compaction must not overlap a push to the same view. Git can report a
|
||||
successful push and install a ref whose parent a concurrent repack has just
|
||||
removed; `tests/git_cruft_concurrency.rs` reproduces this. A grace period
|
||||
cannot prevent it, because the push may start after the repack has chosen
|
||||
what to keep. Compaction therefore takes the family write lease, which every
|
||||
push already holds from before receive-pack starts until its refs and roots
|
||||
are installed. The lease is fair: maintenance queues behind active writers for
|
||||
at most 250 ms, then gives way and retries with backoff from one second to five
|
||||
minutes, so it cannot hold up the pushes behind it.
|
||||
|
||||
Fetches take no lease. A fetch of a pending ref that expires while it is being
|
||||
served may fail. That ref named an upload nobody signed for and is being
|
||||
removed deliberately.
|
||||
|
||||
### Registry and maintenance worker
|
||||
|
||||
`.grasp/staging/<digest>.json` records each staged view, its owed tips, and
|
||||
unsigned ref intentions with absolute expiry deadlines. It is written and
|
||||
fsynced before receive-pack can write an object. One worker
|
||||
processes maintenance requests, which come from staged pushes, promotions and
|
||||
ref deletions, including placeholder expiry. At startup it reads the registry,
|
||||
so interrupted work resumes. Before cleanup, sync and request handling start,
|
||||
recovery reconstructs missing purgatory placeholders from surviving unsigned
|
||||
refs and their durable deadlines. Accepted events and signed events already in
|
||||
purgatory are preserved. Failed pushes with no matching ref create no
|
||||
placeholder. Legacy records without deadlines receive one grace period that
|
||||
is persisted before recovery proceeds, so repeated restarts do not extend it.
|
||||
Unreadable or invalid individual records are logged and skipped without
|
||||
removing their metadata, refs or objects; healthy records still recover. A
|
||||
skipped upload with no checkpoint placeholder remains retained for operator
|
||||
inspection. Failure to enumerate the registry itself still aborts startup.
|
||||
This closes the window between receive-pack and the periodic purgatory
|
||||
checkpoint. A view whose remaining staged objects belong to pending refs is
|
||||
parked until the next request rather than polled.
|
||||
|
||||
### Limits
|
||||
|
||||
- Staging is not a storage quota. It bounds how long an unsigned upload is
|
||||
kept, not how much one may upload.
|
||||
- Archiving a repository stores its view directory as it is, staged objects
|
||||
included. Restoring the archive imports them into the family.
|
||||
- Staged history exists in one view only. Other views cannot read it until it
|
||||
is promoted.
|
||||
|
||||
## Durability and the success fence
|
||||
|
||||
The invariant is:
|
||||
|
||||
> When a client observes a successful push, every Git object needed by the
|
||||
> accepted ref updates is durable in the configured family backend.
|
||||
> accepted ref updates is durable in the configured family backend, or in the
|
||||
> view's staging for an upload that no signed event names yet.
|
||||
|
||||
The local backend satisfies the fence when Git has atomically installed the
|
||||
objects in the family object directory and receive-pack has completed.
|
||||
objects in the family object directory, or in view staging, and receive-pack
|
||||
has completed. Accepting a PR event additionally requires its history to be
|
||||
complete in the family.
|
||||
|
||||
The S3 backend cannot release receive-pack's terminal success immediately.
|
||||
The handler must retain the final protocol status until it has:
|
||||
@@ -224,8 +364,12 @@ Consequences:
|
||||
|
||||
- disable automatic Git maintenance and pruning for family repositories;
|
||||
- create retained roots for every accepted branch, tag, and PR tip;
|
||||
- unsigned uploads stay outside this inventory until a signed event accepts
|
||||
them;
|
||||
- do not use S3 lifecycle deletion on family objects;
|
||||
- never compact by packing only the current visible ref closure;
|
||||
- never compact the family by packing only the current visible ref closure
|
||||
(view staging is compacted this way because it is not part of the
|
||||
inventory, and only once nothing in it is owed to the family);
|
||||
- if pack-count compaction becomes necessary, repack the union of every object
|
||||
in all selected packs, publish the replacement in addition to the old
|
||||
immutable packs, and leave physical deletion to future GC work.
|
||||
@@ -243,8 +387,10 @@ view whose alternate is the identifier family. Therefore:
|
||||
- receive-pack may advertise family base tips as anonymous `.have` lines;
|
||||
- the first contributor push sends only objects the family does not have;
|
||||
- pushes remain restricted to `refs/nostr/<event-id>`;
|
||||
- a push whose PR event is not yet known is staged in the view;
|
||||
- placeholder and expiry cleanup removes view refs or an empty view, never
|
||||
family objects or retained roots.
|
||||
family objects or retained roots. An empty view that still owes history to
|
||||
the family is kept.
|
||||
|
||||
Mirroring a PR into an owner view becomes a ref update after an object
|
||||
availability check. It no longer copies the object graph.
|
||||
@@ -384,9 +530,10 @@ local objects exist.
|
||||
|
||||
Operations that mutate one family are serialized by a per-family lock. This
|
||||
covers pushes to different owner or `/prs/` views with the same identifier,
|
||||
proactive fetches, archive restores, retained/base ref updates, and S3 manifest
|
||||
publication. Read requests take a hydrated family lease; they do not hold the
|
||||
writer lock after their pack set has been pinned.
|
||||
proactive fetches, archive restores, retained/base ref updates, promotion and
|
||||
compaction of view staging, and S3 manifest publication. Read requests take a
|
||||
hydrated family lease; they do not hold the writer lock after their pack set
|
||||
has been pinned.
|
||||
|
||||
View lifecycle locks remain responsible for “may this path be removed?” The
|
||||
family lock is responsible for “is this object inventory and manifest update
|
||||
@@ -397,8 +544,9 @@ atomic?” Neither lock grants authorization.
|
||||
Storage integrity is defined for an object-format/identifier family, not for
|
||||
one owner path. The storage pass checks the family pack set and object graph,
|
||||
then checks every owner and `/prs/` view with the same identifier for the
|
||||
correct alternate and for refs whose targets are available in the family.
|
||||
Multiple independent histories in one identifier are valid, and unreachable
|
||||
correct alternate and for refs whose targets are available in the family. A
|
||||
ref of a staged view whose history is complete in that view's staging is a
|
||||
pending upload, not a fault. Multiple independent histories in one identifier are valid, and unreachable
|
||||
objects are not an error: retained delete-state and rollback data intentionally
|
||||
remain present.
|
||||
|
||||
|
||||
@@ -598,7 +598,7 @@ sequenceDiagram
|
||||
end
|
||||
else No PR event anywhere (git-data-first)
|
||||
GitHandler->>GitProcess: Execute push - accept any commit
|
||||
GitHandler->>Purgatory: add_pr_placeholder(event_id, commit)
|
||||
GitHandler->>Purgatory: add_standard_pr_placeholder(event_id, commit, owner, identifier)
|
||||
GitHandler->>GitClient: Push accepted - awaiting PR event
|
||||
end
|
||||
end
|
||||
@@ -606,6 +606,20 @@ sequenceDiagram
|
||||
|
||||
---
|
||||
|
||||
Normal-endpoint git-first placeholders persist each destination owner, repository
|
||||
identifier, and pushed commit. The 60-second expiry task acquires those repositories'
|
||||
lifecycle write locks before checking the record again. If the event is still absent
|
||||
after 30 minutes, it compare-and-deletes the recorded refs, retaining the placeholder
|
||||
on failure for retry. A database check protects accepted events whose placeholders
|
||||
survived in an older checkpoint. Legacy records without destinations remain readable,
|
||||
but cannot safely drive scoped online ref deletion.
|
||||
|
||||
The objects of a git-first push are staged in the view that received them, not
|
||||
stored in the identifier family. Expiring the ref lets staging maintenance
|
||||
reclaim them. Accepting the PR event first promotes its history into the
|
||||
family; the placeholder is released only after that succeeds. See
|
||||
[Staging of unsigned uploads](git-family-object-storage.md#staging-of-unsigned-uploads).
|
||||
|
||||
## Background Sync
|
||||
|
||||
Purgatory includes a background sync system that fetches git data from remote servers when events arrive before git data.
|
||||
|
||||
+57
-29
@@ -76,6 +76,7 @@ pub async fn authorize_push(
|
||||
|
||||
// Collect all purgatory events that authorize this push
|
||||
let mut purgatory_events = Vec::new();
|
||||
let mut unsigned_refs = HashSet::new();
|
||||
|
||||
// Handle refs/nostr/ refs - validate and collect PR/PR-update events from purgatory
|
||||
if !nostr_refs.is_empty() {
|
||||
@@ -88,35 +89,45 @@ pub async fn authorize_push(
|
||||
// Standard endpoint passes `None` for prs_url: signer / a-tag
|
||||
// identifier are enforced by the surrounding maintainer-set
|
||||
// authorization, not by the URL.
|
||||
match pre_validate_refs_nostr_push(database, purgatory, new_oid, ref_name, None).await {
|
||||
NostrRefPreValidation::Rejected { reason } => {
|
||||
warn!("refs/nostr/ validation failed: {}", reason);
|
||||
return Ok(AuthorizationResult::denied(reason));
|
||||
}
|
||||
NostrRefPreValidation::Authorized {
|
||||
event_from_purgatory,
|
||||
} => {
|
||||
if let Some(event) = event_from_purgatory {
|
||||
debug!("Found matching PR event in purgatory for ref {}", ref_name);
|
||||
purgatory_events.push(event);
|
||||
} else {
|
||||
debug!("Ref {} validated against existing record", ref_name);
|
||||
let event_id = ref_name
|
||||
.strip_prefix("refs/nostr/")
|
||||
.expect("partitioned ref");
|
||||
let track_placeholder =
|
||||
match pre_validate_refs_nostr_push(database, purgatory, new_oid, ref_name, None)
|
||||
.await
|
||||
{
|
||||
NostrRefPreValidation::Rejected { reason } => {
|
||||
warn!("refs/nostr/ validation failed: {}", reason);
|
||||
return Ok(AuthorizationResult::denied(reason));
|
||||
}
|
||||
}
|
||||
NostrRefPreValidation::Unknown => {
|
||||
// No entry in DB or purgatory — create placeholder so
|
||||
// the 30-minute sweep can clean the ref up if the PR
|
||||
// event never arrives. Standard-endpoint placeholders
|
||||
// carry no /prs/ scope.
|
||||
let event_id_hex = ref_name
|
||||
.strip_prefix("refs/nostr/")
|
||||
.expect("shape validated in pre_validate_refs_nostr_push");
|
||||
purgatory.add_pr_placeholder(event_id_hex.to_string(), new_oid.clone());
|
||||
debug!(
|
||||
"Created placeholder for {} - awaiting PR event (will expire in 30min if event doesn't arrive)",
|
||||
event_id_hex
|
||||
);
|
||||
}
|
||||
NostrRefPreValidation::Authorized {
|
||||
event_from_purgatory,
|
||||
signed,
|
||||
} => {
|
||||
if !signed {
|
||||
unsigned_refs.insert(ref_name.clone());
|
||||
}
|
||||
if let Some(event) = event_from_purgatory {
|
||||
purgatory_events.push(event);
|
||||
false
|
||||
} else {
|
||||
purgatory.find_pr_placeholder(event_id).is_some()
|
||||
}
|
||||
}
|
||||
NostrRefPreValidation::Unknown => {
|
||||
unsigned_refs.insert(ref_name.clone());
|
||||
true
|
||||
}
|
||||
};
|
||||
if track_placeholder {
|
||||
// Remember every destination, including subsequent pushes of the
|
||||
// same pending event, so expiry can delete only those exact refs.
|
||||
purgatory.add_standard_pr_placeholder(
|
||||
event_id.to_string(),
|
||||
new_oid.clone(),
|
||||
PublicKey::from_hex(selected_pubkey)?,
|
||||
identifier.to_string(),
|
||||
);
|
||||
}
|
||||
}
|
||||
}
|
||||
@@ -180,6 +191,7 @@ pub async fn authorize_push(
|
||||
state: auth_result.state,
|
||||
maintainers: auth_result.maintainers,
|
||||
purgatory_events,
|
||||
unsigned_refs,
|
||||
});
|
||||
}
|
||||
|
||||
@@ -190,6 +202,7 @@ pub async fn authorize_push(
|
||||
state: None,
|
||||
maintainers: vec![],
|
||||
purgatory_events,
|
||||
unsigned_refs,
|
||||
})
|
||||
}
|
||||
|
||||
@@ -665,6 +678,7 @@ pub async fn get_state_authorization_for_selected_repo(
|
||||
state: None,
|
||||
maintainers: authorized.into_iter().collect(),
|
||||
purgatory_events: vec![],
|
||||
unsigned_refs: HashSet::new(),
|
||||
});
|
||||
}
|
||||
}
|
||||
@@ -736,6 +750,7 @@ pub async fn get_state_authorization_for_selected_repo(
|
||||
state: Some(state),
|
||||
maintainers: authorized.into_iter().collect(),
|
||||
purgatory_events: vec![latest_authorized.clone()],
|
||||
unsigned_refs: HashSet::new(),
|
||||
});
|
||||
} else {
|
||||
warn!(
|
||||
@@ -872,6 +887,9 @@ pub struct AuthorizationResult {
|
||||
pub maintainers: Vec<String>,
|
||||
/// Events from purgatory that authorized this push (state, PR, PR-update events)
|
||||
pub purgatory_events: Vec<Event>,
|
||||
/// Pushed `refs/nostr/` refs that no signed event names yet. Their objects
|
||||
/// are received into view staging instead of the family.
|
||||
pub unsigned_refs: HashSet<String>,
|
||||
}
|
||||
|
||||
impl AuthorizationResult {
|
||||
@@ -883,6 +901,7 @@ impl AuthorizationResult {
|
||||
state: None,
|
||||
maintainers: vec![],
|
||||
purgatory_events: vec![],
|
||||
unsigned_refs: HashSet::new(),
|
||||
}
|
||||
}
|
||||
}
|
||||
@@ -1245,7 +1264,13 @@ pub enum NostrRefPreValidation {
|
||||
/// `authorize_push`) can collect it into `purgatory_events`. `None`
|
||||
/// means the match came from the DB or from a placeholder-only
|
||||
/// purgatory entry — nothing to collect.
|
||||
Authorized { event_from_purgatory: Option<Event> },
|
||||
///
|
||||
/// `signed` is false for a placeholder-only match: no signed event names
|
||||
/// this commit yet, so its objects have not earned family storage.
|
||||
Authorized {
|
||||
event_from_purgatory: Option<Event>,
|
||||
signed: bool,
|
||||
},
|
||||
/// No event with that id is known to the relay yet. The caller may
|
||||
/// create a placeholder (with or without a `/prs/` scope) so the
|
||||
/// purgatory sweep can clean up the ref if the event never arrives.
|
||||
@@ -1315,6 +1340,7 @@ pub async fn pre_validate_refs_nostr_push(
|
||||
}
|
||||
return NostrRefPreValidation::Authorized {
|
||||
event_from_purgatory: None,
|
||||
signed: true,
|
||||
};
|
||||
}
|
||||
Ok(None) => {}
|
||||
@@ -1341,6 +1367,7 @@ pub async fn pre_validate_refs_nostr_push(
|
||||
}
|
||||
return NostrRefPreValidation::Authorized {
|
||||
event_from_purgatory: Some(event),
|
||||
signed: true,
|
||||
};
|
||||
}
|
||||
None => {
|
||||
@@ -1364,6 +1391,7 @@ pub async fn pre_validate_refs_nostr_push(
|
||||
}
|
||||
return NostrRefPreValidation::Authorized {
|
||||
event_from_purgatory: None,
|
||||
signed: false,
|
||||
};
|
||||
}
|
||||
}
|
||||
|
||||
@@ -601,6 +601,20 @@ fn build_expectation(
|
||||
|
||||
for (event_id, entry) in pending_prs {
|
||||
let reference = format!("refs/nostr/{event_id}");
|
||||
if entry.event.is_none() {
|
||||
if let ViewIdentity::Owner { pubkey } = identity {
|
||||
if let Some(record) = entry
|
||||
.standard_refs
|
||||
.iter()
|
||||
.find(|record| record.owner == *pubkey && record.identifier == identifier)
|
||||
{
|
||||
expected
|
||||
.pending_pr_refs
|
||||
.insert(reference, record.commit.clone());
|
||||
continue;
|
||||
}
|
||||
}
|
||||
}
|
||||
match &entry.event {
|
||||
Some(event)
|
||||
if event_applies_to_view(
|
||||
@@ -620,7 +634,10 @@ fn build_expectation(
|
||||
.pending_pr_refs
|
||||
.insert(reference, entry.commit.clone());
|
||||
}
|
||||
None if entry.prs_scope.is_none() && matches!(identity, ViewIdentity::Owner { .. }) => {
|
||||
None if entry.prs_scope.is_none()
|
||||
&& entry.standard_refs.is_empty()
|
||||
&& matches!(identity, ViewIdentity::Owner { .. }) =>
|
||||
{
|
||||
// Legacy standard-endpoint placeholders did not record their
|
||||
// owner/identifier. Preserve a matching ref, but make the
|
||||
// uncertainty visible rather than treating it as authority.
|
||||
@@ -1574,6 +1591,41 @@ mod tests {
|
||||
assert_eq!(expectation.pr_refs.len(), 1);
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn standard_placeholder_authorizes_only_recorded_owner_and_identifier() {
|
||||
let owner = Keys::generate().public_key();
|
||||
let other = Keys::generate().public_key();
|
||||
let purgatory = crate::purgatory::Purgatory::new(PathBuf::new());
|
||||
let id = "ab".repeat(32);
|
||||
let commit = "cd".repeat(20);
|
||||
purgatory.add_standard_pr_placeholder(id.clone(), commit.clone(), owner, "project".into());
|
||||
let pending = vec![(id.clone(), purgatory.find_pr(&id).unwrap())];
|
||||
let data = RepositoryData {
|
||||
announcements: vec![],
|
||||
states: vec![],
|
||||
};
|
||||
for (pubkey, identifier, matches) in [
|
||||
(owner, "project", true),
|
||||
(other, "project", false),
|
||||
(owner, "other", false),
|
||||
] {
|
||||
let expectation = build_expectation(
|
||||
&ViewIdentity::Owner { pubkey },
|
||||
identifier,
|
||||
&data,
|
||||
&[],
|
||||
&pending,
|
||||
&[],
|
||||
None,
|
||||
);
|
||||
assert_eq!(
|
||||
expectation.pending_pr_refs.get(&format!("refs/nostr/{id}")),
|
||||
matches.then_some(&commit)
|
||||
);
|
||||
assert!(expectation.ambiguous_placeholders.is_empty());
|
||||
}
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn owner_view_accepts_only_its_exact_standard_clone_endpoint() {
|
||||
let source_owner = Keys::generate();
|
||||
|
||||
+73
-5
@@ -710,7 +710,7 @@ pub async fn handle_receive_pack(
|
||||
);
|
||||
|
||||
// check push is authorised
|
||||
let _auth_result = match authorize_push(
|
||||
let auth_result = match authorize_push(
|
||||
&database,
|
||||
identifier,
|
||||
owner_pubkey,
|
||||
@@ -788,8 +788,16 @@ pub async fn handle_receive_pack(
|
||||
.map(|plan| &plan.forwarded_body)
|
||||
.unwrap_or(&request_body);
|
||||
|
||||
// Ref updates remain in the selected view; receive-pack's quarantine and
|
||||
// final objects are installed directly in the shared family inventory.
|
||||
let pushed_refs = parse_pushed_refs(&request_body);
|
||||
let staged = if family_lease.is_some() {
|
||||
stage_push(&repo_path, &pushed_refs, &auth_result.unsigned_refs).await?
|
||||
} else {
|
||||
false
|
||||
};
|
||||
|
||||
// Ref updates remain in the selected view. Objects of a push backed by
|
||||
// signed events are installed directly in the shared family inventory;
|
||||
// a staged push keeps them in the view until promotion.
|
||||
let mut git = GitSubprocess::spawn_with_object_directory(
|
||||
GitService::ReceivePack,
|
||||
&repo_path,
|
||||
@@ -797,6 +805,7 @@ pub async fn handle_receive_pack(
|
||||
git_protocol,
|
||||
family_lease
|
||||
.as_ref()
|
||||
.filter(|_| !staged)
|
||||
.map(|lease| lease.family_objects_path.as_path()),
|
||||
)
|
||||
.map_err(GitError::ProcessSpawnFailed)?;
|
||||
@@ -817,7 +826,6 @@ pub async fn handle_receive_pack(
|
||||
// uploading. Streaming sideband progress keeps libgit2 clients from
|
||||
// hitting their per-recv timeout during that otherwise-silent window.
|
||||
let (tx, rx) = mpsc::channel::<Result<Frame<Bytes>, io::Error>>(STREAM_CHANNEL_DEPTH);
|
||||
let pushed_refs = parse_pushed_refs(&request_body);
|
||||
let new_oids: HashSet<String> = pushed_refs
|
||||
.iter()
|
||||
.filter(|(_, new_oid, _)| new_oid != "0000000000000000000000000000000000000000")
|
||||
@@ -852,6 +860,7 @@ pub async fn handle_receive_pack(
|
||||
family_key,
|
||||
family_lease,
|
||||
pushed_refs,
|
||||
staged,
|
||||
push_plan,
|
||||
)
|
||||
.await;
|
||||
@@ -883,6 +892,7 @@ async fn stream_receive_pack_output<S, E, I>(
|
||||
family_key: FamilyKey,
|
||||
family_lease: Option<FamilyWriteLease>,
|
||||
pushed_refs: Vec<(String, String, String)>,
|
||||
staged: bool,
|
||||
mut push_plan: Option<ReceivePackPlan>,
|
||||
) where
|
||||
I: tokio::io::AsyncWrite + Unpin + Send + 'static,
|
||||
@@ -1015,7 +1025,9 @@ async fn stream_receive_pack_output<S, E, I>(
|
||||
|
||||
debug!("Git receive-pack stream completed successfully");
|
||||
|
||||
if family_lease.is_some() {
|
||||
if staged {
|
||||
settle_staged_push(&repo_path).await;
|
||||
} else if family_lease.is_some() {
|
||||
retain_accepted_tips(&storage, &family_key, &repo_path, &pushed_refs);
|
||||
}
|
||||
|
||||
@@ -1153,6 +1165,62 @@ impl std::fmt::Display for GitError {
|
||||
}
|
||||
}
|
||||
|
||||
/// Decide where a push to a thin view stores its objects, before Git runs.
|
||||
///
|
||||
/// A push is staged in the view when it carries a ref that no signed event
|
||||
/// names, or when the view already holds staged objects: the client may have
|
||||
/// omitted objects that only staging holds. Signed tips of a staged push are
|
||||
/// recorded as owed to the family first. The caller holds the family lease.
|
||||
pub(crate) async fn stage_push(
|
||||
repo_path: &std::path::Path,
|
||||
pushed_refs: &[(String, String, String)],
|
||||
unsigned_refs: &HashSet<String>,
|
||||
) -> Result<bool, GitError> {
|
||||
if unsigned_refs.is_empty() && !super::staging::is_staged(repo_path) {
|
||||
return Ok(false);
|
||||
}
|
||||
let owed: Vec<_> = pushed_refs
|
||||
.iter()
|
||||
.filter(|(_, new_oid, name)| {
|
||||
!unsigned_refs.contains(name) && new_oid.bytes().any(|digit| digit != b'0')
|
||||
})
|
||||
.map(|(_, new_oid, name)| super::staging::Tip::new(name, new_oid))
|
||||
.collect();
|
||||
let unsigned: Vec<_> = pushed_refs
|
||||
.iter()
|
||||
.filter(|(_, _, name)| unsigned_refs.contains(name))
|
||||
.map(|(_, oid, name)| super::staging::Tip::new(name, oid))
|
||||
.collect();
|
||||
let view = repo_path.to_owned();
|
||||
tokio::task::spawn_blocking(move || super::staging::stage_upload(&view, &owed, &unsigned))
|
||||
.await
|
||||
.map_err(|error| GitError::Storage(error.to_string()))?
|
||||
.map_err(|error| GitError::Storage(format!("cannot stage push: {error:#}")))
|
||||
}
|
||||
|
||||
/// Promote the signed tips of a staged push while the family lease is held.
|
||||
///
|
||||
/// A failure leaves the tips owed for staging maintenance to retry. The push
|
||||
/// has already succeeded, so it is not reported to the client.
|
||||
pub(crate) async fn settle_staged_push(repo_path: &std::path::Path) {
|
||||
let view = repo_path.to_owned();
|
||||
match tokio::task::spawn_blocking(move || super::staging::settle(&view)).await {
|
||||
Ok(Ok(0)) => {}
|
||||
Ok(Ok(owed)) => warn!(
|
||||
repo = %repo_path.display(),
|
||||
owed,
|
||||
"Signed history of a staged push is not yet in the family"
|
||||
),
|
||||
Ok(Err(error)) => error!(
|
||||
repo = %repo_path.display(),
|
||||
error = %format!("{error:#}"),
|
||||
"Failed to promote signed history of a staged push"
|
||||
),
|
||||
Err(error) => error!(repo = %repo_path.display(), %error, "Staged push promotion panicked"),
|
||||
}
|
||||
super::staging::request_maintenance(repo_path);
|
||||
}
|
||||
|
||||
pub(crate) fn retain_accepted_tips(
|
||||
storage: &LocalGitStorage,
|
||||
family_key: &FamilyKey,
|
||||
|
||||
@@ -629,7 +629,9 @@ pub fn inspect_family(storage: &LocalGitStorage, key: &FamilyKey) -> Result<Fami
|
||||
}
|
||||
for (reference, oid) in list_refs(view)? {
|
||||
refs_checked += 1;
|
||||
if !oid_exists(&family, &oid)? {
|
||||
// A pending upload is complete in its view's staging and is not
|
||||
// family history until its event is accepted.
|
||||
if !oid_exists(&family, &oid)? && !super::staging::holds_history(view, &oid) {
|
||||
missing_oids.insert(oid.clone());
|
||||
missing_ref_targets.push(MissingRefTarget {
|
||||
view: view.clone(),
|
||||
|
||||
@@ -25,6 +25,7 @@ pub mod migration;
|
||||
pub mod process;
|
||||
pub mod protocol;
|
||||
mod receive_pack_plan;
|
||||
pub mod staging;
|
||||
pub mod storage;
|
||||
pub mod subprocess;
|
||||
pub mod sync;
|
||||
@@ -254,6 +255,7 @@ pub fn delete_ref(repo_path: &Path, ref_name: &str) -> Result<(), String> {
|
||||
}
|
||||
|
||||
info!("Deleted ref {} from {}", ref_name, repo_path.display());
|
||||
staging::request_maintenance(repo_path);
|
||||
Ok(())
|
||||
}
|
||||
|
||||
|
||||
@@ -0,0 +1,683 @@
|
||||
//! Unsigned uploads stay in their view until a signed event earns family storage.
|
||||
//!
|
||||
//! A push naming a `refs/nostr/<event-id>` for which no signed event is known
|
||||
//! writes its objects to the view's own object directory instead of the
|
||||
//! identifier family. The family is never garbage-collected, so this keeps
|
||||
//! abandoned uploads reclaimable. Accepting the PR event promotes the tip's
|
||||
//! history into the family first.
|
||||
//!
|
||||
//! While a view holds staged objects every push to it is received into the
|
||||
//! view, because a client may omit objects that only staging holds. Signed
|
||||
//! tips of such pushes are recorded as *owed* before Git runs and stay owed
|
||||
//! until their history is complete in the family alone. Compaction refuses to
|
||||
//! run while anything is owed: the absence of a ref never proves that history
|
||||
//! is disposable, since rollback depends on history no current ref names.
|
||||
//!
|
||||
//! Every function that reads or changes staged objects or the registry record
|
||||
//! requires the caller to hold the family write lease. Pushes already hold it
|
||||
//! for their whole duration, so it also excludes them from compaction.
|
||||
use std::fs::File;
|
||||
use std::io::Write;
|
||||
use std::path::{Path, PathBuf};
|
||||
use std::process::{Command, Output, Stdio};
|
||||
|
||||
use anyhow::{bail, ensure, Context, Result};
|
||||
use bitcoin_hashes::{sha256, Hash};
|
||||
use serde::{Deserialize, Serialize};
|
||||
|
||||
use super::storage::{FamilyKey, LocalGitStorage, ObjectFormat};
|
||||
|
||||
mod recovery;
|
||||
pub use recovery::recover_placeholders;
|
||||
mod worker;
|
||||
pub use worker::{request_maintenance, run_worker};
|
||||
|
||||
const REGISTRY_DIR: &str = "staging";
|
||||
const RETAINED_GLOB: &str = "--glob=refs/grasp/retained/*";
|
||||
|
||||
/// A signed ref tip whose history the family must hold.
|
||||
#[derive(Debug, Clone, PartialEq, Eq, Serialize, Deserialize)]
|
||||
pub struct Tip {
|
||||
pub reference: String,
|
||||
pub oid: String,
|
||||
}
|
||||
|
||||
impl Tip {
|
||||
pub fn new(reference: impl Into<String>, oid: impl Into<String>) -> Self {
|
||||
Self {
|
||||
reference: reference.into(),
|
||||
oid: oid.into(),
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
/// Persistent registration of one view holding staged objects.
|
||||
#[derive(Debug, Default, Serialize, Deserialize)]
|
||||
struct Record {
|
||||
/// View path relative to the Git data root.
|
||||
view: PathBuf,
|
||||
#[serde(default)]
|
||||
owed: Vec<Tip>,
|
||||
/// Unsigned ref intentions persisted before receive-pack, independently of
|
||||
/// the periodic purgatory checkpoint. Wall-clock deadlines survive restarts.
|
||||
#[serde(default)]
|
||||
pending: Vec<PendingUpload>,
|
||||
}
|
||||
|
||||
#[derive(Debug, Serialize, Deserialize)]
|
||||
struct PendingUpload {
|
||||
tip: Tip,
|
||||
expires_at: std::time::SystemTime,
|
||||
}
|
||||
|
||||
/// Outcome of one compaction.
|
||||
#[derive(Debug, PartialEq, Eq)]
|
||||
pub enum Compaction {
|
||||
/// No staged objects remain and the view is no longer registered.
|
||||
Done,
|
||||
/// Staging holds only history that live refs need and the family lacks.
|
||||
Pending,
|
||||
}
|
||||
|
||||
/// A thin view resolved to its family and registry record.
|
||||
struct View {
|
||||
path: PathBuf,
|
||||
storage: LocalGitStorage,
|
||||
key: FamilyKey,
|
||||
record: PathBuf,
|
||||
}
|
||||
|
||||
impl View {
|
||||
/// `None` for a legacy repository that has no family alternate.
|
||||
fn resolve(view: &Path) -> Result<Option<Self>> {
|
||||
let path = std::fs::canonicalize(view)
|
||||
.with_context(|| format!("resolve view {}", view.display()))?;
|
||||
let text = match std::fs::read_to_string(path.join("objects/info/alternates")) {
|
||||
Ok(text) => text,
|
||||
Err(error) if error.kind() == std::io::ErrorKind::NotFound => return Ok(None),
|
||||
Err(error) => return Err(error.into()),
|
||||
};
|
||||
let objects = PathBuf::from(text.lines().next().context("empty family alternate")?);
|
||||
let repo = objects.parent().context("family repository")?;
|
||||
let root = repo.ancestors().nth(4).context("family storage root")?;
|
||||
let identifier = repo
|
||||
.file_name()
|
||||
.and_then(|name| name.to_str())
|
||||
.and_then(|name| name.strip_suffix(".git"))
|
||||
.context("family identifier")?;
|
||||
// The family path names its object format; no Git process is needed.
|
||||
let format = match repo
|
||||
.parent()
|
||||
.and_then(Path::file_name)
|
||||
.and_then(|name| name.to_str())
|
||||
{
|
||||
Some("sha1") => ObjectFormat::Sha1,
|
||||
Some("sha256") => ObjectFormat::Sha256,
|
||||
_ => bail!("unsupported alternate for staging: {}", path.display()),
|
||||
};
|
||||
let storage = LocalGitStorage::new(root);
|
||||
let key = FamilyKey::new(format, identifier)?;
|
||||
ensure!(
|
||||
storage.family_objects_path(&key) == objects,
|
||||
"unsupported alternate for staging: {}",
|
||||
path.display()
|
||||
);
|
||||
let root = std::fs::canonicalize(storage.git_data_path())?;
|
||||
let relative = path
|
||||
.strip_prefix(&root)
|
||||
.context("view outside Git storage")?;
|
||||
let digest = sha256::Hash::hash(relative.as_os_str().as_encoded_bytes());
|
||||
let record = registry_dir(&storage).join(format!("{digest}.json"));
|
||||
Ok(Some(Self {
|
||||
path,
|
||||
storage,
|
||||
key,
|
||||
record,
|
||||
}))
|
||||
}
|
||||
|
||||
fn family(&self) -> PathBuf {
|
||||
self.storage.family_repo_path(&self.key)
|
||||
}
|
||||
|
||||
fn relative(&self) -> Result<PathBuf> {
|
||||
let root = std::fs::canonicalize(self.storage.git_data_path())?;
|
||||
Ok(self.path.strip_prefix(root)?.to_owned())
|
||||
}
|
||||
|
||||
fn load(&self) -> Result<Option<Record>> {
|
||||
match std::fs::read(&self.record) {
|
||||
Ok(bytes) => Ok(Some(serde_json::from_slice(&bytes).with_context(|| {
|
||||
format!("parse staging record {}", self.record.display())
|
||||
})?)),
|
||||
Err(error) if error.kind() == std::io::ErrorKind::NotFound => Ok(None),
|
||||
Err(error) => Err(error.into()),
|
||||
}
|
||||
}
|
||||
|
||||
fn save(&self, record: &Record) -> Result<()> {
|
||||
let directory = self.record.parent().context("staging registry")?;
|
||||
std::fs::create_dir_all(directory)?;
|
||||
let mut file = tempfile::NamedTempFile::new_in(directory)?;
|
||||
file.write_all(&serde_json::to_vec(record)?)?;
|
||||
file.as_file().sync_all()?;
|
||||
file.persist(&self.record)?;
|
||||
File::open(directory)?.sync_all()?;
|
||||
Ok(())
|
||||
}
|
||||
|
||||
fn unregister(&self) -> Result<()> {
|
||||
match std::fs::remove_file(&self.record) {
|
||||
Ok(()) => {}
|
||||
Err(error) if error.kind() == std::io::ErrorKind::NotFound => return Ok(()),
|
||||
Err(error) => return Err(error.into()),
|
||||
}
|
||||
File::open(self.record.parent().context("staging registry")?)?.sync_all()?;
|
||||
Ok(())
|
||||
}
|
||||
}
|
||||
|
||||
fn registry_dir(storage: &LocalGitStorage) -> PathBuf {
|
||||
storage.internal_path().join(REGISTRY_DIR)
|
||||
}
|
||||
|
||||
fn valid_oid(oid: &str) -> bool {
|
||||
matches!(oid.len(), 40 | 64) && oid.bytes().all(|c| c.is_ascii_hexdigit())
|
||||
}
|
||||
|
||||
fn git(repo: &Path, args: &[&str], input: &[u8]) -> Result<Output> {
|
||||
spawn_git(repo, args, input, Stdio::null())
|
||||
}
|
||||
|
||||
/// Run Git and keep its output. Only for commands that print one short line
|
||||
/// per input line, which the reader thread drains while input is written.
|
||||
fn git_output(repo: &Path, args: &[&str], input: &[u8]) -> Result<String> {
|
||||
let output = spawn_git(repo, args, input, Stdio::piped())?;
|
||||
ensure!(
|
||||
output.status.success(),
|
||||
"git {} failed in {}: {}",
|
||||
args.first().copied().unwrap_or_default(),
|
||||
repo.display(),
|
||||
String::from_utf8_lossy(&output.stderr).trim()
|
||||
);
|
||||
Ok(String::from_utf8(output.stdout)?)
|
||||
}
|
||||
|
||||
fn spawn_git(repo: &Path, args: &[&str], input: &[u8], stdout: Stdio) -> Result<Output> {
|
||||
let mut child = Command::new("git")
|
||||
.current_dir(repo)
|
||||
.args(args)
|
||||
.stdin(Stdio::piped())
|
||||
.stdout(stdout)
|
||||
.stderr(Stdio::piped())
|
||||
.spawn()
|
||||
.with_context(|| format!("spawn git {}", args.first().copied().unwrap_or_default()))?;
|
||||
let mut stdin = child.stdin.take().expect("piped stdin");
|
||||
let input = input.to_owned();
|
||||
let writer = std::thread::spawn(move || stdin.write_all(&input));
|
||||
let output = child.wait_with_output()?;
|
||||
match writer.join() {
|
||||
Ok(Err(error)) if error.kind() != std::io::ErrorKind::BrokenPipe => {
|
||||
return Err(error.into())
|
||||
}
|
||||
Ok(_) => {}
|
||||
Err(_) => bail!("git input writer panicked"),
|
||||
}
|
||||
Ok(output)
|
||||
}
|
||||
|
||||
fn run(repo: &Path, args: &[&str]) -> Result<()> {
|
||||
let output = git(repo, args, b"")?;
|
||||
ensure!(
|
||||
output.status.success(),
|
||||
"git {} failed in {}: {}",
|
||||
args.first().copied().unwrap_or_default(),
|
||||
repo.display(),
|
||||
String::from_utf8_lossy(&output.stderr).trim()
|
||||
);
|
||||
Ok(())
|
||||
}
|
||||
|
||||
fn object_exists(repo: &Path, oid: &str) -> Result<bool> {
|
||||
Ok(git(repo, &["cat-file", "-e", oid], b"")?.status.success())
|
||||
}
|
||||
|
||||
/// Whether `oids` and their history are present in the family alone.
|
||||
///
|
||||
/// The walk stops at retained roots. Those were complete when installed and
|
||||
/// the family is append-only, so the cost follows the new history rather than
|
||||
/// the whole repository. Detecting damage behind a root is the integrity
|
||||
/// worker's job; a damaged root fails this check and therefore the promotion.
|
||||
fn complete_in_family(family: &Path, oids: &[&str]) -> Result<bool> {
|
||||
let mut input = Vec::new();
|
||||
for oid in oids {
|
||||
input.extend_from_slice(oid.as_bytes());
|
||||
input.push(b'\n');
|
||||
}
|
||||
input.extend_from_slice(format!("--not\n{RETAINED_GLOB}\n").as_bytes());
|
||||
let output = git(
|
||||
family,
|
||||
&["rev-list", "--objects", "--missing=error", "--stdin"],
|
||||
&input,
|
||||
)?;
|
||||
Ok(output.status.success())
|
||||
}
|
||||
|
||||
/// Whether the view is registered as holding staged objects.
|
||||
///
|
||||
/// Reads one directory entry; safe without the family lease as a hint. A
|
||||
/// decision that depends on it must be made while holding the lease.
|
||||
pub fn is_staged(view: &Path) -> bool {
|
||||
View::resolve(view)
|
||||
.ok()
|
||||
.flatten()
|
||||
.is_some_and(|view| view.record.is_file())
|
||||
}
|
||||
|
||||
/// Whether a staged view holds `oid` with all history the family lacks.
|
||||
///
|
||||
/// The walk stops at the family base tips the view advertises, so its cost
|
||||
/// follows the staged history.
|
||||
pub fn holds_history(view: &Path, oid: &str) -> bool {
|
||||
valid_oid(oid)
|
||||
&& is_staged(view)
|
||||
&& git(
|
||||
view,
|
||||
&[
|
||||
"rev-list",
|
||||
"--objects",
|
||||
"--missing=error",
|
||||
oid,
|
||||
"--not",
|
||||
"--alternate-refs",
|
||||
],
|
||||
b"",
|
||||
)
|
||||
.is_ok_and(|output| output.status.success())
|
||||
}
|
||||
|
||||
/// Whether signed history is still owed to the family from this view.
|
||||
///
|
||||
/// A view that owes history must not be removed, even with no refs left.
|
||||
pub fn owes_history(view: &Path) -> bool {
|
||||
match View::resolve(view).and_then(|view| view.map(|view| view.load()).transpose()) {
|
||||
Ok(record) => record
|
||||
.flatten()
|
||||
.is_some_and(|record| !record.owed.is_empty()),
|
||||
// Unreadable ownership metadata is never permission to delete.
|
||||
Err(_) => true,
|
||||
}
|
||||
}
|
||||
|
||||
/// Register the view and record `owed` tips before Git can write any object.
|
||||
///
|
||||
/// Returns `false` for a legacy repository without a family, which keeps its
|
||||
/// own objects and is never staged.
|
||||
pub fn stage(view: &Path, owed: &[Tip]) -> Result<bool> {
|
||||
stage_upload(view, owed, &[])
|
||||
}
|
||||
|
||||
/// Persist unsigned ref intentions and their expiry before receiving objects.
|
||||
pub fn stage_upload(view: &Path, owed: &[Tip], unsigned: &[Tip]) -> Result<bool> {
|
||||
let Some(view) = View::resolve(view)? else {
|
||||
return Ok(false);
|
||||
};
|
||||
ensure!(
|
||||
owed.iter().all(|tip| valid_oid(&tip.oid)),
|
||||
"invalid owed object ID"
|
||||
);
|
||||
let mut record = match view.load()? {
|
||||
Some(record) => record,
|
||||
None => {
|
||||
// Whatever the view holds before it is first staged predates
|
||||
// staging and cannot be told apart from accepted history.
|
||||
absorb_local_objects(&view)?;
|
||||
Record::default()
|
||||
}
|
||||
};
|
||||
record.view = view.relative()?;
|
||||
let before = record.owed.len();
|
||||
for tip in owed {
|
||||
if !record.owed.contains(tip) {
|
||||
record.owed.push(tip.clone());
|
||||
}
|
||||
}
|
||||
for tip in unsigned {
|
||||
ensure!(valid_oid(&tip.oid), "invalid pending object ID");
|
||||
let id = tip
|
||||
.reference
|
||||
.strip_prefix("refs/nostr/")
|
||||
.context("invalid pending ref")?;
|
||||
ensure!(
|
||||
nostr_sdk::prelude::EventId::from_hex(id).is_ok(),
|
||||
"invalid pending event ID"
|
||||
);
|
||||
record
|
||||
.pending
|
||||
.retain(|pending| pending.tip.reference != tip.reference);
|
||||
record.pending.push(PendingUpload {
|
||||
tip: tip.clone(),
|
||||
expires_at: std::time::SystemTime::now() + crate::purgatory::DEFAULT_EXPIRY,
|
||||
});
|
||||
}
|
||||
if before != record.owed.len() || !unsigned.is_empty() || !view.record.is_file() {
|
||||
view.save(&record)?;
|
||||
}
|
||||
Ok(true)
|
||||
}
|
||||
|
||||
/// Move every object file of the view into the family.
|
||||
///
|
||||
/// Files are linked into the family before they are removed from the view, so
|
||||
/// a concurrent reader of the view always finds each object in one of them.
|
||||
fn absorb_local_objects(view: &View) -> Result<()> {
|
||||
let source = view.path.join("objects");
|
||||
let target = view.storage.family_objects_path(&view.key);
|
||||
let mut moved = Vec::new();
|
||||
for entry in std::fs::read_dir(&source)? {
|
||||
let entry = entry?;
|
||||
let name = entry.file_name();
|
||||
if name == "info" || !entry.file_type()?.is_dir() {
|
||||
continue;
|
||||
}
|
||||
let directory = target.join(&name);
|
||||
let mut files: Vec<_> = std::fs::read_dir(entry.path())?
|
||||
.map(|file| file.map(|file| file.path()))
|
||||
.collect::<std::io::Result<_>>()?;
|
||||
// Git finds a pack through its index, so the index arrives last.
|
||||
files.sort_by_key(|file| file.extension().is_some_and(|ext| ext == "idx"));
|
||||
for file in files {
|
||||
if !file.is_file() {
|
||||
continue;
|
||||
}
|
||||
std::fs::create_dir_all(&directory)?;
|
||||
let destination = directory.join(file.file_name().context("object file name")?);
|
||||
match std::fs::hard_link(&file, &destination) {
|
||||
Ok(()) => {}
|
||||
Err(error) if error.kind() == std::io::ErrorKind::AlreadyExists => {}
|
||||
Err(error) => return Err(error.into()),
|
||||
}
|
||||
moved.push(file);
|
||||
}
|
||||
File::open(&directory)?.sync_all()?;
|
||||
}
|
||||
if moved.is_empty() {
|
||||
return Ok(());
|
||||
}
|
||||
File::open(&target)?.sync_all()?;
|
||||
tracing::info!(
|
||||
view = %view.path.display(),
|
||||
files = moved.len(),
|
||||
"Moved existing view objects into the family before staging"
|
||||
);
|
||||
// Remove indexes first, the reverse of the order they were installed in.
|
||||
moved.sort_by_key(|file| file.extension().is_none_or(|ext| ext != "idx"));
|
||||
for file in moved {
|
||||
std::fs::remove_file(file)?;
|
||||
}
|
||||
Ok(())
|
||||
}
|
||||
|
||||
/// Copy the history of `tips` into the family and retain it there.
|
||||
///
|
||||
/// Succeeds only when every tip is complete in the family alone, which is the
|
||||
/// condition for accepting the event that names it.
|
||||
pub fn promote(view: &Path, tips: &[Tip]) -> Result<()> {
|
||||
let Some(view) = View::resolve(view)? else {
|
||||
return Ok(());
|
||||
};
|
||||
promote_resolved(&view, tips)
|
||||
}
|
||||
|
||||
fn promote_resolved(view: &View, tips: &[Tip]) -> Result<()> {
|
||||
if tips.is_empty() {
|
||||
return Ok(());
|
||||
}
|
||||
ensure!(
|
||||
tips.iter().all(|tip| valid_oid(&tip.oid)),
|
||||
"invalid staged object ID"
|
||||
);
|
||||
let family = view.family();
|
||||
let mut oids: Vec<&str> = tips.iter().map(|tip| tip.oid.as_str()).collect();
|
||||
oids.sort_unstable();
|
||||
oids.dedup();
|
||||
|
||||
if !complete_in_family(&family, &oids)? {
|
||||
let source = view.path.to_str().context("non-UTF-8 view path")?;
|
||||
let mut args = vec![
|
||||
"-c",
|
||||
"uploadpack.allowAnySHA1InWant=true",
|
||||
"fetch",
|
||||
"--no-tags",
|
||||
"--no-write-fetch-head",
|
||||
"--no-auto-maintenance",
|
||||
"--no-recurse-submodules",
|
||||
source,
|
||||
];
|
||||
args.extend_from_slice(&oids);
|
||||
run(&family, &args).context("copy staged history into the family")?;
|
||||
ensure!(
|
||||
complete_in_family(&family, &oids)?,
|
||||
"promoted history is incomplete in the family"
|
||||
);
|
||||
}
|
||||
for tip in tips {
|
||||
view.storage
|
||||
.retain_tip(&view.key, &tip.reference, &tip.oid)?;
|
||||
view.storage
|
||||
.advertise_base_tip(&view.key, &tip.reference, &tip.oid)?;
|
||||
}
|
||||
Ok(())
|
||||
}
|
||||
|
||||
/// Promote every owed tip and return how many remain owed.
|
||||
///
|
||||
/// A tip whose object is in neither store belongs to a push that never
|
||||
/// completed and is dropped. Failures leave their tips owed for a retry.
|
||||
pub fn settle(view: &Path) -> Result<usize> {
|
||||
let Some(view) = View::resolve(view)? else {
|
||||
return Ok(0);
|
||||
};
|
||||
settle_resolved(&view)
|
||||
}
|
||||
|
||||
fn settle_resolved(view: &View) -> Result<usize> {
|
||||
let Some(mut record) = view.load()? else {
|
||||
return Ok(0);
|
||||
};
|
||||
if record.owed.is_empty() {
|
||||
return Ok(0);
|
||||
}
|
||||
let mut present = Vec::new();
|
||||
for tip in std::mem::take(&mut record.owed) {
|
||||
if object_exists(&view.path, &tip.oid)? {
|
||||
present.push(tip);
|
||||
} else {
|
||||
tracing::debug!(
|
||||
view = %view.path.display(),
|
||||
reference = %tip.reference,
|
||||
oid = %tip.oid,
|
||||
"Dropping owed tip of a push that did not complete"
|
||||
);
|
||||
}
|
||||
}
|
||||
// One fetch covers the usual case. After a failure retry each tip, so one
|
||||
// unavailable tip cannot keep independent history out of the family.
|
||||
if let Err(error) = promote_resolved(view, &present) {
|
||||
let batch = present.len() > 1;
|
||||
for tip in present {
|
||||
let failed = if batch {
|
||||
promote_resolved(view, std::slice::from_ref(&tip)).err()
|
||||
} else {
|
||||
None
|
||||
};
|
||||
if !batch || failed.is_some() {
|
||||
tracing::warn!(
|
||||
view = %view.path.display(),
|
||||
reference = %tip.reference,
|
||||
oid = %tip.oid,
|
||||
error = %failed.as_ref().unwrap_or(&error),
|
||||
"Signed history remains staged; promotion will retry"
|
||||
);
|
||||
record.owed.push(tip);
|
||||
}
|
||||
}
|
||||
}
|
||||
view.save(&record)?;
|
||||
Ok(record.owed.len())
|
||||
}
|
||||
|
||||
/// Shrink staging to the history that live refs need and the family lacks.
|
||||
///
|
||||
/// Refuses to run while any tip is owed. Every live ref is a root, so a
|
||||
/// pending upload keeps exactly its own history.
|
||||
pub fn compact(view: &Path) -> Result<Compaction> {
|
||||
let Some(view) = View::resolve(view)? else {
|
||||
return Ok(Compaction::Done);
|
||||
};
|
||||
if !view.record.is_file() {
|
||||
return Ok(Compaction::Done);
|
||||
}
|
||||
// Bound the journal to surviving refs. Failed or expired uploads must not
|
||||
// accumulate metadata while a sibling keeps the view staged indefinitely.
|
||||
if let Some(mut record) = view.load()? {
|
||||
let refs: std::collections::HashMap<_, _> = super::list_refs(&view.path)
|
||||
.map_err(anyhow::Error::msg)?
|
||||
.into_iter()
|
||||
.collect();
|
||||
let before = record.pending.len();
|
||||
record
|
||||
.pending
|
||||
.retain(|pending| refs.get(&pending.tip.reference) == Some(&pending.tip.oid));
|
||||
if record.pending.len() != before {
|
||||
view.save(&record)?;
|
||||
}
|
||||
}
|
||||
let owed = settle_resolved(&view)?;
|
||||
if owed > 0 {
|
||||
bail!("{owed} signed tips are still owed to the family");
|
||||
}
|
||||
run(
|
||||
&view.path,
|
||||
&["repack", "-a", "-d", "-l", "-q", "--no-write-bitmap-index"],
|
||||
)?;
|
||||
run(&view.path, &["prune", "--expire=now"])?;
|
||||
remove_loose_copies(&view)?;
|
||||
if has_local_objects(&view.path)? {
|
||||
return Ok(Compaction::Pending);
|
||||
}
|
||||
view.unregister()?;
|
||||
Ok(Compaction::Done)
|
||||
}
|
||||
|
||||
/// Remove loose objects that the family also holds.
|
||||
///
|
||||
/// Repacking leaves them behind: they are reachable, so pruning keeps them,
|
||||
/// and `--local` keeps them out of the new pack.
|
||||
fn remove_loose_copies(view: &View) -> Result<()> {
|
||||
let objects = view.path.join("objects");
|
||||
let mut loose = Vec::new();
|
||||
for entry in std::fs::read_dir(&objects)? {
|
||||
let entry = entry?;
|
||||
let prefix = entry.file_name();
|
||||
let Some(prefix) = prefix.to_str().filter(|name| name.len() == 2) else {
|
||||
continue;
|
||||
};
|
||||
for object in std::fs::read_dir(entry.path())? {
|
||||
let object = object?;
|
||||
if let Some(rest) = object.file_name().to_str() {
|
||||
let oid = format!("{prefix}{rest}");
|
||||
if valid_oid(&oid) {
|
||||
loose.push((oid, object.path()));
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
if loose.is_empty() {
|
||||
return Ok(());
|
||||
}
|
||||
let mut input = Vec::new();
|
||||
for (oid, _) in &loose {
|
||||
input.extend_from_slice(oid.as_bytes());
|
||||
input.push(b'\n');
|
||||
}
|
||||
let report = git_output(
|
||||
&view.family(),
|
||||
&["cat-file", "--batch-check=%(objectname)"],
|
||||
&input,
|
||||
)?;
|
||||
ensure!(
|
||||
report.lines().count() == loose.len(),
|
||||
"unexpected object report from the family"
|
||||
);
|
||||
for ((oid, path), line) in loose.iter().zip(report.lines()) {
|
||||
if line == oid {
|
||||
std::fs::remove_file(path)?;
|
||||
}
|
||||
}
|
||||
Ok(())
|
||||
}
|
||||
|
||||
fn has_local_objects(view: &Path) -> Result<bool> {
|
||||
for entry in std::fs::read_dir(view.join("objects"))? {
|
||||
let entry = entry?;
|
||||
let name = entry.file_name();
|
||||
if name == "info" || !entry.file_type()?.is_dir() {
|
||||
continue;
|
||||
}
|
||||
if std::fs::read_dir(entry.path())?.next().is_some() {
|
||||
return Ok(true);
|
||||
}
|
||||
}
|
||||
Ok(false)
|
||||
}
|
||||
|
||||
/// Promote an accepted tip out of staging, taking the family lease.
|
||||
///
|
||||
/// A view without staged objects received its history into the family, so
|
||||
/// there is nothing to copy and no lease is taken.
|
||||
pub async fn promote_accepted(view: &Path, tip: Tip) -> Result<()> {
|
||||
promote_accepted_tips(view, vec![tip]).await
|
||||
}
|
||||
|
||||
/// Promote every concrete State tip before accepting or releasing the event.
|
||||
/// A State can adopt an unsigned upload without a subsequent Git push.
|
||||
pub async fn promote_state(
|
||||
view: &Path,
|
||||
state: &crate::nostr::events::RepositoryState,
|
||||
) -> Result<()> {
|
||||
let tips = state
|
||||
.branches
|
||||
.iter()
|
||||
.map(|branch| Tip::new(format!("refs/heads/{}", branch.name), &branch.commit))
|
||||
.chain(
|
||||
state
|
||||
.tags
|
||||
.iter()
|
||||
.map(|tag| Tip::new(format!("refs/tags/{}", tag.name), &tag.commit)),
|
||||
)
|
||||
.filter(|tip| !tip.oid.starts_with("ref: "))
|
||||
.collect();
|
||||
promote_accepted_tips(view, tips).await
|
||||
}
|
||||
|
||||
async fn promote_accepted_tips(view: &Path, tips: Vec<Tip>) -> Result<()> {
|
||||
if !is_staged(view) {
|
||||
return Ok(());
|
||||
}
|
||||
let Some(resolved) = View::resolve(view)? else {
|
||||
return Ok(());
|
||||
};
|
||||
let lease = resolved.storage.write_lease(&resolved.key).await?;
|
||||
let path = resolved.path.clone();
|
||||
tokio::task::spawn_blocking(move || {
|
||||
let _lease = lease;
|
||||
promote(&path, &tips)
|
||||
})
|
||||
.await??;
|
||||
request_maintenance(view);
|
||||
Ok(())
|
||||
}
|
||||
|
||||
#[cfg(test)]
|
||||
mod tests;
|
||||
@@ -0,0 +1,323 @@
|
||||
//! Restore unsigned-upload expiry independently of the purgatory checkpoint.
|
||||
use std::path::Path;
|
||||
use std::time::SystemTime;
|
||||
|
||||
use anyhow::{ensure, Context, Result};
|
||||
use nostr_sdk::prelude::{EventId, FromBech32, PublicKey};
|
||||
|
||||
use super::{registry_dir, PendingUpload, Record, Tip, View};
|
||||
use crate::git::storage::LocalGitStorage;
|
||||
use crate::nostr::SharedDatabase;
|
||||
use crate::purgatory::Purgatory;
|
||||
|
||||
/// Called during startup before background writers and request handling.
|
||||
/// Only surviving refs without a known signed event become placeholders.
|
||||
/// Legacy records get one persisted grace period, never a fresh one per boot.
|
||||
pub async fn recover_placeholders(
|
||||
storage: &LocalGitStorage,
|
||||
purgatory: &Purgatory,
|
||||
database: &SharedDatabase,
|
||||
) -> Result<()> {
|
||||
let entries = match std::fs::read_dir(registry_dir(storage)) {
|
||||
Ok(entries) => entries,
|
||||
Err(error) if error.kind() == std::io::ErrorKind::NotFound => return Ok(()),
|
||||
Err(error) => return Err(error.into()),
|
||||
};
|
||||
for entry in entries {
|
||||
let path = match entry {
|
||||
Ok(entry) => entry.path(),
|
||||
Err(error) => {
|
||||
tracing::error!(%error, "Cannot read staging registry entry; skipping expiry recovery for it");
|
||||
continue;
|
||||
}
|
||||
};
|
||||
if path.extension().is_none_or(|ext| ext != "json") {
|
||||
continue;
|
||||
}
|
||||
if let Err(error) = recover_record(&path, storage, purgatory, database).await {
|
||||
tracing::error!(record = %path.display(), error = %format!("{error:#}"),
|
||||
"Cannot recover staged upload expiry; leaving record and data for inspection");
|
||||
}
|
||||
}
|
||||
Ok(())
|
||||
}
|
||||
|
||||
// Failure is local to this record: without a recovered placeholder, its live
|
||||
// refs remain protected by compaction. Do not delete or rename suspect metadata.
|
||||
async fn recover_record(
|
||||
path: &Path,
|
||||
storage: &LocalGitStorage,
|
||||
purgatory: &Purgatory,
|
||||
database: &SharedDatabase,
|
||||
) -> Result<()> {
|
||||
let mut record: Record = serde_json::from_slice(&std::fs::read(path)?)?;
|
||||
ensure!(
|
||||
!record.view.as_os_str().is_empty()
|
||||
&& record
|
||||
.view
|
||||
.components()
|
||||
.all(|part| matches!(part, std::path::Component::Normal(_))),
|
||||
"invalid staging view path"
|
||||
);
|
||||
let view_path = storage.git_data_path().join(&record.view);
|
||||
if !view_path.try_exists()? {
|
||||
return Ok(());
|
||||
}
|
||||
let view = View::resolve(&view_path)?.context("staged view has no family")?;
|
||||
ensure!(
|
||||
view.record == std::fs::canonicalize(path)?,
|
||||
"staging registry path does not match view"
|
||||
);
|
||||
let parts: Vec<_> = record
|
||||
.view
|
||||
.iter()
|
||||
.map(|part| part.to_str().context("non-UTF-8 view path"))
|
||||
.collect::<Result<_>>()?;
|
||||
let (owner, prs) = match parts.as_slice() {
|
||||
[owner, _] => (PublicKey::from_bech32(owner)?, false),
|
||||
["prs", owner, _] => (PublicKey::from_hex(owner)?, true),
|
||||
_ => anyhow::bail!("unsupported staged view path"),
|
||||
};
|
||||
let refs = crate::git::list_refs(&view.path).map_err(anyhow::Error::msg)?;
|
||||
for (reference, oid) in refs {
|
||||
let Some(id) = reference.strip_prefix("refs/nostr/") else {
|
||||
continue;
|
||||
};
|
||||
let Ok(event_id) = EventId::from_hex(id) else {
|
||||
continue;
|
||||
};
|
||||
if database.event_by_id(&event_id).await?.is_some()
|
||||
|| purgatory
|
||||
.find_pr(id)
|
||||
.is_some_and(|entry| entry.event.is_some())
|
||||
{
|
||||
continue;
|
||||
}
|
||||
let tip = Tip::new(&reference, &oid);
|
||||
let expires_at =
|
||||
if let Some(pending) = record.pending.iter().find(|pending| pending.tip == tip) {
|
||||
pending.expires_at
|
||||
} else {
|
||||
// Older registry versions recorded only the view and owed tips.
|
||||
let expires_at = SystemTime::now() + crate::purgatory::DEFAULT_EXPIRY;
|
||||
record
|
||||
.pending
|
||||
.retain(|pending| pending.tip.reference != reference);
|
||||
record.pending.push(PendingUpload { tip, expires_at });
|
||||
view.save(&record)?;
|
||||
expires_at
|
||||
};
|
||||
purgatory.recover_upload_placeholder(
|
||||
id.to_owned(),
|
||||
oid,
|
||||
(owner, view.key.identifier.clone()),
|
||||
prs,
|
||||
expires_at,
|
||||
);
|
||||
}
|
||||
Ok(())
|
||||
}
|
||||
|
||||
#[cfg(test)]
|
||||
mod tests {
|
||||
use super::super::tests::{commit, reference, Fixture, PENDING};
|
||||
use super::super::{compact, object_exists, stage, stage_upload};
|
||||
use super::*;
|
||||
use nostr_sdk::prelude::*;
|
||||
use std::sync::Arc;
|
||||
use std::time::{Duration, Instant};
|
||||
|
||||
fn database() -> SharedDatabase {
|
||||
Arc::new(nostr_memory::MemoryDatabase::unbounded())
|
||||
}
|
||||
|
||||
fn fixture(prs: bool) -> (Fixture, PublicKey, String) {
|
||||
let owner = Keys::generate().public_key();
|
||||
let parent = if prs {
|
||||
format!("prs/{}", owner.to_hex())
|
||||
} else {
|
||||
owner.to_bech32().unwrap()
|
||||
};
|
||||
let fixture = Fixture::with_view(&parent);
|
||||
stage(&fixture.view, &[]).unwrap();
|
||||
let oid = commit(&fixture.view, "pending", None);
|
||||
stage_upload(&fixture.view, &[], &[Tip::new(PENDING, &oid)]).unwrap();
|
||||
reference(&fixture.view, PENDING, &oid);
|
||||
(fixture, owner, oid)
|
||||
}
|
||||
|
||||
#[tokio::test]
|
||||
async fn bad_records_do_not_block_healthy_recovery_or_delete_data() {
|
||||
for corrupt in [true, false] {
|
||||
let (fixture, _, _) = fixture(false);
|
||||
let bad_path = fixture
|
||||
.storage
|
||||
.git_data_path()
|
||||
.join("unexpected/nested/owner/repo.git");
|
||||
std::fs::create_dir_all(bad_path.parent().unwrap()).unwrap();
|
||||
fixture
|
||||
.storage
|
||||
.create_thin_view(&fixture.key, &bad_path)
|
||||
.unwrap();
|
||||
stage(&bad_path, &[]).unwrap();
|
||||
let oid = commit(&bad_path, "bad record upload", None);
|
||||
let bad_ref = super::super::tests::ABANDONED;
|
||||
stage_upload(&bad_path, &[], &[Tip::new(bad_ref, &oid)]).unwrap();
|
||||
reference(&bad_path, bad_ref, &oid);
|
||||
let record_path = View::resolve(&bad_path).unwrap().unwrap().record;
|
||||
if corrupt {
|
||||
std::fs::write(&record_path, b"{broken json").unwrap();
|
||||
}
|
||||
let before = std::fs::read(&record_path).unwrap();
|
||||
let purgatory = Purgatory::new(fixture.storage.git_data_path());
|
||||
recover_placeholders(&fixture.storage, &purgatory, &database())
|
||||
.await
|
||||
.unwrap();
|
||||
assert!(purgatory
|
||||
.find_pr(PENDING.trim_start_matches("refs/nostr/"))
|
||||
.is_some());
|
||||
assert!(purgatory
|
||||
.find_pr(bad_ref.trim_start_matches("refs/nostr/"))
|
||||
.is_none());
|
||||
assert_eq!(
|
||||
crate::git::get_ref_commit(&bad_path, bad_ref),
|
||||
Some(oid.clone())
|
||||
);
|
||||
assert!(object_exists(&bad_path, &oid).unwrap());
|
||||
assert_eq!(std::fs::read(&record_path).unwrap(), before);
|
||||
}
|
||||
}
|
||||
|
||||
#[tokio::test]
|
||||
async fn uncheckpointed_uploads_recover_original_expiry_and_are_reclaimed() {
|
||||
for prs in [false, true] {
|
||||
let (fixture, owner, oid) = fixture(prs);
|
||||
let view = View::resolve(&fixture.view).unwrap().unwrap();
|
||||
let mut record = view.load().unwrap().unwrap();
|
||||
// Inject an elapsed deadline rather than waiting thirty minutes.
|
||||
record.pending[0].expires_at = SystemTime::UNIX_EPOCH;
|
||||
view.save(&record).unwrap();
|
||||
let db = database();
|
||||
// Repeated recovery must not refresh the durable deadline.
|
||||
for _ in 0..2 {
|
||||
let purgatory = Purgatory::new(fixture.storage.git_data_path());
|
||||
recover_placeholders(&fixture.storage, &purgatory, &db)
|
||||
.await
|
||||
.unwrap();
|
||||
let entry = purgatory
|
||||
.find_pr(PENDING.trim_start_matches("refs/nostr/"))
|
||||
.unwrap();
|
||||
assert!(entry.expires_at <= Instant::now());
|
||||
assert_eq!(entry.commit, oid);
|
||||
if prs {
|
||||
assert_eq!(entry.prs_scope.unwrap().submitter, owner);
|
||||
} else {
|
||||
assert_eq!(entry.standard_refs[0].owner, owner);
|
||||
}
|
||||
}
|
||||
let purgatory = Purgatory::new(fixture.storage.git_data_path());
|
||||
purgatory.set_prs_cleanup_ctx(crate::purgatory::PrsCleanupCtx {
|
||||
git_data_path: fixture.storage.git_data_path().to_owned(),
|
||||
repo_init_locks: Default::default(),
|
||||
});
|
||||
recover_placeholders(&fixture.storage, &purgatory, &db)
|
||||
.await
|
||||
.unwrap();
|
||||
purgatory
|
||||
.cleanup_standard_pr_refs(
|
||||
&crate::nostr::lifecycle::RepositoryLifecycle::in_memory(),
|
||||
&db,
|
||||
)
|
||||
.await;
|
||||
purgatory.cleanup();
|
||||
if prs {
|
||||
assert!(!fixture.view.exists());
|
||||
} else {
|
||||
compact(&fixture.view).unwrap();
|
||||
assert!(!object_exists(&fixture.view, &oid).unwrap());
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
#[tokio::test]
|
||||
async fn legacy_registry_gets_only_one_persisted_grace_period() {
|
||||
let (fixture, _, _) = fixture(false);
|
||||
let view = View::resolve(&fixture.view).unwrap().unwrap();
|
||||
let mut record = view.load().unwrap().unwrap();
|
||||
record.pending.clear();
|
||||
view.save(&record).unwrap();
|
||||
let db = database();
|
||||
let purgatory = Purgatory::new(fixture.storage.git_data_path());
|
||||
recover_placeholders(&fixture.storage, &purgatory, &db)
|
||||
.await
|
||||
.unwrap();
|
||||
let deadline = view.load().unwrap().unwrap().pending[0].expires_at;
|
||||
assert!(deadline > SystemTime::now());
|
||||
let restarted = Purgatory::new(fixture.storage.git_data_path());
|
||||
recover_placeholders(&fixture.storage, &restarted, &db)
|
||||
.await
|
||||
.unwrap();
|
||||
assert_eq!(
|
||||
view.load().unwrap().unwrap().pending[0].expires_at,
|
||||
deadline
|
||||
);
|
||||
}
|
||||
|
||||
#[tokio::test]
|
||||
async fn recovery_preserves_signed_events_and_ignores_failed_ref_updates() {
|
||||
let (fixture, _, oid) = fixture(false);
|
||||
let keys = Keys::generate();
|
||||
let event = EventBuilder::new(Kind::GitPullRequest, "signed")
|
||||
.tags([Tag::custom("c", [&oid])])
|
||||
.finalize(&keys)
|
||||
.unwrap();
|
||||
let reference_name = format!("refs/nostr/{}", event.id);
|
||||
stage_upload(&fixture.view, &[], &[Tip::new(&reference_name, &oid)]).unwrap();
|
||||
reference(&fixture.view, &reference_name, &oid);
|
||||
let db = database();
|
||||
let purgatory = Purgatory::new(fixture.storage.git_data_path());
|
||||
purgatory.add_pr(event.clone(), event.id.to_hex(), oid.clone(), false);
|
||||
recover_placeholders(&fixture.storage, &purgatory, &db)
|
||||
.await
|
||||
.unwrap();
|
||||
assert_eq!(
|
||||
purgatory.find_pr(&event.id.to_hex()).unwrap().event,
|
||||
Some(event.clone())
|
||||
);
|
||||
db.save_event(&event).await.unwrap();
|
||||
let restarted = Purgatory::new(fixture.storage.git_data_path());
|
||||
recover_placeholders(&fixture.storage, &restarted, &db)
|
||||
.await
|
||||
.unwrap();
|
||||
assert!(restarted.find_pr(&event.id.to_hex()).is_none());
|
||||
assert!(object_exists(&fixture.view, &oid).unwrap());
|
||||
// Intent was persisted but receive-pack never installed this ref.
|
||||
let failed = format!("refs/nostr/{}", "ab".repeat(32));
|
||||
stage_upload(&fixture.view, &[], &[Tip::new(&failed, &oid)]).unwrap();
|
||||
recover_placeholders(&fixture.storage, &restarted, &db)
|
||||
.await
|
||||
.unwrap();
|
||||
assert!(restarted
|
||||
.find_pr(failed.trim_start_matches("refs/nostr/"))
|
||||
.is_none());
|
||||
}
|
||||
|
||||
#[tokio::test]
|
||||
async fn recovery_preserves_a_newer_checkpoint_deadline() {
|
||||
let (fixture, owner, oid) = fixture(false);
|
||||
let purgatory = Purgatory::new(fixture.storage.git_data_path());
|
||||
let id = PENDING.trim_start_matches("refs/nostr/");
|
||||
purgatory.recover_upload_placeholder(
|
||||
id.into(),
|
||||
oid,
|
||||
(owner, "repo".into()),
|
||||
false,
|
||||
SystemTime::now() + Duration::from_secs(3600),
|
||||
);
|
||||
let deadline = purgatory.find_pr(id).unwrap().expires_at;
|
||||
recover_placeholders(&fixture.storage, &purgatory, &database())
|
||||
.await
|
||||
.unwrap();
|
||||
assert_eq!(purgatory.find_pr(id).unwrap().expires_at, deadline);
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,307 @@
|
||||
use super::*;
|
||||
use std::process::Command;
|
||||
|
||||
pub(super) struct Fixture {
|
||||
_root: tempfile::TempDir,
|
||||
pub storage: LocalGitStorage,
|
||||
pub key: FamilyKey,
|
||||
pub view: PathBuf,
|
||||
}
|
||||
|
||||
impl Fixture {
|
||||
pub fn new() -> Self {
|
||||
Self::with_view("owner")
|
||||
}
|
||||
|
||||
/// A view at `<parent>/repo.git` below the Git data root.
|
||||
pub fn with_view(parent: &str) -> Self {
|
||||
let root = tempfile::tempdir().unwrap();
|
||||
let storage = LocalGitStorage::new(root.path().canonicalize().unwrap());
|
||||
let key = FamilyKey::sha1("repo").unwrap();
|
||||
storage.ensure_family(&key).unwrap();
|
||||
let view = storage.git_data_path().join(parent).join("repo.git");
|
||||
std::fs::create_dir_all(view.parent().unwrap()).unwrap();
|
||||
storage.create_thin_view(&key, &view).unwrap();
|
||||
Self {
|
||||
_root: root,
|
||||
storage,
|
||||
key,
|
||||
view,
|
||||
}
|
||||
}
|
||||
|
||||
pub fn family(&self) -> PathBuf {
|
||||
self.storage.family_repo_path(&self.key)
|
||||
}
|
||||
|
||||
/// Whether the family holds `oid` and its whole history by itself.
|
||||
fn family_holds(&self, oid: &str) -> bool {
|
||||
Command::new("git")
|
||||
.current_dir(self.family())
|
||||
.args(["rev-list", "--objects", "--missing=error", oid])
|
||||
.output()
|
||||
.unwrap()
|
||||
.status
|
||||
.success()
|
||||
}
|
||||
}
|
||||
|
||||
fn output(repo: &Path, args: &[&str], input: &[u8]) -> String {
|
||||
let mut child = Command::new("git")
|
||||
.current_dir(repo)
|
||||
.args(args)
|
||||
.env("GIT_AUTHOR_NAME", "Test")
|
||||
.env("GIT_AUTHOR_EMAIL", "test@example.com")
|
||||
.env("GIT_COMMITTER_NAME", "Test")
|
||||
.env("GIT_COMMITTER_EMAIL", "test@example.com")
|
||||
.stdin(Stdio::piped())
|
||||
.stdout(Stdio::piped())
|
||||
.stderr(Stdio::piped())
|
||||
.spawn()
|
||||
.unwrap();
|
||||
child.stdin.take().unwrap().write_all(input).unwrap();
|
||||
let output = child.wait_with_output().unwrap();
|
||||
assert!(
|
||||
output.status.success(),
|
||||
"git {args:?}: {}",
|
||||
String::from_utf8_lossy(&output.stderr)
|
||||
);
|
||||
String::from_utf8(output.stdout).unwrap().trim().to_owned()
|
||||
}
|
||||
|
||||
/// Write a commit with one unique blob into `repo`'s own object directory.
|
||||
pub(super) fn commit(repo: &Path, content: &str, parent: Option<&str>) -> String {
|
||||
let blob = output(repo, &["hash-object", "-w", "--stdin"], content.as_bytes());
|
||||
let tree = output(
|
||||
repo,
|
||||
&["mktree"],
|
||||
format!("100644 blob {blob}\tfile\n").as_bytes(),
|
||||
);
|
||||
let mut args = vec!["commit-tree", tree.as_str(), "-m", content];
|
||||
if let Some(parent) = parent {
|
||||
args.extend(["-p", parent]);
|
||||
}
|
||||
output(repo, &args, b"")
|
||||
}
|
||||
|
||||
pub(super) fn reference(repo: &Path, name: &str, oid: &str) {
|
||||
output(repo, &["update-ref", name, oid], b"");
|
||||
}
|
||||
|
||||
pub(super) fn delete_reference(repo: &Path, name: &str) {
|
||||
output(repo, &["update-ref", "-d", name], b"");
|
||||
}
|
||||
|
||||
fn family_objects(fixture: &Fixture) -> String {
|
||||
output(
|
||||
&fixture.family(),
|
||||
&["cat-file", "--batch-all-objects", "--batch-check"],
|
||||
b"",
|
||||
)
|
||||
}
|
||||
|
||||
pub(super) const PENDING: &str =
|
||||
"refs/nostr/1111111111111111111111111111111111111111111111111111111111111111";
|
||||
pub(super) const ABANDONED: &str =
|
||||
"refs/nostr/2222222222222222222222222222222222222222222222222222222222222222";
|
||||
|
||||
#[test]
|
||||
fn abandoned_upload_is_reclaimed_without_touching_the_family() {
|
||||
let fixture = Fixture::new();
|
||||
let before = family_objects(&fixture);
|
||||
assert!(stage(&fixture.view, &[]).unwrap());
|
||||
let tip = commit(&fixture.view, "abandoned", None);
|
||||
reference(&fixture.view, ABANDONED, &tip);
|
||||
|
||||
assert_eq!(compact(&fixture.view).unwrap(), Compaction::Pending);
|
||||
assert!(object_exists(&fixture.view, &tip).unwrap());
|
||||
assert!(is_staged(&fixture.view));
|
||||
|
||||
delete_reference(&fixture.view, ABANDONED);
|
||||
assert_eq!(compact(&fixture.view).unwrap(), Compaction::Done);
|
||||
assert!(!object_exists(&fixture.view, &tip).unwrap());
|
||||
assert!(!is_staged(&fixture.view));
|
||||
assert_eq!(family_objects(&fixture), before);
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn pending_upload_keeps_its_history_when_a_sibling_is_reclaimed() {
|
||||
let fixture = Fixture::new();
|
||||
stage(&fixture.view, &[]).unwrap();
|
||||
let parent = commit(&fixture.view, "pending parent", None);
|
||||
let pending = commit(&fixture.view, "pending", Some(&parent));
|
||||
let abandoned = commit(&fixture.view, "abandoned", None);
|
||||
reference(&fixture.view, PENDING, &pending);
|
||||
reference(&fixture.view, ABANDONED, &abandoned);
|
||||
// Pack both uploads together so reclaiming one must rewrite the pack.
|
||||
assert_eq!(compact(&fixture.view).unwrap(), Compaction::Pending);
|
||||
|
||||
delete_reference(&fixture.view, ABANDONED);
|
||||
assert_eq!(compact(&fixture.view).unwrap(), Compaction::Pending);
|
||||
|
||||
assert!(!object_exists(&fixture.view, &abandoned).unwrap());
|
||||
output(
|
||||
&fixture.view,
|
||||
&["rev-list", "--objects", "--missing=error", &pending],
|
||||
b"",
|
||||
);
|
||||
assert!(!fixture.family_holds(&pending));
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn promotion_makes_history_complete_in_the_family_alone() {
|
||||
let fixture = Fixture::new();
|
||||
stage(&fixture.view, &[]).unwrap();
|
||||
let parent = commit(&fixture.view, "parent", None);
|
||||
let tip = commit(&fixture.view, "tip", Some(&parent));
|
||||
let unrelated = commit(&fixture.view, "unrelated", None);
|
||||
reference(&fixture.view, PENDING, &tip);
|
||||
reference(&fixture.view, ABANDONED, &unrelated);
|
||||
|
||||
promote(&fixture.view, &[Tip::new(PENDING, &tip)]).unwrap();
|
||||
|
||||
assert!(fixture.family_holds(&tip));
|
||||
assert!(
|
||||
!object_exists(&fixture.family(), &unrelated).unwrap(),
|
||||
"promotion copies only the accepted history"
|
||||
);
|
||||
delete_reference(&fixture.view, ABANDONED);
|
||||
assert_eq!(compact(&fixture.view).unwrap(), Compaction::Done);
|
||||
// The view still serves the promoted ref, now through the family.
|
||||
output(
|
||||
&fixture.view,
|
||||
&["rev-list", "--objects", "--missing=error", &tip],
|
||||
b"",
|
||||
);
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn promotion_refuses_incomplete_history() {
|
||||
let fixture = Fixture::new();
|
||||
stage(&fixture.view, &[]).unwrap();
|
||||
let parent = commit(&fixture.view, "parent", None);
|
||||
let tip = commit(&fixture.view, "tip", Some(&parent));
|
||||
reference(&fixture.view, PENDING, &tip);
|
||||
std::fs::remove_file(
|
||||
fixture
|
||||
.view
|
||||
.join("objects")
|
||||
.join(&parent[..2])
|
||||
.join(&parent[2..]),
|
||||
)
|
||||
.unwrap();
|
||||
|
||||
assert!(promote(&fixture.view, &[Tip::new(PENDING, &tip)]).is_err());
|
||||
assert!(!fixture.family_holds(&tip));
|
||||
let retained = output(
|
||||
&fixture.family(),
|
||||
&["for-each-ref", "refs/grasp/retained/"],
|
||||
b"",
|
||||
);
|
||||
assert_eq!(retained, "", "no retention root for incomplete history");
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn owed_history_survives_ref_deletion_until_the_family_holds_it() {
|
||||
let fixture = Fixture::new();
|
||||
let tip_oid = {
|
||||
// Record the signed tip before its objects exist, as a push does.
|
||||
let probe = Fixture::new();
|
||||
commit(&probe.view, "signed", None)
|
||||
};
|
||||
let owed = Tip::new("refs/heads/main", &tip_oid);
|
||||
stage(&fixture.view, std::slice::from_ref(&owed)).unwrap();
|
||||
assert_eq!(commit(&fixture.view, "signed", None), tip_oid);
|
||||
reference(&fixture.view, "refs/heads/main", &tip_oid);
|
||||
|
||||
// Promotion cannot install its retention root while this lock exists.
|
||||
let digest = sha256::Hash::hash(format!("refs/heads/main\0{tip_oid}").as_bytes());
|
||||
let lock = fixture
|
||||
.family()
|
||||
.join(format!("refs/grasp/retained/{digest}.lock"));
|
||||
std::fs::create_dir_all(lock.parent().unwrap()).unwrap();
|
||||
std::fs::write(&lock, b"").unwrap();
|
||||
assert_eq!(settle(&fixture.view).unwrap(), 1);
|
||||
|
||||
// A State deletion removes the ref. Rollback still needs the history.
|
||||
delete_reference(&fixture.view, "refs/heads/main");
|
||||
assert!(
|
||||
compact(&fixture.view).is_err(),
|
||||
"owed tips block compaction"
|
||||
);
|
||||
assert!(object_exists(&fixture.view, &tip_oid).unwrap());
|
||||
assert!(is_staged(&fixture.view));
|
||||
|
||||
std::fs::remove_file(&lock).unwrap();
|
||||
assert_eq!(compact(&fixture.view).unwrap(), Compaction::Done);
|
||||
assert!(fixture.family_holds(&tip_oid));
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn owed_tip_of_a_push_that_never_completed_is_dropped() {
|
||||
let fixture = Fixture::new();
|
||||
let absent = "0123456789012345678901234567890123456789";
|
||||
stage(&fixture.view, &[Tip::new("refs/heads/main", absent)]).unwrap();
|
||||
|
||||
assert_eq!(settle(&fixture.view).unwrap(), 0);
|
||||
assert_eq!(compact(&fixture.view).unwrap(), Compaction::Done);
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn one_unavailable_owed_tip_does_not_hold_back_another() {
|
||||
let fixture = Fixture::new();
|
||||
stage(&fixture.view, &[]).unwrap();
|
||||
let parent = commit(&fixture.view, "broken parent", None);
|
||||
let broken = commit(&fixture.view, "broken", Some(&parent));
|
||||
let sound = commit(&fixture.view, "sound", None);
|
||||
stage(
|
||||
&fixture.view,
|
||||
&[
|
||||
Tip::new("refs/heads/broken", &broken),
|
||||
Tip::new("refs/heads/sound", &sound),
|
||||
],
|
||||
)
|
||||
.unwrap();
|
||||
std::fs::remove_file(
|
||||
fixture
|
||||
.view
|
||||
.join("objects")
|
||||
.join(&parent[..2])
|
||||
.join(&parent[2..]),
|
||||
)
|
||||
.unwrap();
|
||||
|
||||
assert_eq!(settle(&fixture.view).unwrap(), 1);
|
||||
assert!(fixture.family_holds(&sound));
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn legacy_repository_without_a_family_is_never_staged() {
|
||||
let root = tempfile::tempdir().unwrap();
|
||||
output(root.path(), &["init", "--bare", "legacy.git"], b"");
|
||||
let legacy = root.path().join("legacy.git");
|
||||
|
||||
assert!(!stage(&legacy, &[]).unwrap());
|
||||
assert!(!is_staged(&legacy));
|
||||
assert_eq!(compact(&legacy).unwrap(), Compaction::Done);
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn objects_held_before_first_staging_move_to_the_family() {
|
||||
let fixture = Fixture::new();
|
||||
let packed = commit(&fixture.view, "packed before staging", None);
|
||||
reference(&fixture.view, "refs/heads/packed", &packed);
|
||||
output(&fixture.view, &["repack", "-a", "-d", "-l", "-q"], b"");
|
||||
delete_reference(&fixture.view, "refs/heads/packed");
|
||||
let loose = commit(&fixture.view, "loose before staging", None);
|
||||
|
||||
stage(&fixture.view, &[]).unwrap();
|
||||
|
||||
assert!(fixture.family_holds(&packed));
|
||||
assert!(fixture.family_holds(&loose));
|
||||
assert!(!has_local_objects(&fixture.view).unwrap());
|
||||
// Neither commit has a ref, yet compaction cannot reach them any more.
|
||||
assert_eq!(compact(&fixture.view).unwrap(), Compaction::Done);
|
||||
assert!(fixture.family_holds(&packed));
|
||||
assert!(fixture.family_holds(&loose));
|
||||
}
|
||||
@@ -0,0 +1,266 @@
|
||||
//! Event-driven staging maintenance.
|
||||
//!
|
||||
//! Pushes, promotions and ref deletions request maintenance for their view.
|
||||
//! The persistent registry restores those requests after a restart. A view
|
||||
//! whose remaining staged objects belong to pending refs is parked until the
|
||||
//! next request rather than polled.
|
||||
use std::collections::HashMap;
|
||||
use std::path::{Path, PathBuf};
|
||||
use std::sync::{LazyLock, Mutex};
|
||||
use std::time::Duration;
|
||||
|
||||
use anyhow::{Context, Result};
|
||||
use tokio::sync::Notify;
|
||||
use tokio::time::Instant;
|
||||
|
||||
use super::{compact, is_staged, registry_dir, Compaction, Record, View};
|
||||
use crate::git::storage::LocalGitStorage;
|
||||
use crate::grasp06::receive::{remove_prs_repo_if_empty, RepoInitLocks};
|
||||
|
||||
/// How long maintenance queues behind active writers of the family before it
|
||||
/// gives way. The lease is fair, so waiting longer would hold up later pushes.
|
||||
const LEASE_WAIT: Duration = Duration::from_millis(250);
|
||||
const FIRST_RETRY: Duration = Duration::from_secs(1);
|
||||
const LAST_RETRY: Duration = Duration::from_secs(300);
|
||||
|
||||
struct Work {
|
||||
due: Instant,
|
||||
/// Delay before the attempt after next; doubles on each failure.
|
||||
retry: Duration,
|
||||
}
|
||||
|
||||
static QUEUE: LazyLock<Mutex<HashMap<PathBuf, Work>>> = LazyLock::new(Mutex::default);
|
||||
static WAKE: Notify = Notify::const_new();
|
||||
|
||||
fn queue() -> std::sync::MutexGuard<'static, HashMap<PathBuf, Work>> {
|
||||
QUEUE
|
||||
.lock()
|
||||
.unwrap_or_else(std::sync::PoisonError::into_inner)
|
||||
}
|
||||
|
||||
/// Ask the worker to settle and compact a view. Cheap for unstaged views.
|
||||
pub fn request_maintenance(view: &Path) {
|
||||
if !is_staged(view) {
|
||||
return;
|
||||
}
|
||||
let Ok(view) = std::fs::canonicalize(view) else {
|
||||
return;
|
||||
};
|
||||
queue().insert(
|
||||
view,
|
||||
Work {
|
||||
due: Instant::now(),
|
||||
retry: FIRST_RETRY,
|
||||
},
|
||||
);
|
||||
WAKE.notify_waiters();
|
||||
}
|
||||
|
||||
/// Queue every registered view and drop records of views that no longer exist.
|
||||
fn recover(storage: &LocalGitStorage) -> Result<()> {
|
||||
let entries = match std::fs::read_dir(registry_dir(storage)) {
|
||||
Ok(entries) => entries,
|
||||
Err(error) if error.kind() == std::io::ErrorKind::NotFound => return Ok(()),
|
||||
Err(error) => return Err(error.into()),
|
||||
};
|
||||
for entry in entries {
|
||||
let path = entry?.path();
|
||||
if path.extension().is_none_or(|extension| extension != "json") {
|
||||
continue;
|
||||
}
|
||||
let result = (|| -> Result<()> {
|
||||
let record: Record = serde_json::from_slice(&std::fs::read(&path)?)?;
|
||||
anyhow::ensure!(
|
||||
!record.view.as_os_str().is_empty()
|
||||
&& record
|
||||
.view
|
||||
.components()
|
||||
.all(|part| matches!(part, std::path::Component::Normal(_))),
|
||||
"invalid view path"
|
||||
);
|
||||
let view = storage.git_data_path().join(&record.view);
|
||||
if view.try_exists()? {
|
||||
request_maintenance(&view);
|
||||
} else {
|
||||
std::fs::remove_file(&path)?;
|
||||
}
|
||||
Ok(())
|
||||
})();
|
||||
if let Err(error) = result {
|
||||
tracing::warn!(record = %path.display(), %error, "Cannot recover staged Git view");
|
||||
}
|
||||
}
|
||||
Ok(())
|
||||
}
|
||||
|
||||
/// One maintenance attempt. An error means the attempt should be retried.
|
||||
pub(super) async fn maintain(view: &Path, prs_locks: &RepoInitLocks) -> Result<Compaction> {
|
||||
let Some(resolved) = View::resolve(view)? else {
|
||||
return Ok(Compaction::Done);
|
||||
};
|
||||
let lease = tokio::time::timeout(LEASE_WAIT, resolved.storage.write_lease(&resolved.key))
|
||||
.await
|
||||
.context("family is busy")??;
|
||||
let path = resolved.path.clone();
|
||||
let outcome = tokio::task::spawn_blocking(move || {
|
||||
let _lease = lease;
|
||||
compact(&path)
|
||||
})
|
||||
.await??;
|
||||
if outcome == Compaction::Done {
|
||||
remove_prs_repo_if_empty(prs_locks, resolved.storage.git_data_path(), &resolved.path);
|
||||
}
|
||||
Ok(outcome)
|
||||
}
|
||||
|
||||
/// Run staging maintenance until the task is aborted.
|
||||
pub async fn run_worker(storage: LocalGitStorage, prs_locks: RepoInitLocks) {
|
||||
let root = match std::fs::canonicalize(storage.git_data_path()) {
|
||||
Ok(root) => root,
|
||||
Err(error) => {
|
||||
tracing::warn!(%error, "Cannot start staging maintenance");
|
||||
return;
|
||||
}
|
||||
};
|
||||
let recovery = storage.clone();
|
||||
match tokio::task::spawn_blocking(move || recover(&recovery)).await {
|
||||
Ok(Ok(())) => {}
|
||||
result => tracing::warn!(?result, "Staging registry recovery failed"),
|
||||
}
|
||||
loop {
|
||||
// Register for wake-ups before reading the queue, so a request made
|
||||
// between the read and the wait is not missed.
|
||||
let woken = WAKE.notified();
|
||||
tokio::pin!(woken);
|
||||
woken.as_mut().enable();
|
||||
let now = Instant::now();
|
||||
let next = {
|
||||
let mut queue = queue();
|
||||
let next = queue
|
||||
.iter()
|
||||
.filter(|(view, _)| view.starts_with(&root))
|
||||
.min_by_key(|(_, work)| work.due)
|
||||
.map(|(view, work)| (view.clone(), work.due, work.retry));
|
||||
if let Some((view, due, _)) = &next {
|
||||
if *due <= now {
|
||||
queue.remove(view);
|
||||
}
|
||||
}
|
||||
next
|
||||
};
|
||||
match next {
|
||||
Some((view, due, retry)) if due <= now => {
|
||||
if !view.exists() {
|
||||
continue;
|
||||
}
|
||||
if let Err(error) = maintain(&view, &prs_locks).await {
|
||||
tracing::debug!(view = %view.display(), %error, "Staging maintenance will retry");
|
||||
// A request that arrived meanwhile is newer than this failure.
|
||||
queue().entry(view).or_insert(Work {
|
||||
due: Instant::now() + retry,
|
||||
retry: (retry * 2).min(LAST_RETRY),
|
||||
});
|
||||
}
|
||||
}
|
||||
Some((_, due, _)) => {
|
||||
tokio::select! {
|
||||
_ = tokio::time::sleep_until(due) => {}
|
||||
_ = woken => {}
|
||||
}
|
||||
}
|
||||
None => woken.await,
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
#[cfg(test)]
|
||||
mod tests {
|
||||
use super::super::tests::{commit, delete_reference, reference, Fixture, ABANDONED, PENDING};
|
||||
use super::super::{object_exists, stage};
|
||||
use super::*;
|
||||
use crate::grasp06::receive::new_repo_init_locks;
|
||||
|
||||
const DEADLINE: Duration = Duration::from_secs(10);
|
||||
|
||||
async fn wait_until(description: &str, condition: impl Fn() -> bool) {
|
||||
tokio::time::timeout(DEADLINE, async {
|
||||
while !condition() {
|
||||
tokio::task::yield_now().await;
|
||||
}
|
||||
})
|
||||
.await
|
||||
.unwrap_or_else(|_| panic!("timed out waiting for {description}"));
|
||||
}
|
||||
|
||||
#[tokio::test]
|
||||
async fn ref_deletion_wakes_the_worker_to_reclaim_an_abandoned_upload() {
|
||||
let fixture = Fixture::new();
|
||||
stage(&fixture.view, &[]).unwrap();
|
||||
let pending = commit(&fixture.view, "pending", None);
|
||||
let abandoned = commit(&fixture.view, "abandoned", None);
|
||||
reference(&fixture.view, PENDING, &pending);
|
||||
reference(&fixture.view, ABANDONED, &abandoned);
|
||||
let worker = tokio::spawn(run_worker(fixture.storage.clone(), new_repo_init_locks()));
|
||||
|
||||
crate::git::delete_ref(&fixture.view, ABANDONED).unwrap();
|
||||
|
||||
let view = fixture.view.clone();
|
||||
wait_until("the abandoned upload to be reclaimed", || {
|
||||
!object_exists(&view, &abandoned).unwrap()
|
||||
})
|
||||
.await;
|
||||
assert!(object_exists(&fixture.view, &pending).unwrap());
|
||||
assert!(is_staged(&fixture.view), "a pending upload stays staged");
|
||||
worker.abort();
|
||||
}
|
||||
|
||||
#[tokio::test]
|
||||
async fn restart_recovers_a_view_registered_before_it() {
|
||||
let fixture = Fixture::new();
|
||||
stage(&fixture.view, &[]).unwrap();
|
||||
let abandoned = commit(&fixture.view, "abandoned", None);
|
||||
reference(&fixture.view, ABANDONED, &abandoned);
|
||||
// The ref expired without a maintenance request, as after a crash.
|
||||
delete_reference(&fixture.view, ABANDONED);
|
||||
|
||||
let worker = tokio::spawn(run_worker(fixture.storage.clone(), new_repo_init_locks()));
|
||||
|
||||
let view = fixture.view.clone();
|
||||
wait_until("the recovered view to be reclaimed", || !is_staged(&view)).await;
|
||||
assert!(!object_exists(&fixture.view, &abandoned).unwrap());
|
||||
worker.abort();
|
||||
}
|
||||
|
||||
#[tokio::test]
|
||||
async fn maintenance_gives_way_to_a_busy_family_and_succeeds_later() {
|
||||
let fixture = Fixture::new();
|
||||
stage(&fixture.view, &[]).unwrap();
|
||||
let abandoned = commit(&fixture.view, "abandoned", None);
|
||||
let locks = new_repo_init_locks();
|
||||
|
||||
// A push holds the family lease for as long as it runs.
|
||||
let push = fixture.storage.write_lease(&fixture.key).await.unwrap();
|
||||
assert!(maintain(&fixture.view, &locks).await.is_err());
|
||||
assert!(object_exists(&fixture.view, &abandoned).unwrap());
|
||||
|
||||
drop(push);
|
||||
assert_eq!(
|
||||
maintain(&fixture.view, &locks).await.unwrap(),
|
||||
Compaction::Done
|
||||
);
|
||||
assert!(!object_exists(&fixture.view, &abandoned).unwrap());
|
||||
}
|
||||
|
||||
#[tokio::test]
|
||||
async fn empty_prs_repository_is_removed_once_its_staging_is_reclaimed() {
|
||||
let submitter = "a".repeat(64);
|
||||
let fixture = Fixture::with_view(&format!("prs/{submitter}"));
|
||||
stage(&fixture.view, &[]).unwrap();
|
||||
commit(&fixture.view, "abandoned", None);
|
||||
|
||||
let outcome = maintain(&fixture.view, &new_repo_init_locks()).await;
|
||||
|
||||
assert_eq!(outcome.unwrap(), Compaction::Done);
|
||||
assert!(!fixture.view.exists());
|
||||
}
|
||||
}
|
||||
@@ -361,6 +361,20 @@ impl LocalGitStorage {
|
||||
command.env("GIT_OBJECT_DIRECTORY", self.family_objects_path(key));
|
||||
}
|
||||
|
||||
/// Make a Git command run in `view` write objects to the view's family.
|
||||
///
|
||||
/// A legacy repository without a family alternate keeps its own objects.
|
||||
pub fn write_to_family_of(view: &Path, command: &mut Command) {
|
||||
let alternates = std::fs::read_to_string(view.join("objects/info/alternates"));
|
||||
if let Some(objects) = alternates
|
||||
.ok()
|
||||
.as_deref()
|
||||
.and_then(|text| text.lines().next())
|
||||
{
|
||||
command.env("GIT_OBJECT_DIRECTORY", objects);
|
||||
}
|
||||
}
|
||||
|
||||
/// Keep an accepted object tip reachable without exposing it as a view ref.
|
||||
pub fn retain_tip(&self, key: &FamilyKey, source_ref: &str, oid: &str) -> Result<String> {
|
||||
self.update_internal_tip(key, RETAINED_REFS_PREFIX, source_ref, oid)
|
||||
|
||||
+52
-10
@@ -12,7 +12,8 @@ use super::protocol::GitService;
|
||||
|
||||
/// Git subprocess wrapper
|
||||
pub struct GitSubprocess {
|
||||
child: Child,
|
||||
/// `None` once the child has been reaped.
|
||||
child: Option<Child>,
|
||||
}
|
||||
|
||||
impl GitSubprocess {
|
||||
@@ -74,6 +75,7 @@ impl GitSubprocess {
|
||||
cmd.arg("--stateless-rpc");
|
||||
cmd.arg(repo_path);
|
||||
|
||||
cmd.kill_on_drop(true);
|
||||
cmd.stdin(Stdio::piped());
|
||||
cmd.stdout(Stdio::piped());
|
||||
cmd.stderr(Stdio::piped());
|
||||
@@ -89,47 +91,68 @@ impl GitSubprocess {
|
||||
|
||||
let child = cmd.spawn()?;
|
||||
|
||||
Ok(Self { child })
|
||||
Ok(Self { child: Some(child) })
|
||||
}
|
||||
|
||||
/// Get a mutable reference to stdin
|
||||
pub fn stdin(&mut self) -> Option<&mut (impl AsyncWrite + Unpin)> {
|
||||
self.child.stdin.as_mut()
|
||||
self.child.as_mut().expect("live child").stdin.as_mut()
|
||||
}
|
||||
|
||||
/// Get a mutable reference to stdout
|
||||
pub fn stdout(&mut self) -> Option<&mut (impl AsyncRead + Unpin)> {
|
||||
self.child.stdout.as_mut()
|
||||
self.child.as_mut().expect("live child").stdout.as_mut()
|
||||
}
|
||||
|
||||
/// Get a mutable reference to stderr
|
||||
pub fn stderr(&mut self) -> Option<&mut (impl AsyncRead + Unpin)> {
|
||||
self.child.stderr.as_mut()
|
||||
self.child.as_mut().expect("live child").stderr.as_mut()
|
||||
}
|
||||
|
||||
/// Take ownership of stdin
|
||||
pub fn take_stdin(&mut self) -> Option<impl AsyncWrite> {
|
||||
self.child.stdin.take()
|
||||
self.child.as_mut().expect("live child").stdin.take()
|
||||
}
|
||||
|
||||
/// Take ownership of stdout
|
||||
pub fn take_stdout(&mut self) -> Option<impl AsyncRead> {
|
||||
self.child.stdout.take()
|
||||
self.child.as_mut().expect("live child").stdout.take()
|
||||
}
|
||||
|
||||
/// Take ownership of stderr
|
||||
pub fn take_stderr(&mut self) -> Option<impl AsyncRead> {
|
||||
self.child.stderr.take()
|
||||
self.child.as_mut().expect("live child").stderr.take()
|
||||
}
|
||||
|
||||
/// Wait for the subprocess to complete
|
||||
pub async fn wait(mut self) -> std::io::Result<std::process::ExitStatus> {
|
||||
self.child.wait().await
|
||||
let result = self.child.as_mut().expect("live child").wait().await;
|
||||
if result.is_ok() {
|
||||
self.child.take();
|
||||
}
|
||||
result
|
||||
}
|
||||
|
||||
/// Kill the subprocess
|
||||
pub async fn kill(&mut self) -> std::io::Result<()> {
|
||||
self.child.kill().await
|
||||
self.child.as_mut().expect("live child").kill().await
|
||||
}
|
||||
}
|
||||
|
||||
impl Drop for GitSubprocess {
|
||||
fn drop(&mut self) {
|
||||
let Some(mut child) = self.child.take() else {
|
||||
return;
|
||||
};
|
||||
let _ = child.start_kill();
|
||||
if let Ok(runtime) = tokio::runtime::Handle::try_current() {
|
||||
// Blocking tasks cannot be aborted by task cancellation or runtime
|
||||
// shutdown, so the killed child is always reaped.
|
||||
let reaper = runtime.clone();
|
||||
runtime.spawn_blocking(move || {
|
||||
let _ = reaper.block_on(child.wait());
|
||||
});
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
@@ -150,6 +173,25 @@ mod tests {
|
||||
dir
|
||||
}
|
||||
|
||||
#[tokio::test]
|
||||
async fn dropped_subprocess_is_killed_and_reaped() {
|
||||
let repo = create_bare_repo();
|
||||
let child =
|
||||
GitSubprocess::spawn(GitService::ReceivePack, repo.path(), false, None).unwrap();
|
||||
let pid = child.child.as_ref().unwrap().id().unwrap();
|
||||
let process = std::path::PathBuf::from(format!("/proc/{pid}"));
|
||||
assert!(process.exists());
|
||||
drop(child);
|
||||
// A killed but unreaped child remains visible as a zombie.
|
||||
tokio::time::timeout(std::time::Duration::from_secs(5), async {
|
||||
while process.exists() {
|
||||
tokio::task::yield_now().await;
|
||||
}
|
||||
})
|
||||
.await
|
||||
.expect("dropped child was not reaped");
|
||||
}
|
||||
|
||||
#[tokio::test]
|
||||
async fn test_spawn_upload_pack_advertise() {
|
||||
let repo = create_bare_repo();
|
||||
|
||||
+95
-4
@@ -333,13 +333,18 @@ pub(crate) fn copy_single_commit_between_repos(
|
||||
target_repo.display()
|
||||
);
|
||||
|
||||
let output = Command::new("git")
|
||||
let mut fetch = Command::new("git");
|
||||
fetch
|
||||
.args([
|
||||
"fetch",
|
||||
source_repo.to_str().ok_or("Invalid source path")?,
|
||||
commit_hash,
|
||||
])
|
||||
.current_dir(target_repo)
|
||||
.current_dir(target_repo);
|
||||
// Copied history is accepted history: it belongs to the family, never to
|
||||
// the target view's own object directory.
|
||||
crate::git::storage::LocalGitStorage::write_to_family_of(target_repo, &mut fetch);
|
||||
let output = fetch
|
||||
.output()
|
||||
.map_err(|e| format!("Failed to execute git fetch: {}", e))?;
|
||||
|
||||
@@ -545,13 +550,16 @@ pub fn copy_missing_oids_between_repos(
|
||||
|
||||
// Fetch each missing commit from source to target
|
||||
for commit in &missing_commits {
|
||||
let output = Command::new("git")
|
||||
let mut fetch = Command::new("git");
|
||||
fetch
|
||||
.args([
|
||||
"fetch",
|
||||
source_repo.to_str().ok_or("Invalid source path")?,
|
||||
commit,
|
||||
])
|
||||
.current_dir(target_repo)
|
||||
.current_dir(target_repo);
|
||||
crate::git::storage::LocalGitStorage::write_to_family_of(target_repo, &mut fetch);
|
||||
let output = fetch
|
||||
.output()
|
||||
.map_err(|e| format!("Failed to execute git fetch: {}", e))?;
|
||||
|
||||
@@ -1161,6 +1169,16 @@ async fn process_purgatory_state_events(
|
||||
"State event author authorized via maintainer set"
|
||||
);
|
||||
|
||||
// An unsigned upload can satisfy a waiting State. Do not release it
|
||||
// from purgatory until its history is durable independently of staging.
|
||||
if let Err(error) = crate::git::staging::promote_state(source_repo_path, &state).await {
|
||||
result.errors.push(format!(
|
||||
"State {} history could not be stored durably: {error:#}",
|
||||
entry.event.id
|
||||
));
|
||||
continue;
|
||||
}
|
||||
|
||||
// Use unified processing function
|
||||
let process_result = crate::git::process::process_state_with_git_data(
|
||||
&state,
|
||||
@@ -1501,6 +1519,16 @@ async fn process_purgatory_pr_events(
|
||||
let owner_pubkey =
|
||||
extract_owner_from_repo_path(source_repo_path, git_data_path).unwrap_or_default();
|
||||
|
||||
// The event may only be accepted once the family holds its history.
|
||||
let tip = crate::git::staging::Tip::new(format!("refs/nostr/{}", event.id), &entry.commit);
|
||||
if let Err(error) = crate::git::staging::promote_accepted(source_repo_path, tip).await {
|
||||
result.errors.push(format!(
|
||||
"PR {} history could not be stored durably: {error:#}",
|
||||
event.id
|
||||
));
|
||||
continue;
|
||||
}
|
||||
|
||||
// Use unified processing function
|
||||
let process_result = crate::git::process::process_pr_with_git_data(
|
||||
event,
|
||||
@@ -2411,4 +2439,67 @@ mod tests {
|
||||
let result = is_latest_authorized_state(&state, &maintainers, std::slice::from_ref(&state));
|
||||
assert!(result);
|
||||
}
|
||||
|
||||
/// Commit written into `repo`'s own object directory.
|
||||
fn local_commit(repo: &Path, content: &str) -> String {
|
||||
let run = |args: &[&str], input: &str| {
|
||||
use std::io::Write;
|
||||
let mut child = Command::new("git")
|
||||
.current_dir(repo)
|
||||
.args(args)
|
||||
.env("GIT_AUTHOR_NAME", "Test")
|
||||
.env("GIT_AUTHOR_EMAIL", "test@example.com")
|
||||
.env("GIT_COMMITTER_NAME", "Test")
|
||||
.env("GIT_COMMITTER_EMAIL", "test@example.com")
|
||||
.stdin(std::process::Stdio::piped())
|
||||
.stdout(std::process::Stdio::piped())
|
||||
.spawn()
|
||||
.unwrap();
|
||||
child
|
||||
.stdin
|
||||
.take()
|
||||
.unwrap()
|
||||
.write_all(input.as_bytes())
|
||||
.unwrap();
|
||||
let output = child.wait_with_output().unwrap();
|
||||
assert!(output.status.success(), "git {args:?}");
|
||||
String::from_utf8(output.stdout).unwrap().trim().to_owned()
|
||||
};
|
||||
let blob = run(&["hash-object", "-w", "--stdin"], content);
|
||||
let tree = run(&["mktree"], &format!("100644 blob {blob}\tfile\n"));
|
||||
run(&["commit-tree", &tree, "-m", content], "")
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn copied_commit_is_written_to_the_family_not_the_target_view() {
|
||||
use crate::git::storage::{FamilyKey, LocalGitStorage};
|
||||
|
||||
let root = tempfile::tempdir().unwrap();
|
||||
let storage = LocalGitStorage::new(root.path().canonicalize().unwrap());
|
||||
let key = FamilyKey::sha1("repo").unwrap();
|
||||
storage.ensure_family(&key).unwrap();
|
||||
let view = |owner: &str| {
|
||||
let path = storage.git_data_path().join(owner).join("repo.git");
|
||||
std::fs::create_dir_all(path.parent().unwrap()).unwrap();
|
||||
storage.create_thin_view(&key, &path).unwrap();
|
||||
path
|
||||
};
|
||||
let (source, target) = (view("source"), view("target"));
|
||||
let commit = local_commit(&source, "only in the source view");
|
||||
let family = storage.family_repo_path(&key);
|
||||
assert!(!oid_exists(&family, &commit));
|
||||
|
||||
copy_single_commit_between_repos(&source, &target, &commit).unwrap();
|
||||
|
||||
assert!(oid_exists(&family, &commit));
|
||||
let local: Vec<_> = std::fs::read_dir(target.join("objects"))
|
||||
.unwrap()
|
||||
.map(|entry| entry.unwrap())
|
||||
.filter(|entry| entry.file_name() != "info" && entry.file_name() != "pack")
|
||||
.collect();
|
||||
assert!(local.is_empty(), "target view holds objects: {local:?}");
|
||||
assert!(std::fs::read_dir(target.join("objects/pack"))
|
||||
.map(|mut packs| packs.next().is_none())
|
||||
.unwrap_or(true));
|
||||
}
|
||||
}
|
||||
|
||||
@@ -110,6 +110,8 @@ pub fn scan_on_startup(git_data_path: &Path) -> (usize, usize) {
|
||||
}
|
||||
|
||||
match list_refs(&repo_path) {
|
||||
// Signed history still owed to the family outlives its refs.
|
||||
Ok(refs) if refs.is_empty() && crate::git::staging::owes_history(&repo_path) => {}
|
||||
Ok(refs) if refs.is_empty() => {
|
||||
if let Err(e) = std::fs::remove_dir_all(&repo_path) {
|
||||
warn!(
|
||||
|
||||
+72
-21
@@ -79,8 +79,8 @@ use crate::git::authorization::{
|
||||
use crate::git::handlers::{
|
||||
build_git_protocol_error_response, err_pktline_frame, is_git_protocol_error,
|
||||
pump_receive_pack_stdout_to_channel, read_stderr_to_end, record_git_operation,
|
||||
retain_accepted_tips, send_body_bytes, streaming_response, GitError, PumpResult,
|
||||
STREAM_CHANNEL_DEPTH,
|
||||
retain_accepted_tips, send_body_bytes, settle_staged_push, stage_push, streaming_response,
|
||||
GitError, PumpResult, STREAM_CHANNEL_DEPTH,
|
||||
};
|
||||
use crate::git::protocol::GitService;
|
||||
use crate::git::storage::{FamilyKey, FamilyWriteLease, LocalGitStorage};
|
||||
@@ -231,6 +231,7 @@ pub async fn handle_prs_receive_pack(
|
||||
identifier: &prs.identifier,
|
||||
domain,
|
||||
};
|
||||
let mut unsigned_refs = HashSet::new();
|
||||
for (_, new_oid, ref_name) in &pushed_refs {
|
||||
match pre_validate_refs_nostr_push(
|
||||
&database,
|
||||
@@ -255,7 +256,11 @@ pub async fn handle_prs_receive_pack(
|
||||
Some(&request_body),
|
||||
));
|
||||
}
|
||||
NostrRefPreValidation::Authorized { .. } | NostrRefPreValidation::Unknown => {}
|
||||
NostrRefPreValidation::Authorized { signed: true, .. } => {}
|
||||
NostrRefPreValidation::Authorized { signed: false, .. }
|
||||
| NostrRefPreValidation::Unknown => {
|
||||
unsigned_refs.insert(ref_name.clone());
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
@@ -301,12 +306,24 @@ pub async fn handle_prs_receive_pack(
|
||||
// write stdin here, then hand stdout/stderr and all follow-up state to a
|
||||
// detached task that feeds the response body channel.
|
||||
let use_family = storage.is_thin_view(&family_key, &repo_path);
|
||||
let staged = if use_family {
|
||||
match stage_push(&repo_path, &pushed_refs, &unsigned_refs).await {
|
||||
Ok(staged) => staged,
|
||||
Err(e) => {
|
||||
finish_prs_receive_pack(&state, &repo_path);
|
||||
record_git_operation(&metrics, "push", "error");
|
||||
return Err(e);
|
||||
}
|
||||
}
|
||||
} else {
|
||||
false
|
||||
};
|
||||
let mut git = match GitSubprocess::spawn_with_object_directory(
|
||||
GitService::ReceivePack,
|
||||
&repo_path,
|
||||
false,
|
||||
git_protocol,
|
||||
use_family.then_some(&family_lease.family_objects_path),
|
||||
(use_family && !staged).then_some(&family_lease.family_objects_path),
|
||||
)
|
||||
.map_err(GitError::ProcessSpawnFailed)
|
||||
{
|
||||
@@ -370,6 +387,7 @@ pub async fn handle_prs_receive_pack(
|
||||
storage,
|
||||
family_key,
|
||||
use_family.then_some(family_lease),
|
||||
staged,
|
||||
));
|
||||
|
||||
Ok(streaming_response(GitService::ReceivePack, rx))
|
||||
@@ -436,26 +454,52 @@ fn ensure_repo_initialised(
|
||||
fn finish_prs_receive_pack(state: &PrsPathState, repo_path: &Path) {
|
||||
let _g = state.mu.lock().expect("prs path mutex poisoned");
|
||||
state.in_flight.fetch_sub(1, Ordering::Relaxed);
|
||||
if state.in_flight.load(Ordering::Relaxed) == 0 {
|
||||
if let Ok(refs) = list_refs(repo_path) {
|
||||
if refs.is_empty() {
|
||||
if let Err(e) = std::fs::remove_dir_all(repo_path) {
|
||||
warn!(
|
||||
"/prs/ receive-pack: failed to clean up empty repo {}: {}",
|
||||
repo_path.display(),
|
||||
e
|
||||
);
|
||||
} else {
|
||||
debug!(
|
||||
"/prs/ receive-pack: removed empty repo {} (no refs after push)",
|
||||
repo_path.display()
|
||||
);
|
||||
}
|
||||
}
|
||||
remove_idle_empty_repo(state, repo_path);
|
||||
}
|
||||
|
||||
/// Remove a `/prs/` repository that has no refs and no push in flight, so
|
||||
/// abandoned repositories do not accumulate. Returns whether it was removed.
|
||||
///
|
||||
/// The caller holds `state.mu`. That mutex also gates `in_flight` updates, so
|
||||
/// a repository is never removed while a push is being received.
|
||||
pub(crate) fn remove_idle_empty_repo(state: &PrsPathState, repo_path: &Path) -> bool {
|
||||
if state.in_flight.load(Ordering::Relaxed) != 0
|
||||
|| !matches!(list_refs(repo_path), Ok(refs) if refs.is_empty())
|
||||
|| crate::git::staging::owes_history(repo_path)
|
||||
{
|
||||
return false;
|
||||
}
|
||||
match std::fs::remove_dir_all(repo_path) {
|
||||
Ok(()) => {
|
||||
debug!(repo = %repo_path.display(), "Removed zero-ref /prs/ repository");
|
||||
true
|
||||
}
|
||||
Err(error) => {
|
||||
warn!(
|
||||
repo = %repo_path.display(),
|
||||
%error,
|
||||
"Failed to remove zero-ref /prs/ repository"
|
||||
);
|
||||
false
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
/// [`remove_idle_empty_repo`] for callers that do not already hold the path
|
||||
/// mutex. Paths outside `/prs/` are left alone.
|
||||
pub fn remove_prs_repo_if_empty(
|
||||
locks: &RepoInitLocks,
|
||||
git_data_path: &Path,
|
||||
repo_path: &Path,
|
||||
) -> bool {
|
||||
if !crate::grasp06::paths::is_prs_repo_path(repo_path, git_data_path) {
|
||||
return false;
|
||||
}
|
||||
let state = path_state(locks, repo_path);
|
||||
let _g = state.mu.lock().expect("prs path mutex poisoned");
|
||||
remove_idle_empty_repo(&state, repo_path)
|
||||
}
|
||||
|
||||
/// Own the live `/prs/` receive-pack after the request handler has returned its
|
||||
/// streaming response.
|
||||
///
|
||||
@@ -493,6 +537,7 @@ async fn stream_prs_receive_pack_output<S, E>(
|
||||
storage: LocalGitStorage,
|
||||
family_key: FamilyKey,
|
||||
family_lease: Option<FamilyWriteLease>,
|
||||
staged: bool,
|
||||
) where
|
||||
S: tokio::io::AsyncRead + Unpin + Send + 'static,
|
||||
E: tokio::io::AsyncRead + Unpin + Send + 'static,
|
||||
@@ -612,10 +657,16 @@ async fn stream_prs_receive_pack_output<S, E>(
|
||||
.await;
|
||||
}
|
||||
|
||||
if family_lease.is_some() {
|
||||
if staged {
|
||||
settle_staged_push(&repo_path).await;
|
||||
} else if family_lease.is_some() {
|
||||
retain_accepted_tips(&storage, &family_key, &repo_path, &pushed_refs);
|
||||
}
|
||||
|
||||
// Release the family writer before post-push processing: promoting a
|
||||
// waiting event may need to re-enter this same identifier family.
|
||||
drop(family_lease);
|
||||
|
||||
finish_prs_receive_pack(&state, &repo_path);
|
||||
|
||||
// Drive the standard purgatory-release pipeline so PR events already
|
||||
|
||||
@@ -93,34 +93,16 @@ impl DeletionPolicy {
|
||||
.collect()
|
||||
}
|
||||
|
||||
fn cleanup_zero_ref_grasp06_repo_if_idle(&self, repo_path: &PathBuf) {
|
||||
if !crate::grasp06::paths::is_prs_repo_path(repo_path, &self.ctx.git_data_path) {
|
||||
return;
|
||||
}
|
||||
|
||||
let state = crate::grasp06::receive::path_state(&self.ctx.repo_init_locks, repo_path);
|
||||
let _guard = state.mu.lock().expect("prs path mutex poisoned");
|
||||
|
||||
if state.in_flight.load(std::sync::atomic::Ordering::Relaxed) != 0 {
|
||||
return;
|
||||
}
|
||||
|
||||
let is_zero_ref = matches!(git::list_refs(repo_path), Ok(refs) if refs.is_empty());
|
||||
if !is_zero_ref {
|
||||
return;
|
||||
}
|
||||
|
||||
if let Err(e) = std::fs::remove_dir_all(repo_path) {
|
||||
tracing::warn!(
|
||||
repo = %repo_path.display(),
|
||||
error = %e,
|
||||
"Failed to remove zero-ref /prs/ repo while processing deletion request"
|
||||
);
|
||||
return;
|
||||
}
|
||||
|
||||
if let Some(parent) = repo_path.parent() {
|
||||
let _ = std::fs::remove_dir(parent);
|
||||
fn cleanup_zero_ref_grasp06_repo_if_idle(&self, repo_path: &std::path::Path) {
|
||||
let removed = crate::grasp06::receive::remove_prs_repo_if_empty(
|
||||
&self.ctx.repo_init_locks,
|
||||
&self.ctx.git_data_path,
|
||||
repo_path,
|
||||
);
|
||||
if removed {
|
||||
if let Some(parent) = repo_path.parent() {
|
||||
let _ = std::fs::remove_dir(parent);
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
|
||||
@@ -188,9 +188,12 @@ impl DeletionService {
|
||||
};
|
||||
|
||||
if !crate::git::oid_exists(target_repo, oid) {
|
||||
let fetch_output = Command::new("git")
|
||||
let mut fetch = Command::new("git");
|
||||
fetch
|
||||
.args(["fetch", source_repo_str, oid])
|
||||
.current_dir(target_repo)
|
||||
.current_dir(target_repo);
|
||||
LocalGitStorage::write_to_family_of(target_repo, &mut fetch);
|
||||
let fetch_output = fetch
|
||||
.output()
|
||||
.map_err(|e| anyhow::anyhow!("failed to fetch archived OID {oid}: {e}"))?;
|
||||
if !fetch_output.status.success() {
|
||||
|
||||
@@ -65,16 +65,52 @@ impl PrEventPolicy {
|
||||
}
|
||||
};
|
||||
|
||||
// Check for placeholder first (git-data-first scenario)
|
||||
if let Some(placeholder_commit) = self.ctx.purgatory.find_pr_placeholder(&event_id) {
|
||||
// Read the full entry so we can inspect any GRASP-06 scope
|
||||
// recorded when the placeholder was created from a /prs/ push.
|
||||
let prs_scope = self
|
||||
.ctx
|
||||
.purgatory
|
||||
.find_pr(&event_id)
|
||||
.and_then(|entry| entry.prs_scope);
|
||||
|
||||
// A standard-endpoint placeholder is released only once the event's
|
||||
// history is durable or no local upload serves the event.
|
||||
let mut standard_placeholder = false;
|
||||
// A crash may happen after Git installs the ref but before the
|
||||
// purgatory checkpoint. Recover the /prs/ scope only from a signed,
|
||||
// service-local clone URL for this signer and an exact event-id/OID ref.
|
||||
let mut placeholder = self
|
||||
.ctx
|
||||
.purgatory
|
||||
.find_pr_placeholder(&event_id)
|
||||
.map(|commit| {
|
||||
let scope = self
|
||||
.ctx
|
||||
.purgatory
|
||||
.find_pr(&event_id)
|
||||
.and_then(|entry| entry.prs_scope);
|
||||
(commit, scope)
|
||||
});
|
||||
if placeholder.is_none() && self.ctx.config.grasp06_enable {
|
||||
for identifier in crate::grasp06::policy::prs_identifiers_named_by_event_clone_tags(
|
||||
event,
|
||||
&self.ctx.config.service_address(),
|
||||
) {
|
||||
if !git::validate_repository_identifier(&identifier) {
|
||||
continue;
|
||||
}
|
||||
let path = crate::grasp06::paths::prs_repo_path(
|
||||
&self.ctx.git_data_path,
|
||||
&event.pubkey.to_hex(),
|
||||
&identifier,
|
||||
);
|
||||
if git::get_ref_commit(&path, &format!("refs/nostr/{event_id}")).as_deref()
|
||||
== Some(commit.as_str())
|
||||
{
|
||||
placeholder = Some((
|
||||
commit.clone(),
|
||||
Some(crate::purgatory::PrsPlaceholderScope {
|
||||
submitter: event.pubkey,
|
||||
identifier,
|
||||
}),
|
||||
));
|
||||
break;
|
||||
}
|
||||
}
|
||||
}
|
||||
if let Some((placeholder_commit, prs_scope)) = placeholder {
|
||||
if let Some(scope) = prs_scope {
|
||||
// The placeholder was created by a /prs/<submitter>/<id>.git
|
||||
// push (06.md line 12). The arriving event MUST be signed by
|
||||
@@ -146,29 +182,8 @@ impl PrEventPolicy {
|
||||
error = %e,
|
||||
"Failed to delete /prs/ ref while discarding scoped placeholder",
|
||||
);
|
||||
} else if state.in_flight.load(std::sync::atomic::Ordering::Relaxed) == 0
|
||||
&& matches!(
|
||||
crate::git::list_refs(&prs_repo),
|
||||
Ok(refs) if refs.is_empty()
|
||||
)
|
||||
{
|
||||
// No refs left and no push in flight — the
|
||||
// repo is now an empty husk. Drop the bare
|
||||
// dir so /prs/ doesn't accumulate abandoned
|
||||
// repos. Equivalent to the cleanup the
|
||||
// receive handler does on push completion.
|
||||
if let Err(e) = std::fs::remove_dir_all(&prs_repo) {
|
||||
tracing::warn!(
|
||||
repo = %prs_repo.display(),
|
||||
error = %e,
|
||||
"Failed to remove zero-ref /prs/ repo after discarding scoped placeholder",
|
||||
);
|
||||
} else {
|
||||
tracing::debug!(
|
||||
repo = %prs_repo.display(),
|
||||
"Removed zero-ref /prs/ repo after discarding scoped placeholder",
|
||||
);
|
||||
}
|
||||
} else {
|
||||
crate::grasp06::receive::remove_idle_empty_repo(&state, &prs_repo);
|
||||
}
|
||||
}
|
||||
self.ctx.purgatory.remove_pr(&event_id);
|
||||
@@ -211,6 +226,11 @@ impl PrEventPolicy {
|
||||
)
|
||||
.await?;
|
||||
|
||||
// The placeholder stays until the history is durable, so a
|
||||
// failure here leaves the upload to expire normally.
|
||||
self.promote_staged_history(&prs_repo, &event_id, &commit)
|
||||
.await?;
|
||||
|
||||
let process_result = crate::git::process::process_pr_with_git_data(
|
||||
event,
|
||||
&commit,
|
||||
@@ -267,7 +287,7 @@ impl PrEventPolicy {
|
||||
event_id,
|
||||
commit
|
||||
);
|
||||
self.ctx.purgatory.remove_pr(&event_id);
|
||||
standard_placeholder = true;
|
||||
} else {
|
||||
// Standard endpoint placeholder, mismatched commit — original
|
||||
// behaviour: incoming event supersedes.
|
||||
@@ -277,7 +297,7 @@ impl PrEventPolicy {
|
||||
commit,
|
||||
placeholder_commit
|
||||
);
|
||||
self.ctx.purgatory.remove_pr(&event_id);
|
||||
standard_placeholder = true;
|
||||
// Delete incorrect git data (refs/nostr/<event-id>) will be handled below
|
||||
}
|
||||
}
|
||||
@@ -285,6 +305,9 @@ impl PrEventPolicy {
|
||||
let repo_paths = self.find_relevant_repo_paths(event).await?;
|
||||
|
||||
if repo_paths.is_empty() {
|
||||
if standard_placeholder {
|
||||
self.ctx.purgatory.remove_pr(&event_id);
|
||||
}
|
||||
tracing::debug!("No repository paths found for PR event {}", event_id);
|
||||
return Ok(false);
|
||||
}
|
||||
@@ -345,6 +368,14 @@ impl PrEventPolicy {
|
||||
)
|
||||
.unwrap_or_default();
|
||||
|
||||
// The placeholder stays until the history is durable, so a
|
||||
// failure here leaves the upload to expire normally.
|
||||
self.promote_staged_history(&source_repo, &event_id, &commit)
|
||||
.await?;
|
||||
if standard_placeholder {
|
||||
self.ctx.purgatory.remove_pr(&event_id);
|
||||
}
|
||||
|
||||
// Use unified processing function
|
||||
let result = crate::git::process::process_pr_with_git_data(
|
||||
event,
|
||||
@@ -376,6 +407,9 @@ impl PrEventPolicy {
|
||||
|
||||
Ok(true)
|
||||
} else {
|
||||
if standard_placeholder {
|
||||
self.ctx.purgatory.remove_pr(&event_id);
|
||||
}
|
||||
tracing::debug!(
|
||||
"No git data found for PR event {} with commit {}",
|
||||
event_id,
|
||||
@@ -385,6 +419,31 @@ impl PrEventPolicy {
|
||||
}
|
||||
}
|
||||
|
||||
/// Make the history of an accepted PR tip durable in the family.
|
||||
///
|
||||
/// An upload received before its event was known is staged in its view.
|
||||
/// The event may only be accepted once the family alone holds the history.
|
||||
async fn promote_staged_history(
|
||||
&self,
|
||||
repo: &std::path::Path,
|
||||
event_id: &str,
|
||||
commit: &str,
|
||||
) -> Result<()> {
|
||||
let tip = git::staging::Tip::new(format!("refs/nostr/{event_id}"), commit);
|
||||
git::staging::promote_accepted(repo, tip)
|
||||
.await
|
||||
.map_err(|error| {
|
||||
tracing::error!(
|
||||
event_id = %event_id,
|
||||
commit = %commit,
|
||||
repo = %repo.display(),
|
||||
error = %format!("{error:#}"),
|
||||
"Cannot store PR history durably; event not accepted"
|
||||
);
|
||||
anyhow::anyhow!("PR history could not be stored durably")
|
||||
})
|
||||
}
|
||||
|
||||
async fn find_relevant_repo_paths(&self, event: &Event) -> Result<Vec<std::path::PathBuf>> {
|
||||
// Extract ALL `a` tags (repository references) from the PR event
|
||||
let repo_refs: Vec<String> = event
|
||||
|
||||
@@ -218,6 +218,12 @@ impl StatePolicy {
|
||||
event.id,
|
||||
);
|
||||
|
||||
// Ref alignment can consume staged history without another push.
|
||||
// Make it durable before accepting the State or moving any refs.
|
||||
git::staging::promote_state(&repo_with_git_data, &state)
|
||||
.await
|
||||
.context("State history could not be stored durably")?;
|
||||
|
||||
// Use unified processing function
|
||||
let result = crate::git::process::process_state_with_git_data(
|
||||
&state,
|
||||
|
||||
+204
-26
@@ -40,7 +40,7 @@ use std::time::{Duration, Instant, SystemTime};
|
||||
pub use sync::SyncQueueEntry;
|
||||
|
||||
/// Default expiry duration for purgatory entries (30 minutes)
|
||||
const DEFAULT_EXPIRY: Duration = Duration::from_secs(1800);
|
||||
pub(crate) const DEFAULT_EXPIRY: Duration = Duration::from_secs(1800);
|
||||
|
||||
/// Extended expiry for soft-expired announcements (24 hours).
|
||||
///
|
||||
@@ -101,6 +101,8 @@ struct SerializablePrPurgatoryEntry {
|
||||
/// deserialisable.
|
||||
#[serde(default)]
|
||||
prs_scope: Option<types::PrsPlaceholderScope>,
|
||||
#[serde(default)]
|
||||
standard_refs: Vec<types::StandardPrRef>,
|
||||
}
|
||||
|
||||
/// Serializable wrapper for `AnnouncementPurgatoryEntry` with time offsets.
|
||||
@@ -200,7 +202,7 @@ pub struct Purgatory {
|
||||
/// Stored as EventId (hex string) for efficient lookup.
|
||||
expired_events: Arc<DashMap<EventId, Instant>>,
|
||||
|
||||
_git_data_path: PathBuf,
|
||||
git_data_path: PathBuf,
|
||||
|
||||
/// Set once at startup by [`Purgatory::set_prs_cleanup_ctx`] to give
|
||||
/// [`Purgatory::cleanup`] enough information to remove abandoned
|
||||
@@ -234,7 +236,7 @@ impl Purgatory {
|
||||
pr_events: Arc::new(DashMap::new()),
|
||||
sync_queue: Arc::new(DashMap::new()),
|
||||
expired_events: Arc::new(DashMap::new()),
|
||||
_git_data_path: git_data_path.into(),
|
||||
git_data_path: git_data_path.into(),
|
||||
prs_cleanup_ctx: std::sync::OnceLock::new(),
|
||||
}
|
||||
}
|
||||
@@ -493,6 +495,7 @@ impl Purgatory {
|
||||
created_at: now,
|
||||
expires_at: now + DEFAULT_EXPIRY,
|
||||
source,
|
||||
standard_refs: Vec::new(),
|
||||
prs_scope: None,
|
||||
};
|
||||
|
||||
@@ -521,12 +524,185 @@ impl Purgatory {
|
||||
created_at: now,
|
||||
expires_at: now + DEFAULT_EXPIRY,
|
||||
source: types::EventSource::Direct, // Git pushes are direct user actions
|
||||
standard_refs: Vec::new(),
|
||||
prs_scope: None,
|
||||
};
|
||||
|
||||
self.pr_events.insert(event_id, entry);
|
||||
}
|
||||
|
||||
/// Record a normal-endpoint push without forgetting other copies of the ref.
|
||||
pub fn add_standard_pr_placeholder(
|
||||
&self,
|
||||
event_id: String,
|
||||
commit: String,
|
||||
owner: PublicKey,
|
||||
identifier: String,
|
||||
) {
|
||||
let now = Instant::now();
|
||||
let mut entry = self
|
||||
.pr_events
|
||||
.entry(event_id)
|
||||
.or_insert_with(|| PrPurgatoryEntry {
|
||||
event: None,
|
||||
commit: commit.clone(),
|
||||
created_at: now,
|
||||
expires_at: now + DEFAULT_EXPIRY,
|
||||
source: types::EventSource::Direct,
|
||||
prs_scope: None,
|
||||
standard_refs: Vec::new(),
|
||||
});
|
||||
if entry.event.is_some() {
|
||||
return;
|
||||
}
|
||||
entry
|
||||
.standard_refs
|
||||
.retain(|r| r.owner != owner || r.identifier != identifier);
|
||||
entry.standard_refs.push(types::StandardPrRef {
|
||||
owner,
|
||||
identifier,
|
||||
commit,
|
||||
});
|
||||
entry.expires_at = now + DEFAULT_EXPIRY;
|
||||
}
|
||||
|
||||
/// Restore a staged upload before serving requests. Never replace a signed
|
||||
/// purgatory event or shorten a newer checkpoint's placeholder lifetime.
|
||||
pub(crate) fn recover_upload_placeholder(
|
||||
&self,
|
||||
event_id: String,
|
||||
commit: String,
|
||||
destination: (PublicKey, String),
|
||||
prs: bool,
|
||||
expires_at: SystemTime,
|
||||
) {
|
||||
let (owner, identifier) = destination;
|
||||
let existing = self.find_pr(&event_id);
|
||||
if existing.as_ref().is_some_and(|entry| entry.event.is_some()) {
|
||||
return;
|
||||
}
|
||||
let deadline = Instant::now()
|
||||
+ expires_at
|
||||
.duration_since(SystemTime::now())
|
||||
.unwrap_or_default();
|
||||
let deadline = existing.map_or(deadline, |entry| deadline.max(entry.expires_at));
|
||||
if prs {
|
||||
self.add_prs_pr_placeholder(event_id.clone(), commit, owner, identifier);
|
||||
} else {
|
||||
self.add_standard_pr_placeholder(event_id.clone(), commit, owner, identifier);
|
||||
}
|
||||
if let Some(mut entry) = self.pr_events.get_mut(&event_id) {
|
||||
entry.expires_at = deadline;
|
||||
}
|
||||
}
|
||||
|
||||
/// Expire normal-endpoint placeholders under the same lifecycle locks as pushes.
|
||||
/// Recheck the entry after waiting: an event or a new push may have arrived.
|
||||
/// Failed Git operations retain the record for the next sweep.
|
||||
pub async fn cleanup_standard_pr_refs(
|
||||
&self,
|
||||
lifecycle: &crate::nostr::lifecycle::RepositoryLifecycle,
|
||||
database: &crate::nostr::SharedDatabase,
|
||||
) -> usize {
|
||||
let candidates: Vec<_> = self
|
||||
.pr_events
|
||||
.iter()
|
||||
.filter(|e| {
|
||||
e.event.is_none() && !e.standard_refs.is_empty() && e.expires_at <= Instant::now()
|
||||
})
|
||||
.map(|e| (e.key().clone(), e.standard_refs.clone()))
|
||||
.collect();
|
||||
let mut removed = 0;
|
||||
for (id, refs) in candidates {
|
||||
let _guards = lifecycle
|
||||
.write_repositories(refs.iter().map(|r| crate::nostr::lifecycle::RecoveryScope {
|
||||
owner_pubkey_hex: r.owner.to_hex(),
|
||||
identifier: r.identifier.clone(),
|
||||
}))
|
||||
.await;
|
||||
// A checkpoint can retain a placeholder after its event was accepted.
|
||||
// Missing metadata must never cause us to delete a now-authorized ref.
|
||||
let accepted = match EventId::from_hex(&id) {
|
||||
Ok(event_id) => match database.event_by_id(&event_id).await {
|
||||
Ok(event) => event.is_some(),
|
||||
Err(error) => {
|
||||
tracing::warn!(%id, %error, "Cannot check PR acceptance; deferring ref expiry");
|
||||
continue;
|
||||
}
|
||||
},
|
||||
Err(_) => continue,
|
||||
};
|
||||
// Holding the map entry serializes event arrival/removal and placeholder refresh.
|
||||
let dashmap::mapref::entry::Entry::Occupied(mut slot) =
|
||||
self.pr_events.entry(id.clone())
|
||||
else {
|
||||
continue;
|
||||
};
|
||||
let entry = slot.get_mut();
|
||||
let should_remove = (|| {
|
||||
if entry.event.is_some()
|
||||
|| entry.expires_at > Instant::now()
|
||||
|| entry.standard_refs != refs
|
||||
{
|
||||
return false;
|
||||
}
|
||||
if accepted {
|
||||
return true;
|
||||
}
|
||||
let mut ok = true;
|
||||
for r in &refs {
|
||||
let path = self
|
||||
.git_data_path
|
||||
.join(r.owner.to_bech32().expect("public key"))
|
||||
.join(format!("{}.git", r.identifier));
|
||||
match path.try_exists() {
|
||||
Ok(false) => continue,
|
||||
Err(error) => {
|
||||
tracing::warn!(repo = %path.display(), %error, "Cannot inspect PR repository; deferring expiry");
|
||||
ok = false;
|
||||
continue;
|
||||
}
|
||||
Ok(true) => {}
|
||||
}
|
||||
let reference = format!("refs/nostr/{id}");
|
||||
// A previous partial sweep or a rejected push may leave no ref.
|
||||
match crate::git::list_refs(&path) {
|
||||
Ok(refs) if !refs.iter().any(|(name, _)| name == &reference) => continue,
|
||||
Err(error) => {
|
||||
tracing::warn!(repo = %path.display(), %error, "Cannot inspect PR refs; deferring expiry");
|
||||
ok = false;
|
||||
continue;
|
||||
}
|
||||
Ok(_) => {}
|
||||
}
|
||||
// Compare-and-delete prevents expiry from removing a replaced ref.
|
||||
let result = std::process::Command::new("git")
|
||||
.args(["update-ref", "-d", &reference, &r.commit])
|
||||
.current_dir(&path)
|
||||
.output();
|
||||
if !matches!(result, Ok(ref output) if output.status.success()) {
|
||||
tracing::warn!(repo = %path.display(), %reference,
|
||||
"Failed to expire normal PR ref; retaining placeholder for retry");
|
||||
ok = false;
|
||||
} else {
|
||||
// The abandoned upload's staged objects can now be reclaimed.
|
||||
crate::git::staging::request_maintenance(&path);
|
||||
}
|
||||
}
|
||||
if ok && entry.prs_scope.is_some() {
|
||||
entry.standard_refs.clear();
|
||||
return false; // The synchronous sweep still owns the /prs/ ref.
|
||||
}
|
||||
ok
|
||||
})();
|
||||
if should_remove {
|
||||
slot.remove();
|
||||
removed += 1;
|
||||
}
|
||||
}
|
||||
removed
|
||||
}
|
||||
|
||||
/// Add a PR placeholder created by a push to the GRASP-06 `/prs/`
|
||||
/// endpoint (06.md line 12).
|
||||
///
|
||||
@@ -552,19 +728,31 @@ impl Purgatory {
|
||||
identifier: String,
|
||||
) {
|
||||
let now = Instant::now();
|
||||
let entry = PrPurgatoryEntry {
|
||||
let mut slot = self
|
||||
.pr_events
|
||||
.entry(event_id)
|
||||
.or_insert_with(|| PrPurgatoryEntry {
|
||||
event: None,
|
||||
commit: commit.clone(),
|
||||
created_at: now,
|
||||
expires_at: now + DEFAULT_EXPIRY,
|
||||
source: types::EventSource::Direct,
|
||||
prs_scope: None,
|
||||
standard_refs: Vec::new(),
|
||||
});
|
||||
let standard_refs = std::mem::take(&mut slot.standard_refs);
|
||||
*slot = PrPurgatoryEntry {
|
||||
event: None,
|
||||
commit,
|
||||
created_at: now,
|
||||
expires_at: now + DEFAULT_EXPIRY,
|
||||
source: types::EventSource::Direct,
|
||||
standard_refs,
|
||||
prs_scope: Some(types::PrsPlaceholderScope {
|
||||
submitter,
|
||||
identifier,
|
||||
}),
|
||||
};
|
||||
|
||||
self.pr_events.insert(event_id, entry);
|
||||
}
|
||||
|
||||
/// Find state events waiting for a specific repository identifier.
|
||||
@@ -977,7 +1165,7 @@ impl Purgatory {
|
||||
if let Some((repo_path, was_soft_expired)) = revival_info {
|
||||
if was_soft_expired {
|
||||
if !repo_path.exists() {
|
||||
let storage = crate::git::storage::LocalGitStorage::new(&self._git_data_path);
|
||||
let storage = crate::git::storage::LocalGitStorage::new(&self.git_data_path);
|
||||
let result = crate::git::storage::FamilyKey::sha1(identifier)
|
||||
.and_then(|family| storage.create_thin_view(&family, &repo_path));
|
||||
match result {
|
||||
@@ -1301,7 +1489,8 @@ impl Purgatory {
|
||||
.iter()
|
||||
.filter(|entry| {
|
||||
let value = entry.value();
|
||||
value.expires_at <= now
|
||||
value.standard_refs.is_empty()
|
||||
&& value.expires_at <= now
|
||||
&& !value
|
||||
.event
|
||||
.as_ref()
|
||||
@@ -1433,24 +1622,8 @@ impl Purgatory {
|
||||
error = %e,
|
||||
"Failed to delete dangling /prs/ ref during purgatory expiry",
|
||||
);
|
||||
} else if state.in_flight.load(std::sync::atomic::Ordering::Relaxed) == 0
|
||||
&& matches!(
|
||||
crate::git::list_refs(&repo_path),
|
||||
Ok(refs) if refs.is_empty()
|
||||
)
|
||||
{
|
||||
if let Err(e) = std::fs::remove_dir_all(&repo_path) {
|
||||
tracing::warn!(
|
||||
repo = %repo_path.display(),
|
||||
error = %e,
|
||||
"Failed to remove zero-ref /prs/ repo during purgatory expiry",
|
||||
);
|
||||
} else {
|
||||
tracing::debug!(
|
||||
repo = %repo_path.display(),
|
||||
"Removed zero-ref /prs/ repo during purgatory expiry",
|
||||
);
|
||||
}
|
||||
} else {
|
||||
crate::grasp06::receive::remove_idle_empty_repo(&state, &repo_path);
|
||||
}
|
||||
}
|
||||
}
|
||||
@@ -1658,6 +1831,7 @@ impl Purgatory {
|
||||
expires_at_offset_secs: expires_offset.as_secs(),
|
||||
source: e.source,
|
||||
prs_scope: e.prs_scope.clone(),
|
||||
standard_refs: e.standard_refs.clone(),
|
||||
};
|
||||
pr_events.insert(event_id, serializable);
|
||||
}
|
||||
@@ -1830,6 +2004,7 @@ impl Purgatory {
|
||||
expires_at,
|
||||
source: e.source,
|
||||
prs_scope: e.prs_scope,
|
||||
standard_refs: e.standard_refs,
|
||||
};
|
||||
|
||||
self.pr_events.insert(event_id, entry);
|
||||
@@ -3547,3 +3722,6 @@ fn add_prs_pr_placeholder_does_not_overwrite_existing_scoped_placeholder() {
|
||||
assert_eq!(scope.submitter, submitter_b.public_key());
|
||||
assert_eq!(scope.identifier, "repo-b");
|
||||
}
|
||||
|
||||
#[cfg(test)]
|
||||
mod standard_ref_tests;
|
||||
|
||||
@@ -0,0 +1,281 @@
|
||||
use super::*;
|
||||
use crate::nostr::lifecycle::RepositoryLifecycle;
|
||||
fn database() -> crate::nostr::SharedDatabase {
|
||||
Arc::new(nostr_memory::MemoryDatabase::unbounded())
|
||||
}
|
||||
fn refs(path: &Path) -> Result<std::collections::BTreeMap<String, String>, String> {
|
||||
crate::git::list_refs(path).map(|refs| refs.into_iter().collect())
|
||||
}
|
||||
|
||||
fn git(path: &Path, args: &[&str]) -> String {
|
||||
let output = std::process::Command::new("git")
|
||||
.current_dir(path)
|
||||
.env("GIT_AUTHOR_NAME", "Test")
|
||||
.env("GIT_AUTHOR_EMAIL", "test@example.com")
|
||||
.env("GIT_COMMITTER_NAME", "Test")
|
||||
.env("GIT_COMMITTER_EMAIL", "test@example.com")
|
||||
.args(args)
|
||||
.output()
|
||||
.unwrap();
|
||||
assert!(
|
||||
output.status.success(),
|
||||
"{}",
|
||||
String::from_utf8_lossy(&output.stderr)
|
||||
);
|
||||
String::from_utf8(output.stdout).unwrap().trim().to_owned()
|
||||
}
|
||||
|
||||
fn repository(root: &Path, owner: PublicKey, name: &str) -> (PathBuf, String) {
|
||||
let path = root
|
||||
.join(owner.to_bech32().unwrap())
|
||||
.join(format!("{name}.git"));
|
||||
std::fs::create_dir_all(&path).unwrap();
|
||||
git(&path, &["init", "--bare"]);
|
||||
let tree = git(&path, &["mktree"]);
|
||||
let commit = git(&path, &["commit-tree", &tree, "-m", "test"]);
|
||||
(path, commit)
|
||||
}
|
||||
|
||||
fn expire(p: &Purgatory, id: &str) {
|
||||
p.pr_events.get_mut(id).unwrap().expires_at = Instant::now() - Duration::from_secs(1);
|
||||
}
|
||||
|
||||
#[tokio::test]
|
||||
async fn standard_refs_expire_after_restart_without_touching_other_refs() {
|
||||
let root = tempfile::tempdir().unwrap();
|
||||
let owner = Keys::generate().public_key();
|
||||
let p = Purgatory::new(root.path());
|
||||
let id = "ab".repeat(32);
|
||||
let reference = format!("refs/nostr/{id}");
|
||||
let mut paths = Vec::new();
|
||||
for name in ["one", "two"] {
|
||||
let (path, commit) = repository(root.path(), owner, name);
|
||||
git(&path, &["update-ref", &reference, &commit]);
|
||||
git(&path, &["update-ref", "refs/heads/main", &commit]);
|
||||
p.add_standard_pr_placeholder(id.clone(), commit, owner, name.into());
|
||||
paths.push(path);
|
||||
}
|
||||
let (unrelated, commit) = repository(root.path(), owner, "unrelated");
|
||||
git(&unrelated, &["update-ref", &reference, &commit]);
|
||||
let state = root.path().join("purgatory.json");
|
||||
p.save_to_disk(&state).unwrap();
|
||||
let restored = Purgatory::new(root.path());
|
||||
restored.restore_from_disk(&state).unwrap();
|
||||
assert_eq!(restored.find_pr(&id).unwrap().standard_refs.len(), 2);
|
||||
expire(&restored, &id);
|
||||
assert_eq!(
|
||||
restored.cleanup().2,
|
||||
0,
|
||||
"sync sweep must retain cleanup metadata"
|
||||
);
|
||||
assert_eq!(
|
||||
restored
|
||||
.cleanup_standard_pr_refs(&RepositoryLifecycle::in_memory(), &database())
|
||||
.await,
|
||||
1
|
||||
);
|
||||
assert!(restored.find_pr(&id).is_none());
|
||||
for path in paths {
|
||||
let refs = refs(&path).unwrap();
|
||||
assert!(!refs.contains_key(&reference));
|
||||
assert!(refs.contains_key("refs/heads/main"));
|
||||
}
|
||||
assert!(refs(&unrelated).unwrap().contains_key(&reference));
|
||||
}
|
||||
|
||||
#[tokio::test]
|
||||
async fn standard_ref_expiry_retries_lock_failure_and_preserves_replaced_ref() {
|
||||
let root = tempfile::tempdir().unwrap();
|
||||
let owner = Keys::generate().public_key();
|
||||
let (path, commit) = repository(root.path(), owner, "one");
|
||||
let p = Purgatory::new(root.path());
|
||||
let id = "cd".repeat(32);
|
||||
let reference = format!("refs/nostr/{id}");
|
||||
git(&path, &["update-ref", &reference, &commit]);
|
||||
p.add_standard_pr_placeholder(id.clone(), commit.clone(), owner, "one".into());
|
||||
expire(&p, &id);
|
||||
let lifecycle = RepositoryLifecycle::in_memory();
|
||||
let db = database();
|
||||
let lock = path.join(format!("{reference}.lock"));
|
||||
std::fs::write(&lock, b"").unwrap();
|
||||
assert_eq!(p.cleanup_standard_pr_refs(&lifecycle, &db).await, 0);
|
||||
assert!(p.find_pr(&id).is_some());
|
||||
std::fs::remove_file(lock).unwrap();
|
||||
let tree = git(&path, &["mktree"]);
|
||||
let replacement = git(&path, &["commit-tree", &tree, "-m", "replacement"]);
|
||||
git(&path, &["update-ref", &reference, &replacement]);
|
||||
assert_eq!(p.cleanup_standard_pr_refs(&lifecycle, &db).await, 0);
|
||||
assert_eq!(refs(&path).unwrap()[&reference], replacement);
|
||||
git(&path, &["update-ref", &reference, &commit]);
|
||||
assert_eq!(p.cleanup_standard_pr_refs(&lifecycle, &db).await, 1);
|
||||
}
|
||||
|
||||
#[tokio::test]
|
||||
async fn standard_ref_expiry_rechecks_refresh_and_event_arrival_after_push_lock() {
|
||||
let root = tempfile::tempdir().unwrap();
|
||||
let owner = Keys::generate().public_key();
|
||||
let (path, commit) = repository(root.path(), owner, "one");
|
||||
let p = Purgatory::new(root.path());
|
||||
let id = "ef".repeat(32);
|
||||
let reference = format!("refs/nostr/{id}");
|
||||
git(&path, &["update-ref", &reference, &commit]);
|
||||
let lifecycle = RepositoryLifecycle::in_memory();
|
||||
let db = database();
|
||||
for event_arrives in [false, true] {
|
||||
p.add_standard_pr_placeholder(id.clone(), commit.clone(), owner, "one".into());
|
||||
expire(&p, &id);
|
||||
let guard = lifecycle.read_repository(&owner.to_hex(), "one").await;
|
||||
let cleanup = p.cleanup_standard_pr_refs(&lifecycle, &db);
|
||||
tokio::pin!(cleanup);
|
||||
assert!(futures_util::poll!(&mut cleanup).is_pending());
|
||||
if event_arrives {
|
||||
p.remove_pr(&id);
|
||||
} else {
|
||||
p.add_standard_pr_placeholder(id.clone(), commit.clone(), owner, "one".into());
|
||||
}
|
||||
drop(guard);
|
||||
assert_eq!(
|
||||
tokio::time::timeout(Duration::from_secs(5), cleanup)
|
||||
.await
|
||||
.unwrap(),
|
||||
0
|
||||
);
|
||||
assert!(refs(&path).unwrap().contains_key(&reference));
|
||||
}
|
||||
}
|
||||
|
||||
#[tokio::test]
|
||||
async fn standard_push_authorization_records_each_repository_and_legacy_state_still_loads() {
|
||||
let root = tempfile::tempdir().unwrap();
|
||||
let owner = Keys::generate().public_key();
|
||||
let database: crate::nostr::SharedDatabase =
|
||||
Arc::new(nostr_memory::MemoryDatabase::unbounded());
|
||||
let p = Arc::new(Purgatory::new(root.path()));
|
||||
let id = "12".repeat(32);
|
||||
let commit = "34".repeat(20);
|
||||
let line = format!(
|
||||
"{} {commit} refs/nostr/{id}\0report-status\n",
|
||||
"0".repeat(40)
|
||||
);
|
||||
let body = hyper::body::Bytes::from(format!("{:04x}{line}0000", line.len() + 4));
|
||||
for name in ["one", "two", "one"] {
|
||||
let result = crate::git::authorization::authorize_push(
|
||||
&database,
|
||||
name,
|
||||
&owner.to_hex(),
|
||||
&body,
|
||||
&p,
|
||||
root.path(),
|
||||
)
|
||||
.await
|
||||
.unwrap();
|
||||
assert!(result.authorized);
|
||||
}
|
||||
assert_eq!(p.find_pr(&id).unwrap().standard_refs.len(), 2);
|
||||
let state = root.path().join("state.json");
|
||||
p.save_to_disk(&state).unwrap();
|
||||
let mut json: serde_json::Value =
|
||||
serde_json::from_str(&std::fs::read_to_string(&state).unwrap()).unwrap();
|
||||
for entry in json["pr_events"].as_object_mut().unwrap().values_mut() {
|
||||
entry.as_object_mut().unwrap().remove("standard_refs");
|
||||
}
|
||||
std::fs::write(&state, serde_json::to_vec(&json).unwrap()).unwrap();
|
||||
let restored = Purgatory::new(root.path());
|
||||
restored.restore_from_disk(&state).unwrap();
|
||||
assert!(restored.find_pr(&id).unwrap().standard_refs.is_empty());
|
||||
}
|
||||
|
||||
#[tokio::test]
|
||||
async fn standard_and_prs_copies_both_expire() {
|
||||
let root = tempfile::tempdir().unwrap();
|
||||
let owner = Keys::generate().public_key();
|
||||
let (path, commit) = repository(root.path(), owner, "one");
|
||||
let p = Purgatory::new(root.path());
|
||||
p.set_prs_cleanup_ctx(PrsCleanupCtx {
|
||||
git_data_path: root.path().to_path_buf(),
|
||||
repo_init_locks: Default::default(),
|
||||
});
|
||||
let id = "56".repeat(32);
|
||||
let reference = format!("refs/nostr/{id}");
|
||||
git(&path, &["update-ref", &reference, &commit]);
|
||||
p.add_standard_pr_placeholder(id.clone(), commit.clone(), owner, "one".into());
|
||||
p.add_prs_pr_placeholder(id.clone(), commit.clone(), owner, "one".into());
|
||||
let prs = crate::grasp06::paths::prs_repo_path(root.path(), &owner.to_hex(), "one");
|
||||
std::fs::create_dir_all(prs.parent().unwrap()).unwrap();
|
||||
git(
|
||||
root.path(),
|
||||
&[
|
||||
"clone",
|
||||
"--bare",
|
||||
path.to_str().unwrap(),
|
||||
prs.to_str().unwrap(),
|
||||
],
|
||||
);
|
||||
git(&prs, &["update-ref", &reference, &commit]);
|
||||
expire(&p, &id);
|
||||
assert_eq!(
|
||||
p.cleanup_standard_pr_refs(&RepositoryLifecycle::in_memory(), &database())
|
||||
.await,
|
||||
0
|
||||
);
|
||||
assert!(!refs(&path).unwrap().contains_key(&reference));
|
||||
assert_eq!(p.cleanup().2, 1);
|
||||
assert!(!prs.exists());
|
||||
assert!(p.find_pr(&id).is_none());
|
||||
}
|
||||
|
||||
#[tokio::test]
|
||||
async fn standard_ref_expiry_preserves_event_accepted_after_last_checkpoint() {
|
||||
let root = tempfile::tempdir().unwrap();
|
||||
let keys = Keys::generate();
|
||||
let owner = keys.public_key();
|
||||
let (path, commit) = repository(root.path(), owner, "one");
|
||||
let event = EventBuilder::new(Kind::GitPullRequest, "")
|
||||
.tags([Tag::custom("c", [commit.clone()])])
|
||||
.finalize(&keys)
|
||||
.unwrap();
|
||||
let id = event.id.to_hex();
|
||||
let reference = format!("refs/nostr/{id}");
|
||||
git(&path, &["update-ref", &reference, &commit]);
|
||||
let p = Purgatory::new(root.path());
|
||||
p.add_standard_pr_placeholder(id.clone(), commit, owner, "one".into());
|
||||
expire(&p, &id);
|
||||
let db = database();
|
||||
db.save_event(&event).await.unwrap();
|
||||
assert_eq!(
|
||||
p.cleanup_standard_pr_refs(&RepositoryLifecycle::in_memory(), &db)
|
||||
.await,
|
||||
1
|
||||
);
|
||||
assert!(p.find_pr(&id).is_none());
|
||||
assert!(refs(&path).unwrap().contains_key(&reference));
|
||||
}
|
||||
|
||||
#[tokio::test]
|
||||
async fn standard_ref_expiry_retries_after_partial_cleanup() {
|
||||
let root = tempfile::tempdir().unwrap();
|
||||
let owner = Keys::generate().public_key();
|
||||
let p = Purgatory::new(root.path());
|
||||
let id = "78".repeat(32);
|
||||
let reference = format!("refs/nostr/{id}");
|
||||
let mut paths = Vec::new();
|
||||
for name in ["one", "two", "rejected-push"] {
|
||||
let (path, commit) = repository(root.path(), owner, name);
|
||||
if name != "rejected-push" {
|
||||
git(&path, &["update-ref", &reference, &commit]);
|
||||
}
|
||||
p.add_standard_pr_placeholder(id.clone(), commit, owner, name.into());
|
||||
paths.push(path);
|
||||
}
|
||||
expire(&p, &id);
|
||||
let lock = paths[1].join(format!("{reference}.lock"));
|
||||
std::fs::write(&lock, b"").unwrap();
|
||||
let lifecycle = RepositoryLifecycle::in_memory();
|
||||
let db = database();
|
||||
assert_eq!(p.cleanup_standard_pr_refs(&lifecycle, &db).await, 0);
|
||||
assert!(!refs(&paths[0]).unwrap().contains_key(&reference));
|
||||
assert!(refs(&paths[1]).unwrap().contains_key(&reference));
|
||||
std::fs::remove_file(lock).unwrap();
|
||||
assert_eq!(p.cleanup_standard_pr_refs(&lifecycle, &db).await, 1);
|
||||
assert!(p.find_pr(&id).is_none());
|
||||
}
|
||||
@@ -145,6 +145,10 @@ pub struct PrPurgatoryEntry {
|
||||
#[serde(default)]
|
||||
pub source: EventSource,
|
||||
|
||||
/// Exact normal-endpoint refs awaiting this event; persisted for expiry cleanup.
|
||||
#[serde(default)]
|
||||
pub standard_refs: Vec<StandardPrRef>,
|
||||
|
||||
/// If set, this placeholder was created by a push to the GRASP-06
|
||||
/// `/prs/<submitter>/<identifier>.git` endpoint (06.md line 12). When
|
||||
/// the corresponding PR event arrives, its signer MUST equal
|
||||
@@ -205,3 +209,11 @@ pub struct AnnouncementPurgatoryEntry {
|
||||
/// Whether the bare repo has been deleted (soft expiry)
|
||||
pub soft_expired: bool,
|
||||
}
|
||||
|
||||
/// A pushed normal-endpoint ref, including the value cleanup may safely delete.
|
||||
#[derive(Debug, Clone, Serialize, Deserialize, PartialEq, Eq)]
|
||||
pub struct StandardPrRef {
|
||||
pub owner: PublicKey,
|
||||
pub identifier: String,
|
||||
pub commit: String,
|
||||
}
|
||||
|
||||
+20
-9
@@ -249,6 +249,15 @@ impl RelayServer {
|
||||
.await
|
||||
.context("failed deletion lifecycle startup reconciliation")?;
|
||||
|
||||
// Recover upload expiry before cleanup, integrity, sync or HTTP can run.
|
||||
git::staging::recover_placeholders(
|
||||
&git::storage::LocalGitStorage::new(config.effective_git_data_path()),
|
||||
&purgatory,
|
||||
&relay_runtime.stores.database,
|
||||
)
|
||||
.await
|
||||
.context("failed to recover staged upload expiry")?;
|
||||
|
||||
// Make the operator identity discoverable on this relay without
|
||||
// overwriting identity events the owner already published — locally
|
||||
// or on the configured user-index relays. Runs after deletion startup
|
||||
@@ -386,13 +395,22 @@ impl RelayServer {
|
||||
"Crash-safe sync-state checkpoint task started"
|
||||
);
|
||||
|
||||
background_tasks.push(tokio::spawn(git::staging::run_worker(
|
||||
git::storage::LocalGitStorage::new(config.effective_git_data_path()),
|
||||
repo_init_locks.clone(),
|
||||
)));
|
||||
|
||||
// Spawn background cleanup task for purgatory entries (60s interval)
|
||||
let cleanup_purgatory = purgatory.clone();
|
||||
let cleanup_lifecycle = relay_runtime.lifecycle.clone();
|
||||
let cleanup_database = relay_runtime.stores.database.clone();
|
||||
background_tasks.push(tokio::spawn(async move {
|
||||
let mut interval = tokio::time::interval(Duration::from_secs(60));
|
||||
loop {
|
||||
interval.tick().await;
|
||||
let standard_removed = cleanup_purgatory.cleanup_standard_pr_refs(&cleanup_lifecycle, &cleanup_database).await;
|
||||
let (announcement_removed, state_removed, pr_removed) = cleanup_purgatory.cleanup();
|
||||
let pr_removed = pr_removed + standard_removed;
|
||||
if announcement_removed > 0 || state_removed > 0 || pr_removed > 0 {
|
||||
info!(
|
||||
"Purgatory cleanup: removed {} announcements, {} state events, {} PR events",
|
||||
@@ -575,15 +593,8 @@ impl RelayServer {
|
||||
info!("Rejected events cache saved to disk");
|
||||
}
|
||||
|
||||
// Cleanup placeholder refs on shutdown
|
||||
let placeholder_ids = self.purgatory.get_placeholder_event_ids();
|
||||
if !placeholder_ids.is_empty() {
|
||||
info!(
|
||||
"Cleaning up {} placeholder refs/nostr/ refs on shutdown",
|
||||
placeholder_ids.len()
|
||||
);
|
||||
git::cleanup_placeholder_refs(&self.git_data_path, &placeholder_ids);
|
||||
}
|
||||
// Pending refs and their staged objects must survive shutdown together
|
||||
// with the checkpoint. Expiry owns their removal after restart.
|
||||
|
||||
result
|
||||
}
|
||||
|
||||
@@ -451,6 +451,24 @@ impl TestRelay {
|
||||
.await
|
||||
}
|
||||
|
||||
/// Persistent GRASP-06 storage for crash/restart tests.
|
||||
pub async fn start_with_grasp_06_paths(
|
||||
git_data_path: PathBuf,
|
||||
relay_data_path: PathBuf,
|
||||
) -> Self {
|
||||
Self::start_internal(
|
||||
port::reserve_port(),
|
||||
RelayOptions {
|
||||
grasp06_enable: true,
|
||||
lmdb_backend: true,
|
||||
git_data_path: Some(git_data_path),
|
||||
relay_data_path: Some(relay_data_path),
|
||||
..RelayOptions::default()
|
||||
},
|
||||
)
|
||||
.await
|
||||
}
|
||||
|
||||
/// Start a relay on a port that the caller has already reserved.
|
||||
///
|
||||
/// Use this when the test needs the port number *before* the relay
|
||||
@@ -1102,6 +1120,23 @@ impl TestRelay {
|
||||
}
|
||||
|
||||
/// Stop the relay
|
||||
/// Stop with SIGTERM so the relay runs its shutdown path, then reap it.
|
||||
pub async fn stop_gracefully(mut self) {
|
||||
let status = std::process::Command::new("kill")
|
||||
.args(["-TERM", &self.process.id().to_string()])
|
||||
.status()
|
||||
.expect("send SIGTERM to relay");
|
||||
assert!(status.success(), "SIGTERM delivery failed");
|
||||
let deadline = std::time::Instant::now() + std::time::Duration::from_secs(30);
|
||||
while self.process.try_wait().expect("poll relay").is_none() {
|
||||
assert!(
|
||||
std::time::Instant::now() < deadline,
|
||||
"relay did not exit after SIGTERM"
|
||||
);
|
||||
tokio::task::yield_now().await;
|
||||
}
|
||||
}
|
||||
|
||||
pub async fn stop(mut self) {
|
||||
// kill() sends SIGKILL; reap directly instead of guessing a grace period.
|
||||
let _ = self.process.kill();
|
||||
|
||||
@@ -0,0 +1,449 @@
|
||||
//! Git-level probes for concurrent staging expiry. No relay is started here.
|
||||
//! Hooks impose scheduling boundaries without changing Git's commands or results.
|
||||
|
||||
use std::{
|
||||
fs::{self, FileTimes},
|
||||
io::{Read, Write},
|
||||
os::unix::fs::PermissionsExt,
|
||||
path::{Path, PathBuf},
|
||||
process::{Output, Stdio},
|
||||
time::{Duration, SystemTime},
|
||||
};
|
||||
use tempfile::TempDir;
|
||||
use tokio::{
|
||||
io::{AsyncReadExt, AsyncWriteExt},
|
||||
net::{TcpListener, TcpStream},
|
||||
process::{Child, Command},
|
||||
time::timeout,
|
||||
};
|
||||
|
||||
const DEADLINE: Duration = Duration::from_secs(20);
|
||||
const EXPIRED: &str = "refs/nostr/1111111111111111111111111111111111111111111111111111111111111111";
|
||||
const LIVE: &str = "refs/nostr/2222222222222222222222222222222222222222222222222222222222222222";
|
||||
|
||||
fn command(repo: &Path, args: &[&str]) -> Command {
|
||||
let mut cmd = Command::new("git");
|
||||
for (key, _) in std::env::vars_os() {
|
||||
if key.to_string_lossy().starts_with("GIT_") {
|
||||
cmd.env_remove(key);
|
||||
}
|
||||
}
|
||||
cmd.env("GIT_CONFIG_NOSYSTEM", "1")
|
||||
.env("GIT_CONFIG_GLOBAL", "/dev/null")
|
||||
.env("GIT_AUTHOR_NAME", "Test")
|
||||
.env("GIT_AUTHOR_EMAIL", "test@example.com")
|
||||
.env("GIT_COMMITTER_NAME", "Test")
|
||||
.env("GIT_COMMITTER_EMAIL", "test@example.com")
|
||||
.current_dir(repo)
|
||||
.args(args)
|
||||
.kill_on_drop(true)
|
||||
.stdin(Stdio::null())
|
||||
.stdout(Stdio::piped())
|
||||
.stderr(Stdio::piped());
|
||||
cmd
|
||||
}
|
||||
|
||||
async fn output(cmd: &mut Command) -> Output {
|
||||
timeout(DEADLINE, cmd.output()).await.unwrap().unwrap()
|
||||
}
|
||||
|
||||
async fn git(repo: &Path, args: &[&str]) -> String {
|
||||
let result = output(&mut command(repo, args)).await;
|
||||
assert!(result.status.success(), "{args:?}: {result:?}");
|
||||
String::from_utf8(result.stdout).unwrap().trim().to_owned()
|
||||
}
|
||||
|
||||
struct Fixture {
|
||||
_dir: TempDir,
|
||||
view: PathBuf,
|
||||
family: PathBuf,
|
||||
client: PathBuf,
|
||||
base: String,
|
||||
live: String,
|
||||
family_objects: String,
|
||||
}
|
||||
|
||||
impl Fixture {
|
||||
async fn new(old_objects: bool) -> Self {
|
||||
let dir = tempfile::tempdir().unwrap();
|
||||
let view = dir.path().join("view");
|
||||
let family = dir.path().join("family");
|
||||
let client = dir.path().join("client");
|
||||
for repo in [&view, &family, &client] {
|
||||
fs::create_dir(repo).unwrap();
|
||||
git(
|
||||
repo,
|
||||
&["init", "--bare", "--quiet", "--initial-branch=main"],
|
||||
)
|
||||
.await;
|
||||
git(repo, &["config", "gc.auto", "0"]).await;
|
||||
}
|
||||
let family_tree = git(&family, &["mktree"]).await;
|
||||
let family_tip = git(&family, &["commit-tree", &family_tree, "-m", "family"]).await;
|
||||
git(&family, &["update-ref", "refs/heads/main", &family_tip]).await;
|
||||
fs::write(
|
||||
view.join("objects/info/alternates"),
|
||||
format!("{}\n", family.join("objects").display()),
|
||||
)
|
||||
.unwrap();
|
||||
let tree = git(&view, &["mktree"]).await;
|
||||
let base = git(&view, &["commit-tree", &tree, "-m", "expired pending"]).await;
|
||||
let live = git(&view, &["commit-tree", &tree, "-m", "live pending"]).await;
|
||||
git(&view, &["update-ref", EXPIRED, &base]).await;
|
||||
git(&view, &["update-ref", LIVE, &live]).await;
|
||||
let family_objects = Self::objects(&family).await;
|
||||
let fixture = Self {
|
||||
_dir: dir,
|
||||
view,
|
||||
family,
|
||||
client,
|
||||
base,
|
||||
live,
|
||||
family_objects,
|
||||
};
|
||||
fixture.repack().await;
|
||||
if old_objects {
|
||||
// Model elapsed object age without sleeping. The first cruft pack
|
||||
// derives these objects' ages from their previous pack's mtime.
|
||||
let old = SystemTime::now() - Duration::from_secs(7200);
|
||||
let mut packs = 0;
|
||||
for entry in fs::read_dir(fixture.view.join("objects/pack")).unwrap() {
|
||||
let path = entry.unwrap().path();
|
||||
if path.extension().is_some_and(|ext| ext == "pack") {
|
||||
fs::File::open(path)
|
||||
.unwrap()
|
||||
.set_times(FileTimes::new().set_modified(old))
|
||||
.unwrap();
|
||||
packs += 1;
|
||||
}
|
||||
}
|
||||
assert!(packs > 0);
|
||||
}
|
||||
fixture
|
||||
}
|
||||
|
||||
async fn objects(repo: &Path) -> String {
|
||||
git(
|
||||
repo,
|
||||
&[
|
||||
"cat-file",
|
||||
"--batch-all-objects",
|
||||
"--batch-check=%(objectname)",
|
||||
],
|
||||
)
|
||||
.await
|
||||
}
|
||||
|
||||
async fn repack(&self) {
|
||||
git(
|
||||
&self.view,
|
||||
&[
|
||||
"repack",
|
||||
"-d",
|
||||
"-l",
|
||||
"--cruft",
|
||||
"--cruft-expiration=30.minutes.ago",
|
||||
],
|
||||
)
|
||||
.await;
|
||||
}
|
||||
|
||||
async fn expire(&self) {
|
||||
git(&self.view, &["update-ref", "-d", EXPIRED, &self.base]).await;
|
||||
}
|
||||
|
||||
async fn assert_live_and_family_intact(&self) {
|
||||
assert_eq!(git(&self.view, &["rev-parse", LIVE]).await, self.live);
|
||||
git(
|
||||
&self.view,
|
||||
&["rev-list", "--objects", "--missing=error", LIVE],
|
||||
)
|
||||
.await;
|
||||
assert_eq!(Self::objects(&self.family).await, self.family_objects);
|
||||
git(&self.family, &["fsck", "--full"]).await;
|
||||
}
|
||||
}
|
||||
|
||||
struct Gate {
|
||||
listener: TcpListener,
|
||||
hook: PathBuf,
|
||||
}
|
||||
|
||||
impl Gate {
|
||||
async fn new(repo: &Path, name: &str, exec_args: bool) -> Self {
|
||||
let listener = TcpListener::bind("127.0.0.1:0").await.unwrap();
|
||||
let hook = repo.join("hooks").join(name);
|
||||
fs::write(&hook, format!(
|
||||
"#!/bin/sh\n\"$GRASP_CRUFT_TEST_BINARY\" --exact git_hook_barrier --nocapture >&2 || exit 1\n{}\n",
|
||||
if exec_args { "exec \"$@\"" } else { "exit 0" },
|
||||
)).unwrap();
|
||||
fs::set_permissions(&hook, fs::Permissions::from_mode(0o755)).unwrap();
|
||||
Self { listener, hook }
|
||||
}
|
||||
|
||||
fn spawn(&self, cmd: &mut Command) -> Child {
|
||||
cmd.env("GRASP_CRUFT_TEST_BINARY", std::env::current_exe().unwrap())
|
||||
.env(
|
||||
"GRASP_CRUFT_GATE",
|
||||
self.listener.local_addr().unwrap().to_string(),
|
||||
)
|
||||
.spawn()
|
||||
.unwrap()
|
||||
}
|
||||
|
||||
async fn reached(&self) -> TcpStream {
|
||||
timeout(DEADLINE, async {
|
||||
let (mut stream, _) = self.listener.accept().await.unwrap();
|
||||
assert_eq!(stream.read_u8().await.unwrap(), b'R');
|
||||
stream
|
||||
})
|
||||
.await
|
||||
.expect("Git did not reach the hook barrier")
|
||||
}
|
||||
}
|
||||
|
||||
// The hook runs this same test binary in a child process. No Python or global
|
||||
// environment mutation is needed. Outside a hook invocation this is a no-op.
|
||||
#[test]
|
||||
fn git_hook_barrier() {
|
||||
let Ok(address) = std::env::var("GRASP_CRUFT_GATE") else {
|
||||
return;
|
||||
};
|
||||
let mut stream =
|
||||
std::net::TcpStream::connect_timeout(&address.parse().unwrap(), DEADLINE).unwrap();
|
||||
stream.set_read_timeout(Some(DEADLINE)).unwrap();
|
||||
stream.set_write_timeout(Some(DEADLINE)).unwrap();
|
||||
stream.write_all(b"R").unwrap();
|
||||
let mut release = [0];
|
||||
stream.read_exact(&mut release).unwrap();
|
||||
assert_eq!(release, [b'G']);
|
||||
}
|
||||
|
||||
async fn upload_race(expire: bool, old: bool) -> Output {
|
||||
let f = Fixture::new(old).await;
|
||||
let gate = Gate::new(&f.view, "pack-gate", true).await;
|
||||
let upload = format!(
|
||||
"git -c uploadpack.packObjectsHook={} upload-pack",
|
||||
gate.hook.display()
|
||||
);
|
||||
let mut child = gate.spawn(&mut command(
|
||||
&f.client,
|
||||
&[
|
||||
"-c",
|
||||
"protocol.version=0",
|
||||
"fetch",
|
||||
"--no-tags",
|
||||
&format!("--upload-pack={upload}"),
|
||||
f.view.to_str().unwrap(),
|
||||
EXPIRED,
|
||||
],
|
||||
));
|
||||
let mut release = gate.reached().await;
|
||||
assert!(child.try_wait().unwrap().is_none());
|
||||
if expire {
|
||||
f.expire().await;
|
||||
}
|
||||
f.repack().await;
|
||||
f.assert_live_and_family_intact().await;
|
||||
release.write_all(b"G").await.unwrap();
|
||||
let result = timeout(DEADLINE, child.wait_with_output())
|
||||
.await
|
||||
.unwrap()
|
||||
.unwrap();
|
||||
if result.status.success() {
|
||||
assert_eq!(git(&f.client, &["rev-parse", "FETCH_HEAD"]).await, f.base);
|
||||
git(&f.client, &["fsck", "--full"]).await;
|
||||
}
|
||||
// A second fetch demonstrates that failure is confined to the expired ref.
|
||||
git(&f.client, &["fetch", f.view.to_str().unwrap(), LIVE]).await;
|
||||
git(&f.client, &["fsck", "--full"]).await;
|
||||
result
|
||||
}
|
||||
|
||||
#[tokio::test]
|
||||
async fn retained_ref_survives_repack_during_upload() {
|
||||
let result = upload_race(false, true).await;
|
||||
assert!(result.status.success(), "{result:?}");
|
||||
}
|
||||
|
||||
#[tokio::test]
|
||||
async fn recent_objects_survive_repack_after_ref_deletion() {
|
||||
let result = upload_race(true, false).await;
|
||||
assert!(result.status.success(), "{result:?}");
|
||||
}
|
||||
|
||||
#[tokio::test]
|
||||
async fn expired_ref_upload_may_fail_without_harming_live_ref() {
|
||||
let result = upload_race(true, true).await;
|
||||
let stderr = String::from_utf8_lossy(&result.stderr);
|
||||
assert_eq!(result.status.code(), Some(128), "{result:?}");
|
||||
assert!(stderr.contains("bad object"), "{stderr}");
|
||||
assert!(stderr.contains("bad pack header"), "{stderr}");
|
||||
}
|
||||
|
||||
#[tokio::test]
|
||||
// This characterizes a failure of revised gate property 2, not an acceptable
|
||||
// server outcome. Keep it explicit until a revised design prevents this race.
|
||||
async fn concurrent_repack_can_leave_a_successful_push_with_missing_parent() {
|
||||
const NEW: &str = "refs/nostr/3333333333333333333333333333333333333333333333333333333333333333";
|
||||
let f = Fixture::new(true).await;
|
||||
git(
|
||||
&f.client,
|
||||
&[
|
||||
"fetch",
|
||||
f.view.to_str().unwrap(),
|
||||
&format!("{EXPIRED}:refs/heads/base"),
|
||||
],
|
||||
)
|
||||
.await;
|
||||
let advertisement = git(&f.view, &["receive-pack", "--advertise-refs", "."]).await;
|
||||
assert!(advertisement.contains(&format!("{} {EXPIRED}", f.base)));
|
||||
let tree = git(&f.client, &["rev-parse", "refs/heads/base^{tree}"]).await;
|
||||
let new = git(
|
||||
&f.client,
|
||||
&["commit-tree", &tree, "-p", &f.base, "-m", "new pending"],
|
||||
)
|
||||
.await;
|
||||
|
||||
git(
|
||||
&f.client,
|
||||
&["rev-list", "--objects", "--missing=error", &new],
|
||||
)
|
||||
.await;
|
||||
|
||||
// Build the pack a client may send using that advertisement: only the new
|
||||
// commit, omitting its previously advertised parent. No malformed objects.
|
||||
let mut pack = command(&f.client, &["pack-objects", "--stdout", "--revs"])
|
||||
.stdin(Stdio::piped())
|
||||
.spawn()
|
||||
.unwrap();
|
||||
pack.stdin
|
||||
.take()
|
||||
.unwrap()
|
||||
.write_all(format!("{new}\n^{}\n", f.base).as_bytes())
|
||||
.await
|
||||
.unwrap();
|
||||
let pack = timeout(DEADLINE, pack.wait_with_output())
|
||||
.await
|
||||
.unwrap()
|
||||
.unwrap();
|
||||
assert!(pack.status.success(), "{pack:?}");
|
||||
let update = format!("{} {new} {NEW}\0report-status\n", "0".repeat(40));
|
||||
let mut request = format!("{:04x}{update}0000", update.len() + 4).into_bytes();
|
||||
request.extend(pack.stdout);
|
||||
|
||||
// Expiry precedes repack, rather than racing its initial ref snapshot.
|
||||
// The old pack models an idle view eligible for compaction. Any quiet wait
|
||||
// between deletion and repack leaves the following interleaving unchanged:
|
||||
// the new receive-pack begins only AFTER repack has selected its survivors.
|
||||
f.expire().await;
|
||||
let repack_gate = Gate::new(&f.view, "repack-gate", false).await;
|
||||
let wrappers = f._dir.path().join("git-wrappers");
|
||||
fs::create_dir(&wrappers).unwrap();
|
||||
let wrapper = wrappers.join("git");
|
||||
fs::write(
|
||||
&wrapper,
|
||||
r#"#!/bin/sh
|
||||
case " $* " in
|
||||
*" --cruft "*)
|
||||
"$GRASP_CRUFT_REAL_GIT" "$@" || exit $?
|
||||
exec "$GRASP_CRUFT_REPACK_GATE"
|
||||
;;
|
||||
*) exec "$GRASP_CRUFT_REAL_GIT" "$@" ;;
|
||||
esac
|
||||
"#,
|
||||
)
|
||||
.unwrap();
|
||||
fs::set_permissions(&wrapper, fs::Permissions::from_mode(0o755)).unwrap();
|
||||
let real_git = std::env::split_paths(&std::env::var_os("PATH").unwrap())
|
||||
.map(|dir| dir.join("git"))
|
||||
.find(|path| path.is_file())
|
||||
.unwrap();
|
||||
let mut repack = repack_gate.spawn(
|
||||
command(
|
||||
&f.view,
|
||||
&[
|
||||
"repack",
|
||||
"-d",
|
||||
"-l",
|
||||
"--cruft",
|
||||
"--cruft-expiration=30.minutes.ago",
|
||||
],
|
||||
)
|
||||
.env("GIT_EXEC_PATH", &wrappers)
|
||||
.env("GRASP_CRUFT_REAL_GIT", real_git)
|
||||
.env("GRASP_CRUFT_REPACK_GATE", &repack_gate.hook),
|
||||
);
|
||||
let mut finish_repack = repack_gate.reached().await;
|
||||
assert!(repack.try_wait().unwrap().is_none());
|
||||
// Git has finished preparing its cruft pack, but not deleted the old pack.
|
||||
git(&f.view, &["cat-file", "-e", &f.base]).await;
|
||||
|
||||
let receive_gate = Gate::new(&f.view, "pre-receive", false).await;
|
||||
let mut receive = receive_gate.spawn(
|
||||
command(
|
||||
&f.view,
|
||||
&[
|
||||
"-c",
|
||||
&format!("core.hooksPath={}", f.view.join("hooks").display()),
|
||||
"receive-pack",
|
||||
"--stateless-rpc",
|
||||
".",
|
||||
],
|
||||
)
|
||||
.stdin(Stdio::piped()),
|
||||
);
|
||||
receive
|
||||
.stdin
|
||||
.take()
|
||||
.unwrap()
|
||||
.write_all(&request)
|
||||
.await
|
||||
.unwrap();
|
||||
let mut finish_receive = receive_gate.reached().await;
|
||||
assert!(receive.try_wait().unwrap().is_none());
|
||||
// receive-pack has now checked connectivity against the old parent.
|
||||
finish_repack.write_all(b"G").await.unwrap();
|
||||
let repack = timeout(DEADLINE, repack.wait_with_output())
|
||||
.await
|
||||
.unwrap()
|
||||
.unwrap();
|
||||
assert!(repack.status.success(), "{repack:?}");
|
||||
f.assert_live_and_family_intact().await;
|
||||
let parent = output(&mut command(&f.view, &["cat-file", "-e", &f.base])).await;
|
||||
assert!(!parent.status.success());
|
||||
finish_receive.write_all(b"G").await.unwrap();
|
||||
let result = timeout(DEADLINE, receive.wait_with_output())
|
||||
.await
|
||||
.unwrap()
|
||||
.unwrap();
|
||||
let installed = output(&mut command(&f.view, &["rev-parse", "--verify", NEW])).await;
|
||||
let closure = output(&mut command(
|
||||
&f.view,
|
||||
&["rev-list", "--objects", "--missing=error", NEW],
|
||||
))
|
||||
.await;
|
||||
eprintln!(
|
||||
"receive status: {}\nreport: {}\ninstalled ref: {}\nclosure status: {}\nclosure stderr: {}",
|
||||
result.status,
|
||||
String::from_utf8_lossy(&result.stdout),
|
||||
String::from_utf8_lossy(&installed.stdout),
|
||||
closure.status,
|
||||
String::from_utf8_lossy(&closure.stderr)
|
||||
);
|
||||
assert!(result.status.success(), "{result:?}");
|
||||
let report = String::from_utf8_lossy(&result.stdout);
|
||||
assert!(report.contains("unpack ok"), "{report}");
|
||||
assert!(report.contains(&format!("ok {NEW}")), "{report}");
|
||||
assert!(installed.status.success(), "{installed:?}");
|
||||
assert_eq!(String::from_utf8_lossy(&installed.stdout).trim(), new);
|
||||
assert!(
|
||||
!closure.status.success(),
|
||||
"the counterexample no longer reproduces"
|
||||
);
|
||||
assert!(
|
||||
String::from_utf8_lossy(&closure.stderr).contains(&f.base),
|
||||
"{closure:?}"
|
||||
);
|
||||
f.assert_live_and_family_intact().await;
|
||||
}
|
||||
@@ -279,6 +279,9 @@ async fn mixed_deletion_push(atomic: bool, reject_branch: bool, include_deletion
|
||||
let bad = oid.clone();
|
||||
if reject_branch {
|
||||
use std::os::unix::fs::PermissionsExt;
|
||||
// The server inherits ambient Git config, including CI's hostile
|
||||
// hooksPath. Explicitly select this fixture's intentional rejection hook.
|
||||
git(&repo, &["config", "--local", "core.hooksPath", "hooks"]);
|
||||
let hook = repo.join("hooks/update");
|
||||
std::fs::write(&hook, b"#!/bin/sh\n[ \"$1\" != refs/heads/unavailable ]\n").unwrap();
|
||||
std::fs::set_permissions(hook, std::fs::Permissions::from_mode(0o755)).unwrap();
|
||||
|
||||
@@ -369,3 +369,54 @@ isolated_test_with_grasp_06!(
|
||||
test_commit_mismatch_deletes_ref_and_blocks_promotion_with_grasp_06,
|
||||
PushValidationTests::test_commit_mismatch_deletes_ref_and_blocks_promotion
|
||||
);
|
||||
|
||||
/// A crash can lose the purgatory checkpoint after a `/prs/` push installed
|
||||
/// its ref. The arriving PR event must still be matched to that push, using
|
||||
/// only its signed, service-local clone URL and the exact event-id ref.
|
||||
#[tokio::test]
|
||||
async fn pr_event_after_crash_recovers_prs_placeholder_scope() {
|
||||
use nostr_sdk::prelude::*;
|
||||
let persistent = tempfile::tempdir().unwrap();
|
||||
let relay = TestRelay::start_with_grasp_06_paths(
|
||||
persistent.path().join("git"),
|
||||
persistent.path().join("relay"),
|
||||
)
|
||||
.await;
|
||||
let client = AuditClient::new(relay.url(), AuditConfig::isolated())
|
||||
.await
|
||||
.unwrap();
|
||||
let keys = client.keys().clone();
|
||||
let npub = keys.public_key().to_bech32().unwrap();
|
||||
let identifier = "crash-recovered-pr";
|
||||
let local = tempfile::tempdir().unwrap();
|
||||
let commit =
|
||||
common::create_test_repo_with_commit(local.path(), common::CommitVariant::PrTest).unwrap();
|
||||
let clone_url = format!("http://{}/prs/{npub}/{identifier}.git", relay.domain());
|
||||
let event = common::create_pr_event_with_clone(
|
||||
&keys,
|
||||
&format!("30617:{}:{identifier}", keys.public_key().to_hex()),
|
||||
&commit,
|
||||
"Crash-recovered PR",
|
||||
&[&clone_url],
|
||||
)
|
||||
.unwrap();
|
||||
common::push_ref_to_relay(
|
||||
local.path(),
|
||||
&relay.domain(),
|
||||
&format!("prs/{npub}"),
|
||||
identifier,
|
||||
&commit,
|
||||
&format!("refs/nostr/{}", event.id),
|
||||
)
|
||||
.unwrap();
|
||||
// Killing the relay loses any placeholder not yet checkpointed.
|
||||
let relay = relay.restart().await;
|
||||
let client = AuditClient::new(relay.url(), AuditConfig::isolated())
|
||||
.await
|
||||
.unwrap();
|
||||
client.send_event(event.clone()).await.unwrap();
|
||||
common::wait_for_event_served(relay.url(), &event.id, std::time::Duration::from_secs(10))
|
||||
.await
|
||||
.unwrap();
|
||||
relay.stop().await;
|
||||
}
|
||||
|
||||
@@ -0,0 +1,433 @@
|
||||
//! Unsigned uploads are staged in their view and earn family storage only
|
||||
//! when a signed event names them.
|
||||
//!
|
||||
//! ```bash
|
||||
//! cargo test --test pending_upload_staging
|
||||
//! ```
|
||||
|
||||
mod common;
|
||||
|
||||
use std::path::{Path, PathBuf};
|
||||
use std::time::Duration;
|
||||
|
||||
use common::{CommitVariant, TestRelay};
|
||||
use grasp_audit::{AuditClient, AuditConfig};
|
||||
use ngit_grasp::git::staging;
|
||||
use nostr_sdk::prelude::*;
|
||||
|
||||
const DEADLINE: Duration = Duration::from_secs(20);
|
||||
|
||||
fn git(repo: &Path, args: &[&str]) -> std::process::Output {
|
||||
grasp_audit::git_command()
|
||||
.current_dir(repo)
|
||||
.args(args)
|
||||
.output()
|
||||
.expect("spawn git")
|
||||
}
|
||||
|
||||
/// Every object in the family, which has no alternates of its own.
|
||||
fn family_objects(family: &Path) -> String {
|
||||
let output = git(
|
||||
family,
|
||||
&[
|
||||
"cat-file",
|
||||
"--batch-all-objects",
|
||||
"--batch-check=%(objectname)",
|
||||
],
|
||||
);
|
||||
assert!(output.status.success());
|
||||
String::from_utf8(output.stdout).unwrap()
|
||||
}
|
||||
|
||||
/// Object files in the view's own object directory, which is its staging.
|
||||
fn staged_files(view: &Path) -> Vec<PathBuf> {
|
||||
let mut files = Vec::new();
|
||||
for entry in std::fs::read_dir(view.join("objects")).unwrap() {
|
||||
let entry = entry.unwrap();
|
||||
if entry.file_name() == "info" || !entry.file_type().unwrap().is_dir() {
|
||||
continue;
|
||||
}
|
||||
for file in std::fs::read_dir(entry.path()).unwrap() {
|
||||
files.push(file.unwrap().path());
|
||||
}
|
||||
}
|
||||
files
|
||||
}
|
||||
|
||||
/// Whether `repo` holds `tip` and its whole history without any other store.
|
||||
fn holds_history(repo: &Path, tip: &str) -> bool {
|
||||
git(repo, &["rev-list", "--objects", "--missing=error", tip])
|
||||
.status
|
||||
.success()
|
||||
}
|
||||
|
||||
struct Served {
|
||||
_persistent: tempfile::TempDir,
|
||||
relay: TestRelay,
|
||||
client: AuditClient,
|
||||
announcement: Event,
|
||||
state: Event,
|
||||
identifier: String,
|
||||
npub: String,
|
||||
view: PathBuf,
|
||||
family: PathBuf,
|
||||
}
|
||||
|
||||
impl Served {
|
||||
/// A relay serving one repository whose maintainer push has completed.
|
||||
async fn start(name: &str) -> Self {
|
||||
let persistent = tempfile::tempdir().unwrap();
|
||||
let git_data = persistent.path().join("git");
|
||||
let relay = TestRelay::start_with_existing_lmdb_paths(
|
||||
git_data.clone(),
|
||||
persistent.path().join("relay"),
|
||||
None,
|
||||
false,
|
||||
)
|
||||
.await;
|
||||
let client = AuditClient::new(relay.url(), AuditConfig::isolated())
|
||||
.await
|
||||
.unwrap();
|
||||
let (announcement, identifier, state) =
|
||||
common::publish_served_audit_repo_with_state(&client, name).await;
|
||||
let npub = client.public_key().to_bech32().unwrap();
|
||||
let view = git_data.join(&npub).join(format!("{identifier}.git"));
|
||||
let family = git_data
|
||||
.join(".grasp/families/sha1")
|
||||
.join(format!("{identifier}.git"));
|
||||
Self {
|
||||
_persistent: persistent,
|
||||
relay,
|
||||
client,
|
||||
announcement,
|
||||
state,
|
||||
identifier,
|
||||
npub,
|
||||
view,
|
||||
family,
|
||||
}
|
||||
}
|
||||
|
||||
fn pr_event(&self, tip: &str, title: &str) -> Event {
|
||||
common::create_pr_event(
|
||||
self.client.keys(),
|
||||
&common::announcement_coordinate(&self.announcement, &self.identifier),
|
||||
tip,
|
||||
title,
|
||||
)
|
||||
.unwrap()
|
||||
}
|
||||
|
||||
fn push(&self, local: &Path, tip: &str, event: &Event) {
|
||||
common::push_ref_to_relay(
|
||||
local,
|
||||
&self.relay.domain(),
|
||||
&self.npub,
|
||||
&self.identifier,
|
||||
tip,
|
||||
&format!("refs/nostr/{}", event.id),
|
||||
)
|
||||
.unwrap();
|
||||
}
|
||||
|
||||
async fn accept(&self, event: &Event) {
|
||||
let client = AuditClient::new(self.relay.url(), AuditConfig::isolated())
|
||||
.await
|
||||
.unwrap();
|
||||
client.send_event(event.clone()).await.unwrap();
|
||||
common::wait_for_event_served(self.relay.url(), &event.id, DEADLINE)
|
||||
.await
|
||||
.unwrap();
|
||||
}
|
||||
|
||||
async fn wait_until_unstaged(&self) {
|
||||
common::wait_for("staging to be reclaimed", DEADLINE, || async {
|
||||
!staging::is_staged(&self.view) && staged_files(&self.view).is_empty()
|
||||
})
|
||||
.await;
|
||||
}
|
||||
}
|
||||
|
||||
#[tokio::test]
|
||||
async fn signed_push_is_received_into_the_family_without_staging() {
|
||||
let served = Served::start("staging-signed-push").await;
|
||||
|
||||
assert!(!staging::is_staged(&served.view));
|
||||
assert_eq!(staged_files(&served.view), Vec::<PathBuf>::new());
|
||||
let main = ngit_grasp::git::get_ref_commit(&served.view, "refs/heads/main").unwrap();
|
||||
assert!(holds_history(&served.family, &main));
|
||||
|
||||
served.relay.stop().await;
|
||||
}
|
||||
|
||||
#[tokio::test]
|
||||
async fn unsigned_upload_is_staged_across_a_crash_and_promoted_on_acceptance() {
|
||||
let mut served = Served::start("staging-unsigned-upload").await;
|
||||
let family_before = family_objects(&served.family);
|
||||
let local = tempfile::tempdir().unwrap();
|
||||
let tip = common::create_test_repo_with_commit(local.path(), CommitVariant::PrTest).unwrap();
|
||||
let event = served.pr_event(&tip, "staged until signed");
|
||||
|
||||
served.push(local.path(), &tip, &event);
|
||||
|
||||
assert!(staging::is_staged(&served.view));
|
||||
assert!(ngit_grasp::git::oid_exists(&served.view, &tip));
|
||||
assert_eq!(
|
||||
family_objects(&served.family),
|
||||
family_before,
|
||||
"an unsigned upload reached the family"
|
||||
);
|
||||
|
||||
// Killing the relay loses any purgatory placeholder not yet checkpointed;
|
||||
// the staged ref and objects must not depend on it.
|
||||
served.relay = served.relay.restart().await;
|
||||
assert!(staging::is_staged(&served.view));
|
||||
assert!(holds_history(&served.view, &tip));
|
||||
assert_eq!(family_objects(&served.family), family_before);
|
||||
|
||||
served.accept(&event).await;
|
||||
|
||||
assert!(
|
||||
holds_history(&served.family, &tip),
|
||||
"an accepted PR must be complete in the family alone"
|
||||
);
|
||||
served.wait_until_unstaged().await;
|
||||
assert!(holds_history(&served.view, &tip));
|
||||
|
||||
served.relay.stop().await;
|
||||
}
|
||||
|
||||
#[tokio::test]
|
||||
async fn signed_push_onto_staged_history_is_complete_in_the_family() {
|
||||
let served = Served::start("staging-signed-onto-staged").await;
|
||||
let local = tempfile::tempdir().unwrap();
|
||||
let pending_tip =
|
||||
common::create_test_repo_with_commit(local.path(), CommitVariant::PrTest).unwrap();
|
||||
let pending = served.pr_event(&pending_tip, "never signed");
|
||||
served.push(local.path(), &pending_tip, &pending);
|
||||
assert!(staging::is_staged(&served.view));
|
||||
|
||||
// The view advertises the pending tip, so this push omits its objects.
|
||||
let signed_tip = common::add_commit_to_repo(local.path(), CommitVariant::SecondCommit).unwrap();
|
||||
let signed = served.pr_event(&signed_tip, "signed before its push");
|
||||
let client = AuditClient::new(served.relay.url(), AuditConfig::isolated())
|
||||
.await
|
||||
.unwrap();
|
||||
client
|
||||
.send_event_and_note_purgatory(signed.clone())
|
||||
.await
|
||||
.unwrap();
|
||||
served.push(local.path(), &signed_tip, &signed);
|
||||
|
||||
common::wait_for_event_served(served.relay.url(), &signed.id, DEADLINE)
|
||||
.await
|
||||
.unwrap();
|
||||
assert!(
|
||||
holds_history(&served.family, &signed_tip),
|
||||
"signed history depends on objects that only staging holds"
|
||||
);
|
||||
// The pending ref is untouched. Its history is now an ancestor of signed
|
||||
// history, so the family holds it and nothing is left to stage.
|
||||
assert!(holds_history(&served.view, &pending_tip));
|
||||
served.wait_until_unstaged().await;
|
||||
|
||||
served.relay.stop().await;
|
||||
}
|
||||
|
||||
#[tokio::test]
|
||||
async fn maintainer_push_into_a_staged_view_reaches_the_family() {
|
||||
let served = Served::start("staging-maintainer-push").await;
|
||||
let local = tempfile::tempdir().unwrap();
|
||||
let pending_tip =
|
||||
common::create_test_repo_with_commit(local.path(), CommitVariant::PrTest).unwrap();
|
||||
let pending = served.pr_event(&pending_tip, "never signed");
|
||||
served.push(local.path(), &pending_tip, &pending);
|
||||
assert!(staging::is_staged(&served.view));
|
||||
|
||||
let clone =
|
||||
grasp_audit::clone_repo(&served.relay.domain(), &served.npub, &served.identifier).unwrap();
|
||||
std::fs::write(clone.join("next.txt"), "next state").unwrap();
|
||||
assert!(git(&clone, &["add", "."]).status.success());
|
||||
assert!(git(&clone, &["commit", "-m", "next state"])
|
||||
.status
|
||||
.success());
|
||||
let next = String::from_utf8(git(&clone, &["rev-parse", "HEAD"]).stdout)
|
||||
.unwrap()
|
||||
.trim()
|
||||
.to_owned();
|
||||
let state = common::event_ordering::event_after(
|
||||
common::create_state_event(
|
||||
served.client.keys(),
|
||||
&served.identifier,
|
||||
&[("main", &next)],
|
||||
&[],
|
||||
&[],
|
||||
&[],
|
||||
)
|
||||
.unwrap(),
|
||||
served.client.keys(),
|
||||
served.state.created_at,
|
||||
);
|
||||
served
|
||||
.client
|
||||
.send_event_and_note_purgatory(state.clone())
|
||||
.await
|
||||
.unwrap();
|
||||
assert!(grasp_audit::try_push(&clone).unwrap());
|
||||
let _ = std::fs::remove_dir_all(&clone);
|
||||
|
||||
common::wait_for_event_served(served.relay.url(), &state.id, DEADLINE)
|
||||
.await
|
||||
.unwrap();
|
||||
// Rollback after a State deletion depends on this history, whatever
|
||||
// becomes of the view's refs.
|
||||
assert!(holds_history(&served.family, &next));
|
||||
assert!(!staging::owes_history(&served.view));
|
||||
|
||||
served.relay.stop().await;
|
||||
}
|
||||
|
||||
async fn state_adopts_unsigned_upload(state_first: bool) {
|
||||
let served = Served::start("state-adopts-unsigned").await;
|
||||
let original = ngit_grasp::git::get_ref_commit(&served.view, "refs/heads/main").unwrap();
|
||||
let local = tempfile::tempdir().unwrap();
|
||||
let tip = common::create_test_repo_with_commit(local.path(), CommitVariant::PrTest).unwrap();
|
||||
let pending = served.pr_event(&tip, "never published");
|
||||
let state = common::event_ordering::event_after(
|
||||
common::create_state_event(
|
||||
served.client.keys(),
|
||||
&served.identifier,
|
||||
&[("main", &tip)],
|
||||
&[("staged-tag", &tip)],
|
||||
&[],
|
||||
&[],
|
||||
)
|
||||
.unwrap(),
|
||||
served.client.keys(),
|
||||
served.state.created_at,
|
||||
);
|
||||
if state_first {
|
||||
served
|
||||
.client
|
||||
.send_event_and_note_purgatory(state.clone())
|
||||
.await
|
||||
.unwrap();
|
||||
}
|
||||
served.push(local.path(), &tip, &pending);
|
||||
if !state_first {
|
||||
served.accept(&state).await;
|
||||
}
|
||||
common::wait_for_event_served(served.relay.url(), &state.id, DEADLINE)
|
||||
.await
|
||||
.unwrap();
|
||||
assert_eq!(
|
||||
ngit_grasp::git::get_ref_commit(&served.view, "refs/heads/main"),
|
||||
Some(tip.clone())
|
||||
);
|
||||
assert!(
|
||||
holds_history(&served.family, &tip),
|
||||
"accepted State history must leave staging"
|
||||
);
|
||||
|
||||
let next = common::event_ordering::event_after(
|
||||
common::create_state_event(
|
||||
served.client.keys(),
|
||||
&served.identifier,
|
||||
&[("main", &original)],
|
||||
&[],
|
||||
&[],
|
||||
&[],
|
||||
)
|
||||
.unwrap(),
|
||||
served.client.keys(),
|
||||
state.created_at,
|
||||
);
|
||||
served.accept(&next).await;
|
||||
// Stop the writer before simulating expiry and compaction directly.
|
||||
served.relay.stop().await;
|
||||
ngit_grasp::git::delete_ref(&served.view, &format!("refs/nostr/{}", pending.id)).unwrap();
|
||||
staging::compact(&served.view).unwrap();
|
||||
assert!(
|
||||
holds_history(&served.family, &tip),
|
||||
"rollback history survives ref removal"
|
||||
);
|
||||
}
|
||||
|
||||
#[tokio::test]
|
||||
async fn arriving_state_promotes_an_existing_unsigned_upload() {
|
||||
state_adopts_unsigned_upload(false).await;
|
||||
}
|
||||
|
||||
#[tokio::test]
|
||||
async fn waiting_state_promotes_an_unsigned_upload_before_release() {
|
||||
state_adopts_unsigned_upload(true).await;
|
||||
}
|
||||
|
||||
#[tokio::test]
|
||||
async fn crash_without_purgatory_checkpoint_still_expires_unsigned_upload() {
|
||||
let served = Served::start("crash-expiry").await;
|
||||
let local = tempfile::tempdir().unwrap();
|
||||
let tip = common::create_test_repo_with_commit(local.path(), CommitVariant::PrTest).unwrap();
|
||||
let pending = served.pr_event(&tip, "never published");
|
||||
served.push(local.path(), &tip, &pending);
|
||||
let family_before = family_objects(&served.family);
|
||||
let git_data = served._persistent.path().join("git");
|
||||
let relay_data = served._persistent.path().join("relay");
|
||||
served.relay.stop().await;
|
||||
// Reproduce the crash window deterministically: no checkpoint knows this
|
||||
// upload, and its durable deadline has elapsed while the relay was down.
|
||||
let checkpoint = git_data.join("purgatory-state.json");
|
||||
if checkpoint.exists() {
|
||||
std::fs::remove_file(checkpoint).unwrap();
|
||||
}
|
||||
for entry in std::fs::read_dir(git_data.join(".grasp/staging")).unwrap() {
|
||||
let path = entry.unwrap().path();
|
||||
let mut record: serde_json::Value =
|
||||
serde_json::from_slice(&std::fs::read(&path).unwrap()).unwrap();
|
||||
for upload in record["pending"].as_array_mut().unwrap() {
|
||||
upload["expires_at"] = serde_json::to_value(std::time::SystemTime::UNIX_EPOCH).unwrap();
|
||||
}
|
||||
std::fs::write(path, serde_json::to_vec(&record).unwrap()).unwrap();
|
||||
}
|
||||
let relay = TestRelay::start_with_existing_lmdb_paths(git_data, relay_data, None, false).await;
|
||||
common::wait_for(
|
||||
"recovered expired upload to be reclaimed",
|
||||
DEADLINE,
|
||||
|| async { !ngit_grasp::git::oid_exists(&served.view, &tip) },
|
||||
)
|
||||
.await;
|
||||
assert!(
|
||||
ngit_grasp::git::get_ref_commit(&served.view, &format!("refs/nostr/{}", pending.id))
|
||||
.is_none()
|
||||
);
|
||||
assert_eq!(family_objects(&served.family), family_before);
|
||||
relay.stop().await;
|
||||
}
|
||||
|
||||
#[tokio::test]
|
||||
async fn corrupt_staging_record_does_not_prevent_relay_restart() {
|
||||
let mut served = Served::start("corrupt-staging-record").await;
|
||||
let local = tempfile::tempdir().unwrap();
|
||||
let tip = common::create_test_repo_with_commit(local.path(), CommitVariant::PrTest).unwrap();
|
||||
let pending = served.pr_event(&tip, "pending with damaged metadata");
|
||||
served.push(local.path(), &tip, &pending);
|
||||
let registry = served._persistent.path().join("git/.grasp/staging");
|
||||
let records: Vec<_> = std::fs::read_dir(®istry)
|
||||
.unwrap()
|
||||
.map(|entry| entry.unwrap().path())
|
||||
.collect();
|
||||
assert_eq!(records.len(), 1);
|
||||
let damaged = b"{broken json";
|
||||
std::fs::write(&records[0], damaged).unwrap();
|
||||
|
||||
// restart() kills the process and waits for the new relay to be ready.
|
||||
served.relay = served.relay.restart().await;
|
||||
assert_eq!(std::fs::read(&records[0]).unwrap(), damaged);
|
||||
assert_eq!(
|
||||
ngit_grasp::git::get_ref_commit(&served.view, &format!("refs/nostr/{}", pending.id)),
|
||||
Some(tip.clone())
|
||||
);
|
||||
assert!(holds_history(&served.view, &tip));
|
||||
assert!(!holds_history(&served.family, &tip));
|
||||
served.relay.stop().await;
|
||||
}
|
||||
@@ -87,3 +87,56 @@ isolated_purgatory_test!(test_state_event_served_after_git_push);
|
||||
isolated_purgatory_test!(test_pr_event_accepted_into_purgatory_and_isnt_served);
|
||||
isolated_purgatory_test!(test_pr_event_in_purgatory_git_push_accepted);
|
||||
isolated_purgatory_test!(test_pr_event_served_after_git_push);
|
||||
|
||||
#[tokio::test]
|
||||
async fn graceful_shutdown_keeps_pending_pr_refs() {
|
||||
use nostr_sdk::prelude::*;
|
||||
let persistent = tempfile::tempdir().unwrap();
|
||||
let git_data = persistent.path().join("git");
|
||||
let relay = TestRelay::start_with_existing_lmdb_paths(
|
||||
git_data.clone(),
|
||||
persistent.path().join("relay"),
|
||||
None,
|
||||
false,
|
||||
)
|
||||
.await;
|
||||
let client = AuditClient::new(relay.url(), AuditConfig::isolated())
|
||||
.await
|
||||
.unwrap();
|
||||
let (announcement, identifier) =
|
||||
common::publish_served_repo(&client, "shutdown-pending-pr").await;
|
||||
let local = tempfile::tempdir().unwrap();
|
||||
let tip =
|
||||
common::create_test_repo_with_commit(local.path(), common::CommitVariant::PrTest).unwrap();
|
||||
// The event is never sent, so the pushed ref stays a pending placeholder.
|
||||
let event = common::create_pr_event(
|
||||
client.keys(),
|
||||
&common::announcement_coordinate(&announcement, &identifier),
|
||||
&tip,
|
||||
"pending across shutdown",
|
||||
)
|
||||
.unwrap();
|
||||
let npub = client.public_key().to_bech32().unwrap();
|
||||
let reference = format!("refs/nostr/{}", event.id);
|
||||
common::push_ref_to_relay(
|
||||
local.path(),
|
||||
&relay.domain(),
|
||||
&npub,
|
||||
&identifier,
|
||||
&tip,
|
||||
&reference,
|
||||
)
|
||||
.unwrap();
|
||||
let view = git_data.join(&npub).join(format!("{identifier}.git"));
|
||||
assert_eq!(
|
||||
ngit_grasp::git::get_ref_commit(&view, &reference).as_deref(),
|
||||
Some(tip.as_str())
|
||||
);
|
||||
relay.stop_gracefully().await;
|
||||
assert_eq!(
|
||||
ngit_grasp::git::get_ref_commit(&view, &reference).as_deref(),
|
||||
Some(tip.as_str()),
|
||||
"shutdown must keep the pending ref for expiry after restart"
|
||||
);
|
||||
assert!(ngit_grasp::git::oid_exists(&view, &tip));
|
||||
}
|
||||
|
||||
Reference in New Issue
Block a user