mirror of
https://relay.ngit.dev/npub15qydau2hjma6ngxkl2cyar74wzyjshvl65za5k5rl69264ar2exs5cyejr/ngit-grasp.git
synced 2026-10-05 15:08:24 +00:00
DanConwayDev
3200e7b31a
Purgatory git sync listed every needed commit id as an explicit want in
a single `git fetch <url> <oid1> <oid2> ...` and, on each upload-pack
"not our ref" rejection, dropped that one oid and re-sent the entire
remaining batch. Against a state event declaring tips that exist on no
reachable server (production evidence: the `market` repository, whose
declared-but-missing ref tips were crawled against gitnostr.com at
150-270 requests/min, 4,521 "not our ref" upload-pack errors in the
45-minute window on 2026-08-05) this degenerated into one failed
upload-pack round trip per missing tip with O(N^2) want retransmission,
pack data streamed and aborted on every failure, and nothing fetched
until the crawl finished. A single missing object failed every batch
that contained it.
fetch_oids now runs three phases through the same hardened/pinned
subprocess machinery (outbound policy authorization and DNS pinning are
unchanged and now also cover ls-remote):
1. `git ls-remote` compares the remote's advertised refs against the
needed oids; state-event ref tips and `refs/nostr/<event-id>` PR
tips surface in the advertisement, so matching oid sets suffices.
2. One batch `git fetch` of the needed oids the remote advertises.
Advertised oids are always valid wants, so "not our ref" cannot fail
this batch; if the advertisement races a ref update, the pass
degrades to per-oid fetches instead of failing.
3. Residual oids are requested one at a time, so a missing object costs
exactly one small round trip and never aborts the rest. Genuine
remote failures still feed the naughty list and stop the pass while
keeping what the batch already fetched.
Client-side observability the old debug-only retry loop lacked: one
INFO summary line per pass (needed / advertised tips / residual
attempted / residual missing / fetched) plus new counters
ngit_purgatory_git_fetch_passes_total and
ngit_purgatory_git_fetch_oids_total{kind}, so production verification
can observe gitnostr.com's own outbound behaviour rather than only the
serving side.
Reproduced by tests/sync/purgatory_fetch.rs: a real relay serves a
repository with two branch tips behind a new counting smart-HTTP proxy
(tests/common/upload_pack_counting_proxy.rs), and the relay under test
holds a state event declaring those tips plus eight unfetchable ones
(events delivered via a MockRelay bootstrap so purgatory sync takes the
immediate path). On the previous implementation the test fails with a
failed upload-pack request carrying all ten wants; with this change the
first want-carrying request is the successful batch of advertised tips
and every failed request carries at most one want.
Excluded scope: sync-pass scheduling/backoff, URL selection, and the
upload-pack serving side are untouched; the third-party crawler hitting
gitnostr.com only stops when that instance upgrades.
Validation: new scenario test fails on master and passes here (stable
across three runs); full `cargo test` suite green; clippy introduces no
new warnings.