mirror of
https://relay.ngit.dev/npub15qydau2hjma6ngxkl2cyar74wzyjshvl65za5k5rl69264ar2exs5cyejr/ngit-grasp.git
synced 2026-10-05 15:08:24 +00:00
DanConwayDev
3200e7b31a
Purgatory git sync listed every needed commit id as an explicit want in
a single `git fetch <url> <oid1> <oid2> ...` and, on each upload-pack
"not our ref" rejection, dropped that one oid and re-sent the entire
remaining batch. Against a state event declaring tips that exist on no
reachable server (production evidence: the `market` repository, whose
declared-but-missing ref tips were crawled against gitnostr.com at
150-270 requests/min, 4,521 "not our ref" upload-pack errors in the
45-minute window on 2026-08-05) this degenerated into one failed
upload-pack round trip per missing tip with O(N^2) want retransmission,
pack data streamed and aborted on every failure, and nothing fetched
until the crawl finished. A single missing object failed every batch
that contained it.
fetch_oids now runs three phases through the same hardened/pinned
subprocess machinery (outbound policy authorization and DNS pinning are
unchanged and now also cover ls-remote):
1. `git ls-remote` compares the remote's advertised refs against the
needed oids; state-event ref tips and `refs/nostr/<event-id>` PR
tips surface in the advertisement, so matching oid sets suffices.
2. One batch `git fetch` of the needed oids the remote advertises.
Advertised oids are always valid wants, so "not our ref" cannot fail
this batch; if the advertisement races a ref update, the pass
degrades to per-oid fetches instead of failing.
3. Residual oids are requested one at a time, so a missing object costs
exactly one small round trip and never aborts the rest. Genuine
remote failures still feed the naughty list and stop the pass while
keeping what the batch already fetched.
Client-side observability the old debug-only retry loop lacked: one
INFO summary line per pass (needed / advertised tips / residual
attempted / residual missing / fetched) plus new counters
ngit_purgatory_git_fetch_passes_total and
ngit_purgatory_git_fetch_oids_total{kind}, so production verification
can observe gitnostr.com's own outbound behaviour rather than only the
serving side.
Reproduced by tests/sync/purgatory_fetch.rs: a real relay serves a
repository with two branch tips behind a new counting smart-HTTP proxy
(tests/common/upload_pack_counting_proxy.rs), and the relay under test
holds a state event declaring those tips plus eight unfetchable ones
(events delivered via a MockRelay bootstrap so purgatory sync takes the
immediate path). On the previous implementation the test fails with a
failed upload-pack request carrying all ten wants; with this change the
first want-carrying request is the successful batch of advertised tips
and every failed request carries at most one want.
Excluded scope: sync-pass scheduling/backoff, URL selection, and the
upload-pack serving side are untouched; the third-party crawler hitting
gitnostr.com only stops when that instance upgrades.
Validation: new scenario test fails on master and passes here (stable
across three runs); full `cargo test` suite green; clippy introduces no
new warnings.
46 lines
1.4 KiB
Rust
46 lines
1.4 KiB
Rust
//! Proactive Sync Integration Tests
|
|
//!
|
|
//! This test file organizes tests for ngit-grasp's proactive sync functionality.
|
|
//! Tests are grouped into submodules by sync scenario:
|
|
//!
|
|
//! - `historic_sync` - Tests for sync from pre-configured bootstrap relay (historic events)
|
|
//! - `discovery` - Tests for relay discovery from announcement events
|
|
//! - `live_sync` - Tests for real-time sync after connection established
|
|
//! - `tag_variations` - Tests for different Layer 2/3 tag types
|
|
//! - `catchup` - Tests for catchup sync after disconnect (not yet implemented)
|
|
//! - `metrics` - Tests for Prometheus metrics integration
|
|
//!
|
|
//! # Running Tests
|
|
//!
|
|
//! ```bash
|
|
//! # Run all sync tests
|
|
//! cargo test --test sync
|
|
//!
|
|
//! # Run with output
|
|
//! cargo test --test sync -- --nocapture
|
|
//!
|
|
//! # Run specific test
|
|
//! cargo test --test sync test_bootstrap_syncs -- --nocapture
|
|
//!
|
|
//! # Run ignored tests (like catchup)
|
|
//! cargo test --test sync -- --ignored
|
|
//! ```
|
|
|
|
// Include the common test utilities
|
|
mod common;
|
|
|
|
// Include sync test submodules (located in tests/sync/)
|
|
mod sync {
|
|
pub mod catchup;
|
|
pub mod discovery;
|
|
pub mod historic_recovery;
|
|
pub mod historic_sync;
|
|
pub mod live_sync;
|
|
pub mod maintainer_reprocessing;
|
|
pub mod metrics;
|
|
pub mod neg_concurrency;
|
|
pub mod purgatory_fetch;
|
|
pub mod req_concurrency;
|
|
pub mod tag_variations;
|
|
}
|