mirror of
https://relay.ngit.dev/npub15qydau2hjma6ngxkl2cyar74wzyjshvl65za5k5rl69264ar2exs5cyejr/ngit-grasp.git
synced 2026-10-05 23:18:24 +00:00
DanConwayDev
3200e7b31a
Purgatory git sync listed every needed commit id as an explicit want in
a single `git fetch <url> <oid1> <oid2> ...` and, on each upload-pack
"not our ref" rejection, dropped that one oid and re-sent the entire
remaining batch. Against a state event declaring tips that exist on no
reachable server (production evidence: the `market` repository, whose
declared-but-missing ref tips were crawled against gitnostr.com at
150-270 requests/min, 4,521 "not our ref" upload-pack errors in the
45-minute window on 2026-08-05) this degenerated into one failed
upload-pack round trip per missing tip with O(N^2) want retransmission,
pack data streamed and aborted on every failure, and nothing fetched
until the crawl finished. A single missing object failed every batch
that contained it.
fetch_oids now runs three phases through the same hardened/pinned
subprocess machinery (outbound policy authorization and DNS pinning are
unchanged and now also cover ls-remote):
1. `git ls-remote` compares the remote's advertised refs against the
needed oids; state-event ref tips and `refs/nostr/<event-id>` PR
tips surface in the advertisement, so matching oid sets suffices.
2. One batch `git fetch` of the needed oids the remote advertises.
Advertised oids are always valid wants, so "not our ref" cannot fail
this batch; if the advertisement races a ref update, the pass
degrades to per-oid fetches instead of failing.
3. Residual oids are requested one at a time, so a missing object costs
exactly one small round trip and never aborts the rest. Genuine
remote failures still feed the naughty list and stop the pass while
keeping what the batch already fetched.
Client-side observability the old debug-only retry loop lacked: one
INFO summary line per pass (needed / advertised tips / residual
attempted / residual missing / fetched) plus new counters
ngit_purgatory_git_fetch_passes_total and
ngit_purgatory_git_fetch_oids_total{kind}, so production verification
can observe gitnostr.com's own outbound behaviour rather than only the
serving side.
Reproduced by tests/sync/purgatory_fetch.rs: a real relay serves a
repository with two branch tips behind a new counting smart-HTTP proxy
(tests/common/upload_pack_counting_proxy.rs), and the relay under test
holds a state event declaring those tips plus eight unfetchable ones
(events delivered via a MockRelay bootstrap so purgatory sync takes the
immediate path). On the previous implementation the test fails with a
failed upload-pack request carrying all ten wants; with this change the
first want-carrying request is the successful batch of advertised tips
and every failed request carries at most one want.
Excluded scope: sync-pass scheduling/backoff, URL selection, and the
upload-pack serving side are untouched; the third-party crawler hitting
gitnostr.com only stops when that instance upgrades.
Validation: new scenario test fails on master and passes here (stable
across three runs); full `cargo test` suite green; clippy introduces no
new warnings.
ngit-grasp Documentation
Welcome to the ngit-grasp documentation! We use the Diátaxis framework to organize our documentation into four types, each serving a different purpose.
PRACTICAL THEORETICAL
───────── ───────────
LEARNING │ Tutorials │ Explanation │
│ │ │
│ Getting │ Architecture │
│ Started │ Decisions │
│ │ │
├────────────────┼──────────────────┤
│ │ │
WORKING │ How-To │ Reference │
│ Guides │ │
│ │ API Docs │
│ Deployment │ Protocols │
│ Testing │ │
│ │ │
📚 Documentation Types
🎓 Tutorials - Learning by Doing
Purpose: Learn the basics through practical steps
For: Newcomers getting started
Style: Step-by-step lessons with guaranteed outcomes
- Getting Started - Your first ngit-grasp setup
- Running Your First Audit - Using grasp-audit tool
🔧 How-To Guides - Solving Problems
Purpose: Accomplish specific tasks
For: Users with basic knowledge solving real problems
Style: Practical recipes and solutions
- Deploy ngit-grasp - Production deployment guide
- Configure Nix Flakes - Nix development environment
- Run Compliance Tests - GRASP compliance testing
- Upgrade nostr-sdk - Handling SDK upgrades
📖 Reference - Technical Information
Purpose: Look up technical details
For: Users who know what they're looking for
Style: Dry, factual, comprehensive
- Git Protocol - Git Smart HTTP protocol details
- GRASP Protocol - GRASP specification details
- Configuration - All config options
- API Reference - Internal API documentation
💡 Explanation - Understanding Concepts
Purpose: Understand the "why" and design decisions
For: Users wanting deeper understanding
Style: Discussion, context, alternatives
- Architecture Overview - System design and components
- Inline Authorization - Why we chose this approach
- Comparison with ngit-relay - How we differ from reference
- Design Decisions - Key architectural choices
🚀 Quick Start Paths
I'm brand new to ngit-grasp
- Read README.md for project overview
- Follow Getting Started Tutorial
- Understand Architecture Overview
I want to deploy ngit-grasp
- Review Configuration Reference
- Follow Deployment How-To
- Set up monitoring and backups
I want to develop on ngit-grasp
- Follow Getting Started Tutorial
- Read Architecture Overview
- Check Nix Flakes How-To
- Review Test Strategy
I want to understand the design
- Read Inline Authorization Explanation
- Review Design Decisions
- Compare with ngit-relay Comparison
I'm looking for specific information
- Protocol details? → Reference
- Configuration options? → Configuration Reference
- Git protocol? → Git Protocol Reference
📂 Additional Resources
Archive
Historical session notes and completed work. Useful for understanding project evolution but not required reading.
Learnings
DEPRECATED - Being migrated to Diátaxis structure:
- Gotchas → How-To Guides
- Patterns → Reference or Explanation
- Notes → Appropriate category
🤝 Contributing to Documentation
When adding documentation, ask yourself:
Is it a tutorial?
- Does it teach a beginner?
- Is it a complete lesson with guaranteed outcome?
- → Add to
tutorials/
Is it a how-to guide?
- Does it solve a specific problem?
- Is it a recipe for accomplishing a task?
- → Add to
how-to/
Is it reference material?
- Is it technical information?
- Will people look it up when needed?
- → Add to
reference/
Is it explanation?
- Does it explain "why"?
- Does it discuss alternatives or design?
- → Add to
explanation/
See Diátaxis documentation for more guidance.
📊 Project Status
ALPHA - Under active development. Core functionality working, API may change.
Completed
- ✅ grasp-audit compliance testing tool
- ✅ Nix flake development environment
- ✅ nostr-sdk 0.43 upgrade
- ✅ Documentation restructure (Diátaxis)
In Progress
- 🔄 Core ngit-grasp server implementation
- 🔄 GRASP-01 compliance
Planned
- 🔜 GRASP-02 (Proactive Sync)
- 🔜 GRASP-05 (Archive)
🔗 External Links
Documentation structure based on Diátaxis
Last updated: November 4, 2025