mirror of
https://relay.ngit.dev/npub15qydau2hjma6ngxkl2cyar74wzyjshvl65za5k5rl69264ar2exs5cyejr/ngit-grasp.git
synced 2026-10-05 23:18:24 +00:00
DanConwayDev
18eddeedb5
nostr:nevent1qgsx2lyl2e4zvfadwcvkd9fkrcwczj7mf858hy85mwqclwgut8wpg2spz3mhxue69uhhyetvv9ujumn8d96zuer9wcq3yamnwvaz7tm8d96xummnw3ezucm0d5q3kamnwvaz7tmwva5hgtnyv9hxxmmwwashjer9wchxxmmdqqsyuueftzc68d5a3x5d0pecs66y0nacv4pc87kkcfw499adyl5t3ycxqgtta PR-Author: DanConwayDev's Agent nostr:npub1v47f74n2ycn66asev62nv8sas99akj0g0wg0fkup37u3ckwuzs4q7cwtp0 CoverNote: ## What this adds Repository conversations can continue on relays used only by a non-root participant. For example, a reaction to an issue may be stored on one relay, while a reply to that reaction exists only on the reaction author's NIP-65 write relay. Root-author relay discovery and recursive reference queries cannot find that reply unless the participant's mailbox is also searched. This PR adds bounded, historic participant-mailbox coverage: 1. Preserve exact repository-root provenance while deriving the accepted recursive descendant frontier. 2. Treat authors of accepted replies, comments, reactions, zaps and other descendants as thread participants. 3. Accept kind `10002` relay lists only for repository owners, maintainers, root authors and those accepted participants. 4. Derive each participant's NIP-65 read, write and unmarked mailbox relays. 5. Query those mailboxes only for the repository roots and accepted descendant IDs associated with that participant. 6. Pass every returned event through the ordinary write policy, deletion/replacement rules and persistence pipeline. This discovers replies to reactions and other indirect descendants without fetching a participant's unrelated notes or trusting their relay list to bypass repository policy. ## Bounds and scheduling - The existing `NGIT_SYNC_RECURSIVE_DESCENDANT_LIMIT` bounds which indirect descendant IDs remain query roots. Each direct event that tags a repository root has its own bounded branch; the default is 500. - Once a branch reaches that bound, later indirect descendants cannot add more participant authors or query roots through that branch. - Exact root provenance prevents roots from one repository leaking into another participant query. - Participant mailbox coverage is historic only. It does not add permanent non-root participant subscriptions or expand the existing live-sync tier. - Filters use the existing byte-bounded grouping and paginated `RelayConnection::fetch_events` path, including request pacing, background priority, subscription-ledger capacity and EOSE/CLOSED handling. - At most one mailbox worker runs per relay, and at most one new due relay is started per maintenance pass. - Relays progress independently: there is no global mailbox lane, cross-relay success condition or shared completion counter. ## Restart and failure behavior - A probe starts only after both the WebSocket and the sync actor's connection lifecycle are ready. - Ready due relays are preferred, so old unreachable mailbox sources cannot starve connected work. - Completed mailbox workers are handled before new connection results, so a large startup connection queue cannot delay cursor progress or resource release. - Successful filters advance an in-memory, stable-sorted cursor immediately. Completing the last filter schedules the next historic rotation after 24 hours. - Failed filters release their relay worker and retry after five minutes without blocking other relays. - On restart, accepted roots, participants and kind `10002` ownership are reconstructed from LMDB. Filter cursors deliberately restart from the first current group; they are best-effort coverage, not durable exactly-once state. The implementation reuses the existing relay transport. It does not add a mailbox-specific protocol state machine, pending-batch purpose, CLOSE API, NIP-42 retry system, watchdog or connection-queue coordinator. ## Coverage and production evidence The audit motivating this change found 72 events visible through gitworkshop but absent from the pre-change production relay across the `gitworkshop` and `ngit` repositories. The persistent canary served 29 exact misses: six kind-1 notes, three reposts, ten reactions, six comments, two issues, one zap receipt and one repository-follow event. The exact candidate `de5fa6625aa5693afcde7335a572456680aef9df` activated on the isolated archive at 2026-08-14 19:23:49 UTC. Restart reconstruction found 4,291 accepted roots, 403 participant authors and 351 mailbox relays. During the recorded soak: - 63 mailbox filters started and all 63 reached terminal handling: 59 successes and four bounded failures. - Failures received the intended five-minute retry and did not block successful progress on other relays. - `relay.ngit.dev`, `nostr.land`, `nostr.mom`, `haven.danconwaydev.com/inbox` and other relays advanced independently. - `nostr.azzamo.net` completed all six filter groups; the final terminal reported `completed_cycle=true` and scheduled the next probe in 86,400 seconds. - The archive remained active with zero restarts and no process panic or fatal error. The temporary archive test override of `NGIT_SYNC_RECURSIVE_DESCENDANT_LIMIT=2` has been removed. Both the archive and public gitnostr.com have no override and therefore use the default of 500. Public gitnostr.com was not restarted or changed by the archive deployment. ## Validation and review state - Exact tested head: `de5fa6625aa5693afcde7335a572456680aef9df`. - One commit; 1,157 additions and 140 deletions across 11 files. - 767 library tests passed in the exact Nix release build locally and on the production host. - All three proactive Sync+ integration scenarios pass together, including a child found only through a participant reaction author's write mailbox. - Strict all-target Clippy, formatting, diff checks, Nix flake evaluation and a conflict-free merge-tree check against current `master` passed. - One remote build attempt hit the pre-existing `test_entries_expired_during_downtime` timing flake; its source is unchanged by this PR, and the unchanged candidate passed all 767 tests on retry before activation. Recommended for merge. The archive demonstrates successful and failed terminal paths, per-relay independence, restart reconstruction, cursor advancement and a complete historic-to-24-hour-refresh rotation under real startup load.
Explanation
Understanding-oriented documentation - Concepts, design decisions, and the "why" behind ngit-grasp.
What Is Explanation?
Explanation documentation helps you understand concepts and design decisions, providing context and discussing alternatives.
Characteristics:
- ✅ Understanding-oriented (clarify concepts)
- ✅ Theoretical (ideas and design)
- ✅ Discuss alternatives
- ✅ Provide context and background
- ✅ Answer "why" questions
Not explanation:
- ❌ Step-by-step lessons (those are Tutorials)
- ❌ Problem-solving recipes (those are How-To)
- ❌ Technical specifications (those are Reference)
Available Explanation Documentation
Architecture Overview
Understand the system design and component interaction
Topics:
- Overall architecture
- Component responsibilities
- Data flows
- Technology choices
- Design patterns
Read when: You want to understand how ngit-grasp works as a system
Inline Authorization
Why we validate pushes inline instead of using Git hooks
Topics:
- The authorization problem
- Git hooks approach
- Inline approach
- Comparison and trade-offs
- Implementation details
Read when: You want to understand the core architectural decision
Design Decisions
Key architectural choices and their rationale
Topics:
- Inline authorization vs hooks
- Technology stack choices
- Storage design
- API design
- Performance considerations
Read when: You want to know why things are the way they are
Comparison with ngit-relay
How ngit-grasp differs from the reference implementation
Topics:
- Architecture comparison
- Component differences
- Trade-offs
- Migration path
- Compatibility
Read when: You're familiar with ngit-relay and want to understand differences
Purgatory Design
In-memory holding area for events awaiting git data
Topics:
- The "which arrives first?" problem
- Separate storage for state vs PR events
- Late binding for state events
- Bidirectional waiting for PR events
- Authorization during push
Read when: You want to understand how ngit-grasp handles out-of-order event/git data arrival
GRASP-02 Proactive Sync
Relay-to-relay synchronization for repository discovery
Topics:
- Negentropy-based event sync
- Repository announcement discovery
- Relay management and reconnection
- Layer 2 filtering
- Bootstrap and dynamic relay discovery
Read when: You want to understand how ngit-grasp discovers and syncs repositories across relays
GRASP-03 Proactive Sync Plus
NIP-65 inbox/outbox discovery for accepted repository conversations
Read when: You want to understand how accepted conversations are recovered from participant mailboxes
Sync Scaling Constraints and Budgets
Relay-imposed limits and how sync spends them at scale
Topics:
- Verified relay limits (strfry, live NIP-11, our embedded relay)
- Per-connection budget ledger (live vs historic vs fallback)
- Byte-budgeted filter chunking and REQ packing
- Bounded negentropy concurrency
- Multi-connection escalation and serving-side obligations
Read when: You're changing filter construction, subscription management, or sync concurrency, and need the constraint justification
GRASP-02 Purgatory Git Data Fetching
Proactive git data fetching from remote servers
Topics:
- Identifier-based batching
- Exponential backoff with fresh start
- Domain throttling (5 concurrent, 30/min)
- Debounced delays (3min user, 500ms sync)
- 30-minute expiry
- Mock-based testability
Read when: You want to understand how purgatory automatically fetches missing git data
Unified Git Data Sync
Shared processing for git push and purgatory sync paths
Topics:
- Why unify push and sync processing
- OID syncing to owner repos
- Ref alignment logic
- Event release from purgatory
- WebSocket notification
Read when: You want to understand how git data is processed consistently regardless of arrival method
Monitoring Overview
Prometheus metrics and observability
Topics:
- Metrics philosophy
- Connection tracking
- Git operation metrics
- Nostr event metrics
- Privacy considerations
Read when: You want to understand how to monitor ngit-grasp in production
Defensive Measures & Rate Limiting
Protection against abuse, spam, and denial-of-service attacks
Topics:
- Connection and subscription management
- Event publishing rate limits
- Content filtering (blacklists/whitelists)
- Event validation plugin system (WritePolicy/QueryPolicy)
- Relay health management (naughty list, exponential backoff)
- Privacy-preserving IP tracking
- Future enhancements (per-IP rate limiting)
Read when: You want to understand how ngit-grasp protects against abuse and what defensive features are available
GRASP-05 Archive Mode
Read-only mirroring of repositories
Topics:
- Archive whitelist configuration
- Archive-all mode
- Read-only mode defaults
- Use cases for backup/mirror relays
Read when: You want to understand how to run an archive/backup relay
Repository Lifecycle
Handling repository removal, holding, archive, recovery, and purgatory
Topics:
- Repository lifecycle architecture
- Delete disrespector concept
- Preventing left-pad scenarios
- Archival policies
- Holding, recovery, purgatory, and operator curation flows
Read when: You want to understand how ngit-grasp keeps nostr state and git data aligned across deletion, vanish, moderation, recovery, and purgatory flows
Planned Explanation Documentation
GRASP Protocol Design
Status: 🔜 Planned
Topics:
- Why Nostr for Git?
- Authorization model
- Trust and verification
- Decentralization benefits
Storage Architecture
Status: 🔜 Planned
Topics:
- Why separate Git and Nostr storage?
- Indexing strategy
- Performance considerations
- Scaling approach
Testing Philosophy
Status: 🔜 Planned
Topics:
- Why test isolation?
- Integration vs unit tests
- Compliance testing approach
- Test-driven development
Performance Considerations
Status: 🔜 Planned
Topics:
- Async architecture
- Caching strategy
- Database choices
- Bottlenecks and solutions
How to Use Explanation Documentation
- Read to understand - Not to accomplish a task
- Follow your curiosity - Read what interests you
- Connect concepts - Link ideas together
- Question and explore - Think critically
Not sure if this is what you need?
- Want to learn by doing? → Tutorials
- Need to solve a problem? → How-To Guides
- Looking for technical details? → Reference
Contributing Explanation Documentation
When writing explanation:
DO:
- ✅ Discuss concepts and ideas
- ✅ Provide context and background
- ✅ Explain alternatives
- ✅ Use analogies and examples
- ✅ Connect to broader context
- ✅ Answer "why" questions
DON'T:
- ❌ Provide step-by-step instructions (link to Tutorials/How-To)
- ❌ List technical details (link to Reference)
- ❌ Assume you must be comprehensive
- ❌ Avoid opinions (explanation can be opinionated)
Template:
# Explanation: [Topic]
**Purpose:** [What concept/decision this explains]
**Audience:** [Who wants to understand this]
---
## The Problem/Question
[What are we trying to understand?]
---
## Background
[Context and history]
---
## Our Approach
[How we address it]
### Why This Works
[Explanation of benefits]
### Trade-offs
[What we gain and lose]
---
## Alternatives Considered
### [Alternative 1]
**Pros:**
- [Benefits]
**Cons:**
- [Drawbacks]
**Why we didn't choose it:**
[Reasoning]
---
## Conclusion
[Summary of understanding]
---
## Related Documentation
- [Links to relevant docs]
See Diátaxis: Explanation for detailed guidance.
Part of the ngit-grasp documentation using the Diátaxis framework.