ngit-grasp/docs/explanation/architecture.md

ngit-grasp Architecture

Executive Summary

ngit-grasp implements the GRASP protocol in Rust with inline authorization rather than Git hooks. Git push operations are intercepted and validated at the HTTP handler level before reaching the Git repository, eliminating the need for pre-receive hooks.

System Architecture

┌─────────────────────────────────────────────────────────────┐
│                        ngit-grasp                           │
│                     (Single Rust Binary)                    │
├─────────────────────────────────────────────────────────────┤
│                                                             │
│  ┌──────────────────┐              ┌──────────────────┐   │
│  │   HTTP Router    │              │  Nostr Relay     │   │
│  │     (Hyper)      │              │  (nostr-relay-   │   │
│  │                  │              │   builder)       │   │
│  └────────┬─────────┘              └────────┬─────────┘   │
│           │                                 │             │
│           │                                 │             │
│  ┌────────▼──────────────────────────────────▼─────────┐  │
│  │           Shared State & Storage                    │  │
│  │  ┌──────────────┐  ┌──────────────┐                │  │
│  │  │  Repository  │  │  Event Store │                │  │
│  │  │  Manager     │  │  (LMDB/NDB)  │                │  │
│  │  └──────────────┘  └──────────────┘                │  │
│  └─────────────────────────────────────────────────────┘  │
│                                                             │
│  ┌──────────────────────────────────────────────────────┐  │
│  │            Git Protocol Handler                      │  │
│  │                                                      │  │
│  │  1. Receive git-receive-pack request                │  │
│  │  2. Parse ref updates from request                  │  │
│  │  3. Query Nostr relay for state event               │  │
│  │  4. Validate refs against state                     │  │
│  │  5. If valid: spawn git-receive-pack                │  │
│  │  6. If invalid: return HTTP error                   │  │
│  │                                                      │  │
│  └──────────────────────────────────────────────────────┘  │
│                                                             │
└─────────────────────────────────────────────────────────────┘
         │                                      │
         │ HTTP/Git                             │ WebSocket/Nostr
         ▼                                      ▼
    Git Clients                            Nostr Clients

Component Design

1. Main Server (src/main.rs)

Responsibilities:

• Initialize configuration from environment (clap + dotenvy)
• Set up Hyper HTTP server with request routing
• Initialize Nostr relay builder with custom Nip34WritePolicy
• Set up shared storage (LMDB or Memory)
• Handle WebSocket upgrades for Nostr relay
• Handle graceful shutdown

Key Dependencies:

hyper = "1"
tokio = { version = "1", features = ["full"] }
nostr-relay-builder = "0.43"
nostr-sdk = "0.43"
nostr-lmdb = "0.43"

2. HTTP Module (src/http/mod.rs)

Responsibilities:

• Route HTTP requests to appropriate handlers
• WebSocket upgrade for Nostr relay at /
• Git Smart HTTP endpoints at /<npub>/<identifier>.git/*
• Landing pages and NIP-11 document serving
• CORS headers on all responses (GRASP-01 requirement)

Key Implementation Details:

// CORS headers required by GRASP-01 specification
const CORS_ALLOW_ORIGIN: &str = "*";
const CORS_ALLOW_METHODS: &str = "GET, POST";
const CORS_ALLOW_HEADERS: &str = "Content-Type";

/// Add CORS headers to a response builder
fn add_cors_headers(builder: http::response::Builder) -> http::response::Builder {
    builder
        .header("Access-Control-Allow-Origin", CORS_ALLOW_ORIGIN)
        .header("Access-Control-Allow-Methods", CORS_ALLOW_METHODS)
        .header("Access-Control-Allow-Headers", CORS_ALLOW_HEADERS)
}

See src/http/mod.rs:29-84 for the full CORS implementation.

3. Git Module (src/git/)

handlers.rs - Git HTTP Handlers

Implements handlers for Git Smart HTTP protocol:

/// Handle GET /info/refs?service=git-{upload,receive}-pack
pub async fn handle_info_refs(
    repo_path: PathBuf,
    service: GitService,
) -> Result<Response<Full<Bytes>>, GitError>

/// Handle POST /git-upload-pack (clone/fetch)
pub async fn handle_upload_pack(
    repo_path: PathBuf,
    body: Bytes,
) -> Result<Response<Full<Bytes>>, GitError>

/// Handle POST /git-receive-pack (push)
/// THIS IS WHERE THE MAGIC HAPPENS - validates against state before accepting
pub async fn handle_receive_pack(
    repo_path: PathBuf,
    body: Bytes,
    database: SharedDatabase,
    npub: &str,
    identifier: &str,
) -> Result<Response<Full<Bytes>>, GitError>

See src/git/handlers.rs:22-98 for the info-refs implementation.

authorization.rs - Push Validation

Core Logic:

/// Get authorization info for a repository owner
pub async fn get_authorization_for_owner(
    database: &SharedDatabase,
    pubkey: &PublicKey,
    identifier: &str,
) -> Result<AuthorizationResult, AuthorizationError>

/// Validate that pushed refs match the authorized state
pub fn validate_push_refs(
    pushed_refs: &[PushedRef],
    state: &RepositoryState,
) -> Result<(), AuthorizationError>

/// Validate refs/nostr/<event-id> pushes
pub fn validate_nostr_ref_pushes(
    pushed_refs: &[PushedRef],
    database: &SharedDatabase,
) -> Result<(), AuthorizationError>

4. Nostr Module (src/nostr/)

builder.rs - Relay Configuration

The Nip34WritePolicy is the core event validation logic:

/// NIP-34 Write Policy with Full GRASP-01 Event Validation
///
/// Validates all events according to GRASP-01 specification:
/// - Repository announcements must list service in clone and relays tags
///   EXCEPTION: Recursive maintainer announcements are accepted even without
///   listing the service, to enable maintainer chain discovery and GRASP-02 sync
/// - Repository state announcements must have valid structure
/// - Other events must reference accepted repositories or events
/// - Forward references are supported (events referenced by accepted events)
/// - Orphan events with no valid references are rejected
pub struct Nip34WritePolicy {
    domain: String,
    database: SharedDatabase,
    git_data_path: PathBuf,
}

See src/nostr/builder.rs:38-78 for the full policy struct.

events.rs - Event Parsing

Provides structures for parsing NIP-34 events:

/// Parsed repository announcement (Kind 30617)
pub struct RepositoryAnnouncement { ... }

/// Parsed repository state (Kind 30618)
pub struct RepositoryState { ... }

policy/state.rs - State Event Authorization

State events undergo authorization checks at multiple points:

/// State event authorization checks:
/// 1. Announcement must exist for the repository identifier
/// 2. Author must be in maintainer set of accepted announcement
/// 3. Validated on arrival, announcement acceptance, and git data arrival

Defense-in-depth authorization:

• On arrival (StatePolicy): Initial authorization check
• On announcement acceptance: Purgatory re-evaluation of waiting state events
• On git data arrival: Final authorization before database save

5. Purgatory System (src/purgatory/)

The purgatory system solves two related problems:

1. "Which arrives first?" — Either nostr events or git pushes can arrive in any order. Purgatory holds events awaiting their git data counterparts.
2. Misleading empty repository announcements — New announcements are held in purgatory until git data arrives, ensuring clients are never served announcements for repos with no content.

Design Document: See purgatory-design.md for complete design specifications.

Architecture

/// Main purgatory structure with separate stores per event type
pub struct Purgatory {
    /// Announcement events (kind 30617) indexed by (owner, identifier)
    /// Held until git data proves content exists
    announcement_purgatory: DashMap<(PublicKey, String), AnnouncementPurgatoryEntry>,

    /// State events (kind 30618) indexed by repository identifier
    state_events: DashMap<String, Vec<StatePurgatoryEntry>>,
    
    /// PR events (kind 1617/1618) or placeholders indexed by event ID
    pr_events: DashMap<String, PrPurgatoryEntry>,
}

Key Design Principles:

1. Separate Storage: Each event type uses a different indexing strategy

   • Announcements: Indexed by (pubkey, identifier) (unique per owner)
   • State events: Indexed by identifier (multiple events can wait for same repo)
   • PR events: Indexed by event_id (one-to-one mapping)

2. Announcement Purgatory: New announcements are held until git data arrives

   • Bare repo created immediately so pushes can succeed
   • Announcement promoted to database only when git data proves content exists
   • Two-phase soft expiry: bare repo deleted at 30 min, event retained 24h for revival

3. Late Binding: State event refs are extracted at git push time, not event arrival

   • Enables flexible matching when pushes arrive out-of-order
   • Helper functions in helpers.rs handle ref extraction

4. Bidirectional Waiting: Either side can arrive first

   • Event-first: Event waits for git push
   • Git-first: Placeholder created, waits for event

5. Automatic Expiry: 30-minute default expiry, extensible during processing

   • Background cleanup task runs every 60 seconds
   • Removes expired entries from all stores

Data Types

See types.rs for complete definitions:

• RefPair: Ref name + object SHA pair
• AnnouncementPurgatoryEntry: Announcement with bare repo path, relays, and expiry
• StatePurgatoryEntry: State event with metadata
• PrPurgatoryEntry: PR event or placeholder with metadata

Integration Points

Write Policy (src/nostr/policy/):

• Announcement policy routes new announcements to purgatory; replacements accepted immediately
• State policy checks git data existence before adding to purgatory; checks purgatory announcements for authorization
• PR policy checks for placeholders before adding to purgatory
• Events return "purgatory: will not be served until git data arrives" message

Git Handlers (src/git/handlers.rs):

• On git push: Promote announcement from purgatory to database if present
• On git push: Check purgatory for matching state events
• On refs/nostr/* push: Check purgatory for PR events or create placeholders
• Release events from purgatory when git data arrives
• Save released events to database

Main.rs (src/main.rs):

• Creates Arc<Purgatory> at startup
• Passes to both write policy and git handlers
• Spawns background cleanup task (60-second interval)

Thread Safety

• Uses Arc<DashMap> for lock-free concurrent access
• Safe to share between HTTP handlers, WebSocket handlers, and background tasks
• No blocking locks in hot paths

6. Configuration (src/config.rs)

pub struct Config {
    pub domain: String,
    pub owner_npub: String,
    pub relay_name: String,
    pub relay_description: String,
    pub git_data_path: PathBuf,
    pub relay_data_path: PathBuf,
    pub bind_address: SocketAddr,
    pub database_backend: DatabaseBackend,
}

pub enum DatabaseBackend {
    Lmdb,    // Default, production use
    NostrDb, // Alternative
    Memory,  // Testing
}

Configuration is loaded via clap CLI > environment variables > .env > defaults.

Data Flow

Push Operation Flow

1. Git Client → POST /<npub>/<id>.git/git-receive-pack
                ↓
2. HttpService routes to git::handlers::handle_receive_pack()
                ↓
3. Parse ref updates from request body (pkt-line format)
                ↓
4. Extract npub and identifier from URL
                ↓
5. authorization::get_authorization_for_owner()
   ├─ Query database for announcements
   ├─ Build recursive maintainer set
   └─ Get latest authorized state
                ↓
6. authorization::validate_push_refs()
   ├─ Check each ref matches state
   └─ Validate refs/nostr/ pushes
                ↓
7. If VALID:
   ├─ Spawn git-receive-pack subprocess
   ├─ Stream request body to git stdin
   └─ Stream git stdout back to client
                ↓
8. If INVALID:
   └─ Return HTTP 403 with error message

Repository Announcement Flow

1. Nostr Client → EVENT (Kind 30617)
                ↓
2. Nostr relay receives event
                ↓
3. Nip34WritePolicy::admit_event()
   ├─ Check if instance in clone tags
   ├─ Check if instance in relays tags
   ├─ OR: Check if recursive maintainer
   └─ Accept or reject
                ↓
4. If ACCEPTED:
   ├─ Is there an active announcement for (pubkey, identifier) in DB?
   │  ├─ YES → Accept immediately (replacement, repo already proven)
   │  └─ NO  → Route to announcement purgatory
                ↓
5. Announcement Purgatory path:
   ├─ Bare Git repository created immediately at
   │  <git_data_path>/<npub>/<identifier>.git
   ├─ Announcement held in purgatory (not served to clients)
   └─ Awaiting git data to prove content exists
                ↓
6. When git data arrives (push or background sync):
   ├─ Announcement promoted from purgatory to database
   ├─ Event now served to clients
   └─ SyncManager upgrades to Full sync level
                ↓
7. If no git data within 30 minutes:
   ├─ Bare repo deleted (soft expiry)
   ├─ Event retained 24h for potential revival
   └─ Eventually discarded if no git data arrives

State Event Flow

1. Nostr Client → EVENT (Kind 30618)
                ↓
2. Nostr relay receives event
                ↓
3. Nip34WritePolicy::admit_event()
   ├─ Check author is in maintainer set (DB + purgatory announcements)
   ├─ Validate state structure
   └─ Accept or reject
                ↓
4. If ACCEPTED:
   ├─ Does git data already exist for this state?
   │  ├─ YES → Save to database immediately
   │  └─ NO  → Add to state purgatory
                ↓
5. State Purgatory path:
   ├─ Event held in purgatory (not served to clients)
   ├─ Enqueued for background git data sync (3 min delay)
   └─ Awaiting git push or background sync
                ↓
6. When git push arrives:
   ├─ Authorization checks both database AND purgatory
   ├─ If authorized via purgatory state: push proceeds
   ├─ After successful push: state event saved to database
   └─ Removed from purgatory

Testing Strategy

See test-strategy.md for comprehensive testing documentation.

Quick Overview

Integration Tests (tests/):

• Use TestRelay fixture for automatic relay lifecycle
• Each test file in tests/ covers a GRASP-01 requirement

Audit Tests (grasp-audit/):

• Reusable compliance testing for any GRASP implementation
• Spec-mirrored structure in grasp-audit/src/specs/grasp01/

// Example: tests/nip01_compliance.rs
#[tokio::test]
async fn test_nip01_websocket_connection() {
    let relay = TestRelay::start().await;
    // Test NIP-01 compliance...
    relay.stop().await;
}

Performance Considerations

1. Async All The Way

• Use tokio for all I/O
• Non-blocking Git subprocess spawning via GitSubprocess
• Stream large pack files without buffering

2. Shared Database

• Single database instance shared between relay and Git handlers
• Direct queries for push authorization (no WebSocket round-trip)

3. Write Policy Caching

• Maintainer sets computed once per event validation
• State lookups use database indexes

Proactive Sync (GRASP-02)

The ngit-grasp relay implements Proactive Sync of Nostr Eevents, which synchronizes repository data from external relays listed in 30617 repository announcements. This enables the relay to maintain complete repository graphs even when events are published to other listed relays.

Key Features:

• Self-subscription discovery - monitors own relay for announcements and root events to follow
• Three-way diff (compute_actions) determines new subscriptions needed
• Smart reconnection - uses since filter for quick reconnects (<15 min), fresh sync otherwise
• Health tracking with exponential backoff for failing relays
• Daily sync with random 23-25h timer to detect state drift
• Filter consolidation when count exceeds 70 to prevent subscription explosion
• Rejected events index - prevents wasteful re-fetching during negentropy sync

Architecture:

┌─────────────────────────────────────────────────────────────┐
│                       SyncManager                            │
├─────────────────────────────────────────────────────────────┤
│                                                             │
│  ┌──────────────────┐              ┌──────────────────┐    │
│  │  SelfSubscriber  │──actions──▶  │  Main Event Loop │    │
│  │  (own relay)     │              │  (Arc<Mutex>)    │    │
│  └──────────────────┘              └────────┬─────────┘    │
│                                             │              │
│  ┌──────────────────┐              ┌────────▼─────────┐    │
│  │  Daily Timer     │──────────────▶  RelayConnection │    │
│  │  (23-25h random) │              │  per external    │    │
│  └──────────────────┘              │  relay           │    │
│                                    └──────────────────┘    │
│  ┌──────────────────┐                                      │
│  │  Health Tracker  │  Exponential backoff, dead detection │
│  │  (DashMap)       │                                      │
│  └──────────────────┘                                      │
│  ┌──────────────────────────────────────────────────────┐  │
│  │  Rejected Events Index (Two-Tier)                   │  │
│  │  ┌────────────────┐    ┌──────────────────────┐    │  │
│  │  │  Hot Cache     │───▶│  Cold Index          │    │  │
│  │  │  (2 min)       │    │  (7 days)            │    │  │
│  │  │  Full events   │    │  Metadata only       │    │  │
│  │  └────────────────┘    └──────────────────────┘    │  │
│  └──────────────────────────────────────────────────────┘  │
└─────────────────────────────────────────────────────────────┘

Source Code: src/sync/

For full design details, see grasp-02-proactive-sync-v4.md.

Rejected Events Index

The rejected events index solves two critical problems during sync:

1. Negentropy sync efficiency: Prevents repeatedly downloading events that will be rejected again
2. Race condition resolution: Enables immediate re-processing when event dependencies are satisfied

Two-Tier Architecture:

Tier	Duration	Storage	Purpose
Hot Cache	2 minutes	Full events	Immediate re-processing when dependencies arrive
Cold Index	7 days	Metadata only	Prevent re-fetch during negentropy sync

Event Flow:

Event Rejected (e.g., maintainer before owner announcement)
    │
    ├──▶ Store full event in Hot Cache (2 min expiry)
    └──▶ Store metadata in Cold Index (7 day expiry)
    
Dependency Arrives (e.g., owner announcement accepted)
    │
    ├──▶ Invalidate from Cold Index
    ├──▶ Retrieve from Hot Cache (if still available)
    └──▶ Re-process immediately (<1 second vs 24 hours)

Negentropy Sync
    │
    └──▶ Exclude Cold Index IDs from "missing events" calculation

Tracked Events:

• Repository announcements (kind 30617) rejected for not listing this service or maintainer validation failure
• State events (kind 30618) rejected for missing announcements or unauthorized authors

Source Code: src/sync/rejected_index.rs

Contributor PR Submission (GRASP-06)

Optional endpoint at /prs/<npub>/<identifier>.git, gated on NGIT_GRASP06_ENABLE (default off). Contributors push refs/nostr/<event-id> for PR and PR Update events targeting any repository — even repos this relay has no accepted announcement for. The endpoint is unauthenticated at the HTTP level; validity is established by the signed PR / PR Update event and inline acceptance rules.

Module layout:

• src/grasp06/endpoint.rs — URL parsing.
• src/grasp06/paths.rs — on-disk path conventions under <git_data_path>/prs/<hex>/<identifier>.git.
• src/grasp06/fetch.rs — empty-repo synthesis for info/refs and git-upload-pack against repos that don't yet exist on disk.
• src/grasp06/receive.rs — git-receive-pack with init-on-push, strict refs/nostr/<event-id> ref-name validation, and per-ref post-push validation against the database and purgatory. Uses a per-(submitter, identifier) PrsPathState (mutex + in_flight counter) so concurrent pushes to the same path proceed in parallel: the mutex is held only for init+register and decrement+end-of-push cleanup, not across git-receive-pack itself. The same state is consulted (with try_lock in synchronous contexts) by the PR-event policy and the purgatory expiry sweep, which only remove a /prs/ ref or zero-ref bare repo when in_flight == 0.
• src/grasp06/policy.rs — strict clone-tag URL comparator used by the PR-event acceptance relaxation.

/prs/ repos are intentionally isolated from other subsystems: empty-repo cleanup skips the /prs/ subtree, the proactive-sync subsystem never discovers them because subscriptions are built from DB-resident announcements, and the standard repo landing page guards against ever matching a /prs/ path. Full design: GRASP-06 Contributor Pull Request Submission. Operator how-to: Enable GRASP-06.

Future Extensions

GRASP-02: Proactive Sync

GRASP-02 is only partially implemented. still outstanding is the proactive sync of git data for 1. state event and 2. PRs / PR Update refs.

GRASP-05: Archive

Relax the write policy to accept all repository announcements regardless of clone/relays tags.

Deployment

Single Binary

cargo build --release
./target/release/ngit-grasp --domain example.com --owner-npub npub1...

Docker

FROM rust:1.75 as builder
WORKDIR /app
COPY . .
RUN cargo build --release

FROM debian:bookworm-slim
RUN apt-get update && apt-get install -y git && rm -rf /var/lib/apt/lists/*
COPY --from=builder /app/target/release/ngit-grasp /usr/local/bin/
EXPOSE 7334
CMD ["ngit-grasp"]

Systemd

[Unit]
Description=ngit-grasp GRASP server
After=network.target

[Service]
Type=simple
User=git
WorkingDirectory=/opt/ngit-grasp
EnvironmentFile=/opt/ngit-grasp/.env
ExecStart=/usr/local/bin/ngit-grasp
Restart=on-failure

[Install]
WantedBy=multi-user.target

Security Considerations

1. Input Validation: All npub/identifier inputs must be validated
2. Path Traversal: Prevent directory traversal in repository paths
3. DoS Protection: Rate limiting on both HTTP and WebSocket
4. Resource Limits: Limit pack file sizes, event sizes
5. Nostr Event Validation: Strict signature verification (handled by nostr-relay-builder)

Conclusion

ngit-grasp uses inline authorization at the HTTP handler level, giving full control over request handling, WebSocket upgrades, and CORS headers while maintaining full GRASP-01 compliance. The purgatory system ensures that only repositories with actual git content are served to clients, and that events and git data are always consistent when released to the database.

Related Documentation

• Inline Authorization Explanation - Why we chose this approach
• GRASP-02 Proactive Sync v4 Design - Current production sync implementation
• Test Strategy - Comprehensive testing documentation
• GRASP-01 Implementation Learnings - Patterns and lessons learned