ngit-grasp/grasp-audit/README.md

GRASP Audit

A reusable audit and compliance testing tool for GRASP protocol implementations.

Features

• ✅ Shared Fixtures: Fixtures cached and reused across tests (default for CLI)
• ✅ Isolated Testing: Fresh fixtures per test for parallel test isolation
• ✅ Clean Audit Events: Special tags for easy cleanup (no deletion trails)
• ✅ Spec-Mirrored Tests: Test structure matches GRASP protocol exactly
• ✅ Reusable: Can test any GRASP implementation (Rust, Go, Python, etc.)

Quick Start

The fastest way to run GRASP-01 compliance tests:

# Run the test suite against ngit-relay
cd grasp-audit
nix develop -c bash test-ngit-relay.sh --mode test

This automatically:

• ✅ Starts ngit-relay in an isolated Docker container
• ✅ Runs all GRASP-01 compliance tests
• ✅ Cleans up resources when finished

For more options:

./test-ngit-relay.sh --help

Usage Examples

As a Library

use grasp_audit::*;

#[tokio::main]
async fn main() -> Result<()> {
    // Create audit client with shared fixtures (default for CLI)
    let config = AuditConfig::shared();
    let client = AuditClient::new("ws://localhost:7000", config).await?;

    // Run NIP-01 smoke tests
    let results = specs::Nip01SmokeTests::run_all(&client).await;
    results.print_report();

    if !results.all_passed() {
        std::process::exit(1);
    }

    Ok(())
}

As a CLI Tool

# Install
cargo install --path .

# Run smoke tests against local relay (shared fixtures - default)
grasp-audit audit --relay ws://localhost:7000 --spec nip01-smoke

# Run with isolated fixtures (each test gets fresh fixtures)
grasp-audit audit --relay ws://localhost:7000 --mode isolated --spec all

Test Specifications

NIP-01 Smoke Tests (6 tests)

Basic Nostr relay functionality:

1. websocket_connection - Can connect to /
2. send_receive_event - Can send EVENT, get OK
3. create_subscription - Can subscribe with REQ
4. close_subscription - Can close subscriptions
5. reject_invalid_signature - Rejects bad signatures
6. reject_invalid_event_id - Rejects wrong IDs

Why only smoke tests? rust-nostr already has 1000+ tests for NIP-01 compliance. We focus on GRASP-specific behavior.

GRASP-01 Tests (Coming Soon)

• Repository announcement acceptance
• State event handling
• Policy enforcement
• And more...

Audit Event Strategy

All audit events automatically include special tags for isolation and cleanup:

{
  "tags": [
    ["t", "grasp-audit-test-event"],
    ["t", "audit-ci-a1b2c3d4-e5f6-7890-abcd-ef1234567890"],
    ["t", "audit-cleanup-after-1730822334"]
  ]
}

Tag Format:

• ["t", "grasp-audit-test-event"] - Identifies all audit-related events
• ["t", "audit-{run_id}"] - Unique identifier for each audit run
  • Shared mode: audit-audit-{uuid}
  • Isolated mode: audit-isolated-{uuid}
• ["t", "audit-cleanup-after-{unix_timestamp}"] - Cleanup scheduling
  • Default: Current time + 3600 seconds (1 hour)

Benefits:

• Automatic: Tags added automatically to all events via AuditEventBuilder
• Isolation: Each test run has unique ID for event filtering
• Cleanup: Events marked for cleanup after timestamp (direct database cleanup)
• No deletion trails: No NIP-09 deletion events needed
• Discovery: Easy to query all audit events via hashtag

Fixture Modes

The audit tool supports two fixture caching modes that control how test prerequisites are managed.

Shared Mode (Default for CLI)

Fixtures are cached and reused across all tests. Use this when:

• Running the CLI audit tool sequentially
• Tests build on each other's fixtures
• You want efficient resource usage

let config = AuditConfig::shared();

Isolated Mode (For Parallel Tests)

Each test creates fresh fixtures for complete isolation. Use this when:

• Running cargo test in parallel
• Tests must not interfere with each other
• Testing the fixture system itself

let config = AuditConfig::isolated();

CLI Usage

# Default: shared fixtures (efficient for sequential CLI runs)
grasp-audit audit --relay ws://localhost:7000 --spec all

# Isolated fixtures (for testing)
grasp-audit audit --relay ws://localhost:7000 --mode isolated --spec all

Examples

See examples/ directory:

# Simple audit example
cargo run --example simple_audit

Testing

TL;DR: See the Quick Start section for the fastest way to run tests.

Unit Tests

# Enter dev environment (NixOS)
nix develop

# Run unit tests (no relay required)
cargo test

Integration Tests

The recommended approach is test-ngit-relay.sh, which handles all relay lifecycle management automatically.

See the Quick Start section for common usage patterns.

Advanced: Manual Relay Setup

Click to expand manual testing instructions

For advanced use cases where you need direct control over the relay:

# Start relay on a specific port (example uses 18081)
docker run --rm -p 18081:8081 ghcr.io/danconwaydev/ngit-relay:latest

# In another terminal, run tests with RELAY_URL
grasp-audit audit --relay ws://localhost:18081 --mode ci

# or run all ignored tests via cargo
RELAY_URL="ws://localhost:18081" cargo test --lib -- --ignored --nocapture

# or run specific test via cargo
RELAY_URL="ws://localhost:18081" cargo test --lib test_grasp01_nostr_relay_against_relay -- --ignored --nocapture

Test Design Pattern: Fixture-First

To prevent rate-limiting from production relays during testing, we use a fixture-first approach that minimizes relay interactions.

Quick Start for New Tests

1. Create TestContext at test start
2. Get prerequisites via ctx.get_fixture(FixtureKind::...)
3. Build test-specific events using fixtures as base
4. Verify outcomes via send_and_verify_accepted/rejected

Pattern Template

pub async fn test_something(client: &AuditClient) -> TestResult {
    TestResult::new(...)
        .run(|| async {
            // 1. Context
            let ctx = TestContext::new(client);

            // 2. Prerequisites (cached per-TestContext)
            let repo = ctx.get_fixture(FixtureKind::ValidRepo).await?;

            // 3. Test-specific event
            let my_event = client.create_issue(&repo, "Title", "Content", vec![])?;

            // 4. Verify
            send_and_verify_accepted(client, my_event, "description").await?;

            Ok(())
        })
        .await
}

Three-Layer Architecture

┌─────────────────────────────────────────────────────────────────┐
│                    Layer 3: Test Functions                       │
│  Create TestContext, get fixtures, build scenarios, verify       │
├─────────────────────────────────────────────────────────────────┤
│           Layer 2: FixtureKind + TestContext                     │
│  ValidRepo, RepoState, OwnerStateDataPushed, etc.                │
│  Mode-aware caching within TestContext                           │
├─────────────────────────────────────────────────────────────────┤
│               Layer 1: AuditClient                               │
│  event_builder, create_repo_announcement, send_event             │
└─────────────────────────────────────────────────────────────────┘

Available Fixtures

FixtureKind	Provides	Use When
ValidRepo	Accepted repo announcement (kind 30617). Signed by owner keys, lists maintainer in maintainers tag.	Need a repo as prerequisite
RepoWithIssue	Repo + accepted issue (kind 1621)	Testing issue-dependent events
RepoWithComment	Repo + issue + comment (kind 1111)	Testing comment-dependent events
RepoState	Repo + state event (kind 30618). Signed by owner, points to DETERMINISTIC_COMMIT_HASH.	Testing owner state events
PREvent	Repo + PR event (kind 1618). Signed by PR author, points to PR_TEST_COMMIT_HASH.	Testing PR-dependent events
PREventGenerated	PR event built but NOT sent to relay.	Need PR event ID before publishing
PRWrongCommitPushedBeforeEvent	Wrong commit pushed to refs/nostr/<pr-event-id> before PR event sent. Returns unsent PR event.	Testing pre-event ref cleanup
PREventSentAfterWrongPush	PR event sent after wrong commit was pushed. Tests cleanup behavior.	Testing post-event ref cleanup
OwnerStateDataPushed	Full owner push flow: state event + git data pushed. Points to DETERMINISTIC_COMMIT_HASH.	Testing owner push authorization
MaintainerStateDataPushed	Full maintainer push flow: force-pushes over owner's data. Points to MAINTAINER_DETERMINISTIC_COMMIT_HASH.	Testing maintainer push authorization
RecursiveMaintainerStateDataPushed	Full recursive maintainer push flow: Owner → Maintainer → RecursiveMaintainer chain. Points to RECURSIVE_MAINTAINER_DETERMINISTIC_COMMIT_HASH.	Testing recursive maintainer authorization
HeadSetToDevelopBranch	State event with HEAD=refs/heads/develop. Depends on RecursiveMaintainerStateDataPushed.	Testing HEAD branch switching

Deterministic Commit Hashes

Fixtures use deterministic commit hashes for reproducible testing:

Constant	Hash	Used By
DETERMINISTIC_COMMIT_HASH	64ea71d79a57a7acb334cd9651f8aec067c0ce5d	Owner fixtures (RepoState, OwnerStateDataPushed)
MAINTAINER_DETERMINISTIC_COMMIT_HASH	1c2d472c9b71ed51968a66500281a3c4a6840464	MaintainerStateDataPushed
RECURSIVE_MAINTAINER_DETERMINISTIC_COMMIT_HASH	05939b82de66fbdb9c077d0a64fc68522f3cb8e0	RecursiveMaintainerStateDataPushed
PR_TEST_COMMIT_HASH	5d40fb1555a0c28bf4d650515a73aaa54d4d9bfb	PR fixtures (PREvent, PREventGenerated)

Fixture Dependencies

Fixtures automatically resolve their dependencies:

ValidRepo (base)
├── RepoWithIssue → RepoWithComment
├── RepoState
├── PREventGenerated → PRWrongCommitPushedBeforeEvent → PREventSentAfterWrongPush
├── PREvent
└── OwnerStateDataPushed
    └── MaintainerStateDataPushed
        └── RecursiveMaintainerStateDataPushed
            └── HeadSetToDevelopBranch

Fixture Lifecycle: Generate → Send → Verify → DataPushed

Every fixture follows a lifecycle (some stop earlier):

1. GENERATE: Build event via AuditClient.event_builder() (in memory only)
2. SEND: client.send_event(event) transmits to relay (rate-limited operation)
3. VERIFY: Query relay to confirm acceptance/rejection
4. DATA_PUSHED: (DataPushed variants only) Clone repo, create commit, push to git server

Caching happens after the fixture completes - same fixture request returns cached Event.

Note: Some fixtures handle their own event sending (e.g., OwnerStateDataPushed, MaintainerStateDataPushed). These are marked with sends_own_events() -> true.

How TestContext Correlates Events

Each TestContext shares a run_id with all events:

// All events in a TestContext get these tags automatically:
["t", "grasp-audit-test-event"]     // Identifies test events
["t", "audit-{run_id}"]             // Unique ID for this run
["t", "audit-cleanup-after-{ts}"]   // Cleanup timestamp

This enables:

• Event correlation within a test run
• Production relay cleanup scripts
• Test isolation between runs

When NOT to Use Fixtures

Use direct event building (NOT fixtures) when:

• Testing event REJECTION - Build invalid events directly
• Testing signature/ID validation - Need malformed events
• One-off connectivity tests - No prerequisites needed

// Example: Testing rejection (build invalid event directly)
let invalid_event = client.event_builder(Kind::GitRepoAnnouncement, "")
    .tag(Tag::identifier("test"))
    // Missing required 'clone' tag - should be rejected
    .build(client.keys())?;

send_and_verify_rejected(client, invalid_event, "missing clone tag").await?;

Anti-Patterns to Avoid

❌ Creating TestContext inside helper functions - Tests lose cache control

❌ Monolithic setup functions - Mix fixture retrieval with git operations

❌ Direct event creation when fixture exists - Misses caching opportunity

✅ Each test creates own TestContext - Isolation guaranteed

✅ Use fixtures for prerequisites - Caching minimizes relay calls

✅ Build invalid events directly - Only for rejection tests

Architecture

grasp-audit/
├── src/
│   ├── lib.rs              # Public API
│   ├── audit.rs            # Audit config and event tagging
│   ├── client.rs           # Audit client
│   ├── fixtures.rs         # TestContext and FixtureKind
│   ├── result.rs           # Test result types
│   ├── isolation.rs        # Test isolation utilities
│   └── specs/
│       ├── mod.rs
│       └── nip01_smoke.rs  # NIP-01 smoke tests
├── examples/
│   └── simple_audit.rs     # Example usage
└── bin/
    └── grasp-audit.rs      # CLI tool

Development Status

• ✅ Audit framework
• ✅ NIP-01 smoke tests (6 tests)
• 🚧 GRASP-01 relay tests (planned)
• 🚧 GRASP-01 git tests (planned)
• 🚧 Cleanup utilities (planned)

Contributing

This tool is designed to be reusable by any GRASP implementation. Contributions welcome!

License

MIT