mirror of
https://relay.ngit.dev/npub15qydau2hjma6ngxkl2cyar74wzyjshvl65za5k5rl69264ar2exs5cyejr/ngit-grasp.git
synced 2026-10-05 23:18:24 +00:00
313 lines
10 KiB
Markdown
313 lines
10 KiB
Markdown
# Test Strategy for ngit-grasp
|
||
|
||
## Overview
|
||
|
||
This document describes the testing strategy for ngit-grasp, including the **grasp-audit** reusable compliance testing tool and the **integration tests** in the main repository.
|
||
|
||
## Testing Philosophy
|
||
|
||
1. **Specification-Driven**: Tests mirror GRASP-01 protocol structure exactly
|
||
2. **Compliance-First**: Every requirement in the spec has a corresponding test
|
||
3. **Reusable**: The grasp-audit tool can validate any GRASP implementation
|
||
4. **Isolated**: Each test runs with its own relay instance via [`TestRelay`](tests/common/relay.rs:14)
|
||
5. **Clear Failures**: Test failures cite exact spec requirements
|
||
|
||
## Test Pyramid
|
||
|
||
```
|
||
╱╲
|
||
╱ ╲
|
||
╱ E2E╲ ~ 10% End-to-end with real Git
|
||
╱──────╲
|
||
╱ ╲
|
||
╱Compliance╲ ~ 30% GRASP-01 spec validation
|
||
╱────────────╲ (grasp-audit)
|
||
╱ ╲
|
||
╱ Integration ╲ ~ 30% Component interaction
|
||
╱──────────────────╲ (tests/)
|
||
╱ ╲
|
||
╱ Unit Tests ╲ ~ 30% Individual functions
|
||
╱────────────────────────╲ (src/**/tests)
|
||
```
|
||
|
||
## Project Structure
|
||
|
||
### Actual Test Layout
|
||
|
||
```
|
||
ngit-grasp/
|
||
├── tests/ # Integration tests for ngit-grasp
|
||
│ ├── common/
|
||
│ │ ├── mod.rs # Test utilities module
|
||
│ │ └── relay.rs # TestRelay fixture
|
||
│ ├── nip01_compliance.rs # NIP-01 relay compliance
|
||
│ ├── nip11_document.rs # NIP-11 document tests
|
||
│ ├── nip34_announcements.rs # Repository announcement tests
|
||
│ ├── repository_creation.rs # Git repo creation tests
|
||
│ ├── push_authorization.rs # Push validation tests
|
||
│ ├── cors.rs # CORS header tests
|
||
│ └── git_clone.rs # Git clone tests
|
||
│
|
||
└── grasp-audit/ # Reusable GRASP compliance tool
|
||
├── Cargo.toml
|
||
├── flake.nix
|
||
└── src/
|
||
├── lib.rs # Public API
|
||
├── client.rs # AuditClient
|
||
├── audit.rs # AuditConfig, cleanup tags
|
||
├── fixtures.rs # Test fixtures
|
||
└── specs/
|
||
└── grasp01/ # GRASP-01 specification tests
|
||
├── mod.rs # Module exports
|
||
├── nip01_smoke.rs # NIP-01 smoke tests
|
||
├── nip11_document.rs
|
||
├── event_acceptance_policy.rs
|
||
├── cors.rs
|
||
├── git_clone.rs
|
||
├── push_authorization.rs
|
||
├── repository_creation.rs
|
||
└── spec_requirements.rs # Requirement definitions
|
||
```
|
||
|
||
## Integration Tests (tests/)
|
||
|
||
### TestRelay Fixture
|
||
|
||
The [`TestRelay`](tests/common/relay.rs:14) fixture provides automatic relay lifecycle management:
|
||
|
||
```rust
|
||
// From tests/common/relay.rs
|
||
|
||
/// Test relay fixture that manages relay lifecycle
|
||
///
|
||
/// Automatically starts and stops the ngit-grasp relay for testing.
|
||
/// Uses a random port to avoid conflicts and cleans up created repositories.
|
||
pub struct TestRelay {
|
||
process: Child,
|
||
url: String,
|
||
port: u16,
|
||
}
|
||
|
||
impl TestRelay {
|
||
/// Start a test relay instance
|
||
pub async fn start() -> Self { ... }
|
||
|
||
/// Get the relay WebSocket URL
|
||
pub fn url(&self) -> &str { ... }
|
||
|
||
/// Get the relay domain (host:port)
|
||
pub fn domain(&self) -> String { ... }
|
||
|
||
/// Stop the relay
|
||
pub async fn stop(mut self) { ... }
|
||
}
|
||
```
|
||
|
||
### Using TestRelay in Integration Tests
|
||
|
||
From [`tests/nip01_compliance.rs`](tests/nip01_compliance.rs):
|
||
|
||
```rust
|
||
use common::TestRelay;
|
||
use grasp_audit::*;
|
||
|
||
/// Macro to generate isolated integration tests
|
||
macro_rules! isolated_test {
|
||
($test_name:ident) => {
|
||
#[tokio::test]
|
||
async fn $test_name() {
|
||
let relay = TestRelay::start().await;
|
||
let config = AuditConfig::isolated();
|
||
let client = AuditClient::new(relay.url(), config)
|
||
.await
|
||
.expect("Failed to create audit client");
|
||
|
||
let result = specs::Nip01SmokeTests::$test_name(&client).await;
|
||
|
||
relay.stop().await;
|
||
|
||
assert!(
|
||
result.passed,
|
||
"{} failed: {}",
|
||
stringify!($test_name),
|
||
result.error.as_deref().unwrap_or("unknown error")
|
||
);
|
||
}
|
||
};
|
||
}
|
||
|
||
// Generate isolated tests for all NIP-01 smoke tests
|
||
isolated_test!(test_websocket_connection);
|
||
isolated_test!(test_send_receive_event);
|
||
isolated_test!(test_create_subscription);
|
||
```
|
||
|
||
### Running Integration Tests
|
||
|
||
```bash
|
||
# Run all integration tests
|
||
cargo test --test '*'
|
||
|
||
# Run specific test file
|
||
cargo test --test nip01_compliance
|
||
|
||
# Run with output
|
||
cargo test --test nip01_compliance -- --nocapture
|
||
```
|
||
|
||
## GRASP Audit Tool (grasp-audit/)
|
||
|
||
### Purpose
|
||
|
||
The grasp-audit tool is a **reusable GRASP compliance testing library** that can:
|
||
|
||
- Test ngit-grasp for self-validation
|
||
- Test any other GRASP implementation (like ngit-relay)
|
||
- Run in CI/CD for continuous compliance verification
|
||
- Generate compliance reports
|
||
|
||
### Test Suites
|
||
|
||
From [`grasp-audit/src/specs/grasp01/mod.rs`](grasp-audit/src/specs/grasp01/mod.rs):
|
||
|
||
| Suite | Description | Requirements |
|
||
|-------|-------------|--------------|
|
||
| [`Nip01SmokeTests`](grasp-audit/src/specs/grasp01/nip01_smoke.rs) | Basic NIP-01 relay functionality | WebSocket only |
|
||
| [`Nip11DocumentTests`](grasp-audit/src/specs/grasp01/nip11_document.rs) | NIP-11 relay information document | WebSocket only |
|
||
| [`EventAcceptancePolicyTests`](grasp-audit/src/specs/grasp01/event_acceptance_policy.rs) | Event acceptance rules | WebSocket only |
|
||
| [`CorsTests`](grasp-audit/src/specs/grasp01/cors.rs) | CORS headers on Git HTTP endpoints | git-data-dir |
|
||
| [`GitCloneTests`](grasp-audit/src/specs/grasp01/git_clone.rs) | Git clone operations | git-data-dir |
|
||
| [`PushAuthorizationTests`](grasp-audit/src/specs/grasp01/push_authorization.rs) | Push authorization | git-data-dir |
|
||
| [`RepositoryCreationTests`](grasp-audit/src/specs/grasp01/repository_creation.rs) | Repository creation | git-data-dir |
|
||
|
||
### Spec Requirements Database
|
||
|
||
From [`grasp-audit/src/specs/grasp01/spec_requirements.rs`](grasp-audit/src/specs/grasp01/spec_requirements.rs):
|
||
|
||
```rust
|
||
pub struct SpecRequirement {
|
||
pub id: &'static str, // e.g., "GRASP-01:L9"
|
||
pub section: &'static str, // e.g., "Nostr Relay"
|
||
pub level: RequirementLevel, // MUST, SHOULD, MAY
|
||
pub text: &'static str, // Exact text from spec
|
||
pub line: u32, // Line number in spec
|
||
}
|
||
|
||
pub enum RequirementLevel {
|
||
Must,
|
||
Should,
|
||
May,
|
||
}
|
||
```
|
||
|
||
### Automatic Cleanup Tags
|
||
|
||
All audit events include cleanup tags for production safety (from [`grasp-audit/src/audit.rs`](grasp-audit/src/audit.rs)):
|
||
|
||
```rust
|
||
// Automatically added to EVERY audit event:
|
||
["t", "grasp-audit-test-event"] // Marker
|
||
["t", "audit-{run_id}"] // Run isolation
|
||
["t", "audit-cleanup-after-{unix_timestamp}"] // Cleanup time
|
||
```
|
||
|
||
### Running grasp-audit
|
||
|
||
**Testing the reference implementation (ngit-relay):**
|
||
|
||
```bash
|
||
# Use test-ngit-relay.sh for automated relay management
|
||
cd grasp-audit && nix develop -c bash test-ngit-relay.sh --mode test
|
||
|
||
# Or manually:
|
||
docker run --rm -p 18081:8081 ghcr.io/danconwaydev/ngit-relay:latest
|
||
cd grasp-audit
|
||
RELAY_URL="ws://localhost:18081" nix develop -c cargo test --lib -- --ignored --nocapture
|
||
```
|
||
|
||
**Testing ngit-grasp (the main project):**
|
||
|
||
```bash
|
||
# Integration tests use TestRelay fixture - just run:
|
||
cargo test --test '*'
|
||
```
|
||
|
||
## Test Patterns
|
||
|
||
### Isolated Test Pattern
|
||
|
||
Each test runs with its own fresh relay instance:
|
||
|
||
```rust
|
||
#[tokio::test]
|
||
async fn test_something() {
|
||
// Start fresh relay
|
||
let relay = TestRelay::start().await;
|
||
|
||
// Run test
|
||
let client = AuditClient::new(relay.url(), AuditConfig::isolated()).await?;
|
||
// ... test logic ...
|
||
|
||
// Cleanup
|
||
relay.stop().await;
|
||
}
|
||
```
|
||
|
||
### Macro-Based Test Generation
|
||
|
||
For test suites that follow the same pattern, use macros:
|
||
|
||
```rust
|
||
macro_rules! isolated_test {
|
||
($test_name:ident) => {
|
||
#[tokio::test]
|
||
async fn $test_name() {
|
||
let relay = TestRelay::start().await;
|
||
// ... standard setup and teardown ...
|
||
}
|
||
};
|
||
}
|
||
|
||
isolated_test!(test_websocket_connection);
|
||
isolated_test!(test_send_receive_event);
|
||
```
|
||
|
||
## Coverage Targets
|
||
|
||
| Test Type | Coverage Target |
|
||
|-----------|-----------------|
|
||
| Unit Tests | >80% line coverage of `src/` |
|
||
| Integration Tests | All critical user paths |
|
||
| GRASP-01 Compliance | 100% of MUST requirements |
|
||
|
||
## CI/CD Integration
|
||
|
||
### Running All Tests
|
||
|
||
```bash
|
||
# Unit tests (fast, no external dependencies)
|
||
cargo test --lib
|
||
|
||
# Integration tests (requires relay binary built)
|
||
cargo build --release
|
||
cargo test --test '*'
|
||
|
||
# Compliance tests against ngit-relay reference
|
||
cd grasp-audit && nix develop -c bash test-ngit-relay.sh --mode test
|
||
```
|
||
|
||
## Summary
|
||
|
||
| What | Where | Purpose |
|
||
|------|-------|---------|
|
||
| Unit tests | `src/**/tests` modules | Test individual functions |
|
||
| Integration tests | `tests/*.rs` | Test ngit-grasp as a whole |
|
||
| TestRelay fixture | [`tests/common/relay.rs`](tests/common/relay.rs) | Manage relay lifecycle |
|
||
| GRASP audit library | `grasp-audit/` | Reusable compliance testing |
|
||
| GRASP-01 specs | [`grasp-audit/src/specs/grasp01/`](grasp-audit/src/specs/grasp01/) | Spec requirement tests |
|
||
|
||
## Related Documentation
|
||
|
||
- [Architecture](../explanation/architecture.md) - System design
|
||
- [GRASP-01 Implementation Learnings](../learnings/grasp-01-implementation.md) - Patterns and lessons
|
||
- [GRASP Audit Learnings](../learnings/grasp-audit.md) - Audit tool patterns |