refactor: rename AuditMode variants and change CLI default to shared

Breaking change: Renamed AuditMode enum variants for clarity:
- AuditMode::CI -> AuditMode::Isolated (fresh fixtures per test)
- AuditMode::Production -> AuditMode::Shared (reuse fixtures across tests)

Config constructors renamed (with deprecated aliases):
- AuditConfig::ci() -> AuditConfig::isolated()
- AuditConfig::production() -> AuditConfig::shared()

CLI default changed from 'ci' to 'shared' mode, which enables
fixture caching across tests. This fixes the issue where fixtures
were being re-created for every test in CLI mode.

Fixture caching behavior:
- Shared mode (CLI default): Uses client's cache, fixtures reused
- Isolated mode (for cargo test): Local cache per TestContext
This commit is contained in:
DanConwayDev
2025-12-02 21:20:17 +00:00
parent 5c10ca0084
commit 72683beea0
12 changed files with 167 additions and 107 deletions
+36 -24
View File
@@ -4,8 +4,8 @@ A reusable audit and compliance testing tool for GRASP protocol implementations.
## Features
- ✅ **Isolated Testing**: Tests run in parallel with unique audit IDs
- ✅ **Production Audit**: Test live services with minimal impact
- ✅ **Shared Fixtures**: Fixtures cached and reused across tests (default for CLI)
- ✅ **Isolated Testing**: Fresh fixtures per test for parallel test isolation
- ✅ **Clean Audit Events**: Special tags for easy cleanup (no deletion trails)
- ✅ **Spec-Mirrored Tests**: Test structure matches GRASP protocol exactly
- ✅ **Reusable**: Can test any GRASP implementation (Rust, Go, Python, etc.)
@@ -41,8 +41,8 @@ use grasp_audit::*;
#[tokio::main]
async fn main() -> Result<()> {
// Create audit client for CI testing
let config = AuditConfig::ci();
// Create audit client with shared fixtures (default for CLI)
let config = AuditConfig::shared();
let client = AuditClient::new("ws://localhost:7000", config).await?;
// Run NIP-01 smoke tests
@@ -63,11 +63,11 @@ async fn main() -> Result<()> {
# Install
cargo install --path .
# Run smoke tests against local relay
grasp-audit audit --relay ws://localhost:7000 --mode ci --spec nip01-smoke
# Run smoke tests against local relay (shared fixtures - default)
grasp-audit audit --relay ws://localhost:7000 --spec nip01-smoke
# Audit production server
grasp-audit audit --relay wss://grasp.example.com --mode production --spec all
# Run with isolated fixtures (each test gets fresh fixtures)
grasp-audit audit --relay ws://localhost:7000 --mode isolated --spec all
```
## Test Specifications
@@ -110,11 +110,10 @@ All audit events automatically include special tags for isolation and cleanup:
- `["t", "grasp-audit-test-event"]` - Identifies all audit-related events
- `["t", "audit-{run_id}"]` - Unique identifier for each audit run
- CI mode: `audit-ci-{uuid}`
- Production mode: `audit-prod-audit-{timestamp}`
- Shared mode: `audit-audit-{uuid}`
- Isolated mode: `audit-isolated-{uuid}`
- `["t", "audit-cleanup-after-{unix_timestamp}"]` - Cleanup scheduling
- CI mode: Current time + 3600 seconds (1 hour)
- Production mode: Current time + 300 seconds (5 minutes)
- Default: Current time + 3600 seconds (1 hour)
**Benefits:**
@@ -124,27 +123,40 @@ All audit events automatically include special tags for isolation and cleanup:
- **No deletion trails**: No NIP-09 deletion events needed
- **Discovery**: Easy to query all audit events via hashtag
## Modes
## Fixture Modes
### CI Mode (Default)
The audit tool supports two fixture caching modes that control how test prerequisites are managed.
- Tests are isolated by unique run ID
- Tests only see their own events
- Full read/write access
- Cleanup after 1 hour
### Shared Mode (Default for CLI)
Fixtures are cached and reused across all tests. Use this when:
- Running the CLI audit tool sequentially
- Tests build on each other's fixtures
- You want efficient resource usage
```rust
let config = AuditConfig::ci();
let config = AuditConfig::shared();
```
### Production Mode
### Isolated Mode (For Parallel Tests)
- Tests see all events (including real ones)
- Read-only by default (minimal impact)
- Cleanup after 5 minutes
Each test creates fresh fixtures for complete isolation. Use this when:
- Running `cargo test` in parallel
- Tests must not interfere with each other
- Testing the fixture system itself
```rust
let config = AuditConfig::production();
let config = AuditConfig::isolated();
```
### CLI Usage
```bash
# Default: shared fixtures (efficient for sequential CLI runs)
grasp-audit audit --relay ws://localhost:7000 --spec all
# Isolated fixtures (for testing)
grasp-audit audit --relay ws://localhost:7000 --mode isolated --spec all
```
## Examples
+2 -2
View File
@@ -6,8 +6,8 @@ use grasp_audit::*;
#[tokio::main]
async fn main() -> Result<()> {
// Create audit config for CI testing
let config = AuditConfig::ci();
// Create audit config with shared fixtures (default for CLI)
let config = AuditConfig::shared();
println!("GRASP Audit Example");
println!("==================");
+64 -36
View File
@@ -18,46 +18,75 @@ pub struct AuditConfig {
pub read_only: bool,
}
/// Audit mode
/// Audit mode for fixture management
///
/// Controls how test fixtures are cached and shared between tests.
#[derive(Debug, Clone, Copy, PartialEq, Eq)]
pub enum AuditMode {
/// Isolated CI/CD tests - only see own events
CI,
/// Isolated mode - each test creates fresh fixtures
///
/// Use this mode when running tests in parallel (e.g., `cargo test`)
/// where each test needs complete isolation from other tests.
/// Each TestContext gets its own local cache.
Isolated,
/// Production audit - see all events, minimal writes
Production,
/// Shared mode - fixtures are cached and reused across tests
///
/// Use this mode when running the CLI audit tool where tests run
/// sequentially and build on each other's fixtures. This is more
/// efficient as it avoids re-creating the same prerequisite events.
/// All TestContexts share the client's cache.
Shared,
}
impl AuditConfig {
/// Create config for CI/CD testing
pub fn ci() -> Self {
let run_id = format!("ci-{}", &uuid::Uuid::new_v4().to_string()[..8]);
/// Create config for isolated testing (e.g., cargo test)
///
/// Each test creates fresh fixtures for complete test isolation.
/// Use this when running tests in parallel.
pub fn isolated() -> Self {
let run_id = format!("isolated-{}", &uuid::Uuid::new_v4().to_string()[..8]);
Self {
run_id,
mode: AuditMode::CI,
mode: AuditMode::Isolated,
cleanup_after: Timestamp::now() + 3600, // 1 hour from now
read_only: false,
}
}
/// Create config for production audit
pub fn production() -> Self {
let run_id = format!("prod-audit-{}", Timestamp::now().as_u64());
/// Create config for shared fixture mode (default for CLI)
///
/// Fixtures are cached and reused across tests. Use this when
/// running the CLI audit tool where tests run sequentially.
pub fn shared() -> Self {
let run_id = format!("audit-{}", &uuid::Uuid::new_v4().to_string()[..8]);
Self {
run_id,
mode: AuditMode::Production,
cleanup_after: Timestamp::now() + 300, // 5 minutes from now
read_only: true, // Default to read-only for production
mode: AuditMode::Shared,
cleanup_after: Timestamp::now() + 3600, // 1 hour from now
read_only: false,
}
}
/// Alias for isolated() - for backwards compatibility
#[deprecated(since = "0.2.0", note = "Use isolated() instead")]
pub fn ci() -> Self {
Self::isolated()
}
/// Alias for shared() - for backwards compatibility
#[deprecated(since = "0.2.0", note = "Use shared() instead")]
pub fn production() -> Self {
Self::shared()
}
/// Create config with custom run ID
pub fn with_run_id(run_id: String, mode: AuditMode) -> Self {
Self {
run_id,
mode,
cleanup_after: Timestamp::now() + 3600,
read_only: mode == AuditMode::Production,
read_only: false,
}
}
@@ -72,11 +101,10 @@ impl AuditConfig {
///
/// 1. `["t", "grasp-audit-test-event"]` - Identifies all audit-related events
/// 2. `["t", "audit-{run_id}"]` - Unique identifier for this audit run
/// - CI mode: `audit-ci-{uuid}`
/// - Production mode: `audit-prod-audit-{timestamp}`
/// - Isolated mode: `audit-isolated-{uuid}`
/// - Shared mode: `audit-audit-{uuid}`
/// 3. `["t", "audit-cleanup-after-{unix_timestamp}"]` - Cleanup timestamp
/// - CI mode: Current time + 3600 seconds (1 hour)
/// - Production mode: Current time + 300 seconds (5 minutes)
/// - Default: Current time + 3600 seconds (1 hour)
///
/// # Purpose
///
@@ -90,7 +118,7 @@ impl AuditConfig {
/// ```rust
/// use grasp_audit::AuditConfig;
///
/// let config = AuditConfig::ci();
/// let config = AuditConfig::isolated();
/// let tags = config.audit_tags();
///
/// // Tags will look like:
@@ -168,7 +196,7 @@ impl AuditEventBuilder {
/// use nostr_sdk::prelude::*;
/// use grasp_audit::{AuditConfig, AuditEventBuilder};
///
/// let config = AuditConfig::ci();
/// let config = AuditConfig::isolated();
/// let keys = Keys::generate();
///
/// // Create an event with a past timestamp
@@ -209,26 +237,26 @@ mod tests {
use super::*;
#[test]
fn test_ci_config() {
let config = AuditConfig::ci();
assert_eq!(config.mode, AuditMode::CI);
fn test_isolated_config() {
let config = AuditConfig::isolated();
assert_eq!(config.mode, AuditMode::Isolated);
assert!(!config.read_only);
assert!(config.run_id.starts_with("ci-"));
assert!(config.run_id.starts_with("isolated-"));
}
#[test]
fn test_production_config() {
let config = AuditConfig::production();
assert_eq!(config.mode, AuditMode::Production);
assert!(config.read_only);
assert!(config.run_id.starts_with("prod-audit-"));
fn test_shared_config() {
let config = AuditConfig::shared();
assert_eq!(config.mode, AuditMode::Shared);
assert!(!config.read_only);
assert!(config.run_id.starts_with("audit-"));
}
#[test]
fn test_audit_tags() {
use nostr_sdk::prelude::{Alphabet, SingleLetterTag};
let config = AuditConfig::ci();
let config = AuditConfig::isolated();
let tags = config.audit_tags();
assert_eq!(tags.len(), 3);
@@ -252,7 +280,7 @@ mod tests {
// Check for "t" tag with "audit-{run_id}"
assert!(tags.iter().any(|t| {
t.content()
.map(|c| c.starts_with("audit-ci-"))
.map(|c| c.starts_with("audit-isolated-"))
.unwrap_or(false)
}));
@@ -266,7 +294,7 @@ mod tests {
#[test]
fn test_audit_event_builder() {
let config = AuditConfig::ci();
let config = AuditConfig::isolated();
let keys = Keys::generate();
let event = AuditEventBuilder::new(Kind::TextNote, "test", config.clone())
@@ -283,7 +311,7 @@ mod tests {
#[test]
fn test_custom_timestamp_applied() {
let config = AuditConfig::ci();
let config = AuditConfig::isolated();
let keys = Keys::generate();
let custom_ts = Timestamp::from(1700000000);
@@ -302,7 +330,7 @@ mod tests {
#[test]
fn test_default_timestamp_uses_current_time() {
let config = AuditConfig::ci();
let config = AuditConfig::isolated();
let keys = Keys::generate();
let before = Timestamp::now();
+10 -3
View File
@@ -20,8 +20,11 @@ enum Commands {
#[arg(short, long)]
relay: String,
/// Mode: ci or production
#[arg(short, long, default_value = "ci")]
/// Fixture mode: shared (default) or isolated
///
/// - shared: Fixtures are cached and reused across tests (efficient for sequential test runs)
/// - isolated: Each test creates fresh fixtures (for parallel tests like cargo test)
#[arg(short, long, default_value = "shared")]
mode: String,
/// Spec to test (nip01-smoke, nip11, event-acceptance, cors, git-clone, push-auth, repo-creation, all)
@@ -53,10 +56,14 @@ async fn main() -> Result<()> {
spec,
git_data_dir,
} => {
#[allow(deprecated)]
let mut config = match mode.as_str() {
"shared" => AuditConfig::shared(),
"isolated" => AuditConfig::isolated(),
// Backwards compatibility aliases
"ci" => AuditConfig::ci(),
"production" => AuditConfig::production(),
_ => return Err(anyhow!("Invalid mode: {}. Use 'ci' or 'production'", mode)),
_ => return Err(anyhow!("Invalid mode: {}. Use 'shared' or 'isolated'", mode)),
};
// Audit needs to create events to test the relay, so disable read-only mode
+10 -10
View File
@@ -199,7 +199,7 @@ impl AuditClient {
/// ```no_run
/// # use grasp_audit::*;
/// # async fn example() -> anyhow::Result<()> {
/// let config = AuditConfig::ci();
/// let config = AuditConfig::shared();
/// let client = AuditClient::new("ws://localhost:7000", config).await?;
///
/// // Create event with automatic audit tags
@@ -221,8 +221,8 @@ impl AuditClient {
pub async fn query(&self, mut filter: Filter) -> Result<Vec<Event>> {
use nostr_sdk::prelude::{Alphabet, SingleLetterTag};
if self.config.mode == AuditMode::CI {
// In CI mode, only see our own audit events
if self.config.mode == AuditMode::Isolated {
// In Isolated mode, only see our own audit events
// Filter by "t" tags (hashtags)
let t_tag = SingleLetterTag::lowercase(Alphabet::T);
filter = filter
@@ -505,7 +505,7 @@ mod tests {
#[tokio::test]
async fn test_client_creation() {
let config = AuditConfig::ci();
let config = AuditConfig::isolated();
// This will fail if no relay is running, which is expected in tests
// In real usage, there should be a relay at the URL
@@ -514,13 +514,13 @@ mod tests {
// We can't test connection without a running relay
// But we can test that the client is created
if let Ok(client) = result {
assert_eq!(client.config.mode, AuditMode::CI);
assert_eq!(client.config.mode, AuditMode::Isolated);
}
}
#[test]
fn test_event_builder() {
let config = AuditConfig::ci();
let config = AuditConfig::isolated();
let keys = Keys::generate();
let maintainer_keys = Keys::generate();
let recursive_maintainer_keys = Keys::generate();
@@ -543,7 +543,7 @@ mod tests {
#[test]
fn test_audit_tags_automatically_added() {
let config = AuditConfig::ci();
let config = AuditConfig::isolated();
let keys = Keys::generate();
let maintainer_keys = Keys::generate();
let recursive_maintainer_keys = Keys::generate();
@@ -585,8 +585,8 @@ mod tests {
"Missing 'grasp-audit-test-event' tag"
);
assert!(
tag_contents.iter().any(|t| t.starts_with("audit-ci-")),
"Missing 'audit-ci-*' tag"
tag_contents.iter().any(|t| t.starts_with("audit-isolated-")),
"Missing 'audit-isolated-*' tag"
);
assert!(
tag_contents
@@ -604,7 +604,7 @@ mod tests {
#[tokio::test]
async fn test_create_repo_announcement_with_maintainers() {
let config = AuditConfig::ci();
let config = AuditConfig::isolated();
let client = AuditClient::new_test(config);
// Create test maintainer pubkeys (hex format)
+38 -25
View File
@@ -3,19 +3,25 @@
//! This module provides a TestContext abstraction that manages prerequisite events
//! differently based on the audit mode:
//!
//! - **CI Mode (Isolated)**: Creates fresh events for each test, ensuring complete isolation
//! - **Production Mode (Shared)**: Reuses shared fixtures to minimize event publication
//! - **Isolated Mode**: Creates fresh events for each test, ensuring complete isolation.
//! Use this for `cargo test` where tests run in parallel and need isolation.
//! - **Shared Mode**: Reuses shared fixtures across tests to minimize event publication.
//! Use this for CLI audit where tests run sequentially and build on each other.
//!
//! # Cache Sharing Strategy
//!
//! The fixture cache lives on the `AuditClient`, not on `TestContext`. This provides
//! natural cache sharing semantics:
//! The caching behavior depends on the mode:
//!
//! - **CLI mode**: Creates one `AuditClient` → fixtures shared across all tests
//! - **cargo test**: Creates one `AuditClient` per test → fixtures isolated per test
//! - **Shared mode** (default for CLI): Uses the client's fixture cache, shared across
//! all TestContext instances. Fixtures are created once and reused.
//! - **Isolated mode**: Each TestContext has its own local cache. Fixtures are created
//! fresh for each TestContext, providing complete test isolation.
//!
//! This eliminates the need for global state while still enabling fixture reuse
//! when appropriate.
//! # When to Use Each Mode
//!
//! - **CLI audit tool**: Use Shared mode (default). Tests run sequentially and fixtures
//! build on each other efficiently.
//! - **cargo test**: Use Isolated mode. Tests run in parallel and need complete isolation.
//!
//! # What is a Fixture?
//! A fixture represents the state of a repository on a grasp server (events and git refs)
@@ -36,7 +42,7 @@
//! use grasp_audit::*;
//!
//! # async fn example() -> anyhow::Result<()> {
//! let config = AuditConfig::ci();
//! let config = AuditConfig::shared(); // Use shared() for CLI, isolated() for cargo test
//! let client = AuditClient::new("ws://localhost:7000", config).await?;
//! let ctx = TestContext::new(&client);
//!
@@ -341,8 +347,8 @@ pub enum ContextMode {
impl From<AuditMode> for ContextMode {
fn from(mode: AuditMode) -> Self {
match mode {
AuditMode::CI => ContextMode::Isolated,
AuditMode::Production => ContextMode::Shared,
AuditMode::Isolated => ContextMode::Isolated,
AuditMode::Shared => ContextMode::Shared,
}
}
}
@@ -350,30 +356,37 @@ impl From<AuditMode> for ContextMode {
/// Test context for managing prerequisite events
///
/// The TestContext provides mode-aware fixture management:
/// - In Isolated mode: Creates fresh events for each test
/// - In Shared mode: Caches and reuses events across tests
/// - In **Isolated mode**: Each TestContext has its own local cache, creating fresh
/// fixtures for each test. Use this for `cargo test` where tests run in parallel.
/// - In **Shared mode**: Uses the client's fixture cache, shared across all TestContexts.
/// Use this for CLI audit where tests run sequentially and build on each other.
///
/// # Cache Location
/// # Mode Selection
///
/// The fixture cache lives on `AuditClient`, not on `TestContext`. This means:
/// - Multiple `TestContext` instances from the same client share the cache
/// - CLI mode (one client) naturally shares fixtures across all tests
/// - Test mode (one client per test) naturally isolates fixtures
/// The mode is determined by `AuditConfig::mode`:
/// - `AuditConfig::isolated()` → Creates fresh fixtures per TestContext
/// - `AuditConfig::shared()` → Reuses fixtures across all TestContexts (default for CLI)
///
/// # Example
///
/// ```no_run
/// # use grasp_audit::*;
/// # async fn example() -> anyhow::Result<()> {
/// let config = AuditConfig::ci();
/// // For CLI audit (shared fixtures - default)
/// let config = AuditConfig::shared();
/// let client = AuditClient::new("ws://localhost:7000", config).await?;
/// let ctx = TestContext::new(&client);
///
/// // Get a repository fixture
/// // Get a repository fixture - will be reused by subsequent TestContexts
/// let repo = ctx.get_fixture(FixtureKind::ValidRepo).await?;
///
/// // In CI mode: Creates new repo
/// // In Production mode: Returns cached repo
/// // For cargo test (isolated fixtures)
/// let config = AuditConfig::isolated();
/// let client = AuditClient::new("ws://localhost:7000", config).await?;
/// let ctx = TestContext::new(&client);
///
/// // Get a repository fixture - fresh for this TestContext only
/// let repo = ctx.get_fixture(FixtureKind::ValidRepo).await?;
/// # Ok(())
/// # }
/// ```
@@ -1960,9 +1973,9 @@ mod tests {
#[test]
fn test_context_mode_from_audit_mode() {
assert_eq!(ContextMode::from(AuditMode::CI), ContextMode::Isolated);
assert_eq!(ContextMode::from(AuditMode::Isolated), ContextMode::Isolated);
assert_eq!(
ContextMode::from(AuditMode::Production),
ContextMode::from(AuditMode::Shared),
ContextMode::Shared
);
}
@@ -1981,7 +1994,7 @@ mod tests {
#[tokio::test]
async fn test_context_creation() {
let config = AuditConfig::ci();
let config = AuditConfig::isolated();
let client = crate::AuditClient::new_test(config);
let ctx = TestContext::new(&client);
+2 -2
View File
@@ -16,8 +16,8 @@
//!
//! #[tokio::main]
//! async fn main() -> Result<(), Box<dyn std::error::Error>> {
//! // Create audit client for CI testing
//! let config = AuditConfig::ci();
//! // Create audit client with shared fixtures (default for CLI)
//! let config = AuditConfig::shared();
//! let client = AuditClient::new("ws://localhost:7000", config).await?;
//!
//! // Run smoke tests
+1 -1
View File
@@ -459,7 +459,7 @@ mod tests {
.trim_end_matches('/')
.to_string();
let config = AuditConfig::ci();
let config = AuditConfig::isolated();
let client = AuditClient::new(&relay_url, config)
.await
.unwrap_or_else(|_| {
@@ -1139,7 +1139,7 @@ mod tests {
"RELAY_URL environment variable must be set. Example: RELAY_URL=ws://localhost:18081",
);
let config = AuditConfig::ci();
let config = AuditConfig::isolated();
let client = AuditClient::new(&relay_url, config)
.await
.unwrap_or_else(|_| {
+1 -1
View File
@@ -297,7 +297,7 @@ mod tests {
let relay_url = std::env::var("RELAY_URL")
.expect("RELAY_URL environment variable must be set for integration tests");
let config = AuditConfig::ci();
let config = AuditConfig::isolated();
let client = AuditClient::new(&relay_url, config)
.await
.expect("Failed to connect to relay");
@@ -288,7 +288,7 @@ mod tests {
"RELAY_URL environment variable must be set. Example: RELAY_URL=ws://localhost:18081",
);
let config = AuditConfig::ci();
let config = AuditConfig::isolated();
let client = AuditClient::new(&relay_url, config)
.await
.unwrap_or_else(|_| {
@@ -206,4 +206,4 @@ mod tests {
fn test_requirement_count() {
assert_eq!(GRASP_01_REQUIREMENTS.len(), 19);
}
}
}