ngit-grasp/docs/how-to/production-sync-testing.md

How-To: Test Sync Against Production Data

Quick Start Prompt: Run a 30-second production sync test following docs/how-to/production-sync-testing.md. Use the minimal test command with sanitized output. Analyze the log for errors, warnings, and unexpected patterns. Document findings as individual markdown files in work/active-issues/ and suggest code fixes or logging improvements.

Problem: Debug and improve sync behavior using real-world data from production relays
Difficulty: Intermediate
Time: 30 minutes per iteration

Overview

This guide helps you run ngit-grasp's sync system against production relays to discover unexpected errors, inefficiencies, and edge cases that don't appear in controlled tests.

Why production testing matters:

• Real data has inconsistencies, malformed events, and edge cases
• Production relays may behave differently (rate limiting, timeouts, partial NIP-77 support)
• Volume and patterns reveal performance bottlenecks
• Sync discovery leads to cascading subscriptions we can't predict in tests

Prerequisites

• ngit-grasp compiles successfully (cargo build)
• Familiarity with GRASP-02 Proactive Sync
• Understanding of log levels and tracing

Test Setup

1. Choose a Test Identity

Pick a domain with manageable sync volume. Smaller domains mean fewer repos to sync, making logs tractable.

Recommended starting point:

--domain ngit.danconwaydev.com

This domain has few repo announcements listing it, so sync stays manageable.

2. Choose a Bootstrap Relay

The bootstrap relay provides the initial set of announcements to discover repos:

--sync-bootstrap-relay-url wss://git.shakespeare.diy

3. Run with Time Limit

Start with short runs (30 seconds) to capture manageable log volumes:

# Clear any existing data for clean state
rm -rf /tmp/ngit-test-*

# Run for 30 seconds with sanitized output
timeout 30s cargo run -- \
    --sync-bootstrap-relay-url wss://git.shakespeare.diy \
    --domain ngit.danconwaydev.com \
    --git-data-path /tmp/ngit-test-git \
    --relay-data-path /tmp/ngit-test-relay \
    2>&1 | ./scripts/sanitize-logs.sh | tee sync-test.log

Note: The timeout command returns exit code 124, which is expected.

Log Sanitization

Raw logs include full events and hundreds of event IDs per line, making them unwieldy for analysis. The sanitizer truncates long lines:

./scripts/sanitize-logs.sh < raw.log > sanitized.log

# Or pipe directly
cargo run -- [args] 2>&1 | ./scripts/sanitize-logs.sh

Options:

• --head-chars N - First N characters to show (default: 200)
• --tail-chars N - Last N characters to show (default: 100)

Example output:

2024-01-09T10:00:00Z DEBUG sync: Processing events ids=[abc123, def456, ghi789, jkl012...<1847 chars>...xyz999, end123]

What to Look For

Phase 1: Connection & Bootstrap (0-5 seconds)

Expected behavior:

• Connection to bootstrap relay succeeds
• Layer 1 (announcement) subscription starts
• First batch of 30617/30618 events received

Red flags:

• Connection timeout or failure
• NIP-77 negentropy errors (should fall back gracefully)
• Immediate rate limiting

Phase 2: Discovery Cascade (5-15 seconds)

Expected behavior:

• Self-subscriber batches fire as announcements are processed
• New relays discovered from announcement relays tags
• Layer 2 (repo tags) subscriptions created

Red flags:

• Excessive relay discovery (>10 relays rapidly)
• Filter consolidation warnings (>70 filters)
• Missing self-subscriber batch logs

Phase 3: Steady State (15+ seconds)

Expected behavior:

• Historic sync batches completing (EOSE received)
• Periodic health checks running
• Events being saved to database

Red flags:

• Pending batches never confirming
• Repeated connection/disconnect cycles
• Memory growth (check with top in another terminal)

Debugging Checklist

When analyzing logs, look for these patterns:

Errors to Investigate

Pattern	Possible Cause	Action
error (any)	Unexpected failure	Investigate immediately
connection failed	Network/relay issue	Check relay URL, try different relay
rate limit	Too many requests	Check consolidation, increase backoff
negentropy + error	NIP-77 incompatibility	Should fall back - verify it does
timeout	Slow relay or large sync	Increase timeouts or reduce scope

Warnings to Monitor

Pattern	Meaning	Action
consolidating filters	Filter count high	Expected, but frequent = problem
backing off	Health tracker retry	Normal, but watch for excessive
batch failed	Historic sync incomplete	Check which batches, why

Debug Patterns to Verify

Pattern	What it shows
fresh_start	Full sync initiated
quick_reconnect	Incremental sync (<15min gap)
historic sync complete	Sync finished successfully
sync_live	Live subscriptions active
PendingBatch	Items awaiting EOSE confirmation

Iterative Improvement Process

Step 1: Run and Capture

timeout 30s cargo run -- [args] 2>&1 | ./scripts/sanitize-logs.sh > iteration-1.log

Step 2: Identify Issues

Scan logs for errors and unexpected patterns:

grep -i error iteration-1.log
grep -i warn iteration-1.log
grep -i panic iteration-1.log

Step 3: Document Findings

Create individual markdown files in work/active-issues/ for each issue discovered:

# Example: Document a connection timeout issue
cat > work/active-issues/connection-timeout-bootstrap.md <<'EOF'
# Issue: Connection Timeout on Bootstrap Relay

**Discovered:** 2026-01-09
**Status:** Open

## Symptoms

- Connection to wss://git.shakespeare.diy fails after 10s timeout
- Log shows: `error: connection failed: timeout`
- Occurs 100% of time with this relay

## Root Cause

[To be determined]

## Proposed Fix

- Increase connection timeout from 10s to 30s for initial bootstrap
- Add retry logic with exponential backoff
- Consider fallback bootstrap relays

## Code Location

- `src/sync/relay_connection.rs:45` - connection timeout constant
EOF

Why individual files?

• Keeps the how-to guide clean and focused
• Prevents accidental commits of transient issues to tracked files
• Easy to delete resolved issues or archive important ones
• Each file can be worked on independently

Step 4: Fix and Re-test

After code changes, run again to verify the fix.

Step 5: Extend Duration

Once 30-second runs are clean, extend to 2 minutes, then 5 minutes:

timeout 120s cargo run -- [args] 2>&1 | ./scripts/sanitize-logs.sh > iteration-2.log

Logging Improvements

If the logs aren't helpful enough, improve them. Common needs:

Add Context to Existing Logs

// Before
tracing::debug!("Processing events");

// After
tracing::debug!(
    relay = %relay_url,
    event_count = events.len(),
    "Processing events"
);

Add New Log Points

Key places that may need more logging:

• src/sync/mod.rs - SyncManager state transitions
• src/sync/relay_connection.rs - Connection lifecycle
• src/sync/self_subscriber.rs - Batch processing

Reduce Noise

If a log line appears too frequently:

// Change from debug! to trace!
tracing::trace!("Per-event detail that's too noisy");

Active Issues

Issues discovered during production sync testing are tracked in work/active-issues/ as individual markdown files.

View current issues:

ls work/active-issues/

Create a new issue:

# Use kebab-case filename describing the issue
cat > work/active-issues/[issue-name].md <<'EOF'
# Issue: [Short Description]

**Discovered:** [Date]
**Status:** Open

## Symptoms

- Log patterns observed
- Reproduction steps if known

## Root Cause

[To be determined / Known cause]

## Proposed Fix

- Suggested code changes
- Alternative approaches

## Code Location

- File paths and line numbers where changes are needed
EOF

Resolve an issue:

# After fixing, either delete or move to archive
rm work/active-issues/resolved-issue.md
# OR
mv work/active-issues/important-issue.md docs/archive/2026-01-09-important-issue.md

---

Quick Reference

Minimal Test Command

timeout 30s cargo run -- \
    --sync-bootstrap-relay-url wss://git.shakespeare.diy \
    --domain ngit.danconwaydev.com \
    --git-data-path /tmp/ngit-test-git \
    --relay-data-path /tmp/ngit-test-relay \
    2>&1 | ./scripts/sanitize-logs.sh

With Metrics Endpoint

timeout 30s cargo run -- \
    --sync-bootstrap-relay-url wss://git.shakespeare.diy \
    --domain ngit.danconwaydev.com \
    --git-data-path /tmp/ngit-test-git \
    --relay-data-path /tmp/ngit-test-relay \
    --metrics-address 127.0.0.1:9090 \
    2>&1 | ./scripts/sanitize-logs.sh

Then in another terminal: curl http://127.0.0.1:9090/metrics

Different Log Level

The default is DEBUG. For more detail:

RUST_LOG=trace cargo run -- [args]

For less noise:

RUST_LOG=info cargo run -- [args]

---

Part of the ngit-grasp documentation using the Diátaxis framework.