ngit-grasp/docs/how-to/production-sync-testing.md

# How-To: Test Sync Against Production Data

> **Quick Start Prompt:** Check work/active-issues/ for existing issues. If issues exist, pick the most important, fix it, test with cargo test, run clippy and fmt, commit, and report back with a brief 1-2 sentence summary of each issue you identified. If no issues exist, run a 60-second production sync test, analyze logs, create individual issue files in work/active-issues/ (one per issue with minimal description), then report summary listing each issue in 1-2 sentences.

**Problem:** Debug and improve sync behavior using real-world data from production relays  
**Difficulty:** Intermediate  
**Time:** 30 minutes per iteration

## Two-Mode Workflow

This guide operates in two modes:

### Mode 1: Fix Existing Issues
**When:** There are files in `work/active-issues/` (excluding README.md)

1. Check for active issues: `ls work/active-issues/`
2. Pick the most important issue to fix
3. **Review proposed fix and ask for permission before implementing**
4. Implement the fix (after approval)
5. Run `cargo test` to verify tests pass
6. Run `cargo clippy` to check for warnings
7. Run `cargo fmt` to format code
8. Commit changes with descriptive message
9. Report back - **DO NOT** do another issue or run more tests

### Mode 2: Discover New Issues
**When:** No active issues in `work/active-issues/`

1. Run 60-second production sync test (logs saved to `tmp/run-{timestamp}/`)
2. Analyze logs for errors, warnings, unexpected patterns
3. Document each issue as a separate markdown file in `work/active-issues/`
4. Keep issue files minimal - just enough to identify the issue
5. Report brief summary listing each issue in 1-2 sentences
6. **DO NOT** create separate detailed analysis files
7. **DO NOT** do thorough investigation or root cause analysis

## Overview

This guide helps you run ngit-grasp's sync system against production relays to discover unexpected errors, inefficiencies, and edge cases that don't appear in controlled tests.

**Why production testing matters:**
- Real data has inconsistencies, malformed events, and edge cases
- Production relays may behave differently (rate limiting, timeouts, partial NIP-77 support)
- Volume and patterns reveal performance bottlenecks
- Sync discovery leads to cascading subscriptions we can't predict in tests

## Prerequisites

- ngit-grasp compiles successfully (`cargo build`)
- Familiarity with [GRASP-02 Proactive Sync](../explanation/grasp-02-proactive-sync.md)
- Understanding of log levels and tracing

## Test Setup

### 1. Choose a Test Identity

Pick a domain with manageable sync volume. Smaller domains mean fewer repos to sync, making logs tractable.

**Recommended starting point:**
```bash
--domain ngit.danconwaydev.com
```

This domain has few repo announcements listing it, so sync stays manageable.

### 2. Choose a Bootstrap Relay

The bootstrap relay provides the initial set of announcements to discover repos:

```bash
--sync-bootstrap-relay-url wss://git.shakespeare.diy
```

### 3. Run with Time Limit

Run for **60 seconds** to allow the full sync cascade to complete. This duration allows:
- Layer 1: Discovery of repo announcements (0-5s)
- Layer 2: Sending `#a`, `#A`, `#q` filters for repos (5-30s)
- Layer 3: Receiving issues, patches, PRs (30-60s)
- Layer 4: Receiving comments on root events (40-60s)

Each run creates its own subdirectory in `tmp/` to keep data and logs isolated:

```bash
# Create run directory with timestamp
RUN_DIR="tmp/run-$(date +%Y%m%d-%H%M%S)"
mkdir -p "$RUN_DIR"

# Run for 60 seconds, saving both raw and sanitized logs
timeout 60s cargo run -- \
    --sync-bootstrap-relay-url wss://git.shakespeare.diy \
    --domain ngit.danconwaydev.com \
    --git-data-path "$RUN_DIR/git-data" \
    --relay-data-path "$RUN_DIR/relay-data" \
    2>&1 | tee "$RUN_DIR/sync-raw.log" | ./scripts/sanitize-logs.sh | tee "$RUN_DIR/sync.log"
```

**Why 60 seconds?** The sync system uses a 5-second batch window for aggregating discovered repos. Repos discovered late in the sync need time for:
1. Batch window to expire (5s)
2. Layer 2 filters to be sent and processed
3. Layer 3 events (issues/patches/PRs) to be returned
4. Layer 4 filters for root events to be sent
5. Comments and threaded replies to be returned

Testing shows 30 seconds is too short for late-discovered repos to complete the Layer 2→3→4 cascade.

**Note:** The `timeout` command returns exit code 124, which is expected.

**Directory structure after run:**
```
tmp/
└── run-20260109-143022/
    ├── git-data/       # Git repository data
    ├── relay-data/     # Relay database
    ├── sync.log        # Sanitized log (for quick analysis)
    └── sync-raw.log    # Raw log (for full details when needed)
```

**When to use which log:**
- **sync.log** - Use for quick scanning and pattern recognition (long lines truncated)
- **sync-raw.log** - Use when you need full details (e.g., complete rejection reasons, full event data)

## Log Sanitization

Raw logs include full events and hundreds of event IDs per line, making them unwieldy for analysis. The sanitizer truncates long lines:

```bash
./scripts/sanitize-logs.sh < raw.log > sanitized.log

# Or pipe directly
cargo run -- [args] 2>&1 | ./scripts/sanitize-logs.sh
```

**Options:**
- `--head-chars N` - First N characters to show (default: 200)
- `--tail-chars N` - Last N characters to show (default: 100)

Example output:
```
2024-01-09T10:00:00Z DEBUG sync: Processing events ids=[abc123, def456, ghi789, jkl012...<1847 chars>...xyz999, end123]
```

### Retrieving Full Details from Raw Logs

When sanitized logs show truncated messages (e.g., rejection reasons), use the raw log to see the complete content:

```bash
# Find specific error in raw log
grep "Rejected repository announcement" "$RUN_DIR/sync-raw.log"

# Extract full line for specific event ID
grep "note1z5ys7wf3ms5yxhnp3kfw7hpu5asfkx4jngzt5zgs4tm4tnvggnsqjfqeyt" "$RUN_DIR/sync-raw.log"

# View context around a truncated warning
grep -A 2 -B 2 "pattern from sanitized log" "$RUN_DIR/sync-raw.log"
```

The raw log contains complete, untruncated messages including full rejection reasons, event data, and debug details.

## What to Look For

### Phase 1: Connection & Bootstrap (0-5 seconds)

**Expected behavior:**
- Connection to bootstrap relay succeeds
- Layer 1 (announcement) subscription starts
- First batch of 30617/30618 events received

**Red flags:**
- Connection timeout or failure
- NIP-77 negentropy errors (should fall back gracefully)
- Immediate rate limiting

### Phase 2: Discovery Cascade (5-15 seconds)

**Expected behavior:**
- Self-subscriber batches fire as announcements are processed
- New relays discovered from announcement `relays` tags
- Layer 2 (repo tags) subscriptions created

**Red flags:**
- Excessive relay discovery (>10 relays rapidly)
- Filter consolidation warnings (>70 filters)
- Missing self-subscriber batch logs

### Phase 3: Steady State (15+ seconds)

**Expected behavior:**
- Historic sync batches completing (EOSE received)
- Periodic health checks running
- Events being saved to database

**Red flags:**
- Pending batches never confirming
- Repeated connection/disconnect cycles
- Memory growth (check with `top` in another terminal)

## Debugging Checklist

When analyzing logs, look for these patterns:

### Errors to Investigate

| Pattern | Possible Cause | Action |
|---------|----------------|--------|
| `error` (any) | Unexpected failure | Investigate immediately |
| `connection failed` | Network/relay issue | Check relay URL, try different relay |
| `rate limit` | Too many requests | Check consolidation, increase backoff |
| `negentropy` + `error` | NIP-77 incompatibility | Should fall back - verify it does |
| `timeout` | Slow relay or large sync | Increase timeouts or reduce scope |

### Warnings to Monitor

| Pattern | Meaning | Action |
|---------|---------|--------|
| `consolidating filters` | Filter count high | Expected, but frequent = problem |
| `backing off` | Health tracker retry | Normal, but watch for excessive |
| `batch failed` | Historic sync incomplete | Check which batches, why |

### Debug Patterns to Verify

| Pattern | What it shows |
|---------|---------------|
| `fresh_start` | Full sync initiated |
| `quick_reconnect` | Incremental sync (<15min gap) |
| `historic sync complete` | Sync finished successfully |
| `sync_live` | Live subscriptions active |
| `PendingBatch` | Items awaiting EOSE confirmation |

## Mode 1: Fix Existing Issues (Detailed)

When `work/active-issues/` contains issue files:

### Step 1: Check for Active Issues

```bash
ls work/active-issues/
```

If any `.md` files exist (excluding README.md), you're in Mode 1.

### Step 2: Pick Most Important Issue

Review issue files and select based on:
- Severity (errors > warnings > log quality)
- Impact (functionality > performance > UX)
- Complexity (quick fixes first to clear backlog)

### Step 3: Review Proposed Fix and Get Permission

**IMPORTANT:** Before implementing any changes:

1. Read relevant code files to understand the issue
2. Analyze the root cause
3. Propose a fix with explanation of what will change and why
4. Summarize the proposed fix in 2-3 sentences
5. **Ask for user permission to proceed**

**Do NOT implement changes without explicit approval.**

### Step 4: Implement the Fix

After receiving permission, make the necessary code changes based on the issue description and approved plan.

### Step 5: Test, Lint, Format

```bash
# Run tests
cargo test

# Check for warnings
cargo clippy

# Format code
cargo fmt
```

### Step 6: Commit

```bash
git add .
git commit -m "fix: [brief description of what was fixed]"
```

### Step 7: Report Back

**STOP HERE.** Report what was fixed. Do NOT:
- Fix another issue
- Run production sync test
- Do additional investigation

The workflow will cycle back through Mode 1 if more issues remain.

## Mode 2: Discover New Issues (Detailed)

When `work/active-issues/` is empty (or only contains README.md):

### Step 1: Run Production Sync Test

```bash
# Create run directory with timestamp
RUN_DIR="tmp/run-$(date +%Y%m%d-%H%M%S)"
mkdir -p "$RUN_DIR"

# Run 60-second test, saving both raw and sanitized logs
timeout 60s cargo run -- \
    --sync-bootstrap-relay-url wss://git.shakespeare.diy \
    --domain ngit.danconwaydev.com \
    --git-data-path "$RUN_DIR/git-data" \
    --relay-data-path "$RUN_DIR/relay-data" \
    2>&1 | tee "$RUN_DIR/sync-raw.log" | ./scripts/sanitize-logs.sh | tee "$RUN_DIR/sync.log"
```

Each run is isolated in its own timestamped directory under `tmp/`, keeping data and logs organized. Both raw and sanitized logs are saved for flexible analysis.

**Note:** 60 seconds allows the full sync cascade (Layer 1→2→3→4) to complete for late-discovered repos.

### Step 2: Analyze Logs

Scan for errors and unexpected patterns:
```bash
# Find the most recent run
LATEST_RUN=$(ls -1t tmp/run-*/sync.log | head -n1)
LATEST_RAW=$(ls -1t tmp/run-*/sync-raw.log | head -n1)

# Analyze sanitized log for quick scanning
grep -i error "$LATEST_RUN"
grep -i warn "$LATEST_RUN"
grep -i panic "$LATEST_RUN"

# If you find truncated messages, check the raw log for full details
grep "pattern from truncated message" "$LATEST_RAW"
```

### Step 3: Document Issues

Create **one markdown file per issue** in `work/active-issues/`:

```bash
# Example: Minimal issue documentation
cat > work/active-issues/bootstrap-disconnect.md <<'EOF'
# Bootstrap relay disconnects when empty

Bootstrap relay wss://git.shakespeare.diy disconnects after sync finds 0 events. Should persist since user-specified.

Log: "Disconnecting empty relay relay=wss://git.shakespeare.diy"
File: src/sync/mod.rs (check_disconnects function)
EOF
```

**Keep each file brief:**
- Descriptive title (one line)
- What happens (1-2 sentences max)
- Relevant log excerpt (one line)
- File/function location if obvious (one line)
- **NO** separate detailed analysis files
- **NO** root cause analysis
- **NO** proposed solutions (unless immediately obvious)

### Step 4: Report Summary

Provide a brief closing message with 1-2 sentence summary of **each issue** identified:
- State what the issue is
- Where it occurs (file/component)
- Keep it concise

**STOP HERE.** Do NOT:
- Fix the issues immediately
- Create separate detailed analysis markdown files
- Do thorough investigations
- Write lengthy explanations

The workflow will cycle back through Mode 1 to fix issues one at a time.

## Logging Improvements

If the logs aren't helpful enough, improve them. Common needs:

### Add Context to Existing Logs

```rust
// Before
tracing::debug!("Processing events");

// After
tracing::debug!(
    relay = %relay_url,
    event_count = events.len(),
    "Processing events"
);
```

### Add New Log Points

Key places that may need more logging:
- `src/sync/mod.rs` - SyncManager state transitions
- `src/sync/relay_connection.rs` - Connection lifecycle
- `src/sync/self_subscriber.rs` - Batch processing

### Reduce Noise

If a log line appears too frequently:
```rust
// Change from debug! to trace!
tracing::trace!("Per-event detail that's too noisy");
```

## Managing Active Issues

Issues are tracked in `work/active-issues/` as individual markdown files.

**Check for active issues:**
```bash
ls work/active-issues/
```

**After fixing an issue:**
```bash
# Delete the resolved issue file
rm work/active-issues/issue-name.md

# Or archive if important for future reference
mv work/active-issues/issue-name.md docs/archive/2026-01-09-issue-name.md
```

**Issue file format (minimal):**
```markdown
# Brief title

What happens (1-2 sentences).

Log evidence: "relevant log line"
File: src/path/to/file.rs (function_name if known)
```

Keep documentation minimal - just enough to identify and locate the issue.

---

## Workflow Summary

```
Check work/active-issues/
    │
    ├─ Has issues? ──► Mode 1: Pick one issue
    │                          │
    │                          ├─ Review & propose fix
    │                          ├─ Ask permission
    │                          ├─ Fix code (after approval)
    │                          ├─ cargo test
    │                          ├─ cargo clippy
    │                          ├─ cargo fmt
    │                          ├─ git commit
    │                          └─ Report & STOP
    │
    └─ No issues? ──► Mode 2: Run production sync
                               │
                               ├─ timeout 60s cargo run ...
                               ├─ Analyze logs
                               ├─ Document issues (minimal)
                               └─ Report summary & STOP
```

**Key Rules:**
- Only do ONE thing per cycle (fix one issue OR discover issues)
- Always stop after reporting
- Keep issue documentation minimal
- No root cause analysis during discovery

---

## Quick Reference

### Minimal Test Command

```bash
# Create run directory
RUN_DIR="tmp/run-$(date +%Y%m%d-%H%M%S)"
mkdir -p "$RUN_DIR"

# Run test with both raw and sanitized logs (60s for full cascade)
timeout 60s cargo run -- \
    --sync-bootstrap-relay-url wss://git.shakespeare.diy \
    --domain ngit.danconwaydev.com \
    --git-data-path "$RUN_DIR/git-data" \
    --relay-data-path "$RUN_DIR/relay-data" \
    2>&1 | tee "$RUN_DIR/sync-raw.log" | ./scripts/sanitize-logs.sh | tee "$RUN_DIR/sync.log"
```

### With Metrics Endpoint

```bash
# Create run directory
RUN_DIR="tmp/run-$(date +%Y%m%d-%H%M%S)"
mkdir -p "$RUN_DIR"

# Run with metrics and both log formats (60s for full cascade)
timeout 60s cargo run -- \
    --sync-bootstrap-relay-url wss://git.shakespeare.diy \
    --domain ngit.danconwaydev.com \
    --git-data-path "$RUN_DIR/git-data" \
    --relay-data-path "$RUN_DIR/relay-data" \
    --metrics-address 127.0.0.1:9090 \
    2>&1 | tee "$RUN_DIR/sync-raw.log" | ./scripts/sanitize-logs.sh | tee "$RUN_DIR/sync.log"
```

Then in another terminal: `curl http://127.0.0.1:9090/metrics`

### Cleanup Old Runs

```bash
# Remove runs older than 7 days
find tmp/run-* -type d -mtime +7 -exec rm -rf {} +

# Remove all test runs
rm -rf tmp/run-*
```

### Different Log Level

The default is DEBUG. For more detail:
```bash
RUST_LOG=trace cargo run -- [args]
```

For less noise:
```bash
RUST_LOG=info cargo run -- [args]
```

---

*Part of the [ngit-grasp documentation](../README.md) using the [Diátaxis](https://diataxis.fr/) framework.*