# Didactyl Automated Test Harness

## Overview

An automated testing system that starts a Didactyl agent locally (debug build), converses with it via the HTTP API, exercises all tools, monitors logs in real-time, handles agent crashes/restarts, and produces a structured test results report.

**Language:** Python (stdlib only, no external dependencies)  
**Location:** `tests/`  
**Phase 1:** Scripted tests (no LLM driving the tester)  
**Phase 2 (future):** LLM-driven test agent that generates prompts, evaluates responses, and adapts

---

## Architecture

```mermaid
flowchart TB
    subgraph Test Harness - Python
        RUNNER[test_runner.py<br/>orchestrator]
        PROC[agent_process.py<br/>start/stop/restart]
        CLIENT[didactyl_client.py<br/>HTTP API wrapper]
        LOG[log_watcher.py<br/>tail debug.log]
        REPORT[reporter.py<br/>results output]
        
        RUNNER --> PROC
        RUNNER --> CLIENT
        RUNNER --> LOG
        RUNNER --> REPORT
    end
    
    subgraph Test Suites
        TH[test_health]
        TC[test_conversation]
        TI[test_tools_identity]
        TN[test_tools_nostr]
        TS[test_tools_skills]
        TSY[test_tools_system]
        TM[test_tools_memory]
        TCA[test_tools_cashu]
        TB[test_tools_blossom]
        TTO[test_timeouts]
        TE[test_errors]
        TR[test_restart]
    end
    
    RUNNER --> TH & TC & TI & TN & TS & TSY & TM & TCA & TB & TTO & TE & TR
    
    subgraph Didactyl Agent - debug build
        AGENT[didactyl process]
        API[HTTP API :8484]
        LOGFILE[debug.log]
    end
    
    CLIENT -- HTTP --> API
    PROC -- subprocess --> AGENT
    LOG -- tail --> LOGFILE
```

---

## Design Decisions

| Decision | Choice | Rationale |
|---|---|---|
| Language | Python | Best subprocess/HTTP/threading support; already used in project |
| Dependencies | stdlib only | No pip install needed; `urllib`, `subprocess`, `threading`, `json` |
| Test framework | Standalone runner | Self-contained, no pytest dependency |
| Relay strategy | Flexible genesis config | User provides their own test_genesis.jsonc |
| LLM for agent | Real model | User provides API key in test genesis config |
| Tool scope | All tools | Disposable test identity; full coverage |
| Entry point | `tests/run_tests.py` | Single script to run everything |

---

## Directory Structure

```
tests/
├── harness/
│   ├── __init__.py
│   ├── agent_process.py      # start/stop/restart didactyl subprocess
│   ├── didactyl_client.py    # HTTP API wrapper with timeouts
│   ├── log_watcher.py        # real-time log tail + marker system
│   ├── test_runner.py        # orchestrator
│   └── reporter.py           # results formatting (JSON + text)
├── suites/
│   ├── __init__.py
│   ├── test_health.py        # status, context endpoints
│   ├── test_conversation.py  # basic prompt/response
│   ├── test_tools_identity.py # identity and context tools
│   ├── test_tools_nostr.py   # nostr event, messaging, relay tools
│   ├── test_tools_skills.py  # skill and trigger tools
│   ├── test_tools_system.py  # system, local, model, config tools
│   ├── test_tools_memory.py  # task and memory tools
│   ├── test_tools_cashu.py   # cashu wallet tools
│   ├── test_tools_blossom.py # blossom tools
│   ├── test_timeouts.py      # response time assertions
│   ├── test_errors.py        # API error handling paths
│   └── test_restart.py       # crash recovery
├── configs/
│   └── test_genesis.jsonc    # example test config (user fills in secrets)
├── results/                  # test run output (gitignored)
├── run_tests.py              # entry point
├── test.sh                   # existing bash test (kept as-is)
├── blossom_tool_validation_test.c  # existing C test (kept as-is)
└── blossom_tool_validation_test    # existing compiled test (kept as-is)
```

---

## Component Specifications

### 1. agent_process.py — Process Manager

Manages the didactyl subprocess lifecycle.

```python
class AgentProcess:
    def __init__(self, binary_path, config_path, api_port=8484,
                 api_bind="127.0.0.1", debug_level=5, log_file=None):
        """Configure but don't start yet."""
    
    def start(self, timeout=30) -> bool:
        """
        Spawn didactyl as subprocess:
          ./didactyl --config <config_path> --debug <level> 
                     --api-port <port> --api-bind <bind>
        
        Set DIDACTYL_LOG_FILE env var to log_file path.
        Wait for GET /api/status to return 200 (poll with timeout).
        Returns True if agent started successfully.
        """
    
    def stop(self, timeout=10) -> bool:
        """
        Send SIGTERM, wait for clean exit.
        If still alive after timeout, send SIGKILL.
        Returns True if stopped cleanly.
        """
    
    def restart(self, timeout=30) -> bool:
        """stop() then start()."""
    
    def is_alive(self) -> bool:
        """Check process.poll() and optionally /api/status."""
    
    def pid(self) -> int | None:
        """Return PID if running."""
    
    def return_code(self) -> int | None:
        """Return exit code if stopped."""
```

Key details:
- Uses `subprocess.Popen` with `stdout=PIPE, stderr=PIPE`
- Captures stdout/stderr for crash diagnostics
- The health check polls `GET /api/status` every 500ms until success or timeout
- Sets `DIDACTYL_LOG_FILE` to a test-run-specific path like `tests/results/<timestamp>/agent_debug.log`

### 2. didactyl_client.py — HTTP API Client

Thin wrapper around the Didactyl HTTP API using only `urllib`.

```python
class DidactylClient:
    def __init__(self, base_url="https://127.0.0.1:8484", timeout=60,
                 verify_tls=False):
        """Configure base URL and default timeout."""
    
    def status(self) -> dict:
        """GET /api/status"""
    
    def prompt(self, message: str, max_turns: int = 4,
               model: str = None) -> dict:
        """POST /api/prompt/agent — full agent context conversation"""
    
    def prompt_raw(self, messages: list, max_turns: int = 4,
                   model: str = None) -> dict:
        """POST /api/prompt/run — raw messages, no auto-context"""
    
    def prompt_simple(self, system: str, user: str,
                      model: str = None) -> dict:
        """POST /api/prompt/run-simple — no tools"""
    
    def context_current(self) -> dict:
        """GET /api/context/current"""
    
    def context_parts(self) -> dict:
        """GET /api/context/parts"""
    
    def fire_webhook(self, d_tag: str, payload: dict = None) -> dict:
        """POST /api/trigger/<d_tag>"""
```

Key details:
- All methods return parsed JSON dict
- Raises `TimeoutError` if response exceeds timeout
- Raises `ConnectionError` if agent is unreachable
- Raises `APIError(status_code, body)` for non-2xx responses
- Uses `ssl._create_unverified_context()` for local TLS (same as chat CLI)

### 3. log_watcher.py — Real-time Log Monitor

Background thread that tails the agent's debug log file.

```python
class LogWatcher:
    def __init__(self, log_path: str):
        """Configure log file path."""
    
    def start(self):
        """
        Start background thread.
        Open file, seek to end, poll for new lines.
        Store all lines in memory with timestamps.
        """
    
    def stop(self):
        """Stop background thread."""
    
    def set_marker(self, name: str):
        """Record current line count as a named marker."""
    
    def get_lines_since(self, marker: str) -> list[str]:
        """Return all lines captured since the named marker."""
    
    def get_all_lines(self) -> list[str]:
        """Return all captured lines."""
    
    def search(self, pattern: str, since_marker: str = None) -> list[str]:
        """Regex search through captured lines."""
    
    def has_errors(self, since_marker: str = None) -> bool:
        """Check for [ERROR] lines since marker."""
    
    def has_warnings(self, since_marker: str = None) -> bool:
        """Check for [WARN] lines since marker."""
    
    def error_lines(self, since_marker: str = None) -> list[str]:
        """Return all ERROR lines since marker."""
```

Key details:
- Uses `threading.Thread(daemon=True)` for background polling
- Polls file every 100ms for new content
- Handles file rotation (agent restart creates new file)
- Thread-safe access to captured lines via `threading.Lock`

### 4. test_runner.py — Orchestrator

```python
class TestResult:
    name: str
    suite: str
    status: str  # "pass", "fail", "error", "skip", "timeout"
    message: str
    duration_seconds: float
    agent_errors: list[str]  # ERROR lines from log during this test
    details: dict  # arbitrary test-specific data

class TestCase:
    name: str
    description: str
    requires_restart: bool = False
    
    def run(self, client: DidactylClient, log: LogWatcher) -> TestResult:
        """Execute the test and return result."""

class TestRunner:
    def __init__(self, agent: AgentProcess, client: DidactylClient,
                 log: LogWatcher):
        """Configure with harness components."""
    
    def discover_suites(self, suites_dir: str) -> list:
        """Import all test_*.py modules from suites directory."""
    
    def run_all(self, suites: list = None) -> list[TestResult]:
        """
        For each suite:
          1. If test requires restart, restart agent
          2. Set log marker for this test
          3. Run test with timeout wrapper
          4. Capture result + any agent errors from log
          5. If agent crashed, restart and record error
          6. Collect all results
        Return list of TestResult.
        """
    
    def run_suite(self, suite_name: str) -> list[TestResult]:
        """Run a single named suite."""
```

Key details:
- Each test gets a fresh log marker so errors can be correlated
- If `client.prompt()` raises `TimeoutError`, the test is marked "timeout" and the agent is restarted
- If `agent.is_alive()` returns False mid-suite, the agent is restarted and remaining tests continue
- Supports filtering by suite name or test name via CLI args

### 5. reporter.py — Results Output

```python
class Reporter:
    def __init__(self, results: list[TestResult], output_dir: str):
        """Configure with results and output directory."""
    
    def print_summary(self):
        """Print pass/fail/error/skip/timeout counts to stdout."""
    
    def print_details(self):
        """Print each test result with details."""
    
    def write_json(self, path: str):
        """Write full results as JSON for programmatic consumption."""
    
    def write_text(self, path: str):
        """Write human-readable report."""
```

Output format example:
```
== Didactyl Test Results ==
Run: 2026-03-25T09:50:00Z
Agent: v0.0.26
Model: claude-haiku-4.5

Pass:    42
Fail:     3
Error:    1
Timeout:  2
Skip:     0
Total:   48

-- Failures --
[FAIL] test_tools_nostr::nostr_post_kind1
  Expected tool_calls to contain 'nostr_post', got: ['nostr_query']
  Agent errors during test: 0

[FAIL] test_conversation::multi_turn
  Agent returned empty final_response
  Agent errors during test: 2
    [ERROR] [llm.c:234] HTTP 429 rate limited
    [ERROR] [llm.c:240] LLM call failed after 3 retries

[TIMEOUT] test_tools_skills::skill_create
  No response within 60s
  Agent was restarted
```

---

## Test Suite Specifications

### test_health.py

| Test | Prompt/Action | Assertions |
|---|---|---|
| `status_returns_200` | `GET /api/status` | HTTP 200, `success=true` |
| `status_has_fields` | `GET /api/status` | Has `name`, `version`, `pubkey`, `relay_count` |
| `context_current_returns_messages` | `GET /api/context/current` | Has `messages` array, `total_chars > 0` |
| `context_parts_has_system_prompt` | `GET /api/context/parts` | Has part named `system_prompt` |

### test_conversation.py

| Test | Prompt | Assertions |
|---|---|---|
| `simple_greeting` | "Hello, what is your name?" | `final_response` is non-empty string |
| `agent_responds_about_itself` | "What are you? Describe yourself briefly." | `final_response` mentions agent/Didactyl/Nostr |
| `empty_message_handling` | "" (empty string) | Returns error or handles gracefully |
| `very_long_message` | 10000 char string | Returns response or graceful error, no crash |

### test_tools_identity.py

| Test | Prompt | Expected Tool | Assertions |
|---|---|---|---|
| `get_pubkey` | "What is your public key in hex?" | `nostr_pubkey` or `my_pubkey` | Tool called, result has hex pubkey |
| `get_npub` | "What is your npub?" | `nostr_npub` or `my_npub` | Tool called, result has npub1... |
| `agent_identity` | "Tell me about your identity" | `agent_identity` | Tool called, success |
| `agent_version` | "What version are you?" | `agent_version` | Tool called, result has version string |
| `admin_identity` | "Who is your administrator?" | `admin_identity` | Tool called, success |

### test_tools_nostr.py

| Test | Prompt | Expected Tool | Assertions |
|---|---|---|---|
| `nostr_post_kind1` | "Post a test note saying 'Automated test post'" | `nostr_post` | Tool called with kind=1, success, event_id returned |
| `nostr_query_recent` | "Query the 3 most recent kind 1 notes from any author" | `nostr_query` | Tool called, returns events array |
| `nostr_my_events` | "List your recent events" | `nostr_my_events` | Tool called, success |
| `nostr_relay_status` | "What is the status of your relay connections?" | `nostr_relay_status` | Tool called, returns relay info |
| `nostr_dm_send` | "Send a test DM to yourself" | `nostr_dm_send` | Tool called, success |
| `nostr_encode_npub` | "Encode your pubkey as an npub" | `nostr_encode` | Tool called, returns npub |
| `nostr_profile_get` | "Look up your own Nostr profile" | `nostr_profile_get` | Tool called, returns profile |

### test_tools_skills.py

| Test | Prompt | Expected Tool | Assertions |
|---|---|---|---|
| `skill_list` | "List your available skills" | `skill_list` | Tool called, returns skills array |
| `trigger_list` | "List your active triggers" | `trigger_list` | Tool called, success |
| `skill_create_and_remove` | "Create a test skill called 'test-harness-probe' with content 'Test skill' then remove it" | `skill_create`, `skill_remove` | Both tools called, success |

### test_tools_system.py

| Test | Prompt | Expected Tool | Assertions |
|---|---|---|---|
| `tool_list` | "List all your available tools" | `tool_list` | Tool called, returns tools array |
| `model_get` | "What model are you currently using?" | `model_get` | Tool called, returns model info |
| `model_list` | "List available models" | `model_list` | Tool called, success |
| `local_http_fetch` | "Fetch https://httpbin.org/get" | `local_http_fetch` | Tool called, returns HTTP response |
| `config_store_recall` | "Store a test config with d_tag 'test_harness_probe' containing 'hello', then recall it" | `config_store`, `config_recall` | Both called, recalled value matches |

### test_tools_memory.py

| Test | Prompt | Expected Tool | Assertions |
|---|---|---|---|
| `task_list` | "Show me your current task list" | `task_list` or `task_manage` | Tool called, success |
| `task_manage_add_remove` | "Add a task 'test harness probe task' then remove it" | `task_manage` | Tool called with add then remove |
| `memory_save_recall` | "Save 'test harness probe' to memory, then recall your memory" | `memory_save`, `memory_recall` | Both called, recalled contains probe text |

### test_tools_cashu.py

| Test | Prompt | Expected Tool | Assertions |
|---|---|---|---|
| `wallet_balance` | "Check your cashu wallet balance" | `cashu_wallet_balance` | Tool called, success (even if empty) |

Note: Most cashu tools require a configured mint and funded wallet. Phase 1 tests only the read-only balance check. Full cashu testing requires a test mint setup.

### test_tools_blossom.py

| Test | Prompt | Expected Tool | Assertions |
|---|---|---|---|
| `blossom_list` | "List your blossom blobs" | `blossom_list` | Tool called, success (even if empty) |

Note: Full blossom testing requires a configured blossom server. Phase 1 tests only the list operation.

### test_timeouts.py

| Test | Action | Assertions |
|---|---|---|
| `response_within_timeout` | Send simple prompt, measure time | Response received within 60s |
| `status_responds_fast` | `GET /api/status`, measure time | Response within 2s |
| `context_responds_fast` | `GET /api/context/current`, measure time | Response within 5s |

### test_errors.py

| Test | Action | Assertions |
|---|---|---|
| `invalid_json_body` | POST malformed JSON to `/api/prompt/agent` | Returns 400, `success=false` |
| `missing_message_field` | POST `{}` to `/api/prompt/agent` | Returns 400, `success=false` |
| `unknown_endpoint` | GET `/api/nonexistent` | Returns 404 |
| `webhook_nonexistent_dtag` | POST to `/api/trigger/nonexistent-dtag` | Returns 404 |

### test_restart.py

| Test | Action | Assertions |
|---|---|---|
| `clean_restart` | Stop agent, start agent | Agent comes back, `/api/status` works |
| `status_after_restart` | Restart, then `GET /api/status` | Same pubkey, version as before restart |
| `conversation_after_restart` | Restart, then send prompt | Agent responds normally |

---

## Entry Point: run_tests.py

```
Usage:
  python tests/run_tests.py [options]

Options:
  --config PATH       Path to test genesis.jsonc (default: tests/configs/test_genesis.jsonc)
  --binary PATH       Path to didactyl binary (default: ./didactyl)
  --suite NAME        Run only this suite (can repeat)
  --test NAME         Run only this test (can repeat)
  --api-port PORT     API port (default: 8485)
  --timeout SECS      Default response timeout (default: 60)
  --debug-level N     Agent debug level 0-5 (default: 5)
  --output-dir PATH   Results output directory (default: tests/results/<timestamp>)
  --verbose           Print each test result as it runs
  --no-restart        Don't auto-restart agent on crash (fail remaining tests)
```

---

## Test Genesis Config

`tests/configs/test_genesis.jsonc` — an example config the user fills in:

```jsonc
{
  // TEST CONFIGURATION — fill in your test identity secrets
  
  "key": {
    "nsec": "nsec1REPLACE_WITH_DISPOSABLE_TEST_NSEC"
  },
  
  "admin": {
    "pubkey": "npub1REPLACE_WITH_TEST_ADMIN_PUBKEY"
  },
  
  "dm_protocol": "nip04",
  
  "llm": {
    "provider": "openai",
    "api_key": "sk-REPLACE_WITH_API_KEY",
    "model": "claude-haiku-4.5",
    "base_url": "https://api.anthropic.com/v1",
    "max_tokens": 512,
    "temperature": 0.3
  },
  
  "api": {
    "enabled": true,
    "port": 8485,
    "bind_address": "127.0.0.1"
  },
  
  "startup_events": [
    {
      "kind": 0,
      "content_fields": {
        "name": "Didactyl Test Agent",
        "about": "Automated test instance"
      },
      "tags": []
    },
    {
      "kind": 10002,
      "content": "",
      "tags": [
        ["r", "wss://relay.damus.io"],
        ["r", "wss://relay.primal.net"]
      ]
    },
    {
      "kind": 31124,
      "content": "# Test Agent\n\nYou are a test agent. Respond to all requests. Use tools when asked.\n\n{{my_kind0_profile}}\n\nYour npub: {{my_npub}}",
      "tags": [
        ["d", "identity_and_rules"],
        ["app", "didactyl"],
        ["scope", "private"],
        ["trigger", "dm"],
        ["filter", "{\"from\":\"admin\"}"]
      ]
    }
  ]
}
```

---

## Execution Flow

```mermaid
flowchart TD
    START[run_tests.py] --> PARSE[Parse CLI args]
    PARSE --> BUILD[Verify didactyl binary exists]
    BUILD --> CONFIG[Load test genesis config]
    CONFIG --> MKDIR[Create results output dir]
    MKDIR --> LOG_START[Start LogWatcher]
    LOG_START --> AGENT_START[Start AgentProcess]
    AGENT_START --> HEALTH[Wait for /api/status 200]
    
    HEALTH -->|timeout| FAIL_STARTUP[Report startup failure and exit]
    HEALTH -->|success| DISCOVER[Discover test suites]
    
    DISCOVER --> LOOP{Next test?}
    
    LOOP -->|yes| CHECK_ALIVE{Agent alive?}
    CHECK_ALIVE -->|no| RESTART_MID[Restart agent]
    RESTART_MID --> CHECK_ALIVE
    CHECK_ALIVE -->|yes| MARKER[Set log marker]
    MARKER --> RUN_TEST[Run test with timeout]
    
    RUN_TEST -->|pass| RECORD_PASS[Record PASS]
    RUN_TEST -->|fail| RECORD_FAIL[Record FAIL]
    RUN_TEST -->|timeout| RESTART_TIMEOUT[Restart agent]
    RESTART_TIMEOUT --> RECORD_TIMEOUT[Record TIMEOUT]
    RUN_TEST -->|error| RECORD_ERROR[Record ERROR]
    
    RECORD_PASS & RECORD_FAIL & RECORD_TIMEOUT & RECORD_ERROR --> COLLECT_LOGS[Collect agent errors since marker]
    COLLECT_LOGS --> LOOP
    
    LOOP -->|no more| STOP_AGENT[Stop agent]
    STOP_AGENT --> STOP_LOG[Stop LogWatcher]
    STOP_LOG --> REPORT[Generate report]
    REPORT --> EXIT[Exit with code 0 if all pass else 1]
```

---

## Implementation Order

1. **Harness infrastructure** (agent_process, didactyl_client, log_watcher)
2. **Test runner + reporter** (orchestration layer)
3. **test_health.py** (validates the harness itself works)
4. **test_conversation.py** (validates basic agent interaction)
5. **test_errors.py** (validates error handling)
6. **test_timeouts.py** (validates timeout detection)
7. **test_restart.py** (validates process management)
8. **Tool test suites** (one at a time, identity → nostr → skills → system → memory → cashu → blossom)
9. **Entry point + config** (run_tests.py, test_genesis.jsonc)
10. **End-to-end validation** against a running agent

---

## Future: Phase 2 — LLM-Driven Testing

The architecture supports this naturally. In Phase 2:

- Add an `LLMTestAgent` class that wraps an LLM API call
- The LLM receives: tool documentation, test objectives, previous results
- It generates test prompts, evaluates responses, decides next actions
- The `TestCase.run()` method delegates to the LLM agent instead of scripted logic
- The LLM can also analyze agent logs for anomalies
- Potentially: the LLM can generate code patches for bugs it finds

No architectural changes needed — just a new test suite type that uses LLM reasoning instead of scripted assertions.
