140 lines
6.8 KiB
Markdown
140 lines
6.8 KiB
Markdown
# Didactyl Tool Testing Plan
|
|
|
|
## Overview
|
|
|
|
Testing agent-mediated tools is different from unit testing pure functions because the execution path spans multiple boundaries:
|
|
|
|
```
|
|
LLM schema interpretation → JSON argument generation → argument parsing → business logic → Nostr/network I/O → JSON response → LLM interpretation
|
|
```
|
|
|
|
This plan defines three testing layers, ordered by implementation priority.
|
|
|
|
---
|
|
|
|
## Layer 1: Direct Tool Execution
|
|
|
|
Call `tools_execute()` directly with known JSON arguments and assert the JSON response structure.
|
|
|
|
### Implementation
|
|
|
|
Add a `--test-tool <name> <args_json>` CLI flag to `src/main.c` that:
|
|
1. Initializes config and nostr_handler (for network-dependent tools)
|
|
2. Calls `tools_execute(&ctx, name, args_json)`
|
|
3. Prints the raw JSON result to stdout
|
|
4. Exits with 0 if `success: true`, 1 otherwise
|
|
|
|
### Pure Computation Tools (no network needed)
|
|
|
|
| Tool | Test Command | Expected |
|
|
|------|-------------|----------|
|
|
| `nostr_encode` | `--test-tool nostr_encode '{"type":"npub","hex":"<64-char-hex>"}'` | `success: true`, uri starts with `nostr:npub1` |
|
|
| `nostr_decode` | `--test-tool nostr_decode '{"uri":"npub1..."}'` | `success: true`, pubkey is 64-char hex |
|
|
| `nostr_encrypt` | `--test-tool nostr_encrypt '{"recipient_pubkey":"<hex>","plaintext":"hello"}'` | `success: true`, ciphertext is base64 |
|
|
| `nostr_decrypt` | `--test-tool nostr_decrypt '{"sender_pubkey":"<hex>","ciphertext":"<from encrypt>"}'` | `success: true`, plaintext is "hello" |
|
|
|
|
### Network-Dependent Tools (need relay or HTTP)
|
|
|
|
| Tool | Test Command | Expected |
|
|
|------|-------------|----------|
|
|
| `nostr_nip05_lookup` | `--test-tool nostr_nip05_lookup '{"identifier":"_@laantungir.com"}'` | `success: true`, pubkey returned |
|
|
| `nostr_relay_info` | `--test-tool nostr_relay_info '{"relay_url":"wss://relay.damus.io"}'` | `success: true`, info.basic.name present |
|
|
| `nostr_relay_status` | `--test-tool nostr_relay_status '{}'` | `success: true`, relay_count > 0 |
|
|
| `nostr_dm_send` | `--test-tool nostr_dm_send '{"recipient_pubkey":"<hex>","message":"test"}'` | `success: true` |
|
|
| `nostr_post` | `--test-tool nostr_post '{"kind":1,"content":"test"}'` | `success: true`, event_id present |
|
|
| `nostr_delete` | `--test-tool nostr_delete '{"event_ids":["<64-char-hex>"]}'` | `success: true` |
|
|
| `nostr_react` | `--test-tool nostr_react '{"event_id":"<hex>","event_pubkey":"<hex>"}'` | `success: true` |
|
|
| `nostr_profile_get` | `--test-tool nostr_profile_get '{"pubkey":"<hex>"}'` | `success: true`, found: true/false |
|
|
| `nostr_dm_send_nip17` | `--test-tool nostr_dm_send_nip17 '{"recipient_pubkey":"<hex>","message":"test"}'` | `success: true` |
|
|
| `nostr_list_manage` | `--test-tool nostr_list_manage '{"list_kind":10000,"action":"add","items":[["p","<hex>"]]}'` | `success: true` |
|
|
|
|
### Error Case Tests
|
|
|
|
Each tool should also be tested with:
|
|
- Empty args: `'{}'` — should return descriptive error
|
|
- Missing required fields — should return specific error message
|
|
- Invalid hex strings — should reject gracefully
|
|
- Double-encoded JSON string args — should parse correctly (regression for the `invalid arguments JSON` bug)
|
|
|
|
---
|
|
|
|
## Layer 2: Schema-Parse Fidelity
|
|
|
|
Validates that the OpenAI function schema matches what executors actually accept.
|
|
|
|
### Implementation
|
|
|
|
A Python script (`tests/validate_schemas.py`) that:
|
|
1. Runs `didactyl_static --dump-schemas` (new flag) to get the tool schema JSON
|
|
2. For each tool in the schema:
|
|
- Generates a minimal valid argument payload from `required` + `properties`
|
|
- Runs `didactyl_static --test-tool <name> '<generated_json>'`
|
|
- Asserts exit code 0 or expected network error
|
|
3. Reports mismatches between schema and executor expectations
|
|
|
|
### What This Catches
|
|
|
|
- Schema says `required: ["hex"]` but executor checks for `"pubkey"` — mismatch
|
|
- Schema says `type: "integer"` but executor reads it as string
|
|
- Schema advertises parameters the executor ignores
|
|
- Missing required parameters in schema that executor demands
|
|
|
|
---
|
|
|
|
## Layer 3: Agent Integration Testing (Live DM)
|
|
|
|
The most natural test — DM the running agent and verify it uses tools correctly.
|
|
|
|
### Prerequisites
|
|
|
|
- Local relay running at `ws://127.0.0.1:7777`
|
|
- Didactyl running with valid config
|
|
- A separate Nostr client (or script) to send/receive DMs
|
|
|
|
### Test Prompts
|
|
|
|
| # | Tool | DM Prompt | Assert in Response |
|
|
|---|------|-----------|-------------------|
|
|
| 1 | `nostr_encode` | "Encode this pubkey as npub: `<64-char hex>`" | Contains `nostr:npub1` |
|
|
| 2 | `nostr_decode` | "Decode this npub: `npub1...`" | Contains the hex pubkey |
|
|
| 3 | `nostr_dm_send` | "Send a DM to `<your-own-pubkey>` saying 'hello test'" | Confirms sent; you receive it |
|
|
| 4 | `nostr_encrypt` | "Encrypt 'secret message' for `<pubkey>`" | Contains base64 ciphertext |
|
|
| 5 | `nostr_decrypt` | "Decrypt this NIP-44 payload: `<ciphertext from #4>`" | Contains 'secret message' |
|
|
| 6 | `nostr_nip05_lookup` | "Look up `_@laantungir.com`" | Returns a pubkey |
|
|
| 7 | `nostr_relay_info` | "Get NIP-11 info for `wss://relay.damus.io`" | Returns relay name and supported NIPs |
|
|
| 8 | `nostr_relay_status` | "Show me relay connection status" | Lists connected relays with stats |
|
|
| 9 | `nostr_react` | "React with 🤙 to event `<id>` from `<pubkey>`" | Confirms kind 7 published |
|
|
| 10 | `nostr_delete` | "Delete event `<id>`" | Confirms kind 5 published |
|
|
| 11 | `nostr_profile_get` | "Look up the profile for `<pubkey>`" | Returns name/about/picture |
|
|
| 12 | `nostr_post` | "Post a kind 1 note saying 'tool test'" | Confirms published with event_id |
|
|
| 13 | `nostr_list_manage` | "Add `<pubkey>` to my mute list" | Confirms kind 10000 published |
|
|
| 14 | `nostr_dm_send_nip17` | "Send a private NIP-17 DM to `<pubkey>` saying 'gift wrap test'" | Confirms gift wrap sent |
|
|
| 15 | `nostr_post_readme` | "Publish the README to Nostr" | Confirms kind 30023 with d=readme.md |
|
|
|
|
### Verification Methods
|
|
|
|
- **stdout logs**: Watch `[didactyl] executing tool call: <name>` in terminal
|
|
- **context.log**: Full LLM conversation including tool calls and results
|
|
- **Relay inspection**: Query the local relay for published events
|
|
- **DM receipt**: For DM tools, verify the message arrives at the recipient
|
|
|
|
---
|
|
|
|
## Implementation Priority
|
|
|
|
1. **Immediate (no code changes)**: Run Layer 3 tests by DMing the live agent
|
|
2. **Next sprint**: Add `--test-tool` CLI flag for Layer 1
|
|
3. **Later**: Add `--dump-schemas` flag and Python schema validator for Layer 2
|
|
4. **CI integration**: Wrap Layer 1 pure-computation tests in a shell script that runs after `build_static.sh`
|
|
|
|
---
|
|
|
|
## Local Relay Setup for Testing
|
|
|
|
A local relay like `strfry` or `nostr-rs-relay` at `ws://127.0.0.1:7777` provides:
|
|
- All published events are observable and queryable
|
|
- No rate limits or content policies
|
|
- NIP-42 auth testing in isolation
|
|
- Gift-wrapped NIP-17 DMs stay in your test environment
|
|
- Database can be wiped between test runs
|