didactyl/plans/template_tool_unification.md

# Plan: Unify Template Variables with Tools

## Problem

The soul template system (`---template---` in kind 31120) uses `{{variable}}` placeholders that are resolved by a hardcoded C function (`agent_template_resolve_var()`). This is a parallel data-fetching system alongside the existing tool registry. Every new context data source requires:

1. Adding a C function to produce the data
2. Adding an `if (strcmp(...))` branch in the resolver
3. Documenting the new variable name
4. Hoping the template author uses the exact right name

This caused the bug that started this investigation — 5 out of 9 template variables were misspelled/mismatched, producing empty context sections and a provider rejection.

## Goal

**Eliminate template variables entirely.** Template sections fetch their data by calling tools. The tool registry is the single mechanism for getting data into context.

## Design

### New Template Syntax

Replace `content: |` with `tool:` and optional `args:` directives:

```yaml
---template---

- section: admin_identity
  role: system
  tool: admin_identity
  skip_if_empty: true

- section: admin_profile
  role: system
  tool: nostr_admin_profile
  skip_if_empty: true

- section: admin_contacts
  role: system
  tool: nostr_admin_contacts
  skip_if_empty: true

- section: admin_relays
  role: system
  tool: nostr_admin_relays
  skip_if_empty: true

- section: admin_notes
  role: system
  tool: nostr_admin_notes
  skip_if_empty: true

- section: tools
  role: system
  tool: tool_list
  skip_if_empty: true

- section: tasks
  role: system
  tool: task_list
  skip_if_empty: true

- section: dm_history
  role: expand
  limit: 12

- section: conversation
  role: user
  tool: message_current
  skip_if_empty: true
```

A section can still use the old `content:` with `{{variables}}` for static text or mixed content. But the primary mechanism for dynamic data is `tool:`.

### How It Works

```mermaid
flowchart TD
    DM[Incoming DM] --> BUILD[Build context from template]
    BUILD --> SOUL[Emit personality as system msg]
    SOUL --> LOOP[For each template section]
    LOOP --> CHECK{Has tool: directive?}
    CHECK -->|Yes| EXEC["tools_execute(tool_name, args)"]
    EXEC --> FORMAT[Extract content from JSON result]
    FORMAT --> EMIT[Emit as chat message]
    CHECK -->|No| RESOLVE["Resolve {{vars}} from content template<br/>(legacy path, eventually removed)"]
    RESOLVE --> EMIT
    EMIT --> LOOP
    LOOP --> DONE[Complete messages array]
```

### New Context Tools

Add lightweight "context tools" that read from cached in-memory data. These are fast (no network), deterministic, and use the same `tools_execute()` interface as everything else.

| Tool | Returns | Source |
|---|---|---|
| `admin_identity` | Admin pubkey + verification text | config + sender tier |
| `nostr_admin_profile` | Admin kind 0 profile JSON | cached `g_admin_kind0_json` |
| `nostr_admin_contacts` | Admin kind 3 contacts JSON array | cached `g_admin_wot_contacts` |
| `nostr_admin_relays` | Admin kind 10002 relay list JSON | cached `g_admin_kind10002_json` |
| `nostr_admin_notes` | Admin recent kind 1 notes | cached `g_admin_kind1_notes` |
| `task_list` | Current task list from tasks.json | file read (already exists as `task_manage` with `action: list`) |
| `message_current` | Current user message text | passed via tool context |
| `agent_identity` | Agent pubkey, npub | config |

Existing tools that already work for context:
- `tool_list` — returns tool schemas (already exists)
- `task_manage` with `{"action":"list"}` — returns tasks (already exists)
- `local_file_read` — can read any file (already exists)

### Tool Result → Message Content

Tool results are JSON objects like `{"success":true,"content":"..."}`. The template system extracts the `content` field (or a configurable field) as the message text. If the tool returns an error or empty content, and `skip_if_empty: true` is set, the section is omitted.

For tools that return structured data (like `tool_list` returning a schema array), the template can specify a `result_field` or `format` to control extraction:

```yaml
- section: tools
  role: system
  tool: tool_list
  result_field: tools
  skip_if_empty: true
```

### Implementation Phases

#### Phase 1: Add context tools + tool: directive support

1. Add new `context_*` tools to `tools.c` that wrap the existing cached data getters
2. Extend `prompt_template_section_t` with `tool_name` and `tool_args` fields
3. Extend `prompt_template_parse()` to recognize `tool:` and `args:` directives
4. Extend `prompt_template_build_messages()` to call `tools_execute()` when a section has `tool_name` set
5. Update the soul template in `config.jsonc` to use `tool:` directives
6. Keep `content:` + `{{variable}}` working as a fallback for backward compatibility

#### Phase 2: Migrate all sections to tool-based

1. Convert all template sections from `content: {{variable}}` to `tool:` directives
2. Verify all context data flows through tools
3. Update `context_template.md` to reflect new syntax

#### Phase 3: Remove legacy variable resolver

1. Remove `agent_template_resolve_var()` and all its helper functions that are now redundant
2. Remove `prompt_var_resolver_fn` callback from `prompt_template_build_messages()`
3. Clean up dead code in `agent.c`
4. Update all documentation

### Template Section Struct Changes

```c
typedef struct {
    char name[PROMPT_TEMPLATE_MAX_NAME_LEN];
    char role[PROMPT_TEMPLATE_MAX_ROLE_LEN];
    char* content_template;       // legacy: {{variable}} content
    char* tool_name;              // NEW: tool to call for content
    char* tool_args;              // NEW: JSON args for tool call
    char* result_field;           // NEW: which JSON field to extract (default: "content")
    int limit;
    int skip_if_empty;
    char* provider_name;
    char* provider_content_template;
} prompt_template_section_t;
```

### Soul Template Example (After Migration)

```
# Didactyl Agent

You are Didactyl, a sovereign AI agent living on Nostr.
...

---template---

- section: admin_identity
  role: system
  tool: admin_identity
  skip_if_empty: true

- section: admin_profile
  role: system
  tool: nostr_admin_profile
  skip_if_empty: true

- section: admin_contacts
  role: system
  tool: nostr_admin_contacts
  skip_if_empty: true

- section: admin_relays
  role: system
  tool: nostr_admin_relays
  skip_if_empty: true

- section: admin_notes
  role: system
  tool: nostr_admin_notes
  skip_if_empty: true

- section: tools
  role: system
  tool: tool_list
  result_field: tools
  skip_if_empty: true

- section: tasks
  role: system
  tool: task_manage
  args: {"action":"list"}
  result_field: tasks
  skip_if_empty: true

- section: dm_history
  role: expand
  limit: 12

- section: conversation
  role: user
  tool: message_current
  skip_if_empty: true
```

### What Gets Deleted Eventually

- `agent_template_resolve_var()` and all its `if (strcmp(...))` branches
- `build_tool_schemas_json_string()`
- `build_tasks_content_string()` (replaced by `task_manage` tool)
- `build_admin_recent_posts_text()` (replaced by `nostr_admin_notes` tool)
- `build_admin_profile_plain_text()` (replaced by `nostr_admin_profile` tool)
- `build_admin_relay_list_plain_text()` (replaced by `nostr_admin_relays` tool)
- `build_sender_verification_text()` (folded into `admin_identity` tool)
- `build_startup_events_json_string()` (replaced by tool if needed)
- `build_adopted_skills_payload_string()` (replaced by tool if needed)
- `nostr_handler_get_admin_kind3_context()` (just added, will be replaced by tool)
- The `prompt_var_resolver_fn` callback type

### Benefits

1. **Single data-fetching mechanism** — tools are the only way to get data
2. **No more variable name mismatches** — tool names are validated at registration
3. **User-configurable context** — admin can add any tool output to context by editing the soul template
4. **Testable** — `--test-tool nostr_admin_profile` shows exactly what goes into context
5. **Self-documenting** — `tool_list` shows all available context tools with descriptions
6. **Extensible** — new tools automatically become available as context sources
7. **Provider overrides still work** — `provider:` directive can still override formatting per-provider

### Risks

1. **Performance** — Tool calls add function dispatch overhead vs direct variable resolution. Mitigated: context tools read cached data, no network calls.
2. **Backward compatibility** — Old soul templates with `{{variables}}` would break. Mitigated: Phase 1 keeps both paths working.
3. **Tool context threading** — `tools_execute()` needs access to the tools context, which `prompt_template_build_messages()` doesn't currently have. Solution: pass `tools_context_t*` to the build function.