didactyl/plans/prompt_templates_coding.md

# Prompt Template System — Coding Plan

Design doc: `plans/prompt_templates.md`

## Overview

Replace the hardcoded context assembly in `src/agent.c` with a template-driven system. The template lives inside the soul event content (kind 31120), delimited by `---template---`. If no template marker is found, fall back to the current hardcoded assembly for backward compatibility.

---

## Phase 1: New Files — Template Parser & Builder

### Step 1.1: Create `src/prompt_template.h`

Header with data structures and public API.

```c
#ifndef DIDACTYL_PROMPT_TEMPLATE_H
#define DIDACTYL_PROMPT_TEMPLATE_H

#include "cjson/cJSON.h"
#include "config.h"

#define PROMPT_TEMPLATE_MAX_SECTIONS 32
#define PROMPT_TEMPLATE_MAX_NAME_LEN 64
#define PROMPT_TEMPLATE_MAX_ROLE_LEN 16
#define PROMPT_TEMPLATE_MARKER "---template---"

typedef struct {
    char name[PROMPT_TEMPLATE_MAX_NAME_LEN];
    char role[PROMPT_TEMPLATE_MAX_ROLE_LEN];   // system, user, assistant, expand
    char* content_template;                     // content with {{var}} placeholders, or NULL
    int limit;                                  // for expand sections, 0 = default
} prompt_template_section_t;

typedef struct {
    char* personality;                          // everything above ---template---
    prompt_template_section_t sections[PROMPT_TEMPLATE_MAX_SECTIONS];
    int section_count;
} prompt_template_t;

// Variable resolver callback: given a variable name, return a malloc'd string or NULL.
// The caller frees the returned string.
typedef char* (*prompt_var_resolver_fn)(const char* var_name, void* user_data);

// Parse soul content into a template. Returns 0 on success, -1 if no template found.
// On success, caller must call prompt_template_free() when done.
// On -1 (no template), out_template is zeroed — caller should use hardcoded fallback.
int prompt_template_parse(const char* soul_content, prompt_template_t* out_template);

// Build a cJSON messages array from a parsed template.
// resolver_fn is called for each {{variable}} encountered.
// dm_history_messages is a cJSON array of user/assistant messages for "expand" sections.
// Returns a new cJSON array (caller owns it), or NULL on error.
cJSON* prompt_template_build_messages(
    const prompt_template_t* tmpl,
    prompt_var_resolver_fn resolver_fn,
    void* resolver_user_data,
    cJSON* dm_history_messages
);

// Free internals of a parsed template (does not free the struct itself).
void prompt_template_free(prompt_template_t* tmpl);

// Get the section name for a message index (for logging/API).
// Returns the section name string or NULL if idx is out of range.
const char* prompt_template_section_name_at(const prompt_template_t* tmpl, int section_idx);

#endif
```

### Step 1.2: Create `src/prompt_template.c`

Implementation file with three main components:

#### 1.2a: Template Parser — `prompt_template_parse()`

Logic:
1. Search `soul_content` for the string `"\n---template---\n"` (with newlines on both sides, or at start/end of string).
2. If not found, return -1 (no template).
3. Split: everything before the marker → `tmpl->personality` (strdup'd).
4. Everything after the marker → parse as template sections.
5. Template section format is line-oriented YAML-like:

```
- section: admin_identity
  role: system
  limit: 0
  content: |
    This is your administrator! Admin pubkey: {{admin_pubkey}}
```

Parsing rules:
- Lines starting with `- section:` begin a new section.
- `role:` sets the role (default: `system`).
- `limit:` sets the limit integer (default: 0).
- `content: |` starts a multi-line content block. All subsequent lines indented by 4+ spaces (or until the next `- section:` line) are the content template.
- `{{variable_name}}` placeholders in content are left as-is during parsing; they are resolved at build time.

Edge cases:
- Trim leading/trailing whitespace from section names and roles.
- If `content:` is a single line (not `|`), treat the rest of the line as the content.
- Cap at `PROMPT_TEMPLATE_MAX_SECTIONS`.

#### 1.2b: Variable Resolver — `prompt_template_build_messages()`

Logic:
1. Create a new cJSON array.
2. First, append the personality as a system message (role=system, content=personality).
3. For each section in order:
   - If `role` is `"expand"`: insert the `dm_history_messages` array items here, limited by `section.limit` (take last N if limit > 0).
   - Otherwise: resolve `{{var}}` placeholders in `content_template` by calling `resolver_fn(var_name, user_data)`. Build the resolved string. Append as a message with the configured role.
4. Return the array.

Variable resolution:
- Scan content_template for `{{` ... `}}` pairs.
- Extract the variable name (trimmed).
- Call `resolver_fn(name, user_data)`.
- If resolver returns NULL, substitute empty string.
- If resolver returns a string, substitute it and free the returned string.
- Build the final resolved content by concatenating literal segments and resolved values.

#### 1.2c: Cleanup — `prompt_template_free()`

- Free `tmpl->personality`.
- For each section, free `content_template`.
- Zero the struct.

---

## Phase 2: Variable Resolver in `src/agent.c`

### Step 2.1: Create a resolver function

Add a static function in `src/agent.c`:

```c
static char* agent_resolve_template_var(const char* var_name, void* user_data);
```

This function maps variable names to data sources:

| Variable Name | Source | Implementation |
|---|---|---|
| `admin_pubkey` | `g_cfg->admin.pubkey` | `strdup(g_cfg->admin.pubkey)` |
| `admin_kind0_json` | `nostr_handler_get_admin_kind0_context()` | Already returns malloc'd string |
| `admin_kind10002_json` | `nostr_handler_get_admin_kind10002_context()` | Already returns malloc'd string |
| `startup_events_json` | Serialize startup events | Reuse logic from current `append_startup_events_context()` |
| `adopted_skills_content` | Build skills string | Reuse logic from current `append_adopted_skills_context()` |
| `admin_notes_content` | `nostr_handler_get_admin_kind1_notes_context()` | Already returns malloc'd string |
| `agent_pubkey` | `g_cfg->keys.public_key_hex` | `strdup(g_cfg->keys.public_key_hex)` |

The `user_data` parameter is unused (NULL) since the resolver accesses globals.

### Step 2.2: Extract helper functions from existing code

Refactor the following existing static functions to return malloc'd strings instead of appending directly to a cJSON array:

- Extract startup events serialization from `append_startup_events_context()` (lines 383-434) into a new `static char* build_startup_events_string(void)`.
- Extract adopted skills content from `append_adopted_skills_context()` (lines ~744-940) into a new `static char* build_adopted_skills_string(void)`.

These helpers are called by the resolver function.

---

## Phase 3: Wire Template into Agent

### Step 3.1: Parse template at init time

In `agent_init()` (or when `g_system_context` is set), after the soul content is available:

```c
static prompt_template_t g_prompt_template;
static int g_has_template = 0;
```

After `g_system_context` is assigned, call:

```c
g_has_template = (prompt_template_parse(g_system_context, &g_prompt_template) == 0);
```

If `g_has_template` is true, `g_prompt_template.personality` replaces `g_system_context` for the system prompt message.

### Step 3.2: Modify `agent_build_admin_messages_json()`

Current location: `src/agent.c:1092`

Current signature (unchanged):
```c
int agent_build_admin_messages_json(const char* current_user_message, char** out_messages_json);
```

New logic:

```c
if (g_has_template) {
    // Build DM history as a cJSON array
    cJSON* dm_history = build_dm_history_array(current_user_message);
    
    // Build messages from template
    cJSON* messages = prompt_template_build_messages(
        &g_prompt_template,
        agent_resolve_template_var,
        NULL,
        dm_history
    );
    
    cJSON_Delete(dm_history);
    
    if (!messages) {
        return -1;
    }
    
    char* json = cJSON_PrintUnformatted(messages);
    cJSON_Delete(messages);
    *out_messages_json = json;
    return json ? 0 : -1;
} else {
    // Existing hardcoded assembly (current code, unchanged)
    ...
}
```

### Step 3.3: Extract DM history builder

Extract the DM history logic from `append_recent_admin_dm_history()` into a function that returns a cJSON array of user/assistant messages:

```c
static cJSON* build_dm_history_array(const char* current_user_message);
```

This is used by the template builder for `role: expand` sections.

---

## Phase 4: Context Log Formatting

### Step 4.1: Update `format_context_payload_for_log()`

Current location: `src/agent.c:245`

When `g_has_template` is true, use section names from the template instead of `detect_context_section()`:

- Message 0 is always `system_prompt` (the personality).
- Messages 1..N map to template sections by index.
- For `expand` sections, multiple messages share the same section name.

Change the log header from:
```
Message 01 | role=system | section=system_prompt
```
To:
```
Section: system_prompt | role=system
```

### Step 4.2: Update `classify_part_name()` in `src/http_api.c`

Current location: `src/http_api.c:113`

Add a new exported function from `src/agent.h`:
```c
const char* agent_get_section_name_for_message(int message_index);
```

This returns the template section name if a template is active, or falls back to the existing content-prefix detection.

In `classify_part_name()`, call this function first. If it returns non-NULL, use it. Otherwise fall back to the existing `strncmp` chain.

---

## Phase 5: Build System Updates

### Step 5.1: Update `Makefile`

Add `$(SRC_DIR)/prompt_template.c` to the `SRCS` list (after `trigger_manager.c`, before `http_api.c`).

### Step 5.2: Update `Dockerfile.alpine-musl`

Add `src/prompt_template.c` to the gcc command line (after `src/trigger_manager.c`, before `src/http_api.c`).

---

## Phase 6: Default Template in Soul

### Step 6.1: Update `config.json.example`

The startup events should include a soul event (kind 31120) with a `---template---` section that matches the current hardcoded behavior:

```markdown
# Didactyl Agent

You are Didactyl, a sovereign AI agent living on Nostr.
...existing soul content...

---template---

- section: admin_identity
  role: system
  content: |
    This is your administrator! Admin pubkey (hex): {{admin_pubkey}}

- section: admin_profile
  role: system
  content: |
    Administrator profile (JSON): {{admin_kind0_json}}

- section: admin_relay_list
  role: system
  content: |
    Administrator relay list (JSON): {{admin_kind10002_json}}

- section: startup_events
  role: system
  content: |
    Startup events memory (kinds/content/tags): {{startup_events_json}}

- section: adopted_skills
  role: system
  content: |
    {{adopted_skills_content}}

- section: dm_history
  role: expand
  limit: 12

- section: admin_notes
  role: system
  limit: 10
  content: |
    Administrator recent public notes: {{admin_notes_content}}
```

---

## Phase 7: Testing

### Step 7.1: Backward compatibility test

1. Build with `make`.
2. Run with existing config (no `---template---` in soul).
3. Send a DM and verify `context.log` output matches pre-change format.
4. Verify `/api/context/current` returns expected structure.

### Step 7.2: Template test

1. Edit the soul event content to include `---template---` section.
2. Restart agent.
3. Send a DM and verify `context.log` shows section-named headers.
4. Verify `/api/context/current` returns section names from template.
5. Verify the LLM receives the correct messages in the correct order.

### Step 7.3: Section reordering test

1. Move `admin_notes` section above `adopted_skills` in the template.
2. Restart and verify the order changes in `context.log`.

### Step 7.4: Limit test

1. Set `dm_history` limit to 4 (instead of 12).
2. Verify only 4 DM history turns appear in context.

---

## File Change Summary

| File | Action | Description |
|---|---|---|
| `src/prompt_template.h` | **NEW** | Data structures and API |
| `src/prompt_template.c` | **NEW** | Parser, variable resolver, context builder |
| `src/agent.c` | **MODIFY** | Add template globals, resolver function, wire into `agent_build_admin_messages_json()`, update log formatter |
| `src/agent.h` | **MODIFY** | Add `agent_get_section_name_for_message()` export |
| `src/http_api.c` | **MODIFY** | Update `classify_part_name()` to use template section names |
| `Makefile` | **MODIFY** | Add `prompt_template.c` to SRCS |
| `Dockerfile.alpine-musl` | **MODIFY** | Add `prompt_template.c` to gcc command |
| `config.json.example` | **MODIFY** | Update soul event to include template section |

---

## Implementation Order

1. `src/prompt_template.h` — data structures and API declarations
2. `src/prompt_template.c` — parser, builder, free
3. `Makefile` + `Dockerfile.alpine-musl` — add new source file
4. Build and verify compilation
5. `src/agent.c` — extract helper functions (`build_startup_events_string`, `build_adopted_skills_string`, `build_dm_history_array`)
6. `src/agent.c` — add resolver function and template globals
7. `src/agent.c` — wire template into `agent_build_admin_messages_json()`
8. `src/agent.c` — update `format_context_payload_for_log()`
9. `src/agent.h` + `src/http_api.c` — section name API for classify_part_name
10. Build and test backward compatibility (no template in soul)
11. `config.json.example` — add template to soul event
12. Test with template soul
13. Test section reordering and limit changes