sovereign_browser/plans/embedded-agent.md

Embedded Agent — In-Browser LLM with First-Class MCP Access

Concurrency model — long-running tasks

The agent loop runs on a background GThread, not the GTK main thread. This is critical for long-running tasks like "follow each link, download images, write a summary for each" which may involve dozens of LLM calls and hundreds of tool calls.

sequenceDiagram
    participant UI as Chat UI main thread
    participant Chat as agent_chat.c
    participant Thread as Agent Loop background thread
    participant LLM as LLM API
    participant Main as GTK main loop
    participant Tools as agent_tools_dispatch

    UI->>Chat: POST sovereign://agents/send?text=...
    Chat->>Thread: g_thread_new agent_loop_run
    Chat-->>UI: 200 OK session started
    loop ReAct iterations
        Thread->>LLM: POST chat/completions blocking
        LLM-->>Thread: response + tool_calls
        Thread->>Thread: persist assistant message
        alt has tool_calls
            loop each tool call
                Thread->>Main: g_idle_add dispatch_tool
                Main->>Tools: agent_tools_dispatch sync JS eval
                Tools-->>Main: result
                Main-->>Thread: result via GAsyncQueue
                Thread->>Thread: persist tool result
            end
        else no tool_calls
            Thread->>Thread: mark session complete
        end
    end
    Thread->>Main: g_idle_add update_ui final

Threading rules

1. LLM HTTP calls happen on the background thread — soup_session_send blocks in-thread, the GTK UI stays responsive.

2. Browser tool dispatch (snapshot, click, eval, etc.) MUST run on the GTK main thread because WebKitGTK is not thread-safe. The background thread schedules each tool call via g_idle_add() and waits for the result on a GAsyncQueue or a per-call GMainLoop (same pattern the MCP HTTP handler uses for sync JS eval).

3. Filesystem + shell tools run directly on the background thread — they don't touch GTK or WebKit, so no main-thread hop needed. This keeps long shell commands from blocking the UI.

4. SQLite writes use the existing SQLITE_OPEN_FULLMUTEX connection (thread-safe). The background thread can call db_kv_set / agent_chat_store_* directly.

5. UI updates happen via g_idle_add() — the background thread pushes status updates (current tool, iteration count, partial output) to a shared struct, and an idle callback renders them in the chat page.

6. Cancellation — a g_atomic_int cancel flag is checked at the top of each loop iteration. The chat UI's "Stop" button sets it. The background thread exits cleanly at the next check point.

Iteration cap

The default max_iterations is 100 (configurable on sovereign://agents), not 20. Long tasks like "follow 30 links" need multiple tool calls per link (open, snapshot, extract, download, write) — easily 100+ calls. The cap is a safety valve, not a tight limit. The user can raise it in settings for very long batch jobs.

Status polling

The chat UI polls sovereign://agents/status?session=... every 500ms while a session is active. The response includes:

• state: idle | thinking | tool_call | complete | error | cancelled
• iteration: current iteration number
• current_tool: name of the tool being executed (if any)
• last_message: most recent assistant text (for progressive display)
• error: error message if state is error

This gives the user live visibility into long-running tasks without streaming complexity.

Goal

Embed an LLM-powered agent directly inside sovereign_browser. The user types ; <message> in the URL bar to talk to the agent. The agent has first-class access to the browser's own MCP tool set (snapshot, click, eval, tabs, etc.) plus full filesystem and shell access (the browser runs in a dedicated Qubes qube, so arbitrary command execution is acceptable and intended).

Provider config (base URL, API key, model name) is managed on a new sovereign://agents internal page. Chat history is persisted per-session in SQLite so it can be fed back as context on follow-up messages.

User-facing flow

flowchart LR
    User[User types in URL bar] --> Check{Starts with semicolon?}
    Check -- yes --> Agent[Embedded Agent Chat UI]
    Check -- no --> Nav[Normal navigation]
    Agent --> LLM[OpenAI-compatible API call]
    LLM --> ToolLoop{Tool calls in response?}
    ToolLoop -- yes --> Dispatch[agent_tools_dispatch via internal MCP loopback]
    Dispatch --> ToolLoop
    ToolLoop -- no --> Render[Render assistant message in chat UI]
    Render --> User

Architecture

flowchart TB
    subgraph URLBar
        Entry[GtkEntry on_url_activate]
    end
    subgraph AgentModule
        Router[agent_chat_route_input]
        Client[agent_llm.c — HTTP client]
        Loop[agent_loop.c — tool-call loop]
        Store[agent_chat_store.c — SQLite persistence]
    end
    subgraph BrowserCore
        Tools[agent_tools_dispatch]
        FsTools[agent_fs_tools.c — fs + shell tools]
        Bridge[nostr_bridge.c — sovereign:// scheme]
    end
    subgraph External
        API[OpenAI-compatible API]
    end
    subgraph UI
        ChatPage[sovereign://agents/chat — HTML/JS chat UI]
        ConfigPage[sovereign://agents — provider config]
    end

    Entry --> Router
    Router -->|chat message| Client
    Client -->|HTTPS POST| API
    API -->|JSON response| Client
    Client --> Loop
    Loop -->|tool call| Tools
    Loop -->|fs/shell call| FsTools
    Tools --> BrowserCore
    Loop --> Store
    Store --> ChatPage
    Bridge --> ChatPage
    Bridge --> ConfigPage

Key design decisions

1. Reuse agent_tools_dispatch() — The embedded agent calls the exact same C dispatch function the external MCP server uses. No parallel tool implementation. The agent passes conn = NULL so async JS tools use the sync agent_js_eval_sync() path (same as the MCP HTTP handler).

2. New system tools (agent_fs_tools.c) — fs_read, fs_write, fs_list, fs_mkdir, fs_delete, shell_exec. These are registered alongside the existing browser tools so both the embedded agent and external MCP clients can use them. Full shell access — the Qubes qube provides the sandbox.

3. OpenAI-compatible HTTP client (agent_llm.c) — Uses libsoup-3.0 (already linked) to POST to {base_url}/chat/completions with the tools array built from the same tool_defs[] catalog in agent_mcp.c. Streams or polls; parses tool calls from the response. One client covers OpenAI, OpenRouter, Ollama, LM Studio, Groq, etc. via base URL + key.

4. Tool-call loop (agent_loop.c) — Standard ReAct loop, runs on a background GThread (see Concurrency model above):

   • Build messages array (system prompt + persisted history + new user msg).
   • Call LLM (blocking HTTP on the background thread).
   • If response contains tool_calls, dispatch each:
     • Browser tools → hop to GTK main thread via g_idle_add() + wait on GAsyncQueue (WebKitGTK is not thread-safe).
     • Filesystem/shell tools → run directly on the background thread.
   • Append tool results to messages, call LLM again.
   • Repeat until no more tool calls → render final assistant text.
   • Cap at N iterations (default 100, configurable) to prevent infinite loops. Check cancel flag at the top of each iteration.

5. Chat persistence (agent_chat_store.c) — New SQLite tables (agent_sessions, agent_messages) in the per-user browser.db. Each session has an id, title, created_at. Messages store role (user/assistant/tool), content, and tool-call JSON. The URL-bar ; command reuses the most recent session (or starts a new one if none exists).

6. sovereign://agents config page — Rendered by nostr_bridge.c (same pattern as sovereign://settings). Fields: provider name, base URL, API key, model name, system prompt (optional). Saved via sovereign://agents/set?key=...&value=... to the key_value table (existing db_kv_set). API key stored in the key_value table; since this is a dedicated qube, that's acceptable.

7. sovereign://agents/chat chat UI — A lightweight HTML/JS page (rendered by nostr_bridge.c) that:

   • Shows the message history (loaded from SQLite via a sovereign://agents/messages?session=... endpoint).
   • Has an input box for follow-up messages (POSTed to sovereign://agents/send?session=...&text=...).
   • Polls sovereign://agents/status?session=... for in-progress tool calls and streams the assistant's final response.
   • The URL-bar ; shortcut navigates here with the message pre-filled and auto-submits.

8. URL-bar ; routing — In on_url_activate, check if text[0] == ';'. If so, extract the message (text + 1, trimmed) and call agent_chat_route_input(message) instead of normalize_url(). The function opens sovereign://agents/chat in the active tab (or a new tab if the active tab isn't already the chat page) and kicks off the agent loop. If the message is empty (; alone), just open the chat page without sending.

9. System prompt — A default system prompt explains the agent's capabilities: it controls a web browser via MCP tools, has filesystem and shell access, and should use the snapshot+ref pattern for page interaction. The user can override this on sovereign://agents.

New files

File	Purpose
src/agent_llm.h / src/agent_llm.c	OpenAI-compatible HTTP client (libsoup). Sends chat-completions request with tools, parses response + tool_calls.
src/agent_loop.h / src/agent_loop.c	ReAct tool-call loop. Orchestrates LLM calls ↔ tool dispatch.
src/agent_chat_store.h / src/agent_chat_store.c	SQLite persistence for chat sessions and messages.
src/agent_fs_tools.h / src/agent_fs_tools.c	Filesystem + shell tools (fs_read, fs_write, fs_list, fs_mkdir, fs_delete, shell_exec).
src/agent_chat.h / src/agent_chat.c	High-level entry point: agent_chat_route_input(), session management, bridges URL-bar → loop → UI.

Modified files

File	Change
src/tab_manager.c	on_url_activate: detect ; prefix, route to agent_chat_route_input().
src/nostr_bridge.c	Add sovereign://agents, sovereign://agents/chat, sovereign://agents/set, sovereign://agents/messages, sovereign://agents/send, sovereign://agents/status route handlers.
src/agent_mcp.c	Export tool_defs[] / build_tools_list() (or move to a shared header) so agent_llm.c can build the OpenAI tools array from the same catalog. Add fs/shell tool defs.
src/agent_tools.c	Dispatch fs/shell tools (or route them via a new agent_fs_tools_dispatch() called from the same dispatcher).
src/db.c / src/db.h	Add agent_sessions + agent_messages tables and CRUD functions.
src/settings.h / src/settings.c	Add agent provider settings fields (base_url, api_key, model, system_prompt).
Makefile	Add new .c files to SRC.

SQLite schema

CREATE TABLE IF NOT EXISTS agent_sessions (
    id          TEXT PRIMARY KEY,   -- UUID
    title       TEXT,
    created_at  INTEGER,
    updated_at  INTEGER
);

CREATE TABLE IF NOT EXISTS agent_messages (
    id          INTEGER PRIMARY KEY AUTOINCREMENT,
    session_id  TEXT,
    role        TEXT,               -- 'user' | 'assistant' | 'tool' | 'system'
    content     TEXT,               -- message text (or tool result JSON)
    tool_calls  TEXT,               -- JSON array of tool calls (assistant msgs)
    tool_call_id TEXT,              -- for role='tool': which call this answers
    created_at  INTEGER,
    FOREIGN KEY (session_id) REFERENCES agent_sessions(id)
);
CREATE INDEX IF NOT EXISTS idx_agent_messages_session
    ON agent_messages(session_id, created_at);

Tool catalog additions (fs + shell)

Tool	Description
fs_read	Read a file's contents (text). Params: path.
fs_write	Write text to a file (overwrite). Params: path, content.
fs_list	List directory entries. Params: path.
fs_mkdir	Create a directory (recursive). Params: path.
fs_delete	Delete a file or empty directory. Params: path.
shell_exec	Run a shell command, return stdout+stderr+exit code. Params: command, timeout_ms (default 30000).

OpenAI tools array format

The agent_llm.c client builds the tools field from the shared tool_defs[] array, converting each entry to the OpenAI format:

{
  "type": "function",
  "function": {
    "name": "snapshot",
    "description": "Get the accessibility tree...",
    "parameters": { ...schema... }
  }
}

Concurrency note

See the Concurrency model section at the top of this document for the full design. Summary: the agent loop runs on a background GThread. LLM HTTP calls block in-thread. Browser tool calls hop to the GTK main thread via g_idle_add() + GAsyncQueue (WebKitGTK is not thread-safe). Filesystem and shell tools run directly on the background thread. The UI polls sovereign://agents/status for live progress. A cancel flag allows the user to stop long-running tasks.

Phasing

The work breaks into two phases that can be implemented sequentially:

Phase 1 — Foundation: fs/shell tools, LLM client, tool-call loop, chat persistence, sovereign://agents config page, URL-bar ; routing, basic chat UI. After Phase 1 the example task ("save links to ~/temp/links.txt") works end-to-end.

Phase 2 — Polish: streaming responses, tool-call progress display in the chat UI, session list / history sidebar, editable system prompt per session, multi-session support from the URL bar, error recovery UI.