c-relay/plans/main_thread_offload_plan.md

# Main Thread CPU Offload Plan

## Problem Statement

The main `c_relay` thread consumes ~56% CPU while the DB worker threads (`db-read-1..4`, `db-write`) sit near 0%. All protocol handling, JSON parsing, event validation, subscription matching, and message queueing happens synchronously inside `nostr_relay_callback()` on the main libwebsockets event-loop thread.

## Current Architecture

```mermaid
flowchart TD
    LWS[lws_service - main thread] --> CB[nostr_relay_callback]
    CB --> PARSE[JSON parse - cJSON_Parse]
    CB --> VALIDATE[Signature verify - nostr_validate_unified_request]
    CB --> STORE[store_event - builds payload]
    STORE --> SYNC_WRITE[thread_pool_execute_store_event_sync]
    SYNC_WRITE --> |blocks main thread| DB_WRITE[db-write thread]
    DB_WRITE --> |signal| SYNC_WRITE
    CB --> REQ[handle_req_message - builds SQL]
    REQ --> SYNC_READ[thread_pool_execute_req_sync]
    SYNC_READ --> |blocks main thread| DB_READ[db-read-N thread]
    DB_READ --> |signal| SYNC_READ
    CB --> BROADCAST[broadcast_event_to_subscriptions]
    BROADCAST --> QUEUE[queue_message per subscriber]
    CB --> WRITEABLE[LWS_CALLBACK_SERVER_WRITEABLE]
    WRITEABLE --> DRAIN[process_message_queue - lws_write]
```

### Why the main thread is hot

| Work item | Where | Cost |
|-----------|-------|------|
| JSON parsing | `cJSON_Parse` in `LWS_CALLBACK_RECEIVE` | Medium - per message |
| Signature verification | `nostr_validate_unified_request` - ed25519 crypto | **High** - per EVENT |
| SQL query building | `handle_req_message` filter-to-SQL loop | Low-Medium |
| Sync DB wait | `thread_pool_execute_*_sync` - pthread_cond_wait | Blocks but yields CPU |
| Result iteration + expiration check | Row loop in `handle_req_message` with `cJSON_Parse` per row | Medium-High for large result sets |
| Subscription matching | `broadcast_event_to_subscriptions` - filter matching | Medium - scales with subscriber count |
| Message serialization + queueing | `snprintf` + `queue_message_take_ownership` per subscriber | Medium |
| Config lookups | `get_config_int/bool` called repeatedly in hot paths | Low but frequent |

### Key constraint: libwebsockets is single-threaded

libwebsockets requires that **all `lws_write`, `lws_callback_on_writable`, and `lws_close_reason` calls happen from the service thread** (the thread running `lws_service`). This means we cannot directly write to WebSocket connections from worker threads. However, we **can** do computation on worker threads and post results back to the main thread for I/O.

## Offload Strategy

### Phase 1: Async EVENT Processing (Highest Impact)

Convert EVENT handling from synchronous to async. Currently the main thread does: parse → validate → store → broadcast, all blocking. Instead:

1. **Main thread**: Parse JSON (fast), extract event ID for dedup check, then submit a job to a new "event processing" worker thread
2. **Worker thread**: Signature verification (expensive crypto), store_event (already goes to DB thread), prepare broadcast payload
3. **Main thread callback**: Receive result via `wake_loop_cb` + `lws_cancel_service`, send OK response and broadcast to subscribers

```mermaid
flowchart TD
    LWS[lws_service - main thread] --> RECEIVE[LWS_CALLBACK_RECEIVE]
    RECEIVE --> PARSE[JSON parse + dedup check]
    PARSE --> SUBMIT[Submit to event-worker queue]
    SUBMIT --> LWS
    
    WORKER[event-worker thread] --> VERIFY[Signature verification]
    VERIFY --> STORE_DB[store_event via DB pool]
    STORE_DB --> PREP[Prepare broadcast payload]
    PREP --> RESULT_Q[Push result to completion queue]
    RESULT_Q --> WAKE[lws_cancel_service]
    
    LWS2[lws_service wakes] --> POLL[Poll completion queue]
    POLL --> OK[Send OK response via queue_message]
    POLL --> BCAST[broadcast_event_to_subscriptions]
```

**What this offloads**: ed25519 signature verification (~the most expensive per-event operation), event classification, tag serialization, and the synchronous DB store wait.

**What stays on main thread**: JSON parse (needed to extract event ID for dedup), OK response writing, broadcast fan-out (requires lws access).

### Phase 2: Async REQ Query Execution (Medium Impact)

Convert REQ handling from sync to async:

1. **Main thread**: Parse filters, build SQL, create subscription, submit query job
2. **DB reader thread**: Execute query (already happens, but currently blocks main thread via `_sync`)
3. **Main thread callback**: Iterate results, queue EVENT messages, send EOSE

This is simpler than Phase 1 because the thread pool already supports async submission via `thread_pool_submit_read` with a `result_cb`. The `_sync` wrappers just add a condvar wait on top. We need to:
- Use `thread_pool_submit_read` directly instead of `thread_pool_execute_req_sync`
- Store pending REQ context (sub_id, wsi, pss) so the callback can complete the work
- In the result callback, push results to a completion queue and call `lws_cancel_service`
- On the main thread, drain the completion queue and send EVENT + EOSE messages

### Phase 3: Reduce Per-Row Overhead in REQ Results (Low-Medium Impact)

Currently each row from a REQ query gets `cJSON_Parse` just to check NIP-40 expiration. This is wasteful:

- Option A: Add an `expiration` column to the events table so expiration filtering can be done in SQL
- Option B: Store expiration timestamp in a fast-parse format (extract during INSERT, store as integer column)
- Option C: Use string search on the raw JSON for the expiration tag instead of full parse

### Phase 4: Config Value Caching (Low Impact, Easy Win)

`get_config_int` and `get_config_bool` are called on every message in hot paths. These do SQLite queries. Cache config values in memory with a TTL or invalidation signal, so the main loop only refreshes them periodically (already done for `debug_level` every 60s — extend to all hot-path config values).

## Implementation Priority

| Phase | Impact | Risk | Complexity |
|-------|--------|------|------------|
| Phase 1: Async EVENT | **High** - removes crypto from main thread | Medium - async state management | Medium-High |
| Phase 2: Async REQ | **Medium** - unblocks main thread during queries | Low - infrastructure exists | Medium |
| Phase 3: Expiration optimization | **Low-Medium** - reduces per-row parse cost | Low | Low |
| Phase 4: Config caching | **Low** - reduces DB round-trips | Very Low | Low |

## Detailed Design: Phase 1 (Async EVENT Processing)

### New Components

#### Completion Queue (`src/completion_queue.h/.c`)
A thread-safe FIFO queue for posting results from worker threads back to the main thread:

```c
typedef struct {
    int type;           // COMPLETION_TYPE_EVENT_RESULT, COMPLETION_TYPE_REQ_RESULT, etc.
    void* data;         // Type-specific result data
    struct lws* wsi;    // Target WebSocket connection
    void* pss;          // Per-session data
} completion_item_t;

int completion_queue_init(void);
int completion_queue_push(completion_item_t* item);
completion_item_t* completion_queue_pop(void);  // Non-blocking
void completion_queue_shutdown(void);
```

#### Event Worker Thread
A dedicated pthread that processes EVENT validation/storage:

```c
typedef struct {
    cJSON* event;           // Parsed event JSON - ownership transferred
    cJSON* full_message;    // Full message JSON for context
    struct lws* wsi;
    void* pss;
    char sub_id[64];        // For response routing
} event_work_item_t;
```

#### Main Loop Integration
Add a completion queue drain step to the main event loop. After `lws_service` returns (either from timeout or `lws_cancel_service` wake), check the completion queue:

```c
while (g_server_running && !g_shutdown_flag) {
    int result = lws_service(ws_context, 1000);
    
    // NEW: Drain completion queue
    completion_item_t* item;
    while ((item = completion_queue_pop()) != NULL) {
        process_completion(item);  // Send OK, broadcast, etc.
        free(item);
    }
    
    // ... existing periodic checks ...
}
```

### Changes to Existing Code

1. **`nostr_relay_callback` EVENT path**: After JSON parse and dedup check, instead of calling `nostr_validate_unified_request` + `store_event` + `broadcast_event_to_subscriptions` synchronously, submit an `event_work_item_t` to the event worker queue and return 0 immediately.

2. **`store_event`**: No changes needed — it already uses `thread_pool_execute_store_event_sync` which will run on the DB writer thread. The event worker thread will call it.

3. **`broadcast_event_to_subscriptions`**: No changes needed — it will be called from the main thread when processing the completion item, which is correct since it calls `queue_message_take_ownership` (requires lws thread).

### Thread Safety Considerations

- The `cJSON* event` object must be fully owned by the worker thread during processing. The main thread must not access it after submission.
- The `wsi` and `pss` pointers could become invalid if the client disconnects while the event is being processed. The completion handler must validate that the connection is still alive before sending the OK response.
- A generation counter or epoch on `pss` can detect stale references.

## What Cannot Be Offloaded

- **`lws_write` / `queue_message`**: Must happen on the lws service thread
- **`lws_callback_on_writable`**: Must happen on the lws service thread  
- **`lws_close_reason`**: Must happen on the lws service thread
- **Subscription list iteration for broadcast**: Accesses `lws_wsi_user` which is lws-internal

These are fundamental libwebsockets constraints. The pattern is always: do computation off-thread, post result to completion queue, wake main thread, do I/O on main thread.

## Expected Impact

With Phase 1 alone, the main thread would no longer perform:
- ed25519 signature verification (~100-500μs per event depending on CPU)
- Synchronous DB store wait (~50-200μs per event)
- Event classification, tag serialization, JSON serialization for storage

This should reduce main-thread CPU by roughly 30-50% for EVENT-heavy workloads, shifting that work to the event worker thread and DB threads.