c-relay-pg/plans/main_thread_offload_plan.md

Main Thread CPU Offload Plan

Problem Statement

The main c_relay_pg thread consumes ~56% CPU while the DB worker threads (db-read-1..4, db-write) sit near 0%. All protocol handling, JSON parsing, event validation, subscription matching, and message queueing happens synchronously inside nostr_relay_callback() on the main libwebsockets event-loop thread.

Current Architecture

flowchart TD
    LWS[lws_service - main thread] --> CB[nostr_relay_callback]
    CB --> PARSE[JSON parse - cJSON_Parse]
    CB --> VALIDATE[Signature verify - nostr_validate_unified_request]
    CB --> STORE[store_event - builds payload]
    STORE --> SYNC_WRITE[thread_pool_execute_store_event_sync]
    SYNC_WRITE --> |blocks main thread| DB_WRITE[db-write thread]
    DB_WRITE --> |signal| SYNC_WRITE
    CB --> REQ[handle_req_message - builds SQL]
    REQ --> SYNC_READ[thread_pool_execute_req_sync]
    SYNC_READ --> |blocks main thread| DB_READ[db-read-N thread]
    DB_READ --> |signal| SYNC_READ
    CB --> BROADCAST[broadcast_event_to_subscriptions]
    BROADCAST --> QUEUE[queue_message per subscriber]
    CB --> WRITEABLE[LWS_CALLBACK_SERVER_WRITEABLE]
    WRITEABLE --> DRAIN[process_message_queue - lws_write]

Why the main thread is hot

Work item	Where	Cost
JSON parsing	cJSON_Parse in LWS_CALLBACK_RECEIVE	Medium - per message
Signature verification	nostr_validate_unified_request - ed25519 crypto	High - per EVENT
SQL query building	handle_req_message filter-to-SQL loop	Low-Medium
Sync DB wait	thread_pool_execute_*_sync - pthread_cond_wait	Blocks but yields CPU
Result iteration + expiration check	Row loop in handle_req_message with cJSON_Parse per row	Medium-High for large result sets
Subscription matching	broadcast_event_to_subscriptions - filter matching	Medium - scales with subscriber count
Message serialization + queueing	snprintf + queue_message_take_ownership per subscriber	Medium
Config lookups	get_config_int/bool called repeatedly in hot paths	Low but frequent

Key constraint: libwebsockets is single-threaded

libwebsockets requires that all lws_write, lws_callback_on_writable, and lws_close_reason calls happen from the service thread (the thread running lws_service). This means we cannot directly write to WebSocket connections from worker threads. However, we can do computation on worker threads and post results back to the main thread for I/O.

Offload Strategy

Phase 1: Async EVENT Processing (Highest Impact)

Convert EVENT handling from synchronous to async. Currently the main thread does: parse → validate → store → broadcast, all blocking. Instead:

1. Main thread: Parse JSON (fast), extract event ID for dedup check, then submit a job to a new "event processing" worker thread
2. Worker thread: Signature verification (expensive crypto), store_event (already goes to DB thread), prepare broadcast payload
3. Main thread callback: Receive result via wake_loop_cb + lws_cancel_service, send OK response and broadcast to subscribers

flowchart TD
    LWS[lws_service - main thread] --> RECEIVE[LWS_CALLBACK_RECEIVE]
    RECEIVE --> PARSE[JSON parse + dedup check]
    PARSE --> SUBMIT[Submit to event-worker queue]
    SUBMIT --> LWS
    
    WORKER[event-worker thread] --> VERIFY[Signature verification]
    VERIFY --> STORE_DB[store_event via DB pool]
    STORE_DB --> PREP[Prepare broadcast payload]
    PREP --> RESULT_Q[Push result to completion queue]
    RESULT_Q --> WAKE[lws_cancel_service]
    
    LWS2[lws_service wakes] --> POLL[Poll completion queue]
    POLL --> OK[Send OK response via queue_message]
    POLL --> BCAST[broadcast_event_to_subscriptions]

What this offloads: ed25519 signature verification (~the most expensive per-event operation), event classification, tag serialization, and the synchronous DB store wait.

What stays on main thread: JSON parse (needed to extract event ID for dedup), OK response writing, broadcast fan-out (requires lws access).

Phase 2: Async REQ Query Execution (Medium Impact)

Convert REQ handling from sync to async:

1. Main thread: Parse filters, build SQL, create subscription, submit query job
2. DB reader thread: Execute query (already happens, but currently blocks main thread via _sync)
3. Main thread callback: Iterate results, queue EVENT messages, send EOSE

This is simpler than Phase 1 because the thread pool already supports async submission via thread_pool_submit_read with a result_cb. The _sync wrappers just add a condvar wait on top. We need to:

• Use thread_pool_submit_read directly instead of thread_pool_execute_req_sync
• Store pending REQ context (sub_id, wsi, pss) so the callback can complete the work
• In the result callback, push results to a completion queue and call lws_cancel_service
• On the main thread, drain the completion queue and send EVENT + EOSE messages

Phase 3: Reduce Per-Row Overhead in REQ Results (Low-Medium Impact)

Currently each row from a REQ query gets cJSON_Parse just to check NIP-40 expiration. This is wasteful:

• Option A: Add an expiration column to the events table so expiration filtering can be done in SQL
• Option B: Store expiration timestamp in a fast-parse format (extract during INSERT, store as integer column)
• Option C: Use string search on the raw JSON for the expiration tag instead of full parse

Phase 4: Config Value Caching (Low Impact, Easy Win)

get_config_int and get_config_bool are called on every message in hot paths. These do SQLite queries. Cache config values in memory with a TTL or invalidation signal, so the main loop only refreshes them periodically (already done for debug_level every 60s — extend to all hot-path config values).

Implementation Priority

Phase	Impact	Risk	Complexity
Phase 1: Async EVENT	High - removes crypto from main thread	Medium - async state management	Medium-High
Phase 2: Async REQ	Medium - unblocks main thread during queries	Low - infrastructure exists	Medium
Phase 3: Expiration optimization	Low-Medium - reduces per-row parse cost	Low	Low
Phase 4: Config caching	Low - reduces DB round-trips	Very Low	Low

Detailed Design: Phase 1 (Async EVENT Processing)

New Components

Completion Queue (src/completion_queue.h/.c)

A thread-safe FIFO queue for posting results from worker threads back to the main thread:

typedef struct {
    int type;           // COMPLETION_TYPE_EVENT_RESULT, COMPLETION_TYPE_REQ_RESULT, etc.
    void* data;         // Type-specific result data
    struct lws* wsi;    // Target WebSocket connection
    void* pss;          // Per-session data
} completion_item_t;

int completion_queue_init(void);
int completion_queue_push(completion_item_t* item);
completion_item_t* completion_queue_pop(void);  // Non-blocking
void completion_queue_shutdown(void);

Event Worker Thread

A dedicated pthread that processes EVENT validation/storage:

typedef struct {
    cJSON* event;           // Parsed event JSON - ownership transferred
    cJSON* full_message;    // Full message JSON for context
    struct lws* wsi;
    void* pss;
    char sub_id[64];        // For response routing
} event_work_item_t;

Main Loop Integration

Add a completion queue drain step to the main event loop. After lws_service returns (either from timeout or lws_cancel_service wake), check the completion queue:

while (g_server_running && !g_shutdown_flag) {
    int result = lws_service(ws_context, 1000);
    
    // NEW: Drain completion queue
    completion_item_t* item;
    while ((item = completion_queue_pop()) != NULL) {
        process_completion(item);  // Send OK, broadcast, etc.
        free(item);
    }
    
    // ... existing periodic checks ...
}

Changes to Existing Code

1. nostr_relay_callback EVENT path: After JSON parse and dedup check, instead of calling nostr_validate_unified_request + store_event + broadcast_event_to_subscriptions synchronously, submit an event_work_item_t to the event worker queue and return 0 immediately.

2. store_event: No changes needed — it already uses thread_pool_execute_store_event_sync which will run on the DB writer thread. The event worker thread will call it.

3. broadcast_event_to_subscriptions: No changes needed — it will be called from the main thread when processing the completion item, which is correct since it calls queue_message_take_ownership (requires lws thread).

Thread Safety Considerations

• The cJSON* event object must be fully owned by the worker thread during processing. The main thread must not access it after submission.
• The wsi and pss pointers could become invalid if the client disconnects while the event is being processed. The completion handler must validate that the connection is still alive before sending the OK response.
• A generation counter or epoch on pss can detect stale references.

What Cannot Be Offloaded

• lws_write / queue_message: Must happen on the lws service thread
• lws_callback_on_writable: Must happen on the lws service thread
• lws_close_reason: Must happen on the lws service thread
• Subscription list iteration for broadcast: Accesses lws_wsi_user which is lws-internal

These are fundamental libwebsockets constraints. The pattern is always: do computation off-thread, post result to completion queue, wake main thread, do I/O on main thread.

Expected Impact

With Phase 1 alone, the main thread would no longer perform:

• ed25519 signature verification (~100-500μs per event depending on CPU)
• Synchronous DB store wait (~50-200μs per event)
• Event classification, tag serialization, JSON serialization for storage

This should reduce main-thread CPU by roughly 30-50% for EVENT-heavy workloads, shifting that work to the event worker thread and DB threads.