c-relay/plans/sqlite_off_main_thread_plan.md

# Plan: Remove All SQLite from lws-main Thread

## Problem Statement

After implementing the 4 offload fixes (v2.1.4), `lws-main` still consumes **53.87% avg CPU** (down from 68.71%). Perf profiling shows **100% of the top symbols are SQLite** — B-tree traversal, page cache, VFS reads. The worker threads (`db-read`, `db-write`, `event-worker`) are nearly idle at 0.0–0.1%.

The goal is to eliminate all synchronous SQLite access from `lws-main` so it only does: JSON parsing, WebSocket I/O, subscription matching (in-memory), and message queuing.

## Thread Architecture Change

Reduce from 7 threads to 4:

```
Current:  lws-main, db-read-1, db-read-2, db-read-3, db-read-4, db-write, event-worker
Proposed: lws-main, db-read, db-write, event-worker
```

The 4 reader threads are overkill — they average 0.07–0.10% CPU each. A single `db-read` thread is sufficient for the current workload. This can be made configurable later if needed.

## Remaining SQLite Calls on lws-main

Every synchronous SQLite call that currently runs on `lws-main`:

### Hot Path — Per-Event (every incoming EVENT message)

| Call | Location | Frequency | Cost |
|------|----------|-----------|------|
| `event_id_exists_in_db()` | websockets.c:1324, 2134 | Every EVENT | ~10-50us per B-tree lookup |
| `get_config_value("relay_pubkey")` | websockets.c:2050 | Every kind-14 EVENT | SQLite SELECT |
| `get_config_value("admin_pubkey")` | websockets.c:2073 | Every kind-23456 EVENT | SQLite SELECT |

### Hot Path — Per-Connection (every new WebSocket connection)

| Call | Location | Frequency | Cost |
|------|----------|-----------|------|
| `get_config_bool("trust_proxy_headers")` | websockets.c:1071 | Every connection | SQLite SELECT |
| `get_config_bool("nip42_auth_required_events")` | websockets.c:1134 | Every connection | SQLite SELECT |
| `get_config_bool("nip42_auth_required_subscriptions")` | websockets.c:1135 | Every connection | SQLite SELECT |

### Hot Path — Per-REQ (every subscription request)

| Call | Location | Frequency | Cost |
|------|----------|-----------|------|
| `get_config_bool("expiration_enabled")` | main.c:1424 | Every REQ | SQLite SELECT |
| `get_config_bool("expiration_filter")` | main.c:1425 | Every REQ | SQLite SELECT |
| `check_database_auth_rules()` | websockets.c:1706 | Every REQ when WoT enabled | SQLite SELECT |

### Medium Path — Per-COUNT (NIP-45)

| Call | Location | Frequency | Cost |
|------|----------|-----------|------|
| `thread_pool_execute_count_sync()` | websockets.c:3689 | Every COUNT message | Blocks main thread waiting for db-read |

### Low Path — Periodic (every 60s or on timer)

| Call | Location | Frequency | Cost |
|------|----------|-----------|------|
| `refresh_hot_config_if_needed()` | websockets.c:228-247 | Every 5s | 8 SQLite SELECTs |
| `generate_and_post_status_event()` | websockets.c:3213 | Every N hours | store_event + broadcast |
| `ip_ban_cleanup/log_stats` | websockets.c:3004-3005 | Every 60s | SQLite reads/writes |

### Per-EVENT — Validator Path (called from event-worker AND sync path)

| Call | Location | Frequency | Cost |
|------|----------|-----------|------|
| `get_config_value("admin_pubkey")` | request_validator.c:308 | Every kind-23456 EVENT | SQLite SELECT |
| `get_config_value("nip42_auth_enabled")` | request_validator.c:326 | Every EVENT | SQLite SELECT |
| `get_config_bool("pow_enabled")` | request_validator.c:370 | Every EVENT | SQLite SELECT |
| `get_config_int("pow_min_difficulty")` | request_validator.c:371 | Every EVENT | SQLite SELECT |
| `get_config_int("pow_validation_flags")` | request_validator.c:372 | Every EVENT | SQLite SELECT |
| `get_config_int("nip40_expiration_grace_period")` | request_validator.c:432 | Every EVENT with expiration | SQLite SELECT |
| `get_config_value("auth_enabled")` | request_validator.c:212 | Every EVENT (reload_auth_config) | SQLite SELECT |
| `get_config_value("auth_rules_enabled")` | request_validator.c:220 | Every EVENT (reload_auth_config) | SQLite SELECT |
| `db_count_active_whitelist_rules()` | request_validator.c:536 | Every 5s (cached) | Opens separate SQLite connection |

**Note:** These run on the **event-worker** thread for async-eligible events, but on **lws-main** for special kinds (14, 1059, 23456) that go through the sync validation path. They also run on lws-main for the `nostr_validate_unified_request()` call in the sync fallback.

### Per-EVENT — NIP-13 PoW Validation

| Call | Location | Frequency | Cost |
|------|----------|-----------|------|
| `get_config_bool("pow_enabled")` | nip013.c:27 | Every EVENT | SQLite SELECT |
| `get_config_int("pow_min_difficulty")` | nip013.c:28 | Every EVENT | SQLite SELECT |
| `get_config_value("pow_mode")` | nip013.c:29 | Every EVENT | SQLite SELECT |

### Per-Broadcast — NIP-40 Expiration Check

| Call | Location | Frequency | Cost |
|------|----------|-----------|------|
| `get_config_bool("expiration_enabled")` | subscriptions.c:764 | Every broadcast | SQLite SELECT |
| `get_config_bool("expiration_filter")` | subscriptions.c:765 | Every broadcast | SQLite SELECT |

### Per-NIP-11 Request (HTTP, not WebSocket)

| Call | Location | Frequency | Cost |
|------|----------|-----------|------|
| 15+ `get_config_value()` calls | nip011.c:96-110 | Every NIP-11 HTTP request | 15 SQLite SELECTs |
| 10+ `get_config_int()`/`get_config_bool()` calls | nip011.c:113-123 | Every NIP-11 HTTP request | 10 SQLite SELECTs |

### Per-NIP-42 Auth Event

| Call | Location | Frequency | Cost |
|------|----------|-----------|------|
| `get_config_value("relay_url")` | nip042.c:99 | Every AUTH event | SQLite SELECT |
| `get_config_int("relay_port")` | nip042.c:103 | Every AUTH event (fallback) | SQLite SELECT |

### Per-NIP-40 Expiration Check (EVENT submission)

| Call | Location | Frequency | Cost |
|------|----------|-----------|------|
| 5 `get_config_bool()`/`get_config_int()` calls | nip040.c:37-41 | Every EVENT with expiration tag | 5 SQLite SELECTs |

### Subscription Lifecycle (per sub create/close/disconnect)

| Call | Location | Frequency | Cost |
|------|----------|-----------|------|
| `db_log_subscription_created()` | subscriptions.c:1043 | Every REQ | SQLite INSERT |
| `db_log_subscription_closed()` | subscriptions.c:1053 | Every CLOSE | SQLite INSERT + UPDATE |
| `db_log_subscription_disconnected()` | subscriptions.c:1060 | Every disconnect | SQLite UPDATE + INSERT |
| `db_update_subscription_events_sent()` | subscriptions.c:1088 | Every broadcast match | SQLite UPDATE |
| `db_cleanup_orphaned_subscriptions()` | subscriptions.c:1100 | Startup only | SQLite DELETE |

### IP Ban System (per-connection and periodic)

| Call | Location | Frequency | Cost |
|------|----------|-----------|------|
| `get_config_value("idle_ban_whitelist")` | ip_ban.c:254 | Every ban check | SQLite SELECT |
| `get_config_bool("auth_fail_ban_enabled")` | ip_ban.c:301, 343 | Every ban check + failure | SQLite SELECT |
| `get_config_bool("idle_ban_enabled")` | ip_ban.c:307, 394 | Every ban check + idle failure | SQLite SELECT |
| 6+ `get_config_int()` calls | ip_ban.c:345-399, 464-465 | Every failure recording | SQLite SELECTs |
| `db_prepare()`/`db_step_stmt()` for ip_bans | ip_ban.c:141-238 | Load at startup + periodic save | SQLite reads/writes |

### Special Kind Sync Path (kinds 14, 1059, 23456)

| Call | Location | Frequency | Cost |
|------|----------|-----------|------|
| `store_event()` | websockets.c:1548-1629 | Every special-kind EVENT | Full sync store |
| `process_admin_event_in_config()` | websockets.c:1468 | Every kind-23456 | Config DB writes |
| `process_nip17_admin_message()` | websockets.c:1523 | Every kind-1059 when enabled | Decrypt + DB |
| `process_dm_stats_command()` | websockets.c:3240 | Admin DM stats | Decrypt + query + encrypt |
| `generate_stats_json()` | api.c:1167 (via dm_admin.c:805) | Admin stats command | Multiple SQLite queries |

### DM Admin Path (called from lws-main for kind 1059/14)

| Call | Location | Frequency | Cost |
|------|----------|-----------|------|
| `get_config_value("relay_pubkey")` | dm_admin.c:358, 451 | Every admin DM | SQLite SELECT |
| `get_config_value("admin_pubkey")` | dm_admin.c:504 | Every admin DM | SQLite SELECT |
| `get_config_int("nip59_timestamp_max_delay_sec")` | dm_admin.c:373 | Every admin DM | SQLite SELECT |
| `get_config_int("wot_enabled")` | dm_admin.c:613, 640, 660 | WoT commands | SQLite SELECT |

### API/Monitoring (called from lws-main periodic timer)

| Call | Location | Frequency | Cost |
|------|----------|-----------|------|
| `generate_stats_json()` | api.c:1167 | Status post + admin stats | ~10 SQLite queries |
| `query_time_based_statistics()` | api.c:102 | Monitoring events | SQLite queries |
| `query_subscription_details()` | api.c:197 | Monitoring events | SQLite queries |
| `get_config_value("relay_pubkey")` | api.c:419, 821, 1291 | Every monitoring event | SQLite SELECT |
| `get_config_int("kind_1_status_posts_hours")` | api.c:547 | Status post check | SQLite SELECT |
| `get_config_int("kind_24567_reporting_throttle_sec")` | api.c:60 | Monitoring throttle | SQLite SELECT |

## Implementation Plan

### Fix 1: Create Global Config Cache — Eliminate ALL get_config_* SQLite Calls

**Impact: Very High** — eliminates 50+ distinct `get_config_value/int/bool` call sites across 8 source files

The audit found `get_config_value/int/bool` calls in: `websockets.c`, `main.c`, `subscriptions.c`, `request_validator.c`, `nip011.c`, `nip013.c`, `nip040.c`, `nip042.c`, `ip_ban.c`, `dm_admin.c`, `api.c`, `config.c`. Every single one does a `SELECT value FROM config WHERE key=?` SQLite query.

**Approach:** Replace `get_config_value_from_table()` with an in-memory hash map that is loaded once at startup and refreshed periodically (every 5s) or on admin config change events.

**Changes:**

1. **New: `config_cache` in config.c** — A simple hash map or fixed array of key-value pairs loaded from the config table:
   ```c
   typedef struct {
       char key[64];
       char value[512];
   } config_cache_entry_t;
   
   static config_cache_entry_t g_config_cache[256];
   static int g_config_cache_count = 0;
   static time_t g_config_cache_last_refresh = 0;
   static pthread_mutex_t g_config_cache_mutex;
   #define CONFIG_CACHE_TTL_SEC 5
   ```

2. **Modify `get_config_value_from_table()`** — Instead of querying SQLite, look up in the in-memory cache. If cache is stale, refresh from SQLite (but only once per TTL period, not per call).

3. **Add `config_cache_refresh()`** — Loads all rows from `SELECT key, value FROM config` into the hash map. Called:
   - Once at startup after config table is populated
   - Every 5 seconds from the existing `refresh_hot_config_if_needed()` timer
   - Immediately after `process_admin_event_in_config()` modifies config

4. **Thread safety:** The cache is read from lws-main and event-worker threads. Use a read-write lock or double-buffer pattern. Since config changes are rare, a simple mutex with short hold time is fine.

5. **Remove the existing `hot_config_cache_t`** in websockets.c — it becomes redundant since the global config cache serves the same purpose with broader coverage.

This single change eliminates SQLite calls from:
- Per-connection: `trust_proxy_headers`, `nip42_auth_required_*` (websockets.c)
- Per-REQ: `expiration_enabled`, `expiration_filter` (main.c)
- Per-EVENT: `admin_pubkey`, `relay_pubkey`, `nip42_auth_enabled`, `pow_*` (request_validator.c, nip013.c)
- Per-broadcast: `expiration_enabled`, `expiration_filter` (subscriptions.c)
- Per-NIP-11: 25+ config values (nip011.c)
- Per-NIP-42: `relay_url`, `relay_port` (nip042.c)
- Per-NIP-40: 5 expiration config values (nip040.c)
- Per-IP-ban: `idle_ban_*`, `auth_fail_*` config values (ip_ban.c)
- Per-admin-DM: `relay_pubkey`, `admin_pubkey`, `wot_enabled` (dm_admin.c)
- Per-monitoring: `relay_pubkey`, throttle values (api.c)

### Fix 2: Move Duplicate Check to Event Worker Thread

**Impact: High** — eliminates the most frequent per-EVENT SQLite call from lws-main

Currently `event_id_exists_in_db()` runs on lws-main before async submission. Move it into the `async_event_worker_main()` function, which already has its own SQLite connection.

**Changes:**
- Remove `event_id_exists_in_db()` calls from both EVENT paths in websockets.c (lines 1324 and 2134)
- Add the duplicate check at the start of `async_event_worker_main()` before `nostr_validate_unified_request()`
- If duplicate found, set `completion->success = 1` and `completion->error_message = "duplicate: already have this event"` and skip crypto
- The completion handler already sends OK responses, so this works seamlessly

**Trade-off:** Without the early dedup check on lws-main, duplicate events will be submitted to the event worker queue and consume a queue slot + thread wakeup before being detected. This is acceptable because:
- The worker thread dedup check is still fast (B-tree index lookup)
- It avoids the much larger cost of running SQLite on lws-main for every event
- Queue depth is 4096, so even under heavy duplicate traffic this won't overflow

### Fix 3: Make COUNT Queries Async

**Impact: Medium** — eliminates blocking wait on lws-main for NIP-45 COUNT

Currently `handle_count_message()` calls `thread_pool_execute_count_sync()` which blocks lws-main waiting for the db-read worker. Convert to the same async pattern used for REQ queries.

**Changes:**
- Create `count_async_state_t` similar to `req_async_state_t`
- Create `count_async_completion_t` with count result
- Add `submit_count_query_async()` that uses `thread_pool_submit_read()` with a callback
- The callback pushes to a count completion queue
- Add `process_count_async_completions()` to drain the queue on lws-main and send COUNT responses
- Call it from the main event loop alongside `process_req_async_completions()`

### Fix 4: Cache Auth Rules Check Result

**Impact: Low-Medium** — eliminates per-REQ SQLite when WoT is enabled

`check_database_auth_rules()` at websockets.c:1706 runs a SQLite query for every REQ when `wot_enabled == 2`. Cache the result per-session.

**Changes:**
- Add `wot_checked` and `wot_allowed` fields to `per_session_data`
- On first REQ, call `check_database_auth_rules()` and cache the result
- On subsequent REQs from the same session, use the cached value
- Reset cache when auth state changes (e.g., after NIP-42 AUTH)

### Fix 5: Move Special-Kind EVENT Store to Event Worker

**Impact: Low** — special kinds are rare but currently do full sync store on lws-main

Kinds 14, 1059, and 23456 are currently excluded from async processing because they have special handling. However, the `store_event()` call within those paths still blocks lws-main.

**Changes:**
- For kind 1059 and kind 14: after special processing completes, submit the store to the event worker instead of calling `store_event()` synchronously
- For kind 23456: admin events are not stored (processed by admin API), so no change needed
- This requires the special processing (decryption, admin check) to still happen on lws-main (it needs `pss` context), but the final DB write goes async

### Fix 6: Reduce Reader Threads from 4 to 1

**Impact: Resource savings** — 3 fewer threads, 3 fewer SQLite connections

**Changes:**
- Change default `reader_threads` in `thread_pool_init()` from 4 to 1
- Or make it configurable via config with default 1
- The single `db-read` thread handles all REQ and COUNT queries sequentially
- If future profiling shows the read thread becoming a bottleneck, increase back to 2+

### Fix 7: Move Subscription Logging Off Main Thread

**Impact: Medium** — eliminates per-REQ INSERT, per-CLOSE UPDATE, per-broadcast UPDATE

The subscription lifecycle functions in subscriptions.c do synchronous SQLite writes on lws-main:
- `db_log_subscription_created()` — INSERT on every REQ
- `db_log_subscription_closed()` — INSERT + UPDATE on every CLOSE
- `db_log_subscription_disconnected()` — UPDATE + INSERT on every disconnect
- `db_update_subscription_events_sent()` — UPDATE on every broadcast match

**Changes:**
- Submit these as fire-and-forget write jobs to the db-write thread via `thread_pool_submit_write()`
- Create a new job type `THREAD_POOL_JOB_SUBSCRIPTION_LOG` or simply use a generic callback job
- The main thread doesn't need to wait for these to complete — they're purely observational logging

### Fix 8: Move Periodic Tasks Off Main Thread

**Impact: Low** — these run every 60s but can still cause latency spikes

- `generate_and_post_status_event()`: Submit as an async job to the event worker
- `ip_ban_cleanup()`/`ip_ban_log_stats()`/`ip_ban_save()`: These do SQLite reads/writes. Move to a periodic job on the db-write thread
- Config cache refresh (Fix 1) replaces the old `refresh_hot_config_if_needed()` SQLite queries

### Fix 9: Cache NIP-11 Response

**Impact: Low-Medium** — eliminates 25+ SQLite SELECTs per NIP-11 HTTP request

`generate_relay_info_json()` in nip011.c calls `get_config_value()` 15 times and `get_config_int()`/`get_config_bool()` 10 times. After Fix 1 (global config cache), these become in-memory lookups. But we can go further:

**Changes:**
- Cache the serialized NIP-11 JSON response string with a TTL (e.g., 60 seconds)
- On NIP-11 request, return the cached string directly without rebuilding
- Invalidate on admin config change events

## Implementation Order

| # | Fix | Risk | Complexity |
|---|-----|------|------------|
| 1 | Global config cache (replaces all get_config_* SQLite) | Low | Medium |
| 2 | Move dedup check to event worker | Low | Low |
| 3 | Async COUNT queries | Low | Medium |
| 4 | Cache WoT auth rules per-session | Very Low | Low |
| 5 | Async store for special kinds | Medium | Medium |
| 6 | Reduce reader threads to 1 | Very Low | Trivial |
| 7 | Move subscription logging off main thread | Low | Low-Medium |
| 8 | Move periodic tasks off main thread | Low | Medium |
| 9 | Cache NIP-11 response | Very Low | Low |

## Expected Impact

After all 9 fixes, `lws-main` should have **zero direct SQLite calls**. Its work becomes:
- JSON parse incoming messages (~fast)
- In-memory subscription matching (~fast)
- In-memory config lookups (~fast, hash map)
- Message formatting and queuing (~fast)
- `lws_service()` I/O (~fast)
- Draining completion queues (~fast)

### Impact by fix

| Fix | SQLite calls eliminated | Frequency |
|-----|------------------------|-----------|
| Fix 1: Global config cache | ~50+ `get_config_*` call sites across 8 files | Per-event, per-connection, per-REQ, per-broadcast, per-NIP-11 |
| Fix 2: Dedup to event worker | 2 `event_id_exists_in_db()` calls | Per-EVENT |
| Fix 3: Async COUNT | 1 `thread_pool_execute_count_sync()` | Per-COUNT message |
| Fix 4: WoT cache per-session | 1 `check_database_auth_rules()` | Per-REQ when WoT enabled |
| Fix 5: Async special-kind store | ~6 `store_event()` calls | Per special-kind EVENT |
| Fix 6: Reduce readers | N/A (resource savings) | N/A |
| Fix 7: Async subscription logging | 4 `db_log_*`/`db_update_*` calls | Per-REQ, per-CLOSE, per-broadcast |
| Fix 8: Async periodic tasks | `generate_stats_json()`, `ip_ban_*` | Every 60s |
| Fix 9: Cache NIP-11 | 25+ config lookups per request | Per NIP-11 HTTP request |

Estimated main-thread CPU reduction: from **54% to ~5-10%** (mostly WebSocket I/O, subscription matching, and JSON parsing).

## Thread Model After Changes

```
lws-main     — WebSocket I/O, JSON parse, subscription match, message queue
               NO SQLite. Pure compute + I/O.

db-read      — All SELECT queries: REQ results, COUNT results, config refresh,
               auth rule checks, event dedup checks via completion queue

db-write     — All INSERT/UPDATE: event storage, tag inserts, IP ban persistence,
               status event generation

event-worker — EVENT validation: signature verification, then submits store to
               db-write via completion queue, results flow back to lws-main
```

```mermaid
flowchart LR
    CLIENT[Nostr Clients] --> LWS[lws-main<br/>WebSocket I/O<br/>JSON parse<br/>Sub matching<br/>NO SQLite]
    
    LWS -->|EVENT submit| EW[event-worker<br/>Sig verify<br/>Dedup check]
    EW -->|store job| DW[db-write<br/>INSERT events<br/>INSERT tags<br/>IP ban save]
    EW -->|completion| LWS
    DW -->|completion| LWS
    
    LWS -->|REQ/COUNT submit| DR[db-read<br/>SELECT queries<br/>Config refresh<br/>Auth checks]
    DR -->|completion| LWS
    
    LWS -->|OK/EVENT/EOSE| CLIENT
```