c-relay/plans/sqlite_off_main_thread_plan.md

Plan: Remove All SQLite from lws-main Thread

Problem Statement

After implementing the 4 offload fixes (v2.1.4), lws-main still consumes 53.87% avg CPU (down from 68.71%). Perf profiling shows 100% of the top symbols are SQLite — B-tree traversal, page cache, VFS reads. The worker threads (db-read, db-write, event-worker) are nearly idle at 0.0–0.1%.

The goal is to eliminate all synchronous SQLite access from lws-main so it only does: JSON parsing, WebSocket I/O, subscription matching (in-memory), and message queuing.

Thread Architecture Change

Reduce from 7 threads to 4:

Current:  lws-main, db-read-1, db-read-2, db-read-3, db-read-4, db-write, event-worker
Proposed: lws-main, db-read, db-write, event-worker

The 4 reader threads are overkill — they average 0.07–0.10% CPU each. A single db-read thread is sufficient for the current workload. This can be made configurable later if needed.

Remaining SQLite Calls on lws-main

Every synchronous SQLite call that currently runs on lws-main:

Hot Path — Per-Event (every incoming EVENT message)

Call	Location	Frequency	Cost
event_id_exists_in_db()	websockets.c:1324, 2134	Every EVENT	~10-50us per B-tree lookup
get_config_value("relay_pubkey")	websockets.c:2050	Every kind-14 EVENT	SQLite SELECT
get_config_value("admin_pubkey")	websockets.c:2073	Every kind-23456 EVENT	SQLite SELECT

Hot Path — Per-Connection (every new WebSocket connection)

Call	Location	Frequency	Cost
get_config_bool("trust_proxy_headers")	websockets.c:1071	Every connection	SQLite SELECT
get_config_bool("nip42_auth_required_events")	websockets.c:1134	Every connection	SQLite SELECT
get_config_bool("nip42_auth_required_subscriptions")	websockets.c:1135	Every connection	SQLite SELECT

Hot Path — Per-REQ (every subscription request)

Call	Location	Frequency	Cost
get_config_bool("expiration_enabled")	main.c:1424	Every REQ	SQLite SELECT
get_config_bool("expiration_filter")	main.c:1425	Every REQ	SQLite SELECT
check_database_auth_rules()	websockets.c:1706	Every REQ when WoT enabled	SQLite SELECT

Medium Path — Per-COUNT (NIP-45)

Call	Location	Frequency	Cost
thread_pool_execute_count_sync()	websockets.c:3689	Every COUNT message	Blocks main thread waiting for db-read

Low Path — Periodic (every 60s or on timer)

Call	Location	Frequency	Cost
refresh_hot_config_if_needed()	websockets.c:228-247	Every 5s	8 SQLite SELECTs
generate_and_post_status_event()	websockets.c:3213	Every N hours	store_event + broadcast
ip_ban_cleanup/log_stats	websockets.c:3004-3005	Every 60s	SQLite reads/writes

Per-EVENT — Validator Path (called from event-worker AND sync path)

Call	Location	Frequency	Cost
get_config_value("admin_pubkey")	request_validator.c:308	Every kind-23456 EVENT	SQLite SELECT
get_config_value("nip42_auth_enabled")	request_validator.c:326	Every EVENT	SQLite SELECT
get_config_bool("pow_enabled")	request_validator.c:370	Every EVENT	SQLite SELECT
get_config_int("pow_min_difficulty")	request_validator.c:371	Every EVENT	SQLite SELECT
get_config_int("pow_validation_flags")	request_validator.c:372	Every EVENT	SQLite SELECT
get_config_int("nip40_expiration_grace_period")	request_validator.c:432	Every EVENT with expiration	SQLite SELECT
get_config_value("auth_enabled")	request_validator.c:212	Every EVENT (reload_auth_config)	SQLite SELECT
get_config_value("auth_rules_enabled")	request_validator.c:220	Every EVENT (reload_auth_config)	SQLite SELECT
db_count_active_whitelist_rules()	request_validator.c:536	Every 5s (cached)	Opens separate SQLite connection

Note: These run on the event-worker thread for async-eligible events, but on lws-main for special kinds (14, 1059, 23456) that go through the sync validation path. They also run on lws-main for the nostr_validate_unified_request() call in the sync fallback.

Per-EVENT — NIP-13 PoW Validation

Call	Location	Frequency	Cost
get_config_bool("pow_enabled")	nip013.c:27	Every EVENT	SQLite SELECT
get_config_int("pow_min_difficulty")	nip013.c:28	Every EVENT	SQLite SELECT
get_config_value("pow_mode")	nip013.c:29	Every EVENT	SQLite SELECT

Per-Broadcast — NIP-40 Expiration Check

Call	Location	Frequency	Cost
get_config_bool("expiration_enabled")	subscriptions.c:764	Every broadcast	SQLite SELECT
get_config_bool("expiration_filter")	subscriptions.c:765	Every broadcast	SQLite SELECT

Per-NIP-11 Request (HTTP, not WebSocket)

Call	Location	Frequency	Cost
15+ get_config_value() calls	nip011.c:96-110	Every NIP-11 HTTP request	15 SQLite SELECTs
10+ get_config_int()/get_config_bool() calls	nip011.c:113-123	Every NIP-11 HTTP request	10 SQLite SELECTs

Per-NIP-42 Auth Event

Call	Location	Frequency	Cost
get_config_value("relay_url")	nip042.c:99	Every AUTH event	SQLite SELECT
get_config_int("relay_port")	nip042.c:103	Every AUTH event (fallback)	SQLite SELECT

Per-NIP-40 Expiration Check (EVENT submission)

Call	Location	Frequency	Cost
5 get_config_bool()/get_config_int() calls	nip040.c:37-41	Every EVENT with expiration tag	5 SQLite SELECTs

Subscription Lifecycle (per sub create/close/disconnect)

Call	Location	Frequency	Cost
db_log_subscription_created()	subscriptions.c:1043	Every REQ	SQLite INSERT
db_log_subscription_closed()	subscriptions.c:1053	Every CLOSE	SQLite INSERT + UPDATE
db_log_subscription_disconnected()	subscriptions.c:1060	Every disconnect	SQLite UPDATE + INSERT
db_update_subscription_events_sent()	subscriptions.c:1088	Every broadcast match	SQLite UPDATE
db_cleanup_orphaned_subscriptions()	subscriptions.c:1100	Startup only	SQLite DELETE

IP Ban System (per-connection and periodic)

Call	Location	Frequency	Cost
get_config_value("idle_ban_whitelist")	ip_ban.c:254	Every ban check	SQLite SELECT
get_config_bool("auth_fail_ban_enabled")	ip_ban.c:301, 343	Every ban check + failure	SQLite SELECT
get_config_bool("idle_ban_enabled")	ip_ban.c:307, 394	Every ban check + idle failure	SQLite SELECT
6+ get_config_int() calls	ip_ban.c:345-399, 464-465	Every failure recording	SQLite SELECTs
db_prepare()/db_step_stmt() for ip_bans	ip_ban.c:141-238	Load at startup + periodic save	SQLite reads/writes

Special Kind Sync Path (kinds 14, 1059, 23456)

Call	Location	Frequency	Cost
store_event()	websockets.c:1548-1629	Every special-kind EVENT	Full sync store
process_admin_event_in_config()	websockets.c:1468	Every kind-23456	Config DB writes
process_nip17_admin_message()	websockets.c:1523	Every kind-1059 when enabled	Decrypt + DB
process_dm_stats_command()	websockets.c:3240	Admin DM stats	Decrypt + query + encrypt
generate_stats_json()	api.c:1167 (via dm_admin.c:805)	Admin stats command	Multiple SQLite queries

DM Admin Path (called from lws-main for kind 1059/14)

Call	Location	Frequency	Cost
get_config_value("relay_pubkey")	dm_admin.c:358, 451	Every admin DM	SQLite SELECT
get_config_value("admin_pubkey")	dm_admin.c:504	Every admin DM	SQLite SELECT
get_config_int("nip59_timestamp_max_delay_sec")	dm_admin.c:373	Every admin DM	SQLite SELECT
get_config_int("wot_enabled")	dm_admin.c:613, 640, 660	WoT commands	SQLite SELECT

API/Monitoring (called from lws-main periodic timer)

Call	Location	Frequency	Cost
generate_stats_json()	api.c:1167	Status post + admin stats	~10 SQLite queries
query_time_based_statistics()	api.c:102	Monitoring events	SQLite queries
query_subscription_details()	api.c:197	Monitoring events	SQLite queries
get_config_value("relay_pubkey")	api.c:419, 821, 1291	Every monitoring event	SQLite SELECT
get_config_int("kind_1_status_posts_hours")	api.c:547	Status post check	SQLite SELECT
get_config_int("kind_24567_reporting_throttle_sec")	api.c:60	Monitoring throttle	SQLite SELECT

Implementation Plan

Fix 1: Create Global Config Cache — Eliminate ALL get_config_* SQLite Calls

Impact: Very High — eliminates 50+ distinct get_config_value/int/bool call sites across 8 source files

The audit found get_config_value/int/bool calls in: websockets.c, main.c, subscriptions.c, request_validator.c, nip011.c, nip013.c, nip040.c, nip042.c, ip_ban.c, dm_admin.c, api.c, config.c. Every single one does a SELECT value FROM config WHERE key=? SQLite query.

Approach: Replace get_config_value_from_table() with an in-memory hash map that is loaded once at startup and refreshed periodically (every 5s) or on admin config change events.

Changes:

1. New: config_cache in config.c — A simple hash map or fixed array of key-value pairs loaded from the config table:

   typedef struct {
       char key[64];
       char value[512];
   } config_cache_entry_t;
   
   static config_cache_entry_t g_config_cache[256];
   static int g_config_cache_count = 0;
   static time_t g_config_cache_last_refresh = 0;
   static pthread_mutex_t g_config_cache_mutex;
   #define CONFIG_CACHE_TTL_SEC 5
   

2. Modify get_config_value_from_table() — Instead of querying SQLite, look up in the in-memory cache. If cache is stale, refresh from SQLite (but only once per TTL period, not per call).

3. Add config_cache_refresh() — Loads all rows from SELECT key, value FROM config into the hash map. Called:

   • Once at startup after config table is populated
   • Every 5 seconds from the existing refresh_hot_config_if_needed() timer
   • Immediately after process_admin_event_in_config() modifies config

4. Thread safety: The cache is read from lws-main and event-worker threads. Use a read-write lock or double-buffer pattern. Since config changes are rare, a simple mutex with short hold time is fine.

5. Remove the existing hot_config_cache_t in websockets.c — it becomes redundant since the global config cache serves the same purpose with broader coverage.

This single change eliminates SQLite calls from:

• Per-connection: trust_proxy_headers, nip42_auth_required_* (websockets.c)
• Per-REQ: expiration_enabled, expiration_filter (main.c)
• Per-EVENT: admin_pubkey, relay_pubkey, nip42_auth_enabled, pow_* (request_validator.c, nip013.c)
• Per-broadcast: expiration_enabled, expiration_filter (subscriptions.c)
• Per-NIP-11: 25+ config values (nip011.c)
• Per-NIP-42: relay_url, relay_port (nip042.c)
• Per-NIP-40: 5 expiration config values (nip040.c)
• Per-IP-ban: idle_ban_*, auth_fail_* config values (ip_ban.c)
• Per-admin-DM: relay_pubkey, admin_pubkey, wot_enabled (dm_admin.c)
• Per-monitoring: relay_pubkey, throttle values (api.c)

Fix 2: Move Duplicate Check to Event Worker Thread

Impact: High — eliminates the most frequent per-EVENT SQLite call from lws-main

Currently event_id_exists_in_db() runs on lws-main before async submission. Move it into the async_event_worker_main() function, which already has its own SQLite connection.

Changes:

• Remove event_id_exists_in_db() calls from both EVENT paths in websockets.c (lines 1324 and 2134)
• Add the duplicate check at the start of async_event_worker_main() before nostr_validate_unified_request()
• If duplicate found, set completion->success = 1 and completion->error_message = "duplicate: already have this event" and skip crypto
• The completion handler already sends OK responses, so this works seamlessly

Trade-off: Without the early dedup check on lws-main, duplicate events will be submitted to the event worker queue and consume a queue slot + thread wakeup before being detected. This is acceptable because:

• The worker thread dedup check is still fast (B-tree index lookup)
• It avoids the much larger cost of running SQLite on lws-main for every event
• Queue depth is 4096, so even under heavy duplicate traffic this won't overflow

Fix 3: Make COUNT Queries Async

Impact: Medium — eliminates blocking wait on lws-main for NIP-45 COUNT

Currently handle_count_message() calls thread_pool_execute_count_sync() which blocks lws-main waiting for the db-read worker. Convert to the same async pattern used for REQ queries.

Changes:

• Create count_async_state_t similar to req_async_state_t
• Create count_async_completion_t with count result
• Add submit_count_query_async() that uses thread_pool_submit_read() with a callback
• The callback pushes to a count completion queue
• Add process_count_async_completions() to drain the queue on lws-main and send COUNT responses
• Call it from the main event loop alongside process_req_async_completions()

Fix 4: Cache Auth Rules Check Result

Impact: Low-Medium — eliminates per-REQ SQLite when WoT is enabled

check_database_auth_rules() at websockets.c:1706 runs a SQLite query for every REQ when wot_enabled == 2. Cache the result per-session.

Changes:

• Add wot_checked and wot_allowed fields to per_session_data
• On first REQ, call check_database_auth_rules() and cache the result
• On subsequent REQs from the same session, use the cached value
• Reset cache when auth state changes (e.g., after NIP-42 AUTH)

Fix 5: Move Special-Kind EVENT Store to Event Worker

Impact: Low — special kinds are rare but currently do full sync store on lws-main

Kinds 14, 1059, and 23456 are currently excluded from async processing because they have special handling. However, the store_event() call within those paths still blocks lws-main.

Changes:

• For kind 1059 and kind 14: after special processing completes, submit the store to the event worker instead of calling store_event() synchronously
• For kind 23456: admin events are not stored (processed by admin API), so no change needed
• This requires the special processing (decryption, admin check) to still happen on lws-main (it needs pss context), but the final DB write goes async

Fix 6: Reduce Reader Threads from 4 to 1

Impact: Resource savings — 3 fewer threads, 3 fewer SQLite connections

Changes:

• Change default reader_threads in thread_pool_init() from 4 to 1
• Or make it configurable via config with default 1
• The single db-read thread handles all REQ and COUNT queries sequentially
• If future profiling shows the read thread becoming a bottleneck, increase back to 2+

Fix 7: Move Subscription Logging Off Main Thread

Impact: Medium — eliminates per-REQ INSERT, per-CLOSE UPDATE, per-broadcast UPDATE

The subscription lifecycle functions in subscriptions.c do synchronous SQLite writes on lws-main:

• db_log_subscription_created() — INSERT on every REQ
• db_log_subscription_closed() — INSERT + UPDATE on every CLOSE
• db_log_subscription_disconnected() — UPDATE + INSERT on every disconnect
• db_update_subscription_events_sent() — UPDATE on every broadcast match

Changes:

• Submit these as fire-and-forget write jobs to the db-write thread via thread_pool_submit_write()
• Create a new job type THREAD_POOL_JOB_SUBSCRIPTION_LOG or simply use a generic callback job
• The main thread doesn't need to wait for these to complete — they're purely observational logging

Fix 8: Move Periodic Tasks Off Main Thread

Impact: Low — these run every 60s but can still cause latency spikes

• generate_and_post_status_event(): Submit as an async job to the event worker
• ip_ban_cleanup()/ip_ban_log_stats()/ip_ban_save(): These do SQLite reads/writes. Move to a periodic job on the db-write thread
• Config cache refresh (Fix 1) replaces the old refresh_hot_config_if_needed() SQLite queries

Fix 9: Cache NIP-11 Response

Impact: Low-Medium — eliminates 25+ SQLite SELECTs per NIP-11 HTTP request

generate_relay_info_json() in nip011.c calls get_config_value() 15 times and get_config_int()/get_config_bool() 10 times. After Fix 1 (global config cache), these become in-memory lookups. But we can go further:

Changes:

• Cache the serialized NIP-11 JSON response string with a TTL (e.g., 60 seconds)
• On NIP-11 request, return the cached string directly without rebuilding
• Invalidate on admin config change events

Implementation Order

#	Fix	Risk	Complexity
1	Global config cache (replaces all get_config_* SQLite)	Low	Medium
2	Move dedup check to event worker	Low	Low
3	Async COUNT queries	Low	Medium
4	Cache WoT auth rules per-session	Very Low	Low
5	Async store for special kinds	Medium	Medium
6	Reduce reader threads to 1	Very Low	Trivial
7	Move subscription logging off main thread	Low	Low-Medium
8	Move periodic tasks off main thread	Low	Medium
9	Cache NIP-11 response	Very Low	Low

Expected Impact

After all 9 fixes, lws-main should have zero direct SQLite calls. Its work becomes:

• JSON parse incoming messages (~fast)
• In-memory subscription matching (~fast)
• In-memory config lookups (~fast, hash map)
• Message formatting and queuing (~fast)
• lws_service() I/O (~fast)
• Draining completion queues (~fast)

Impact by fix

Fix	SQLite calls eliminated	Frequency
Fix 1: Global config cache	~50+ get_config_* call sites across 8 files	Per-event, per-connection, per-REQ, per-broadcast, per-NIP-11
Fix 2: Dedup to event worker	2 event_id_exists_in_db() calls	Per-EVENT
Fix 3: Async COUNT	1 thread_pool_execute_count_sync()	Per-COUNT message
Fix 4: WoT cache per-session	1 check_database_auth_rules()	Per-REQ when WoT enabled
Fix 5: Async special-kind store	~6 store_event() calls	Per special-kind EVENT
Fix 6: Reduce readers	N/A (resource savings)	N/A
Fix 7: Async subscription logging	4 db_log_*/db_update_* calls	Per-REQ, per-CLOSE, per-broadcast
Fix 8: Async periodic tasks	generate_stats_json(), ip_ban_*	Every 60s
Fix 9: Cache NIP-11	25+ config lookups per request	Per NIP-11 HTTP request

Estimated main-thread CPU reduction: from 54% to ~5-10% (mostly WebSocket I/O, subscription matching, and JSON parsing).

Thread Model After Changes

lws-main     — WebSocket I/O, JSON parse, subscription match, message queue
               NO SQLite. Pure compute + I/O.

db-read      — All SELECT queries: REQ results, COUNT results, config refresh,
               auth rule checks, event dedup checks via completion queue

db-write     — All INSERT/UPDATE: event storage, tag inserts, IP ban persistence,
               status event generation

event-worker — EVENT validation: signature verification, then submits store to
               db-write via completion queue, results flow back to lws-main

flowchart LR
    CLIENT[Nostr Clients] --> LWS[lws-main<br/>WebSocket I/O<br/>JSON parse<br/>Sub matching<br/>NO SQLite]
    
    LWS -->|EVENT submit| EW[event-worker<br/>Sig verify<br/>Dedup check]
    EW -->|store job| DW[db-write<br/>INSERT events<br/>INSERT tags<br/>IP ban save]
    EW -->|completion| LWS
    DW -->|completion| LWS
    
    LWS -->|REQ/COUNT submit| DR[db-read<br/>SELECT queries<br/>Config refresh<br/>Auth checks]
    DR -->|completion| LWS
    
    LWS -->|OK/EVENT/EOSE| CLIENT