Files

T

Laan Tungir ce9ae5a488 v2.1.5 - Offload duplicate and COUNT paths, add config/NIP-11 caching, and reduce DB reader threads

2026-04-01 16:15:46 -04:00

20 KiB

Raw Permalink Blame History

Plan: Remove All SQLite from lws-main Thread

Problem Statement

After implementing the 4 offload fixes (v2.1.4), lws-main still consumes 53.87% avg CPU (down from 68.71%). Perf profiling shows 100% of the top symbols are SQLite — B-tree traversal, page cache, VFS reads. The worker threads (db-read, db-write, event-worker) are nearly idle at 0.0–0.1%.

The goal is to eliminate all synchronous SQLite access from lws-main so it only does: JSON parsing, WebSocket I/O, subscription matching (in-memory), and message queuing.

Thread Architecture Change

Reduce from 7 threads to 4:

Current:  lws-main, db-read-1, db-read-2, db-read-3, db-read-4, db-write, event-worker
Proposed: lws-main, db-read, db-write, event-worker

The 4 reader threads are overkill — they average 0.07–0.10% CPU each. A single db-read thread is sufficient for the current workload. This can be made configurable later if needed.

Remaining SQLite Calls on lws-main

Every synchronous SQLite call that currently runs on lws-main:

Hot Path — Per-Event (every incoming EVENT message)

Call	Location	Frequency	Cost
`event_id_exists_in_db()`	websockets.c:1324, 2134	Every EVENT	~10-50us per B-tree lookup
`get_config_value("relay_pubkey")`	websockets.c:2050	Every kind-14 EVENT	SQLite SELECT
`get_config_value("admin_pubkey")`	websockets.c:2073	Every kind-23456 EVENT	SQLite SELECT

Hot Path — Per-Connection (every new WebSocket connection)

Call	Location	Frequency	Cost
`get_config_bool("trust_proxy_headers")`	websockets.c:1071	Every connection	SQLite SELECT
`get_config_bool("nip42_auth_required_events")`	websockets.c:1134	Every connection	SQLite SELECT
`get_config_bool("nip42_auth_required_subscriptions")`	websockets.c:1135	Every connection	SQLite SELECT

Hot Path — Per-REQ (every subscription request)

Call	Location	Frequency	Cost
`get_config_bool("expiration_enabled")`	main.c:1424	Every REQ	SQLite SELECT
`get_config_bool("expiration_filter")`	main.c:1425	Every REQ	SQLite SELECT
`check_database_auth_rules()`	websockets.c:1706	Every REQ when WoT enabled	SQLite SELECT

Medium Path — Per-COUNT (NIP-45)

Call	Location	Frequency	Cost
`thread_pool_execute_count_sync()`	websockets.c:3689	Every COUNT message	Blocks main thread waiting for db-read

Low Path — Periodic (every 60s or on timer)

Call	Location	Frequency	Cost
`refresh_hot_config_if_needed()`	websockets.c:228-247	Every 5s	8 SQLite SELECTs
`generate_and_post_status_event()`	websockets.c:3213	Every N hours	store_event + broadcast
`ip_ban_cleanup/log_stats`	websockets.c:3004-3005	Every 60s	SQLite reads/writes

Per-EVENT — Validator Path (called from event-worker AND sync path)

Call	Location	Frequency	Cost
`get_config_value("admin_pubkey")`	request_validator.c:308	Every kind-23456 EVENT	SQLite SELECT
`get_config_value("nip42_auth_enabled")`	request_validator.c:326	Every EVENT	SQLite SELECT
`get_config_bool("pow_enabled")`	request_validator.c:370	Every EVENT	SQLite SELECT
`get_config_int("pow_min_difficulty")`	request_validator.c:371	Every EVENT	SQLite SELECT
`get_config_int("pow_validation_flags")`	request_validator.c:372	Every EVENT	SQLite SELECT
`get_config_int("nip40_expiration_grace_period")`	request_validator.c:432	Every EVENT with expiration	SQLite SELECT
`get_config_value("auth_enabled")`	request_validator.c:212	Every EVENT (reload_auth_config)	SQLite SELECT
`get_config_value("auth_rules_enabled")`	request_validator.c:220	Every EVENT (reload_auth_config)	SQLite SELECT
`db_count_active_whitelist_rules()`	request_validator.c:536	Every 5s (cached)	Opens separate SQLite connection

Note: These run on the event-worker thread for async-eligible events, but on lws-main for special kinds (14, 1059, 23456) that go through the sync validation path. They also run on lws-main for the nostr_validate_unified_request() call in the sync fallback.

Per-EVENT — NIP-13 PoW Validation

Call	Location	Frequency	Cost
`get_config_bool("pow_enabled")`	nip013.c:27	Every EVENT	SQLite SELECT
`get_config_int("pow_min_difficulty")`	nip013.c:28	Every EVENT	SQLite SELECT
`get_config_value("pow_mode")`	nip013.c:29	Every EVENT	SQLite SELECT

Per-Broadcast — NIP-40 Expiration Check

Call	Location	Frequency	Cost
`get_config_bool("expiration_enabled")`	subscriptions.c:764	Every broadcast	SQLite SELECT
`get_config_bool("expiration_filter")`	subscriptions.c:765	Every broadcast	SQLite SELECT

Per-NIP-11 Request (HTTP, not WebSocket)

Call	Location	Frequency	Cost
15+ `get_config_value()` calls	nip011.c:96-110	Every NIP-11 HTTP request	15 SQLite SELECTs
10+ `get_config_int()`/`get_config_bool()` calls	nip011.c:113-123	Every NIP-11 HTTP request	10 SQLite SELECTs

Per-NIP-42 Auth Event

Call	Location	Frequency	Cost
`get_config_value("relay_url")`	nip042.c:99	Every AUTH event	SQLite SELECT
`get_config_int("relay_port")`	nip042.c:103	Every AUTH event (fallback)	SQLite SELECT

Per-NIP-40 Expiration Check (EVENT submission)

Call	Location	Frequency	Cost
5 `get_config_bool()`/`get_config_int()` calls	nip040.c:37-41	Every EVENT with expiration tag	5 SQLite SELECTs

Subscription Lifecycle (per sub create/close/disconnect)

Call	Location	Frequency	Cost
`db_log_subscription_created()`	subscriptions.c:1043	Every REQ	SQLite INSERT
`db_log_subscription_closed()`	subscriptions.c:1053	Every CLOSE	SQLite INSERT + UPDATE
`db_log_subscription_disconnected()`	subscriptions.c:1060	Every disconnect	SQLite UPDATE + INSERT
`db_update_subscription_events_sent()`	subscriptions.c:1088	Every broadcast match	SQLite UPDATE
`db_cleanup_orphaned_subscriptions()`	subscriptions.c:1100	Startup only	SQLite DELETE

IP Ban System (per-connection and periodic)

Call	Location	Frequency	Cost
`get_config_value("idle_ban_whitelist")`	ip_ban.c:254	Every ban check	SQLite SELECT
`get_config_bool("auth_fail_ban_enabled")`	ip_ban.c:301, 343	Every ban check + failure	SQLite SELECT
`get_config_bool("idle_ban_enabled")`	ip_ban.c:307, 394	Every ban check + idle failure	SQLite SELECT
6+ `get_config_int()` calls	ip_ban.c:345-399, 464-465	Every failure recording	SQLite SELECTs
`db_prepare()`/`db_step_stmt()` for ip_bans	ip_ban.c:141-238	Load at startup + periodic save	SQLite reads/writes

Special Kind Sync Path (kinds 14, 1059, 23456)

Call	Location	Frequency	Cost
`store_event()`	websockets.c:1548-1629	Every special-kind EVENT	Full sync store
`process_admin_event_in_config()`	websockets.c:1468	Every kind-23456	Config DB writes
`process_nip17_admin_message()`	websockets.c:1523	Every kind-1059 when enabled	Decrypt + DB
`process_dm_stats_command()`	websockets.c:3240	Admin DM stats	Decrypt + query + encrypt
`generate_stats_json()`	api.c:1167 (via dm_admin.c:805)	Admin stats command	Multiple SQLite queries

DM Admin Path (called from lws-main for kind 1059/14)

Call	Location	Frequency	Cost
`get_config_value("relay_pubkey")`	dm_admin.c:358, 451	Every admin DM	SQLite SELECT
`get_config_value("admin_pubkey")`	dm_admin.c:504	Every admin DM	SQLite SELECT
`get_config_int("nip59_timestamp_max_delay_sec")`	dm_admin.c:373	Every admin DM	SQLite SELECT
`get_config_int("wot_enabled")`	dm_admin.c:613, 640, 660	WoT commands	SQLite SELECT

API/Monitoring (called from lws-main periodic timer)

Call	Location	Frequency	Cost
`generate_stats_json()`	api.c:1167	Status post + admin stats	~10 SQLite queries
`query_time_based_statistics()`	api.c:102	Monitoring events	SQLite queries
`query_subscription_details()`	api.c:197	Monitoring events	SQLite queries
`get_config_value("relay_pubkey")`	api.c:419, 821, 1291	Every monitoring event	SQLite SELECT
`get_config_int("kind_1_status_posts_hours")`	api.c:547	Status post check	SQLite SELECT
`get_config_int("kind_24567_reporting_throttle_sec")`	api.c:60	Monitoring throttle	SQLite SELECT

Implementation Plan

Fix 1: Create Global Config Cache — Eliminate ALL get_config_* SQLite Calls

Impact: Very High — eliminates 50+ distinct get_config_value/int/bool call sites across 8 source files

The audit found get_config_value/int/bool calls in: websockets.c, main.c, subscriptions.c, request_validator.c, nip011.c, nip013.c, nip040.c, nip042.c, ip_ban.c, dm_admin.c, api.c, config.c. Every single one does a SELECT value FROM config WHERE key=? SQLite query.

Approach: Replace get_config_value_from_table() with an in-memory hash map that is loaded once at startup and refreshed periodically (every 5s) or on admin config change events.

Changes:

New: config_cache in config.c — A simple hash map or fixed array of key-value pairs loaded from the config table:

typedef struct {
    char key[64];
    char value[512];
} config_cache_entry_t;

static config_cache_entry_t g_config_cache[256];
static int g_config_cache_count = 0;
static time_t g_config_cache_last_refresh = 0;
static pthread_mutex_t g_config_cache_mutex;
#define CONFIG_CACHE_TTL_SEC 5

Modify get_config_value_from_table() — Instead of querying SQLite, look up in the in-memory cache. If cache is stale, refresh from SQLite (but only once per TTL period, not per call).
Add config_cache_refresh() — Loads all rows from SELECT key, value FROM config into the hash map. Called:
- Once at startup after config table is populated
- Every 5 seconds from the existing refresh_hot_config_if_needed() timer
- Immediately after process_admin_event_in_config() modifies config
Thread safety: The cache is read from lws-main and event-worker threads. Use a read-write lock or double-buffer pattern. Since config changes are rare, a simple mutex with short hold time is fine.
Remove the existing hot_config_cache_t in websockets.c — it becomes redundant since the global config cache serves the same purpose with broader coverage.

This single change eliminates SQLite calls from:

Per-connection: trust_proxy_headers, nip42_auth_required_* (websockets.c)
Per-REQ: expiration_enabled, expiration_filter (main.c)
Per-EVENT: admin_pubkey, relay_pubkey, nip42_auth_enabled, pow_* (request_validator.c, nip013.c)
Per-broadcast: expiration_enabled, expiration_filter (subscriptions.c)
Per-NIP-11: 25+ config values (nip011.c)
Per-NIP-42: relay_url, relay_port (nip042.c)
Per-NIP-40: 5 expiration config values (nip040.c)
Per-IP-ban: idle_ban_*, auth_fail_* config values (ip_ban.c)
Per-admin-DM: relay_pubkey, admin_pubkey, wot_enabled (dm_admin.c)
Per-monitoring: relay_pubkey, throttle values (api.c)

Fix 2: Move Duplicate Check to Event Worker Thread

Impact: High — eliminates the most frequent per-EVENT SQLite call from lws-main

Currently event_id_exists_in_db() runs on lws-main before async submission. Move it into the async_event_worker_main() function, which already has its own SQLite connection.

Changes:

Remove event_id_exists_in_db() calls from both EVENT paths in websockets.c (lines 1324 and 2134)
Add the duplicate check at the start of async_event_worker_main() before nostr_validate_unified_request()
If duplicate found, set completion->success = 1 and completion->error_message = "duplicate: already have this event" and skip crypto
The completion handler already sends OK responses, so this works seamlessly

Trade-off: Without the early dedup check on lws-main, duplicate events will be submitted to the event worker queue and consume a queue slot + thread wakeup before being detected. This is acceptable because:

The worker thread dedup check is still fast (B-tree index lookup)
It avoids the much larger cost of running SQLite on lws-main for every event
Queue depth is 4096, so even under heavy duplicate traffic this won't overflow

Fix 3: Make COUNT Queries Async

Impact: Medium — eliminates blocking wait on lws-main for NIP-45 COUNT

Currently handle_count_message() calls thread_pool_execute_count_sync() which blocks lws-main waiting for the db-read worker. Convert to the same async pattern used for REQ queries.

Changes:

Create count_async_state_t similar to req_async_state_t
Create count_async_completion_t with count result
Add submit_count_query_async() that uses thread_pool_submit_read() with a callback
The callback pushes to a count completion queue
Add process_count_async_completions() to drain the queue on lws-main and send COUNT responses
Call it from the main event loop alongside process_req_async_completions()

Fix 4: Cache Auth Rules Check Result

Impact: Low-Medium — eliminates per-REQ SQLite when WoT is enabled

check_database_auth_rules() at websockets.c:1706 runs a SQLite query for every REQ when wot_enabled == 2. Cache the result per-session.

Changes:

Add wot_checked and wot_allowed fields to per_session_data
On first REQ, call check_database_auth_rules() and cache the result
On subsequent REQs from the same session, use the cached value
Reset cache when auth state changes (e.g., after NIP-42 AUTH)

Fix 5: Move Special-Kind EVENT Store to Event Worker

Impact: Low — special kinds are rare but currently do full sync store on lws-main

Kinds 14, 1059, and 23456 are currently excluded from async processing because they have special handling. However, the store_event() call within those paths still blocks lws-main.

Changes:

For kind 1059 and kind 14: after special processing completes, submit the store to the event worker instead of calling store_event() synchronously
For kind 23456: admin events are not stored (processed by admin API), so no change needed
This requires the special processing (decryption, admin check) to still happen on lws-main (it needs pss context), but the final DB write goes async

Fix 6: Reduce Reader Threads from 4 to 1

Impact: Resource savings — 3 fewer threads, 3 fewer SQLite connections

Changes:

Change default reader_threads in thread_pool_init() from 4 to 1
Or make it configurable via config with default 1
The single db-read thread handles all REQ and COUNT queries sequentially
If future profiling shows the read thread becoming a bottleneck, increase back to 2+

Fix 7: Move Subscription Logging Off Main Thread

Impact: Medium — eliminates per-REQ INSERT, per-CLOSE UPDATE, per-broadcast UPDATE

The subscription lifecycle functions in subscriptions.c do synchronous SQLite writes on lws-main:

db_log_subscription_created() — INSERT on every REQ
db_log_subscription_closed() — INSERT + UPDATE on every CLOSE
db_log_subscription_disconnected() — UPDATE + INSERT on every disconnect
db_update_subscription_events_sent() — UPDATE on every broadcast match

Changes:

Submit these as fire-and-forget write jobs to the db-write thread via thread_pool_submit_write()
Create a new job type THREAD_POOL_JOB_SUBSCRIPTION_LOG or simply use a generic callback job
The main thread doesn't need to wait for these to complete — they're purely observational logging

Fix 8: Move Periodic Tasks Off Main Thread

Impact: Low — these run every 60s but can still cause latency spikes

generate_and_post_status_event(): Submit as an async job to the event worker
ip_ban_cleanup()/ip_ban_log_stats()/ip_ban_save(): These do SQLite reads/writes. Move to a periodic job on the db-write thread
Config cache refresh (Fix 1) replaces the old refresh_hot_config_if_needed() SQLite queries

Fix 9: Cache NIP-11 Response

Impact: Low-Medium — eliminates 25+ SQLite SELECTs per NIP-11 HTTP request

generate_relay_info_json() in nip011.c calls get_config_value() 15 times and get_config_int()/get_config_bool() 10 times. After Fix 1 (global config cache), these become in-memory lookups. But we can go further:

Changes:

Cache the serialized NIP-11 JSON response string with a TTL (e.g., 60 seconds)
On NIP-11 request, return the cached string directly without rebuilding
Invalidate on admin config change events

Implementation Order

#	Fix	Risk	Complexity
1	Global config cache (replaces all get_config_* SQLite)	Low	Medium
2	Move dedup check to event worker	Low	Low
3	Async COUNT queries	Low	Medium
4	Cache WoT auth rules per-session	Very Low	Low
5	Async store for special kinds	Medium	Medium
6	Reduce reader threads to 1	Very Low	Trivial
7	Move subscription logging off main thread	Low	Low-Medium
8	Move periodic tasks off main thread	Low	Medium
9	Cache NIP-11 response	Very Low	Low

Expected Impact

After all 9 fixes, lws-main should have zero direct SQLite calls. Its work becomes:

JSON parse incoming messages (~fast)
In-memory subscription matching (~fast)
In-memory config lookups (~fast, hash map)
Message formatting and queuing (~fast)
lws_service() I/O (~fast)
Draining completion queues (~fast)

Impact by fix

Fix	SQLite calls eliminated	Frequency
Fix 1: Global config cache	~50+ `get_config_*` call sites across 8 files	Per-event, per-connection, per-REQ, per-broadcast, per-NIP-11
Fix 2: Dedup to event worker	2 `event_id_exists_in_db()` calls	Per-EVENT
Fix 3: Async COUNT	1 `thread_pool_execute_count_sync()`	Per-COUNT message
Fix 4: WoT cache per-session	1 `check_database_auth_rules()`	Per-REQ when WoT enabled
Fix 5: Async special-kind store	~6 `store_event()` calls	Per special-kind EVENT
Fix 6: Reduce readers	N/A (resource savings)	N/A
Fix 7: Async subscription logging	4 `db_log_`/`db_update_` calls	Per-REQ, per-CLOSE, per-broadcast
Fix 8: Async periodic tasks	`generate_stats_json()`, `ip_ban_*`	Every 60s
Fix 9: Cache NIP-11	25+ config lookups per request	Per NIP-11 HTTP request

Estimated main-thread CPU reduction: from 54% to ~5-10% (mostly WebSocket I/O, subscription matching, and JSON parsing).

Thread Model After Changes

lws-main     — WebSocket I/O, JSON parse, subscription match, message queue
               NO SQLite. Pure compute + I/O.

db-read      — All SELECT queries: REQ results, COUNT results, config refresh,
               auth rule checks, event dedup checks via completion queue

db-write     — All INSERT/UPDATE: event storage, tag inserts, IP ban persistence,
               status event generation

event-worker — EVENT validation: signature verification, then submits store to
               db-write via completion queue, results flow back to lws-main

flowchart LR
    CLIENT[Nostr Clients] --> LWS[lws-main<br/>WebSocket I/O<br/>JSON parse<br/>Sub matching<br/>NO SQLite]
    
    LWS -->|EVENT submit| EW[event-worker<br/>Sig verify<br/>Dedup check]
    EW -->|store job| DW[db-write<br/>INSERT events<br/>INSERT tags<br/>IP ban save]
    EW -->|completion| LWS
    DW -->|completion| LWS
    
    LWS -->|REQ/COUNT submit| DR[db-read<br/>SELECT queries<br/>Config refresh<br/>Auth checks]
    DR -->|completion| LWS
    
    LWS -->|OK/EVENT/EOSE| CLIENT

20 KiB Raw Permalink Blame History Unescape Escape

Plan: Remove All SQLite from lws-main Thread

Problem Statement

Thread Architecture Change

Remaining SQLite Calls on lws-main

Hot Path — Per-Event (every incoming EVENT message)

Hot Path — Per-Connection (every new WebSocket connection)

Hot Path — Per-REQ (every subscription request)

Medium Path — Per-COUNT (NIP-45)

Low Path — Periodic (every 60s or on timer)

Per-EVENT — Validator Path (called from event-worker AND sync path)

Per-EVENT — NIP-13 PoW Validation

Per-Broadcast — NIP-40 Expiration Check

Per-NIP-11 Request (HTTP, not WebSocket)

Per-NIP-42 Auth Event

Per-NIP-40 Expiration Check (EVENT submission)

Subscription Lifecycle (per sub create/close/disconnect)

IP Ban System (per-connection and periodic)

Special Kind Sync Path (kinds 14, 1059, 23456)

DM Admin Path (called from lws-main for kind 1059/14)

API/Monitoring (called from lws-main periodic timer)

Implementation Plan

Fix 1: Create Global Config Cache — Eliminate ALL get_config_* SQLite Calls

Fix 2: Move Duplicate Check to Event Worker Thread

Fix 3: Make COUNT Queries Async

Fix 4: Cache Auth Rules Check Result

Fix 5: Move Special-Kind EVENT Store to Event Worker

Fix 6: Reduce Reader Threads from 4 to 1

Fix 7: Move Subscription Logging Off Main Thread

Fix 8: Move Periodic Tasks Off Main Thread

Fix 9: Cache NIP-11 Response

Implementation Order

Expected Impact

Impact by fix

Thread Model After Changes

20 KiB

Raw Permalink Blame History